[ { "path": ".gitignore", "content": "# use glob syntax\nsyntax: glob\n\n*~\ndebug/*\nrelease/*\nprofile/*\ndist/graphlabapi*\ndoc/doxygen/*\ndoc/doxygen_internal/*\ndeps/*\nconfigure.deps\nconfig.log\nnew_news.txt\ncxxtest/cxxtest/*.pyc\n\nextapis/java_jni/dist\nextapis/java_jni/bin\n\ndemoapps/coloring/matlab_tools/*.tsv\ndemoapps/shortest_path/matlab_tools/*.tsv\ndemoapps/pagerank/matlab_tools/*.tsv\n\ntags\n*swp\n\\.mchg\\.mchg\n.ycm_extra_conf*\n" }, { "path": "BINARY_README", "content": "\n Graphlab Binary Release\n -----------------------\n\n=======\nLicense\n=======\n\nGraphLab is free software licensed under the Apache 2.0 License. See\nlicense/LICENSE.txt for details.\n\n============\nIntroduction\n============\n\nGraphLab PowerGraph is a graph-based, high performance, distributed computation framework\nwritten in C++. \n\nGraphLab Features:\n\nUnified multicore/\ndistributed API: write once run anywhere \n\nTuned for performance: optimized C++ execution engine leverages extensive \n multi-threading and asynchronous IO \n\nScalable: Run on large cluster deployments by\n intelligently placing data and computation \n\nHDFS Integration: Access your data directly from HDFS \n\nPowerful Machine \nLearning Toolkits: Tackle challenging machine \n learning problems with ease\n\n\n========\nMac OS X\n========\nThe Mac OS X build is restricted and is compiled without HDFS support, nor\ndistributed capabilities. All toolkits therefore run entirely on one machine\nin shared memory.\n\nThese binaries have been tested to run on OS X 10.6 or greater. Though they\nmay work on 10.5 as well.\n\n=====\nLinux\n=====\nThe Linux builds are more complex due to varying glibc versions as well as the\nneed to support both OpenMPI and MPICH2. FOUR binary releases are provided\nPlease obtain the package closest matching your needs. In particular, it is\nimportant to match the MPI library you have on your cluster: \nthey are not compatible.\n\nThe binary packages are:\n - graphlab_openmpi : Supports OpenMPI 1.3 and 1.4. Supports HDFS and thus\n requires a working JVM (whether or not you use \n HDFS).\n - graphlab_mpich2 : Supports MPICH2. Supports HDFS and thus\n requires a working JVM (whether or not you use \n HDFS).\n - graphlab_openmpi_no_jvm : Supports OpenMPI 1.3 and 1.4. Does not support\n HDFS. Does not require a working JVM. Loading\n files distributed will therefore require either\n a distributed file system or a common NFS\n share.\n - graphlab_mpich2_no_jvm : Supports MPICH2. Does not support HDFS. Does not \n require a working JVM. Loading files distributed \n will therefore require either a distributed file \n system or a common NFS share. If you want to run\n to run only in shared memory, this is also the\n package to use.\n\nEach binary is paired with a matching shell script which sets up the\nexecution environment (by forcing the load of provided versions of system\nbinaries). Several environment variables are used to control the process. \n\n JAVA_HOME\n Either JAVA_HOME or JVM_SO_PATH must be set. \n This must point to the Java home directory.\n For instance: /usr/lib/jvm/java-6-openjdk\n This was tested with Oracle's implementation of Java (sun-jdk or open-jdk).\n\n JVM_SO_PATH\n Either JAVA_HOME or JVM_SO_PATH must be set. \n Optional. The script will expect to find libjvm.so in \n $JAVA_HOME/jre/lib/amd64/client/libjvm.so or\n $JAVA_HOME/jre/lib/amd64/server/libjvm.so\n If libjvm.so is not in either locations, the script will fail. In which \n case, you should set this variable to the directory containing libjvm.so.\n\n USE_SYSTEM_LIBS\n Optional. If set, the system's glibc (and other system dependencies) \n will be used instead of the provided versions. \n\n\nHDFS Capability is compiled in. If HDFS is to be used, the \n\"hadoop\" command must be available.\n\n\nAdditional Notes for OpenMPI\n----------------------------\nUnlike the MPICH2 build, the OpenMPI build is unable to static link\neverything. For now, we provide only binary releases built against \nOpenMPI 1.3.2 which should be binary compatible with 1.4. \n\n=============\nBuild Details\n=============\n\nMac Build:\nCompiled on OS X 10.8\nApple clang version 4.0 (tags/Apple/clang-421.0.57) (based on LLVM 3.1svn)\n-O3 -march=x86-64 -mtune=generic -mmacosx-version-min=10.5 \n\nLinux Build:\nTo support maximal compatibility, this was compiled on an \nold Linux distribution.\n\nCompiled on Ubuntu 9.04, GCC 4.3.3\n-O3 -march=x86-64 -mtune=generic \n\nOpenMPI: 1.3.2 (ABI Compatible with 1.4)\nMPICH2: 1.5\n\n" }, { "path": "CMakeLists.txt", "content": "project(GraphLab)\n\n# We require the most recent version of cmake and automatically\n# install the correct version when using the cmake lists\ncmake_minimum_required(VERSION 2.8)\n\n# Libraries linked via full path no longer produce linker search paths.\ncmake_policy(SET CMP0003 NEW)\n# Preprocessor definition values are now escaped automatically.\ncmake_policy(SET CMP0005 NEW)\n\n# Determine where additional GraphLab specific cmake modules are\n# defined\nset(CMAKE_MODULE_PATH ${GraphLab_SOURCE_DIR}/cmake)\n\n# TODO: what is this used by\nset(BUILD_SHARED_LIBS ON)\n\ninclude(CheckLibraryExists)\ninclude(CheckFunctionExists)\ninclude(ExternalProject)\ninclude(CheckCXXSourceCompiles)\n\n# check we are on a 64 bit machine (else fail)\nif( CMAKE_SIZEOF_VOID_P EQUAL 4 )\n message( SEND_ERROR \"GraphLab does not support 32 bit systems. Please switch to 64 bit system and try again\" )\nendif ( CMAKE_SIZEOF_VOID_P EQUAL 4 )\n\n# # Display information about cmake variables\n# include(CMakePrintSystemInformation)\n\n# set include path\ninclude_directories(\n ${GraphLab_SOURCE_DIR}/src\n ${GraphLab_SOURCE_DIR}/cxxtest\n ${GraphLab_SOURCE_DIR}/deps/local/include)\n\n# set link path\nlink_directories(${GraphLab_SOURCE_DIR}/deps/local/lib)\n\nadd_definitions(-DUSE_DYNAMIC_LOCAL_GRAPH)\n\nif(NO_OPENMP)\n set(OPENMP_C_FLAGS \"\")\n set(OPENMP_LIBRARIES \"\")\n add_definitions(-D__NO_OPENMP__)\nelse()\n set(OPENMP_C_FLAGS \"-fopenmp\")\n set(OPENMP_LIBRARIES \"gomp\")\nendif()\n\nif (APPLE)\nlink_libraries(pthread ${OPENMP_LIBRARIES})\nelse()\nlink_libraries(pthread rt ${OPENMP_LIBRARIES})\nendif()\n\nset(CPP11_FLAGS \"\")\nif(CPP11)\n message(STATUS \"C++11 Enabled\")\n set(CPP11_FLAGS \"-std=c++11 -Wno-enum-compare -Wno-conversion-null\")\nendif()\n\nif(VID32)\n message(STATUS \"Using 32bit vertex id types\")\n add_definitions(-DUSE_VID32)\nendif()\n\n\n# Shared compiler flags used by all builds (debug, profile, release)\nset(COMPILER_FLAGS \"-Wall -g ${CPP11_FLAGS} ${OPENMP_C_FLAGS}\" CACHE STRING \"common compiler options\")\n\n# Disable address space randomization for OSX lion and above\nif (APPLE)\n set(CMAKE_EXE_LINKER_FLAGS \"${CMAKE_EXE_LINKER_FLAGS} -Xlinker -no_pie\")\nendif()\n\nset(CMAKE_EXE_LINKER_FLAGS \"${CMAKE_EXE_LINKER_FLAGS} -Wl,-rpath,${GraphLab_SOURCE_DIR}/deps/local/lib\")\n\n# Set subdirectories\nsubdirs(src tests demoapps toolkits)\nif(EXPERIMENTAL)\n if (IS_DIRECTORY ${GraphLab_SOURCE_DIR}/experimental)\n subdirs(experimental)\n endif()\nendif()\n\nif (IS_DIRECTORY ${GraphLab_SOURCE_DIR}/apps)\n subdirs(apps)\nendif()\n\n# Set installation rules\ninstall(DIRECTORY src/\n DESTINATION include\n FILES_MATCHING PATTERN \"*.hpp\" PATTERN \"*.h\"\n PATTERN \".svn\" EXCLUDE\n )\ninstall(CODE \"execute_process(COMMAND ./scripts/install_deps.sh\n ${CMAKE_INSTALL_PREFIX} WORKING_DIRECTORY\n ${CMAKE_CURRENT_SOURCE_DIR})\")\n\n# set(EXTERNAL_PROJECT_FLAGS\n# \"-I${GraphLab_SOURCE_DIR}/deps/local/include -L${GraphLab_SOURCE_DIR}/deps/local/lib\")\n\n\n\n### ===================================================================\n# ## Download News update\n# if(NOT DISABLE_NEWS)\n# message(STATUS\n# \"\\n\"\n# \"===============================================================\\n\"\n# \"Downloading the news from graphlab.org: \")\n# file(DOWNLOAD \"http://graphlab.org/news.txt\"\n# ${GraphLab_SOURCE_DIR}/new_news.txt\n# TIMEOUT 3\n# STATUS download_status)\n# # Track usage v2 news url\n# file(DOWNLOAD \"http://c.statcounter.com/7396904/0/a1b29b21/1/\"\n# ${GraphLab_SOURCE_DIR}/temporary_download_counter.tmp\n# TIMEOUT 3\n# STATUS dlstatus)\n# file(REMOVE ${GraphLab_SOURCE_DIR}/temporary_download_counter.tmp)\n# file(READ ${GraphLab_SOURCE_DIR}/new_news.txt news_update)\n# message(STATUS\n# \"Finished.\\n\"\n# ${news_update}\n# \"\\n\"\n# \"===============================================================\")\n# endif()\n\n\n\n\n\n\n## ============================================================================\n## ============================================================================\n## ============================================================================\n# Build external dependencies\n\n# Build eigen =================================================================\n# TODO: Move to toolkits.\n# While eigen is not used in the core GraphLab library we found it\n# useful to have a matrix library. For now we download eigen from\n# our servers since the eigen servers use SSL which is not supported\n# by cmake DOWNLOAD_COMMAND hg clone\n# https://bitbucket.org/eigen/eigen/\nExternalProject_Add(eigen\n PREFIX ${GraphLab_SOURCE_DIR}/deps/eigen\n URL http://bitbucket.org/eigen/eigen/get/3.1.2.tar.bz2\n URL_MD5 e9c081360dde5e7dcb8eba3c8430fde2\n CONFIGURE_COMMAND \"\"\n BUILD_COMMAND \"\"\n BUILD_IN_SOURCE 1\n INSTALL_COMMAND cp -r Eigen unsupported /\n INSTALL_DIR ${GraphLab_SOURCE_DIR}/deps/local/include)\nadd_definitions(-DHAS_EIGEN)\n\nmacro(requires_eigen NAME)\n add_dependencies(${NAME} eigen)\nendmacro(requires_eigen)\n\n\n# libbz =================================================================\nExternalProject_Add(libbz2\n PREFIX ${GraphLab_SOURCE_DIR}/deps/libbz2\n URL http://www.bzip.org/1.0.6/bzip2-1.0.6.tar.gz\n URL_MD5 00b516f4704d4a7cb50a1d97e6e8e15b\n INSTALL_DIR ${GraphLab_SOURCE_DIR}/deps/local\n CONFIGURE_COMMAND \"\"\n PATCH_COMMAND patch -N -p0 -i ${GraphLab_SOURCE_DIR}/patches/libbz2_fpic.patch || true\n BUILD_IN_SOURCE 1\n BUILD_COMMAND make install PREFIX=\n INSTALL_COMMAND \"\" )\n\n\n\n\n# Build Boost =================================================================\n# GraphLab relies heavily on the boost library for parsing program\n# options, filesystem support, random number generation, and\n# unordered containers.\nExternalProject_Add(boost\n PREFIX ${GraphLab_SOURCE_DIR}/deps/boost\n URL \"http://tcpdiag.dl.sourceforge.net/project/boost/boost/1.53.0/boost_1_53_0.tar.gz\" \n URL_MD5 57a9e2047c0f511c4dfcf00eb5eb2fbb\n BUILD_IN_SOURCE 1\n CONFIGURE_COMMAND\n ./bootstrap.sh\n --with-libraries=filesystem\n --with-libraries=program_options\n --with-libraries=system\n --with-libraries=iostreams\n --with-libraries=date_time\n --with-libraries=random\n --with-libraries=context\n --prefix=\n BUILD_COMMAND\n C_INCLUDE_PATH=${GraphLab_SOURCE_DIR}/deps/local/include\n CPLUS_INCLUDE_PATH=${GraphLab_SOURCE_DIR}/deps/local/include\n LIBRARY_PATH=${GraphLab_SOURCE_DIR}/deps/local/lib\n ./b2 install link=static variant=release threading=multi runtime-link=static\n INSTALL_COMMAND \"\"\n INSTALL_DIR ${GraphLab_SOURCE_DIR}/deps/local )\nset(BOOST_ROOT ${GraphLab_SOURCE_DIR}/deps/local )\nset(BOOST_LIBS_DIR ${GraphLab_SOURCE_DIR}/deps/local/lib)\nset(Boost_LIBRARIES\n ${BOOST_LIBS_DIR}/libboost_filesystem.a\n ${BOOST_LIBS_DIR}/libboost_program_options.a\n ${BOOST_LIBS_DIR}/libboost_system.a\n ${BOOST_LIBS_DIR}/libboost_iostreams.a\n ${BOOST_LIBS_DIR}/libboost_context.a\n ${BOOST_LIBS_DIR}/libboost_date_time.a)\nadd_dependencies(boost libbz2)\nmessage(STATUS \"Boost libs: \" ${Boost_LIBRARIES})\n# add_definitions(-DBOOST_DATE_TIME_POSIX_TIME_STD_CONFIG)\n# add_definitions(-DBOOST_ALL_DYN_LINK)\n# set(Boost_SHARED_LIBRARIES \"\")\nforeach(blib ${Boost_LIBRARIES})\n message(STATUS \"Boost libs: \" ${blib})\n string(REGEX REPLACE \"\\\\.a$\" ${CMAKE_SHARED_LIBRARY_SUFFIX} bout ${blib})\n message(STATUS \"Boost dyn libs: \" ${bout})\n set(Boost_SHARED_LIBRARIES ${Boost_SHARED_LIBRARIES} ${bout})\nendforeach()\nmessage(STATUS \"Boost Shared libs: \" ${Boost_SHARED_LIBRARIES})\n\nif(NOT NO_TCMALLOC)\n if(APPLE)\n set (tcmalloc_shared \"--enable-shared=yes\")\n else()\n set (tcmalloc_shared \"--enable-shared=no\")\n endif()\n\n # TCMalloc ===================================================================\n # We use tcmalloc for improved memory allocation performance\n ExternalProject_Add(libtcmalloc\n PREFIX ${GraphLab_SOURCE_DIR}/deps/tcmalloc\n # Some users can't access domain googlecode.com ,This is a spare URL\n # URL http://sourceforge.jp/projects/sfnet_gperftools.mirror/downloads/gperftools-2.0.tar.gz\n URL http://gperftools.googlecode.com/files/gperftools-2.0.tar.gz\n URL_MD5 13f6e8961bc6a26749783137995786b6\n PATCH_COMMAND patch -N -p0 -i ${GraphLab_SOURCE_DIR}/patches/tcmalloc.patch || true\n CONFIGURE_COMMAND /configure --enable-frame-pointers --prefix= ${tcmalloc_shared}\n INSTALL_DIR ${GraphLab_SOURCE_DIR}/deps/local)\n #link_libraries(tcmalloc)\n set(TCMALLOC-FOUND 1)\n add_definitions(-DHAS_TCMALLOC)\nendif()\n\n\n\n# Lib Jpeg =================================================================\n#ExternalProject_Add(libjpeg\n# PREFIX ${GraphLab_SOURCE_DIR}/deps/libjpeg\n# URL http://www.ijg.org/files/jpegsrc.v8d.tar.gz\n# INSTALL_DIR ${GraphLab_SOURCE_DIR}/deps/local\n# CONFIGURE_COMMAND /configure --prefix=\n# BUILD_IN_SOURCE 1)\n#\n\n\n# v8 ======================================================================\n\n# ImageMagick =================================================================\n# # set(imagemagick_configure_cmd\n# # bash -c \"C_INCLUDE_PATH=${GraphLab_SOURCE_DIR}/deps/local/include CPLUS_INCLUDE_PATH=${GraphLab_SOURCE_DIR}/deps/local/include LIBRARY_PATH=${GraphLab_SOURCE_DIR}/deps/local/lib LDFLAGS='-L${GraphLab_SOURCE_DIR}/deps/local/lib' ./configure --prefix=${GraphLab_SOURCE_DIR}/deps/local --without-threads --disable-openmp --disable-opencl\")\n# # ExternalProject_Add(imagemagick\n# # PREFIX ${GraphLab_SOURCE_DIR}/deps/imagemagick\n# # URL http://www.imagemagick.org/download/ImageMagick.tar.gz\n# # # URL_MD5 010b63a2542c4ec4918c8cb431c00356\n# # BUILD_IN_SOURCE 1\n# # CONFIGURE_COMMAND ${imagemagick_configure_cmd}\n# # BUILD_COMMAND\n# # C_INCLUDE_PATH=${GraphLab_SOURCE_DIR}/deps/local/include\n# # CPLUS_INCLUDE_PATH=${GraphLab_SOURCE_DIR}/deps/local/include\n# # LIBRARY_PATH=${GraphLab_SOURCE_DIR}/deps/local/lib\n# # # LDFLAGS=-L${GraphLab_SOURCE_DIR}/deps/local/lib\n# # make\n# # INSTALL_DIR ${GraphLab_SOURCE_DIR}/deps/local)\n#ExternalProject_Add(imagemagick\n# PREFIX ${GraphLab_SOURCE_DIR}/deps/imagemagick\n# URL http://www.imagemagick.org/download/ImageMagick.tar.gz\n# # URL_MD5 010b63a2542c4ec4918c8cb431c00356\n# CONFIGURE_COMMAND\n# C_INCLUDE_PATH=${GraphLab_SOURCE_DIR}/deps/local/include\n# CPLUS_INCLUDE_PATH=${GraphLab_SOURCE_DIR}/deps/local/include\n# LIBRARY_PATH=${GraphLab_SOURCE_DIR}/deps/local/lib\n# LDFLAGS=-L${GraphLab_SOURCE_DIR}/deps/local/lib\n# /configure\n# --prefix=\n# --without-threads\n# --disable-openmp\n# --disable-opencl\n# BUILD_COMMAND\n# C_INCLUDE_PATH=${GraphLab_SOURCE_DIR}/deps/local/include\n# CPLUS_INCLUDE_PATH=${GraphLab_SOURCE_DIR}/deps/local/include\n# LIBRARY_PATH=${GraphLab_SOURCE_DIR}/deps/local/lib\n# LDFLAGS=-L${GraphLab_SOURCE_DIR}/deps/local/lib\n# make\n# INSTALL_DIR ${GraphLab_SOURCE_DIR}/deps/local)\n#include_directories(${GraphLab_SOURCE_DIR}/deps/local/include/ImageMagick)\n#add_dependencies(imagemagick libjpeg)\n\n# macro(requires_imagemagick NAME)\n# target_link_libraries(${NAME} Magick++ jpeg)\n# add_dependencies(${NAME} imagemagick libjpeg)\n# endmacro(requires_imagemagick)\n\n# OpenCV =================================================================\nExternalProject_Add(opencv\n PREFIX ${GraphLab_SOURCE_DIR}/deps/opencv\n URL http://sourceforge.net/projects/opencvlibrary/files/opencv-unix/2.4.9/opencv-2.4.9.zip/download\n # URL_MD5 010b63a2542c4ec4918c8cb431c00356\n PATCH_COMMAND patch -N -p0 cmake/OpenCVModule.cmake -i ${GraphLab_SOURCE_DIR}/patches/opencv_apple_rpath.patch || true\n CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=\n -DBUILD_TESTS=OFF -DBUILD_PERF_TESTS=OFF -DBUILD_PACKAGE=OFF -DBUILD_EXAMPLES=OFF -DOPENCV_BUILD_3RDPARTY_LIBS=ON -DBUILD_SHARED_LIBS=ON -DBUILD_DOCS=OFF -DBUILD_JPEG=ON -DCMAKE_INCLUDE_PATH=${GraphLab_SOURCE_DIR}/deps/local/include -DWITH_CUBLAS=OFF -DWITH_1394=OFF -DWITH_AVFOUNDATION=OFF -DWITH_CUDA=OFF -DWITH_CUFFT=OFF -DWITH_FFMPEG=OFF -DWITH_GSTREAMER=OFF -DWITH_GTK=OFF -DWITH_QUICKTIME=OFF -DWITH_VIDEOINPUT=OFF -DWITH_XIMEA=OFF -DWITH_XINE=OFF -DWITH_V4L=OFF -DWITH_UNICAP=OFF -DWITH_QT=OFF -DWITH_JASPER=NO -DWITH_TIFF=NO -DWITH_OPENCL=OFF -DCMAKE_LIBRARY_PATH=${GraphLab_SOURCE_DIR}/deps/local/lib\n INSTALL_DIR ${GraphLab_SOURCE_DIR}/deps/local)\n\n# add_dependencies(opencv eigen)\n\nmacro(requires_opencv NAME)\n get_target_property(tmp ${NAME} COMPILE_FLAGS)\n if (NOT tmp)\n set(tmp \"-I${CMAKE_SOURCE_DIR}/deps/local/include/opencv\")\n else()\n set(tmp \"${tmp} -I${CMAKE_SOURCE_DIR}/deps/local/include/opencv\")\n endif()\n set_target_properties(${NAME} PROPERTIES COMPILE_FLAGS \"${tmp}\")\n target_link_libraries(${NAME} opencv_core opencv_imgproc opencv_calib3d opencv_video opencv_features2d opencv_ml opencv_highgui opencv_objdetect opencv_contrib opencv_legacy opencv_contrib opencv_stitching)\n add_dependencies(${NAME} opencv)\nendmacro(requires_opencv)\n\n# zlib =======================================================================\n# We rely on the zlib library to read gzip compressed files\n# (using boost iostreams).\n# check_library_exists(z zlibVersion \"\" ZLIB_FOUND)\n# if (ZLIB_FOUND)\n# add_definitions(-DHAS_ZLIB)\n# link_libraries(z)\n# endif()\n\n# libevent ====================================================================\n# LibEvent is used in the RPC layer to manage the interaction between the\n# TCP stack and the event handler threads\n\nif (APPLE)\n ExternalProject_Add(libevent\n PREFIX ${GraphLab_SOURCE_DIR}/deps/event\n URL http://iweb.dl.sourceforge.net/project/levent/libevent/libevent-2.0/libevent-2.0.18-stable.tar.gz\n URL_MD5 aa1ce9bc0dee7b8084f6855765f2c86a\n CONFIGURE_COMMAND /configure --prefix= --disable-openssl --enable-shared=no\n INSTALL_DIR ${GraphLab_SOURCE_DIR}/deps/local\n )\nelse()\n ExternalProject_Add(libevent\n PREFIX ${GraphLab_SOURCE_DIR}/deps/event\n URL http://iweb.dl.sourceforge.net/project/levent/libevent/libevent-2.0/libevent-2.0.18-stable.tar.gz\n URL_MD5 aa1ce9bc0dee7b8084f6855765f2c86a\n CONFIGURE_COMMAND /configure --prefix= --disable-openssl --enable-shared=no\n INSTALL_DIR ${GraphLab_SOURCE_DIR}/deps/local\n INSTALL_COMMAND prefix=/ make install && ${GraphLab_SOURCE_DIR}/patches/libevent_clean_and_remap.sh /lib\n )\nendif()\n#link_libraries(event)\n#link_libraries(event_pthreads)\n\n\n# libjson ====================================================================\n# Lib Json is used to support json serialization for long term storage of\n# graph data.\n#ExternalProject_Add(libjson\n# PREFIX ${GraphLab_SOURCE_DIR}/deps/json\n# URL http://graphlab.org/deps/libjson_7.6.0.zip\n# URL_MD5 dcb326038bd9b710b8f717580c647833\n# BUILD_IN_SOURCE 1\n# CONFIGURE_COMMAND \"\"\n# PATCH_COMMAND patch -N -p1 -i ${GraphLab_SOURCE_DIR}/patches/libjson.patch || true\n# BUILD_COMMAND make\n# INSTALL_COMMAND prefix=/ make install\n# INSTALL_DIR ${GraphLab_SOURCE_DIR}/deps/local\n# )\n\n\n# zookeeper ================================================================\n\nExternalProject_Add(zookeeper\n PREFIX ${GraphLab_SOURCE_DIR}/deps/zookeeper\n URL http://mirror.metrocast.net/apache/zookeeper/zookeeper-3.5.1-alpha/zookeeper-3.5.1-alpha.tar.gz\n URL_MD5 d85f9751724d3f20f792803b61c4db24\n PATCH_COMMAND ${CMAKE_COMMAND} -E copy_directory ${GraphLab_SOURCE_DIR}/patches/zookeeper/ \n BUILD_IN_SOURCE 1\n CONFIGURE_COMMAND ./configure --prefix= --disable-shared\n INSTALL_DIR ${GraphLab_SOURCE_DIR}/deps/local)\n\n## =============================================================\n# JNI =========================================================================\n# We use JNI for jvm interfacing and for hdfs support. We currently only\n# detect the presence of java and do not install our own version.\n\nif(NOT NO_JAVAC)\n find_package(Java)\n find_package(JNI)\n\n\n if((EXISTS ${JAVA_INCLUDE_PATH}/jni.h))\n set(JNI_REALLY_FOUND 1 CACHE BOOL \"Tests whether the header actually exists\")\n message(STATUS \"jni.h was found at \" ${JAVA_INCLUDE_PATH}/jni.h)\n include_directories(${JNI_INCLUDE_DIRS})\n ## Guess java home location\n if(EXISTS $ENV{JAVA_HOME} )\n set(JAVA_HOME $ENV{JAVA_HOME} CACHE STRING \"Location of Java Home\")\n message(STATUS \"Java home set by user: \" ${JAVA_HOME})\n else()\n string(REGEX REPLACE \"\\\\/include\" \"\" JAVA_HOME_GUESS ${JAVA_AWT_INCLUDE_PATH})\n set(JAVA_HOME ${JAVA_HOME_GUESS} CACHE STRING \"Location of Java Home\")\n message(STATUS \"Java Home guessed: \" ${JAVA_HOME})\n endif()\n else( )\n set(JNI_REALLY_FOUND 0 CACHE BOOL \"Tests whether the header actually exists\")\n message(STATUS \"jni.h was not found at \" ${JAVA_INCLUDE_PATH}/jni.h)\n endif( )\n\n # Ant =========================================================================\n # Ant is used to build the graphlab JVM interface\n find_package(Ant)\n\n # LibHDFS =====================================================================\n # If JNI is found we install libhdfs which allows programs to read and write\n # to hdfs filesystems\n if( JNI_REALLY_FOUND )\n message(STATUS \"Building libhdfs\")\n ExternalProject_Add(hadoop\n PREFIX ${GraphLab_SOURCE_DIR}/deps/hadoop\n URL https://archive.apache.org/dist/hadoop/core/hadoop-1.0.1/hadoop-1.0.1.tar.gz\n# URL http://www.gtlib.gatech.edu/pub/apache/hadoop/common/hadoop-1.0.1/hadoop-1.0.1.tar.gz\n URL_MD5 e627d9b688c4de03cba8313bd0bba148\n UPDATE_COMMAND chmod +x /src/c++/libhdfs/install-sh /src/c++/libhdfs/configure\n PATCH_COMMAND patch -N -p1 -i ${GraphLab_SOURCE_DIR}/patches/libhdfs.patch || true\n BUILD_IN_SOURCE 1\n CONFIGURE_COMMAND /src/c++/libhdfs/configure JVM_ARCH=tune=generic --prefix= --with-java=${JAVA_HOME} --enable-shared=no --enable-static=yes\n INSTALL_DIR ${GraphLab_SOURCE_DIR}/deps/local\n TEST_COMMAND bash -c \"mkdir -p /include && cp /src/c++/libhdfs/hdfs.h /include\")\n set(HADOOP_FOUND 1 CACHE BOOL \"Hadoop was found\")\n add_definitions(-DHAS_HADOOP)\n # link_libraries(hdfs ${JAVA_JVM_LIBRARY})\n endif( JNI_REALLY_FOUND )\n\n\n\nendif()\n\n\n\n# MPI =========================================================================\n# MPI is currently used to establish communication between rpc nodes\n#sgr this name is confusing. i like WITH_MPI better. likewise for NO_OPENMP\nif(NO_MPI)\n message(STATUS \"MPI disabled as a configure option.\")\nelse()\n #find_package(MPICH2)\n find_package(MPI) \n if(MPI_FOUND)\n message(STATUS \"MPI Found: \" ${MPI_LIBRARY} ${MPI_EXTRA_LIBRARY})\n add_definitions(-DHAS_MPI)\n include_directories(${MPI_INCLUDE_PATH})\n else(MPI_FOUND)\n message(STATUS \"MPI Not Found! Distributed Executables will not be compiled\")\n set(MPI_LIBRARY \"\")\n set(MPI_EXTRA_LIBRARY \"\")\n # ExternalProject_Add(openmpi\n # PREFIX ${GraphLab_SOURCE_DIR}/deps/openmpi\n # URL http://www.open-mpi.org/software/ompi/v1.4/downloads/openmpi-1.4.5.tar.gz\n # CONFIGURE_COMMAND /configure --prefix=\n # INSTALL_DIR ${GraphLab_SOURCE_DIR}/deps/local)\n endif(MPI_FOUND)\nendif()\n\n\n\n#ExternalProject_Add(qthreads\n# PREFIX ${GraphLab_SOURCE_DIR}/deps/qthreads\n# URL http://graphlab.org/deps/qthreads.tar.bz2\n# URL_MD5 fd94ae41a06f3756c1042d47cfa671d3\n# INSTALL_DIR ${GraphLab_SOURCE_DIR}/deps/local\n# BUILD_IN_SOURCE 1\n# CONFIGURE_COMMAND ./configure --prefix= --enable-oversubscription --with-scheduler=sherwood --enable-lf-febs -enable-condwait-queue\n# )\n\n\n\n## =============================================================\n# Core graphlab requirements\nmacro(requires_core_deps NAME)\n target_link_libraries(${NAME}\n ${Boost_LIBRARIES}\n z\n event event_pthreads\n zookeeper_mt)\n add_dependencies(${NAME} boost libevent zookeeper)\n if(MPI_FOUND)\n target_link_libraries(${NAME} ${MPI_LIBRARY} ${MPI_EXTRA_LIBRARY})\n endif(MPI_FOUND)\n if(HADOOP_FOUND)\n target_link_libraries(${NAME} hdfs ${JAVA_JVM_LIBRARY})\n add_dependencies(${NAME} hadoop)\n endif(HADOOP_FOUND)\n if(NOT NO_TCMALLOC)\n target_link_libraries(${NAME} tcmalloc)\n add_dependencies(${NAME} libtcmalloc)\n endif()\nendmacro(requires_core_deps)\n\n\n\n\n\n## ============================================================================\n# System Checks\n\n## ============================================================================\n# Test for cpu affinity support\nset(CMAKE_REQUIRED_LIBRARIES \"pthread\")\ncheck_function_exists(pthread_setaffinity_np HAS_SET_AFFINITY)\nset(CMAKE_REQUIRED_LIBRARIES ${crlbackup})\n\ninclude(CheckCXXCompilerFlag)\n## ============================================================================\n# check if MARCH is set\nif (NOT MARCH)\n #set to native if supported\n check_cxx_compiler_flag(-march=native HAS_MARCH_NATIVE)\n if(HAS_MARCH_NATIVE)\n set(MARCH \"native\")\n else()\n set(MARCH \"x86-64\")\n endif()\nendif()\n\n# check if MTUNE is set\nif (NOT MTUNE)\n #set to native if supported\n check_cxx_compiler_flag(-mtune=native HAS_MTUNE_NATIVE)\n if(HAS_MTUNE_NATIVE)\n set(MTUNE \"native\")\n else()\n set(MTUNE \"x86-64\")\n endif()\nendif()\n\n## ============================================================================\n# Setup compiler\n\n# If profiling mode is enabled then additional profiling flags are set for\n# the compiler\nif (COMPILE_PROFILING MATCHES 1)\n set(PROFILING_FLAGS \"-DUSE_EVENT_LOG -DUSE_TRACEPOINT\")\nelse()\n set(PROFILING_FLAGS \"\")\nendif()\n\n# check for SSE instruction set (needed for CRC32)\n#check_cxx_compiler_flag(-msse4.2 HAS_SSE42)\n#set(SSE42_FLAG \"\")\n#if (HAS_SSE42)\n# set(SSE42_FLAG \"-msse4.2\")\n#endif()\n\n\n#disable Wno-unused-local-typedefs if available\ncheck_cxx_compiler_flag(-Wno-unused-local-typedefs HAS_WNO_LOCAL_TYPEDEFS)\nif(HAS_WNO_LOCAL_TYPEDEFS)\n set(WNO_LOCAL_TYPEDEFS \"-Wno-unused-local-typedefs\")\nelse()\n set(WNO_LOCAL_TYPEDEFS \"\")\nendif()\n\n\n# Set the debug flags\nset(CMAKE_C_FLAGS_DEBUG\n \"-O0 -Wno-attributes -march=${MARCH} -Winit-self ${PROFILING_FLAGS} ${COMPILER_FLAGS}\"\n CACHE STRING \"compiler options\" FORCE)\nset(CMAKE_CXX_FLAGS_DEBUG\n \"-O0 ${WNO_LOCAL_TYPEDEFS} -Wno-attributes -march=${MARCH} -Winit-self ${PROFILING_FLAGS} ${COMPILER_FLAGS}\"\n CACHE STRING \"compiler options\" FORCE)\n\nset(CMAKE_C_FLAGS_RELEASE\n \"-O3 -Wno-attributes -march=${MARCH} -mtune=${MTUNE} ${PROFILING_FLAGS} ${COMPILER_FLAGS}\"\n CACHE STRING \"compiler options\" FORCE)\nset(CMAKE_CXX_FLAGS_RELEASE\n \"-O3 ${WNO_LOCAL_TYPEDEFS} -Wno-attributes -march=${MARCH} -mtune=${MTUNE} ${PROFILING_FLAGS} ${COMPILER_FLAGS}\"\n CACHE STRING \"compiler options\" FORCE)\n\nset(CMAKE_REQUIRED_FLAGS ${CMAKE_CXX_FLAGS_RELEASE})\ncheck_cxx_source_compiles(\"int main(int argc, char** argv) { return __builtin_ia32_crc32di(0,0);}\" HAS_CRC32)\nset(CMAKE_REQUIRED_FLAGS \"\")\nif(HAS_CRC32)\n set(CMAKE_CXX_FLAGS_RELEASE\n \"${CMAKE_CXX_FLAGS_RELEASE} -DHAS_BUILTIN_CRC32\"\n CACHE STRING \"compiler options\" FORCE)\nendif()\n\n# check for gcc 4.8\n\n\n# # Mex setup\n# set(CMAKE_CXX_FLAGS_MEX\n# \"${CMAKE_CXX_FLAGS_RELEASE} -D_GNU_SOURCE -fPIC -fno-omit-frame-pointer -pthread \")\n# set(CMAKE_C_FLAGS_MEX\n# \"${CMAKE_C_FLAGS_RELEASE} -D_GNU_SOURCE -fexceptions -fPIC -fno-omit-frame-pointer -pthread \")\n\n\nif (CMAKE_BUILD_TYPE MATCHES \"Release\")\n message(STATUS \"Release build with C++ flags: \" ${CMAKE_CXX_FLAGS_RELEASE})\n message(STATUS \"Release build with C flags: \" ${CMAKE_C_FLAGS_RELEASE})\nelseif(CMAKE_BUILD_TYPE MATCHES \"Debug\")\n message(STATUS \"Debug build with C++ flags: \" ${CMAKE_CXX_FLAGS_DEBUG})\n message(STATUS \"Debug build with C flags: \" ${CMAKE_C_FLAGS_DEBUG})\nelseif(CMAKE_BUILD_TYPE MATCHES \"Mex\")\n message(STATUS \"Mex CXX\" ${CMAKE_CXX_FLAGS_MEX})\n message(STATUS \"Mex C\" ${CMAKE_C_FLAGS_MEX})\nelse()\n message(WARNING \"Unknown build type: \" ${CMAKE_BUILD_TYPE} \"!\")\nendif()\n\n\n\n\n\n\n\n## ============================================================================\n## ============================================================================\n## ============================================================================\n# Setup testing tools\n# Make sure testing is enabled\nenable_testing()\n# Use Python interpreter\nfind_package(PythonInterp)\nset(CXXTESTGEN ${CMAKE_SOURCE_DIR}/cxxtest/cxxtestgen)\n# create a macro to define a test\nmacro(ADD_CXXTEST NAME)\n if(PYTHONINTERP_FOUND)\n add_custom_command(\n OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${NAME}.cpp\n COMMAND\n ${PYTHON_EXECUTABLE} ${CXXTESTGEN}\n --runner=ErrorPrinter\n -o ${CMAKE_CURRENT_BINARY_DIR}/${NAME}.cpp ${ARGV}\n DEPENDS ${ARGV}\n WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}\n )\n endif(PYTHONINTERP_FOUND)\n add_graphlab_executable(${NAME}test ${CMAKE_CURRENT_BINARY_DIR}/${NAME}.cpp)\n set_source_files_properties( ${CMAKE_CURRENT_BINARY_DIR}/${NAME}.cpp\n PROPERTIES COMPILE_FLAGS \"-I${CMAKE_CURRENT_SOURCE_DIR}\" )\n\n add_test(${NAME} ${NAME}test)\nendmacro(ADD_CXXTEST)\n\n\n# macro(ADD_CPPTEST NAME)\n# add_graphlab_executable(${NAME}test ${NAME})\n# set_source_files_properties(${NAME}\n# PROPERTIES COMPILE_FLAGS \"-I${CMAKE_CURRENT_SOURCE_DIR}\" )\n# add_test(${NAME} ${NAME}test)\n# endmacro(ADD_CPPTEST)\n\n\n\n\n\n\n## ============================================================================\n## ============================================================================\n## ============================================================================\n# Build Macros\n\n# copy_file ===================================================================\n# copy a single file into build environment\nmacro(copy_file NAME)\n message(STATUS \"Copying File: \" ${NAME})\n file(INSTALL ${CMAKE_CURRENT_SOURCE_DIR}/${NAME}\n DESTINATION ${CMAKE_CURRENT_BINARY_DIR} )\nendmacro(copy_file NAME)\n\n# copy_files ==================================================================\n# copy all files matching a pattern into the build environment\nmacro(copy_files NAME)\n message(STATUS \"Copying Files: \" ${NAME})\n file(INSTALL ${CMAKE_CURRENT_SOURCE_DIR}/\n DESTINATION ${CMAKE_CURRENT_BINARY_DIR}\n FILES_MATCHING PATTERN ${NAME} )\nendmacro(copy_files NAME)\n\n\n# macro(EXEC file prog)\n# add_custom_command(\n# OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${file}\n# COMMAND\n# ${prog}\n# DEPENDS ${prog}\n# WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}\n# )\n# endmacro(EXEC)\n\n# add_graphlab_executable =====================================================\n# copy all files matching a pattern into the build environment\nmacro(add_graphlab_executable NAME)\n add_executable(${NAME} ${ARGN})\n target_link_libraries(${NAME} graphlab)\n add_dependencies(${NAME} graphlab)\n requires_core_deps(${NAME})\nendmacro(add_graphlab_executable)\n\n\nmacro(add_extension_executable NAME)\n add_executable(${NAME} extension_main.cpp ${ARGN})\n# if(!APPLE)\n#set_target_properties(${NAME} PROPERTIES LINK_FLAGS -Wl,-wrap,main)\n# endif()\n target_link_libraries(${NAME} graphlab graphlab_extension)\n add_dependencies(${NAME} graphlab graphlab_extension)\n requires_core_deps(${NAME})\nendmacro(add_extension_executable)\n\n\n\nadd_custom_target(external_dependencies)\nrequires_core_deps(external_dependencies)\n\n# add_jni_library =============================================================\n# If jni was found then create a jni library. Otherwise generate a warning\nmacro(add_jni_library NAME)\n # Only build if JNI was found\n if (JNI_REALLY_FOUND)\n include_directories(\n ${JAVA_INCLUDE_PATH}\n ${JAVA_INCLUDE_PATH}/linux\n )\n message(STATUS \"Detected JNI library \" ${NAME})\n add_library(${NAME} SHARED ${ARGN})\n target_link_libraries(${NAME} graphlab_pic)\n target_link_libraries(${NAME} ${Boost_SHARED_LIBRARIES})\n # IF (APPLE)\n # SET(CMAKE_SHARED_MODULE_CREATE_CXX_FLAGS \"-dynamiclib -m64 \")\n # SET_TARGET_PROPERTIES(${NAME} PROPERTIES SUFFIX .jnilib)\n # ENDIF (APPLE)\n elseif ()\n message(STATUS \"Not building \" ${NAME} \" because JNI was not found\")\n endif ()\nendmacro(add_jni_library)\n" }, { "path": "Doxyfile", "content": "# Doxyfile 1.5.8\n\n# This file describes the settings to be used by the documentation system\n# doxygen (www.doxygen.org) for a project\n#\n# All text after a hash (#) is considered a comment and will be ignored\n# The format is:\n# TAG = value [value, ...]\n# For lists items can also be appended using:\n# TAG += value [value, ...]\n# Values that contain spaces should be placed between quotes (\" \")\n\n#---------------------------------------------------------------------------\n# Project related configuration options\n#---------------------------------------------------------------------------\n\n# This tag specifies the encoding used for all characters in the config file \n# that follow. The default is UTF-8 which is also the encoding used for all \n# text before the first occurrence of this tag. Doxygen uses libiconv (or the \n# iconv built into libc) for the transcoding. See \n# http://www.gnu.org/software/libiconv for the list of possible encodings.\n\nDOXYFILE_ENCODING = UTF-8\n\n# The PROJECT_NAME tag is a single word (or a sequence of words surrounded \n# by quotes) that should identify the project.\n\nPROJECT_NAME = \"GraphLab: Distributed Graph-Parallel API\"\n\n# The PROJECT_NUMBER tag can be used to enter a project or revision number. \n# This could be handy for archiving the generated documentation or \n# if some version control system is used.\n\nPROJECT_NUMBER = 2.2\n\n# The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) \n# base path where the generated documentation will be put. \n# If a relative path is entered, it will be relative to the location \n# where doxygen was started. If left blank the current directory will be used.\n\nOUTPUT_DIRECTORY = doc/doxygen\n\n# If the CREATE_SUBDIRS tag is set to YES, then doxygen will create \n# 4096 sub-directories (in 2 levels) under the output directory of each output \n# format and will distribute the generated files over these directories. \n# Enabling this option can be useful when feeding doxygen a huge amount of \n# source files, where putting all generated files in the same directory would \n# otherwise cause performance problems for the file system.\n\nCREATE_SUBDIRS = NO\n\n# The OUTPUT_LANGUAGE tag is used to specify the language in which all \n# documentation generated by doxygen is written. Doxygen will use this \n# information to generate all constant output in the proper language. \n# The default language is English, other supported languages are: \n# Afrikaans, Arabic, Brazilian, Catalan, Chinese, Chinese-Traditional, \n# Croatian, Czech, Danish, Dutch, Farsi, Finnish, French, German, Greek, \n# Hungarian, Italian, Japanese, Japanese-en (Japanese with English messages), \n# Korean, Korean-en, Lithuanian, Norwegian, Macedonian, Persian, Polish, \n# Portuguese, Romanian, Russian, Serbian, Serbian-Cyrilic, Slovak, Slovene, \n# Spanish, Swedish, and Ukrainian.\n\nOUTPUT_LANGUAGE = English\n\n# If the BRIEF_MEMBER_DESC tag is set to YES (the default) Doxygen will \n# include brief member descriptions after the members that are listed in \n# the file and class documentation (similar to JavaDoc). \n# Set to NO to disable this.\n\nBRIEF_MEMBER_DESC = YES\n\n# If the REPEAT_BRIEF tag is set to YES (the default) Doxygen will prepend \n# the brief description of a member or function before the detailed description. \n# Note: if both HIDE_UNDOC_MEMBERS and BRIEF_MEMBER_DESC are set to NO, the \n# brief descriptions will be completely suppressed.\n\nREPEAT_BRIEF = YES\n\n# This tag implements a quasi-intelligent brief description abbreviator \n# that is used to form the text in various listings. Each string \n# in this list, if found as the leading text of the brief description, will be \n# stripped from the text and the result after processing the whole list, is \n# used as the annotated text. Otherwise, the brief description is used as-is. \n# If left blank, the following values are used (\"$name\" is automatically \n# replaced with the name of the entity): \"The $name class\" \"The $name widget\" \n# \"The $name file\" \"is\" \"provides\" \"specifies\" \"contains\" \n# \"represents\" \"a\" \"an\" \"the\"\n\nABBREVIATE_BRIEF = \n\n# If the ALWAYS_DETAILED_SEC and REPEAT_BRIEF tags are both set to YES then \n# Doxygen will generate a detailed section even if there is only a brief \n# description.\n\nALWAYS_DETAILED_SEC = NO\n\n# If the INLINE_INHERITED_MEMB tag is set to YES, doxygen will show all \n# inherited members of a class in the documentation of that class as if those \n# members were ordinary class members. Constructors, destructors and assignment \n# operators of the base classes will not be shown.\n\nINLINE_INHERITED_MEMB = YES\n\n# If the FULL_PATH_NAMES tag is set to YES then Doxygen will prepend the full \n# path before files name in the file list and in the header files. If set \n# to NO the shortest path that makes the file name unique will be used.\n\nFULL_PATH_NAMES = YES\n\n# If the FULL_PATH_NAMES tag is set to YES then the STRIP_FROM_PATH tag \n# can be used to strip a user-defined part of the path. Stripping is \n# only done if one of the specified strings matches the left-hand part of \n# the path. The tag can be used to show relative paths in the file list. \n# If left blank the directory from which doxygen is run is used as the \n# path to strip.\n\nSTRIP_FROM_PATH = src/ \n\n# The STRIP_FROM_INC_PATH tag can be used to strip a user-defined part of \n# the path mentioned in the documentation of a class, which tells \n# the reader which header file to include in order to use a class. \n# If left blank only the name of the header file containing the class \n# definition is used. Otherwise one should specify the include paths that \n# are normally passed to the compiler using the -I flag.\n\nSTRIP_FROM_INC_PATH = src/ \n\n# If the SHORT_NAMES tag is set to YES, doxygen will generate much shorter \n# (but less readable) file names. This can be useful is your file systems \n# doesn't support long names like on DOS, Mac, or CD-ROM.\n\nSHORT_NAMES = NO\n\n# If the JAVADOC_AUTOBRIEF tag is set to YES then Doxygen \n# will interpret the first line (until the first dot) of a JavaDoc-style \n# comment as the brief description. If set to NO, the JavaDoc \n# comments will behave just like regular Qt-style comments \n# (thus requiring an explicit @brief command for a brief description.)\n\nJAVADOC_AUTOBRIEF = NO\n\n# If the QT_AUTOBRIEF tag is set to YES then Doxygen will \n# interpret the first line (until the first dot) of a Qt-style \n# comment as the brief description. If set to NO, the comments \n# will behave just like regular Qt-style comments (thus requiring \n# an explicit \\brief command for a brief description.)\n\nQT_AUTOBRIEF = NO\n\n# The MULTILINE_CPP_IS_BRIEF tag can be set to YES to make Doxygen \n# treat a multi-line C++ special comment block (i.e. a block of //! or /// \n# comments) as a brief description. This used to be the default behaviour. \n# The new default is to treat a multi-line C++ comment block as a detailed \n# description. Set this tag to YES if you prefer the old behaviour instead.\n\nMULTILINE_CPP_IS_BRIEF = NO\n\n# If the INHERIT_DOCS tag is set to YES (the default) then an undocumented \n# member inherits the documentation from any documented member that it \n# re-implements.\n\nINHERIT_DOCS = YES\n\n# If the SEPARATE_MEMBER_PAGES tag is set to YES, then doxygen will produce \n# a new page for each member. If set to NO, the documentation of a member will \n# be part of the file/class/namespace that contains it.\n\nSEPARATE_MEMBER_PAGES = NO\n\n# The TAB_SIZE tag can be used to set the number of spaces in a tab. \n# Doxygen uses this value to replace tabs by spaces in code fragments.\n\nTAB_SIZE = 2\n\n# This tag can be used to specify a number of aliases that acts \n# as commands in the documentation. An alias has the form \"name=value\". \n# For example adding \"sideeffect=\\par Side Effects:\\n\" will allow you to \n# put the command \\sideeffect (or @sideeffect) in the documentation, which \n# will result in a user-defined paragraph with heading \"Side Effects:\". \n# You can put \\n's in the value part of an alias to insert newlines.\n\nALIASES = \n\n# Set the OPTIMIZE_OUTPUT_FOR_C tag to YES if your project consists of C \n# sources only. Doxygen will then generate output that is more tailored for C. \n# For instance, some of the names that are used will be different. The list \n# of all members will be omitted, etc.\n\nOPTIMIZE_OUTPUT_FOR_C = NO\n\n# Set the OPTIMIZE_OUTPUT_JAVA tag to YES if your project consists of Java \n# sources only. Doxygen will then generate output that is more tailored for \n# Java. For instance, namespaces will be presented as packages, qualified \n# scopes will look different, etc.\n\nOPTIMIZE_OUTPUT_JAVA = NO\n\n# Set the OPTIMIZE_FOR_FORTRAN tag to YES if your project consists of Fortran \n# sources only. Doxygen will then generate output that is more tailored for \n# Fortran.\n\nOPTIMIZE_FOR_FORTRAN = NO\n\n# Set the OPTIMIZE_OUTPUT_VHDL tag to YES if your project consists of VHDL \n# sources. Doxygen will then generate output that is tailored for \n# VHDL.\n\nOPTIMIZE_OUTPUT_VHDL = NO\n\n# Doxygen selects the parser to use depending on the extension of the files it parses. \n# With this tag you can assign which parser to use for a given extension. \n# Doxygen has a built-in mapping, but you can override or extend it using this tag. \n# The format is ext=language, where ext is a file extension, and language is one of \n# the parsers supported by doxygen: IDL, Java, Javascript, C#, C, C++, D, PHP, \n# Objective-C, Python, Fortran, VHDL, C, C++. For instance to make doxygen treat \n# .inc files as Fortran files (default is PHP), and .f files as C (default is Fortran), \n# use: inc=Fortran f=C\n\nEXTENSION_MAPPING = \n\n# If you use STL classes (i.e. std::string, std::vector, etc.) but do not want \n# to include (a tag file for) the STL sources as input, then you should \n# set this tag to YES in order to let doxygen match functions declarations and \n# definitions whose arguments contain STL classes (e.g. func(std::string); v.s. \n# func(std::string) {}). This also make the inheritance and collaboration \n# diagrams that involve STL classes more complete and accurate.\n\nBUILTIN_STL_SUPPORT = YES\n\n# If you use Microsoft's C++/CLI language, you should set this option to YES to \n# enable parsing support.\n\nCPP_CLI_SUPPORT = NO\n\n# Set the SIP_SUPPORT tag to YES if your project consists of sip sources only. \n# Doxygen will parse them like normal C++ but will assume all classes use public \n# instead of private inheritance when no explicit protection keyword is present.\n\nSIP_SUPPORT = NO\n\n# For Microsoft's IDL there are propget and propput attributes to indicate getter \n# and setter methods for a property. Setting this option to YES (the default) \n# will make doxygen to replace the get and set methods by a property in the \n# documentation. This will only work if the methods are indeed getting or \n# setting a simple type. If this is not the case, or you want to show the \n# methods anyway, you should set this option to NO.\n\nIDL_PROPERTY_SUPPORT = NO\n\n# If member grouping is used in the documentation and the DISTRIBUTE_GROUP_DOC \n# tag is set to YES, then doxygen will reuse the documentation of the first \n# member in the group (if any) for the other members of the group. By default \n# all members of a group must be documented explicitly.\n\nDISTRIBUTE_GROUP_DOC = NO\n\n# Set the SUBGROUPING tag to YES (the default) to allow class member groups of \n# the same type (for instance a group of public functions) to be put as a \n# subgroup of that type (e.g. under the Public Functions section). Set it to \n# NO to prevent subgrouping. Alternatively, this can be done per class using \n# the \\nosubgrouping command.\n\nSUBGROUPING = YES\n\n# When TYPEDEF_HIDES_STRUCT is enabled, a typedef of a struct, union, or enum \n# is documented as struct, union, or enum with the name of the typedef. So \n# typedef struct TypeS {} TypeT, will appear in the documentation as a struct \n# with name TypeT. When disabled the typedef will appear as a member of a file, \n# namespace, or class. And the struct will be named TypeS. This can typically \n# be useful for C code in case the coding convention dictates that all compound \n# types are typedef'ed and only the typedef is referenced, never the tag name.\n\nTYPEDEF_HIDES_STRUCT = NO\n\n# The SYMBOL_CACHE_SIZE determines the size of the internal cache use to \n# determine which symbols to keep in memory and which to flush to disk. \n# When the cache is full, less often used symbols will be written to disk. \n# For small to medium size projects (<1000 input files) the default value is \n# probably good enough. For larger projects a too small cache size can cause \n# doxygen to be busy swapping symbols to and from disk most of the time \n# causing a significant performance penality. \n# If the system has enough physical memory increasing the cache will improve the \n# performance by keeping more symbols in memory. Note that the value works on \n# a logarithmic scale so increasing the size by one will rougly double the \n# memory usage. The cache size is given by this formula: \n# 2^(16+SYMBOL_CACHE_SIZE). The valid range is 0..9, the default is 0, \n# corresponding to a cache size of 2^16 = 65536 symbols\n\nSYMBOL_CACHE_SIZE = 0\n\n#---------------------------------------------------------------------------\n# Build related configuration options\n#---------------------------------------------------------------------------\n\n# If the EXTRACT_ALL tag is set to YES doxygen will assume all entities in \n# documentation are documented, even if no documentation was available. \n# Private class members and static file members will be hidden unless \n# the EXTRACT_PRIVATE and EXTRACT_STATIC tags are set to YES\n\nEXTRACT_ALL = NO\n\n# If the EXTRACT_PRIVATE tag is set to YES all private members of a class \n# will be included in the documentation.\n\nEXTRACT_PRIVATE = NO\n\n# If the EXTRACT_STATIC tag is set to YES all static members of a file \n# will be included in the documentation.\n\nEXTRACT_STATIC = NO\n\n# If the EXTRACT_LOCAL_CLASSES tag is set to YES classes (and structs) \n# defined locally in source files will be included in the documentation. \n# If set to NO only classes defined in header files are included.\n\nEXTRACT_LOCAL_CLASSES = NO\n\n# This flag is only useful for Objective-C code. When set to YES local \n# methods, which are defined in the implementation section but not in \n# the interface are included in the documentation. \n# If set to NO (the default) only methods in the interface are included.\n\nEXTRACT_LOCAL_METHODS = NO\n\n# If this flag is set to YES, the members of anonymous namespaces will be \n# extracted and appear in the documentation as a namespace called \n# 'anonymous_namespace{file}', where file will be replaced with the base \n# name of the file that contains the anonymous namespace. By default \n# anonymous namespace are hidden.\n\nEXTRACT_ANON_NSPACES = NO\n\n# If the HIDE_UNDOC_MEMBERS tag is set to YES, Doxygen will hide all \n# undocumented members of documented classes, files or namespaces. \n# If set to NO (the default) these members will be included in the \n# various overviews, but no documentation section is generated. \n# This option has no effect if EXTRACT_ALL is enabled.\n\nHIDE_UNDOC_MEMBERS = NO\n\n# If the HIDE_UNDOC_CLASSES tag is set to YES, Doxygen will hide all \n# undocumented classes that are normally visible in the class hierarchy. \n# If set to NO (the default) these classes will be included in the various \n# overviews. This option has no effect if EXTRACT_ALL is enabled.\n\nHIDE_UNDOC_CLASSES = YES\n\n# If the HIDE_FRIEND_COMPOUNDS tag is set to YES, Doxygen will hide all \n# friend (class|struct|union) declarations. \n# If set to NO (the default) these declarations will be included in the \n# documentation.\n\nHIDE_FRIEND_COMPOUNDS = YES\n\n# If the HIDE_IN_BODY_DOCS tag is set to YES, Doxygen will hide any \n# documentation blocks found inside the body of a function. \n# If set to NO (the default) these blocks will be appended to the \n# function's detailed documentation block.\n\nHIDE_IN_BODY_DOCS = YES\n\n# The INTERNAL_DOCS tag determines if documentation \n# that is typed after a \\internal command is included. If the tag is set \n# to NO (the default) then the documentation will be excluded. \n# Set it to YES to include the internal documentation.\n\nINTERNAL_DOCS = NO\n\n# If the CASE_SENSE_NAMES tag is set to NO then Doxygen will only generate \n# file names in lower-case letters. If set to YES upper-case letters are also \n# allowed. This is useful if you have classes or files whose names only differ \n# in case and if your file system supports case sensitive file names. Windows \n# and Mac users are advised to set this option to NO.\n\nCASE_SENSE_NAMES = YES\n\n# If the HIDE_SCOPE_NAMES tag is set to NO (the default) then Doxygen \n# will show members with their full class and namespace scopes in the \n# documentation. If set to YES the scope will be hidden.\n\nHIDE_SCOPE_NAMES = NO\n\n# If the SHOW_INCLUDE_FILES tag is set to YES (the default) then Doxygen \n# will put a list of the files that are included by a file in the documentation \n# of that file.\n\nSHOW_INCLUDE_FILES = YES\n\n# If the INLINE_INFO tag is set to YES (the default) then a tag [inline] \n# is inserted in the documentation for inline members.\n\nINLINE_INFO = YES\n\n# If the SORT_MEMBER_DOCS tag is set to YES (the default) then doxygen \n# will sort the (detailed) documentation of file and class members \n# alphabetically by member name. If set to NO the members will appear in \n# declaration order.\n\nSORT_MEMBER_DOCS = YES\n\n# If the SORT_BRIEF_DOCS tag is set to YES then doxygen will sort the \n# brief documentation of file, namespace and class members alphabetically \n# by member name. If set to NO (the default) the members will appear in \n# declaration order.\n\nSORT_BRIEF_DOCS = NO\n\n# If the SORT_GROUP_NAMES tag is set to YES then doxygen will sort the \n# hierarchy of group names into alphabetical order. If set to NO (the default) \n# the group names will appear in their defined order.\n\nSORT_GROUP_NAMES = NO\n\n# If the SORT_BY_SCOPE_NAME tag is set to YES, the class list will be \n# sorted by fully-qualified names, including namespaces. If set to \n# NO (the default), the class list will be sorted only by class name, \n# not including the namespace part. \n# Note: This option is not very useful if HIDE_SCOPE_NAMES is set to YES. \n# Note: This option applies only to the class list, not to the \n# alphabetical list.\n\nSORT_BY_SCOPE_NAME = NO\n\n# The GENERATE_TODOLIST tag can be used to enable (YES) or \n# disable (NO) the todo list. This list is created by putting \\todo \n# commands in the documentation.\n\nGENERATE_TODOLIST = NO\n\n# The GENERATE_TESTLIST tag can be used to enable (YES) or \n# disable (NO) the test list. This list is created by putting \\test \n# commands in the documentation.\n\nGENERATE_TESTLIST = NO\n\n# The GENERATE_BUGLIST tag can be used to enable (YES) or \n# disable (NO) the bug list. This list is created by putting \\bug \n# commands in the documentation.\n\nGENERATE_BUGLIST = NO\n\n# The GENERATE_DEPRECATEDLIST tag can be used to enable (YES) or \n# disable (NO) the deprecated list. This list is created by putting \n# \\deprecated commands in the documentation.\n\nGENERATE_DEPRECATEDLIST= NO\n\n# The ENABLED_SECTIONS tag can be used to enable conditional \n# documentation sections, marked by \\if sectionname ... \\endif.\n\nENABLED_SECTIONS = \n\n# The MAX_INITIALIZER_LINES tag determines the maximum number of lines \n# the initial value of a variable or define consists of for it to appear in \n# the documentation. If the initializer consists of more lines than specified \n# here it will be hidden. Use a value of 0 to hide initializers completely. \n# The appearance of the initializer of individual variables and defines in the \n# documentation can be controlled using \\showinitializer or \\hideinitializer \n# command in the documentation regardless of this setting.\n\nMAX_INITIALIZER_LINES = 30\n\n# Set the SHOW_USED_FILES tag to NO to disable the list of files generated \n# at the bottom of the documentation of classes and structs. If set to YES the \n# list will mention the files that were used to generate the documentation.\n\nSHOW_USED_FILES = YES\n\n# If the sources in your project are distributed over multiple directories \n# then setting the SHOW_DIRECTORIES tag to YES will show the directory hierarchy \n# in the documentation. The default is NO.\n\nSHOW_DIRECTORIES = YES\n\n# Set the SHOW_FILES tag to NO to disable the generation of the Files page. \n# This will remove the Files entry from the Quick Index and from the \n# Folder Tree View (if specified). The default is YES.\n\nSHOW_FILES = YES\n\n# Set the SHOW_NAMESPACES tag to NO to disable the generation of the \n# Namespaces page. \n# This will remove the Namespaces entry from the Quick Index \n# and from the Folder Tree View (if specified). The default is YES.\n\nSHOW_NAMESPACES = YES\n\n# The FILE_VERSION_FILTER tag can be used to specify a program or script that \n# doxygen should invoke to get the current version for each file (typically from \n# the version control system). Doxygen will invoke the program by executing (via \n# popen()) the command , where is the value of \n# the FILE_VERSION_FILTER tag, and is the name of an input file \n# provided by doxygen. Whatever the program writes to standard output \n# is used as the file version. See the manual for examples.\n\nFILE_VERSION_FILTER = \n\n# The LAYOUT_FILE tag can be used to specify a layout file which will be parsed by \n# doxygen. The layout file controls the global structure of the generated output files \n# in an output format independent way. The create the layout file that represents \n# doxygen's defaults, run doxygen with the -l option. You can optionally specify a \n# file name after the option, if omitted DoxygenLayout.xml will be used as the name \n# of the layout file.\n\nLAYOUT_FILE = \n\n#---------------------------------------------------------------------------\n# configuration options related to warning and progress messages\n#---------------------------------------------------------------------------\n\n# The QUIET tag can be used to turn on/off the messages that are generated \n# by doxygen. Possible values are YES and NO. If left blank NO is used.\n\nQUIET = NO\n\n# The WARNINGS tag can be used to turn on/off the warning messages that are \n# generated by doxygen. Possible values are YES and NO. If left blank \n# NO is used.\n\nWARNINGS = YES\n\n# If WARN_IF_UNDOCUMENTED is set to YES, then doxygen will generate warnings \n# for undocumented members. If EXTRACT_ALL is set to YES then this flag will \n# automatically be disabled.\n\nWARN_IF_UNDOCUMENTED = YES\n\n# If WARN_IF_DOC_ERROR is set to YES, doxygen will generate warnings for \n# potential errors in the documentation, such as not documenting some \n# parameters in a documented function, or documenting parameters that \n# don't exist or using markup commands wrongly.\n\nWARN_IF_DOC_ERROR = YES\n\n# This WARN_NO_PARAMDOC option can be abled to get warnings for \n# functions that are documented, but have no documentation for their parameters \n# or return value. If set to NO (the default) doxygen will only warn about \n# wrong or incomplete parameter documentation, but not about the absence of \n# documentation.\n\nWARN_NO_PARAMDOC = NO\n\n# The WARN_FORMAT tag determines the format of the warning messages that \n# doxygen can produce. The string should contain the $file, $line, and $text \n# tags, which will be replaced by the file and line number from which the \n# warning originated and the warning text. Optionally the format may contain \n# $version, which will be replaced by the version of the file (if it could \n# be obtained via FILE_VERSION_FILTER)\n\nWARN_FORMAT = \"$file:$line: $text\"\n\n# The WARN_LOGFILE tag can be used to specify a file to which warning \n# and error messages should be written. If left blank the output is written \n# to stderr.\n\nWARN_LOGFILE = doxygen.log \n\n#---------------------------------------------------------------------------\n# configuration options related to the input files\n#---------------------------------------------------------------------------\n\n# The INPUT tag can be used to specify the files and/or directories that contain \n# documented source files. You may enter file names like \"myfile.cpp\" or \n# directories like \"/usr/src/myproject\". Separate the files or directories \n# with spaces.\n\nINPUT = src \\\n demoapps \\\n toolkits\n\n# This tag can be used to specify the character encoding of the source files \n# that doxygen parses. Internally doxygen uses the UTF-8 encoding, which is \n# also the default input encoding. Doxygen uses libiconv (or the iconv built \n# into libc) for the transcoding. See http://www.gnu.org/software/libiconv for \n# the list of possible encodings.\n\nINPUT_ENCODING = UTF-8\n\n# If the value of the INPUT tag contains directories, you can use the \n# FILE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp \n# and *.h) to filter out the source-files in the directories. If left \n# blank the following patterns are tested: \n# *.c *.cc *.cxx *.cpp *.c++ *.java *.ii *.ixx *.ipp *.i++ *.inl *.h *.hh *.hxx \n# *.hpp *.h++ *.idl *.odl *.cs *.php *.php3 *.inc *.m *.mm *.py *.f90\n\nFILE_PATTERNS = *.hpp *.cpp *.dox \n\n# The RECURSIVE tag can be used to turn specify whether or not subdirectories \n# should be searched for input files as well. Possible values are YES and NO. \n# If left blank NO is used.\n\nRECURSIVE = YES\n\n# The EXCLUDE tag can be used to specify files and/or directories that should \n# excluded from the INPUT source files. This way you can easily exclude a \n# subdirectory from a directory tree whose root is specified with the INPUT tag.\n\nEXCLUDE = src/graphlab/matlab src/graphlab/gpu \n\n# The EXCLUDE_SYMLINKS tag can be used select whether or not files or \n# directories that are symbolic links (a Unix filesystem feature) are excluded \n# from the input.\n\nEXCLUDE_SYMLINKS = NO\n\n# If the value of the INPUT tag contains directories, you can use the \n# EXCLUDE_PATTERNS tag to specify one or more wildcard patterns to exclude \n# certain files from those directories. Note that the wildcards are matched \n# against the file with absolute path, so to exclude all test directories \n# for example use the pattern */test/*\n\nEXCLUDE_PATTERNS = */src/graphlab/rpc/*issue.hpp */src/graphlab/rpc/*dispatch.hpp */toolkits/*cpp */toolkits/*hpp\n\n# The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names \n# (namespaces, classes, functions, etc.) that should be excluded from the \n# output. The symbol name can be a fully qualified name, a word, or if the \n# wildcard * is used, a substring. Examples: ANamespace, AClass, \n# AClass::ANamespace, ANamespace::*Test\n\nEXCLUDE_SYMBOLS = graphlab::archive_detail::* graphlab::dc_impl::*\n\n# The EXAMPLE_PATH tag can be used to specify one or more files or \n# directories that contain example code fragments that are included (see \n# the \\include command).\n\nEXAMPLE_PATH = \n\n# If the value of the EXAMPLE_PATH tag contains directories, you can use the \n# EXAMPLE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp \n# and *.h) to filter out the source-files in the directories. If left \n# blank all files are included.\n\nEXAMPLE_PATTERNS = \n\n# If the EXAMPLE_RECURSIVE tag is set to YES then subdirectories will be \n# searched for input files to be used with the \\include or \\dontinclude \n# commands irrespective of the value of the RECURSIVE tag. \n# Possible values are YES and NO. If left blank NO is used.\n\nEXAMPLE_RECURSIVE = NO\n\n# The IMAGE_PATH tag can be used to specify one or more files or \n# directories that contain image that are included in the documentation (see \n# the \\image command).\n\nIMAGE_PATH = doc/images\n\n# The INPUT_FILTER tag can be used to specify a program that doxygen should \n# invoke to filter for each input file. Doxygen will invoke the filter program \n# by executing (via popen()) the command , where \n# is the value of the INPUT_FILTER tag, and is the name of an \n# input file. Doxygen will then use the output that the filter program writes \n# to standard output. \n# If FILTER_PATTERNS is specified, this tag will be \n# ignored.\n\nINPUT_FILTER = \n\n# The FILTER_PATTERNS tag can be used to specify filters on a per file pattern \n# basis. \n# Doxygen will compare the file name with each pattern and apply the \n# filter if there is a match. \n# The filters are a list of the form: \n# pattern=filter (like *.cpp=my_cpp_filter). See INPUT_FILTER for further \n# info on how filters are used. If FILTER_PATTERNS is empty, INPUT_FILTER \n# is applied to all files.\n\nFILTER_PATTERNS = \n\n# If the FILTER_SOURCE_FILES tag is set to YES, the input filter (if set using \n# INPUT_FILTER) will be used to filter the input files when producing source \n# files to browse (i.e. when SOURCE_BROWSER is set to YES).\n\nFILTER_SOURCE_FILES = NO\n\n#---------------------------------------------------------------------------\n# configuration options related to source browsing\n#---------------------------------------------------------------------------\n\n# If the SOURCE_BROWSER tag is set to YES then a list of source files will \n# be generated. Documented entities will be cross-referenced with these sources. \n# Note: To get rid of all source code in the generated output, make sure also \n# VERBATIM_HEADERS is set to NO.\n\nSOURCE_BROWSER = YES\n\n# Setting the INLINE_SOURCES tag to YES will include the body \n# of functions and classes directly in the documentation.\n\nINLINE_SOURCES = NO\n\n# Setting the STRIP_CODE_COMMENTS tag to YES (the default) will instruct \n# doxygen to hide any special comment blocks from generated source code \n# fragments. Normal C and C++ comments will always remain visible.\n\nSTRIP_CODE_COMMENTS = NO\n\n# If the REFERENCED_BY_RELATION tag is set to YES \n# then for each documented function all documented \n# functions referencing it will be listed.\n\nREFERENCED_BY_RELATION = NO\n\n# If the REFERENCES_RELATION tag is set to YES \n# then for each documented function all documented entities \n# called/used by that function will be listed.\n\nREFERENCES_RELATION = NO\n\n# If the REFERENCES_LINK_SOURCE tag is set to YES (the default) \n# and SOURCE_BROWSER tag is set to YES, then the hyperlinks from \n# functions in REFERENCES_RELATION and REFERENCED_BY_RELATION lists will \n# link to the source code. \n# Otherwise they will link to the documentation.\n\nREFERENCES_LINK_SOURCE = YES\n\n# If the USE_HTAGS tag is set to YES then the references to source code \n# will point to the HTML generated by the htags(1) tool instead of doxygen \n# built-in source browser. The htags tool is part of GNU's global source \n# tagging system (see http://www.gnu.org/software/global/global.html). You \n# will need version 4.8.6 or higher.\n\nUSE_HTAGS = NO\n\n# If the VERBATIM_HEADERS tag is set to YES (the default) then Doxygen \n# will generate a verbatim copy of the header file for each class for \n# which an include is specified. Set to NO to disable this.\n\nVERBATIM_HEADERS = NO\n\n#---------------------------------------------------------------------------\n# configuration options related to the alphabetical class index\n#---------------------------------------------------------------------------\n\n# If the ALPHABETICAL_INDEX tag is set to YES, an alphabetical index \n# of all compounds will be generated. Enable this if the project \n# contains a lot of classes, structs, unions or interfaces.\n\nALPHABETICAL_INDEX = YES\n\n# If the alphabetical index is enabled (see ALPHABETICAL_INDEX) then \n# the COLS_IN_ALPHA_INDEX tag can be used to specify the number of columns \n# in which this list will be split (can be a number in the range [1..20])\n\nCOLS_IN_ALPHA_INDEX = 5\n\n# In case all classes in a project start with a common prefix, all \n# classes will be put under the same header in the alphabetical index. \n# The IGNORE_PREFIX tag can be used to specify one or more prefixes that \n# should be ignored while generating the index headers.\n\nIGNORE_PREFIX = \n\n#---------------------------------------------------------------------------\n# configuration options related to the HTML output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_HTML tag is set to YES (the default) Doxygen will \n# generate HTML output.\n\nGENERATE_HTML = YES\n\n# The HTML_OUTPUT tag is used to specify where the HTML docs will be put. \n# If a relative path is entered the value of OUTPUT_DIRECTORY will be \n# put in front of it. If left blank `html' will be used as the default path.\n\nHTML_OUTPUT = html\n\n# The HTML_FILE_EXTENSION tag can be used to specify the file extension for \n# each generated HTML page (for example: .htm,.php,.asp). If it is left blank \n# doxygen will generate files with .html extension.\n\nHTML_FILE_EXTENSION = .html\n\n# The HTML_HEADER tag can be used to specify a personal HTML header for \n# each generated HTML page. If it is left blank doxygen will generate a \n# standard header.\n\nHTML_HEADER = \n\n# The HTML_FOOTER tag can be used to specify a personal HTML footer for \n# each generated HTML page. If it is left blank doxygen will generate a \n# standard footer.\n\nHTML_FOOTER = \n\n# The HTML_STYLESHEET tag can be used to specify a user-defined cascading \n# style sheet that is used by each HTML page. It can be used to \n# fine-tune the look of the HTML output. If the tag is left blank doxygen \n# will generate a default style sheet. Note that doxygen will try to copy \n# the style sheet file to the HTML output directory, so don't put your own \n# stylesheet in the HTML output directory as well, or it will be erased!\n\nHTML_STYLESHEET = \n\n# If the HTML_ALIGN_MEMBERS tag is set to YES, the members of classes, \n# files or namespaces will be aligned in HTML using tables. If set to \n# NO a bullet list will be used.\n\nHTML_ALIGN_MEMBERS = YES\n\n# If the HTML_DYNAMIC_SECTIONS tag is set to YES then the generated HTML \n# documentation will contain sections that can be hidden and shown after the \n# page has loaded. For this to work a browser that supports \n# JavaScript and DHTML is required (for instance Mozilla 1.0+, Firefox \n# Netscape 6.0+, Internet explorer 5.0+, Konqueror, or Safari).\n\nHTML_DYNAMIC_SECTIONS = YES\n\n# If the GENERATE_DOCSET tag is set to YES, additional index files \n# will be generated that can be used as input for Apple's Xcode 3 \n# integrated development environment, introduced with OSX 10.5 (Leopard). \n# To create a documentation set, doxygen will generate a Makefile in the \n# HTML output directory. Running make will produce the docset in that \n# directory and running \"make install\" will install the docset in \n# ~/Library/Developer/Shared/Documentation/DocSets so that Xcode will find \n# it at startup. \n# See http://developer.apple.com/tools/creatingdocsetswithdoxygen.html for more information.\n\nGENERATE_DOCSET = NO\n\n# When GENERATE_DOCSET tag is set to YES, this tag determines the name of the \n# feed. A documentation feed provides an umbrella under which multiple \n# documentation sets from a single provider (such as a company or product suite) \n# can be grouped.\n\nDOCSET_FEEDNAME = \"Doxygen generated docs\"\n\n# When GENERATE_DOCSET tag is set to YES, this tag specifies a string that \n# should uniquely identify the documentation set bundle. This should be a \n# reverse domain-name style string, e.g. com.mycompany.MyDocSet. Doxygen \n# will append .docset to the name.\n\nDOCSET_BUNDLE_ID = org.doxygen.Project\n\n# If the GENERATE_HTMLHELP tag is set to YES, additional index files \n# will be generated that can be used as input for tools like the \n# Microsoft HTML help workshop to generate a compiled HTML help file (.chm) \n# of the generated HTML documentation.\n\nGENERATE_HTMLHELP = NO\n\n# If the GENERATE_HTMLHELP tag is set to YES, the CHM_FILE tag can \n# be used to specify the file name of the resulting .chm file. You \n# can add a path in front of the file if the result should not be \n# written to the html output directory.\n\nCHM_FILE = \n\n# If the GENERATE_HTMLHELP tag is set to YES, the HHC_LOCATION tag can \n# be used to specify the location (absolute path including file name) of \n# the HTML help compiler (hhc.exe). If non-empty doxygen will try to run \n# the HTML help compiler on the generated index.hhp.\n\nHHC_LOCATION = \n\n# If the GENERATE_HTMLHELP tag is set to YES, the GENERATE_CHI flag \n# controls if a separate .chi index file is generated (YES) or that \n# it should be included in the master .chm file (NO).\n\nGENERATE_CHI = NO\n\n# If the GENERATE_HTMLHELP tag is set to YES, the CHM_INDEX_ENCODING \n# is used to encode HtmlHelp index (hhk), content (hhc) and project file \n# content.\n\nCHM_INDEX_ENCODING = \n\n# If the GENERATE_HTMLHELP tag is set to YES, the BINARY_TOC flag \n# controls whether a binary table of contents is generated (YES) or a \n# normal table of contents (NO) in the .chm file.\n\nBINARY_TOC = NO\n\n# The TOC_EXPAND flag can be set to YES to add extra items for group members \n# to the contents of the HTML help documentation and to the tree view.\n\nTOC_EXPAND = NO\n\n# If the GENERATE_QHP tag is set to YES and both QHP_NAMESPACE and QHP_VIRTUAL_FOLDER \n# are set, an additional index file will be generated that can be used as input for \n# Qt's qhelpgenerator to generate a Qt Compressed Help (.qch) of the generated \n# HTML documentation.\n\nGENERATE_QHP = NO\n\n# If the QHG_LOCATION tag is specified, the QCH_FILE tag can \n# be used to specify the file name of the resulting .qch file. \n# The path specified is relative to the HTML output folder.\n\nQCH_FILE = \n\n# The QHP_NAMESPACE tag specifies the namespace to use when generating \n# Qt Help Project output. For more information please see \n# http://doc.trolltech.com/qthelpproject.html#namespace\n\nQHP_NAMESPACE = \n\n# The QHP_VIRTUAL_FOLDER tag specifies the namespace to use when generating \n# Qt Help Project output. For more information please see \n# http://doc.trolltech.com/qthelpproject.html#virtual-folders\n\nQHP_VIRTUAL_FOLDER = doc\n\n# If QHP_CUST_FILTER_NAME is set, it specifies the name of a custom filter to add. \n# For more information please see \n# http://doc.trolltech.com/qthelpproject.html#custom-filters\n\nQHP_CUST_FILTER_NAME = \n\n# The QHP_CUST_FILT_ATTRS tag specifies the list of the attributes of the custom filter to add.For more information please see \n# Qt Help Project / Custom Filters.\n\nQHP_CUST_FILTER_ATTRS = \n\n# The QHP_SECT_FILTER_ATTRS tag specifies the list of the attributes this project's \n# filter section matches. \n# Qt Help Project / Filter Attributes.\n\nQHP_SECT_FILTER_ATTRS = \n\n# If the GENERATE_QHP tag is set to YES, the QHG_LOCATION tag can \n# be used to specify the location of Qt's qhelpgenerator. \n# If non-empty doxygen will try to run qhelpgenerator on the generated \n# .qhp file.\n\nQHG_LOCATION = \n\n# The DISABLE_INDEX tag can be used to turn on/off the condensed index at \n# top of each HTML page. The value NO (the default) enables the index and \n# the value YES disables it.\n\nDISABLE_INDEX = NO\n\n# This tag can be used to set the number of enum values (range [1..20]) \n# that doxygen will group on one line in the generated HTML documentation.\n\nENUM_VALUES_PER_LINE = 4\n\n# The GENERATE_TREEVIEW tag is used to specify whether a tree-like index \n# structure should be generated to display hierarchical information. \n# If the tag value is set to FRAME, a side panel will be generated \n# containing a tree-like index structure (just like the one that \n# is generated for HTML Help). For this to work a browser that supports \n# JavaScript, DHTML, CSS and frames is required (for instance Mozilla 1.0+, \n# Netscape 6.0+, Internet explorer 5.0+, or Konqueror). Windows users are \n# probably better off using the HTML help feature. Other possible values \n# for this tag are: HIERARCHIES, which will generate the Groups, Directories, \n# and Class Hierarchy pages using a tree view instead of an ordered list; \n# ALL, which combines the behavior of FRAME and HIERARCHIES; and NONE, which \n# disables this behavior completely. For backwards compatibility with previous \n# releases of Doxygen, the values YES and NO are equivalent to FRAME and NONE \n# respectively.\n\nGENERATE_TREEVIEW = YES\n\n# If the treeview is enabled (see GENERATE_TREEVIEW) then this tag can be \n# used to set the initial width (in pixels) of the frame in which the tree \n# is shown.\n\nTREEVIEW_WIDTH = 250\n\n# Use this tag to change the font size of Latex formulas included \n# as images in the HTML documentation. The default is 10. Note that \n# when you change the font size after a successful doxygen run you need \n# to manually remove any form_*.png images from the HTML output directory \n# to force them to be regenerated.\n\nFORMULA_FONTSIZE = 10\n\n#---------------------------------------------------------------------------\n# configuration options related to the LaTeX output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_LATEX tag is set to YES (the default) Doxygen will \n# generate Latex output.\n\nGENERATE_LATEX = NO\n\n# The LATEX_OUTPUT tag is used to specify where the LaTeX docs will be put. \n# If a relative path is entered the value of OUTPUT_DIRECTORY will be \n# put in front of it. If left blank `latex' will be used as the default path.\n\nLATEX_OUTPUT = latex\n\n# The LATEX_CMD_NAME tag can be used to specify the LaTeX command name to be \n# invoked. If left blank `latex' will be used as the default command name.\n\nLATEX_CMD_NAME = latex\n\n# The MAKEINDEX_CMD_NAME tag can be used to specify the command name to \n# generate index for LaTeX. If left blank `makeindex' will be used as the \n# default command name.\n\nMAKEINDEX_CMD_NAME = makeindex\n\n# If the COMPACT_LATEX tag is set to YES Doxygen generates more compact \n# LaTeX documents. This may be useful for small projects and may help to \n# save some trees in general.\n\nCOMPACT_LATEX = NO\n\n# The PAPER_TYPE tag can be used to set the paper type that is used \n# by the printer. Possible values are: a4, a4wide, letter, legal and \n# executive. If left blank a4wide will be used.\n\nPAPER_TYPE = a4wide\n\n# The EXTRA_PACKAGES tag can be to specify one or more names of LaTeX \n# packages that should be included in the LaTeX output.\n\nEXTRA_PACKAGES = \n\n# The LATEX_HEADER tag can be used to specify a personal LaTeX header for \n# the generated latex document. The header should contain everything until \n# the first chapter. If it is left blank doxygen will generate a \n# standard header. Notice: only use this tag if you know what you are doing!\n\nLATEX_HEADER = \n\n# If the PDF_HYPERLINKS tag is set to YES, the LaTeX that is generated \n# is prepared for conversion to pdf (using ps2pdf). The pdf file will \n# contain links (just like the HTML output) instead of page references \n# This makes the output suitable for online browsing using a pdf viewer.\n\nPDF_HYPERLINKS = YES\n\n# If the USE_PDFLATEX tag is set to YES, pdflatex will be used instead of \n# plain latex in the generated Makefile. Set this option to YES to get a \n# higher quality PDF documentation.\n\nUSE_PDFLATEX = YES\n\n# If the LATEX_BATCHMODE tag is set to YES, doxygen will add the \\\\batchmode. \n# command to the generated LaTeX files. This will instruct LaTeX to keep \n# running if errors occur, instead of asking the user for help. \n# This option is also used when generating formulas in HTML.\n\nLATEX_BATCHMODE = NO\n\n# If LATEX_HIDE_INDICES is set to YES then doxygen will not \n# include the index chapters (such as File Index, Compound Index, etc.) \n# in the output.\n\nLATEX_HIDE_INDICES = NO\n\n#---------------------------------------------------------------------------\n# configuration options related to the RTF output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_RTF tag is set to YES Doxygen will generate RTF output \n# The RTF output is optimized for Word 97 and may not look very pretty with \n# other RTF readers or editors.\n\nGENERATE_RTF = NO\n\n# The RTF_OUTPUT tag is used to specify where the RTF docs will be put. \n# If a relative path is entered the value of OUTPUT_DIRECTORY will be \n# put in front of it. If left blank `rtf' will be used as the default path.\n\nRTF_OUTPUT = rtf\n\n# If the COMPACT_RTF tag is set to YES Doxygen generates more compact \n# RTF documents. This may be useful for small projects and may help to \n# save some trees in general.\n\nCOMPACT_RTF = NO\n\n# If the RTF_HYPERLINKS tag is set to YES, the RTF that is generated \n# will contain hyperlink fields. The RTF file will \n# contain links (just like the HTML output) instead of page references. \n# This makes the output suitable for online browsing using WORD or other \n# programs which support those fields. \n# Note: wordpad (write) and others do not support links.\n\nRTF_HYPERLINKS = NO\n\n# Load stylesheet definitions from file. Syntax is similar to doxygen's \n# config file, i.e. a series of assignments. You only have to provide \n# replacements, missing definitions are set to their default value.\n\nRTF_STYLESHEET_FILE = \n\n# Set optional variables used in the generation of an rtf document. \n# Syntax is similar to doxygen's config file.\n\nRTF_EXTENSIONS_FILE = \n\n#---------------------------------------------------------------------------\n# configuration options related to the man page output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_MAN tag is set to YES (the default) Doxygen will \n# generate man pages\n\nGENERATE_MAN = NO\n\n# The MAN_OUTPUT tag is used to specify where the man pages will be put. \n# If a relative path is entered the value of OUTPUT_DIRECTORY will be \n# put in front of it. If left blank `man' will be used as the default path.\n\nMAN_OUTPUT = man\n\n# The MAN_EXTENSION tag determines the extension that is added to \n# the generated man pages (default is the subroutine's section .3)\n\nMAN_EXTENSION = .3\n\n# If the MAN_LINKS tag is set to YES and Doxygen generates man output, \n# then it will generate one additional man file for each entity \n# documented in the real man page(s). These additional files \n# only source the real man page, but without them the man command \n# would be unable to find the correct page. The default is NO.\n\nMAN_LINKS = NO\n\n#---------------------------------------------------------------------------\n# configuration options related to the XML output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_XML tag is set to YES Doxygen will \n# generate an XML file that captures the structure of \n# the code including all documentation.\n\nGENERATE_XML = NO\n\n# The XML_OUTPUT tag is used to specify where the XML pages will be put. \n# If a relative path is entered the value of OUTPUT_DIRECTORY will be \n# put in front of it. If left blank `xml' will be used as the default path.\n\nXML_OUTPUT = xml\n\n# The XML_SCHEMA tag can be used to specify an XML schema, \n# which can be used by a validating XML parser to check the \n# syntax of the XML files.\n\nXML_SCHEMA = \n\n# The XML_DTD tag can be used to specify an XML DTD, \n# which can be used by a validating XML parser to check the \n# syntax of the XML files.\n\nXML_DTD = \n\n# If the XML_PROGRAMLISTING tag is set to YES Doxygen will \n# dump the program listings (including syntax highlighting \n# and cross-referencing information) to the XML output. Note that \n# enabling this will significantly increase the size of the XML output.\n\nXML_PROGRAMLISTING = YES\n\n#---------------------------------------------------------------------------\n# configuration options for the AutoGen Definitions output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_AUTOGEN_DEF tag is set to YES Doxygen will \n# generate an AutoGen Definitions (see autogen.sf.net) file \n# that captures the structure of the code including all \n# documentation. Note that this feature is still experimental \n# and incomplete at the moment.\n\nGENERATE_AUTOGEN_DEF = NO\n\n#---------------------------------------------------------------------------\n# configuration options related to the Perl module output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_PERLMOD tag is set to YES Doxygen will \n# generate a Perl module file that captures the structure of \n# the code including all documentation. Note that this \n# feature is still experimental and incomplete at the \n# moment.\n\nGENERATE_PERLMOD = NO\n\n# If the PERLMOD_LATEX tag is set to YES Doxygen will generate \n# the necessary Makefile rules, Perl scripts and LaTeX code to be able \n# to generate PDF and DVI output from the Perl module output.\n\nPERLMOD_LATEX = NO\n\n# If the PERLMOD_PRETTY tag is set to YES the Perl module output will be \n# nicely formatted so it can be parsed by a human reader. \n# This is useful \n# if you want to understand what is going on. \n# On the other hand, if this \n# tag is set to NO the size of the Perl module output will be much smaller \n# and Perl will parse it just the same.\n\nPERLMOD_PRETTY = YES\n\n# The names of the make variables in the generated doxyrules.make file \n# are prefixed with the string contained in PERLMOD_MAKEVAR_PREFIX. \n# This is useful so different doxyrules.make files included by the same \n# Makefile don't overwrite each other's variables.\n\nPERLMOD_MAKEVAR_PREFIX = \n\n#---------------------------------------------------------------------------\n# Configuration options related to the preprocessor \n#---------------------------------------------------------------------------\n\n# If the ENABLE_PREPROCESSING tag is set to YES (the default) Doxygen will \n# evaluate all C-preprocessor directives found in the sources and include \n# files.\n\nENABLE_PREPROCESSING = YES \n\n# If the MACRO_EXPANSION tag is set to YES Doxygen will expand all macro \n# names in the source code. If set to NO (the default) only conditional \n# compilation will be performed. Macro expansion can be done in a controlled \n# way by setting EXPAND_ONLY_PREDEF to YES.\n\nMACRO_EXPANSION = NO\n\n# If the EXPAND_ONLY_PREDEF and MACRO_EXPANSION tags are both set to YES \n# then the macro expansion is limited to the macros specified with the \n# PREDEFINED and EXPAND_AS_DEFINED tags.\n\nEXPAND_ONLY_PREDEF = YES\n\n# If the SEARCH_INCLUDES tag is set to YES (the default) the includes files \n# in the INCLUDE_PATH (see below) will be search if a #include is found.\n\nSEARCH_INCLUDES = YES\n\n# The INCLUDE_PATH tag can be used to specify one or more directories that \n# contain include files that are not input files but should be processed by \n# the preprocessor.\n\nINCLUDE_PATH = \n\n# You can use the INCLUDE_FILE_PATTERNS tag to specify one or more wildcard \n# patterns (like *.h and *.hpp) to filter out the header-files in the \n# directories. If left blank, the patterns specified with FILE_PATTERNS will \n# be used.\n\nINCLUDE_FILE_PATTERNS = \n\n# The PREDEFINED tag can be used to specify one or more macro names that \n# are defined before the preprocessor is started (similar to the -D option of \n# gcc). The argument of the tag is a list of macros of the form: name \n# or name=definition (no spaces). If the definition and the = are \n# omitted =1 is assumed. To prevent a macro definition from being \n# undefined via #undef or recursively expanded use the := operator \n# instead of the = operator.\n\nPREDEFINED = DOXYGEN_DOCUMENTATION GRAPHLAB_SERIALIZE_HPP \n\n# If the MACRO_EXPANSION and EXPAND_ONLY_PREDEF tags are set to YES then \n# this tag can be used to specify a list of macro names that should be expanded. \n# The macro definition that is found in the sources will be used. \n# Use the PREDEFINED tag if you want to use a different macro definition.\n\nEXPAND_AS_DEFINED = RPC_DEFAULT_NUMHANDLERTHREADS RPC_DEFAULT_COMMTYPE \n\n# If the SKIP_FUNCTION_MACROS tag is set to YES (the default) then \n# doxygen's preprocessor will remove all function-like macros that are alone \n# on a line, have an all uppercase name, and do not end with a semicolon. Such \n# function macros are typically used for boiler-plate code, and will confuse \n# the parser if not removed.\n\nSKIP_FUNCTION_MACROS = YES\n\n#---------------------------------------------------------------------------\n# Configuration::additions related to external references \n#---------------------------------------------------------------------------\n\n# The TAGFILES option can be used to specify one or more tagfiles. \n# Optionally an initial location of the external documentation \n# can be added for each tagfile. The format of a tag file without \n# this location is as follows: \n# \n# TAGFILES = file1 file2 ... \n# Adding location for the tag files is done as follows: \n# \n# TAGFILES = file1=loc1 \"file2 = loc2\" ... \n# where \"loc1\" and \"loc2\" can be relative or absolute paths or \n# URLs. If a location is present for each tag, the installdox tool \n# does not have to be run to correct the links. \n# Note that each tag file must have a unique name \n# (where the name does NOT include the path) \n# If a tag file is not located in the directory in which doxygen \n# is run, you must also specify the path to the tagfile here.\n\nTAGFILES = \n\n# When a file name is specified after GENERATE_TAGFILE, doxygen will create \n# a tag file that is based on the input files it reads.\n\nGENERATE_TAGFILE = \n\n# If the ALLEXTERNALS tag is set to YES all external classes will be listed \n# in the class index. If set to NO only the inherited external classes \n# will be listed.\n\nALLEXTERNALS = NO\n\n# If the EXTERNAL_GROUPS tag is set to YES all external groups will be listed \n# in the modules index. If set to NO, only the current project's groups will \n# be listed.\n\nEXTERNAL_GROUPS = YES\n\n# The PERL_PATH should be the absolute path and name of the perl script \n# interpreter (i.e. the result of `which perl').\n\nPERL_PATH = /usr/bin/perl\n\n#---------------------------------------------------------------------------\n# Configuration options related to the dot tool \n#---------------------------------------------------------------------------\n\n# If the CLASS_DIAGRAMS tag is set to YES (the default) Doxygen will \n# generate a inheritance diagram (in HTML, RTF and LaTeX) for classes with base \n# or super classes. Setting the tag to NO turns the diagrams off. Note that \n# this option is superseded by the HAVE_DOT option below. This is only a \n# fallback. It is recommended to install and use dot, since it yields more \n# powerful graphs.\n\nCLASS_DIAGRAMS = NO\n\n# You can define message sequence charts within doxygen comments using the \\msc \n# command. Doxygen will then run the mscgen tool (see \n# http://www.mcternan.me.uk/mscgen/) to produce the chart and insert it in the \n# documentation. The MSCGEN_PATH tag allows you to specify the directory where \n# the mscgen tool resides. If left empty the tool is assumed to be found in the \n# default search path.\n\nMSCGEN_PATH = \n\n# If set to YES, the inheritance and collaboration graphs will hide \n# inheritance and usage relations if the target is undocumented \n# or is not a class.\n\nHIDE_UNDOC_RELATIONS = YES\n\n# If you set the HAVE_DOT tag to YES then doxygen will assume the dot tool is \n# available from the path. This tool is part of Graphviz, a graph visualization \n# toolkit from AT&T and Lucent Bell Labs. The other options in this section \n# have no effect if this option is set to NO (the default)\n\nHAVE_DOT = NO\n\n# By default doxygen will write a font called FreeSans.ttf to the output \n# directory and reference it in all dot files that doxygen generates. This \n# font does not include all possible unicode characters however, so when you need \n# these (or just want a differently looking font) you can specify the font name \n# using DOT_FONTNAME. You need need to make sure dot is able to find the font, \n# which can be done by putting it in a standard location or by setting the \n# DOTFONTPATH environment variable or by setting DOT_FONTPATH to the directory \n# containing the font.\n\nDOT_FONTNAME = FreeSans\n\n# The DOT_FONTSIZE tag can be used to set the size of the font of dot graphs. \n# The default size is 10pt.\n\nDOT_FONTSIZE = 10\n\n# By default doxygen will tell dot to use the output directory to look for the \n# FreeSans.ttf font (which doxygen will put there itself). If you specify a \n# different font using DOT_FONTNAME you can set the path where dot \n# can find it using this tag.\n\nDOT_FONTPATH = \n\n# If the CLASS_GRAPH and HAVE_DOT tags are set to YES then doxygen \n# will generate a graph for each documented class showing the direct and \n# indirect inheritance relations. Setting this tag to YES will force the \n# the CLASS_DIAGRAMS tag to NO.\n\nCLASS_GRAPH = NO\n\n# If the COLLABORATION_GRAPH and HAVE_DOT tags are set to YES then doxygen \n# will generate a graph for each documented class showing the direct and \n# indirect implementation dependencies (inheritance, containment, and \n# class references variables) of the class with other documented classes.\n\nCOLLABORATION_GRAPH = NO\n\n# If the GROUP_GRAPHS and HAVE_DOT tags are set to YES then doxygen \n# will generate a graph for groups, showing the direct groups dependencies\n\nGROUP_GRAPHS = NO\n\n# If the UML_LOOK tag is set to YES doxygen will generate inheritance and \n# collaboration diagrams in a style similar to the OMG's Unified Modeling \n# Language.\n\nUML_LOOK = NO\n\n# If set to YES, the inheritance and collaboration graphs will show the \n# relations between templates and their instances.\n\nTEMPLATE_RELATIONS = NO\n\n# If the ENABLE_PREPROCESSING, SEARCH_INCLUDES, INCLUDE_GRAPH, and HAVE_DOT \n# tags are set to YES then doxygen will generate a graph for each documented \n# file showing the direct and indirect include dependencies of the file with \n# other documented files.\n\nINCLUDE_GRAPH = NO\n\n# If the ENABLE_PREPROCESSING, SEARCH_INCLUDES, INCLUDED_BY_GRAPH, and \n# HAVE_DOT tags are set to YES then doxygen will generate a graph for each \n# documented header file showing the documented files that directly or \n# indirectly include this file.\n\nINCLUDED_BY_GRAPH = NO\n\n# If the CALL_GRAPH and HAVE_DOT options are set to YES then \n# doxygen will generate a call dependency graph for every global function \n# or class method. Note that enabling this option will significantly increase \n# the time of a run. So in most cases it will be better to enable call graphs \n# for selected functions only using the \\callgraph command.\n\nCALL_GRAPH = NO\n\n# If the CALLER_GRAPH and HAVE_DOT tags are set to YES then \n# doxygen will generate a caller dependency graph for every global function \n# or class method. Note that enabling this option will significantly increase \n# the time of a run. So in most cases it will be better to enable caller \n# graphs for selected functions only using the \\callergraph command.\n\nCALLER_GRAPH = NO\n\n# If the GRAPHICAL_HIERARCHY and HAVE_DOT tags are set to YES then doxygen \n# will graphical hierarchy of all classes instead of a textual one.\n\nGRAPHICAL_HIERARCHY = YES\n\n# If the DIRECTORY_GRAPH, SHOW_DIRECTORIES and HAVE_DOT tags are set to YES \n# then doxygen will show the dependencies a directory has on other directories \n# in a graphical way. The dependency relations are determined by the #include \n# relations between the files in the directories.\n\nDIRECTORY_GRAPH = YES\n\n# The DOT_IMAGE_FORMAT tag can be used to set the image format of the images \n# generated by dot. Possible values are png, jpg, or gif \n# If left blank png will be used.\n\nDOT_IMAGE_FORMAT = png\n\n# The tag DOT_PATH can be used to specify the path where the dot tool can be \n# found. If left blank, it is assumed the dot tool can be found in the path.\n\nDOT_PATH = \n\n# The DOTFILE_DIRS tag can be used to specify one or more directories that \n# contain dot files that are included in the documentation (see the \n# \\dotfile command).\n\nDOTFILE_DIRS = \n\n# The DOT_GRAPH_MAX_NODES tag can be used to set the maximum number of \n# nodes that will be shown in the graph. If the number of nodes in a graph \n# becomes larger than this value, doxygen will truncate the graph, which is \n# visualized by representing a node as a red box. Note that doxygen if the \n# number of direct children of the root node in a graph is already larger than \n# DOT_GRAPH_MAX_NODES then the graph will not be shown at all. Also note \n# that the size of a graph can be further restricted by MAX_DOT_GRAPH_DEPTH.\n\nDOT_GRAPH_MAX_NODES = 50\n\n# The MAX_DOT_GRAPH_DEPTH tag can be used to set the maximum depth of the \n# graphs generated by dot. A depth value of 3 means that only nodes reachable \n# from the root by following a path via at most 3 edges will be shown. Nodes \n# that lay further from the root node will be omitted. Note that setting this \n# option to 1 or 2 may greatly reduce the computation time needed for large \n# code bases. Also note that the size of a graph can be further restricted by \n# DOT_GRAPH_MAX_NODES. Using a depth of 0 means no depth restriction.\n\nMAX_DOT_GRAPH_DEPTH = 0\n\n# Set the DOT_TRANSPARENT tag to YES to generate images with a transparent \n# background. This is disabled by default, because dot on Windows does not \n# seem to support this out of the box. Warning: Depending on the platform used, \n# enabling this option may lead to badly anti-aliased labels on the edges of \n# a graph (i.e. they become hard to read).\n\nDOT_TRANSPARENT = NO\n\n# Set the DOT_MULTI_TARGETS tag to YES allow dot to generate multiple output \n# files in one run (i.e. multiple -o and -T options on the command line). This \n# makes dot run faster, but since only newer versions of dot (>1.8.10) \n# support this, this feature is disabled by default.\n\nDOT_MULTI_TARGETS = NO\n\n# If the GENERATE_LEGEND tag is set to YES (the default) Doxygen will \n# generate a legend page explaining the meaning of the various boxes and \n# arrows in the dot generated graphs.\n\nGENERATE_LEGEND = YES\n\n# If the DOT_CLEANUP tag is set to YES (the default) Doxygen will \n# remove the intermediate dot files that are used to generate \n# the various graphs.\n\nDOT_CLEANUP = YES\n\n#---------------------------------------------------------------------------\n# Options related to the search engine\n#---------------------------------------------------------------------------\n\n# The SEARCHENGINE tag specifies whether or not a search engine should be \n# used. If set to NO the values of all tags below this one will be ignored.\n\nSEARCHENGINE = YES\n" }, { "path": "Doxyfile_internal", "content": "# Doxyfile 1.5.8\n\n# This file describes the settings to be used by the documentation system\n# doxygen (www.doxygen.org) for a project\n#\n# All text after a hash (#) is considered a comment and will be ignored\n# The format is:\n# TAG = value [value, ...]\n# For lists items can also be appended using:\n# TAG += value [value, ...]\n# Values that contain spaces should be placed between quotes (\" \")\n\n#---------------------------------------------------------------------------\n# Project related configuration options\n#---------------------------------------------------------------------------\n\n# This tag specifies the encoding used for all characters in the config file \n# that follow. The default is UTF-8 which is also the encoding used for all \n# text before the first occurrence of this tag. Doxygen uses libiconv (or the \n# iconv built into libc) for the transcoding. See \n# http://www.gnu.org/software/libiconv for the list of possible encodings.\n\nDOXYFILE_ENCODING = UTF-8\n\n# The PROJECT_NAME tag is a single word (or a sequence of words surrounded \n# by quotes) that should identify the project.\n\nPROJECT_NAME = \"GraphLab: Distributed Graph-Parallel API\"\n\n# The PROJECT_NUMBER tag can be used to enter a project or revision number. \n# This could be handy for archiving the generated documentation or \n# if some version control system is used.\n\nPROJECT_NUMBER = 2.2\n\n# The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) \n# base path where the generated documentation will be put. \n# If a relative path is entered, it will be relative to the location \n# where doxygen was started. If left blank the current directory will be used.\n\nOUTPUT_DIRECTORY = doc/doxygen_internal\n\n# If the CREATE_SUBDIRS tag is set to YES, then doxygen will create \n# 4096 sub-directories (in 2 levels) under the output directory of each output \n# format and will distribute the generated files over these directories. \n# Enabling this option can be useful when feeding doxygen a huge amount of \n# source files, where putting all generated files in the same directory would \n# otherwise cause performance problems for the file system.\n\nCREATE_SUBDIRS = NO\n\n# The OUTPUT_LANGUAGE tag is used to specify the language in which all \n# documentation generated by doxygen is written. Doxygen will use this \n# information to generate all constant output in the proper language. \n# The default language is English, other supported languages are: \n# Afrikaans, Arabic, Brazilian, Catalan, Chinese, Chinese-Traditional, \n# Croatian, Czech, Danish, Dutch, Farsi, Finnish, French, German, Greek, \n# Hungarian, Italian, Japanese, Japanese-en (Japanese with English messages), \n# Korean, Korean-en, Lithuanian, Norwegian, Macedonian, Persian, Polish, \n# Portuguese, Romanian, Russian, Serbian, Serbian-Cyrilic, Slovak, Slovene, \n# Spanish, Swedish, and Ukrainian.\n\nOUTPUT_LANGUAGE = English\n\n# If the BRIEF_MEMBER_DESC tag is set to YES (the default) Doxygen will \n# include brief member descriptions after the members that are listed in \n# the file and class documentation (similar to JavaDoc). \n# Set to NO to disable this.\n\nBRIEF_MEMBER_DESC = YES\n\n# If the REPEAT_BRIEF tag is set to YES (the default) Doxygen will prepend \n# the brief description of a member or function before the detailed description. \n# Note: if both HIDE_UNDOC_MEMBERS and BRIEF_MEMBER_DESC are set to NO, the \n# brief descriptions will be completely suppressed.\n\nREPEAT_BRIEF = YES\n\n# This tag implements a quasi-intelligent brief description abbreviator \n# that is used to form the text in various listings. Each string \n# in this list, if found as the leading text of the brief description, will be \n# stripped from the text and the result after processing the whole list, is \n# used as the annotated text. Otherwise, the brief description is used as-is. \n# If left blank, the following values are used (\"$name\" is automatically \n# replaced with the name of the entity): \"The $name class\" \"The $name widget\" \n# \"The $name file\" \"is\" \"provides\" \"specifies\" \"contains\" \n# \"represents\" \"a\" \"an\" \"the\"\n\nABBREVIATE_BRIEF = \n\n# If the ALWAYS_DETAILED_SEC and REPEAT_BRIEF tags are both set to YES then \n# Doxygen will generate a detailed section even if there is only a brief \n# description.\n\nALWAYS_DETAILED_SEC = NO\n\n# If the INLINE_INHERITED_MEMB tag is set to YES, doxygen will show all \n# inherited members of a class in the documentation of that class as if those \n# members were ordinary class members. Constructors, destructors and assignment \n# operators of the base classes will not be shown.\n\nINLINE_INHERITED_MEMB = YES\n\n# If the FULL_PATH_NAMES tag is set to YES then Doxygen will prepend the full \n# path before files name in the file list and in the header files. If set \n# to NO the shortest path that makes the file name unique will be used.\n\nFULL_PATH_NAMES = YES\n\n# If the FULL_PATH_NAMES tag is set to YES then the STRIP_FROM_PATH tag \n# can be used to strip a user-defined part of the path. Stripping is \n# only done if one of the specified strings matches the left-hand part of \n# the path. The tag can be used to show relative paths in the file list. \n# If left blank the directory from which doxygen is run is used as the \n# path to strip.\n\nSTRIP_FROM_PATH = src/ \n\n# The STRIP_FROM_INC_PATH tag can be used to strip a user-defined part of \n# the path mentioned in the documentation of a class, which tells \n# the reader which header file to include in order to use a class. \n# If left blank only the name of the header file containing the class \n# definition is used. Otherwise one should specify the include paths that \n# are normally passed to the compiler using the -I flag.\n\nSTRIP_FROM_INC_PATH = src/ \n\n# If the SHORT_NAMES tag is set to YES, doxygen will generate much shorter \n# (but less readable) file names. This can be useful is your file systems \n# doesn't support long names like on DOS, Mac, or CD-ROM.\n\nSHORT_NAMES = NO\n\n# If the JAVADOC_AUTOBRIEF tag is set to YES then Doxygen \n# will interpret the first line (until the first dot) of a JavaDoc-style \n# comment as the brief description. If set to NO, the JavaDoc \n# comments will behave just like regular Qt-style comments \n# (thus requiring an explicit @brief command for a brief description.)\n\nJAVADOC_AUTOBRIEF = NO\n\n# If the QT_AUTOBRIEF tag is set to YES then Doxygen will \n# interpret the first line (until the first dot) of a Qt-style \n# comment as the brief description. If set to NO, the comments \n# will behave just like regular Qt-style comments (thus requiring \n# an explicit \\brief command for a brief description.)\n\nQT_AUTOBRIEF = NO\n\n# The MULTILINE_CPP_IS_BRIEF tag can be set to YES to make Doxygen \n# treat a multi-line C++ special comment block (i.e. a block of //! or /// \n# comments) as a brief description. This used to be the default behaviour. \n# The new default is to treat a multi-line C++ comment block as a detailed \n# description. Set this tag to YES if you prefer the old behaviour instead.\n\nMULTILINE_CPP_IS_BRIEF = NO\n\n# If the INHERIT_DOCS tag is set to YES (the default) then an undocumented \n# member inherits the documentation from any documented member that it \n# re-implements.\n\nINHERIT_DOCS = YES\n\n# If the SEPARATE_MEMBER_PAGES tag is set to YES, then doxygen will produce \n# a new page for each member. If set to NO, the documentation of a member will \n# be part of the file/class/namespace that contains it.\n\nSEPARATE_MEMBER_PAGES = NO\n\n# The TAB_SIZE tag can be used to set the number of spaces in a tab. \n# Doxygen uses this value to replace tabs by spaces in code fragments.\n\nTAB_SIZE = 2\n\n# This tag can be used to specify a number of aliases that acts \n# as commands in the documentation. An alias has the form \"name=value\". \n# For example adding \"sideeffect=\\par Side Effects:\\n\" will allow you to \n# put the command \\sideeffect (or @sideeffect) in the documentation, which \n# will result in a user-defined paragraph with heading \"Side Effects:\". \n# You can put \\n's in the value part of an alias to insert newlines.\n\nALIASES = \n\n# Set the OPTIMIZE_OUTPUT_FOR_C tag to YES if your project consists of C \n# sources only. Doxygen will then generate output that is more tailored for C. \n# For instance, some of the names that are used will be different. The list \n# of all members will be omitted, etc.\n\nOPTIMIZE_OUTPUT_FOR_C = NO\n\n# Set the OPTIMIZE_OUTPUT_JAVA tag to YES if your project consists of Java \n# sources only. Doxygen will then generate output that is more tailored for \n# Java. For instance, namespaces will be presented as packages, qualified \n# scopes will look different, etc.\n\nOPTIMIZE_OUTPUT_JAVA = NO\n\n# Set the OPTIMIZE_FOR_FORTRAN tag to YES if your project consists of Fortran \n# sources only. Doxygen will then generate output that is more tailored for \n# Fortran.\n\nOPTIMIZE_FOR_FORTRAN = NO\n\n# Set the OPTIMIZE_OUTPUT_VHDL tag to YES if your project consists of VHDL \n# sources. Doxygen will then generate output that is tailored for \n# VHDL.\n\nOPTIMIZE_OUTPUT_VHDL = NO\n\n# Doxygen selects the parser to use depending on the extension of the files it parses. \n# With this tag you can assign which parser to use for a given extension. \n# Doxygen has a built-in mapping, but you can override or extend it using this tag. \n# The format is ext=language, where ext is a file extension, and language is one of \n# the parsers supported by doxygen: IDL, Java, Javascript, C#, C, C++, D, PHP, \n# Objective-C, Python, Fortran, VHDL, C, C++. For instance to make doxygen treat \n# .inc files as Fortran files (default is PHP), and .f files as C (default is Fortran), \n# use: inc=Fortran f=C\n\nEXTENSION_MAPPING = \n\n# If you use STL classes (i.e. std::string, std::vector, etc.) but do not want \n# to include (a tag file for) the STL sources as input, then you should \n# set this tag to YES in order to let doxygen match functions declarations and \n# definitions whose arguments contain STL classes (e.g. func(std::string); v.s. \n# func(std::string) {}). This also make the inheritance and collaboration \n# diagrams that involve STL classes more complete and accurate.\n\nBUILTIN_STL_SUPPORT = YES\n\n# If you use Microsoft's C++/CLI language, you should set this option to YES to \n# enable parsing support.\n\nCPP_CLI_SUPPORT = NO\n\n# Set the SIP_SUPPORT tag to YES if your project consists of sip sources only. \n# Doxygen will parse them like normal C++ but will assume all classes use public \n# instead of private inheritance when no explicit protection keyword is present.\n\nSIP_SUPPORT = NO\n\n# For Microsoft's IDL there are propget and propput attributes to indicate getter \n# and setter methods for a property. Setting this option to YES (the default) \n# will make doxygen to replace the get and set methods by a property in the \n# documentation. This will only work if the methods are indeed getting or \n# setting a simple type. If this is not the case, or you want to show the \n# methods anyway, you should set this option to NO.\n\nIDL_PROPERTY_SUPPORT = NO\n\n# If member grouping is used in the documentation and the DISTRIBUTE_GROUP_DOC \n# tag is set to YES, then doxygen will reuse the documentation of the first \n# member in the group (if any) for the other members of the group. By default \n# all members of a group must be documented explicitly.\n\nDISTRIBUTE_GROUP_DOC = NO\n\n# Set the SUBGROUPING tag to YES (the default) to allow class member groups of \n# the same type (for instance a group of public functions) to be put as a \n# subgroup of that type (e.g. under the Public Functions section). Set it to \n# NO to prevent subgrouping. Alternatively, this can be done per class using \n# the \\nosubgrouping command.\n\nSUBGROUPING = YES\n\n# When TYPEDEF_HIDES_STRUCT is enabled, a typedef of a struct, union, or enum \n# is documented as struct, union, or enum with the name of the typedef. So \n# typedef struct TypeS {} TypeT, will appear in the documentation as a struct \n# with name TypeT. When disabled the typedef will appear as a member of a file, \n# namespace, or class. And the struct will be named TypeS. This can typically \n# be useful for C code in case the coding convention dictates that all compound \n# types are typedef'ed and only the typedef is referenced, never the tag name.\n\nTYPEDEF_HIDES_STRUCT = NO\n\n# The SYMBOL_CACHE_SIZE determines the size of the internal cache use to \n# determine which symbols to keep in memory and which to flush to disk. \n# When the cache is full, less often used symbols will be written to disk. \n# For small to medium size projects (<1000 input files) the default value is \n# probably good enough. For larger projects a too small cache size can cause \n# doxygen to be busy swapping symbols to and from disk most of the time \n# causing a significant performance penality. \n# If the system has enough physical memory increasing the cache will improve the \n# performance by keeping more symbols in memory. Note that the value works on \n# a logarithmic scale so increasing the size by one will rougly double the \n# memory usage. The cache size is given by this formula: \n# 2^(16+SYMBOL_CACHE_SIZE). The valid range is 0..9, the default is 0, \n# corresponding to a cache size of 2^16 = 65536 symbols\n\nSYMBOL_CACHE_SIZE = 0\n\n#---------------------------------------------------------------------------\n# Build related configuration options\n#---------------------------------------------------------------------------\n\n# If the EXTRACT_ALL tag is set to YES doxygen will assume all entities in \n# documentation are documented, even if no documentation was available. \n# Private class members and static file members will be hidden unless \n# the EXTRACT_PRIVATE and EXTRACT_STATIC tags are set to YES\n\nEXTRACT_ALL = NO\n\n# If the EXTRACT_PRIVATE tag is set to YES all private members of a class \n# will be included in the documentation.\n\nEXTRACT_PRIVATE = NO\n\n# If the EXTRACT_STATIC tag is set to YES all static members of a file \n# will be included in the documentation.\n\nEXTRACT_STATIC = NO\n\n# If the EXTRACT_LOCAL_CLASSES tag is set to YES classes (and structs) \n# defined locally in source files will be included in the documentation. \n# If set to NO only classes defined in header files are included.\n\nEXTRACT_LOCAL_CLASSES = NO\n\n# This flag is only useful for Objective-C code. When set to YES local \n# methods, which are defined in the implementation section but not in \n# the interface are included in the documentation. \n# If set to NO (the default) only methods in the interface are included.\n\nEXTRACT_LOCAL_METHODS = NO\n\n# If this flag is set to YES, the members of anonymous namespaces will be \n# extracted and appear in the documentation as a namespace called \n# 'anonymous_namespace{file}', where file will be replaced with the base \n# name of the file that contains the anonymous namespace. By default \n# anonymous namespace are hidden.\n\nEXTRACT_ANON_NSPACES = NO\n\n# If the HIDE_UNDOC_MEMBERS tag is set to YES, Doxygen will hide all \n# undocumented members of documented classes, files or namespaces. \n# If set to NO (the default) these members will be included in the \n# various overviews, but no documentation section is generated. \n# This option has no effect if EXTRACT_ALL is enabled.\n\nHIDE_UNDOC_MEMBERS = NO\n\n# If the HIDE_UNDOC_CLASSES tag is set to YES, Doxygen will hide all \n# undocumented classes that are normally visible in the class hierarchy. \n# If set to NO (the default) these classes will be included in the various \n# overviews. This option has no effect if EXTRACT_ALL is enabled.\n\nHIDE_UNDOC_CLASSES = YES\n\n# If the HIDE_FRIEND_COMPOUNDS tag is set to YES, Doxygen will hide all \n# friend (class|struct|union) declarations. \n# If set to NO (the default) these declarations will be included in the \n# documentation.\n\nHIDE_FRIEND_COMPOUNDS = YES\n\n# If the HIDE_IN_BODY_DOCS tag is set to YES, Doxygen will hide any \n# documentation blocks found inside the body of a function. \n# If set to NO (the default) these blocks will be appended to the \n# function's detailed documentation block.\n\nHIDE_IN_BODY_DOCS = YES\n\n# The INTERNAL_DOCS tag determines if documentation \n# that is typed after a \\internal command is included. If the tag is set \n# to NO (the default) then the documentation will be excluded. \n# Set it to YES to include the internal documentation.\n\nINTERNAL_DOCS = YES \n\n# If the CASE_SENSE_NAMES tag is set to NO then Doxygen will only generate \n# file names in lower-case letters. If set to YES upper-case letters are also \n# allowed. This is useful if you have classes or files whose names only differ \n# in case and if your file system supports case sensitive file names. Windows \n# and Mac users are advised to set this option to NO.\n\nCASE_SENSE_NAMES = YES\n\n# If the HIDE_SCOPE_NAMES tag is set to NO (the default) then Doxygen \n# will show members with their full class and namespace scopes in the \n# documentation. If set to YES the scope will be hidden.\n\nHIDE_SCOPE_NAMES = NO\n\n# If the SHOW_INCLUDE_FILES tag is set to YES (the default) then Doxygen \n# will put a list of the files that are included by a file in the documentation \n# of that file.\n\nSHOW_INCLUDE_FILES = YES\n\n# If the INLINE_INFO tag is set to YES (the default) then a tag [inline] \n# is inserted in the documentation for inline members.\n\nINLINE_INFO = YES\n\n# If the SORT_MEMBER_DOCS tag is set to YES (the default) then doxygen \n# will sort the (detailed) documentation of file and class members \n# alphabetically by member name. If set to NO the members will appear in \n# declaration order.\n\nSORT_MEMBER_DOCS = YES\n\n# If the SORT_BRIEF_DOCS tag is set to YES then doxygen will sort the \n# brief documentation of file, namespace and class members alphabetically \n# by member name. If set to NO (the default) the members will appear in \n# declaration order.\n\nSORT_BRIEF_DOCS = NO\n\n# If the SORT_GROUP_NAMES tag is set to YES then doxygen will sort the \n# hierarchy of group names into alphabetical order. If set to NO (the default) \n# the group names will appear in their defined order.\n\nSORT_GROUP_NAMES = NO\n\n# If the SORT_BY_SCOPE_NAME tag is set to YES, the class list will be \n# sorted by fully-qualified names, including namespaces. If set to \n# NO (the default), the class list will be sorted only by class name, \n# not including the namespace part. \n# Note: This option is not very useful if HIDE_SCOPE_NAMES is set to YES. \n# Note: This option applies only to the class list, not to the \n# alphabetical list.\n\nSORT_BY_SCOPE_NAME = NO\n\n# The GENERATE_TODOLIST tag can be used to enable (YES) or \n# disable (NO) the todo list. This list is created by putting \\todo \n# commands in the documentation.\n\nGENERATE_TODOLIST = NO\n\n# The GENERATE_TESTLIST tag can be used to enable (YES) or \n# disable (NO) the test list. This list is created by putting \\test \n# commands in the documentation.\n\nGENERATE_TESTLIST = NO\n\n# The GENERATE_BUGLIST tag can be used to enable (YES) or \n# disable (NO) the bug list. This list is created by putting \\bug \n# commands in the documentation.\n\nGENERATE_BUGLIST = NO\n\n# The GENERATE_DEPRECATEDLIST tag can be used to enable (YES) or \n# disable (NO) the deprecated list. This list is created by putting \n# \\deprecated commands in the documentation.\n\nGENERATE_DEPRECATEDLIST= NO\n\n# The ENABLED_SECTIONS tag can be used to enable conditional \n# documentation sections, marked by \\if sectionname ... \\endif.\n\nENABLED_SECTIONS = GRAPHLAB_INTERNAL\n\n# The MAX_INITIALIZER_LINES tag determines the maximum number of lines \n# the initial value of a variable or define consists of for it to appear in \n# the documentation. If the initializer consists of more lines than specified \n# here it will be hidden. Use a value of 0 to hide initializers completely. \n# The appearance of the initializer of individual variables and defines in the \n# documentation can be controlled using \\showinitializer or \\hideinitializer \n# command in the documentation regardless of this setting.\n\nMAX_INITIALIZER_LINES = 30\n\n# Set the SHOW_USED_FILES tag to NO to disable the list of files generated \n# at the bottom of the documentation of classes and structs. If set to YES the \n# list will mention the files that were used to generate the documentation.\n\nSHOW_USED_FILES = YES\n\n# If the sources in your project are distributed over multiple directories \n# then setting the SHOW_DIRECTORIES tag to YES will show the directory hierarchy \n# in the documentation. The default is NO.\n\nSHOW_DIRECTORIES = YES\n\n# Set the SHOW_FILES tag to NO to disable the generation of the Files page. \n# This will remove the Files entry from the Quick Index and from the \n# Folder Tree View (if specified). The default is YES.\n\nSHOW_FILES = YES\n\n# Set the SHOW_NAMESPACES tag to NO to disable the generation of the \n# Namespaces page. \n# This will remove the Namespaces entry from the Quick Index \n# and from the Folder Tree View (if specified). The default is YES.\n\nSHOW_NAMESPACES = YES\n\n# The FILE_VERSION_FILTER tag can be used to specify a program or script that \n# doxygen should invoke to get the current version for each file (typically from \n# the version control system). Doxygen will invoke the program by executing (via \n# popen()) the command , where is the value of \n# the FILE_VERSION_FILTER tag, and is the name of an input file \n# provided by doxygen. Whatever the program writes to standard output \n# is used as the file version. See the manual for examples.\n\nFILE_VERSION_FILTER = \n\n# The LAYOUT_FILE tag can be used to specify a layout file which will be parsed by \n# doxygen. The layout file controls the global structure of the generated output files \n# in an output format independent way. The create the layout file that represents \n# doxygen's defaults, run doxygen with the -l option. You can optionally specify a \n# file name after the option, if omitted DoxygenLayout.xml will be used as the name \n# of the layout file.\n\nLAYOUT_FILE = \n\n#---------------------------------------------------------------------------\n# configuration options related to warning and progress messages\n#---------------------------------------------------------------------------\n\n# The QUIET tag can be used to turn on/off the messages that are generated \n# by doxygen. Possible values are YES and NO. If left blank NO is used.\n\nQUIET = NO\n\n# The WARNINGS tag can be used to turn on/off the warning messages that are \n# generated by doxygen. Possible values are YES and NO. If left blank \n# NO is used.\n\nWARNINGS = YES\n\n# If WARN_IF_UNDOCUMENTED is set to YES, then doxygen will generate warnings \n# for undocumented members. If EXTRACT_ALL is set to YES then this flag will \n# automatically be disabled.\n\nWARN_IF_UNDOCUMENTED = YES\n\n# If WARN_IF_DOC_ERROR is set to YES, doxygen will generate warnings for \n# potential errors in the documentation, such as not documenting some \n# parameters in a documented function, or documenting parameters that \n# don't exist or using markup commands wrongly.\n\nWARN_IF_DOC_ERROR = YES\n\n# This WARN_NO_PARAMDOC option can be abled to get warnings for \n# functions that are documented, but have no documentation for their parameters \n# or return value. If set to NO (the default) doxygen will only warn about \n# wrong or incomplete parameter documentation, but not about the absence of \n# documentation.\n\nWARN_NO_PARAMDOC = NO\n\n# The WARN_FORMAT tag determines the format of the warning messages that \n# doxygen can produce. The string should contain the $file, $line, and $text \n# tags, which will be replaced by the file and line number from which the \n# warning originated and the warning text. Optionally the format may contain \n# $version, which will be replaced by the version of the file (if it could \n# be obtained via FILE_VERSION_FILTER)\n\nWARN_FORMAT = \"$file:$line: $text\"\n\n# The WARN_LOGFILE tag can be used to specify a file to which warning \n# and error messages should be written. If left blank the output is written \n# to stderr.\n\nWARN_LOGFILE = doxygen.log \n\n#---------------------------------------------------------------------------\n# configuration options related to the input files\n#---------------------------------------------------------------------------\n\n# The INPUT tag can be used to specify the files and/or directories that contain \n# documented source files. You may enter file names like \"myfile.cpp\" or \n# directories like \"/usr/src/myproject\". Separate the files or directories \n# with spaces.\n\nINPUT = src \\\n demoapps \\\n toolkits\n\n# This tag can be used to specify the character encoding of the source files \n# that doxygen parses. Internally doxygen uses the UTF-8 encoding, which is \n# also the default input encoding. Doxygen uses libiconv (or the iconv built \n# into libc) for the transcoding. See http://www.gnu.org/software/libiconv for \n# the list of possible encodings.\n\nINPUT_ENCODING = UTF-8\n\n# If the value of the INPUT tag contains directories, you can use the \n# FILE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp \n# and *.h) to filter out the source-files in the directories. If left \n# blank the following patterns are tested: \n# *.c *.cc *.cxx *.cpp *.c++ *.java *.ii *.ixx *.ipp *.i++ *.inl *.h *.hh *.hxx \n# *.hpp *.h++ *.idl *.odl *.cs *.php *.php3 *.inc *.m *.mm *.py *.f90\n\nFILE_PATTERNS = *.hpp *.cpp *.dox \n\n# The RECURSIVE tag can be used to turn specify whether or not subdirectories \n# should be searched for input files as well. Possible values are YES and NO. \n# If left blank NO is used.\n\nRECURSIVE = YES\n\n# The EXCLUDE tag can be used to specify files and/or directories that should \n# excluded from the INPUT source files. This way you can easily exclude a \n# subdirectory from a directory tree whose root is specified with the INPUT tag.\n\nEXCLUDE = src/graphlab/matlab src/graphlab/gpu \n\n# The EXCLUDE_SYMLINKS tag can be used select whether or not files or \n# directories that are symbolic links (a Unix filesystem feature) are excluded \n# from the input.\n\nEXCLUDE_SYMLINKS = NO\n\n# If the value of the INPUT tag contains directories, you can use the \n# EXCLUDE_PATTERNS tag to specify one or more wildcard patterns to exclude \n# certain files from those directories. Note that the wildcards are matched \n# against the file with absolute path, so to exclude all test directories \n# for example use the pattern */test/*\n\nEXCLUDE_PATTERNS = */src/graphlab/rpc/*issue.hpp */src/graphlab/rpc/*dispatch.hpp */toolkits/*cpp */toolkits/*hpp\n\n# The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names \n# (namespaces, classes, functions, etc.) that should be excluded from the \n# output. The symbol name can be a fully qualified name, a word, or if the \n# wildcard * is used, a substring. Examples: ANamespace, AClass, \n# AClass::ANamespace, ANamespace::*Test\n\nEXCLUDE_SYMBOLS = graphlab::archive_detail::* graphlab::dc_impl::*\n\n# The EXAMPLE_PATH tag can be used to specify one or more files or \n# directories that contain example code fragments that are included (see \n# the \\include command).\n\nEXAMPLE_PATH = \n\n# If the value of the EXAMPLE_PATH tag contains directories, you can use the \n# EXAMPLE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp \n# and *.h) to filter out the source-files in the directories. If left \n# blank all files are included.\n\nEXAMPLE_PATTERNS = \n\n# If the EXAMPLE_RECURSIVE tag is set to YES then subdirectories will be \n# searched for input files to be used with the \\include or \\dontinclude \n# commands irrespective of the value of the RECURSIVE tag. \n# Possible values are YES and NO. If left blank NO is used.\n\nEXAMPLE_RECURSIVE = NO\n\n# The IMAGE_PATH tag can be used to specify one or more files or \n# directories that contain image that are included in the documentation (see \n# the \\image command).\n\nIMAGE_PATH = doc/images\n\n# The INPUT_FILTER tag can be used to specify a program that doxygen should \n# invoke to filter for each input file. Doxygen will invoke the filter program \n# by executing (via popen()) the command , where \n# is the value of the INPUT_FILTER tag, and is the name of an \n# input file. Doxygen will then use the output that the filter program writes \n# to standard output. \n# If FILTER_PATTERNS is specified, this tag will be \n# ignored.\n\nINPUT_FILTER = \n\n# The FILTER_PATTERNS tag can be used to specify filters on a per file pattern \n# basis. \n# Doxygen will compare the file name with each pattern and apply the \n# filter if there is a match. \n# The filters are a list of the form: \n# pattern=filter (like *.cpp=my_cpp_filter). See INPUT_FILTER for further \n# info on how filters are used. If FILTER_PATTERNS is empty, INPUT_FILTER \n# is applied to all files.\n\nFILTER_PATTERNS = \n\n# If the FILTER_SOURCE_FILES tag is set to YES, the input filter (if set using \n# INPUT_FILTER) will be used to filter the input files when producing source \n# files to browse (i.e. when SOURCE_BROWSER is set to YES).\n\nFILTER_SOURCE_FILES = NO\n\n#---------------------------------------------------------------------------\n# configuration options related to source browsing\n#---------------------------------------------------------------------------\n\n# If the SOURCE_BROWSER tag is set to YES then a list of source files will \n# be generated. Documented entities will be cross-referenced with these sources. \n# Note: To get rid of all source code in the generated output, make sure also \n# VERBATIM_HEADERS is set to NO.\n\nSOURCE_BROWSER = YES\n\n# Setting the INLINE_SOURCES tag to YES will include the body \n# of functions and classes directly in the documentation.\n\nINLINE_SOURCES = NO\n\n# Setting the STRIP_CODE_COMMENTS tag to YES (the default) will instruct \n# doxygen to hide any special comment blocks from generated source code \n# fragments. Normal C and C++ comments will always remain visible.\n\nSTRIP_CODE_COMMENTS = NO\n\n# If the REFERENCED_BY_RELATION tag is set to YES \n# then for each documented function all documented \n# functions referencing it will be listed.\n\nREFERENCED_BY_RELATION = NO\n\n# If the REFERENCES_RELATION tag is set to YES \n# then for each documented function all documented entities \n# called/used by that function will be listed.\n\nREFERENCES_RELATION = NO\n\n# If the REFERENCES_LINK_SOURCE tag is set to YES (the default) \n# and SOURCE_BROWSER tag is set to YES, then the hyperlinks from \n# functions in REFERENCES_RELATION and REFERENCED_BY_RELATION lists will \n# link to the source code. \n# Otherwise they will link to the documentation.\n\nREFERENCES_LINK_SOURCE = YES\n\n# If the USE_HTAGS tag is set to YES then the references to source code \n# will point to the HTML generated by the htags(1) tool instead of doxygen \n# built-in source browser. The htags tool is part of GNU's global source \n# tagging system (see http://www.gnu.org/software/global/global.html). You \n# will need version 4.8.6 or higher.\n\nUSE_HTAGS = NO\n\n# If the VERBATIM_HEADERS tag is set to YES (the default) then Doxygen \n# will generate a verbatim copy of the header file for each class for \n# which an include is specified. Set to NO to disable this.\n\nVERBATIM_HEADERS = NO\n\n#---------------------------------------------------------------------------\n# configuration options related to the alphabetical class index\n#---------------------------------------------------------------------------\n\n# If the ALPHABETICAL_INDEX tag is set to YES, an alphabetical index \n# of all compounds will be generated. Enable this if the project \n# contains a lot of classes, structs, unions or interfaces.\n\nALPHABETICAL_INDEX = YES\n\n# If the alphabetical index is enabled (see ALPHABETICAL_INDEX) then \n# the COLS_IN_ALPHA_INDEX tag can be used to specify the number of columns \n# in which this list will be split (can be a number in the range [1..20])\n\nCOLS_IN_ALPHA_INDEX = 5\n\n# In case all classes in a project start with a common prefix, all \n# classes will be put under the same header in the alphabetical index. \n# The IGNORE_PREFIX tag can be used to specify one or more prefixes that \n# should be ignored while generating the index headers.\n\nIGNORE_PREFIX = \n\n#---------------------------------------------------------------------------\n# configuration options related to the HTML output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_HTML tag is set to YES (the default) Doxygen will \n# generate HTML output.\n\nGENERATE_HTML = YES\n\n# The HTML_OUTPUT tag is used to specify where the HTML docs will be put. \n# If a relative path is entered the value of OUTPUT_DIRECTORY will be \n# put in front of it. If left blank `html' will be used as the default path.\n\nHTML_OUTPUT = html\n\n# The HTML_FILE_EXTENSION tag can be used to specify the file extension for \n# each generated HTML page (for example: .htm,.php,.asp). If it is left blank \n# doxygen will generate files with .html extension.\n\nHTML_FILE_EXTENSION = .html\n\n# The HTML_HEADER tag can be used to specify a personal HTML header for \n# each generated HTML page. If it is left blank doxygen will generate a \n# standard header.\n\nHTML_HEADER = \n\n# The HTML_FOOTER tag can be used to specify a personal HTML footer for \n# each generated HTML page. If it is left blank doxygen will generate a \n# standard footer.\n\nHTML_FOOTER = \n\n# The HTML_STYLESHEET tag can be used to specify a user-defined cascading \n# style sheet that is used by each HTML page. It can be used to \n# fine-tune the look of the HTML output. If the tag is left blank doxygen \n# will generate a default style sheet. Note that doxygen will try to copy \n# the style sheet file to the HTML output directory, so don't put your own \n# stylesheet in the HTML output directory as well, or it will be erased!\n\nHTML_STYLESHEET = \n\n# If the HTML_ALIGN_MEMBERS tag is set to YES, the members of classes, \n# files or namespaces will be aligned in HTML using tables. If set to \n# NO a bullet list will be used.\n\nHTML_ALIGN_MEMBERS = YES\n\n# If the HTML_DYNAMIC_SECTIONS tag is set to YES then the generated HTML \n# documentation will contain sections that can be hidden and shown after the \n# page has loaded. For this to work a browser that supports \n# JavaScript and DHTML is required (for instance Mozilla 1.0+, Firefox \n# Netscape 6.0+, Internet explorer 5.0+, Konqueror, or Safari).\n\nHTML_DYNAMIC_SECTIONS = YES\n\n# If the GENERATE_DOCSET tag is set to YES, additional index files \n# will be generated that can be used as input for Apple's Xcode 3 \n# integrated development environment, introduced with OSX 10.5 (Leopard). \n# To create a documentation set, doxygen will generate a Makefile in the \n# HTML output directory. Running make will produce the docset in that \n# directory and running \"make install\" will install the docset in \n# ~/Library/Developer/Shared/Documentation/DocSets so that Xcode will find \n# it at startup. \n# See http://developer.apple.com/tools/creatingdocsetswithdoxygen.html for more information.\n\nGENERATE_DOCSET = NO\n\n# When GENERATE_DOCSET tag is set to YES, this tag determines the name of the \n# feed. A documentation feed provides an umbrella under which multiple \n# documentation sets from a single provider (such as a company or product suite) \n# can be grouped.\n\nDOCSET_FEEDNAME = \"Doxygen generated docs\"\n\n# When GENERATE_DOCSET tag is set to YES, this tag specifies a string that \n# should uniquely identify the documentation set bundle. This should be a \n# reverse domain-name style string, e.g. com.mycompany.MyDocSet. Doxygen \n# will append .docset to the name.\n\nDOCSET_BUNDLE_ID = org.doxygen.Project\n\n# If the GENERATE_HTMLHELP tag is set to YES, additional index files \n# will be generated that can be used as input for tools like the \n# Microsoft HTML help workshop to generate a compiled HTML help file (.chm) \n# of the generated HTML documentation.\n\nGENERATE_HTMLHELP = NO\n\n# If the GENERATE_HTMLHELP tag is set to YES, the CHM_FILE tag can \n# be used to specify the file name of the resulting .chm file. You \n# can add a path in front of the file if the result should not be \n# written to the html output directory.\n\nCHM_FILE = \n\n# If the GENERATE_HTMLHELP tag is set to YES, the HHC_LOCATION tag can \n# be used to specify the location (absolute path including file name) of \n# the HTML help compiler (hhc.exe). If non-empty doxygen will try to run \n# the HTML help compiler on the generated index.hhp.\n\nHHC_LOCATION = \n\n# If the GENERATE_HTMLHELP tag is set to YES, the GENERATE_CHI flag \n# controls if a separate .chi index file is generated (YES) or that \n# it should be included in the master .chm file (NO).\n\nGENERATE_CHI = NO\n\n# If the GENERATE_HTMLHELP tag is set to YES, the CHM_INDEX_ENCODING \n# is used to encode HtmlHelp index (hhk), content (hhc) and project file \n# content.\n\nCHM_INDEX_ENCODING = \n\n# If the GENERATE_HTMLHELP tag is set to YES, the BINARY_TOC flag \n# controls whether a binary table of contents is generated (YES) or a \n# normal table of contents (NO) in the .chm file.\n\nBINARY_TOC = NO\n\n# The TOC_EXPAND flag can be set to YES to add extra items for group members \n# to the contents of the HTML help documentation and to the tree view.\n\nTOC_EXPAND = NO\n\n# If the GENERATE_QHP tag is set to YES and both QHP_NAMESPACE and QHP_VIRTUAL_FOLDER \n# are set, an additional index file will be generated that can be used as input for \n# Qt's qhelpgenerator to generate a Qt Compressed Help (.qch) of the generated \n# HTML documentation.\n\nGENERATE_QHP = NO\n\n# If the QHG_LOCATION tag is specified, the QCH_FILE tag can \n# be used to specify the file name of the resulting .qch file. \n# The path specified is relative to the HTML output folder.\n\nQCH_FILE = \n\n# The QHP_NAMESPACE tag specifies the namespace to use when generating \n# Qt Help Project output. For more information please see \n# http://doc.trolltech.com/qthelpproject.html#namespace\n\nQHP_NAMESPACE = \n\n# The QHP_VIRTUAL_FOLDER tag specifies the namespace to use when generating \n# Qt Help Project output. For more information please see \n# http://doc.trolltech.com/qthelpproject.html#virtual-folders\n\nQHP_VIRTUAL_FOLDER = doc\n\n# If QHP_CUST_FILTER_NAME is set, it specifies the name of a custom filter to add. \n# For more information please see \n# http://doc.trolltech.com/qthelpproject.html#custom-filters\n\nQHP_CUST_FILTER_NAME = \n\n# The QHP_CUST_FILT_ATTRS tag specifies the list of the attributes of the custom filter to add.For more information please see \n# Qt Help Project / Custom Filters.\n\nQHP_CUST_FILTER_ATTRS = \n\n# The QHP_SECT_FILTER_ATTRS tag specifies the list of the attributes this project's \n# filter section matches. \n# Qt Help Project / Filter Attributes.\n\nQHP_SECT_FILTER_ATTRS = \n\n# If the GENERATE_QHP tag is set to YES, the QHG_LOCATION tag can \n# be used to specify the location of Qt's qhelpgenerator. \n# If non-empty doxygen will try to run qhelpgenerator on the generated \n# .qhp file.\n\nQHG_LOCATION = \n\n# The DISABLE_INDEX tag can be used to turn on/off the condensed index at \n# top of each HTML page. The value NO (the default) enables the index and \n# the value YES disables it.\n\nDISABLE_INDEX = NO\n\n# This tag can be used to set the number of enum values (range [1..20]) \n# that doxygen will group on one line in the generated HTML documentation.\n\nENUM_VALUES_PER_LINE = 4\n\n# The GENERATE_TREEVIEW tag is used to specify whether a tree-like index \n# structure should be generated to display hierarchical information. \n# If the tag value is set to FRAME, a side panel will be generated \n# containing a tree-like index structure (just like the one that \n# is generated for HTML Help). For this to work a browser that supports \n# JavaScript, DHTML, CSS and frames is required (for instance Mozilla 1.0+, \n# Netscape 6.0+, Internet explorer 5.0+, or Konqueror). Windows users are \n# probably better off using the HTML help feature. Other possible values \n# for this tag are: HIERARCHIES, which will generate the Groups, Directories, \n# and Class Hierarchy pages using a tree view instead of an ordered list; \n# ALL, which combines the behavior of FRAME and HIERARCHIES; and NONE, which \n# disables this behavior completely. For backwards compatibility with previous \n# releases of Doxygen, the values YES and NO are equivalent to FRAME and NONE \n# respectively.\n\nGENERATE_TREEVIEW = YES\n\n# If the treeview is enabled (see GENERATE_TREEVIEW) then this tag can be \n# used to set the initial width (in pixels) of the frame in which the tree \n# is shown.\n\nTREEVIEW_WIDTH = 250\n\n# Use this tag to change the font size of Latex formulas included \n# as images in the HTML documentation. The default is 10. Note that \n# when you change the font size after a successful doxygen run you need \n# to manually remove any form_*.png images from the HTML output directory \n# to force them to be regenerated.\n\nFORMULA_FONTSIZE = 10\n\n#---------------------------------------------------------------------------\n# configuration options related to the LaTeX output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_LATEX tag is set to YES (the default) Doxygen will \n# generate Latex output.\n\nGENERATE_LATEX = NO\n\n# The LATEX_OUTPUT tag is used to specify where the LaTeX docs will be put. \n# If a relative path is entered the value of OUTPUT_DIRECTORY will be \n# put in front of it. If left blank `latex' will be used as the default path.\n\nLATEX_OUTPUT = latex\n\n# The LATEX_CMD_NAME tag can be used to specify the LaTeX command name to be \n# invoked. If left blank `latex' will be used as the default command name.\n\nLATEX_CMD_NAME = latex\n\n# The MAKEINDEX_CMD_NAME tag can be used to specify the command name to \n# generate index for LaTeX. If left blank `makeindex' will be used as the \n# default command name.\n\nMAKEINDEX_CMD_NAME = makeindex\n\n# If the COMPACT_LATEX tag is set to YES Doxygen generates more compact \n# LaTeX documents. This may be useful for small projects and may help to \n# save some trees in general.\n\nCOMPACT_LATEX = NO\n\n# The PAPER_TYPE tag can be used to set the paper type that is used \n# by the printer. Possible values are: a4, a4wide, letter, legal and \n# executive. If left blank a4wide will be used.\n\nPAPER_TYPE = a4wide\n\n# The EXTRA_PACKAGES tag can be to specify one or more names of LaTeX \n# packages that should be included in the LaTeX output.\n\nEXTRA_PACKAGES = \n\n# The LATEX_HEADER tag can be used to specify a personal LaTeX header for \n# the generated latex document. The header should contain everything until \n# the first chapter. If it is left blank doxygen will generate a \n# standard header. Notice: only use this tag if you know what you are doing!\n\nLATEX_HEADER = \n\n# If the PDF_HYPERLINKS tag is set to YES, the LaTeX that is generated \n# is prepared for conversion to pdf (using ps2pdf). The pdf file will \n# contain links (just like the HTML output) instead of page references \n# This makes the output suitable for online browsing using a pdf viewer.\n\nPDF_HYPERLINKS = YES\n\n# If the USE_PDFLATEX tag is set to YES, pdflatex will be used instead of \n# plain latex in the generated Makefile. Set this option to YES to get a \n# higher quality PDF documentation.\n\nUSE_PDFLATEX = YES\n\n# If the LATEX_BATCHMODE tag is set to YES, doxygen will add the \\\\batchmode. \n# command to the generated LaTeX files. This will instruct LaTeX to keep \n# running if errors occur, instead of asking the user for help. \n# This option is also used when generating formulas in HTML.\n\nLATEX_BATCHMODE = NO\n\n# If LATEX_HIDE_INDICES is set to YES then doxygen will not \n# include the index chapters (such as File Index, Compound Index, etc.) \n# in the output.\n\nLATEX_HIDE_INDICES = NO\n\n#---------------------------------------------------------------------------\n# configuration options related to the RTF output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_RTF tag is set to YES Doxygen will generate RTF output \n# The RTF output is optimized for Word 97 and may not look very pretty with \n# other RTF readers or editors.\n\nGENERATE_RTF = NO\n\n# The RTF_OUTPUT tag is used to specify where the RTF docs will be put. \n# If a relative path is entered the value of OUTPUT_DIRECTORY will be \n# put in front of it. If left blank `rtf' will be used as the default path.\n\nRTF_OUTPUT = rtf\n\n# If the COMPACT_RTF tag is set to YES Doxygen generates more compact \n# RTF documents. This may be useful for small projects and may help to \n# save some trees in general.\n\nCOMPACT_RTF = NO\n\n# If the RTF_HYPERLINKS tag is set to YES, the RTF that is generated \n# will contain hyperlink fields. The RTF file will \n# contain links (just like the HTML output) instead of page references. \n# This makes the output suitable for online browsing using WORD or other \n# programs which support those fields. \n# Note: wordpad (write) and others do not support links.\n\nRTF_HYPERLINKS = NO\n\n# Load stylesheet definitions from file. Syntax is similar to doxygen's \n# config file, i.e. a series of assignments. You only have to provide \n# replacements, missing definitions are set to their default value.\n\nRTF_STYLESHEET_FILE = \n\n# Set optional variables used in the generation of an rtf document. \n# Syntax is similar to doxygen's config file.\n\nRTF_EXTENSIONS_FILE = \n\n#---------------------------------------------------------------------------\n# configuration options related to the man page output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_MAN tag is set to YES (the default) Doxygen will \n# generate man pages\n\nGENERATE_MAN = NO\n\n# The MAN_OUTPUT tag is used to specify where the man pages will be put. \n# If a relative path is entered the value of OUTPUT_DIRECTORY will be \n# put in front of it. If left blank `man' will be used as the default path.\n\nMAN_OUTPUT = man\n\n# The MAN_EXTENSION tag determines the extension that is added to \n# the generated man pages (default is the subroutine's section .3)\n\nMAN_EXTENSION = .3\n\n# If the MAN_LINKS tag is set to YES and Doxygen generates man output, \n# then it will generate one additional man file for each entity \n# documented in the real man page(s). These additional files \n# only source the real man page, but without them the man command \n# would be unable to find the correct page. The default is NO.\n\nMAN_LINKS = NO\n\n#---------------------------------------------------------------------------\n# configuration options related to the XML output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_XML tag is set to YES Doxygen will \n# generate an XML file that captures the structure of \n# the code including all documentation.\n\nGENERATE_XML = NO\n\n# The XML_OUTPUT tag is used to specify where the XML pages will be put. \n# If a relative path is entered the value of OUTPUT_DIRECTORY will be \n# put in front of it. If left blank `xml' will be used as the default path.\n\nXML_OUTPUT = xml\n\n# The XML_SCHEMA tag can be used to specify an XML schema, \n# which can be used by a validating XML parser to check the \n# syntax of the XML files.\n\nXML_SCHEMA = \n\n# The XML_DTD tag can be used to specify an XML DTD, \n# which can be used by a validating XML parser to check the \n# syntax of the XML files.\n\nXML_DTD = \n\n# If the XML_PROGRAMLISTING tag is set to YES Doxygen will \n# dump the program listings (including syntax highlighting \n# and cross-referencing information) to the XML output. Note that \n# enabling this will significantly increase the size of the XML output.\n\nXML_PROGRAMLISTING = YES\n\n#---------------------------------------------------------------------------\n# configuration options for the AutoGen Definitions output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_AUTOGEN_DEF tag is set to YES Doxygen will \n# generate an AutoGen Definitions (see autogen.sf.net) file \n# that captures the structure of the code including all \n# documentation. Note that this feature is still experimental \n# and incomplete at the moment.\n\nGENERATE_AUTOGEN_DEF = NO\n\n#---------------------------------------------------------------------------\n# configuration options related to the Perl module output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_PERLMOD tag is set to YES Doxygen will \n# generate a Perl module file that captures the structure of \n# the code including all documentation. Note that this \n# feature is still experimental and incomplete at the \n# moment.\n\nGENERATE_PERLMOD = NO\n\n# If the PERLMOD_LATEX tag is set to YES Doxygen will generate \n# the necessary Makefile rules, Perl scripts and LaTeX code to be able \n# to generate PDF and DVI output from the Perl module output.\n\nPERLMOD_LATEX = NO\n\n# If the PERLMOD_PRETTY tag is set to YES the Perl module output will be \n# nicely formatted so it can be parsed by a human reader. \n# This is useful \n# if you want to understand what is going on. \n# On the other hand, if this \n# tag is set to NO the size of the Perl module output will be much smaller \n# and Perl will parse it just the same.\n\nPERLMOD_PRETTY = YES\n\n# The names of the make variables in the generated doxyrules.make file \n# are prefixed with the string contained in PERLMOD_MAKEVAR_PREFIX. \n# This is useful so different doxyrules.make files included by the same \n# Makefile don't overwrite each other's variables.\n\nPERLMOD_MAKEVAR_PREFIX = \n\n#---------------------------------------------------------------------------\n# Configuration options related to the preprocessor \n#---------------------------------------------------------------------------\n\n# If the ENABLE_PREPROCESSING tag is set to YES (the default) Doxygen will \n# evaluate all C-preprocessor directives found in the sources and include \n# files.\n\nENABLE_PREPROCESSING = YES \n\n# If the MACRO_EXPANSION tag is set to YES Doxygen will expand all macro \n# names in the source code. If set to NO (the default) only conditional \n# compilation will be performed. Macro expansion can be done in a controlled \n# way by setting EXPAND_ONLY_PREDEF to YES.\n\nMACRO_EXPANSION = NO\n\n# If the EXPAND_ONLY_PREDEF and MACRO_EXPANSION tags are both set to YES \n# then the macro expansion is limited to the macros specified with the \n# PREDEFINED and EXPAND_AS_DEFINED tags.\n\nEXPAND_ONLY_PREDEF = YES\n\n# If the SEARCH_INCLUDES tag is set to YES (the default) the includes files \n# in the INCLUDE_PATH (see below) will be search if a #include is found.\n\nSEARCH_INCLUDES = YES\n\n# The INCLUDE_PATH tag can be used to specify one or more directories that \n# contain include files that are not input files but should be processed by \n# the preprocessor.\n\nINCLUDE_PATH = \n\n# You can use the INCLUDE_FILE_PATTERNS tag to specify one or more wildcard \n# patterns (like *.h and *.hpp) to filter out the header-files in the \n# directories. If left blank, the patterns specified with FILE_PATTERNS will \n# be used.\n\nINCLUDE_FILE_PATTERNS = \n\n# The PREDEFINED tag can be used to specify one or more macro names that \n# are defined before the preprocessor is started (similar to the -D option of \n# gcc). The argument of the tag is a list of macros of the form: name \n# or name=definition (no spaces). If the definition and the = are \n# omitted =1 is assumed. To prevent a macro definition from being \n# undefined via #undef or recursively expanded use the := operator \n# instead of the = operator.\n\nPREDEFINED = DOXYGEN_DOCUMENTATION GRAPHLAB_SERIALIZE_HPP \n\n# If the MACRO_EXPANSION and EXPAND_ONLY_PREDEF tags are set to YES then \n# this tag can be used to specify a list of macro names that should be expanded. \n# The macro definition that is found in the sources will be used. \n# Use the PREDEFINED tag if you want to use a different macro definition.\n\nEXPAND_AS_DEFINED = RPC_DEFAULT_NUMHANDLERTHREADS RPC_DEFAULT_COMMTYPE \n\n# If the SKIP_FUNCTION_MACROS tag is set to YES (the default) then \n# doxygen's preprocessor will remove all function-like macros that are alone \n# on a line, have an all uppercase name, and do not end with a semicolon. Such \n# function macros are typically used for boiler-plate code, and will confuse \n# the parser if not removed.\n\nSKIP_FUNCTION_MACROS = YES\n\n#---------------------------------------------------------------------------\n# Configuration::additions related to external references \n#---------------------------------------------------------------------------\n\n# The TAGFILES option can be used to specify one or more tagfiles. \n# Optionally an initial location of the external documentation \n# can be added for each tagfile. The format of a tag file without \n# this location is as follows: \n# \n# TAGFILES = file1 file2 ... \n# Adding location for the tag files is done as follows: \n# \n# TAGFILES = file1=loc1 \"file2 = loc2\" ... \n# where \"loc1\" and \"loc2\" can be relative or absolute paths or \n# URLs. If a location is present for each tag, the installdox tool \n# does not have to be run to correct the links. \n# Note that each tag file must have a unique name \n# (where the name does NOT include the path) \n# If a tag file is not located in the directory in which doxygen \n# is run, you must also specify the path to the tagfile here.\n\nTAGFILES = \n\n# When a file name is specified after GENERATE_TAGFILE, doxygen will create \n# a tag file that is based on the input files it reads.\n\nGENERATE_TAGFILE = \n\n# If the ALLEXTERNALS tag is set to YES all external classes will be listed \n# in the class index. If set to NO only the inherited external classes \n# will be listed.\n\nALLEXTERNALS = NO\n\n# If the EXTERNAL_GROUPS tag is set to YES all external groups will be listed \n# in the modules index. If set to NO, only the current project's groups will \n# be listed.\n\nEXTERNAL_GROUPS = YES\n\n# The PERL_PATH should be the absolute path and name of the perl script \n# interpreter (i.e. the result of `which perl').\n\nPERL_PATH = /usr/bin/perl\n\n#---------------------------------------------------------------------------\n# Configuration options related to the dot tool \n#---------------------------------------------------------------------------\n\n# If the CLASS_DIAGRAMS tag is set to YES (the default) Doxygen will \n# generate a inheritance diagram (in HTML, RTF and LaTeX) for classes with base \n# or super classes. Setting the tag to NO turns the diagrams off. Note that \n# this option is superseded by the HAVE_DOT option below. This is only a \n# fallback. It is recommended to install and use dot, since it yields more \n# powerful graphs.\n\nCLASS_DIAGRAMS = NO\n\n# You can define message sequence charts within doxygen comments using the \\msc \n# command. Doxygen will then run the mscgen tool (see \n# http://www.mcternan.me.uk/mscgen/) to produce the chart and insert it in the \n# documentation. The MSCGEN_PATH tag allows you to specify the directory where \n# the mscgen tool resides. If left empty the tool is assumed to be found in the \n# default search path.\n\nMSCGEN_PATH = \n\n# If set to YES, the inheritance and collaboration graphs will hide \n# inheritance and usage relations if the target is undocumented \n# or is not a class.\n\nHIDE_UNDOC_RELATIONS = YES\n\n# If you set the HAVE_DOT tag to YES then doxygen will assume the dot tool is \n# available from the path. This tool is part of Graphviz, a graph visualization \n# toolkit from AT&T and Lucent Bell Labs. The other options in this section \n# have no effect if this option is set to NO (the default)\n\nHAVE_DOT = NO\n\n# By default doxygen will write a font called FreeSans.ttf to the output \n# directory and reference it in all dot files that doxygen generates. This \n# font does not include all possible unicode characters however, so when you need \n# these (or just want a differently looking font) you can specify the font name \n# using DOT_FONTNAME. You need need to make sure dot is able to find the font, \n# which can be done by putting it in a standard location or by setting the \n# DOTFONTPATH environment variable or by setting DOT_FONTPATH to the directory \n# containing the font.\n\nDOT_FONTNAME = FreeSans\n\n# The DOT_FONTSIZE tag can be used to set the size of the font of dot graphs. \n# The default size is 10pt.\n\nDOT_FONTSIZE = 10\n\n# By default doxygen will tell dot to use the output directory to look for the \n# FreeSans.ttf font (which doxygen will put there itself). If you specify a \n# different font using DOT_FONTNAME you can set the path where dot \n# can find it using this tag.\n\nDOT_FONTPATH = \n\n# If the CLASS_GRAPH and HAVE_DOT tags are set to YES then doxygen \n# will generate a graph for each documented class showing the direct and \n# indirect inheritance relations. Setting this tag to YES will force the \n# the CLASS_DIAGRAMS tag to NO.\n\nCLASS_GRAPH = NO\n\n# If the COLLABORATION_GRAPH and HAVE_DOT tags are set to YES then doxygen \n# will generate a graph for each documented class showing the direct and \n# indirect implementation dependencies (inheritance, containment, and \n# class references variables) of the class with other documented classes.\n\nCOLLABORATION_GRAPH = NO\n\n# If the GROUP_GRAPHS and HAVE_DOT tags are set to YES then doxygen \n# will generate a graph for groups, showing the direct groups dependencies\n\nGROUP_GRAPHS = NO\n\n# If the UML_LOOK tag is set to YES doxygen will generate inheritance and \n# collaboration diagrams in a style similar to the OMG's Unified Modeling \n# Language.\n\nUML_LOOK = NO\n\n# If set to YES, the inheritance and collaboration graphs will show the \n# relations between templates and their instances.\n\nTEMPLATE_RELATIONS = NO\n\n# If the ENABLE_PREPROCESSING, SEARCH_INCLUDES, INCLUDE_GRAPH, and HAVE_DOT \n# tags are set to YES then doxygen will generate a graph for each documented \n# file showing the direct and indirect include dependencies of the file with \n# other documented files.\n\nINCLUDE_GRAPH = NO\n\n# If the ENABLE_PREPROCESSING, SEARCH_INCLUDES, INCLUDED_BY_GRAPH, and \n# HAVE_DOT tags are set to YES then doxygen will generate a graph for each \n# documented header file showing the documented files that directly or \n# indirectly include this file.\n\nINCLUDED_BY_GRAPH = NO\n\n# If the CALL_GRAPH and HAVE_DOT options are set to YES then \n# doxygen will generate a call dependency graph for every global function \n# or class method. Note that enabling this option will significantly increase \n# the time of a run. So in most cases it will be better to enable call graphs \n# for selected functions only using the \\callgraph command.\n\nCALL_GRAPH = NO\n\n# If the CALLER_GRAPH and HAVE_DOT tags are set to YES then \n# doxygen will generate a caller dependency graph for every global function \n# or class method. Note that enabling this option will significantly increase \n# the time of a run. So in most cases it will be better to enable caller \n# graphs for selected functions only using the \\callergraph command.\n\nCALLER_GRAPH = NO\n\n# If the GRAPHICAL_HIERARCHY and HAVE_DOT tags are set to YES then doxygen \n# will graphical hierarchy of all classes instead of a textual one.\n\nGRAPHICAL_HIERARCHY = YES\n\n# If the DIRECTORY_GRAPH, SHOW_DIRECTORIES and HAVE_DOT tags are set to YES \n# then doxygen will show the dependencies a directory has on other directories \n# in a graphical way. The dependency relations are determined by the #include \n# relations between the files in the directories.\n\nDIRECTORY_GRAPH = YES\n\n# The DOT_IMAGE_FORMAT tag can be used to set the image format of the images \n# generated by dot. Possible values are png, jpg, or gif \n# If left blank png will be used.\n\nDOT_IMAGE_FORMAT = png\n\n# The tag DOT_PATH can be used to specify the path where the dot tool can be \n# found. If left blank, it is assumed the dot tool can be found in the path.\n\nDOT_PATH = \n\n# The DOTFILE_DIRS tag can be used to specify one or more directories that \n# contain dot files that are included in the documentation (see the \n# \\dotfile command).\n\nDOTFILE_DIRS = \n\n# The DOT_GRAPH_MAX_NODES tag can be used to set the maximum number of \n# nodes that will be shown in the graph. If the number of nodes in a graph \n# becomes larger than this value, doxygen will truncate the graph, which is \n# visualized by representing a node as a red box. Note that doxygen if the \n# number of direct children of the root node in a graph is already larger than \n# DOT_GRAPH_MAX_NODES then the graph will not be shown at all. Also note \n# that the size of a graph can be further restricted by MAX_DOT_GRAPH_DEPTH.\n\nDOT_GRAPH_MAX_NODES = 50\n\n# The MAX_DOT_GRAPH_DEPTH tag can be used to set the maximum depth of the \n# graphs generated by dot. A depth value of 3 means that only nodes reachable \n# from the root by following a path via at most 3 edges will be shown. Nodes \n# that lay further from the root node will be omitted. Note that setting this \n# option to 1 or 2 may greatly reduce the computation time needed for large \n# code bases. Also note that the size of a graph can be further restricted by \n# DOT_GRAPH_MAX_NODES. Using a depth of 0 means no depth restriction.\n\nMAX_DOT_GRAPH_DEPTH = 0\n\n# Set the DOT_TRANSPARENT tag to YES to generate images with a transparent \n# background. This is disabled by default, because dot on Windows does not \n# seem to support this out of the box. Warning: Depending on the platform used, \n# enabling this option may lead to badly anti-aliased labels on the edges of \n# a graph (i.e. they become hard to read).\n\nDOT_TRANSPARENT = NO\n\n# Set the DOT_MULTI_TARGETS tag to YES allow dot to generate multiple output \n# files in one run (i.e. multiple -o and -T options on the command line). This \n# makes dot run faster, but since only newer versions of dot (>1.8.10) \n# support this, this feature is disabled by default.\n\nDOT_MULTI_TARGETS = NO\n\n# If the GENERATE_LEGEND tag is set to YES (the default) Doxygen will \n# generate a legend page explaining the meaning of the various boxes and \n# arrows in the dot generated graphs.\n\nGENERATE_LEGEND = YES\n\n# If the DOT_CLEANUP tag is set to YES (the default) Doxygen will \n# remove the intermediate dot files that are used to generate \n# the various graphs.\n\nDOT_CLEANUP = YES\n\n#---------------------------------------------------------------------------\n# Options related to the search engine\n#---------------------------------------------------------------------------\n\n# The SEARCHENGINE tag specifies whether or not a search engine should be \n# used. If set to NO the values of all tags below this one will be ignored.\n\nSEARCHENGINE = YES\n" }, { "path": "README.md", "content": "# GraphLab PowerGraph v2.2\n\n## UPDATE: For a signficant evolution of this codebase, see GraphLab Create which is available for download at [turi.com](https://turi.com)\n\n## History\nIn 2013, the team that created GraphLab PowerGraph started the Seattle-based company, GraphLab, Inc. The learnings from GraphLab PowerGraph and GraphChi projects have culminated into GraphLab Create, a enterprise-class data science platform for data scientists and software engineers that can simplify building and deploying advanced machine learning models as a RESTful predictive service. In January 2015, GraphLab, Inc. was renamed to Turi. See [turi.com](https://turi.com) for more information. \n\n## Status\nGraphLab PowerGraph is no longer in active development by the founding team. GraphLab PowerGraph is now supported by the community at [http://forum.turi.com/](http://forum.turi.com/). \n\n# Introduction\n\nGraphLab PowerGraph is a graph-based, high performance, distributed computation framework written in C++. \n\nThe GraphLab PowerGraph academic project was started in 2009 at Carnegie Mellon University to develop a new parallel computation abstraction tailored to machine learning. GraphLab PowerGraph 1.0 employed shared-memory design. In GraphLab PowerGraph 2.1, the framework was redesigned to target the distributed environment. It addressed the difficulties with real-world power-law graphs and achieved unparalleled performance at the time. In GraphLab PowerGraph 2.2, the Warp System was introduced and provided a new flexible, distributed architecture around fine-grained user-mode threading (fibers). The Warp System allows one to easily extend the abstraction, to improve optimization for example, while also improving usability.\n\nGraphLab PowerGraph is the culmination of 4-years of research and development into graph computation, distributed computing, and machine learning. GraphLab PowerGraph scales to graphs with billions of vertices and edges easily, performing orders of magnitude faster than competing systems. GraphLab PowerGraph combines advances in machine learning algorithms, asynchronous distributed graph computation, prioritized scheduling, and graph placement with optimized low-level system design and efficient data-structures to achieve unmatched performance and scalability in challenging machine learning tasks.\n\nRelated is GraphChi, a spin-off project separate from the GraphLab PowerGraph project. GraphChi was designed to run very large graph computations on just a single machine, by using a novel algorithm for processing the graph from disk (SSD or hard drive) enabling a single desktop computer (actually a Mac Mini) to tackle problems that previously demanded an entire cluster. For more information, see [https://github.com/GraphChi](https://github.com/GraphChi).\n\n# License\n\n\nGraphLab PowerGraph is released under the [Apache 2 license](http://www.apache.org/licenses/LICENSE-2.0.html).\n\nIf you use GraphLab PowerGraph in your research, please cite our paper:\n```\n @inproceedings{Low+al:uai10graphlab,\n title = {GraphLab: A New Parallel Framework for Machine Learning},\n author = {Yucheng Low and\n Joseph Gonzalez and\n Aapo Kyrola and\n Danny Bickson and\n Carlos Guestrin and\n Joseph M. Hellerstein},\n booktitle = {Conference on Uncertainty in Artificial Intelligence (UAI)},\n month = {July},\n year = {2010}\n }\n```\n\n# Academic and Conference Papers\n\nJoseph E. Gonzalez, Yucheng Low, Haijie Gu, Danny Bickson, and Carlos Guestrin (2012). \"[PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs](https://www.usenix.org/conference/osdi12/technical-sessions/presentation/gonzalez).\" Proceedings of the 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI '12).\n\nYucheng Low, Joseph Gonzalez, Aapo Kyrola, Danny Bickson, Carlos Guestrin and Joseph M. Hellerstein (2012). \"[Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud](http://vldb.org/pvldb/vol5/p716_yuchenglow_vldb2012.pdf).\" Proceedings of the VLDB Endowment (PVLDB).\n\nYucheng Low, Joseph Gonzalez, Aapo Kyrola, Danny Bickson, Carlos Guestrin, and Joseph M. Hellerstein (2010). \"[GraphLab: A New Parallel Framework for Machine Learning](http://arxiv.org/pdf/1006.4990v1.pdf).\" Conference on Uncertainty in Artificial Intelligence (UAI).\n\nLi, Kevin; Gibson, Charles; Ho, David; Zhou, Qi; Kim, Jason; Buhisi, Omar; Brown, Donald E.; Gerber, Matthew, \"[Assessment of machine learning algorithms in cloud computing frameworks](http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=6549501)\", Systems and Information Engineering Design Symposium (SIEDS), 2013 IEEE, pp.98,103, 26-26 April 2013\n\n[Towards Benchmarking Graph-Processing Platforms](http://sc13.supercomputing.org/sites/default/files/PostersArchive/post152.html). by Yong Guo (Delft University of Technology), Marcin Biczak (Delft University of Technology), Ana Lucia Varbanescu (University of Amsterdam), Alexandru Iosup (Delft University of Technology), Claudio Martella (VU University Amsterdam), Theodore L. Willke (Intel Corporation), in Super Computing 13\n\nAapo Kyrola, Guy Blelloch, and Carlos Guestrin (2012). \"[GraphChi: Large-Scale Graph computation on Just a PC](https://www.usenix.org/conference/osdi12/technical-sessions/presentation/kyrola).\" Proceedings of the 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI '12).\n\n\n# The Software Stack\n\nThe GraphLab PowerGraph project consists of a core API and a collection of high-performance machine learning and data mining toolkits built on top. The API is written in C++ and built on top of standard cluster and cloud technologies. Inter-process communication is accomplished over TCP-IP and MPI is used to launch and manage GraphLab PowerGraph programs. Each process is multithreaded to fully utilize the multicore resources available on modern cluster nodes. It supports reading and writing to both Posix and HDFS filesystems.\n\n![GraphLab PowerGraph Software Stack](images/gl_os_software_stack.png \"GraphLab Software Stack\")\n\nGraphLab PowerGraph has a large selection of machine learning methods already implemented (see /toolkits directory in this repo). You can also implement your own algorithms on top of the graph programming API (a certain degree of C++ knowledge is required).\n\nGraphLab PowerGraph Feature Highlights\n--------------------------------------\n\n* **Unified multicore/distributed API:** write once run anywhere \n\n* **Tuned for performance:** optimized C++ execution engine leverages extensive multi-threading and asynchronous IO \n\n* **Scalable:** Run on large cluster deployments by intelligently placing data and computation \n\n* **HDFS Integration:** Access your data directly from HDFS \n\n* **Powerful Machine Learning Toolkits:** Tackle challenging machine learning problems with ease\n\n## Building\n\nThe current version of GraphLab PowerGraph was tested on Ubuntu Linux 64-bit 10.04, 11.04 (Natty), 12.04 (Pangolin) as well as Mac OS X 10.7 (Lion) and Mac OS X 10.8 (Mountain Lion). It requires a 64-bit operating system.\n\n# Dependencies\n\nTo simplify installation, GraphLab PowerGraph currently downloads and builds most of its required dependencies using CMake’s External Project feature. This also means the first build could take a long time.\n\nThere are however, a few dependencies which must be manually satisfied.\n\n* On OS X: g++ (>= 4.2) or clang (>= 3.0) [Required]\n + Required for compiling GraphLab.\n\n* On Linux: g++ (>= 4.3) or clang (>= 3.0) [Required]\n + Required for compiling GraphLab.\n\n* *nix build tools: patch, make [Required]\n + Should come with most Mac/Linux systems by default. Recent Ubuntu version will require to install the build-essential package.\n\n* zlib [Required]\n + Comes with most Mac/Linux systems by default. Recent Ubuntu version will require the zlib1g-dev package.\n\n* Open MPI or MPICH2 [Strongly Recommended]\n + Required for running GraphLab distributed. \n\n* JDK 6 or greater [Optional]\n + Required for HDFS support \n\n## Satisfying Dependencies on Mac OS X\n\nInstalling XCode with the command line tools (in XCode 4.3 you have to do this manually in the XCode Preferences -> Download pane), satisfies all of these dependencies.\n\n## Satisfying Dependencies on Ubuntu\n\nAll the dependencies can be satisfied from the repository:\n\n sudo apt-get update\n sudo apt-get install gcc g++ build-essential libopenmpi-dev openmpi-bin default-jdk cmake zlib1g-dev git\n\n# Downloading GraphLab PowerGraph\n\nYou can download GraphLab PowerGraph directly from the Github Repository. Github also offers a zip download of the repository if you do not have git.\n\nThe git command line for cloning the repository is:\n\n git clone https://github.com/graphlab-code/graphlab.git\n cd graphlab\n\n\n# Compiling and Running\n\n```\n./configure\n```\n\nIn the graphlabapi directory, will create two sub-directories, release/ and debug/ . cd into either of these directories and running make will build the release or the debug versions respectively. Note that this will compile all of GraphLab, including all toolkits. Since some toolkits require additional dependencies (for instance, the Computer Vision toolkit needs OpenCV), this will also download and build all optional dependencies.\n\nWe recommend using make’s parallel build feature to accelerate the compilation process. For instance:\n\n```\nmake -j4\n```\n\nwill perform up to 4 build tasks in parallel. When building in release/ mode, GraphLab does require a large amount of memory to compile with the heaviest toolkit requiring 1GB of RAM.\n\nAlternatively, if you know exactly which toolkit you want to build, cd into the toolkit’s sub-directory and running make, will be significantly faster as it will only download the minimal set of dependencies for that toolkit. For instance:\n\n```\ncd release/toolkits/graph_analytics\nmake -j4\n```\n\nwill build only the Graph Analytics toolkit and will not need to obtain OpenCV, Eigen, etc used by the other toolkits.\n\n## Compilation Issues\nIf you encounter issues please post the following on the [GraphLab forum](http://forum.graphlab.com).\n\n* detailed description of the problem you are facing\n* OS and OS version\n* output of uname -a\n* hardware of the machine\n* utput of g++ -v and clang++ -v\n* contents of graphlab/config.log and graphlab/configure.deps\n\n# Writing Your Own Apps\n\nThere are two ways to write your own apps.\n\n* To work in the GraphLab PowerGraph source tree, (recommended)\n* Install and link against Graphlab PowerGraph (not recommended)\n\n\n## 1: Working in the GraphLab PowerGraph Source Tree\n\nThis is the best option if you just want to try using GraphLab PowerGraph quickly. GraphLab PowerGraph\nuses the CMake build system which enables you to quickly create\na C++ project without having to write complicated Makefiles. \n\n1. Create your own sub-directory in the apps/ directory. for example apps/my_app\n \n2. Create a CMakeLists.txt in apps/my_app containing the following lines:\n\n project(GraphLab) \n add_graphlab_executable(my_app [List of cpp files space separated]) \n\n3. Substituting the right values into the square brackets. For instance:\n\n project(GraphLab) \n add_graphlab_executable(my_app my_app.cpp) \n\n4. Running \"make\" in the apps/ directory of any of the build directories \nshould compile your app. If your app does not show up, try running\n\n cd [the GraphLab API directory]\n touch apps/CMakeLists.txt\n\n\n## 2: Installing and Linking Against GraphLab PowerGraph\n\nTo install and use GraphLab PowerGraph this way will require your system\nto completely satisfy all remaining dependencies, which GraphLab PowerGraph normally \nbuilds automatically. This path is not extensively tested and is \n**not recommended**\n\nYou will require the following additional dependencies\n - libevent (>=2.0.18)\n - libjson (>=7.6.0)\n - libboost (>=1.53)\n - libhdfs (required for HDFS support)\n - tcmalloc (optional)\n\nFollow the instructions in the [Compiling] section to build the release/ \nversion of the library. Then cd into the release/ build directory and \nrun make install . This will install the following:\n\n* include/graphlab.hpp\n + The primary GraphLab header \n* include/graphlab/...\n + The folder containing the headers for the rest of the GraphLab library \n* lib/libgraphlab.a\n + The GraphLab static library.\n \nOnce you have installed GraphLab PowerGraph you can compile your program by running:\n\n```\ng++ -O3 -pthread -lzookeeper_mt -lzookeeper_st -lboost_context -lz -ltcmalloc -levent -levent_pthreads -ljson -lboost_filesystem -lboost_program_options -lboost_system -lboost_iostreams -lboost_date_time -lhdfs -lgraphlab hello_world.cpp\n```\n \nIf you have compiled with MPI support, you will also need\n\n -lmpi -lmpi++ \n \n# Tutorials\nSee [tutorials](TUTORIALS.md)\n\n# Datasets\nThe following are data sets links we found useful when getting started with GraphLab PowerGraph.\n\n##Social Graphs\n* [Stanford Large Network Dataset (SNAP)](http://snap.stanford.edu/data/index.html)\n* [Laboratory for Web Algorithms](http://law.di.unimi.it/datasets.php)\n\n##Collaborative Filtering\n* [Million Song dataset](http://labrosa.ee.columbia.edu/millionsong/)\n* [Movielens dataset GroupLens](http://grouplens.org/datasets/movielens/)\n* [KDD Cup 2012 by Tencent, Inc.](https://www.kddcup2012.org/)\n* [University of Florida sparse matrix collection](http://www.cise.ufl.edu/research/sparse/matrices/)\n\n##Classification\n* [Airline on time performance](http://stat-computing.org/dataexpo/2009/)\n* [SF restaurants](http://missionlocal.org/san-francisco-restaurant-health-inspections/)\n\n##Misc\n* [Amazon Web Services public datasets](http://aws.amazon.com/datasets)\n \n# Release Notes\n##### **map_reduce_vertices/edges and transform_vertices/edges are not parallelized on Mac OS X**\n\nThese operations currently rely on OpenMP for parallelism.\n\nOn OS X 10.6 and earlier, gcc 4.2 has several OpenMP bugs and is not stable enough to use reliably.\n\nOn OS X 10.7, the clang\n++ compiler does not yet support OpenMP.\n\n##### **map_reduce_vertices/edges and transform_vertices/edges use a lot more processors than what was specified in –ncpus**\n\nThis is related to the question above. While there is a simple temporary solution (omp_set_num_threads), we intend to properly resolve the issue by not using openMP at all.\n\n##### **Unable to launch distributed GraphLab when each machine has multiple network interfaces**\n\nThe communication initialization currently takes the first non-localhost IP address as the machine’s IP. A more reliable solution will be to use the hostname used by MPI.\n" }, { "path": "TUTORIALS.md", "content": "# GraphLab PowerGraph Tutorials\n\n##Table of Contents\n* [Deploying on AWS EC2 Cluster](#ec2)\n* [Deploying in a Cluster](#cluster)\n* [Deploying on a single multicore machine](#multicore)\n* [Benchmarking on AWS EC2](#benchmarking)\n* [Fine tuning GraphLab PowerGraph performance](#perf_tuning)\n\n\n# Deploying in AWS EC2 Cluster\n\n## Step 0: Requirements\n* You should have Amazon EC2 account eligible to run on us-east-1a zone.\n\n* Find out using the Amazon AWS console your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (under your account name on the top right corner-> security credentials -> access keys)\n\n* You should have a keypair attached to the zone you are running on (in our example us-east-1a) as explained here. You will need to know your keypair name (graphlabkey in our example), and the location of the private key (~/.ssh/graphlabkey.pem in our example).\n\n* Install [boto](https://pypi.python.org/pypi/boto/). This is the AWS Python client. To install, run: \n\n```\nsudo pip install boto\n```\n\n* Download and install GraphLab PowerGraph using the instructions in the [README.md](README.md).\n\n\n## Step 1: Environment Setup\n\nEdit your .bashrc or .bash_profile or .profile files (remember to source it after editing, using the bash command “source <filename>”)\n\n```\nexport AWS_ACCESS_KEY_ID=[ Your access key ]\nexport AWS_SECRET_ACCESS_KEY=[ Your access key secret ]\n```\n\n## Step 2: Start the cluster\n\n```\ncd ~/graphlabapi/scripts/ec2\n./gl-ec2 -i ~/.ssh/graphlab.pem -k graphlabkey -s 1 launch launchtest\n```\n\n(In the above command, we created a 2-node cluster in us-east-1a zone. -s is the number of slaves, launch is the action, and launchtest is the name of the cluster)\n\n## Step 3: Update GraphLab PowerGraph\n\n```\n./gl-ec2 -i ~/.ssh/graphlab.pem -k graphlabkey update launchtest\n```\n\n## Step 4: Run Alternating Least Squares Demo\n\nThis step runs ALS (alternating least squares) in a cluster using small netflix subset.\nIt first downloads the data from the web: [http://www.select.cs.cmu.edu/code/graphlab/datasets/smallnetflix_mm.train](http://www.select.cs.cmu.edu/code/graphlab/datasets/smallnetflix_mm.train) and [http://www.select.cs.cmu.edu/code/graphlab/datasets/smallnetflix_mm.validate](http://www.select.cs.cmu.edu/code/graphlab/datasets/smallnetflix_mm.validate), copy it into HDFS, and runs 5 alternating least squares iterations:\n\n```\n./gl-ec2 -i ~/.ssh/graphlab.pem -k graphlabkey als_demo launchtest\n```\n\nAfter the run is completed, login to the master node and view the output files in the folder ~/graphlabapi/release/toolkits/collaborative_filtering/ The algorithm and exact format is explained in the API docs.\n\n## Step 5: Shutdown the Cluster\n\n```\n./gl-ec2 -i ~/.ssh/graphlab.pem -k grpahlabkey destroy launchtest\n```\n\n## Other Useful Commands:\n\nLogin into the master node using\n\n```\n./gl-ec2 -i ~/.ssh/graphlab.pem -s 1 login launchtest\n```\n\n\n# Deploying in a Cluster\n\n## Step 0: Install GraphLab PowerGraph on one of your cluster nodes.\n\nInstall GraphLab PowerGraph, using instructions in the [README.md](README.md), on your master node (one of your cluster machines).\n\n## Step 1: Copy GraphLab PowerGraph files to all machines.\n\n1) Create a file called in your home directory called “machines” with the names of all the MPI nodes participate in the computation.\n\nFor example:\n\n```\ncat ~/machines\nmynode1.some.random.domain\nmynode2.some.random.domain\n...\nmynode18.some.random.domain\n```\n2) Verify you have the machines files from section 1) in your root folder of all of the machines.\n\n3) You will need to setup password-less SSH between the master node and all other machines.\n\nVerify it is possible to ssh without password between any pairs of machines. These [instructions](http://www.linuxproblem.org/art_9.html) explain how to setup ssh without passswords.\n\nBefore proceeding, verify that this is setup correctly; check that the following connects to the remote machine without prompting for a password:\n\n```\n# from machine mynode1.some.random.domain\nssh mynode2.some.random.domain\n```\n\n4) On the node you installed GraphLab on, run the following commands to copy GraphLab files to the rest of the machines:\n\n```\ncd ~/graphlab/release/toolkits\n~/graphlab/scripts/mpirsync\ncd ~/graphlab/deps/local\n~/graphlab/scripts/mpirsync\n```\n\nThis step will only work if the file you created in step 1 was named \"machines\" and located in your home directory.\n\nIn order for mpirsync to run properly all machines must have all network ports open.\n\n## Step 2a: Run PageRank on a synthetic graph\n\nThis step runs the [PageRank](http://en.wikipedia.org/wiki/PageRank) algorithm on a synthetic generated graph of 100,000 nodes. It spawns two GraphLab mpi instances (-n 2).\n```\nmpiexec -n 2 -hostfile ~/machines /path/to/pagerank --powerlaw=100000\n```\n\n## Step 2: Run GraphLab PowerGraph ALS using subset of Netflix data\n\nThis step runs ALS (alternating least squares) in a cluster using small netflix susbset.\nIt first downloads an anonymized, synthetic Netflix dataset from the web: [http://www.select.cs.cmu.edu/code/graphlab/datasets/smallnetflix_mm.train](http://www.select.cs.cmu.edu/code/graphlab/datasets/smallnetflix_mm.train) and [http://www.select.cs.cmu.edu/code/graphlab/datasets/smallnetflix_mm.validate](http://www.select.cs.cmu.edu/code/graphlab/datasets/smallnetflix_mm.validate), and runs 5 alternating least squares iterations. After the run is completed, you can login into any of the nodes and view the output files in the folder ~/graphlab/release/toolkits/collaborative_filtering/\n\n ```\n cd /some/ns/folder/\nmkdir smallnetflix\ncd smallnetflix/\nwget http://www.select.cs.cmu.edu/code/graphlab/datasets/smallnetflix_mm.train\nwget http://www.select.cs.cmu.edu/code/graphlab/datasets/smallnetflix_mm.validate\n```\nNow run GraphLab:\n\n````\nmpiexec -n 2 -hostfile ~/machines /path/to/als --matrix /some/ns/folder/smallnetflix/ --max_iter=3 --ncpus=1 --minval=1 --maxval=5 --predictions=out_file\n```\nWhere -n is the number of MPI nodes, and –ncpus is the number of deployed cores on each MPI node.\n\nmachines is a file which includes a list of the machines you like to deploy on (each machine in a new line)\n\nNote: this section assumes you have a network storage (ns) folder where the input can be stored.\nAlternatively, you can split the input into several disjoint files, and store the subsets on the cluster machines.\n\nNote: Don’t forget to change /path/to/als and /some/ns/folder to your actual folder path!\n\nNote: For mpich2, use -f instead of -hostfile.\n\n## Step 3:\n\n[Fine tuning graphlab deployment](#perf_tuning).\n\n## Errors and their resolution:\n\n### Error:\n\n```\n/mnt/info/home/daroczyb/als: error while loading shared libraries: libevent_pthreads-2.0.so.5: cannot open shared object file: No such file or directory\n```\n\n**Solution:**\n\nYou should define LD_LIBRARY_PATH to point to the location of libevent_pthreads, this is done with the -x mpi command, for example:\n\n```\nmpiexec --hostfile machines -x LD_LIBRARY_PATH=/home/daroczyb/graphlab/deps/local/lib/ /mnt/info/home/daroczyb/als /mnt/info/home/daroczyb/smallnetflix_mm.train\n```\n\n### Error:\n\n```\nmnt/info/home/daroczyb/als: error while loading shared libraries: libjvm.so: cannot open shared object file: No such file or directory\n```\n\n**Solution:**\n\nPoint LD_LIBRARY_PATH to the location of libjvm.so using the -x mpi command:\n\n```\nmpiexec --hostfile machines -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/daroczyb/graphlab/deps/local/lib/:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/ /mnt/info/home/daroczyb/als /mnt/info/home/daroczyb/smallnetflix_mm.train\n```\n\n### Error:\n\n```\nproblem with execution of /graphlab/release/toolkits/collaborative_filtering/als on debian1: [Errno 2] No such file or directory\n```\n\n**Solution:**\n\nYou should verify the executable is found on the same path on all machines.\n\n### Error:\n\na prompt asking for password when running mpiexec\n\n**Solution:** Use the following [instructions](http://www.linuxproblem.org/art_9.html) to allow connection with a public/private key pair (no password).\n\n### Error:\n\n```\nException in thread \"main\" java.lang.IllegalArgumentException: Wrong FS: hdfs://[domain]:9000/user/[user_name]/data.txt, expected: file:///\n at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)\n at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)\n at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:307)\n at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:842)\n at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:867)\n at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:487)\n Call to org.apache.hadoop.fs.FileSystem::listStatus failed!\n WARNING: distributed_graph.hpp(load_from_hdfs:1889): No files found matching hdfs://[domain]:9000/user/[user_name]/data.txt\n```\n\n**Solution:**\nVerify you classpath includes all hadoop required folders.\n\n### Error:\n\nJust after TCP Communication layer is constructed: \n```\nBAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES, EXITCODE: 11, CLEANING UP REMAINING PROCESSES, YOU CAN IGNORE THE BELOW CLEANUP MESSAGES\n```\nor:\n\n```\n[xyzserver:22296] *** Process received signal *** mpiexec noticed that process rank 0 with PID 22296 on node xyzserver exited on signal 11 (Segmentation fault).\n```\n\n**Solution:**\n\nCheck that all machines have access to, or are using the same binary\n\n\n#Deployment on a single multicore machine\n\n## Preliminaries:\n\n## Step 0: Install GraphLab on one of your cluster nodes.\n\nUsing the instructions [here](/projects/source.html) on your master node (one of your cluster machines), except invoke the configure script with the ‘–no_mpi’ flag.\nDon’t forget to use\n```\n./configure --no_mpi\n```\n\nwhen configuring GraphLab.\n\n## Step 1: Run GraphLab ALS\n\nThis step runs ALS (alternating least squares) in a cluster using small netflix susbset. It first downloads the data from the web, runs 5 alternating least squares iterations. After the run is completed, the output files will be created in the running folder (the folder graphlab/release/toolkits/collaborative_filtering/) \n\n```\ncd graphlab/release/toolkits/collaborative_filtering/\nmkdir smallnetflix\ncd smallnetflix/\nwget http://www.select.cs.cmu.edu/code/graphlab/datasets/smallnetflix_mm.train\nwget http://www.select.cs.cmu.edu/code/graphlab/datasets/smallnetflix_mm.validate\ncd ..\n```\n\nNow run GraphLab:\n\n```\n./als --matrix ./smallnetflix/ --max_iter=5 --ncpus=1 --predictions=out_file\n```\n \nWhere –ncpus is the number of deployed cores.\n\n\n# Benchmarking on AWS EC2\n\nA commonly repeating task is evaluation of GraphLab performance and scaling properties on a cluster. To help jump start benchmarking we have created this tutorial.\n\n## Step 0: Requirements\n\n1. You should have Amazon EC2 account eligible to run on us-west zone.\n2. Find out using the Amazon AWS console your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (under your account name on the top right corner-> security credentials -> access keys)\n3. You should have a keypair attached to the zone you are running on (in our example us-west) as explained [here](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html). You will need to know your keypair name (amazonec2 in our example), and the location of the private key (~/.ssh/amazonec2.pem in our example).\n4. Install boto. This is the AWS Python client. To install, run: `sudo pip boto`.\n5. Download and install GraphLab using the instructions [here](/projects/source.html).\n\n## Step 1: Recommended setting\n\nWe recommend using high performance computing instances (like cc2.8xlarge) since we observed a significant improved performance especially related to variation in cluster load and network utilization. The scripts also allow using regular instances.\n\nTo avoid ec2 unexpected loads, we recommend repeating each experiment a few times and computing the average.\n\n## Step 2: Environment Setup\n\nEdit your .bashrc or .bash_profile or .profile files (remember to source it after editing, using the bash command “source ”)\n\n```\nexport AWS_ACCESS_KEY_ID=[ Your access key ]\nexport AWS_SECRET_ACCESS_KEY=[ Your access key secret ]\n```\n\n## Step 3: configure benchmarking\n\nEdit the [benchmark_ec2.sh](https://github.com/graphlab-code/graphlab/blob/master/scripts/ec2/benchmark_ec2.sh) script found under graphlab/scripts/ec2\n1. Select the requested algorithms of the following options:\n ```\nALS=1 # alternating least squares\nSVD=1 # singular value decomposition\nPR=1 # pagerank\n```\n(Setting an algorithm to 0 will disable its run).\n2. Select the number of slaves (any number between 0 to n) by setting the MAX_SLAVES variable.\n3. Select the number of experiment repeats (any number between 0 to n) by setting the MAX_RETRY variable. The benchmarking script, spawns an ec2 cluster of size n machines, and then tests the requested algorithm using 0, 1, … n-1 slaves. Each experiment is repeated MAX_RETRY times.\n\n### Step 3: Perform benchmarking\n\n```\ncd ~/graphlabapi/scripts/ec2\n./benchmark_ec2.sh\n```\nIt is advised to redirect the benchmarking output to file, for example on bash:\n\n ```\n ./benchmark_ec2 > output 2>&1\n ```\n\n### Step 4: Processing the results\n\nFor detecting final runtime for ALS/SVD\n\n```\ngrep \"Runtime\" output\n```\nFor detecting final runtime for PR:\n\n```\ngrep \"Finished Running\" output\n```\nYou will need to manually compute the average runtime for each case. A recommended metric to use is the “speedup” curve, which is the time for executing on a single machine divided by the time executing on k machines. The optimal result is linear speedup, namely running on k machines speeds up the algorithm k times vs. running on a single machine.\n\n### Step 5: behind the scenes\n\nHere is a more detailed explanation of the benchmarking process. The benchmarking is calling gl-ec2 script which calls [gl_ec2.py](https://github.com/graphlab-code/graphlab/blob/master/scripts/ec2/gl_ec2.py) script.\n1. The “launch” command to start a graphlab cluster with X machines.\n2. The “update” command to get the latest version of graphlab from git, recompile it, and disseminate the binary to the salves\n3. The “als_demo”, “svd_demo”, “pagerank_demo” command benchmark ALS/SVD/PR algorithms. It first downloads a dataset from the web and then calls graphlab with the right command lines to issue a run on the downloaded dataset. For PR we use the [LiveJournal](http://snap.stanford.edu/data/soc-LiveJournal1.html) dataset. For ALS/SVD we use a [netflix like synthetic sample](http://www.select.cs.cmu.edu/code/graphlab/datasets/smallnetflix_mm.train).\n4. In case you would like to benchmark a different dataset, you can edit the dataset URL in the gl_ec2.py example.\n5. In case you would like to benchmark a different algorithm, you can add an additional youralgo_demo section into the gl_ec2.py script.\n6. In case you would like to bechmark a regular instance, simply change the following line in gl_ec2.py from\n\n````\n./gl-ec2 -i ~/.ssh/amazonec2.pem -k amazonec2 -a hpc -s $MAX_SLAVES -t cc2.8xlarge launch hpctest\n```\nto:\n```\n./gl-ec2 -i ~/.ssh/amazonec2.pem -k amazonec2 -s $MAX_SLAVES -t m1.xlarge launch hpctest\n```\n\n### Advanced topics.\n\nIn case you like to work in a different ec2 region (than the default us-west):\n\nFor us-east region, those are the provided AMIs:\n\nStandard: ami-31360458, high performance: ami-39360450.\n\nYou should\n\n1. add the following line just before: [gl_ec2.py](https://github.com/graphlab-code/graphlab/blob/master/scripts/ec2/gl_ec2.py#L223)\n \n```\nopts.ami = \"ami-31360458\"\n```\n2. run with the additional command line argument:\n```\n-r us-east-1\n```\n\n### Support\n\nIf you encounter any problem when trying to run this benchmarking feel free to post on [forum.graphlab.com](http://forum.graphlab.com)\n\n\n# Fine tuning GraphLab PowerGraph performance\n\nThis section contains tips and examples on how to setup GraphLab properly on your cluster and how to squeeze performance.\n\n## 0: Compile in release\n\nVerify you compiled graphlab in the release subfolder (and not in debug subfolder). Compiling in release may speed execution up to x10 times!\n\nTip: Always compile in release when testing performance.\n\n## 1: Understanding input graph loading\n\nGraphLab PowerGraph has built in parallel loading of the input graph. However, for efficient parallel loading, the input file should be split into multiple disjoint sub files. When using a single input file, the graph loading becomes serial (which is bad!).\n\nEach MPIinstance has a single loader of the input graph attached to it (does not matter how many cpus are used by that MPI instance).\n\nTip: Always split your input file into at least as many MPI processes you are using.\n\n## 2: Verify MPI is working correctly\n\nYou can test your MPI setup as follows:\n\n1. Compile the release/demoapps/rpc subfolder (using “cd release/demoapps/rpc/; make”). Copy the files generated by the compile to all machines.\n2. Run:\n\n```\nmpiexec -n 2 --hostfile ~/machines /home/ubuntu/graphlab/release/demoapps/rpc/rpc_example1\n```\nAs part of the output, you should see something like this:\n\n```\nTCP Communication layer constructed.\nTCP Communication layer constructed.\n\n10\n5 plus 1 is : 6\n11 plus 1 is : 12\n```\n\nIf you get something else, please report an error as explained below\n\n## 3: Fine tuning of the partitioning.\n\nPrevious to the program execution, the graph is first loaded into memory and partitioned into the different cluster machines. It is possible to try different partitioning strategies. This is done using the following flags:\n\n```\n--graph_opts=\"ingress=oblivious\n```\n\nor\n\n````\n--graph_opts=\"ingress=grid\" # works for power of 2 sized cluster i.e. 2,4,8,.. machines\n```\n\nFor different graphs, different partitioning methods may give different performance gains.\n\n## 4: Setting ncpus\n\nThe –ncpus option let you set the number of cores used to perform computation. Prior to 2.1.4644 this defaults to 2. After 2.1.4644, this defaults to #cores – 2. When run in the distributed setting, the maximum number this should be set to is #cores – 2 since 2 cores should be reserved for communication.\n" }, { "path": "apps/CMakeLists.txt", "content": "project(GraphLab)\n\n# link_libraries(${Boost_LIBRARIES})\n# link_libraries(${GraphLab_LIBRARIES})\n\n\n\nmacro(add_all_subdirectories retval curdir)\n file(GLOB sub-dir RELATIVE ${curdir} *)\n set(list_of_dirs \"\")\n foreach(dir ${sub-dir})\n if(IS_DIRECTORY ${curdir}/${dir})\n STRING(SUBSTRING ${dir} 0 1 firstchar)\n if(${firstchar} STREQUAL \".\" OR ${firstchar} STREQUAL \"_\" )\n else(${firstchar} STREQUAL \".\" OR ${firstchar} STREQUAL \"_\")\n set(list_of_dirs ${list_of_dirs} ${dir})\n message(STATUS \"Detected App: \" ${dir})\n add_subdirectory(${dir})\n endif()\n endif()\n endforeach()\n set(${retval} ${list_of_dirs})\nendmacro()\n\nadd_all_subdirectories(retval, ${CMAKE_CURRENT_SOURCE_DIR})\n\n" }, { "path": "apps/cascades/CMakeLists.txt", "content": "project(cascades)\nadd_graphlab_executable(cascades cascades.cpp)\n" }, { "path": "apps/cascades/cascades.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \n#include \n#include \n#include \n\ndouble infection_chance;\ndouble recovery_chance;\n\nenum Status {INFECTED, SUSCEPTIBLE, RECOVERED};\n\n// The vertex data is its status (S, I, or R) \ntypedef Status vertex_data_type;\n\n// infected_status counts the number of infected neighbors, since\n// the number of infected neighbors determines how likely a susceptible node is\n// to be infected\nstruct infected_status: public graphlab::IS_POD_TYPE {\n int value;\n vertex_data_type status;\n\n infected_status() {\n value = 0;\n status = INFECTED;\n }\n\n infected_status& operator+=(const infected_status& other) {\n if (other.status == INFECTED) {\n this->value++;\n }\n\n return *this;\n }\n};\n\n\ntypedef infected_status gather_type;\n \n// The graph type is determined by the vertex and edge data types\ntypedef graphlab::distributed_graph graph_type;\n\nbool line_parser(graph_type& graph, const std::string& filename, const std::string& textline) {\n std::stringstream strm(textline);\n graphlab::vertex_id_type vid;\n\n char label;\n // first entry in the line is a vertex ID\n strm >> vid;\n\n // next entry is their status (S, I, or R)\n strm >> label;\n\n vertex_data_type statusLabel;\n if (label == 'S') {\n statusLabel = SUSCEPTIBLE;\n } else if (label == 'I') {\n statusLabel = INFECTED;\n } else {\n statusLabel = RECOVERED;\n }\n\n \n // insert this vertex with its label \n graph.add_vertex(vid, statusLabel);\n\n // while there are elements in the line, continue to read until we fail\n while(1) {\n graphlab::vertex_id_type other_vid;\n strm >> other_vid;\n if (strm.fail()) {\n break;\n }\n graph.add_edge(vid, other_vid);\n }\n\n return true;\n}\n\nclass cascades:\n public graphlab::ivertex_program,\n public graphlab::IS_POD_TYPE {\n\n public:\n edge_dir_type gather_edges(icontext_type& context, const vertex_type& vertex) const {\n return graphlab::ALL_EDGES;\n }\n\n gather_type gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n // figure out which data to get from the edge.\n bool isEdgeSource = (vertex.id() == edge.source().id());\n vertex_data_type neighbor_status = isEdgeSource ? edge.target().data() : edge.source().data();\n\n // create infected_status and add neighbor's status to it. \n\n infected_status status;\n status.status = neighbor_status;\n status.value = 1;\n \n return status;\n }\n\n void apply(icontext_type& context, vertex_type& vertex, const gather_type& total) {\n vertex_data_type old_data = vertex.data();\n\n vertex_data_type result = old_data;\n double random_value;\n\n // if vertex.data == RECOVERED, don't do anything\n // if vertex.data == INFECTED, roll on recovery_chance to see if recovery\n // occurs \n // if vertex.data == SUSCEPTIBLE, then do (total) dice rolls (each time comparing\n // the result to infection_chance). if any of them show up as positive,\n // set vertex.data to INFECTED. \n\n if (old_data != RECOVERED) {\n if (old_data == INFECTED) {\n random_value = ((double)rand())/RAND_MAX;\n if (random_value <= recovery_chance) {\n result = RECOVERED;\n }\n } else if (old_data == SUSCEPTIBLE) {\n for (int i = 0; i < total.value; i++) {\n random_value = ((double)rand())/RAND_MAX;\n if (random_value <= infection_chance) {\n result = INFECTED;\n break;\n }\n }\n }\n }\n\n vertex.data() = result;\n\n if (result == INFECTED) {\n context.signal(vertex);\n }\n }\n\n edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n return graphlab::NO_EDGES;\n }\n };\n\nstruct cascades_writer{\n std::string save_vertex(graph_type::vertex_type v) {\n std::stringstream strm;\n\n Status status = v.data();\n char vertex_data;\n \n // Convert the status back into a char\n if (status == INFECTED) {\n vertex_data = 'I';\n } else if (status == SUSCEPTIBLE) {\n vertex_data = 'S';\n } else {\n vertex_data = 'R';\n }\n strm << v.id() << \"\\t\" << vertex_data << \"\\n\";\n return strm.str();\n }\n std::string save_edge (graph_type::edge_type e) { return \"\"; }\n};\n\n\nint main(int argc, char** argv) {\n srand((unsigned) time(0));\n // Initialize control plain using mpi\n graphlab::mpi_tools::init(argc, argv);\n graphlab::distributed_control dc;\n global_logger().set_log_level(LOG_INFO);\n \n // Parse command line options -----------------------------------------------\n graphlab::command_line_options clopts(\"Label Propagation algorithm.\");\n std::string graph_dir;\n std::string execution_type = \"synchronous\";\n double recovery = -1;\n double infection = -1;\n size_t iterations = -1;\n\n clopts.attach_option(\"graph\", graph_dir, \"The graph file. Required \");\n clopts.add_positional(\"graph\");\n clopts.attach_option(\"execution\", execution_type, \"Execution type (synchronous or asynchronous)\");\n\n std::string saveprefix;\n clopts.attach_option(\"saveprefix\", saveprefix,\n \"If set, will save the resultant pagerank to a \"\n \"sequence of files with prefix saveprefix\");\n\n clopts.attach_option(\"recovery chance\", recovery, \"Chance of recovery for an infected individual at each step. Required.\");\n clopts.attach_option(\"infection chance\", infection, \"Chance of infection for a susceptible individual per person at each step. Required.\");\n\n clopts.attach_option(\"iterations\", iterations, \"If set, will force the use of synchronous engine overriding any engine option set by the --engine parameter. Runs cascades for a fixed number of iterations. Also overrides the max_iterations option in the engine.\");\n\n if(!clopts.parse(argc, argv)) {\n dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n return EXIT_FAILURE;\n }\n if (graph_dir == \"\") {\n dc.cout() << \"Graph not specified. Cannot continue\";\n return EXIT_FAILURE;\n }\n\n if (recovery == -1) {\n dc.cout() << \"Recovery chance not specified. Cannot continue\";\n return EXIT_FAILURE;\n }\n\n if (infection == -1) {\n dc.cout() << \"Infection chance not specified. Cannot continue\";\n return EXIT_FAILURE;\n }\n\n infection_chance = infection;\n recovery_chance = recovery;\n \n if (iterations != -1) {\n // make sure this is the synchronous engine\n dc.cout() << \"--iterations set. Forcing Synchronous engine, and running for \"\n << iterations << \" iterations.\" << std::endl;\n clopts.get_engine_args().set_option(\"type\", \"synchronous\");\n clopts.get_engine_args().set_option(\"max_iterations\", iterations);\n }\n \n // Build the graph ----------------------------------------------------------\n graph_type graph(dc);\n dc.cout() << \"Loading graph using line parser\" << std::endl;\n graph.load(graph_dir, line_parser);\n // must call finalize before querying the graph\n graph.finalize();\n\n dc.cout() << \"#vertices: \" << graph.num_vertices() << \" #edges:\" << graph.num_edges() << std::endl;\n\n graphlab::omni_engine engine(dc, graph, execution_type, clopts);\n\n engine.signal_all();\n engine.start();\n\n const float runtime = engine.elapsed_seconds();\n dc.cout() << \"Finished Running engine in \" << runtime << \" seconds.\" << std::endl;\n\n if (saveprefix != \"\") {\n graph.save(saveprefix, cascades_writer(),\n false, // do not gzip\n true, //save vertices\n false); // do not save edges \n }\n \n\n graphlab::mpi_tools::finalize();\n return EXIT_SUCCESS;\n}\n" }, { "path": "apps/concomp/CMakeLists.txt", "content": "project(example)\nadd_graphlab_executable(concomp concomp.cpp)\n" }, { "path": "apps/concomp/concomp.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \n\n// The vertex data is just the id value\ntypedef graphlab::vertex_id_type vertex_data_type;\ntypedef graphlab::vertex_id_type edge_data_type;\n\ntypedef double gather_type;\n\n// The graph type is determined by the vertex and edge data types\ntypedef graphlab::distributed_graph graph_type;\n\n/*\n * A simple function used by graph.transform_vertices(init_vertex);\n * to initialize the vertex data to the vertex id\n */\nvoid init_vertex(graph_type::vertex_type& vertex) { vertex.data() = -1; }\n\nstruct min_combiner : public graphlab::IS_POD_TYPE {\n vertex_data_type value;\n\n min_combiner() {\n value = -1;\n }\n \n min_combiner& operator+=(const min_combiner& other) { \n if (other.value < value) {\n value = other.value;\n }\n return *this; \n }\n};\n\nclass concomp :\n public graphlab::ivertex_program,\n public graphlab::IS_POD_TYPE {\n\n // set changed to determine which edges to scatter on\n bool changed;\n\n // local copy of the message value\n vertex_data_type message_value; \npublic:\n // Receive inbound message (minimum data of adjacent vertices)\n void init(icontext_type& context, const vertex_type& vertex, const message_type& message) {\n // message.value == 4294967295 on first run, so init message_value to vertex data.\n if (message.value == 4294967295) {\n message_value = vertex.id();\n } else {\n // else, set the local copy to the message parameter.\n message_value = message.value;\n }\n\n }\n\n edge_dir_type gather_edges(icontext_type& context, const vertex_type& vertex) const {\n return graphlab::NO_EDGES;\n }\n\n // Change the vertex data if any of its neighbors have a lower data value.\n void apply(icontext_type& context, vertex_type& vertex, const gather_type& total) {\n // mark if values differ to determine which edges to scatter on.\n if (message_value < vertex.data()) {\n changed = true;\n vertex.data() = message_value;\n } else {\n changed = false;\n }\n }\n\n edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n // If the vertex data changed, scatter along all edges. Otherwise stop.\n if (changed) {\n return graphlab::ALL_EDGES;\n } else {\n return graphlab::NO_EDGES;\n }\n }\n\n // Scatter to scatter_edges edges with the new message value.\n void scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n bool isEdgeSource = (vertex.id() == edge.source().id());\n bool hasSameData = isEdgeSource ? (vertex.data() == edge.target().data()) : (vertex.data() == edge.source().data()) ;\n if (!hasSameData) {\n min_combiner combiner;\n combiner.value = message_value;\n\n context.signal(isEdgeSource ? edge.target() : edge.source(), combiner);\n }\n }\n};\n\n/* We want to save the final graph so we define a write which will be\n* used in graph.save(\"path/prefix\", concomp_writer()) to save the graph.\n*/\nstruct concomp_writer {\n std::string save_vertex(graph_type::vertex_type v) {\n std::stringstream strm;\n strm << v.id() << \"\\t\" << v.data() << \"\\n\";\n return strm.str();\n }\n std::string save_edge(graph_type::edge_type e) { return \"\"; }\n}; // end of concomp writer\n \n\nint main(int argc, char** argv) {\n // Initialize control plain using mpi\n graphlab::mpi_tools::init(argc, argv);\n graphlab::distributed_control dc;\n global_logger().set_log_level(LOG_INFO);\n \n // Parse command line options -----------------------------------------------\n graphlab::command_line_options clopts(\"Connected Components algorithm.\");\n std::string graph_dir;\n std::string format = \"snap\";\n std::string execution_type = \"synchronous\";\n clopts.attach_option(\"graph\", graph_dir, \"The graph file. Required \");\n clopts.add_positional(\"graph\");\n clopts.attach_option(\"format\", format, \"The graph file format\");\n clopts.attach_option(\"execution\", execution_type, \"Execution type (synchronous or asynchronous)\");\n\n std::string saveprefix;\n clopts.attach_option(\"saveprefix\", saveprefix,\n \"If set, will save the resultant pagerank to a \"\n \"sequence of files with prefix saveprefix\");\n\n if(!clopts.parse(argc, argv)) {\n dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n return EXIT_FAILURE;\n }\n if (graph_dir == \"\") {\n dc.cout() << \"Graph not specified. Cannot continue\";\n return EXIT_FAILURE;\n }\n \n // Build the graph ----------------------------------------------------------\n graph_type graph(dc, clopts);\n dc.cout() << \"Loading graph in format: \"<< format << std::endl;\n graph.load_format(graph_dir, format);\n // must call finalize before querying the graph\n graph.finalize();\n\n graph.transform_vertices(init_vertex);\n\n dc.cout() << \"#vertices: \" << graph.num_vertices() << \" #edges:\" << graph.num_edges() << std::endl;\n\n graphlab::omni_engine engine(dc, graph, execution_type, clopts);\n\n min_combiner initial_message;\n initial_message.value = -1;\n\n engine.signal_all(initial_message);\n\n engine.start();\n\n const float runtime = engine.elapsed_seconds();\n dc.cout() << \"Finished Running engine in \" << runtime << \" seconds.\" << std::endl;\n\n if (saveprefix != \"\") {\n graph.save(saveprefix, concomp_writer(),\n false, // do not gzip\n true, // save vertices\n false); // do not save edges\n }\n\n graphlab::mpi_tools::finalize();\n\n return EXIT_SUCCESS;\n}\n" }, { "path": "apps/example/CMakeLists.txt", "content": "project(example)\nadd_graphlab_executable(hello_world hello_world.cpp)\n" }, { "path": "apps/example/hello_world.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \n\n// The vertex data is just the pagerank value (a float)\ntypedef float vertex_data_type;\n\n// There is no edge data in the pagerank application\ntypedef float edge_data_type;\ntypedef float message_type;\n\n// The graph type is determined by the vertex and edge data types\ntypedef graphlab::distributed_graph graph_type;\n\n/*\n * A simple function used by graph.transform_vertices(init_vertex);\n * to initialize the vertex data.\n */\nvoid init_vertex(graph_type::vertex_type& vertex) { vertex.data() = vertex.id(); }\n\nstruct min_combiner {\n graphlab::vertex_id_type v;\n min_combiner& operator+=(const min_combiner& other) { \n v = std::min(v, other.v); \n return *this; \n }\n};\n\nclass concomp :\n public graphlab::ivertex_program,\n public graphlab::IS_POD_TYPE {\n bool changed;\npublic:\n float gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n float edge_data = edge.source().data();\n float vertex_data = vertex.data();\n\n std::cout << \"current vertex id: \" << vertex.id() << \" data: \" << vertex.data() << \"\\n\";\n\n\n std::cout << \"\\tedge vertex id: \" << edge.source().id() << \" data: \" << edge_data << \"\\n\";\n if (edge_data < vertex_data) {\n std::cout << \"returning edge data: \" << edge_data << \"\\n\";\n return edge_data;\n } else {\n std::cout << \"returning vertex data: \" << vertex_data << \"\\n\";\n return vertex_data;\n }\n }\n\n void apply(icontext_type& context, vertex_type& vertex, const gather_type& smallest) {\n std::cout << \"vertex id: \" << vertex.id() << \" data: \" << vertex.data() << \"\\n\";\n std::cout << \"smallest: \" << smallest << \"\\n\";\n\n if (smallest < vertex.data()) {\n vertex.data() = smallest;\n changed = true;\n } else {\n changed = false;\n }\n }\n\n edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n if (changed) {\n return graphlab::ALL_EDGES;\n } else {\n return graphlab::NO_EDGES;\n }\n }\n\n /* The scatter function just signal adjacent pages */\n void scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n context.signal(edge.target());\n }\n};\n\n/* We want to save the final graph so we define a write which will be\n * used in graph.save(\"path/prefix\", concomp_writer()) to save the graph.\n */\nstruct concomp_writer {\n std::string save_vertex(graph_type::vertex_type v) {\n std::stringstream strm;\n strm << v.id() << \"\\t\" << v.data() << \"\\n\";\n return strm.str();\n }\n std::string save_edge(graph_type::edge_type e) { return \"\"; }\n}; // end of concomp writer\n \n\nint main(int argc, char** argv) {\n // Initialize control plain using mpi\n graphlab::mpi_tools::init(argc, argv);\n graphlab::distributed_control dc;\n global_logger().set_log_level(LOG_INFO);\n \n // Parse command line options -----------------------------------------------\n graphlab::command_line_options clopts(\"PageRank algorithm.\");\n std::string graph_dir;\n std::string format = \"snap\";\n clopts.attach_option(\"graph\", graph_dir, \"The graph file. Required \");\n clopts.add_positional(\"graph\");\n clopts.attach_option(\"format\", format, \"The graph file format\");\n if(!clopts.parse(argc, argv)) {\n dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n return EXIT_FAILURE;\n }\n if (graph_dir == \"\") {\n dc.cout() << \"Graph not specified. Cannot continue\";\n return EXIT_FAILURE;\n }\n \n // Build the graph ----------------------------------------------------------\n graph_type graph(dc, clopts);\n dc.cout() << \"Loading graph in format: \"<< format << std::endl;\n graph.load_format(graph_dir, format);\n // must call finalize before querying the graph\n graph.finalize();\n\n graph.transform_vertices(init_vertex);\n\n dc.cout() << \"#vertices: \" << graph.num_vertices() << \" #edges:\" << graph.num_edges() << std::endl;\n\n graphlab::omni_engine engine(dc, graph, \"synchronous\", clopts);\n engine.signal_all();\n\n engine.start();\n\n const float runtime = engine.elapsed_seconds();\n dc.cout() << \"Finished Running engine in \" << runtime << \" seconds.\" << std::endl;\n\n graph.save(\"output\" + graph_dir + \".txt\", concomp_writer(),\n false, // do not gzip\n true, // save vertices\n false); // do not save edges\n}\n" }, { "path": "apps/label_propagation/CMakeLists.txt", "content": "project(label_propagation)\nadd_graphlab_executable(label_propagation label_propagation.cpp)\n" }, { "path": "apps/label_propagation/label_propagation.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \n\nstruct label_counter {\n std::map label_count;\n\n label_counter() {\n }\n \n label_counter& operator+=(const label_counter& other) { \n for ( std::map::const_iterator iter = other.label_count.begin();\n iter != other.label_count.end(); ++iter ) {\n label_count[iter->first] += iter->second;\n }\n\n return *this; \n }\n\n void save(graphlab::oarchive& oarc) const {\n oarc << label_count;\n }\n\n void load(graphlab::iarchive& iarc) {\n iarc >> label_count;\n }\n};\n\n// The vertex data is its label \ntypedef std::string vertex_data_type;\n\ntypedef label_counter gather_type;\n \n// The graph type is determined by the vertex and edge data types\ntypedef graphlab::distributed_graph graph_type;\n\nbool line_parser(graph_type& graph, const std::string& filename, const std::string& textline) {\n std::stringstream strm(textline);\n graphlab::vertex_id_type vid;\n std::string label;\n // first entry in the line is a vertex ID\n strm >> vid;\n strm >> label;\n // insert this vertex with its label \n graph.add_vertex(vid, label);\n // while there are elements in the line, continue to read until we fail\n while(1){\n graphlab::vertex_id_type other_vid;\n strm >> other_vid;\n if (strm.fail())\n break;\n graph.add_edge(vid, other_vid);\n }\n\n return true;\n}\n\nclass labelpropagation :\n public graphlab::ivertex_program,\n public graphlab::IS_POD_TYPE {\n bool changed;\n\n public:\n edge_dir_type gather_edges(icontext_type& context, const vertex_type& vertex) const {\n return graphlab::ALL_EDGES;\n }\n\n gather_type gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n // figure out which data to get from the edge.\n bool isEdgeSource = (vertex.id() == edge.source().id());\n std::string neighbor_label = isEdgeSource ? edge.target().data() : edge.source().data();\n\n // make a label_counter and place the neighbor data in it\n label_counter counter;\n counter.label_count[neighbor_label] = 1;\n\n\n // gather_type is a label counter, so += will add neighbor counts to the\n // label_count map.\n return counter;\n }\n\n void apply(icontext_type& context, vertex_type& vertex, const gather_type& total) {\n\n int maxCount = 0;\n\n std::string maxLabel = vertex.data();\n\n // Figure out which label of the vertex's neighbors' labels is most common\n for ( std::map::const_iterator iter = total.label_count.begin();\n iter != total.label_count.end(); ++iter ) {\n if (iter->second > maxCount) {\n maxCount = iter->second;\n maxLabel = iter->first;\n }\n }\n\n \n // if maxLabel differs to vertex data, mark vertex as changed and update\n // its data.\n if ((vertex.data()).compare(maxLabel) != 0) {\n changed = true;\n vertex.data() = maxLabel;\n } else {\n changed = false;\n }\n }\n\n edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n // if vertex data changes, scatter to all edges.\n if (changed) {\n return graphlab::ALL_EDGES;\n } else {\n return graphlab::NO_EDGES;\n }\n }\n\n void scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n bool isEdgeSource = (vertex.id() == edge.source().id());\n\n context.signal(isEdgeSource ? edge.target() : edge.source()); \n }\n };\n\nstruct labelpropagation_writer {\n std::string save_vertex(graph_type::vertex_type v) {\n std::stringstream strm;\n strm << v.id() << \"\\t\" << v.data() << \"\\n\";\n return strm.str();\n }\n std::string save_edge (graph_type::edge_type e) { return \"\"; }\n};\n\n\nint main(int argc, char** argv) {\n // Initialize control plain using mpi\n graphlab::mpi_tools::init(argc, argv);\n graphlab::distributed_control dc;\n global_logger().set_log_level(LOG_INFO);\n \n // Parse command line options -----------------------------------------------\n graphlab::command_line_options clopts(\"Label Propagation algorithm.\");\n std::string graph_dir;\n std::string execution_type = \"synchronous\";\n clopts.attach_option(\"graph\", graph_dir, \"The graph file. Required \");\n clopts.add_positional(\"graph\");\n clopts.attach_option(\"execution\", execution_type, \"Execution type (synchronous or asynchronous)\");\n\n std::string saveprefix;\n clopts.attach_option(\"saveprefix\", saveprefix,\n \"If set, will save the resultant pagerank to a \"\n \"sequence of files with prefix saveprefix\");\n\n if(!clopts.parse(argc, argv)) {\n dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n return EXIT_FAILURE;\n }\n if (graph_dir == \"\") {\n dc.cout() << \"Graph not specified. Cannot continue\";\n return EXIT_FAILURE;\n }\n \n // Build the graph ----------------------------------------------------------\n graph_type graph(dc);\n dc.cout() << \"Loading graph using line parser\" << std::endl;\n graph.load(graph_dir, line_parser);\n // must call finalize before querying the graph\n graph.finalize();\n\n dc.cout() << \"#vertices: \" << graph.num_vertices() << \" #edges:\" << graph.num_edges() << std::endl;\n\n graphlab::omni_engine engine(dc, graph, execution_type, clopts);\n\n engine.signal_all();\n engine.start();\n\n const float runtime = engine.elapsed_seconds();\n dc.cout() << \"Finished Running engine in \" << runtime << \" seconds.\" << std::endl;\n\n if (saveprefix != \"\") {\n graph.save(saveprefix, labelpropagation_writer(),\n false, // do not gzip\n true, //save vertices\n false); // do not save edges \n }\n \n\n graphlab::mpi_tools::finalize();\n return EXIT_SUCCESS;\n}\n" }, { "path": "cmake/FindAnt.cmake", "content": "# - Try to find Ant\nfind_file(ANT_EXEC NAMES ant ant.sh ant.bat PATHS $ENV{ANT_HOME}/bin)\n\nINCLUDE(FindPackageHandleStandardArgs)\nFIND_PACKAGE_HANDLE_STANDARD_ARGS(ANT DEFAULT_MSG ANT_EXEC)\n\nMARK_AS_ADVANCED(ANT_EXEC)" }, { "path": "cmake/FindBoost.cmake", "content": "# - Try to find Boost include dirs and libraries\n# Usage of this module as follows:\n#\n# SET(Boost_USE_STATIC_LIBS ON)\n# SET(Boost_USE_MULTITHREAD OFF)\n# FIND_PACKAGE( Boost 1.34.1 COMPONENTS date_time filesystem iostreams ... )\n#\n# The Boost_ADDITIONAL_VERSIONS variable can be used to specify a list of\n# boost version numbers that should be taken into account when searching\n# for the libraries. Unfortunately boost puts the version number into the\n# actual filename for the libraries, so this might be needed in the future\n# when new Boost versions are released.\n#\n# Currently this module searches for the following version numbers:\n# 1.33, 1.33.0, 1.33.1, 1.34, 1.34.0, 1.34.1, 1.35, 1.35.0, 1.35.1, 1.36, \n# 1.36.0, 1.36.1\n#\n# The components list needs to be the actual names of boost libraries, that is\n# the part of the actual library files that differ on different libraries. So\n# its \"date_time\" for \"libboost_date_time...\". Anything else will result in\n# errors\n#\n# You can provide a minimum version number that should be used. If you provide this \n# version number and specify the REQUIRED attribute, this module will fail if it\n# can't find the specified or a later version. If you specify a version number this is\n# automatically put into the considered list of version numbers and thus doesn't need\n# to be specified in the Boost_ADDITIONAL_VERSIONS variable\n#\n# Variables used by this module, they can change the default behaviour and need to be set\n# before calling find_package:\n# Boost_USE_MULTITHREAD Can be set to OFF to use the non-multithreaded\n# boost libraries. Defaults to ON.\n# Boost_USE_STATIC_LIBS Can be set to ON to force the use of the static\n# boost libraries. Defaults to OFF.\n# Boost_ADDITIONAL_VERSIONS A list of version numbers to use for searching\n# the boost include directory. The default list\n# of version numbers is:\n# 1.33, 1.33.0, 1.33.1, 1.34, 1.34.0, 1.34.1, \n# 1.35, 1.35.0, 1.35.1, 1.36, 1.36.0, 1.36.1\n# If you want to look for an older or newer\n# version set this variable to a list of\n# strings, where each string contains a number, i.e.\n# SET(Boost_ADDITIONAL_VERSIONS \"0.99.0\" \"1.35.0\")\n# BOOST_ROOT or BOOSTROOT Preferred installation prefix for searching for Boost,\n# set this if the module has problems finding the proper Boost installation\n# BOOST_INCLUDEDIR Set this to the include directory of Boost, if the\n# module has problems finding the proper Boost installation\n# BOOST_LIBRARYDIR Set this to the lib directory of Boost, if the\n# module has problems finding the proper Boost installation\n#\n# The last three variables are available also as environment variables\n#\n#\n# Variables defined by this module:\n#\n# Boost_FOUND System has Boost, this means the include dir was found,\n# as well as all the libraries specified in the COMPONENTS list\n# Boost_INCLUDE_DIRS Boost include directories, not cached\n# Boost_INCLUDE_DIR This is almost the same as above, but this one is cached and may be\n# modified by advanced users\n# Boost_LIBRARIES Link these to use the Boost libraries that you specified, not cached\n# Boost_LIBRARY_DIRS The path to where the Boost library files are.\n# Boost_VERSION The version number of the boost libraries that have been found,\n# same as in version.hpp from Boost\n# Boost_LIB_VERSION The version number in filename form as its appended to the library filenames\n# Boost_MAJOR_VERSION major version number of boost\n# Boost_MINOR_VERSION minor version number of boost\n# Boost_SUBMINOR_VERSION subminor version number of boost\n# Boost_LIB_DIAGNOSTIC_DEFINITIONS Only set on windows. Can be used with add_definitions \n# to print diagnostic information about the automatic \n# linking done on windows.\n\n# For each component you list the following variables are set.\n# ATTENTION: The component names need to be in lower case, just as the boost\n# library names however the cmake variables use upper case for the component\n# part. So you'd get Boost_SERIALIZATION_FOUND for example.\n#\n# Boost_${COMPONENT}_FOUND True IF the Boost library \"component\" was found.\n# Boost_${COMPONENT}_LIBRARY The absolute path of the Boost library \"component\".\n# Boost_${COMPONENT}_LIBRARY_DEBUG The absolute path of the debug version of the\n# Boost library \"component\".\n# Boost_${COMPONENT}_LIBRARY_RELEASE The absolute path of the release version of the\n# Boost library \"component\"\n#\n# Copyright (c) 2006-2008 Andreas Schneider \n# Copyright (c) 2007 Wengo\n# Copyright (c) 2007 Mike Jackson\n# Copyright (c) 2008 Andreas Pakulat \n#\n# Redistribution AND use is allowed according to the terms of the New\n# BSD license.\n# For details see the accompanying COPYING-CMAKE-SCRIPTS file.\n#\nOPTION(Boost_USE_MULTITHREADED \n \"Use the multithreaded versions of the Boost libraries\" OFF)\n\nif (Boost_FIND_VERSION_EXACT)\n if (Boost_FIND_VERSION_PATCH)\n set( _boost_TEST_VERSIONS \n \"${Boost_FIND_VERSION_MAJOR}.${Boost_FIND_VERSION_MINOR}.${Boost_FIND_VERSION_PATCH}\")\n else (Boost_FIND_VERSION_PATCH)\n set( _boost_TEST_VERSIONS \n \"${Boost_FIND_VERSION_MAJOR}.${Boost_FIND_VERSION_MINOR}.0\"\n \"${Boost_FIND_VERSION_MAJOR}.${Boost_FIND_VERSION_MINOR}\")\n endif (Boost_FIND_VERSION_PATCH)\nelse (Boost_FIND_VERSION_EXACT)\n set( _boost_TEST_VERSIONS ${Boost_ADDITIONAL_VERSIONS} \n \"1.43\" \"1.42\" \"1.41\" \"1.40\" \"1.39\" \"1.38\" \"1.37\" )\nendif (Boost_FIND_VERSION_EXACT)\n\n# The reason that we failed to find Boost. This will be set to a\n# user-friendly message when we fail to find some necessary piece of\n# Boost.\nset(Boost_ERROR_REASON)\n\n############################################\n#\n# Check the existence of the libraries.\n#\n############################################\n# This macro was taken directly from the FindQt4.cmake file that is included\n# with the CMake distribution. This is NOT my work. All work was done by the\n# original authors of the FindQt4.cmake file. Only minor modifications were\n# made to remove references to Qt and make this file more generally applicable\n#########################################################################\n\nMACRO (_Boost_ADJUST_LIB_VARS basename)\n IF (Boost_INCLUDE_DIR )\n IF (Boost_${basename}_LIBRARY_DEBUG AND Boost_${basename}_LIBRARY_RELEASE)\n # if the generator supports configuration types then set\n # optimized and debug libraries, or if the CMAKE_BUILD_TYPE has a value\n IF (CMAKE_CONFIGURATION_TYPES OR CMAKE_BUILD_TYPE)\n SET(Boost_${basename}_LIBRARY optimized ${Boost_${basename}_LIBRARY_RELEASE} debug ${Boost_${basename}_LIBRARY_DEBUG})\n ELSE(CMAKE_CONFIGURATION_TYPES OR CMAKE_BUILD_TYPE)\n # if there are no configuration types and CMAKE_BUILD_TYPE has no value\n # then just use the release libraries\n SET(Boost_${basename}_LIBRARY ${Boost_${basename}_LIBRARY_RELEASE} )\n ENDIF(CMAKE_CONFIGURATION_TYPES OR CMAKE_BUILD_TYPE)\n SET(Boost_${basename}_LIBRARIES optimized ${Boost_${basename}_LIBRARY_RELEASE} debug ${Boost_${basename}_LIBRARY_DEBUG})\n ENDIF (Boost_${basename}_LIBRARY_DEBUG AND Boost_${basename}_LIBRARY_RELEASE)\n\n # if only the release version was found, set the debug variable also to the release version\n IF (Boost_${basename}_LIBRARY_RELEASE AND NOT Boost_${basename}_LIBRARY_DEBUG)\n SET(Boost_${basename}_LIBRARY_DEBUG ${Boost_${basename}_LIBRARY_RELEASE})\n SET(Boost_${basename}_LIBRARY ${Boost_${basename}_LIBRARY_RELEASE})\n SET(Boost_${basename}_LIBRARIES ${Boost_${basename}_LIBRARY_RELEASE})\n ENDIF (Boost_${basename}_LIBRARY_RELEASE AND NOT Boost_${basename}_LIBRARY_DEBUG)\n\n # if only the debug version was found, set the release variable also to the debug version\n IF (Boost_${basename}_LIBRARY_DEBUG AND NOT Boost_${basename}_LIBRARY_RELEASE)\n SET(Boost_${basename}_LIBRARY_RELEASE ${Boost_${basename}_LIBRARY_DEBUG})\n SET(Boost_${basename}_LIBRARY ${Boost_${basename}_LIBRARY_DEBUG})\n SET(Boost_${basename}_LIBRARIES ${Boost_${basename}_LIBRARY_DEBUG})\n ENDIF (Boost_${basename}_LIBRARY_DEBUG AND NOT Boost_${basename}_LIBRARY_RELEASE)\n \n IF (Boost_${basename}_LIBRARY)\n SET(Boost_${basename}_LIBRARY ${Boost_${basename}_LIBRARY} CACHE FILEPATH \"The Boost ${basename} library\")\n GET_FILENAME_COMPONENT(Boost_LIBRARY_DIRS \"${Boost_${basename}_LIBRARY}\" PATH)\n SET(Boost_LIBRARY_DIRS ${Boost_LIBRARY_DIRS} CACHE FILEPATH \"Boost library directory\")\n SET(Boost_${basename}_FOUND ON CACHE INTERNAL \"Whether the Boost ${basename} library found\")\n ENDIF (Boost_${basename}_LIBRARY)\n\n ENDIF (Boost_INCLUDE_DIR )\n # Make variables changeble to the advanced user\n MARK_AS_ADVANCED(\n Boost_${basename}_LIBRARY\n Boost_${basename}_LIBRARY_RELEASE\n Boost_${basename}_LIBRARY_DEBUG\n )\nENDMACRO (_Boost_ADJUST_LIB_VARS)\n\n#-------------------------------------------------------------------------------\n\n\nSET( _boost_IN_CACHE TRUE)\nIF(Boost_INCLUDE_DIR)\n FOREACH(COMPONENT ${Boost_FIND_COMPONENTS})\n STRING(TOUPPER ${COMPONENT} COMPONENT)\n IF(NOT Boost_${COMPONENT}_FOUND)\n SET( _boost_IN_CACHE FALSE)\n ENDIF(NOT Boost_${COMPONENT}_FOUND)\n ENDFOREACH(COMPONENT)\nELSE(Boost_INCLUDE_DIR)\n SET( _boost_IN_CACHE FALSE)\nENDIF(Boost_INCLUDE_DIR)\n\nIF (_boost_IN_CACHE)\n # in cache already\n SET(Boost_FOUND TRUE)\n FOREACH(COMPONENT ${Boost_FIND_COMPONENTS})\n STRING(TOUPPER ${COMPONENT} COMPONENT)\n _Boost_ADJUST_LIB_VARS( ${COMPONENT} )\n SET(Boost_LIBRARIES ${Boost_LIBRARIES} ${Boost_${COMPONENT}_LIBRARY})\n ENDFOREACH(COMPONENT)\n SET(Boost_INCLUDE_DIRS ${Boost_INCLUDE_DIR})\n IF(Boost_VERSION AND NOT \"${Boost_VERSION}\" STREQUAL \"0\")\n MATH(EXPR Boost_MAJOR_VERSION \"${Boost_VERSION} / 100000\")\n MATH(EXPR Boost_MINOR_VERSION \"${Boost_VERSION} / 100 % 1000\")\n MATH(EXPR Boost_SUBMINOR_VERSION \"${Boost_VERSION} % 100\")\n ENDIF(Boost_VERSION AND NOT \"${Boost_VERSION}\" STREQUAL \"0\")\nELSE (_boost_IN_CACHE)\n # Need to search for boost\n\n IF(WIN32)\n # In windows, automatic linking is performed, so you do not have\n # to specify the libraries. If you are linking to a dynamic\n # runtime, then you can choose to link to either a static or a\n # dynamic Boost library, the default is to do a static link. You\n # can alter this for a specific library \"whatever\" by defining\n # BOOST_WHATEVER_DYN_LINK to force Boost library \"whatever\" to be\n # linked dynamically. Alternatively you can force all Boost\n # libraries to dynamic link by defining BOOST_ALL_DYN_LINK.\n \n # This feature can be disabled for Boost library \"whatever\" by\n # defining BOOST_WHATEVER_NO_LIB, or for all of Boost by defining\n # BOOST_ALL_NO_LIB.\n \n # If you want to observe which libraries are being linked against\n # then defining BOOST_LIB_DIAGNOSTIC will cause the auto-linking\n # code to emit a #pragma message each time a library is selected\n # for linking.\n SET(Boost_LIB_DIAGNOSTIC_DEFINITIONS \n \"-DBOOST_LIB_DIAGNOSTIC\" CACHE STRING \"Boost diagnostic define\")\n ENDIF(WIN32)\n\n SET(_boost_INCLUDE_SEARCH_DIRS\n C:/boost/include\n \"C:/boost\"\n \"$ENV{ProgramFiles}/boost/boost_${Boost_FIND_VERSION_MAJOR}_${Boost_FIND_VERSION_MINOR}_${Boost_FIND_VERSION_PATCH}\"\n \"$ENV{ProgramFiles}/Boost\"\n /sw/local/include\n )\n\n SET(_boost_LIBRARIES_SEARCH_DIRS\n C:/boost/lib\n \"C:/boost\"\n \"$ENV{ProgramFiles}/boost/boost_${Boost_FIND_VERSION_MAJOR}_${Boost_FIND_VERSION_MINOR}_${Boost_FIND_VERSION_PATCH}/lib\"\n \"$ENV{ProgramFiles}/Boost\"\n /sw/local/lib\n )\n\n # If BOOST_ROOT was defined in the environment, use it.\n if (NOT BOOST_ROOT AND NOT $ENV{BOOST_ROOT} STREQUAL \"\")\n set(BOOST_ROOT $ENV{BOOST_ROOT})\n endif(NOT BOOST_ROOT AND NOT $ENV{BOOST_ROOT} STREQUAL \"\")\n\n # If BOOSTROOT was defined in the environment, use it.\n if (NOT BOOST_ROOT AND NOT $ENV{BOOSTROOT} STREQUAL \"\")\n set(BOOST_ROOT $ENV{BOOSTROOT})\n endif(NOT BOOST_ROOT AND NOT $ENV{BOOSTROOT} STREQUAL \"\")\n\n # If BOOST_INCLUDEDIR was defined in the environment, use it.\n IF( NOT $ENV{BOOST_INCLUDEDIR} STREQUAL \"\" )\n set(BOOST_INCLUDEDIR $ENV{BOOST_INCLUDEDIR})\n ENDIF( NOT $ENV{BOOST_INCLUDEDIR} STREQUAL \"\" )\n\n # If BOOST_LIBRARYDIR was defined in the environment, use it.\n IF( NOT $ENV{BOOST_LIBRARYDIR} STREQUAL \"\" )\n set(BOOST_LIBRARYDIR $ENV{BOOST_LIBRARYDIR})\n ENDIF( NOT $ENV{BOOST_LIBRARYDIR} STREQUAL \"\" )\n\n IF( BOOST_ROOT )\n file(TO_CMAKE_PATH ${BOOST_ROOT} BOOST_ROOT)\n SET(_boost_INCLUDE_SEARCH_DIRS \n ${BOOST_ROOT}/include \n ${BOOST_ROOT}\n ${_boost_INCLUDE_SEARCH_DIRS})\n SET(_boost_LIBRARIES_SEARCH_DIRS \n ${BOOST_ROOT}/lib \n ${BOOST_ROOT}/stage/lib \n ${_boost_LIBRARIES_SEARCH_DIRS})\n ENDIF( BOOST_ROOT )\n\n IF( BOOST_INCLUDEDIR )\n file(TO_CMAKE_PATH ${BOOST_INCLUDEDIR} BOOST_INCLUDEDIR)\n SET(_boost_INCLUDE_SEARCH_DIRS \n ${BOOST_INCLUDEDIR} ${_boost_INCLUDE_SEARCH_DIRS})\n ENDIF( BOOST_INCLUDEDIR )\n\n IF( BOOST_LIBRARYDIR )\n file(TO_CMAKE_PATH ${BOOST_LIBRARYDIR} BOOST_LIBRARYDIR)\n SET(_boost_LIBRARIES_SEARCH_DIRS \n ${BOOST_LIBRARYDIR} ${_boost_LIBRARIES_SEARCH_DIRS})\n ENDIF( BOOST_LIBRARYDIR )\n\n # Try to find Boost by stepping backwards through the Boost versions\n # we know about.\n IF( NOT Boost_INCLUDE_DIR )\n # Build a list of path suffixes for each version.\n SET(_boost_PATH_SUFFIXES)\n FOREACH(_boost_VER ${_boost_TEST_VERSIONS})\n # Add in a path suffix, based on the required version, ideally\n # we could read this from version.hpp, but for that to work we'd\n # need to know the include dir already\n if (WIN32 AND NOT CYGWIN)\n set(_boost_PATH_SUFFIX boost_${_boost_VER})\n else (WIN32 AND NOT CYGWIN)\n set(_boost_PATH_SUFFIX boost-${_boost_VER})\n endif (WIN32 AND NOT CYGWIN)\n\n IF(_boost_PATH_SUFFIX MATCHES \"[0-9]+\\\\.[0-9]+\\\\.[0-9]+\")\n STRING(REGEX REPLACE \"([0-9]+)\\\\.([0-9]+)\\\\.([0-9]+)\" \"\\\\1_\\\\2_\\\\3\" \n _boost_PATH_SUFFIX ${_boost_PATH_SUFFIX})\n ELSEIF(_boost_PATH_SUFFIX MATCHES \"[0-9]+\\\\.[0-9]+\")\n STRING(REGEX REPLACE \"([0-9]+)\\\\.([0-9]+)\" \"\\\\1_\\\\2\" \n _boost_PATH_SUFFIX ${_boost_PATH_SUFFIX})\n ENDIF(_boost_PATH_SUFFIX MATCHES \"[0-9]+\\\\.[0-9]+\\\\.[0-9]+\")\n LIST(APPEND _boost_PATH_SUFFIXES \"${_boost_PATH_SUFFIX}\")\n ENDFOREACH(_boost_VER)\n\n # Look for a standard boost header file.\n FIND_PATH(Boost_INCLUDE_DIR\n NAMES boost/config.hpp\n HINTS ${_boost_INCLUDE_SEARCH_DIRS}\n PATH_SUFFIXES ${_boost_PATH_SUFFIXES}\n )\n ENDIF( NOT Boost_INCLUDE_DIR )\n\n IF(Boost_INCLUDE_DIR)\n # Extract Boost_VERSION and Boost_LIB_VERSION from version.hpp\n # Read the whole file:\n #\n SET(BOOST_VERSION 0)\n SET(BOOST_LIB_VERSION \"\")\n FILE(READ \"${Boost_INCLUDE_DIR}/boost/version.hpp\" _boost_VERSION_HPP_CONTENTS)\n \n STRING(REGEX REPLACE \".*#define BOOST_VERSION ([0-9]+).*\" \"\\\\1\" Boost_VERSION \"${_boost_VERSION_HPP_CONTENTS}\")\n STRING(REGEX REPLACE \".*#define BOOST_LIB_VERSION \\\"([0-9_]+)\\\".*\" \"\\\\1\" Boost_LIB_VERSION \"${_boost_VERSION_HPP_CONTENTS}\")\n \n SET(Boost_LIB_VERSION ${Boost_LIB_VERSION} CACHE INTERNAL \"The library version string for boost libraries\")\n SET(Boost_VERSION ${Boost_VERSION} CACHE INTERNAL \"The version number for boost libraries\")\n \n IF(NOT \"${Boost_VERSION}\" STREQUAL \"0\")\n MATH(EXPR Boost_MAJOR_VERSION \"${Boost_VERSION} / 100000\")\n MATH(EXPR Boost_MINOR_VERSION \"${Boost_VERSION} / 100 % 1000\")\n MATH(EXPR Boost_SUBMINOR_VERSION \"${Boost_VERSION} % 100\")\n\n set(Boost_ERROR_REASON\n \"${Boost_ERROR_REASON}Boost version: ${Boost_MAJOR_VERSION}.${Boost_MINOR_VERSION}.${Boost_SUBMINOR_VERSION}\\nBoost include path: ${Boost_INCLUDE_DIR}\")\n ENDIF(NOT \"${Boost_VERSION}\" STREQUAL \"0\")\n ELSE(Boost_INCLUDE_DIR)\n set(Boost_ERROR_REASON\n \"${Boost_ERROR_REASON}Unable to find the Boost header files. Please set BOOST_ROOT to the root directory containing Boost or BOOST_INCLUDEDIR to the directory containing Boost's headers.\")\n ENDIF(Boost_INCLUDE_DIR)\n\n # Setting some more suffixes for the library\n SET (Boost_LIB_PREFIX \"\")\n\n # Stano: cygwin does not seem to need the lib prefix, is this MSVC-only thing?\n # IF ( WIN32 AND Boost_USE_STATIC_LIBS )\n # SET (Boost_LIB_PREFIX \"lib\")\n # ENDIF ( WIN32 AND Boost_USE_STATIC_LIBS )\n\n IF ( MSVC AND Boost_USE_STATIC_LIBS )\n SET (Boost_LIB_PREFIX \"lib\")\n ENDIF ( MSVC AND Boost_USE_STATIC_LIBS )\n\n SET (_boost_COMPILER \"-gcc\")\n IF (MSVC90)\n SET (_boost_COMPILER \"-vc90\")\n ELSEIF (MSVC80)\n SET (_boost_COMPILER \"-vc80\")\n ELSEIF (MSVC71)\n SET (_boost_COMPILER \"-vc71\")\n ENDIF(MSVC90)\n IF (MINGW)\n EXEC_PROGRAM(${CMAKE_CXX_COMPILER}\n ARGS -dumpversion\n OUTPUT_VARIABLE _boost_COMPILER_VERSION\n )\n STRING(REGEX REPLACE \"([0-9])\\\\.([0-9])\\\\.[0-9]\" \"\\\\1\\\\2\"\n _boost_COMPILER_VERSION ${_boost_COMPILER_VERSION})\n SET (_boost_COMPILER \"-mgw${_boost_COMPILER_VERSION}\")\n ENDIF(MINGW)\n IF (UNIX)\n IF (NOT CMAKE_COMPILER_IS_GNUCC)\n # We assume that we have the Intel compiler.\n SET (_boost_COMPILER \"-il\")\n ELSE (NOT CMAKE_COMPILER_IS_GNUCC)\n # Determine which version of GCC we have.\n EXEC_PROGRAM(${CMAKE_CXX_COMPILER}\n ARGS -dumpversion\n OUTPUT_VARIABLE _boost_COMPILER_VERSION\n )\n STRING(REGEX REPLACE \"([0-9])\\\\.([0-9])\\\\.[0-9]\" \"\\\\1\\\\2\"\n _boost_COMPILER_VERSION ${_boost_COMPILER_VERSION})\n IF(APPLE)\n IF(Boost_MINOR_VERSION)\n IF(${Boost_MINOR_VERSION} GREATER 35)\n # In Boost 1.36.0 and newer, the mangled compiler name used\n # on Mac OS X/Darwin is \"xgcc\".\n SET(_boost_COMPILER \"-xgcc${_boost_COMPILER_VERSION}\")\n ELSE(${Boost_MINOR_VERSION} GREATER 35)\n # In Boost <= 1.35.0, there is no mangled compiler name for\n # the Mac OS X/Darwin version of GCC.\n SET(_boost_COMPILER \"\")\n ENDIF(${Boost_MINOR_VERSION} GREATER 35)\n ELSE(Boost_MINOR_VERSION)\n # We don't know the Boost version, so assume it's\n # pre-1.36.0.\n SET(_boost_COMPILER \"\")\n ENDIF(Boost_MINOR_VERSION)\n ELSE(APPLE)\n SET (_boost_COMPILER \"-gcc${_boost_COMPILER_VERSION}\")\n ENDIF(APPLE)\n ENDIF (NOT CMAKE_COMPILER_IS_GNUCC)\n ENDIF(UNIX)\n\n\n\n SET (_boost_MULTITHREADED \"-mt\")\n\n IF( NOT Boost_USE_MULTITHREADED )\n SET (_boost_MULTITHREADED \"\")\n ENDIF( NOT Boost_USE_MULTITHREADED )\n\n SET( _boost_STATIC_TAG \"\")\n IF (WIN32)\n IF(MSVC)\n SET (_boost_ABI_TAG \"g\")\n ENDIF(MSVC)\n IF( Boost_USE_STATIC_LIBS )\n SET( _boost_STATIC_TAG \"-s\")\n ENDIF( Boost_USE_STATIC_LIBS )\n ENDIF(WIN32)\n SET (_boost_ABI_TAG \"${_boost_ABI_TAG}d\")\n\n # ------------------------------------------------------------------------\n # Begin finding boost libraries\n # ------------------------------------------------------------------------\n FOREACH(COMPONENT ${Boost_FIND_COMPONENTS})\n STRING(TOUPPER ${COMPONENT} UPPERCOMPONENT)\n SET( Boost_${UPPERCOMPONENT}_LIBRARY \"Boost_${UPPERCOMPONENT}_LIBRARY-NOTFOUND\" )\n SET( Boost_${UPPERCOMPONENT}_LIBRARY_RELEASE \"Boost_${UPPERCOMPONENT}_LIBRARY_RELEASE-NOTFOUND\" )\n SET( Boost_${UPPERCOMPONENT}_LIBRARY_DEBUG \"Boost_${UPPERCOMPONENT}_LIBRARY_DEBUG-NOTFOUND\")\n\n # Support preference of static libs by adjusting CMAKE_FIND_LIBRARY_SUFFIXES\n IF( Boost_USE_STATIC_LIBS )\n SET( _boost_ORIG_CMAKE_FIND_LIBRARY_SUFFIXES ${CMAKE_FIND_LIBRARY_SUFFIXES})\n IF(WIN32)\n SET(CMAKE_FIND_LIBRARY_SUFFIXES .lib .a ${CMAKE_FIND_LIBRARY_SUFFIXES})\n ELSE(WIN32)\n SET(CMAKE_FIND_LIBRARY_SUFFIXES .a ${CMAKE_FIND_LIBRARY_SUFFIXES})\n ENDIF(WIN32)\n ENDIF( Boost_USE_STATIC_LIBS )\n\n FIND_LIBRARY(Boost_${UPPERCOMPONENT}_LIBRARY_RELEASE\n NAMES ${Boost_LIB_PREFIX}boost_${COMPONENT}${_boost_COMPILER}${_boost_MULTITHREADED}-${Boost_LIB_VERSION}\n ${Boost_LIB_PREFIX}boost_${COMPONENT}${_boost_COMPILER}${_boost_MULTITHREADED}${_boost_STATIC_TAG}-${Boost_LIB_VERSION}\n ${Boost_LIB_PREFIX}boost_${COMPONENT}${_boost_MULTITHREADED}\n ${Boost_LIB_PREFIX}boost_${COMPONENT}${_boost_MULTITHREADED}${_boost_STATIC_TAG}\n ${Boost_LIB_PREFIX}boost_${COMPONENT}\n HINTS ${_boost_LIBRARIES_SEARCH_DIRS}\n )\n\n FIND_LIBRARY(Boost_${UPPERCOMPONENT}_LIBRARY_DEBUG\n NAMES ${Boost_LIB_PREFIX}boost_${COMPONENT}${_boost_COMPILER}${_boost_MULTITHREADED}-${_boost_ABI_TAG}-${Boost_LIB_VERSION}\n ${Boost_LIB_PREFIX}boost_${COMPONENT}${_boost_COMPILER}${_boost_MULTITHREADED}${_boost_STATIC_TAG}${_boost_ABI_TAG}-${Boost_LIB_VERSION}\n ${Boost_LIB_PREFIX}boost_${COMPONENT}${_boost_MULTITHREADED}-${_boost_ABI_TAG}\n ${Boost_LIB_PREFIX}boost_${COMPONENT}${_boost_MULTITHREADED}${_boost_STATIC_TAG}${_boost_ABI_TAG}\n ${Boost_LIB_PREFIX}boost_${COMPONENT}-${_boost_ABI_TAG}\n HINTS ${_boost_LIBRARIES_SEARCH_DIRS}\n )\n\n _Boost_ADJUST_LIB_VARS(${UPPERCOMPONENT})\n IF( Boost_USE_STATIC_LIBS )\n SET(CMAKE_FIND_LIBRARY_SUFFIXES ${_boost_ORIG_CMAKE_FIND_LIBRARY_SUFFIXES})\n ENDIF( Boost_USE_STATIC_LIBS )\n ENDFOREACH(COMPONENT)\n # ------------------------------------------------------------------------\n # End finding boost libraries\n # ------------------------------------------------------------------------\n\n SET(Boost_INCLUDE_DIRS\n ${Boost_INCLUDE_DIR}\n )\n\n SET(Boost_FOUND FALSE)\n IF(Boost_INCLUDE_DIR)\n SET( Boost_FOUND TRUE )\n\n # Check the version of Boost against the requested version.\n if (Boost_FIND_VERSION AND NOT Boost_FIND_VERSION_MINOR)\n message(SEND_ERROR \"When requesting a specific version of Boost, you must provide at least the major and minor version numbers, e.g., 1.34\")\n endif (Boost_FIND_VERSION AND NOT Boost_FIND_VERSION_MINOR)\n if(Boost_MAJOR_VERSION LESS \"${Boost_FIND_VERSION_MAJOR}\" )\n set( Boost_FOUND FALSE )\n set(_Boost_VERSION_AGE \"old\")\n elseif(Boost_MAJOR_VERSION EQUAL \"${Boost_FIND_VERSION_MAJOR}\" )\n if(Boost_MINOR_VERSION LESS \"${Boost_FIND_VERSION_MINOR}\" )\n set( Boost_FOUND FALSE )\n set(_Boost_VERSION_AGE \"old\")\n elseif(Boost_MINOR_VERSION EQUAL \"${Boost_FIND_VERSION_MINOR}\" )\n if( Boost_FIND_VERSION_PATCH AND Boost_SUBMINOR_VERSION LESS \"${Boost_FIND_VERSION_PATCH}\" )\n set( Boost_FOUND FALSE )\n set(_Boost_VERSION_AGE \"old\")\n endif( Boost_FIND_VERSION_PATCH AND Boost_SUBMINOR_VERSION LESS \"${Boost_FIND_VERSION_PATCH}\" )\n endif( Boost_MINOR_VERSION LESS \"${Boost_FIND_VERSION_MINOR}\" )\n endif( Boost_MAJOR_VERSION LESS \"${Boost_FIND_VERSION_MAJOR}\" )\n\n if (Boost_FOUND AND Boost_FIND_VERSION_EXACT)\n # If the user requested an exact version of Boost, check\n # that. We already know that the Boost version we have is >= the\n # requested version.\n set(_Boost_VERSION_AGE \"new\")\n\n # If the user didn't specify a patchlevel, it's 0.\n if (NOT Boost_FIND_VERSION_PATCH)\n set(Boost_FIND_VERSION_PATCH 0)\n endif (NOT Boost_FIND_VERSION_PATCH)\n \n # We'll set Boost_FOUND true again if we have an exact version match.\n set(Boost_FOUND FALSE)\n if(Boost_MAJOR_VERSION EQUAL \"${Boost_FIND_VERSION_MAJOR}\" )\n if(Boost_MINOR_VERSION EQUAL \"${Boost_FIND_VERSION_MINOR}\" )\n if(Boost_SUBMINOR_VERSION EQUAL \"${Boost_FIND_VERSION_PATCH}\" )\n set( Boost_FOUND TRUE )\n endif(Boost_SUBMINOR_VERSION EQUAL \"${Boost_FIND_VERSION_PATCH}\" )\n endif( Boost_MINOR_VERSION EQUAL \"${Boost_FIND_VERSION_MINOR}\" )\n endif( Boost_MAJOR_VERSION EQUAL \"${Boost_FIND_VERSION_MAJOR}\" )\n endif (Boost_FOUND AND Boost_FIND_VERSION_EXACT)\n\n if(NOT Boost_FOUND)\n # State that we found a version of Boost that is too new or too old.\n set(Boost_ERROR_REASON\n \"${Boost_ERROR_REASON}\\nDetected version of Boost is too ${_Boost_VERSION_AGE}. Requested version was ${Boost_FIND_VERSION_MAJOR}.${Boost_FIND_VERSION_MINOR}\")\n if (Boost_FIND_VERSION_PATCH)\n set(Boost_ERROR_REASON \n \"${Boost_ERROR_REASON}.${Boost_FIND_VERSION_PATCH}\")\n endif (Boost_FIND_VERSION_PATCH)\n if (NOT Boost_FIND_VERSION_EXACT)\n set(Boost_ERROR_REASON \"${Boost_ERROR_REASON} (or newer)\")\n endif (NOT Boost_FIND_VERSION_EXACT)\n set(Boost_ERROR_REASON \"${Boost_ERROR_REASON}.\")\n endif (NOT Boost_FOUND)\n\n if (Boost_FOUND)\n set(_boost_CHECKED_COMPONENT FALSE)\n set(_Boost_MISSING_COMPONENTS)\n foreach(COMPONENT ${Boost_FIND_COMPONENTS})\n string(TOUPPER ${COMPONENT} COMPONENT)\n set(_boost_CHECKED_COMPONENT TRUE)\n if(NOT Boost_${COMPONENT}_FOUND)\n string(TOLOWER ${COMPONENT} COMPONENT)\n list(APPEND _Boost_MISSING_COMPONENTS ${COMPONENT})\n set( Boost_FOUND FALSE)\n endif(NOT Boost_${COMPONENT}_FOUND)\n endforeach(COMPONENT)\n endif (Boost_FOUND)\n\n if (_Boost_MISSING_COMPONENTS)\n # We were unable to find some libraries, so generate a sensible\n # error message that lists the libraries we were unable to find.\n set(Boost_ERROR_REASON\n \"${Boost_ERROR_REASON}\\nThe following Boost libraries could not be found:\\n\")\n foreach(COMPONENT ${_Boost_MISSING_COMPONENTS})\n set(Boost_ERROR_REASON\n \"${Boost_ERROR_REASON} boost_${COMPONENT}\\n\")\n endforeach(COMPONENT)\n\n list(LENGTH Boost_FIND_COMPONENTS Boost_NUM_COMPONENTS_WANTED)\n list(LENGTH _Boost_MISSING_COMPONENTS Boost_NUM_MISSING_COMPONENTS)\n if (${Boost_NUM_COMPONENTS_WANTED} EQUAL ${Boost_NUM_MISSING_COMPONENTS})\n set(Boost_ERROR_REASON\n \"${Boost_ERROR_REASON}No Boost libraries were found. You may need to set Boost_LIBRARYDIR to the directory containing Boost libraries or BOOST_ROOT to the location of Boost.\")\n else (${Boost_NUM_COMPONENTS_WANTED} EQUAL ${Boost_NUM_MISSING_COMPONENTS})\n set(Boost_ERROR_REASON\n \"${Boost_ERROR_REASON}Some (but not all) of the required Boost libraries were found. You may need to install these additional Boost libraries. Alternatively, set Boost_LIBRARYDIR to the directory containing Boost libraries or BOOST_ROOT to the location of Boost.\")\n endif (${Boost_NUM_COMPONENTS_WANTED} EQUAL ${Boost_NUM_MISSING_COMPONENTS})\n endif (_Boost_MISSING_COMPONENTS)\n\n IF( NOT Boost_LIBRARY_DIRS AND NOT _boost_CHECKED_COMPONENT )\n # Compatibility Code for backwards compatibility with CMake\n # 2.4's FindBoost module.\n\n # Look for the boost library path.\n # Note that the user may not have installed any libraries\n # so it is quite possible the Boost_LIBRARY_PATH may not exist.\n SET(_boost_LIB_DIR ${Boost_INCLUDE_DIR})\n \n IF(\"${_boost_LIB_DIR}\" MATCHES \"boost-[0-9]+\")\n GET_FILENAME_COMPONENT(_boost_LIB_DIR ${_boost_LIB_DIR} PATH)\n ENDIF (\"${_boost_LIB_DIR}\" MATCHES \"boost-[0-9]+\")\n \n IF(\"${_boost_LIB_DIR}\" MATCHES \"/include$\")\n # Strip off the trailing \"/include\" in the path.\n GET_FILENAME_COMPONENT(_boost_LIB_DIR ${_boost_LIB_DIR} PATH)\n ENDIF(\"${_boost_LIB_DIR}\" MATCHES \"/include$\")\n \n IF(EXISTS \"${_boost_LIB_DIR}/lib\")\n SET (_boost_LIB_DIR ${_boost_LIB_DIR}/lib)\n ELSE(EXISTS \"${_boost_LIB_DIR}/lib\")\n IF(EXISTS \"${_boost_LIB_DIR}/stage/lib\")\n SET(_boost_LIB_DIR ${_boost_LIB_DIR}/stage/lib)\n ELSE(EXISTS \"${_boost_LIB_DIR}/stage/lib\")\n SET(_boost_LIB_DIR \"\")\n ENDIF(EXISTS \"${_boost_LIB_DIR}/stage/lib\")\n ENDIF(EXISTS \"${_boost_LIB_DIR}/lib\")\n \n IF(_boost_LIB_DIR AND EXISTS \"${_boost_LIB_DIR}\")\n SET(Boost_LIBRARY_DIRS ${_boost_LIB_DIR} CACHE FILEPATH \"Boost library directory\")\n ENDIF(_boost_LIB_DIR AND EXISTS \"${_boost_LIB_DIR}\")\n\n ENDIF( NOT Boost_LIBRARY_DIRS AND NOT _boost_CHECKED_COMPONENT )\n\n ELSE(Boost_INCLUDE_DIR)\n SET( Boost_FOUND FALSE)\n ENDIF(Boost_INCLUDE_DIR)\n\n IF (Boost_FOUND)\n IF (NOT Boost_FIND_QUIETLY)\n MESSAGE(STATUS \"Boost version: ${Boost_MAJOR_VERSION}.${Boost_MINOR_VERSION}.${Boost_SUBMINOR_VERSION}\")\n ENDIF(NOT Boost_FIND_QUIETLY)\n IF (NOT Boost_FIND_QUIETLY)\n MESSAGE(STATUS \"Found the following Boost libraries:\")\n ENDIF(NOT Boost_FIND_QUIETLY)\n FOREACH ( COMPONENT ${Boost_FIND_COMPONENTS} )\n STRING( TOUPPER ${COMPONENT} UPPERCOMPONENT )\n IF ( Boost_${UPPERCOMPONENT}_FOUND )\n IF (NOT Boost_FIND_QUIETLY)\n MESSAGE (STATUS \" ${COMPONENT}\")\n ENDIF(NOT Boost_FIND_QUIETLY)\n SET(Boost_LIBRARIES ${Boost_LIBRARIES} ${Boost_${UPPERCOMPONENT}_LIBRARY})\n ENDIF ( Boost_${UPPERCOMPONENT}_FOUND )\n ENDFOREACH(COMPONENT)\n ELSE (Boost_FOUND)\n IF (Boost_FIND_REQUIRED)\n message(SEND_ERROR \"Unable to find the requested Boost libraries.\\n${Boost_ERROR_REASON}\")\n ENDIF(Boost_FIND_REQUIRED)\n ENDIF(Boost_FOUND)\n\n # Under Windows, automatic linking is performed, so no need to specify the libraries.\n IF (WIN32)\n IF (NOT MINGW)\n SET(Boost_LIBRARIES \"\")\n ENDIF (NOT MINGW)\n ENDIF(WIN32)\n\n # show the Boost_INCLUDE_DIRS AND Boost_LIBRARIES variables only in the advanced view\n MARK_AS_ADVANCED(Boost_INCLUDE_DIR\n Boost_INCLUDE_DIRS\n Boost_LIBRARY_DIRS\n Boost_USE_MULTITHREADED\n )\nENDIF(_boost_IN_CACHE)\n\n" }, { "path": "cmake/FindCUDA/make2cmake.cmake", "content": "# James Bigler, NVIDIA Corp (nvidia.com - jbigler)\n# Abe Stephens, SCI Institute -- http://www.sci.utah.edu/~abe/FindCuda.html\n#\n# Copyright (c) 2008 - 2009 NVIDIA Corporation. All rights reserved.\n#\n# Copyright (c) 2007-2009\n# Scientific Computing and Imaging Institute, University of Utah\n#\n# This code is licensed under the MIT License. See the FindCUDA.cmake script\n# for the text of the license.\n\n# The MIT License\n#\n# License for the specific language governing rights and limitations under\n# Permission is hereby granted, free of charge, to any person obtaining a\n# copy of this software and associated documentation files (the \"Software\"),\n# to deal in the Software without restriction, including without limitation\n# the rights to use, copy, modify, merge, publish, distribute, sublicense,\n# and/or sell copies of the Software, and to permit persons to whom the\n# Software is furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included\n# in all copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS\n# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL\n# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\n# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER\n# DEALINGS IN THE SOFTWARE.\n#\n\n#######################################################################\n# This converts a file written in makefile syntax into one that can be included\n# by CMake.\n\nfile(READ ${input_file} depend_text)\n\nif (${depend_text} MATCHES \".+\")\n\n # message(\"FOUND DEPENDS\")\n\n # Remember, four backslashes is escaped to one backslash in the string.\n string(REGEX REPLACE \"\\\\\\\\ \" \" \" depend_text ${depend_text})\n\n # This works for the nvcc -M generated dependency files.\n string(REGEX REPLACE \"^.* : \" \"\" depend_text ${depend_text})\n string(REGEX REPLACE \"[ \\\\\\\\]*\\n\" \";\" depend_text ${depend_text})\n\n set(dependency_list \"\")\n\n foreach(file ${depend_text})\n\n string(REGEX REPLACE \"^ +\" \"\" file ${file})\n\n if(NOT IS_DIRECTORY ${file})\n # If softlinks start to matter, we should change this to REALPATH. For now we need\n # to flatten paths, because nvcc can generate stuff like /bin/../include instead of\n # just /include.\n get_filename_component(file_absolute \"${file}\" ABSOLUTE)\n list(APPEND dependency_list \"${file_absolute}\")\n endif(NOT IS_DIRECTORY ${file})\n\n endforeach(file)\n\nelse()\n # message(\"FOUND NO DEPENDS\")\nendif()\n\n# Remove the duplicate entries and sort them.\nlist(REMOVE_DUPLICATES dependency_list)\nlist(SORT dependency_list)\n\nforeach(file ${dependency_list})\n set(cuda_nvcc_depend \"${cuda_nvcc_depend} \\\"${file}\\\"\\n\")\nendforeach()\n\nfile(WRITE ${output_file} \"# Generated by: make2cmake.cmake\\nSET(CUDA_NVCC_DEPEND\\n ${cuda_nvcc_depend})\\n\\n\")\n" }, { "path": "cmake/FindCUDA/parse_cubin.cmake", "content": "# James Bigler, NVIDIA Corp (nvidia.com - jbigler)\n# Abe Stephens, SCI Institute -- http://www.sci.utah.edu/~abe/FindCuda.html\n#\n# Copyright (c) 2008 - 2009 NVIDIA Corporation. All rights reserved.\n#\n# Copyright (c) 2007-2009\n# Scientific Computing and Imaging Institute, University of Utah\n#\n# This code is licensed under the MIT License. See the FindCUDA.cmake script\n# for the text of the license.\n\n# The MIT License\n#\n# License for the specific language governing rights and limitations under\n# Permission is hereby granted, free of charge, to any person obtaining a\n# copy of this software and associated documentation files (the \"Software\"),\n# to deal in the Software without restriction, including without limitation\n# the rights to use, copy, modify, merge, publish, distribute, sublicense,\n# and/or sell copies of the Software, and to permit persons to whom the\n# Software is furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included\n# in all copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS\n# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL\n# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\n# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER\n# DEALINGS IN THE SOFTWARE.\n#\n\n#######################################################################\n# Parses a .cubin file produced by nvcc and reports statistics about the file.\n\n\nfile(READ ${input_file} file_text)\n\nif (${file_text} MATCHES \".+\")\n\n # Remember, four backslashes is escaped to one backslash in the string.\n string(REGEX REPLACE \";\" \"\\\\\\\\;\" file_text ${file_text})\n string(REGEX REPLACE \"\\ncode\" \";code\" file_text ${file_text})\n\n list(LENGTH file_text len)\n\n foreach(line ${file_text})\n\n # Only look at \"code { }\" blocks.\n if(line MATCHES \"^code\")\n\n # Break into individual lines.\n string(REGEX REPLACE \"\\n\" \";\" line ${line})\n\n foreach(entry ${line})\n\n # Extract kernel names.\n if (${entry} MATCHES \"[^g]name = ([^ ]+)\")\n string(REGEX REPLACE \".* = ([^ ]+)\" \"\\\\1\" entry ${entry})\n\n # Check to see if the kernel name starts with \"_\"\n set(skip FALSE)\n # if (${entry} MATCHES \"^_\")\n # Skip the rest of this block.\n # message(\"Skipping ${entry}\")\n # set(skip TRUE)\n # else (${entry} MATCHES \"^_\")\n message(\"Kernel: ${entry}\")\n # endif (${entry} MATCHES \"^_\")\n\n endif(${entry} MATCHES \"[^g]name = ([^ ]+)\")\n\n # Skip the rest of the block if necessary\n if(NOT skip)\n\n # Registers\n if (${entry} MATCHES \"reg([ ]+)=([ ]+)([^ ]+)\")\n string(REGEX REPLACE \".*([ ]+)=([ ]+)([^ ]+)\" \"\\\\3\" entry ${entry})\n message(\"Registers: ${entry}\")\n endif()\n\n # Local memory\n if (${entry} MATCHES \"lmem([ ]+)=([ ]+)([^ ]+)\")\n string(REGEX REPLACE \".*([ ]+)=([ ]+)([^ ]+)\" \"\\\\3\" entry ${entry})\n message(\"Local: ${entry}\")\n endif()\n\n # Shared memory\n if (${entry} MATCHES \"smem([ ]+)=([ ]+)([^ ]+)\")\n string(REGEX REPLACE \".*([ ]+)=([ ]+)([^ ]+)\" \"\\\\3\" entry ${entry})\n message(\"Shared: ${entry}\")\n endif()\n\n if (${entry} MATCHES \"^}\")\n message(\"\")\n endif()\n\n endif(NOT skip)\n\n\n endforeach(entry)\n\n endif(line MATCHES \"^code\")\n\n endforeach(line)\n\nelse()\n # message(\"FOUND NO DEPENDS\")\nendif()\n\n\n" }, { "path": "cmake/FindCUDA/run_nvcc.cmake", "content": "# James Bigler, NVIDIA Corp (nvidia.com - jbigler)\n#\n# Copyright (c) 2008 - 2009 NVIDIA Corporation. All rights reserved.\n#\n# This code is licensed under the MIT License. See the FindCUDA.cmake script\n# for the text of the license.\n\n# The MIT License\n#\n# License for the specific language governing rights and limitations under\n# Permission is hereby granted, free of charge, to any person obtaining a\n# copy of this software and associated documentation files (the \"Software\"),\n# to deal in the Software without restriction, including without limitation\n# the rights to use, copy, modify, merge, publish, distribute, sublicense,\n# and/or sell copies of the Software, and to permit persons to whom the\n# Software is furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included\n# in all copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS\n# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL\n# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\n# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER\n# DEALINGS IN THE SOFTWARE.\n\n\n##########################################################################\n# This file runs the nvcc commands to produce the desired output file along with\n# the dependency file needed by CMake to compute dependencies. In addition the\n# file checks the output of each command and if the command fails it deletes the\n# output files.\n\n# Input variables\n#\n# verbose:BOOL=<> OFF: Be as quiet as possible (default)\n# ON : Describe each step\n#\n# build_configuration:STRING=<> Typically one of Debug, MinSizeRel, Release, or\n# RelWithDebInfo, but it should match one of the\n# entries in CUDA_HOST_FLAGS. This is the build\n# configuration used when compiling the code. If\n# blank or unspecified Debug is assumed as this is\n# what CMake does.\n#\n# generated_file:STRING=<> File to generate. This argument must be passed in.\n#\n# generated_cubin_file:STRING=<> File to generate. This argument must be passed\n# in if build_cubin is true.\n\nif(NOT generated_file)\n message(FATAL_ERROR \"You must specify generated_file on the command line\")\nendif()\n\n# Set these up as variables to make reading the generated file easier\nset(CMAKE_COMMAND \"@CMAKE_COMMAND@\")\nset(source_file \"@source_file@\")\nset(NVCC_generated_dependency_file \"@NVCC_generated_dependency_file@\")\nset(cmake_dependency_file \"@cmake_dependency_file@\")\nset(CUDA_make2cmake \"@CUDA_make2cmake@\")\nset(CUDA_parse_cubin \"@CUDA_parse_cubin@\")\nset(build_cubin @build_cubin@)\n# We won't actually use these variables for now, but we need to set this, in\n# order to force this file to be run again if it changes.\nset(generated_file_path \"@generated_file_path@\")\nset(generated_file_internal \"@generated_file@\")\nset(generated_cubin_file_internal \"@generated_cubin_file@\")\n\nset(CUDA_NVCC_EXECUTABLE \"@CUDA_NVCC_EXECUTABLE@\")\nset(CUDA_NVCC_FLAGS \"@CUDA_NVCC_FLAGS@;;@CUDA_WRAP_OPTION_NVCC_FLAGS@\")\n@CUDA_NVCC_FLAGS_CONFIG@\nset(nvcc_flags \"@nvcc_flags@\")\nset(CUDA_NVCC_INCLUDE_ARGS \"@CUDA_NVCC_INCLUDE_ARGS@\")\nset(format_flag \"@format_flag@\")\n\nif(build_cubin AND NOT generated_cubin_file)\n message(FATAL_ERROR \"You must specify generated_cubin_file on the command line\")\nendif()\n\n# This is the list of host compilation flags. It C or CXX should already have\n# been chosen by FindCUDA.cmake.\n@CUDA_HOST_FLAGS@\n\n# Take the compiler flags and package them up to be sent to the compiler via -Xcompiler\nset(nvcc_host_compiler_flags \"\")\n# If we weren't given a build_configuration, use Debug.\nif(NOT build_configuration)\n set(build_configuration Debug)\nendif()\nstring(TOUPPER \"${build_configuration}\" build_configuration)\n#message(\"CUDA_NVCC_HOST_COMPILER_FLAGS = ${CUDA_NVCC_HOST_COMPILER_FLAGS}\")\nforeach(flag ${CMAKE_HOST_FLAGS} ${CMAKE_HOST_FLAGS_${build_configuration}})\n # Extra quotes are added around each flag to help nvcc parse out flags with spaces.\n set(nvcc_host_compiler_flags \"${nvcc_host_compiler_flags},\\\"${flag}\\\"\")\nendforeach()\nif (nvcc_host_compiler_flags)\n set(nvcc_host_compiler_flags \"-Xcompiler\" ${nvcc_host_compiler_flags})\nendif()\n#message(\"nvcc_host_compiler_flags = \\\"${nvcc_host_compiler_flags}\\\"\")\n# Add the build specific configuration flags\nlist(APPEND CUDA_NVCC_FLAGS ${CUDA_NVCC_FLAGS_${build_configuration}})\n\nif(DEFINED CCBIN)\n set(CCBIN -ccbin \"${CCBIN}\")\nendif()\n\n# cuda_execute_process - Executes a command with optional command echo and status message.\n#\n# status - Status message to print if verbose is true\n# command - COMMAND argument from the usual execute_process argument structure\n# ARGN - Remaining arguments are the command with arguments\n#\n# CUDA_result - return value from running the command\n#\n# Make this a macro instead of a function, so that things like RESULT_VARIABLE\n# and other return variables are present after executing the process.\nmacro(cuda_execute_process status command)\n set(_command ${command})\n if(NOT _command STREQUAL \"COMMAND\")\n message(FATAL_ERROR \"Malformed call to cuda_execute_process. Missing COMMAND as second argument. (command = ${command})\")\n endif()\n if(verbose)\n execute_process(COMMAND \"${CMAKE_COMMAND}\" -E echo -- ${status})\n # Now we need to build up our command string. We are accounting for quotes\n # and spaces, anything else is left up to the user to fix if they want to\n # copy and paste a runnable command line.\n set(cuda_execute_process_string)\n foreach(arg ${ARGN})\n # If there are quotes, excape them, so they come through.\n string(REPLACE \"\\\"\" \"\\\\\\\"\" arg ${arg})\n # Args with spaces need quotes around them to get them to be parsed as a single argument.\n if(arg MATCHES \" \")\n list(APPEND cuda_execute_process_string \"\\\"${arg}\\\"\")\n else()\n list(APPEND cuda_execute_process_string ${arg})\n endif()\n endforeach()\n # Echo the command\n execute_process(COMMAND ${CMAKE_COMMAND} -E echo ${cuda_execute_process_string})\n endif(verbose)\n # Run the command\n execute_process(COMMAND ${ARGN} RESULT_VARIABLE CUDA_result )\nendmacro()\n\n# Delete the target file\ncuda_execute_process(\n \"Removing ${generated_file}\"\n COMMAND \"${CMAKE_COMMAND}\" -E remove \"${generated_file}\"\n )\n\n# For CUDA 2.3 and below, -G -M doesn't work, so remove the -G flag\n# for dependency generation and hope for the best.\nset(depends_CUDA_NVCC_FLAGS \"${CUDA_NVCC_FLAGS}\")\nset(CUDA_VERSION @CUDA_VERSION@)\nif(CUDA_VERSION VERSION_LESS \"3.0\")\n cmake_policy(PUSH)\n # CMake policy 0007 NEW states that empty list elements are not\n # ignored. I'm just setting it to avoid the warning that's printed.\n cmake_policy(SET CMP0007 NEW)\n # Note that this will remove all occurances of -G.\n list(REMOVE_ITEM depends_CUDA_NVCC_FLAGS \"-G\")\n cmake_policy(POP)\nendif()\n\n# nvcc doesn't define __CUDACC__ for some reason when generating dependency files. This\n# can cause incorrect dependencies when #including files based on this macro which is\n# defined in the generating passes of nvcc invokation. We will go ahead and manually\n# define this for now until a future version fixes this bug.\nset(CUDACC_DEFINE -D__CUDACC__)\n\n# Generate the dependency file\ncuda_execute_process(\n \"Generating dependency file: ${NVCC_generated_dependency_file}\"\n COMMAND \"${CUDA_NVCC_EXECUTABLE}\"\n -M\n ${CUDACC_DEFINE}\n \"${source_file}\"\n -o \"${NVCC_generated_dependency_file}\"\n ${CCBIN}\n ${nvcc_flags}\n ${nvcc_host_compiler_flags}\n ${depends_CUDA_NVCC_FLAGS}\n -DNVCC\n ${CUDA_NVCC_INCLUDE_ARGS}\n )\n\nif(CUDA_result)\n message(FATAL_ERROR \"Error generating ${generated_file}\")\nendif()\n\n# Generate the cmake readable dependency file to a temp file. Don't put the\n# quotes just around the filenames for the input_file and output_file variables.\n# CMake will pass the quotes through and not be able to find the file.\ncuda_execute_process(\n \"Generating temporary cmake readable file: ${cmake_dependency_file}.tmp\"\n COMMAND \"${CMAKE_COMMAND}\"\n -D \"input_file:FILEPATH=${NVCC_generated_dependency_file}\"\n -D \"output_file:FILEPATH=${cmake_dependency_file}.tmp\"\n -P \"${CUDA_make2cmake}\"\n )\n\nif(CUDA_result)\n message(FATAL_ERROR \"Error generating ${generated_file}\")\nendif()\n\n# Copy the file if it is different\ncuda_execute_process(\n \"Copy if different ${cmake_dependency_file}.tmp to ${cmake_dependency_file}\"\n COMMAND \"${CMAKE_COMMAND}\" -E copy_if_different \"${cmake_dependency_file}.tmp\" \"${cmake_dependency_file}\"\n )\n\nif(CUDA_result)\n message(FATAL_ERROR \"Error generating ${generated_file}\")\nendif()\n\n# Delete the temporary file\ncuda_execute_process(\n \"Removing ${cmake_dependency_file}.tmp and ${NVCC_generated_dependency_file}\"\n COMMAND \"${CMAKE_COMMAND}\" -E remove \"${cmake_dependency_file}.tmp\" \"${NVCC_generated_dependency_file}\"\n )\n\nif(CUDA_result)\n message(FATAL_ERROR \"Error generating ${generated_file}\")\nendif()\n\n# Generate the code\ncuda_execute_process(\n \"Generating ${generated_file}\"\n COMMAND \"${CUDA_NVCC_EXECUTABLE}\"\n \"${source_file}\"\n ${format_flag} -o \"${generated_file}\"\n ${CCBIN}\n ${nvcc_flags}\n ${nvcc_host_compiler_flags}\n ${CUDA_NVCC_FLAGS}\n -DNVCC\n ${CUDA_NVCC_INCLUDE_ARGS}\n )\n\nif(CUDA_result)\n # Since nvcc can sometimes leave half done files make sure that we delete the output file.\n cuda_execute_process(\n \"Removing ${generated_file}\"\n COMMAND \"${CMAKE_COMMAND}\" -E remove \"${generated_file}\"\n )\n message(FATAL_ERROR \"Error generating file ${generated_file}\")\nelse()\n if(verbose)\n message(\"Generated ${generated_file} successfully.\")\n endif()\nendif()\n\n# Cubin resource report commands.\nif( build_cubin )\n # Run with -cubin to produce resource usage report.\n cuda_execute_process(\n \"Generating ${generated_cubin_file}\"\n COMMAND \"${CUDA_NVCC_EXECUTABLE}\"\n \"${source_file}\"\n ${CUDA_NVCC_FLAGS}\n ${nvcc_flags}\n ${CCBIN}\n ${nvcc_host_compiler_flags}\n -DNVCC\n -cubin\n -o \"${generated_cubin_file}\"\n ${CUDA_NVCC_INCLUDE_ARGS}\n )\n\n # Execute the parser script.\n cuda_execute_process(\n \"Executing the parser script\"\n COMMAND \"${CMAKE_COMMAND}\"\n -D \"input_file:STRING=${generated_cubin_file}\"\n -P \"${CUDA_parse_cubin}\"\n )\n\nendif( build_cubin )\n" }, { "path": "cmake/FindCUDA.cmake", "content": "# - Tools for building CUDA C files: libraries and build dependencies.\n# This script locates the NVIDIA CUDA C tools. It should work on linux, windows,\n# and mac and should be reasonably up to date with CUDA C releases.\n#\n# This script makes use of the standard find_package arguments of ,\n# REQUIRED and QUIET. CUDA_FOUND will report if an acceptable version of CUDA\n# was found.\n#\n# The script will prompt the user to specify CUDA_TOOLKIT_ROOT_DIR if the prefix\n# cannot be determined by the location of nvcc in the system path and REQUIRED\n# is specified to find_package(). To use a different installed version of the\n# toolkit set the environment variable CUDA_BIN_PATH before running cmake\n# (e.g. CUDA_BIN_PATH=/usr/local/cuda1.0 instead of the default /usr/local/cuda)\n# or set CUDA_TOOLKIT_ROOT_DIR after configuring. If you change the value of\n# CUDA_TOOLKIT_ROOT_DIR, various components that depend on the path will be\n# relocated.\n#\n# It might be necessary to set CUDA_TOOLKIT_ROOT_DIR manually on certain\n# platforms, or to use a cuda runtime not installed in the default location. In\n# newer versions of the toolkit the cuda library is included with the graphics\n# driver- be sure that the driver version matches what is needed by the cuda\n# runtime version.\n#\n# The following variables affect the behavior of the macros in the script (in\n# alphebetical order). Note that any of these flags can be changed multiple\n# times in the same directory before calling CUDA_ADD_EXECUTABLE,\n# CUDA_ADD_LIBRARY, CUDA_COMPILE, CUDA_COMPILE_PTX or CUDA_WRAP_SRCS.\n#\n# CUDA_64_BIT_DEVICE_CODE (Default matches host bit size)\n# -- Set to ON to compile for 64 bit device code, OFF for 32 bit device code.\n# Note that making this different from the host code when generating object\n# or C files from CUDA code just won't work, because size_t gets defined by\n# nvcc in the generated source. If you compile to PTX and then load the\n# file yourself, you can mix bit sizes between device and host.\n#\n# CUDA_ATTACH_VS_BUILD_RULE_TO_CUDA_FILE (Default ON)\n# -- Set to ON if you want the custom build rule to be attached to the source\n# file in Visual Studio. Turn OFF if you add the same cuda file to multiple\n# targets.\n#\n# This allows the user to build the target from the CUDA file; however, bad\n# things can happen if the CUDA source file is added to multiple targets.\n# When performing parallel builds it is possible for the custom build\n# command to be run more than once and in parallel causing cryptic build\n# errors. VS runs the rules for every source file in the target, and a\n# source can have only one rule no matter how many projects it is added to.\n# When the rule is run from multiple targets race conditions can occur on\n# the generated file. Eventually everything will get built, but if the user\n# is unaware of this behavior, there may be confusion. It would be nice if\n# this script could detect the reuse of source files across multiple targets\n# and turn the option off for the user, but no good solution could be found.\n#\n# CUDA_BUILD_CUBIN (Default OFF)\n# -- Set to ON to enable and extra compilation pass with the -cubin option in\n# Device mode. The output is parsed and register, shared memory usage is\n# printed during build.\n#\n# CUDA_BUILD_EMULATION (Default OFF for device mode)\n# -- Set to ON for Emulation mode. -D_DEVICEEMU is defined for CUDA C files\n# when CUDA_BUILD_EMULATION is TRUE.\n#\n# CUDA_GENERATED_OUTPUT_DIR (Default CMAKE_CURRENT_BINARY_DIR)\n# -- Set to the path you wish to have the generated files placed. If it is\n# blank output files will be placed in CMAKE_CURRENT_BINARY_DIR.\n# Intermediate files will always be placed in\n# CMAKE_CURRENT_BINARY_DIR/CMakeFiles.\n#\n# CUDA_HOST_COMPILATION_CPP (Default ON)\n# -- Set to OFF for C compilation of host code.\n#\n# CUDA_NVCC_FLAGS\n# CUDA_NVCC_FLAGS_\n# -- Additional NVCC command line arguments. NOTE: multiple arguments must be\n# semi-colon delimited (e.g. --compiler-options;-Wall)\n#\n# CUDA_PROPAGATE_HOST_FLAGS (Default ON)\n# -- Set to ON to propagate CMAKE_{C,CXX}_FLAGS and their configuration\n# dependent counterparts (e.g. CMAKE_C_FLAGS_DEBUG) automatically to the\n# host compiler through nvcc's -Xcompiler flag. This helps make the\n# generated host code match the rest of the system better. Sometimes\n# certain flags give nvcc problems, and this will help you turn the flag\n# propagation off. This does not affect the flags supplied directly to nvcc\n# via CUDA_NVCC_FLAGS or through the OPTION flags specified through\n# CUDA_ADD_LIBRARY, CUDA_ADD_EXECUTABLE, or CUDA_WRAP_SRCS. Flags used for\n# shared library compilation are not affected by this flag.\n#\n# CUDA_VERBOSE_BUILD (Default OFF)\n# -- Set to ON to see all the commands used when building the CUDA file. When\n# using a Makefile generator the value defaults to VERBOSE (run make\n# VERBOSE=1 to see output), although setting CUDA_VERBOSE_BUILD to ON will\n# always print the output.\n#\n# The script creates the following macros (in alphebetical order):\n#\n# CUDA_ADD_CUFFT_TO_TARGET( cuda_target )\n# -- Adds the cufft library to the target (can be any target). Handles whether\n# you are in emulation mode or not.\n#\n# CUDA_ADD_CUBLAS_TO_TARGET( cuda_target )\n# -- Adds the cublas library to the target (can be any target). Handles\n# whether you are in emulation mode or not.\n#\n# CUDA_ADD_EXECUTABLE( cuda_target file0 file1 ...\n# [WIN32] [MACOSX_BUNDLE] [EXCLUDE_FROM_ALL] [OPTIONS ...] )\n# -- Creates an executable \"cuda_target\" which is made up of the files\n# specified. All of the non CUDA C files are compiled using the standard\n# build rules specified by CMAKE and the cuda files are compiled to object\n# files using nvcc and the host compiler. In addition CUDA_INCLUDE_DIRS is\n# added automatically to include_directories(). Some standard CMake target\n# calls can be used on the target after calling this macro\n# (e.g. set_target_properties and target_link_libraries), but setting\n# properties that adjust compilation flags will not affect code compiled by\n# nvcc. Such flags should be modified before calling CUDA_ADD_EXECUTABLE,\n# CUDA_ADD_LIBRARY or CUDA_WRAP_SRCS.\n#\n# CUDA_ADD_LIBRARY( cuda_target file0 file1 ...\n# [STATIC | SHARED | MODULE] [EXCLUDE_FROM_ALL] [OPTIONS ...] )\n# -- Same as CUDA_ADD_EXECUTABLE except that a library is created.\n#\n# CUDA_BUILD_CLEAN_TARGET()\n# -- Creates a convience target that deletes all the dependency files\n# generated. You should make clean after running this target to ensure the\n# dependency files get regenerated.\n#\n# CUDA_COMPILE( generated_files file0 file1 ... [STATIC | SHARED | MODULE]\n# [OPTIONS ...] )\n# -- Returns a list of generated files from the input source files to be used\n# with ADD_LIBRARY or ADD_EXECUTABLE.\n#\n# CUDA_COMPILE_PTX( generated_files file0 file1 ... [OPTIONS ...] )\n# -- Returns a list of PTX files generated from the input source files.\n#\n# CUDA_INCLUDE_DIRECTORIES( path0 path1 ... )\n# -- Sets the directories that should be passed to nvcc\n# (e.g. nvcc -Ipath0 -Ipath1 ... ). These paths usually contain other .cu\n# files.\n#\n# CUDA_WRAP_SRCS ( cuda_target format generated_files file0 file1 ...\n# [STATIC | SHARED | MODULE] [OPTIONS ...] )\n# -- This is where all the magic happens. CUDA_ADD_EXECUTABLE,\n# CUDA_ADD_LIBRARY, CUDA_COMPILE, and CUDA_COMPILE_PTX all call this\n# function under the hood.\n#\n# Given the list of files (file0 file1 ... fileN) this macro generates\n# custom commands that generate either PTX or linkable objects (use \"PTX\" or\n# \"OBJ\" for the format argument to switch). Files that don't end with .cu\n# or have the HEADER_FILE_ONLY property are ignored.\n#\n# The arguments passed in after OPTIONS are extra command line options to\n# give to nvcc. You can also specify per configuration options by\n# specifying the name of the configuration followed by the options. General\n# options must preceed configuration specific options. Not all\n# configurations need to be specified, only the ones provided will be used.\n#\n# OPTIONS -DFLAG=2 \"-DFLAG_OTHER=space in flag\"\n# DEBUG -g\n# RELEASE --use_fast_math\n# RELWITHDEBINFO --use_fast_math;-g\n# MINSIZEREL --use_fast_math\n#\n# For certain configurations (namely VS generating object files with\n# CUDA_ATTACH_VS_BUILD_RULE_TO_CUDA_FILE set to ON), no generated file will\n# be produced for the given cuda file. This is because when you add the\n# cuda file to Visual Studio it knows that this file produces an object file\n# and will link in the resulting object file automatically.\n#\n# This script will also generate a separate cmake script that is used at\n# build time to invoke nvcc. This is for serveral reasons.\n#\n# 1. nvcc can return negative numbers as return values which confuses\n# Visual Studio into thinking that the command succeeded. The script now\n# checks the error codes and produces errors when there was a problem.\n#\n# 2. nvcc has been known to not delete incomplete results when it\n# encounters problems. This confuses build systems into thinking the\n# target was generated when in fact an unusable file exists. The script\n# now deletes the output files if there was an error.\n#\n# 3. By putting all the options that affect the build into a file and then\n# make the build rule dependent on the file, the output files will be\n# regenerated when the options change.\n#\n# This script also looks at optional arguments STATIC, SHARED, or MODULE to\n# determine when to target the object compilation for a shared library.\n# BUILD_SHARED_LIBS is ignored in CUDA_WRAP_SRCS, but it is respected in\n# CUDA_ADD_LIBRARY. On some systems special flags are added for building\n# objects intended for shared libraries. A preprocessor macro,\n# _EXPORTS is defined when a shared library compilation is\n# detected.\n#\n# Flags passed into add_definitions with -D or /D are passed along to nvcc.\n#\n# The script defines the following variables:\n#\n# CUDA_VERSION_MAJOR -- The major version of cuda as reported by nvcc.\n# CUDA_VERSION_MINOR -- The minor version.\n# CUDA_VERSION\n# CUDA_VERSION_STRING -- CUDA_VERSION_MAJOR.CUDA_VERSION_MINOR\n#\n# CUDA_TOOLKIT_ROOT_DIR -- Path to the CUDA Toolkit (defined if not set).\n# CUDA_SDK_ROOT_DIR -- Path to the CUDA SDK. Use this to find files in the\n# SDK. This script will not directly support finding\n# specific libraries or headers, as that isn't\n# supported by NVIDIA. If you want to change\n# libraries when the path changes see the\n# FindCUDA.cmake script for an example of how to clear\n# these variables. There are also examples of how to\n# use the CUDA_SDK_ROOT_DIR to locate headers or\n# libraries, if you so choose (at your own risk).\n# CUDA_INCLUDE_DIRS -- Include directory for cuda headers. Added automatically\n# for CUDA_ADD_EXECUTABLE and CUDA_ADD_LIBRARY.\n# CUDA_LIBRARIES -- Cuda RT library.\n# CUDA_CUFFT_LIBRARIES -- Device or emulation library for the Cuda FFT\n# implementation (alternative to:\n# CUDA_ADD_CUFFT_TO_TARGET macro)\n# CUDA_CUBLAS_LIBRARIES -- Device or emulation library for the Cuda BLAS\n# implementation (alterative to:\n# CUDA_ADD_CUBLAS_TO_TARGET macro).\n#\n#\n# James Bigler, NVIDIA Corp (nvidia.com - jbigler)\n# Abe Stephens, SCI Institute -- http://www.sci.utah.edu/~abe/FindCuda.html\n#\n# Copyright (c) 2008 - 2009 NVIDIA Corporation. All rights reserved.\n#\n# Copyright (c) 2007-2009\n# Scientific Computing and Imaging Institute, University of Utah\n#\n# This code is licensed under the MIT License. See the FindCUDA.cmake script\n# for the text of the license.\n\n# The MIT License\n#\n# License for the specific language governing rights and limitations under\n# Permission is hereby granted, free of charge, to any person obtaining a\n# copy of this software and associated documentation files (the \"Software\"),\n# to deal in the Software without restriction, including without limitation\n# the rights to use, copy, modify, merge, publish, distribute, sublicense,\n# and/or sell copies of the Software, and to permit persons to whom the\n# Software is furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included\n# in all copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS\n# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL\n# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\n# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER\n# DEALINGS IN THE SOFTWARE.\n#\n###############################################################################\n\n# FindCUDA.cmake\n\n# We need to have at least this version to support the VERSION_LESS argument to 'if' (2.6.2) and unset (2.6.3)\ncmake_policy(PUSH)\ncmake_minimum_required(VERSION 2.6.2)\ncmake_policy(POP)\n\n# This macro helps us find the location of helper files we will need the full path to\nmacro(CUDA_FIND_HELPER_FILE _name _extension)\n set(_full_name \"${_name}.${_extension}\")\n # CMAKE_CURRENT_LIST_FILE contains the full path to the file currently being\n # processed. Using this variable, we can pull out the current path, and\n # provide a way to get access to the other files we need local to here.\n get_filename_component(CMAKE_CURRENT_LIST_DIR \"${CMAKE_CURRENT_LIST_FILE}\" PATH)\n find_file(CUDA_${_name} ${_full_name} PATHS ${CMAKE_CURRENT_LIST_DIR}/FindCUDA NO_DEFAULT_PATH)\n if(NOT CUDA_${_name})\n set(error_message \"${_full_name} not found in CMAKE_MODULE_PATH\")\n if(CUDA_FIND_REQUIRED)\n message(FATAL_ERROR \"${error_message}\")\n else(CUDA_FIND_REQUIRED)\n if(NOT CUDA_FIND_QUIETLY)\n message(STATUS \"${error_message}\")\n endif(NOT CUDA_FIND_QUIETLY)\n endif(CUDA_FIND_REQUIRED)\n endif(NOT CUDA_${_name})\n # Set this variable as internal, so the user isn't bugged with it.\n set(CUDA_${_name} ${CUDA_${_name}} CACHE INTERNAL \"Location of ${_full_name}\" FORCE)\nendmacro(CUDA_FIND_HELPER_FILE)\n\n#####################################################################\n## CUDA_INCLUDE_NVCC_DEPENDENCIES\n##\n\n# So we want to try and include the dependency file if it exists. If\n# it doesn't exist then we need to create an empty one, so we can\n# include it.\n\n# If it does exist, then we need to check to see if all the files it\n# depends on exist. If they don't then we should clear the dependency\n# file and regenerate it later. This covers the case where a header\n# file has disappeared or moved.\n\nmacro(CUDA_INCLUDE_NVCC_DEPENDENCIES dependency_file)\n set(CUDA_NVCC_DEPEND)\n set(CUDA_NVCC_DEPEND_REGENERATE FALSE)\n\n\n # Include the dependency file. Create it first if it doesn't exist . The\n # INCLUDE puts a dependency that will force CMake to rerun and bring in the\n # new info when it changes. DO NOT REMOVE THIS (as I did and spent a few\n # hours figuring out why it didn't work.\n if(NOT EXISTS ${dependency_file})\n file(WRITE ${dependency_file} \"#FindCUDA.cmake generated file. Do not edit.\\n\")\n endif()\n # Always include this file to force CMake to run again next\n # invocation and rebuild the dependencies.\n #message(\"including dependency_file = ${dependency_file}\")\n include(${dependency_file})\n\n # Now we need to verify the existence of all the included files\n # here. If they aren't there we need to just blank this variable and\n # make the file regenerate again.\n# if(DEFINED CUDA_NVCC_DEPEND)\n# message(\"CUDA_NVCC_DEPEND set\")\n# else()\n# message(\"CUDA_NVCC_DEPEND NOT set\")\n# endif()\n if(CUDA_NVCC_DEPEND)\n #message(\"CUDA_NVCC_DEPEND true\")\n foreach(f ${CUDA_NVCC_DEPEND})\n #message(\"searching for ${f}\")\n if(NOT EXISTS ${f})\n #message(\"file ${f} not found\")\n set(CUDA_NVCC_DEPEND_REGENERATE TRUE)\n endif()\n endforeach(f)\n else(CUDA_NVCC_DEPEND)\n #message(\"CUDA_NVCC_DEPEND false\")\n # No dependencies, so regenerate the file.\n set(CUDA_NVCC_DEPEND_REGENERATE TRUE)\n endif(CUDA_NVCC_DEPEND)\n\n #message(\"CUDA_NVCC_DEPEND_REGENERATE = ${CUDA_NVCC_DEPEND_REGENERATE}\")\n # No incoming dependencies, so we need to generate them. Make the\n # output depend on the dependency file itself, which should cause the\n # rule to re-run.\n if(CUDA_NVCC_DEPEND_REGENERATE)\n file(WRITE ${dependency_file} \"#FindCUDA.cmake generated file. Do not edit.\\n\")\n endif(CUDA_NVCC_DEPEND_REGENERATE)\n\nendmacro(CUDA_INCLUDE_NVCC_DEPENDENCIES)\n\n###############################################################################\n###############################################################################\n# Setup variables' defaults\n###############################################################################\n###############################################################################\n\n# Allow the user to specify if the device code is supposed to be 32 or 64 bit.\nif(CMAKE_SIZEOF_VOID_P EQUAL 8)\n set(CUDA_64_BIT_DEVICE_CODE_DEFAULT ON)\nelse()\n set(CUDA_64_BIT_DEVICE_CODE_DEFAULT OFF)\nendif()\noption(CUDA_64_BIT_DEVICE_CODE \"Compile device code in 64 bit mode\" ${CUDA_64_BIT_DEVICE_CODE_DEFAULT})\n\n# Attach the build rule to the source file in VS. This option\noption(CUDA_ATTACH_VS_BUILD_RULE_TO_CUDA_FILE \"Attach the build rule to the CUDA source file. Enable only when the CUDA source file is added to at most one target.\" ON)\n\n# Prints out extra information about the cuda file during compilation\noption(CUDA_BUILD_CUBIN \"Generate and parse .cubin files in Device mode.\" OFF)\n\n# Set whether we are using emulation or device mode.\noption(CUDA_BUILD_EMULATION \"Build in Emulation mode\" OFF)\n\n# Where to put the generated output.\nset(CUDA_GENERATED_OUTPUT_DIR \"\" CACHE PATH \"Directory to put all the output files. If blank it will default to the CMAKE_CURRENT_BINARY_DIR\")\n\n# Parse HOST_COMPILATION mode.\noption(CUDA_HOST_COMPILATION_CPP \"Generated file extension\" ON)\n\n# Extra user settable flags\nset(CUDA_NVCC_FLAGS \"\" CACHE STRING \"Semi-colon delimit multiple arguments.\")\n\n# Propagate the host flags to the host compiler via -Xcompiler\noption(CUDA_PROPAGATE_HOST_FLAGS \"Propage C/CXX_FLAGS and friends to the host compiler via -Xcompile\" ON)\n\n# Specifies whether the commands used when compiling the .cu file will be printed out.\noption(CUDA_VERBOSE_BUILD \"Print out the commands run while compiling the CUDA source file. With the Makefile generator this defaults to VERBOSE variable specified on the command line, but can be forced on with this option.\" OFF)\n\nmark_as_advanced(\n CUDA_64_BIT_DEVICE_CODE\n CUDA_ATTACH_VS_BUILD_RULE_TO_CUDA_FILE\n CUDA_GENERATED_OUTPUT_DIR\n CUDA_HOST_COMPILATION_CPP\n CUDA_NVCC_FLAGS\n CUDA_PROPAGATE_HOST_FLAGS\n )\n\n# Makefile and similar generators don't define CMAKE_CONFIGURATION_TYPES, so we\n# need to add another entry for the CMAKE_BUILD_TYPE. We also need to add the\n# standerd set of 4 build types (Debug, MinSizeRel, Release, and RelWithDebInfo)\n# for completeness. We need run this loop in order to accomodate the addition\n# of extra configuration types. Duplicate entries will be removed by\n# REMOVE_DUPLICATES.\nset(CUDA_configuration_types ${CMAKE_CONFIGURATION_TYPES} ${CMAKE_BUILD_TYPE} Debug MinSizeRel Release RelWithDebInfo)\nlist(REMOVE_DUPLICATES CUDA_configuration_types)\nforeach(config ${CUDA_configuration_types})\n string(TOUPPER ${config} config_upper)\n set(CUDA_NVCC_FLAGS_${config_upper} \"\" CACHE STRING \"Semi-colon delimit multiple arguments.\")\n mark_as_advanced(CUDA_NVCC_FLAGS_${config_upper})\nendforeach()\n\n###############################################################################\n###############################################################################\n# Locate CUDA, Set Build Type, etc.\n###############################################################################\n###############################################################################\n\n# Check to see if the CUDA_TOOLKIT_ROOT_DIR and CUDA_SDK_ROOT_DIR have changed,\n# if they have then clear the cache variables, so that will be detected again.\nif(NOT \"${CUDA_TOOLKIT_ROOT_DIR}\" STREQUAL \"${CUDA_TOOLKIT_ROOT_DIR_INTERNAL}\")\n unset(CUDA_NVCC_EXECUTABLE CACHE)\n unset(CUDA_VERSION CACHE)\n unset(CUDA_TOOLKIT_INCLUDE CACHE)\n unset(CUDA_CUDART_LIBRARY CACHE)\n unset(CUDA_CUDA_LIBRARY CACHE)\n unset(CUDA_cublas_LIBRARY CACHE)\n unset(CUDA_cublasemu_LIBRARY CACHE)\n unset(CUDA_cufft_LIBRARY CACHE)\n unset(CUDA_cufftemu_LIBRARY CACHE)\nendif()\n\nif(NOT \"${CUDA_SDK_ROOT_DIR}\" STREQUAL \"${CUDA_SDK_ROOT_DIR_INTERNAL}\")\n # No specific variables to catch. Use this kind of code before calling\n # find_package(CUDA) to clean up any variables that may depend on this path.\n\n # unset(MY_SPECIAL_CUDA_SDK_INCLUDE_DIR CACHE)\n # unset(MY_SPECIAL_CUDA_SDK_LIBRARY CACHE)\nendif()\n\n# Search for the cuda distribution.\nif(NOT CUDA_TOOLKIT_ROOT_DIR)\n\n # Search in the CUDA_BIN_PATH first.\n find_path(CUDA_TOOLKIT_ROOT_DIR\n NAMES nvcc nvcc.exe\n PATHS ENV CUDA_BIN_PATH\n DOC \"Toolkit location.\"\n NO_DEFAULT_PATH\n )\n # Now search default paths\n find_path(CUDA_TOOLKIT_ROOT_DIR\n NAMES nvcc nvcc.exe\n PATHS /usr/local/bin\n /usr/local/cuda/bin\n DOC \"Toolkit location.\"\n )\n\n if (CUDA_TOOLKIT_ROOT_DIR)\n string(REGEX REPLACE \"[/\\\\\\\\]?bin[64]*[/\\\\\\\\]?$\" \"\" CUDA_TOOLKIT_ROOT_DIR ${CUDA_TOOLKIT_ROOT_DIR})\n # We need to force this back into the cache.\n set(CUDA_TOOLKIT_ROOT_DIR ${CUDA_TOOLKIT_ROOT_DIR} CACHE PATH \"Toolkit location.\" FORCE)\n endif(CUDA_TOOLKIT_ROOT_DIR)\n if (NOT EXISTS ${CUDA_TOOLKIT_ROOT_DIR})\n if(CUDA_FIND_REQUIRED)\n message(FATAL_ERROR \"Specify CUDA_TOOLKIT_ROOT_DIR\")\n elseif(NOT CUDA_FIND_QUIETLY)\n message(STATUS \"CUDA_TOOLKIT_ROOT_DIR not found or specified\")\n endif()\n endif (NOT EXISTS ${CUDA_TOOLKIT_ROOT_DIR})\nendif (NOT CUDA_TOOLKIT_ROOT_DIR)\n\n# CUDA_NVCC_EXECUTABLE\nfind_program(CUDA_NVCC_EXECUTABLE\n NAMES nvcc\n PATHS \"${CUDA_TOOLKIT_ROOT_DIR}/bin\"\n \"${CUDA_TOOLKIT_ROOT_DIR}/bin64\"\n ENV CUDA_BIN_PATH\n NO_DEFAULT_PATH\n )\n# Search default search paths, after we search our own set of paths.\nfind_program(CUDA_NVCC_EXECUTABLE nvcc)\nmark_as_advanced(CUDA_NVCC_EXECUTABLE)\n\nif(CUDA_NVCC_EXECUTABLE AND NOT CUDA_VERSION)\n # Compute the version.\n execute_process (COMMAND ${CUDA_NVCC_EXECUTABLE} \"--version\" OUTPUT_VARIABLE NVCC_OUT)\n string(REGEX REPLACE \".*release ([0-9]+)\\\\.([0-9]+).*\" \"\\\\1\" CUDA_VERSION_MAJOR ${NVCC_OUT})\n string(REGEX REPLACE \".*release ([0-9]+)\\\\.([0-9]+).*\" \"\\\\2\" CUDA_VERSION_MINOR ${NVCC_OUT})\n set(CUDA_VERSION \"${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}\" CACHE STRING \"Version of CUDA as computed from nvcc.\")\n mark_as_advanced(CUDA_VERSION)\nendif()\n\n# Always set this convenience variable\nset(CUDA_VERSION_STRING \"${CUDA_VERSION}\")\n\n# Here we need to determine if the version we found is acceptable. We will\n# assume that is unless CUDA_FIND_VERSION_EXACT or CUDA_FIND_VERSION is\n# specified. The presence of either of these options checks the version\n# string and signals if the version is acceptable or not.\nset(_cuda_version_acceptable TRUE)\n#\nif(CUDA_FIND_VERSION_EXACT AND NOT CUDA_VERSION VERSION_EQUAL CUDA_FIND_VERSION)\n set(_cuda_version_acceptable FALSE)\nendif()\n#\nif(CUDA_FIND_VERSION AND CUDA_VERSION VERSION_LESS CUDA_FIND_VERSION)\n set(_cuda_version_acceptable FALSE)\nendif()\n#\nif(NOT _cuda_version_acceptable)\n set(_cuda_error_message \"Requested CUDA version ${CUDA_FIND_VERSION}, but found unacceptable version ${CUDA_VERSION}\")\n if(CUDA_FIND_REQUIRED)\n message(\"${_cuda_error_message}\")\n elseif(NOT CUDA_FIND_QUIETLY)\n message(\"${_cuda_error_message}\")\n endif()\nendif()\n\n# CUDA_TOOLKIT_INCLUDE\nfind_path(CUDA_TOOLKIT_INCLUDE\n device_functions.h # Header included in toolkit\n PATHS \"${CUDA_TOOLKIT_ROOT_DIR}/include\"\n ENV CUDA_INC_PATH\n NO_DEFAULT_PATH\n )\n# Search default search paths, after we search our own set of paths.\nfind_path(CUDA_TOOLKIT_INCLUDE device_functions.h)\nmark_as_advanced(CUDA_TOOLKIT_INCLUDE)\n\n# Set the user list of include dir to nothing to initialize it.\nset (CUDA_NVCC_INCLUDE_ARGS_USER \"\")\nset (CUDA_INCLUDE_DIRS ${CUDA_TOOLKIT_INCLUDE})\n\nmacro(FIND_LIBRARY_LOCAL_FIRST _var _names _doc)\n if(CMAKE_SIZEOF_VOID_P EQUAL 8)\n set(_cuda_64bit_lib_dir \"${CUDA_TOOLKIT_ROOT_DIR}/lib64\")\n endif()\n find_library(${_var}\n NAMES ${_names}\n PATHS ${_cuda_64bit_lib_dir}\n \"${CUDA_TOOLKIT_ROOT_DIR}/lib\"\n ENV CUDA_LIB_PATH\n DOC ${_doc}\n NO_DEFAULT_PATH\n )\n # Search default search paths, after we search our own set of paths.\n find_library(${_var} NAMES ${_names} DOC ${_doc})\nendmacro()\n\n# CUDA_LIBRARIES\nfind_library_local_first(CUDA_CUDART_LIBRARY cudart \"\\\"cudart\\\" library\")\nset(CUDA_LIBRARIES ${CUDA_CUDART_LIBRARY})\nif(APPLE)\n # We need to add the path to cudart to the linker using rpath, since the\n # library name for the cuda libraries is prepended with @rpath.\n get_filename_component(_cuda_path_to_cudart \"${CUDA_CUDART_LIBRARY}\" PATH)\n if(_cuda_path_to_cudart)\n list(APPEND CUDA_LIBRARIES -Wl,-rpath \"-Wl,${_cuda_path_to_cudart}\")\n endif()\nendif()\n\n# 1.1 toolkit on linux doesn't appear to have a separate library on\n# some platforms.\nfind_library_local_first(CUDA_CUDA_LIBRARY cuda \"\\\"cuda\\\" library (older versions only).\")\n\n# Add cuda library to the link line only if it is found.\nif (CUDA_CUDA_LIBRARY)\n set(CUDA_LIBRARIES ${CUDA_LIBRARIES} ${CUDA_CUDA_LIBRARY})\nendif(CUDA_CUDA_LIBRARY)\n\nmark_as_advanced(\n CUDA_CUDA_LIBRARY\n CUDA_CUDART_LIBRARY\n )\n\n#######################\n# Look for some of the toolkit helper libraries\nmacro(FIND_CUDA_HELPER_LIBS _name)\n find_library_local_first(CUDA_${_name}_LIBRARY ${_name} \"\\\"${_name}\\\" library\")\n mark_as_advanced(CUDA_${_name}_LIBRARY)\nendmacro(FIND_CUDA_HELPER_LIBS)\n\n# Search for cufft and cublas libraries.\nfind_cuda_helper_libs(cufftemu)\nfind_cuda_helper_libs(cublasemu)\nfind_cuda_helper_libs(cufft)\nfind_cuda_helper_libs(cublas)\n\nif (CUDA_BUILD_EMULATION)\n set(CUDA_CUFFT_LIBRARIES ${CUDA_cufftemu_LIBRARY})\n set(CUDA_CUBLAS_LIBRARIES ${CUDA_cublasemu_LIBRARY})\nelse()\n set(CUDA_CUFFT_LIBRARIES ${CUDA_cufft_LIBRARY})\n set(CUDA_CUBLAS_LIBRARIES ${CUDA_cublas_LIBRARY})\nendif()\n\n########################\n# Look for the SDK stuff\nfind_path(CUDA_SDK_ROOT_DIR common/inc/cutil.h\n \"$ENV{NVSDKCUDA_ROOT}\"\n \"[HKEY_LOCAL_MACHINE\\\\SOFTWARE\\\\NVIDIA Corporation\\\\Installed Products\\\\NVIDIA SDK 10\\\\Compute;InstallDir]\"\n \"/Developer/GPU\\ Computing/C\"\n )\n\n# Keep the CUDA_SDK_ROOT_DIR first in order to be able to override the\n# environment variables.\nset(CUDA_SDK_SEARCH_PATH\n \"${CUDA_SDK_ROOT_DIR}\"\n \"${CUDA_TOOLKIT_ROOT_DIR}/local/NVSDK0.2\"\n \"${CUDA_TOOLKIT_ROOT_DIR}/NVSDK0.2\"\n \"${CUDA_TOOLKIT_ROOT_DIR}/NV_CUDA_SDK\"\n \"$ENV{HOME}/NVIDIA_CUDA_SDK\"\n \"$ENV{HOME}/NVIDIA_CUDA_SDK_MACOSX\"\n \"/Developer/CUDA\"\n )\n\n# Example of how to find an include file from the CUDA_SDK_ROOT_DIR\n\n# find_path(CUDA_CUT_INCLUDE_DIR\n# cutil.h\n# PATHS ${CUDA_SDK_SEARCH_PATH}\n# PATH_SUFFIXES \"common/inc\"\n# DOC \"Location of cutil.h\"\n# NO_DEFAULT_PATH\n# )\n# # Now search system paths\n# find_path(CUDA_CUT_INCLUDE_DIR cutil.h DOC \"Location of cutil.h\")\n\n# mark_as_advanced(CUDA_CUT_INCLUDE_DIR)\n\n\n# Example of how to find a library in the CUDA_SDK_ROOT_DIR\n\n# # cutil library is called cutil64 for 64 bit builds on windows. We don't want\n# # to get these confused, so we are setting the name based on the word size of\n# # the build.\n\n# if(CMAKE_SIZEOF_VOID_P EQUAL 8)\n# set(cuda_cutil_name cutil64)\n# else(CMAKE_SIZEOF_VOID_P EQUAL 8)\n# set(cuda_cutil_name cutil32)\n# endif(CMAKE_SIZEOF_VOID_P EQUAL 8)\n\n# find_library(CUDA_CUT_LIBRARY\n# NAMES cutil ${cuda_cutil_name}\n# PATHS ${CUDA_SDK_SEARCH_PATH}\n# # The new version of the sdk shows up in common/lib, but the old one is in lib\n# PATH_SUFFIXES \"common/lib\" \"lib\"\n# DOC \"Location of cutil library\"\n# NO_DEFAULT_PATH\n# )\n# # Now search system paths\n# find_library(CUDA_CUT_LIBRARY NAMES cutil ${cuda_cutil_name} DOC \"Location of cutil library\")\n# mark_as_advanced(CUDA_CUT_LIBRARY)\n# set(CUDA_CUT_LIBRARIES ${CUDA_CUT_LIBRARY})\n\n\n\n#############################\n# Check for required components\nset(CUDA_FOUND TRUE)\n\nset(CUDA_TOOLKIT_ROOT_DIR_INTERNAL \"${CUDA_TOOLKIT_ROOT_DIR}\" CACHE INTERNAL\n \"This is the value of the last time CUDA_TOOLKIT_ROOT_DIR was set successfully.\" FORCE)\nset(CUDA_SDK_ROOT_DIR_INTERNAL \"${CUDA_SDK_ROOT_DIR}\" CACHE INTERNAL\n \"This is the value of the last time CUDA_SDK_ROOT_DIR was set successfully.\" FORCE)\n\ninclude(FindPackageHandleStandardArgs)\nfind_package_handle_standard_args(CUDA DEFAULT_MSG\n CUDA_TOOLKIT_ROOT_DIR\n CUDA_NVCC_EXECUTABLE\n CUDA_INCLUDE_DIRS\n CUDA_CUDART_LIBRARY\n _cuda_version_acceptable\n )\n\n\n\n###############################################################################\n###############################################################################\n# Macros\n###############################################################################\n###############################################################################\n\n###############################################################################\n# Add include directories to pass to the nvcc command.\nmacro(CUDA_INCLUDE_DIRECTORIES)\n foreach(dir ${ARGN})\n list(APPEND CUDA_NVCC_INCLUDE_ARGS_USER \"-I${dir}\")\n endforeach(dir ${ARGN})\nendmacro(CUDA_INCLUDE_DIRECTORIES)\n\n\n##############################################################################\ncuda_find_helper_file(parse_cubin cmake)\ncuda_find_helper_file(make2cmake cmake)\ncuda_find_helper_file(run_nvcc cmake)\n\n##############################################################################\n# Separate the OPTIONS out from the sources\n#\nmacro(CUDA_GET_SOURCES_AND_OPTIONS _sources _cmake_options _options)\n set( ${_sources} )\n set( ${_cmake_options} )\n set( ${_options} )\n set( _found_options FALSE )\n foreach(arg ${ARGN})\n if(arg STREQUAL \"OPTIONS\")\n set( _found_options TRUE )\n elseif(\n arg STREQUAL \"WIN32\" OR\n arg STREQUAL \"MACOSX_BUNDLE\" OR\n arg STREQUAL \"EXCLUDE_FROM_ALL\" OR\n arg STREQUAL \"STATIC\" OR\n arg STREQUAL \"SHARED\" OR\n arg STREQUAL \"MODULE\"\n )\n list(APPEND ${_cmake_options} \"${arg}\")\n else()\n if ( _found_options )\n list(APPEND ${_options} \"${arg}\")\n else()\n # Assume this is a file\n list(APPEND ${_sources} \"${arg}\")\n endif()\n endif()\n endforeach()\nendmacro()\n\n##############################################################################\n# Parse the OPTIONS from ARGN and set the variables prefixed by _option_prefix\n#\nmacro(CUDA_PARSE_NVCC_OPTIONS _option_prefix)\n set( _found_config )\n foreach(arg ${ARGN})\n # Determine if we are dealing with a perconfiguration flag\n foreach(config ${CUDA_configuration_types})\n string(TOUPPER ${config} config_upper)\n if (arg STREQUAL \"${config_upper}\")\n set( _found_config _${arg})\n # Set arg to nothing to keep it from being processed further\n set( arg )\n endif()\n endforeach()\n\n if ( arg )\n list(APPEND ${_option_prefix}${_found_config} \"${arg}\")\n endif()\n endforeach()\nendmacro()\n\n##############################################################################\n# Helper to add the include directory for CUDA only once\nfunction(CUDA_ADD_CUDA_INCLUDE_ONCE)\n get_directory_property(_include_directories INCLUDE_DIRECTORIES)\n set(_add TRUE)\n if(_include_directories)\n foreach(dir ${_include_directories})\n if(\"${dir}\" STREQUAL \"${CUDA_INCLUDE_DIRS}\")\n set(_add FALSE)\n endif()\n endforeach()\n endif()\n if(_add)\n include_directories(${CUDA_INCLUDE_DIRS})\n endif()\nendfunction()\n\nfunction(CUDA_BUILD_SHARED_LIBRARY shared_flag)\n set(cmake_args ${ARGN})\n # If SHARED, MODULE, or STATIC aren't already in the list of arguments, then\n # add SHARED or STATIC based on the value of BUILD_SHARED_LIBS.\n list(FIND cmake_args SHARED _cuda_found_SHARED)\n list(FIND cmake_args MODULE _cuda_found_MODULE)\n list(FIND cmake_args STATIC _cuda_found_STATIC)\n if( _cuda_found_SHARED GREATER -1 OR\n _cuda_found_MODULE GREATER -1 OR\n _cuda_found_STATIC GREATER -1)\n set(_cuda_build_shared_libs)\n else()\n if (BUILD_SHARED_LIBS)\n set(_cuda_build_shared_libs SHARED)\n else()\n set(_cuda_build_shared_libs STATIC)\n endif()\n endif()\n set(${shared_flag} ${_cuda_build_shared_libs} PARENT_SCOPE)\nendfunction()\n\n##############################################################################\n# This helper macro populates the following variables and setups up custom\n# commands and targets to invoke the nvcc compiler to generate C or PTX source\n# dependant upon the format parameter. The compiler is invoked once with -M\n# to generate a dependency file and a second time with -cuda or -ptx to generate\n# a .cpp or .ptx file.\n# INPUT:\n# cuda_target - Target name\n# format - PTX or OBJ\n# FILE1 .. FILEN - The remaining arguments are the sources to be wrapped.\n# OPTIONS - Extra options to NVCC\n# OUTPUT:\n# generated_files - List of generated files\n##############################################################################\n##############################################################################\n\nmacro(CUDA_WRAP_SRCS cuda_target format generated_files)\n\n if( ${format} MATCHES \"PTX\" )\n set( compile_to_ptx ON )\n elseif( ${format} MATCHES \"OBJ\")\n set( compile_to_ptx OFF )\n else()\n message( FATAL_ERROR \"Invalid format flag passed to CUDA_WRAP_SRCS: '${format}'. Use OBJ or PTX.\")\n endif()\n\n # Set up all the command line flags here, so that they can be overriden on a per target basis.\n\n set(nvcc_flags \"\")\n\n # Emulation if the card isn't present.\n if (CUDA_BUILD_EMULATION)\n # Emulation.\n set(nvcc_flags ${nvcc_flags} --device-emulation -D_DEVICEEMU -g)\n else(CUDA_BUILD_EMULATION)\n # Device mode. No flags necessary.\n endif(CUDA_BUILD_EMULATION)\n\n if(CUDA_HOST_COMPILATION_CPP)\n set(CUDA_C_OR_CXX CXX)\n else(CUDA_HOST_COMPILATION_CPP)\n if(CUDA_VERSION VERSION_LESS \"3.0\")\n set(nvcc_flags ${nvcc_flags} --host-compilation C)\n else()\n message(WARNING \"--host-compilation flag is deprecated in CUDA version >= 3.0. Removing --host-compilation C flag\" )\n endif()\n set(CUDA_C_OR_CXX C)\n endif(CUDA_HOST_COMPILATION_CPP)\n\n set(generated_extension ${CMAKE_${CUDA_C_OR_CXX}_OUTPUT_EXTENSION})\n\n if(CUDA_64_BIT_DEVICE_CODE)\n set(nvcc_flags ${nvcc_flags} -m64)\n else()\n set(nvcc_flags ${nvcc_flags} -m32)\n endif()\n\n # This needs to be passed in at this stage, because VS needs to fill out the\n # value of VCInstallDir from within VS.\n if(CMAKE_GENERATOR MATCHES \"Visual Studio\")\n if( CMAKE_SIZEOF_VOID_P EQUAL 8 )\n # Add nvcc flag for 64b Windows\n set(ccbin_flags -D \"\\\"CCBIN:PATH=$(VCInstallDir)bin\\\"\" )\n endif()\n endif()\n\n # Figure out which configure we will use and pass that in as an argument to\n # the script. We need to defer the decision until compilation time, because\n # for VS projects we won't know if we are making a debug or release build\n # until build time.\n if(CMAKE_GENERATOR MATCHES \"Visual Studio\")\n set( CUDA_build_configuration \"$(ConfigurationName)\" )\n else()\n set( CUDA_build_configuration \"${CMAKE_BUILD_TYPE}\")\n endif()\n\n # Initialize our list of includes with the user ones followed by the CUDA system ones.\n set(CUDA_NVCC_INCLUDE_ARGS ${CUDA_NVCC_INCLUDE_ARGS_USER} \"-I${CUDA_INCLUDE_DIRS}\")\n # Get the include directories for this directory and use them for our nvcc command.\n get_directory_property(CUDA_NVCC_INCLUDE_DIRECTORIES INCLUDE_DIRECTORIES)\n if(CUDA_NVCC_INCLUDE_DIRECTORIES)\n foreach(dir ${CUDA_NVCC_INCLUDE_DIRECTORIES})\n list(APPEND CUDA_NVCC_INCLUDE_ARGS \"-I${dir}\")\n endforeach()\n endif()\n\n # Reset these variables\n set(CUDA_WRAP_OPTION_NVCC_FLAGS)\n foreach(config ${CUDA_configuration_types})\n string(TOUPPER ${config} config_upper)\n set(CUDA_WRAP_OPTION_NVCC_FLAGS_${config_upper})\n endforeach()\n\n CUDA_GET_SOURCES_AND_OPTIONS(_cuda_wrap_sources _cuda_wrap_cmake_options _cuda_wrap_options ${ARGN})\n CUDA_PARSE_NVCC_OPTIONS(CUDA_WRAP_OPTION_NVCC_FLAGS ${_cuda_wrap_options})\n\n # Figure out if we are building a shared library. BUILD_SHARED_LIBS is\n # respected in CUDA_ADD_LIBRARY.\n set(_cuda_build_shared_libs FALSE)\n # SHARED, MODULE\n list(FIND _cuda_wrap_cmake_options SHARED _cuda_found_SHARED)\n list(FIND _cuda_wrap_cmake_options MODULE _cuda_found_MODULE)\n if(_cuda_found_SHARED GREATER -1 OR _cuda_found_MODULE GREATER -1)\n set(_cuda_build_shared_libs TRUE)\n endif()\n # STATIC\n list(FIND _cuda_wrap_cmake_options STATIC _cuda_found_STATIC)\n if(_cuda_found_STATIC GREATER -1)\n set(_cuda_build_shared_libs FALSE)\n endif()\n\n # CUDA_HOST_FLAGS\n if(_cuda_build_shared_libs)\n # If we are setting up code for a shared library, then we need to add extra flags for\n # compiling objects for shared libraries.\n set(CUDA_HOST_SHARED_FLAGS ${CMAKE_SHARED_LIBRARY_${CUDA_C_OR_CXX}_FLAGS})\n else()\n set(CUDA_HOST_SHARED_FLAGS)\n endif()\n # Only add the CMAKE_{C,CXX}_FLAGS if we are propagating host flags. We\n # always need to set the SHARED_FLAGS, though.\n if(CUDA_PROPAGATE_HOST_FLAGS)\n set(CUDA_HOST_FLAGS \"set(CMAKE_HOST_FLAGS ${CMAKE_${CUDA_C_OR_CXX}_FLAGS} ${CUDA_HOST_SHARED_FLAGS})\")\n else()\n set(CUDA_HOST_FLAGS \"set(CMAKE_HOST_FLAGS ${CUDA_HOST_SHARED_FLAGS})\")\n endif()\n\n set(CUDA_NVCC_FLAGS_CONFIG \"# Build specific configuration flags\")\n # Loop over all the configuration types to generate appropriate flags for run_nvcc.cmake\n foreach(config ${CUDA_configuration_types})\n string(TOUPPER ${config} config_upper)\n # CMAKE_FLAGS are strings and not lists. By not putting quotes around CMAKE_FLAGS\n # we convert the strings to lists (like we want).\n\n if(CUDA_PROPAGATE_HOST_FLAGS)\n # nvcc chokes on -g3, so replace it with -g\n if(CMAKE_COMPILER_IS_GNUCC)\n string(REPLACE \"-g3\" \"-g\" _cuda_C_FLAGS \"${CMAKE_${CUDA_C_OR_CXX}_FLAGS_${config_upper}}\")\n else()\n set(_cuda_C_FLAGS \"${CMAKE_${CUDA_C_OR_CXX}_FLAGS_${config_upper}}\")\n endif()\n\n set(CUDA_HOST_FLAGS \"${CUDA_HOST_FLAGS}\\nset(CMAKE_HOST_FLAGS_${config_upper} ${_cuda_C_FLAGS})\")\n endif()\n\n # Note that if we ever want CUDA_NVCC_FLAGS_ to be string (instead of a list\n # like it is currently), we can remove the quotes around the\n # ${CUDA_NVCC_FLAGS_${config_upper}} variable like the CMAKE_HOST_FLAGS_ variable.\n set(CUDA_NVCC_FLAGS_CONFIG \"${CUDA_NVCC_FLAGS_CONFIG}\\nset(CUDA_NVCC_FLAGS_${config_upper} \\\"${CUDA_NVCC_FLAGS_${config_upper}};;${CUDA_WRAP_OPTION_NVCC_FLAGS_${config_upper}}\\\")\")\n endforeach()\n\n if(compile_to_ptx)\n # Don't use any of the host compilation flags for PTX targets.\n set(CUDA_HOST_FLAGS)\n set(CUDA_NVCC_FLAGS_CONFIG)\n endif()\n\n # Get the list of definitions from the directory property\n get_directory_property(CUDA_NVCC_DEFINITIONS COMPILE_DEFINITIONS)\n if(CUDA_NVCC_DEFINITIONS)\n foreach(_definition ${CUDA_NVCC_DEFINITIONS})\n list(APPEND nvcc_flags \"-D${_definition}\")\n endforeach()\n endif()\n\n if(_cuda_build_shared_libs)\n list(APPEND nvcc_flags \"-D${cuda_target}_EXPORTS\")\n endif()\n\n # Determine output directory\n if(CUDA_GENERATED_OUTPUT_DIR)\n set(cuda_compile_output_dir \"${CUDA_GENERATED_OUTPUT_DIR}\")\n else()\n set(cuda_compile_output_dir \"${CMAKE_CURRENT_BINARY_DIR}\")\n endif()\n\n # Reset the output variable\n set(_cuda_wrap_generated_files \"\")\n\n # Iterate over the macro arguments and create custom\n # commands for all the .cu files.\n foreach(file ${ARGN})\n # Ignore any file marked as a HEADER_FILE_ONLY\n get_source_file_property(_is_header ${file} HEADER_FILE_ONLY)\n if(${file} MATCHES \".*\\\\.cu$\" AND NOT _is_header)\n\n # Add a custom target to generate a c or ptx file. ######################\n\n get_filename_component( basename ${file} NAME )\n if( compile_to_ptx )\n set(generated_file_path \"${cuda_compile_output_dir}\")\n set(generated_file_basename \"${cuda_target}_generated_${basename}.ptx\")\n set(format_flag \"-ptx\")\n file(MAKE_DIRECTORY \"${cuda_compile_output_dir}\")\n else( compile_to_ptx )\n set(generated_file_path \"${cuda_compile_output_dir}/${CMAKE_CFG_INTDIR}\")\n set(generated_file_basename \"${cuda_target}_generated_${basename}${generated_extension}\")\n set(format_flag \"-c\")\n endif( compile_to_ptx )\n\n # Set all of our file names. Make sure that whatever filenames that have\n # generated_file_path in them get passed in through as a command line\n # argument, so that the ${CMAKE_CFG_INTDIR} gets expanded at run time\n # instead of configure time.\n set(generated_file \"${generated_file_path}/${generated_file_basename}\")\n set(cmake_dependency_file \"${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/${generated_file_basename}.depend\")\n set(NVCC_generated_dependency_file \"${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/${generated_file_basename}.NVCC-depend\")\n set(generated_cubin_file \"${generated_file_path}/${generated_file_basename}.cubin.txt\")\n set(custom_target_script \"${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/${generated_file_basename}.cmake\")\n\n # Setup properties for obj files:\n if( NOT compile_to_ptx )\n set_source_files_properties(\"${generated_file}\"\n PROPERTIES\n EXTERNAL_OBJECT true # This is an object file not to be compiled, but only be linked.\n )\n endif()\n\n # Don't add CMAKE_CURRENT_SOURCE_DIR if the path is already an absolute path.\n get_filename_component(file_path \"${file}\" PATH)\n if(IS_ABSOLUTE \"${file_path}\")\n set(source_file \"${file}\")\n else()\n set(source_file \"${CMAKE_CURRENT_SOURCE_DIR}/${file}\")\n endif()\n\n # Bring in the dependencies. Creates a variable CUDA_NVCC_DEPEND #######\n cuda_include_nvcc_dependencies(${cmake_dependency_file})\n\n # Convience string for output ###########################################\n if(CUDA_BUILD_EMULATION)\n set(cuda_build_type \"Emulation\")\n else(CUDA_BUILD_EMULATION)\n set(cuda_build_type \"Device\")\n endif(CUDA_BUILD_EMULATION)\n\n # Build the NVCC made dependency file ###################################\n set(build_cubin OFF)\n if ( NOT CUDA_BUILD_EMULATION AND CUDA_BUILD_CUBIN )\n if ( NOT compile_to_ptx )\n set ( build_cubin ON )\n endif( NOT compile_to_ptx )\n endif( NOT CUDA_BUILD_EMULATION AND CUDA_BUILD_CUBIN )\n\n # Configure the build script\n configure_file(\"${CUDA_run_nvcc}\" \"${custom_target_script}\" @ONLY)\n\n # So if a user specifies the same cuda file as input more than once, you\n # can have bad things happen with dependencies. Here we check an option\n # to see if this is the behavior they want.\n if(CUDA_ATTACH_VS_BUILD_RULE_TO_CUDA_FILE)\n set(main_dep MAIN_DEPENDENCY ${source_file})\n else()\n set(main_dep DEPENDS ${source_file})\n endif()\n\n if(CUDA_VERBOSE_BUILD)\n set(verbose_output ON)\n elseif(CMAKE_GENERATOR MATCHES \"Makefiles\")\n set(verbose_output \"$(VERBOSE)\")\n else()\n set(verbose_output OFF)\n endif()\n\n # Create up the comment string\n file(RELATIVE_PATH generated_file_relative_path \"${CMAKE_BINARY_DIR}\" \"${generated_file}\")\n if(compile_to_ptx)\n set(cuda_build_comment_string \"Building NVCC ptx file ${generated_file_relative_path}\")\n else()\n set(cuda_build_comment_string \"Building NVCC (${cuda_build_type}) object ${generated_file_relative_path}\")\n endif()\n\n # Build the generated file and dependency file ##########################\n add_custom_command(\n OUTPUT ${generated_file}\n # These output files depend on the source_file and the contents of cmake_dependency_file\n ${main_dep}\n DEPENDS ${CUDA_NVCC_DEPEND}\n DEPENDS ${custom_target_script}\n # Make sure the output directory exists before trying to write to it.\n COMMAND ${CMAKE_COMMAND} -E make_directory \"${generated_file_path}\"\n COMMAND ${CMAKE_COMMAND} ARGS\n -D verbose:BOOL=${verbose_output}\n ${ccbin_flags}\n -D build_configuration:STRING=${CUDA_build_configuration}\n -D \"generated_file:STRING=${generated_file}\"\n -D \"generated_cubin_file:STRING=${generated_cubin_file}\"\n -P \"${custom_target_script}\"\n COMMENT \"${cuda_build_comment_string}\"\n )\n\n # Make sure the build system knows the file is generated.\n set_source_files_properties(${generated_file} PROPERTIES GENERATED TRUE)\n\n # Don't add the object file to the list of generated files if we are using\n # visual studio and we are attaching the build rule to the cuda file. VS\n # will add our object file to the linker automatically for us.\n set(cuda_add_generated_file TRUE)\n\n if(NOT compile_to_ptx AND CMAKE_GENERATOR MATCHES \"Visual Studio\" AND CUDA_ATTACH_VS_BUILD_RULE_TO_CUDA_FILE)\n # Visual Studio 8 crashes when you close the solution when you don't add the object file.\n if(NOT CMAKE_GENERATOR MATCHES \"Visual Studio 8\")\n #message(\"Not adding ${generated_file}\")\n set(cuda_add_generated_file FALSE)\n endif()\n endif()\n\n if(cuda_add_generated_file)\n list(APPEND _cuda_wrap_generated_files ${generated_file})\n endif()\n\n # Add the other files that we want cmake to clean on a cleanup ##########\n list(APPEND CUDA_ADDITIONAL_CLEAN_FILES \"${cmake_dependency_file}\")\n list(REMOVE_DUPLICATES CUDA_ADDITIONAL_CLEAN_FILES)\n set(CUDA_ADDITIONAL_CLEAN_FILES ${CUDA_ADDITIONAL_CLEAN_FILES} CACHE INTERNAL \"List of intermediate files that are part of the cuda dependency scanning.\")\n\n endif(${file} MATCHES \".*\\\\.cu$\" AND NOT _is_header)\n endforeach(file)\n\n # Set the return parameter\n set(${generated_files} ${_cuda_wrap_generated_files})\nendmacro(CUDA_WRAP_SRCS)\n\n\n###############################################################################\n###############################################################################\n# ADD LIBRARY\n###############################################################################\n###############################################################################\nmacro(CUDA_ADD_LIBRARY cuda_target)\n\n CUDA_ADD_CUDA_INCLUDE_ONCE()\n\n # Separate the sources from the options\n CUDA_GET_SOURCES_AND_OPTIONS(_sources _cmake_options _options ${ARGN})\n CUDA_BUILD_SHARED_LIBRARY(_cuda_shared_flag ${ARGN})\n # Create custom commands and targets for each file.\n CUDA_WRAP_SRCS( ${cuda_target} OBJ _generated_files ${_sources}\n ${_cmake_options} ${_cuda_shared_flag}\n OPTIONS ${_options} )\n\n # Add the library.\n add_library(${cuda_target} ${_cmake_options}\n ${_generated_files}\n ${_sources}\n )\n\n target_link_libraries(${cuda_target}\n ${CUDA_LIBRARIES}\n )\n\n # We need to set the linker language based on what the expected generated file\n # would be. CUDA_C_OR_CXX is computed based on CUDA_HOST_COMPILATION_CPP.\n set_target_properties(${cuda_target}\n PROPERTIES\n LINKER_LANGUAGE ${CUDA_C_OR_CXX}\n )\n\nendmacro(CUDA_ADD_LIBRARY cuda_target)\n\n\n###############################################################################\n###############################################################################\n# ADD EXECUTABLE\n###############################################################################\n###############################################################################\nmacro(CUDA_ADD_EXECUTABLE cuda_target)\n\n CUDA_ADD_CUDA_INCLUDE_ONCE()\n\n # Separate the sources from the options\n CUDA_GET_SOURCES_AND_OPTIONS(_sources _cmake_options _options ${ARGN})\n # Create custom commands and targets for each file.\n CUDA_WRAP_SRCS( ${cuda_target} OBJ _generated_files ${_sources} OPTIONS ${_options} )\n\n # Add the library.\n add_executable(${cuda_target} ${_cmake_options}\n ${_generated_files}\n ${_sources}\n )\n\n target_link_libraries(${cuda_target}\n ${CUDA_LIBRARIES}\n )\n\n # We need to set the linker language based on what the expected generated file\n # would be. CUDA_C_OR_CXX is computed based on CUDA_HOST_COMPILATION_CPP.\n set_target_properties(${cuda_target}\n PROPERTIES\n LINKER_LANGUAGE ${CUDA_C_OR_CXX}\n )\n\nendmacro(CUDA_ADD_EXECUTABLE cuda_target)\n\n\n###############################################################################\n###############################################################################\n# CUDA COMPILE\n###############################################################################\n###############################################################################\nmacro(CUDA_COMPILE generated_files)\n\n # Separate the sources from the options\n CUDA_GET_SOURCES_AND_OPTIONS(_sources _cmake_options _options ${ARGN})\n # Create custom commands and targets for each file.\n CUDA_WRAP_SRCS( cuda_compile OBJ _generated_files ${_sources} ${_cmake_options}\n OPTIONS ${_options} )\n\n set( ${generated_files} ${_generated_files})\n\nendmacro(CUDA_COMPILE)\n\n\n###############################################################################\n###############################################################################\n# CUDA COMPILE PTX\n###############################################################################\n###############################################################################\nmacro(CUDA_COMPILE_PTX generated_files)\n\n # Separate the sources from the options\n CUDA_GET_SOURCES_AND_OPTIONS(_sources _cmake_options _options ${ARGN})\n # Create custom commands and targets for each file.\n CUDA_WRAP_SRCS( cuda_compile_ptx PTX _generated_files ${_sources} ${_cmake_options}\n OPTIONS ${_options} )\n\n set( ${generated_files} ${_generated_files})\n\nendmacro(CUDA_COMPILE_PTX)\n\n###############################################################################\n###############################################################################\n# CUDA ADD CUFFT TO TARGET\n###############################################################################\n###############################################################################\nmacro(CUDA_ADD_CUFFT_TO_TARGET target)\n if (CUDA_BUILD_EMULATION)\n target_link_libraries(${target} ${CUDA_cufftemu_LIBRARY})\n else()\n target_link_libraries(${target} ${CUDA_cufft_LIBRARY})\n endif()\nendmacro()\n\n###############################################################################\n###############################################################################\n# CUDA ADD CUBLAS TO TARGET\n###############################################################################\n###############################################################################\nmacro(CUDA_ADD_CUBLAS_TO_TARGET target)\n if (CUDA_BUILD_EMULATION)\n target_link_libraries(${target} ${CUDA_cublasemu_LIBRARY})\n else()\n target_link_libraries(${target} ${CUDA_cublas_LIBRARY})\n endif()\nendmacro()\n\n###############################################################################\n###############################################################################\n# CUDA BUILD CLEAN TARGET\n###############################################################################\n###############################################################################\nmacro(CUDA_BUILD_CLEAN_TARGET)\n # Call this after you add all your CUDA targets, and you will get a convience\n # target. You should also make clean after running this target to get the\n # build system to generate all the code again.\n\n set(cuda_clean_target_name clean_cuda_depends)\n if (CMAKE_GENERATOR MATCHES \"Visual Studio\")\n string(TOUPPER ${cuda_clean_target_name} cuda_clean_target_name)\n endif()\n add_custom_target(${cuda_clean_target_name}\n COMMAND ${CMAKE_COMMAND} -E remove ${CUDA_ADDITIONAL_CLEAN_FILES})\n\n # Clear out the variable, so the next time we configure it will be empty.\n # This is useful so that the files won't persist in the list after targets\n # have been removed.\n set(CUDA_ADDITIONAL_CLEAN_FILES \"\" CACHE INTERNAL \"List of intermediate files that are part of the cuda dependency scanning.\")\nendmacro(CUDA_BUILD_CLEAN_TARGET)\n" }, { "path": "cmake/FindLibJpeg.cmake", "content": "# Copyright (c) 2009, Whispersoft s.r.l.\n# All rights reserved.\n#\n# Redistribution and use in source and binary forms, with or without\n# modification, are permitted provided that the following conditions are\n# met:\n#\n# * Redistributions of source code must retain the above copyright\n# notice, this list of conditions and the following disclaimer.\n# * Redistributions in binary form must reproduce the above\n# copyright notice, this list of conditions and the following disclaimer\n# in the documentation and/or other materials provided with the\n# distribution.\n# * Neither the name of Whispersoft s.r.l. nor the names of its\n# contributors may be used to endorse or promote products derived from\n# this software without specific prior written permission.\n#\n# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n# \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n#\n\n# Finds LibJpeg library\n#\n# LibJpeg_INCLUDE_DIR - where to find jpeglib.h, etc.\n# LibJpeg_LIBRARIES - List of libraries when using LibJpeg.\n# LibJpeg_FOUND - True if LibJpeg found.\n#\n\nif (LibJpeg_INCLUDE_DIR)\n # Already in cache, be silent\n set(LibJpeg_FIND_QUIETLY TRUE)\nendif (LibJpeg_INCLUDE_DIR)\n\nfind_path(LibJpeg_INCLUDE_DIR jpeglib.h\n /opt/local/include\n /usr/local/include\n /usr/include\n)\n\nset(LibJpeg_NAMES jpeg)\nfind_library(LibJpeg_LIBRARY\n NAMES ${LibJpeg_NAMES}\n PATHS /usr/lib /usr/local/lib /opt/local/lib\n)\n\nif (LibJpeg_INCLUDE_DIR AND LibJpeg_LIBRARY)\n set(LibJpeg_FOUND TRUE)\n set( LibJpeg_LIBRARIES ${LibJpeg_LIBRARY} )\nelse (LibJpeg_INCLUDE_DIR AND LibJpeg_LIBRARY)\n set(LibJpeg_FOUND FALSE)\n set(LibJpeg_LIBRARIES)\nendif (LibJpeg_INCLUDE_DIR AND LibJpeg_LIBRARY)\n\nif (LibJpeg_FOUND)\n if (NOT LibJpeg_FIND_QUIETLY)\n message(STATUS \"Found LibJpeg: ${LibJpeg_LIBRARY}\")\n endif (NOT LibJpeg_FIND_QUIETLY)\nelse (LibJpeg_FOUND)\n if (LibJpeg_FIND_REQUIRED)\n message(STATUS \"Looked for LibJpeg libraries named ${LibJpeg_NAMES}.\")\n message(STATUS \"Include file detected: [${LibJpeg_INCLUDE_DIR}].\")\n message(STATUS \"Lib file detected: [${LibJpeg_LIBRARY}].\")\n message(FATAL_ERROR \"=========> Could NOT find LibJpeg library\")\n endif (LibJpeg_FIND_REQUIRED)\nendif (LibJpeg_FOUND)\n\nmark_as_advanced(\n LibJpeg_LIBRARY\n LibJpeg_INCLUDE_DIR\n )\n" }, { "path": "cmake/FindMPICH2.cmake", "content": "# - Message Passing Interface (MPI) module.\n# \n# The Message Passing Interface (MPI) is a library used to write\n# high-performance parallel applications that use message passing, and\n# is typically deployed on a cluster. MPI is a standard interface\n# (defined by the MPI forum) for which many implementations are\n# available. All of these implementations have somewhat different\n# compilation approaches (different include paths, libraries to link\n# against, etc.), and this module tries to smooth out those differences.\n#\n# This module will set the following variables:\n# MPI_FOUND TRUE if we have found MPI\n# MPI_COMPILE_FLAGS Compilation flags for MPI programs\n# MPI_INCLUDE_PATH Include path(s) for MPI header\n# MPI_LINK_FLAGS Linking flags for MPI programs\n# MPI_LIBRARY First MPI library to link against (cached)\n# MPI_EXTRA_LIBRARY Extra MPI libraries to link against (cached)\n# MPI_LIBRARIES All libraries to link MPI programs against\n# MPIEXEC Executable for running MPI programs\n# MPIEXEC_NUMPROC_FLAG Flag to pass to MPIEXEC before giving it the\n# number of processors to run on\n# MPIEXEC_PREFLAGS Flags to pass to MPIEXEC directly before the\n# executable to run.\n# MPIEXEC_POSTFLAGS Flags to pass to MPIEXEC after all other flags.\n#\n# This module will attempt to auto-detect these settings, first by\n# looking for a MPI compiler, which many MPI implementations provide\n# as a pass-through to the native compiler to simplify the compilation\n# of MPI programs. The MPI compiler is stored in the cache variable\n# MPI_COMPILER, and will attempt to look for commonly-named drivers\n# mpic++, mpicxx, mpiCC, or mpicc. If the compiler driver is found and\n# recognized, it will be used to set all of the module variables. To\n# skip this auto-detection, set MPI_LIBRARY and MPI_INCLUDE_PATH in\n# the CMake cache.\n#\n# If no compiler driver is found or the compiler driver is not\n# recognized, this module will then search for common include paths\n# and library names to try to detect MPI. \n#\n# If CMake initially finds a different MPI than was intended, and you\n# want to use the MPI compiler auto-detection for a different MPI\n# implementation, set MPI_COMPILER to the MPI compiler driver you want\n# to use (e.g., mpicxx) and then set MPI_LIBRARY to the string\n# MPI_LIBRARY-NOTFOUND. When you re-configure, auto-detection of MPI\n# will run again with the newly-specified MPI_COMPILER.\n#\n# When using MPIEXEC to execute MPI applications, you should typically\n# use all of the MPIEXEC flags as follows:\n# ${MPIEXEC} ${MPIEXEC_NUMPROC_FLAG} PROCS ${MPIEXEC_PREFLAGS} EXECUTABLE\n# ${MPIEXEC_POSTFLAGS} ARGS\n# where PROCS is the number of processors on which to execute the program,\n# EXECUTABLE is the MPI program, and ARGS are the arguments to pass to the \n# MPI program.\n\n# Try to find the MPI driver program\nfind_program(MPI_COMPILER \n NAMES mpicxx\n DOC \"MPI compiler. Used only to detect MPI compilation flags.\")\nmark_as_advanced(MPI_COMPILER)\n\nfind_program(MPIEXEC\n NAMES mpiexec \n DOC \"Executable for running MPI programs.\")\n\nset(MPIEXEC_NUMPROC_FLAG \"-np\" CACHE STRING \"Flag used by MPI to specify the number of processes for MPIEXEC; the next option will be the number of processes.\")\nset(MPIEXEC_PREFLAGS \"\" CACHE STRING \"These flags will be directly before the executable that is being run by MPIEXEC.\")\nset(MPIEXEC_POSTFLAGS \"\" CACHE STRING \"These flags will come after all flags given to MPIEXEC.\")\nset(MPIEXEC_MAX_NUMPROCS \"2\" CACHE STRING \"Maximum number of processors available to run MPI applications.\")\nmark_as_advanced(MPIEXEC MPIEXEC_NUMPROC_FLAG MPIEXEC_PREFLAGS \n MPIEXEC_POSTFLAGS MPIEXEC_MAX_NUMPROCS)\n\nif (MPI_INCLUDE_PATH AND MPI_LIBRARY)\n # Do nothing: we already have MPI_INCLUDE_PATH and MPI_LIBRARY in\n # the cache, and we don't want to override those settings.\nelseif (MPI_COMPILER)\n # Check whether the -showme:compile option works. This indicates\n # that we have either Open MPI or a newer version of LAM-MPI, and\n # implies that -showme:link will also work.\n exec_program(${MPI_COMPILER} \n ARGS -showme:compile \n OUTPUT_VARIABLE MPI_COMPILE_CMDLINE\n RETURN_VALUE MPI_COMPILER_RETURN)\n\n if (MPI_COMPILER_RETURN EQUAL 0)\n # If we appear to have -showme:compile, then we should also have\n # -showme:link. Try it.\n exec_program(${MPI_COMPILER} \n ARGS -showme:link\n OUTPUT_VARIABLE MPI_LINK_CMDLINE\n RETURN_VALUE MPI_COMPILER_RETURN)\n\n # Note that we probably have -showme:incdirs and -showme:libdirs\n # as well.\n set(MPI_COMPILER_MAY_HAVE_INCLIBDIRS TRUE)\n endif (MPI_COMPILER_RETURN EQUAL 0)\n\n if (MPI_COMPILER_RETURN EQUAL 0)\n # Do nothing: we have our command lines now\n else (MPI_COMPILER_RETURN EQUAL 0)\n # Older versions of LAM-MPI have \"-showme\". Try it.\n exec_program(${MPI_COMPILER} \n ARGS -showme\n OUTPUT_VARIABLE MPI_COMPILE_CMDLINE\n RETURN_VALUE MPI_COMPILER_RETURN)\n endif (MPI_COMPILER_RETURN EQUAL 0) \n\n if (MPI_COMPILER_RETURN EQUAL 0)\n # Do nothing: we have our command lines now\n else (MPI_COMPILER_RETURN EQUAL 0)\n # MPICH uses \"-show\". Try it.\n exec_program(${MPI_COMPILER} \n ARGS -show\n OUTPUT_VARIABLE MPI_COMPILE_CMDLINE\n RETURN_VALUE MPI_COMPILER_RETURN)\n endif (MPI_COMPILER_RETURN EQUAL 0) \n\n if (MPI_COMPILER_RETURN EQUAL 0)\n # We have our command lines, but we might need to copy\n # MPI_COMPILE_CMDLINE into MPI_LINK_CMDLINE, if the underlying\n if (NOT MPI_LINK_CMDLINE)\n SET(MPI_LINK_CMDLINE ${MPI_COMPILE_CMDLINE})\n endif (NOT MPI_LINK_CMDLINE)\n else (MPI_COMPILER_RETURN EQUAL 0)\n message(STATUS \"Unable to determine MPI from MPI driver ${MPI_COMPILER}\")\n endif (MPI_COMPILER_RETURN EQUAL 0)\nendif (MPI_INCLUDE_PATH AND MPI_LIBRARY)\n\nif (MPI_INCLUDE_PATH AND MPI_LIBRARY)\n # Do nothing: we already have MPI_INCLUDE_PATH and MPI_LIBRARY in\n # the cache, and we don't want to override those settings.\nelseif (MPI_COMPILE_CMDLINE)\n # Extract compile flags from the compile command line.\n string(REGEX MATCHALL \"-D([^\\\" ]+|\\\"[^\\\"]+\\\")\" MPI_ALL_COMPILE_FLAGS \"${MPI_COMPILE_CMDLINE}\")\n set(MPI_COMPILE_FLAGS_WORK)\n foreach(FLAG ${MPI_ALL_COMPILE_FLAGS})\n if (MPI_COMPILE_FLAGS_WORK)\n set(MPI_COMPILE_FLAGS_WORK \"${MPI_COMPILE_FLAGS_WORK} ${FLAG}\")\n else(MPI_COMPILE_FLAGS_WORK)\n set(MPI_COMPILE_FLAGS_WORK ${FLAG})\n endif(MPI_COMPILE_FLAGS_WORK)\n endforeach(FLAG)\n\n # Extract include paths from compile command line\n string(REGEX MATCHALL \"-I([^\\\" ]+|\\\"[^\\\"]+\\\")\" MPI_ALL_INCLUDE_PATHS \"${MPI_COMPILE_CMDLINE}\")\n set(MPI_INCLUDE_PATH_WORK)\n foreach(IPATH ${MPI_ALL_INCLUDE_PATHS})\n string(REGEX REPLACE \"^-I\" \"\" IPATH ${IPATH})\n string(REGEX REPLACE \"//\" \"/\" IPATH ${IPATH})\n list(APPEND MPI_INCLUDE_PATH_WORK ${IPATH})\n endforeach(IPATH)\n \n if (NOT MPI_INCLUDE_PATH_WORK)\n if (MPI_COMPILER_MAY_HAVE_INCLIBDIRS)\n # The compile command line didn't have any include paths on it,\n # but we may have -showme:incdirs. Use it.\n exec_program(${MPI_COMPILER} \n ARGS -showme:incdirs\n OUTPUT_VARIABLE MPI_INCLUDE_PATH_WORK\n RETURN_VALUE MPI_COMPILER_RETURN)\n separate_arguments(MPI_INCLUDE_PATH_WORK)\n endif (MPI_COMPILER_MAY_HAVE_INCLIBDIRS)\n endif (NOT MPI_INCLUDE_PATH_WORK)\n\n if (NOT MPI_INCLUDE_PATH_WORK)\n # If all else fails, just search for mpi.h in the normal include\n # paths.\n find_path(MPI_INCLUDE_PATH mpi.h)\n set(MPI_INCLUDE_PATH_WORK ${MPI_INCLUDE_PATH})\n endif (NOT MPI_INCLUDE_PATH_WORK)\n\n # Extract linker paths from the link command line\n string(REGEX MATCHALL \"-L([^\\\" ]+|\\\"[^\\\"]+\\\")\" MPI_ALL_LINK_PATHS \"${MPI_LINK_CMDLINE}\")\n set(MPI_LINK_PATH)\n foreach(LPATH ${MPI_ALL_LINK_PATHS})\n string(REGEX REPLACE \"^-L\" \"\" LPATH ${LPATH})\n string(REGEX REPLACE \"//\" \"/\" LPATH ${LPATH})\n list(APPEND MPI_LINK_PATH ${LPATH})\n endforeach(LPATH)\n\n if (NOT MPI_LINK_PATH)\n if (MPI_COMPILER_MAY_HAVE_INCLIBDIRS)\n # The compile command line didn't have any linking paths on it,\n # but we may have -showme:libdirs. Use it.\n exec_program(${MPI_COMPILER} \n ARGS -showme:libdirs\n OUTPUT_VARIABLE MPI_LINK_PATH\n RETURN_VALUE MPI_COMPILER_RETURN)\n separate_arguments(MPI_LINK_PATH)\n endif (MPI_COMPILER_MAY_HAVE_INCLIBDIRS)\n endif (NOT MPI_LINK_PATH)\n\n # Extract linker flags from the link command line\n string(REGEX MATCHALL \"-Wl,([^\\\" ]+|\\\"[^\\\"]+\\\")\" MPI_ALL_LINK_FLAGS \"${MPI_LINK_CMDLINE}\")\n set(MPI_LINK_FLAGS_WORK)\n foreach(FLAG ${MPI_ALL_LINK_FLAGS})\n if (MPI_LINK_FLAGS_WORK)\n set(MPI_LINK_FLAGS_WORK \"${MPI_LINK_FLAGS_WORK} ${FLAG}\")\n else(MPI_LINK_FLAGS_WORK)\n set(MPI_LINK_FLAGS_WORK ${FLAG})\n endif(MPI_LINK_FLAGS_WORK)\n endforeach(FLAG)\n\n # Extract the set of libraries to link against from the link command\n # line\n string(REGEX MATCHALL \"-l([^\\\" ]+|\\\"[^\\\"]+\\\")\" MPI_LIBNAMES \"${MPI_LINK_CMDLINE}\")\n message(STATUS \"MPI libnames: ${MPI_LIBNAMES}\")\n # Determine full path names for all of the libraries that one needs\n # to link against in an MPI program\n set(MPI_LIBRARIES)\n foreach(LIB ${MPI_LIBNAMES})\n string(REGEX REPLACE \"^-l\" \"\" LIB ${LIB})\n set(MPI_LIB \"MPI_LIB-NOTFOUND\" CACHE FILEPATH \"Cleared\" FORCE)\n find_library(MPI_LIB ${LIB} HINTS ${MPI_LINK_PATH})\n if (MPI_LIB)\n list(APPEND MPI_LIBRARIES ${MPI_LIB})\n else (MPI_LIB)\n # message(STATUS \"Unable to find MPI library ${LIB}\")\n endif (MPI_LIB)\n endforeach(LIB)\n set(MPI_LIB \"MPI_LIB-NOTFOUND\" CACHE INTERNAL \"Scratch variable for MPI detection\" FORCE)\n\n message(STATUS \"MPI libraries: ${MPI_LIBRARIES}\")\n\n # Chop MPI_LIBRARIES into the old-style MPI_LIBRARY and\n # MPI_EXTRA_LIBRARY.\n list(LENGTH MPI_LIBRARIES MPI_NUMLIBS)\n list(LENGTH MPI_LIBNAMES MPI_NUMLIBS_EXPECTED)\n# if (MPI_NUMLIBS EQUAL MPI_NUMLIBS_EXPECTED)\n list(GET MPI_LIBRARIES 0 MPI_LIBRARY_WORK)\n set(MPI_LIBRARY ${MPI_LIBRARY_WORK} CACHE FILEPATH \"MPI library to link against\" FORCE)\n# else (MPI_NUMLIBS EQUAL MPI_NUMLIBS_EXPECTED)\n# set(MPI_LIBRARY \"MPI_LIBRARY-NOTFOUND\" CACHE FILEPATH \"MPI library to link against\" FORCE)\n# endif (MPI_NUMLIBS EQUAL MPI_NUMLIBS_EXPECTED)\n if (MPI_NUMLIBS GREATER 1)\n set(MPI_EXTRA_LIBRARY_WORK ${MPI_LIBRARIES})\n list(REMOVE_AT MPI_EXTRA_LIBRARY_WORK 0)\n set(MPI_EXTRA_LIBRARY ${MPI_EXTRA_LIBRARY_WORK} CACHE STRING \"Extra MPI libraries to link against\" FORCE)\n else (MPI_NUMLIBS GREATER 1)\n set(MPI_EXTRA_LIBRARY \"MPI_EXTRA_LIBRARY-NOTFOUND\" CACHE STRING \"Extra MPI libraries to link against\" FORCE)\n endif (MPI_NUMLIBS GREATER 1)\n\n # Set up all of the appropriate cache entries\n set(MPI_COMPILE_FLAGS ${MPI_COMPILE_FLAGS_WORK} CACHE STRING \"MPI compilation flags\" FORCE)\n set(MPI_INCLUDE_PATH ${MPI_INCLUDE_PATH_WORK} CACHE STRING \"MPI include path\" FORCE)\n set(MPI_LINK_FLAGS ${MPI_LINK_FLAGS_WORK} CACHE STRING \"MPI linking flags\" FORCE)\nelse (MPI_COMPILE_CMDLINE)\n find_path(MPI_INCLUDE_PATH mpi.h \n /usr/local/include \n /usr/include \n /usr/include/mpi\n /usr/include/mpich2\n /usr/local/mpi/include\n \"C:/Program Files/MPICH/SDK/Include\" \n \"$ENV{SystemDrive}/Program Files/MPICH2/include\"\n \"$ENV{SystemDrive}/Program Files/Microsoft Compute Cluster Pack/Include\"\n )\n \n # Decide between 32-bit and 64-bit libraries for Microsoft's MPI\n if (CMAKE_CL_64)\n set(MS_MPI_ARCH_DIR amd64)\n else (CMAKE_CL_64)\n set(MS_MPI_ARCH_DIR i386)\n endif (CMAKE_CL_64)\n \n find_library(MPI_LIBRARY \n NAMES mpi mpich msmpi\n PATHS /usr/lib /usr/local/lib /usr/local/mpi/lib \n \"C:/Program Files/MPICH/SDK/Lib\" \n \"$ENV{SystemDrive}/Program Files/MPICH/SDK/Lib\"\n \"$ENV{SystemDrive}/Program Files/Microsoft Compute Cluster Pack/Lib/${MS_MPI_ARCH_DIR}\"\n )\n find_library(MPI_LIBRARY \n NAMES mpich2\n PATHS\n \"$ENV{SystemDrive}/Program Files/MPICH2/Lib\")\n\n find_library(MPI_EXTRA_LIBRARY \n NAMES mpi++\n PATHS /usr/lib /usr/local/lib /usr/local/mpi/lib \n \"C:/Program Files/MPICH/SDK/Lib\" \n DOC \"Extra MPI libraries to link against.\")\n\n set(MPI_COMPILE_FLAGS \"\" CACHE STRING \"MPI compilation flags\")\n set(MPI_LINK_FLAGS \"\" CACHE STRING \"MPI linking flags\")\nendif (MPI_INCLUDE_PATH AND MPI_LIBRARY)\n\n# on BlueGene/L the MPI lib is named libmpich.rts.a, there also these additional libs are required\nif(\"${MPI_LIBRARY}\" MATCHES \"mpich.rts\")\n set(MPI_EXTRA_LIBRARY ${MPI_EXTRA_LIBRARY} msglayer.rts devices.rts rts.rts devices.rts)\n set(MPI_LIBRARY ${MPI_LIBRARY} msglayer.rts devices.rts rts.rts devices.rts)\nendif(\"${MPI_LIBRARY}\" MATCHES \"mpich.rts\")\n\n# Set up extra variables to conform to \nif (MPI_EXTRA_LIBRARY)\n set(MPI_LIBRARIES ${MPI_LIBRARY} ${MPI_EXTRA_LIBRARY})\nelse (MPI_EXTRA_LIBRARY)\n set(MPI_LIBRARIES ${MPI_LIBRARY})\nendif (MPI_EXTRA_LIBRARY)\n\nif (MPI_INCLUDE_PATH AND MPI_LIBRARY)\n set(MPI_FOUND TRUE)\nelse (MPI_INCLUDE_PATH AND MPI_LIBRARY)\n set(MPI_FOUND FALSE)\nendif (MPI_INCLUDE_PATH AND MPI_LIBRARY)\n\ninclude(FindPackageHandleStandardArgs)\n# handle the QUIETLY and REQUIRED arguments \nfind_package_handle_standard_args(MPI DEFAULT_MSG MPI_LIBRARY MPI_INCLUDE_PATH)\n\nmark_as_advanced(MPI_INCLUDE_PATH MPI_COMPILE_FLAGS MPI_LINK_FLAGS MPI_LIBRARY \n MPI_EXTRA_LIBRARY)\n" }, { "path": "cmake/FindMatlab.cmake", "content": "# - this module looks for Matlab\n# Defines:\n# MATLAB_INCLUDE_DIR: include path for mex.h, engine.h\n# MATLAB_LIBRARIES: required libraries: libmex, etc\n# MATLAB_MAT_LIBRARY: path to libmat.lib\n# MATLAB_MEX_LIBRARY: path to libmex.lib\n# MATLAB_MX_LIBRARY: path to libmx.lib\n# MATLAB_ENG_LIBRARY: path to libeng.lib\n\n\nSET(MATLAB_FOUND 0)\nIF(WIN32)\n IF(${CMAKE_GENERATOR} MATCHES \"Visual Studio .*\" OR ${CMAKE_GENERATOR} MATCHES \"NMake Makefiles\")\n SET(MATLAB_ROOT \"[HKEY_LOCAL_MACHINE\\\\SOFTWARE\\\\MathWorks\\\\MATLAB\\\\7.0;MATLABROOT]/extern/lib/win32/microsoft/\")\n ELSE(${CMAKE_GENERATOR} MATCHES \"Visual Studio .*\" OR ${CMAKE_GENERATOR} MATCHES \"NMake Makefiles\")\n IF(${CMAKE_GENERATOR} MATCHES \"Borland\")\n # Same here, there are also: bcc50 and bcc51 directories\n SET(MATLAB_ROOT \"[HKEY_LOCAL_MACHINE\\\\SOFTWARE\\\\MathWorks\\\\MATLAB\\\\7.0;MATLABROOT]/extern/lib/win32/microsoft/bcc54\")\n ELSE(${CMAKE_GENERATOR} MATCHES \"Borland\")\n MESSAGE(FATAL_ERROR \"Generator not compatible: ${CMAKE_GENERATOR}\")\n ENDIF(${CMAKE_GENERATOR} MATCHES \"Borland\")\n ENDIF(${CMAKE_GENERATOR} MATCHES \"Visual Studio .*\" OR ${CMAKE_GENERATOR} MATCHES \"NMake Makefiles\")\n FIND_LIBRARY(MATLAB_MEX_LIBRARY\n libmex\n ${MATLAB_ROOT}\n )\n FIND_LIBRARY(MATLAB_MX_LIBRARY\n libmx\n ${MATLAB_ROOT}\n )\n FIND_LIBRARY(MATLAB_ENG_LIBRARY\n libeng\n ${MATLAB_ROOT}\n )\n FIND_LIBRARY(MATLAB_MAT_LIBRARY\n libmat\n ${MATLAB_ROOT}\n )\n\n FIND_PATH(MATLAB_INCLUDE_DIR\n \"mex.h\"\n \"[HKEY_LOCAL_MACHINE\\\\SOFTWARE\\\\MathWorks\\\\MATLAB\\\\7.0;MATLABROOT]/extern/include\"\n )\nELSE( WIN32 )\n IF(NOT MATLAB_ROOT)\n IF($ENV{MATLAB_ROOT})\n SET(MATLAB_ROOT $ENV{MATLAB_ROOT})\n ELSE($ENV{MATLAB_ROOT})\n SET(MATLAB_ROOT /opt/matlab)\n ENDIF($ENV{MATLAB_ROOT}) \n ENDIF(NOT MATLAB_ROOT)\n IF(CMAKE_SIZEOF_VOID_P EQUAL 4)\n # Regular x86\n SET(MATLAB_SYS\n ${MATLAB_ROOT}/bin/glnx86\n )\n ELSE(CMAKE_SIZEOF_VOID_P EQUAL 4)\n # AMD64:\n SET(MATLAB_SYS\n ${MATLAB_ROOT}/bin/glnxa64\n )\n ENDIF(CMAKE_SIZEOF_VOID_P EQUAL 4)\n FIND_LIBRARY(MATLAB_MEX_LIBRARY\n mex\n ${MATLAB_SYS}\n )\n FIND_LIBRARY(MATLAB_MX_LIBRARY\n mx\n ${MATLAB_SYS}\n )\n FIND_LIBRARY(MATLAB_MAT_LIBRARY\n mat\n ${MATLAB_SYS}\n )\n FIND_LIBRARY(MATLAB_ENG_LIBRARY\n eng\n ${MATLAB_SYS}\n )\n FIND_PATH(MATLAB_INCLUDE_DIR\n \"mex.h\"\n ${MATLAB_ROOT}/extern/include\n )\n\nENDIF(WIN32)\n\n# This is common to UNIX and Win32:\nSET(MATLAB_LIBRARIES\n ${MATLAB_MEX_LIBRARY}\n ${MATLAB_MX_LIBRARY}\n ${MATLAB_ENG_LIBRARY}\n)\n\nIF(MATLAB_INCLUDE_DIR \n AND MATLAB_MEX_LIBRARY \n AND MATLAB_MAT_LIBRARY\n AND MATLAB_ENG_LIBRARY\n AND MATLAB_MX_LIBRARY)\n SET(MATLAB_LIBRARIES ${MATLAB_MX_LIBRARY} ${MATLAB_MEX_LIBRARY} ${MATLAB_ENG_LIBRARY} ${MATLAB_MAT_LIBRARY})\nENDIF(MATLAB_INCLUDE_DIR \n AND MATLAB_MEX_LIBRARY \n AND MATLAB_MAT_LIBRARY\n AND MATLAB_ENG_LIBRARY\n AND MATLAB_MX_LIBRARY)\n\nMARK_AS_ADVANCED(\n MATLAB_MEX_LIBRARY\n MATLAB_MX_LIBRARY\n MATLAB_ENG_LIBRARY\n MATLAB_INCLUDE_DIR\n MATLAB_ROOT\n)\n\nINCLUDE(FindPackageHandleStandardArgs)\nFIND_PACKAGE_HANDLE_STANDARD_ARGS(Matlab \n MATLAB_INCLUDE_DIR \n MATLAB_MEX_LIBRARY \n MATLAB_MAT_LIBRARY\n MATLAB_ENG_LIBRARY\n MATLAB_MX_LIBRARY )\n\n" }, { "path": "cmake/FindPerftools.cmake", "content": "## Obtained from:\n#\n# http://tracker.icir.org/bro/browser/branches/matthias/vast/cmake/FindPerftools.cmake?order=date\n# \n# \n\n# Copyright (c) 2009, International Computer Science Institute All\n# rights reserved.\n#\n# Redistribution and use in source and binary forms, with or without\n# modification, are permitted provided that the following conditions\n# are met:\n#\n# * Redistributions of source code must retain the above copyright\n# notice, this list of conditions and the following disclaimer.\n#\n# * Redistributions in binary form must reproduce the above\n# copyright notice, this list of conditions and the following\n# disclaimer in the documentation and/or other materials\n# provided with the distribution.\n#\n# * Neither the name of the International Computer Science\n# Institute the names of its contributors may be used to endorse\n# or promote products derived from this software without\n# specific prior written permission.\n#\n# THIS SOFTWARE IS PROVIDED BY International Computer Science\n# Institute ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,\n# INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF\n# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n# DISCLAIMED. IN NO EVENT SHALL International Computer Science\n# Institute BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,\n# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,\n# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR\n# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY\n# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE\n# USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH\n# DAMAGE.\n\n\n# - Try to find Google perftools include dirs and libraries \n#\n# Usage of this module as follows:\n#\n# find_package(Perftools)\n#\n# Variables used by this module, they can change the default behaviour and need\n# to be set before calling find_package:\n#\n# PERFTOOLS_ROOT Preferred installation prefix for searching for\n# Perftools, set this if the module has problems\n# finding the proper installation path.\n# PERFTOOLS_INCLUDEDIR Set this to the include directory of the Google\n# perftools, if the module has problems finding the\n# installation path.\n# PERFTOOLS_LIBRARYDIR Set this to the library directory of the Google\n# perftools if the module has problems finding the\n# proper installation path.\n#\n# Variables defined by this module:\n#\n# Perftools_FOUND System has Google perftools, this means the\n# include dir and all the libraries were found. \n# Perftools_INCLUDE_DIRS Google perftools include directories. \n# Perftools_LIBRARIES Link these to use the Google perftools libraries.\n#\n# Perftools_TCMALLOC_LIBRARY Path to the tcmalloc library.\n# Perftools_STACKTRACE_LIBRARY Path to the stacktrace library.\n# Perftools_PROFILER_LIBRARY Path to the profiler library.\n\nif (PERFTOOLS_ROOT)\n set(Perftools_ADDITIONAL_INCLUDE_SEARCH_DIRS ${PERFTOOLS_ROOT}/include)\n set(Perftools_ADDITIONAL_LIBRARY_SEARCH_DIRS ${PERFTOOLS_ROOT}/lib)\nendif ()\n\nif (PERFTOOLS_INCLUDEDIR)\n set(Perftools_ADDITIONAL_INCLUDE_SEARCH_DIRS ${PERFTOOLS_ROOT}/include)\nendif ()\n\nif (PERFTOOLS_LIBRARYDIR)\n set(Perftools_ADDITIONAL_LIBRARY_SEARCH_DIRS ${PERFTOOLS_ROOT}/lib)\nendif ()\n\n\nif (Perftools_LIBRARIES AND Perftools_INCLUDE_DIRS)\n # In cache already.\n set(Perftools_FOUND true)\nelse ()\n find_path(Perftools_INCLUDE_DIRS\n NAMES\n google/heap-profiler.h\n PATHS\n ${Perftools_ADDITIONAL_INCLUDE_SEARCH_DIRS}\n /usr/local/include\n /opt/local/include\n /sw/include\n /usr/include\n )\n\n # tcmalloc\n set(tcmalloc_names ${tcmalloc_names} tcmalloc)\n find_library(perftools_tcmalloc_library\n NAMES \n ${tcmalloc_names}\n PATHS \n ${Perftools_ADDITIONAL_LIBRARY_SEARCH_DIRS}\n /usr/local/lib\n /opt/local/lib\n /sw/lib\n /usr/lib\n )\n\n if (perftools_tcmalloc_library AND Perftools_INCLUDE_DIRS)\n set(Perftools_TCMALLOC_LIBRARY ${perftools_tcmalloc_library})\n set(Perftools_LIBRARIES \n ${Perftools_LIBRARIES} ${perftools_tcmalloc_library})\n set(Perftools_FOUND true)\n else ()\n set(Perftools_FOUND false)\n endif ()\n\n\n # stacktrace\n set(stacktrace_names ${stacktrace_names} stacktrace)\n find_library(perftools_stacktrace_library\n NAMES \n ${stacktrace_names}\n PATHS \n ${Perftools_ADDITIONAL_LIBRARY_SEARCH_DIRS}\n /usr/local/lib\n /opt/local/lib\n /sw/lib\n /usr/lib\n )\n\n if (perftools_stacktrace_library AND Perftools_INCLUDE_DIRS)\n set(Perftools_STACKTRACE_LIBRARY ${perftools_stacktrace_library})\n set(Perftools_LIBRARIES \n ${Perftools_LIBRARIES} ${perftools_stacktrace_library})\n endif ()\n\n\n # profiler\n set(profiler_names ${profiler_names} profiler)\n find_library(perftools_profiler_library\n NAMES \n ${profiler_names}\n PATHS \n ${Perftools_ADDITIONAL_LIBRARY_SEARCH_DIRS}\n /usr/local/lib\n /opt/local/lib\n /sw/lib\n /usr/lib\n )\n\n if (perftools_profiler_library AND Perftools_INCLUDE_DIRS)\n set(Perftools_PROFILER_LIBRARY ${perftools_profiler_library})\n set(Perftools_LIBRARIES \n ${Perftools_LIBRARIES} ${perftools_profiler_library})\n endif ()\n\n if (Perftools_FOUND)\n if (NOT Perftools_FIND_QUIETLY)\n message(STATUS \"Found Google perftools\")\n endif ()\n else ()\n if (Perftools_FIND_REQUIRED)\n message(FATAL_ERROR \"Could not find Google perftools\")\n endif ()\n endif ()\n\n mark_as_advanced(\n Perftools_INCLUDE_DIRS\n Perftools_LIBRARIES\n Perftools_TCMALLOC_LIBRARY\n Perftools_STACKTRACE_LIBRARY\n Perftools_PROFILER_LIBRARY\n )\nendif()\n" }, { "path": "cmake/Mex_stub.cpp", "content": "// Adopted from: http://www.cmake.org/Wiki/images/7/72/Mex_stub.cpp\n// and tutorial: http://www.cmake.org/Wiki/CMake:MatlabMex\n// on June 9, 2010 (akyrola)\n\n#include \"mex.h\"\n\n\nextern void __mexFunction__(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]);\nextern void __at_exit__();\n\nstatic void at_exit();\n\nvoid mexFunction(int nlhs, mxArray *plhs[],\n\t\t int nrhs, const mxArray *prhs[])\n{\n \n mexAtExit(&at_exit);\n\n __mexFunction__(nlhs, plhs, nrhs, prhs);\n\n}\n\nstatic void at_exit()\n{\n __at_exit__(); \n}\n" }, { "path": "cmake/mex_link.sh", "content": "#!/bin/sh\n\nOUTPUT=$1\nSTATIC_LIB_NAME=$OUTPUT\nMOVE_LOCATION=$2\nBASEDIR=$3\nLINKFILES=$4\nLINKER_FLAGS= -shared -Wl,--version-script,/afs/cs.cmu.edu/misc/matlab/amd64_f7/7.9/lib/matlab7/extern/lib/glnxa64/mexFunction.map -Wl,--no-undefined\n\necho $OUTPUT\necho $STATIC_LIB_NAME\n\n \n\necho 'Running godawful mex linking hack...'\n# mex_stub.o compiled with:\ng++ -g -Wall -fPIC -ansi -D_GNU_SOURCE -fPIC -fno-omit-frame-pointer -pthread -DMATLAB_MEX_FILE -lmx -lmex -lmat -lm -I/afs/cs.cmu.edu/local/matlab/amd64_f7/7.9/lib/matlab7/extern/include -c ${BASEDIR}/cmake/Mex_stub.cpp -o mex_stub.o\nmex -g -cxx CC='gcc' CXX='g++' LD='g++' -L./ -lglib-2.0 -l$STATIC_LIB_NAME $LINKER_FLAGS -output $OUTPUT mex_stub.o $LINKFILES\n#mv $OUTPUT.mexa64 $MOVE_LOCATION\n\n#-lpthread -lgthread-2.0 -lrt -DMX_COMPAT_32" }, { "path": "configure", "content": "#!/bin/bash\n##=============================================================================\n## Support code\nfunction download_file {\n # detect wget\n echo \"Downloading $2 from $1 ...\"\n if [ -z `which wget` ] ; then\n if [ -z `which curl` ] ; then\n echo \"Unable to find either curl or wget! Cannot proceed with\n automatic install.\"\n exit 1\n fi\n curl $1 -o $2\n else\n wget $1 -O $2\n fi\n} # end of download file\nfunction print_help {\n echo \"Usage: ./configure [--prefix=PREFIX] [--experimental]\"\n echo\n echo \" --cleanup remove all build directories\"\n echo\n echo \" --prefix=[PREFIX] GraphLab Installation target directory. Defaults to /usr/local\"\n echo\n echo \" --ide=[Xcode] Specify the ide to use when building GraphLab.\"\n echo\n echo \" --no_openmp Disables OpenMP. Disabled by default on Mac.\"\n echo\n echo \" --no_mpi Disables MPI (disables distributed execution).\"\n echo\n echo \" --no_jvm Disable JVM features including HDFS integration.\"\n echo\n echo \" --no_tcmalloc Disable using tcmalloc instead of malloc.\"\n echo\n echo \" --experimental Turns on undocumented experimental capabilities. \"\n echo\n echo \" --c++11 Turns on C++11 experimental features. \"\n echo\n echo \" --vid32 Switch to 32bit vertex ids.\"\n echo\n echo \" -D var=value Specify definitions to be passed on to cmake.\"\n\n exit 1\n} # end of print help\nfunction run_cleanup {\n #!/bin/bash\n echo \"This script completely erases all build folders including dependencies!!!\"\n echo \"Are you sure you want to continue? (yes or no)\"\n read yesorno;\n if [ \"$yesorno\" == \"yes\" ]; then\n echo \"Removing release and debug folders\";\n rm -rf release debug deps configure.deps\n else\n echo \"Doing nothing!\";\n fi\n exit 1\n} # end of run cleanup\nfunction unknown_option {\n echo \"Unrecognized option: $1\"\n echo \"To get help, run ./configure --help\"\n exit 1\n} # end of unknown option\n\n\n\n\n## this function is broken\n# function check_version {\n# local version=$1 check=$2\n# local winner=$(echo -e \"$version\\n$check\" | sed '/^$/d' | sort -nr | head -1)\n# [[ \"$winner\" = \"$version\" ]] && return 0\n# return 1\n# } # end of check version\n\n## Obtained from forum:\n# http://stackoverflow.com/questions/4023830/bash-how-compare-two-strings-in-version-format\n# Return 0 if version are equal\n# Returns 1 if version 1 is larger\n# Returns 2 if version 2 is larger\nfunction check_version {\n if [[ $1 == $2 ]]\n then\n return 0\n fi\n local IFS=.\n local i ver1=($1) ver2=($2)\n # fill empty fields in ver1 with zeros\n for ((i=${#ver1[@]}; i<${#ver2[@]}; i++))\n do\n ver1[i]=0\n done\n for ((i=0; i<${#ver1[@]}; i++))\n do\n if [[ -z ${ver2[i]} ]]\n then\n # fill empty fields in ver2 with zeros\n ver2[i]=0\n fi\n if ((10#${ver1[i]} > 10#${ver2[i]}))\n then\n return 1\n fi\n if ((10#${ver1[i]} < 10#${ver2[i]}))\n then\n return 2\n fi\n done\n return 0\n}\n\n\n##=============================================================================\n## Main configuration processing\n\n## Define some defaults which are modified by the script and whatever\n## is defined in configure.deps\nRELEASE_DIR=release\nDEBUG_DIR=debug\nINSTALL_DIR=/usr/local\nEXPERIMENTAL=false\nGRAPHLAB_HOME=$PWD\nDEPS_PREFIX=$PWD/deps/local\nNO_OPENMP=false\nNO_MPI=false\nNO_TCMALLOC=false\nCPP11=false\nVID32=false\nCFLAGS=\"\"\n\n# if mac detected, force no_openmp flags by default\nif [[ $OSTYPE == darwin* ]]; then\n NO_OPENMP=true\nfi\n\n\n## The defaults can be overwritten be editing the configure.deps file\nif [ -f configure.deps ]; then\n# source configure.deps\n # We delete the configure deps and then recreate it each time using\n # the original values along with any modifications made by the\n # configure logic.\n rm configure.deps\nfi\n\n\n# Parse command line configure flags ------------------------------------------\nwhile [ $# -gt 0 ]\n do case $1 in\n --help) print_help=1 ;;\n --cleanup) run_cleanup=1 ;;\n --no_openmp) no_openmp=1 ;;\n --no_mpi) no_mpi=1 ;;\n --no_tcmalloc) no_tcmalloc=1 ;;\n --no_jvm) no_jvm=1 ;;\n --experimental) experimental=1 ;;\n --c++11) cpp11=1 ;;\n --vid32) vid32=1 ;;\n --prefix=*) prefix=${1##--prefix=} ;;\n --ide=*) ide=${1##--ide=} ;;\n -D) CFLAGS=\"$CFLAGS -D $2\"; shift ;;\n *) unknown_option $1 ;;\n esac\n shift\ndone\n\nif [ $print_help ]; then\n print_help;\nfi\n\n\nif [ $run_cleanup ]; then\n run_cleanup\nfi\n\n# Extra generator setting (passed in as an argument)\nif [[ -n $ide ]]; then\n GENERATOR=\"-G $ide\"\nfi\nif [ $no_openmp ]; then\n NO_OPENMP=true\nfi\nif [ $no_mpi ]; then\n NO_MPI=true\nfi\nif [ $no_tcmalloc ]; then\n NO_TCMALLOC=true\nfi\nif [ $experimental ]; then\n EXPERIMENTAL=true\nfi\nif [ $cpp11 ]; then\n CPP11=true\nfi\nif [ $vid32 ]; then\n VID32=true\nfi\n\nif [[ -n $prefix ]]; then\n INSTALL_DIR=$prefix\nfi\n\n\n# If not specified we assume gcc and g++ are the default c and c++\n# compilers\nif [[ -z $CC ]]; then\n CC=gcc\nfi\nif [[ -z $CXX ]]; then\n CXX=g++\nfi\n\n\n# Test java\nif [[ -z $JAVAC ]]; then\n JAVAC=javac\nfi\n# set -e\n# set -o pipefail\n\nif ! $JAVAC -version >> /dev/null; then\n JAVAC=\"\"\nfi\n\nif [[ -n $no_jvm ]]; then\n echo \"JVM disabled\"\n JAVAC=\"\"\nfi\n\n## Begin logging in config.log\nLOG_FILE=config.log\ndate | tee $LOG_FILE\n\n\n\n\n## ===================================================================\n## Setup CMake\n# Automatically detect and install a sufficiently new version of\n# cmake\n\n## Install cmake\nif [ `which cmake` ]; then\n #test cmake version\n echo \"Testing existing cmake version...\"\n currentversion=`cmake --version | awk -F \"patch\" '{print $1;}' | tr -dc '[0-9].'`\n echo \"Detected $currentversion . Required 2.8.3\"\n check_version $currentversion \"2.8.3\"\n if [ $? -ne 2 ]; then\n echo \"CMake version is good\"\n CMAKE=\"cmake\"\n fi\nfi\n\n# CMake not found and there is a cmake in the deps directory!\nif [ -z $CMAKE ] && [ -f $DEPS_PREFIX/bin/cmake ]; then\n #test cmake version\n echo \"Testing existing cmake version...\"\n currentversion=`$DEPS_PREFIX/bin/cmake --version | awk -F \"patch\" '{print $1;}' | tr -dc '[0-9].'`\n echo \"Detected ${currentversion}. Required 2.8.3\"\n check_version $currentversion \"2.8.3\"\n if [ $? -ne 2 ]; then\n echo \"CMake version is good\"\n CMAKE=$DEPS_PREFIX/bin/cmake\n fi\nfi\n\n\nif [ -z $CMAKE ]; then\n echo \"This script will now proceed to download CMake and set it up in\"\n echo \"the local graphlab/deps directory. The GraphLab compilation \"\n echo \"process will be directed to use graphlab/deps/cmake.\"\n pushd .\n mkdir deps\n cd deps\n\n # get the cmake software page\n rm -f software.html\n download_file \"http://www.cmake.org/cmake/resources/software.html\" software.html\n # look for the first tar.gz I can download\n cmakedownload=`grep -m 1 -o -e \"href=\\\"http://www\\\\.cmake.*\\\\.tar\\\\.gz\\\"\" software.html | grep -o -e \"http.*\\\\.tar\\\\.gz\"`\n if [ -z \"$cmakedownload\" ] ; then\n echo \"Unable to locate CMake package. You will have to install it yourself.\"\n exit 1\n fi\n rm -f cmake.tar.gz\n set -e\n download_file $cmakedownload cmake.tar.gz\n tar -xzvf cmake.tar.gz\n # cd into the extracted directory and install\n cd cmake-*\n ./configure --prefix=$DEPS_PREFIX\n make -j2\n make install\n set +e\n popd\n CMAKE=$DEPS_PREFIX/bin/cmake\n echo \"CMAKE=$CMAKE\" >> configure.deps\nfi\n\n## ============================================================================\n# Regenerate the configure.deps\necho -e \"# Release build directory:\" >> configure.deps\necho -e \"\\t RELEASE_DIR=$RELEASE_DIR\" >> configure.deps\n\necho -e \"# Debug build directory (optimization disabled):\" >> configure.deps\necho -e \"\\t DEBUG_DIR=$DEBUG_DIR\" >> configure.deps\n\necho -e \"# Directory in which graphlab is installed (prefix):\" >> configure.deps\necho -e \"\\t INSTALL_DIR=$INSTALL_DIR\" >> configure.deps\n\necho -e \"# Is experimental (research) code enabled:\" >> configure.deps\necho -e \"\\t EXPERIMENTAL=$EXPERIMENTAL\" >> configure.deps\n\necho -e \"# The graphlab home directory: \" >> configure.deps\necho -e \"\\t GRAPHLAB_HOME=$GRAPHLAB_HOME\" >> configure.deps\n\necho -e \"# The directory in which graphlab installs external dependencies:\" >> configure.deps\necho -e \"\\t DEPS_PREFIX=$DEPS_PREFIX\" >> configure.deps\n\necho -e \"# Use OpenMP? This can accelerate some graph building code: \" >> configure.deps\necho -e \"\\t NO_OPENMP=$NO_OPENMP\" >> configure.deps\n\necho -e \"# Use MPI? Without MPI GraphLab cannot run distributed: \" >> configure.deps\necho -e \"\\t NO_MPI=$NO_MPI\" >> configure.deps\n\necho -e \"# Use tcmalloc? Thread-Caching Malloc improves memory allocation: \" >> configure.deps\necho -e \"\\t NO_TCMALLOC=$NO_TCMALLOC\" >> configure.deps\n\necho -e \"# The c compiler to use: \" >> configure.deps\necho -e \"\\t CC=$CC\" >> configure.deps\n\necho -e \"# The c++ compiler to use: \" >> configure.deps\necho -e \"\\t CXX=$CXX\" >> configure.deps\n\necho -e \"# Any addition user defined CFLAGS: \" >> configure.deps\necho -e \"\\t CFLAGS=$CFLAGS\" >> configure.deps\n\necho -e \"# The Java compiler: \" >> configure.deps\necho -e \"\\t JAVAC=$JAVAC\" >> configure.deps\n\necho -e \"# The cmake binary used to geneate the project:\" >> configure.deps\necho -e \"\\t CMAKE=$CMAKE\" >> configure.deps\n\n\n\nmkdir -p deps/local/lib\n\necho \"======================= BUILD CONFIGURATION ========================\"\necho \"System Information: \" | tee -a $LOG_FILE\nuname -v | tee -a $LOG_FILE\necho \"Compiler Information: \" | tee -a $LOG_FILE\n$CC --version | tee -a $LOG_FILE\n$CXX --version | tee -a $LOG_FILE\n$CMAKE --version | tee -a $LOG_FILE\nif [[ -n $JAVAC ]]; then\n $JAVAC -version | tee -a $LOG_FILE\nfi\n\necho \"======================= Config File ================================\"\n\ncat configure.deps | tee -a $LOG_FILE\n\n\n### Add addition config flags =================================================\nCFLAGS=\"$CFLAGS -D NO_OPENMP:BOOL=$NO_OPENMP\"\nCFLAGS=\"$CFLAGS -D NO_MPI:BOOL=$NO_MPI\"\nCFLAGS=\"$CFLAGS -D NO_TCMALLOC:BOOL=$NO_TCMALLOC\"\nCFLAGS=\"$CFLAGS -D CMAKE_INSTALL_PREFIX:STRING=$INSTALL_DIR\"\nCFLAGS=\"$CFLAGS -D EXPERIMENTAL:BOOL=$EXPERIMENTAL\"\nCFLAGS=\"$CFLAGS -D CPP11:BOOL=$CPP11\"\nCFLAGS=\"$CFLAGS -D VID32:BOOL=$VID32\"\nif [ -z $JAVAC ]; then\n CFLAGS=\"$CFLAGS -D NO_JAVAC:BOOL=1\"\nfi\n\n## ============================================================================\n# Run Cmake\n\n\nset -e\nset -o pipefail\n\n\n\necho -e \"\\n\\n\\n======================= Release ========================\" \\\n | tee -a $LOG_FILE\n\nif [ ! -d $RELEASE_DIR ]; then\n mkdir $RELEASE_DIR\nfi\ncd $RELEASE_DIR\nrm -f CMakeCache.txt\nbuild_cmd=\"$CMAKE \\\n $GENERATOR \\\n -D CMAKE_BUILD_TYPE=Release \\\n $CFLAGS \\\n ../.\"\necho $build_cmd | tee -a \"../$LOG_FILE\"\neval $build_cmd | tee -a \"../$LOG_FILE\"\ncd $GRAPHLAB_HOME\n\n\necho -e \"\\n\\n\\n======================= Debug =========================\" \\\n | tee -a $LOG_FILE\n\nif [ ! -d $DEBUG_DIR ]; then\n mkdir $DEBUG_DIR\nfi\ncd $DEBUG_DIR\nrm -f CMakeCache.txt\nbuild_cmd=\"$CMAKE \\\n $GENERATOR \\\n -D CMAKE_BUILD_TYPE=Debug \\\n $CFLAGS \\\n ../.\"\necho $build_cmd | tee -a ../$LOG_FILE\neval $build_cmd | tee -a ../$LOG_FILE\ncd $GRAPHLAB_HOME\n\necho \"*****************************************************\"\necho \"* Important Note *\"\necho \"* -------------- *\"\necho \"* You do not have to build everything! *\"\necho \"* *\"\necho \"* Everything takes a very long time, and a ton of *\"\necho \"* of memory to build. You can just cd into toolkit *\"\necho \"* directory you want, and just build that. For *\"\necho \"* instance, if I just want the release build of *\"\necho \"* PageRank, I could: *\"\necho \"* *\"\necho \"* cd release/toolkits/graph_analytics *\"\necho \"* make -j2 *\"\necho \"* *\"\necho \"* Use at most [RAM in GB] parallel builds. The *\"\necho \"* compilation consumes a lot of RAM. i.e. if you *\"\necho \"* have 4 GB of RAM, do not do more than make -j4. *\"\necho \"*****************************************************\"\n" }, { "path": "cxxtest/cxxtest/Descriptions.cpp", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__Descriptions_cpp__\n#define __cxxtest__Descriptions_cpp__\n\n#include \n\nnamespace CxxTest\n{\n TestDescription::~TestDescription() {}\n SuiteDescription::~SuiteDescription() {}\n WorldDescription::~WorldDescription() {}\n \n //\n // Convert total tests to string\n //\n#ifndef _CXXTEST_FACTOR\n char *WorldDescription::strTotalTests( char *s ) const\n {\n numberToString( numTotalTests(), s );\n return s;\n }\n#else // _CXXTEST_FACTOR\n char *WorldDescription::strTotalTests( char *s ) const\n {\n char *p = numberToString( numTotalTests(), s );\n\n if ( numTotalTests() <= 1 )\n return s;\n\n unsigned n = numTotalTests();\n unsigned numFactors = 0;\n\n for ( unsigned factor = 2; (factor * factor) <= n; factor += (factor == 2) ? 1 : 2 ) {\n unsigned power;\n\n for ( power = 0; (n % factor) == 0; n /= factor )\n ++ power;\n\n if ( !power )\n continue;\n\n p = numberToString( factor, copyString( p, (numFactors == 0) ? \" = \" : \" * \" ) );\n if ( power > 1 )\n p = numberToString( power, copyString( p, \"^\" ) );\n ++ numFactors;\n }\n\n if ( n > 1 ) {\n if ( !numFactors )\n copyString( p, tracker().failedTests() ? \" :(\" : tracker().warnings() ? \" :|\" : \" :)\" );\n else\n numberToString( n, copyString( p, \" * \" ) );\n }\n return s;\n }\n#endif // _CXXTEST_FACTOR\n}\n\n#endif // __cxxtest__Descriptions_cpp__\n" }, { "path": "cxxtest/cxxtest/Descriptions.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__Descriptions_h__\n#define __cxxtest__Descriptions_h__\n\n//\n// TestDescription, SuiteDescription and WorldDescription\n// hold information about tests so they can be run and reported.\n//\n\n#include \n\nnamespace CxxTest \n{\n class TestSuite;\n\n class TestDescription : public Link\n {\n public:\n virtual ~TestDescription();\n \n virtual const char *file() const = 0;\n virtual int line() const = 0;\n virtual const char *testName() const = 0;\n virtual const char *suiteName() const = 0;\n \n virtual void run() = 0;\n virtual bool setUp() = 0;\n virtual bool tearDown() = 0;\n\n virtual const TestDescription *next() const = 0;\n virtual TestDescription *next() = 0; \n };\n\n class SuiteDescription : public Link\n {\n public:\n virtual ~SuiteDescription();\n \n virtual const char *file() const = 0;\n virtual int line() const = 0;\n virtual const char *suiteName() const = 0;\n virtual TestSuite *suite() const = 0;\n \n virtual unsigned numTests() const = 0;\n virtual const TestDescription &testDescription( unsigned /*i*/ ) const = 0;\n\n virtual TestDescription *firstTest() = 0;\n virtual const TestDescription *firstTest() const = 0;\n virtual SuiteDescription *next() = 0;\n virtual const SuiteDescription *next() const = 0;\n\n virtual void activateAllTests() = 0;\n virtual bool leaveOnly( const char * /*testName*/ ) = 0;\n\n virtual bool setUp() = 0;\n virtual bool tearDown() = 0;\n };\n\n class WorldDescription : public Link\n {\n public:\n virtual ~WorldDescription();\n \n virtual const char *worldName() const { return \"cxxtest\"; }\n virtual unsigned numSuites( void ) const = 0;\n virtual unsigned numTotalTests( void ) const = 0;\n virtual const SuiteDescription &suiteDescription( unsigned /*i*/ ) const = 0;\n\n enum { MAX_STRLEN_TOTAL_TESTS = 32 };\n char *strTotalTests( char * /*buffer*/ ) const;\n\n virtual SuiteDescription *firstSuite() = 0;\n virtual const SuiteDescription *firstSuite() const = 0;\n\n virtual void activateAllTests() = 0;\n virtual bool leaveOnly( const char * /*suiteName*/, const char * /*testName*/ = 0 ) = 0;\n };\n}\n\n#endif // __cxxtest__Descriptions_h__\n\n" }, { "path": "cxxtest/cxxtest/DummyDescriptions.cpp", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#include \n\nnamespace CxxTest \n{\n DummyTestDescription::DummyTestDescription() {}\n \n const char *DummyTestDescription::file() const { return \"\"; }\n int DummyTestDescription::line() const { return 0; }\n const char *DummyTestDescription::testName() const { return \"\"; }\n const char *DummyTestDescription::suiteName() const { return \"\"; }\n bool DummyTestDescription::setUp() { return true;}\n void DummyTestDescription::run() {}\n bool DummyTestDescription::tearDown() { return true;}\n\n TestDescription *DummyTestDescription::next() { return 0; }\n const TestDescription *DummyTestDescription::next() const { return 0; }\n \n DummySuiteDescription::DummySuiteDescription() : _test() {}\n \n const char *DummySuiteDescription::file() const { return \"\"; }\n int DummySuiteDescription::line() const { return 0; }\n const char *DummySuiteDescription::suiteName() const { return \"\"; }\n TestSuite *DummySuiteDescription::suite() const { return 0; }\n unsigned DummySuiteDescription::numTests() const { return 0; }\n const TestDescription &DummySuiteDescription::testDescription( unsigned ) const { return _test; }\n SuiteDescription *DummySuiteDescription::next() { return 0; }\n TestDescription *DummySuiteDescription::firstTest() { return 0; }\n const SuiteDescription *DummySuiteDescription::next() const { return 0; }\n const TestDescription *DummySuiteDescription::firstTest() const { return 0; }\n void DummySuiteDescription::activateAllTests() {}\n bool DummySuiteDescription::leaveOnly( const char * /*testName*/ ) { return false; }\n \n bool DummySuiteDescription::setUp() { return true;}\n bool DummySuiteDescription::tearDown() { return true;}\n\n DummyWorldDescription::DummyWorldDescription() : _suite() {}\n \n unsigned DummyWorldDescription::numSuites( void ) const { return 0; }\n unsigned DummyWorldDescription::numTotalTests( void ) const { return 0; }\n const SuiteDescription &DummyWorldDescription::suiteDescription( unsigned ) const { return _suite; }\n SuiteDescription *DummyWorldDescription::firstSuite() { return 0; }\n const SuiteDescription *DummyWorldDescription::firstSuite() const { return 0; }\n void DummyWorldDescription::activateAllTests() {}\n bool DummyWorldDescription::leaveOnly( const char * /*suiteName*/, const char * /*testName*/ ) { return false; }\n \n bool DummyWorldDescription::setUp() { return true;}\n bool DummyWorldDescription::tearDown() { return true;}\n}\n\n" }, { "path": "cxxtest/cxxtest/DummyDescriptions.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__DummyDescriptions_h__\n#define __cxxtest__DummyDescriptions_h__\n\n//\n// DummyTestDescription, DummySuiteDescription and DummyWorldDescription\n//\n\n#include \n\nnamespace CxxTest \n{\n class DummyTestDescription : public TestDescription\n {\n public:\n DummyTestDescription();\n \n const char *file() const;\n int line() const;\n const char *testName() const;\n const char *suiteName() const;\n bool setUp();\n void run();\n bool tearDown();\n\n TestDescription *next();\n const TestDescription *next() const;\n };\n\n class DummySuiteDescription : public SuiteDescription\n { \n public:\n DummySuiteDescription();\n \n const char *file() const;\n int line() const;\n const char *suiteName() const;\n TestSuite *suite() const;\n unsigned numTests() const;\n const TestDescription &testDescription( unsigned ) const;\n SuiteDescription *next();\n TestDescription *firstTest();\n const SuiteDescription *next() const;\n const TestDescription *firstTest() const;\n void activateAllTests();\n bool leaveOnly( const char * /*testName*/ );\n \n bool setUp();\n bool tearDown();\n\n private:\n DummyTestDescription _test;\n };\n\n class DummyWorldDescription : public WorldDescription\n {\n public:\n DummyWorldDescription();\n \n unsigned numSuites( void ) const;\n unsigned numTotalTests( void ) const;\n const SuiteDescription &suiteDescription( unsigned ) const;\n SuiteDescription *firstSuite();\n const SuiteDescription *firstSuite() const;\n void activateAllTests();\n bool leaveOnly( const char * /*suiteName*/, const char * /*testName*/ = 0 );\n \n bool setUp();\n bool tearDown();\n\n private:\n DummySuiteDescription _suite;\n };\n}\n\n#endif // __cxxtest__DummyDescriptions_h__\n\n" }, { "path": "cxxtest/cxxtest/ErrorFormatter.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__ErrorFormatter_h__\n#define __cxxtest__ErrorFormatter_h__\n\n//\n// The ErrorFormatter is a TestListener that\n// prints reports of the errors to an output\n// stream. Since we cannot rely on the standard\n// iostreams, this header defines a base class\n// analogout to std::ostream.\n//\n\n#include \n#include \n#include \n#include \n#include \n\nnamespace CxxTest\n{\n class OutputStream\n {\n public:\n virtual ~OutputStream() {}\n virtual void flush() {};\n virtual OutputStream &operator<<( unsigned /*number*/ ) { return *this; }\n virtual OutputStream &operator<<( const char * /*string*/ ) { return *this; }\n\n typedef void (*Manipulator)( OutputStream & );\n \n virtual OutputStream &operator<<( Manipulator m ) { m( *this ); return *this; }\n static void endl( OutputStream &o ) { (o << \"\\n\").flush(); }\n };\n\n class ErrorFormatter : public TestListener\n {\n public:\n ErrorFormatter( OutputStream *o, const char *preLine = \":\", const char *postLine = \"\" ) :\n _dotting( true ),\n _reported( false ),\n _o(o),\n _preLine(preLine),\n _postLine(postLine)\n {\n }\n\n int run()\n {\n TestRunner::runAllTests( *this );\n return tracker().failedTests();\n }\n\n void enterWorld( const WorldDescription & /*desc*/ )\n {\n (*_o) << \"Running \" << totalTests;\n _o->flush();\n _dotting = true;\n _reported = false;\n }\n\n static void totalTests( OutputStream &o )\n {\n char s[WorldDescription::MAX_STRLEN_TOTAL_TESTS];\n const WorldDescription &wd = tracker().world();\n o << wd.strTotalTests( s ) << (wd.numTotalTests() == 1 ? \" test\" : \" tests\");\n }\n\n void enterSuite( const SuiteDescription & )\n {\n _reported = false;\n }\n\n void enterTest( const TestDescription & )\n {\n _reported = false;\n }\n\n void leaveTest( const TestDescription & )\n {\n if ( !tracker().testFailed() ) {\n (*_o) << \".\";\n _o->flush();\n fflush(stdout);\n _dotting = true;\n }\n }\n\n void leaveWorld( const WorldDescription &desc )\n {\n if ( !tracker().failedTests() ) {\n (*_o) << \"OK!\" << endl;\n return;\n }\n newLine();\n (*_o) << \"Failed \" << tracker().failedTests() << \" of \" << totalTests << endl;\n unsigned numPassed = desc.numTotalTests() - tracker().failedTests();\n (*_o) << \"Success rate: \" << (numPassed * 100 / desc.numTotalTests()) << \"%\" << endl;\n }\n\n void trace( const char *file, int line, const char *expression )\n {\n stop( file, line ) << \"Trace: \" <<\n expression << endl;\n }\n\n void warning( const char *file, int line, const char *expression )\n {\n stop( file, line ) << \"Warning: \" <<\n expression << endl;\n }\n\n void failedTest( const char *file, int line, const char *expression )\n {\n stop( file, line ) << \"Error: Test failed: \" <<\n expression << endl;\n }\n\n void failedAssert( const char *file, int line, const char *expression )\n {\n stop( file, line ) << \"Error: Assertion failed: \" <<\n expression << endl;\n }\n\n void failedAssertEquals( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *x, const char *y )\n {\n stop( file, line ) << \"Error: Expected (\" <<\n xStr << \" == \" << yStr << \"), found (\" <<\n x << \" != \" << y << \")\" << endl;\n }\n\n void failedAssertSameData( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *sizeStr, const void *x,\n const void *y, unsigned size )\n {\n stop( file, line ) << \"Error: Expected \" << sizeStr << \" (\" << size << \") bytes to be equal at (\" <<\n xStr << \") and (\" << yStr << \"), found:\" << endl;\n dump( x, size );\n (*_o) << \" differs from\" << endl;\n dump( y, size );\n }\n\n void failedAssertSameFiles( const char* file, int line,\n const char*, const char*,\n const char* explanation\n )\n {\n stop( file, line ) << \"Error: \" << explanation << endl;\n }\n\n void failedAssertDelta( const char *file, int line,\n const char *xStr, const char *yStr, const char *dStr,\n const char *x, const char *y, const char *d )\n {\n stop( file, line ) << \"Error: Expected (\" <<\n xStr << \" == \" << yStr << \") up to \" << dStr << \" (\" << d << \"), found (\" <<\n x << \" != \" << y << \")\" << endl;\n }\n\n void failedAssertDiffers( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *value )\n {\n stop( file, line ) << \"Error: Expected (\" <<\n xStr << \" != \" << yStr << \"), found (\" <<\n value << \")\" << endl;\n }\n\n void failedAssertLessThan( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *x, const char *y )\n {\n stop( file, line ) << \"Error: Expected (\" <<\n xStr << \" < \" << yStr << \"), found (\" <<\n x << \" >= \" << y << \")\" << endl;\n }\n\n void failedAssertLessThanEquals( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *x, const char *y )\n {\n stop( file, line ) << \"Error: Expected (\" <<\n xStr << \" <= \" << yStr << \"), found (\" <<\n x << \" > \" << y << \")\" << endl;\n }\n\n void failedAssertRelation( const char *file, int line,\n const char *relation, const char *xStr, const char *yStr,\n const char *x, const char *y )\n {\n stop( file, line ) << \"Error: Expected \" << relation << \"( \" <<\n xStr << \", \" << yStr << \" ), found !\" << relation << \"( \" << x << \", \" << y << \" )\" << endl;\n }\n\n void failedAssertPredicate( const char *file, int line,\n const char *predicate, const char *xStr, const char *x )\n {\n stop( file, line ) << \"Error: Expected \" << predicate << \"( \" <<\n xStr << \" ), found !\" << predicate << \"( \" << x << \" )\" << endl;\n }\n\n void failedAssertThrows( const char *file, int line,\n const char *expression, const char *type,\n bool otherThrown )\n {\n stop( file, line ) << \"Error: Expected (\" << expression << \") to throw (\" <<\n type << \") but it \" << (otherThrown ? \"threw something else\" : \"didn't throw\") <<\n endl;\n }\n\n void failedAssertThrowsNot( const char *file, int line, const char *expression )\n {\n stop( file, line ) << \"Error: Expected (\" << expression << \") not to throw, but it did\" <<\n endl;\n }\n\n protected:\n OutputStream *outputStream() const\n {\n return _o;\n }\n\n private:\n ErrorFormatter( const ErrorFormatter & );\n ErrorFormatter &operator=( const ErrorFormatter & );\n \n OutputStream &stop( const char *file, int line )\n {\n newLine();\n reportTest();\n return (*_o) << file << _preLine << line << _postLine << \": \";\n }\n\n void newLine( void )\n {\n if ( _dotting ) {\n (*_o) << endl;\n _dotting = false;\n }\n }\n\n void reportTest( void )\n {\n if( _reported )\n return;\n (*_o) << \"In \" << tracker().suite().suiteName() << \"::\" << tracker().test().testName() << \":\" << endl;\n _reported = true;\n }\n\n void dump( const void *buffer, unsigned size )\n {\n if ( !buffer )\n dumpNull();\n else\n dumpBuffer( buffer, size );\n }\n\n void dumpNull()\n {\n (*_o) << \" (null)\" << endl;\n }\n \n void dumpBuffer( const void *buffer, unsigned size )\n {\n unsigned dumpSize = size;\n if ( maxDumpSize() && dumpSize > maxDumpSize() )\n dumpSize = maxDumpSize();\n\n const unsigned char *p = (const unsigned char *)buffer;\n (*_o) << \" { \";\n for ( unsigned i = 0; i < dumpSize; ++ i )\n (*_o) << byteToHex( *p++ ) << \" \";\n if ( dumpSize < size )\n (*_o) << \"... \";\n (*_o) << \"}\" << endl;\n }\n\n static void endl( OutputStream &o )\n {\n OutputStream::endl( o );\n }\n\n bool _dotting;\n bool _reported;\n OutputStream *_o;\n const char *_preLine;\n const char *_postLine;\n };\n}\n\n#endif // __cxxtest__ErrorFormatter_h__\n\n" }, { "path": "cxxtest/cxxtest/ErrorPrinter.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__ErrorPrinter_h__\n#define __cxxtest__ErrorPrinter_h__\n\n//\n// The ErrorPrinter is a simple TestListener that\n// just prints \"OK\" if everything goes well, otherwise\n// reports the error in the format of compiler messages.\n// The ErrorPrinter uses std::cout\n//\n\n#include \n\n#ifndef _CXXTEST_HAVE_STD\n# define _CXXTEST_HAVE_STD\n#endif // _CXXTEST_HAVE_STD\n\n#include \n#include \n\n#ifdef _CXXTEST_OLD_STD\n# include \n#else // !_CXXTEST_OLD_STD\n# include \n#endif // _CXXTEST_OLD_STD\n\nnamespace CxxTest \n{\n class ErrorPrinter : public ErrorFormatter\n {\n public:\n ErrorPrinter( CXXTEST_STD(ostream) &o = CXXTEST_STD(cout), const char *preLine = \":\", const char *postLine = \"\" ) :\n ErrorFormatter( new Adapter(o), preLine, postLine ) {}\n virtual ~ErrorPrinter() { delete outputStream(); }\n\n private:\n class Adapter : public OutputStream\n {\n CXXTEST_STD(ostream) &_o;\n public:\n Adapter( CXXTEST_STD(ostream) &o ) : _o(o) {}\n void flush() { _o.flush(); }\n OutputStream &operator<<( const char *s ) { _o << s; return *this; }\n OutputStream &operator<<( Manipulator m ) { return OutputStream::operator<<( m ); }\n OutputStream &operator<<( unsigned i )\n {\n char s[1 + 3 * sizeof(unsigned)];\n numberToString( i, s );\n _o << s;\n return *this;\n }\n };\n };\n}\n\n#endif // __cxxtest__ErrorPrinter_h__\n" }, { "path": "cxxtest/cxxtest/Flags.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__Flags_h__\n#define __cxxtest__Flags_h__\n\n//\n// These are the flags that control CxxTest\n//\n\n#if !defined(CXXTEST_FLAGS)\n# define CXXTEST_FLAGS\n#endif // !CXXTEST_FLAGS\n\n#if defined(CXXTEST_HAVE_EH) && !defined(_CXXTEST_HAVE_EH)\n# define _CXXTEST_HAVE_EH\n#endif // CXXTEST_HAVE_EH\n\n#if defined(CXXTEST_HAVE_STD) && !defined(_CXXTEST_HAVE_STD)\n# define _CXXTEST_HAVE_STD\n#endif // CXXTEST_HAVE_STD\n\n#if defined(CXXTEST_OLD_TEMPLATE_SYNTAX) && !defined(_CXXTEST_OLD_TEMPLATE_SYNTAX)\n# define _CXXTEST_OLD_TEMPLATE_SYNTAX\n#endif // CXXTEST_OLD_TEMPLATE_SYNTAX\n\n#if defined(CXXTEST_OLD_STD) && !defined(_CXXTEST_OLD_STD)\n# define _CXXTEST_OLD_STD\n#endif // CXXTEST_OLD_STD\n\n#if defined(CXXTEST_ABORT_TEST_ON_FAIL) && !defined(_CXXTEST_ABORT_TEST_ON_FAIL)\n# define _CXXTEST_ABORT_TEST_ON_FAIL\n#endif // CXXTEST_ABORT_TEST_ON_FAIL\n\n#if defined(CXXTEST_NO_COPY_CONST) && !defined(_CXXTEST_NO_COPY_CONST)\n# define _CXXTEST_NO_COPY_CONST\n#endif // CXXTEST_NO_COPY_CONST\n\n#if defined(CXXTEST_FACTOR) && !defined(_CXXTEST_FACTOR)\n# define _CXXTEST_FACTOR\n#endif // CXXTEST_FACTOR\n\n#if defined(CXXTEST_PARTIAL_TEMPLATE_SPECIALIZATION) && !defined(_CXXTEST_PARTIAL_TEMPLATE_SPECIALIZATION)\n# define _CXXTEST_PARTIAL_TEMPLATE_SPECIALIZATION\n#endif // CXXTEST_PARTIAL_TEMPLATE_SPECIALIZATION\n\n#if defined(CXXTEST_LONGLONG)\n# if defined(_CXXTEST_LONGLONG)\n# undef _CXXTEST_LONGLONG\n# endif\n# define _CXXTEST_LONGLONG CXXTEST_LONGLONG\n#endif // CXXTEST_LONGLONG\n\n#ifndef CXXTEST_MAX_DUMP_SIZE\n# define CXXTEST_MAX_DUMP_SIZE 0\n#endif // CXXTEST_MAX_DUMP_SIZE\n\n#if defined(_CXXTEST_ABORT_TEST_ON_FAIL) && !defined(CXXTEST_DEFAULT_ABORT)\n# define CXXTEST_DEFAULT_ABORT true\n#endif // _CXXTEST_ABORT_TEST_ON_FAIL && !CXXTEST_DEFAULT_ABORT\n\n#if !defined(CXXTEST_DEFAULT_ABORT)\n# define CXXTEST_DEFAULT_ABORT false\n#endif // !CXXTEST_DEFAULT_ABORT\n\n#if defined(_CXXTEST_ABORT_TEST_ON_FAIL) && !defined(_CXXTEST_HAVE_EH)\n# warning \"CXXTEST_ABORT_TEST_ON_FAIL is meaningless without CXXTEST_HAVE_EH\"\n# undef _CXXTEST_ABORT_TEST_ON_FAIL\n#endif // _CXXTEST_ABORT_TEST_ON_FAIL && !_CXXTEST_HAVE_EH\n\n//\n// Some minimal per-compiler configuration to allow us to compile\n//\n\n#ifdef __BORLANDC__\n# if __BORLANDC__ <= 0x520 // Borland C++ 5.2 or earlier\n# ifndef _CXXTEST_OLD_STD\n# define _CXXTEST_OLD_STD\n# endif\n# ifndef _CXXTEST_OLD_TEMPLATE_SYNTAX\n# define _CXXTEST_OLD_TEMPLATE_SYNTAX\n# endif\n# endif\n# if __BORLANDC__ >= 0x540 // C++ Builder 4.0 or later\n# ifndef _CXXTEST_NO_COPY_CONST\n# define _CXXTEST_NO_COPY_CONST\n# endif\n# ifndef _CXXTEST_LONGLONG\n# define _CXXTEST_LONGLONG __int64\n# endif\n# endif\n#endif // __BORLANDC__\n\n#ifdef _MSC_VER // Visual C++\n# ifndef _CXXTEST_LONGLONG\n# define _CXXTEST_LONGLONG __int64\n# endif\n# if (_MSC_VER >= 0x51E)\n# ifndef _CXXTEST_PARTIAL_TEMPLATE_SPECIALIZATION\n# define _CXXTEST_PARTIAL_TEMPLATE_SPECIALIZATION\n# endif\n# endif\n# pragma warning( disable : 4127 )\n# pragma warning( disable : 4290 )\n# pragma warning( disable : 4511 )\n# pragma warning( disable : 4512 )\n# pragma warning( disable : 4514 )\n#endif // _MSC_VER\n\n#ifdef __GNUC__\n# if (__GNUC__ > 2) || (__GNUC__ == 2 && __GNUC_MINOR__ >= 9)\n# ifndef _CXXTEST_PARTIAL_TEMPLATE_SPECIALIZATION\n# define _CXXTEST_PARTIAL_TEMPLATE_SPECIALIZATION\n# endif\n# endif\n# if defined(__LONG_LONG_MAX__) && !defined(__cplusplus)\n# define _CXXTEST_LONGLONG long long\n# endif\n#endif // __GNUC__\n\n#ifdef __DMC__ // Digital Mars\n# ifndef _CXXTEST_OLD_STD\n# define _CXXTEST_OLD_STD\n# endif\n#endif\n\n#ifdef __SUNPRO_CC // Sun Studio C++\n# if __SUNPRO_CC >= 0x510\n# ifndef _CXXTEST_PARTIAL_TEMPLATE_SPECIALIZATION\n# define _CXXTEST_PARTIAL_TEMPLATE_SPECIALIZATION\n# endif\n# endif\n#endif\n\n#ifdef __xlC__ // IBM XL C/C++\n// Partial specialization may be supported before 7.0.0.3, but it is\n// definitely supported after.\n# if __xlC__ >= 0x0700\n# ifndef _CXXTEST_PARTIAL_TEMPLATE_SPECIALIZATION\n# define _CXXTEST_PARTIAL_TEMPLATE_SPECIALIZATION\n# endif\n# endif\n#endif\n\n#endif // __cxxtest__Flags_h__\n\n" }, { "path": "cxxtest/cxxtest/GlobalFixture.cpp", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__GlobalFixture_cpp__\n#define __cxxtest__GlobalFixture_cpp__\n\n#include \n\nnamespace CxxTest \n{\n bool GlobalFixture::setUpWorld() { return true; }\n bool GlobalFixture::tearDownWorld() { return true; }\n bool GlobalFixture::setUp() { return true; }\n bool GlobalFixture::tearDown() { return true; }\n \n GlobalFixture::GlobalFixture() { attach( _list ); }\n GlobalFixture::~GlobalFixture() { detach( _list ); }\n \n GlobalFixture *GlobalFixture::firstGlobalFixture() { return (GlobalFixture *)_list.head(); }\n GlobalFixture *GlobalFixture::lastGlobalFixture() { return (GlobalFixture *)_list.tail(); }\n GlobalFixture *GlobalFixture::nextGlobalFixture() { return (GlobalFixture *)next(); }\n GlobalFixture *GlobalFixture::prevGlobalFixture() { return (GlobalFixture *)prev(); }\n}\n\n#endif // __cxxtest__GlobalFixture_cpp__\n\n" }, { "path": "cxxtest/cxxtest/GlobalFixture.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__GlobalFixture_h__\n#define __cxxtest__GlobalFixture_h__\n\n#include \n\nnamespace CxxTest \n{\n class GlobalFixture : public Link\n {\n public:\n virtual bool setUpWorld();\n virtual bool tearDownWorld();\n virtual bool setUp();\n virtual bool tearDown();\n \n GlobalFixture();\n ~GlobalFixture();\n \n static GlobalFixture *firstGlobalFixture();\n static GlobalFixture *lastGlobalFixture();\n GlobalFixture *nextGlobalFixture();\n GlobalFixture *prevGlobalFixture();\n\n private:\n static List _list;\n };\n}\n\n#endif // __cxxtest__GlobalFixture_h__\n\n" }, { "path": "cxxtest/cxxtest/Gui.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __CXXTEST__GUI_H\n#define __CXXTEST__GUI_H\n\n//\n// GuiListener is a simple base class for the differes GUIs\n// GuiTuiRunner combines a GUI with a text-mode error formatter\n//\n\n#include \n\nnamespace CxxTest\n{\n class GuiListener : public TestListener\n {\n public:\n GuiListener() : _state( GREEN_BAR ) {}\n virtual ~GuiListener() {}\n\n virtual void runGui( int &argc, char **argv, TestListener &listener )\n {\n enterGui( argc, argv );\n TestRunner::runAllTests( listener );\n leaveGui(); \n }\n\n virtual void enterGui( int & /*argc*/, char ** /*argv*/ ) {}\n virtual void leaveGui() {}\n \n //\n // The easy way is to implement these functions:\n // \n virtual void guiEnterWorld( unsigned /*numTotalTests*/ ) {}\n virtual void guiEnterSuite( const char * /*suiteName*/ ) {}\n virtual void guiEnterTest( const char * /*suiteName*/, const char * /*testName*/ ) {}\n virtual void yellowBar() {}\n virtual void redBar() {}\n\n //\n // The hard way is this:\n //\n void enterWorld( const WorldDescription &d ) { guiEnterWorld( d.numTotalTests() ); }\n void enterSuite( const SuiteDescription &d ) { guiEnterSuite( d.suiteName() ); }\n void enterTest( const TestDescription &d ) { guiEnterTest( d.suiteName(), d.testName() ); }\n void leaveTest( const TestDescription & ) {}\n void leaveSuite( const SuiteDescription & ) {}\n void leaveWorld( const WorldDescription & ) {}\n \n void warning( const char * /*file*/, int /*line*/, const char * /*expression*/ )\n {\n yellowBarSafe();\n }\n \n void failedTest( const char * /*file*/, int /*line*/, const char * /*expression*/ )\n {\n redBarSafe();\n }\n \n void failedAssert( const char * /*file*/, int /*line*/, const char * /*expression*/ )\n {\n redBarSafe();\n }\n \n void failedAssertEquals( const char * /*file*/, int /*line*/,\n const char * /*xStr*/, const char * /*yStr*/,\n const char * /*x*/, const char * /*y*/ )\n {\n redBarSafe();\n }\n\n void failedAssertSameData( const char * /*file*/, int /*line*/,\n const char * /*xStr*/, const char * /*yStr*/,\n const char * /*sizeStr*/, const void * /*x*/,\n const void * /*y*/, unsigned /*size*/ )\n {\n redBarSafe();\n }\n \n void failedAssertDelta( const char * /*file*/, int /*line*/,\n const char * /*xStr*/, const char * /*yStr*/, const char * /*dStr*/,\n const char * /*x*/, const char * /*y*/, const char * /*d*/ )\n {\n redBarSafe();\n }\n \n void failedAssertDiffers( const char * /*file*/, int /*line*/,\n const char * /*xStr*/, const char * /*yStr*/,\n const char * /*value*/ )\n {\n redBarSafe();\n }\n \n void failedAssertLessThan( const char * /*file*/, int /*line*/,\n const char * /*xStr*/, const char * /*yStr*/,\n const char * /*x*/, const char * /*y*/ )\n {\n redBarSafe();\n }\n \n void failedAssertLessThanEquals( const char * /*file*/, int /*line*/,\n const char * /*xStr*/, const char * /*yStr*/,\n const char * /*x*/, const char * /*y*/ )\n {\n redBarSafe();\n }\n \n void failedAssertPredicate( const char * /*file*/, int /*line*/,\n const char * /*predicate*/, const char * /*xStr*/, const char * /*x*/ )\n {\n redBarSafe();\n }\n \n void failedAssertRelation( const char * /*file*/, int /*line*/,\n const char * /*relation*/, const char * /*xStr*/, const char * /*yStr*/,\n const char * /*x*/, const char * /*y*/ )\n {\n redBarSafe();\n }\n \n void failedAssertThrows( const char * /*file*/, int /*line*/,\n const char * /*expression*/, const char * /*type*/,\n bool /*otherThrown*/ )\n {\n redBarSafe();\n }\n \n void failedAssertThrowsNot( const char * /*file*/, int /*line*/,\n const char * /*expression*/ )\n {\n redBarSafe();\n }\n\n protected:\n void yellowBarSafe()\n {\n if ( _state < YELLOW_BAR ) {\n yellowBar();\n _state = YELLOW_BAR;\n }\n }\n\n void redBarSafe()\n {\n if ( _state < RED_BAR ) {\n redBar();\n _state = RED_BAR;\n }\n }\n \n private:\n enum { GREEN_BAR, YELLOW_BAR, RED_BAR } _state;\n };\n\n template\n class GuiTuiRunner : public TeeListener\n {\n int* _argc;\n char **_argv;\n GuiT _gui;\n TuiT _tui;\n \n public:\n GuiTuiRunner() : _argc(0), _argv(0) {}\n\n void process_commandline( int& argc, char** argv )\n { \n _argc=&argc;\n _argv=argv;\n setFirst( _gui );\n setSecond( _tui );\n }\n\n int run()\n {\n _gui.runGui( *_argc, _argv, *this );\n return tracker().failedTests();\n }\n };\n}\n\n#endif //__CXXTEST__GUI_H\n\n" }, { "path": "cxxtest/cxxtest/LinkedList.cpp", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__LinkedList_cpp__\n#define __cxxtest__LinkedList_cpp__\n\n#include \n\nnamespace CxxTest\n{\n List GlobalFixture::_list = { 0, 0 };\n List RealSuiteDescription::_suites = { 0, 0 };\n\n void List::initialize()\n {\n _head = _tail = 0;\n }\n \n Link *List::head()\n {\n Link *l = _head;\n while ( l && !l->active() )\n l = l->next();\n return l;\n }\n\n const Link *List::head() const\n {\n Link *l = _head;\n while ( l && !l->active() )\n l = l->next();\n return l;\n }\n\n Link *List::tail()\n {\n Link *l = _tail;\n while ( l && !l->active() )\n l = l->prev();\n return l;\n }\n\n const Link *List::tail() const\n {\n Link *l = _tail;\n while ( l && !l->active() )\n l = l->prev();\n return l;\n }\n\n bool List::empty() const\n {\n return (_head == 0);\n }\n\n unsigned List::size() const\n {\n unsigned count = 0;\n for ( const Link *l = head(); l != 0; l = l->next() )\n ++ count;\n return count;\n }\n\n Link *List::nth( unsigned n )\n {\n Link *l = head();\n while ( n -- )\n l = l->next();\n return l;\n }\n\n void List::activateAll()\n {\n for ( Link *l = _head; l != 0; l = l->justNext() )\n l->setActive( true );\n }\n\n void List::leaveOnly( const Link &link )\n {\n for ( Link *l = head(); l != 0; l = l->next() )\n if ( l != &link )\n l->setActive( false );\n }\n\n Link::Link() :\n _next( 0 ),\n _prev( 0 ),\n _active( true )\n {\n }\n\n Link::~Link()\n {\n }\n \n bool Link::active() const\n {\n return _active;\n }\n \n void Link::setActive( bool value )\n {\n _active = value;\n }\n\n Link * Link::justNext()\n {\n return _next;\n }\n \n Link * Link::justPrev()\n {\n return _prev;\n }\n \n Link * Link::next()\n {\n Link *l = _next;\n while ( l && !l->_active )\n l = l->_next;\n return l;\n }\n \n Link * Link::prev()\n {\n Link *l = _prev;\n while ( l && !l->_active )\n l = l->_prev;\n return l;\n }\n \n const Link * Link::next() const\n {\n Link *l = _next;\n while ( l && !l->_active )\n l = l->_next;\n return l;\n }\n \n const Link * Link::prev() const\n {\n Link *l = _prev;\n while ( l && !l->_active )\n l = l->_prev;\n return l;\n }\n \n void Link::attach( List &l )\n {\n if ( l._tail )\n l._tail->_next = this;\n\n _prev = l._tail;\n _next = 0;\n \n if ( l._head == 0 )\n l._head = this;\n l._tail = this;\n }\n\n void Link::detach( List &l )\n {\n if ( _prev )\n _prev->_next = _next;\n else\n l._head = _next;\n \n if ( _next )\n _next->_prev = _prev;\n else\n l._tail = _prev;\n }\n}\n\n#endif // __cxxtest__LinkedList_cpp__\n" }, { "path": "cxxtest/cxxtest/LinkedList.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__LinkedList_h__\n#define __cxxtest__LinkedList_h__\n\n#include \n\nnamespace CxxTest \n{\n struct List;\n class Link;\n\n struct List\n {\n Link *_head;\n Link *_tail;\n\n void initialize();\n\n Link *head();\n const Link *head() const;\n Link *tail();\n const Link *tail() const;\n\n bool empty() const;\n unsigned size() const;\n Link *nth( unsigned n );\n\n void activateAll();\n void leaveOnly( const Link &link );\n };\n\n class Link\n { \n public:\n Link();\n virtual ~Link();\n\n bool active() const;\n void setActive( bool value = true );\n\n Link *justNext();\n Link *justPrev();\n \n Link *next();\n Link *prev();\n const Link *next() const;\n const Link *prev() const;\n\n void attach( List &l );\n void detach( List &l );\n\n private:\n Link *_next;\n Link *_prev;\n bool _active;\n\n Link( const Link & );\n Link &operator=( const Link & );\n };\n}\n\n#endif // __cxxtest__LinkedList_h__\n\n" }, { "path": "cxxtest/cxxtest/Mock.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__Mock_h__\n#define __cxxtest__Mock_h__\n\nnamespace dummy_mock_ns {}\n\n//\n// The default namespace is T::\n//\n#ifndef CXXTEST_MOCK_NAMESPACE\n# define CXXTEST_MOCK_NAMESPACE T\n#endif // CXXTEST_MOCK_NAMESPACE\n\n//\n// MockTraits: What to return when no mock object has been created\n//\n#define __CXXTEST_MOCK__TRAITS \\\n namespace CXXTEST_MOCK_NAMESPACE \\\n { \\\n template \\\n class MockTraits \\\n { \\\n public: \\\n static T defaultValue() { return 0; } \\\n }; \\\n }\n\n//\n// extern \"C\" when needed\n//\n#ifdef __cplusplus\n# define CXXTEST_EXTERN_C extern \"C\"\n#else\n# define CXXTEST_EXTERN_C\n#endif // __cplusplus\n\n//\n// Prototypes: For \"normal\" headers\n//\n#define __CXXTEST_MOCK__PROTOTYPE( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n namespace CXXTEST_MOCK_NAMESPACE { TYPE NAME ARGS; }\n\n#define __CXXTEST_MOCK_VOID__PROTOTYPE( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK__PROTOTYPE( MOCK, void, NAME, ARGS, REAL, CALL )\n\n#define __CXXTEST_SUPPLY__PROTOTYPE( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n TYPE REAL ARGS;\n\n#define __CXXTEST_SUPPLY_VOID__PROTOTYPE( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_SUPPLY__PROTOTYPE( MOCK, void, NAME, ARGS, REAL, CALL )\n\n//\n// Class declarations: For test files\n//\n#define __CXXTEST_MOCK__CLASS_DECLARATION( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n namespace CXXTEST_MOCK_NAMESPACE { \\\n class Base_##MOCK : public CxxTest::Link \\\n { \\\n public: \\\n Base_##MOCK(); \\\n ~Base_##MOCK(); \\\n bool setUp(); \\\n bool tearDown(); \\\n \\\n static Base_##MOCK ¤t(); \\\n \\\n virtual TYPE NAME ARGS = 0; \\\n \\\n private: \\\n static CxxTest::List _list; \\\n }; \\\n \\\n class Real_##MOCK : public Base_##MOCK \\\n { \\\n public: \\\n TYPE NAME ARGS; \\\n }; \\\n \\\n class _Unimplemented_##MOCK : public Base_##MOCK \\\n { \\\n public: \\\n TYPE NAME ARGS; \\\n }; \\\n }\n\n#define __CXXTEST_MOCK_VOID__CLASS_DECLARATION( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK__CLASS_DECLARATION( MOCK, void, NAME, ARGS, REAL, CALL )\n\n#define __CXXTEST_SUPPLY__CLASS_DECLARATION( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n namespace CXXTEST_MOCK_NAMESPACE { \\\n class Base_##MOCK : public CxxTest::Link \\\n { \\\n public: \\\n Base_##MOCK(); \\\n ~Base_##MOCK(); \\\n bool setUp(); \\\n bool tearDown(); \\\n \\\n static Base_##MOCK ¤t(); \\\n \\\n virtual TYPE NAME ARGS = 0; \\\n \\\n private: \\\n static CxxTest::List _list; \\\n }; \\\n \\\n class _Unimplemented_##MOCK : public Base_##MOCK \\\n { \\\n public: \\\n TYPE NAME ARGS; \\\n }; \\\n }\n\n#define __CXXTEST_SUPPLY_VOID__CLASS_DECLARATION( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_SUPPLY__CLASS_DECLARATION( MOCK, void, NAME, ARGS, REAL, CALL )\n\n//\n// Class implementation: For test source files\n//\n#define __CXXTEST_MOCK__COMMON_CLASS_IMPLEMENTATION( MOCK, NAME ) \\\n namespace CXXTEST_MOCK_NAMESPACE { \\\n \\\n CxxTest::List Base_##MOCK::_list = { 0, 0 }; \\\n \\\n Base_##MOCK::Base_##MOCK() { attach( _list ); } \\\n Base_##MOCK::~Base_##MOCK() { detach( _list ); } \\\n bool Base_##MOCK::setUp() { return true; } \\\n bool Base_##MOCK::tearDown() { return true; } \\\n \\\n Base_##MOCK &Base_##MOCK::current() \\\n { \\\n if ( _list.empty() ) \\\n static _Unimplemented_##MOCK unimplemented; \\\n return *(Base_##MOCK *)_list.tail(); \\\n } \\\n }\n\n#define __CXXTEST_MOCK__CLASS_IMPLEMENTATION( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK__COMMON_CLASS_IMPLEMENTATION( MOCK, NAME ) \\\n namespace CXXTEST_MOCK_NAMESPACE { \\\n TYPE Real_##MOCK::NAME ARGS \\\n { \\\n return REAL CALL; \\\n } \\\n \\\n TYPE _Unimplemented_##MOCK::NAME ARGS \\\n { \\\n while ( false ) \\\n return NAME CALL; \\\n __CXXTEST_MOCK_UNIMPLEMENTED( NAME, ARGS ); \\\n return MockTraits::defaultValue(); \\\n } \\\n \\\n TYPE NAME ARGS \\\n { \\\n return Base_##MOCK::current().NAME CALL; \\\n } \\\n }\n\n#define __CXXTEST_MOCK_VOID__CLASS_IMPLEMENTATION( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK__COMMON_CLASS_IMPLEMENTATION( MOCK, NAME ) \\\n namespace CXXTEST_MOCK_NAMESPACE { \\\n void Real_##MOCK::NAME ARGS \\\n { \\\n REAL CALL; \\\n } \\\n \\\n void _Unimplemented_##MOCK::NAME ARGS \\\n { \\\n while ( false ) \\\n NAME CALL; \\\n __CXXTEST_MOCK_UNIMPLEMENTED( NAME, ARGS ); \\\n } \\\n \\\n void NAME ARGS \\\n { \\\n Base_##MOCK::current().NAME CALL; \\\n } \\\n }\n\n#define __CXXTEST_SUPPLY__CLASS_IMPLEMENTATION( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK__COMMON_CLASS_IMPLEMENTATION( MOCK, NAME ) \\\n namespace CXXTEST_MOCK_NAMESPACE { \\\n TYPE _Unimplemented_##MOCK::NAME ARGS \\\n { \\\n while ( false ) \\\n return NAME CALL; \\\n __CXXTEST_MOCK_UNIMPLEMENTED( NAME, ARGS ); \\\n return MockTraits::defaultValue(); \\\n } \\\n } \\\n \\\n TYPE REAL ARGS \\\n { \\\n return CXXTEST_MOCK_NAMESPACE::Base_##MOCK::current().NAME CALL; \\\n }\n\n#define __CXXTEST_SUPPLY_VOID__CLASS_IMPLEMENTATION( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK__COMMON_CLASS_IMPLEMENTATION( MOCK, NAME ) \\\n namespace CXXTEST_MOCK_NAMESPACE { \\\n void _Unimplemented_##MOCK::NAME ARGS \\\n { \\\n while ( false ) \\\n NAME CALL; \\\n __CXXTEST_MOCK_UNIMPLEMENTED( NAME, ARGS ); \\\n } \\\n } \\\n \\\n void REAL ARGS \\\n { \\\n CXXTEST_MOCK_NAMESPACE::Base_##MOCK::current().NAME CALL; \\\n } \\\n\n//\n// Error for calling mock function w/o object\n//\n#define __CXXTEST_MOCK_UNIMPLEMENTED( NAME, ARGS ) \\\n TS_FAIL( CXXTEST_MOCK_NAMESPACE_STR #NAME #ARGS \" called with no \" \\\n CXXTEST_MOCK_NAMESPACE_STR \"Base_\" #NAME \" object\" ); \\\n\n#define CXXTEST_MOCK_NAMESPACE_STR __CXXTEST_STR(CXXTEST_MOCK_NAMESPACE) \"::\"\n#define __CXXTEST_STR(X) __CXXTEST_XSTR(X)\n#define __CXXTEST_XSTR(X) #X\n\n#if defined(CXXTEST_MOCK_TEST_SOURCE_FILE)\n//\n// Test source file: Prototypes, class declarations and implementation\n//\n#include \n\n__CXXTEST_MOCK__TRAITS\n\n#define CXXTEST_MOCK( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK__PROTOTYPE( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK__CLASS_DECLARATION( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK__CLASS_IMPLEMENTATION( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n using namespace dummy_mock_ns\n\n#define CXXTEST_MOCK_VOID( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK_VOID__PROTOTYPE( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK_VOID__CLASS_DECLARATION( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK_VOID__CLASS_IMPLEMENTATION( MOCK, NAME, ARGS, REAL, CALL ) \\\n using namespace dummy_mock_ns\n\n#define CXXTEST_SUPPLY( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_SUPPLY__PROTOTYPE( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_SUPPLY__CLASS_DECLARATION( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_SUPPLY__CLASS_IMPLEMENTATION( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n using namespace dummy_mock_ns\n\n#define CXXTEST_SUPPLY_VOID( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_SUPPLY_VOID__PROTOTYPE( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_SUPPLY_VOID__CLASS_DECLARATION( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_SUPPLY_VOID__CLASS_IMPLEMENTATION( MOCK, NAME, ARGS, REAL, CALL ) \\\n using namespace dummy_mock_ns\n\n#elif defined(CXXTEST_FLAGS) || defined(CXXTEST_RUNNING)\n//\n// Test file other than source: Prototypes and class declarations\n//\n#include \n\n__CXXTEST_MOCK__TRAITS;\n\n#define CXXTEST_MOCK( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK__PROTOTYPE( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK__CLASS_DECLARATION( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n using namespace dummy_mock_ns\n\n#define CXXTEST_MOCK_VOID( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK_VOID__PROTOTYPE( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK_VOID__CLASS_DECLARATION( MOCK, NAME, ARGS, REAL, CALL ) \\\n using namespace dummy_mock_ns\n\n#define CXXTEST_SUPPLY( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_SUPPLY__PROTOTYPE( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_SUPPLY__CLASS_DECLARATION( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n using namespace dummy_mock_ns\n\n#define CXXTEST_SUPPLY_VOID( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_SUPPLY_VOID__PROTOTYPE( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_SUPPLY_VOID__CLASS_DECLARATION( MOCK, NAME, ARGS, REAL, CALL ) \\\n using namespace dummy_mock_ns\n\n#elif defined(CXXTEST_MOCK_REAL_SOURCE_FILE)\n//\n// Real source file: \"Real\" implementations\n//\n#define CXXTEST_MOCK( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n namespace CXXTEST_MOCK_NAMESPACE { TYPE NAME ARGS { return REAL CALL; } } using namespace dummy_mock_ns\n\n#define CXXTEST_MOCK_VOID( MOCK, NAME, ARGS, REAL, CALL ) \\\n namespace CXXTEST_MOCK_NAMESPACE { void NAME ARGS { REAL CALL; } } using namespace dummy_mock_ns\n\n#else\n//\n// Ordinary header file: Just prototypes\n//\n\n#define CXXTEST_MOCK( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK__PROTOTYPE( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n using namespace dummy_mock_ns\n\n#define CXXTEST_MOCK_VOID( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_MOCK_VOID__PROTOTYPE( MOCK, NAME, ARGS, REAL, CALL ) \\\n using namespace dummy_mock_ns\n\n#define CXXTEST_SUPPLY( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_SUPPLY__PROTOTYPE( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n using namespace dummy_mock_ns\n\n#define CXXTEST_SUPPLY_VOID( MOCK, NAME, ARGS, REAL, CALL ) \\\n __CXXTEST_SUPPLY_VOID__PROTOTYPE( MOCK, NAME, ARGS, REAL, CALL ) \\\n using namespace dummy_mock_ns\n\n#endif // Ordinary header file\n\n//\n// How to supply extern \"C\" functions\n//\n#define CXXTEST_SUPPLY_C( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n CXXTEST_EXTERN_C __CXXTEST_SUPPLY__PROTOTYPE( MOCK, TYPE, NAME, ARGS, REAL, CALL ) \\\n CXXTEST_SUPPLY( MOCK, TYPE, NAME, ARGS, REAL, CALL )\n\n#define CXXTEST_SUPPLY_VOID_C( MOCK, NAME, ARGS, REAL, CALL ) \\\n CXXTEST_EXTERN_C __CXXTEST_SUPPLY_VOID__PROTOTYPE( MOCK, NAME, ARGS, REAL, CALL ) \\\n CXXTEST_SUPPLY_VOID( MOCK, NAME, ARGS, REAL, CALL )\n\n//\n// Usually we mean the global namespace\n//\n#define CXXTEST_MOCK_GLOBAL( TYPE, NAME, ARGS, CALL ) \\\n CXXTEST_MOCK( NAME, TYPE, NAME, ARGS, ::NAME, CALL )\n\n#define CXXTEST_MOCK_VOID_GLOBAL( NAME, ARGS, CALL ) \\\n CXXTEST_MOCK_VOID( NAME, NAME, ARGS, ::NAME, CALL )\n\n#define CXXTEST_SUPPLY_GLOBAL( TYPE, NAME, ARGS, CALL ) \\\n CXXTEST_SUPPLY( NAME, TYPE, NAME, ARGS, NAME, CALL )\n\n#define CXXTEST_SUPPLY_VOID_GLOBAL( NAME, ARGS, CALL ) \\\n CXXTEST_SUPPLY_VOID( NAME, NAME, ARGS, NAME, CALL )\n\n#define CXXTEST_SUPPLY_GLOBAL_C( TYPE, NAME, ARGS, CALL ) \\\n CXXTEST_SUPPLY_C( NAME, TYPE, NAME, ARGS, NAME, CALL )\n\n#define CXXTEST_SUPPLY_VOID_GLOBAL_C( NAME, ARGS, CALL ) \\\n CXXTEST_SUPPLY_VOID_C( NAME, NAME, ARGS, NAME, CALL )\n\n//\n// What to return when no mock object has been created.\n// The default value of 0 usually works, but some cases may need this.\n//\n#define CXXTEST_MOCK_DEFAULT_VALUE( TYPE, VALUE ) \\\n namespace CXXTEST_MOCK_NAMESPACE \\\n { \\\n template<> \\\n class MockTraits \\\n { \\\n public: \\\n static TYPE defaultValue() { return VALUE; } \\\n }; \\\n } using namespace dummy_mock_ns\n\n#endif // __cxxtest__Mock_h__\n" }, { "path": "cxxtest/cxxtest/ParenPrinter.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__ParenPrinter_h__\n#define __cxxtest__ParenPrinter_h__\n\n//\n// The ParenPrinter is identical to the ErrorPrinter, except it\n// prints the line number in a format expected by some compilers\n// (notably, MSVC).\n//\n\n#include \n\nnamespace CxxTest \n{\n class ParenPrinter : public ErrorPrinter\n {\n public:\n ParenPrinter( CXXTEST_STD(ostream) &o = CXXTEST_STD(cout) ) : ErrorPrinter( o, \"(\", \")\" ) {}\n };\n}\n\n#endif // __cxxtest__ParenPrinter_h__\n" }, { "path": "cxxtest/cxxtest/QtGui.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__QtGui_h__\n#define __cxxtest__QtGui_h__\n\n//\n// The QtGui displays a simple progress bar using the Qt Toolkit. It\n// has been tested with versions 2.x and 3.x.\n// \n// Apart from normal Qt command-line arguments, it accepts the following options:\n// -minimized Start minimized, pop up on error\n// -keep Don't close the window at the end\n// -title TITLE Set the window caption\n//\n// If both are -minimized and -keep specified, GUI will only keep the\n// window if it's in focus.\n//\n\n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\nnamespace CxxTest\n{\n class QtGui : public GuiListener\n {\n public:\n void enterGui( int &argc, char **argv )\n {\n parseCommandLine( argc, argv );\n createApplication( argc, argv );\n }\n\n void enterWorld( const WorldDescription &wd )\n {\n createWindow( wd );\n processEvents();\n }\n\n void guiEnterSuite( const char *suiteName )\n {\n showSuiteName( suiteName );\n }\n\n void guiEnterTest( const char *suiteName, const char *testName )\n {\n setCaption( suiteName, testName );\n advanceProgressBar();\n showTestName( testName );\n showTestsDone( _progressBar->progress() );\n processEvents();\n }\n\n void yellowBar()\n {\n setColor( 255, 255, 0 );\n setIcon( QMessageBox::Warning );\n getTotalTests();\n processEvents();\n }\n \n void redBar()\n {\n if ( _startMinimized && _mainWindow->isMinimized() )\n showNormal();\n setColor( 255, 0, 0 );\n setIcon( QMessageBox::Critical );\n getTotalTests();\n processEvents();\n }\n\n void leaveGui()\n {\n if ( keep() ) {\n showSummary();\n _application->exec();\n }\n else\n _mainWindow->close( true );\n }\n\n private:\n QString _title;\n bool _startMinimized, _keep;\n unsigned _numTotalTests;\n QString _strTotalTests;\n QApplication *_application;\n QWidget *_mainWindow;\n QVBoxLayout *_layout;\n QProgressBar *_progressBar;\n QStatusBar *_statusBar;\n QLabel *_suiteName, *_testName, *_testsDone;\n\n void parseCommandLine( int argc, char **argv )\n {\n _startMinimized = _keep = false;\n _title = argv[0];\n \n for ( int i = 1; i < argc; ++ i ) {\n QString arg( argv[i] );\n if ( arg == \"-minimized\" )\n _startMinimized = true;\n else if ( arg == \"-keep\" )\n _keep = true;\n else if ( arg == \"-title\" && (i + 1 < argc) )\n _title = argv[++i];\n }\n }\n\n void createApplication( int &argc, char **argv )\n {\n _application = new QApplication( argc, argv );\n } \n \n void createWindow( const WorldDescription &wd )\n {\n getTotalTests( wd ); \n createMainWindow();\n createProgressBar();\n createStatusBar();\n setMainWidget();\n if ( _startMinimized )\n showMinimized();\n else\n showNormal();\n }\n\n void getTotalTests()\n {\n getTotalTests( tracker().world() );\n }\n\n void getTotalTests( const WorldDescription &wd )\n {\n _numTotalTests = wd.numTotalTests();\n char s[WorldDescription::MAX_STRLEN_TOTAL_TESTS];\n _strTotalTests = wd.strTotalTests( s );\n }\n\n void createMainWindow()\n {\n _mainWindow = new QWidget();\n _layout = new QVBoxLayout( _mainWindow );\n }\n\n void createProgressBar()\n {\n _layout->addWidget( _progressBar = new QProgressBar( _numTotalTests, _mainWindow ) );\n _progressBar->setProgress( 0 );\n setColor( 0, 255, 0 );\n setIcon( QMessageBox::Information );\n }\n\n void createStatusBar()\n {\n _layout->addWidget( _statusBar = new QStatusBar( _mainWindow ) );\n _statusBar->addWidget( _suiteName = new QLabel( _statusBar ), 2 );\n _statusBar->addWidget( _testName = new QLabel( _statusBar ), 4 );\n _statusBar->addWidget( _testsDone = new QLabel( _statusBar ), 1 );\n }\n\n void setMainWidget()\n {\n _application->setMainWidget( _mainWindow );\n }\n\n void showMinimized()\n {\n _mainWindow->showMinimized();\n }\n\n void showNormal()\n {\n _mainWindow->showNormal();\n centerWindow();\n }\n\n void setCaption( const QString &suiteName, const QString &testName )\n {\n _mainWindow->setCaption( _title + \" - \" + suiteName + \"::\" + testName + \"()\" );\n }\n\n void showSuiteName( const QString &suiteName )\n {\n _suiteName->setText( \"class \" + suiteName );\n }\n\n void advanceProgressBar()\n {\n _progressBar->setProgress( _progressBar->progress() + 1 );\n }\n\n void showTestName( const QString &testName )\n {\n _testName->setText( testName + \"()\" );\n }\n\n void showTestsDone( unsigned testsDone )\n {\n _testsDone->setText( asString( testsDone ) + \" of \" + _strTotalTests );\n }\n\n static QString asString( unsigned n )\n {\n return QString::number( n );\n }\n\n void setColor( int r, int g, int b )\n {\n QPalette palette = _progressBar->palette();\n palette.setColor( QColorGroup::Highlight, QColor( r, g, b ) );\n _progressBar->setPalette( palette );\n }\n\n void setIcon( QMessageBox::Icon icon )\n {\n#if QT_VERSION >= 0x030000\n _mainWindow->setIcon( QMessageBox::standardIcon( icon ) );\n#else // Qt version < 3.0.0\n _mainWindow->setIcon( QMessageBox::standardIcon( icon, QApplication::style().guiStyle() ) );\n#endif // QT_VERSION\n }\n\n void processEvents()\n {\n _application->processEvents();\n }\n\n void centerWindow()\n {\n QWidget *desktop = QApplication::desktop();\n int xCenter = desktop->x() + (desktop->width() / 2);\n int yCenter = desktop->y() + (desktop->height() / 2);\n \n int windowWidth = (desktop->width() * 4) / 5;\n int windowHeight = _mainWindow->height();\n _mainWindow->setGeometry( xCenter - (windowWidth / 2), yCenter - (windowHeight / 2), windowWidth, windowHeight );\n }\n\n bool keep()\n {\n if ( !_keep )\n return false;\n if ( !_startMinimized )\n return true;\n return (_mainWindow == _application->activeWindow());\n }\n\n void showSummary()\n {\n QString summary = _strTotalTests + (_numTotalTests == 1 ? \" test\" : \" tests\");\n if ( tracker().failedTests() )\n summary = \"Failed \" + asString( tracker().failedTests() ) + \" of \" + summary;\n else\n summary = summary + \" passed\";\n\n _mainWindow->setCaption( _title + \" - \" + summary );\n\n _statusBar->removeWidget( _suiteName );\n _statusBar->removeWidget( _testName );\n _testsDone->setText( summary );\n }\n };\n}\n\n#endif // __cxxtest__QtGui_h__\n" }, { "path": "cxxtest/cxxtest/RealDescriptions.cpp", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__RealDescriptions_cpp__\n#define __cxxtest__RealDescriptions_cpp__\n\n//\n// NOTE: If an error occur during world construction/deletion, CxxTest cannot\n// know where the error originated.\n//\n\n#include \n\nnamespace CxxTest \n{\n RealTestDescription::RealTestDescription()\n {\n }\n \n RealTestDescription::RealTestDescription( List &argList,\n SuiteDescription &argSuite,\n unsigned argLine,\n const char *argTestName )\n {\n initialize( argList, argSuite, argLine, argTestName );\n }\n\n void RealTestDescription::initialize( List &argList,\n SuiteDescription &argSuite,\n unsigned argLine,\n const char *argTestName )\n {\n _suite = &argSuite;\n _line = argLine;\n _testName = argTestName;\n attach( argList );\n }\n \n bool RealTestDescription::setUp()\n {\n if ( !suite() )\n return false;\n\n for ( GlobalFixture *gf = GlobalFixture::firstGlobalFixture(); gf != 0; gf = gf->nextGlobalFixture() ) {\n bool ok;\n _TS_TRY { ok = gf->setUp(); }\n _TS_LAST_CATCH( { ok = false; } );\n\n if ( !ok ) {\n doFailTest( file(), line(), \"Error in GlobalFixture::setUp()\" );\n return false;\n }\n }\n\n _TS_TRY {\n bool ok = false;\n _TSM_ASSERT_THROWS_NOTHING( file(), line(), \"Exception thrown from setUp()\", suite()->setUp(); ok=true );\n if (ok == false) return ok;\n }\n _TS_CATCH_ABORT( { return false; } );\n\n return true;\n }\n\n bool RealTestDescription::tearDown()\n {\n if ( !suite() )\n return false;\n\n _TS_TRY {\n _TSM_ASSERT_THROWS_NOTHING( file(), line(), \"Exception thrown from tearDown()\", suite()->tearDown() );\n }\n _TS_CATCH_ABORT( { return false; } );\n\n for ( GlobalFixture *gf = GlobalFixture::lastGlobalFixture(); gf != 0; gf = gf->prevGlobalFixture() ) {\n bool ok;\n _TS_TRY { ok = gf->tearDown(); }\n _TS_LAST_CATCH( { ok = false; } );\n\n if ( !ok ) {\n doFailTest( file(), line(), \"Error in GlobalFixture::tearDown()\" );\n return false;\n }\n }\n\n return true;\n }\n\n const char *RealTestDescription::file() const { return _suite->file(); }\n int RealTestDescription::line() const { return _line; }\n const char *RealTestDescription::testName() const { return _testName; }\n const char *RealTestDescription::suiteName() const { return _suite->suiteName(); }\n\n TestDescription *RealTestDescription::next() { return (RealTestDescription *)Link::next(); }\n const TestDescription *RealTestDescription::next() const { return (const RealTestDescription *)Link::next(); }\n\n TestSuite *RealTestDescription::suite() const { return _suite->suite(); }\n\n void RealTestDescription::run()\n {\n _TS_TRY { runTest(); }\n _TS_CATCH_ABORT( {} )\n ___TSM_CATCH( file(), line(), \"Exception thrown from test\" );\n }\n \n RealSuiteDescription::RealSuiteDescription() {}\n RealSuiteDescription::RealSuiteDescription( const char *argFile,\n unsigned argLine,\n const char *argSuiteName,\n List &argTests )\n {\n initialize( argFile, argLine, argSuiteName, argTests );\n }\n\n void RealSuiteDescription::initialize( const char *argFile,\n unsigned argLine,\n const char *argSuiteName,\n List &argTests )\n {\n _file = argFile;\n _line = argLine;\n _suiteName = argSuiteName;\n _tests = &argTests;\n \n attach( _suites );\n }\n\n const char *RealSuiteDescription::file() const { return _file; }\n int RealSuiteDescription::line() const { return _line; }\n const char *RealSuiteDescription::suiteName() const { return _suiteName; }\n\n TestDescription *RealSuiteDescription::firstTest() { return (RealTestDescription *)_tests->head(); }\n const TestDescription *RealSuiteDescription::firstTest() const { return (const RealTestDescription *)_tests->head(); }\n SuiteDescription *RealSuiteDescription::next() { return (RealSuiteDescription *)Link::next(); }\n const SuiteDescription *RealSuiteDescription::next() const { return (const RealSuiteDescription *)Link::next(); }\n \n unsigned RealSuiteDescription::numTests() const { return _tests->size(); }\n \n const TestDescription &RealSuiteDescription::testDescription( unsigned i ) const\n {\n return *(RealTestDescription *)_tests->nth( i );\n }\n\n void RealSuiteDescription::activateAllTests()\n {\n _tests->activateAll();\n }\n \n bool RealSuiteDescription::leaveOnly( const char *testName )\n {\n for ( TestDescription *td = firstTest(); td != 0; td = td->next() ) {\n if ( stringsEqual( td->testName(), testName ) ) {\n _tests->leaveOnly( *td );\n return true;\n }\n }\n return false; \n }\n \n StaticSuiteDescription::StaticSuiteDescription() {}\n StaticSuiteDescription::StaticSuiteDescription( const char *argFile, unsigned argLine,\n const char *argSuiteName, TestSuite &argSuite,\n List &argTests ) :\n RealSuiteDescription( argFile, argLine, argSuiteName, argTests )\n {\n doInitialize( argSuite );\n }\n\n void StaticSuiteDescription::initialize( const char *argFile, unsigned argLine,\n const char *argSuiteName, TestSuite &argSuite,\n List &argTests )\n {\n RealSuiteDescription::initialize( argFile, argLine, argSuiteName, argTests );\n doInitialize( argSuite );\n }\n \n void StaticSuiteDescription::doInitialize( TestSuite &argSuite )\n {\n _suite = &argSuite;\n }\n\n TestSuite *StaticSuiteDescription::suite() const\n {\n return _suite;\n }\n\n bool StaticSuiteDescription::setUp() { return true; }\n bool StaticSuiteDescription::tearDown() { return true; }\n\n CommonDynamicSuiteDescription::CommonDynamicSuiteDescription() {}\n CommonDynamicSuiteDescription::CommonDynamicSuiteDescription( const char *argFile, unsigned argLine,\n const char *argSuiteName, List &argTests,\n unsigned argCreateLine, unsigned argDestroyLine ) :\n RealSuiteDescription( argFile, argLine, argSuiteName, argTests )\n {\n doInitialize( argCreateLine, argDestroyLine );\n }\n\n void CommonDynamicSuiteDescription::initialize( const char *argFile, unsigned argLine,\n const char *argSuiteName, List &argTests,\n unsigned argCreateLine, unsigned argDestroyLine )\n {\n RealSuiteDescription::initialize( argFile, argLine, argSuiteName, argTests );\n doInitialize( argCreateLine, argDestroyLine );\n }\n\n void CommonDynamicSuiteDescription::doInitialize( unsigned argCreateLine, unsigned argDestroyLine )\n {\n _createLine = argCreateLine;\n _destroyLine = argDestroyLine;\n }\n \n List &RealWorldDescription::suites()\n {\n return RealSuiteDescription::_suites;\n }\n \n unsigned RealWorldDescription::numSuites( void ) const\n {\n return suites().size();\n }\n \n unsigned RealWorldDescription::numTotalTests( void ) const\n {\n unsigned count = 0;\n for ( const SuiteDescription *sd = firstSuite(); sd != 0; sd = sd->next() )\n count += sd->numTests();\n return count;\n }\n \n SuiteDescription *RealWorldDescription::firstSuite()\n {\n return (RealSuiteDescription *)suites().head();\n }\n\n const SuiteDescription *RealWorldDescription::firstSuite() const\n {\n return (const RealSuiteDescription *)suites().head();\n }\n\n const SuiteDescription &RealWorldDescription::suiteDescription( unsigned i ) const\n {\n return *(const RealSuiteDescription *)suites().nth( i );\n }\n\n void RealWorldDescription::activateAllTests()\n {\n suites().activateAll();\n for ( SuiteDescription *sd = firstSuite(); sd != 0; sd = sd->next() )\n sd->activateAllTests();\n }\n\n bool RealWorldDescription::leaveOnly( const char *suiteName, const char *testName )\n {\n for ( SuiteDescription *sd = firstSuite(); sd != 0; sd = sd->next() ) {\n if ( stringsEqual( sd->suiteName(), suiteName ) ) {\n if ( testName )\n if ( !sd->leaveOnly( testName ) )\n return false;\n suites().leaveOnly( *sd );\n return true;\n }\n }\n return false;\n }\n \n bool RealWorldDescription::setUp()\n {\n for ( GlobalFixture *gf = GlobalFixture::firstGlobalFixture(); gf != 0; gf = gf->nextGlobalFixture() ) {\n bool ok;\n _TS_TRY { \n ok = gf->setUpWorld(); \n if (tracker().testFailed()) {\n tracker().initialize();\n ok = false;\n }\n }\n _TS_LAST_CATCH( { ok = false; } );\n\n if ( !ok ) {\n reportError( \"Error setting up world\" );\n return false;\n }\n }\n\n return true;\n }\n\n bool RealWorldDescription::tearDown()\n {\n for ( GlobalFixture *gf = GlobalFixture::lastGlobalFixture(); gf != 0; gf = gf->prevGlobalFixture() ) {\n bool ok;\n _TS_TRY { ok = gf->tearDownWorld(); }\n _TS_LAST_CATCH( { ok = false; } );\n\n if ( !ok ) {\n reportError( \"Error tearing down world\" );\n return false;\n }\n }\n\n return true;\n }\n\n void RealWorldDescription::reportError( const char *message )\n {\n doWarn( __FILE__, 5, message );\n }\n\n void activateAllTests()\n {\n RealWorldDescription().activateAllTests();\n }\n\n bool leaveOnly( const char *suiteName, const char *testName )\n {\n return RealWorldDescription().leaveOnly( suiteName, testName );\n }\n}\n\n#endif // __cxxtest__RealDescriptions_cpp__\n\n" }, { "path": "cxxtest/cxxtest/RealDescriptions.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__RealDescriptions_h__\n#define __cxxtest__RealDescriptions_h__\n\n//\n// The \"real\" description classes\n//\n\n#include \n#include \n#include \n\nnamespace CxxTest \n{\n class RealTestDescription : public TestDescription\n {\n public:\n RealTestDescription();\n RealTestDescription( List &argList, SuiteDescription &argSuite, unsigned argLine, const char *argTestName );\n void initialize( List &argList, SuiteDescription &argSuite, unsigned argLine, const char *argTestName );\n \n const char *file() const;\n int line() const;\n const char *testName() const;\n const char *suiteName() const;\n\n TestDescription *next();\n const TestDescription *next() const;\n\n TestSuite *suite() const;\n\n bool setUp();\n void run(); \n bool tearDown();\n \n private:\n RealTestDescription( const RealTestDescription & );\n RealTestDescription &operator=( const RealTestDescription & );\n\n virtual void runTest() = 0;\n \n SuiteDescription *_suite;\n int _line;\n const char *_testName;\n };\n\n class RealSuiteDescription : public SuiteDescription\n {\n public:\n RealSuiteDescription();\n RealSuiteDescription( const char *argFile, unsigned argLine, const char *argSuiteName, List &argTests );\n \n void initialize( const char *argFile, unsigned argLine, const char *argSuiteName, List &argTests );\n\n const char *file() const;\n int line() const;\n const char *suiteName() const;\n\n TestDescription *firstTest();\n const TestDescription *firstTest() const;\n SuiteDescription *next();\n const SuiteDescription *next() const;\n \n unsigned numTests() const;\n const TestDescription &testDescription( unsigned i ) const;\n\n void activateAllTests();\n bool leaveOnly( const char *testName );\n \n private:\n RealSuiteDescription( const RealSuiteDescription & );\n RealSuiteDescription &operator=( const RealSuiteDescription & );\n \n const char *_file;\n int _line;\n const char *_suiteName;\n List *_tests;\n\n static List _suites;\n friend class RealWorldDescription;\n };\n\n class StaticSuiteDescription : public RealSuiteDescription\n {\n public:\n StaticSuiteDescription();\n StaticSuiteDescription( const char *argFile, unsigned argLine,\n const char *argSuiteName, TestSuite &argSuite,\n List &argTests );\n\n void initialize( const char *argFile, unsigned argLine,\n const char *argSuiteName, TestSuite &argSuite,\n List &argTests );\n TestSuite *suite() const;\n \n bool setUp();\n bool tearDown();\n \n private:\n StaticSuiteDescription( const StaticSuiteDescription & );\n StaticSuiteDescription &operator=( const StaticSuiteDescription & );\n\n void doInitialize( TestSuite &argSuite );\n \n TestSuite *_suite;\n };\n\n class CommonDynamicSuiteDescription : public RealSuiteDescription\n {\n public:\n CommonDynamicSuiteDescription();\n CommonDynamicSuiteDescription( const char *argFile, unsigned argLine,\n const char *argSuiteName, List &argTests,\n unsigned argCreateLine, unsigned argDestroyLine );\n\n void initialize( const char *argFile, unsigned argLine,\n const char *argSuiteName, List &argTests,\n unsigned argCreateLine, unsigned argDestroyLine );\n\n protected:\n unsigned _createLine, _destroyLine;\n\n private:\n void doInitialize( unsigned argCreateLine, unsigned argDestroyLine );\n };\n \n template\n class DynamicSuiteDescription : public CommonDynamicSuiteDescription\n {\n public:\n DynamicSuiteDescription() {}\n DynamicSuiteDescription( const char *argFile, unsigned argLine,\n const char *argSuiteName, List &argTests,\n S *&argSuite, unsigned argCreateLine,\n unsigned argDestroyLine ) :\n CommonDynamicSuiteDescription( argFile, argLine, argSuiteName, argTests, argCreateLine, argDestroyLine )\n {\n _suite = &argSuite;\n }\n\n void initialize( const char *argFile, unsigned argLine,\n const char *argSuiteName, List &argTests,\n S *&argSuite, unsigned argCreateLine,\n unsigned argDestroyLine )\n {\n CommonDynamicSuiteDescription::initialize( argFile, argLine,\n argSuiteName, argTests,\n argCreateLine, argDestroyLine );\n _suite = &argSuite;\n }\n\n TestSuite *suite() const { return realSuite(); }\n\n bool setUp();\n bool tearDown();\n \n private:\n S *realSuite() const { return *_suite; }\n void setSuite( S *s ) { *_suite = s; }\n\n void createSuite()\n {\n setSuite( S::createSuite() );\n }\n \n void destroySuite()\n {\n S *s = realSuite();\n setSuite( 0 );\n S::destroySuite( s );\n }\n\n S **_suite;\n };\n\n template\n bool DynamicSuiteDescription::setUp()\n {\n _TS_TRY {\n _TSM_ASSERT_THROWS_NOTHING( file(), _createLine, \"Exception thrown from createSuite()\", createSuite() );\n _TSM_ASSERT( file(), _createLine, \"createSuite() failed\", suite() != 0 );\n }\n _TS_CATCH_ABORT( { return false; } );\n\n return (suite() != 0);\n }\n\n template\n bool DynamicSuiteDescription::tearDown()\n {\n if ( !_suite )\n return true;\n \n _TS_TRY {\n _TSM_ASSERT_THROWS_NOTHING( file(), _destroyLine, \"destroySuite() failed\", destroySuite() );\n }\n _TS_CATCH_ABORT( { return false; } );\n\n return true;\n }\n \n class RealWorldDescription : public WorldDescription\n {\n public:\n static List &suites();\n const char *worldName() const { return _worldName;}\n unsigned numSuites( void ) const;\n unsigned numTotalTests( void ) const;\n SuiteDescription *firstSuite();\n const SuiteDescription *firstSuite() const;\n const SuiteDescription &suiteDescription( unsigned i ) const;\n void activateAllTests();\n bool leaveOnly( const char *suiteName, const char *testName = 0 );\n \n bool setUp();\n bool tearDown();\n static void reportError( const char *message );\n\n static const char *_worldName;\n };\n\n void activateAllTests();\n bool leaveOnly( const char *suiteName, const char *testName = 0 );\n}\n\n#endif // __cxxtest__RealDescriptions_h__\n\n" }, { "path": "cxxtest/cxxtest/Root.cpp", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__Root_cpp__\n#define __cxxtest__Root_cpp__\n\n//\n// This file holds the \"root\" of CxxTest, i.e.\n// the parts that must be in a source file file.\n//\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#endif // __cxxtest__Root_cpp__\n" }, { "path": "cxxtest/cxxtest/SelfTest.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest_SelfTest_h__\n#define __cxxtest_SelfTest_h__\n\n#define CXXTEST_SUITE(name)\n#define CXXTEST_CODE(member)\n\n#endif // __cxxtest_SelfTest_h__\n" }, { "path": "cxxtest/cxxtest/StdHeaders.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest_StdHeaders_h__\n#define __cxxtest_StdHeaders_h__\n\n//\n// This file basically #includes the STL headers.\n// It exists to support warning level 4 in Visual C++\n//\n\n#ifdef _MSC_VER\n# pragma warning( push, 1 )\n#endif // _MSC_VER\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#ifdef _MSC_VER\n# pragma warning( pop )\n#endif // _MSC_VER\n\n#endif // __cxxtest_StdHeaders_h__\n" }, { "path": "cxxtest/cxxtest/StdTestSuite.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__StdTestSuite_h__\n#define __cxxtest__StdTestSuite_h__\n\n//\n// This provides explicit partial specializations for STL-based\n// TestSuite comparison functions\n//\n\nnamespace CxxTest {\n\n#ifdef _CXXTEST_PARTIAL_TEMPLATE_SPECIALIZATION\n\ntemplate\nstruct delta, std::vector, D>\n{\n static bool test(std::vector x, std::vector y, D d)\n {\n if ( x.size() != y.size() )\n return false;\n for(size_t i = 0; i::test(x[i], y[i], d) )\n return false;\n return true;\n }\n};\n\ntemplate\nstruct delta, std::list, D>\n{\n static bool test(std::list x, std::list y, D d)\n {\n typename std::list::const_iterator x_it = x.begin();\n typename std::list::const_iterator y_it = y.begin();\n for(; x_it != x.end(); ++x_it, ++y_it)\n {\n if ( y_it == y.end() )\n return false;\n if ( ! delta::test(*x_it, *y_it, d) )\n return false;\n }\n return y_it == y.end();\n }\n};\n\n#endif\n\n} // namespace CxxTest\n\n#endif // __cxxtest__StdTestSuite_h__\n\n" }, { "path": "cxxtest/cxxtest/StdValueTraits.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest_StdValueTraits_h__\n#define __cxxtest_StdValueTraits_h__\n\n//\n// This file defines ValueTraits for std:: stuff.\n// It is #included by if you\n// define CXXTEST_HAVE_STD\n//\n\n#include \n#include \n\n#ifdef _CXXTEST_OLD_STD\n# define CXXTEST_STD(x) x\n#else // !_CXXTEST_OLD_STD\n# define CXXTEST_STD(x) std::x\n#endif // _CXXTEST_OLD_STD\n\n#ifndef CXXTEST_USER_VALUE_TRAITS\n\nnamespace CxxTest\n{\n //\n // NOTE: This should have been\n // template\n // class ValueTraits< std::basic_string > {};\n // But MSVC doesn't support it (yet).\n //\n\n //\n // If we have std::string, we might as well use it\n //\n class StdTraitsBase\n {\n public:\n StdTraitsBase &operator<<( const CXXTEST_STD(string) &s ) { _s += s; return *this; }\n const char *asString() const { return _s.c_str(); }\n\n private:\n CXXTEST_STD(string) _s;\n };\n \n //\n // std::string\n //\n CXXTEST_TEMPLATE_INSTANTIATION\n class ValueTraits : public StdTraitsBase\n {\n public:\n ValueTraits( const CXXTEST_STD(string) &s )\n {\n *this << \"\\\"\";\n for ( unsigned i = 0; i < s.length(); ++ i ) {\n char c[sizeof(\"\\\\xXX\")];\n charToString( s[i], c );\n *this << c;\n }\n *this << \"\\\"\";\n }\n };\n\n CXXTEST_COPY_CONST_TRAITS( CXXTEST_STD(string) );\n\n#ifndef _CXXTEST_OLD_STD\n //\n // std::wstring\n //\n CXXTEST_TEMPLATE_INSTANTIATION\n class ValueTraits)> : public StdTraitsBase\n {\n public:\n ValueTraits( const CXXTEST_STD(basic_string) &s )\n {\n *this << \"L\\\"\";\n for ( unsigned i = 0; i < s.length(); ++ i ) {\n char c[sizeof(\"\\\\x12345678\")];\n charToString( (unsigned long)s[i], c );\n *this << c;\n }\n *this << \"\\\"\";\n }\n };\n\n CXXTEST_COPY_CONST_TRAITS( CXXTEST_STD(basic_string) );\n#endif // _CXXTEST_OLD_STD\n\n //\n // Convert a range defined by iterators to a string\n // This is useful for almost all STL containers\n //\n template\n void dumpRange( Stream &s, Iterator first, Iterator last )\n {\n if ( first == last ) {\n s << \"{}\";\n return;\n }\n \n s << \"{ \";\n while ( first != last ) {\n s << TS_AS_STRING(*first);\n if ( ++ first != last )\n s << \", \";\n }\n s << \" }\";\n }\n\n#ifdef _CXXTEST_PARTIAL_TEMPLATE_SPECIALIZATION\n //\n // std::pair\n //\n template\n class ValueTraits< CXXTEST_STD(pair) > : public StdTraitsBase\n {\n public:\n ValueTraits( const CXXTEST_STD(pair) &p ) \n {\n *this << \"<\" << TS_AS_STRING( p.first ) << \", \" << TS_AS_STRING( p.second ) << \">\";\n }\n };\n\n //\n // std::vector\n //\n template\n class ValueTraits< CXXTEST_STD(vector) > : public StdTraitsBase\n {\n public:\n ValueTraits( const CXXTEST_STD(vector) &v )\n {\n dumpRange( *this, v.begin(), v.end() );\n }\n };\n\n //\n // std::list\n //\n template\n class ValueTraits< CXXTEST_STD(list) > : public StdTraitsBase\n {\n public:\n ValueTraits( const CXXTEST_STD(list) &l )\n {\n dumpRange( *this, l.begin(), l.end() );\n }\n };\n\n //\n // std::set\n //\n template\n class ValueTraits< CXXTEST_STD(set) > : public StdTraitsBase\n {\n public:\n ValueTraits( const CXXTEST_STD(set) &s )\n {\n dumpRange( *this, s.begin(), s.end() );\n }\n };\n\n //\n // std::map\n //\n template\n class ValueTraits< CXXTEST_STD(map) > : public StdTraitsBase\n {\n public:\n ValueTraits( const CXXTEST_STD(map) &m )\n {\n dumpRange( *this, m.begin(), m.end() );\n }\n }; \n\n //\n // std::deque\n //\n template\n class ValueTraits< CXXTEST_STD(deque) > : public StdTraitsBase\n {\n public:\n ValueTraits( const CXXTEST_STD(deque) &d )\n {\n dumpRange( *this, d.begin(), d.end() );\n }\n };\n\n //\n // std::multiset\n //\n template\n class ValueTraits< CXXTEST_STD(multiset) > : public StdTraitsBase\n {\n public:\n ValueTraits( const CXXTEST_STD(multiset) &ms )\n {\n dumpRange( *this, ms.begin(), ms.end() );\n }\n };\n\n //\n // std::multimap\n //\n template\n class ValueTraits< CXXTEST_STD(multimap) > : public StdTraitsBase\n {\n public:\n ValueTraits( const CXXTEST_STD(multimap) &mm )\n {\n dumpRange( *this, mm.begin(), mm.end() );\n }\n };\n\n //\n // std::complex\n //\n template\n class ValueTraits< CXXTEST_STD(complex) > : public StdTraitsBase\n {\n public:\n ValueTraits( const CXXTEST_STD(complex) &c )\n {\n if ( !c.imag() )\n *this << TS_AS_STRING(c.real());\n else if ( !c.real() )\n *this << \"(\" << TS_AS_STRING(c.imag()) << \" * i)\";\n else\n *this << \"(\" << TS_AS_STRING(c.real()) << \" + \" << TS_AS_STRING(c.imag()) << \" * i)\";\n }\n };\n#endif // _CXXTEST_PARTIAL_TEMPLATE_SPECIALIZATION\n}\n\n#endif // CXXTEST_USER_VALUE_TRAITS\n\n#endif // __cxxtest_StdValueTraits_h__\n" }, { "path": "cxxtest/cxxtest/StdioFilePrinter.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__StdioFilePrinter_h__\n#define __cxxtest__StdioFilePrinter_h__\n\n//\n// The StdioFilePrinter is a simple TestListener that\n// just prints \"OK\" if everything goes well, otherwise\n// reports the error in the format of compiler messages.\n// This class uses , i.e. FILE * and fprintf().\n//\n\n#include \n#include \n\nnamespace CxxTest \n{\n class StdioFilePrinter : public ErrorFormatter\n {\n public:\n StdioFilePrinter( FILE *o, const char *preLine = \":\", const char *postLine = \"\" ) :\n ErrorFormatter( new Adapter(o), preLine, postLine ) {}\n virtual ~StdioFilePrinter() { delete outputStream(); }\n\n private:\n class Adapter : public OutputStream\n {\n Adapter( const Adapter & );\n Adapter &operator=( const Adapter & );\n \n FILE *_o;\n \n public:\n Adapter( FILE *o ) : _o(o) {}\n void flush() { fflush( _o ); }\n OutputStream &operator<<( unsigned i ) { fprintf( _o, \"%u\", i ); return *this; }\n OutputStream &operator<<( const char *s ) { fputs( s, _o ); return *this; }\n OutputStream &operator<<( Manipulator m ) { return OutputStream::operator<<( m ); }\n };\n };\n}\n\n#endif // __cxxtest__StdioFilePrinter_h__\n" }, { "path": "cxxtest/cxxtest/StdioPrinter.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__StdioPrinter_h__\n#define __cxxtest__StdioPrinter_h__\n\n//\n// The StdioPrinter is an StdioFilePrinter which defaults to stdout.\n// This should have been called StdOutPrinter or something, but the name\n// has been historically used.\n//\n\n#include \n\nnamespace CxxTest \n{\n class StdioPrinter : public StdioFilePrinter\n {\n public:\n StdioPrinter( FILE *o = stdout, const char *preLine = \":\", const char *postLine = \"\" ) :\n StdioFilePrinter( o, preLine, postLine ) {}\n };\n}\n\n#endif // __cxxtest__StdioPrinter_h__\n" }, { "path": "cxxtest/cxxtest/TeeListener.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__TeeListener_h__\n#define __cxxtest__TeeListener_h__\n\n//\n// A TeeListener notifies two \"regular\" TestListeners\n//\n\n#include \n#include \n\nnamespace CxxTest\n{\n class TeeListener : public TestListener\n {\n public:\n TeeListener()\n {\n setFirst( _dummy );\n setSecond( _dummy );\n }\n\n virtual ~TeeListener()\n {\n }\n\n void setFirst( TestListener &first )\n {\n _first = &first;\n }\n\n void setSecond( TestListener &second )\n {\n _second = &second;\n }\n\n void enterWorld( const WorldDescription &d )\n {\n _first->enterWorld( d );\n _second->enterWorld( d );\n }\n\n void enterSuite( const SuiteDescription &d )\n {\n _first->enterSuite( d );\n _second->enterSuite( d );\n }\n \n void enterTest( const TestDescription &d )\n {\n _first->enterTest( d );\n _second->enterTest( d );\n }\n \n void trace( const char *file, int line, const char *expression )\n {\n _first->trace( file, line, expression );\n _second->trace( file, line, expression );\n }\n \n void warning( const char *file, int line, const char *expression )\n {\n _first->warning( file, line, expression );\n _second->warning( file, line, expression );\n }\n \n void failedTest( const char *file, int line, const char *expression )\n {\n _first->failedTest( file, line, expression );\n _second->failedTest( file, line, expression );\n }\n \n void failedAssert( const char *file, int line, const char *expression )\n {\n _first->failedAssert( file, line, expression );\n _second->failedAssert( file, line, expression );\n }\n \n void failedAssertEquals( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *x, const char *y )\n {\n _first->failedAssertEquals( file, line, xStr, yStr, x, y );\n _second->failedAssertEquals( file, line, xStr, yStr, x, y );\n }\n\n void failedAssertSameData( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *sizeStr, const void *x,\n const void *y, unsigned size )\n {\n _first->failedAssertSameData( file, line, xStr, yStr, sizeStr, x, y, size );\n _second->failedAssertSameData( file, line, xStr, yStr, sizeStr, x, y, size );\n }\n \n void failedAssertSameFiles( const char* file, int line, const char* file1, const char* file2, const char* explanation)\n {\n _first->failedAssertSameFiles( file, line, file1, file2, explanation );\n _second->failedAssertSameFiles( file, line, file1, file2, explanation );\n }\n\n void failedAssertDelta( const char *file, int line,\n const char *xStr, const char *yStr, const char *dStr,\n const char *x, const char *y, const char *d )\n {\n _first->failedAssertDelta( file, line, xStr, yStr, dStr, x, y, d );\n _second->failedAssertDelta( file, line, xStr, yStr, dStr, x, y, d );\n }\n \n void failedAssertDiffers( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *value )\n {\n _first->failedAssertDiffers( file, line, xStr, yStr, value );\n _second->failedAssertDiffers( file, line, xStr, yStr, value );\n }\n \n void failedAssertLessThan( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *x, const char *y )\n {\n _first->failedAssertLessThan( file, line, xStr, yStr, x, y );\n _second->failedAssertLessThan( file, line, xStr, yStr, x, y );\n }\n \n void failedAssertLessThanEquals( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *x, const char *y )\n {\n _first->failedAssertLessThanEquals( file, line, xStr, yStr, x, y );\n _second->failedAssertLessThanEquals( file, line, xStr, yStr, x, y );\n }\n \n void failedAssertPredicate( const char *file, int line,\n const char *predicate, const char *xStr, const char *x )\n {\n _first->failedAssertPredicate( file, line, predicate, xStr, x );\n _second->failedAssertPredicate( file, line, predicate, xStr, x );\n }\n \n void failedAssertRelation( const char *file, int line,\n const char *relation, const char *xStr, const char *yStr,\n const char *x, const char *y )\n {\n _first->failedAssertRelation( file, line, relation, xStr, yStr, x, y );\n _second->failedAssertRelation( file, line, relation, xStr, yStr, x, y );\n }\n \n void failedAssertThrows( const char *file, int line,\n const char *expression, const char *type,\n bool otherThrown )\n {\n _first->failedAssertThrows( file, line, expression, type, otherThrown );\n _second->failedAssertThrows( file, line, expression, type, otherThrown );\n }\n \n void failedAssertThrowsNot( const char *file, int line,\n const char *expression )\n {\n _first->failedAssertThrowsNot( file, line, expression );\n _second->failedAssertThrowsNot( file, line, expression );\n }\n \n void leaveTest( const TestDescription &d )\n {\n _first->leaveTest(d);\n _second->leaveTest(d);\n }\n \n void leaveSuite( const SuiteDescription &d )\n {\n _first->leaveSuite(d);\n _second->leaveSuite(d);\n }\n \n void leaveWorld( const WorldDescription &d )\n {\n _first->leaveWorld(d);\n _second->leaveWorld(d);\n }\n\n private:\n TestListener *_first, *_second;\n TestListener _dummy;\n };\n}\n\n\n#endif // __cxxtest__TeeListener_h__\n" }, { "path": "cxxtest/cxxtest/TestListener.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__TestListener_h__\n#define __cxxtest__TestListener_h__\n\n//\n// TestListener is the base class for all \"listeners\",\n// i.e. classes that receive notifications of the\n// testing process.\n//\n// The names of the parameters are in comments to avoid\n// \"unused parameter\" warnings.\n//\n\n#include \n\nnamespace CxxTest\n{\n class TestListener\n {\n public:\n TestListener() {}\n virtual ~TestListener() {}\n virtual void process_commandline(int& /*argc*/, char** /*argv*/) {}\n \n virtual void enterWorld( const WorldDescription & /*desc*/ ) {}\n virtual void enterSuite( const SuiteDescription & /*desc*/ ) {}\n virtual void enterTest( const TestDescription & /*desc*/ ) {}\n virtual void trace( const char * /*file*/, int /*line*/,\n const char * /*expression*/ ) {}\n virtual void warning( const char * /*file*/, int /*line*/,\n const char * /*expression*/ ) {}\n virtual void failedTest( const char * /*file*/, int /*line*/,\n const char * /*expression*/ ) {}\n virtual void failedAssert( const char * /*file*/, int /*line*/,\n const char * /*expression*/ ) {}\n virtual void failedAssertEquals( const char * /*file*/, int /*line*/,\n const char * /*xStr*/, const char * /*yStr*/,\n const char * /*x*/, const char * /*y*/ ) {}\n virtual void failedAssertSameData( const char * /*file*/, int /*line*/,\n const char * /*xStr*/, const char * /*yStr*/,\n const char * /*sizeStr*/, const void * /*x*/,\n const void * /*y*/, unsigned /*size*/ ) {}\n virtual void failedAssertDelta( const char * /*file*/, int /*line*/,\n const char * /*xStr*/, const char * /*yStr*/,\n const char * /*dStr*/, const char * /*x*/,\n const char * /*y*/, const char * /*d*/ ) {}\n virtual void failedAssertDiffers( const char * /*file*/, int /*line*/,\n const char * /*xStr*/, const char * /*yStr*/,\n const char * /*value*/ ) {}\n virtual void failedAssertLessThan( const char * /*file*/, int /*line*/,\n const char * /*xStr*/, const char * /*yStr*/,\n const char * /*x*/, const char * /*y*/ ) {}\n virtual void failedAssertLessThanEquals( const char * /*file*/, int /*line*/,\n const char * /*xStr*/, const char * /*yStr*/,\n const char * /*x*/, const char * /*y*/ ) {}\n virtual void failedAssertPredicate( const char * /*file*/, int /*line*/,\n const char * /*predicate*/, const char * /*xStr*/, const char * /*x*/ ) {}\n virtual void failedAssertRelation( const char * /*file*/, int /*line*/,\n const char * /*relation*/, const char * /*xStr*/, const char * /*yStr*/,\n const char * /*x*/, const char * /*y*/ ) {}\n virtual void failedAssertThrows( const char * /*file*/, int /*line*/,\n const char * /*expression*/, const char * /*type*/,\n bool /*otherThrown*/ ) {}\n virtual void failedAssertThrowsNot( const char * /*file*/, int /*line*/,\n const char * /*expression*/ ) {}\n\t virtual void failedAssertSameFiles( const char* /*file*/, int /*line*/, \n const char* , const char*, const char* ) {}\n virtual void leaveTest( const TestDescription & /*desc*/ ) {}\n virtual void leaveSuite( const SuiteDescription & /*desc*/ ) {}\n virtual void leaveWorld( const WorldDescription & /*desc*/ ) {}\n };\n}\n\n#endif // __cxxtest__TestListener_h__\n\n" }, { "path": "cxxtest/cxxtest/TestMain.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __CxxTestMain_h\n#define __CxxTestMain_h\n\n#include \n#include \n\n#ifndef _CXXTEST_HAVE_STD\n# define _CXXTEST_HAVE_STD\n#endif // _CXXTEST_HAVE_STD\n\n#include \n\n#ifdef _CXXTEST_OLD_STD\n# include \n# include \n#else // !_CXXTEST_OLD_STD\n# include \n# include \n#endif // _CXXTEST_OLD_STD\n\nnamespace CxxTest\n{\n\ninline void print_help(const char* name)\n{\n CXXTEST_STD(cerr) << name << \" \" << CXXTEST_STD(endl);\n CXXTEST_STD(cerr) << name << \" \" << CXXTEST_STD(endl);\n CXXTEST_STD(cerr) << name << \" -h\" << CXXTEST_STD(endl);\n CXXTEST_STD(cerr) << name << \" --help\" << CXXTEST_STD(endl);\n CXXTEST_STD(cerr) << name << \" --help-tests\" << CXXTEST_STD(endl);\n CXXTEST_STD(cerr) << name << \" -v Enable tracing output.\" << CXXTEST_STD(endl);\n}\n\n\ntemplate \nint Main(TesterT& tmp, int argc, char* argv[])\n{ \n//\n// Parse the command-line arguments. The default behavior is to run all tests\n//\n// This is a primitive parser, but I'm not sure what sort of portable\n// parser should be used in cxxtest.\n//\n\n//\n// Print command-line syntax\n//\nfor (int i=1; inext() )\n for ( TestDescription *td = sd->firstTest(); td; td = td->next() )\n CXXTEST_STD(cout) << td->suiteName() << \" \" << td->testName() << CXXTEST_STD(endl);\n return 0;\n }\n}\n\n//\n// Process command-line options here.\n//\nwhile ((argc > 1) && (argv[1][0] == '-')) {\n if (CXXTEST_STD(strcmp)(argv[1],\"-v\") == 0) {\n tracker().print_tracing = true;\n }\n else {\n CXXTEST_STD(cerr) << \"ERROR: unknown option '\" << argv[1] << \"'\" << CXXTEST_STD(endl);\n return -1;\n }\n for (int i=1; i<(argc-1); i++)\n argv[i] = argv[i+1];\n argc--;\n }\n\n//\n// Run experiments\n//\nbool status=false;\nif ((argc==2) && (argv[1][0] != '-')) {\n status=leaveOnly(argv[1]);\n if (!status) {\n CXXTEST_STD(cerr) << \"ERROR: unknown suite '\" << argv[1] << \"'\" << CXXTEST_STD(endl);\n return -1;\n }\n }\nif ((argc==3) && (argv[1][0] != '-')) {\n status=leaveOnly(argv[1],argv[2]);\n if (!status) {\n CXXTEST_STD(cerr) << \"ERROR: unknown test '\" << argv[1] << \"::\" << argv[2] << \"'\" << CXXTEST_STD(endl);\n return -1;\n }\n }\n\ntmp.process_commandline(argc,argv);\nreturn tmp.run();\n}\n\n}\n#endif\n\n" }, { "path": "cxxtest/cxxtest/TestRunner.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest_TestRunner_h__\n#define __cxxtest_TestRunner_h__\n\n//\n// TestRunner is the class that runs all the tests.\n// To use it, create an object that implements the TestListener\n// interface and call TestRunner::runAllTests( myListener );\n// \n\n#include \n#include \n#include \n#include \n\nnamespace CxxTest \n{\n class TestRunner\n {\n public:\n static void runAllTests( TestListener &listener )\n {\n tracker().setListener( &listener );\n _TS_TRY { TestRunner().runWorld(); }\n _TS_LAST_CATCH( { tracker().failedTest( __FILE__, __LINE__, \"Exception thrown from world\" ); } );\n tracker().setListener( 0 );\n }\n\n static void runAllTests( TestListener *listener )\n {\n if ( listener ) {\n listener->warning( __FILE__, __LINE__, \"Deprecated; Use runAllTests( TestListener & )\" );\n runAllTests( *listener );\n }\n } \n \n private:\n void runWorld()\n {\n RealWorldDescription wd;\n WorldGuard sg;\n \n tracker().enterWorld( wd );\n if ( wd.setUp() ) {\n for ( SuiteDescription *sd = wd.firstSuite(); sd; sd = sd->next() )\n if ( sd->active() )\n runSuite( *sd );\n \n wd.tearDown();\n }\n tracker().leaveWorld( wd );\n }\n \n void runSuite( SuiteDescription &sd )\n {\n StateGuard sg;\n \n tracker().enterSuite( sd );\n if ( sd.setUp() ) {\n for ( TestDescription *td = sd.firstTest(); td; td = td->next() )\n if ( td->active() )\n runTest( *td );\n\n sd.tearDown();\n }\n tracker().leaveSuite( sd );\n }\n\n void runTest( TestDescription &td )\n {\n StateGuard sg;\n \n tracker().enterTest( td );\n if ( td.setUp() ) {\n td.run();\n td.tearDown();\n }\n tracker().leaveTest( td );\n }\n \n class StateGuard\n {\n#ifdef _CXXTEST_HAVE_EH\n bool _abortTestOnFail;\n#endif // _CXXTEST_HAVE_EH\n unsigned _maxDumpSize;\n \n public:\n StateGuard()\n {\n#ifdef _CXXTEST_HAVE_EH\n _abortTestOnFail = abortTestOnFail();\n#endif // _CXXTEST_HAVE_EH\n _maxDumpSize = maxDumpSize();\n }\n \n ~StateGuard()\n {\n#ifdef _CXXTEST_HAVE_EH\n setAbortTestOnFail( _abortTestOnFail );\n#endif // _CXXTEST_HAVE_EH\n setMaxDumpSize( _maxDumpSize );\n }\n };\n\n class WorldGuard : public StateGuard\n {\n public:\n WorldGuard() : StateGuard()\n {\n#ifdef _CXXTEST_HAVE_EH\n setAbortTestOnFail( CXXTEST_DEFAULT_ABORT );\n#endif // _CXXTEST_HAVE_EH\n setMaxDumpSize( CXXTEST_MAX_DUMP_SIZE );\n }\n };\n };\n\n //\n // For --no-static-init\n //\n void initialize();\n}\n\n\n#endif // __cxxtest_TestRunner_h__\n" }, { "path": "cxxtest/cxxtest/TestSuite.cpp", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__TestSuite_cpp__\n#define __cxxtest__TestSuite_cpp__\n\n#include \n#if defined(_CXXTEST_HAVE_STD)\n#include \n#endif\n\nnamespace CxxTest\n{\n //\n // TestSuite members\n //\n TestSuite::~TestSuite() {}\n void TestSuite::setUp() {}\n void TestSuite::tearDown() {}\n\n //\n // Test-aborting stuff\n //\n static bool currentAbortTestOnFail = false;\n\n bool abortTestOnFail()\n {\n return currentAbortTestOnFail;\n }\n\n void setAbortTestOnFail( bool value )\n {\n currentAbortTestOnFail = value;\n }\n \n void doAbortTest()\n {\n# if defined(_CXXTEST_HAVE_EH)\n if ( currentAbortTestOnFail )\n throw AbortTest();\n# endif // _CXXTEST_HAVE_EH\n }\n\n //\n // Max dump size\n //\n static unsigned currentMaxDumpSize = CXXTEST_MAX_DUMP_SIZE;\n\n unsigned maxDumpSize()\n {\n return currentMaxDumpSize;\n }\n \n void setMaxDumpSize( unsigned value )\n {\n currentMaxDumpSize = value;\n }\n\n //\n // Some non-template functions\n //\n void doTrace( const char *file, int line, const char *message )\n {\n if (tracker().print_tracing) {\n tracker().trace( file, line, message );\n }\n }\n\n void doWarn( const char *file, int line, const char *message )\n {\n tracker().warning( file, line, message );\n }\n\n void doFailTest( const char *file, int line, const char *message )\n {\n tracker().failedTest( file, line, message );\n TS_ABORT();\n }\n\n void doFailAssert( const char *file, int line,\n const char *expression, const char *message )\n {\n if ( message )\n tracker().failedTest( file, line, message );\n tracker().failedAssert( file, line, expression );\n TS_ABORT();\n }\n\n bool sameData( const void *x, const void *y, unsigned size )\n {\n if ( size == 0 )\n return true;\n \n if ( x == y )\n return true;\n\n if ( !x || !y )\n return false;\n\n const char *cx = (const char *)x;\n const char *cy = (const char *)y;\n while ( size -- )\n if ( *cx++ != *cy++ )\n return false;\n\n return true;\n }\n\n void doAssertSameData( const char *file, int line,\n const char *xExpr, const void *x,\n const char *yExpr, const void *y,\n const char *sizeExpr, unsigned size,\n const char *message )\n {\n if ( !sameData( x, y, size ) ) {\n if ( message )\n tracker().failedTest( file, line, message );\n tracker().failedAssertSameData( file, line, xExpr, yExpr, sizeExpr, x, y, size );\n TS_ABORT();\n }\n }\n\n//#if defined(_CXXTEST_HAVE_STD)\n bool sameFiles( const char* file1, const char* file2, std::ostringstream& explanation)\n {\n std::string ppprev_line;\n std::string pprev_line;\n std::string prev_line;\n std::string curr_line;\n\n std::ifstream is1;\n is1.open(file1);\n std::ifstream is2;\n is2.open(file2);\n if (!is1) {\n explanation << \"File '\" << file1 << \"' does not exist!\";\n return false;\n }\n if (!is2) {\n explanation << \"File '\" << file2 << \"' does not exist!\";\n return false;\n }\n\n int nline=1;\n char c1, c2;\n while (1) {\n is1.get(c1);\n is2.get(c2);\n if (!is1 && !is2) return true;\n if (!is1) {\n explanation << \"File '\" << file1 << \"' ended before file '\" << file2 << \"' (line \" << nline << \")\";\n explanation << std::endl << \"= \" << ppprev_line << std::endl << \"= \" << pprev_line << std::endl << \"= \" << prev_line << std::endl << \"< \" << curr_line;\n is1.get(c1);\n while (is1 && (c1 != '\\n')) {\n explanation << c1;\n is1.get(c1);\n }\n explanation << std::endl;\n return false;\n }\n if (!is2) {\n explanation << \"File '\" << file2 << \"' ended before file '\" << file1 << \"' (line \" << nline << \")\";\n explanation << std::endl << \"= \" << ppprev_line << std::endl << \"= \" << pprev_line << std::endl << \"= \" << prev_line << std::endl << \"> \" << curr_line;\n is2.get(c2);\n while (is2 && (c2 != '\\n')) {\n explanation << c2;\n is2.get(c2);\n }\n explanation << std::endl;\n return false;\n }\n if (c1 != c2) {\n explanation << \"Files '\" << file1 << \"' and '\" << file2 << \"' differ at line \" << nline;\n explanation << std::endl << \"= \" << ppprev_line << std::endl << \"= \" << pprev_line << std::endl << \"= \" << prev_line;\n\n explanation << std::endl << \"< \" << curr_line;\n is2.get(c1);\n while (is1 && (c1 != '\\n')) {\n explanation << c1;\n is2.get(c1);\n }\n explanation << std::endl;\n\n explanation << std::endl << \"> \" << curr_line;\n is2.get(c2);\n while (is2 && (c2 != '\\n')) {\n explanation << c2;\n is2.get(c2);\n }\n explanation << std::endl;\n\n return false;\n }\n if (c1 == '\\n') {\n ppprev_line = pprev_line;\n pprev_line = prev_line;\n prev_line = curr_line;\n curr_line = \"\";\n nline++;\n }\n else {\n curr_line += c1;\n }\n }\n }\n//#endif\n\n void doAssertSameFiles( const char* file, int line,\n const char* file1, const char* file2,\n const char* message)\n {\n#if defined(_CXXTEST_HAVE_STD)\n std::ostringstream explanation;\n if ( !sameFiles( file1, file2, explanation ) ) {\n if ( message )\n tracker().failedTest( file, line, message );\n tracker().failedAssertSameFiles( file, line, file1, file2, explanation.str().c_str());\n TS_ABORT();\n }\n#else\n tracker().failedAssertSameFiles( file, line, file1, file2, \"This test is only supported when --have-std is enabled\");\n TS_ABORT();\n#endif\n }\n\n void doFailAssertThrows( const char *file, int line,\n const char *expr, const char *type,\n bool otherThrown,\n const char *message,\n const char *exception )\n {\n if ( exception )\n tracker().failedTest( file, line, exception );\n if ( message )\n tracker().failedTest( file, line, message );\n \n tracker().failedAssertThrows( file, line, expr, type, otherThrown );\n TS_ABORT();\n }\n\n void doFailAssertThrowsNot( const char *file, int line,\n const char *expression, const char *message,\n const char *exception )\n {\n if ( exception )\n tracker().failedTest( file, line, exception );\n if ( message )\n tracker().failedTest( file, line, message );\n \n tracker().failedAssertThrowsNot( file, line, expression );\n TS_ABORT();\n }\n}\n\n#endif // __cxxtest__TestSuite_cpp__\n\n" }, { "path": "cxxtest/cxxtest/TestSuite.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__TestSuite_h__\n#define __cxxtest__TestSuite_h__\n\n//\n// class TestSuite is the base class for all test suites.\n// To define a test suite, derive from this class and add\n// member functions called void test*();\n//\n\n#include \n#include \n#include \n#include \n#include \n\n#if defined(_CXXTEST_HAVE_STD)\n# include \n#endif // _CXXTEST_HAVE_STD\n\nnamespace CxxTest\n{\n class TestSuite\n {\n public:\n virtual ~TestSuite();\n virtual void setUp();\n virtual void tearDown();\n };\n \n class AbortTest {};\n void doAbortTest();\n# define TS_ABORT() CxxTest::doAbortTest()\n \n bool abortTestOnFail();\n void setAbortTestOnFail( bool value = CXXTEST_DEFAULT_ABORT );\n\n unsigned maxDumpSize();\n void setMaxDumpSize( unsigned value = CXXTEST_MAX_DUMP_SIZE );\n\n void doTrace( const char *file, int line, const char *message );\n void doWarn( const char *file, int line, const char *message );\n void doFailTest( const char *file, int line, const char *message );\n void doFailAssert( const char *file, int line, const char *expression, const char *message );\n\n template\n struct equals {\n static bool test( X x, Y y )\n {\n return (x == y);\n }\n };\n\n template\n void doAssertEquals( const char *file, int line,\n const char *xExpr, X x,\n const char *yExpr, Y y,\n const char *message )\n {\n if ( !equals::test( x, y ) ) {\n if ( message )\n tracker().failedTest( file, line, message );\n tracker().failedAssertEquals( file, line, xExpr, yExpr, TS_AS_STRING(x), TS_AS_STRING(y) );\n TS_ABORT();\n }\n }\n\n bool sameData( const void *x, const void *y, unsigned size );\n\n void doAssertSameData( const char *file, int line,\n const char *xExpr, const void *x,\n const char *yExpr, const void *y,\n const char *sizeExpr, unsigned size,\n const char *message );\n\n//#if defined(_CXXTEST_HAVE_STD)\n bool sameFiles( const char* file1, const char* file2, std::ostringstream& explanation);\n//#endif\n\n template\n struct differs {\n static bool test( X x, Y y )\n {\n return !(x == y);\n }\n };\n\n template\n void doAssertDiffers( const char *file, int line,\n const char *xExpr, X x,\n const char *yExpr, Y y,\n const char *message )\n {\n if ( !differs::test( x, y ) ) {\n if ( message )\n tracker().failedTest( file, line, message );\n tracker().failedAssertDiffers( file, line, xExpr, yExpr, TS_AS_STRING(x) );\n TS_ABORT();\n }\n }\n\n template\n struct lessThan {\n static bool test( X x, Y y )\n {\n return (x < y);\n }\n };\n\n template\n void doAssertLessThan( const char *file, int line,\n const char *xExpr, X x,\n const char *yExpr, Y y,\n const char *message )\n {\n if ( !lessThan::test(x, y) ) {\n if ( message )\n tracker().failedTest( file, line, message );\n tracker().failedAssertLessThan( file, line, xExpr, yExpr, TS_AS_STRING(x), TS_AS_STRING(y) );\n TS_ABORT();\n }\n }\n\n template\n struct lessThanEquals {\n static bool test( X x, Y y )\n {\n return (x <= y);\n }\n };\n\n template\n void doAssertLessThanEquals( const char *file, int line,\n const char *xExpr, X x,\n const char *yExpr, Y y,\n const char *message )\n {\n if ( !lessThanEquals::test( x, y ) ) {\n if ( message )\n tracker().failedTest( file, line, message );\n tracker().failedAssertLessThanEquals( file, line, xExpr, yExpr, TS_AS_STRING(x), TS_AS_STRING(y) );\n TS_ABORT();\n }\n }\n\n template\n void doAssertPredicate( const char *file, int line,\n const char *pExpr, const P &p,\n const char *xExpr, X x,\n const char *message )\n {\n if ( !p( x ) ) {\n if ( message )\n tracker().failedTest( file, line, message );\n tracker().failedAssertPredicate( file, line, pExpr, xExpr, TS_AS_STRING(x) );\n TS_ABORT();\n }\n }\n\n template\n void doAssertRelation( const char *file, int line,\n const char *rExpr, const R &r, \n const char *xExpr, X x,\n const char *yExpr, Y y,\n const char *message )\n {\n if ( !r( x, y ) ) {\n if ( message )\n tracker().failedTest( file, line, message );\n tracker().failedAssertRelation( file, line, rExpr, xExpr, yExpr, TS_AS_STRING(x), TS_AS_STRING(y) );\n TS_ABORT();\n }\n }\n\n // An indirection template so the compiler can determine what type \n // \"X +/- D\" should be\n template\n bool delta_le_helper( X x, Y y )\n { \n return lessThanEquals::test(x,y); \n }\n\n template\n struct delta {\n static bool test( X x, Y y, D d )\n {\n return delta_le_helper(x-d, y) && delta_le_helper(y, x+d);\n //(y >= x - d) && (y <= x + d));\n }\n };\n\n template\n void doAssertDelta( const char *file, int line,\n const char *xExpr, X x,\n const char *yExpr, Y y,\n const char *dExpr, D d,\n const char *message )\n {\n if ( !delta::test( x, y, d ) ) {\n if ( message )\n tracker().failedTest( file, line, message );\n \n tracker().failedAssertDelta( file, line, xExpr, yExpr, dExpr,\n TS_AS_STRING(x), TS_AS_STRING(y), TS_AS_STRING(d) );\n TS_ABORT();\n }\n }\n\n void doFailAssertThrows( const char *file, int line,\n const char *expr, const char *type,\n bool otherThrown,\n const char *message,\n const char *exception = 0 );\n \n void doFailAssertThrowsNot( const char *file, int line,\n const char *expression, const char *message,\n const char *exception = 0 );\n\n void doAssertSameFiles( const char* file, int line,\n const char* file1, const char* file2,\n const char* message);\n\n# ifdef _CXXTEST_HAVE_EH\n# define _TS_TRY try\n# define _TS_CATCH_TYPE(t, b) catch t b\n# define _TS_CATCH_ABORT(b) _TS_CATCH_TYPE( (const CxxTest::AbortTest &), b )\n# define _TS_LAST_CATCH(b) _TS_CATCH_TYPE( (...), b )\n# define _TSM_LAST_CATCH(f,l,m) _TS_LAST_CATCH( { (CxxTest::tracker()).failedTest(f,l,m); TS_ABORT(); } )\n# ifdef _CXXTEST_HAVE_STD\n# define _TS_CATCH_STD(e,b) _TS_CATCH_TYPE( (const std::exception& e), b )\n# else // !_CXXTEST_HAVE_STD\n# define _TS_CATCH_STD(e,b)\n# endif // _CXXTEST_HAVE_STD\n# define ___TSM_CATCH(f,l,m) \\\n _TS_CATCH_STD(e, { (CxxTest::tracker()).failedTest(f,l,e.what()); TS_ABORT(); }) \\\n _TSM_LAST_CATCH(f,l,m)\n# define __TSM_CATCH(f,l,m) \\\n _TS_CATCH_ABORT( { throw; } ) \\\n ___TSM_CATCH(f,l,m)\n# define __TS_CATCH(f,l) __TSM_CATCH(f,l,\"Unhandled exception\")\n# define _TS_CATCH __TS_CATCH(__FILE__,__LINE__)\n# else // !_CXXTEST_HAVE_EH\n# define _TS_TRY\n# define ___TSM_CATCH(f,l,m)\n# define __TSM_CATCH(f,l,m)\n# define __TS_CATCH(f,l)\n# define _TS_CATCH\n# define _TS_CATCH_TYPE(t, b)\n# define _TS_LAST_CATCH(b)\n# define _TS_CATCH_STD(e,b)\n# define _TS_CATCH_ABORT(b)\n# endif // _CXXTEST_HAVE_EH\n\n // TS_TRACE\n# define _TS_TRACE(f,l,e) CxxTest::doTrace( (f), (l), TS_AS_STRING(e) )\n# define TS_TRACE(e) _TS_TRACE( __FILE__, __LINE__, e )\n\n // TS_WARN\n# define _TS_WARN(f,l,e) CxxTest::doWarn( (f), (l), TS_AS_STRING(e) )\n# define TS_WARN(e) _TS_WARN( __FILE__, __LINE__, e )\n\n // TS_FAIL\n# define _TS_FAIL(f,l,e) CxxTest::doFailTest( (f), (l), TS_AS_STRING(e) )\n# define TS_FAIL(e) _TS_FAIL( __FILE__, __LINE__, e )\n\n // TS_ASSERT\n# define ___ETS_ASSERT(f,l,e,m) { if ( !(e) ) CxxTest::doFailAssert( (f), (l), #e, (m) ); }\n# define ___TS_ASSERT(f,l,e,m) { _TS_TRY { ___ETS_ASSERT(f,l,e,m); } __TS_CATCH(f,l) }\n \n# define _ETS_ASSERT(f,l,e) ___ETS_ASSERT(f,l,e,0)\n# define _TS_ASSERT(f,l,e) ___TS_ASSERT(f,l,e,0)\n \n# define ETS_ASSERT(e) _ETS_ASSERT(__FILE__,__LINE__,e)\n# define TS_ASSERT(e) _TS_ASSERT(__FILE__,__LINE__,e)\n \n# define _ETSM_ASSERT(f,l,m,e) ___ETS_ASSERT(f,l,e,TS_AS_STRING(m) )\n# define _TSM_ASSERT(f,l,m,e) ___TS_ASSERT(f,l,e,TS_AS_STRING(m) )\n\n# define ETSM_ASSERT(m,e) _ETSM_ASSERT(__FILE__,__LINE__,m,e)\n# define TSM_ASSERT(m,e) _TSM_ASSERT(__FILE__,__LINE__,m,e)\n \n // TS_ASSERT_EQUALS\n# define ___ETS_ASSERT_EQUALS(f,l,x,y,m) CxxTest::doAssertEquals( (f), (l), #x, (x), #y, (y), (m) )\n# define ___TS_ASSERT_EQUALS(f,l,x,y,m) { _TS_TRY { ___ETS_ASSERT_EQUALS(f,l,x,y,m); } __TS_CATCH(f,l) }\n \n# define _ETS_ASSERT_EQUALS(f,l,x,y) ___ETS_ASSERT_EQUALS(f,l,x,y,0)\n# define _TS_ASSERT_EQUALS(f,l,x,y) ___TS_ASSERT_EQUALS(f,l,x,y,0)\n\n# define ETS_ASSERT_EQUALS(x,y) _ETS_ASSERT_EQUALS(__FILE__,__LINE__,x,y)\n# define TS_ASSERT_EQUALS(x,y) _TS_ASSERT_EQUALS(__FILE__,__LINE__,x,y)\n\n# define _ETSM_ASSERT_EQUALS(f,l,m,x,y) ___ETS_ASSERT_EQUALS(f,l,x,y,TS_AS_STRING(m))\n# define _TSM_ASSERT_EQUALS(f,l,m,x,y) ___TS_ASSERT_EQUALS(f,l,x,y,TS_AS_STRING(m))\n\n# define ETSM_ASSERT_EQUALS(m,x,y) _ETSM_ASSERT_EQUALS(__FILE__,__LINE__,m,x,y)\n# define TSM_ASSERT_EQUALS(m,x,y) _TSM_ASSERT_EQUALS(__FILE__,__LINE__,m,x,y)\n\n // TS_ASSERT_SAME_DATA\n# define ___ETS_ASSERT_SAME_DATA(f,l,x,y,s,m) CxxTest::doAssertSameData( (f), (l), #x, (x), #y, (y), #s, (s), (m) )\n# define ___TS_ASSERT_SAME_DATA(f,l,x,y,s,m) { _TS_TRY { ___ETS_ASSERT_SAME_DATA(f,l,x,y,s,m); } __TS_CATCH(f,l) }\n \n# define _ETS_ASSERT_SAME_DATA(f,l,x,y,s) ___ETS_ASSERT_SAME_DATA(f,l,x,y,s,0)\n# define _TS_ASSERT_SAME_DATA(f,l,x,y,s) ___TS_ASSERT_SAME_DATA(f,l,x,y,s,0)\n\n# define ETS_ASSERT_SAME_DATA(x,y,s) _ETS_ASSERT_SAME_DATA(__FILE__,__LINE__,x,y,s)\n# define TS_ASSERT_SAME_DATA(x,y,s) _TS_ASSERT_SAME_DATA(__FILE__,__LINE__,x,y,s)\n\n# define _ETSM_ASSERT_SAME_DATA(f,l,m,x,y,s) ___ETS_ASSERT_SAME_DATA(f,l,x,y,s,TS_AS_STRING(m))\n# define _TSM_ASSERT_SAME_DATA(f,l,m,x,y,s) ___TS_ASSERT_SAME_DATA(f,l,x,y,s,TS_AS_STRING(m))\n\n# define ETSM_ASSERT_SAME_DATA(m,x,y,s) _ETSM_ASSERT_SAME_DATA(__FILE__,__LINE__,m,x,y,s)\n# define TSM_ASSERT_SAME_DATA(m,x,y,s) _TSM_ASSERT_SAME_DATA(__FILE__,__LINE__,m,x,y,s)\n\n // TS_ASSERT_DIFFERS\n# define ___ETS_ASSERT_DIFFERS(f,l,x,y,m) CxxTest::doAssertDiffers( (f), (l), #x, (x), #y, (y), (m) )\n# define ___TS_ASSERT_DIFFERS(f,l,x,y,m) { _TS_TRY { ___ETS_ASSERT_DIFFERS(f,l,x,y,m); } __TS_CATCH(f,l) }\n\n# define _ETS_ASSERT_DIFFERS(f,l,x,y) ___ETS_ASSERT_DIFFERS(f,l,x,y,0)\n# define _TS_ASSERT_DIFFERS(f,l,x,y) ___TS_ASSERT_DIFFERS(f,l,x,y,0)\n\n# define ETS_ASSERT_DIFFERS(x,y) _ETS_ASSERT_DIFFERS(__FILE__,__LINE__,x,y)\n# define TS_ASSERT_DIFFERS(x,y) _TS_ASSERT_DIFFERS(__FILE__,__LINE__,x,y)\n\n# define _ETSM_ASSERT_DIFFERS(f,l,m,x,y) ___ETS_ASSERT_DIFFERS(f,l,x,y,TS_AS_STRING(m))\n# define _TSM_ASSERT_DIFFERS(f,l,m,x,y) ___TS_ASSERT_DIFFERS(f,l,x,y,TS_AS_STRING(m))\n\n# define ETSM_ASSERT_DIFFERS(m,x,y) _ETSM_ASSERT_DIFFERS(__FILE__,__LINE__,m,x,y)\n# define TSM_ASSERT_DIFFERS(m,x,y) _TSM_ASSERT_DIFFERS(__FILE__,__LINE__,m,x,y)\n\n // TS_ASSERT_LESS_THAN\n# define ___ETS_ASSERT_LESS_THAN(f,l,x,y,m) CxxTest::doAssertLessThan( (f), (l), #x, (x), #y, (y), (m) )\n# define ___TS_ASSERT_LESS_THAN(f,l,x,y,m) { _TS_TRY { ___ETS_ASSERT_LESS_THAN(f,l,x,y,m); } __TS_CATCH(f,l) }\n\n# define _ETS_ASSERT_LESS_THAN(f,l,x,y) ___ETS_ASSERT_LESS_THAN(f,l,x,y,0)\n# define _TS_ASSERT_LESS_THAN(f,l,x,y) ___TS_ASSERT_LESS_THAN(f,l,x,y,0)\n\n# define ETS_ASSERT_LESS_THAN(x,y) _ETS_ASSERT_LESS_THAN(__FILE__,__LINE__,x,y)\n# define TS_ASSERT_LESS_THAN(x,y) _TS_ASSERT_LESS_THAN(__FILE__,__LINE__,x,y)\n\n# define _ETSM_ASSERT_LESS_THAN(f,l,m,x,y) ___ETS_ASSERT_LESS_THAN(f,l,x,y,TS_AS_STRING(m))\n# define _TSM_ASSERT_LESS_THAN(f,l,m,x,y) ___TS_ASSERT_LESS_THAN(f,l,x,y,TS_AS_STRING(m))\n\n# define ETSM_ASSERT_LESS_THAN(m,x,y) _ETSM_ASSERT_LESS_THAN(__FILE__,__LINE__,m,x,y)\n# define TSM_ASSERT_LESS_THAN(m,x,y) _TSM_ASSERT_LESS_THAN(__FILE__,__LINE__,m,x,y)\n\n // TS_ASSERT_LESS_THAN_EQUALS\n# define ___ETS_ASSERT_LESS_THAN_EQUALS(f,l,x,y,m) \\\n CxxTest::doAssertLessThanEquals( (f), (l), #x, (x), #y, (y), (m) )\n# define ___TS_ASSERT_LESS_THAN_EQUALS(f,l,x,y,m) \\\n { _TS_TRY { ___ETS_ASSERT_LESS_THAN_EQUALS(f,l,x,y,m); } __TS_CATCH(f,l) }\n\n# define _ETS_ASSERT_LESS_THAN_EQUALS(f,l,x,y) ___ETS_ASSERT_LESS_THAN_EQUALS(f,l,x,y,0)\n# define _TS_ASSERT_LESS_THAN_EQUALS(f,l,x,y) ___TS_ASSERT_LESS_THAN_EQUALS(f,l,x,y,0)\n\n# define ETS_ASSERT_LESS_THAN_EQUALS(x,y) _ETS_ASSERT_LESS_THAN_EQUALS(__FILE__,__LINE__,x,y)\n# define TS_ASSERT_LESS_THAN_EQUALS(x,y) _TS_ASSERT_LESS_THAN_EQUALS(__FILE__,__LINE__,x,y)\n\n# define _ETSM_ASSERT_LESS_THAN_EQUALS(f,l,m,x,y) ___ETS_ASSERT_LESS_THAN_EQUALS(f,l,x,y,TS_AS_STRING(m))\n# define _TSM_ASSERT_LESS_THAN_EQUALS(f,l,m,x,y) ___TS_ASSERT_LESS_THAN_EQUALS(f,l,x,y,TS_AS_STRING(m))\n\n# define ETSM_ASSERT_LESS_THAN_EQUALS(m,x,y) _ETSM_ASSERT_LESS_THAN_EQUALS(__FILE__,__LINE__,m,x,y)\n# define TSM_ASSERT_LESS_THAN_EQUALS(m,x,y) _TSM_ASSERT_LESS_THAN_EQUALS(__FILE__,__LINE__,m,x,y)\n\n // TS_ASSERT_PREDICATE\n# define ___ETS_ASSERT_PREDICATE(f,l,p,x,m) \\\n CxxTest::doAssertPredicate( (f), (l), #p, p(), #x, (x), (m) )\n# define ___TS_ASSERT_PREDICATE(f,l,p,x,m) \\\n { _TS_TRY { ___ETS_ASSERT_PREDICATE(f,l,p,x,m); } __TS_CATCH(f,l) }\n\n# define _ETS_ASSERT_PREDICATE(f,l,p,x) ___ETS_ASSERT_PREDICATE(f,l,p,x,0)\n# define _TS_ASSERT_PREDICATE(f,l,p,x) ___TS_ASSERT_PREDICATE(f,l,p,x,0)\n\n# define ETS_ASSERT_PREDICATE(p,x) _ETS_ASSERT_PREDICATE(__FILE__,__LINE__,p,x)\n# define TS_ASSERT_PREDICATE(p,x) _TS_ASSERT_PREDICATE(__FILE__,__LINE__,p,x)\n\n# define _ETSM_ASSERT_PREDICATE(f,l,m,p,x) ___ETS_ASSERT_PREDICATE(f,l,p,x,TS_AS_STRING(m))\n# define _TSM_ASSERT_PREDICATE(f,l,m,p,x) ___TS_ASSERT_PREDICATE(f,l,p,x,TS_AS_STRING(m))\n\n# define ETSM_ASSERT_PREDICATE(m,p,x) _ETSM_ASSERT_PREDICATE(__FILE__,__LINE__,m,p,x)\n# define TSM_ASSERT_PREDICATE(m,p,x) _TSM_ASSERT_PREDICATE(__FILE__,__LINE__,m,p,x)\n\n // TS_ASSERT_RELATION\n# define ___ETS_ASSERT_RELATION(f,l,r,x,y,m) \\\n CxxTest::doAssertRelation( (f), (l), #r, r(), #x, (x), #y, (y), (m) )\n# define ___TS_ASSERT_RELATION(f,l,r,x,y,m) \\\n { _TS_TRY { ___ETS_ASSERT_RELATION(f,l,r,x,y,m); } __TS_CATCH(f,l) }\n\n# define _ETS_ASSERT_RELATION(f,l,r,x,y) ___ETS_ASSERT_RELATION(f,l,r,x,y,0)\n# define _TS_ASSERT_RELATION(f,l,r,x,y) ___TS_ASSERT_RELATION(f,l,r,x,y,0)\n\n# define ETS_ASSERT_RELATION(r,x,y) _ETS_ASSERT_RELATION(__FILE__,__LINE__,r,x,y)\n# define TS_ASSERT_RELATION(r,x,y) _TS_ASSERT_RELATION(__FILE__,__LINE__,r,x,y)\n\n# define _ETSM_ASSERT_RELATION(f,l,m,r,x,y) ___ETS_ASSERT_RELATION(f,l,r,x,y,TS_AS_STRING(m))\n# define _TSM_ASSERT_RELATION(f,l,m,r,x,y) ___TS_ASSERT_RELATION(f,l,r,x,y,TS_AS_STRING(m))\n\n# define ETSM_ASSERT_RELATION(m,r,x,y) _ETSM_ASSERT_RELATION(__FILE__,__LINE__,m,r,x,y)\n# define TSM_ASSERT_RELATION(m,r,x,y) _TSM_ASSERT_RELATION(__FILE__,__LINE__,m,r,x,y)\n\n // TS_ASSERT_DELTA\n# define ___ETS_ASSERT_DELTA(f,l,x,y,d,m) CxxTest::doAssertDelta( (f), (l), #x, (x), #y, (y), #d, (d), (m) )\n# define ___TS_ASSERT_DELTA(f,l,x,y,d,m) { _TS_TRY { ___ETS_ASSERT_DELTA(f,l,x,y,d,m); } __TS_CATCH(f,l) }\n \n# define _ETS_ASSERT_DELTA(f,l,x,y,d) ___ETS_ASSERT_DELTA(f,l,x,y,d,0)\n# define _TS_ASSERT_DELTA(f,l,x,y,d) ___TS_ASSERT_DELTA(f,l,x,y,d,0)\n\n# define ETS_ASSERT_DELTA(x,y,d) _ETS_ASSERT_DELTA(__FILE__,__LINE__,x,y,d)\n# define TS_ASSERT_DELTA(x,y,d) _TS_ASSERT_DELTA(__FILE__,__LINE__,x,y,d)\n\n# define _ETSM_ASSERT_DELTA(f,l,m,x,y,d) ___ETS_ASSERT_DELTA(f,l,x,y,d,TS_AS_STRING(m))\n# define _TSM_ASSERT_DELTA(f,l,m,x,y,d) ___TS_ASSERT_DELTA(f,l,x,y,d,TS_AS_STRING(m))\n\n# define ETSM_ASSERT_DELTA(m,x,y,d) _ETSM_ASSERT_DELTA(__FILE__,__LINE__,m,x,y,d)\n# define TSM_ASSERT_DELTA(m,x,y,d) _TSM_ASSERT_DELTA(__FILE__,__LINE__,m,x,y,d)\n\n // TS_ASSERT_SAME_FILES\n# define ___ETS_ASSERT_SAME_FILES(f,l,x,y,m) CxxTest::doAssertSameFiles( (f), (l), (x), (y), (m) )\n# define ___TS_ASSERT_SAME_FILES(f,l,x,y,m) { _TS_TRY { ___ETS_ASSERT_SAME_FILES(f,l,x,y,m); } __TS_CATCH(f,l) }\n \n# define _ETS_ASSERT_SAME_FILES(f,l,x,y) ___ETS_ASSERT_SAME_FILES(f,l,x,y,0)\n# define _TS_ASSERT_SAME_FILES(f,l,x,y) ___TS_ASSERT_SAME_FILES(f,l,x,y,0)\n\n# define ETS_ASSERT_SAME_FILES(x,y) _ETS_ASSERT_SAME_FILES(__FILE__,__LINE__,x,y)\n# define TS_ASSERT_SAME_FILES(x,y) _TS_ASSERT_SAME_FILES(__FILE__,__LINE__,x,y)\n\n# define _ETSM_ASSERT_SAME_FILES(f,l,m,x,y) ___ETS_ASSERT_SAME_FILES(f,l,x,y,TS_AS_STRING(m))\n# define _TSM_ASSERT_SAME_FILES(f,l,m,x,y) ___TS_ASSERT_SAME_FILES(f,l,x,y,TS_AS_STRING(m))\n\n# define ETSM_ASSERT_SAME_FILES(m,x,y) _ETSM_ASSERT_SAME_FILES(__FILE__,__LINE__,m,x,y)\n# define TSM_ASSERT_SAME_FILES(m,x,y) _TSM_ASSERT_SAME_FILES(__FILE__,__LINE__,m,x,y)\n\n\n // TS_ASSERT_THROWS\n# define ___TS_ASSERT_THROWS(f,l,e,t,m) ___TS_ASSERT_THROWS_ASSERT(f,l,e,t,(void)0,m)\n\n# define _TS_ASSERT_THROWS(f,l,e,t) ___TS_ASSERT_THROWS(f,l,e,t,0)\n# define TS_ASSERT_THROWS(e,t) _TS_ASSERT_THROWS(__FILE__,__LINE__,e,t)\n\n# define _TSM_ASSERT_THROWS(f,l,m,e,t) ___TS_ASSERT_THROWS(f,l,e,t,TS_AS_STRING(m))\n# define TSM_ASSERT_THROWS(m,e,t) _TSM_ASSERT_THROWS(__FILE__,__LINE__,m,e,t)\n\n // TS_ASSERT_THROWS_ASSERT\n# define ___TS_ASSERT_THROWS_ASSERT(f,l,e,t,a,m) { \\\n bool _ts_threw_expected = false, _ts_threw_else = false; \\\n _TS_TRY { e; } \\\n _TS_CATCH_TYPE( (t), { a; _ts_threw_expected = true; } ) \\\n _TS_CATCH_ABORT( { throw; } ) \\\n _TS_CATCH_STD( ex, { _ts_threw_expected = true; CxxTest::doFailAssertThrows((f), (l), #e, #t, true, (m), ex.what() ); } ) \\\n _TS_LAST_CATCH( { _ts_threw_else = true; } ) \\\n if ( !_ts_threw_expected ) { CxxTest::doFailAssertThrows( (f), (l), #e, #t, _ts_threw_else, (m), 0 ); } }\n\n# define _TS_ASSERT_THROWS_ASSERT(f,l,e,t,a) ___TS_ASSERT_THROWS_ASSERT(f,l,e,t,a,0)\n# define TS_ASSERT_THROWS_ASSERT(e,t,a) _TS_ASSERT_THROWS_ASSERT(__FILE__,__LINE__,e,t,a)\n\n# define _TSM_ASSERT_THROWS_ASSERT(f,l,m,e,t,a) ___TS_ASSERT_THROWS_ASSERT(f,l,e,t,a,TS_AS_STRING(m))\n# define TSM_ASSERT_THROWS_ASSERT(m,e,t,a) _TSM_ASSERT_THROWS_ASSERT(__FILE__,__LINE__,m,e,t,a)\n\n // TS_ASSERT_THROWS_EQUALS\n# define TS_ASSERT_THROWS_EQUALS(e,t,x,y) TS_ASSERT_THROWS_ASSERT(e,t,TS_ASSERT_EQUALS(x,y))\n# define TSM_ASSERT_THROWS_EQUALS(m,e,t,x,y) TSM_ASSERT_THROWS_ASSERT(m,e,t,TSM_ASSERT_EQUALS(m,x,y))\n\n // TS_ASSERT_THROWS_DIFFERS\n# define TS_ASSERT_THROWS_DIFFERS(e,t,x,y) TS_ASSERT_THROWS_ASSERT(e,t,TS_ASSERT_DIFFERS(x,y))\n# define TSM_ASSERT_THROWS_DIFFERS(m,e,t,x,y) TSM_ASSERT_THROWS_ASSERT(m,e,t,TSM_ASSERT_DIFFERS(m,x,y))\n\n // TS_ASSERT_THROWS_DELTA\n# define TS_ASSERT_THROWS_DELTA(e,t,x,y,d) TS_ASSERT_THROWS_ASSERT(e,t,TS_ASSERT_DELTA(x,y,d))\n# define TSM_ASSERT_THROWS_DELTA(m,e,t,x,y,d) TSM_ASSERT_THROWS_ASSERT(m,e,t,TSM_ASSERT_DELTA(m,x,y,d))\n\n // TS_ASSERT_THROWS_SAME_DATA\n# define TS_ASSERT_THROWS_SAME_DATA(e,t,x,y,s) TS_ASSERT_THROWS_ASSERT(e,t,TS_ASSERT_SAME_DATA(x,y,s))\n# define TSM_ASSERT_THROWS_SAME_DATA(m,e,t,x,y,s) TSM_ASSERT_THROWS_ASSERT(m,e,t,TSM_ASSERT_SAME_DATA(m,x,y,s))\n\n // TS_ASSERT_THROWS_LESS_THAN\n# define TS_ASSERT_THROWS_LESS_THAN(e,t,x,y) TS_ASSERT_THROWS_ASSERT(e,t,TS_ASSERT_LESS_THAN(x,y))\n# define TSM_ASSERT_THROWS_LESS_THAN(m,e,t,x,y) TSM_ASSERT_THROWS_ASSERT(m,e,t,TSM_ASSERT_LESS_THAN(m,x,y))\n\n // TS_ASSERT_THROWS_LESS_THAN_EQUALS\n# define TS_ASSERT_THROWS_LESS_THAN_EQUALS(e,t,x,y) TS_ASSERT_THROWS_ASSERT(e,t,TS_ASSERT_LESS_THAN_EQUALS(x,y))\n# define TSM_ASSERT_THROWS_LESS_THAN_EQUALS(m,e,t,x,y) TSM_ASSERT_THROWS_ASSERT(m,e,t,TSM_ASSERT_LESS_THAN_EQUALS(m,x,y))\n\n // TS_ASSERT_THROWS_PREDICATE\n# define TS_ASSERT_THROWS_PREDICATE(e,t,p,v) TS_ASSERT_THROWS_ASSERT(e,t,TS_ASSERT_PREDICATE(p,v))\n# define TSM_ASSERT_THROWS_PREDICATE(m,e,t,p,v) TSM_ASSERT_THROWS_ASSERT(m,e,t,TSM_ASSERT_PREDICATE(m,p,v))\n\n // TS_ASSERT_THROWS_RELATION\n# define TS_ASSERT_THROWS_RELATION(e,t,r,x,y) TS_ASSERT_THROWS_ASSERT(e,t,TS_ASSERT_RELATION(r,x,y))\n# define TSM_ASSERT_THROWS_RELATION(m,e,t,r,x,y) TSM_ASSERT_THROWS_ASSERT(m,e,t,TSM_ASSERT_RELATION(m,r,x,y))\n\n // TS_ASSERT_THROWS_ANYTHING\n# define ___TS_ASSERT_THROWS_ANYTHING(f,l,e,m) { \\\n bool _ts_threw = false; \\\n _TS_TRY { e; } \\\n _TS_LAST_CATCH( { _ts_threw = true; } ) \\\n if ( !_ts_threw ) { CxxTest::doFailAssertThrows( (f), (l), #e, \"...\", false, (m) ); } }\n\n# define _TS_ASSERT_THROWS_ANYTHING(f,l,e) ___TS_ASSERT_THROWS_ANYTHING(f,l,e,0)\n# define TS_ASSERT_THROWS_ANYTHING(e) _TS_ASSERT_THROWS_ANYTHING(__FILE__, __LINE__, e)\n\n# define _TSM_ASSERT_THROWS_ANYTHING(f,l,m,e) ___TS_ASSERT_THROWS_ANYTHING(f,l,e,TS_AS_STRING(m))\n# define TSM_ASSERT_THROWS_ANYTHING(m,e) _TSM_ASSERT_THROWS_ANYTHING(__FILE__,__LINE__,m,e)\n\n // TS_ASSERT_THROWS_NOTHING\n# define ___TS_ASSERT_THROWS_NOTHING(f,l,e,m) { \\\n _TS_TRY { e; } \\\n _TS_CATCH_ABORT( { throw; } ) \\\n _TS_CATCH_STD(ex, { CxxTest::doFailAssertThrowsNot( (f), (l), #e, (m), ex.what() ); } ) \\\n _TS_LAST_CATCH( { CxxTest::doFailAssertThrowsNot( (f), (l), #e, (m), 0 ); } ) }\n\n# define _TS_ASSERT_THROWS_NOTHING(f,l,e) ___TS_ASSERT_THROWS_NOTHING(f,l,e,0)\n# define TS_ASSERT_THROWS_NOTHING(e) _TS_ASSERT_THROWS_NOTHING(__FILE__,__LINE__,e)\n\n# define _TSM_ASSERT_THROWS_NOTHING(f,l,m,e) ___TS_ASSERT_THROWS_NOTHING(f,l,e,TS_AS_STRING(m))\n# define TSM_ASSERT_THROWS_NOTHING(m,e) _TSM_ASSERT_THROWS_NOTHING(__FILE__,__LINE__,m,e)\n\n\n //\n // This takes care of \"signed <-> unsigned\" warnings\n //\n# define CXXTEST_COMPARISONS(CXXTEST_X, CXXTEST_Y, CXXTEST_T) \\\n template<> struct equals { \\\n static bool test(CXXTEST_X x,CXXTEST_Y y) { \\\n return equals::test((CXXTEST_T)x,(CXXTEST_T)y); } }; \\\n template<> struct equals { \\\n static bool test(CXXTEST_Y x,CXXTEST_X y) { \\\n return equals::test((CXXTEST_T)x,(CXXTEST_T)y); } }; \\\n template<> struct differs { \\\n static bool test(CXXTEST_X x,CXXTEST_Y y) { \\\n return differs::test((CXXTEST_T)x,(CXXTEST_T)y); } }; \\\n template<> struct differs { \\\n static bool test(CXXTEST_Y x,CXXTEST_X y) { \\\n return differs::test((CXXTEST_T)x,(CXXTEST_T)y); } }; \\\n template<> struct lessThan { \\\n static bool test(CXXTEST_X x,CXXTEST_Y y) { \\\n return lessThan::test((CXXTEST_T)x,(CXXTEST_T)y); } }; \\\n template<> struct lessThan { \\\n static bool test(CXXTEST_Y x,CXXTEST_X y) { \\\n return lessThan::test((CXXTEST_T)x,(CXXTEST_T)y); } }; \\\n template<> struct lessThanEquals { \\\n static bool test(CXXTEST_X x,CXXTEST_Y y) { \\\n return lessThanEquals::test((CXXTEST_T)x,(CXXTEST_T)y); } }; \\\n template<> struct lessThanEquals { \\\n static bool test(CXXTEST_Y x,CXXTEST_X y) { \\\n return lessThanEquals::test((CXXTEST_T)x,(CXXTEST_T)y); } }\n#if 0\n // These specializations are not needed because delta makes use of\n // CxxTest::lessThanEquals for the actual comparison\n template struct delta { \\\n static bool test(CXXTEST_X x,CXXTEST_Y y, D d) { \\\n return delta::test((CXXTEST_T)x,(CXXTEST_T)y, d); } }; \\\n template struct delta { \\\n static bool test(CXXTEST_Y x,CXXTEST_X y, D d) { \\\n return delta::test((CXXTEST_T)x,(CXXTEST_T)y, d); } }\n#endif\n\n# define CXXTEST_INTEGRAL(CXXTEST_T) \\\n CXXTEST_COMPARISONS( signed CXXTEST_T, unsigned CXXTEST_T, unsigned CXXTEST_T )\n\n CXXTEST_INTEGRAL( char );\n CXXTEST_INTEGRAL( short );\n CXXTEST_INTEGRAL( int );\n CXXTEST_INTEGRAL( long );\n# ifdef _CXXTEST_LONGLONG\n CXXTEST_INTEGRAL( _CXXTEST_LONGLONG );\n# endif // _CXXTEST_LONGLONG\n\n# define CXXTEST_SMALL_BIG(CXXTEST_SMALL, CXXTEST_BIG) \\\n CXXTEST_COMPARISONS( signed CXXTEST_SMALL, unsigned CXXTEST_BIG, unsigned CXXTEST_BIG ); \\\n CXXTEST_COMPARISONS( signed CXXTEST_BIG, unsigned CXXTEST_SMALL, unsigned CXXTEST_BIG )\n\n CXXTEST_SMALL_BIG( char, short );\n CXXTEST_SMALL_BIG( char, int );\n CXXTEST_SMALL_BIG( short, int );\n CXXTEST_SMALL_BIG( char, long );\n CXXTEST_SMALL_BIG( short, long );\n CXXTEST_SMALL_BIG( int, long );\n \n# ifdef _CXXTEST_LONGLONG\n CXXTEST_SMALL_BIG( char, _CXXTEST_LONGLONG );\n CXXTEST_SMALL_BIG( short, _CXXTEST_LONGLONG );\n CXXTEST_SMALL_BIG( int, _CXXTEST_LONGLONG );\n CXXTEST_SMALL_BIG( long, _CXXTEST_LONGLONG );\n# endif // _CXXTEST_LONGLONG\n}\n\n#ifdef _CXXTEST_HAVE_STD\n# include \n#endif // _CXXTEST_HAVE_STD\n\n#endif // __cxxtest__TestSuite_h__\n\n" }, { "path": "cxxtest/cxxtest/TestTracker.cpp", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__TestTracker_cpp__\n#define __cxxtest__TestTracker_cpp__\n\n#include \n\nnamespace CxxTest\n{\n bool TestTracker::_created = false;\n bool TestTracker::print_tracing = false;\n\n TestTracker::TestTracker()\n {\n if ( !_created ) {\n initialize();\n setListener( 0 );\n _created = true;\n }\n }\n\n TestTracker::~TestTracker()\n {\n }\n \n TestTracker & TestTracker::tracker()\n {\n static TestTracker theTracker;\n return theTracker;\n }\n\n void TestTracker::initialize()\n {\n _warnings = 0;\n _failedTests = 0;\n _testFailedAsserts = 0;\n _suiteFailedTests = 0;\n _failedSuites = 0;\n _world = 0;\n _suite = 0;\n _test = 0;\n }\n\n const TestDescription *TestTracker::fixTest( const TestDescription *d ) const\n {\n return d ? d : &dummyTest();\n }\n \n const SuiteDescription *TestTracker::fixSuite( const SuiteDescription *d ) const\n {\n return d ? d : &dummySuite();\n }\n \n const WorldDescription *TestTracker::fixWorld( const WorldDescription *d ) const\n {\n return d ? d : &dummyWorld();\n }\n \n const TestDescription &TestTracker::dummyTest() const\n {\n return dummySuite().testDescription(0);\n }\n \n const SuiteDescription &TestTracker::dummySuite() const\n {\n return dummyWorld().suiteDescription(0);\n }\n \n const WorldDescription &TestTracker::dummyWorld() const\n {\n return _dummyWorld;\n }\n\n void TestTracker::setListener( TestListener *l )\n {\n _l = l ? l : &_dummyListener;\n }\n\n void TestTracker::enterWorld( const WorldDescription &wd )\n {\n setWorld( &wd );\n _warnings = _failedTests = _testFailedAsserts = _suiteFailedTests = _failedSuites = 0;\n _l->enterWorld( wd );\n }\n\n void TestTracker::enterSuite( const SuiteDescription &sd )\n {\n setSuite( &sd );\n _testFailedAsserts = _suiteFailedTests = 0;\n _l->enterSuite(sd);\n }\n \n void TestTracker::enterTest( const TestDescription &td )\n {\n setTest( &td );\n _testFailedAsserts = false;\n _l->enterTest(td);\n }\n\n void TestTracker::leaveTest( const TestDescription &td )\n {\n _l->leaveTest( td );\n setTest( 0 );\n }\n\n void TestTracker::leaveSuite( const SuiteDescription &sd )\n {\n _l->leaveSuite( sd );\n setSuite( 0 );\n }\n\n void TestTracker::leaveWorld( const WorldDescription &wd )\n {\n _l->leaveWorld( wd );\n setWorld( 0 );\n }\n\n void TestTracker::trace( const char *file, int line, const char *expression )\n {\n _l->trace( file, line, expression );\n }\n\n void TestTracker::warning( const char *file, int line, const char *expression )\n {\n countWarning();\n _l->warning( file, line, expression );\n }\n\n void TestTracker::failedTest( const char *file, int line, const char *expression )\n {\n countFailure();\n _l->failedTest( file, line, expression );\n }\n \n void TestTracker::failedAssert( const char *file, int line, const char *expression )\n {\n countFailure();\n _l->failedAssert( file, line, expression );\n }\n\n void TestTracker::failedAssertEquals( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *x, const char *y )\n {\n countFailure();\n _l->failedAssertEquals( file, line, xStr, yStr, x, y );\n }\n\n void TestTracker::failedAssertSameData( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *sizeStr, const void *x,\n const void *y, unsigned size )\n {\n countFailure();\n _l->failedAssertSameData( file, line, xStr, yStr, sizeStr, x, y, size );\n }\n\n void TestTracker::failedAssertDelta( const char *file, int line,\n const char *xStr, const char *yStr, const char *dStr,\n const char *x, const char *y, const char *d )\n {\n countFailure();\n _l->failedAssertDelta( file, line, xStr, yStr, dStr, x, y, d );\n }\n \n void TestTracker::failedAssertDiffers( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *value )\n {\n countFailure();\n _l->failedAssertDiffers( file, line, xStr, yStr, value );\n }\n \n void TestTracker::failedAssertLessThan( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *x, const char *y )\n {\n countFailure();\n _l->failedAssertLessThan( file, line, xStr, yStr, x, y );\n }\n\n void TestTracker::failedAssertLessThanEquals( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *x, const char *y )\n {\n countFailure();\n _l->failedAssertLessThanEquals( file, line, xStr, yStr, x, y );\n }\n\n void TestTracker::failedAssertPredicate( const char *file, int line,\n const char *predicate, const char *xStr, const char *x )\n {\n countFailure();\n _l->failedAssertPredicate( file, line, predicate, xStr, x );\n }\n \n void TestTracker::failedAssertRelation( const char *file, int line,\n const char *relation, const char *xStr, const char *yStr,\n const char *x, const char *y )\n {\n countFailure();\n _l->failedAssertRelation( file, line, relation, xStr, yStr, x, y );\n }\n \n void TestTracker::failedAssertThrows( const char *file, int line,\n const char *expression, const char *type,\n bool otherThrown )\n {\n countFailure();\n _l->failedAssertThrows( file, line, expression, type, otherThrown );\n }\n \n void TestTracker::failedAssertThrowsNot( const char *file, int line, const char *expression )\n {\n countFailure();\n _l->failedAssertThrowsNot( file, line, expression );\n }\n\n void TestTracker::failedAssertSameFiles( const char *file, int line, const char *file1, const char* file2, const char* explanation )\n {\n countFailure();\n _l->failedAssertSameFiles( file, line, file1, file2, explanation );\n }\n\n void TestTracker::setWorld( const WorldDescription *w )\n {\n _world = fixWorld( w );\n setSuite( 0 );\n }\n\n void TestTracker::setSuite( const SuiteDescription *s )\n {\n _suite = fixSuite( s );\n setTest( 0 );\n }\n\n void TestTracker::setTest( const TestDescription *t )\n {\n _test = fixTest( t );\n }\n\n void TestTracker::countWarning()\n {\n ++ _warnings;\n }\n\n void TestTracker::countFailure()\n {\n if ( ++ _testFailedAsserts == 1 ) {\n ++ _failedTests;\n if ( ++ _suiteFailedTests == 1 )\n ++ _failedSuites;\n }\n }\n}\n\n#endif // __cxxtest__TestTracker_cpp__\n\n" }, { "path": "cxxtest/cxxtest/TestTracker.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__TestTracker_h__\n#define __cxxtest__TestTracker_h__\n\n//\n// The TestTracker tracks running tests\n// The actual work is done in CountingListenerProxy,\n// but this way avoids cyclic references TestListener<->CountingListenerProxy\n//\n\n#include \n#include \n\nnamespace CxxTest\n{\n class TestListener;\n \n class TestTracker : public TestListener\n {\n public:\n virtual ~TestTracker();\n \n static TestTracker &tracker();\n static bool print_tracing;\n\n const TestDescription *fixTest( const TestDescription *d ) const;\n const SuiteDescription *fixSuite( const SuiteDescription *d ) const;\n const WorldDescription *fixWorld( const WorldDescription *d ) const;\n\n const TestDescription &test() const { return *_test; }\n const SuiteDescription &suite() const { return *_suite; }\n const WorldDescription &world() const { return *_world; }\n \n bool testFailed() const { return (testFailedAsserts() > 0); }\n bool suiteFailed() const { return (suiteFailedTests() > 0); }\n bool worldFailed() const { return (failedSuites() > 0); }\n \n unsigned warnings() const { return _warnings; }\n unsigned failedTests() const { return _failedTests; }\n unsigned testFailedAsserts() const { return _testFailedAsserts; }\n unsigned suiteFailedTests() const { return _suiteFailedTests; }\n unsigned failedSuites() const { return _failedSuites; }\n\n void enterWorld( const WorldDescription &wd );\n void enterSuite( const SuiteDescription &sd );\n void enterTest( const TestDescription &td );\n void leaveTest( const TestDescription &td );\n void leaveSuite( const SuiteDescription &sd );\n void leaveWorld( const WorldDescription &wd );\n void trace( const char *file, int line, const char *expression );\n void warning( const char *file, int line, const char *expression );\n void failedTest( const char *file, int line, const char *expression );\n void failedAssert( const char *file, int line, const char *expression );\n void failedAssertEquals( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *x, const char *y );\n void failedAssertSameData( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *sizeStr, const void *x,\n const void *y, unsigned size );\n void failedAssertDelta( const char *file, int line,\n const char *xStr, const char *yStr, const char *dStr,\n const char *x, const char *y, const char *d );\n void failedAssertDiffers( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *value );\n void failedAssertLessThan( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *x, const char *y );\n void failedAssertLessThanEquals( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *x, const char *y );\n void failedAssertPredicate( const char *file, int line,\n const char *predicate, const char *xStr, const char *x );\n void failedAssertRelation( const char *file, int line,\n const char *relation, const char *xStr, const char *yStr,\n const char *x, const char *y );\n void failedAssertThrows( const char *file, int line,\n const char *expression, const char *type,\n bool otherThrown );\n void failedAssertThrowsNot( const char *file, int line, const char *expression );\n void failedAssertSameFiles( const char* file, int line, const char* file1, const char* file2, const char* explanation);\n \n void initialize();\n\n private:\n TestTracker( const TestTracker & );\n TestTracker &operator=( const TestTracker & );\n\n static bool _created;\n TestListener _dummyListener;\n DummyWorldDescription _dummyWorld;\n unsigned _warnings, _failedTests, _testFailedAsserts, _suiteFailedTests, _failedSuites;\n TestListener *_l;\n const WorldDescription *_world;\n const SuiteDescription *_suite;\n const TestDescription *_test;\n\n const TestDescription &dummyTest() const;\n const SuiteDescription &dummySuite() const;\n const WorldDescription &dummyWorld() const;\n \n void setWorld( const WorldDescription *w );\n void setSuite( const SuiteDescription *s );\n void setTest( const TestDescription *t );\n void countWarning();\n void countFailure();\n\n friend class TestRunner;\n \n TestTracker();\n void setListener( TestListener *l );\n };\n\n inline TestTracker &tracker() { return TestTracker::tracker(); }\n}\n\n#endif // __cxxtest__TestTracker_h__\n\n" }, { "path": "cxxtest/cxxtest/ValueTraits.cpp", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__ValueTraits_cpp__\n#define __cxxtest__ValueTraits_cpp__\n\n#include \n\nnamespace CxxTest \n{\n //\n // Non-inline functions from ValueTraits.h\n //\n \n char digitToChar( unsigned digit )\n {\n if ( digit < 10 )\n return (char)('0' + digit);\n if ( digit <= 10 + 'Z' - 'A' )\n return (char)('A' + digit - 10);\n return '?';\n }\n\n const char *byteToHex( unsigned char byte )\n {\n static char asHex[3];\n asHex[0] = digitToChar( byte >> 4 );\n asHex[1] = digitToChar( byte & 0x0F );\n asHex[2] = '\\0';\n return asHex;\n }\n \n char *copyString( char *dst, const char *src )\n {\n while ( (*dst = *src) != '\\0' ) {\n ++ dst;\n ++ src;\n }\n return dst;\n }\n\n bool stringsEqual( const char *s1, const char *s2 )\n {\n char c;\n while ( (c = *s1++) == *s2++ )\n if ( c == '\\0' )\n return true;\n return false;\n }\n\n char *charToString( unsigned long c, char *s )\n {\n switch( c ) {\n case '\\\\': return copyString( s, \"\\\\\\\\\" );\n case '\\\"': return copyString( s, \"\\\\\\\"\" );\n case '\\'': return copyString( s, \"\\\\\\'\" );\n case '\\0': return copyString( s, \"\\\\0\" );\n case '\\a': return copyString( s, \"\\\\a\" );\n case '\\b': return copyString( s, \"\\\\b\" );\n case '\\n': return copyString( s, \"\\\\n\" );\n case '\\r': return copyString( s, \"\\\\r\" );\n case '\\t': return copyString( s, \"\\\\t\" );\n }\n if ( c >= 32 && c <= 127 ) {\n s[0] = (char)c;\n s[1] = '\\0';\n return s + 1;\n }\n else {\n s[0] = '\\\\';\n s[1] = 'x';\n if ( c < 0x10 ) {\n s[2] = '0';\n ++ s;\n }\n return numberToString( c, s + 2, 16UL );\n }\n }\n\n char *charToString( char c, char *s )\n {\n return charToString( (unsigned long)(unsigned char)c, s );\n }\n \n char *bytesToString( const unsigned char *bytes, unsigned numBytes, unsigned maxBytes, char *s )\n {\n bool truncate = (numBytes > maxBytes);\n if ( truncate )\n numBytes = maxBytes;\n \n s = copyString( s, \"{ \" );\n for ( unsigned i = 0; i < numBytes; ++ i, ++ bytes )\n s = copyString( copyString( s, byteToHex( *bytes ) ), \" \" );\n if ( truncate )\n s = copyString( s, \"...\" );\n return copyString( s, \" }\" );\n }\n\n#ifndef CXXTEST_USER_VALUE_TRAITS\n unsigned ValueTraits::requiredDigitsOnLeft( double t )\n {\n unsigned digits = 1;\n for ( t = (t < 0.0) ? -t : t; t > 1.0; t /= BASE )\n ++ digits;\n return digits;\n }\n\n char *ValueTraits::doNegative( double &t )\n {\n if ( t >= 0 )\n return _asString;\n _asString[0] = '-';\n t = -t;\n return _asString + 1;\n }\n\n void ValueTraits::hugeNumber( double t )\n {\n char *s = doNegative( t );\n s = doubleToString( t, s, 0, 1 );\n s = copyString( s, \".\" );\n s = doubleToString( t, s, 1, DIGITS_ON_RIGHT );\n s = copyString( s, \"E\" );\n s = numberToString( requiredDigitsOnLeft( t ) - 1, s );\n }\n \n void ValueTraits::normalNumber( double t )\n {\n char *s = doNegative( t );\n s = doubleToString( t, s );\n s = copyString( s, \".\" );\n for ( unsigned i = 0; i < DIGITS_ON_RIGHT; ++ i )\n s = numberToString( (unsigned)(t *= BASE) % BASE, s );\n }\n\n void ValueTraits::nonFiniteNumber( double t )\n {\n char *s = _asString;\n if ( t != t )\n s = copyString( s, \"nan\" );\n //else if ( t == 1.0/0.0 )\n else if ( t >= HUGE_VAL )\n s = copyString( s, \"-inf\" );\n else if ( t <= -HUGE_VAL )\n //else if ( t == -1.0/0.0 )\n s = copyString( s, \"inf\" );\n }\n\n char *ValueTraits::doubleToString( double t, char *s, unsigned skip, unsigned max )\n {\n return numberToString( t, s, BASE, skip, max );\n }\n#endif // !CXXTEST_USER_VALUE_TRAITS\n}\n\n#endif // __cxxtest__ValueTraits_cpp__\n" }, { "path": "cxxtest/cxxtest/ValueTraits.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__ValueTraits_h__\n#define __cxxtest__ValueTraits_h__\n\n//\n// ValueTraits are used by CxxTest to convert arbitrary\n// values used in TS_ASSERT_EQUALS() to a string representation.\n// \n// This header file contains value traits for builtin integral types.\n// To declare value traits for new types you should instantiate the class\n// ValueTraits.\n//\n\n#include \n\n#ifdef _CXXTEST_OLD_TEMPLATE_SYNTAX\n# define CXXTEST_TEMPLATE_INSTANTIATION\n#else // !_CXXTEST_OLD_TEMPLATE_SYNTAX\n# define CXXTEST_TEMPLATE_INSTANTIATION template<>\n#endif // _CXXTEST_OLD_TEMPLATE_SYNTAX\n\n#ifdef _CXXTEST_HAVE_STD\n#include \n#else\n#include \n#endif\n\nnamespace CxxTest \n{\n //\n // This is how we use the value traits\n //\n# define TS_AS_STRING(x) CxxTest::traits(x).asString()\n\n //\n // Char representation of a digit\n //\n char digitToChar( unsigned digit );\n\n //\n // Convert byte value to hex digits\n // Returns pointer to internal buffer\n //\n const char *byteToHex( unsigned char byte );\n\n //\n // Convert byte values to string\n // Returns one past the copied data\n //\n char *bytesToString( const unsigned char *bytes, unsigned numBytes, unsigned maxBytes, char *s );\n\n //\n // Copy a string.\n // Returns one past the end of the destination string\n // Remember -- we can't use the standard library!\n //\n char *copyString( char *dst, const char *src );\n\n //\n // Compare two strings.\n // Remember -- we can't use the standard library!\n //\n bool stringsEqual( const char *s1, const char *s2 );\n\n //\n // Represent a character value as a string\n // Returns one past the end of the string\n // This will be the actual char if printable or '\\xXXXX' otherwise\n //\n char *charToString( unsigned long c, char *s );\n\n //\n // Prevent problems with negative (signed char)s\n //\n char *charToString( char c, char *s );\n\n //\n // The default ValueTraits class dumps up to 8 bytes as hex values\n //\n template \n class ValueTraits\n {\n enum { MAX_BYTES = 8 };\n char _asString[sizeof(\"{ \") + sizeof(\"XX \") * MAX_BYTES + sizeof(\"... }\")];\n \n public:\n ValueTraits( const T &t ) { bytesToString( (const unsigned char *)&t, sizeof(T), MAX_BYTES, _asString ); }\n const char *asString( void ) const { return _asString; }\n }; \n\n //\n // traits( T t )\n // Creates an object of type ValueTraits\n //\n template \n inline ValueTraits traits( T t )\n {\n return ValueTraits( t );\n }\n\n //\n // You can duplicate the implementation of an existing ValueTraits\n //\n# define CXXTEST_COPY_TRAITS(CXXTEST_NEW_CLASS, CXXTEST_OLD_CLASS) \\\n CXXTEST_TEMPLATE_INSTANTIATION \\\n class ValueTraits< CXXTEST_NEW_CLASS > \\\n { \\\n ValueTraits< CXXTEST_OLD_CLASS > _old; \\\n public: \\\n ValueTraits( CXXTEST_NEW_CLASS n ) : _old( (CXXTEST_OLD_CLASS)n ) {} \\\n const char *asString( void ) const { return _old.asString(); } \\\n }\n\n //\n // Certain compilers need separate declarations for T and const T\n //\n# ifdef _CXXTEST_NO_COPY_CONST\n# define CXXTEST_COPY_CONST_TRAITS(CXXTEST_CLASS)\n# else // !_CXXTEST_NO_COPY_CONST\n# define CXXTEST_COPY_CONST_TRAITS(CXXTEST_CLASS) CXXTEST_COPY_TRAITS(CXXTEST_CLASS, const CXXTEST_CLASS)\n# endif // _CXXTEST_NO_COPY_CONST\n \n //\n // Avoid compiler warnings about unsigned types always >= 0\n //\n template inline bool negative( N n ) { return n < 0; }\n template inline N abs( N n ) { return negative(n) ? -n : n; }\n\n# define CXXTEST_NON_NEGATIVE(Type) \\\n CXXTEST_TEMPLATE_INSTANTIATION \\\n inline bool negative( Type ) { return false; } \\\n CXXTEST_TEMPLATE_INSTANTIATION \\\n inline Type abs( Type value ) { return value; }\n\n CXXTEST_NON_NEGATIVE( bool )\n CXXTEST_NON_NEGATIVE( unsigned char )\n CXXTEST_NON_NEGATIVE( unsigned short int )\n CXXTEST_NON_NEGATIVE( unsigned int )\n CXXTEST_NON_NEGATIVE( unsigned long int )\n# ifdef _CXXTEST_LONGLONG\n CXXTEST_NON_NEGATIVE( unsigned _CXXTEST_LONGLONG )\n# endif // _CXXTEST_LONGLONG\n\n //\n // Represent (integral) number as a string\n // Returns one past the end of the string\n // Remember -- we can't use the standard library!\n //\n template\n char *numberToString( N n, char *s,\n N base = 10,\n unsigned skipDigits = 0,\n unsigned maxDigits = (unsigned)-1 )\n {\n if ( negative(n) ) {\n *s++ = '-';\n n = abs(n);\n }\n \n N digit = 1;\n while ( digit <= (n / base) )\n digit *= base;\n N digitValue;\n for ( ; digit >= 1 && skipDigits; n -= digit * digitValue, digit /= base, -- skipDigits )\n digitValue = (unsigned)(n / digit);\n for ( ; digit >= 1 && maxDigits; n -= digit * digitValue, digit /= base, -- maxDigits )\n *s++ = digitToChar( (unsigned)(digitValue = (unsigned)(n / digit)) );\n\n *s = '\\0';\n return s;\n }\n\n //\n // All the specific ValueTraits follow.\n // You can #define CXXTEST_USER_VALUE_TRAITS if you don't want them\n //\n \n#ifndef CXXTEST_USER_VALUE_TRAITS\n //\n // ValueTraits: const char * const &\n // This is used for printing strings, as in TS_FAIL( \"Message\" )\n //\n CXXTEST_TEMPLATE_INSTANTIATION\n class ValueTraits\n {\n ValueTraits &operator=( const ValueTraits & );\n const char *_asString;\n \n public:\n ValueTraits( const char * const &value ) : _asString( value ) {}\n ValueTraits( const ValueTraits &other ) : _asString( other._asString ) {}\n const char *asString( void ) const { return _asString; }\n };\n\n CXXTEST_COPY_TRAITS( const char *, const char * const & );\n CXXTEST_COPY_TRAITS( char *, const char * const & );\n\n //\n // ValueTraits: bool\n // \n CXXTEST_TEMPLATE_INSTANTIATION\n class ValueTraits\n {\n bool _value;\n \n public:\n ValueTraits( const bool value ) : _value( value ) {}\n const char *asString( void ) const { return _value ? \"true\" : \"false\"; }\n };\n\n CXXTEST_COPY_CONST_TRAITS( bool );\n\n# ifdef _CXXTEST_LONGLONG\n //\n // ValueTraits: signed long long\n //\n CXXTEST_TEMPLATE_INSTANTIATION\n class ValueTraits\n {\n typedef _CXXTEST_LONGLONG T;\n char _asString[2 + 3 * sizeof(T)];\n public:\n ValueTraits( T t ) { numberToString( t, _asString ); }\n const char *asString( void ) const { return _asString; }\n };\n\n CXXTEST_COPY_CONST_TRAITS( signed _CXXTEST_LONGLONG );\n\n //\n // ValueTraits: unsigned long long\n //\n CXXTEST_TEMPLATE_INSTANTIATION\n class ValueTraits\n {\n typedef unsigned _CXXTEST_LONGLONG T;\n char _asString[1 + 3 * sizeof(T)];\n public:\n ValueTraits( T t ) { numberToString( t, _asString ); }\n const char *asString( void ) const { return _asString; }\n };\n\n CXXTEST_COPY_CONST_TRAITS( unsigned _CXXTEST_LONGLONG );\n# endif // _CXXTEST_LONGLONG\n\n //\n // ValueTraits: signed long\n //\n CXXTEST_TEMPLATE_INSTANTIATION\n class ValueTraits\n {\n typedef signed long int T;\n char _asString[2 + 3 * sizeof(T)];\n public:\n ValueTraits( T t ) { numberToString( t, _asString ); }\n const char *asString( void ) const { return _asString; }\n };\n\n CXXTEST_COPY_CONST_TRAITS( signed long int );\n \n //\n // ValueTraits: unsigned long\n //\n CXXTEST_TEMPLATE_INSTANTIATION\n class ValueTraits\n {\n typedef unsigned long int T;\n char _asString[1 + 3 * sizeof(T)];\n public:\n ValueTraits( T t ) { numberToString( t, _asString ); }\n const char *asString( void ) const { return _asString; }\n };\n\n CXXTEST_COPY_CONST_TRAITS( unsigned long int );\n \n //\n // All decimals are the same as the long version\n //\n \n CXXTEST_COPY_TRAITS( const signed int, const signed long int );\n CXXTEST_COPY_TRAITS( const unsigned int, const unsigned long int );\n CXXTEST_COPY_TRAITS( const signed short int, const signed long int );\n CXXTEST_COPY_TRAITS( const unsigned short int, const unsigned long int );\n CXXTEST_COPY_TRAITS( const unsigned char, const unsigned long int );\n \n CXXTEST_COPY_CONST_TRAITS( signed int );\n CXXTEST_COPY_CONST_TRAITS( unsigned int );\n CXXTEST_COPY_CONST_TRAITS( signed short int );\n CXXTEST_COPY_CONST_TRAITS( unsigned short int );\n CXXTEST_COPY_CONST_TRAITS( unsigned char );\n\n //\n // ValueTraits: char\n // Returns 'x' for printable chars, '\\x??' for others\n //\n CXXTEST_TEMPLATE_INSTANTIATION\n class ValueTraits\n {\n char _asString[sizeof(\"'\\\\xXX'\")];\n public:\n ValueTraits( char c ) { copyString( charToString( c, copyString( _asString, \"'\" ) ), \"'\" ); }\n const char *asString( void ) const { return _asString; }\n };\n\n CXXTEST_COPY_CONST_TRAITS( char );\n\n //\n // ValueTraits: signed char\n // Same as char, some compilers need it\n //\n CXXTEST_COPY_TRAITS( const signed char, const char );\n CXXTEST_COPY_CONST_TRAITS( signed char );\n\n //\n // ValueTraits: double\n //\n CXXTEST_TEMPLATE_INSTANTIATION\n class ValueTraits\n {\n public:\n ValueTraits( double t ) \n {\n //if ( ( t != t ) || ( t >= 1.0/0.0 ) || ( t == -1.0/0.0 ) )\n if ( ( t != t ) || ( t >= HUGE_VAL ) || ( t == -HUGE_VAL ) )\n nonFiniteNumber( t );\n else if ( requiredDigitsOnLeft( t ) > MAX_DIGITS_ON_LEFT )\n hugeNumber( t );\n else\n normalNumber( t );\n }\n\n const char *asString( void ) const { return _asString; }\n \n private:\n enum { MAX_DIGITS_ON_LEFT = 24, DIGITS_ON_RIGHT = 4, BASE = 10 };\n char _asString[1 + MAX_DIGITS_ON_LEFT + 1 + DIGITS_ON_RIGHT + 1];\n\n static unsigned requiredDigitsOnLeft( double t );\n char *doNegative( double &t );\n void hugeNumber( double t );\n void normalNumber( double t );\n void nonFiniteNumber( double t );\n char *doubleToString( double t, char *s, unsigned skip = 0, unsigned max = (unsigned)-1 );\n };\n\n CXXTEST_COPY_CONST_TRAITS( double );\n\n //\n // ValueTraits: float\n //\n CXXTEST_COPY_TRAITS( const float, const double );\n CXXTEST_COPY_CONST_TRAITS( float );\n#endif // !CXXTEST_USER_VALUE_TRAITS\n}\n\n#ifdef _CXXTEST_HAVE_STD\n# include \n#endif // _CXXTEST_HAVE_STD\n\nnamespace dummy_enum_ns {}\n\n//\n// CXXTEST_ENUM_TRAITS\n//\n#define CXXTEST_ENUM_TRAITS( TYPE, VALUES ) \\\n namespace CxxTest \\\n { \\\n CXXTEST_TEMPLATE_INSTANTIATION \\\n class ValueTraits \\\n { \\\n TYPE _value; \\\n char _fallback[sizeof(\"(\" #TYPE \")\") + 3 * sizeof(TYPE)]; \\\n public: \\\n ValueTraits( TYPE value ) { \\\n _value = value; \\\n numberToString( _value, copyString( _fallback, \"(\" #TYPE \")\" ) ); \\\n } \\\n const char *asString( void ) const \\\n { \\\n switch ( _value ) \\\n { \\\n VALUES \\\n default: return _fallback; \\\n } \\\n } \\\n }; \\\n } using namespace dummy_enum_ns\n\n#define CXXTEST_ENUM_MEMBER( MEMBER ) \\\n case MEMBER: return #MEMBER;\n\n#endif // __cxxtest__ValueTraits_h__\n" }, { "path": "cxxtest/cxxtest/Win32Gui.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__Win32Gui_h__\n#define __cxxtest__Win32Gui_h__\n\n//\n// The Win32Gui displays a simple progress bar using the Win32 API.\n//\n// It accepts the following command line options:\n// -minimized Start minimized, pop up on error\n// -keep Don't close the window at the end\n// -title TITLE Set the window caption\n//\n// If both -minimized and -keep are specified, GUI will only keep the\n// window if it's in focus.\n//\n// N.B. If you're wondering why this class doesn't use any standard\n// library or STL ( would have been nice) it's because it only\n// uses \"straight\" Win32 API.\n//\n\n#include \n\n#include \n#include \n\nnamespace CxxTest\n{\n class Win32Gui : public GuiListener\n {\n public:\n void enterGui( int &argc, char **argv )\n\t{\n\t parseCommandLine( argc, argv );\n\t}\n\t\n void enterWorld( const WorldDescription &wd )\n {\n getTotalTests( wd );\n _testsDone = 0;\n startGuiThread();\n }\n\n\tvoid guiEnterSuite( const char *suiteName )\n\t{\n\t showSuiteName( suiteName );\n reset( _suiteStart );\n\t}\n\n void guiEnterTest( const char *suiteName, const char *testName )\n {\n ++ _testsDone;\n setTestCaption( suiteName, testName );\n showTestName( testName );\n\t showTestsDone();\n progressBarMessage( PBM_STEPIT );\n reset( _testStart );\n }\n\n void yellowBar()\n {\n\t setColor( 255, 255, 0 );\n setIcon( IDI_WARNING );\n getTotalTests();\n }\n \n void redBar()\n {\n if ( _startMinimized )\n showMainWindow( SW_SHOWNORMAL );\n\t setColor( 255, 0, 0 );\n\t setIcon( IDI_ERROR );\n getTotalTests();\n }\n\n void leaveGui()\n {\n if ( keep() )\n {\n showSummary();\n WaitForSingleObject( _gui, INFINITE );\n }\n DestroyWindow( _mainWindow );\n }\n\n private:\n const char *_title;\n bool _startMinimized, _keep;\n HANDLE _gui;\n WNDCLASSEX _windowClass;\n HWND _mainWindow, _progressBar, _statusBar;\n HANDLE _canStartTests;\n unsigned _numTotalTests, _testsDone;\n char _strTotalTests[WorldDescription::MAX_STRLEN_TOTAL_TESTS];\n enum { \n STATUS_SUITE_NAME, STATUS_SUITE_TIME,\n STATUS_TEST_NAME, STATUS_TEST_TIME,\n STATUS_TESTS_DONE, STATUS_WORLD_TIME,\n STATUS_TOTAL_PARTS \n };\n int _statusWidths[STATUS_TOTAL_PARTS];\n unsigned _statusOffsets[STATUS_TOTAL_PARTS];\n unsigned _statusTotal;\n char _statusTestsDone[sizeof(\"1000000000 of (100%)\") + WorldDescription::MAX_STRLEN_TOTAL_TESTS];\n DWORD _worldStart, _suiteStart, _testStart;\n char _timeString[sizeof(\"00:00:00\")];\n\n void parseCommandLine( int argc, char **argv )\n {\n _startMinimized = _keep = false;\n\t _title = argv[0];\n \n for ( int i = 1; i < argc; ++ i )\n {\n if ( !lstrcmpA( argv[i], \"-minimized\" ) )\n _startMinimized = true;\n else if ( !lstrcmpA( argv[i], \"-keep\" ) )\n _keep = true;\n else if ( !lstrcmpA( argv[i], \"-title\" ) && (i + 1 < argc) )\n _title = argv[++i];\n }\n }\n \n void getTotalTests()\n {\n getTotalTests( tracker().world() );\n }\n\n void getTotalTests( const WorldDescription &wd )\n {\n _numTotalTests = wd.numTotalTests();\n wd.strTotalTests( _strTotalTests );\n }\n\n void startGuiThread()\n {\n _canStartTests = CreateEvent( NULL, TRUE, FALSE, NULL );\n\t\t\tDWORD threadId;\n _gui = CreateThread( NULL, 0, &(Win32Gui::guiThread), (LPVOID)this, 0, &threadId );\n WaitForSingleObject( _canStartTests, INFINITE );\n }\n\n static DWORD WINAPI guiThread( LPVOID parameter )\n {\n ((Win32Gui *)parameter)->gui();\n return 0;\n }\n\n void gui()\n {\n registerWindowClass();\n createMainWindow();\n initCommonControls();\n createProgressBar();\n createStatusBar();\n centerMainWindow();\n showMainWindow();\n startTimer();\n startTests();\n\n messageLoop();\n }\n\n void registerWindowClass()\n {\n _windowClass.cbSize = sizeof(_windowClass);\n _windowClass.style = CS_HREDRAW | CS_VREDRAW;\n _windowClass.lpfnWndProc = &(Win32Gui::windowProcedure);\n _windowClass.cbClsExtra = 0;\n _windowClass.cbWndExtra = sizeof(LONG);\n _windowClass.hInstance = (HINSTANCE)NULL;\n _windowClass.hIcon = (HICON)NULL;\n _windowClass.hCursor = (HCURSOR)NULL;\n _windowClass.hbrBackground = (HBRUSH)(COLOR_WINDOW + 1);\n _windowClass.lpszMenuName = NULL;\n _windowClass.lpszClassName = TEXT(\"CxxTest Window Class\");\n _windowClass.hIconSm = (HICON)NULL;\n\n RegisterClassEx( &_windowClass );\n }\n\n void createMainWindow()\n {\n _mainWindow = createWindow( _windowClass.lpszClassName, WS_OVERLAPPEDWINDOW );\n }\n\n void initCommonControls()\n {\n HMODULE dll = LoadLibraryA( \"comctl32.dll\" );\n if ( !dll )\n\t\treturn;\n\t\t\n\t typedef void (WINAPI *FUNC)( void );\n\t FUNC func = (FUNC)GetProcAddress( dll, \"InitCommonControls\" );\n\t if ( !func )\n return;\n\n\t func();\n }\n\n void createProgressBar()\n {\n _progressBar = createWindow( PROGRESS_CLASS, WS_CHILD | WS_VISIBLE | PBS_SMOOTH, _mainWindow );\n\n#ifdef PBM_SETRANGE32\n progressBarMessage( PBM_SETRANGE32, 0, _numTotalTests );\n#else // No PBM_SETRANGE32, use PBM_SETRANGE\n\t progressBarMessage( PBM_SETRANGE, 0, MAKELPARAM( 0, (WORD)_numTotalTests ) );\n#endif // PBM_SETRANGE32\n progressBarMessage( PBM_SETPOS, 0 );\n progressBarMessage( PBM_SETSTEP, 1 );\n greenBar();\n UpdateWindow( _progressBar );\n }\n\n void createStatusBar()\n {\n _statusBar = createWindow( STATUSCLASSNAME, WS_CHILD | WS_VISIBLE, _mainWindow );\n setRatios( 4, 1, 3, 1, 3, 1 );\n }\n\n void setRatios( unsigned suiteNameRatio, unsigned suiteTimeRatio,\n unsigned testNameRatio, unsigned testTimeRatio,\n unsigned testsDoneRatio, unsigned worldTimeRatio )\n {\n _statusTotal = 0;\n _statusOffsets[STATUS_SUITE_NAME] = (_statusTotal += suiteNameRatio);\n _statusOffsets[STATUS_SUITE_TIME] = (_statusTotal += suiteTimeRatio);\n _statusOffsets[STATUS_TEST_NAME] = (_statusTotal += testNameRatio);\n _statusOffsets[STATUS_TEST_TIME] = (_statusTotal += testTimeRatio);\n _statusOffsets[STATUS_TESTS_DONE] = (_statusTotal += testsDoneRatio);\n _statusOffsets[STATUS_WORLD_TIME] = (_statusTotal += worldTimeRatio);\n }\n\n HWND createWindow( LPCTSTR className, DWORD style, HWND parent = (HWND)NULL )\n {\n return CreateWindow( className, NULL, style, 0, 0, 0, 0, parent,\n (HMENU)NULL, (HINSTANCE)NULL, (LPVOID)this );\n }\n\n void progressBarMessage( UINT message, WPARAM wParam = 0, LPARAM lParam = 0 )\n {\n SendMessage( _progressBar, message, wParam, lParam );\n }\n\n void centerMainWindow()\n {\n RECT screen;\n getScreenArea( screen );\n\n LONG screenWidth = screen.right - screen.left;\n LONG screenHeight = screen.bottom - screen.top;\n\n LONG xCenter = (screen.right + screen.left) / 2;\n LONG yCenter = (screen.bottom + screen.top) / 2;\n\n LONG windowWidth = (screenWidth * 4) / 5;\n LONG windowHeight = screenHeight / 10;\n LONG minimumHeight = 2 * (GetSystemMetrics( SM_CYCAPTION ) + GetSystemMetrics( SM_CYFRAME ));\n if ( windowHeight < minimumHeight )\n windowHeight = minimumHeight;\n\n SetWindowPos( _mainWindow, HWND_TOP,\n xCenter - (windowWidth / 2), yCenter - (windowHeight / 2),\n windowWidth, windowHeight, 0 );\n }\n\n void getScreenArea( RECT &area )\n {\n if ( !getScreenAreaWithoutTaskbar( area ) )\n getWholeScreenArea( area );\n }\n\n bool getScreenAreaWithoutTaskbar( RECT &area )\n {\n return (SystemParametersInfo( SPI_GETWORKAREA, sizeof(RECT), &area, 0 ) != 0);\n }\n\n void getWholeScreenArea( RECT &area )\n {\n area.left = area.top = 0;\n area.right = GetSystemMetrics( SM_CXSCREEN );\n area.bottom = GetSystemMetrics( SM_CYSCREEN );\n }\n\n void showMainWindow()\n {\n showMainWindow( _startMinimized ? SW_MINIMIZE : SW_SHOWNORMAL );\n UpdateWindow( _mainWindow );\n }\n\n void showMainWindow( int mode )\n {\n ShowWindow( _mainWindow, mode );\n }\n\n enum { TIMER_ID = 1, TIMER_DELAY = 1000 };\n\n void startTimer()\n {\n reset( _worldStart );\n reset( _suiteStart );\n reset( _testStart );\n SetTimer( _mainWindow, TIMER_ID, TIMER_DELAY, 0 );\n }\n\n void reset( DWORD &tick )\n {\n tick = GetTickCount();\n }\n\n void startTests()\n {\n SetEvent( _canStartTests );\n }\n\n void messageLoop()\n {\n MSG message;\n while ( BOOL haveMessage = GetMessage( &message, NULL, 0, 0 ) )\n if ( haveMessage != -1 )\n DispatchMessage( &message );\n }\n\n static LRESULT CALLBACK windowProcedure( HWND window, UINT message, WPARAM wParam, LPARAM lParam )\n {\n if ( message == WM_CREATE )\n setUp( window, (LPCREATESTRUCT)lParam );\n\n Win32Gui *that = (Win32Gui *)GetWindowLong( window, GWL_USERDATA );\n return that->handle( window, message, wParam, lParam );\n }\n\n static void setUp( HWND window, LPCREATESTRUCT create )\n {\n SetWindowLong( window, GWL_USERDATA, (LONG)create->lpCreateParams );\n }\n\n LRESULT handle( HWND window, UINT message, WPARAM wParam, LPARAM lParam )\n {\n switch ( message )\n {\n case WM_SIZE: resizeControls(); break;\n\n case WM_TIMER: updateTime(); break;\n\n case WM_CLOSE:\n case WM_DESTROY:\n case WM_QUIT:\n ExitProcess( tracker().failedTests() );\n\n default: return DefWindowProc( window, message, wParam, lParam );\n }\n return 0;\n }\n\n void resizeControls()\n {\n RECT r;\n GetClientRect( _mainWindow, &r );\n LONG width = r.right - r.left;\n LONG height = r.bottom - r.top;\n\n GetClientRect( _statusBar, &r );\n LONG statusHeight = r.bottom - r.top;\n LONG resizeGripWidth = statusHeight;\n LONG progressHeight = height - statusHeight;\n\n SetWindowPos( _progressBar, HWND_TOP, 0, 0, width, progressHeight, 0 );\n SetWindowPos( _statusBar, HWND_TOP, 0, progressHeight, width, statusHeight, 0 );\n setStatusParts( width - resizeGripWidth );\n }\n\n void setStatusParts( LONG width )\n {\n for ( unsigned i = 0; i < STATUS_TOTAL_PARTS; ++ i )\n _statusWidths[i] = (width * _statusOffsets[i]) / _statusTotal;\n\n statusBarMessage( SB_SETPARTS, STATUS_TOTAL_PARTS, _statusWidths );\n }\n\n void statusBarMessage( UINT message, WPARAM wParam = 0, const void *lParam = 0 )\n {\n SendMessage( _statusBar, message, wParam, (LPARAM)lParam );\n }\n\n void greenBar()\n {\n setColor( 0, 255, 0 );\n setIcon( IDI_INFORMATION );\n }\n\n#ifdef PBM_SETBARCOLOR\n void setColor( BYTE red, BYTE green, BYTE blue )\n {\n progressBarMessage( PBM_SETBARCOLOR, 0, RGB( red, green, blue ) );\n }\n#else // !PBM_SETBARCOLOR\n void setColor( BYTE, BYTE, BYTE ) \n {\n }\n#endif // PBM_SETBARCOLOR\n\n void setIcon( LPCTSTR icon )\n {\n SendMessage( _mainWindow, WM_SETICON, ICON_BIG, (LPARAM)loadStandardIcon( icon ) );\n }\n\n HICON loadStandardIcon( LPCTSTR icon )\n {\n return LoadIcon( (HINSTANCE)NULL, icon );\n }\n\n void setTestCaption( const char *suiteName, const char *testName )\n {\n setCaption( suiteName, \"::\", testName, \"()\" );\n }\n\n void setCaption( const char *a = \"\", const char *b = \"\", const char *c = \"\", const char *d = \"\" )\n {\n unsigned length = lstrlenA( _title ) + sizeof( \" - \" ) +\n lstrlenA( a ) + lstrlenA( b ) + lstrlenA( c ) + lstrlenA( d );\n char *name = allocate( length );\n lstrcpyA( name, _title );\n lstrcatA( name, \" - \" );\n lstrcatA( name, a );\n lstrcatA( name, b );\n lstrcatA( name, c );\n lstrcatA( name, d );\n SetWindowTextA( _mainWindow, name );\n deallocate( name );\n }\n\n void showSuiteName( const char *suiteName )\n {\n setStatusPart( STATUS_SUITE_NAME, suiteName );\n\t}\n\n\tvoid showTestName( const char *testName )\n\t{\n setStatusPart( STATUS_TEST_NAME, testName );\n\t}\n\n\tvoid showTestsDone()\n\t{\n wsprintfA( _statusTestsDone, \"%u of %s (%u%%)\",\n _testsDone, _strTotalTests,\n (_testsDone * 100) / _numTotalTests );\n setStatusPart( STATUS_TESTS_DONE, _statusTestsDone );\n }\n\n void updateTime()\n {\n setStatusTime( STATUS_WORLD_TIME, _worldStart );\n setStatusTime( STATUS_SUITE_TIME, _suiteStart );\n setStatusTime( STATUS_TEST_TIME, _testStart );\n }\n\n void setStatusTime( unsigned part, DWORD start )\n {\n unsigned total = (GetTickCount() - start) / 1000;\n unsigned hours = total / 3600;\n unsigned minutes = (total / 60) % 60;\n unsigned seconds = total % 60;\n\n if ( hours )\n wsprintfA( _timeString, \"%u:%02u:%02u\", hours, minutes, seconds );\n else\n wsprintfA( _timeString, \"%02u:%02u\", minutes, seconds );\n\n setStatusPart( part, _timeString );\n }\n\n bool keep()\n {\n if ( !_keep )\n return false;\n if ( !_startMinimized )\n return true;\n return (_mainWindow == GetForegroundWindow());\n }\n\n void showSummary()\n {\n stopTimer();\n setSummaryStatusBar();\n setSummaryCaption();\n }\n\n void setStatusPart( unsigned part, const char *text )\n {\n statusBarMessage( SB_SETTEXTA, part, text );\n }\n\n void stopTimer()\n {\n KillTimer( _mainWindow, TIMER_ID );\n setStatusTime( STATUS_WORLD_TIME, _worldStart );\n }\n\n void setSummaryStatusBar()\n {\n setRatios( 0, 0, 0, 0, 1, 1 );\n resizeControls();\n \n const char *tests = (_numTotalTests == 1) ? \"test\" : \"tests\";\n if ( tracker().failedTests() )\n wsprintfA( _statusTestsDone, \"Failed %u of %s %s\",\n tracker().failedTests(), _strTotalTests, tests );\n else\n wsprintfA( _statusTestsDone, \"%s %s passed\", _strTotalTests, tests );\n\n setStatusPart( STATUS_TESTS_DONE, _statusTestsDone );\n }\n\n void setSummaryCaption()\n {\n setCaption( _statusTestsDone );\n }\n\n char *allocate( unsigned length )\n {\n return (char *)HeapAlloc( GetProcessHeap(), 0, length );\n }\n\n void deallocate( char *data )\n {\n HeapFree( GetProcessHeap(), 0, data );\n }\n };\n}\n\n#endif // __cxxtest__Win32Gui_h__\n" }, { "path": "cxxtest/cxxtest/X11Gui.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__X11Gui_h__\n#define __cxxtest__X11Gui_h__\n\n//\n// X11Gui displays a simple progress bar using X11\n// \n// It accepts the following command-line arguments:\n// -title - Sets the application title\n// -fn or -font <font> - Sets the font\n// -bg or -background <color> - Sets the background color (default=Grey)\n// -fg or -foreground <color> - Sets the text color (default=Black)\n// -green/-yellow/-red <color> - Sets the colors of the bar\n//\n\n#include <cxxtest/Gui.h>\n\n#include <X11/Xlib.h>\n#include <X11/Xutil.h>\n#include <stdio.h>\n#include <stdlib.h>\n#include <string.h>\n\nnamespace CxxTest\n{\n class X11Gui : public GuiListener\n {\n public:\n void enterGui( int &argc, char **argv )\n {\n parseCommandLine( argc, argv );\n }\n \n void enterWorld( const WorldDescription &wd )\n {\n openDisplay();\n if ( _display ) {\n createColors();\n createWindow();\n createGc();\n createFont();\n centerWindow();\n initializeEvents();\n initializeBar( wd );\n processEvents();\n }\n }\n \n void guiEnterTest( const char *suiteName, const char *testName )\n {\n if ( _display ) {\n ++ _testsDone;\n setWindowName( suiteName, testName );\n redraw();\n }\n }\n \n void yellowBar()\n {\n if ( _display ) {\n _barColor = getColor( _yellowName );\n getTotalTests();\n processEvents();\n }\n }\n\n void redBar()\n {\n if ( _display ) {\n _barColor = getColor( _redName );\n getTotalTests();\n processEvents();\n }\n }\n\n void leaveGui()\n {\n if ( _display ) {\n freeFontInfo();\n destroyGc();\n destroyWindow();\n closeDisplay();\n }\n }\n\n private:\n const char *_programName;\n Display *_display;\n Window _window;\n unsigned _numTotalTests, _testsDone;\n char _strTotalTests[WorldDescription::MAX_STRLEN_TOTAL_TESTS];\n const char *_foregroundName, *_backgroundName;\n const char *_greenName, *_yellowName, *_redName;\n unsigned long _foreground, _background, _barColor;\n int _width, _height;\n GC _gc;\n const char *_fontName;\n XID _fontId;\n XFontStruct *_fontInfo;\n int _textHeight, _textDescent;\n long _eventMask;\n Colormap _colormap;\n\n void parseCommandLine( int &argc, char **argv )\n {\n _programName = argv[0];\n\n _fontName = 0;\n _foregroundName = \"Black\";\n _backgroundName = \"Grey\";\n _greenName = \"Green\";\n _yellowName = \"Yellow\";\n _redName = \"Red\";\n\n for ( int i = 1; i + 1 < argc; ++ i ) {\n if ( !strcmp( argv[i], \"-title\" ) )\n _programName = argv[++ i];\n else if ( !strcmp( argv[i], \"-fn\" ) || !strcmp( argv[i], \"-font\" ) )\n _fontName = argv[++ i];\n else if ( !strcmp( argv[i], \"-fg\" ) || !strcmp( argv[i], \"-foreground\" ) )\n _foregroundName = argv[++ i];\n else if ( !strcmp( argv[i], \"-bg\" ) || !strcmp( argv[i], \"-background\" ) )\n _backgroundName = argv[++ i];\n else if ( !strcmp( argv[i], \"-green\" ) )\n _greenName = argv[++ i];\n else if ( !strcmp( argv[i], \"-yellow\" ) )\n _yellowName = argv[++ i];\n else if ( !strcmp( argv[i], \"-red\" ) )\n _redName = argv[++ i];\n }\n }\n\n void openDisplay()\n {\n _display = XOpenDisplay( NULL );\n }\n\n void createColors()\n {\n _colormap = DefaultColormap( _display, 0 );\n _foreground = getColor( _foregroundName );\n _background = getColor( _backgroundName );\n }\n\n unsigned long getColor( const char *colorName )\n {\n XColor color;\n XParseColor( _display, _colormap, colorName, &color );\n XAllocColor( _display, _colormap, &color );\n return color.pixel;\n }\n \n void createWindow()\n {\n _window = XCreateSimpleWindow( _display, RootWindow( _display, 0 ), 0, 0, 1, 1, 0, 0, _background );\n }\n\n void createGc()\n {\n _gc = XCreateGC( _display, _window, 0, 0 );\n }\n\n void createFont()\n {\n if ( !loadFont() )\n useDefaultFont();\n getFontInfo();\n _textHeight = _fontInfo->ascent + _fontInfo->descent;\n _textDescent = _fontInfo->descent;\n }\n\n bool loadFont()\n {\n if ( !_fontName )\n return false;\n _fontId = XLoadFont( _display, _fontName );\n return (XSetFont( _display, _gc, _fontId ) == Success);\n }\n\n void useDefaultFont()\n {\n _fontId = XGContextFromGC( _gc );\n }\n\n void getFontInfo()\n {\n _fontInfo = XQueryFont( _display, _fontId );\n }\n\n void freeFontInfo()\n {\n XFreeFontInfo( NULL, _fontInfo, 1 );\n }\n\n void initializeEvents()\n {\n _eventMask = ExposureMask;\n XSelectInput( _display, _window, _eventMask );\n }\n\n void initializeBar( const WorldDescription &wd )\n {\n getTotalTests( wd );\n _testsDone = 0;\n _barColor = getColor( _greenName );\n }\n\n void getTotalTests()\n {\n getTotalTests( tracker().world() );\n }\n\n void getTotalTests( const WorldDescription &wd )\n {\n _numTotalTests = wd.numTotalTests();\n wd.strTotalTests( _strTotalTests );\n }\n\n void centerWindow()\n {\n XMapWindow( _display, _window );\n \n Screen *screen = XDefaultScreenOfDisplay( _display );\n int screenWidth = WidthOfScreen( screen );\n int screenHeight = HeightOfScreen( screen );\n int xCenter = screenWidth / 2;\n int yCenter = screenHeight / 2;\n\n _width = (screenWidth * 4) / 5;\n _height = screenHeight / 14;\n \n XMoveResizeWindow( _display, _window, xCenter - (_width / 2), yCenter - (_height / 2), _width, _height );\n }\n\n void processEvents()\n {\n redraw();\n \n XEvent event;\n while( XCheckMaskEvent( _display, _eventMask, &event ) )\n redraw();\n }\n\n void setWindowName( const char *suiteName, const char *testName )\n {\n unsigned length = strlen( _programName ) + strlen( suiteName ) + strlen( testName ) + sizeof( \" - ::()\" );\n char *name = (char *)malloc( length );\n sprintf( name, \"%s - %s::%s()\", _programName, suiteName, testName );\n XSetStandardProperties( _display, _window, name, 0, 0, 0, 0, 0 );\n free( name );\n }\n\n void redraw()\n {\n getWindowSize();\n drawSolidBar();\n drawDividers();\n drawPercentage();\n flush();\n }\n\n void getWindowSize()\n {\n XWindowAttributes attributes;\n XGetWindowAttributes( _display, _window, &attributes );\n _width = attributes.width;\n _height = attributes.height;\n }\n\n void drawSolidBar()\n {\n unsigned barWidth = (_width * _testsDone) / _numTotalTests;\n\n XSetForeground( _display, _gc, _barColor );\n XFillRectangle( _display, _window, _gc, 0, 0, barWidth, _height );\n\n XSetForeground( _display, _gc, _background );\n XFillRectangle( _display, _window, _gc, barWidth, 0, _width + 1 - barWidth, _height );\n }\n\n void drawDividers()\n {\n if(_width / _numTotalTests < 5)\n return;\n for ( unsigned i = 1; i < _testsDone; ++ i ) {\n int x = (_width * i) / _numTotalTests;\n XDrawLine( _display, _window, _gc, x, 0, x, _height);\n }\n }\n\n void drawPercentage()\n {\n XSetForeground( _display, _gc, _foreground );\n \n char str[sizeof(\"1000000000 of \") + sizeof(_strTotalTests) + sizeof(\" (100%)\")];\n sprintf( str, \"%u of %s (%u%%)\", _testsDone, _strTotalTests, (_testsDone * 100) / _numTotalTests );\n unsigned len = strlen( str );\n\n int textWidth = XTextWidth( _fontInfo, str, len );\n\n XDrawString( _display, _window, _gc,\n (_width - textWidth) / 2, ((_height + _textHeight) / 2) - _textDescent,\n str, len );\n }\n\n void flush()\n {\n XFlush( _display );\n }\n\n void destroyGc()\n {\n XFreeGC( _display, _gc );\n }\n\n void destroyWindow()\n {\n XDestroyWindow( _display, _window );\n }\n\n void closeDisplay()\n {\n XCloseDisplay( _display );\n }\n };\n}\n\n#endif //__cxxtest__X11Gui_h__\n" }, { "path": "cxxtest/cxxtest/XUnitPrinter.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __CXXTEST__XUNIT_PRINTER_H\n#define __CXXTEST__XUNIT_PRINTER_H\n\n//\n// XUnitPrinter combines an ErrorPrinter with an XML formatter.\n//\n\n#include <cxxtest/TeeListener.h>\n#include <cxxtest/ErrorPrinter.h>\n#include <cxxtest/XmlPrinter.h>\n\nnamespace CxxTest\n{\n class XUnitPrinter : public TeeListener\n {\n public:\n\n XmlPrinter xml_printer;\n ErrorPrinter error_printer;\n \n XUnitPrinter( CXXTEST_STD(ostream) &o = CXXTEST_STD(cout) )\n : xml_printer(o)\n {\n setFirst( error_printer );\n setSecond( xml_printer );\n }\n\n int run()\n {\n TestRunner::runAllTests( *this );\n return tracker().failedTests();\n }\n };\n}\n\n#endif //__CXXTEST__XUNIT_PRINTER_H\n\n" }, { "path": "cxxtest/cxxtest/XmlFormatter.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n// Licensed under the LGPL, see http://www.gnu.org/licenses/lgpl.html\n\n#ifndef __CXXTEST__XMLFORMATTER_H\n#define __CXXTEST__XMLFORMATTER_H\n\n//\n// The XmlFormatter is a TestListener that\n// prints reports of the errors to an output\n// stream in the form of an XML document.\n//\n\n// The following definitions are used if stack trace support is enabled,\n// to give the traces an easily-parsable XML format. If stack tracing is\n// not enabled, then these definitions will be ignored.\n#define CXXTEST_STACK_TRACE_ESCAPE_AS_XML\n#define CXXTEST_STACK_TRACE_NO_ESCAPE_FILELINE_AFFIXES\n\n#define CXXTEST_STACK_TRACE_INITIAL_PREFIX \"<stack-frame function=\\\"\"\n#define CXXTEST_STACK_TRACE_INITIAL_SUFFIX \"\\\"/>\\n\"\n#define CXXTEST_STACK_TRACE_OTHER_PREFIX CXXTEST_STACK_TRACE_INITIAL_PREFIX\n#define CXXTEST_STACK_TRACE_OTHER_SUFFIX CXXTEST_STACK_TRACE_INITIAL_SUFFIX\n#define CXXTEST_STACK_TRACE_ELLIDED_MESSAGE \"\"\n#define CXXTEST_STACK_TRACE_FILELINE_PREFIX \"\\\" location=\\\"\"\n#define CXXTEST_STACK_TRACE_FILELINE_SUFFIX \"\"\n\n\n#include <cxxtest/TestRunner.h>\n#include <cxxtest/TestListener.h>\n#include <cxxtest/TestTracker.h>\n#include <cxxtest/ValueTraits.h>\n#include <cxxtest/ErrorFormatter.h>\n#include <cxxtest/StdHeaders.h>\n#include <iostream>\n#include <sstream>\n#include <cstring>\n#include <cstdio>\n\nnamespace CxxTest\n{\n class TeeOutputStreams\n {\n private:\n class teebuffer : public std::basic_streambuf<char>\n {\n typedef std::basic_streambuf<char> streambuf_t;\n public:\n teebuffer(streambuf_t * buf1, streambuf_t * buf2)\n : buffer1(buf1), buffer2(buf2)\n {}\n\n virtual int overflow(int c)\n {\n if (c == EOF)\n return !EOF;\n else\n {\n int const ans1 = buffer1->sputc(c);\n int const ans2 = buffer2->sputc(c);\n return ans1 == EOF || ans2 == EOF ? EOF : c;\n }\n }\n\n virtual int sync()\n {\n int ans1 = buffer1->pubsync();\n int ans2 = buffer2->pubsync();\n return ans1 || ans2 ? -1 : 0;\n }\n\n streambuf_t * buffer1;\n streambuf_t * buffer2;\n };\n\n public:\n TeeOutputStreams(std::ostream& _cout, std::ostream& _cerr)\n : out(),\n err(),\n orig_cout(_cout),\n orig_cerr(_cerr),\n tee_out(out.rdbuf(), _cout.rdbuf()),\n tee_err(err.rdbuf(), _cerr.rdbuf())\n {\n orig_cout.rdbuf(&tee_out);\n orig_cerr.rdbuf(&tee_err);\n }\n\n ~TeeOutputStreams()\n {\n orig_cout.rdbuf(tee_out.buffer2);\n orig_cerr.rdbuf(tee_err.buffer2);\n }\n\n std::stringstream out;\n std::stringstream err;\n\n private:\n std::ostream& orig_cout;\n std::ostream& orig_cerr;\n teebuffer tee_out;\n teebuffer tee_err;\n };\n\n class ElementInfo\n {\n public:\n std::string name;\n std::stringstream value;\n std::map<std::string,std::string> attribute;\n\n ElementInfo()\n : name(), value(), attribute()\n {}\n\n ElementInfo(const ElementInfo& rhs)\n : name(rhs.name), value(rhs.value.str()), attribute(rhs.attribute)\n {}\n\n ElementInfo& operator=(const ElementInfo& rhs)\n {\n name = rhs.name;\n value.str(rhs.value.str());\n attribute = rhs.attribute;\n return *this;\n }\n\n template <class Type>\n void add(const std::string& name_, Type& value_)\n {\n std::ostringstream os;\n os << value_;\n attribute[name_] = os.str();\n }\n\n void write(OutputStream& os) {\n os << \" <\" << name.c_str() << \" \";\n std::map<std::string,std::string>::iterator curr=attribute.begin();\n std::map<std::string,std::string>::iterator end =attribute.end();\n while (curr != end) {\n os << curr->first.c_str() \n << \"=\\\"\" << curr->second.c_str() << \"\\\" \";\n curr++;\n }\n if (value.str().empty()) {\n os << \"/>\";\n }\n else {\n os << \">\" << escape(value.str()).c_str() \n << \"</\" << name.c_str() << \">\";\n }\n os.endl(os);\n }\n\n std::string escape(const std::string& str)\n {\n std::string escStr = \"\";\n for(size_t i = 0; i < str.length(); i++)\n {\n switch(str[i])\n {\n case '\"': escStr += \""\"; break;\n case '\\'': escStr += \"'\"; break;\n case '<': escStr += \"<\"; break;\n case '>': escStr += \">\"; break;\n case '&': escStr += \"&\"; break;\n default: escStr += str[i]; break;\n }\n }\n return escStr;\n }\n\n };\n\n class TestCaseInfo\n {\n public:\n\n TestCaseInfo() : fail(false), error(false), runtime(0.0) {}\n std::string className;\n std::string testName;\n std::string line;\n bool fail;\n bool error;\n double runtime;\n std::list<ElementInfo> elements;\n typedef std::list<ElementInfo>::iterator element_t;\n std::string world;\n\n element_t add_element(const std::string& name)\n {\n element_t elt = elements.insert(elements.end(), ElementInfo());\n elt->name=name;\n return elt;\n }\n\n element_t update_element(const std::string& name)\n {\n element_t elt = elements.begin();\n while ( elt != elements.end() )\n {\n if ( elt->name == name )\n return elt;\n }\n return add_element(name);\n }\n\n void write( OutputStream &o )\n {\n o << \" <testcase classname=\\\"\" << className.c_str() \n << \"\\\" name=\\\"\" << testName.c_str() \n << \"\\\" line=\\\"\" << line.c_str() << \"\\\"\";\n bool elts=false;\n element_t curr = elements.begin();\n element_t end = elements.end();\n while (curr != end) {\n if (!elts) {\n o << \">\";\n o.endl(o);\n elts=true;\n }\n curr->write(o);\n curr++;\n }\n if (elts)\n o << \" </testcase>\";\n else\n o << \" />\";\n o.endl(o);\n }\n\n };\n\n class XmlFormatter : public TestListener\n {\n public:\n XmlFormatter( OutputStream *o, OutputStream *ostr, std::ostringstream *os) \n : _o(o), _ostr(ostr), _os(os), stream_redirect(NULL) \n {}\n\n std::list<TestCaseInfo> info;\n std::list<TestCaseInfo>::iterator testcase;\n typedef std::list<ElementInfo>::iterator element_t;\n std::string classname;\n int ntests;\n int nfail;\n int nerror;\n double totaltime;\n\n int run()\n {\n TestRunner::runAllTests( *this );\n return tracker().failedTests();\n }\n\n void enterWorld( const WorldDescription & /*desc*/ )\n {\n ntests=0;\n nfail=0;\n nerror=0;\n totaltime=0;\n }\n\n static void totalTests( OutputStream &o )\n {\n char s[WorldDescription::MAX_STRLEN_TOTAL_TESTS];\n const WorldDescription &wd = tracker().world();\n o << wd.strTotalTests( s ) \n << (wd.numTotalTests() == 1 ? \" test\" : \" tests\");\n }\n\n void enterSuite( const SuiteDescription& desc )\n {\n classname = desc.suiteName();\n // replace \"::\" namespace with java-style \".\"\n size_t pos = 0;\n while( (pos = classname.find(\"::\", pos)) !=\n CXXTEST_STD(string::npos) )\n classname.replace(pos, 2, \".\");\n while ( ! classname.empty() && classname[0] == '.' )\n classname.erase(0,1);\n\n //CXXTEST_STD(cout) << \"HERE \" << desc.file() << \" \" \n // << classname << CXXTEST_STD(endl);\n\n //classname=desc.suiteName();\n //(*_o) << \"file=\\\"\" << desc.file() << \"\\\" \";\n //(*_o) << \"line=\\\"\" << desc.line() << \"\\\"\";\n //_o->flush();\n }\n\n void leaveSuite( const SuiteDescription & )\n {\n std::list<TestCaseInfo>::iterator curr = info.begin();\n std::list<TestCaseInfo>::iterator end = info.end();\n while (curr != end) {\n if (curr->fail) nfail++;\n if (curr->error) nerror++;\n totaltime += curr->runtime;\n ntests++;\n curr++;\n }\n curr = info.begin();\n end = info.end();\n while (curr != end) {\n (*curr).write(*_ostr);\n curr++;\n }\n info.clear();\n }\n\n void enterTest( const TestDescription & desc )\n {\n testcase = info.insert(info.end(),TestCaseInfo());\n testcase->testName = desc.testName();\n testcase->className = classname;\n std::ostringstream os;\n os << desc.line();\n testcase->line = os.str();\n\n if ( stream_redirect )\n CXXTEST_STD(cerr) << \"ERROR: The stream_redirect != NULL\" \n << CXXTEST_STD(endl);\n\n stream_redirect = \n new TeeOutputStreams(CXXTEST_STD(cout), CXXTEST_STD(cerr));\n }\n\n void leaveTest( const TestDescription & )\n {\n if ( stream_redirect != NULL )\n {\n std::string out = stream_redirect->out.str();\n if ( ! out.empty() )\n {\n // silently ignore the '.'\n if ( out[0] != '.' || out.size() > 1 )\n testcase->add_element(\"system-out\")->value << out;\n }\n if ( ! stream_redirect->err.str().empty() )\n testcase->add_element(\"system-err\")->value << stream_redirect->err.str();\n\n delete stream_redirect;\n stream_redirect = NULL;\n }\n }\n\n void leaveWorld( const WorldDescription& desc )\n {\n std::ostringstream os;\n os << totaltime;\n (*_o) << \"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\" ?>\" << endl;\n (*_o) << \"<testsuite name=\\\"\" << desc.worldName() << \"\\\" \";\n (*_o) << \" tests=\\\"\" << ntests \n << \"\\\" errors=\\\"\" << nerror \n << \"\\\" failures=\\\"\" << nfail \n << \"\\\" time=\\\"\" << os.str().c_str() << \"\\\" >\";\n _o->endl(*_o);\n (*_o) << _os->str().c_str();\n _os->clear();\n (*_o) << \"</testsuite>\" << endl;\n _o->flush();\n }\n\n void trace( const char* /*file*/, int line, const char *expression )\n {\n element_t elt = testcase->add_element(\"trace\");\n elt->add(\"line\",line);\n elt->value << expression;\n }\n\n void warning( const char* /*file*/, int line, const char *expression )\n {\n element_t elt = testcase->add_element(\"warning\");\n elt->add(\"line\",line);\n elt->value << expression;\n }\n\n void failedTest( const char* file, int line, const char* expression )\n {\n testFailure( file, line, \"failure\") << \"Test failed: \" << expression;\n }\n\n void failedAssert( const char *file, int line, const char *expression )\n {\n testFailure( file, line, \"failedAssert\" ) \n << \"Assertion failed: \" << expression;\n }\n\n void failedAssertEquals( const char *file, int line,\n const char* xStr, const char* yStr,\n const char *x, const char *y )\n {\n testFailure( file, line, \"failedAssertEquals\" )\n << \"Error: Expected (\"\n << xStr << \" == \" << yStr << \"), found (\"\n << x << \" != \" << y << \")\";\n }\n\n void failedAssertSameData( const char *file, int line,\n const char *xStr, const char *yStr, const char *sizeStr,\n const void* /*x*/, const void* /*y*/, unsigned size )\n {\n testFailure( file, line, \"failedAssertSameData\")\n << \"Error: Expected \" << sizeStr \n << \" (\" << size << \") bytes to be equal at (\"\n << xStr << \") and (\" << yStr << \"), found\";\n }\n\n void failedAssertSameFiles( const char *file, int line,\n const char *, const char *,\n const char* explanation\n )\n {\n testFailure( file, line, \"failedAssertSameFiles\" )\n << \"Error: \" << explanation;\n }\n\n void failedAssertDelta( const char *file, int line,\n const char *xStr, const char *yStr, const char *dStr,\n const char *x, const char *y, const char *d )\n {\n testFailure( file, line, \"failedAssertDelta\" )\n << \"Error: Expected (\" \n << xStr << \" == \" << yStr << \") up to \" << dStr \n << \" (\" << d << \"), found (\" \n << x << \" != \" << y << \")\";\n }\n\n void failedAssertDiffers( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *value )\n {\n testFailure( file, line, \"failedAssertDiffers\" )\n << \"Error: Expected (\" \n << xStr << \" != \" << yStr << \"), found (\" \n << value << \")\";\n }\n\n void failedAssertLessThan( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *x, const char *y )\n {\n testFailure( file, line, \"failedAssertLessThan\" )\n << \"Error: Expected (\" <<\n xStr << \" < \" << yStr << \"), found (\" <<\n x << \" >= \" << y << \")\";\n }\n\n void failedAssertLessThanEquals( const char *file, int line,\n const char *xStr, const char *yStr,\n const char *x, const char *y )\n {\n testFailure( file, line, \"failedAssertLessThanEquals\" )\n << \"Error: Expected (\" <<\n xStr << \" <= \" << yStr << \"), found (\" <<\n x << \" > \" << y << \")\";\n }\n\n void failedAssertRelation( const char *file, int line,\n const char *relation, const char *xStr, const char *yStr,\n const char *x, const char *y )\n {\n testFailure( file, line, \"failedAssertRelation\" )\n << \"Error: Expected \" << relation << \"( \" <<\n xStr << \", \" << yStr << \" ), found !\" << relation \n << \"( \" << x << \", \" << y << \" )\";\n }\n\n void failedAssertPredicate( const char *file, int line,\n const char *predicate, const char *xStr, const char *x )\n {\n testFailure( file, line, \"failedAssertPredicate\" )\n << \"Error: Expected \" << predicate << \"( \" <<\n xStr << \" ), found !\" << predicate << \"( \" << x << \" )\";\n }\n\n void failedAssertThrows( const char *file, int line,\n const char *expression, const char *type,\n bool otherThrown )\n {\n testFailure( file, line, \"failedAssertThrows\" )\n << \"Error: Expected (\" << expression << \") to throw (\" <<\n type << \") but it \" \n << (otherThrown ? \"threw something else\" : \"didn't throw\");\n }\n\n void failedAssertThrowsNot( const char *file, int line, const char *expression )\n {\n testFailure( file, line, \"failedAssertThrowsNot\" )\n << \"Error: Expected (\" << expression \n << \") not to throw, but it did\";\n }\n\n protected:\n\n OutputStream *outputStream() const\n {\n return _o;\n }\n\n OutputStream *outputFileStream() const\n {\n return _ostr;\n }\n\n private:\n XmlFormatter( const XmlFormatter & );\n XmlFormatter &operator=( const XmlFormatter & );\n\n std::stringstream& testFailure( const char* file, int line, const char *failureType)\n {\n testcase->fail=true;\n element_t elt = testcase->update_element(\"failure\");\n if ( elt->value.str().empty() )\n {\n elt->add(\"type\",failureType);\n elt->add(\"line\",line);\n elt->add(\"file\",file);\n }\n else\n elt->value << CXXTEST_STD(endl);\n return elt->value;\n //failedTest(file,line,message.c_str());\n }\n\n#if 0\n void attributeBinary( const char* name, const void *value, unsigned size )\n {\n (*_o) << name;\n (*_o) << \"=\\\"\";\n dump(value, size);\n (*_o) << \"\\\" \";\n }\n\n void dump( const void *buffer, unsigned size )\n {\n if (!buffer) return;\n\n unsigned dumpSize = size;\n if ( maxDumpSize() && dumpSize > maxDumpSize() )\n dumpSize = maxDumpSize();\n\n const unsigned char *p = (const unsigned char *)buffer;\n for ( unsigned i = 0; i < dumpSize; ++ i )\n (*_o) << byteToHex( *p++ ) << \" \";\n if ( dumpSize < size )\n (*_o) << \"... \";\n }\n#endif\n\n static void endl( OutputStream &o )\n {\n OutputStream::endl( o );\n }\n\n OutputStream *_o;\n OutputStream *_ostr;\n std::ostringstream *_os;\n\n TeeOutputStreams *stream_redirect;\n };\n}\n\n#endif // __CXXTEST__XMLFORMATTER_H\n\n" }, { "path": "cxxtest/cxxtest/XmlPrinter.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__XmlPrinter_h__\n#define __cxxtest__XmlPrinter_h__\n\n//\n// The XmlPrinter is a simple TestListener that\n// prints JUnit style xml to the output stream\n//\n\n\n#include <cxxtest/Flags.h>\n\n#ifndef _CXXTEST_HAVE_STD\n# define _CXXTEST_HAVE_STD\n#endif // _CXXTEST_HAVE_STD\n\n#include <cxxtest/XmlFormatter.h>\n#include <cxxtest/StdValueTraits.h>\n\n#include <sstream>\n#ifdef _CXXTEST_OLD_STD\n# include <iostream.h>\n#else // !_CXXTEST_OLD_STD\n# include <iostream>\n#endif // _CXXTEST_OLD_STD\n\nnamespace CxxTest\n{\n class XmlPrinter : public XmlFormatter\n {\n public:\n XmlPrinter( CXXTEST_STD(ostream) &o = CXXTEST_STD(cout), const char* /*preLine*/ = \":\", const char* /*postLine*/ = \"\" ) :\n XmlFormatter( new Adapter(o), new Adapter(ostr), &ostr ) {}\n\n virtual ~XmlPrinter() \n {\n delete outputStream(); \n delete outputFileStream(); \n }\n\n private:\n\n std::ostringstream ostr;\n\n class Adapter : public OutputStream\n {\n CXXTEST_STD(ostream) &_o;\n public:\n Adapter( CXXTEST_STD(ostream) &o ) : _o(o) {}\n void flush() { _o.flush(); }\n OutputStream &operator<<( const char *s ) { _o << s; return *this; }\n OutputStream &operator<<( Manipulator m ) { return OutputStream::operator<<( m ); }\n OutputStream &operator<<( unsigned i )\n {\n char s[1 + 3 * sizeof(unsigned)];\n numberToString( i, s );\n _o << s;\n return *this;\n }\n };\n };\n}\n\n#endif // __cxxtest__XmlPrinter_h__\n\n" }, { "path": "cxxtest/cxxtest/YesNoRunner.h", "content": "/*\n-------------------------------------------------------------------------\n CxxTest: A lightweight C++ unit testing library.\n Copyright (c) 2008 Sandia Corporation.\n This software is distributed under the LGPL License v2.1\n For more information, see the COPYING file in the top CxxTest directory.\n Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n the U.S. Government retains certain rights in this software.\n-------------------------------------------------------------------------\n*/\n\n#ifndef __cxxtest__YesNoRunner_h__\n#define __cxxtest__YesNoRunner_h__\n\n//\n// The YesNoRunner is a simple TestListener that\n// just returns true iff all tests passed.\n//\n\n#include <cxxtest/TestRunner.h>\n#include <cxxtest/TestListener.h>\n\nnamespace CxxTest \n{\n class YesNoRunner : public TestListener\n {\n public:\n YesNoRunner()\n {\n }\n \n int run()\n {\n TestRunner::runAllTests( *this );\n return tracker().failedTests();\n }\n };\n}\n\n#endif // __cxxtest__YesNoRunner_h__\n" }, { "path": "cxxtest/cxxtest/__init__.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n\"\"\"cxxtest: A Python package that supports the CxxTest test framework for C/C++.\n\n.. _CxxTest: http://cxxtest.tigris.org/\n\nCxxTest is a unit testing framework for C++ that is similar in\nspirit to JUnit, CppUnit, and xUnit. CxxTest is easy to use because\nit does not require precompiling a CxxTest testing library, it\nemploys no advanced features of C++ (e.g. RTTI) and it supports a\nvery flexible form of test discovery.\n\nThe cxxtest Python package includes capabilities for parsing C/C++ source files and generating\nCxxTest drivers.\n\"\"\"\n\nfrom cxxtest.__release__ import __version__, __date__\n__date__\n__version__\n\n__maintainer__ = \"William E. Hart\"\n__maintainer_email__ = \"whart222@gmail.com\"\n__license__ = \"LGPL\"\n__url__ = \"http://cxxtest.tigris.org/\"\n\nfrom cxxtest.cxxtestgen import *\n" }, { "path": "cxxtest/cxxtest/__release__.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n\"\"\" Release Information for cxxtest \"\"\"\n\n__version__ = '4.0.2'\n__date__ = \"2012-01-02\"\n" }, { "path": "cxxtest/cxxtest/cxx_parser.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n# vim: fileencoding=utf-8\n\n#\n# This is a PLY parser for the entire ANSI C++ grammar. This grammar was \n# adapted from the FOG grammar developed by E. D. Willink. See\n#\n# http://www.computing.surrey.ac.uk/research/dsrg/fog/\n#\n# for further details.\n#\n# The goal of this grammar is to extract information about class, function and\n# class method declarations, along with their associated scope. Thus, this \n# grammar can be used to analyze classes in an inheritance heirarchy, and then\n# enumerate the methods in a derived class.\n#\n# This grammar parses blocks of <>, (), [] and {} in a generic manner. Thus,\n# There are several capabilities that this grammar does not support:\n#\n# 1. Ambiguous template specification. This grammar cannot parse template\n# specifications that do not have paired <>'s in their declaration. In\n# particular, ambiguous declarations like\n#\n# foo<A, c<3 >();\n#\n# cannot be correctly parsed.\n#\n# 2. Template class specialization. Although the goal of this grammar is to\n# extract class information, specialization of templated classes is\n# not supported. When a template class definition is parsed, it's \n# declaration is archived without information about the template\n# parameters. Class specializations will be stored separately, and \n# thus they can be processed after the fact. However, this grammar\n# does not attempt to correctly process properties of class inheritence\n# when template class specialization is employed.\n#\n\n#\n# TODO: document usage of this file\n#\n\nfrom __future__ import division\n\nimport os\nimport ply.lex as lex\nimport ply.yacc as yacc\nimport re\ntry:\n from collections import OrderedDict\nexcept ImportError:\n from ordereddict import OrderedDict\n\nlexer = None\nscope_lineno = 0\nidentifier_lineno = {}\n_parse_info=None\n_parsedata=None\nnoExceptionLogic = True\n\ndef ply_init(data):\n global _parsedata\n _parsedata=data\n\n\nclass Scope(object):\n\n def __init__(self,name,abs_name,scope_t,base_classes,lineno):\n self.function=[]\n self.name=name\n self.scope_t=scope_t\n self.sub_scopes=[]\n self.base_classes=base_classes\n self.abs_name=abs_name\n self.lineno=lineno\n \n def insert(self,scope):\n self.sub_scopes.append(scope)\n\n\nclass CppInfo(object):\n\n def __init__(self, filter=None):\n self.verbose=0\n if filter is None:\n self.filter=re.compile(\"[Tt][Ee][Ss][Tt]|createSuite|destroySuite\")\n else:\n self.filter=filter\n self.scopes=[\"\"]\n self.index=OrderedDict()\n self.index[\"\"]=Scope(\"\",\"::\",\"namespace\",[],1)\n self.function=[]\n\n def push_scope(self,ns,scope_t,base_classes=[]):\n name = self.scopes[-1]+\"::\"+ns\n if self.verbose>=2:\n print \"-- Starting \"+scope_t+\" \"+name\n self.scopes.append(name)\n self.index[name] = Scope(ns,name,scope_t,base_classes,scope_lineno-1)\n\n def pop_scope(self):\n scope = self.scopes.pop()\n if self.verbose>=2:\n print \"-- Stopping \"+scope\n return scope\n\n def add_function(self, fn):\n fn = str(fn)\n if self.filter.search(fn):\n self.index[self.scopes[-1]].function.append((fn, identifier_lineno.get(fn,lexer.lineno-1)))\n tmp = self.scopes[-1]+\"::\"+fn\n if self.verbose==2:\n print \"-- Function declaration \"+fn+\" \"+tmp\n elif self.verbose==1:\n print \"-- Function declaration \"+tmp\n\n def get_functions(self,name,quiet=False):\n if name == \"::\":\n name = \"\"\n scope = self.index[name]\n fns=scope.function\n for key in scope.base_classes:\n cname = self.find_class(key,scope)\n if cname is None:\n if not quiet:\n print \"Defined classes: \",list(self.index.keys())\n print \"WARNING: Unknown class \"+key\n else:\n fns += self.get_functions(cname,quiet)\n return fns\n \n def find_class(self,name,scope):\n if ':' in name:\n if name in self.index:\n return name\n else:\n return None \n tmp = scope.abs_name.split(':')\n name1 = \":\".join(tmp[:-1] + [name])\n if name1 in self.index:\n return name1\n name2 = \"::\"+name\n if name2 in self.index:\n return name2\n return None\n\n def __repr__(self):\n return str(self)\n\n def is_baseclass(self,cls,base):\n '''Returns true if base is a base-class of cls'''\n if cls in self.index:\n bases = self.index[cls]\n elif \"::\"+cls in self.index:\n bases = self.index[\"::\"+cls]\n else:\n return False\n #raise IOError, \"Unknown class \"+cls\n if base in bases.base_classes:\n return True\n for name in bases.base_classes:\n if self.is_baseclass(name,base):\n return True\n return False\n\n def __str__(self):\n ans=\"\"\n keys = list(self.index.keys())\n keys.sort()\n for key in keys:\n scope = self.index[key]\n ans += scope.scope_t+\" \"+scope.abs_name+\"\\n\"\n if scope.scope_t == \"class\":\n ans += \" Base Classes: \"+str(scope.base_classes)+\"\\n\"\n for fn in self.get_functions(scope.abs_name):\n ans += \" \"+fn+\"\\n\"\n else:\n for fn in scope.function:\n ans += \" \"+fn+\"\\n\"\n return ans\n\n\ndef flatten(x):\n \"\"\"Flatten nested list\"\"\"\n try:\n strtypes = basestring\n except: # for python3 etc\n strtypes = (str, bytes)\n\n result = []\n for el in x:\n if hasattr(el, \"__iter__\") and not isinstance(el, strtypes):\n result.extend(flatten(el))\n else:\n result.append(el)\n return result\n\n#\n# The lexer (and/or a preprocessor) is expected to identify the following\n#\n# Punctuation:\n#\n#\nliterals = \"+-*/%^&|~!<>=:()?.\\'\\\"\\\\@$;,\"\n\n#\nreserved = {\n 'private' : 'PRIVATE',\n 'protected' : 'PROTECTED',\n 'public' : 'PUBLIC',\n\n 'bool' : 'BOOL',\n 'char' : 'CHAR',\n 'double' : 'DOUBLE',\n 'float' : 'FLOAT',\n 'int' : 'INT',\n 'long' : 'LONG',\n 'short' : 'SHORT',\n 'signed' : 'SIGNED',\n 'unsigned' : 'UNSIGNED',\n 'void' : 'VOID',\n 'wchar_t' : 'WCHAR_T',\n\n 'class' : 'CLASS',\n 'enum' : 'ENUM',\n 'namespace' : 'NAMESPACE',\n 'struct' : 'STRUCT',\n 'typename' : 'TYPENAME',\n 'union' : 'UNION',\n\n 'const' : 'CONST',\n 'volatile' : 'VOLATILE',\n\n 'auto' : 'AUTO',\n 'explicit' : 'EXPLICIT',\n 'export' : 'EXPORT',\n 'extern' : 'EXTERN',\n '__extension__' : 'EXTENSION',\n 'friend' : 'FRIEND',\n 'inline' : 'INLINE',\n 'mutable' : 'MUTABLE',\n 'register' : 'REGISTER',\n 'static' : 'STATIC',\n 'template' : 'TEMPLATE',\n 'typedef' : 'TYPEDEF',\n 'using' : 'USING',\n 'virtual' : 'VIRTUAL',\n\n 'asm' : 'ASM',\n 'break' : 'BREAK',\n 'case' : 'CASE',\n 'catch' : 'CATCH',\n 'const_cast' : 'CONST_CAST',\n 'continue' : 'CONTINUE',\n 'default' : 'DEFAULT',\n 'delete' : 'DELETE',\n 'do' : 'DO',\n 'dynamic_cast' : 'DYNAMIC_CAST',\n 'else' : 'ELSE',\n 'false' : 'FALSE',\n 'for' : 'FOR',\n 'goto' : 'GOTO',\n 'if' : 'IF',\n 'new' : 'NEW',\n 'operator' : 'OPERATOR',\n 'reinterpret_cast' : 'REINTERPRET_CAST',\n 'return' : 'RETURN',\n 'sizeof' : 'SIZEOF',\n 'static_cast' : 'STATIC_CAST',\n 'switch' : 'SWITCH',\n 'this' : 'THIS',\n 'throw' : 'THROW',\n 'true' : 'TRUE',\n 'try' : 'TRY',\n 'typeid' : 'TYPEID',\n 'while' : 'WHILE',\n '\"C\"' : 'CLiteral',\n '\"C++\"' : 'CppLiteral',\n\n '__attribute__' : 'ATTRIBUTE',\n '__cdecl__' : 'CDECL',\n '__typeof' : 'uTYPEOF',\n 'typeof' : 'TYPEOF', \n\n 'CXXTEST_STD' : 'CXXTEST_STD'\n}\n \ntokens = [\n \"CharacterLiteral\",\n \"FloatingLiteral\",\n \"Identifier\",\n \"IntegerLiteral\",\n \"StringLiteral\",\n \"RBRACE\",\n \"LBRACE\",\n \"RBRACKET\",\n \"LBRACKET\",\n \"ARROW\",\n \"ARROW_STAR\",\n \"DEC\",\n \"EQ\",\n \"GE\",\n \"INC\",\n \"LE\",\n \"LOG_AND\",\n \"LOG_OR\",\n \"NE\",\n \"SHL\",\n \"SHR\",\n \"ASS_ADD\",\n \"ASS_AND\",\n \"ASS_DIV\",\n \"ASS_MOD\",\n \"ASS_MUL\",\n \"ASS_OR\",\n \"ASS_SHL\",\n \"ASS_SHR\",\n \"ASS_SUB\",\n \"ASS_XOR\",\n \"DOT_STAR\",\n \"ELLIPSIS\",\n \"SCOPE\",\n] + list(reserved.values())\n\nt_ignore = \" \\t\\r\"\n\nt_LBRACE = r\"(\\{)|(<%)\"\nt_RBRACE = r\"(\\})|(%>)\"\nt_LBRACKET = r\"(\\[)|(<:)\"\nt_RBRACKET = r\"(\\])|(:>)\"\nt_ARROW = r\"->\"\nt_ARROW_STAR = r\"->\\*\"\nt_DEC = r\"--\"\nt_EQ = r\"==\"\nt_GE = r\">=\"\nt_INC = r\"\\+\\+\"\nt_LE = r\"<=\"\nt_LOG_AND = r\"&&\"\nt_LOG_OR = r\"\\|\\|\"\nt_NE = r\"!=\"\nt_SHL = r\"<<\"\nt_SHR = r\">>\"\nt_ASS_ADD = r\"\\+=\"\nt_ASS_AND = r\"&=\"\nt_ASS_DIV = r\"/=\"\nt_ASS_MOD = r\"%=\"\nt_ASS_MUL = r\"\\*=\"\nt_ASS_OR = r\"\\|=\"\nt_ASS_SHL = r\"<<=\"\nt_ASS_SHR = r\">>=\"\nt_ASS_SUB = r\"-=\"\nt_ASS_XOR = r\"^=\"\nt_DOT_STAR = r\"\\.\\*\"\nt_ELLIPSIS = r\"\\.\\.\\.\"\nt_SCOPE = r\"::\"\n\n# Discard comments\ndef t_COMMENT(t):\n r'(/\\*(.|\\n)*?\\*/)|(//.*?\\n)|(\\#.*?\\n)'\n t.lexer.lineno += t.value.count(\"\\n\")\n\nt_IntegerLiteral = r'(0x[0-9A-F]+)|([0-9]+(L){0,1})'\nt_FloatingLiteral = r\"[0-9]+[eE\\.\\+-]+[eE\\.\\+\\-0-9]+\"\nt_CharacterLiteral = r'\\'([^\\'\\\\]|\\\\.)*\\''\n#t_StringLiteral = r'\"([^\"\\\\]|\\\\.)*\"'\ndef t_StringLiteral(t):\n r'\"([^\"\\\\]|\\\\.)*\"'\n t.type = reserved.get(t.value,'StringLiteral')\n return t\n\ndef t_Identifier(t):\n r\"[a-zA-Z_][a-zA-Z_0-9\\.]*\"\n t.type = reserved.get(t.value,'Identifier')\n return t\n\n\ndef t_error(t):\n print \"Illegal character '%s'\" % t.value[0]\n #raise IOError, \"Parse error\"\n #t.lexer.skip()\n\ndef t_newline(t):\n r'[\\n]+'\n t.lexer.lineno += len(t.value)\n\nprecedence = (\n ( 'right', 'SHIFT_THERE', 'REDUCE_HERE_MOSTLY', 'SCOPE'),\n ( 'nonassoc', 'ELSE', 'INC', 'DEC', '+', '-', '*', '&', 'LBRACKET', 'LBRACE', '<', ':', ')')\n )\n\nstart = 'translation_unit'\n\n#\n# The %prec resolves the 14.2-3 ambiguity:\n# Identifier '<' is forced to go through the is-it-a-template-name test\n# All names absorb TEMPLATE with the name, so that no template_test is \n# performed for them. This requires all potential declarations within an \n# expression to perpetuate this policy and thereby guarantee the ultimate \n# coverage of explicit_instantiation.\n#\n# The %prec also resolves a conflict in identifier : which is forced to be a \n# shift of a label for a labeled-statement rather than a reduction for the \n# name of a bit-field or generalised constructor. This is pretty dubious \n# syntactically but correct for all semantic possibilities. The shift is \n# only activated when the ambiguity exists at the start of a statement. \n# In this context a bit-field declaration or constructor definition are not \n# allowed.\n#\n\ndef p_identifier(p):\n '''identifier : Identifier\n | CXXTEST_STD '(' Identifier ')'\n '''\n if p[1][0] in ('t','T','c','d'):\n identifier_lineno[p[1]] = p.lineno(1)\n p[0] = p[1]\n\ndef p_id(p):\n '''id : identifier %prec SHIFT_THERE\n | template_decl\n | TEMPLATE id\n '''\n p[0] = get_rest(p)\n\ndef p_global_scope(p):\n '''global_scope : SCOPE\n '''\n p[0] = get_rest(p)\n\ndef p_id_scope(p):\n '''id_scope : id SCOPE'''\n p[0] = get_rest(p)\n\ndef p_id_scope_seq(p):\n '''id_scope_seq : id_scope\n | id_scope id_scope_seq\n '''\n p[0] = get_rest(p)\n\n#\n# A :: B :: C; is ambiguous How much is type and how much name ?\n# The %prec maximises the (type) length which is the 7.1-2 semantic constraint.\n#\ndef p_nested_id(p):\n '''nested_id : id %prec SHIFT_THERE\n | id_scope nested_id\n '''\n p[0] = get_rest(p)\n\ndef p_scoped_id(p):\n '''scoped_id : nested_id\n | global_scope nested_id\n | id_scope_seq\n | global_scope id_scope_seq\n '''\n global scope_lineno\n scope_lineno = lexer.lineno\n data = flatten(get_rest(p))\n if data[0] != None:\n p[0] = \"\".join(data)\n\n#\n# destructor_id has to be held back to avoid a conflict with a one's \n# complement as per 5.3.1-9, It gets put back only when scoped or in a \n# declarator_id, which is only used as an explicit member name.\n# Declarations of an unscoped destructor are always parsed as a one's \n# complement.\n#\ndef p_destructor_id(p):\n '''destructor_id : '~' id\n | TEMPLATE destructor_id\n '''\n p[0]=get_rest(p)\n\n#def p_template_id(p):\n# '''template_id : empty\n# | TEMPLATE\n# '''\n# pass\n\ndef p_template_decl(p):\n '''template_decl : identifier '<' nonlgt_seq_opt '>'\n '''\n #\n # WEH: should we include the lt/gt symbols to indicate that this is a\n # template class? How is that going to be used later???\n #\n #p[0] = [p[1] ,\"<\",\">\"]\n p[0] = p[1]\n\ndef p_special_function_id(p):\n '''special_function_id : conversion_function_id\n | operator_function_id\n | TEMPLATE special_function_id\n '''\n p[0]=get_rest(p)\n\ndef p_nested_special_function_id(p):\n '''nested_special_function_id : special_function_id\n | id_scope destructor_id\n | id_scope nested_special_function_id\n '''\n p[0]=get_rest(p)\n\ndef p_scoped_special_function_id(p):\n '''scoped_special_function_id : nested_special_function_id\n | global_scope nested_special_function_id\n '''\n p[0]=get_rest(p)\n\n# declarator-id is all names in all scopes, except reserved words\ndef p_declarator_id(p):\n '''declarator_id : scoped_id\n | scoped_special_function_id\n | destructor_id\n '''\n p[0]=p[1]\n\n#\n# The standard defines pseudo-destructors in terms of type-name, which is \n# class/enum/typedef, of which class-name is covered by a normal destructor. \n# pseudo-destructors are supposed to support ~int() in templates, so the \n# grammar here covers built-in names. Other names are covered by the lack \n# of identifier/type discrimination.\n#\ndef p_built_in_type_id(p):\n '''built_in_type_id : built_in_type_specifier\n | built_in_type_id built_in_type_specifier\n '''\n pass\n\ndef p_pseudo_destructor_id(p):\n '''pseudo_destructor_id : built_in_type_id SCOPE '~' built_in_type_id\n | '~' built_in_type_id\n | TEMPLATE pseudo_destructor_id\n '''\n pass\n\ndef p_nested_pseudo_destructor_id(p):\n '''nested_pseudo_destructor_id : pseudo_destructor_id\n | id_scope nested_pseudo_destructor_id\n '''\n pass\n\ndef p_scoped_pseudo_destructor_id(p):\n '''scoped_pseudo_destructor_id : nested_pseudo_destructor_id\n | global_scope scoped_pseudo_destructor_id\n '''\n pass\n\n#-------------------------------------------------------------------------------\n# A.2 Lexical conventions\n#-------------------------------------------------------------------------------\n#\n\ndef p_literal(p):\n '''literal : IntegerLiteral\n | CharacterLiteral\n | FloatingLiteral\n | StringLiteral\n | TRUE\n | FALSE\n '''\n pass\n\n#-------------------------------------------------------------------------------\n# A.3 Basic concepts\n#-------------------------------------------------------------------------------\ndef p_translation_unit(p):\n '''translation_unit : declaration_seq_opt\n '''\n pass\n\n#-------------------------------------------------------------------------------\n# A.4 Expressions\n#-------------------------------------------------------------------------------\n#\n# primary_expression covers an arbitrary sequence of all names with the \n# exception of an unscoped destructor, which is parsed as its unary expression \n# which is the correct disambiguation (when ambiguous). This eliminates the \n# traditional A(B) meaning A B ambiguity, since we never have to tack an A \n# onto the front of something that might start with (. The name length got \n# maximised ab initio. The downside is that semantic interpretation must split \n# the names up again.\n#\n# Unification of the declaration and expression syntax means that unary and \n# binary pointer declarator operators:\n# int * * name\n# are parsed as binary and unary arithmetic operators (int) * (*name). Since \n# type information is not used\n# ambiguities resulting from a cast\n# (cast)*(value)\n# are resolved to favour the binary rather than the cast unary to ease AST \n# clean-up. The cast-call ambiguity must be resolved to the cast to ensure \n# that (a)(b)c can be parsed.\n#\n# The problem of the functional cast ambiguity\n# name(arg)\n# as call or declaration is avoided by maximising the name within the parsing \n# kernel. So primary_id_expression picks up \n# extern long int const var = 5;\n# as an assignment to the syntax parsed as \"extern long int const var\". The \n# presence of two names is parsed so that \"extern long into const\" is \n# distinguished from \"var\" considerably simplifying subsequent \n# semantic resolution.\n#\n# The generalised name is a concatenation of potential type-names (scoped \n# identifiers or built-in sequences) plus optionally one of the special names \n# such as an operator-function-id, conversion-function-id or destructor as the \n# final name. \n#\n\ndef get_rest(p):\n return [p[i] for i in range(1, len(p))]\n\ndef p_primary_expression(p):\n '''primary_expression : literal\n | THIS\n | suffix_decl_specified_ids\n | abstract_expression %prec REDUCE_HERE_MOSTLY\n '''\n p[0] = get_rest(p)\n\n#\n# Abstract-expression covers the () and [] of abstract-declarators.\n#\ndef p_abstract_expression(p):\n '''abstract_expression : parenthesis_clause\n | LBRACKET bexpression_opt RBRACKET\n | TEMPLATE abstract_expression\n '''\n pass\n\ndef p_postfix_expression(p):\n '''postfix_expression : primary_expression\n | postfix_expression parenthesis_clause\n | postfix_expression LBRACKET bexpression_opt RBRACKET\n | postfix_expression LBRACKET bexpression_opt RBRACKET attributes\n | postfix_expression '.' declarator_id\n | postfix_expression '.' scoped_pseudo_destructor_id\n | postfix_expression ARROW declarator_id\n | postfix_expression ARROW scoped_pseudo_destructor_id \n | postfix_expression INC\n | postfix_expression DEC\n | DYNAMIC_CAST '<' nonlgt_seq_opt '>' '(' expression ')'\n | STATIC_CAST '<' nonlgt_seq_opt '>' '(' expression ')'\n | REINTERPRET_CAST '<' nonlgt_seq_opt '>' '(' expression ')'\n | CONST_CAST '<' nonlgt_seq_opt '>' '(' expression ')'\n | TYPEID parameters_clause\n '''\n #print \"HERE\",str(p[1])\n p[0] = get_rest(p)\n\ndef p_bexpression_opt(p):\n '''bexpression_opt : empty\n | bexpression\n '''\n pass\n\ndef p_bexpression(p):\n '''bexpression : nonbracket_seq\n | nonbracket_seq bexpression_seq bexpression_clause nonbracket_seq_opt\n | bexpression_seq bexpression_clause nonbracket_seq_opt\n '''\n pass\n\ndef p_bexpression_seq(p):\n '''bexpression_seq : empty\n | bexpression_seq bexpression_clause nonbracket_seq_opt\n '''\n pass\n\ndef p_bexpression_clause(p):\n '''bexpression_clause : LBRACKET bexpression_opt RBRACKET\n '''\n pass\n\n\n\ndef p_expression_list_opt(p):\n '''expression_list_opt : empty\n | expression_list\n '''\n pass\n\ndef p_expression_list(p):\n '''expression_list : assignment_expression\n | expression_list ',' assignment_expression\n '''\n pass\n\ndef p_unary_expression(p):\n '''unary_expression : postfix_expression\n | INC cast_expression\n | DEC cast_expression\n | ptr_operator cast_expression\n | suffix_decl_specified_scope star_ptr_operator cast_expression\n | '+' cast_expression\n | '-' cast_expression\n | '!' cast_expression\n | '~' cast_expression\n | SIZEOF unary_expression\n | new_expression\n | global_scope new_expression\n | delete_expression\n | global_scope delete_expression\n '''\n p[0] = get_rest(p)\n\ndef p_delete_expression(p):\n '''delete_expression : DELETE cast_expression\n '''\n pass\n\ndef p_new_expression(p):\n '''new_expression : NEW new_type_id new_initializer_opt\n | NEW parameters_clause new_type_id new_initializer_opt\n | NEW parameters_clause\n | NEW parameters_clause parameters_clause new_initializer_opt\n '''\n pass\n\ndef p_new_type_id(p):\n '''new_type_id : type_specifier ptr_operator_seq_opt\n | type_specifier new_declarator\n | type_specifier new_type_id\n '''\n pass\n\ndef p_new_declarator(p):\n '''new_declarator : ptr_operator new_declarator\n | direct_new_declarator\n '''\n pass\n\ndef p_direct_new_declarator(p):\n '''direct_new_declarator : LBRACKET bexpression_opt RBRACKET\n | direct_new_declarator LBRACKET bexpression RBRACKET\n '''\n pass\n\ndef p_new_initializer_opt(p):\n '''new_initializer_opt : empty\n | '(' expression_list_opt ')'\n '''\n pass\n\n#\n# cast-expression is generalised to support a [] as well as a () prefix. This covers the omission of \n# DELETE[] which when followed by a parenthesised expression was ambiguous. It also covers the gcc \n# indexed array initialisation for free.\n#\ndef p_cast_expression(p):\n '''cast_expression : unary_expression\n | abstract_expression cast_expression\n '''\n p[0] = get_rest(p)\n\ndef p_pm_expression(p):\n '''pm_expression : cast_expression\n | pm_expression DOT_STAR cast_expression\n | pm_expression ARROW_STAR cast_expression\n '''\n p[0] = get_rest(p)\n\ndef p_multiplicative_expression(p):\n '''multiplicative_expression : pm_expression\n | multiplicative_expression star_ptr_operator pm_expression\n | multiplicative_expression '/' pm_expression\n | multiplicative_expression '%' pm_expression\n '''\n p[0] = get_rest(p)\n\ndef p_additive_expression(p):\n '''additive_expression : multiplicative_expression\n | additive_expression '+' multiplicative_expression\n | additive_expression '-' multiplicative_expression\n '''\n p[0] = get_rest(p)\n\ndef p_shift_expression(p):\n '''shift_expression : additive_expression\n | shift_expression SHL additive_expression\n | shift_expression SHR additive_expression\n '''\n p[0] = get_rest(p)\n\n# | relational_expression '<' shift_expression\n# | relational_expression '>' shift_expression\n# | relational_expression LE shift_expression\n# | relational_expression GE shift_expression\ndef p_relational_expression(p):\n '''relational_expression : shift_expression\n '''\n p[0] = get_rest(p)\n\ndef p_equality_expression(p):\n '''equality_expression : relational_expression\n | equality_expression EQ relational_expression\n | equality_expression NE relational_expression\n '''\n p[0] = get_rest(p)\n\ndef p_and_expression(p):\n '''and_expression : equality_expression\n | and_expression '&' equality_expression\n '''\n p[0] = get_rest(p)\n\ndef p_exclusive_or_expression(p):\n '''exclusive_or_expression : and_expression\n | exclusive_or_expression '^' and_expression\n '''\n p[0] = get_rest(p)\n\ndef p_inclusive_or_expression(p):\n '''inclusive_or_expression : exclusive_or_expression\n | inclusive_or_expression '|' exclusive_or_expression\n '''\n p[0] = get_rest(p)\n\ndef p_logical_and_expression(p):\n '''logical_and_expression : inclusive_or_expression\n | logical_and_expression LOG_AND inclusive_or_expression\n '''\n p[0] = get_rest(p)\n\ndef p_logical_or_expression(p):\n '''logical_or_expression : logical_and_expression\n | logical_or_expression LOG_OR logical_and_expression\n '''\n p[0] = get_rest(p)\n\ndef p_conditional_expression(p):\n '''conditional_expression : logical_or_expression\n | logical_or_expression '?' expression ':' assignment_expression\n '''\n p[0] = get_rest(p)\n\n\n#\n# assignment-expression is generalised to cover the simple assignment of a braced initializer in order to \n# contribute to the coverage of parameter-declaration and init-declaration.\n#\n# | logical_or_expression assignment_operator assignment_expression\ndef p_assignment_expression(p):\n '''assignment_expression : conditional_expression\n | logical_or_expression assignment_operator nonsemicolon_seq\n | logical_or_expression '=' braced_initializer\n | throw_expression\n '''\n p[0]=get_rest(p)\n\ndef p_assignment_operator(p):\n '''assignment_operator : '=' \n | ASS_ADD\n | ASS_AND\n | ASS_DIV\n | ASS_MOD\n | ASS_MUL\n | ASS_OR\n | ASS_SHL\n | ASS_SHR\n | ASS_SUB\n | ASS_XOR\n '''\n pass\n\n#\n# expression is widely used and usually single-element, so the reductions are arranged so that a\n# single-element expression is returned as is. Multi-element expressions are parsed as a list that\n# may then behave polymorphically as an element or be compacted to an element.\n#\n\ndef p_expression(p):\n '''expression : assignment_expression\n | expression_list ',' assignment_expression\n '''\n p[0] = get_rest(p)\n\ndef p_constant_expression(p):\n '''constant_expression : conditional_expression\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.5 Statements\n#---------------------------------------------------------------------------------------------------\n# Parsing statements is easy once simple_declaration has been generalised to cover expression_statement.\n#\n#\n# The use of extern here is a hack. The 'extern \"C\" {}' block gets parsed\n# as a function, so when nested 'extern \"C\"' declarations exist, they don't\n# work because the block is viewed as a list of statements... :(\n#\ndef p_statement(p):\n '''statement : compound_statement\n | declaration_statement\n | try_block\n | labeled_statement\n | selection_statement\n | iteration_statement\n | jump_statement\n '''\n pass\n\ndef p_compound_statement(p):\n '''compound_statement : LBRACE statement_seq_opt RBRACE\n '''\n pass\n\ndef p_statement_seq_opt(p):\n '''statement_seq_opt : empty\n | statement_seq_opt statement\n '''\n pass\n\n#\n# The dangling else conflict is resolved to the innermost if.\n#\ndef p_selection_statement(p):\n '''selection_statement : IF '(' condition ')' statement %prec SHIFT_THERE\n | IF '(' condition ')' statement ELSE statement\n | SWITCH '(' condition ')' statement\n '''\n pass\n\ndef p_condition_opt(p):\n '''condition_opt : empty\n | condition\n '''\n pass\n\ndef p_condition(p):\n '''condition : nonparen_seq\n | nonparen_seq condition_seq parameters_clause nonparen_seq_opt\n | condition_seq parameters_clause nonparen_seq_opt\n '''\n pass\n\ndef p_condition_seq(p):\n '''condition_seq : empty\n | condition_seq parameters_clause nonparen_seq_opt\n '''\n pass\n\ndef p_labeled_statement(p):\n '''labeled_statement : identifier ':' statement\n | CASE constant_expression ':' statement\n | DEFAULT ':' statement\n '''\n pass\n\ndef p_try_block(p):\n '''try_block : TRY compound_statement handler_seq\n '''\n global noExceptionLogic\n noExceptionLogic=False\n\ndef p_jump_statement(p):\n '''jump_statement : BREAK ';'\n | CONTINUE ';'\n | RETURN nonsemicolon_seq ';'\n | GOTO identifier ';'\n '''\n pass\n\ndef p_iteration_statement(p):\n '''iteration_statement : WHILE '(' condition ')' statement\n | DO statement WHILE '(' expression ')' ';'\n | FOR '(' nonparen_seq_opt ')' statement\n '''\n pass\n\ndef p_declaration_statement(p):\n '''declaration_statement : block_declaration\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.6 Declarations\n#---------------------------------------------------------------------------------------------------\ndef p_compound_declaration(p):\n '''compound_declaration : LBRACE declaration_seq_opt RBRACE \n '''\n pass\n\ndef p_declaration_seq_opt(p):\n '''declaration_seq_opt : empty\n | declaration_seq_opt declaration\n '''\n pass\n\ndef p_declaration(p):\n '''declaration : block_declaration\n | function_definition\n | template_declaration\n | explicit_specialization\n | specialised_declaration\n '''\n pass\n\ndef p_specialised_declaration(p):\n '''specialised_declaration : linkage_specification\n | namespace_definition\n | TEMPLATE specialised_declaration\n '''\n pass\n\ndef p_block_declaration(p):\n '''block_declaration : simple_declaration\n | specialised_block_declaration\n '''\n pass\n\ndef p_specialised_block_declaration(p):\n '''specialised_block_declaration : asm_definition\n | namespace_alias_definition\n | using_declaration\n | using_directive\n | TEMPLATE specialised_block_declaration\n '''\n pass\n\ndef p_simple_declaration(p):\n '''simple_declaration : ';'\n | init_declaration ';'\n | init_declarations ';'\n | decl_specifier_prefix simple_declaration\n '''\n global _parse_info\n if len(p) == 3:\n if p[2] == \";\":\n decl = p[1]\n else:\n decl = p[2]\n if decl is not None:\n fp = flatten(decl)\n if len(fp) >= 2 and fp[0] is not None and fp[0]!=\"operator\" and fp[1] == '(':\n p[0] = fp[0]\n _parse_info.add_function(fp[0])\n\n#\n# A decl-specifier following a ptr_operator provokes a shift-reduce conflict for * const name which is resolved in favour of the pointer, and implemented by providing versions of decl-specifier guaranteed not to start with a cv_qualifier. decl-specifiers are implemented type-centrically. That is the semantic constraint that there must be a type is exploited to impose structure, but actually eliminate very little syntax. built-in types are multi-name and so need a different policy.\n#\n# non-type decl-specifiers are bound to the left-most type in a decl-specifier-seq, by parsing from the right and attaching suffixes to the right-hand type. Finally residual prefixes attach to the left. \n#\ndef p_suffix_built_in_decl_specifier_raw(p):\n '''suffix_built_in_decl_specifier_raw : built_in_type_specifier\n | suffix_built_in_decl_specifier_raw built_in_type_specifier\n | suffix_built_in_decl_specifier_raw decl_specifier_suffix\n '''\n pass\n\ndef p_suffix_built_in_decl_specifier(p):\n '''suffix_built_in_decl_specifier : suffix_built_in_decl_specifier_raw\n | TEMPLATE suffix_built_in_decl_specifier\n '''\n pass\n\n# | id_scope_seq\n# | SCOPE id_scope_seq\ndef p_suffix_named_decl_specifier(p):\n '''suffix_named_decl_specifier : scoped_id \n | elaborate_type_specifier \n | suffix_named_decl_specifier decl_specifier_suffix\n '''\n p[0]=get_rest(p)\n\ndef p_suffix_named_decl_specifier_bi(p):\n '''suffix_named_decl_specifier_bi : suffix_named_decl_specifier\n | suffix_named_decl_specifier suffix_built_in_decl_specifier_raw\n '''\n p[0] = get_rest(p)\n #print \"HERE\",get_rest(p)\n\ndef p_suffix_named_decl_specifiers(p):\n '''suffix_named_decl_specifiers : suffix_named_decl_specifier_bi\n | suffix_named_decl_specifiers suffix_named_decl_specifier_bi\n '''\n p[0] = get_rest(p)\n\ndef p_suffix_named_decl_specifiers_sf(p):\n '''suffix_named_decl_specifiers_sf : scoped_special_function_id\n | suffix_named_decl_specifiers\n | suffix_named_decl_specifiers scoped_special_function_id\n '''\n #print \"HERE\",get_rest(p)\n p[0] = get_rest(p)\n\ndef p_suffix_decl_specified_ids(p):\n '''suffix_decl_specified_ids : suffix_built_in_decl_specifier\n | suffix_built_in_decl_specifier suffix_named_decl_specifiers_sf\n | suffix_named_decl_specifiers_sf\n '''\n if len(p) == 3:\n p[0] = p[2]\n else:\n p[0] = p[1]\n\ndef p_suffix_decl_specified_scope(p):\n '''suffix_decl_specified_scope : suffix_named_decl_specifiers SCOPE\n | suffix_built_in_decl_specifier suffix_named_decl_specifiers SCOPE\n | suffix_built_in_decl_specifier SCOPE\n '''\n p[0] = get_rest(p)\n\ndef p_decl_specifier_affix(p):\n '''decl_specifier_affix : storage_class_specifier\n | function_specifier\n | FRIEND\n | TYPEDEF\n | cv_qualifier\n '''\n pass\n\ndef p_decl_specifier_suffix(p):\n '''decl_specifier_suffix : decl_specifier_affix\n '''\n pass\n\ndef p_decl_specifier_prefix(p):\n '''decl_specifier_prefix : decl_specifier_affix\n | TEMPLATE decl_specifier_prefix\n '''\n pass\n\ndef p_storage_class_specifier(p):\n '''storage_class_specifier : REGISTER \n | STATIC \n | MUTABLE\n | EXTERN %prec SHIFT_THERE\n | EXTENSION\n | AUTO\n '''\n pass\n\ndef p_function_specifier(p):\n '''function_specifier : EXPLICIT\n | INLINE\n | VIRTUAL\n '''\n pass\n\ndef p_type_specifier(p):\n '''type_specifier : simple_type_specifier\n | elaborate_type_specifier\n | cv_qualifier\n '''\n pass\n\ndef p_elaborate_type_specifier(p):\n '''elaborate_type_specifier : class_specifier\n | enum_specifier\n | elaborated_type_specifier\n | TEMPLATE elaborate_type_specifier\n '''\n pass\n\ndef p_simple_type_specifier(p):\n '''simple_type_specifier : scoped_id\n | scoped_id attributes\n | built_in_type_specifier\n '''\n p[0] = p[1]\n\ndef p_built_in_type_specifier(p):\n '''built_in_type_specifier : Xbuilt_in_type_specifier\n | Xbuilt_in_type_specifier attributes\n '''\n pass\n\ndef p_attributes(p):\n '''attributes : attribute\n | attributes attribute\n '''\n pass\n\ndef p_attribute(p):\n '''attribute : ATTRIBUTE '(' parameters_clause ')'\n '''\n\ndef p_Xbuilt_in_type_specifier(p):\n '''Xbuilt_in_type_specifier : CHAR \n | WCHAR_T \n | BOOL \n | SHORT \n | INT \n | LONG \n | SIGNED \n | UNSIGNED \n | FLOAT \n | DOUBLE \n | VOID\n | uTYPEOF parameters_clause\n | TYPEOF parameters_clause\n '''\n pass\n\n#\n# The over-general use of declaration_expression to cover decl-specifier-seq_opt declarator in a function-definition means that\n# class X { };\n# could be a function-definition or a class-specifier.\n# enum X { };\n# could be a function-definition or an enum-specifier.\n# The function-definition is not syntactically valid so resolving the false conflict in favour of the\n# elaborated_type_specifier is correct.\n#\ndef p_elaborated_type_specifier(p):\n '''elaborated_type_specifier : class_key scoped_id %prec SHIFT_THERE\n | elaborated_enum_specifier\n | TYPENAME scoped_id\n '''\n pass\n\ndef p_elaborated_enum_specifier(p):\n '''elaborated_enum_specifier : ENUM scoped_id %prec SHIFT_THERE\n '''\n pass\n\ndef p_enum_specifier(p):\n '''enum_specifier : ENUM scoped_id enumerator_clause\n | ENUM enumerator_clause\n '''\n pass\n\ndef p_enumerator_clause(p):\n '''enumerator_clause : LBRACE enumerator_list_ecarb\n | LBRACE enumerator_list enumerator_list_ecarb\n | LBRACE enumerator_list ',' enumerator_definition_ecarb\n '''\n pass\n\ndef p_enumerator_list_ecarb(p):\n '''enumerator_list_ecarb : RBRACE\n '''\n pass\n\ndef p_enumerator_definition_ecarb(p):\n '''enumerator_definition_ecarb : RBRACE\n '''\n pass\n\ndef p_enumerator_definition_filler(p):\n '''enumerator_definition_filler : empty\n '''\n pass\n\ndef p_enumerator_list_head(p):\n '''enumerator_list_head : enumerator_definition_filler\n | enumerator_list ',' enumerator_definition_filler\n '''\n pass\n\ndef p_enumerator_list(p):\n '''enumerator_list : enumerator_list_head enumerator_definition\n '''\n pass\n\ndef p_enumerator_definition(p):\n '''enumerator_definition : enumerator\n | enumerator '=' constant_expression\n '''\n pass\n\ndef p_enumerator(p):\n '''enumerator : identifier\n '''\n pass\n\ndef p_namespace_definition(p):\n '''namespace_definition : NAMESPACE scoped_id push_scope compound_declaration\n | NAMESPACE push_scope compound_declaration\n '''\n global _parse_info\n scope = _parse_info.pop_scope()\n\ndef p_namespace_alias_definition(p):\n '''namespace_alias_definition : NAMESPACE scoped_id '=' scoped_id ';'\n '''\n pass\n\ndef p_push_scope(p):\n '''push_scope : empty'''\n global _parse_info\n if p[-2] == \"namespace\":\n scope=p[-1]\n else:\n scope=\"\"\n _parse_info.push_scope(scope,\"namespace\")\n\ndef p_using_declaration(p):\n '''using_declaration : USING declarator_id ';'\n | USING TYPENAME declarator_id ';'\n '''\n pass\n\ndef p_using_directive(p):\n '''using_directive : USING NAMESPACE scoped_id ';'\n '''\n pass\n\n# '''asm_definition : ASM '(' StringLiteral ')' ';'\ndef p_asm_definition(p):\n '''asm_definition : ASM '(' nonparen_seq_opt ')' ';'\n '''\n pass\n\ndef p_linkage_specification(p):\n '''linkage_specification : EXTERN CLiteral declaration\n | EXTERN CLiteral compound_declaration\n | EXTERN CppLiteral declaration\n | EXTERN CppLiteral compound_declaration\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.7 Declarators\n#---------------------------------------------------------------------------------------------------\n#\n# init-declarator is named init_declaration to reflect the embedded decl-specifier-seq_opt\n#\n\ndef p_init_declarations(p):\n '''init_declarations : assignment_expression ',' init_declaration\n | init_declarations ',' init_declaration\n '''\n p[0]=get_rest(p)\n\ndef p_init_declaration(p):\n '''init_declaration : assignment_expression\n '''\n p[0]=get_rest(p)\n\ndef p_star_ptr_operator(p):\n '''star_ptr_operator : '*'\n | star_ptr_operator cv_qualifier\n '''\n pass\n\ndef p_nested_ptr_operator(p):\n '''nested_ptr_operator : star_ptr_operator\n | id_scope nested_ptr_operator\n '''\n pass\n\ndef p_ptr_operator(p):\n '''ptr_operator : '&'\n | nested_ptr_operator\n | global_scope nested_ptr_operator\n '''\n pass\n\ndef p_ptr_operator_seq(p):\n '''ptr_operator_seq : ptr_operator\n | ptr_operator ptr_operator_seq\n '''\n pass\n\n#\n# Independently coded to localise the shift-reduce conflict: sharing just needs another %prec\n#\ndef p_ptr_operator_seq_opt(p):\n '''ptr_operator_seq_opt : empty %prec SHIFT_THERE\n | ptr_operator ptr_operator_seq_opt\n '''\n pass\n\ndef p_cv_qualifier_seq_opt(p):\n '''cv_qualifier_seq_opt : empty\n | cv_qualifier_seq_opt cv_qualifier\n '''\n pass\n\n# TODO: verify that we should include attributes here\ndef p_cv_qualifier(p):\n '''cv_qualifier : CONST \n | VOLATILE\n | attributes\n '''\n pass\n\ndef p_type_id(p):\n '''type_id : type_specifier abstract_declarator_opt\n | type_specifier type_id\n '''\n pass\n\ndef p_abstract_declarator_opt(p):\n '''abstract_declarator_opt : empty\n | ptr_operator abstract_declarator_opt\n | direct_abstract_declarator\n '''\n pass\n\ndef p_direct_abstract_declarator_opt(p):\n '''direct_abstract_declarator_opt : empty\n | direct_abstract_declarator\n '''\n pass\n\ndef p_direct_abstract_declarator(p):\n '''direct_abstract_declarator : direct_abstract_declarator_opt parenthesis_clause\n | direct_abstract_declarator_opt LBRACKET RBRACKET\n | direct_abstract_declarator_opt LBRACKET bexpression RBRACKET\n '''\n pass\n\ndef p_parenthesis_clause(p):\n '''parenthesis_clause : parameters_clause cv_qualifier_seq_opt\n | parameters_clause cv_qualifier_seq_opt exception_specification\n '''\n p[0] = ['(',')']\n\ndef p_parameters_clause(p):\n '''parameters_clause : '(' condition_opt ')'\n '''\n p[0] = ['(',')']\n\n#\n# A typed abstract qualifier such as\n# Class * ...\n# looks like a multiply, so pointers are parsed as their binary operation equivalents that\n# ultimately terminate with a degenerate right hand term.\n#\ndef p_abstract_pointer_declaration(p):\n '''abstract_pointer_declaration : ptr_operator_seq\n | multiplicative_expression star_ptr_operator ptr_operator_seq_opt\n '''\n pass\n\ndef p_abstract_parameter_declaration(p):\n '''abstract_parameter_declaration : abstract_pointer_declaration\n | and_expression '&'\n | and_expression '&' abstract_pointer_declaration\n '''\n pass\n\ndef p_special_parameter_declaration(p):\n '''special_parameter_declaration : abstract_parameter_declaration\n | abstract_parameter_declaration '=' assignment_expression\n | ELLIPSIS\n '''\n pass\n\ndef p_parameter_declaration(p):\n '''parameter_declaration : assignment_expression\n | special_parameter_declaration\n | decl_specifier_prefix parameter_declaration\n '''\n pass\n\n#\n# function_definition includes constructor, destructor, implicit int definitions too. A local destructor is successfully parsed as a function-declaration but the ~ was treated as a unary operator. constructor_head is the prefix ambiguity between a constructor and a member-init-list starting with a bit-field.\n#\ndef p_function_definition(p):\n '''function_definition : ctor_definition\n | func_definition\n '''\n pass\n\ndef p_func_definition(p):\n '''func_definition : assignment_expression function_try_block\n | assignment_expression function_body\n | decl_specifier_prefix func_definition\n '''\n global _parse_info\n if p[2] is not None and p[2][0] == '{':\n decl = flatten(p[1])\n #print \"HERE\",decl\n if decl[-1] == ')':\n decl=decl[-3]\n else:\n decl=decl[-1]\n p[0] = decl\n if decl != \"operator\":\n _parse_info.add_function(decl)\n else:\n p[0] = p[2]\n\ndef p_ctor_definition(p):\n '''ctor_definition : constructor_head function_try_block\n | constructor_head function_body\n | decl_specifier_prefix ctor_definition\n '''\n if p[2] is None or p[2][0] == \"try\" or p[2][0] == '{':\n p[0]=p[1]\n else:\n p[0]=p[1]\n\ndef p_constructor_head(p):\n '''constructor_head : bit_field_init_declaration\n | constructor_head ',' assignment_expression\n '''\n p[0]=p[1]\n\ndef p_function_try_block(p):\n '''function_try_block : TRY function_block handler_seq\n '''\n global noExceptionLogic\n noExceptionLogic=False\n p[0] = ['try']\n\ndef p_function_block(p):\n '''function_block : ctor_initializer_opt function_body\n '''\n pass\n\ndef p_function_body(p):\n '''function_body : LBRACE nonbrace_seq_opt RBRACE \n '''\n p[0] = ['{','}']\n\ndef p_initializer_clause(p):\n '''initializer_clause : assignment_expression\n | braced_initializer\n '''\n pass\n\ndef p_braced_initializer(p):\n '''braced_initializer : LBRACE initializer_list RBRACE\n | LBRACE initializer_list ',' RBRACE\n | LBRACE RBRACE\n '''\n pass\n\ndef p_initializer_list(p):\n '''initializer_list : initializer_clause\n | initializer_list ',' initializer_clause\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.8 Classes\n#---------------------------------------------------------------------------------------------------\n#\n# An anonymous bit-field declaration may look very like inheritance:\n# const int B = 3;\n# class A : B ;\n# The two usages are too distant to try to create and enforce a common prefix so we have to resort to\n# a parser hack by backtracking. Inheritance is much the most likely so we mark the input stream context\n# and try to parse a base-clause. If we successfully reach a { the base-clause is ok and inheritance was\n# the correct choice so we unmark and continue. If we fail to find the { an error token causes \n# back-tracking to the alternative parse in elaborated_type_specifier which regenerates the : and \n# declares unconditional success.\n#\n\ndef p_class_specifier_head(p):\n '''class_specifier_head : class_key scoped_id ':' base_specifier_list LBRACE\n | class_key ':' base_specifier_list LBRACE\n | class_key scoped_id LBRACE\n | class_key LBRACE\n '''\n global _parse_info\n base_classes=[]\n if len(p) == 6:\n scope = p[2]\n base_classes = p[4]\n elif len(p) == 4:\n scope = p[2]\n elif len(p) == 5:\n base_classes = p[3]\n else:\n scope = \"\"\n _parse_info.push_scope(scope,p[1],base_classes)\n \n\ndef p_class_key(p):\n '''class_key : CLASS \n | STRUCT \n | UNION\n '''\n p[0] = p[1]\n\ndef p_class_specifier(p):\n '''class_specifier : class_specifier_head member_specification_opt RBRACE\n '''\n scope = _parse_info.pop_scope()\n\ndef p_member_specification_opt(p):\n '''member_specification_opt : empty\n | member_specification_opt member_declaration\n '''\n pass\n\ndef p_member_declaration(p):\n '''member_declaration : accessibility_specifier\n | simple_member_declaration\n | function_definition\n | using_declaration\n | template_declaration\n '''\n p[0] = get_rest(p)\n #print \"Decl\",get_rest(p)\n\n#\n# The generality of constructor names (there need be no parenthesised argument list) means that that\n# name : f(g), h(i)\n# could be the start of a constructor or the start of an anonymous bit-field. An ambiguity is avoided by\n# parsing the ctor-initializer of a function_definition as a bit-field.\n#\ndef p_simple_member_declaration(p):\n '''simple_member_declaration : ';'\n | assignment_expression ';'\n | constructor_head ';'\n | member_init_declarations ';'\n | decl_specifier_prefix simple_member_declaration\n '''\n global _parse_info\n decl = flatten(get_rest(p))\n if len(decl) >= 4 and decl[-3] == \"(\":\n _parse_info.add_function(decl[-4])\n\ndef p_member_init_declarations(p):\n '''member_init_declarations : assignment_expression ',' member_init_declaration\n | constructor_head ',' bit_field_init_declaration\n | member_init_declarations ',' member_init_declaration\n '''\n pass\n\ndef p_member_init_declaration(p):\n '''member_init_declaration : assignment_expression\n | bit_field_init_declaration\n '''\n pass\n\ndef p_accessibility_specifier(p):\n '''accessibility_specifier : access_specifier ':'\n '''\n pass\n\ndef p_bit_field_declaration(p):\n '''bit_field_declaration : assignment_expression ':' bit_field_width\n | ':' bit_field_width\n '''\n if len(p) == 4:\n p[0]=p[1]\n\ndef p_bit_field_width(p):\n '''bit_field_width : logical_or_expression\n | logical_or_expression '?' bit_field_width ':' bit_field_width\n '''\n pass\n\ndef p_bit_field_init_declaration(p):\n '''bit_field_init_declaration : bit_field_declaration\n | bit_field_declaration '=' initializer_clause\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.9 Derived classes\n#---------------------------------------------------------------------------------------------------\ndef p_base_specifier_list(p):\n '''base_specifier_list : base_specifier\n | base_specifier_list ',' base_specifier\n '''\n if len(p) == 2:\n p[0] = [p[1]]\n else:\n p[0] = p[1]+[p[3]]\n\ndef p_base_specifier(p):\n '''base_specifier : scoped_id\n | access_specifier base_specifier\n | VIRTUAL base_specifier\n '''\n if len(p) == 2:\n p[0] = p[1]\n else:\n p[0] = p[2]\n\ndef p_access_specifier(p):\n '''access_specifier : PRIVATE \n | PROTECTED \n | PUBLIC\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.10 Special member functions\n#---------------------------------------------------------------------------------------------------\ndef p_conversion_function_id(p):\n '''conversion_function_id : OPERATOR conversion_type_id\n '''\n p[0] = ['operator']\n\ndef p_conversion_type_id(p):\n '''conversion_type_id : type_specifier ptr_operator_seq_opt\n | type_specifier conversion_type_id\n '''\n pass\n\n#\n# Ctor-initialisers can look like a bit field declaration, given the generalisation of names:\n# Class(Type) : m1(1), m2(2) { }\n# NonClass(bit_field) : int(2), second_variable, ...\n# The grammar below is used within a function_try_block or function_definition.\n# See simple_member_declaration for use in normal member function_definition.\n#\ndef p_ctor_initializer_opt(p):\n '''ctor_initializer_opt : empty\n | ctor_initializer\n '''\n pass\n\ndef p_ctor_initializer(p):\n '''ctor_initializer : ':' mem_initializer_list\n '''\n pass\n\ndef p_mem_initializer_list(p):\n '''mem_initializer_list : mem_initializer\n | mem_initializer_list_head mem_initializer\n '''\n pass\n\ndef p_mem_initializer_list_head(p):\n '''mem_initializer_list_head : mem_initializer_list ','\n '''\n pass\n\ndef p_mem_initializer(p):\n '''mem_initializer : mem_initializer_id '(' expression_list_opt ')'\n '''\n pass\n\ndef p_mem_initializer_id(p):\n '''mem_initializer_id : scoped_id\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.11 Overloading\n#---------------------------------------------------------------------------------------------------\n\ndef p_operator_function_id(p):\n '''operator_function_id : OPERATOR operator\n | OPERATOR '(' ')'\n | OPERATOR LBRACKET RBRACKET\n | OPERATOR '<'\n | OPERATOR '>'\n | OPERATOR operator '<' nonlgt_seq_opt '>'\n '''\n p[0] = [\"operator\"]\n\n#\n# It is not clear from the ANSI standard whether spaces are permitted in delete[]. If not then it can\n# be recognised and returned as DELETE_ARRAY by the lexer. Assuming spaces are permitted there is an\n# ambiguity created by the over generalised nature of expressions. operator new is a valid delarator-id\n# which we may have an undimensioned array of. Semantic rubbish, but syntactically valid. Since the\n# array form is covered by the declarator consideration we can exclude the operator here. The need\n# for a semantic rescue can be eliminated at the expense of a couple of shift-reduce conflicts by\n# removing the comments on the next four lines.\n#\ndef p_operator(p):\n '''operator : NEW\n | DELETE\n | '+'\n | '-'\n | '*'\n | '/'\n | '%'\n | '^'\n | '&'\n | '|'\n | '~'\n | '!'\n | '='\n | ASS_ADD\n | ASS_SUB\n | ASS_MUL\n | ASS_DIV\n | ASS_MOD\n | ASS_XOR\n | ASS_AND\n | ASS_OR\n | SHL\n | SHR\n | ASS_SHR\n | ASS_SHL\n | EQ\n | NE\n | LE\n | GE\n | LOG_AND\n | LOG_OR\n | INC\n | DEC\n | ','\n | ARROW_STAR\n | ARROW\n '''\n p[0]=p[1]\n\n# | IF\n# | SWITCH\n# | WHILE\n# | FOR\n# | DO\ndef p_reserved(p):\n '''reserved : PRIVATE\n | CLiteral\n | CppLiteral\n | IF\n | SWITCH\n | WHILE\n | FOR\n | DO\n | PROTECTED\n | PUBLIC\n | BOOL\n | CHAR\n | DOUBLE\n | FLOAT\n | INT\n | LONG\n | SHORT\n | SIGNED\n | UNSIGNED\n | VOID\n | WCHAR_T\n | CLASS\n | ENUM\n | NAMESPACE\n | STRUCT\n | TYPENAME\n | UNION\n | CONST\n | VOLATILE\n | AUTO\n | EXPLICIT\n | EXPORT\n | EXTERN\n | FRIEND\n | INLINE\n | MUTABLE\n | REGISTER\n | STATIC\n | TEMPLATE\n | TYPEDEF\n | USING\n | VIRTUAL\n | ASM\n | BREAK\n | CASE\n | CATCH\n | CONST_CAST\n | CONTINUE\n | DEFAULT\n | DYNAMIC_CAST\n | ELSE\n | FALSE\n | GOTO\n | OPERATOR\n | REINTERPRET_CAST\n | RETURN\n | SIZEOF\n | STATIC_CAST\n | THIS\n | THROW\n | TRUE\n | TRY\n | TYPEID\n | ATTRIBUTE\n | CDECL\n | TYPEOF\n | uTYPEOF\n '''\n if p[1] in ('try', 'catch', 'throw'):\n global noExceptionLogic\n noExceptionLogic=False\n\n#---------------------------------------------------------------------------------------------------\n# A.12 Templates\n#---------------------------------------------------------------------------------------------------\ndef p_template_declaration(p):\n '''template_declaration : template_parameter_clause declaration\n | EXPORT template_declaration\n '''\n pass\n\ndef p_template_parameter_clause(p):\n '''template_parameter_clause : TEMPLATE '<' nonlgt_seq_opt '>'\n '''\n pass\n\n#\n# Generalised naming makes identifier a valid declaration, so TEMPLATE identifier is too.\n# The TEMPLATE prefix is therefore folded into all names, parenthesis_clause and decl_specifier_prefix.\n#\n# explicit_instantiation: TEMPLATE declaration\n#\ndef p_explicit_specialization(p):\n '''explicit_specialization : TEMPLATE '<' '>' declaration\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.13 Exception Handling\n#---------------------------------------------------------------------------------------------------\ndef p_handler_seq(p):\n '''handler_seq : handler\n | handler handler_seq\n '''\n pass\n\ndef p_handler(p):\n '''handler : CATCH '(' exception_declaration ')' compound_statement\n '''\n global noExceptionLogic\n noExceptionLogic=False\n\ndef p_exception_declaration(p):\n '''exception_declaration : parameter_declaration\n '''\n pass\n\ndef p_throw_expression(p):\n '''throw_expression : THROW\n | THROW assignment_expression\n '''\n global noExceptionLogic\n noExceptionLogic=False\n\ndef p_exception_specification(p):\n '''exception_specification : THROW '(' ')'\n | THROW '(' type_id_list ')'\n '''\n global noExceptionLogic\n noExceptionLogic=False\n\ndef p_type_id_list(p):\n '''type_id_list : type_id\n | type_id_list ',' type_id\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# Misc productions\n#---------------------------------------------------------------------------------------------------\ndef p_nonsemicolon_seq(p):\n '''nonsemicolon_seq : empty\n | nonsemicolon_seq nonsemicolon\n '''\n pass\n\ndef p_nonsemicolon(p):\n '''nonsemicolon : misc\n | '('\n | ')'\n | '<'\n | '>'\n | LBRACKET nonbracket_seq_opt RBRACKET\n | LBRACE nonbrace_seq_opt RBRACE\n '''\n pass\n\ndef p_nonparen_seq_opt(p):\n '''nonparen_seq_opt : empty\n | nonparen_seq_opt nonparen\n '''\n pass\n\ndef p_nonparen_seq(p):\n '''nonparen_seq : nonparen\n | nonparen_seq nonparen\n '''\n pass\n\ndef p_nonparen(p):\n '''nonparen : misc\n | '<'\n | '>'\n | ';'\n | LBRACKET nonbracket_seq_opt RBRACKET\n | LBRACE nonbrace_seq_opt RBRACE\n '''\n pass\n\ndef p_nonbracket_seq_opt(p):\n '''nonbracket_seq_opt : empty\n | nonbracket_seq_opt nonbracket\n '''\n pass\n\ndef p_nonbracket_seq(p):\n '''nonbracket_seq : nonbracket\n | nonbracket_seq nonbracket\n '''\n pass\n\ndef p_nonbracket(p):\n '''nonbracket : misc\n | '<'\n | '>'\n | '('\n | ')'\n | ';'\n | LBRACKET nonbracket_seq_opt RBRACKET\n | LBRACE nonbrace_seq_opt RBRACE\n '''\n pass\n\ndef p_nonbrace_seq_opt(p):\n '''nonbrace_seq_opt : empty\n | nonbrace_seq_opt nonbrace\n '''\n pass\n\ndef p_nonbrace(p):\n '''nonbrace : misc\n | '<'\n | '>'\n | '('\n | ')'\n | ';'\n | LBRACKET nonbracket_seq_opt RBRACKET\n | LBRACE nonbrace_seq_opt RBRACE\n '''\n pass\n\ndef p_nonlgt_seq_opt(p):\n '''nonlgt_seq_opt : empty\n | nonlgt_seq_opt nonlgt\n '''\n pass\n\ndef p_nonlgt(p):\n '''nonlgt : misc\n | '('\n | ')'\n | LBRACKET nonbracket_seq_opt RBRACKET\n | '<' nonlgt_seq_opt '>'\n | ';'\n '''\n pass\n\ndef p_misc(p):\n '''misc : operator\n | identifier\n | IntegerLiteral\n | CharacterLiteral\n | FloatingLiteral\n | StringLiteral\n | reserved\n | '?'\n | ':'\n | '.'\n | SCOPE\n | ELLIPSIS\n | EXTENSION\n '''\n pass\n\ndef p_empty(p):\n '''empty : '''\n pass\n\n\n\n#\n# Compute column.\n# input is the input text string\n# token is a token instance\n#\ndef _find_column(input,token):\n ''' TODO '''\n i = token.lexpos\n while i > 0:\n if input[i] == '\\n': break\n i -= 1\n column = (token.lexpos - i)+1\n return column\n\ndef p_error(p):\n if p is None:\n tmp = \"Syntax error at end of file.\"\n else:\n tmp = \"Syntax error at token \"\n if p.type is \"\":\n tmp = tmp + \"''\"\n else:\n tmp = tmp + str(p.type)\n tmp = tmp + \" with value '\"+str(p.value)+\"'\"\n tmp = tmp + \" in line \" + str(lexer.lineno-1)\n tmp = tmp + \" at column \"+str(_find_column(_parsedata,p))\n raise IOError( tmp )\n\n\n\n#\n# The function that performs the parsing\n#\ndef parse_cpp(data=None, filename=None, debug=0, optimize=0, verbose=False, func_filter=None):\n if debug > 0:\n print \"Debugging parse_cpp!\"\n #\n # Always remove the parser.out file, which is generated to create debugging\n #\n if os.path.exists(\"parser.out\"):\n os.remove(\"parser.out\")\n #\n # Remove the parsetab.py* files. These apparently need to be removed\n # to ensure the creation of a parser.out file.\n #\n if os.path.exists(\"parsetab.py\"):\n os.remove(\"parsetab.py\")\n if os.path.exists(\"parsetab.pyc\"):\n os.remove(\"parsetab.pyc\")\n global debugging\n debugging=True\n #\n # Build lexer\n #\n global lexer\n lexer = lex.lex()\n #\n # Initialize parse object\n #\n global _parse_info\n _parse_info = CppInfo(filter=func_filter)\n _parse_info.verbose=verbose\n #\n # Build yaccer\n #\n write_table = not os.path.exists(\"parsetab.py\")\n yacc.yacc(debug=debug, optimize=optimize, write_tables=write_table)\n #\n # Parse the file\n #\n global _parsedata\n if not data is None:\n _parsedata=data\n ply_init(_parsedata)\n yacc.parse(data,debug=debug)\n elif not filename is None:\n f = open(filename)\n data = f.read()\n f.close()\n _parsedata=data\n ply_init(_parsedata)\n yacc.parse(data, debug=debug)\n else:\n return None\n #\n if not noExceptionLogic:\n _parse_info.noExceptionLogic = False\n else:\n for key in identifier_lineno:\n if 'ASSERT_THROWS' in key:\n _parse_info.noExceptionLogic = False\n break\n _parse_info.noExceptionLogic = True\n #\n return _parse_info\n\n\n\nimport sys\n\nif __name__ == '__main__':\n #\n # This MAIN routine parses a sequence of files provided at the command\n # line. If '-v' is included, then a verbose parsing output is \n # generated.\n #\n for arg in sys.argv[1:]:\n if arg == \"-v\":\n continue\n print \"Parsing file '\"+arg+\"'\"\n if '-v' in sys.argv:\n parse_cpp(filename=arg,debug=2,verbose=2)\n else:\n parse_cpp(filename=arg,verbose=2)\n #\n # Print the _parse_info object summary for this file.\n # This illustrates how class inheritance can be used to \n # deduce class members.\n # \n print str(_parse_info)\n\n" }, { "path": "cxxtest/cxxtest/cxxtest_fog.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n#\n# TODO: add line number info\n# TODO: add test function names\n#\n\nfrom __future__ import division\n\nimport sys\nimport re\n#from os.path import abspath, dirname\n#sys.path.insert(0, dirname(dirname(abspath(__file__))))\n#sys.path.insert(0, dirname(dirname(abspath(__file__)))+\"/cxx_parse\")\nfrom cxxtest_misc import abort\nimport cxx_parser\nimport re\n\ndef cstr( str ):\n '''Convert a string to its C representation'''\n return '\"' + re.sub('\\\\\\\\', '\\\\\\\\\\\\\\\\', str ) + '\"'\n\ndef scanInputFiles(files, _options):\n '''Scan all input files for test suites'''\n suites=[]\n for file in files:\n try:\n print \"Parsing file \"+file,\n sys.stdout.flush()\n parse_info = cxx_parser.parse_cpp(filename=file,optimize=1)\n except IOError, err:\n print \" error.\"\n print str(err)\n continue\n print \"done.\" \n sys.stdout.flush()\n #\n # WEH: see if it really makes sense to use parse information to\n # initialize this data. I don't think so...\n #\n _options.haveStandardLibrary=1\n if not parse_info.noExceptionLogic:\n _options.haveExceptionHandling=1\n #\n keys = list(parse_info.index.keys())\n tpat = re.compile(\"[Tt][Ee][Ss][Tt]\")\n for key in keys:\n if parse_info.index[key].scope_t == \"class\" and parse_info.is_baseclass(key,\"CxxTest::TestSuite\"):\n name=parse_info.index[key].name\n suite = { 'name' : name,\n 'file' : file,\n 'cfile' : cstr(file),\n 'line' : str(parse_info.index[key].lineno),\n 'generated' : 0,\n 'object' : 'suite_%s' % name,\n 'dobject' : 'suiteDescription_%s' % name,\n 'tlist' : 'Tests_%s' % name,\n 'tests' : [],\n 'lines' : [] }\n for fn in parse_info.get_functions(key,quiet=True):\n tname = fn[0]\n lineno = str(fn[1])\n if tname.startswith('createSuite'):\n # Indicate that we're using a dynamically generated test suite\n suite['create'] = str(lineno) # (unknown line)\n if tname.startswith('destroySuite'):\n # Indicate that we're using a dynamically generated test suite\n suite['destroy'] = str(lineno) # (unknown line)\n if not tpat.match(tname):\n # Skip non-test methods\n continue\n test = { 'name' : tname,\n 'suite' : suite,\n 'class' : 'TestDescription_suite_%s_%s' % (suite['name'], tname),\n 'object' : 'testDescription_suite_%s_%s' % (suite['name'], tname),\n 'line' : lineno,\n }\n suite['tests'].append(test)\n suites.append(suite)\n\n if not _options.root:\n ntests = 0\n for suite in suites:\n ntests += len(suite['tests'])\n if ntests == 0:\n abort( 'No tests defined' )\n #\n return [_options, suites]\n\n" }, { "path": "cxxtest/cxxtest/cxxtest_misc.py", "content": "#!/usr/bin/python\n#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\nimport sys\n\ndef abort( problem ):\n '''Print error message and exit'''\n sys.stderr.write( '\\n' )\n sys.stderr.write( problem )\n sys.stderr.write( '\\n\\n' )\n sys.exit(2)\n\n" }, { "path": "cxxtest/cxxtest/cxxtest_parser.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\nfrom __future__ import division\n\nimport codecs\nimport re\n#import sys\n#import getopt\n#import glob\nfrom cxxtest.cxxtest_misc import abort\n\n# Global variables\nsuites = []\nsuite = None\ninBlock = 0\noptions=None\n\ndef scanInputFiles(files, _options):\n '''Scan all input files for test suites'''\n global options\n options=_options\n for file in files:\n scanInputFile(file)\n global suites\n if len(suites) is 0 and not options.root:\n abort( 'No tests defined' )\n return [options,suites]\n\nlineCont_re = re.compile('(.*)\\\\\\s*$')\ndef scanInputFile(fileName):\n '''Scan single input file for test suites'''\n # mode 'rb' is problematic in python3 - byte arrays don't behave the same as\n # strings.\n # As far as the choice of the default encoding: utf-8 chews through\n # everything that the previous ascii codec could, plus most of new code.\n # TODO: figure out how to do this properly - like autodetect encoding from\n # file header.\n file = codecs.open(fileName, mode='r', encoding='utf-8')\n prev = \"\"\n lineNo = 0\n contNo = 0\n while 1:\n line = file.readline()\n if not line:\n break\n lineNo += 1\n\n m = lineCont_re.match(line)\n if m:\n prev += m.group(1) + \" \"\n contNo += 1\n else:\n scanInputLine( fileName, lineNo - contNo, prev + line )\n contNo = 0\n prev = \"\"\n if contNo:\n scanInputLine( fileName, lineNo - contNo, prev + line )\n \n closeSuite()\n file.close()\n\ndef scanInputLine( fileName, lineNo, line ):\n '''Scan single input line for interesting stuff'''\n scanLineForExceptionHandling( line )\n scanLineForStandardLibrary( line )\n\n scanLineForSuiteStart( fileName, lineNo, line )\n\n global suite\n if suite:\n scanLineInsideSuite( suite, lineNo, line )\n\ndef scanLineInsideSuite( suite, lineNo, line ):\n '''Analyze line which is part of a suite'''\n global inBlock\n if lineBelongsToSuite( suite, lineNo, line ):\n scanLineForTest( suite, lineNo, line )\n scanLineForCreate( suite, lineNo, line )\n scanLineForDestroy( suite, lineNo, line )\n\ndef lineBelongsToSuite( suite, lineNo, line ):\n '''Returns whether current line is part of the current suite.\n This can be false when we are in a generated suite outside of CXXTEST_CODE() blocks\n If the suite is generated, adds the line to the list of lines'''\n if not suite['generated']:\n return 1\n\n global inBlock\n if not inBlock:\n inBlock = lineStartsBlock( line )\n if inBlock:\n inBlock = addLineToBlock( suite, lineNo, line )\n return inBlock\n\n\nstd_re = re.compile( r\"\\b(std\\s*::|CXXTEST_STD|using\\s+namespace\\s+std\\b|^\\s*\\#\\s*include\\s+<[a-z0-9]+>)\" )\ndef scanLineForStandardLibrary( line ):\n '''Check if current line uses standard library'''\n global options\n if not options.haveStandardLibrary and std_re.search(line):\n if not options.noStandardLibrary:\n options.haveStandardLibrary = 1\n\nexception_re = re.compile( r\"\\b(throw|try|catch|TSM?_ASSERT_THROWS[A-Z_]*)\\b\" )\ndef scanLineForExceptionHandling( line ):\n '''Check if current line uses exception handling'''\n global options\n if not options.haveExceptionHandling and exception_re.search(line):\n if not options.noExceptionHandling:\n options.haveExceptionHandling = 1\n\nclassdef = '(?:::\\s*)?(?:\\w+\\s*::\\s*)*\\w+'\nbaseclassdef = '(?:public|private|protected)\\s+%s' % (classdef,)\ngeneral_suite = r\"\\bclass\\s+(%s)\\s*:(?:\\s*%s\\s*,)*\\s*public\\s+\" \\\n % (classdef, baseclassdef,)\ntestsuite = '(?:(?:::)?\\s*CxxTest\\s*::\\s*)?TestSuite'\nsuites_re = { re.compile( general_suite + testsuite ) : None }\ngeneratedSuite_re = re.compile( r'\\bCXXTEST_SUITE\\s*\\(\\s*(\\w*)\\s*\\)' )\ndef scanLineForSuiteStart( fileName, lineNo, line ):\n '''Check if current line starts a new test suite'''\n for i in list(suites_re.items()):\n m = i[0].search( line )\n if m:\n suite = startSuite( m.group(1), fileName, lineNo, 0 )\n if i[1] is not None:\n for test in i[1]['tests']:\n addTest(suite, test['name'], test['line'])\n break\n m = generatedSuite_re.search( line )\n if m:\n sys.stdout.write( \"%s:%s: Warning: Inline test suites are deprecated.\\n\" % (fileName, lineNo) )\n startSuite( m.group(1), fileName, lineNo, 1 )\n\ndef startSuite( name, file, line, generated ):\n '''Start scanning a new suite'''\n global suite\n closeSuite()\n object_name = name.replace(':',\"_\")\n suite = { 'name' : name,\n 'file' : file,\n 'cfile' : cstr(file),\n 'line' : line,\n 'generated' : generated,\n 'object' : 'suite_%s' % object_name,\n 'dobject' : 'suiteDescription_%s' % object_name,\n 'tlist' : 'Tests_%s' % object_name,\n 'tests' : [],\n 'lines' : [] }\n suites_re[re.compile( general_suite + name )] = suite\n return suite\n\ndef lineStartsBlock( line ):\n '''Check if current line starts a new CXXTEST_CODE() block'''\n return re.search( r'\\bCXXTEST_CODE\\s*\\(', line ) is not None\n\ntest_re = re.compile( r'^([^/]|/[^/])*\\bvoid\\s+([Tt]est\\w+)\\s*\\(\\s*(void)?\\s*\\)' )\ndef scanLineForTest( suite, lineNo, line ):\n '''Check if current line starts a test'''\n m = test_re.search( line )\n if m:\n addTest( suite, m.group(2), lineNo )\n\ndef addTest( suite, name, line ):\n '''Add a test function to the current suite'''\n test = { 'name' : name,\n 'suite' : suite,\n 'class' : 'TestDescription_%s_%s' % (suite['object'], name),\n 'object' : 'testDescription_%s_%s' % (suite['object'], name),\n 'line' : line,\n }\n suite['tests'].append( test )\n\ndef addLineToBlock( suite, lineNo, line ):\n '''Append the line to the current CXXTEST_CODE() block'''\n line = fixBlockLine( suite, lineNo, line )\n line = re.sub( r'^.*\\{\\{', '', line )\n \n e = re.search( r'\\}\\}', line )\n if e:\n line = line[:e.start()]\n suite['lines'].append( line )\n return e is None\n\ndef fixBlockLine( suite, lineNo, line):\n '''Change all [E]TS_ macros used in a line to _[E]TS_ macros with the correct file/line'''\n return re.sub( r'\\b(E?TSM?_(ASSERT[A-Z_]*|FAIL))\\s*\\(',\n r'_\\1(%s,%s,' % (suite['cfile'], lineNo),\n line, 0 )\n\ncreate_re = re.compile( r'\\bstatic\\s+\\w+\\s*\\*\\s*createSuite\\s*\\(\\s*(void)?\\s*\\)' )\ndef scanLineForCreate( suite, lineNo, line ):\n '''Check if current line defines a createSuite() function'''\n if create_re.search( line ):\n addSuiteCreateDestroy( suite, 'create', lineNo )\n\ndestroy_re = re.compile( r'\\bstatic\\s+void\\s+destroySuite\\s*\\(\\s*\\w+\\s*\\*\\s*\\w*\\s*\\)' )\ndef scanLineForDestroy( suite, lineNo, line ):\n '''Check if current line defines a destroySuite() function'''\n if destroy_re.search( line ):\n addSuiteCreateDestroy( suite, 'destroy', lineNo )\n\ndef cstr( s ):\n '''Convert a string to its C representation'''\n return '\"' + s.replace( '\\\\', '\\\\\\\\' ) + '\"'\n\n\ndef addSuiteCreateDestroy( suite, which, line ):\n '''Add createSuite()/destroySuite() to current suite'''\n if which in suite:\n abort( '%s:%s: %sSuite() already declared' % ( suite['file'], str(line), which ) )\n suite[which] = line\n\ndef closeSuite():\n '''Close current suite and add it to the list if valid'''\n global suite\n if suite is not None:\n if len(suite['tests']) is not 0:\n verifySuite(suite)\n rememberSuite(suite)\n suite = None\n\ndef verifySuite(suite):\n '''Verify current suite is legal'''\n if 'create' in suite and 'destroy' not in suite:\n abort( '%s:%s: Suite %s has createSuite() but no destroySuite()' %\n (suite['file'], suite['create'], suite['name']) )\n elif 'destroy' in suite and 'create' not in suite:\n abort( '%s:%s: Suite %s has destroySuite() but no createSuite()' %\n (suite['file'], suite['destroy'], suite['name']) )\n\ndef rememberSuite(suite):\n '''Add current suite to list'''\n global suites\n suites.append( suite )\n\n" }, { "path": "cxxtest/cxxtest/cxxtestgen.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n# vim: fileencoding=utf-8\n\nfrom __future__ import division\n# the above import important for forward-compatibility with python3,\n# which is already the default in archlinux!\n\n__all__ = ['main']\n\nimport __release__\nimport os\nimport sys\nimport re\nimport glob\nfrom optparse import OptionParser\nimport cxxtest_parser\n\ntry:\n import cxxtest_fog\n imported_fog=True\nexcept ImportError:\n imported_fog=False\n\nfrom cxxtest_misc import abort\n\noptions = []\nsuites = []\n\nwrotePreamble = 0\nwroteWorld = 0\nlastIncluded = ''\n\ndef main(args=sys.argv):\n '''The main program'''\n #\n # Reset global state\n #\n global wrotePreamble\n wrotePreamble=0\n global wroteWorld\n wroteWorld=0\n global lastIncluded\n lastIncluded = ''\n\n global suites\n global options\n files = parseCommandline(args)\n if imported_fog and options.fog:\n [options,suites] = cxxtest_fog.scanInputFiles( files, options )\n else:\n [options,suites] = cxxtest_parser.scanInputFiles( files, options )\n writeOutput()\n\ndef parseCommandline(args):\n '''Analyze command line arguments'''\n global imported_fog\n global options\n parser = OptionParser(\"%prog [options] [<filename> ...]\")\n parser.add_option(\"--version\",\n action=\"store_true\", dest=\"version\", default=False,\n help=\"Write the CxxTest version.\")\n parser.add_option(\"-o\", \"--output\",\n dest=\"outputFileName\", default=None, metavar=\"NAME\",\n help=\"Write output to file NAME.\")\n parser.add_option(\"-w\",\"--world\", dest=\"world\", default=\"cxxtest\",\n help=\"The label of the tests, used to name the XML results.\")\n parser.add_option(\"\", \"--include\", action=\"append\",\n dest=\"headers\", default=[], metavar=\"HEADER\",\n help=\"Include file HEADER in the test runner before other headers.\")\n parser.add_option(\"\", \"--abort-on-fail\",\n action=\"store_true\", dest=\"abortOnFail\", default=False,\n help=\"Abort tests on failed asserts (like xUnit).\")\n parser.add_option(\"\", \"--main\",\n action=\"store\", dest=\"main\", default=\"main\",\n help=\"Specify an alternative name for the main() function.\")\n parser.add_option(\"\", \"--headers\",\n action=\"store\", dest=\"header_filename\", default=None,\n help=\"Specify a filename that contains a list of header files that are processed to generate a test runner.\")\n parser.add_option(\"\", \"--runner\",\n dest=\"runner\", default=\"\", metavar=\"CLASS\",\n help=\"Create a test runner that processes test events using the class CxxTest::CLASS.\")\n parser.add_option(\"\", \"--gui\",\n dest=\"gui\", metavar=\"CLASS\",\n help=\"Create a GUI test runner that processes test events using the class CxxTest::CLASS. (deprecated)\")\n parser.add_option(\"\", \"--error-printer\",\n action=\"store_true\", dest=\"error_printer\", default=False,\n help=\"Create a test runner using the ErrorPrinter class, and allow the use of the standard library.\")\n parser.add_option(\"\", \"--xunit-printer\",\n action=\"store_true\", dest=\"xunit_printer\", default=False,\n help=\"Create a test runner using the XUnitPrinter class.\")\n parser.add_option(\"\", \"--xunit-file\", dest=\"xunit_file\", default=\"\",\n help=\"The file to which the XML summary is written for test runners using the XUnitPrinter class. The default XML filename is TEST-<world>.xml, where <world> is the value of the --world option. (default: cxxtest)\")\n parser.add_option(\"\", \"--have-std\",\n action=\"store_true\", dest=\"haveStandardLibrary\", default=False,\n help=\"Use the standard library (even if not found in tests).\")\n parser.add_option(\"\", \"--no-std\",\n action=\"store_true\", dest=\"noStandardLibrary\", default=False,\n help=\"Do not use standard library (even if found in tests).\")\n parser.add_option(\"\", \"--have-eh\",\n action=\"store_true\", dest=\"haveExceptionHandling\", default=False,\n help=\"Use exception handling (even if not found in tests).\")\n parser.add_option(\"\", \"--no-eh\",\n action=\"store_true\", dest=\"noExceptionHandling\", default=False,\n help=\"Do not use exception handling (even if found in tests).\")\n parser.add_option(\"\", \"--longlong\",\n dest=\"longlong\", default=None, metavar=\"TYPE\",\n help=\"Use TYPE as for long long integers. (default: not supported)\")\n parser.add_option(\"\", \"--no-static-init\",\n action=\"store_true\", dest=\"noStaticInit\", default=False,\n help=\"Do not rely on static initialization in the test runner.\")\n parser.add_option(\"\", \"--template\",\n dest=\"templateFileName\", default=None, metavar=\"TEMPLATE\",\n help=\"Generate the test runner using file TEMPLATE to define a template.\")\n parser.add_option(\"\", \"--root\",\n action=\"store_true\", dest=\"root\", default=False,\n help=\"Write the main() function and global data for a test runner.\")\n parser.add_option(\"\", \"--part\",\n action=\"store_true\", dest=\"part\", default=False,\n help=\"Write the tester classes for a test runner.\")\n #parser.add_option(\"\", \"--factor\",\n #action=\"store_true\", dest=\"factor\", default=False,\n #help=\"Declare the _CXXTEST_FACTOR macro. (deprecated)\")\n if imported_fog:\n fog_help = \"Use new FOG C++ parser\"\n else:\n fog_help = \"Use new FOG C++ parser (disabled)\"\n parser.add_option(\"-f\", \"--fog-parser\",\n action=\"store_true\",\n dest=\"fog\",\n default=False,\n help=fog_help\n )\n\n (options, args) = parser.parse_args(args=args)\n if not options.header_filename is None:\n if not os.path.exists(options.header_filename):\n abort( \"ERROR: the file '%s' does not exist!\" % options.header_filename )\n INPUT = open(options.header_filename)\n headers = [line.strip() for line in INPUT]\n args.extend( headers )\n INPUT.close()\n\n if options.fog and not imported_fog:\n abort( \"Cannot use the FOG parser. Check that the 'ply' package is installed. The 'ordereddict' package is also required if running Python 2.6\")\n\n if options.version:\n printVersion()\n\n # the cxxtest builder relies on this behaviour! don't remove\n if options.runner == 'none':\n options.runner = None\n\n if options.xunit_printer or options.runner == \"XUnitPrinter\":\n options.xunit_printer=True\n options.runner=\"XUnitPrinter\"\n if len(args) > 1:\n if options.xunit_file == \"\":\n if options.world == \"\":\n options.world = \"cxxtest\"\n options.xunit_file=\"TEST-\"+options.world+\".xml\"\n elif options.xunit_file == \"\":\n if options.world == \"\":\n options.world = \"cxxtest\"\n options.xunit_file=\"TEST-\"+options.world+\".xml\"\n\n if options.error_printer:\n options.runner= \"ErrorPrinter\"\n options.haveStandardLibrary = True\n \n if options.noStaticInit and (options.root or options.part):\n abort( '--no-static-init cannot be used with --root/--part' )\n\n if options.gui and not options.runner:\n options.runner = 'StdioPrinter'\n\n files = setFiles(args[1:])\n if len(files) == 0 and not options.root:\n sys.stderr.write(parser.error(\"No input files found\"))\n\n return files\n\n\ndef printVersion():\n '''Print CxxTest version and exit'''\n sys.stdout.write( \"This is CxxTest version %s.\\n\" % __release__.__version__ )\n sys.exit(0)\n\ndef setFiles(patterns ):\n '''Set input files specified on command line'''\n files = expandWildcards( patterns )\n return files\n\ndef expandWildcards( patterns ):\n '''Expand all wildcards in an array (glob)'''\n fileNames = []\n for pathName in patterns:\n patternFiles = glob.glob( pathName )\n for fileName in patternFiles:\n fileNames.append( fixBackslashes( fileName ) )\n return fileNames\n\ndef fixBackslashes( fileName ):\n '''Convert backslashes to slashes in file name'''\n return re.sub( r'\\\\', '/', fileName, 0 )\n\n\ndef writeOutput():\n '''Create output file'''\n if options.templateFileName:\n writeTemplateOutput()\n else:\n writeSimpleOutput()\n\ndef writeSimpleOutput():\n '''Create output not based on template'''\n output = startOutputFile()\n writePreamble( output )\n if options.root or not options.part:\n writeMain( output )\n\n if len(suites) > 0:\n output.write(\"bool \"+suites[0]['object']+\"_init = false;\\n\")\n\n writeWorld( output )\n output.close()\n\ninclude_re = re.compile( r\"\\s*\\#\\s*include\\s+<cxxtest/\" )\npreamble_re = re.compile( r\"^\\s*<CxxTest\\s+preamble>\\s*$\" )\nworld_re = re.compile( r\"^\\s*<CxxTest\\s+world>\\s*$\" )\ndef writeTemplateOutput():\n '''Create output based on template file'''\n template = open(options.templateFileName)\n output = startOutputFile()\n while 1:\n line = template.readline()\n if not line:\n break;\n if include_re.search( line ):\n writePreamble( output )\n output.write( line )\n elif preamble_re.search( line ):\n writePreamble( output )\n elif world_re.search( line ):\n if len(suites) > 0:\n output.write(\"bool \"+suites[0]['object']+\"_init = false;\\n\")\n writeWorld( output )\n else:\n output.write( line )\n template.close()\n output.close()\n\ndef startOutputFile():\n '''Create output file and write header'''\n if options.outputFileName is not None:\n output = open( options.outputFileName, 'w' )\n else:\n output = sys.stdout\n output.write( \"/* Generated file, do not edit */\\n\\n\" )\n return output\n\ndef writePreamble( output ):\n '''Write the CxxTest header (#includes and #defines)'''\n global wrotePreamble\n if wrotePreamble: return\n output.write( \"#ifndef CXXTEST_RUNNING\\n\" )\n output.write( \"#define CXXTEST_RUNNING\\n\" )\n output.write( \"#endif\\n\" )\n output.write( \"\\n\" )\n if options.xunit_printer:\n output.write( \"#include <fstream>\\n\" )\n if options.haveStandardLibrary:\n output.write( \"#define _CXXTEST_HAVE_STD\\n\" )\n if options.haveExceptionHandling:\n output.write( \"#define _CXXTEST_HAVE_EH\\n\" )\n if options.abortOnFail:\n output.write( \"#define _CXXTEST_ABORT_TEST_ON_FAIL\\n\" )\n if options.longlong:\n output.write( \"#define _CXXTEST_LONGLONG %s\\n\" % options.longlong )\n #if options.factor:\n #output.write( \"#define _CXXTEST_FACTOR\\n\" )\n for header in options.headers:\n output.write( \"#include \\\"%s\\\"\\n\" % header )\n output.write( \"#include <cxxtest/TestListener.h>\\n\" )\n output.write( \"#include <cxxtest/TestTracker.h>\\n\" )\n output.write( \"#include <cxxtest/TestRunner.h>\\n\" )\n output.write( \"#include <cxxtest/RealDescriptions.h>\\n\" )\n output.write( \"#include <cxxtest/TestMain.h>\\n\" )\n if options.runner:\n output.write( \"#include <cxxtest/%s.h>\\n\" % options.runner )\n if options.gui:\n output.write( \"#include <cxxtest/%s.h>\\n\" % options.gui )\n output.write( \"\\n\" )\n wrotePreamble = 1\n\ndef writeMain( output ):\n '''Write the main() function for the test runner'''\n if not (options.gui or options.runner):\n return\n output.write( 'int %s( int argc, char *argv[] ) {\\n' % options.main )\n output.write( ' int status;\\n' )\n if options.noStaticInit:\n output.write( ' CxxTest::initialize();\\n' )\n if options.gui:\n tester_t = \"CxxTest::GuiTuiRunner<CxxTest::%s, CxxTest::%s> \" % (options.gui, options.runner)\n else:\n tester_t = \"CxxTest::%s\" % (options.runner)\n if options.xunit_printer:\n output.write( ' std::ofstream ofstr(\"%s\");\\n' % options.xunit_file )\n output.write( ' %s tmp(ofstr);\\n' % tester_t )\n output.write( ' CxxTest::RealWorldDescription::_worldName = \"%s\";\\n' % options.world )\n else:\n output.write( ' %s tmp;\\n' % tester_t )\n output.write( ' status = CxxTest::Main<%s>( tmp, argc, argv );\\n' % tester_t )\n output.write( ' return status;\\n')\n output.write( '}\\n' )\n\n\ndef writeWorld( output ):\n '''Write the world definitions'''\n global wroteWorld\n if wroteWorld: return\n writePreamble( output )\n writeSuites( output )\n if options.root or not options.part:\n writeRoot( output )\n writeWorldDescr( output )\n if options.noStaticInit:\n writeInitialize( output )\n wroteWorld = 1\n\ndef writeSuites(output):\n '''Write all TestDescriptions and SuiteDescriptions'''\n for suite in suites:\n writeInclude( output, suite['file'] )\n if isGenerated(suite):\n generateSuite( output, suite )\n if isDynamic(suite):\n writeSuitePointer( output, suite )\n else:\n writeSuiteObject( output, suite )\n writeTestList( output, suite )\n writeSuiteDescription( output, suite )\n writeTestDescriptions( output, suite )\n\ndef isGenerated(suite):\n '''Checks whether a suite class should be created'''\n return suite['generated']\n\ndef isDynamic(suite):\n '''Checks whether a suite is dynamic'''\n return 'create' in suite\n\ndef writeInclude(output, file):\n '''Add #include \"file\" statement'''\n global lastIncluded\n if file == lastIncluded: return\n output.writelines( [ '#include \"', file, '\"\\n\\n' ] )\n lastIncluded = file\n\ndef generateSuite( output, suite ):\n '''Write a suite declared with CXXTEST_SUITE()'''\n output.write( 'class %s : public CxxTest::TestSuite {\\n' % suite['name'] )\n output.write( 'public:\\n' )\n for line in suite['lines']:\n output.write(line)\n output.write( '};\\n\\n' )\n\ndef writeSuitePointer( output, suite ):\n '''Create static suite pointer object for dynamic suites'''\n if options.noStaticInit:\n output.write( 'static %s *%s;\\n\\n' % (suite['name'], suite['object']) )\n else:\n output.write( 'static %s *%s = 0;\\n\\n' % (suite['name'], suite['object']) )\n\ndef writeSuiteObject( output, suite ):\n '''Create static suite object for non-dynamic suites'''\n output.writelines( [ \"static \", suite['name'], \" \", suite['object'], \";\\n\\n\" ] )\n\ndef writeTestList( output, suite ):\n '''Write the head of the test linked list for a suite'''\n if options.noStaticInit:\n output.write( 'static CxxTest::List %s;\\n' % suite['tlist'] )\n else:\n output.write( 'static CxxTest::List %s = { 0, 0 };\\n' % suite['tlist'] )\n\ndef writeWorldDescr( output ):\n '''Write the static name of the world name'''\n if options.noStaticInit:\n output.write( 'const char* CxxTest::RealWorldDescription::_worldName;\\n' )\n else:\n output.write( 'const char* CxxTest::RealWorldDescription::_worldName = \"cxxtest\";\\n' )\n\ndef writeTestDescriptions( output, suite ):\n '''Write all test descriptions for a suite'''\n for test in suite['tests']:\n writeTestDescription( output, suite, test )\n\ndef writeTestDescription( output, suite, test ):\n '''Write test description object'''\n output.write( 'static class %s : public CxxTest::RealTestDescription {\\n' % test['class'] )\n output.write( 'public:\\n' )\n if not options.noStaticInit:\n output.write( ' %s() : CxxTest::RealTestDescription( %s, %s, %s, \"%s\" ) {}\\n' %\n (test['class'], suite['tlist'], suite['dobject'], test['line'], test['name']) )\n output.write( ' void runTest() { %s }\\n' % runBody( suite, test ) )\n output.write( '} %s;\\n\\n' % test['object'] )\n\ndef runBody( suite, test ):\n '''Body of TestDescription::run()'''\n if isDynamic(suite): return dynamicRun( suite, test )\n else: return staticRun( suite, test )\n\ndef dynamicRun( suite, test ):\n '''Body of TestDescription::run() for test in a dynamic suite'''\n return 'if ( ' + suite['object'] + ' ) ' + suite['object'] + '->' + test['name'] + '();'\n \ndef staticRun( suite, test ):\n '''Body of TestDescription::run() for test in a non-dynamic suite'''\n return suite['object'] + '.' + test['name'] + '();'\n \ndef writeSuiteDescription( output, suite ):\n '''Write SuiteDescription object'''\n if isDynamic( suite ):\n writeDynamicDescription( output, suite )\n else:\n writeStaticDescription( output, suite )\n\ndef writeDynamicDescription( output, suite ):\n '''Write SuiteDescription for a dynamic suite'''\n output.write( 'CxxTest::DynamicSuiteDescription<%s> %s' % (suite['name'], suite['dobject']) )\n if not options.noStaticInit:\n output.write( '( %s, %s, \"%s\", %s, %s, %s, %s )' %\n (suite['cfile'], suite['line'], suite['name'], suite['tlist'],\n suite['object'], suite['create'], suite['destroy']) )\n output.write( ';\\n\\n' )\n\ndef writeStaticDescription( output, suite ):\n '''Write SuiteDescription for a static suite'''\n output.write( 'CxxTest::StaticSuiteDescription %s' % suite['dobject'] )\n if not options.noStaticInit:\n output.write( '( %s, %s, \"%s\", %s, %s )' %\n (suite['cfile'], suite['line'], suite['name'], suite['object'], suite['tlist']) )\n output.write( ';\\n\\n' )\n\ndef writeRoot(output):\n '''Write static members of CxxTest classes'''\n output.write( '#include <cxxtest/Root.cpp>\\n' )\n\ndef writeInitialize(output):\n '''Write CxxTest::initialize(), which replaces static initialization'''\n output.write( 'namespace CxxTest {\\n' )\n output.write( ' void initialize()\\n' )\n output.write( ' {\\n' )\n for suite in suites:\n output.write( ' %s.initialize();\\n' % suite['tlist'] )\n if isDynamic(suite):\n output.write( ' %s = 0;\\n' % suite['object'] )\n output.write( ' %s.initialize( %s, %s, \"%s\", %s, %s, %s, %s );\\n' %\n (suite['dobject'], suite['cfile'], suite['line'], suite['name'],\n suite['tlist'], suite['object'], suite['create'], suite['destroy']) )\n else:\n output.write( ' %s.initialize( %s, %s, \"%s\", %s, %s );\\n' %\n (suite['dobject'], suite['cfile'], suite['line'], suite['name'],\n suite['object'], suite['tlist']) )\n\n for test in suite['tests']:\n output.write( ' %s.initialize( %s, %s, %s, \"%s\" );\\n' %\n (test['object'], suite['tlist'], suite['dobject'], test['line'], test['name']) )\n\n output.write( ' }\\n' )\n output.write( '}\\n' )\n\n" }, { "path": "cxxtest/cxxtestgen", "content": "#! /usr/bin/env python\n#\n# The CxxTest driver script, which uses the cxxtest Python package.\n#\n\nimport sys\nimport os\nfrom os.path import realpath, dirname\nif sys.version_info < (3,0):\n sys.path.insert(0, dirname(dirname(realpath(__file__)))+os.sep+'python')\nelse:\n sys.path.insert(0, dirname(dirname(realpath(__file__)))+os.sep+'python'+os.sep+'python3')\nsys.path.append(\".\")\n\nimport cxxtest\n\ncxxtest.main(sys.argv)\n\n" }, { "path": "cxxtest/python/README.txt", "content": "CxxTest Python Package\n======================\n\nThe CxxTest Python package includes utilities that are used by the\nCxxTest unit testing framework. Specifically, this Python package\nsupports C++ parsing and code generation done in the cxxtestgen \nscript.\n\n" }, { "path": "cxxtest/python/convert.py", "content": "#\n# Execute this script to copy the cxxtest/*.py files\n# and run 2to3 to convert them to Python 3.\n#\n\nimport glob\nimport subprocess\nimport os\nimport shutil\n\nos.chdir('cxxtest')\nfor file in glob.glob('*.py'):\n shutil.copyfile(file, '../python3/cxxtest/'+file)\n#\nos.chdir('../python3/cxxtest')\n#\nfor file in glob.glob('*.py'):\n subprocess.call('2to3 -w '+file, shell=True)\n\n" }, { "path": "cxxtest/python/cxxtest/__init__.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n\"\"\"cxxtest: A Python package that supports the CxxTest test framework for C/C++.\n\n.. _CxxTest: http://cxxtest.tigris.org/\n\nCxxTest is a unit testing framework for C++ that is similar in\nspirit to JUnit, CppUnit, and xUnit. CxxTest is easy to use because\nit does not require precompiling a CxxTest testing library, it\nemploys no advanced features of C++ (e.g. RTTI) and it supports a\nvery flexible form of test discovery.\n\nThe cxxtest Python package includes capabilities for parsing C/C++ source files and generating\nCxxTest drivers.\n\"\"\"\n\nfrom cxxtest.__release__ import __version__, __date__\n__date__\n__version__\n\n__maintainer__ = \"William E. Hart\"\n__maintainer_email__ = \"whart222@gmail.com\"\n__license__ = \"LGPL\"\n__url__ = \"http://cxxtest.tigris.org/\"\n\nfrom cxxtest.cxxtestgen import *\n" }, { "path": "cxxtest/python/cxxtest/__release__.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n\"\"\" Release Information for cxxtest \"\"\"\n\n__version__ = '4.0.2'\n__date__ = \"2012-01-02\"\n" }, { "path": "cxxtest/python/cxxtest/cxx_parser.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n# vim: fileencoding=utf-8\n\n#\n# This is a PLY parser for the entire ANSI C++ grammar. This grammar was \n# adapted from the FOG grammar developed by E. D. Willink. See\n#\n# http://www.computing.surrey.ac.uk/research/dsrg/fog/\n#\n# for further details.\n#\n# The goal of this grammar is to extract information about class, function and\n# class method declarations, along with their associated scope. Thus, this \n# grammar can be used to analyze classes in an inheritance heirarchy, and then\n# enumerate the methods in a derived class.\n#\n# This grammar parses blocks of <>, (), [] and {} in a generic manner. Thus,\n# There are several capabilities that this grammar does not support:\n#\n# 1. Ambiguous template specification. This grammar cannot parse template\n# specifications that do not have paired <>'s in their declaration. In\n# particular, ambiguous declarations like\n#\n# foo<A, c<3 >();\n#\n# cannot be correctly parsed.\n#\n# 2. Template class specialization. Although the goal of this grammar is to\n# extract class information, specialization of templated classes is\n# not supported. When a template class definition is parsed, it's \n# declaration is archived without information about the template\n# parameters. Class specializations will be stored separately, and \n# thus they can be processed after the fact. However, this grammar\n# does not attempt to correctly process properties of class inheritence\n# when template class specialization is employed.\n#\n\n#\n# TODO: document usage of this file\n#\n\nfrom __future__ import division\n\nimport os\nimport ply.lex as lex\nimport ply.yacc as yacc\nimport re\ntry:\n from collections import OrderedDict\nexcept ImportError:\n from ordereddict import OrderedDict\n\nlexer = None\nscope_lineno = 0\nidentifier_lineno = {}\n_parse_info=None\n_parsedata=None\nnoExceptionLogic = True\n\ndef ply_init(data):\n global _parsedata\n _parsedata=data\n\n\nclass Scope(object):\n\n def __init__(self,name,abs_name,scope_t,base_classes,lineno):\n self.function=[]\n self.name=name\n self.scope_t=scope_t\n self.sub_scopes=[]\n self.base_classes=base_classes\n self.abs_name=abs_name\n self.lineno=lineno\n \n def insert(self,scope):\n self.sub_scopes.append(scope)\n\n\nclass CppInfo(object):\n\n def __init__(self, filter=None):\n self.verbose=0\n if filter is None:\n self.filter=re.compile(\"[Tt][Ee][Ss][Tt]|createSuite|destroySuite\")\n else:\n self.filter=filter\n self.scopes=[\"\"]\n self.index=OrderedDict()\n self.index[\"\"]=Scope(\"\",\"::\",\"namespace\",[],1)\n self.function=[]\n\n def push_scope(self,ns,scope_t,base_classes=[]):\n name = self.scopes[-1]+\"::\"+ns\n if self.verbose>=2:\n print \"-- Starting \"+scope_t+\" \"+name\n self.scopes.append(name)\n self.index[name] = Scope(ns,name,scope_t,base_classes,scope_lineno-1)\n\n def pop_scope(self):\n scope = self.scopes.pop()\n if self.verbose>=2:\n print \"-- Stopping \"+scope\n return scope\n\n def add_function(self, fn):\n fn = str(fn)\n if self.filter.search(fn):\n self.index[self.scopes[-1]].function.append((fn, identifier_lineno.get(fn,lexer.lineno-1)))\n tmp = self.scopes[-1]+\"::\"+fn\n if self.verbose==2:\n print \"-- Function declaration \"+fn+\" \"+tmp\n elif self.verbose==1:\n print \"-- Function declaration \"+tmp\n\n def get_functions(self,name,quiet=False):\n if name == \"::\":\n name = \"\"\n scope = self.index[name]\n fns=scope.function\n for key in scope.base_classes:\n cname = self.find_class(key,scope)\n if cname is None:\n if not quiet:\n print \"Defined classes: \",list(self.index.keys())\n print \"WARNING: Unknown class \"+key\n else:\n fns += self.get_functions(cname,quiet)\n return fns\n \n def find_class(self,name,scope):\n if ':' in name:\n if name in self.index:\n return name\n else:\n return None \n tmp = scope.abs_name.split(':')\n name1 = \":\".join(tmp[:-1] + [name])\n if name1 in self.index:\n return name1\n name2 = \"::\"+name\n if name2 in self.index:\n return name2\n return None\n\n def __repr__(self):\n return str(self)\n\n def is_baseclass(self,cls,base):\n '''Returns true if base is a base-class of cls'''\n if cls in self.index:\n bases = self.index[cls]\n elif \"::\"+cls in self.index:\n bases = self.index[\"::\"+cls]\n else:\n return False\n #raise IOError, \"Unknown class \"+cls\n if base in bases.base_classes:\n return True\n for name in bases.base_classes:\n if self.is_baseclass(name,base):\n return True\n return False\n\n def __str__(self):\n ans=\"\"\n keys = list(self.index.keys())\n keys.sort()\n for key in keys:\n scope = self.index[key]\n ans += scope.scope_t+\" \"+scope.abs_name+\"\\n\"\n if scope.scope_t == \"class\":\n ans += \" Base Classes: \"+str(scope.base_classes)+\"\\n\"\n for fn in self.get_functions(scope.abs_name):\n ans += \" \"+fn+\"\\n\"\n else:\n for fn in scope.function:\n ans += \" \"+fn+\"\\n\"\n return ans\n\n\ndef flatten(x):\n \"\"\"Flatten nested list\"\"\"\n try:\n strtypes = basestring\n except: # for python3 etc\n strtypes = (str, bytes)\n\n result = []\n for el in x:\n if hasattr(el, \"__iter__\") and not isinstance(el, strtypes):\n result.extend(flatten(el))\n else:\n result.append(el)\n return result\n\n#\n# The lexer (and/or a preprocessor) is expected to identify the following\n#\n# Punctuation:\n#\n#\nliterals = \"+-*/%^&|~!<>=:()?.\\'\\\"\\\\@$;,\"\n\n#\nreserved = {\n 'private' : 'PRIVATE',\n 'protected' : 'PROTECTED',\n 'public' : 'PUBLIC',\n\n 'bool' : 'BOOL',\n 'char' : 'CHAR',\n 'double' : 'DOUBLE',\n 'float' : 'FLOAT',\n 'int' : 'INT',\n 'long' : 'LONG',\n 'short' : 'SHORT',\n 'signed' : 'SIGNED',\n 'unsigned' : 'UNSIGNED',\n 'void' : 'VOID',\n 'wchar_t' : 'WCHAR_T',\n\n 'class' : 'CLASS',\n 'enum' : 'ENUM',\n 'namespace' : 'NAMESPACE',\n 'struct' : 'STRUCT',\n 'typename' : 'TYPENAME',\n 'union' : 'UNION',\n\n 'const' : 'CONST',\n 'volatile' : 'VOLATILE',\n\n 'auto' : 'AUTO',\n 'explicit' : 'EXPLICIT',\n 'export' : 'EXPORT',\n 'extern' : 'EXTERN',\n '__extension__' : 'EXTENSION',\n 'friend' : 'FRIEND',\n 'inline' : 'INLINE',\n 'mutable' : 'MUTABLE',\n 'register' : 'REGISTER',\n 'static' : 'STATIC',\n 'template' : 'TEMPLATE',\n 'typedef' : 'TYPEDEF',\n 'using' : 'USING',\n 'virtual' : 'VIRTUAL',\n\n 'asm' : 'ASM',\n 'break' : 'BREAK',\n 'case' : 'CASE',\n 'catch' : 'CATCH',\n 'const_cast' : 'CONST_CAST',\n 'continue' : 'CONTINUE',\n 'default' : 'DEFAULT',\n 'delete' : 'DELETE',\n 'do' : 'DO',\n 'dynamic_cast' : 'DYNAMIC_CAST',\n 'else' : 'ELSE',\n 'false' : 'FALSE',\n 'for' : 'FOR',\n 'goto' : 'GOTO',\n 'if' : 'IF',\n 'new' : 'NEW',\n 'operator' : 'OPERATOR',\n 'reinterpret_cast' : 'REINTERPRET_CAST',\n 'return' : 'RETURN',\n 'sizeof' : 'SIZEOF',\n 'static_cast' : 'STATIC_CAST',\n 'switch' : 'SWITCH',\n 'this' : 'THIS',\n 'throw' : 'THROW',\n 'true' : 'TRUE',\n 'try' : 'TRY',\n 'typeid' : 'TYPEID',\n 'while' : 'WHILE',\n '\"C\"' : 'CLiteral',\n '\"C++\"' : 'CppLiteral',\n\n '__attribute__' : 'ATTRIBUTE',\n '__cdecl__' : 'CDECL',\n '__typeof' : 'uTYPEOF',\n 'typeof' : 'TYPEOF', \n\n 'CXXTEST_STD' : 'CXXTEST_STD'\n}\n \ntokens = [\n \"CharacterLiteral\",\n \"FloatingLiteral\",\n \"Identifier\",\n \"IntegerLiteral\",\n \"StringLiteral\",\n \"RBRACE\",\n \"LBRACE\",\n \"RBRACKET\",\n \"LBRACKET\",\n \"ARROW\",\n \"ARROW_STAR\",\n \"DEC\",\n \"EQ\",\n \"GE\",\n \"INC\",\n \"LE\",\n \"LOG_AND\",\n \"LOG_OR\",\n \"NE\",\n \"SHL\",\n \"SHR\",\n \"ASS_ADD\",\n \"ASS_AND\",\n \"ASS_DIV\",\n \"ASS_MOD\",\n \"ASS_MUL\",\n \"ASS_OR\",\n \"ASS_SHL\",\n \"ASS_SHR\",\n \"ASS_SUB\",\n \"ASS_XOR\",\n \"DOT_STAR\",\n \"ELLIPSIS\",\n \"SCOPE\",\n] + list(reserved.values())\n\nt_ignore = \" \\t\\r\"\n\nt_LBRACE = r\"(\\{)|(<%)\"\nt_RBRACE = r\"(\\})|(%>)\"\nt_LBRACKET = r\"(\\[)|(<:)\"\nt_RBRACKET = r\"(\\])|(:>)\"\nt_ARROW = r\"->\"\nt_ARROW_STAR = r\"->\\*\"\nt_DEC = r\"--\"\nt_EQ = r\"==\"\nt_GE = r\">=\"\nt_INC = r\"\\+\\+\"\nt_LE = r\"<=\"\nt_LOG_AND = r\"&&\"\nt_LOG_OR = r\"\\|\\|\"\nt_NE = r\"!=\"\nt_SHL = r\"<<\"\nt_SHR = r\">>\"\nt_ASS_ADD = r\"\\+=\"\nt_ASS_AND = r\"&=\"\nt_ASS_DIV = r\"/=\"\nt_ASS_MOD = r\"%=\"\nt_ASS_MUL = r\"\\*=\"\nt_ASS_OR = r\"\\|=\"\nt_ASS_SHL = r\"<<=\"\nt_ASS_SHR = r\">>=\"\nt_ASS_SUB = r\"-=\"\nt_ASS_XOR = r\"^=\"\nt_DOT_STAR = r\"\\.\\*\"\nt_ELLIPSIS = r\"\\.\\.\\.\"\nt_SCOPE = r\"::\"\n\n# Discard comments\ndef t_COMMENT(t):\n r'(/\\*(.|\\n)*?\\*/)|(//.*?\\n)|(\\#.*?\\n)'\n t.lexer.lineno += t.value.count(\"\\n\")\n\nt_IntegerLiteral = r'(0x[0-9A-F]+)|([0-9]+(L){0,1})'\nt_FloatingLiteral = r\"[0-9]+[eE\\.\\+-]+[eE\\.\\+\\-0-9]+\"\nt_CharacterLiteral = r'\\'([^\\'\\\\]|\\\\.)*\\''\n#t_StringLiteral = r'\"([^\"\\\\]|\\\\.)*\"'\ndef t_StringLiteral(t):\n r'\"([^\"\\\\]|\\\\.)*\"'\n t.type = reserved.get(t.value,'StringLiteral')\n return t\n\ndef t_Identifier(t):\n r\"[a-zA-Z_][a-zA-Z_0-9\\.]*\"\n t.type = reserved.get(t.value,'Identifier')\n return t\n\n\ndef t_error(t):\n print \"Illegal character '%s'\" % t.value[0]\n #raise IOError, \"Parse error\"\n #t.lexer.skip()\n\ndef t_newline(t):\n r'[\\n]+'\n t.lexer.lineno += len(t.value)\n\nprecedence = (\n ( 'right', 'SHIFT_THERE', 'REDUCE_HERE_MOSTLY', 'SCOPE'),\n ( 'nonassoc', 'ELSE', 'INC', 'DEC', '+', '-', '*', '&', 'LBRACKET', 'LBRACE', '<', ':', ')')\n )\n\nstart = 'translation_unit'\n\n#\n# The %prec resolves the 14.2-3 ambiguity:\n# Identifier '<' is forced to go through the is-it-a-template-name test\n# All names absorb TEMPLATE with the name, so that no template_test is \n# performed for them. This requires all potential declarations within an \n# expression to perpetuate this policy and thereby guarantee the ultimate \n# coverage of explicit_instantiation.\n#\n# The %prec also resolves a conflict in identifier : which is forced to be a \n# shift of a label for a labeled-statement rather than a reduction for the \n# name of a bit-field or generalised constructor. This is pretty dubious \n# syntactically but correct for all semantic possibilities. The shift is \n# only activated when the ambiguity exists at the start of a statement. \n# In this context a bit-field declaration or constructor definition are not \n# allowed.\n#\n\ndef p_identifier(p):\n '''identifier : Identifier\n | CXXTEST_STD '(' Identifier ')'\n '''\n if p[1][0] in ('t','T','c','d'):\n identifier_lineno[p[1]] = p.lineno(1)\n p[0] = p[1]\n\ndef p_id(p):\n '''id : identifier %prec SHIFT_THERE\n | template_decl\n | TEMPLATE id\n '''\n p[0] = get_rest(p)\n\ndef p_global_scope(p):\n '''global_scope : SCOPE\n '''\n p[0] = get_rest(p)\n\ndef p_id_scope(p):\n '''id_scope : id SCOPE'''\n p[0] = get_rest(p)\n\ndef p_id_scope_seq(p):\n '''id_scope_seq : id_scope\n | id_scope id_scope_seq\n '''\n p[0] = get_rest(p)\n\n#\n# A :: B :: C; is ambiguous How much is type and how much name ?\n# The %prec maximises the (type) length which is the 7.1-2 semantic constraint.\n#\ndef p_nested_id(p):\n '''nested_id : id %prec SHIFT_THERE\n | id_scope nested_id\n '''\n p[0] = get_rest(p)\n\ndef p_scoped_id(p):\n '''scoped_id : nested_id\n | global_scope nested_id\n | id_scope_seq\n | global_scope id_scope_seq\n '''\n global scope_lineno\n scope_lineno = lexer.lineno\n data = flatten(get_rest(p))\n if data[0] != None:\n p[0] = \"\".join(data)\n\n#\n# destructor_id has to be held back to avoid a conflict with a one's \n# complement as per 5.3.1-9, It gets put back only when scoped or in a \n# declarator_id, which is only used as an explicit member name.\n# Declarations of an unscoped destructor are always parsed as a one's \n# complement.\n#\ndef p_destructor_id(p):\n '''destructor_id : '~' id\n | TEMPLATE destructor_id\n '''\n p[0]=get_rest(p)\n\n#def p_template_id(p):\n# '''template_id : empty\n# | TEMPLATE\n# '''\n# pass\n\ndef p_template_decl(p):\n '''template_decl : identifier '<' nonlgt_seq_opt '>'\n '''\n #\n # WEH: should we include the lt/gt symbols to indicate that this is a\n # template class? How is that going to be used later???\n #\n #p[0] = [p[1] ,\"<\",\">\"]\n p[0] = p[1]\n\ndef p_special_function_id(p):\n '''special_function_id : conversion_function_id\n | operator_function_id\n | TEMPLATE special_function_id\n '''\n p[0]=get_rest(p)\n\ndef p_nested_special_function_id(p):\n '''nested_special_function_id : special_function_id\n | id_scope destructor_id\n | id_scope nested_special_function_id\n '''\n p[0]=get_rest(p)\n\ndef p_scoped_special_function_id(p):\n '''scoped_special_function_id : nested_special_function_id\n | global_scope nested_special_function_id\n '''\n p[0]=get_rest(p)\n\n# declarator-id is all names in all scopes, except reserved words\ndef p_declarator_id(p):\n '''declarator_id : scoped_id\n | scoped_special_function_id\n | destructor_id\n '''\n p[0]=p[1]\n\n#\n# The standard defines pseudo-destructors in terms of type-name, which is \n# class/enum/typedef, of which class-name is covered by a normal destructor. \n# pseudo-destructors are supposed to support ~int() in templates, so the \n# grammar here covers built-in names. Other names are covered by the lack \n# of identifier/type discrimination.\n#\ndef p_built_in_type_id(p):\n '''built_in_type_id : built_in_type_specifier\n | built_in_type_id built_in_type_specifier\n '''\n pass\n\ndef p_pseudo_destructor_id(p):\n '''pseudo_destructor_id : built_in_type_id SCOPE '~' built_in_type_id\n | '~' built_in_type_id\n | TEMPLATE pseudo_destructor_id\n '''\n pass\n\ndef p_nested_pseudo_destructor_id(p):\n '''nested_pseudo_destructor_id : pseudo_destructor_id\n | id_scope nested_pseudo_destructor_id\n '''\n pass\n\ndef p_scoped_pseudo_destructor_id(p):\n '''scoped_pseudo_destructor_id : nested_pseudo_destructor_id\n | global_scope scoped_pseudo_destructor_id\n '''\n pass\n\n#-------------------------------------------------------------------------------\n# A.2 Lexical conventions\n#-------------------------------------------------------------------------------\n#\n\ndef p_literal(p):\n '''literal : IntegerLiteral\n | CharacterLiteral\n | FloatingLiteral\n | StringLiteral\n | TRUE\n | FALSE\n '''\n pass\n\n#-------------------------------------------------------------------------------\n# A.3 Basic concepts\n#-------------------------------------------------------------------------------\ndef p_translation_unit(p):\n '''translation_unit : declaration_seq_opt\n '''\n pass\n\n#-------------------------------------------------------------------------------\n# A.4 Expressions\n#-------------------------------------------------------------------------------\n#\n# primary_expression covers an arbitrary sequence of all names with the \n# exception of an unscoped destructor, which is parsed as its unary expression \n# which is the correct disambiguation (when ambiguous). This eliminates the \n# traditional A(B) meaning A B ambiguity, since we never have to tack an A \n# onto the front of something that might start with (. The name length got \n# maximised ab initio. The downside is that semantic interpretation must split \n# the names up again.\n#\n# Unification of the declaration and expression syntax means that unary and \n# binary pointer declarator operators:\n# int * * name\n# are parsed as binary and unary arithmetic operators (int) * (*name). Since \n# type information is not used\n# ambiguities resulting from a cast\n# (cast)*(value)\n# are resolved to favour the binary rather than the cast unary to ease AST \n# clean-up. The cast-call ambiguity must be resolved to the cast to ensure \n# that (a)(b)c can be parsed.\n#\n# The problem of the functional cast ambiguity\n# name(arg)\n# as call or declaration is avoided by maximising the name within the parsing \n# kernel. So primary_id_expression picks up \n# extern long int const var = 5;\n# as an assignment to the syntax parsed as \"extern long int const var\". The \n# presence of two names is parsed so that \"extern long into const\" is \n# distinguished from \"var\" considerably simplifying subsequent \n# semantic resolution.\n#\n# The generalised name is a concatenation of potential type-names (scoped \n# identifiers or built-in sequences) plus optionally one of the special names \n# such as an operator-function-id, conversion-function-id or destructor as the \n# final name. \n#\n\ndef get_rest(p):\n return [p[i] for i in range(1, len(p))]\n\ndef p_primary_expression(p):\n '''primary_expression : literal\n | THIS\n | suffix_decl_specified_ids\n | abstract_expression %prec REDUCE_HERE_MOSTLY\n '''\n p[0] = get_rest(p)\n\n#\n# Abstract-expression covers the () and [] of abstract-declarators.\n#\ndef p_abstract_expression(p):\n '''abstract_expression : parenthesis_clause\n | LBRACKET bexpression_opt RBRACKET\n | TEMPLATE abstract_expression\n '''\n pass\n\ndef p_postfix_expression(p):\n '''postfix_expression : primary_expression\n | postfix_expression parenthesis_clause\n | postfix_expression LBRACKET bexpression_opt RBRACKET\n | postfix_expression LBRACKET bexpression_opt RBRACKET attributes\n | postfix_expression '.' declarator_id\n | postfix_expression '.' scoped_pseudo_destructor_id\n | postfix_expression ARROW declarator_id\n | postfix_expression ARROW scoped_pseudo_destructor_id \n | postfix_expression INC\n | postfix_expression DEC\n | DYNAMIC_CAST '<' nonlgt_seq_opt '>' '(' expression ')'\n | STATIC_CAST '<' nonlgt_seq_opt '>' '(' expression ')'\n | REINTERPRET_CAST '<' nonlgt_seq_opt '>' '(' expression ')'\n | CONST_CAST '<' nonlgt_seq_opt '>' '(' expression ')'\n | TYPEID parameters_clause\n '''\n #print \"HERE\",str(p[1])\n p[0] = get_rest(p)\n\ndef p_bexpression_opt(p):\n '''bexpression_opt : empty\n | bexpression\n '''\n pass\n\ndef p_bexpression(p):\n '''bexpression : nonbracket_seq\n | nonbracket_seq bexpression_seq bexpression_clause nonbracket_seq_opt\n | bexpression_seq bexpression_clause nonbracket_seq_opt\n '''\n pass\n\ndef p_bexpression_seq(p):\n '''bexpression_seq : empty\n | bexpression_seq bexpression_clause nonbracket_seq_opt\n '''\n pass\n\ndef p_bexpression_clause(p):\n '''bexpression_clause : LBRACKET bexpression_opt RBRACKET\n '''\n pass\n\n\n\ndef p_expression_list_opt(p):\n '''expression_list_opt : empty\n | expression_list\n '''\n pass\n\ndef p_expression_list(p):\n '''expression_list : assignment_expression\n | expression_list ',' assignment_expression\n '''\n pass\n\ndef p_unary_expression(p):\n '''unary_expression : postfix_expression\n | INC cast_expression\n | DEC cast_expression\n | ptr_operator cast_expression\n | suffix_decl_specified_scope star_ptr_operator cast_expression\n | '+' cast_expression\n | '-' cast_expression\n | '!' cast_expression\n | '~' cast_expression\n | SIZEOF unary_expression\n | new_expression\n | global_scope new_expression\n | delete_expression\n | global_scope delete_expression\n '''\n p[0] = get_rest(p)\n\ndef p_delete_expression(p):\n '''delete_expression : DELETE cast_expression\n '''\n pass\n\ndef p_new_expression(p):\n '''new_expression : NEW new_type_id new_initializer_opt\n | NEW parameters_clause new_type_id new_initializer_opt\n | NEW parameters_clause\n | NEW parameters_clause parameters_clause new_initializer_opt\n '''\n pass\n\ndef p_new_type_id(p):\n '''new_type_id : type_specifier ptr_operator_seq_opt\n | type_specifier new_declarator\n | type_specifier new_type_id\n '''\n pass\n\ndef p_new_declarator(p):\n '''new_declarator : ptr_operator new_declarator\n | direct_new_declarator\n '''\n pass\n\ndef p_direct_new_declarator(p):\n '''direct_new_declarator : LBRACKET bexpression_opt RBRACKET\n | direct_new_declarator LBRACKET bexpression RBRACKET\n '''\n pass\n\ndef p_new_initializer_opt(p):\n '''new_initializer_opt : empty\n | '(' expression_list_opt ')'\n '''\n pass\n\n#\n# cast-expression is generalised to support a [] as well as a () prefix. This covers the omission of \n# DELETE[] which when followed by a parenthesised expression was ambiguous. It also covers the gcc \n# indexed array initialisation for free.\n#\ndef p_cast_expression(p):\n '''cast_expression : unary_expression\n | abstract_expression cast_expression\n '''\n p[0] = get_rest(p)\n\ndef p_pm_expression(p):\n '''pm_expression : cast_expression\n | pm_expression DOT_STAR cast_expression\n | pm_expression ARROW_STAR cast_expression\n '''\n p[0] = get_rest(p)\n\ndef p_multiplicative_expression(p):\n '''multiplicative_expression : pm_expression\n | multiplicative_expression star_ptr_operator pm_expression\n | multiplicative_expression '/' pm_expression\n | multiplicative_expression '%' pm_expression\n '''\n p[0] = get_rest(p)\n\ndef p_additive_expression(p):\n '''additive_expression : multiplicative_expression\n | additive_expression '+' multiplicative_expression\n | additive_expression '-' multiplicative_expression\n '''\n p[0] = get_rest(p)\n\ndef p_shift_expression(p):\n '''shift_expression : additive_expression\n | shift_expression SHL additive_expression\n | shift_expression SHR additive_expression\n '''\n p[0] = get_rest(p)\n\n# | relational_expression '<' shift_expression\n# | relational_expression '>' shift_expression\n# | relational_expression LE shift_expression\n# | relational_expression GE shift_expression\ndef p_relational_expression(p):\n '''relational_expression : shift_expression\n '''\n p[0] = get_rest(p)\n\ndef p_equality_expression(p):\n '''equality_expression : relational_expression\n | equality_expression EQ relational_expression\n | equality_expression NE relational_expression\n '''\n p[0] = get_rest(p)\n\ndef p_and_expression(p):\n '''and_expression : equality_expression\n | and_expression '&' equality_expression\n '''\n p[0] = get_rest(p)\n\ndef p_exclusive_or_expression(p):\n '''exclusive_or_expression : and_expression\n | exclusive_or_expression '^' and_expression\n '''\n p[0] = get_rest(p)\n\ndef p_inclusive_or_expression(p):\n '''inclusive_or_expression : exclusive_or_expression\n | inclusive_or_expression '|' exclusive_or_expression\n '''\n p[0] = get_rest(p)\n\ndef p_logical_and_expression(p):\n '''logical_and_expression : inclusive_or_expression\n | logical_and_expression LOG_AND inclusive_or_expression\n '''\n p[0] = get_rest(p)\n\ndef p_logical_or_expression(p):\n '''logical_or_expression : logical_and_expression\n | logical_or_expression LOG_OR logical_and_expression\n '''\n p[0] = get_rest(p)\n\ndef p_conditional_expression(p):\n '''conditional_expression : logical_or_expression\n | logical_or_expression '?' expression ':' assignment_expression\n '''\n p[0] = get_rest(p)\n\n\n#\n# assignment-expression is generalised to cover the simple assignment of a braced initializer in order to \n# contribute to the coverage of parameter-declaration and init-declaration.\n#\n# | logical_or_expression assignment_operator assignment_expression\ndef p_assignment_expression(p):\n '''assignment_expression : conditional_expression\n | logical_or_expression assignment_operator nonsemicolon_seq\n | logical_or_expression '=' braced_initializer\n | throw_expression\n '''\n p[0]=get_rest(p)\n\ndef p_assignment_operator(p):\n '''assignment_operator : '=' \n | ASS_ADD\n | ASS_AND\n | ASS_DIV\n | ASS_MOD\n | ASS_MUL\n | ASS_OR\n | ASS_SHL\n | ASS_SHR\n | ASS_SUB\n | ASS_XOR\n '''\n pass\n\n#\n# expression is widely used and usually single-element, so the reductions are arranged so that a\n# single-element expression is returned as is. Multi-element expressions are parsed as a list that\n# may then behave polymorphically as an element or be compacted to an element.\n#\n\ndef p_expression(p):\n '''expression : assignment_expression\n | expression_list ',' assignment_expression\n '''\n p[0] = get_rest(p)\n\ndef p_constant_expression(p):\n '''constant_expression : conditional_expression\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.5 Statements\n#---------------------------------------------------------------------------------------------------\n# Parsing statements is easy once simple_declaration has been generalised to cover expression_statement.\n#\n#\n# The use of extern here is a hack. The 'extern \"C\" {}' block gets parsed\n# as a function, so when nested 'extern \"C\"' declarations exist, they don't\n# work because the block is viewed as a list of statements... :(\n#\ndef p_statement(p):\n '''statement : compound_statement\n | declaration_statement\n | try_block\n | labeled_statement\n | selection_statement\n | iteration_statement\n | jump_statement\n '''\n pass\n\ndef p_compound_statement(p):\n '''compound_statement : LBRACE statement_seq_opt RBRACE\n '''\n pass\n\ndef p_statement_seq_opt(p):\n '''statement_seq_opt : empty\n | statement_seq_opt statement\n '''\n pass\n\n#\n# The dangling else conflict is resolved to the innermost if.\n#\ndef p_selection_statement(p):\n '''selection_statement : IF '(' condition ')' statement %prec SHIFT_THERE\n | IF '(' condition ')' statement ELSE statement\n | SWITCH '(' condition ')' statement\n '''\n pass\n\ndef p_condition_opt(p):\n '''condition_opt : empty\n | condition\n '''\n pass\n\ndef p_condition(p):\n '''condition : nonparen_seq\n | nonparen_seq condition_seq parameters_clause nonparen_seq_opt\n | condition_seq parameters_clause nonparen_seq_opt\n '''\n pass\n\ndef p_condition_seq(p):\n '''condition_seq : empty\n | condition_seq parameters_clause nonparen_seq_opt\n '''\n pass\n\ndef p_labeled_statement(p):\n '''labeled_statement : identifier ':' statement\n | CASE constant_expression ':' statement\n | DEFAULT ':' statement\n '''\n pass\n\ndef p_try_block(p):\n '''try_block : TRY compound_statement handler_seq\n '''\n global noExceptionLogic\n noExceptionLogic=False\n\ndef p_jump_statement(p):\n '''jump_statement : BREAK ';'\n | CONTINUE ';'\n | RETURN nonsemicolon_seq ';'\n | GOTO identifier ';'\n '''\n pass\n\ndef p_iteration_statement(p):\n '''iteration_statement : WHILE '(' condition ')' statement\n | DO statement WHILE '(' expression ')' ';'\n | FOR '(' nonparen_seq_opt ')' statement\n '''\n pass\n\ndef p_declaration_statement(p):\n '''declaration_statement : block_declaration\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.6 Declarations\n#---------------------------------------------------------------------------------------------------\ndef p_compound_declaration(p):\n '''compound_declaration : LBRACE declaration_seq_opt RBRACE \n '''\n pass\n\ndef p_declaration_seq_opt(p):\n '''declaration_seq_opt : empty\n | declaration_seq_opt declaration\n '''\n pass\n\ndef p_declaration(p):\n '''declaration : block_declaration\n | function_definition\n | template_declaration\n | explicit_specialization\n | specialised_declaration\n '''\n pass\n\ndef p_specialised_declaration(p):\n '''specialised_declaration : linkage_specification\n | namespace_definition\n | TEMPLATE specialised_declaration\n '''\n pass\n\ndef p_block_declaration(p):\n '''block_declaration : simple_declaration\n | specialised_block_declaration\n '''\n pass\n\ndef p_specialised_block_declaration(p):\n '''specialised_block_declaration : asm_definition\n | namespace_alias_definition\n | using_declaration\n | using_directive\n | TEMPLATE specialised_block_declaration\n '''\n pass\n\ndef p_simple_declaration(p):\n '''simple_declaration : ';'\n | init_declaration ';'\n | init_declarations ';'\n | decl_specifier_prefix simple_declaration\n '''\n global _parse_info\n if len(p) == 3:\n if p[2] == \";\":\n decl = p[1]\n else:\n decl = p[2]\n if decl is not None:\n fp = flatten(decl)\n if len(fp) >= 2 and fp[0] is not None and fp[0]!=\"operator\" and fp[1] == '(':\n p[0] = fp[0]\n _parse_info.add_function(fp[0])\n\n#\n# A decl-specifier following a ptr_operator provokes a shift-reduce conflict for * const name which is resolved in favour of the pointer, and implemented by providing versions of decl-specifier guaranteed not to start with a cv_qualifier. decl-specifiers are implemented type-centrically. That is the semantic constraint that there must be a type is exploited to impose structure, but actually eliminate very little syntax. built-in types are multi-name and so need a different policy.\n#\n# non-type decl-specifiers are bound to the left-most type in a decl-specifier-seq, by parsing from the right and attaching suffixes to the right-hand type. Finally residual prefixes attach to the left. \n#\ndef p_suffix_built_in_decl_specifier_raw(p):\n '''suffix_built_in_decl_specifier_raw : built_in_type_specifier\n | suffix_built_in_decl_specifier_raw built_in_type_specifier\n | suffix_built_in_decl_specifier_raw decl_specifier_suffix\n '''\n pass\n\ndef p_suffix_built_in_decl_specifier(p):\n '''suffix_built_in_decl_specifier : suffix_built_in_decl_specifier_raw\n | TEMPLATE suffix_built_in_decl_specifier\n '''\n pass\n\n# | id_scope_seq\n# | SCOPE id_scope_seq\ndef p_suffix_named_decl_specifier(p):\n '''suffix_named_decl_specifier : scoped_id \n | elaborate_type_specifier \n | suffix_named_decl_specifier decl_specifier_suffix\n '''\n p[0]=get_rest(p)\n\ndef p_suffix_named_decl_specifier_bi(p):\n '''suffix_named_decl_specifier_bi : suffix_named_decl_specifier\n | suffix_named_decl_specifier suffix_built_in_decl_specifier_raw\n '''\n p[0] = get_rest(p)\n #print \"HERE\",get_rest(p)\n\ndef p_suffix_named_decl_specifiers(p):\n '''suffix_named_decl_specifiers : suffix_named_decl_specifier_bi\n | suffix_named_decl_specifiers suffix_named_decl_specifier_bi\n '''\n p[0] = get_rest(p)\n\ndef p_suffix_named_decl_specifiers_sf(p):\n '''suffix_named_decl_specifiers_sf : scoped_special_function_id\n | suffix_named_decl_specifiers\n | suffix_named_decl_specifiers scoped_special_function_id\n '''\n #print \"HERE\",get_rest(p)\n p[0] = get_rest(p)\n\ndef p_suffix_decl_specified_ids(p):\n '''suffix_decl_specified_ids : suffix_built_in_decl_specifier\n | suffix_built_in_decl_specifier suffix_named_decl_specifiers_sf\n | suffix_named_decl_specifiers_sf\n '''\n if len(p) == 3:\n p[0] = p[2]\n else:\n p[0] = p[1]\n\ndef p_suffix_decl_specified_scope(p):\n '''suffix_decl_specified_scope : suffix_named_decl_specifiers SCOPE\n | suffix_built_in_decl_specifier suffix_named_decl_specifiers SCOPE\n | suffix_built_in_decl_specifier SCOPE\n '''\n p[0] = get_rest(p)\n\ndef p_decl_specifier_affix(p):\n '''decl_specifier_affix : storage_class_specifier\n | function_specifier\n | FRIEND\n | TYPEDEF\n | cv_qualifier\n '''\n pass\n\ndef p_decl_specifier_suffix(p):\n '''decl_specifier_suffix : decl_specifier_affix\n '''\n pass\n\ndef p_decl_specifier_prefix(p):\n '''decl_specifier_prefix : decl_specifier_affix\n | TEMPLATE decl_specifier_prefix\n '''\n pass\n\ndef p_storage_class_specifier(p):\n '''storage_class_specifier : REGISTER \n | STATIC \n | MUTABLE\n | EXTERN %prec SHIFT_THERE\n | EXTENSION\n | AUTO\n '''\n pass\n\ndef p_function_specifier(p):\n '''function_specifier : EXPLICIT\n | INLINE\n | VIRTUAL\n '''\n pass\n\ndef p_type_specifier(p):\n '''type_specifier : simple_type_specifier\n | elaborate_type_specifier\n | cv_qualifier\n '''\n pass\n\ndef p_elaborate_type_specifier(p):\n '''elaborate_type_specifier : class_specifier\n | enum_specifier\n | elaborated_type_specifier\n | TEMPLATE elaborate_type_specifier\n '''\n pass\n\ndef p_simple_type_specifier(p):\n '''simple_type_specifier : scoped_id\n | scoped_id attributes\n | built_in_type_specifier\n '''\n p[0] = p[1]\n\ndef p_built_in_type_specifier(p):\n '''built_in_type_specifier : Xbuilt_in_type_specifier\n | Xbuilt_in_type_specifier attributes\n '''\n pass\n\ndef p_attributes(p):\n '''attributes : attribute\n | attributes attribute\n '''\n pass\n\ndef p_attribute(p):\n '''attribute : ATTRIBUTE '(' parameters_clause ')'\n '''\n\ndef p_Xbuilt_in_type_specifier(p):\n '''Xbuilt_in_type_specifier : CHAR \n | WCHAR_T \n | BOOL \n | SHORT \n | INT \n | LONG \n | SIGNED \n | UNSIGNED \n | FLOAT \n | DOUBLE \n | VOID\n | uTYPEOF parameters_clause\n | TYPEOF parameters_clause\n '''\n pass\n\n#\n# The over-general use of declaration_expression to cover decl-specifier-seq_opt declarator in a function-definition means that\n# class X { };\n# could be a function-definition or a class-specifier.\n# enum X { };\n# could be a function-definition or an enum-specifier.\n# The function-definition is not syntactically valid so resolving the false conflict in favour of the\n# elaborated_type_specifier is correct.\n#\ndef p_elaborated_type_specifier(p):\n '''elaborated_type_specifier : class_key scoped_id %prec SHIFT_THERE\n | elaborated_enum_specifier\n | TYPENAME scoped_id\n '''\n pass\n\ndef p_elaborated_enum_specifier(p):\n '''elaborated_enum_specifier : ENUM scoped_id %prec SHIFT_THERE\n '''\n pass\n\ndef p_enum_specifier(p):\n '''enum_specifier : ENUM scoped_id enumerator_clause\n | ENUM enumerator_clause\n '''\n pass\n\ndef p_enumerator_clause(p):\n '''enumerator_clause : LBRACE enumerator_list_ecarb\n | LBRACE enumerator_list enumerator_list_ecarb\n | LBRACE enumerator_list ',' enumerator_definition_ecarb\n '''\n pass\n\ndef p_enumerator_list_ecarb(p):\n '''enumerator_list_ecarb : RBRACE\n '''\n pass\n\ndef p_enumerator_definition_ecarb(p):\n '''enumerator_definition_ecarb : RBRACE\n '''\n pass\n\ndef p_enumerator_definition_filler(p):\n '''enumerator_definition_filler : empty\n '''\n pass\n\ndef p_enumerator_list_head(p):\n '''enumerator_list_head : enumerator_definition_filler\n | enumerator_list ',' enumerator_definition_filler\n '''\n pass\n\ndef p_enumerator_list(p):\n '''enumerator_list : enumerator_list_head enumerator_definition\n '''\n pass\n\ndef p_enumerator_definition(p):\n '''enumerator_definition : enumerator\n | enumerator '=' constant_expression\n '''\n pass\n\ndef p_enumerator(p):\n '''enumerator : identifier\n '''\n pass\n\ndef p_namespace_definition(p):\n '''namespace_definition : NAMESPACE scoped_id push_scope compound_declaration\n | NAMESPACE push_scope compound_declaration\n '''\n global _parse_info\n scope = _parse_info.pop_scope()\n\ndef p_namespace_alias_definition(p):\n '''namespace_alias_definition : NAMESPACE scoped_id '=' scoped_id ';'\n '''\n pass\n\ndef p_push_scope(p):\n '''push_scope : empty'''\n global _parse_info\n if p[-2] == \"namespace\":\n scope=p[-1]\n else:\n scope=\"\"\n _parse_info.push_scope(scope,\"namespace\")\n\ndef p_using_declaration(p):\n '''using_declaration : USING declarator_id ';'\n | USING TYPENAME declarator_id ';'\n '''\n pass\n\ndef p_using_directive(p):\n '''using_directive : USING NAMESPACE scoped_id ';'\n '''\n pass\n\n# '''asm_definition : ASM '(' StringLiteral ')' ';'\ndef p_asm_definition(p):\n '''asm_definition : ASM '(' nonparen_seq_opt ')' ';'\n '''\n pass\n\ndef p_linkage_specification(p):\n '''linkage_specification : EXTERN CLiteral declaration\n | EXTERN CLiteral compound_declaration\n | EXTERN CppLiteral declaration\n | EXTERN CppLiteral compound_declaration\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.7 Declarators\n#---------------------------------------------------------------------------------------------------\n#\n# init-declarator is named init_declaration to reflect the embedded decl-specifier-seq_opt\n#\n\ndef p_init_declarations(p):\n '''init_declarations : assignment_expression ',' init_declaration\n | init_declarations ',' init_declaration\n '''\n p[0]=get_rest(p)\n\ndef p_init_declaration(p):\n '''init_declaration : assignment_expression\n '''\n p[0]=get_rest(p)\n\ndef p_star_ptr_operator(p):\n '''star_ptr_operator : '*'\n | star_ptr_operator cv_qualifier\n '''\n pass\n\ndef p_nested_ptr_operator(p):\n '''nested_ptr_operator : star_ptr_operator\n | id_scope nested_ptr_operator\n '''\n pass\n\ndef p_ptr_operator(p):\n '''ptr_operator : '&'\n | nested_ptr_operator\n | global_scope nested_ptr_operator\n '''\n pass\n\ndef p_ptr_operator_seq(p):\n '''ptr_operator_seq : ptr_operator\n | ptr_operator ptr_operator_seq\n '''\n pass\n\n#\n# Independently coded to localise the shift-reduce conflict: sharing just needs another %prec\n#\ndef p_ptr_operator_seq_opt(p):\n '''ptr_operator_seq_opt : empty %prec SHIFT_THERE\n | ptr_operator ptr_operator_seq_opt\n '''\n pass\n\ndef p_cv_qualifier_seq_opt(p):\n '''cv_qualifier_seq_opt : empty\n | cv_qualifier_seq_opt cv_qualifier\n '''\n pass\n\n# TODO: verify that we should include attributes here\ndef p_cv_qualifier(p):\n '''cv_qualifier : CONST \n | VOLATILE\n | attributes\n '''\n pass\n\ndef p_type_id(p):\n '''type_id : type_specifier abstract_declarator_opt\n | type_specifier type_id\n '''\n pass\n\ndef p_abstract_declarator_opt(p):\n '''abstract_declarator_opt : empty\n | ptr_operator abstract_declarator_opt\n | direct_abstract_declarator\n '''\n pass\n\ndef p_direct_abstract_declarator_opt(p):\n '''direct_abstract_declarator_opt : empty\n | direct_abstract_declarator\n '''\n pass\n\ndef p_direct_abstract_declarator(p):\n '''direct_abstract_declarator : direct_abstract_declarator_opt parenthesis_clause\n | direct_abstract_declarator_opt LBRACKET RBRACKET\n | direct_abstract_declarator_opt LBRACKET bexpression RBRACKET\n '''\n pass\n\ndef p_parenthesis_clause(p):\n '''parenthesis_clause : parameters_clause cv_qualifier_seq_opt\n | parameters_clause cv_qualifier_seq_opt exception_specification\n '''\n p[0] = ['(',')']\n\ndef p_parameters_clause(p):\n '''parameters_clause : '(' condition_opt ')'\n '''\n p[0] = ['(',')']\n\n#\n# A typed abstract qualifier such as\n# Class * ...\n# looks like a multiply, so pointers are parsed as their binary operation equivalents that\n# ultimately terminate with a degenerate right hand term.\n#\ndef p_abstract_pointer_declaration(p):\n '''abstract_pointer_declaration : ptr_operator_seq\n | multiplicative_expression star_ptr_operator ptr_operator_seq_opt\n '''\n pass\n\ndef p_abstract_parameter_declaration(p):\n '''abstract_parameter_declaration : abstract_pointer_declaration\n | and_expression '&'\n | and_expression '&' abstract_pointer_declaration\n '''\n pass\n\ndef p_special_parameter_declaration(p):\n '''special_parameter_declaration : abstract_parameter_declaration\n | abstract_parameter_declaration '=' assignment_expression\n | ELLIPSIS\n '''\n pass\n\ndef p_parameter_declaration(p):\n '''parameter_declaration : assignment_expression\n | special_parameter_declaration\n | decl_specifier_prefix parameter_declaration\n '''\n pass\n\n#\n# function_definition includes constructor, destructor, implicit int definitions too. A local destructor is successfully parsed as a function-declaration but the ~ was treated as a unary operator. constructor_head is the prefix ambiguity between a constructor and a member-init-list starting with a bit-field.\n#\ndef p_function_definition(p):\n '''function_definition : ctor_definition\n | func_definition\n '''\n pass\n\ndef p_func_definition(p):\n '''func_definition : assignment_expression function_try_block\n | assignment_expression function_body\n | decl_specifier_prefix func_definition\n '''\n global _parse_info\n if p[2] is not None and p[2][0] == '{':\n decl = flatten(p[1])\n #print \"HERE\",decl\n if decl[-1] == ')':\n decl=decl[-3]\n else:\n decl=decl[-1]\n p[0] = decl\n if decl != \"operator\":\n _parse_info.add_function(decl)\n else:\n p[0] = p[2]\n\ndef p_ctor_definition(p):\n '''ctor_definition : constructor_head function_try_block\n | constructor_head function_body\n | decl_specifier_prefix ctor_definition\n '''\n if p[2] is None or p[2][0] == \"try\" or p[2][0] == '{':\n p[0]=p[1]\n else:\n p[0]=p[1]\n\ndef p_constructor_head(p):\n '''constructor_head : bit_field_init_declaration\n | constructor_head ',' assignment_expression\n '''\n p[0]=p[1]\n\ndef p_function_try_block(p):\n '''function_try_block : TRY function_block handler_seq\n '''\n global noExceptionLogic\n noExceptionLogic=False\n p[0] = ['try']\n\ndef p_function_block(p):\n '''function_block : ctor_initializer_opt function_body\n '''\n pass\n\ndef p_function_body(p):\n '''function_body : LBRACE nonbrace_seq_opt RBRACE \n '''\n p[0] = ['{','}']\n\ndef p_initializer_clause(p):\n '''initializer_clause : assignment_expression\n | braced_initializer\n '''\n pass\n\ndef p_braced_initializer(p):\n '''braced_initializer : LBRACE initializer_list RBRACE\n | LBRACE initializer_list ',' RBRACE\n | LBRACE RBRACE\n '''\n pass\n\ndef p_initializer_list(p):\n '''initializer_list : initializer_clause\n | initializer_list ',' initializer_clause\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.8 Classes\n#---------------------------------------------------------------------------------------------------\n#\n# An anonymous bit-field declaration may look very like inheritance:\n# const int B = 3;\n# class A : B ;\n# The two usages are too distant to try to create and enforce a common prefix so we have to resort to\n# a parser hack by backtracking. Inheritance is much the most likely so we mark the input stream context\n# and try to parse a base-clause. If we successfully reach a { the base-clause is ok and inheritance was\n# the correct choice so we unmark and continue. If we fail to find the { an error token causes \n# back-tracking to the alternative parse in elaborated_type_specifier which regenerates the : and \n# declares unconditional success.\n#\n\ndef p_class_specifier_head(p):\n '''class_specifier_head : class_key scoped_id ':' base_specifier_list LBRACE\n | class_key ':' base_specifier_list LBRACE\n | class_key scoped_id LBRACE\n | class_key LBRACE\n '''\n global _parse_info\n base_classes=[]\n if len(p) == 6:\n scope = p[2]\n base_classes = p[4]\n elif len(p) == 4:\n scope = p[2]\n elif len(p) == 5:\n base_classes = p[3]\n else:\n scope = \"\"\n _parse_info.push_scope(scope,p[1],base_classes)\n \n\ndef p_class_key(p):\n '''class_key : CLASS \n | STRUCT \n | UNION\n '''\n p[0] = p[1]\n\ndef p_class_specifier(p):\n '''class_specifier : class_specifier_head member_specification_opt RBRACE\n '''\n scope = _parse_info.pop_scope()\n\ndef p_member_specification_opt(p):\n '''member_specification_opt : empty\n | member_specification_opt member_declaration\n '''\n pass\n\ndef p_member_declaration(p):\n '''member_declaration : accessibility_specifier\n | simple_member_declaration\n | function_definition\n | using_declaration\n | template_declaration\n '''\n p[0] = get_rest(p)\n #print \"Decl\",get_rest(p)\n\n#\n# The generality of constructor names (there need be no parenthesised argument list) means that that\n# name : f(g), h(i)\n# could be the start of a constructor or the start of an anonymous bit-field. An ambiguity is avoided by\n# parsing the ctor-initializer of a function_definition as a bit-field.\n#\ndef p_simple_member_declaration(p):\n '''simple_member_declaration : ';'\n | assignment_expression ';'\n | constructor_head ';'\n | member_init_declarations ';'\n | decl_specifier_prefix simple_member_declaration\n '''\n global _parse_info\n decl = flatten(get_rest(p))\n if len(decl) >= 4 and decl[-3] == \"(\":\n _parse_info.add_function(decl[-4])\n\ndef p_member_init_declarations(p):\n '''member_init_declarations : assignment_expression ',' member_init_declaration\n | constructor_head ',' bit_field_init_declaration\n | member_init_declarations ',' member_init_declaration\n '''\n pass\n\ndef p_member_init_declaration(p):\n '''member_init_declaration : assignment_expression\n | bit_field_init_declaration\n '''\n pass\n\ndef p_accessibility_specifier(p):\n '''accessibility_specifier : access_specifier ':'\n '''\n pass\n\ndef p_bit_field_declaration(p):\n '''bit_field_declaration : assignment_expression ':' bit_field_width\n | ':' bit_field_width\n '''\n if len(p) == 4:\n p[0]=p[1]\n\ndef p_bit_field_width(p):\n '''bit_field_width : logical_or_expression\n | logical_or_expression '?' bit_field_width ':' bit_field_width\n '''\n pass\n\ndef p_bit_field_init_declaration(p):\n '''bit_field_init_declaration : bit_field_declaration\n | bit_field_declaration '=' initializer_clause\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.9 Derived classes\n#---------------------------------------------------------------------------------------------------\ndef p_base_specifier_list(p):\n '''base_specifier_list : base_specifier\n | base_specifier_list ',' base_specifier\n '''\n if len(p) == 2:\n p[0] = [p[1]]\n else:\n p[0] = p[1]+[p[3]]\n\ndef p_base_specifier(p):\n '''base_specifier : scoped_id\n | access_specifier base_specifier\n | VIRTUAL base_specifier\n '''\n if len(p) == 2:\n p[0] = p[1]\n else:\n p[0] = p[2]\n\ndef p_access_specifier(p):\n '''access_specifier : PRIVATE \n | PROTECTED \n | PUBLIC\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.10 Special member functions\n#---------------------------------------------------------------------------------------------------\ndef p_conversion_function_id(p):\n '''conversion_function_id : OPERATOR conversion_type_id\n '''\n p[0] = ['operator']\n\ndef p_conversion_type_id(p):\n '''conversion_type_id : type_specifier ptr_operator_seq_opt\n | type_specifier conversion_type_id\n '''\n pass\n\n#\n# Ctor-initialisers can look like a bit field declaration, given the generalisation of names:\n# Class(Type) : m1(1), m2(2) { }\n# NonClass(bit_field) : int(2), second_variable, ...\n# The grammar below is used within a function_try_block or function_definition.\n# See simple_member_declaration for use in normal member function_definition.\n#\ndef p_ctor_initializer_opt(p):\n '''ctor_initializer_opt : empty\n | ctor_initializer\n '''\n pass\n\ndef p_ctor_initializer(p):\n '''ctor_initializer : ':' mem_initializer_list\n '''\n pass\n\ndef p_mem_initializer_list(p):\n '''mem_initializer_list : mem_initializer\n | mem_initializer_list_head mem_initializer\n '''\n pass\n\ndef p_mem_initializer_list_head(p):\n '''mem_initializer_list_head : mem_initializer_list ','\n '''\n pass\n\ndef p_mem_initializer(p):\n '''mem_initializer : mem_initializer_id '(' expression_list_opt ')'\n '''\n pass\n\ndef p_mem_initializer_id(p):\n '''mem_initializer_id : scoped_id\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.11 Overloading\n#---------------------------------------------------------------------------------------------------\n\ndef p_operator_function_id(p):\n '''operator_function_id : OPERATOR operator\n | OPERATOR '(' ')'\n | OPERATOR LBRACKET RBRACKET\n | OPERATOR '<'\n | OPERATOR '>'\n | OPERATOR operator '<' nonlgt_seq_opt '>'\n '''\n p[0] = [\"operator\"]\n\n#\n# It is not clear from the ANSI standard whether spaces are permitted in delete[]. If not then it can\n# be recognised and returned as DELETE_ARRAY by the lexer. Assuming spaces are permitted there is an\n# ambiguity created by the over generalised nature of expressions. operator new is a valid delarator-id\n# which we may have an undimensioned array of. Semantic rubbish, but syntactically valid. Since the\n# array form is covered by the declarator consideration we can exclude the operator here. The need\n# for a semantic rescue can be eliminated at the expense of a couple of shift-reduce conflicts by\n# removing the comments on the next four lines.\n#\ndef p_operator(p):\n '''operator : NEW\n | DELETE\n | '+'\n | '-'\n | '*'\n | '/'\n | '%'\n | '^'\n | '&'\n | '|'\n | '~'\n | '!'\n | '='\n | ASS_ADD\n | ASS_SUB\n | ASS_MUL\n | ASS_DIV\n | ASS_MOD\n | ASS_XOR\n | ASS_AND\n | ASS_OR\n | SHL\n | SHR\n | ASS_SHR\n | ASS_SHL\n | EQ\n | NE\n | LE\n | GE\n | LOG_AND\n | LOG_OR\n | INC\n | DEC\n | ','\n | ARROW_STAR\n | ARROW\n '''\n p[0]=p[1]\n\n# | IF\n# | SWITCH\n# | WHILE\n# | FOR\n# | DO\ndef p_reserved(p):\n '''reserved : PRIVATE\n | CLiteral\n | CppLiteral\n | IF\n | SWITCH\n | WHILE\n | FOR\n | DO\n | PROTECTED\n | PUBLIC\n | BOOL\n | CHAR\n | DOUBLE\n | FLOAT\n | INT\n | LONG\n | SHORT\n | SIGNED\n | UNSIGNED\n | VOID\n | WCHAR_T\n | CLASS\n | ENUM\n | NAMESPACE\n | STRUCT\n | TYPENAME\n | UNION\n | CONST\n | VOLATILE\n | AUTO\n | EXPLICIT\n | EXPORT\n | EXTERN\n | FRIEND\n | INLINE\n | MUTABLE\n | REGISTER\n | STATIC\n | TEMPLATE\n | TYPEDEF\n | USING\n | VIRTUAL\n | ASM\n | BREAK\n | CASE\n | CATCH\n | CONST_CAST\n | CONTINUE\n | DEFAULT\n | DYNAMIC_CAST\n | ELSE\n | FALSE\n | GOTO\n | OPERATOR\n | REINTERPRET_CAST\n | RETURN\n | SIZEOF\n | STATIC_CAST\n | THIS\n | THROW\n | TRUE\n | TRY\n | TYPEID\n | ATTRIBUTE\n | CDECL\n | TYPEOF\n | uTYPEOF\n '''\n if p[1] in ('try', 'catch', 'throw'):\n global noExceptionLogic\n noExceptionLogic=False\n\n#---------------------------------------------------------------------------------------------------\n# A.12 Templates\n#---------------------------------------------------------------------------------------------------\ndef p_template_declaration(p):\n '''template_declaration : template_parameter_clause declaration\n | EXPORT template_declaration\n '''\n pass\n\ndef p_template_parameter_clause(p):\n '''template_parameter_clause : TEMPLATE '<' nonlgt_seq_opt '>'\n '''\n pass\n\n#\n# Generalised naming makes identifier a valid declaration, so TEMPLATE identifier is too.\n# The TEMPLATE prefix is therefore folded into all names, parenthesis_clause and decl_specifier_prefix.\n#\n# explicit_instantiation: TEMPLATE declaration\n#\ndef p_explicit_specialization(p):\n '''explicit_specialization : TEMPLATE '<' '>' declaration\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.13 Exception Handling\n#---------------------------------------------------------------------------------------------------\ndef p_handler_seq(p):\n '''handler_seq : handler\n | handler handler_seq\n '''\n pass\n\ndef p_handler(p):\n '''handler : CATCH '(' exception_declaration ')' compound_statement\n '''\n global noExceptionLogic\n noExceptionLogic=False\n\ndef p_exception_declaration(p):\n '''exception_declaration : parameter_declaration\n '''\n pass\n\ndef p_throw_expression(p):\n '''throw_expression : THROW\n | THROW assignment_expression\n '''\n global noExceptionLogic\n noExceptionLogic=False\n\ndef p_exception_specification(p):\n '''exception_specification : THROW '(' ')'\n | THROW '(' type_id_list ')'\n '''\n global noExceptionLogic\n noExceptionLogic=False\n\ndef p_type_id_list(p):\n '''type_id_list : type_id\n | type_id_list ',' type_id\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# Misc productions\n#---------------------------------------------------------------------------------------------------\ndef p_nonsemicolon_seq(p):\n '''nonsemicolon_seq : empty\n | nonsemicolon_seq nonsemicolon\n '''\n pass\n\ndef p_nonsemicolon(p):\n '''nonsemicolon : misc\n | '('\n | ')'\n | '<'\n | '>'\n | LBRACKET nonbracket_seq_opt RBRACKET\n | LBRACE nonbrace_seq_opt RBRACE\n '''\n pass\n\ndef p_nonparen_seq_opt(p):\n '''nonparen_seq_opt : empty\n | nonparen_seq_opt nonparen\n '''\n pass\n\ndef p_nonparen_seq(p):\n '''nonparen_seq : nonparen\n | nonparen_seq nonparen\n '''\n pass\n\ndef p_nonparen(p):\n '''nonparen : misc\n | '<'\n | '>'\n | ';'\n | LBRACKET nonbracket_seq_opt RBRACKET\n | LBRACE nonbrace_seq_opt RBRACE\n '''\n pass\n\ndef p_nonbracket_seq_opt(p):\n '''nonbracket_seq_opt : empty\n | nonbracket_seq_opt nonbracket\n '''\n pass\n\ndef p_nonbracket_seq(p):\n '''nonbracket_seq : nonbracket\n | nonbracket_seq nonbracket\n '''\n pass\n\ndef p_nonbracket(p):\n '''nonbracket : misc\n | '<'\n | '>'\n | '('\n | ')'\n | ';'\n | LBRACKET nonbracket_seq_opt RBRACKET\n | LBRACE nonbrace_seq_opt RBRACE\n '''\n pass\n\ndef p_nonbrace_seq_opt(p):\n '''nonbrace_seq_opt : empty\n | nonbrace_seq_opt nonbrace\n '''\n pass\n\ndef p_nonbrace(p):\n '''nonbrace : misc\n | '<'\n | '>'\n | '('\n | ')'\n | ';'\n | LBRACKET nonbracket_seq_opt RBRACKET\n | LBRACE nonbrace_seq_opt RBRACE\n '''\n pass\n\ndef p_nonlgt_seq_opt(p):\n '''nonlgt_seq_opt : empty\n | nonlgt_seq_opt nonlgt\n '''\n pass\n\ndef p_nonlgt(p):\n '''nonlgt : misc\n | '('\n | ')'\n | LBRACKET nonbracket_seq_opt RBRACKET\n | '<' nonlgt_seq_opt '>'\n | ';'\n '''\n pass\n\ndef p_misc(p):\n '''misc : operator\n | identifier\n | IntegerLiteral\n | CharacterLiteral\n | FloatingLiteral\n | StringLiteral\n | reserved\n | '?'\n | ':'\n | '.'\n | SCOPE\n | ELLIPSIS\n | EXTENSION\n '''\n pass\n\ndef p_empty(p):\n '''empty : '''\n pass\n\n\n\n#\n# Compute column.\n# input is the input text string\n# token is a token instance\n#\ndef _find_column(input,token):\n ''' TODO '''\n i = token.lexpos\n while i > 0:\n if input[i] == '\\n': break\n i -= 1\n column = (token.lexpos - i)+1\n return column\n\ndef p_error(p):\n if p is None:\n tmp = \"Syntax error at end of file.\"\n else:\n tmp = \"Syntax error at token \"\n if p.type is \"\":\n tmp = tmp + \"''\"\n else:\n tmp = tmp + str(p.type)\n tmp = tmp + \" with value '\"+str(p.value)+\"'\"\n tmp = tmp + \" in line \" + str(lexer.lineno-1)\n tmp = tmp + \" at column \"+str(_find_column(_parsedata,p))\n raise IOError( tmp )\n\n\n\n#\n# The function that performs the parsing\n#\ndef parse_cpp(data=None, filename=None, debug=0, optimize=0, verbose=False, func_filter=None):\n if debug > 0:\n print \"Debugging parse_cpp!\"\n #\n # Always remove the parser.out file, which is generated to create debugging\n #\n if os.path.exists(\"parser.out\"):\n os.remove(\"parser.out\")\n #\n # Remove the parsetab.py* files. These apparently need to be removed\n # to ensure the creation of a parser.out file.\n #\n if os.path.exists(\"parsetab.py\"):\n os.remove(\"parsetab.py\")\n if os.path.exists(\"parsetab.pyc\"):\n os.remove(\"parsetab.pyc\")\n global debugging\n debugging=True\n #\n # Build lexer\n #\n global lexer\n lexer = lex.lex()\n #\n # Initialize parse object\n #\n global _parse_info\n _parse_info = CppInfo(filter=func_filter)\n _parse_info.verbose=verbose\n #\n # Build yaccer\n #\n write_table = not os.path.exists(\"parsetab.py\")\n yacc.yacc(debug=debug, optimize=optimize, write_tables=write_table)\n #\n # Parse the file\n #\n global _parsedata\n if not data is None:\n _parsedata=data\n ply_init(_parsedata)\n yacc.parse(data,debug=debug)\n elif not filename is None:\n f = open(filename)\n data = f.read()\n f.close()\n _parsedata=data\n ply_init(_parsedata)\n yacc.parse(data, debug=debug)\n else:\n return None\n #\n if not noExceptionLogic:\n _parse_info.noExceptionLogic = False\n else:\n for key in identifier_lineno:\n if 'ASSERT_THROWS' in key:\n _parse_info.noExceptionLogic = False\n break\n _parse_info.noExceptionLogic = True\n #\n return _parse_info\n\n\n\nimport sys\n\nif __name__ == '__main__':\n #\n # This MAIN routine parses a sequence of files provided at the command\n # line. If '-v' is included, then a verbose parsing output is \n # generated.\n #\n for arg in sys.argv[1:]:\n if arg == \"-v\":\n continue\n print \"Parsing file '\"+arg+\"'\"\n if '-v' in sys.argv:\n parse_cpp(filename=arg,debug=2,verbose=2)\n else:\n parse_cpp(filename=arg,verbose=2)\n #\n # Print the _parse_info object summary for this file.\n # This illustrates how class inheritance can be used to \n # deduce class members.\n # \n print str(_parse_info)\n\n" }, { "path": "cxxtest/python/cxxtest/cxxtest_fog.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n#\n# TODO: add line number info\n# TODO: add test function names\n#\n\nfrom __future__ import division\n\nimport sys\nimport re\n#from os.path import abspath, dirname\n#sys.path.insert(0, dirname(dirname(abspath(__file__))))\n#sys.path.insert(0, dirname(dirname(abspath(__file__)))+\"/cxx_parse\")\nfrom cxxtest_misc import abort\nimport cxx_parser\nimport re\n\ndef cstr( str ):\n '''Convert a string to its C representation'''\n return '\"' + re.sub('\\\\\\\\', '\\\\\\\\\\\\\\\\', str ) + '\"'\n\ndef scanInputFiles(files, _options):\n '''Scan all input files for test suites'''\n suites=[]\n for file in files:\n try:\n print \"Parsing file \"+file,\n sys.stdout.flush()\n parse_info = cxx_parser.parse_cpp(filename=file,optimize=1)\n except IOError, err:\n print \" error.\"\n print str(err)\n continue\n print \"done.\" \n sys.stdout.flush()\n #\n # WEH: see if it really makes sense to use parse information to\n # initialize this data. I don't think so...\n #\n _options.haveStandardLibrary=1\n if not parse_info.noExceptionLogic:\n _options.haveExceptionHandling=1\n #\n keys = list(parse_info.index.keys())\n tpat = re.compile(\"[Tt][Ee][Ss][Tt]\")\n for key in keys:\n if parse_info.index[key].scope_t == \"class\" and parse_info.is_baseclass(key,\"CxxTest::TestSuite\"):\n name=parse_info.index[key].name\n suite = { 'name' : name,\n 'file' : file,\n 'cfile' : cstr(file),\n 'line' : str(parse_info.index[key].lineno),\n 'generated' : 0,\n 'object' : 'suite_%s' % name,\n 'dobject' : 'suiteDescription_%s' % name,\n 'tlist' : 'Tests_%s' % name,\n 'tests' : [],\n 'lines' : [] }\n for fn in parse_info.get_functions(key,quiet=True):\n tname = fn[0]\n lineno = str(fn[1])\n if tname.startswith('createSuite'):\n # Indicate that we're using a dynamically generated test suite\n suite['create'] = str(lineno) # (unknown line)\n if tname.startswith('destroySuite'):\n # Indicate that we're using a dynamically generated test suite\n suite['destroy'] = str(lineno) # (unknown line)\n if not tpat.match(tname):\n # Skip non-test methods\n continue\n test = { 'name' : tname,\n 'suite' : suite,\n 'class' : 'TestDescription_suite_%s_%s' % (suite['name'], tname),\n 'object' : 'testDescription_suite_%s_%s' % (suite['name'], tname),\n 'line' : lineno,\n }\n suite['tests'].append(test)\n suites.append(suite)\n\n if not _options.root:\n ntests = 0\n for suite in suites:\n ntests += len(suite['tests'])\n if ntests == 0:\n abort( 'No tests defined' )\n #\n return [_options, suites]\n\n" }, { "path": "cxxtest/python/cxxtest/cxxtest_misc.py", "content": "#!/usr/bin/python\n#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\nimport sys\n\ndef abort( problem ):\n '''Print error message and exit'''\n sys.stderr.write( '\\n' )\n sys.stderr.write( problem )\n sys.stderr.write( '\\n\\n' )\n sys.exit(2)\n\n" }, { "path": "cxxtest/python/cxxtest/cxxtest_parser.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\nfrom __future__ import division\n\nimport codecs\nimport re\n#import sys\n#import getopt\n#import glob\nfrom cxxtest.cxxtest_misc import abort\n\n# Global variables\nsuites = []\nsuite = None\ninBlock = 0\noptions=None\n\ndef scanInputFiles(files, _options):\n '''Scan all input files for test suites'''\n global options\n options=_options\n for file in files:\n scanInputFile(file)\n global suites\n if len(suites) is 0 and not options.root:\n abort( 'No tests defined' )\n return [options,suites]\n\nlineCont_re = re.compile('(.*)\\\\\\s*$')\ndef scanInputFile(fileName):\n '''Scan single input file for test suites'''\n # mode 'rb' is problematic in python3 - byte arrays don't behave the same as\n # strings.\n # As far as the choice of the default encoding: utf-8 chews through\n # everything that the previous ascii codec could, plus most of new code.\n # TODO: figure out how to do this properly - like autodetect encoding from\n # file header.\n file = codecs.open(fileName, mode='r', encoding='utf-8')\n prev = \"\"\n lineNo = 0\n contNo = 0\n while 1:\n line = file.readline()\n if not line:\n break\n lineNo += 1\n\n m = lineCont_re.match(line)\n if m:\n prev += m.group(1) + \" \"\n contNo += 1\n else:\n scanInputLine( fileName, lineNo - contNo, prev + line )\n contNo = 0\n prev = \"\"\n if contNo:\n scanInputLine( fileName, lineNo - contNo, prev + line )\n \n closeSuite()\n file.close()\n\ndef scanInputLine( fileName, lineNo, line ):\n '''Scan single input line for interesting stuff'''\n scanLineForExceptionHandling( line )\n scanLineForStandardLibrary( line )\n\n scanLineForSuiteStart( fileName, lineNo, line )\n\n global suite\n if suite:\n scanLineInsideSuite( suite, lineNo, line )\n\ndef scanLineInsideSuite( suite, lineNo, line ):\n '''Analyze line which is part of a suite'''\n global inBlock\n if lineBelongsToSuite( suite, lineNo, line ):\n scanLineForTest( suite, lineNo, line )\n scanLineForCreate( suite, lineNo, line )\n scanLineForDestroy( suite, lineNo, line )\n\ndef lineBelongsToSuite( suite, lineNo, line ):\n '''Returns whether current line is part of the current suite.\n This can be false when we are in a generated suite outside of CXXTEST_CODE() blocks\n If the suite is generated, adds the line to the list of lines'''\n if not suite['generated']:\n return 1\n\n global inBlock\n if not inBlock:\n inBlock = lineStartsBlock( line )\n if inBlock:\n inBlock = addLineToBlock( suite, lineNo, line )\n return inBlock\n\n\nstd_re = re.compile( r\"\\b(std\\s*::|CXXTEST_STD|using\\s+namespace\\s+std\\b|^\\s*\\#\\s*include\\s+<[a-z0-9]+>)\" )\ndef scanLineForStandardLibrary( line ):\n '''Check if current line uses standard library'''\n global options\n if not options.haveStandardLibrary and std_re.search(line):\n if not options.noStandardLibrary:\n options.haveStandardLibrary = 1\n\nexception_re = re.compile( r\"\\b(throw|try|catch|TSM?_ASSERT_THROWS[A-Z_]*)\\b\" )\ndef scanLineForExceptionHandling( line ):\n '''Check if current line uses exception handling'''\n global options\n if not options.haveExceptionHandling and exception_re.search(line):\n if not options.noExceptionHandling:\n options.haveExceptionHandling = 1\n\nclassdef = '(?:::\\s*)?(?:\\w+\\s*::\\s*)*\\w+'\nbaseclassdef = '(?:public|private|protected)\\s+%s' % (classdef,)\ngeneral_suite = r\"\\bclass\\s+(%s)\\s*:(?:\\s*%s\\s*,)*\\s*public\\s+\" \\\n % (classdef, baseclassdef,)\ntestsuite = '(?:(?:::)?\\s*CxxTest\\s*::\\s*)?TestSuite'\nsuites_re = { re.compile( general_suite + testsuite ) : None }\ngeneratedSuite_re = re.compile( r'\\bCXXTEST_SUITE\\s*\\(\\s*(\\w*)\\s*\\)' )\ndef scanLineForSuiteStart( fileName, lineNo, line ):\n '''Check if current line starts a new test suite'''\n for i in list(suites_re.items()):\n m = i[0].search( line )\n if m:\n suite = startSuite( m.group(1), fileName, lineNo, 0 )\n if i[1] is not None:\n for test in i[1]['tests']:\n addTest(suite, test['name'], test['line'])\n break\n m = generatedSuite_re.search( line )\n if m:\n sys.stdout.write( \"%s:%s: Warning: Inline test suites are deprecated.\\n\" % (fileName, lineNo) )\n startSuite( m.group(1), fileName, lineNo, 1 )\n\ndef startSuite( name, file, line, generated ):\n '''Start scanning a new suite'''\n global suite\n closeSuite()\n object_name = name.replace(':',\"_\")\n suite = { 'name' : name,\n 'file' : file,\n 'cfile' : cstr(file),\n 'line' : line,\n 'generated' : generated,\n 'object' : 'suite_%s' % object_name,\n 'dobject' : 'suiteDescription_%s' % object_name,\n 'tlist' : 'Tests_%s' % object_name,\n 'tests' : [],\n 'lines' : [] }\n suites_re[re.compile( general_suite + name )] = suite\n return suite\n\ndef lineStartsBlock( line ):\n '''Check if current line starts a new CXXTEST_CODE() block'''\n return re.search( r'\\bCXXTEST_CODE\\s*\\(', line ) is not None\n\ntest_re = re.compile( r'^([^/]|/[^/])*\\bvoid\\s+([Tt]est\\w+)\\s*\\(\\s*(void)?\\s*\\)' )\ndef scanLineForTest( suite, lineNo, line ):\n '''Check if current line starts a test'''\n m = test_re.search( line )\n if m:\n addTest( suite, m.group(2), lineNo )\n\ndef addTest( suite, name, line ):\n '''Add a test function to the current suite'''\n test = { 'name' : name,\n 'suite' : suite,\n 'class' : 'TestDescription_%s_%s' % (suite['object'], name),\n 'object' : 'testDescription_%s_%s' % (suite['object'], name),\n 'line' : line,\n }\n suite['tests'].append( test )\n\ndef addLineToBlock( suite, lineNo, line ):\n '''Append the line to the current CXXTEST_CODE() block'''\n line = fixBlockLine( suite, lineNo, line )\n line = re.sub( r'^.*\\{\\{', '', line )\n \n e = re.search( r'\\}\\}', line )\n if e:\n line = line[:e.start()]\n suite['lines'].append( line )\n return e is None\n\ndef fixBlockLine( suite, lineNo, line):\n '''Change all [E]TS_ macros used in a line to _[E]TS_ macros with the correct file/line'''\n return re.sub( r'\\b(E?TSM?_(ASSERT[A-Z_]*|FAIL))\\s*\\(',\n r'_\\1(%s,%s,' % (suite['cfile'], lineNo),\n line, 0 )\n\ncreate_re = re.compile( r'\\bstatic\\s+\\w+\\s*\\*\\s*createSuite\\s*\\(\\s*(void)?\\s*\\)' )\ndef scanLineForCreate( suite, lineNo, line ):\n '''Check if current line defines a createSuite() function'''\n if create_re.search( line ):\n addSuiteCreateDestroy( suite, 'create', lineNo )\n\ndestroy_re = re.compile( r'\\bstatic\\s+void\\s+destroySuite\\s*\\(\\s*\\w+\\s*\\*\\s*\\w*\\s*\\)' )\ndef scanLineForDestroy( suite, lineNo, line ):\n '''Check if current line defines a destroySuite() function'''\n if destroy_re.search( line ):\n addSuiteCreateDestroy( suite, 'destroy', lineNo )\n\ndef cstr( s ):\n '''Convert a string to its C representation'''\n return '\"' + s.replace( '\\\\', '\\\\\\\\' ) + '\"'\n\n\ndef addSuiteCreateDestroy( suite, which, line ):\n '''Add createSuite()/destroySuite() to current suite'''\n if which in suite:\n abort( '%s:%s: %sSuite() already declared' % ( suite['file'], str(line), which ) )\n suite[which] = line\n\ndef closeSuite():\n '''Close current suite and add it to the list if valid'''\n global suite\n if suite is not None:\n if len(suite['tests']) is not 0:\n verifySuite(suite)\n rememberSuite(suite)\n suite = None\n\ndef verifySuite(suite):\n '''Verify current suite is legal'''\n if 'create' in suite and 'destroy' not in suite:\n abort( '%s:%s: Suite %s has createSuite() but no destroySuite()' %\n (suite['file'], suite['create'], suite['name']) )\n elif 'destroy' in suite and 'create' not in suite:\n abort( '%s:%s: Suite %s has destroySuite() but no createSuite()' %\n (suite['file'], suite['destroy'], suite['name']) )\n\ndef rememberSuite(suite):\n '''Add current suite to list'''\n global suites\n suites.append( suite )\n\n" }, { "path": "cxxtest/python/cxxtest/cxxtestgen.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n# vim: fileencoding=utf-8\n\nfrom __future__ import division\n# the above import important for forward-compatibility with python3,\n# which is already the default in archlinux!\n\n__all__ = ['main']\n\nimport __release__\nimport os\nimport sys\nimport re\nimport glob\nfrom optparse import OptionParser\nimport cxxtest_parser\n\ntry:\n import cxxtest_fog\n imported_fog=True\nexcept ImportError:\n imported_fog=False\n\nfrom cxxtest_misc import abort\n\noptions = []\nsuites = []\n\nwrotePreamble = 0\nwroteWorld = 0\nlastIncluded = ''\n\ndef main(args=sys.argv):\n '''The main program'''\n #\n # Reset global state\n #\n global wrotePreamble\n wrotePreamble=0\n global wroteWorld\n wroteWorld=0\n global lastIncluded\n lastIncluded = ''\n\n global suites\n global options\n files = parseCommandline(args)\n if imported_fog and options.fog:\n [options,suites] = cxxtest_fog.scanInputFiles( files, options )\n else:\n [options,suites] = cxxtest_parser.scanInputFiles( files, options )\n writeOutput()\n\ndef parseCommandline(args):\n '''Analyze command line arguments'''\n global imported_fog\n global options\n parser = OptionParser(\"%prog [options] [<filename> ...]\")\n parser.add_option(\"--version\",\n action=\"store_true\", dest=\"version\", default=False,\n help=\"Write the CxxTest version.\")\n parser.add_option(\"-o\", \"--output\",\n dest=\"outputFileName\", default=None, metavar=\"NAME\",\n help=\"Write output to file NAME.\")\n parser.add_option(\"-w\",\"--world\", dest=\"world\", default=\"cxxtest\",\n help=\"The label of the tests, used to name the XML results.\")\n parser.add_option(\"\", \"--include\", action=\"append\",\n dest=\"headers\", default=[], metavar=\"HEADER\",\n help=\"Include file HEADER in the test runner before other headers.\")\n parser.add_option(\"\", \"--abort-on-fail\",\n action=\"store_true\", dest=\"abortOnFail\", default=False,\n help=\"Abort tests on failed asserts (like xUnit).\")\n parser.add_option(\"\", \"--main\",\n action=\"store\", dest=\"main\", default=\"main\",\n help=\"Specify an alternative name for the main() function.\")\n parser.add_option(\"\", \"--headers\",\n action=\"store\", dest=\"header_filename\", default=None,\n help=\"Specify a filename that contains a list of header files that are processed to generate a test runner.\")\n parser.add_option(\"\", \"--runner\",\n dest=\"runner\", default=\"\", metavar=\"CLASS\",\n help=\"Create a test runner that processes test events using the class CxxTest::CLASS.\")\n parser.add_option(\"\", \"--gui\",\n dest=\"gui\", metavar=\"CLASS\",\n help=\"Create a GUI test runner that processes test events using the class CxxTest::CLASS. (deprecated)\")\n parser.add_option(\"\", \"--error-printer\",\n action=\"store_true\", dest=\"error_printer\", default=False,\n help=\"Create a test runner using the ErrorPrinter class, and allow the use of the standard library.\")\n parser.add_option(\"\", \"--xunit-printer\",\n action=\"store_true\", dest=\"xunit_printer\", default=False,\n help=\"Create a test runner using the XUnitPrinter class.\")\n parser.add_option(\"\", \"--xunit-file\", dest=\"xunit_file\", default=\"\",\n help=\"The file to which the XML summary is written for test runners using the XUnitPrinter class. The default XML filename is TEST-<world>.xml, where <world> is the value of the --world option. (default: cxxtest)\")\n parser.add_option(\"\", \"--have-std\",\n action=\"store_true\", dest=\"haveStandardLibrary\", default=False,\n help=\"Use the standard library (even if not found in tests).\")\n parser.add_option(\"\", \"--no-std\",\n action=\"store_true\", dest=\"noStandardLibrary\", default=False,\n help=\"Do not use standard library (even if found in tests).\")\n parser.add_option(\"\", \"--have-eh\",\n action=\"store_true\", dest=\"haveExceptionHandling\", default=False,\n help=\"Use exception handling (even if not found in tests).\")\n parser.add_option(\"\", \"--no-eh\",\n action=\"store_true\", dest=\"noExceptionHandling\", default=False,\n help=\"Do not use exception handling (even if found in tests).\")\n parser.add_option(\"\", \"--longlong\",\n dest=\"longlong\", default=None, metavar=\"TYPE\",\n help=\"Use TYPE as for long long integers. (default: not supported)\")\n parser.add_option(\"\", \"--no-static-init\",\n action=\"store_true\", dest=\"noStaticInit\", default=False,\n help=\"Do not rely on static initialization in the test runner.\")\n parser.add_option(\"\", \"--template\",\n dest=\"templateFileName\", default=None, metavar=\"TEMPLATE\",\n help=\"Generate the test runner using file TEMPLATE to define a template.\")\n parser.add_option(\"\", \"--root\",\n action=\"store_true\", dest=\"root\", default=False,\n help=\"Write the main() function and global data for a test runner.\")\n parser.add_option(\"\", \"--part\",\n action=\"store_true\", dest=\"part\", default=False,\n help=\"Write the tester classes for a test runner.\")\n #parser.add_option(\"\", \"--factor\",\n #action=\"store_true\", dest=\"factor\", default=False,\n #help=\"Declare the _CXXTEST_FACTOR macro. (deprecated)\")\n if imported_fog:\n fog_help = \"Use new FOG C++ parser\"\n else:\n fog_help = \"Use new FOG C++ parser (disabled)\"\n parser.add_option(\"-f\", \"--fog-parser\",\n action=\"store_true\",\n dest=\"fog\",\n default=False,\n help=fog_help\n )\n\n (options, args) = parser.parse_args(args=args)\n if not options.header_filename is None:\n if not os.path.exists(options.header_filename):\n abort( \"ERROR: the file '%s' does not exist!\" % options.header_filename )\n INPUT = open(options.header_filename)\n headers = [line.strip() for line in INPUT]\n args.extend( headers )\n INPUT.close()\n\n if options.fog and not imported_fog:\n abort( \"Cannot use the FOG parser. Check that the 'ply' package is installed. The 'ordereddict' package is also required if running Python 2.6\")\n\n if options.version:\n printVersion()\n\n # the cxxtest builder relies on this behaviour! don't remove\n if options.runner == 'none':\n options.runner = None\n\n if options.xunit_printer or options.runner == \"XUnitPrinter\":\n options.xunit_printer=True\n options.runner=\"XUnitPrinter\"\n if len(args) > 1:\n if options.xunit_file == \"\":\n if options.world == \"\":\n options.world = \"cxxtest\"\n options.xunit_file=\"TEST-\"+options.world+\".xml\"\n elif options.xunit_file == \"\":\n if options.world == \"\":\n options.world = \"cxxtest\"\n options.xunit_file=\"TEST-\"+options.world+\".xml\"\n\n if options.error_printer:\n options.runner= \"ErrorPrinter\"\n options.haveStandardLibrary = True\n \n if options.noStaticInit and (options.root or options.part):\n abort( '--no-static-init cannot be used with --root/--part' )\n\n if options.gui and not options.runner:\n options.runner = 'StdioPrinter'\n\n files = setFiles(args[1:])\n if len(files) == 0 and not options.root:\n sys.stderr.write(parser.error(\"No input files found\"))\n\n return files\n\n\ndef printVersion():\n '''Print CxxTest version and exit'''\n sys.stdout.write( \"This is CxxTest version %s.\\n\" % __release__.__version__ )\n sys.exit(0)\n\ndef setFiles(patterns ):\n '''Set input files specified on command line'''\n files = expandWildcards( patterns )\n return files\n\ndef expandWildcards( patterns ):\n '''Expand all wildcards in an array (glob)'''\n fileNames = []\n for pathName in patterns:\n patternFiles = glob.glob( pathName )\n for fileName in patternFiles:\n fileNames.append( fixBackslashes( fileName ) )\n return fileNames\n\ndef fixBackslashes( fileName ):\n '''Convert backslashes to slashes in file name'''\n return re.sub( r'\\\\', '/', fileName, 0 )\n\n\ndef writeOutput():\n '''Create output file'''\n if options.templateFileName:\n writeTemplateOutput()\n else:\n writeSimpleOutput()\n\ndef writeSimpleOutput():\n '''Create output not based on template'''\n output = startOutputFile()\n writePreamble( output )\n if options.root or not options.part:\n writeMain( output )\n\n if len(suites) > 0:\n output.write(\"bool \"+suites[0]['object']+\"_init = false;\\n\")\n\n writeWorld( output )\n output.close()\n\ninclude_re = re.compile( r\"\\s*\\#\\s*include\\s+<cxxtest/\" )\npreamble_re = re.compile( r\"^\\s*<CxxTest\\s+preamble>\\s*$\" )\nworld_re = re.compile( r\"^\\s*<CxxTest\\s+world>\\s*$\" )\ndef writeTemplateOutput():\n '''Create output based on template file'''\n template = open(options.templateFileName)\n output = startOutputFile()\n while 1:\n line = template.readline()\n if not line:\n break;\n if include_re.search( line ):\n writePreamble( output )\n output.write( line )\n elif preamble_re.search( line ):\n writePreamble( output )\n elif world_re.search( line ):\n if len(suites) > 0:\n output.write(\"bool \"+suites[0]['object']+\"_init = false;\\n\")\n writeWorld( output )\n else:\n output.write( line )\n template.close()\n output.close()\n\ndef startOutputFile():\n '''Create output file and write header'''\n if options.outputFileName is not None:\n output = open( options.outputFileName, 'w' )\n else:\n output = sys.stdout\n output.write( \"/* Generated file, do not edit */\\n\\n\" )\n return output\n\ndef writePreamble( output ):\n '''Write the CxxTest header (#includes and #defines)'''\n global wrotePreamble\n if wrotePreamble: return\n output.write( \"#ifndef CXXTEST_RUNNING\\n\" )\n output.write( \"#define CXXTEST_RUNNING\\n\" )\n output.write( \"#endif\\n\" )\n output.write( \"\\n\" )\n if options.xunit_printer:\n output.write( \"#include <fstream>\\n\" )\n if options.haveStandardLibrary:\n output.write( \"#define _CXXTEST_HAVE_STD\\n\" )\n if options.haveExceptionHandling:\n output.write( \"#define _CXXTEST_HAVE_EH\\n\" )\n if options.abortOnFail:\n output.write( \"#define _CXXTEST_ABORT_TEST_ON_FAIL\\n\" )\n if options.longlong:\n output.write( \"#define _CXXTEST_LONGLONG %s\\n\" % options.longlong )\n #if options.factor:\n #output.write( \"#define _CXXTEST_FACTOR\\n\" )\n for header in options.headers:\n output.write( \"#include \\\"%s\\\"\\n\" % header )\n output.write( \"#include <cxxtest/TestListener.h>\\n\" )\n output.write( \"#include <cxxtest/TestTracker.h>\\n\" )\n output.write( \"#include <cxxtest/TestRunner.h>\\n\" )\n output.write( \"#include <cxxtest/RealDescriptions.h>\\n\" )\n output.write( \"#include <cxxtest/TestMain.h>\\n\" )\n if options.runner:\n output.write( \"#include <cxxtest/%s.h>\\n\" % options.runner )\n if options.gui:\n output.write( \"#include <cxxtest/%s.h>\\n\" % options.gui )\n output.write( \"\\n\" )\n wrotePreamble = 1\n\ndef writeMain( output ):\n '''Write the main() function for the test runner'''\n if not (options.gui or options.runner):\n return\n output.write( 'int %s( int argc, char *argv[] ) {\\n' % options.main )\n output.write( ' int status;\\n' )\n if options.noStaticInit:\n output.write( ' CxxTest::initialize();\\n' )\n if options.gui:\n tester_t = \"CxxTest::GuiTuiRunner<CxxTest::%s, CxxTest::%s> \" % (options.gui, options.runner)\n else:\n tester_t = \"CxxTest::%s\" % (options.runner)\n if options.xunit_printer:\n output.write( ' std::ofstream ofstr(\"%s\");\\n' % options.xunit_file )\n output.write( ' %s tmp(ofstr);\\n' % tester_t )\n output.write( ' CxxTest::RealWorldDescription::_worldName = \"%s\";\\n' % options.world )\n else:\n output.write( ' %s tmp;\\n' % tester_t )\n output.write( ' status = CxxTest::Main<%s>( tmp, argc, argv );\\n' % tester_t )\n output.write( ' return status;\\n')\n output.write( '}\\n' )\n\n\ndef writeWorld( output ):\n '''Write the world definitions'''\n global wroteWorld\n if wroteWorld: return\n writePreamble( output )\n writeSuites( output )\n if options.root or not options.part:\n writeRoot( output )\n writeWorldDescr( output )\n if options.noStaticInit:\n writeInitialize( output )\n wroteWorld = 1\n\ndef writeSuites(output):\n '''Write all TestDescriptions and SuiteDescriptions'''\n for suite in suites:\n writeInclude( output, suite['file'] )\n if isGenerated(suite):\n generateSuite( output, suite )\n if isDynamic(suite):\n writeSuitePointer( output, suite )\n else:\n writeSuiteObject( output, suite )\n writeTestList( output, suite )\n writeSuiteDescription( output, suite )\n writeTestDescriptions( output, suite )\n\ndef isGenerated(suite):\n '''Checks whether a suite class should be created'''\n return suite['generated']\n\ndef isDynamic(suite):\n '''Checks whether a suite is dynamic'''\n return 'create' in suite\n\ndef writeInclude(output, file):\n '''Add #include \"file\" statement'''\n global lastIncluded\n if file == lastIncluded: return\n output.writelines( [ '#include \"', file, '\"\\n\\n' ] )\n lastIncluded = file\n\ndef generateSuite( output, suite ):\n '''Write a suite declared with CXXTEST_SUITE()'''\n output.write( 'class %s : public CxxTest::TestSuite {\\n' % suite['name'] )\n output.write( 'public:\\n' )\n for line in suite['lines']:\n output.write(line)\n output.write( '};\\n\\n' )\n\ndef writeSuitePointer( output, suite ):\n '''Create static suite pointer object for dynamic suites'''\n if options.noStaticInit:\n output.write( 'static %s *%s;\\n\\n' % (suite['name'], suite['object']) )\n else:\n output.write( 'static %s *%s = 0;\\n\\n' % (suite['name'], suite['object']) )\n\ndef writeSuiteObject( output, suite ):\n '''Create static suite object for non-dynamic suites'''\n output.writelines( [ \"static \", suite['name'], \" \", suite['object'], \";\\n\\n\" ] )\n\ndef writeTestList( output, suite ):\n '''Write the head of the test linked list for a suite'''\n if options.noStaticInit:\n output.write( 'static CxxTest::List %s;\\n' % suite['tlist'] )\n else:\n output.write( 'static CxxTest::List %s = { 0, 0 };\\n' % suite['tlist'] )\n\ndef writeWorldDescr( output ):\n '''Write the static name of the world name'''\n if options.noStaticInit:\n output.write( 'const char* CxxTest::RealWorldDescription::_worldName;\\n' )\n else:\n output.write( 'const char* CxxTest::RealWorldDescription::_worldName = \"cxxtest\";\\n' )\n\ndef writeTestDescriptions( output, suite ):\n '''Write all test descriptions for a suite'''\n for test in suite['tests']:\n writeTestDescription( output, suite, test )\n\ndef writeTestDescription( output, suite, test ):\n '''Write test description object'''\n output.write( 'static class %s : public CxxTest::RealTestDescription {\\n' % test['class'] )\n output.write( 'public:\\n' )\n if not options.noStaticInit:\n output.write( ' %s() : CxxTest::RealTestDescription( %s, %s, %s, \"%s\" ) {}\\n' %\n (test['class'], suite['tlist'], suite['dobject'], test['line'], test['name']) )\n output.write( ' void runTest() { %s }\\n' % runBody( suite, test ) )\n output.write( '} %s;\\n\\n' % test['object'] )\n\ndef runBody( suite, test ):\n '''Body of TestDescription::run()'''\n if isDynamic(suite): return dynamicRun( suite, test )\n else: return staticRun( suite, test )\n\ndef dynamicRun( suite, test ):\n '''Body of TestDescription::run() for test in a dynamic suite'''\n return 'if ( ' + suite['object'] + ' ) ' + suite['object'] + '->' + test['name'] + '();'\n \ndef staticRun( suite, test ):\n '''Body of TestDescription::run() for test in a non-dynamic suite'''\n return suite['object'] + '.' + test['name'] + '();'\n \ndef writeSuiteDescription( output, suite ):\n '''Write SuiteDescription object'''\n if isDynamic( suite ):\n writeDynamicDescription( output, suite )\n else:\n writeStaticDescription( output, suite )\n\ndef writeDynamicDescription( output, suite ):\n '''Write SuiteDescription for a dynamic suite'''\n output.write( 'CxxTest::DynamicSuiteDescription<%s> %s' % (suite['name'], suite['dobject']) )\n if not options.noStaticInit:\n output.write( '( %s, %s, \"%s\", %s, %s, %s, %s )' %\n (suite['cfile'], suite['line'], suite['name'], suite['tlist'],\n suite['object'], suite['create'], suite['destroy']) )\n output.write( ';\\n\\n' )\n\ndef writeStaticDescription( output, suite ):\n '''Write SuiteDescription for a static suite'''\n output.write( 'CxxTest::StaticSuiteDescription %s' % suite['dobject'] )\n if not options.noStaticInit:\n output.write( '( %s, %s, \"%s\", %s, %s )' %\n (suite['cfile'], suite['line'], suite['name'], suite['object'], suite['tlist']) )\n output.write( ';\\n\\n' )\n\ndef writeRoot(output):\n '''Write static members of CxxTest classes'''\n output.write( '#include <cxxtest/Root.cpp>\\n' )\n\ndef writeInitialize(output):\n '''Write CxxTest::initialize(), which replaces static initialization'''\n output.write( 'namespace CxxTest {\\n' )\n output.write( ' void initialize()\\n' )\n output.write( ' {\\n' )\n for suite in suites:\n output.write( ' %s.initialize();\\n' % suite['tlist'] )\n if isDynamic(suite):\n output.write( ' %s = 0;\\n' % suite['object'] )\n output.write( ' %s.initialize( %s, %s, \"%s\", %s, %s, %s, %s );\\n' %\n (suite['dobject'], suite['cfile'], suite['line'], suite['name'],\n suite['tlist'], suite['object'], suite['create'], suite['destroy']) )\n else:\n output.write( ' %s.initialize( %s, %s, \"%s\", %s, %s );\\n' %\n (suite['dobject'], suite['cfile'], suite['line'], suite['name'],\n suite['object'], suite['tlist']) )\n\n for test in suite['tests']:\n output.write( ' %s.initialize( %s, %s, %s, \"%s\" );\\n' %\n (test['object'], suite['tlist'], suite['dobject'], test['line'], test['name']) )\n\n output.write( ' }\\n' )\n output.write( '}\\n' )\n\n" }, { "path": "cxxtest/python/python3/cxxtest/__init__.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n\"\"\"cxxtest: A Python package that supports the CxxTest test framework for C/C++.\n\n.. _CxxTest: http://cxxtest.tigris.org/\n\nCxxTest is a unit testing framework for C++ that is similar in\nspirit to JUnit, CppUnit, and xUnit. CxxTest is easy to use because\nit does not require precompiling a CxxTest testing library, it\nemploys no advanced features of C++ (e.g. RTTI) and it supports a\nvery flexible form of test discovery.\n\nThe cxxtest Python package includes capabilities for parsing C/C++ source files and generating\nCxxTest drivers.\n\"\"\"\n\nfrom cxxtest.__release__ import __version__, __date__\n__date__\n__version__\n\n__maintainer__ = \"William E. Hart\"\n__maintainer_email__ = \"whart222@gmail.com\"\n__license__ = \"LGPL\"\n__url__ = \"http://cxxtest.tigris.org/\"\n\nfrom cxxtest.cxxtestgen import *\n" }, { "path": "cxxtest/python/python3/cxxtest/__release__.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n\"\"\" Release Information for cxxtest \"\"\"\n\n__version__ = '4.0.2'\n__date__ = \"2012-01-02\"\n" }, { "path": "cxxtest/python/python3/cxxtest/cxx_parser.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n# vim: fileencoding=utf-8\n\n#\n# This is a PLY parser for the entire ANSI C++ grammar. This grammar was \n# adapted from the FOG grammar developed by E. D. Willink. See\n#\n# http://www.computing.surrey.ac.uk/research/dsrg/fog/\n#\n# for further details.\n#\n# The goal of this grammar is to extract information about class, function and\n# class method declarations, along with their associated scope. Thus, this \n# grammar can be used to analyze classes in an inheritance heirarchy, and then\n# enumerate the methods in a derived class.\n#\n# This grammar parses blocks of <>, (), [] and {} in a generic manner. Thus,\n# There are several capabilities that this grammar does not support:\n#\n# 1. Ambiguous template specification. This grammar cannot parse template\n# specifications that do not have paired <>'s in their declaration. In\n# particular, ambiguous declarations like\n#\n# foo<A, c<3 >();\n#\n# cannot be correctly parsed.\n#\n# 2. Template class specialization. Although the goal of this grammar is to\n# extract class information, specialization of templated classes is\n# not supported. When a template class definition is parsed, it's \n# declaration is archived without information about the template\n# parameters. Class specializations will be stored separately, and \n# thus they can be processed after the fact. However, this grammar\n# does not attempt to correctly process properties of class inheritence\n# when template class specialization is employed.\n#\n\n#\n# TODO: document usage of this file\n#\n\n\n\nimport os\nimport ply.lex as lex\nimport ply.yacc as yacc\nimport re\ntry:\n from collections import OrderedDict\nexcept ImportError:\n from ordereddict import OrderedDict\n\nlexer = None\nscope_lineno = 0\nidentifier_lineno = {}\n_parse_info=None\n_parsedata=None\nnoExceptionLogic = True\n\ndef ply_init(data):\n global _parsedata\n _parsedata=data\n\n\nclass Scope(object):\n\n def __init__(self,name,abs_name,scope_t,base_classes,lineno):\n self.function=[]\n self.name=name\n self.scope_t=scope_t\n self.sub_scopes=[]\n self.base_classes=base_classes\n self.abs_name=abs_name\n self.lineno=lineno\n \n def insert(self,scope):\n self.sub_scopes.append(scope)\n\n\nclass CppInfo(object):\n\n def __init__(self, filter=None):\n self.verbose=0\n if filter is None:\n self.filter=re.compile(\"[Tt][Ee][Ss][Tt]|createSuite|destroySuite\")\n else:\n self.filter=filter\n self.scopes=[\"\"]\n self.index=OrderedDict()\n self.index[\"\"]=Scope(\"\",\"::\",\"namespace\",[],1)\n self.function=[]\n\n def push_scope(self,ns,scope_t,base_classes=[]):\n name = self.scopes[-1]+\"::\"+ns\n if self.verbose>=2:\n print(\"-- Starting \"+scope_t+\" \"+name)\n self.scopes.append(name)\n self.index[name] = Scope(ns,name,scope_t,base_classes,scope_lineno-1)\n\n def pop_scope(self):\n scope = self.scopes.pop()\n if self.verbose>=2:\n print(\"-- Stopping \"+scope)\n return scope\n\n def add_function(self, fn):\n fn = str(fn)\n if self.filter.search(fn):\n self.index[self.scopes[-1]].function.append((fn, identifier_lineno.get(fn,lexer.lineno-1)))\n tmp = self.scopes[-1]+\"::\"+fn\n if self.verbose==2:\n print(\"-- Function declaration \"+fn+\" \"+tmp)\n elif self.verbose==1:\n print(\"-- Function declaration \"+tmp)\n\n def get_functions(self,name,quiet=False):\n if name == \"::\":\n name = \"\"\n scope = self.index[name]\n fns=scope.function\n for key in scope.base_classes:\n cname = self.find_class(key,scope)\n if cname is None:\n if not quiet:\n print(\"Defined classes: \",list(self.index.keys()))\n print(\"WARNING: Unknown class \"+key)\n else:\n fns += self.get_functions(cname,quiet)\n return fns\n \n def find_class(self,name,scope):\n if ':' in name:\n if name in self.index:\n return name\n else:\n return None \n tmp = scope.abs_name.split(':')\n name1 = \":\".join(tmp[:-1] + [name])\n if name1 in self.index:\n return name1\n name2 = \"::\"+name\n if name2 in self.index:\n return name2\n return None\n\n def __repr__(self):\n return str(self)\n\n def is_baseclass(self,cls,base):\n '''Returns true if base is a base-class of cls'''\n if cls in self.index:\n bases = self.index[cls]\n elif \"::\"+cls in self.index:\n bases = self.index[\"::\"+cls]\n else:\n return False\n #raise IOError, \"Unknown class \"+cls\n if base in bases.base_classes:\n return True\n for name in bases.base_classes:\n if self.is_baseclass(name,base):\n return True\n return False\n\n def __str__(self):\n ans=\"\"\n keys = list(self.index.keys())\n keys.sort()\n for key in keys:\n scope = self.index[key]\n ans += scope.scope_t+\" \"+scope.abs_name+\"\\n\"\n if scope.scope_t == \"class\":\n ans += \" Base Classes: \"+str(scope.base_classes)+\"\\n\"\n for fn in self.get_functions(scope.abs_name):\n ans += \" \"+fn+\"\\n\"\n else:\n for fn in scope.function:\n ans += \" \"+fn+\"\\n\"\n return ans\n\n\ndef flatten(x):\n \"\"\"Flatten nested list\"\"\"\n try:\n strtypes = str\n except: # for python3 etc\n strtypes = (str, bytes)\n\n result = []\n for el in x:\n if hasattr(el, \"__iter__\") and not isinstance(el, strtypes):\n result.extend(flatten(el))\n else:\n result.append(el)\n return result\n\n#\n# The lexer (and/or a preprocessor) is expected to identify the following\n#\n# Punctuation:\n#\n#\nliterals = \"+-*/%^&|~!<>=:()?.\\'\\\"\\\\@$;,\"\n\n#\nreserved = {\n 'private' : 'PRIVATE',\n 'protected' : 'PROTECTED',\n 'public' : 'PUBLIC',\n\n 'bool' : 'BOOL',\n 'char' : 'CHAR',\n 'double' : 'DOUBLE',\n 'float' : 'FLOAT',\n 'int' : 'INT',\n 'long' : 'LONG',\n 'short' : 'SHORT',\n 'signed' : 'SIGNED',\n 'unsigned' : 'UNSIGNED',\n 'void' : 'VOID',\n 'wchar_t' : 'WCHAR_T',\n\n 'class' : 'CLASS',\n 'enum' : 'ENUM',\n 'namespace' : 'NAMESPACE',\n 'struct' : 'STRUCT',\n 'typename' : 'TYPENAME',\n 'union' : 'UNION',\n\n 'const' : 'CONST',\n 'volatile' : 'VOLATILE',\n\n 'auto' : 'AUTO',\n 'explicit' : 'EXPLICIT',\n 'export' : 'EXPORT',\n 'extern' : 'EXTERN',\n '__extension__' : 'EXTENSION',\n 'friend' : 'FRIEND',\n 'inline' : 'INLINE',\n 'mutable' : 'MUTABLE',\n 'register' : 'REGISTER',\n 'static' : 'STATIC',\n 'template' : 'TEMPLATE',\n 'typedef' : 'TYPEDEF',\n 'using' : 'USING',\n 'virtual' : 'VIRTUAL',\n\n 'asm' : 'ASM',\n 'break' : 'BREAK',\n 'case' : 'CASE',\n 'catch' : 'CATCH',\n 'const_cast' : 'CONST_CAST',\n 'continue' : 'CONTINUE',\n 'default' : 'DEFAULT',\n 'delete' : 'DELETE',\n 'do' : 'DO',\n 'dynamic_cast' : 'DYNAMIC_CAST',\n 'else' : 'ELSE',\n 'false' : 'FALSE',\n 'for' : 'FOR',\n 'goto' : 'GOTO',\n 'if' : 'IF',\n 'new' : 'NEW',\n 'operator' : 'OPERATOR',\n 'reinterpret_cast' : 'REINTERPRET_CAST',\n 'return' : 'RETURN',\n 'sizeof' : 'SIZEOF',\n 'static_cast' : 'STATIC_CAST',\n 'switch' : 'SWITCH',\n 'this' : 'THIS',\n 'throw' : 'THROW',\n 'true' : 'TRUE',\n 'try' : 'TRY',\n 'typeid' : 'TYPEID',\n 'while' : 'WHILE',\n '\"C\"' : 'CLiteral',\n '\"C++\"' : 'CppLiteral',\n\n '__attribute__' : 'ATTRIBUTE',\n '__cdecl__' : 'CDECL',\n '__typeof' : 'uTYPEOF',\n 'typeof' : 'TYPEOF', \n\n 'CXXTEST_STD' : 'CXXTEST_STD'\n}\n \ntokens = [\n \"CharacterLiteral\",\n \"FloatingLiteral\",\n \"Identifier\",\n \"IntegerLiteral\",\n \"StringLiteral\",\n \"RBRACE\",\n \"LBRACE\",\n \"RBRACKET\",\n \"LBRACKET\",\n \"ARROW\",\n \"ARROW_STAR\",\n \"DEC\",\n \"EQ\",\n \"GE\",\n \"INC\",\n \"LE\",\n \"LOG_AND\",\n \"LOG_OR\",\n \"NE\",\n \"SHL\",\n \"SHR\",\n \"ASS_ADD\",\n \"ASS_AND\",\n \"ASS_DIV\",\n \"ASS_MOD\",\n \"ASS_MUL\",\n \"ASS_OR\",\n \"ASS_SHL\",\n \"ASS_SHR\",\n \"ASS_SUB\",\n \"ASS_XOR\",\n \"DOT_STAR\",\n \"ELLIPSIS\",\n \"SCOPE\",\n] + list(reserved.values())\n\nt_ignore = \" \\t\\r\"\n\nt_LBRACE = r\"(\\{)|(<%)\"\nt_RBRACE = r\"(\\})|(%>)\"\nt_LBRACKET = r\"(\\[)|(<:)\"\nt_RBRACKET = r\"(\\])|(:>)\"\nt_ARROW = r\"->\"\nt_ARROW_STAR = r\"->\\*\"\nt_DEC = r\"--\"\nt_EQ = r\"==\"\nt_GE = r\">=\"\nt_INC = r\"\\+\\+\"\nt_LE = r\"<=\"\nt_LOG_AND = r\"&&\"\nt_LOG_OR = r\"\\|\\|\"\nt_NE = r\"!=\"\nt_SHL = r\"<<\"\nt_SHR = r\">>\"\nt_ASS_ADD = r\"\\+=\"\nt_ASS_AND = r\"&=\"\nt_ASS_DIV = r\"/=\"\nt_ASS_MOD = r\"%=\"\nt_ASS_MUL = r\"\\*=\"\nt_ASS_OR = r\"\\|=\"\nt_ASS_SHL = r\"<<=\"\nt_ASS_SHR = r\">>=\"\nt_ASS_SUB = r\"-=\"\nt_ASS_XOR = r\"^=\"\nt_DOT_STAR = r\"\\.\\*\"\nt_ELLIPSIS = r\"\\.\\.\\.\"\nt_SCOPE = r\"::\"\n\n# Discard comments\ndef t_COMMENT(t):\n r'(/\\*(.|\\n)*?\\*/)|(//.*?\\n)|(\\#.*?\\n)'\n t.lexer.lineno += t.value.count(\"\\n\")\n\nt_IntegerLiteral = r'(0x[0-9A-F]+)|([0-9]+(L){0,1})'\nt_FloatingLiteral = r\"[0-9]+[eE\\.\\+-]+[eE\\.\\+\\-0-9]+\"\nt_CharacterLiteral = r'\\'([^\\'\\\\]|\\\\.)*\\''\n#t_StringLiteral = r'\"([^\"\\\\]|\\\\.)*\"'\ndef t_StringLiteral(t):\n r'\"([^\"\\\\]|\\\\.)*\"'\n t.type = reserved.get(t.value,'StringLiteral')\n return t\n\ndef t_Identifier(t):\n r\"[a-zA-Z_][a-zA-Z_0-9\\.]*\"\n t.type = reserved.get(t.value,'Identifier')\n return t\n\n\ndef t_error(t):\n print(\"Illegal character '%s'\" % t.value[0])\n #raise IOError, \"Parse error\"\n #t.lexer.skip()\n\ndef t_newline(t):\n r'[\\n]+'\n t.lexer.lineno += len(t.value)\n\nprecedence = (\n ( 'right', 'SHIFT_THERE', 'REDUCE_HERE_MOSTLY', 'SCOPE'),\n ( 'nonassoc', 'ELSE', 'INC', 'DEC', '+', '-', '*', '&', 'LBRACKET', 'LBRACE', '<', ':', ')')\n )\n\nstart = 'translation_unit'\n\n#\n# The %prec resolves the 14.2-3 ambiguity:\n# Identifier '<' is forced to go through the is-it-a-template-name test\n# All names absorb TEMPLATE with the name, so that no template_test is \n# performed for them. This requires all potential declarations within an \n# expression to perpetuate this policy and thereby guarantee the ultimate \n# coverage of explicit_instantiation.\n#\n# The %prec also resolves a conflict in identifier : which is forced to be a \n# shift of a label for a labeled-statement rather than a reduction for the \n# name of a bit-field or generalised constructor. This is pretty dubious \n# syntactically but correct for all semantic possibilities. The shift is \n# only activated when the ambiguity exists at the start of a statement. \n# In this context a bit-field declaration or constructor definition are not \n# allowed.\n#\n\ndef p_identifier(p):\n '''identifier : Identifier\n | CXXTEST_STD '(' Identifier ')'\n '''\n if p[1][0] in ('t','T','c','d'):\n identifier_lineno[p[1]] = p.lineno(1)\n p[0] = p[1]\n\ndef p_id(p):\n '''id : identifier %prec SHIFT_THERE\n | template_decl\n | TEMPLATE id\n '''\n p[0] = get_rest(p)\n\ndef p_global_scope(p):\n '''global_scope : SCOPE\n '''\n p[0] = get_rest(p)\n\ndef p_id_scope(p):\n '''id_scope : id SCOPE'''\n p[0] = get_rest(p)\n\ndef p_id_scope_seq(p):\n '''id_scope_seq : id_scope\n | id_scope id_scope_seq\n '''\n p[0] = get_rest(p)\n\n#\n# A :: B :: C; is ambiguous How much is type and how much name ?\n# The %prec maximises the (type) length which is the 7.1-2 semantic constraint.\n#\ndef p_nested_id(p):\n '''nested_id : id %prec SHIFT_THERE\n | id_scope nested_id\n '''\n p[0] = get_rest(p)\n\ndef p_scoped_id(p):\n '''scoped_id : nested_id\n | global_scope nested_id\n | id_scope_seq\n | global_scope id_scope_seq\n '''\n global scope_lineno\n scope_lineno = lexer.lineno\n data = flatten(get_rest(p))\n if data[0] != None:\n p[0] = \"\".join(data)\n\n#\n# destructor_id has to be held back to avoid a conflict with a one's \n# complement as per 5.3.1-9, It gets put back only when scoped or in a \n# declarator_id, which is only used as an explicit member name.\n# Declarations of an unscoped destructor are always parsed as a one's \n# complement.\n#\ndef p_destructor_id(p):\n '''destructor_id : '~' id\n | TEMPLATE destructor_id\n '''\n p[0]=get_rest(p)\n\n#def p_template_id(p):\n# '''template_id : empty\n# | TEMPLATE\n# '''\n# pass\n\ndef p_template_decl(p):\n '''template_decl : identifier '<' nonlgt_seq_opt '>'\n '''\n #\n # WEH: should we include the lt/gt symbols to indicate that this is a\n # template class? How is that going to be used later???\n #\n #p[0] = [p[1] ,\"<\",\">\"]\n p[0] = p[1]\n\ndef p_special_function_id(p):\n '''special_function_id : conversion_function_id\n | operator_function_id\n | TEMPLATE special_function_id\n '''\n p[0]=get_rest(p)\n\ndef p_nested_special_function_id(p):\n '''nested_special_function_id : special_function_id\n | id_scope destructor_id\n | id_scope nested_special_function_id\n '''\n p[0]=get_rest(p)\n\ndef p_scoped_special_function_id(p):\n '''scoped_special_function_id : nested_special_function_id\n | global_scope nested_special_function_id\n '''\n p[0]=get_rest(p)\n\n# declarator-id is all names in all scopes, except reserved words\ndef p_declarator_id(p):\n '''declarator_id : scoped_id\n | scoped_special_function_id\n | destructor_id\n '''\n p[0]=p[1]\n\n#\n# The standard defines pseudo-destructors in terms of type-name, which is \n# class/enum/typedef, of which class-name is covered by a normal destructor. \n# pseudo-destructors are supposed to support ~int() in templates, so the \n# grammar here covers built-in names. Other names are covered by the lack \n# of identifier/type discrimination.\n#\ndef p_built_in_type_id(p):\n '''built_in_type_id : built_in_type_specifier\n | built_in_type_id built_in_type_specifier\n '''\n pass\n\ndef p_pseudo_destructor_id(p):\n '''pseudo_destructor_id : built_in_type_id SCOPE '~' built_in_type_id\n | '~' built_in_type_id\n | TEMPLATE pseudo_destructor_id\n '''\n pass\n\ndef p_nested_pseudo_destructor_id(p):\n '''nested_pseudo_destructor_id : pseudo_destructor_id\n | id_scope nested_pseudo_destructor_id\n '''\n pass\n\ndef p_scoped_pseudo_destructor_id(p):\n '''scoped_pseudo_destructor_id : nested_pseudo_destructor_id\n | global_scope scoped_pseudo_destructor_id\n '''\n pass\n\n#-------------------------------------------------------------------------------\n# A.2 Lexical conventions\n#-------------------------------------------------------------------------------\n#\n\ndef p_literal(p):\n '''literal : IntegerLiteral\n | CharacterLiteral\n | FloatingLiteral\n | StringLiteral\n | TRUE\n | FALSE\n '''\n pass\n\n#-------------------------------------------------------------------------------\n# A.3 Basic concepts\n#-------------------------------------------------------------------------------\ndef p_translation_unit(p):\n '''translation_unit : declaration_seq_opt\n '''\n pass\n\n#-------------------------------------------------------------------------------\n# A.4 Expressions\n#-------------------------------------------------------------------------------\n#\n# primary_expression covers an arbitrary sequence of all names with the \n# exception of an unscoped destructor, which is parsed as its unary expression \n# which is the correct disambiguation (when ambiguous). This eliminates the \n# traditional A(B) meaning A B ambiguity, since we never have to tack an A \n# onto the front of something that might start with (. The name length got \n# maximised ab initio. The downside is that semantic interpretation must split \n# the names up again.\n#\n# Unification of the declaration and expression syntax means that unary and \n# binary pointer declarator operators:\n# int * * name\n# are parsed as binary and unary arithmetic operators (int) * (*name). Since \n# type information is not used\n# ambiguities resulting from a cast\n# (cast)*(value)\n# are resolved to favour the binary rather than the cast unary to ease AST \n# clean-up. The cast-call ambiguity must be resolved to the cast to ensure \n# that (a)(b)c can be parsed.\n#\n# The problem of the functional cast ambiguity\n# name(arg)\n# as call or declaration is avoided by maximising the name within the parsing \n# kernel. So primary_id_expression picks up \n# extern long int const var = 5;\n# as an assignment to the syntax parsed as \"extern long int const var\". The \n# presence of two names is parsed so that \"extern long into const\" is \n# distinguished from \"var\" considerably simplifying subsequent \n# semantic resolution.\n#\n# The generalised name is a concatenation of potential type-names (scoped \n# identifiers or built-in sequences) plus optionally one of the special names \n# such as an operator-function-id, conversion-function-id or destructor as the \n# final name. \n#\n\ndef get_rest(p):\n return [p[i] for i in range(1, len(p))]\n\ndef p_primary_expression(p):\n '''primary_expression : literal\n | THIS\n | suffix_decl_specified_ids\n | abstract_expression %prec REDUCE_HERE_MOSTLY\n '''\n p[0] = get_rest(p)\n\n#\n# Abstract-expression covers the () and [] of abstract-declarators.\n#\ndef p_abstract_expression(p):\n '''abstract_expression : parenthesis_clause\n | LBRACKET bexpression_opt RBRACKET\n | TEMPLATE abstract_expression\n '''\n pass\n\ndef p_postfix_expression(p):\n '''postfix_expression : primary_expression\n | postfix_expression parenthesis_clause\n | postfix_expression LBRACKET bexpression_opt RBRACKET\n | postfix_expression LBRACKET bexpression_opt RBRACKET attributes\n | postfix_expression '.' declarator_id\n | postfix_expression '.' scoped_pseudo_destructor_id\n | postfix_expression ARROW declarator_id\n | postfix_expression ARROW scoped_pseudo_destructor_id \n | postfix_expression INC\n | postfix_expression DEC\n | DYNAMIC_CAST '<' nonlgt_seq_opt '>' '(' expression ')'\n | STATIC_CAST '<' nonlgt_seq_opt '>' '(' expression ')'\n | REINTERPRET_CAST '<' nonlgt_seq_opt '>' '(' expression ')'\n | CONST_CAST '<' nonlgt_seq_opt '>' '(' expression ')'\n | TYPEID parameters_clause\n '''\n #print \"HERE\",str(p[1])\n p[0] = get_rest(p)\n\ndef p_bexpression_opt(p):\n '''bexpression_opt : empty\n | bexpression\n '''\n pass\n\ndef p_bexpression(p):\n '''bexpression : nonbracket_seq\n | nonbracket_seq bexpression_seq bexpression_clause nonbracket_seq_opt\n | bexpression_seq bexpression_clause nonbracket_seq_opt\n '''\n pass\n\ndef p_bexpression_seq(p):\n '''bexpression_seq : empty\n | bexpression_seq bexpression_clause nonbracket_seq_opt\n '''\n pass\n\ndef p_bexpression_clause(p):\n '''bexpression_clause : LBRACKET bexpression_opt RBRACKET\n '''\n pass\n\n\n\ndef p_expression_list_opt(p):\n '''expression_list_opt : empty\n | expression_list\n '''\n pass\n\ndef p_expression_list(p):\n '''expression_list : assignment_expression\n | expression_list ',' assignment_expression\n '''\n pass\n\ndef p_unary_expression(p):\n '''unary_expression : postfix_expression\n | INC cast_expression\n | DEC cast_expression\n | ptr_operator cast_expression\n | suffix_decl_specified_scope star_ptr_operator cast_expression\n | '+' cast_expression\n | '-' cast_expression\n | '!' cast_expression\n | '~' cast_expression\n | SIZEOF unary_expression\n | new_expression\n | global_scope new_expression\n | delete_expression\n | global_scope delete_expression\n '''\n p[0] = get_rest(p)\n\ndef p_delete_expression(p):\n '''delete_expression : DELETE cast_expression\n '''\n pass\n\ndef p_new_expression(p):\n '''new_expression : NEW new_type_id new_initializer_opt\n | NEW parameters_clause new_type_id new_initializer_opt\n | NEW parameters_clause\n | NEW parameters_clause parameters_clause new_initializer_opt\n '''\n pass\n\ndef p_new_type_id(p):\n '''new_type_id : type_specifier ptr_operator_seq_opt\n | type_specifier new_declarator\n | type_specifier new_type_id\n '''\n pass\n\ndef p_new_declarator(p):\n '''new_declarator : ptr_operator new_declarator\n | direct_new_declarator\n '''\n pass\n\ndef p_direct_new_declarator(p):\n '''direct_new_declarator : LBRACKET bexpression_opt RBRACKET\n | direct_new_declarator LBRACKET bexpression RBRACKET\n '''\n pass\n\ndef p_new_initializer_opt(p):\n '''new_initializer_opt : empty\n | '(' expression_list_opt ')'\n '''\n pass\n\n#\n# cast-expression is generalised to support a [] as well as a () prefix. This covers the omission of \n# DELETE[] which when followed by a parenthesised expression was ambiguous. It also covers the gcc \n# indexed array initialisation for free.\n#\ndef p_cast_expression(p):\n '''cast_expression : unary_expression\n | abstract_expression cast_expression\n '''\n p[0] = get_rest(p)\n\ndef p_pm_expression(p):\n '''pm_expression : cast_expression\n | pm_expression DOT_STAR cast_expression\n | pm_expression ARROW_STAR cast_expression\n '''\n p[0] = get_rest(p)\n\ndef p_multiplicative_expression(p):\n '''multiplicative_expression : pm_expression\n | multiplicative_expression star_ptr_operator pm_expression\n | multiplicative_expression '/' pm_expression\n | multiplicative_expression '%' pm_expression\n '''\n p[0] = get_rest(p)\n\ndef p_additive_expression(p):\n '''additive_expression : multiplicative_expression\n | additive_expression '+' multiplicative_expression\n | additive_expression '-' multiplicative_expression\n '''\n p[0] = get_rest(p)\n\ndef p_shift_expression(p):\n '''shift_expression : additive_expression\n | shift_expression SHL additive_expression\n | shift_expression SHR additive_expression\n '''\n p[0] = get_rest(p)\n\n# | relational_expression '<' shift_expression\n# | relational_expression '>' shift_expression\n# | relational_expression LE shift_expression\n# | relational_expression GE shift_expression\ndef p_relational_expression(p):\n '''relational_expression : shift_expression\n '''\n p[0] = get_rest(p)\n\ndef p_equality_expression(p):\n '''equality_expression : relational_expression\n | equality_expression EQ relational_expression\n | equality_expression NE relational_expression\n '''\n p[0] = get_rest(p)\n\ndef p_and_expression(p):\n '''and_expression : equality_expression\n | and_expression '&' equality_expression\n '''\n p[0] = get_rest(p)\n\ndef p_exclusive_or_expression(p):\n '''exclusive_or_expression : and_expression\n | exclusive_or_expression '^' and_expression\n '''\n p[0] = get_rest(p)\n\ndef p_inclusive_or_expression(p):\n '''inclusive_or_expression : exclusive_or_expression\n | inclusive_or_expression '|' exclusive_or_expression\n '''\n p[0] = get_rest(p)\n\ndef p_logical_and_expression(p):\n '''logical_and_expression : inclusive_or_expression\n | logical_and_expression LOG_AND inclusive_or_expression\n '''\n p[0] = get_rest(p)\n\ndef p_logical_or_expression(p):\n '''logical_or_expression : logical_and_expression\n | logical_or_expression LOG_OR logical_and_expression\n '''\n p[0] = get_rest(p)\n\ndef p_conditional_expression(p):\n '''conditional_expression : logical_or_expression\n | logical_or_expression '?' expression ':' assignment_expression\n '''\n p[0] = get_rest(p)\n\n\n#\n# assignment-expression is generalised to cover the simple assignment of a braced initializer in order to \n# contribute to the coverage of parameter-declaration and init-declaration.\n#\n# | logical_or_expression assignment_operator assignment_expression\ndef p_assignment_expression(p):\n '''assignment_expression : conditional_expression\n | logical_or_expression assignment_operator nonsemicolon_seq\n | logical_or_expression '=' braced_initializer\n | throw_expression\n '''\n p[0]=get_rest(p)\n\ndef p_assignment_operator(p):\n '''assignment_operator : '=' \n | ASS_ADD\n | ASS_AND\n | ASS_DIV\n | ASS_MOD\n | ASS_MUL\n | ASS_OR\n | ASS_SHL\n | ASS_SHR\n | ASS_SUB\n | ASS_XOR\n '''\n pass\n\n#\n# expression is widely used and usually single-element, so the reductions are arranged so that a\n# single-element expression is returned as is. Multi-element expressions are parsed as a list that\n# may then behave polymorphically as an element or be compacted to an element.\n#\n\ndef p_expression(p):\n '''expression : assignment_expression\n | expression_list ',' assignment_expression\n '''\n p[0] = get_rest(p)\n\ndef p_constant_expression(p):\n '''constant_expression : conditional_expression\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.5 Statements\n#---------------------------------------------------------------------------------------------------\n# Parsing statements is easy once simple_declaration has been generalised to cover expression_statement.\n#\n#\n# The use of extern here is a hack. The 'extern \"C\" {}' block gets parsed\n# as a function, so when nested 'extern \"C\"' declarations exist, they don't\n# work because the block is viewed as a list of statements... :(\n#\ndef p_statement(p):\n '''statement : compound_statement\n | declaration_statement\n | try_block\n | labeled_statement\n | selection_statement\n | iteration_statement\n | jump_statement\n '''\n pass\n\ndef p_compound_statement(p):\n '''compound_statement : LBRACE statement_seq_opt RBRACE\n '''\n pass\n\ndef p_statement_seq_opt(p):\n '''statement_seq_opt : empty\n | statement_seq_opt statement\n '''\n pass\n\n#\n# The dangling else conflict is resolved to the innermost if.\n#\ndef p_selection_statement(p):\n '''selection_statement : IF '(' condition ')' statement %prec SHIFT_THERE\n | IF '(' condition ')' statement ELSE statement\n | SWITCH '(' condition ')' statement\n '''\n pass\n\ndef p_condition_opt(p):\n '''condition_opt : empty\n | condition\n '''\n pass\n\ndef p_condition(p):\n '''condition : nonparen_seq\n | nonparen_seq condition_seq parameters_clause nonparen_seq_opt\n | condition_seq parameters_clause nonparen_seq_opt\n '''\n pass\n\ndef p_condition_seq(p):\n '''condition_seq : empty\n | condition_seq parameters_clause nonparen_seq_opt\n '''\n pass\n\ndef p_labeled_statement(p):\n '''labeled_statement : identifier ':' statement\n | CASE constant_expression ':' statement\n | DEFAULT ':' statement\n '''\n pass\n\ndef p_try_block(p):\n '''try_block : TRY compound_statement handler_seq\n '''\n global noExceptionLogic\n noExceptionLogic=False\n\ndef p_jump_statement(p):\n '''jump_statement : BREAK ';'\n | CONTINUE ';'\n | RETURN nonsemicolon_seq ';'\n | GOTO identifier ';'\n '''\n pass\n\ndef p_iteration_statement(p):\n '''iteration_statement : WHILE '(' condition ')' statement\n | DO statement WHILE '(' expression ')' ';'\n | FOR '(' nonparen_seq_opt ')' statement\n '''\n pass\n\ndef p_declaration_statement(p):\n '''declaration_statement : block_declaration\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.6 Declarations\n#---------------------------------------------------------------------------------------------------\ndef p_compound_declaration(p):\n '''compound_declaration : LBRACE declaration_seq_opt RBRACE \n '''\n pass\n\ndef p_declaration_seq_opt(p):\n '''declaration_seq_opt : empty\n | declaration_seq_opt declaration\n '''\n pass\n\ndef p_declaration(p):\n '''declaration : block_declaration\n | function_definition\n | template_declaration\n | explicit_specialization\n | specialised_declaration\n '''\n pass\n\ndef p_specialised_declaration(p):\n '''specialised_declaration : linkage_specification\n | namespace_definition\n | TEMPLATE specialised_declaration\n '''\n pass\n\ndef p_block_declaration(p):\n '''block_declaration : simple_declaration\n | specialised_block_declaration\n '''\n pass\n\ndef p_specialised_block_declaration(p):\n '''specialised_block_declaration : asm_definition\n | namespace_alias_definition\n | using_declaration\n | using_directive\n | TEMPLATE specialised_block_declaration\n '''\n pass\n\ndef p_simple_declaration(p):\n '''simple_declaration : ';'\n | init_declaration ';'\n | init_declarations ';'\n | decl_specifier_prefix simple_declaration\n '''\n global _parse_info\n if len(p) == 3:\n if p[2] == \";\":\n decl = p[1]\n else:\n decl = p[2]\n if decl is not None:\n fp = flatten(decl)\n if len(fp) >= 2 and fp[0] is not None and fp[0]!=\"operator\" and fp[1] == '(':\n p[0] = fp[0]\n _parse_info.add_function(fp[0])\n\n#\n# A decl-specifier following a ptr_operator provokes a shift-reduce conflict for * const name which is resolved in favour of the pointer, and implemented by providing versions of decl-specifier guaranteed not to start with a cv_qualifier. decl-specifiers are implemented type-centrically. That is the semantic constraint that there must be a type is exploited to impose structure, but actually eliminate very little syntax. built-in types are multi-name and so need a different policy.\n#\n# non-type decl-specifiers are bound to the left-most type in a decl-specifier-seq, by parsing from the right and attaching suffixes to the right-hand type. Finally residual prefixes attach to the left. \n#\ndef p_suffix_built_in_decl_specifier_raw(p):\n '''suffix_built_in_decl_specifier_raw : built_in_type_specifier\n | suffix_built_in_decl_specifier_raw built_in_type_specifier\n | suffix_built_in_decl_specifier_raw decl_specifier_suffix\n '''\n pass\n\ndef p_suffix_built_in_decl_specifier(p):\n '''suffix_built_in_decl_specifier : suffix_built_in_decl_specifier_raw\n | TEMPLATE suffix_built_in_decl_specifier\n '''\n pass\n\n# | id_scope_seq\n# | SCOPE id_scope_seq\ndef p_suffix_named_decl_specifier(p):\n '''suffix_named_decl_specifier : scoped_id \n | elaborate_type_specifier \n | suffix_named_decl_specifier decl_specifier_suffix\n '''\n p[0]=get_rest(p)\n\ndef p_suffix_named_decl_specifier_bi(p):\n '''suffix_named_decl_specifier_bi : suffix_named_decl_specifier\n | suffix_named_decl_specifier suffix_built_in_decl_specifier_raw\n '''\n p[0] = get_rest(p)\n #print \"HERE\",get_rest(p)\n\ndef p_suffix_named_decl_specifiers(p):\n '''suffix_named_decl_specifiers : suffix_named_decl_specifier_bi\n | suffix_named_decl_specifiers suffix_named_decl_specifier_bi\n '''\n p[0] = get_rest(p)\n\ndef p_suffix_named_decl_specifiers_sf(p):\n '''suffix_named_decl_specifiers_sf : scoped_special_function_id\n | suffix_named_decl_specifiers\n | suffix_named_decl_specifiers scoped_special_function_id\n '''\n #print \"HERE\",get_rest(p)\n p[0] = get_rest(p)\n\ndef p_suffix_decl_specified_ids(p):\n '''suffix_decl_specified_ids : suffix_built_in_decl_specifier\n | suffix_built_in_decl_specifier suffix_named_decl_specifiers_sf\n | suffix_named_decl_specifiers_sf\n '''\n if len(p) == 3:\n p[0] = p[2]\n else:\n p[0] = p[1]\n\ndef p_suffix_decl_specified_scope(p):\n '''suffix_decl_specified_scope : suffix_named_decl_specifiers SCOPE\n | suffix_built_in_decl_specifier suffix_named_decl_specifiers SCOPE\n | suffix_built_in_decl_specifier SCOPE\n '''\n p[0] = get_rest(p)\n\ndef p_decl_specifier_affix(p):\n '''decl_specifier_affix : storage_class_specifier\n | function_specifier\n | FRIEND\n | TYPEDEF\n | cv_qualifier\n '''\n pass\n\ndef p_decl_specifier_suffix(p):\n '''decl_specifier_suffix : decl_specifier_affix\n '''\n pass\n\ndef p_decl_specifier_prefix(p):\n '''decl_specifier_prefix : decl_specifier_affix\n | TEMPLATE decl_specifier_prefix\n '''\n pass\n\ndef p_storage_class_specifier(p):\n '''storage_class_specifier : REGISTER \n | STATIC \n | MUTABLE\n | EXTERN %prec SHIFT_THERE\n | EXTENSION\n | AUTO\n '''\n pass\n\ndef p_function_specifier(p):\n '''function_specifier : EXPLICIT\n | INLINE\n | VIRTUAL\n '''\n pass\n\ndef p_type_specifier(p):\n '''type_specifier : simple_type_specifier\n | elaborate_type_specifier\n | cv_qualifier\n '''\n pass\n\ndef p_elaborate_type_specifier(p):\n '''elaborate_type_specifier : class_specifier\n | enum_specifier\n | elaborated_type_specifier\n | TEMPLATE elaborate_type_specifier\n '''\n pass\n\ndef p_simple_type_specifier(p):\n '''simple_type_specifier : scoped_id\n | scoped_id attributes\n | built_in_type_specifier\n '''\n p[0] = p[1]\n\ndef p_built_in_type_specifier(p):\n '''built_in_type_specifier : Xbuilt_in_type_specifier\n | Xbuilt_in_type_specifier attributes\n '''\n pass\n\ndef p_attributes(p):\n '''attributes : attribute\n | attributes attribute\n '''\n pass\n\ndef p_attribute(p):\n '''attribute : ATTRIBUTE '(' parameters_clause ')'\n '''\n\ndef p_Xbuilt_in_type_specifier(p):\n '''Xbuilt_in_type_specifier : CHAR \n | WCHAR_T \n | BOOL \n | SHORT \n | INT \n | LONG \n | SIGNED \n | UNSIGNED \n | FLOAT \n | DOUBLE \n | VOID\n | uTYPEOF parameters_clause\n | TYPEOF parameters_clause\n '''\n pass\n\n#\n# The over-general use of declaration_expression to cover decl-specifier-seq_opt declarator in a function-definition means that\n# class X { };\n# could be a function-definition or a class-specifier.\n# enum X { };\n# could be a function-definition or an enum-specifier.\n# The function-definition is not syntactically valid so resolving the false conflict in favour of the\n# elaborated_type_specifier is correct.\n#\ndef p_elaborated_type_specifier(p):\n '''elaborated_type_specifier : class_key scoped_id %prec SHIFT_THERE\n | elaborated_enum_specifier\n | TYPENAME scoped_id\n '''\n pass\n\ndef p_elaborated_enum_specifier(p):\n '''elaborated_enum_specifier : ENUM scoped_id %prec SHIFT_THERE\n '''\n pass\n\ndef p_enum_specifier(p):\n '''enum_specifier : ENUM scoped_id enumerator_clause\n | ENUM enumerator_clause\n '''\n pass\n\ndef p_enumerator_clause(p):\n '''enumerator_clause : LBRACE enumerator_list_ecarb\n | LBRACE enumerator_list enumerator_list_ecarb\n | LBRACE enumerator_list ',' enumerator_definition_ecarb\n '''\n pass\n\ndef p_enumerator_list_ecarb(p):\n '''enumerator_list_ecarb : RBRACE\n '''\n pass\n\ndef p_enumerator_definition_ecarb(p):\n '''enumerator_definition_ecarb : RBRACE\n '''\n pass\n\ndef p_enumerator_definition_filler(p):\n '''enumerator_definition_filler : empty\n '''\n pass\n\ndef p_enumerator_list_head(p):\n '''enumerator_list_head : enumerator_definition_filler\n | enumerator_list ',' enumerator_definition_filler\n '''\n pass\n\ndef p_enumerator_list(p):\n '''enumerator_list : enumerator_list_head enumerator_definition\n '''\n pass\n\ndef p_enumerator_definition(p):\n '''enumerator_definition : enumerator\n | enumerator '=' constant_expression\n '''\n pass\n\ndef p_enumerator(p):\n '''enumerator : identifier\n '''\n pass\n\ndef p_namespace_definition(p):\n '''namespace_definition : NAMESPACE scoped_id push_scope compound_declaration\n | NAMESPACE push_scope compound_declaration\n '''\n global _parse_info\n scope = _parse_info.pop_scope()\n\ndef p_namespace_alias_definition(p):\n '''namespace_alias_definition : NAMESPACE scoped_id '=' scoped_id ';'\n '''\n pass\n\ndef p_push_scope(p):\n '''push_scope : empty'''\n global _parse_info\n if p[-2] == \"namespace\":\n scope=p[-1]\n else:\n scope=\"\"\n _parse_info.push_scope(scope,\"namespace\")\n\ndef p_using_declaration(p):\n '''using_declaration : USING declarator_id ';'\n | USING TYPENAME declarator_id ';'\n '''\n pass\n\ndef p_using_directive(p):\n '''using_directive : USING NAMESPACE scoped_id ';'\n '''\n pass\n\n# '''asm_definition : ASM '(' StringLiteral ')' ';'\ndef p_asm_definition(p):\n '''asm_definition : ASM '(' nonparen_seq_opt ')' ';'\n '''\n pass\n\ndef p_linkage_specification(p):\n '''linkage_specification : EXTERN CLiteral declaration\n | EXTERN CLiteral compound_declaration\n | EXTERN CppLiteral declaration\n | EXTERN CppLiteral compound_declaration\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.7 Declarators\n#---------------------------------------------------------------------------------------------------\n#\n# init-declarator is named init_declaration to reflect the embedded decl-specifier-seq_opt\n#\n\ndef p_init_declarations(p):\n '''init_declarations : assignment_expression ',' init_declaration\n | init_declarations ',' init_declaration\n '''\n p[0]=get_rest(p)\n\ndef p_init_declaration(p):\n '''init_declaration : assignment_expression\n '''\n p[0]=get_rest(p)\n\ndef p_star_ptr_operator(p):\n '''star_ptr_operator : '*'\n | star_ptr_operator cv_qualifier\n '''\n pass\n\ndef p_nested_ptr_operator(p):\n '''nested_ptr_operator : star_ptr_operator\n | id_scope nested_ptr_operator\n '''\n pass\n\ndef p_ptr_operator(p):\n '''ptr_operator : '&'\n | nested_ptr_operator\n | global_scope nested_ptr_operator\n '''\n pass\n\ndef p_ptr_operator_seq(p):\n '''ptr_operator_seq : ptr_operator\n | ptr_operator ptr_operator_seq\n '''\n pass\n\n#\n# Independently coded to localise the shift-reduce conflict: sharing just needs another %prec\n#\ndef p_ptr_operator_seq_opt(p):\n '''ptr_operator_seq_opt : empty %prec SHIFT_THERE\n | ptr_operator ptr_operator_seq_opt\n '''\n pass\n\ndef p_cv_qualifier_seq_opt(p):\n '''cv_qualifier_seq_opt : empty\n | cv_qualifier_seq_opt cv_qualifier\n '''\n pass\n\n# TODO: verify that we should include attributes here\ndef p_cv_qualifier(p):\n '''cv_qualifier : CONST \n | VOLATILE\n | attributes\n '''\n pass\n\ndef p_type_id(p):\n '''type_id : type_specifier abstract_declarator_opt\n | type_specifier type_id\n '''\n pass\n\ndef p_abstract_declarator_opt(p):\n '''abstract_declarator_opt : empty\n | ptr_operator abstract_declarator_opt\n | direct_abstract_declarator\n '''\n pass\n\ndef p_direct_abstract_declarator_opt(p):\n '''direct_abstract_declarator_opt : empty\n | direct_abstract_declarator\n '''\n pass\n\ndef p_direct_abstract_declarator(p):\n '''direct_abstract_declarator : direct_abstract_declarator_opt parenthesis_clause\n | direct_abstract_declarator_opt LBRACKET RBRACKET\n | direct_abstract_declarator_opt LBRACKET bexpression RBRACKET\n '''\n pass\n\ndef p_parenthesis_clause(p):\n '''parenthesis_clause : parameters_clause cv_qualifier_seq_opt\n | parameters_clause cv_qualifier_seq_opt exception_specification\n '''\n p[0] = ['(',')']\n\ndef p_parameters_clause(p):\n '''parameters_clause : '(' condition_opt ')'\n '''\n p[0] = ['(',')']\n\n#\n# A typed abstract qualifier such as\n# Class * ...\n# looks like a multiply, so pointers are parsed as their binary operation equivalents that\n# ultimately terminate with a degenerate right hand term.\n#\ndef p_abstract_pointer_declaration(p):\n '''abstract_pointer_declaration : ptr_operator_seq\n | multiplicative_expression star_ptr_operator ptr_operator_seq_opt\n '''\n pass\n\ndef p_abstract_parameter_declaration(p):\n '''abstract_parameter_declaration : abstract_pointer_declaration\n | and_expression '&'\n | and_expression '&' abstract_pointer_declaration\n '''\n pass\n\ndef p_special_parameter_declaration(p):\n '''special_parameter_declaration : abstract_parameter_declaration\n | abstract_parameter_declaration '=' assignment_expression\n | ELLIPSIS\n '''\n pass\n\ndef p_parameter_declaration(p):\n '''parameter_declaration : assignment_expression\n | special_parameter_declaration\n | decl_specifier_prefix parameter_declaration\n '''\n pass\n\n#\n# function_definition includes constructor, destructor, implicit int definitions too. A local destructor is successfully parsed as a function-declaration but the ~ was treated as a unary operator. constructor_head is the prefix ambiguity between a constructor and a member-init-list starting with a bit-field.\n#\ndef p_function_definition(p):\n '''function_definition : ctor_definition\n | func_definition\n '''\n pass\n\ndef p_func_definition(p):\n '''func_definition : assignment_expression function_try_block\n | assignment_expression function_body\n | decl_specifier_prefix func_definition\n '''\n global _parse_info\n if p[2] is not None and p[2][0] == '{':\n decl = flatten(p[1])\n #print \"HERE\",decl\n if decl[-1] == ')':\n decl=decl[-3]\n else:\n decl=decl[-1]\n p[0] = decl\n if decl != \"operator\":\n _parse_info.add_function(decl)\n else:\n p[0] = p[2]\n\ndef p_ctor_definition(p):\n '''ctor_definition : constructor_head function_try_block\n | constructor_head function_body\n | decl_specifier_prefix ctor_definition\n '''\n if p[2] is None or p[2][0] == \"try\" or p[2][0] == '{':\n p[0]=p[1]\n else:\n p[0]=p[1]\n\ndef p_constructor_head(p):\n '''constructor_head : bit_field_init_declaration\n | constructor_head ',' assignment_expression\n '''\n p[0]=p[1]\n\ndef p_function_try_block(p):\n '''function_try_block : TRY function_block handler_seq\n '''\n global noExceptionLogic\n noExceptionLogic=False\n p[0] = ['try']\n\ndef p_function_block(p):\n '''function_block : ctor_initializer_opt function_body\n '''\n pass\n\ndef p_function_body(p):\n '''function_body : LBRACE nonbrace_seq_opt RBRACE \n '''\n p[0] = ['{','}']\n\ndef p_initializer_clause(p):\n '''initializer_clause : assignment_expression\n | braced_initializer\n '''\n pass\n\ndef p_braced_initializer(p):\n '''braced_initializer : LBRACE initializer_list RBRACE\n | LBRACE initializer_list ',' RBRACE\n | LBRACE RBRACE\n '''\n pass\n\ndef p_initializer_list(p):\n '''initializer_list : initializer_clause\n | initializer_list ',' initializer_clause\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.8 Classes\n#---------------------------------------------------------------------------------------------------\n#\n# An anonymous bit-field declaration may look very like inheritance:\n# const int B = 3;\n# class A : B ;\n# The two usages are too distant to try to create and enforce a common prefix so we have to resort to\n# a parser hack by backtracking. Inheritance is much the most likely so we mark the input stream context\n# and try to parse a base-clause. If we successfully reach a { the base-clause is ok and inheritance was\n# the correct choice so we unmark and continue. If we fail to find the { an error token causes \n# back-tracking to the alternative parse in elaborated_type_specifier which regenerates the : and \n# declares unconditional success.\n#\n\ndef p_class_specifier_head(p):\n '''class_specifier_head : class_key scoped_id ':' base_specifier_list LBRACE\n | class_key ':' base_specifier_list LBRACE\n | class_key scoped_id LBRACE\n | class_key LBRACE\n '''\n global _parse_info\n base_classes=[]\n if len(p) == 6:\n scope = p[2]\n base_classes = p[4]\n elif len(p) == 4:\n scope = p[2]\n elif len(p) == 5:\n base_classes = p[3]\n else:\n scope = \"\"\n _parse_info.push_scope(scope,p[1],base_classes)\n \n\ndef p_class_key(p):\n '''class_key : CLASS \n | STRUCT \n | UNION\n '''\n p[0] = p[1]\n\ndef p_class_specifier(p):\n '''class_specifier : class_specifier_head member_specification_opt RBRACE\n '''\n scope = _parse_info.pop_scope()\n\ndef p_member_specification_opt(p):\n '''member_specification_opt : empty\n | member_specification_opt member_declaration\n '''\n pass\n\ndef p_member_declaration(p):\n '''member_declaration : accessibility_specifier\n | simple_member_declaration\n | function_definition\n | using_declaration\n | template_declaration\n '''\n p[0] = get_rest(p)\n #print \"Decl\",get_rest(p)\n\n#\n# The generality of constructor names (there need be no parenthesised argument list) means that that\n# name : f(g), h(i)\n# could be the start of a constructor or the start of an anonymous bit-field. An ambiguity is avoided by\n# parsing the ctor-initializer of a function_definition as a bit-field.\n#\ndef p_simple_member_declaration(p):\n '''simple_member_declaration : ';'\n | assignment_expression ';'\n | constructor_head ';'\n | member_init_declarations ';'\n | decl_specifier_prefix simple_member_declaration\n '''\n global _parse_info\n decl = flatten(get_rest(p))\n if len(decl) >= 4 and decl[-3] == \"(\":\n _parse_info.add_function(decl[-4])\n\ndef p_member_init_declarations(p):\n '''member_init_declarations : assignment_expression ',' member_init_declaration\n | constructor_head ',' bit_field_init_declaration\n | member_init_declarations ',' member_init_declaration\n '''\n pass\n\ndef p_member_init_declaration(p):\n '''member_init_declaration : assignment_expression\n | bit_field_init_declaration\n '''\n pass\n\ndef p_accessibility_specifier(p):\n '''accessibility_specifier : access_specifier ':'\n '''\n pass\n\ndef p_bit_field_declaration(p):\n '''bit_field_declaration : assignment_expression ':' bit_field_width\n | ':' bit_field_width\n '''\n if len(p) == 4:\n p[0]=p[1]\n\ndef p_bit_field_width(p):\n '''bit_field_width : logical_or_expression\n | logical_or_expression '?' bit_field_width ':' bit_field_width\n '''\n pass\n\ndef p_bit_field_init_declaration(p):\n '''bit_field_init_declaration : bit_field_declaration\n | bit_field_declaration '=' initializer_clause\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.9 Derived classes\n#---------------------------------------------------------------------------------------------------\ndef p_base_specifier_list(p):\n '''base_specifier_list : base_specifier\n | base_specifier_list ',' base_specifier\n '''\n if len(p) == 2:\n p[0] = [p[1]]\n else:\n p[0] = p[1]+[p[3]]\n\ndef p_base_specifier(p):\n '''base_specifier : scoped_id\n | access_specifier base_specifier\n | VIRTUAL base_specifier\n '''\n if len(p) == 2:\n p[0] = p[1]\n else:\n p[0] = p[2]\n\ndef p_access_specifier(p):\n '''access_specifier : PRIVATE \n | PROTECTED \n | PUBLIC\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.10 Special member functions\n#---------------------------------------------------------------------------------------------------\ndef p_conversion_function_id(p):\n '''conversion_function_id : OPERATOR conversion_type_id\n '''\n p[0] = ['operator']\n\ndef p_conversion_type_id(p):\n '''conversion_type_id : type_specifier ptr_operator_seq_opt\n | type_specifier conversion_type_id\n '''\n pass\n\n#\n# Ctor-initialisers can look like a bit field declaration, given the generalisation of names:\n# Class(Type) : m1(1), m2(2) { }\n# NonClass(bit_field) : int(2), second_variable, ...\n# The grammar below is used within a function_try_block or function_definition.\n# See simple_member_declaration for use in normal member function_definition.\n#\ndef p_ctor_initializer_opt(p):\n '''ctor_initializer_opt : empty\n | ctor_initializer\n '''\n pass\n\ndef p_ctor_initializer(p):\n '''ctor_initializer : ':' mem_initializer_list\n '''\n pass\n\ndef p_mem_initializer_list(p):\n '''mem_initializer_list : mem_initializer\n | mem_initializer_list_head mem_initializer\n '''\n pass\n\ndef p_mem_initializer_list_head(p):\n '''mem_initializer_list_head : mem_initializer_list ','\n '''\n pass\n\ndef p_mem_initializer(p):\n '''mem_initializer : mem_initializer_id '(' expression_list_opt ')'\n '''\n pass\n\ndef p_mem_initializer_id(p):\n '''mem_initializer_id : scoped_id\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.11 Overloading\n#---------------------------------------------------------------------------------------------------\n\ndef p_operator_function_id(p):\n '''operator_function_id : OPERATOR operator\n | OPERATOR '(' ')'\n | OPERATOR LBRACKET RBRACKET\n | OPERATOR '<'\n | OPERATOR '>'\n | OPERATOR operator '<' nonlgt_seq_opt '>'\n '''\n p[0] = [\"operator\"]\n\n#\n# It is not clear from the ANSI standard whether spaces are permitted in delete[]. If not then it can\n# be recognised and returned as DELETE_ARRAY by the lexer. Assuming spaces are permitted there is an\n# ambiguity created by the over generalised nature of expressions. operator new is a valid delarator-id\n# which we may have an undimensioned array of. Semantic rubbish, but syntactically valid. Since the\n# array form is covered by the declarator consideration we can exclude the operator here. The need\n# for a semantic rescue can be eliminated at the expense of a couple of shift-reduce conflicts by\n# removing the comments on the next four lines.\n#\ndef p_operator(p):\n '''operator : NEW\n | DELETE\n | '+'\n | '-'\n | '*'\n | '/'\n | '%'\n | '^'\n | '&'\n | '|'\n | '~'\n | '!'\n | '='\n | ASS_ADD\n | ASS_SUB\n | ASS_MUL\n | ASS_DIV\n | ASS_MOD\n | ASS_XOR\n | ASS_AND\n | ASS_OR\n | SHL\n | SHR\n | ASS_SHR\n | ASS_SHL\n | EQ\n | NE\n | LE\n | GE\n | LOG_AND\n | LOG_OR\n | INC\n | DEC\n | ','\n | ARROW_STAR\n | ARROW\n '''\n p[0]=p[1]\n\n# | IF\n# | SWITCH\n# | WHILE\n# | FOR\n# | DO\ndef p_reserved(p):\n '''reserved : PRIVATE\n | CLiteral\n | CppLiteral\n | IF\n | SWITCH\n | WHILE\n | FOR\n | DO\n | PROTECTED\n | PUBLIC\n | BOOL\n | CHAR\n | DOUBLE\n | FLOAT\n | INT\n | LONG\n | SHORT\n | SIGNED\n | UNSIGNED\n | VOID\n | WCHAR_T\n | CLASS\n | ENUM\n | NAMESPACE\n | STRUCT\n | TYPENAME\n | UNION\n | CONST\n | VOLATILE\n | AUTO\n | EXPLICIT\n | EXPORT\n | EXTERN\n | FRIEND\n | INLINE\n | MUTABLE\n | REGISTER\n | STATIC\n | TEMPLATE\n | TYPEDEF\n | USING\n | VIRTUAL\n | ASM\n | BREAK\n | CASE\n | CATCH\n | CONST_CAST\n | CONTINUE\n | DEFAULT\n | DYNAMIC_CAST\n | ELSE\n | FALSE\n | GOTO\n | OPERATOR\n | REINTERPRET_CAST\n | RETURN\n | SIZEOF\n | STATIC_CAST\n | THIS\n | THROW\n | TRUE\n | TRY\n | TYPEID\n | ATTRIBUTE\n | CDECL\n | TYPEOF\n | uTYPEOF\n '''\n if p[1] in ('try', 'catch', 'throw'):\n global noExceptionLogic\n noExceptionLogic=False\n\n#---------------------------------------------------------------------------------------------------\n# A.12 Templates\n#---------------------------------------------------------------------------------------------------\ndef p_template_declaration(p):\n '''template_declaration : template_parameter_clause declaration\n | EXPORT template_declaration\n '''\n pass\n\ndef p_template_parameter_clause(p):\n '''template_parameter_clause : TEMPLATE '<' nonlgt_seq_opt '>'\n '''\n pass\n\n#\n# Generalised naming makes identifier a valid declaration, so TEMPLATE identifier is too.\n# The TEMPLATE prefix is therefore folded into all names, parenthesis_clause and decl_specifier_prefix.\n#\n# explicit_instantiation: TEMPLATE declaration\n#\ndef p_explicit_specialization(p):\n '''explicit_specialization : TEMPLATE '<' '>' declaration\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# A.13 Exception Handling\n#---------------------------------------------------------------------------------------------------\ndef p_handler_seq(p):\n '''handler_seq : handler\n | handler handler_seq\n '''\n pass\n\ndef p_handler(p):\n '''handler : CATCH '(' exception_declaration ')' compound_statement\n '''\n global noExceptionLogic\n noExceptionLogic=False\n\ndef p_exception_declaration(p):\n '''exception_declaration : parameter_declaration\n '''\n pass\n\ndef p_throw_expression(p):\n '''throw_expression : THROW\n | THROW assignment_expression\n '''\n global noExceptionLogic\n noExceptionLogic=False\n\ndef p_exception_specification(p):\n '''exception_specification : THROW '(' ')'\n | THROW '(' type_id_list ')'\n '''\n global noExceptionLogic\n noExceptionLogic=False\n\ndef p_type_id_list(p):\n '''type_id_list : type_id\n | type_id_list ',' type_id\n '''\n pass\n\n#---------------------------------------------------------------------------------------------------\n# Misc productions\n#---------------------------------------------------------------------------------------------------\ndef p_nonsemicolon_seq(p):\n '''nonsemicolon_seq : empty\n | nonsemicolon_seq nonsemicolon\n '''\n pass\n\ndef p_nonsemicolon(p):\n '''nonsemicolon : misc\n | '('\n | ')'\n | '<'\n | '>'\n | LBRACKET nonbracket_seq_opt RBRACKET\n | LBRACE nonbrace_seq_opt RBRACE\n '''\n pass\n\ndef p_nonparen_seq_opt(p):\n '''nonparen_seq_opt : empty\n | nonparen_seq_opt nonparen\n '''\n pass\n\ndef p_nonparen_seq(p):\n '''nonparen_seq : nonparen\n | nonparen_seq nonparen\n '''\n pass\n\ndef p_nonparen(p):\n '''nonparen : misc\n | '<'\n | '>'\n | ';'\n | LBRACKET nonbracket_seq_opt RBRACKET\n | LBRACE nonbrace_seq_opt RBRACE\n '''\n pass\n\ndef p_nonbracket_seq_opt(p):\n '''nonbracket_seq_opt : empty\n | nonbracket_seq_opt nonbracket\n '''\n pass\n\ndef p_nonbracket_seq(p):\n '''nonbracket_seq : nonbracket\n | nonbracket_seq nonbracket\n '''\n pass\n\ndef p_nonbracket(p):\n '''nonbracket : misc\n | '<'\n | '>'\n | '('\n | ')'\n | ';'\n | LBRACKET nonbracket_seq_opt RBRACKET\n | LBRACE nonbrace_seq_opt RBRACE\n '''\n pass\n\ndef p_nonbrace_seq_opt(p):\n '''nonbrace_seq_opt : empty\n | nonbrace_seq_opt nonbrace\n '''\n pass\n\ndef p_nonbrace(p):\n '''nonbrace : misc\n | '<'\n | '>'\n | '('\n | ')'\n | ';'\n | LBRACKET nonbracket_seq_opt RBRACKET\n | LBRACE nonbrace_seq_opt RBRACE\n '''\n pass\n\ndef p_nonlgt_seq_opt(p):\n '''nonlgt_seq_opt : empty\n | nonlgt_seq_opt nonlgt\n '''\n pass\n\ndef p_nonlgt(p):\n '''nonlgt : misc\n | '('\n | ')'\n | LBRACKET nonbracket_seq_opt RBRACKET\n | '<' nonlgt_seq_opt '>'\n | ';'\n '''\n pass\n\ndef p_misc(p):\n '''misc : operator\n | identifier\n | IntegerLiteral\n | CharacterLiteral\n | FloatingLiteral\n | StringLiteral\n | reserved\n | '?'\n | ':'\n | '.'\n | SCOPE\n | ELLIPSIS\n | EXTENSION\n '''\n pass\n\ndef p_empty(p):\n '''empty : '''\n pass\n\n\n\n#\n# Compute column.\n# input is the input text string\n# token is a token instance\n#\ndef _find_column(input,token):\n ''' TODO '''\n i = token.lexpos\n while i > 0:\n if input[i] == '\\n': break\n i -= 1\n column = (token.lexpos - i)+1\n return column\n\ndef p_error(p):\n if p is None:\n tmp = \"Syntax error at end of file.\"\n else:\n tmp = \"Syntax error at token \"\n if p.type is \"\":\n tmp = tmp + \"''\"\n else:\n tmp = tmp + str(p.type)\n tmp = tmp + \" with value '\"+str(p.value)+\"'\"\n tmp = tmp + \" in line \" + str(lexer.lineno-1)\n tmp = tmp + \" at column \"+str(_find_column(_parsedata,p))\n raise IOError( tmp )\n\n\n\n#\n# The function that performs the parsing\n#\ndef parse_cpp(data=None, filename=None, debug=0, optimize=0, verbose=False, func_filter=None):\n if debug > 0:\n print(\"Debugging parse_cpp!\")\n #\n # Always remove the parser.out file, which is generated to create debugging\n #\n if os.path.exists(\"parser.out\"):\n os.remove(\"parser.out\")\n #\n # Remove the parsetab.py* files. These apparently need to be removed\n # to ensure the creation of a parser.out file.\n #\n if os.path.exists(\"parsetab.py\"):\n os.remove(\"parsetab.py\")\n if os.path.exists(\"parsetab.pyc\"):\n os.remove(\"parsetab.pyc\")\n global debugging\n debugging=True\n #\n # Build lexer\n #\n global lexer\n lexer = lex.lex()\n #\n # Initialize parse object\n #\n global _parse_info\n _parse_info = CppInfo(filter=func_filter)\n _parse_info.verbose=verbose\n #\n # Build yaccer\n #\n write_table = not os.path.exists(\"parsetab.py\")\n yacc.yacc(debug=debug, optimize=optimize, write_tables=write_table)\n #\n # Parse the file\n #\n global _parsedata\n if not data is None:\n _parsedata=data\n ply_init(_parsedata)\n yacc.parse(data,debug=debug)\n elif not filename is None:\n f = open(filename)\n data = f.read()\n f.close()\n _parsedata=data\n ply_init(_parsedata)\n yacc.parse(data, debug=debug)\n else:\n return None\n #\n if not noExceptionLogic:\n _parse_info.noExceptionLogic = False\n else:\n for key in identifier_lineno:\n if 'ASSERT_THROWS' in key:\n _parse_info.noExceptionLogic = False\n break\n _parse_info.noExceptionLogic = True\n #\n return _parse_info\n\n\n\nimport sys\n\nif __name__ == '__main__':\n #\n # This MAIN routine parses a sequence of files provided at the command\n # line. If '-v' is included, then a verbose parsing output is \n # generated.\n #\n for arg in sys.argv[1:]:\n if arg == \"-v\":\n continue\n print(\"Parsing file '\"+arg+\"'\")\n if '-v' in sys.argv:\n parse_cpp(filename=arg,debug=2,verbose=2)\n else:\n parse_cpp(filename=arg,verbose=2)\n #\n # Print the _parse_info object summary for this file.\n # This illustrates how class inheritance can be used to \n # deduce class members.\n # \n print(str(_parse_info))\n\n" }, { "path": "cxxtest/python/python3/cxxtest/cxxtest_fog.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n#\n# TODO: add line number info\n# TODO: add test function names\n#\n\n\n\nimport sys\nimport re\n#from os.path import abspath, dirname\n#sys.path.insert(0, dirname(dirname(abspath(__file__))))\n#sys.path.insert(0, dirname(dirname(abspath(__file__)))+\"/cxx_parse\")\nfrom .cxxtest_misc import abort\nfrom . import cxx_parser\nimport re\n\ndef cstr( str ):\n '''Convert a string to its C representation'''\n return '\"' + re.sub('\\\\\\\\', '\\\\\\\\\\\\\\\\', str ) + '\"'\n\ndef scanInputFiles(files, _options):\n '''Scan all input files for test suites'''\n suites=[]\n for file in files:\n try:\n print(\"Parsing file \"+file, end=' ')\n sys.stdout.flush()\n parse_info = cxx_parser.parse_cpp(filename=file,optimize=1)\n except IOError as err:\n print(\" error.\")\n print(str(err))\n continue\n print(\"done.\") \n sys.stdout.flush()\n #\n # WEH: see if it really makes sense to use parse information to\n # initialize this data. I don't think so...\n #\n _options.haveStandardLibrary=1\n if not parse_info.noExceptionLogic:\n _options.haveExceptionHandling=1\n #\n keys = list(parse_info.index.keys())\n tpat = re.compile(\"[Tt][Ee][Ss][Tt]\")\n for key in keys:\n if parse_info.index[key].scope_t == \"class\" and parse_info.is_baseclass(key,\"CxxTest::TestSuite\"):\n name=parse_info.index[key].name\n suite = { 'name' : name,\n 'file' : file,\n 'cfile' : cstr(file),\n 'line' : str(parse_info.index[key].lineno),\n 'generated' : 0,\n 'object' : 'suite_%s' % name,\n 'dobject' : 'suiteDescription_%s' % name,\n 'tlist' : 'Tests_%s' % name,\n 'tests' : [],\n 'lines' : [] }\n for fn in parse_info.get_functions(key,quiet=True):\n tname = fn[0]\n lineno = str(fn[1])\n if tname.startswith('createSuite'):\n # Indicate that we're using a dynamically generated test suite\n suite['create'] = str(lineno) # (unknown line)\n if tname.startswith('destroySuite'):\n # Indicate that we're using a dynamically generated test suite\n suite['destroy'] = str(lineno) # (unknown line)\n if not tpat.match(tname):\n # Skip non-test methods\n continue\n test = { 'name' : tname,\n 'suite' : suite,\n 'class' : 'TestDescription_suite_%s_%s' % (suite['name'], tname),\n 'object' : 'testDescription_suite_%s_%s' % (suite['name'], tname),\n 'line' : lineno,\n }\n suite['tests'].append(test)\n suites.append(suite)\n\n if not _options.root:\n ntests = 0\n for suite in suites:\n ntests += len(suite['tests'])\n if ntests == 0:\n abort( 'No tests defined' )\n #\n return [_options, suites]\n\n" }, { "path": "cxxtest/python/python3/cxxtest/cxxtest_misc.py", "content": "#!/usr/bin/python\n#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\nimport sys\n\ndef abort( problem ):\n '''Print error message and exit'''\n sys.stderr.write( '\\n' )\n sys.stderr.write( problem )\n sys.stderr.write( '\\n\\n' )\n sys.exit(2)\n\n" }, { "path": "cxxtest/python/python3/cxxtest/cxxtest_parser.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n\n\nimport codecs\nimport re\n#import sys\n#import getopt\n#import glob\nfrom cxxtest.cxxtest_misc import abort\n\n# Global variables\nsuites = []\nsuite = None\ninBlock = 0\noptions=None\n\ndef scanInputFiles(files, _options):\n '''Scan all input files for test suites'''\n global options\n options=_options\n for file in files:\n scanInputFile(file)\n global suites\n if len(suites) is 0 and not options.root:\n abort( 'No tests defined' )\n return [options,suites]\n\nlineCont_re = re.compile('(.*)\\\\\\s*$')\ndef scanInputFile(fileName):\n '''Scan single input file for test suites'''\n # mode 'rb' is problematic in python3 - byte arrays don't behave the same as\n # strings.\n # As far as the choice of the default encoding: utf-8 chews through\n # everything that the previous ascii codec could, plus most of new code.\n # TODO: figure out how to do this properly - like autodetect encoding from\n # file header.\n file = codecs.open(fileName, mode='r', encoding='utf-8')\n prev = \"\"\n lineNo = 0\n contNo = 0\n while 1:\n line = file.readline()\n if not line:\n break\n lineNo += 1\n\n m = lineCont_re.match(line)\n if m:\n prev += m.group(1) + \" \"\n contNo += 1\n else:\n scanInputLine( fileName, lineNo - contNo, prev + line )\n contNo = 0\n prev = \"\"\n if contNo:\n scanInputLine( fileName, lineNo - contNo, prev + line )\n \n closeSuite()\n file.close()\n\ndef scanInputLine( fileName, lineNo, line ):\n '''Scan single input line for interesting stuff'''\n scanLineForExceptionHandling( line )\n scanLineForStandardLibrary( line )\n\n scanLineForSuiteStart( fileName, lineNo, line )\n\n global suite\n if suite:\n scanLineInsideSuite( suite, lineNo, line )\n\ndef scanLineInsideSuite( suite, lineNo, line ):\n '''Analyze line which is part of a suite'''\n global inBlock\n if lineBelongsToSuite( suite, lineNo, line ):\n scanLineForTest( suite, lineNo, line )\n scanLineForCreate( suite, lineNo, line )\n scanLineForDestroy( suite, lineNo, line )\n\ndef lineBelongsToSuite( suite, lineNo, line ):\n '''Returns whether current line is part of the current suite.\n This can be false when we are in a generated suite outside of CXXTEST_CODE() blocks\n If the suite is generated, adds the line to the list of lines'''\n if not suite['generated']:\n return 1\n\n global inBlock\n if not inBlock:\n inBlock = lineStartsBlock( line )\n if inBlock:\n inBlock = addLineToBlock( suite, lineNo, line )\n return inBlock\n\n\nstd_re = re.compile( r\"\\b(std\\s*::|CXXTEST_STD|using\\s+namespace\\s+std\\b|^\\s*\\#\\s*include\\s+<[a-z0-9]+>)\" )\ndef scanLineForStandardLibrary( line ):\n '''Check if current line uses standard library'''\n global options\n if not options.haveStandardLibrary and std_re.search(line):\n if not options.noStandardLibrary:\n options.haveStandardLibrary = 1\n\nexception_re = re.compile( r\"\\b(throw|try|catch|TSM?_ASSERT_THROWS[A-Z_]*)\\b\" )\ndef scanLineForExceptionHandling( line ):\n '''Check if current line uses exception handling'''\n global options\n if not options.haveExceptionHandling and exception_re.search(line):\n if not options.noExceptionHandling:\n options.haveExceptionHandling = 1\n\nclassdef = '(?:::\\s*)?(?:\\w+\\s*::\\s*)*\\w+'\nbaseclassdef = '(?:public|private|protected)\\s+%s' % (classdef,)\ngeneral_suite = r\"\\bclass\\s+(%s)\\s*:(?:\\s*%s\\s*,)*\\s*public\\s+\" \\\n % (classdef, baseclassdef,)\ntestsuite = '(?:(?:::)?\\s*CxxTest\\s*::\\s*)?TestSuite'\nsuites_re = { re.compile( general_suite + testsuite ) : None }\ngeneratedSuite_re = re.compile( r'\\bCXXTEST_SUITE\\s*\\(\\s*(\\w*)\\s*\\)' )\ndef scanLineForSuiteStart( fileName, lineNo, line ):\n '''Check if current line starts a new test suite'''\n for i in list(suites_re.items()):\n m = i[0].search( line )\n if m:\n suite = startSuite( m.group(1), fileName, lineNo, 0 )\n if i[1] is not None:\n for test in i[1]['tests']:\n addTest(suite, test['name'], test['line'])\n break\n m = generatedSuite_re.search( line )\n if m:\n sys.stdout.write( \"%s:%s: Warning: Inline test suites are deprecated.\\n\" % (fileName, lineNo) )\n startSuite( m.group(1), fileName, lineNo, 1 )\n\ndef startSuite( name, file, line, generated ):\n '''Start scanning a new suite'''\n global suite\n closeSuite()\n object_name = name.replace(':',\"_\")\n suite = { 'name' : name,\n 'file' : file,\n 'cfile' : cstr(file),\n 'line' : line,\n 'generated' : generated,\n 'object' : 'suite_%s' % object_name,\n 'dobject' : 'suiteDescription_%s' % object_name,\n 'tlist' : 'Tests_%s' % object_name,\n 'tests' : [],\n 'lines' : [] }\n suites_re[re.compile( general_suite + name )] = suite\n return suite\n\ndef lineStartsBlock( line ):\n '''Check if current line starts a new CXXTEST_CODE() block'''\n return re.search( r'\\bCXXTEST_CODE\\s*\\(', line ) is not None\n\ntest_re = re.compile( r'^([^/]|/[^/])*\\bvoid\\s+([Tt]est\\w+)\\s*\\(\\s*(void)?\\s*\\)' )\ndef scanLineForTest( suite, lineNo, line ):\n '''Check if current line starts a test'''\n m = test_re.search( line )\n if m:\n addTest( suite, m.group(2), lineNo )\n\ndef addTest( suite, name, line ):\n '''Add a test function to the current suite'''\n test = { 'name' : name,\n 'suite' : suite,\n 'class' : 'TestDescription_%s_%s' % (suite['object'], name),\n 'object' : 'testDescription_%s_%s' % (suite['object'], name),\n 'line' : line,\n }\n suite['tests'].append( test )\n\ndef addLineToBlock( suite, lineNo, line ):\n '''Append the line to the current CXXTEST_CODE() block'''\n line = fixBlockLine( suite, lineNo, line )\n line = re.sub( r'^.*\\{\\{', '', line )\n \n e = re.search( r'\\}\\}', line )\n if e:\n line = line[:e.start()]\n suite['lines'].append( line )\n return e is None\n\ndef fixBlockLine( suite, lineNo, line):\n '''Change all [E]TS_ macros used in a line to _[E]TS_ macros with the correct file/line'''\n return re.sub( r'\\b(E?TSM?_(ASSERT[A-Z_]*|FAIL))\\s*\\(',\n r'_\\1(%s,%s,' % (suite['cfile'], lineNo),\n line, 0 )\n\ncreate_re = re.compile( r'\\bstatic\\s+\\w+\\s*\\*\\s*createSuite\\s*\\(\\s*(void)?\\s*\\)' )\ndef scanLineForCreate( suite, lineNo, line ):\n '''Check if current line defines a createSuite() function'''\n if create_re.search( line ):\n addSuiteCreateDestroy( suite, 'create', lineNo )\n\ndestroy_re = re.compile( r'\\bstatic\\s+void\\s+destroySuite\\s*\\(\\s*\\w+\\s*\\*\\s*\\w*\\s*\\)' )\ndef scanLineForDestroy( suite, lineNo, line ):\n '''Check if current line defines a destroySuite() function'''\n if destroy_re.search( line ):\n addSuiteCreateDestroy( suite, 'destroy', lineNo )\n\ndef cstr( s ):\n '''Convert a string to its C representation'''\n return '\"' + s.replace( '\\\\', '\\\\\\\\' ) + '\"'\n\n\ndef addSuiteCreateDestroy( suite, which, line ):\n '''Add createSuite()/destroySuite() to current suite'''\n if which in suite:\n abort( '%s:%s: %sSuite() already declared' % ( suite['file'], str(line), which ) )\n suite[which] = line\n\ndef closeSuite():\n '''Close current suite and add it to the list if valid'''\n global suite\n if suite is not None:\n if len(suite['tests']) is not 0:\n verifySuite(suite)\n rememberSuite(suite)\n suite = None\n\ndef verifySuite(suite):\n '''Verify current suite is legal'''\n if 'create' in suite and 'destroy' not in suite:\n abort( '%s:%s: Suite %s has createSuite() but no destroySuite()' %\n (suite['file'], suite['create'], suite['name']) )\n elif 'destroy' in suite and 'create' not in suite:\n abort( '%s:%s: Suite %s has destroySuite() but no createSuite()' %\n (suite['file'], suite['destroy'], suite['name']) )\n\ndef rememberSuite(suite):\n '''Add current suite to list'''\n global suites\n suites.append( suite )\n\n" }, { "path": "cxxtest/python/python3/cxxtest/cxxtestgen.py", "content": "#-------------------------------------------------------------------------\n# CxxTest: A lightweight C++ unit testing library.\n# Copyright (c) 2008 Sandia Corporation.\n# This software is distributed under the LGPL License v2.1\n# For more information, see the COPYING file in the top CxxTest directory.\n# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,\n# the U.S. Government retains certain rights in this software.\n#-------------------------------------------------------------------------\n\n# vim: fileencoding=utf-8\n\n\n# the above import important for forward-compatibility with python3,\n# which is already the default in archlinux!\n\n__all__ = ['main']\n\nfrom . import __release__\nimport os\nimport sys\nimport re\nimport glob\nfrom optparse import OptionParser\nfrom . import cxxtest_parser\n\ntry:\n from . import cxxtest_fog\n imported_fog=True\nexcept ImportError:\n imported_fog=False\n\nfrom .cxxtest_misc import abort\n\noptions = []\nsuites = []\n\nwrotePreamble = 0\nwroteWorld = 0\nlastIncluded = ''\n\ndef main(args=sys.argv):\n '''The main program'''\n #\n # Reset global state\n #\n global wrotePreamble\n wrotePreamble=0\n global wroteWorld\n wroteWorld=0\n global lastIncluded\n lastIncluded = ''\n\n global suites\n global options\n files = parseCommandline(args)\n if imported_fog and options.fog:\n [options,suites] = cxxtest_fog.scanInputFiles( files, options )\n else:\n [options,suites] = cxxtest_parser.scanInputFiles( files, options )\n writeOutput()\n\ndef parseCommandline(args):\n '''Analyze command line arguments'''\n global imported_fog\n global options\n parser = OptionParser(\"%prog [options] [<filename> ...]\")\n parser.add_option(\"--version\",\n action=\"store_true\", dest=\"version\", default=False,\n help=\"Write the CxxTest version.\")\n parser.add_option(\"-o\", \"--output\",\n dest=\"outputFileName\", default=None, metavar=\"NAME\",\n help=\"Write output to file NAME.\")\n parser.add_option(\"-w\",\"--world\", dest=\"world\", default=\"cxxtest\",\n help=\"The label of the tests, used to name the XML results.\")\n parser.add_option(\"\", \"--include\", action=\"append\",\n dest=\"headers\", default=[], metavar=\"HEADER\",\n help=\"Include file HEADER in the test runner before other headers.\")\n parser.add_option(\"\", \"--abort-on-fail\",\n action=\"store_true\", dest=\"abortOnFail\", default=False,\n help=\"Abort tests on failed asserts (like xUnit).\")\n parser.add_option(\"\", \"--main\",\n action=\"store\", dest=\"main\", default=\"main\",\n help=\"Specify an alternative name for the main() function.\")\n parser.add_option(\"\", \"--headers\",\n action=\"store\", dest=\"header_filename\", default=None,\n help=\"Specify a filename that contains a list of header files that are processed to generate a test runner.\")\n parser.add_option(\"\", \"--runner\",\n dest=\"runner\", default=\"\", metavar=\"CLASS\",\n help=\"Create a test runner that processes test events using the class CxxTest::CLASS.\")\n parser.add_option(\"\", \"--gui\",\n dest=\"gui\", metavar=\"CLASS\",\n help=\"Create a GUI test runner that processes test events using the class CxxTest::CLASS. (deprecated)\")\n parser.add_option(\"\", \"--error-printer\",\n action=\"store_true\", dest=\"error_printer\", default=False,\n help=\"Create a test runner using the ErrorPrinter class, and allow the use of the standard library.\")\n parser.add_option(\"\", \"--xunit-printer\",\n action=\"store_true\", dest=\"xunit_printer\", default=False,\n help=\"Create a test runner using the XUnitPrinter class.\")\n parser.add_option(\"\", \"--xunit-file\", dest=\"xunit_file\", default=\"\",\n help=\"The file to which the XML summary is written for test runners using the XUnitPrinter class. The default XML filename is TEST-<world>.xml, where <world> is the value of the --world option. (default: cxxtest)\")\n parser.add_option(\"\", \"--have-std\",\n action=\"store_true\", dest=\"haveStandardLibrary\", default=False,\n help=\"Use the standard library (even if not found in tests).\")\n parser.add_option(\"\", \"--no-std\",\n action=\"store_true\", dest=\"noStandardLibrary\", default=False,\n help=\"Do not use standard library (even if found in tests).\")\n parser.add_option(\"\", \"--have-eh\",\n action=\"store_true\", dest=\"haveExceptionHandling\", default=False,\n help=\"Use exception handling (even if not found in tests).\")\n parser.add_option(\"\", \"--no-eh\",\n action=\"store_true\", dest=\"noExceptionHandling\", default=False,\n help=\"Do not use exception handling (even if found in tests).\")\n parser.add_option(\"\", \"--longlong\",\n dest=\"longlong\", default=None, metavar=\"TYPE\",\n help=\"Use TYPE as for long long integers. (default: not supported)\")\n parser.add_option(\"\", \"--no-static-init\",\n action=\"store_true\", dest=\"noStaticInit\", default=False,\n help=\"Do not rely on static initialization in the test runner.\")\n parser.add_option(\"\", \"--template\",\n dest=\"templateFileName\", default=None, metavar=\"TEMPLATE\",\n help=\"Generate the test runner using file TEMPLATE to define a template.\")\n parser.add_option(\"\", \"--root\",\n action=\"store_true\", dest=\"root\", default=False,\n help=\"Write the main() function and global data for a test runner.\")\n parser.add_option(\"\", \"--part\",\n action=\"store_true\", dest=\"part\", default=False,\n help=\"Write the tester classes for a test runner.\")\n #parser.add_option(\"\", \"--factor\",\n #action=\"store_true\", dest=\"factor\", default=False,\n #help=\"Declare the _CXXTEST_FACTOR macro. (deprecated)\")\n if imported_fog:\n fog_help = \"Use new FOG C++ parser\"\n else:\n fog_help = \"Use new FOG C++ parser (disabled)\"\n parser.add_option(\"-f\", \"--fog-parser\",\n action=\"store_true\",\n dest=\"fog\",\n default=False,\n help=fog_help\n )\n\n (options, args) = parser.parse_args(args=args)\n if not options.header_filename is None:\n if not os.path.exists(options.header_filename):\n abort( \"ERROR: the file '%s' does not exist!\" % options.header_filename )\n INPUT = open(options.header_filename)\n headers = [line.strip() for line in INPUT]\n args.extend( headers )\n INPUT.close()\n\n if options.fog and not imported_fog:\n abort( \"Cannot use the FOG parser. Check that the 'ply' package is installed. The 'ordereddict' package is also required if running Python 2.6\")\n\n if options.version:\n printVersion()\n\n # the cxxtest builder relies on this behaviour! don't remove\n if options.runner == 'none':\n options.runner = None\n\n if options.xunit_printer or options.runner == \"XUnitPrinter\":\n options.xunit_printer=True\n options.runner=\"XUnitPrinter\"\n if len(args) > 1:\n if options.xunit_file == \"\":\n if options.world == \"\":\n options.world = \"cxxtest\"\n options.xunit_file=\"TEST-\"+options.world+\".xml\"\n elif options.xunit_file == \"\":\n if options.world == \"\":\n options.world = \"cxxtest\"\n options.xunit_file=\"TEST-\"+options.world+\".xml\"\n\n if options.error_printer:\n options.runner= \"ErrorPrinter\"\n options.haveStandardLibrary = True\n \n if options.noStaticInit and (options.root or options.part):\n abort( '--no-static-init cannot be used with --root/--part' )\n\n if options.gui and not options.runner:\n options.runner = 'StdioPrinter'\n\n files = setFiles(args[1:])\n if len(files) == 0 and not options.root:\n sys.stderr.write(parser.error(\"No input files found\"))\n\n return files\n\n\ndef printVersion():\n '''Print CxxTest version and exit'''\n sys.stdout.write( \"This is CxxTest version %s.\\n\" % __release__.__version__ )\n sys.exit(0)\n\ndef setFiles(patterns ):\n '''Set input files specified on command line'''\n files = expandWildcards( patterns )\n return files\n\ndef expandWildcards( patterns ):\n '''Expand all wildcards in an array (glob)'''\n fileNames = []\n for pathName in patterns:\n patternFiles = glob.glob( pathName )\n for fileName in patternFiles:\n fileNames.append( fixBackslashes( fileName ) )\n return fileNames\n\ndef fixBackslashes( fileName ):\n '''Convert backslashes to slashes in file name'''\n return re.sub( r'\\\\', '/', fileName, 0 )\n\n\ndef writeOutput():\n '''Create output file'''\n if options.templateFileName:\n writeTemplateOutput()\n else:\n writeSimpleOutput()\n\ndef writeSimpleOutput():\n '''Create output not based on template'''\n output = startOutputFile()\n writePreamble( output )\n if options.root or not options.part:\n writeMain( output )\n\n if len(suites) > 0:\n output.write(\"bool \"+suites[0]['object']+\"_init = false;\\n\")\n\n writeWorld( output )\n output.close()\n\ninclude_re = re.compile( r\"\\s*\\#\\s*include\\s+<cxxtest/\" )\npreamble_re = re.compile( r\"^\\s*<CxxTest\\s+preamble>\\s*$\" )\nworld_re = re.compile( r\"^\\s*<CxxTest\\s+world>\\s*$\" )\ndef writeTemplateOutput():\n '''Create output based on template file'''\n template = open(options.templateFileName)\n output = startOutputFile()\n while 1:\n line = template.readline()\n if not line:\n break;\n if include_re.search( line ):\n writePreamble( output )\n output.write( line )\n elif preamble_re.search( line ):\n writePreamble( output )\n elif world_re.search( line ):\n if len(suites) > 0:\n output.write(\"bool \"+suites[0]['object']+\"_init = false;\\n\")\n writeWorld( output )\n else:\n output.write( line )\n template.close()\n output.close()\n\ndef startOutputFile():\n '''Create output file and write header'''\n if options.outputFileName is not None:\n output = open( options.outputFileName, 'w' )\n else:\n output = sys.stdout\n output.write( \"/* Generated file, do not edit */\\n\\n\" )\n return output\n\ndef writePreamble( output ):\n '''Write the CxxTest header (#includes and #defines)'''\n global wrotePreamble\n if wrotePreamble: return\n output.write( \"#ifndef CXXTEST_RUNNING\\n\" )\n output.write( \"#define CXXTEST_RUNNING\\n\" )\n output.write( \"#endif\\n\" )\n output.write( \"\\n\" )\n if options.xunit_printer:\n output.write( \"#include <fstream>\\n\" )\n if options.haveStandardLibrary:\n output.write( \"#define _CXXTEST_HAVE_STD\\n\" )\n if options.haveExceptionHandling:\n output.write( \"#define _CXXTEST_HAVE_EH\\n\" )\n if options.abortOnFail:\n output.write( \"#define _CXXTEST_ABORT_TEST_ON_FAIL\\n\" )\n if options.longlong:\n output.write( \"#define _CXXTEST_LONGLONG %s\\n\" % options.longlong )\n #if options.factor:\n #output.write( \"#define _CXXTEST_FACTOR\\n\" )\n for header in options.headers:\n output.write( \"#include \\\"%s\\\"\\n\" % header )\n output.write( \"#include <cxxtest/TestListener.h>\\n\" )\n output.write( \"#include <cxxtest/TestTracker.h>\\n\" )\n output.write( \"#include <cxxtest/TestRunner.h>\\n\" )\n output.write( \"#include <cxxtest/RealDescriptions.h>\\n\" )\n output.write( \"#include <cxxtest/TestMain.h>\\n\" )\n if options.runner:\n output.write( \"#include <cxxtest/%s.h>\\n\" % options.runner )\n if options.gui:\n output.write( \"#include <cxxtest/%s.h>\\n\" % options.gui )\n output.write( \"\\n\" )\n wrotePreamble = 1\n\ndef writeMain( output ):\n '''Write the main() function for the test runner'''\n if not (options.gui or options.runner):\n return\n output.write( 'int %s( int argc, char *argv[] ) {\\n' % options.main )\n output.write( ' int status;\\n' )\n if options.noStaticInit:\n output.write( ' CxxTest::initialize();\\n' )\n if options.gui:\n tester_t = \"CxxTest::GuiTuiRunner<CxxTest::%s, CxxTest::%s> \" % (options.gui, options.runner)\n else:\n tester_t = \"CxxTest::%s\" % (options.runner)\n if options.xunit_printer:\n output.write( ' std::ofstream ofstr(\"%s\");\\n' % options.xunit_file )\n output.write( ' %s tmp(ofstr);\\n' % tester_t )\n output.write( ' CxxTest::RealWorldDescription::_worldName = \"%s\";\\n' % options.world )\n else:\n output.write( ' %s tmp;\\n' % tester_t )\n output.write( ' status = CxxTest::Main<%s>( tmp, argc, argv );\\n' % tester_t )\n output.write( ' return status;\\n')\n output.write( '}\\n' )\n\n\ndef writeWorld( output ):\n '''Write the world definitions'''\n global wroteWorld\n if wroteWorld: return\n writePreamble( output )\n writeSuites( output )\n if options.root or not options.part:\n writeRoot( output )\n writeWorldDescr( output )\n if options.noStaticInit:\n writeInitialize( output )\n wroteWorld = 1\n\ndef writeSuites(output):\n '''Write all TestDescriptions and SuiteDescriptions'''\n for suite in suites:\n writeInclude( output, suite['file'] )\n if isGenerated(suite):\n generateSuite( output, suite )\n if isDynamic(suite):\n writeSuitePointer( output, suite )\n else:\n writeSuiteObject( output, suite )\n writeTestList( output, suite )\n writeSuiteDescription( output, suite )\n writeTestDescriptions( output, suite )\n\ndef isGenerated(suite):\n '''Checks whether a suite class should be created'''\n return suite['generated']\n\ndef isDynamic(suite):\n '''Checks whether a suite is dynamic'''\n return 'create' in suite\n\ndef writeInclude(output, file):\n '''Add #include \"file\" statement'''\n global lastIncluded\n if file == lastIncluded: return\n output.writelines( [ '#include \"', file, '\"\\n\\n' ] )\n lastIncluded = file\n\ndef generateSuite( output, suite ):\n '''Write a suite declared with CXXTEST_SUITE()'''\n output.write( 'class %s : public CxxTest::TestSuite {\\n' % suite['name'] )\n output.write( 'public:\\n' )\n for line in suite['lines']:\n output.write(line)\n output.write( '};\\n\\n' )\n\ndef writeSuitePointer( output, suite ):\n '''Create static suite pointer object for dynamic suites'''\n if options.noStaticInit:\n output.write( 'static %s *%s;\\n\\n' % (suite['name'], suite['object']) )\n else:\n output.write( 'static %s *%s = 0;\\n\\n' % (suite['name'], suite['object']) )\n\ndef writeSuiteObject( output, suite ):\n '''Create static suite object for non-dynamic suites'''\n output.writelines( [ \"static \", suite['name'], \" \", suite['object'], \";\\n\\n\" ] )\n\ndef writeTestList( output, suite ):\n '''Write the head of the test linked list for a suite'''\n if options.noStaticInit:\n output.write( 'static CxxTest::List %s;\\n' % suite['tlist'] )\n else:\n output.write( 'static CxxTest::List %s = { 0, 0 };\\n' % suite['tlist'] )\n\ndef writeWorldDescr( output ):\n '''Write the static name of the world name'''\n if options.noStaticInit:\n output.write( 'const char* CxxTest::RealWorldDescription::_worldName;\\n' )\n else:\n output.write( 'const char* CxxTest::RealWorldDescription::_worldName = \"cxxtest\";\\n' )\n\ndef writeTestDescriptions( output, suite ):\n '''Write all test descriptions for a suite'''\n for test in suite['tests']:\n writeTestDescription( output, suite, test )\n\ndef writeTestDescription( output, suite, test ):\n '''Write test description object'''\n output.write( 'static class %s : public CxxTest::RealTestDescription {\\n' % test['class'] )\n output.write( 'public:\\n' )\n if not options.noStaticInit:\n output.write( ' %s() : CxxTest::RealTestDescription( %s, %s, %s, \"%s\" ) {}\\n' %\n (test['class'], suite['tlist'], suite['dobject'], test['line'], test['name']) )\n output.write( ' void runTest() { %s }\\n' % runBody( suite, test ) )\n output.write( '} %s;\\n\\n' % test['object'] )\n\ndef runBody( suite, test ):\n '''Body of TestDescription::run()'''\n if isDynamic(suite): return dynamicRun( suite, test )\n else: return staticRun( suite, test )\n\ndef dynamicRun( suite, test ):\n '''Body of TestDescription::run() for test in a dynamic suite'''\n return 'if ( ' + suite['object'] + ' ) ' + suite['object'] + '->' + test['name'] + '();'\n \ndef staticRun( suite, test ):\n '''Body of TestDescription::run() for test in a non-dynamic suite'''\n return suite['object'] + '.' + test['name'] + '();'\n \ndef writeSuiteDescription( output, suite ):\n '''Write SuiteDescription object'''\n if isDynamic( suite ):\n writeDynamicDescription( output, suite )\n else:\n writeStaticDescription( output, suite )\n\ndef writeDynamicDescription( output, suite ):\n '''Write SuiteDescription for a dynamic suite'''\n output.write( 'CxxTest::DynamicSuiteDescription<%s> %s' % (suite['name'], suite['dobject']) )\n if not options.noStaticInit:\n output.write( '( %s, %s, \"%s\", %s, %s, %s, %s )' %\n (suite['cfile'], suite['line'], suite['name'], suite['tlist'],\n suite['object'], suite['create'], suite['destroy']) )\n output.write( ';\\n\\n' )\n\ndef writeStaticDescription( output, suite ):\n '''Write SuiteDescription for a static suite'''\n output.write( 'CxxTest::StaticSuiteDescription %s' % suite['dobject'] )\n if not options.noStaticInit:\n output.write( '( %s, %s, \"%s\", %s, %s )' %\n (suite['cfile'], suite['line'], suite['name'], suite['object'], suite['tlist']) )\n output.write( ';\\n\\n' )\n\ndef writeRoot(output):\n '''Write static members of CxxTest classes'''\n output.write( '#include <cxxtest/Root.cpp>\\n' )\n\ndef writeInitialize(output):\n '''Write CxxTest::initialize(), which replaces static initialization'''\n output.write( 'namespace CxxTest {\\n' )\n output.write( ' void initialize()\\n' )\n output.write( ' {\\n' )\n for suite in suites:\n output.write( ' %s.initialize();\\n' % suite['tlist'] )\n if isDynamic(suite):\n output.write( ' %s = 0;\\n' % suite['object'] )\n output.write( ' %s.initialize( %s, %s, \"%s\", %s, %s, %s, %s );\\n' %\n (suite['dobject'], suite['cfile'], suite['line'], suite['name'],\n suite['tlist'], suite['object'], suite['create'], suite['destroy']) )\n else:\n output.write( ' %s.initialize( %s, %s, \"%s\", %s, %s );\\n' %\n (suite['dobject'], suite['cfile'], suite['line'], suite['name'],\n suite['object'], suite['tlist']) )\n\n for test in suite['tests']:\n output.write( ' %s.initialize( %s, %s, %s, \"%s\" );\\n' %\n (test['object'], suite['tlist'], suite['dobject'], test['line'], test['name']) )\n\n output.write( ' }\\n' )\n output.write( '}\\n' )\n\n" }, { "path": "cxxtest/python/python3/scripts/cxxtestgen", "content": "#! python\n\nimport cxxtest.cxxtestgen\n\ncxxtest.cxxtestgen.main()\n" }, { "path": "cxxtest/python/scripts/cxxtestgen", "content": "#! python\n\nimport cxxtest.cxxtestgen\n\ncxxtest.cxxtestgen.main()\n" }, { "path": "cxxtest/python/setup.py", "content": "\"\"\"\nScript to generate the installer for cxxtest.\n\"\"\"\n\nclassifiers = \"\"\"\\\nDevelopment Status :: 4 - Beta\nIntended Audience :: End Users/Desktop\nLicense :: OSI Approved :: LGPL License\nNatural Language :: English\nOperating System :: Microsoft :: Windows\nOperating System :: Unix\nProgramming Language :: Python\nTopic :: Software Development :: Libraries :: Python Modules\n\"\"\"\n\nimport os\nimport sys\nfrom os.path import realpath, dirname\nif sys.version_info >= (3,0):\n sys.path.insert(0, dirname(realpath(__file__))+os.sep+'python3')\n os.chdir('python3')\n\nimport cxxtest\n\ntry:\n from setuptools import setup\nexcept ImportError:\n from distutils.core import setup\n\ndoclines = cxxtest.__doc__.split(\"\\n\")\n\nsetup(name=\"cxxtest\",\n version=cxxtest.__version__,\n maintainer=cxxtest.__maintainer__,\n maintainer_email=cxxtest.__maintainer_email__,\n url = cxxtest.__url__,\n license = cxxtest.__license__,\n platforms = [\"any\"],\n description = doclines[0],\n classifiers = filter(None, classifiers.split(\"\\n\")),\n long_description = \"\\n\".join(doclines[2:]),\n packages=['cxxtest'],\n keywords=['utility'],\n scripts=['scripts/cxxtestgen']\n #\n # The entry_points option is not supported by distutils.core\n #\n #entry_points=\"\"\"\n #[console_scripts]\n #cxxtestgen = cxxtest.cxxtestgen:main\n #\"\"\"\n )\n\n" }, { "path": "demoapps/CMakeLists.txt", "content": "project(GraphLab)\n\n# link_libraries(${Boost_LIBRARIES})\n# link_libraries(${GraphLab_LIBRARIES})\n\n\n\nmacro(add_all_subdirectories retval curdir)\n file(GLOB sub-dir RELATIVE ${curdir} *)\n set(list_of_dirs \"\")\n foreach(dir ${sub-dir})\n if(IS_DIRECTORY ${curdir}/${dir})\n STRING(SUBSTRING ${dir} 0 1 firstchar)\n if(${firstchar} STREQUAL \".\" OR ${firstchar} STREQUAL \"_\" )\n else(${firstchar} STREQUAL \".\" OR ${firstchar} STREQUAL \"_\")\n set(list_of_dirs ${list_of_dirs} ${dir})\n message(STATUS \"Detected demo app: \" ${dir})\n add_subdirectory(${dir})\n endif()\n endif()\n endforeach()\n set(${retval} ${list_of_dirs})\nendmacro()\n\nadd_all_subdirectories(retval, ${CMAKE_CURRENT_SOURCE_DIR})\n\n" }, { "path": "demoapps/dsl/CMakeLists.txt", "content": "project(GraphLab)\n\n# add_graphlab_executable(gl_server gl_server.cpp)\n\n# add_library(gen_impl SHARED gen_impl.cpp)\n" }, { "path": "demoapps/dsl/gen_impl.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <cmath>\n#include <iostream>\n#include \"graph_typedefs.gen\"\n\nusing namespace graphlab;\n\n#define ALPHA 0.87\n\nextern \"C\" void user_program(user_funs* f) {\n float prev = f->get_vertex_data();\n float neighbors = f->reduce_neighbors(IN_EDGES);\n //neighbors = neighbors/out_edges\n float curr = ALPHA*neighbors + (1-ALPHA);\n f->set_vertex_data(curr);\n float last_change = std::abs(curr - prev);\n std::cout << \"last change was: \" << last_change << std::endl;\n if (last_change > 0.01) {\n\tf->signal_neighbors(OUT_EDGES);\n }\n std::cout.flush();\n}\n\nextern \"C\" void vertex_reduce(vertex_data_type& a, const vertex_data_type& b) {\n a += b;\n}\n\n" }, { "path": "demoapps/dsl/gl_server.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <vector>\n#include <string>\n#include <fstream>\n\n#include <graphlab.hpp>\n#include <graphlab/engine/gl3engine.hpp>\n\n#include <dlfcn.h>\n\nusing namespace graphlab;\n\n#include \"graph_typedefs.gen\"\n\n#define GET_EDGE_DATA 100\n#define GET_VERTEX_DATA 101\n\ntypedef distributed_graph<vertex_data_type, edge_data_type> graph_type;\ntypedef gl3engine<graph_type> engine_type;\n\n\n//here we keep one global copy of the user defined function pointers\nvoid (*user_program)(user_funs*) = NULL;\nvoid (*vertex_reduce)(vertex_data_type&,const vertex_data_type&) = NULL;\nvoid (*edge_reduce)(edge_data_type&,const edge_data_type&) = NULL;\n\n//here are definitions of functions passed to dynamically linked user code\ngraph_type::vertex_data_type get_vertex_data(graph_type::vertex_type* v) {\n return v->data();\n}\nvoid set_vertex_data(graph_type::vertex_data_type& d, graph_type::vertex_type* v) {\n v->data() = d;\n}\n\n// graph_type::edge_data_type edge_get_map(graph_type::edge_type& e) {\n// return e.data();\n// }\ngraph_type::vertex_data_type vertex_get_map(const graph_type::vertex_type& v) {\n return v.data();\n}\n\n// void edge_get_reduce(edge_data_type& ev, const edge_data_type ed) {\n// edge_reduce(ev,ed);\n// }\n\nvoid vertex_get_reduce(vertex_data_type& ev, const vertex_data_type ed) {\n vertex_reduce(ev,ed);\n}\n\n//wonder why this isn't working. hm...\n// void edge_get_reduce(std::vector<graph_type::edge_data_type>& ev, const graph_type::edge_data_type ed) {\n// ev.push_back(ed);\n// }\n\n// void vertex_get_reduce(std::vector<graph_type::vertex_data_type>& vv, const graph_type::edge_data_type vd) {\n// vv.push_back(vd);\n// }\n\n// std::vector<edge_data_type> get_neighboring_edges() {\n// return current_context->map_reduce<std::vector<edge_data_type> >(GET_EDGE_DATA, ALL_EDGES);\n// }\n// std::vector<vertex_data_type> get_neighboring_vertices() {\n// return current_context->map_reduce<std::vector<vertex_data_type> >(GET_VERTEX_DATA, ALL_EDGES);\n// }\n\nvertex_data_type reduce_neighbors(edge_dir_type d, engine_type::context_type* ctx) {\n return ctx->map_reduce<vertex_data_type>(GET_VERTEX_DATA, d);\n}\n\nvoid signal_neighbors(edge_dir_type d, engine_type::context_type* ctx) {\n ctx->broadcast_signal(d);\n}\n\n// edge_data_type reduce_edges() {\n// return current_context->map_reduce<edge_data_type>(GET_EDGE_DATA, ALL_EDGES);\n// }\n\nvoid server_program(engine_type::context_type& context,\n\t\t graph_type::vertex_type& vertex,\n\t\t const engine_type::message_type& unused) {\n\n\n\n //capture functions for user to call\n user_funs f;\n f.vtx = &vertex;\n f.ctx = &context;\n f._get_vertex_data = (vertex_data_type (*)(void*))get_vertex_data;\n f._set_vertex_data = (void (*)(vertex_data_type&,void*))set_vertex_data;\n f._reduce_neighbors = (vertex_data_type (*)(edge_dir_type,void*))reduce_neighbors;\n f._signal_neighbors = (void (*)(edge_dir_type,void*))signal_neighbors;\n //f.reduce_edges = reduce_edges;\n\n //call dynamically linked user function\n user_program(&f);\n \n // float prev = vertex.data();\n // // map reduce over neighbors\n // vertex.data() = 0.15 + 0.85 *\n // \tcontext.map_reduce<float>(PAGERANK_MAP_REDUCE, IN_EDGES);\n\n // float last_change = std::fabs((vertex.data()- prev) / vertex.num_out_edges());\n // if (last_change > TOLERANCE) {\n // \t// signals out neighbors if I change substantially\n // \tcontext.broadcast_signal(OUT_EDGES);\n // }\n}\n\n\n\nvoid init_vertex(graph_type::vertex_type& vertex) { vertex.data() = 1; }\n\nint main(int argc, char** argv) {\n\n // Initialize control plain using mpi\n graphlab::mpi_tools::init(argc, argv);\n graphlab::distributed_control dc;\n global_logger().set_log_level(LOG_INFO);\n\n // Parse command line options -----------------------------------------------\n graphlab::command_line_options clopts(\"PageRank algorithm.\");\n std::string graph_dir;\n std::string format = \"adj\";\n std::string exec_type = \"synchronous\";\n clopts.attach_option(\"graph\", graph_dir,\n \"The graph file. Required \");\n clopts.add_positional(\"graph\");\n clopts.attach_option(\"format\", format,\n \"The graph file format\");\n clopts.attach_option(\"engine\", exec_type, \n \"The engine type synchronous or asynchronous\");\n //clopts.attach_option(\"tol\", TOLERANCE,\n // \"The permissible change at convergence.\");\n std::string saveprefix;\n clopts.attach_option(\"saveprefix\", saveprefix,\n \"If set, will save the resultant pagerank to a \"\n \"sequence of files with prefix saveprefix\");\n //set single threaded\n //clopts.set_ncpus(1);\n if(!clopts.parse(argc, argv)) {\n dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n return EXIT_FAILURE;\n }\n\n if (graph_dir == \"\") {\n dc.cout() << \"Graph not specified. Cannot continue\";\n return EXIT_FAILURE;\n }\n\n\n // Dynamically link the functions\n dc.cout() << \"Starting dynamic linking\" << std::endl;\n void* handle = dlopen(\"/home/emullen/graphlab/graphlab2.2/graphlabapi/debug/demoapps/dsl/libgen_impl.so\", RTLD_LAZY);\n if (handle == NULL) {\n dc.cout() << dlerror() << std::endl;\n assert(handle != NULL);\n }\n\n // pagerank_map = (float (*)(const graphlab::distributed_graph<float, graphlab::empty>::vertex_type&))dlsym(pagerank_handle, \"pagerank_map\");\n // assert(pagerank_map != NULL);\n // pagerank_combine = (void (*)(float&, const float&))dlsym(pagerank_handle, \"pagerank_combine\");\n // assert(pagerank_combine != NULL);\n user_program = (void (*)(user_funs*))dlsym(handle, \"user_program\");\n assert(user_program != NULL);\n vertex_reduce = (void (*)(vertex_data_type&,const vertex_data_type&))dlsym(handle, \"vertex_reduce\");\n assert(vertex_reduce != NULL);\n // edge_reduce = (void (*)(edge_data_type&,const edge_data_type&))dlsym(handle, \"edge_reduce\");\n // assert(edge_reduce != NULL);\n dc.cout() << \"Finished dynamic linking\" << std::endl;\n\n // Build the graph ----------------------------------------------------------\n graph_type graph(dc, clopts);\n dc.cout() << \"Loading graph in format: \"<< format << std::endl;\n graph.load_format(graph_dir, format);\n // must call finalize before querying the graph\n graph.finalize();\n dc.cout() << \"#vertices: \" << graph.num_vertices()\n << \" #edges:\" << graph.num_edges() << std::endl;\n\n // Initialize the vertex data\n graph.transform_vertices(init_vertex);\n\n // Running The Engine -------------------------------------------------------\n engine_type engine(dc, graph, clopts);\n\n // register the map reduce operation before usage\n // Each task registration must have a distinct ID ranging fro 0 to 223\n //engine.register_map_reduce(PAGERANK_MAP_REDUCE,\n // pagerank_map,\n // pagerank_combine);\n // engine.register_map_reduce(GET_EDGE_DATA,\n // \t\t\t edge_get_map,\n // \t\t\t edge_get_reduce);\n engine.register_map_reduce(GET_VERTEX_DATA,\n\t\t\t vertex_get_map,\n\t\t\t vertex_get_reduce);\n\n engine.set_vertex_program(server_program);\n engine.signal_all();\n engine.wait();\n \n // Save the final graph -----------------------------------------------------\n // if (saveprefix != \"\") {\n // graph.save(saveprefix, pagerank_writer(),\n // false, // do not gzip\n // true, // save vertices\n // false); // do not save edges\n // }\n\n dlclose(handle);\n\n // Tear-down communication layer and quit -----------------------------------\n graphlab::mpi_tools::finalize();\n return EXIT_SUCCESS;\n} // End of main\n" }, { "path": "demoapps/dsl/graph_typedefs.gen", "content": "#include <vector>\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <ostream>\n\ntypedef float vertex_data_type;\ntypedef float edge_data_type;\n\ntypedef struct {\n void* ctx; \n void* vtx;\n vertex_data_type get_vertex_data() {\n\treturn _get_vertex_data(vtx);\n }\n void set_vertex_data(vertex_data_type vd) {\n\t_set_vertex_data(vd,vtx);\n }\n vertex_data_type reduce_neighbors(graphlab::edge_dir_type e) {\n\treturn _reduce_neighbors(e,ctx);\n }\n void signal_neighbors(graphlab::edge_dir_type e) {\n\t_signal_neighbors(e,ctx);\n }\n vertex_data_type (*_get_vertex_data)(void*);\n void (*_set_vertex_data)(vertex_data_type&,void*);\n //std::vector<edge_data_type> (*get_neighboring_edges)();\n //std::vector<vertex_data_type> (*get_neighboring_vertices)();\n vertex_data_type (*_reduce_neighbors)(graphlab::edge_dir_type,void*);\n void (*_signal_neighbors)(graphlab::edge_dir_type,void*);\n //edge_data_type (*reduce_edges)();\n} user_funs;\n" }, { "path": "demoapps/dsl/impl.graphlab", "content": "types {\n vertextype float\n edgetype float\n}\n\n\ndef update() {\n //get a local copy of the vertex data\n float prev = get_vertex_data()\n\n //get a sum of the neighbors' values\n //uses vertex_reduce defined below\n float neighbors = reduce_neighbors(IN_EDGES)\n\n //calculate updated value\n float curr = 0.85*neighbors + 0.15\n\n //modify our data\n set_vertex_data(curr)\n\n //calculate change\n float last_change = curr - prev\n if (last_change < 0) {\n last_change = -1 * last_change\n }\n\n //signal neighbors\n if (last_change > 0.01) {\n signal_neighbors(OUT_EDGES)\n }\n}\n\ndef vertex_reduce(a:vertextype,b:vertextype) {\n a += b\n}\n" }, { "path": "demoapps/dsl/scala_impl/dsl.scala", "content": "import scala.util.parsing.combinator.syntactical._\n\n\nobject GLParser extends StandardTokenParsers {\n val keywords = List(\"update\",\"reduce_edges\",\"reduce_vertices\",\"print\")\n val types = List(\"bool\",\"int\",\"float\")\n\n lexical.reserved ++= keywords\n lexical.reserved ++= types\n\n lexical.delimiters ++= List(\"{\",\"}\",\"(\",\")\",\".\",\",\",\"+\",\"-\",\"*\",\"/\",\"<\",\">\",\"=\")\n\n def numLit:Parser[Int] = opt(\"-\") ~ numericLit ^^ {\n case None ~ n => n.toInt\n case Some(_) ~ n => (-1)*n.toInt\n }\n\n\n\n}\n" }, { "path": "demoapps/pagerank/CMakeLists.txt", "content": "project(GraphLab)\n\n\nadd_graphlab_executable(simple_pagerank simple_pagerank.cpp)\n\nadd_graphlab_executable(warp_parfor_pagerank warp_parfor_pagerank.cpp)\n\nadd_graphlab_executable(warp_engine_pagerank warp_engine_pagerank.cpp)\n" }, { "path": "demoapps/pagerank/simple_pagerank.cpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include <vector>\n#include <string>\n#include <fstream>\n\n#include <graphlab.hpp>\n// #include <graphlab/macros_def.hpp>\n\n// Global random reset probability\nfloat RESET_PROB = 0.15;\n\nfloat TOLERANCE = 1.0E-2;\n\n// The vertex data is just the pagerank value (a float)\ntypedef float vertex_data_type;\n\n// There is no edge data in the pagerank application\ntypedef graphlab::empty edge_data_type;\n\n// The graph type is determined by the vertex and edge data types\ntypedef graphlab::distributed_graph<vertex_data_type, edge_data_type> graph_type;\n\n/*\n * A simple function used by graph.transform_vertices(init_vertex);\n * to initialize the vertes data.\n */\nvoid init_vertex(graph_type::vertex_type& vertex) { vertex.data() = 1; }\n\n\n\n/*\n * The factorized page rank update function extends ivertex_program\n * specifying the:\n *\n * 1) graph_type\n * 2) gather_type: float (returned by the gather function). Note\n * that the gather type is not strictly needed here since it is\n * assumed to be the same as the vertex_data_type unless\n * otherwise specified\n *\n * In addition ivertex program also takes a message type which is\n * assumed to be empty. Since we do not need messages no message type\n * is provided.\n *\n * pagerank also extends graphlab::IS_POD_TYPE (is plain old data type)\n * which tells graphlab that the pagerank program can be serialized\n * (converted to a byte stream) by directly reading its in memory\n * representation. If a vertex program does not exted\n * graphlab::IS_POD_TYPE it must implement load and save functions.\n */\nclass pagerank :\n public graphlab::ivertex_program<graph_type, float>,\n public graphlab::IS_POD_TYPE {\n float last_change;\npublic:\n /* Gather the weighted rank of the adjacent page */\n float gather(icontext_type& context, const vertex_type& vertex,\n edge_type& edge) const {\n return ((1.0 - RESET_PROB) / edge.source().num_out_edges()) *\n edge.source().data();\n }\n\n /* Use the total rank of adjacent pages to update this page */\n void apply(icontext_type& context, vertex_type& vertex,\n const gather_type& total) {\n const double newval = total + RESET_PROB;\n last_change = std::fabs(newval - vertex.data());\n vertex.data() = newval;\n }\n\n /* The scatter edges depend on whether the pagerank has converged */\n edge_dir_type scatter_edges(icontext_type& context,\n const vertex_type& vertex) const {\n if (last_change > TOLERANCE) return graphlab::OUT_EDGES;\n else return graphlab::NO_EDGES;\n }\n\n /* The scatter function just signal adjacent pages */\n void scatter(icontext_type& context, const vertex_type& vertex,\n edge_type& edge) const {\n context.signal(edge.target());\n }\n}; // end of factorized_pagerank update functor\n\n\n/*\n * We want to save the final graph so we define a write which will be\n * used in graph.save(\"path/prefix\", pagerank_writer()) to save the graph.\n */\nstruct pagerank_writer {\n std::string save_vertex(graph_type::vertex_type v) {\n std::stringstream strm;\n strm << v.id() << \"\\t\" << v.data() << \"\\n\";\n return strm.str();\n }\n std::string save_edge(graph_type::edge_type e) { return \"\"; }\n}; // end of pagerank writer\n\n\n\nint main(int argc, char** argv) {\n // Initialize control plain using mpi\n graphlab::mpi_tools::init(argc, argv);\n graphlab::distributed_control dc;\n global_logger().set_log_level(LOG_INFO);\n\n // Parse command line options -----------------------------------------------\n graphlab::command_line_options clopts(\"PageRank algorithm.\");\n std::string graph_dir;\n std::string format = \"adj\";\n std::string exec_type = \"synchronous\";\n clopts.attach_option(\"graph\", graph_dir,\n \"The graph file. Required \");\n clopts.add_positional(\"graph\");\n clopts.attach_option(\"format\", format,\n \"The graph file format\");\n clopts.attach_option(\"engine\", exec_type, \n \"The engine type synchronous or asynchronous\");\n clopts.attach_option(\"tol\", TOLERANCE,\n \"The permissible change at convergence.\");\n std::string saveprefix;\n clopts.attach_option(\"saveprefix\", saveprefix,\n \"If set, will save the resultant pagerank to a \"\n \"sequence of files with prefix saveprefix\");\n\n if(!clopts.parse(argc, argv)) {\n dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n return EXIT_FAILURE;\n }\n\n if (graph_dir == \"\") {\n dc.cout() << \"Graph not specified. Cannot continue\";\n return EXIT_FAILURE;\n }\n\n // Build the graph ----------------------------------------------------------\n graph_type graph(dc, clopts);\n dc.cout() << \"Loading graph in format: \"<< format << std::endl;\n graph.load_format(graph_dir, format);\n // must call finalize before querying the graph\n graph.finalize();\n dc.cout() << \"#vertices: \" << graph.num_vertices()\n << \" #edges:\" << graph.num_edges() << std::endl;\n\n // Initialize the vertex data\n graph.transform_vertices(init_vertex);\n\n // Running The Engine -------------------------------------------------------\n graphlab::omni_engine<pagerank> engine(dc, graph, exec_type, clopts);\n engine.signal_all();\n engine.start();\n const float runtime = engine.elapsed_seconds();\n dc.cout() << \"Finished Running engine in \" << runtime\n << \" seconds.\" << std::endl;\n\n // Save the final graph -----------------------------------------------------\n if (saveprefix != \"\") {\n graph.save(saveprefix, pagerank_writer(),\n false, // do not gzip\n true, // save vertices\n false); // do not save edges\n }\n\n // Tear-down communication layer and quit -----------------------------------\n graphlab::mpi_tools::finalize();\n return EXIT_SUCCESS;\n} // End of main\n\n\n\n" }, { "path": "demoapps/pagerank/warp_engine_pagerank.cpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include <vector>\n#include <string>\n#include <fstream>\n\n#include <graphlab.hpp>\n#include <graphlab/engine/warp_engine.hpp>\nusing namespace graphlab;\n\n// The graph type is determined by the vertex and edge data types\ntypedef distributed_graph<float , graphlab::empty> graph_type;\ntypedef warp::warp_engine<graph_type> warp_engine_type;\n\n/*\n * A simple function used by graph.transform_vertices(init_vertex);\n * to initialize the vertes data.\n */\nvoid init_vertex(graph_type::vertex_type& vertex) { vertex.data() = 1; }\n\n\nfloat pagerank_map(graph_type::edge_type edge, graph_type::vertex_type other) {\n return other.data() / other.num_out_edges();\n}\n\n\nvoid signal_neighbor(warp_engine_type::context& context,\n graph_type::edge_type edge, graph_type::vertex_type other) {\n context.signal(other);\n}\n\n\n\nvoid pagerank(warp_engine_type::context& context,\n graph_type::vertex_type vertex) {\n float old_vdata = vertex.data();\n vertex.data() = 0.15 + 0.85 * warp::map_reduce_neighborhood(vertex,\n IN_EDGES,\n pagerank_map);\n if (std::fabs(old_vdata - vertex.data()) > 1E-2) {\n warp::broadcast_neighborhood(context, vertex, OUT_EDGES, signal_neighbor);\n }\n}\n\n\n\n\n\n\n/*\n * We want to save the final graph so we define a write which will be\n * used in graph.save(\"path/prefix\", pagerank_writer()) to save the graph.\n */\nstruct pagerank_writer {\n std::string save_vertex(graph_type::vertex_type v) {\n std::stringstream strm;\n strm << v.id() << \"\\t\" << v.data() << \"\\n\";\n return strm.str();\n }\n std::string save_edge(graph_type::edge_type e) { return \"\"; }\n}; // end of pagerank writer\n\n\nint main(int argc, char** argv) {\n global_logger().set_log_level(LOG_INFO);\n // Initialize control plain using mpi\n mpi_tools::init(argc, argv);\n distributed_control dc;\n\n // Parse command line options -----------------------------------------------\n command_line_options clopts(\"PageRank algorithm.\");\n std::string graph_dir;\n std::string format = \"tsv\";\n clopts.attach_option(\"graph\", graph_dir, \"The graph file. \");\n clopts.attach_option(\"format\", format, \"The graph file format\");\n size_t iterations = 10;\n clopts.attach_option(\"iterations\", iterations,\n \"Number of asynchronous iterations to run\");\n std::string saveprefix;\n clopts.attach_option(\"saveprefix\", saveprefix,\n \"Prefix to save the output pagerank in\");\n\n if(!clopts.parse(argc, argv)) {\n dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n return EXIT_FAILURE;\n }\n\n // Build the graph ----------------------------------------------------------\n graph_type graph(dc, clopts);\n dc.cout() << \"Loading graph in format: \"<< format << std::endl;\n graph.load_format(graph_dir, format);\n // must call finalize before querying the graph\n graph.finalize();\n\n // Initialize the vertex data\n graph.transform_vertices(init_vertex);\n\n timer ti;\n warp_engine_type engine(dc, graph, clopts);\n engine.signal_all();\n engine.set_update_function(pagerank);\n engine.start();\n\n dc.cout() << \"Finished Running in \" << ti.current_time()\n << \" seconds.\" << std::endl;\n\n\n // Save the final graph -----------------------------------------------------\n if (saveprefix != \"\") {\n graph.save(saveprefix, pagerank_writer(),\n false, // do not gzip\n true, // save vertices\n false); // do not save edges\n }\n\n mpi_tools::finalize();\n} // End of main\n\n\n\n\n" }, { "path": "demoapps/pagerank/warp_parfor_pagerank.cpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include <vector>\n#include <string>\n#include <fstream>\n\n#include <graphlab.hpp>\n#include <graphlab/engine/warp_graph_mapreduce.hpp>\n#include <graphlab/engine/warp_parfor_all_vertices.hpp>\n#include <graphlab/engine/warp_graph_transform.hpp>\nusing namespace graphlab;\n\n// The graph type is determined by the vertex and edge data types\ntypedef distributed_graph<float , float> graph_type;\n\n/*\n * A simple function used by graph.transform_vertices(init_vertex);\n * to initialize the vertes data.\n */\nvoid init_vertex(graph_type::vertex_type& vertex) { vertex.data() = 1; }\n\n\nfloat pagerank_map(graph_type::edge_type edge, graph_type::vertex_type other) {\n return other.data() / other.num_out_edges();\n}\n\nvoid pagerank(graph_type::vertex_type vertex) {\n vertex.data() = 0.15 + 0.85 * warp::map_reduce_neighborhood(vertex,\n IN_EDGES,\n pagerank_map);\n}\n\n/*\n * We want to save the final graph so we define a write which will be\n * used in graph.save(\"path/prefix\", pagerank_writer()) to save the graph.\n */\nstruct pagerank_writer {\n std::string save_vertex(graph_type::vertex_type v) {\n std::stringstream strm;\n strm << v.id() << \"\\t\" << v.data() << \"\\n\";\n return strm.str();\n }\n std::string save_edge(graph_type::edge_type e) { return \"\"; }\n}; // end of pagerank writer\n\n\nint main(int argc, char** argv) {\n // Initialize control plain using mpi\n mpi_tools::init(argc, argv);\n distributed_control dc;\n\n // Parse command line options -----------------------------------------------\n command_line_options clopts(\"PageRank algorithm.\");\n std::string graph_dir;\n std::string format = \"tsv\";\n clopts.attach_option(\"graph\", graph_dir, \"The graph file. \");\n clopts.attach_option(\"format\", format, \"The graph file format\");\n size_t iterations = 10;\n clopts.attach_option(\"iterations\", iterations,\n \"Number of asynchronous iterations to run\");\n std::string saveprefix;\n clopts.attach_option(\"saveprefix\", saveprefix,\n \"Prefix to save the output pagerank in\");\n\n if(!clopts.parse(argc, argv)) {\n dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n return EXIT_FAILURE;\n }\n\n // Build the graph ----------------------------------------------------------\n graph_type graph(dc, clopts);\n dc.cout() << \"Loading graph in format: \"<< format << std::endl;\n graph.load_format(graph_dir, format);\n // must call finalize before querying the graph\n graph.finalize();\n\n // Initialize the vertex data\n graph.transform_vertices(init_vertex);\n\n timer ti;\n for (size_t i = 0;i < iterations; ++i) {\n warp::parfor_all_vertices(graph, pagerank);\n std::cout << \"Iteration \" << i << \" complete\\n\";\n }\n\n dc.cout() << \"Finished Running in \" << ti.current_time()\n << \" seconds.\" << std::endl;\n\n\n // Save the final graph -----------------------------------------------------\n if (saveprefix != \"\") {\n graph.save(saveprefix, pagerank_writer(),\n false, // do not gzip\n true, // save vertices\n false); // do not save edges\n }\n\n mpi_tools::finalize();\n} // End of main\n\n\n\n\n" }, { "path": "demoapps/rpc/CMakeLists.txt", "content": "project(GraphLab)\n\nadd_graphlab_executable(rpc_example1 rpc_example1.cpp)\nadd_graphlab_executable(rpc_example2 rpc_example2.cpp)\nadd_graphlab_executable(rpc_example3 rpc_example3.cpp)\nadd_graphlab_executable(rpc_example4 rpc_example4.cpp)\nadd_graphlab_executable(rpc_example5 rpc_example5.cpp)\nadd_graphlab_executable(rpc_example6 rpc_example6.cpp)\nadd_graphlab_executable(rpc_example7 rpc_example7.cpp)\nadd_graphlab_executable(rpc_example8 rpc_example8.cpp)\nadd_graphlab_executable(rpc_example9 rpc_example9.cpp)\n\n#add_graphlab_executable(barrier_test barrier_test.cpp)\nadd_graphlab_executable(dht_performance_test dht_performance_test.cpp)\n\nadd_graphlab_executable(rpc_call_perf_test rpc_call_perf_test.cpp)\n\nadd_graphlab_executable(fiber_future_test fiber_future_test.cpp)\nadd_graphlab_executable(obj_fiber_future_test obj_fiber_future_test.cpp)\n" }, { "path": "demoapps/rpc/dht_performance_test.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <iostream>\n#include <graphlab/util/timer.hpp>\n#include <graphlab/util/mpi_tools.hpp>\n#include <graphlab/util/generics/any.hpp>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_init_from_mpi.hpp> \n#include <graphlab/rpc/dht.hpp>\n#include <graphlab/logger/logger.hpp>\nusing namespace graphlab;\n\nstd::string randstring(size_t len) {\n std::string str;\n str.resize(len);\n const char *charset=\"ab\";\n size_t charsetlen = 64;\n for (size_t i = 0;i < len; ++i) {\n str[i] = charset[rand() % charsetlen];\n }\n return str;\n}\n\nint main(int argc, char ** argv) {\n mpi_tools::init(argc, argv);\n distributed_control dc;\n \n std::cout << \"I am machine id \" << dc.procid() \n << \" in \" << dc.numprocs() << \" machines\"<<std::endl;\n dht<std::string, std::string> testdht(dc);\n dc.barrier(); \n std::vector<std::pair<std::string, std::string> > data;\n const size_t NUMSTRINGS = 10000;\n const size_t strlen[4] = {16, 128, 1024, 10240};\n // fill rate\n for (size_t l = 0; l < 4; ++l) {\n timer ti;\n if (dc.procid() == 0) {\n std::cout << \"String Length = \" << strlen[l] << std::endl;\n data.clear();\n for (size_t i = 0;i < NUMSTRINGS; ++i) {\n data.push_back(std::make_pair(randstring(8), randstring(strlen[l])));\n }\n std::cout << \"10k random strings generated\" << std::endl;\n std::cout << \"Starting set\" << std::endl;\n ti.start();\n for (size_t i = 0;i < NUMSTRINGS; ++i) {\n testdht.set(data[i].first, data[i].second);\n if (i % 100000 == 0) {\n std::cout << \".\";\n std::cout.flush();\n }\n }\n std::cout << \"10k insertions in \" << ti.current_time();\n }\n dc.full_barrier();\n if (dc.procid() == 0) std::cout << \"\\t\" << ti.current_time() << \" \" << double(strlen[l]*NUMSTRINGS)/ti.current_time()/1024/1024 << std::endl;\n // dc.barrier();\n // get rate\n if (dc.procid() == 0) {\n std::cout << \"Starting get\" << std::endl;\n\n timer ti;\n ti.start();\n for (size_t i = 0;i < NUMSTRINGS; ++i) {\n std::pair<bool, std::string> ret = testdht.get(data[i].first);\n assert(ret.first);\n if (i % 100 == 0) {\n std::cout << \".\";\n std::cout.flush();\n }\n }\n std::cout << \"10k reads in \" << ti.current_time() << std::endl;\n }\n\n if (dc.procid() == 0) {\n std::cout << \"Starting background gets\" << std::endl;\n\n timer ti;\n std::vector<request_future<std::pair<bool, std::string> > > futures;\n futures.resize(NUMSTRINGS);\n ti.start();\n for (size_t i = 0;i < NUMSTRINGS; ++i) {\n futures[i] = testdht.get_future(data[i].first);\n }\n std::cout << \"gets issued.\" << std::endl;\n for (size_t i = 0;i < NUMSTRINGS; ++i) {\n std::pair<bool, std::string> ret = futures[i]();\n if (i % 100 == 0) {\n std::cout << \".\";\n std::cout.flush();\n }\n }\n std::cout << \"10k reads in \" << ti.current_time() << std::endl;\n }\n\n testdht.clear();\n }\n dc.barrier();\n testdht.print_stats();\n mpi_tools::finalize();\n}\n" }, { "path": "demoapps/rpc/fiber_future_test.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <iostream>\n#include <graphlab/util/timer.hpp>\n#include <graphlab/util/mpi_tools.hpp>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/parallel/atomic.hpp>\n#include <graphlab/parallel/fiber_group.hpp>\n#include <graphlab/parallel/fiber_remote_request.hpp>\nusing namespace graphlab;\n\natomic<size_t> complete_count;\n\nsize_t some_remote_function(size_t a) {\n return a;\n}\n\nvoid test_fiber(size_t sequential_count) {\n for (size_t i = 0;i < sequential_count; ++i) {\n request_future<size_t> ret = fiber_remote_request(1, some_remote_function, 1);\n complete_count.inc(ret()); \n }\n}\n\nint main(int argc, char** argv) {\n mpi_tools::init(argc, argv);\n distributed_control dc;\n timer ti;\n // with fibers\n if (dc.procid() == 0) {\n fiber_group group(8192);\n for (int i = 0;i < 1600000; ++i) {\n group.launch(boost::bind(test_fiber, 1));\n if (i % 100000 == 0) std::cout << i << \"\\n\";\n }\n group.join();\n std::cout << \"completed requests: \" << complete_count.value << \" in \" << ti.current_time() << \"\\n\"; \n }\n\n dc.barrier();\n mpi_tools::finalize();\n}\n\n" }, { "path": "demoapps/rpc/obj_fiber_future_test.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <iostream>\n#include <graphlab/util/timer.hpp>\n#include <graphlab/util/mpi_tools.hpp>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/parallel/atomic.hpp>\n#include <graphlab/parallel/fiber_group.hpp>\n#include <graphlab/parallel/fiber_remote_request.hpp>\nusing namespace graphlab;\n\nstruct testclass {\n dc_dist_object<testclass> rmi;\n atomic<size_t> complete_count;\n\n testclass(distributed_control& dc): rmi(dc, this) { }\n\n\n size_t some_remote_function(size_t a) {\n return a;\n }\n\n void test_fiber(size_t sequential_count) {\n for (size_t i = 0;i < sequential_count; ++i) {\n request_future<size_t> ret = object_fiber_remote_request(rmi, 1, &testclass::some_remote_function, 1);\n complete_count.inc(ret()); \n }\n }\n};\n\n\nint main(int argc, char** argv) {\n mpi_tools::init(argc, argv);\n distributed_control dc;\n testclass tc(dc);\n timer ti;\n // with fibers\n if (dc.procid() == 0) {\n fiber_group group(4096);\n for (int i = 0;i < 1600000; ++i) {\n group.launch(boost::bind(&testclass::test_fiber, &tc, 1));\n }\n group.join();\n std::cout << \"completed requests: \" << tc.complete_count.value << \" in \" << ti.current_time() << \"\\n\"; \n }\n\n dc.barrier();\n mpi_tools::finalize();\n}\n\n" }, { "path": "demoapps/rpc/rpc_call_perf_test.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_dist_object.hpp>\n#include <graphlab/rpc/dc_init_from_mpi.hpp>\n#include <graphlab/util/mpi_tools.hpp>\n#include <graphlab/util/timer.hpp>\nusing namespace graphlab;\n\n#define SEND_LIMIT (64 * 1024 * 1024)\n#define SEND_LIMIT_PRINT \"64MB\"\nstruct teststruct {\n\n dc_dist_object<teststruct> rmi;\n teststruct(distributed_control &dc):rmi(dc, this) {\n dc.barrier();\n }\n\n /**\n * Receiver\n */\n\n atomic<size_t> ctr;\n void receive_ints(size_t i0, size_t i1, size_t i2, size_t i3) {\n ctr.inc();\n }\n\n\n void receive_vector(const std::vector<size_t> &s) {\n ctr.inc();\n }\n\n void receive_string(const std::string &s) {\n ctr.inc();\n }\n\n\n /**\n * Short Sends With Remote Call\n */\n\n void perform_short_sends_0(size_t number) {\n for (size_t i = 0;i < number; ++i) {\n rmi.remote_call(1, &teststruct::receive_ints, 100,100,1000,5000000);\n }\n }\n\n void perform_long_sends_0(size_t length, size_t number) {\n std::vector<size_t> v(length, 5000000);\n for (size_t i = 0;i < number; ++i) {\n rmi.remote_call(1, &teststruct::receive_vector, v);\n }\n }\n\n\n void perform_string_sends_0(size_t length, size_t number) {\n std::string s(length, 1);\n for (size_t i = 0;i < number; ++i) {\n rmi.remote_call(1, &teststruct::receive_string, s);\n }\n }\n\n\n\n void print_res(double t1, double t2, double t3) {\n std::cout << \"Calls Sent at \";\n std::cout << SEND_LIMIT / t1 / 1024 / 1024 << \" MB/s\\n\";\n std::cout << \"Receive Completed at \";\n std::cout << SEND_LIMIT / t3 / 1024 / 1024 << \" MB/s\\n\\n\";\n\n }\n void run_short_sends_0() {\n if (rmi.procid() == 1) {\n rmi.full_barrier();\n return;\n }\n timer ti;\n std::cout << \"Single Threaded \" << SEND_LIMIT_PRINT << \" sends, 4 integer blocks\\n\";\n ti.start();\n size_t numsends = SEND_LIMIT / (sizeof(size_t) * 4);\n perform_short_sends_0(numsends);\n double t1 = ti.current_time();\n rmi.dc().flush();\n double t2 = ti.current_time();\n rmi.full_barrier();\n double t3 = ti.current_time();\n print_res(t1,t2,t3);\n }\n\n\n void run_threaded_short_sends_0(size_t numthreads) {\n if (rmi.procid() == 1) {\n rmi.full_barrier();\n return;\n }\n timer ti;\n std::cout << numthreads << \" threaded \" << SEND_LIMIT_PRINT << \" sends, 4 integer blocks\\n\";\n ti.start();\n fiber_group thrgrp;\n size_t numsends = SEND_LIMIT / (sizeof(size_t) * 4 * numthreads);\n for (size_t i = 0; i < numthreads; ++i) {\n fiber_control::affinity_type affinity;\n affinity.clear(); affinity.set_bit(i % fiber_control::get_instance().num_workers());\n thrgrp.launch(boost::bind(&teststruct::perform_short_sends_0, this, numsends), affinity);\n }\n thrgrp.join();\n double t1 = ti.current_time();\n rmi.dc().flush();\n double t2 = ti.current_time();\n rmi.full_barrier();\n double t3 = ti.current_time();\n print_res(t1,t2,t3);\n }\n\n\n\n void run_string_sends_0(size_t length) {\n if (rmi.procid() == 1) {\n rmi.full_barrier();\n return;\n }\n timer ti;\n size_t numsends = SEND_LIMIT / (length);\n std::cout << \"Single Threaded \" << SEND_LIMIT_PRINT <<\" sends, \" << length << \" bytes * \"<< numsends << \"\\n\";\n ti.start();\n size_t rd = rdtsc();\n perform_string_sends_0(length, numsends);\n size_t rd2 = rdtsc();\n std::cout << \"Completed in: \" << ti.current_time() << \" seconds\\n\";\n std::cout << (rd2 - rd) / numsends << \" cycles per call\\n\";\n double t1 = ti.current_time();\n rmi.dc().flush();\n std::cout << \"Flush in: \" << ti.current_time() << \" seconds\\n\";\n double t2 = ti.current_time();\n rmi.full_barrier();\n std::cout << \"Receive Complete in: \" << ti.current_time() << \" seconds\\n\";\n double t3 = ti.current_time();\n print_res(t1,t2,t3);\n }\n\n\n void run_threaded_string_sends_0(size_t length, size_t numthreads) {\n if (rmi.procid() == 1) {\n rmi.full_barrier();\n return;\n }\n timer ti;\n std::cout << numthreads << \" threaded \" << SEND_LIMIT_PRINT <<\" sends, \"\n << length << \" bytes\\n\";\n ti.start();\n size_t numsends = SEND_LIMIT / (length * numthreads);\n size_t rd = rdtsc();\n fiber_group thrgrp;\n for (size_t i = 0; i < numthreads; ++i) {\n fiber_control::affinity_type affinity;\n affinity.clear(); affinity.set_bit(i % fiber_control::get_instance().num_workers());\n thrgrp.launch(boost::bind(&teststruct::perform_string_sends_0, this, length, numsends), affinity);\n }\n thrgrp.join();\n size_t rd2 = rdtsc();\n std::cout << (rd2 - rd) / (numthreads * numsends) << \" cycles per call\\n\";\n double t1 = ti.current_time();\n rmi.dc().flush();\n double t2 = ti.current_time();\n rmi.full_barrier();\n double t3 = ti.current_time();\n print_res(t1,t2,t3);\n }\n\n};\n\n\nint main(int argc, char** argv) {\n // init MPI\n mpi_tools::init(argc, argv);\n distributed_control dc;\n\n if (dc.numprocs() != 2) {\n std::cout << \"Run with exactly 2 MPI nodes.\\n\";\n return 0;\n }\n dc.barrier();\n teststruct ts(dc);\n /*\n ts.run_short_sends_0();\n ts.run_threaded_short_sends_0(2);\n ts.run_threaded_short_sends_0(4);\n ts.run_threaded_short_sends_0(8);\n ts.run_threaded_short_sends_0(16);\n ts.run_short_pod_sends_0();\n ts.run_threaded_short_pod_sends_0(2);\n ts.run_threaded_short_pod_sends_0(4);\n ts.run_threaded_short_pod_sends_0(8);\n ts.run_threaded_short_pod_sends_0(16);\n ts.run_long_sends_0(1024);\n ts.run_threaded_long_sends_0(1024, 2);\n ts.run_threaded_long_sends_0(1024, 4);\n ts.run_threaded_long_sends_0(1024, 8);\n ts.run_threaded_long_sends_0(1024, 16);\n ts.run_long_sends_0(10240);\n ts.run_threaded_long_sends_0(10240, 2);\n ts.run_threaded_long_sends_0(10240, 4);\n ts.run_threaded_long_sends_0(10240, 8);\n ts.run_threaded_long_sends_0(10240, 16);\n */\n for (size_t i = 4; i < 24; ++i) {\n ts.run_string_sends_0(1<<i);\n }\n\n\n ts.run_threaded_string_sends_0(16, 1);\n ts.run_threaded_string_sends_0(16, 2);\n ts.run_threaded_string_sends_0(16, 4);\n ts.run_threaded_string_sends_0(16, 8);\n ts.run_threaded_string_sends_0(16, 16);\n dc.barrier();\n mpi_tools::finalize();\n}\n" }, { "path": "demoapps/rpc/rpc_example1.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <iostream>\n#include <graphlab/rpc/dc.hpp>\nusing namespace graphlab;\n\n\nvoid print(int val) {\n std::cout << val << std::endl;\n}\n\nint add_one(int val) {\n return val + 1;\n}\n\n\nint main(int argc, char ** argv) {\n // init MPI\n global_logger().set_log_level(LOG_INFO);\n mpi_tools::init(argc, argv);\n distributed_control dc;\n \n if (dc.numprocs() != 2) {\n std::cout<< \"RPC Example 1: Basic Synchronous RPC\\n\";\n std::cout << \"Run with exactly 2 MPI nodes.\\n\";\n return 0;\n }\n \n if (dc.procid() == 0) {\n dc.control_call(1, print, 10);\n std::cout << \"5 plus 1 is : \" << dc.remote_request(1, add_one, 5) << std::endl;\n std::cout << \"11 plus 1 is : \" << dc.remote_request(1, add_one, 11) << std::endl;\n }\n dc.barrier();\n // terminate MPI\n mpi_tools::finalize();\n}\n\n" }, { "path": "demoapps/rpc/rpc_example2.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n#include <iostream>\n#include <string>\n#include <sstream>\n#include <vector>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/serialization/serialization_includes.hpp>\nusing namespace graphlab;\n\n\nvoid print(std::string val) {\n std::cout << val << std::endl;\n}\n\nstd::vector<int> add_one(std::vector<int> val) {\n val.push_back(1);\n return val;\n}\n\n\nint main(int argc, char ** argv) {\n // init MPI\n mpi_tools::init(argc, argv);\n distributed_control dc;\n \n if (dc.numprocs() != 2) {\n std::cout<< \"RPC Example 2: Asynchronous RPC with Built-in Serialization\\n\";\n std::cout << \"Run with exactly 2 MPI nodes.\\n\";\n return 0;\n }\n \n dc.barrier();\n if (dc.procid() == 0) {\n dc.remote_call(1, print, \"hello world!\");\n // Create a vector with a few elements\n std::vector<int> vec;\n vec.push_back(1); vec.push_back(2);\n // Call the remote machine \n vec = dc.remote_request(1, add_one, vec);\n \n std::stringstream strm;\n // Print the vector \n for (size_t i = 0; i < vec.size(); ++i) {\n strm << vec[i] << \", \";\n }\n strm << std::endl;\n strm.flush();\n dc.remote_call(1, print, strm.str());\n }\n dc.barrier();\n\n mpi_tools::finalize();\n}\n\n" }, { "path": "demoapps/rpc/rpc_example3.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n\n#include <iostream>\n#include <string>\n#include <map>\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/rpc/dc.hpp>\nusing namespace graphlab;\n\nstruct teststruct: public IS_POD_TYPE{\n int a;\n double b;\n};\n\n\nvoid print(std::map<int, teststruct> &data1, \n std::string data2) {\n std::cout << \"1.a = \" << data1[1].a << std::endl;\n std::cout << \"10.b = \" << data1[10].b << std::endl;\n std::cout << \"string = \" << data2 << std::endl;\n}\n\n\n\nint main(int argc, char ** argv) {\n // init MPI\n mpi_tools::init(argc, argv);\n distributed_control dc;\n\n if (dc.numprocs() != 2) {\n std::cout<< \"RPC Example 3: Asynchronous RPC with Struct POD Serialization\\n\";\n std::cout << \"Run with exactly 2 MPI nodes.\\n\";\n return 0;\n } \n \n if (dc.procid() == 0) {\n std::map<int, teststruct> data;\n data[1].a = 10;\n data[2].b = 15.0;\n dc.remote_call(1, print, data, \"hello world!\");\n }\n dc.barrier();\n\n mpi_tools::finalize();\n}\n\n" }, { "path": "demoapps/rpc/rpc_example4.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <iostream>\n#include <string>\n#include <map>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/serialization/serialization_includes.hpp>\n\nusing namespace graphlab;\n\nstruct teststruct{\n int a;\n double b;\n std::vector<std::string> name;\n \n void save(oarchive &arc) const {\n arc << a << b << name;\n };\n void load(iarchive &arc) {\n arc >> a >> b >> name;\n };\n};\n\n\nvoid print(std::map<int, teststruct> &data1, \n std::string data2) {\n std::cout << \"1.a = \" << data1[1].a << std::endl;\n std::cout << \"2.b = \" << data1[2].b << std::endl;\n std::cout << \"string = \" << data2 << std::endl;\n}\n\n\n\nint main(int argc, char ** argv) {\n mpi_tools::init(argc, argv);\n distributed_control dc;\n\n if (dc.numprocs() != 2) {\n std::cout<< \"RPC Example 4: Asynchronous RPC with Manual Serialization\\n\";\n std::cout << \"Run with exactly 2 MPI nodes.\\n\";\n return 0;\n }\n \n if (dc.procid() == 0) {\n std::map<int, teststruct> data;\n data[1].a = 10;\n data[2].b = 15.0;\n dc.remote_call(1, print, data, \"hello world!\");\n }\n dc.barrier();\n \n mpi_tools::finalize();\n}\n" }, { "path": "demoapps/rpc/rpc_example5.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n#include <iostream>\n#include <cstdio>\n#include <graphlab/rpc/dc.hpp>\nusing namespace graphlab;\n\n\nint main(int argc, char ** argv) {\n mpi_tools::init(argc, argv);\n distributed_control dc;\n\n if (dc.numprocs() != 2) {\n\n std::cout<< \"RPC Example 5: Asynchronous RPC to printf \\n\";\n std::cout << \"Run with exactly 2 MPI nodes.\\n\";\n return 0;\n }\n \n if (dc.procid() == 0) {\n dc.remote_call(1, printf, \"%d + %f = %s\\n\", 1, 2.0, \"three\");\n }\n dc.barrier();\n \n mpi_tools::finalize();\n}\n" }, { "path": "demoapps/rpc/rpc_example6.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <iostream>\n#include <string>\n#include <vector>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/util/generics/any.hpp>\nusing namespace graphlab;\n\n\nvoid print(any val) {\n val.print(std::cout);\n std::cout << std::endl;\n}\n\n\nint main(int argc, char ** argv) {\n mpi_tools::init(argc, argv);\n distributed_control dc;\n\n if (dc.numprocs() != 2) {\n std::cout<< \"RPC Example 6: Asynchronous RPC with any \\n\";\n std::cout << \"Run with exactly 2 MPI nodes.\\n\";\n return 0;\n }\n\n if (dc.procid() == 0) {\n dc.remote_call(1, print, any(15));\n dc.remote_call(1, print, any(10.5));\n dc.remote_call(1, print, any(std::string(\"hello world\"))); \n }\n \n int i = dc.procid() == 0 ? 10 : 100;\n dc.broadcast(i, dc.procid() == 0);\n std::cout << i << std::endl;\n assert(i == 10);\n \n mpi_tools::finalize();\n}\n" }, { "path": "demoapps/rpc/rpc_example7.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <iostream>\n#include <cstdio>\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_dist_object.hpp>\nusing namespace graphlab;\n\ntemplate <typename T>\nclass distributed_vector {\n private:\n dc_dist_object<distributed_vector<T> > rmi; // The local RMI object\n std::map<size_t, T> data; // storage\n mutex lock; // protect the storage\n public:\n distributed_vector(distributed_control &dc):rmi(dc, this) { };\n \n ///Reads the value at key i\n T get(size_t i) {\n // find the owning machine\n procid_t owningmachine = i % rmi.dc().numprocs();\n \n if (owningmachine == rmi.dc().procid()) {\n // if I own the data. just read and return it\n T ret;\n lock.lock();\n ret = data[i];\n lock.unlock();\n return ret;\n }\n else {\n // otherwise I need to go to another machine\n return rmi.remote_request(owningmachine, \n &distributed_vector<T>::get, \n i);\n }\n }\n \n /// Sets the value at key i\n void set(size_t i, const T& val) {\n // find the owning machine\n procid_t owningmachine = i % rmi.dc().numprocs();\n \n if (owningmachine == rmi.dc().procid()) {\n // if I own the data set it\n lock.lock();\n data[i] = val;\n lock.unlock();\n }\n else {\n // forward the write to another machine\n rmi.remote_request(owningmachine, \n &distributed_vector<T>::set, \n i, \n val);\n }\n }\n};\n\nint main(int argc, char ** argv) {\n mpi_tools::init(argc, argv);\n distributed_control dc;\n\n if (dc.numprocs() != 2) {\n std::cout<< \"RPC Example 7: Distributed Object\\n\";\n std::cout << \"Run with exactly 2 MPI nodes.\\n\";\n return 0;\n }\n \n size_t i = 10;\n dc.all_reduce(i);\n std::cout << i << \"\\n\";\n // create a distributed vector\n distributed_vector<std::string> vec(dc);\n dc.barrier();\n if (dc.procid() == 0) {\n vec.set(10, \"set from 0\");\n vec.set(11, \"set from 0\");\n }\n else {\n vec.set(1, \"set from 1\");\n vec.set(2, \"set from 1\");\n }\n dc.barrier();\n if (dc.procid() == 0) {\n std::cout << vec.get(1) << \"\\n\"; \n std::cout << vec.get(2) << \"\\n\"; \n std::cout << vec.get(10) << \"\\n\";\n std::cout << vec.get(11) << std::endl;\n }\n dc.barrier();\n if (dc.procid() == 1) {\n std::cout << vec.get(1) << \"\\n\"; \n std::cout << vec.get(2) << \"\\n\"; \n std::cout << vec.get(10) << \"\\n\";\n std::cout << vec.get(11) << std::endl;\n }\n dc.barrier();\n \n mpi_tools::finalize();\n}\n" }, { "path": "demoapps/rpc/rpc_example8.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include <vector>\n#include <iostream>\n#include <graphlab/rpc/dc.hpp>\nusing namespace graphlab;\n\n\nvoid print(distributed_control &dc, procid_t caller, int val) {\n std::cout << dc.procid() << \": Receiving print with value : \" << val << std::endl;\n}\n\n\n\nint main(int argc, char ** argv) {\n mpi_tools::init(argc, argv);\n distributed_control dc;\n\n if (dc.numprocs() != 4) {\n std::cout<< \"RPC Example 8: Basic Broadcast Test\\n\";\n std::cout << \"Run with exactly 4 MPI nodes.\\n\";\n return 0;\n }\n \n if (dc.procid() == 0) {\n std::cout << \"First set of calls... Proc 1 and 3 should receive\" << std::endl;\n std::vector<procid_t> s;\n s.push_back(1); s.push_back(3);\n dc.remote_call(s.begin(), s.end(), print, 1);\n }\n dc.full_barrier();\n \n if (dc.procid() == 0) {\n std::cout << \"Second set of calls... Proc 0 and 2 should receive\" << std::endl;\n std::vector<procid_t> s;\n s.push_back(2); s.push_back(0);\n dc.remote_call(s.begin(), s.end(), print, 1);\n }\n dc.full_barrier();\n // terminate MPI\n mpi_tools::finalize();\n}\n" }, { "path": "demoapps/rpc/rpc_example9.cpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include <vector>\n#include <iostream>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/serialization/iarchive.hpp>\nusing namespace graphlab;\n\n\nstruct test_struct {\n dc_dist_object<test_struct> rmi;\n test_struct(distributed_control &dc):rmi(dc, this) {\n dc.barrier();\n }\n\n void test_blob(size_t len, wild_pointer w) {\n assert(len == sizeof(procid_t));\n std::cout << \"split call from : \" << *reinterpret_cast<const procid_t*>(w.ptr) << \"\\n\";\n }\n\n void print(int val) {\n std::cout << rmi.procid() << \": Receiving print with value : \" << val << std::endl;\n }\n\n void test() {\n if (rmi.procid() == 0) {\n std::cout << \"First set of calls... Proc 1 and 3 should receive\" << std::endl;\n std::vector<procid_t> s;\n s.push_back(1); s.push_back(3);\n rmi.remote_call(s.begin(), s.end(), &test_struct::print, 1);\n }\n rmi.full_barrier();\n\n if (rmi.procid() == 0) {\n std::cout << \"Second set of calls... Proc 0 and 2 should receive\" << std::endl;\n std::vector<procid_t> s;\n s.push_back(2); s.push_back(0);\n rmi.remote_call(s.begin(), s.end(), &test_struct::print, 1);\n }\n rmi.full_barrier();\n\n oarchive* oarc = rmi.split_call_begin(&test_struct::test_blob);\n (*oarc) << rmi.procid();\n rmi.split_call_end(1, oarc);\n rmi.full_barrier();\n }\n};\n\nint main(int argc, char ** argv) {\n mpi_tools::init(argc, argv);\n distributed_control dc;\n\n if (dc.numprocs() != 4) {\n std::cout<< \"RPC Example 8: Basic Broadcast Test\\n\";\n std::cout << \"Run with exactly 4 MPI nodes.\\n\";\n return 0;\n }\n test_struct ts(dc);\n ts.test();\n\n // terminate MPI\n mpi_tools::finalize();\n}\n" }, { "path": "dist/README", "content": "This folder contains the distribution files created by the make_dist\nscript in the project root." }, { "path": "doc/README", "content": "To generate docs, run \"doxygen\" in the graphlab/ directory\nDoxygen >= 1.8 recommended\n\n" }, { "path": "doc/images/cycle_triangle.dot", "content": "digraph g{\n A [label=\"A\",style=filled, color=green, shape=circle];\n b [label=\"\", shape=circle];\n c [label=\"\", shape=circle];\n { rank=same; b; c}\n nodesep=0.6;\n A->b ;\n A->c [dir=back];\n b->c ;\n}\n" }, { "path": "doc/images/example_webgraph.dot", "content": "digraph webgraph {\n a [label=\"1: a.com\"];\n b [label=\"4: b.org\"];\n c [label=\"10: c.edu\"];\n d [label=\"11: d.gov\"];\n a->b;\n a->c;\n b->c;\n c->d;\n c->a;\n d->c;\n}\n" }, { "path": "doc/images/graph_format_example.dot", "content": "digraph webgraph {\n a [label=\"1\"];\n b [label=\"2\"];\n c [label=\"5\"];\n d [label=\"7\"];\n a->b;\n a->c;\n c->d;\n d->a;\n d->c;\n}\n" }, { "path": "doc/images/in_triangle.dot", "content": "digraph g{\n A [label=\"A\",style=filled, color=green, shape=circle];\n b [label=\"\", shape=circle];\n c [label=\"\", shape=circle];\n { rank=same; b; c}\n nodesep=0.6;\n A->b [dir=back];\n A->c [dir=back];\n b->c [dir=none,style=dashed];\n}\n" }, { "path": "doc/images/out_triangle.dot", "content": "digraph g{\n A [label=\"A\",style=filled, color=green, shape=circle];\n b [label=\"\", shape=circle];\n c [label=\"\", shape=circle];\n { rank=same; b; c}\n nodesep=0.6;\n A->b;\n A->c;\n b->c [dir=none,style=dashed];\n}\n" }, { "path": "doc/images/through_triangle.dot", "content": "digraph g{\n A [label=\"A\",style=filled, color=green, shape=circle];\n b [label=\"\", shape=circle];\n c [label=\"\", shape=circle];\n { rank=same; b; c}\n nodesep=0.6;\n A->b ;\n A->c [dir=back];\n c->b ;\n}\n" }, { "path": "license/LICENSE.txt", "content": "\n Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n" }, { "path": "license/LICENSE_prepend.txt", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n" }, { "path": "license/corporate_CLA.txt", "content": " Carnegie Mellon University \n Corporate Contributor License Agreement\n based on \n The Apache Software Foundation\nSoftware Grant and Corporate Contributor License Agreement (\"Agreement\")\n http://www.apache.org/licenses/\n (v r190612)\n\n\nThank you for your interest in GraphLab. In order to clarify the\nintellectual property license granted with Contributions from any\nperson or entity, Carnegie Mellon University (\"CMU\") must have a\nContributor License Agreement (CLA) on file that has been signed by\neach Contributor, indicating agreement to the license terms\nbelow. This license is for your protection as a Contributor as well as\nthe protection of CMU and its users; it does not change your rights to\nuse your own Contributions for any other purpose.\n\nThis version of the Agreement allows an entity (the \"Corporation\") to\nsubmit Contributions to CMU, to authorize Contributions submitted by\nits designated employees to CMU, and to grant copyright and patent\nlicenses thereto.\n\nIf you have not already done so, please complete and sign, then scan\nand email a pdf file of this Agreement to support-graphlab@cs.cmu.edu.\nAlternatively, you may send it by facsimile to +1-412-268-2205. If\nnecessary, send an original signed Agreement to:\n\n Carlos Guestrin\n 6105 Gates Hillman Complex \n Machine Learning Department \n Carnegie Mellon University \n 5000 Forbes Avenue \n Pittsburgh, PA 15213, U.S.A.\n\nPlease read this document carefully before signing and keep a copy for\nyour records.\n\n\n Corporation name: ________________________________________________\n\n Corporation address: ________________________________________________\n\n ________________________________________________\n\n ________________________________________________\n\n Point of Contact: ________________________________________________\n\n E-Mail: ________________________________________________\n\n Telephone: _____________________ Fax: _____________________\n\n\nYou accept and agree to the following terms and conditions for Your\npresent and future Contributions submitted to CMU. In return, CMU\nshall not use Your Contributions in a way that is contrary to the\npublic benefit or inconsistent with its nonprofit status and bylaws in\neffect at the time of the Contribution. Except for the license granted\nherein to CMU and recipients of software distributed by CMU, You\nreserve all right, title, and interest in and to Your Contributions.\n\n1. Definitions.\n\n \"You\" (or \"Your\") shall mean the copyright owner or legal entity\n authorized by the copyright owner that is making this Agreement\n with CMU. For legal entities, the entity making a Contribution and\n all other entities that control, are controlled by, or are under\n common control with that entity are considered to be a single\n Contributor. For the purposes of this definition, \"control\" means\n (i) the power, direct or indirect, to cause the direction or\n management of such entity, whether by contract or otherwise, or\n (ii) ownership of fifty percent (50%) or more of the outstanding\n shares, or (iii) beneficial ownership of such entity.\n\n \"Contribution\" shall mean the code, documentation or other original\n works of authorship expressly identified in Schedule B, as well as\n any original work of authorship, including any modifications or\n additions to an existing work, that is intentionally submitted by\n You to CMU for inclusion in, or documentation of, any of the\n products owned or managed by CMU (the \"Work\"). For the purposes of\n this definition, \"submitted\" means any form of electronic, verbal,\n or written communication sent to CMU or its representatives,\n including but not limited to communication on electronic mailing\n lists, source code control systems, and issue tracking systems that\n are managed by, or on behalf of, CMU for the purpose of discussing\n and improving the Work, but excluding communication that is\n conspicuously marked or otherwise designated in writing by You as\n \"Not a Contribution.\"\n\n2. Grant of Copyright License. Subject to the terms and conditions of\n this Agreement, You hereby grant to CMU and to recipients of\n software distributed by CMU a perpetual, worldwide, non-exclusive,\n no-charge, royalty-free, irrevocable copyright license to\n reproduce, prepare derivative works of, publicly display, publicly\n perform, sublicense, and distribute Your Contributions and such\n derivative works.\n\n3. Grant of Patent License. Subject to the terms and conditions of\n this Agreement, You hereby grant to CMU and to recipients of\n software distributed by CMU a perpetual, worldwide, non-exclusive,\n no-charge, royalty-free, irrevocable (except as stated in this\n section) patent license to make, have made, use, offer to sell,\n sell, import, and otherwise transfer the Work, where such license\n applies only to those patent claims licensable by You that are\n necessarily infringed by Your Contribution(s) alone or by\n combination of Your Contribution(s) with the Work to which such\n Contribution(s) were submitted. If any entity institutes patent\n litigation against You or any other entity (including a cross-claim\n or counterclaim in a lawsuit) alleging that your Contribution, or\n the Work to which you have contributed, constitutes direct or\n contributory patent infringement, then any patent licenses granted\n to that entity under this Agreement for that Contribution or Work\n shall terminate as of the date such litigation is filed.\n\n4. You represent that You are legally entitled to grant the above\n license. You represent further that each employee of the\n Corporation designated on Schedule A below (or in a subsequent\n written modification to that Schedule) is authorized to submit\n Contributions on behalf of the Corporation.\n\n5. You represent that each of Your Contributions is Your original\n creation (see section 7 for submissions on behalf of others).\n\n6. You are not expected to provide support for Your Contributions,\n except to the extent You desire to provide support. You may provide\n support for free, for a fee, or not at all. Unless required by\n applicable law or agreed to in writing, You provide Your\n Contributions on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS\n OF ANY KIND, either express or implied, including, without\n limitation, any warranties or conditions of TITLE,\n NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR\n PURPOSE.\n\n7. Should You wish to submit work that is not Your original creation,\n You may submit it to CMU separately from any Contribution,\n identifying the complete details of its source and of any license\n or other restriction (including, but not limited to, related\n patents, trademarks, and license agreements) of which you are\n personally aware, and conspicuously marking the work as \"Submitted\n on behalf of a third-party: [named here]\".\n\n8. It is your responsibility to notify CMU when any change is required\n to the list of designated employees authorized to submit\n Contributions on behalf of the Corporation, or to the Corporation's\n Point of Contact with CMU.\n\n\n\n Please sign: __________________________________ Date: _______________\n\n Title: __________________________________\n\n Corporation: __________________________________\n\n\nSchedule A\n\n [Initial list of designated employees. NB: authorization is not\n tied to particular Contributions.]\n\n\n\n\nSchedule B\n\n [Identification of optional concurrent software grant. Would be\n left blank or omitted if there is no concurrent software grant.]\n\n" }, { "path": "license/individual_CLA.txt", "content": " Carnegie Mellon University \n Individual Contributor License Agreement\n based on \n The Apache Software Foundation\n Individual Contributor License Agreement (\"Agreement\") V2.0\n http://www.apache.org/licenses/\n\nThank you for your interest in GraphLab. In order to clarify the\nintellectual property license granted with Contributions from any\nperson or entity, Carnegie Mellon University (\"CMU\") must have a\nContributor License Agreement (\"CLA\") on file that has been signed by\neach Contributor, indicating agreement to the license terms\nbelow. This license is for your protection as a Contributor as well as\nthe protection of CMU and its users; it does not change your rights to\nuse your own Contributions for any other purpose.\n\nIf you have not already done so, please complete and sign, then scan\nand email a pdf file of this Agreement to support-graphlab@cs.cmu.edu.\nAlternatively, you may send it by facsimile to +1-412-268-2205. If\nnecessary, send an original signed Agreement to:\n\n Carlos Guestrin\n 6105 Gates Hillman Complex \n Machine Learning Department \n Carnegie Mellon University \n 5000 Forbes Avenue \n Pittsburgh, PA 15213, U.S.A.\n\nPlease read this document carefully before signing and keep a copy for\nyour records.\n\n Full name: ______________________________________________________\n\n Mailing Address: ________________________________________________\n\n _________________________________________________________________\n\n Country: ______________________________________________________\n\n Telephone: ______________________________________________________\n\n Facsimile: ______________________________________________________\n\n E-Mail: ______________________________________________________\n \nYou accept and agree to the following terms and conditions for Your\npresent and future Contributions submitted to CMU. In return, CMU\nshall not use Your Contributions in a way that is contrary to the\npublic benefit or inconsistent with its nonprofit status and bylaws in\neffect at the time of the Contribution. Except for the license granted\nherein to CMU and recipients of software distributed by CMU, You\nreserve all right, title, and interest in and to Your Contributions.\n\n1. Definitions.\n\n \"You\" (or \"Your\") shall mean the copyright owner or legal entity\n authorized by the copyright owner that is making this Agreement\n with CMU. For legal entities, the entity making a Contribution and\n all other entities that control, are controlled by, or are under\n common control with that entity are considered to be a single\n Contributor. For the purposes of this definition, \"control\" means\n (i) the power, direct or indirect, to cause the direction or\n management of such entity, whether by contract or otherwise, or\n (ii) ownership of fifty percent (50%) or more of the outstanding\n shares, or (iii) beneficial ownership of such entity.\n\n \"Contribution\" shall mean any original work of authorship,\n including any modifications or additions to an existing work, that\n is intentionally submitted by You to CMU for inclusion in, or\n documentation of, any of the products owned or managed by CMU (the\n \"Work\"). For the purposes of this definition, \"submitted\" means any\n form of electronic, verbal, or written communication sent to CMU or\n its representatives, including but not limited to communication on\n electronic mailing lists, source code control systems, and issue\n tracking systems that are managed by, or on behalf of, CMU for the\n purpose of discussing and improving the Work, but excluding\n communication that is conspicuously marked or otherwise designated\n in writing by You as \"Not a Contribution.\"\n\n2. Grant of Copyright License. Subject to the terms and conditions of\n this Agreement, You hereby grant to CMU and to recipients of\n software distributed by CMU a perpetual, worldwide, non-exclusive,\n no-charge, royalty-free, irrevocable copyright license to\n reproduce, prepare derivative works of, publicly display, publicly\n perform, sublicense, and distribute Your Contributions and such\n derivative works.\n\n3. Grant of Patent License. Subject to the terms and conditions of\n this Agreement, You hereby grant to CMU and to recipients of\n software distributed by CMU a perpetual, worldwide, non-exclusive,\n no-charge, royalty-free, irrevocable (except as stated in this\n section) patent license to make, have made, use, offer to sell,\n sell, import, and otherwise transfer the Work, where such license\n applies only to those patent claims licensable by You that are\n necessarily infringed by Your Contribution(s) alone or by\n combination of Your Contribution(s) with the Work to which such\n Contribution(s) was submitted. If any entity institutes patent\n litigation against You or any other entity (including a cross-claim\n or counterclaim in a lawsuit) alleging that your Contribution, or\n the Work to which you have contributed, constitutes direct or\n contributory patent infringement, then any patent licenses granted\n to that entity under this Agreement for that Contribution or Work\n shall terminate as of the date such litigation is filed.\n\n4. You represent that you are legally entitled to grant the above\n license. If your employer(s) has rights to intellectual property\n that you create that includes your Contributions, you represent\n that you have received permission to make Contributions on behalf\n of that employer, that your employer has waived such rights for\n your Contributions to CMU, or that your employer has executed a\n separate Corporate CLA with CMU.\n\n5. You represent that each of Your Contributions is Your original\n creation (see section 7 for submissions on behalf of others). You\n represent that Your Contribution submissions include complete\n details of any third-party license or other restriction (including,\n but not limited to, related patents and trademarks) of which you\n are personally aware and which are associated with any part of Your\n Contributions.\n\n6. You are not expected to provide support for Your Contributions,\n except to the extent You desire to provide support. You may provide\n support for free, for a fee, or not at all. Unless required by\n applicable law or agreed to in writing, You provide Your\n Contributions on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS\n OF ANY KIND, either express or implied, including, without\n limitation, any warranties or conditions of TITLE, NON-\n INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.\n\n7. Should You wish to submit work that is not Your original creation,\n You may submit it to CMU separately from any Contribution,\n identifying the complete details of its source and of any license\n or other restriction (including, but not limited to, related\n patents, trademarks, and license agreements) of which you are\n personally aware, and conspicuously marking the work as \"Submitted\n on behalf of a third-party: [named here]\".\n\n8. You agree to notify CMU of any facts or circumstances of which you\n become aware that would make these representations inaccurate in\n any respect.\n\n\nPlease sign: __________________________________ Date: ________________\n" }, { "path": "matlab/eventlog_parser.m", "content": "function ret = eventlog_parser(eventlogfile)\nf = fopen(eventlogfile);\nres = textscan(f, '%s %f %f %f %f %f %f', 'Delimiter', '\\t');\nfclose(f);\nnames= res{1};\ntimes = res{2}; \nminimum = res{3};\naverage = res{4};\nmaximum = res{5};\ntotal = res{6};\nrate = res{7};\n\nuniquenames = unique(names);\nnumentries = length(names);\nret = {};\nif (isempty(uniquenames))\n return\nend\n\n\nfor i = 1:length(uniquenames)\n ret{i} = struct('name', [], ...\n 'times', [], ...\n 'minimum', [], ...\n 'average', [], ...\n 'maximum', [], ...\n 'total', [], ...\n 'rate', []);\n\n ret{i}.name = uniquenames{i};\n for j = 1:numentries\n if (strcmp(uniquenames{i}, names{j}))\n ret{i}.times = [ret{i}.times, times(j)];\n ret{i}.minimum = [ret{i}.minimum, minimum(j)];\n ret{i}.average = [ret{i}.average, average(j)];\n ret{i}.maximum = [ret{i}.maximum, maximum(j)];\n ret{i}.total = [ret{i}.total, total(j)];\n ret{i}.rate = [ret{i}.rate , rate(j)];\n end\n end\nend\nend" }, { "path": "patches/boost.patch", "content": "diff -rupN boost_1_53_0_old/boost/cstdint.hpp boost_1_53_0/boost/cstdint.hpp\nindex ea84b65..697b67b 100644\n--- boost_1_53_0_old/boost/cstdint.hpp\n+++ boost_1_53_0/boost/cstdint.hpp\n@@ -39,9 +39,15 @@\n // Note that GLIBC is a bit inconsistent about whether int64_t is defined or not\n // depending upon what headers happen to have been included first...\n // so we disable use of stdint.h when GLIBC does not define __GLIBC_HAVE_LONG_LONG.\n-// See https://svn.boost.org/trac/boost/ticket/3548 and http://sources.redhat.com/bugzilla/show_bug.cgi?id=10990\n+// See issues:\n+// https://svn.boost.org/trac/boost/ticket/3548\n+// http://sources.redhat.com/bugzilla/show_bug.cgi?id=10990\n+// https://svn.boost.org/trac/boost/ticket/8973\n //\n-#if defined(BOOST_HAS_STDINT_H) && (!defined(__GLIBC__) || defined(__GLIBC_HAVE_LONG_LONG))\n+#if defined(BOOST_HAS_STDINT_H) \\\n+ && (!defined(__GLIBC__) \\\n+ || defined(__GLIBC_HAVE_LONG_LONG) \\\n+ || (defined(__GLIBC__) && ((__GLIBC__ > 2) || ((__GLIBC__ == 2) && (__GLIBC_MINOR__ >= 17)))))\n \n // The following #include is an implementation artifact; not part of interface.\n # ifdef __hpux\n\n" }, { "path": "patches/libbz2_fpic.patch", "content": "--- Makefile\t2010-09-10 18:46:02.000000000 -0400\n+++ Makefile\t2012-05-31 13:21:15.211233533 -0400\n@@ -21,7 +21,7 @@\n LDFLAGS=\n \n BIGFILES=-D_FILE_OFFSET_BITS=64\n-CFLAGS=-Wall -Winline -O2 -g $(BIGFILES)\n+CFLAGS=-fpic -fPIC -Wall -Winline -O2 -g $(BIGFILES)\n \n # Where you want it installed when you do 'make install'\n PREFIX=/usr/local\n" }, { "path": "patches/libevent_clean_and_remap.sh", "content": "#!/bin/bash\nif [ $# -ne 1 ]\nthen\n echo \"Missing argument: directory where libevent libraries reside\"\n exit 1\nfi\n\nCURDIR=\"$( cd \"$( dirname \"${BASH_SOURCE[0]}\" )\" && pwd )\"\ncd $1\nrm -f libevent*.so\nobjcopy --redefine-syms=$CURDIR/libevent_remap_file.txt libevent_pthreads.a\nobjcopy --redefine-syms=$CURDIR/libevent_remap_file.txt libevent.a\n\n" }, { "path": "patches/libevent_remap_file.txt", "content": "accept_socket_cb gl_accept_socket_cb\nadvance_last_with_data.isra.5 gl_advance_last_with_data.isra.5\narc4_count gl_arc4_count\narc4rand_lock gl_arc4rand_lock\narc4_seeded_ok gl_arc4_seeded_ok\narc4_stir gl_arc4_stir\narc4_stir_pid gl_arc4_stir_pid\nbe_filter_ctrl gl_be_filter_ctrl\nbe_filter_destruct gl_be_filter_destruct\nbe_filter_disable gl_be_filter_disable\nbe_filter_enable gl_be_filter_enable\nbe_filter_eventcb gl_be_filter_eventcb\nbe_filter_flush gl_be_filter_flush\nbe_filter_process_input gl_be_filter_process_input\nbe_filter_process_output gl_be_filter_process_output\nbe_filter_readcb gl_be_filter_readcb\nbe_filter_writecb gl_be_filter_writecb\nbe_null_filter gl_be_null_filter\nbe_pair_destruct gl_be_pair_destruct\nbe_pair_disable gl_be_pair_disable\nbe_pair_enable gl_be_pair_enable\nbe_pair_flush gl_be_pair_flush\nbe_pair_outbuf_cb gl_be_pair_outbuf_cb\nbe_pair_transfer gl_be_pair_transfer\nbe_socket_adj_timeouts gl_be_socket_adj_timeouts\nbe_socket_ctrl gl_be_socket_ctrl\nbe_socket_destruct gl_be_socket_destruct\nbe_socket_disable gl_be_socket_disable\nbe_socket_enable gl_be_socket_enable\nbe_socket_flush gl_be_socket_flush\nbe_underlying_writebuf_full.isra.2 gl_be_underlying_writebuf_full.isra.2\n_bev_group_random_element.isra.0 gl__bev_group_random_element.isra.0\n_bev_group_refill_callback gl__bev_group_refill_callback\n_bev_group_suspend_reading gl__bev_group_suspend_reading\n_bev_group_suspend_writing gl__bev_group_suspend_writing\n_bev_group_unsuspend_reading gl__bev_group_unsuspend_reading\n_bev_group_unsuspend_writing gl__bev_group_unsuspend_writing\n_bev_refill_callback gl__bev_refill_callback\nbind_socket_ai gl_bind_socket_ai\nbind_socket gl_bind_socket\nbracket_addr_ok gl_bracket_addr_ok\n_bufferevent_add_event gl__bufferevent_add_event\nbufferevent_add_to_rate_limit_group gl_bufferevent_add_to_rate_limit_group\nbufferevent_base_set gl_bufferevent_base_set\nbufferevent_connect_getaddrinfo_cb gl_bufferevent_connect_getaddrinfo_cb\n_bufferevent_decref_and_unlock gl__bufferevent_decref_and_unlock\nbufferevent_decref gl_bufferevent_decref\n_bufferevent_decrement_read_buckets gl__bufferevent_decrement_read_buckets\nbufferevent_decrement_read_limit gl_bufferevent_decrement_read_limit\n_bufferevent_decrement_write_buckets gl__bufferevent_decrement_write_buckets\nbufferevent_decrement_write_limit gl_bufferevent_decrement_write_limit\n_bufferevent_del_generic_timeout_cbs gl__bufferevent_del_generic_timeout_cbs\nbufferevent_disable gl_bufferevent_disable\nbufferevent_disable_hard gl_bufferevent_disable_hard\nbufferevent_enable gl_bufferevent_enable\nbufferevent_enable_locking gl_bufferevent_enable_locking\nbufferevent_filtered_outbuf_cb gl_bufferevent_filtered_outbuf_cb\nbufferevent_filter_new gl_bufferevent_filter_new\nbufferevent_flush gl_bufferevent_flush\nbufferevent_free gl_bufferevent_free\n_bufferevent_generic_adj_timeouts gl__bufferevent_generic_adj_timeouts\nbufferevent_generic_read_timeout_cb gl_bufferevent_generic_read_timeout_cb\nbufferevent_generic_write_timeout_cb gl_bufferevent_generic_write_timeout_cb\nbufferevent_get_base gl_bufferevent_get_base\nbufferevent_get_enabled gl_bufferevent_get_enabled\nbufferevent_getfd gl_bufferevent_getfd\nbufferevent_get_input gl_bufferevent_get_input\nbufferevent_get_max_to_read gl_bufferevent_get_max_to_read\nbufferevent_get_max_to_write gl_bufferevent_get_max_to_write\nbufferevent_get_output gl_bufferevent_get_output\nbufferevent_get_read_limit gl_bufferevent_get_read_limit\n_bufferevent_get_read_max gl__bufferevent_get_read_max\nbufferevent_get_underlying gl_bufferevent_get_underlying\nbufferevent_get_write_limit gl_bufferevent_get_write_limit\n_bufferevent_get_write_max gl__bufferevent_get_write_max\nbufferevent_inbuf_wm_cb gl_bufferevent_inbuf_wm_cb\n_bufferevent_incref_and_lock gl__bufferevent_incref_and_lock\nbufferevent_incref gl_bufferevent_incref\nbufferevent_init_common gl_bufferevent_init_common\n_bufferevent_init_generic_timeout_cbs gl__bufferevent_init_generic_timeout_cbs\nbufferevent_lock gl_bufferevent_lock\nbufferevent_new gl_bufferevent_new\nbufferevent_ops_filter gl_bufferevent_ops_filter\nbufferevent_ops_pair gl_bufferevent_ops_pair\nbufferevent_ops_socket gl_bufferevent_ops_socket\nbufferevent_pair_elt_new gl_bufferevent_pair_elt_new\nbufferevent_pair_get_partner gl_bufferevent_pair_get_partner\nbufferevent_pair_new gl_bufferevent_pair_new\nbufferevent_priority_set gl_bufferevent_priority_set\nbufferevent_rate_limit_group_decrement_read gl_bufferevent_rate_limit_group_decrement_read\nbufferevent_rate_limit_group_decrement_write gl_bufferevent_rate_limit_group_decrement_write\nbufferevent_rate_limit_group_free gl_bufferevent_rate_limit_group_free\nbufferevent_rate_limit_group_get_read_limit gl_bufferevent_rate_limit_group_get_read_limit\nbufferevent_rate_limit_group_get_totals gl_bufferevent_rate_limit_group_get_totals\nbufferevent_rate_limit_group_get_write_limit gl_bufferevent_rate_limit_group_get_write_limit\nbufferevent_rate_limit_group_new gl_bufferevent_rate_limit_group_new\nbufferevent_rate_limit_group_reset_totals gl_bufferevent_rate_limit_group_reset_totals\nbufferevent_rate_limit_group_set_cfg gl_bufferevent_rate_limit_group_set_cfg\nbufferevent_rate_limit_group_set_min_share gl_bufferevent_rate_limit_group_set_min_share\nbufferevent_read_buffer gl_bufferevent_read_buffer\nbufferevent_readcb gl_bufferevent_readcb\nbufferevent_read gl_bufferevent_read\nbufferevent_remove_from_rate_limit_group gl_bufferevent_remove_from_rate_limit_group\nbufferevent_remove_from_rate_limit_group_internal gl_bufferevent_remove_from_rate_limit_group_internal\nbufferevent_run_deferred_callbacks_locked gl_bufferevent_run_deferred_callbacks_locked\nbufferevent_run_deferred_callbacks_unlocked gl_bufferevent_run_deferred_callbacks_unlocked\n_bufferevent_run_eventcb gl__bufferevent_run_eventcb\n_bufferevent_run_readcb gl__bufferevent_run_readcb\n_bufferevent_run_writecb gl__bufferevent_run_writecb\nbufferevent_setcb gl_bufferevent_setcb\nbufferevent_setfd gl_bufferevent_setfd\nbufferevent_set_rate_limit gl_bufferevent_set_rate_limit\nbufferevent_settimeout gl_bufferevent_settimeout\nbufferevent_set_timeouts gl_bufferevent_set_timeouts\nbufferevent_setwatermark gl_bufferevent_setwatermark\nbufferevent_socket_connect gl_bufferevent_socket_connect\nbufferevent_socket_connect_hostname gl_bufferevent_socket_connect_hostname\nbufferevent_socket_get_dns_error gl_bufferevent_socket_get_dns_error\nbufferevent_socket_new gl_bufferevent_socket_new\nbufferevent_socket_outbuf_cb gl_bufferevent_socket_outbuf_cb\nbufferevent_suspend_read gl_bufferevent_suspend_read\nbufferevent_suspend_write gl_bufferevent_suspend_write\nbufferevent_unlock gl_bufferevent_unlock\nbufferevent_unsuspend_read gl_bufferevent_unsuspend_read\nbufferevent_unsuspend_write gl_bufferevent_unsuspend_write\nbufferevent_write_buffer gl_bufferevent_write_buffer\nbufferevent_writecb gl_bufferevent_writecb\nbufferevent_write gl_bufferevent_write\nclient_error_phrases gl_client_error_phrases\ncommon_timeout_callback gl_common_timeout_callback\ncommon_timeout_schedule.isra.23 gl_common_timeout_schedule.isra.23\nCOPY_CHAIN.part.7 gl_COPY_CHAIN.part.7\nCSWTCH.118 gl_CSWTCH.118\nCSWTCH.190 gl_CSWTCH.190\nCSWTCH.365 gl_CSWTCH.365\nCSWTCH.4 gl_CSWTCH.4\ncurrent_base gl_current_base\ndebug_cond_wait gl_debug_cond_wait\ndebug_lock_alloc gl_debug_lock_alloc\ndebug_lock_free gl_debug_lock_free\ndebug_lock_lock gl_debug_lock_lock\ndebug_lock_unlock gl_debug_lock_unlock\ndecode_int64_internal.constprop.0 gl_decode_int64_internal.constprop.0\ndecode_int_internal gl_decode_int_internal\ndecode_tag_internal gl_decode_tag_internal\ndnsname_to_labels gl_dnsname_to_labels\nend_of_path gl_end_of_path\nepoll_apply_one_change.isra.0 gl_epoll_apply_one_change.isra.0\nepoll_dealloc gl_epoll_dealloc\nepoll_dispatch gl_epoll_dispatch\nepoll_init gl_epoll_init\nepoll_nochangelist_add gl_epoll_nochangelist_add\nepoll_nochangelist_del gl_epoll_nochangelist_del\nepollops_changelist gl_epollops_changelist\nepollops gl_epollops\nerror_codes.5989 gl_error_codes.5989\nevbuffer_add_buffer gl_evbuffer_add_buffer\nevbuffer_add_cb gl_evbuffer_add_cb\nevbuffer_add_file gl_evbuffer_add_file\nevbuffer_add gl_evbuffer_add\nevbuffer_add_printf gl_evbuffer_add_printf\nevbuffer_add_reference gl_evbuffer_add_reference\nevbuffer_add_vprintf gl_evbuffer_add_vprintf\nevbuffer_cb_clear_flags gl_evbuffer_cb_clear_flags\nevbuffer_cb_set_flags gl_evbuffer_cb_set_flags\nevbuffer_chain_align gl_evbuffer_chain_align\nevbuffer_chain_insert gl_evbuffer_chain_insert\nevbuffer_chain_new gl_evbuffer_chain_new\n_evbuffer_chain_pin gl__evbuffer_chain_pin\n_evbuffer_chain_unpin gl__evbuffer_chain_unpin\nevbuffer_clear_flags gl_evbuffer_clear_flags\nevbuffer_commit_space gl_evbuffer_commit_space\nevbuffer_copyout gl_evbuffer_copyout\n_evbuffer_decref_and_unlock gl__evbuffer_decref_and_unlock\nevbuffer_defer_callbacks gl_evbuffer_defer_callbacks\nevbuffer_deferred_callback gl_evbuffer_deferred_callback\nevbuffer_drain gl_evbuffer_drain\nevbuffer_enable_locking gl_evbuffer_enable_locking\n_evbuffer_expand_fast gl__evbuffer_expand_fast\nevbuffer_expand gl_evbuffer_expand\nevbuffer_expand_singlechain gl_evbuffer_expand_singlechain\nevbuffer_find_eol_char gl_evbuffer_find_eol_char\nevbuffer_find gl_evbuffer_find\nevbuffer_free gl_evbuffer_free\nevbuffer_freeze gl_evbuffer_freeze\nevbuffer_get_contiguous_space gl_evbuffer_get_contiguous_space\nevbuffer_get_length gl_evbuffer_get_length\n_evbuffer_incref_and_lock gl__evbuffer_incref_and_lock\n_evbuffer_incref gl__evbuffer_incref\nevbuffer_invoke_callbacks gl_evbuffer_invoke_callbacks\nevbuffer_lock gl_evbuffer_lock\nevbuffer_new gl_evbuffer_new\nevbuffer_peek gl_evbuffer_peek\nevbuffer_prepend_buffer gl_evbuffer_prepend_buffer\nevbuffer_prepend gl_evbuffer_prepend\nevbuffer_ptr_memcmp.isra.12 gl_evbuffer_ptr_memcmp.isra.12\nevbuffer_ptr_set gl_evbuffer_ptr_set\nevbuffer_pullup gl_evbuffer_pullup\nevbuffer_read gl_evbuffer_read\nevbuffer_readline gl_evbuffer_readline\nevbuffer_readln gl_evbuffer_readln\n_evbuffer_read_setup_vecs gl__evbuffer_read_setup_vecs\nevbuffer_remove_all_callbacks.isra.4 gl_evbuffer_remove_all_callbacks.isra.4\nevbuffer_remove_buffer gl_evbuffer_remove_buffer\nevbuffer_remove_cb_entry gl_evbuffer_remove_cb_entry\nevbuffer_remove_cb gl_evbuffer_remove_cb\nevbuffer_remove gl_evbuffer_remove\nevbuffer_reserve_space gl_evbuffer_reserve_space\nevbuffer_run_callbacks gl_evbuffer_run_callbacks\nevbuffer_search_eol gl_evbuffer_search_eol\nevbuffer_search gl_evbuffer_search\nevbuffer_search_range gl_evbuffer_search_range\nevbuffer_setcb gl_evbuffer_setcb\nevbuffer_set_flags gl_evbuffer_set_flags\nevbuffer_set_parent gl_evbuffer_set_parent\n_evbuffer_testing_use_linear_file_access gl__evbuffer_testing_use_linear_file_access\n_evbuffer_testing_use_mmap gl__evbuffer_testing_use_mmap\n_evbuffer_testing_use_sendfile gl__evbuffer_testing_use_sendfile\nevbuffer_unfreeze gl_evbuffer_unfreeze\nevbuffer_unlock gl_evbuffer_unlock\nevbuffer_write_atmost gl_evbuffer_write_atmost\nevbuffer_write gl_evbuffer_write\nevbuffer_write_iovec.isra.14 gl_evbuffer_write_iovec.isra.14\nevconnlistener_disable gl_evconnlistener_disable\nevconnlistener_enable gl_evconnlistener_enable\nevconnlistener_event_ops gl_evconnlistener_event_ops\nevconnlistener_free gl_evconnlistener_free\nevconnlistener_get_base gl_evconnlistener_get_base\nevconnlistener_get_fd gl_evconnlistener_get_fd\nevconnlistener_new_bind gl_evconnlistener_new_bind\nevconnlistener_new gl_evconnlistener_new\nevconnlistener_set_cb gl_evconnlistener_set_cb\nevconnlistener_set_error_cb gl_evconnlistener_set_error_cb\nevdns_add_server_port gl_evdns_add_server_port\nevdns_add_server_port_with_base gl_evdns_add_server_port_with_base\nevdns_base_clear_nameservers_and_suspend gl_evdns_base_clear_nameservers_and_suspend\nevdns_base_count_nameservers gl_evdns_base_count_nameservers\nevdns_base_free_and_unlock gl_evdns_base_free_and_unlock\nevdns_base_free gl_evdns_base_free\nevdns_base_load_hosts gl_evdns_base_load_hosts\nevdns_base_nameserver_add gl_evdns_base_nameserver_add\nevdns_base_nameserver_ip_add gl_evdns_base_nameserver_ip_add\nevdns_base_nameserver_sockaddr_add gl_evdns_base_nameserver_sockaddr_add\nevdns_base_new gl_evdns_base_new\nevdns_base_parse_hosts_line.isra.18 gl_evdns_base_parse_hosts_line.isra.18\nevdns_base_resolv_conf_parse gl_evdns_base_resolv_conf_parse\nevdns_base_resolve_ipv4 gl_evdns_base_resolve_ipv4\nevdns_base_resolve_ipv6 gl_evdns_base_resolve_ipv6\nevdns_base_resolve_reverse gl_evdns_base_resolve_reverse\nevdns_base_resolve_reverse_ipv6 gl_evdns_base_resolve_reverse_ipv6\nevdns_base_resume gl_evdns_base_resume\nevdns_base_search_add gl_evdns_base_search_add\nevdns_base_search_clear gl_evdns_base_search_clear\nevdns_base_search_ndots_set gl_evdns_base_search_ndots_set\nevdns_base_set_max_requests_inflight gl_evdns_base_set_max_requests_inflight\nevdns_base_set_option gl_evdns_base_set_option\nevdns_base_set_option_impl gl_evdns_base_set_option_impl\nevdns_cancel_request gl_evdns_cancel_request\nevdns_clear_nameservers_and_suspend gl_evdns_clear_nameservers_and_suspend\nevdns_close_server_port gl_evdns_close_server_port\nevdns_count_nameservers gl_evdns_count_nameservers\nevdns_err_to_string gl_evdns_err_to_string\nevdns_getaddrinfo_cancel gl_evdns_getaddrinfo_cancel\nevdns_getaddrinfo gl_evdns_getaddrinfo\nevdns_getaddrinfo_gotresolve gl_evdns_getaddrinfo_gotresolve\nevdns_getaddrinfo_impl gl_evdns_getaddrinfo_impl\nevdns_getaddrinfo_timeout_cb gl_evdns_getaddrinfo_timeout_cb\nevdns_get_global_base gl_evdns_get_global_base\nevdns_init gl_evdns_init\nevdns_log_fn gl_evdns_log_fn\n_evdns_log gl__evdns_log\nevdns_nameserver_add gl_evdns_nameserver_add\n_evdns_nameserver_add_impl gl__evdns_nameserver_add_impl\nevdns_nameserver_ip_add gl_evdns_nameserver_ip_add\nevdns_request_insert gl_evdns_request_insert\nevdns_request_remove gl_evdns_request_remove\nevdns_requests_pump_waiting_queue gl_evdns_requests_pump_waiting_queue\nevdns_request_timeout_callback gl_evdns_request_timeout_callback\nevdns_request_transmit gl_evdns_request_transmit\nevdns_resolv_conf_parse gl_evdns_resolv_conf_parse\nevdns_resolve_ipv4 gl_evdns_resolve_ipv4\nevdns_resolve_ipv6 gl_evdns_resolve_ipv6\nevdns_resolve_reverse gl_evdns_resolve_reverse\nevdns_resolve_reverse_ipv6 gl_evdns_resolve_reverse_ipv6\nevdns_resume gl_evdns_resume\nevdns_search_add gl_evdns_search_add\nevdns_search_clear gl_evdns_search_clear\nevdns_search_ndots_set gl_evdns_search_ndots_set\nevdns_server_request_add_aaaa_reply gl_evdns_server_request_add_aaaa_reply\nevdns_server_request_add_a_reply gl_evdns_server_request_add_a_reply\nevdns_server_request_add_cname_reply gl_evdns_server_request_add_cname_reply\nevdns_server_request_add_ptr_reply gl_evdns_server_request_add_ptr_reply\nevdns_server_request_add_reply gl_evdns_server_request_add_reply\nevdns_server_request_drop gl_evdns_server_request_drop\nevdns_server_request_format_response gl_evdns_server_request_format_response\nevdns_server_request_get_requesting_addr gl_evdns_server_request_get_requesting_addr\nevdns_server_request_respond gl_evdns_server_request_respond\nevdns_server_request_set_flags gl_evdns_server_request_set_flags\nevdns_set_log_fn gl_evdns_set_log_fn\nevdns_set_option gl_evdns_set_option\nevdns_set_random_bytes_fn gl_evdns_set_random_bytes_fn\nevdns_set_transaction_id_fn gl_evdns_set_transaction_id_fn\nevdns_shutdown gl_evdns_shutdown\nevdns_transmit gl_evdns_transmit\nevent_active gl_event_active\nevent_active_nolock gl_event_active_nolock\nevent_add gl_event_add\nevent_add_internal.part.22 gl_event_add_internal.part.22\nevent_assign gl_event_assign\nevent_base_add_virtual gl_event_base_add_virtual\nevent_base_assert_ok gl_event_base_assert_ok\nevent_base_del_virtual gl_event_base_del_virtual\nevent_base_dispatch gl_event_base_dispatch\nevent_base_dump_events gl_event_base_dump_events\nevent_base_free gl_event_base_free\nevent_base_get_deferred_cb_queue gl_event_base_get_deferred_cb_queue\nevent_base_get_features gl_event_base_get_features\nevent_base_get_method gl_event_base_get_method\nevent_base_gettimeofday_cached gl_event_base_gettimeofday_cached\nevent_base_got_break gl_event_base_got_break\nevent_base_got_exit gl_event_base_got_exit\nevent_base_init_common_timeout gl_event_base_init_common_timeout\nevent_base_loopbreak gl_event_base_loopbreak\nevent_base_loopexit gl_event_base_loopexit\nevent_base_loop gl_event_base_loop\nevent_base_new gl_event_base_new\nevent_base_new_with_config gl_event_base_new_with_config\nevent_base_once gl_event_base_once\nevent_base_priority_init gl_event_base_priority_init\nevent_base_set gl_event_base_set\nevent_base_start_iocp gl_event_base_start_iocp\nevent_base_stop_iocp gl_event_base_stop_iocp\nevent_changelist_add gl_event_changelist_add\nevent_changelist_del gl_event_changelist_del\nevent_changelist_freemem gl_event_changelist_freemem\nevent_changelist_get_or_construct gl_event_changelist_get_or_construct\nevent_changelist_init gl_event_changelist_init\nevent_changelist_remove_all gl_event_changelist_remove_all\nevent_config_avoid_method gl_event_config_avoid_method\nevent_config_free gl_event_config_free\nevent_config_new gl_event_config_new\nevent_config_require_features gl_event_config_require_features\nevent_config_set_flag gl_event_config_set_flag\nevent_config_set_num_cpus_hint gl_event_config_set_num_cpus_hint\nevent_debug_map_HT_CLEAR gl_event_debug_map_HT_CLEAR\nevent_debug_map_HT_GROW gl_event_debug_map_HT_GROW\n_event_debug_map_HT_REP_IS_BAD gl__event_debug_map_HT_REP_IS_BAD\n_event_debug_map_lock gl__event_debug_map_lock\nevent_debug_map_PRIMES gl_event_debug_map_PRIMES\n_event_debug_mode_on gl__event_debug_mode_on\nevent_debug_mode_too_late gl_event_debug_mode_too_late\nevent_debug_unassign gl_event_debug_unassign\n_event_debugx gl__event_debugx\nevent_deferred_cb_cancel gl_event_deferred_cb_cancel\nevent_deferred_cb_init gl_event_deferred_cb_init\nevent_deferred_cb_queue_init gl_event_deferred_cb_queue_init\nevent_deferred_cb_schedule gl_event_deferred_cb_schedule\nevent_del gl_event_del\nevent_del_internal.part.20 gl_event_del_internal.part.20\nevent_dispatch gl_event_dispatch\nevent_enable_debug_mode gl_event_enable_debug_mode\nevent_err gl_event_err\nevent_errx gl_event_errx\nevent_exit gl_event_exit\nevent_free gl_event_free\nevent_get_assignment gl_event_get_assignment\nevent_get_base gl_event_get_base\nevent_get_callback_arg gl_event_get_callback_arg\nevent_get_callback gl_event_get_callback\nevent_get_events gl_event_get_events\nevent_get_fd gl_event_get_fd\nevent_get_method gl_event_get_method\nevent_get_struct_event_size gl_event_get_struct_event_size\nevent_get_supported_methods gl_event_get_supported_methods\nevent_get_version gl_event_get_version\nevent_get_version_number gl_event_get_version_number\nevent_global_current_base_ gl_event_global_current_base_\nevent_global_setup_locks_ gl_event_global_setup_locks_\nevent_init gl_event_init\nevent_initialized gl_event_initialized\nevent_listener_destroy gl_event_listener_destroy\nevent_listener_disable gl_event_listener_disable\nevent_listener_enable gl_event_listener_enable\nevent_listener_getbase gl_event_listener_getbase\nevent_listener_getfd gl_event_listener_getfd\nevent_loopbreak gl_event_loopbreak\nevent_loopexit_cb gl_event_loopexit_cb\nevent_loopexit gl_event_loopexit\nevent_loop gl_event_loop\nevent_mm_calloc_ gl_event_mm_calloc_\nevent_mm_free_ gl_event_mm_free_\nevent_mm_malloc_ gl_event_mm_malloc_\nevent_mm_realloc_ gl_event_mm_realloc_\nevent_mm_strdup_ gl_event_mm_strdup_\nevent_msgx gl_event_msgx\nevent_new gl_event_new\nevent_once_cb gl_event_once_cb\nevent_once gl_event_once\neventops gl_eventops\nevent_pending gl_event_pending\nevent_priority_init gl_event_priority_init\nevent_priority_set gl_event_priority_set\nevent_queue_insert gl_event_queue_insert\nevent_queue_remove gl_event_queue_remove\nevent_reinit gl_event_reinit\nevent_set_fatal_callback gl_event_set_fatal_callback\nevent_set gl_event_set\nevent_set_log_callback gl_event_set_log_callback\nevent_set_mem_functions gl_event_set_mem_functions\nevent_sock_err gl_event_sock_err\nevent_sock_warn gl_event_sock_warn\n_event_strlcpy gl__event_strlcpy\nevent_warn gl_event_warn\nevent_warnx gl_event_warnx\nevhttp_accept_socket gl_evhttp_accept_socket\nevhttp_accept_socket_with_handle gl_evhttp_accept_socket_with_handle\nevhttp_add_header gl_evhttp_add_header\nevhttp_add_header_internal.isra.3 gl_evhttp_add_header_internal.isra.3\nevhttp_add_server_alias gl_evhttp_add_server_alias\nevhttp_add_virtual_host gl_evhttp_add_virtual_host\nevhttp_associate_new_request_with_connection gl_evhttp_associate_new_request_with_connection\nevhttp_bind_listener gl_evhttp_bind_listener\nevhttp_bind_socket gl_evhttp_bind_socket\nevhttp_bind_socket_with_handle gl_evhttp_bind_socket_with_handle\nevhttp_bound_socket_get_fd gl_evhttp_bound_socket_get_fd\nevhttp_bound_socket_get_listener gl_evhttp_bound_socket_get_listener\nevhttp_cancel_request gl_evhttp_cancel_request\nevhttp_clear_headers gl_evhttp_clear_headers\nevhttp_connection_base_new gl_evhttp_connection_base_new\nevhttp_connection_cb_cleanup gl_evhttp_connection_cb_cleanup\nevhttp_connection_cb gl_evhttp_connection_cb\nevhttp_connection_connect gl_evhttp_connection_connect\nevhttp_connection_connect.part.25 gl_evhttp_connection_connect.part.25\nevhttp_connection_done gl_evhttp_connection_done\nevhttp_connection_fail gl_evhttp_connection_fail\nevhttp_connection_free gl_evhttp_connection_free\nevhttp_connection_get_base gl_evhttp_connection_get_base\nevhttp_connection_get_bufferevent gl_evhttp_connection_get_bufferevent\nevhttp_connection_get_peer gl_evhttp_connection_get_peer\nevhttp_connection_new gl_evhttp_connection_new\nevhttp_connection_reset gl_evhttp_connection_reset\nevhttp_connection_retry gl_evhttp_connection_retry\nevhttp_connection_set_base gl_evhttp_connection_set_base\nevhttp_connection_set_closecb gl_evhttp_connection_set_closecb\nevhttp_connection_set_local_address gl_evhttp_connection_set_local_address\nevhttp_connection_set_local_port gl_evhttp_connection_set_local_port\nevhttp_connection_set_max_body_size gl_evhttp_connection_set_max_body_size\nevhttp_connection_set_max_headers_size gl_evhttp_connection_set_max_headers_size\nevhttp_connection_set_retries gl_evhttp_connection_set_retries\nevhttp_connection_set_timeout gl_evhttp_connection_set_timeout\nevhttp_decode_uri gl_evhttp_decode_uri\nevhttp_decode_uri_internal gl_evhttp_decode_uri_internal\nevhttp_deferred_read_cb gl_evhttp_deferred_read_cb\nevhttp_del_accept_socket gl_evhttp_del_accept_socket\nevhttp_del_cb gl_evhttp_del_cb\nevhttp_encode_uri gl_evhttp_encode_uri\nevhttp_error_cb gl_evhttp_error_cb\nevhttp_find_alias gl_evhttp_find_alias\nevhttp_find_header gl_evhttp_find_header\nevhttp_free gl_evhttp_free\nevhttp_get_body gl_evhttp_get_body\nevhttp_handle_request gl_evhttp_handle_request\nevhttp_htmlescape gl_evhttp_htmlescape\nevhttp_is_connection_close gl_evhttp_is_connection_close\nevhttp_is_connection_keepalive gl_evhttp_is_connection_keepalive\nevhttp_make_header gl_evhttp_make_header\nevhttp_make_request gl_evhttp_make_request\nevhttp_new gl_evhttp_new\nevhttp_new_object gl_evhttp_new_object\nevhttp_parse_firstline gl_evhttp_parse_firstline\nevhttp_parse_headers gl_evhttp_parse_headers\nevhttp_parse_query gl_evhttp_parse_query\nevhttp_parse_query_impl gl_evhttp_parse_query_impl\nevhttp_parse_query_str gl_evhttp_parse_query_str\nevhttp_read_body gl_evhttp_read_body\nevhttp_read_cb gl_evhttp_read_cb\nevhttp_read_header gl_evhttp_read_header\nevhttp_read_trailer gl_evhttp_read_trailer\nevhttp_remove_header gl_evhttp_remove_header\nevhttp_remove_server_alias gl_evhttp_remove_server_alias\nevhttp_remove_virtual_host gl_evhttp_remove_virtual_host\nevhttp_request_dispatch gl_evhttp_request_dispatch\nevhttp_request_free gl_evhttp_request_free\nevhttp_request_free.part.23 gl_evhttp_request_free.part.23\nevhttp_request_get_command gl_evhttp_request_get_command\nevhttp_request_get_connection gl_evhttp_request_get_connection\nevhttp_request_get_evhttp_uri gl_evhttp_request_get_evhttp_uri\nevhttp_request_get_host gl_evhttp_request_get_host\nevhttp_request_get_input_buffer gl_evhttp_request_get_input_buffer\nevhttp_request_get_input_headers gl_evhttp_request_get_input_headers\nevhttp_request_get_output_buffer gl_evhttp_request_get_output_buffer\nevhttp_request_get_output_headers gl_evhttp_request_get_output_headers\nevhttp_request_get_response_code gl_evhttp_request_get_response_code\nevhttp_request_get_uri gl_evhttp_request_get_uri\nevhttp_request_is_owned gl_evhttp_request_is_owned\nevhttp_request_new gl_evhttp_request_new\nevhttp_request_own gl_evhttp_request_own\nevhttp_request_set_chunked_cb gl_evhttp_request_set_chunked_cb\nevhttp_response_code gl_evhttp_response_code\nevhttp_send_continue_done gl_evhttp_send_continue_done\nevhttp_send_done gl_evhttp_send_done\nevhttp_send_error gl_evhttp_send_error\nevhttp_send_page gl_evhttp_send_page\nevhttp_send.part.27 gl_evhttp_send.part.27\nevhttp_send_reply_chunk gl_evhttp_send_reply_chunk\nevhttp_send_reply_end gl_evhttp_send_reply_end\nevhttp_send_reply gl_evhttp_send_reply\nevhttp_send_reply_start gl_evhttp_send_reply_start\nevhttp_set_allowed_methods gl_evhttp_set_allowed_methods\nevhttp_set_cb gl_evhttp_set_cb\nevhttp_set_gencb gl_evhttp_set_gencb\nevhttp_set_max_body_size gl_evhttp_set_max_body_size\nevhttp_set_max_headers_size gl_evhttp_set_max_headers_size\nevhttp_set_timeout gl_evhttp_set_timeout\nevhttp_start gl_evhttp_start\nevhttp_start_read gl_evhttp_start_read\nevhttp_uridecode gl_evhttp_uridecode\nevhttp_uriencode gl_evhttp_uriencode\nevhttp_uri_free gl_evhttp_uri_free\nevhttp_uri_get_fragment gl_evhttp_uri_get_fragment\nevhttp_uri_get_host gl_evhttp_uri_get_host\nevhttp_uri_get_path gl_evhttp_uri_get_path\nevhttp_uri_get_port gl_evhttp_uri_get_port\nevhttp_uri_get_query gl_evhttp_uri_get_query\nevhttp_uri_get_scheme gl_evhttp_uri_get_scheme\nevhttp_uri_get_userinfo gl_evhttp_uri_get_userinfo\nevhttp_uri_join gl_evhttp_uri_join\nevhttp_uri_new gl_evhttp_uri_new\nevhttp_uri_parse gl_evhttp_uri_parse\nevhttp_uri_parse_with_flags gl_evhttp_uri_parse_with_flags\nevhttp_uri_set_flags gl_evhttp_uri_set_flags\nevhttp_uri_set_fragment gl_evhttp_uri_set_fragment\nevhttp_uri_set_host gl_evhttp_uri_set_host\nevhttp_uri_set_path gl_evhttp_uri_set_path\nevhttp_uri_set_port gl_evhttp_uri_set_port\nevhttp_uri_set_query gl_evhttp_uri_set_query\nevhttp_uri_set_scheme gl_evhttp_uri_set_scheme\nevhttp_uri_set_userinfo gl_evhttp_uri_set_userinfo\nevhttp_write_buffer.constprop.31 gl_evhttp_write_buffer.constprop.31\nevhttp_write_cb gl_evhttp_write_cb\nevhttp_write_connectioncb gl_evhttp_write_connectioncb\nevmap_check_integrity gl_evmap_check_integrity\nevmap_io_active gl_evmap_io_active\nevmap_io_add gl_evmap_io_add\nevmap_io_clear gl_evmap_io_clear\nevmap_io_del gl_evmap_io_del\nevmap_io_get_fdinfo gl_evmap_io_get_fdinfo\nevmap_io_initmap gl_evmap_io_initmap\nevmap_make_space.constprop.1 gl_evmap_make_space.constprop.1\nevmap_signal_active gl_evmap_signal_active\nevmap_signal_add gl_evmap_signal_add\nevmap_signal_clear gl_evmap_signal_clear\nevmap_signal_del gl_evmap_signal_del\nevmap_signal_initmap gl_evmap_signal_initmap\nevrpc_add_hook gl_evrpc_add_hook\nevrpc_construct_uri gl_evrpc_construct_uri\nevrpc_free gl_evrpc_free\nevrpc_get_reply gl_evrpc_get_reply\nevrpc_get_request gl_evrpc_get_request\nevrpc_hook_add_meta gl_evrpc_hook_add_meta\nevrpc_hook_find_meta gl_evrpc_hook_find_meta\nevrpc_hook_get_connection gl_evrpc_hook_get_connection\nevrpc_hook_meta_new gl_evrpc_hook_meta_new\nevrpc_init gl_evrpc_init\nevrpc_make_request_ctx gl_evrpc_make_request_ctx\nevrpc_make_request gl_evrpc_make_request\nevrpc_meta_data_free gl_evrpc_meta_data_free\nevrpc_pause_request gl_evrpc_pause_request\nevrpc_pool_add_connection gl_evrpc_pool_add_connection\nevrpc_pool_free gl_evrpc_pool_free\nevrpc_pool_new gl_evrpc_pool_new\nevrpc_pool_remove_connection gl_evrpc_pool_remove_connection\nevrpc_pool_schedule gl_evrpc_pool_schedule\nevrpc_pool_set_timeout gl_evrpc_pool_set_timeout\nevrpc_register_generic gl_evrpc_register_generic\nevrpc_register_rpc gl_evrpc_register_rpc\nevrpc_remove_hook gl_evrpc_remove_hook\nevrpc_reply_done_closure gl_evrpc_reply_done_closure\nevrpc_reply_done gl_evrpc_reply_done\nevrpc_reqstate_free gl_evrpc_reqstate_free\nevrpc_request_cb_closure gl_evrpc_request_cb_closure\nevrpc_request_cb gl_evrpc_request_cb\nevrpc_request_done_closure gl_evrpc_request_done_closure\nevrpc_request_done gl_evrpc_request_done\nevrpc_request_get_pool gl_evrpc_request_get_pool\nevrpc_request_set_cb gl_evrpc_request_set_cb\nevrpc_request_set_pool gl_evrpc_request_set_pool\nevrpc_request_timeout gl_evrpc_request_timeout\nevrpc_request_wrapper_free gl_evrpc_request_wrapper_free\nevrpc_resume_request gl_evrpc_resume_request\nevrpc_schedule_request_closure gl_evrpc_schedule_request_closure\nevrpc_schedule_request gl_evrpc_schedule_request\nevrpc_send_request_generic gl_evrpc_send_request_generic\nevrpc_unregister_rpc gl_evrpc_unregister_rpc\nevsig_add gl_evsig_add\nevsig_base_fd gl_evsig_base_fd\nevsig_base gl_evsig_base\nevsig_base_lock gl_evsig_base_lock\nevsig_base_n_signals_added gl_evsig_base_n_signals_added\nevsig_cb gl_evsig_cb\nevsig_dealloc gl_evsig_dealloc\nevsig_del gl_evsig_del\nevsig_global_setup_locks_ gl_evsig_global_setup_locks_\nevsig_handler gl_evsig_handler\nevsig_init gl_evsig_init\nevsigops gl_evsigops\n_evsig_restore_handler gl__evsig_restore_handler\nevsig_set_base gl_evsig_set_base\n_evsig_set_handler gl__evsig_set_handler\nevtag_consume gl_evtag_consume\nevtag_decode_int64 gl_evtag_decode_int64\nevtag_decode_int gl_evtag_decode_int\nevtag_decode_tag gl_evtag_decode_tag\nevtag_encode_int64 gl_evtag_encode_int64\nevtag_encode_int gl_evtag_encode_int\nevtag_encode_tag gl_evtag_encode_tag\nevtag_init gl_evtag_init\nevtag_marshal_buffer gl_evtag_marshal_buffer\nevtag_marshal gl_evtag_marshal\nevtag_marshal_int64 gl_evtag_marshal_int64\nevtag_marshal_int gl_evtag_marshal_int\nevtag_marshal_string gl_evtag_marshal_string\nevtag_marshal_timeval gl_evtag_marshal_timeval\nevtag_payload_length gl_evtag_payload_length\nevtag_peek gl_evtag_peek\nevtag_peek_length gl_evtag_peek_length\nevtag_unmarshal_fixed gl_evtag_unmarshal_fixed\nevtag_unmarshal gl_evtag_unmarshal\nevtag_unmarshal_header gl_evtag_unmarshal_header\nevtag_unmarshal_int64 gl_evtag_unmarshal_int64\nevtag_unmarshal_int gl_evtag_unmarshal_int\nevtag_unmarshal_string gl_evtag_unmarshal_string\nevtag_unmarshal_timeval gl_evtag_unmarshal_timeval\n_evthread_cond_fns gl__evthread_cond_fns\n_evthread_debug_get_real_lock gl__evthread_debug_get_real_lock\nevthread_debug_lock_mark_locked.isra.0 gl_evthread_debug_lock_mark_locked.isra.0\nevthread_debug_lock_mark_unlocked gl_evthread_debug_lock_mark_unlocked\nevthread_enable_lock_debuging gl_evthread_enable_lock_debuging\n_evthread_id_fn gl__evthread_id_fn\n_evthread_is_debug_lock_held gl__evthread_is_debug_lock_held\n_evthread_lock_debugging_enabled gl__evthread_lock_debugging_enabled\n_evthread_lock_fns gl__evthread_lock_fns\nevthread_make_base_notifiable gl_evthread_make_base_notifiable\nevthread_make_base_notifiable.part.28 gl_evthread_make_base_notifiable.part.28\nevthread_notify_base_default gl_evthread_notify_base_default\nevthread_notify_base_eventfd gl_evthread_notify_base_eventfd\nevthread_notify_base gl_evthread_notify_base\nevthread_notify_drain_default gl_evthread_notify_drain_default\nevthread_notify_drain_eventfd gl_evthread_notify_drain_eventfd\nevthread_set_condition_callbacks gl_evthread_set_condition_callbacks\nevthread_set_id_callback gl_evthread_set_id_callback\nevthread_set_lock_callbacks gl_evthread_set_lock_callbacks\nevthread_setup_global_lock_ gl_evthread_setup_global_lock_\nev_token_bucket_cfg_free gl_ev_token_bucket_cfg_free\nev_token_bucket_cfg_new gl_ev_token_bucket_cfg_new\nev_token_bucket_get_tick gl_ev_token_bucket_get_tick\nev_token_bucket_init gl_ev_token_bucket_init\nev_token_bucket_update gl_ev_token_bucket_update\nevutil_addrinfo_append gl_evutil_addrinfo_append\nevutil_adjust_hints_for_addrconfig gl_evutil_adjust_hints_for_addrconfig\nevutil_ascii_strcasecmp gl_evutil_ascii_strcasecmp\nevutil_ascii_strncasecmp gl_evutil_ascii_strncasecmp\nevutil_closesocket gl_evutil_closesocket\nevutil_ersatz_socketpair gl_evutil_ersatz_socketpair\nevutil_format_sockaddr_port gl_evutil_format_sockaddr_port\nevutil_freeaddrinfo gl_evutil_freeaddrinfo\nevutil_gai_strerror gl_evutil_gai_strerror\nevutil_getaddrinfo_async gl_evutil_getaddrinfo_async\nevutil_getaddrinfo_common gl_evutil_getaddrinfo_common\nevutil_getaddrinfo gl_evutil_getaddrinfo\nevutil_getaddrinfo_infer_protocols.isra.0 gl_evutil_getaddrinfo_infer_protocols.isra.0\nevutil_getenv gl_evutil_getenv\nevutil_hex_char_to_int gl_evutil_hex_char_to_int\nevutil_inet_ntop gl_evutil_inet_ntop\nevutil_inet_ntop.part.4 gl_evutil_inet_ntop.part.4\nevutil_inet_pton gl_evutil_inet_pton\nEVUTIL_ISALNUM gl_EVUTIL_ISALNUM\nEVUTIL_ISALNUM_TABLE gl_EVUTIL_ISALNUM_TABLE\nEVUTIL_ISALPHA gl_EVUTIL_ISALPHA\nEVUTIL_ISALPHA_TABLE gl_EVUTIL_ISALPHA_TABLE\nEVUTIL_ISDIGIT gl_EVUTIL_ISDIGIT\nEVUTIL_ISDIGIT_TABLE gl_EVUTIL_ISDIGIT_TABLE\nEVUTIL_ISLOWER gl_EVUTIL_ISLOWER\nEVUTIL_ISLOWER_TABLE gl_EVUTIL_ISLOWER_TABLE\nEVUTIL_ISPRINT gl_EVUTIL_ISPRINT\nEVUTIL_ISPRINT_TABLE gl_EVUTIL_ISPRINT_TABLE\nEVUTIL_ISSPACE gl_EVUTIL_ISSPACE\nEVUTIL_ISSPACE_TABLE gl_EVUTIL_ISSPACE_TABLE\nEVUTIL_ISUPPER gl_EVUTIL_ISUPPER\nEVUTIL_ISUPPER_TABLE gl_EVUTIL_ISUPPER_TABLE\nEVUTIL_ISXDIGIT gl_EVUTIL_ISXDIGIT\nEVUTIL_ISXDIGIT_TABLE gl_EVUTIL_ISXDIGIT_TABLE\nevutil_make_listen_socket_reuseable gl_evutil_make_listen_socket_reuseable\nevutil_make_socket_closeonexec gl_evutil_make_socket_closeonexec\nevutil_make_socket_nonblocking gl_evutil_make_socket_nonblocking\nevutil_new_addrinfo gl_evutil_new_addrinfo\nevutil_open_closeonexec gl_evutil_open_closeonexec\nevutil_parse_sockaddr_port gl_evutil_parse_sockaddr_port\nevutil_read_file gl_evutil_read_file\nevutil_secure_rng_add_bytes gl_evutil_secure_rng_add_bytes\nevutil_secure_rng_get_bytes gl_evutil_secure_rng_get_bytes\nevutil_secure_rng_global_setup_locks_ gl_evutil_secure_rng_global_setup_locks_\nevutil_secure_rng_init gl_evutil_secure_rng_init\nevutil_set_evdns_getaddrinfo_fn gl_evutil_set_evdns_getaddrinfo_fn\nevutil_snprintf gl_evutil_snprintf\nevutil_sockaddr_cmp gl_evutil_sockaddr_cmp\nevutil_sockaddr_is_loopback gl_evutil_sockaddr_is_loopback\nevutil_socket_connect gl_evutil_socket_connect\nevutil_socket_finished_connecting gl_evutil_socket_finished_connecting\nevutil_socketpair gl_evutil_socketpair\nevutil_strtoll gl_evutil_strtoll\nEVUTIL_TOLOWER gl_EVUTIL_TOLOWER\nEVUTIL_TOLOWER_TABLE gl_EVUTIL_TOLOWER_TABLE\nEVUTIL_TOUPPER gl_EVUTIL_TOUPPER\nEVUTIL_TOUPPER_TABLE gl_EVUTIL_TOUPPER_TABLE\nevutil_tv_to_msec gl_evutil_tv_to_msec\nevutil_vsnprintf gl_evutil_vsnprintf\n_evutil_weakrand gl__evutil_weakrand\nfatal_fn gl_fatal_fn\nfilenames.5800 gl_filenames.5800\nfree_getaddrinfo_request gl_free_getaddrinfo_request\n__func__.4053 gl___func__.4053\n__func__.4058 gl___func__.4058\n__func__.4066 gl___func__.4066\n__func__.4071 gl___func__.4071\n__func__.4085 gl___func__.4085\n__func__.4104 gl___func__.4104\n__func__.4489 gl___func__.4489\n__func__.4709 gl___func__.4709\n__func__.4844 gl___func__.4844\n__func__.4877 gl___func__.4877\n__func__.4950 gl___func__.4950\n__func__.5088 gl___func__.5088\n__func__.5479 gl___func__.5479\n__func__.5538 gl___func__.5538\n__func__.5553 gl___func__.5553\n__func__.5556 gl___func__.5556\n__func__.5563 gl___func__.5563\n__func__.5568 gl___func__.5568\n__func__.5605 gl___func__.5605\n__func__.5609 gl___func__.5609\n__func__.5624 gl___func__.5624\n__func__.5634 gl___func__.5634\n__func__.5654 gl___func__.5654\n__func__.5665 gl___func__.5665\n__func__.5670 gl___func__.5670\n__func__.5671 gl___func__.5671\n__func__.5675 gl___func__.5675\n__func__.5679 gl___func__.5679\n__func__.5683 gl___func__.5683\n__func__.5685 gl___func__.5685\n__func__.5692 gl___func__.5692\n__func__.5693 gl___func__.5693\n__func__.5711 gl___func__.5711\n__func__.5713 gl___func__.5713\n__func__.5727 gl___func__.5727\n__func__.5733 gl___func__.5733\n__func__.5743 gl___func__.5743\n__func__.5744 gl___func__.5744\n__func__.5754 gl___func__.5754\n__func__.5775 gl___func__.5775\n__func__.5885 gl___func__.5885\n__func__.5899 gl___func__.5899\n__func__.5905 gl___func__.5905\n__func__.5913 gl___func__.5913\n__func__.5923 gl___func__.5923\n__func__.5938 gl___func__.5938\n__func__.5943 gl___func__.5943\n__func__.5947 gl___func__.5947\n__func__.5972 gl___func__.5972\n__func__.5980 gl___func__.5980\n__func__.5990 gl___func__.5990\n__func__.6040 gl___func__.6040\n__func__.6062 gl___func__.6062\n__func__.6080 gl___func__.6080\n__func__.6128 gl___func__.6128\n__func__.6129 gl___func__.6129\n__func__.6135 gl___func__.6135\n__func__.6154 gl___func__.6154\n__func__.6162 gl___func__.6162\n__func__.6173 gl___func__.6173\n__func__.6177 gl___func__.6177\n__func__.6186 gl___func__.6186\n__func__.6188 gl___func__.6188\n__func__.6192 gl___func__.6192\n__func__.6195 gl___func__.6195\n__func__.6200 gl___func__.6200\n__func__.6206 gl___func__.6206\n__func__.6212 gl___func__.6212\n__func__.6213 gl___func__.6213\n__func__.6214 gl___func__.6214\n__func__.6219 gl___func__.6219\n__func__.6223 gl___func__.6223\n__func__.6234 gl___func__.6234\n__func__.6247 gl___func__.6247\n__func__.6261 gl___func__.6261\n__func__.6263 gl___func__.6263\n__func__.6286 gl___func__.6286\n__func__.6287 gl___func__.6287\n__func__.6294 gl___func__.6294\n__func__.6300 gl___func__.6300\n__func__.6302 gl___func__.6302\n__func__.6309 gl___func__.6309\n__func__.6315 gl___func__.6315\n__func__.6320 gl___func__.6320\n__func__.6325 gl___func__.6325\n__func__.6329 gl___func__.6329\n__func__.6354 gl___func__.6354\n__func__.6358 gl___func__.6358\n__func__.6362 gl___func__.6362\n__func__.6369 gl___func__.6369\n__func__.6374 gl___func__.6374\n__func__.6379 gl___func__.6379\n__func__.6384 gl___func__.6384\n__func__.6404 gl___func__.6404\n__func__.6414 gl___func__.6414\n__func__.6430 gl___func__.6430\n__func__.6450 gl___func__.6450\n__func__.6452 gl___func__.6452\n__func__.6460 gl___func__.6460\n__func__.6467 gl___func__.6467\n__func__.6469 gl___func__.6469\n__func__.6481 gl___func__.6481\n__func__.6489 gl___func__.6489\n__func__.6498 gl___func__.6498\n__func__.6506 gl___func__.6506\n__func__.6509 gl___func__.6509\n__func__.6511 gl___func__.6511\n__func__.6515 gl___func__.6515\n__func__.6516 gl___func__.6516\n__func__.6524 gl___func__.6524\n__func__.6533 gl___func__.6533\n__func__.6539 gl___func__.6539\n__func__.6540 gl___func__.6540\n__func__.6542 gl___func__.6542\n__func__.6548 gl___func__.6548\n__func__.6554 gl___func__.6554\n__func__.6559 gl___func__.6559\n__func__.6560 gl___func__.6560\n__func__.6565 gl___func__.6565\n__func__.6566 gl___func__.6566\n__func__.6572 gl___func__.6572\n__func__.6578 gl___func__.6578\n__func__.6585 gl___func__.6585\n__func__.6594 gl___func__.6594\n__func__.6603 gl___func__.6603\n__func__.6612 gl___func__.6612\n__func__.6616 gl___func__.6616\n__func__.6621 gl___func__.6621\n__func__.6623 gl___func__.6623\n__func__.6624 gl___func__.6624\n__func__.6631 gl___func__.6631\n__func__.6640 gl___func__.6640\n__func__.6663 gl___func__.6663\n__func__.6666 gl___func__.6666\n__func__.6675 gl___func__.6675\n__func__.6680 gl___func__.6680\n__func__.6685 gl___func__.6685\n__func__.6698 gl___func__.6698\n__func__.6701 gl___func__.6701\n__func__.6709 gl___func__.6709\n__func__.6710 gl___func__.6710\n__func__.6714 gl___func__.6714\n__func__.6719 gl___func__.6719\n__func__.6720 gl___func__.6720\n__func__.6726 gl___func__.6726\n__func__.6757 gl___func__.6757\n__func__.6768 gl___func__.6768\n__func__.6787 gl___func__.6787\n__func__.6802 gl___func__.6802\n__func__.6825 gl___func__.6825\n__func__.6831 gl___func__.6831\n__func__.6835 gl___func__.6835\n__func__.6836 gl___func__.6836\n__func__.6839 gl___func__.6839\n__func__.6842 gl___func__.6842\n__func__.6865 gl___func__.6865\n__func__.6880 gl___func__.6880\n__func__.6890 gl___func__.6890\n__func__.6900 gl___func__.6900\n__func__.6906 gl___func__.6906\n__func__.6911 gl___func__.6911\n__func__.6939 gl___func__.6939\n__func__.6960 gl___func__.6960\n__func__.6982 gl___func__.6982\n__func__.7005 gl___func__.7005\n__func__.7025 gl___func__.7025\n__func__.7036 gl___func__.7036\n__func__.7049 gl___func__.7049\n__func__.7054 gl___func__.7054\n__func__.7059 gl___func__.7059\n__func__.7105 gl___func__.7105\n__func__.7142 gl___func__.7142\n__func__.7156 gl___func__.7156\n__func__.7171 gl___func__.7171\n__func__.7219 gl___func__.7219\n__func__.7249 gl___func__.7249\n__func__.7285 gl___func__.7285\n__func__.7334 gl___func__.7334\n__func__.7371 gl___func__.7371\n__func__.7383 gl___func__.7383\n__func__.7392 gl___func__.7392\n__func__.7416 gl___func__.7416\n__func__.7423 gl___func__.7423\n__func__.7438 gl___func__.7438\n__func__.7450 gl___func__.7450\n__func__.7493 gl___func__.7493\n__func__.7532 gl___func__.7532\n__func__.7540 gl___func__.7540\n__func__.7554 gl___func__.7554\n__func__.7666 gl___func__.7666\n__func__.7698 gl___func__.7698\n__func__.7790 gl___func__.7790\n__func__.7822 gl___func__.7822\n__func__.7860 gl___func__.7860\n__func__.7893 gl___func__.7893\n__func__.7901 gl___func__.7901\n__func__.7955 gl___func__.7955\n__func__.8001 gl___func__.8001\n__func__.8054 gl___func__.8054\n__func__.8094 gl___func__.8094\n__func__.8099 gl___func__.8099\n__func__.8104 gl___func__.8104\n__func__.8113 gl___func__.8113\n__func__.8118 gl___func__.8118\n__func__.8123 gl___func__.8123\ngettime gl_gettime\nglobal_debug_map gl_global_debug_map\nhad_ipv4_address gl_had_ipv4_address\nhad_ipv6_address gl_had_ipv6_address\ninformational_phrases gl_informational_phrases\n.LC24 gl_.LC24\n.LC7 gl_.LC7\nlistener_decref_and_unlock gl_listener_decref_and_unlock\nlistener_read_cb gl_listener_read_cb\nlog_fn gl_log_fn\nLOOPBACK_S6.5935 gl_LOOPBACK_S6.5935\nmethods.6197 gl_methods.6197\n_mm_free_fn gl__mm_free_fn\n_mm_malloc_fn gl__mm_malloc_fn\n_mm_realloc_fn gl__mm_realloc_fn\nname_from_addr.constprop.29 gl_name_from_addr.constprop.29\nname_parse gl_name_parse\nnameserver_failed gl_nameserver_failed\nnameserver_pick gl_nameserver_pick\nnameserver_probe_callback gl_nameserver_probe_callback\nnameserver_prod_callback gl_nameserver_prod_callback\nnameserver_read gl_nameserver_read\nnameserver_ready_callback gl_nameserver_ready_callback\nnameserver_up gl_nameserver_up\nnameserver_write_waiting gl_nameserver_write_waiting\nneed_numeric_port_hack_ gl_need_numeric_port_hack_\nneed_socktype_protocol_hack_ gl_need_socktype_protocol_hack_\nnotify_base_cbq_callback gl_notify_base_cbq_callback\n.o: gl_.o:\no: gl_o:\n_original_cond_fns gl__original_cond_fns\n_original_lock_fns gl__original_lock_fns\nparse_numeric_servname gl_parse_numeric_servname\npoll_add gl_poll_add\npoll_dealloc gl_poll_dealloc\npoll_del gl_poll_del\npoll_dispatch gl_poll_dispatch\npoll_init gl_poll_init\npollops gl_pollops\nprefix_suffix_match.constprop.30 gl_prefix_suffix_match.constprop.30\nPRESERVE_PINNED gl_PRESERVE_PINNED\nredirection_phrases gl_redirection_phrases\nregname_ok gl_regname_ok\nreply_handle gl_reply_handle\nreply_run_callback gl_reply_run_callback\nreply_schedule_callback gl_reply_schedule_callback\nrequest_find_from_trans_id gl_request_find_from_trans_id\nrequest_finished gl_request_finished\nrequest_new.isra.16 gl_request_new.isra.16\nrequest_submit gl_request_submit\nresolv_conf_parse_line gl_resolv_conf_parse_line\nresponse_classes gl_response_classes\nRESTORE_PINNED.part.9 gl_RESTORE_PINNED.part.9\nrs gl_rs\nrs_initialized gl_rs_initialized\nscheme_ok gl_scheme_ok\nsearch_make_new.isra.10 gl_search_make_new.isra.10\nsearch_postfix_add.isra.11 gl_search_postfix_add.isra.11\nsearch_request_finished gl_search_request_finished\nsearch_request_new gl_search_request_new\nsearch_set_from_hostname gl_search_set_from_hostname\nsearch_state_decref gl_search_state_decref\nsearch_state_new gl_search_state_new\nselect_add gl_select_add\nselect_dealloc gl_select_dealloc\nselect_del gl_select_del\nselect_dispatch gl_select_dispatch\nselect_free_selectop gl_select_free_selectop\nselect_init gl_select_init\nselectops gl_selectops\nselect_resize gl_select_resize\nserver_error_phrases gl_server_error_phrases\nserver_port_flush gl_server_port_flush\nserver_port_free gl_server_port_free\nserver_port_read gl_server_port_read\nserver_port_ready_callback gl_server_port_ready_callback\nserver_request_free_answers gl_server_request_free_answers\nserver_request_free gl_server_request_free\nsignals.5673 gl_signals.5673\nsockaddr_getport gl_sockaddr_getport\nsockaddr_setport gl_sockaddr_setport\nstrtoint gl_strtoint\nstrtotimeval gl_strtotimeval\nsuccess_phrases gl_success_phrases\ntested_for_getaddrinfo_hacks gl_tested_for_getaddrinfo_hacks\ntest_for_getaddrinfo_hacks gl_test_for_getaddrinfo_hacks\ntransaction_id_pick gl_transaction_id_pick\nupcast.part.0 gl_upcast.part.0\nuri_chars gl_uri_chars\nuse_mmap gl_use_mmap\nuse_monotonic gl_use_monotonic\nuse_monotonic_initialized.6057 gl_use_monotonic_initialized.6057\nuserinfo_ok gl_userinfo_ok\nuse_sendfile gl_use_sendfile\n_warn_helper gl__warn_helper\nZERO_CHAIN.part.8 gl_ZERO_CHAIN.part.8\n" }, { "path": "patches/libhdfs.patch", "content": "diff -rupN hadoop-1.0.1_old/src/c++/libhdfs/hdfsJniHelper.c hadoop-1.0.1/src/c++/libhdfs/hdfsJniHelper.c\n--- hadoop-1.0.1_old/src/c++/libhdfs/hdfsJniHelper.c\t2012-04-03 11:46:00.000000000 -0400\n+++ hadoop-1.0.1/src/c++/libhdfs/hdfsJniHelper.c\t2012-04-03 11:54:18.000000000 -0400\n@@ -15,7 +15,9 @@\n */\n \n #include <string.h> \n+#if !defined(__APPLE__)\n #include <error.h>\n+#endif /* __APPLE__ */\n #include \"hdfsJniHelper.h\"\n \n static pthread_mutex_t hdfsHashMutex = PTHREAD_MUTEX_INITIALIZER;\n\ndiff -rupN hadoop-1.0.1_old/src/c++/libhdfs/configure hadoop-1.0.1/src/c++/libhdfs/configure\n--- hadoop-1.0.1_old/src/c++/libhdfs/configure\n+++ hadoop-1.0.1/src/c++/libhdfs/configure\n@@ -12038,7 +12038,7 @@ echo \"${ECHO_T}ohh u there ... $LIB_JVM_DIR\" >&6; }\n if test \"$supported_os\" != \"darwin\"\n then\n CFLAGS=\"$CFLAGS -m${JVM_ARCH} -I$JAVA_HOME/include -I$JAVA_HOME/include/$supported_os\"\n- LDFLAGS=\"$LDFLAGS -m${JVM_ARCH} -L$LIB_JVM_DIR -ljvm -shared -Wl,-x\"\n+ LDFLAGS=\"$LDFLAGS -m${JVM_ARCH} -L$LIB_JVM_DIR -ljvm -Wl,-x\"\n { echo \"$as_me:$LINENO: result: VALUE OF JVM_ARCH IS :$JVM_ARCH\" >&5\n echo \"${ECHO_T}VALUE OF JVM_ARCH IS :$JVM_ARCH\" >&6; }\n fi\n\ndiff -rupN hadoop-1.0.1_old/src/c++/libhdfs/configure.ac hadoop-1.0.1/src/c++/libhdfs/configure.ac\n--- hadoop-1.0.1_old/src/c++/libhdfs/configure.ac\n+++ hadoop-1.0.1/src/c++/libhdfs/configure.ac\n@@ -80,7 +80,7 @@ AP_JVM_LIBDIR()\n if test \"$supported_os\" != \"darwin\"\n then\n CFLAGS=\"$CFLAGS -m${JVM_ARCH} -I$JAVA_HOME/include -I$JAVA_HOME/include/$supported_os\"\n- LDFLAGS=\"$LDFLAGS -m${JVM_ARCH} -L$LIB_JVM_DIR -ljvm -shared -Wl,-x\"\n+ LDFLAGS=\"$LDFLAGS -m${JVM_ARCH} -L$LIB_JVM_DIR -ljvm -Wl,-x\"\n AC_MSG_RESULT([VALUE OF JVM_ARCH IS :$JVM_ARCH])\n fi\n \ndiff -rupN hadoop-1.0.1_old/src/c++/libhdfs/hdfs.c hadoop-1.0.1/src/c++/libhdfs/hdfs.c\n--- hadoop-1.0.1_old/src/c++/libhdfs/hdfs.c\n+++ hadoop-1.0.1/src/c++/libhdfs/hdfs.c\n@@ -252,7 +252,7 @@ hdfsFS hdfsConnectAsUser(const char* host, tPort port, const char *user)\n cURI = malloc(strlen(host)+16);\n sprintf(cURI, \"hdfs://%s:%d\", host, (int)(port));\n if (cURI == NULL) {\n- fprintf (stderr, \"Couldn't allocate an object of size %d\",\n+ fprintf (stderr, \"Couldn't allocate an object of size %zu\",\n strlen(host) + 16);\n errno = EINTERNAL; \n goto done; \n\n" }, { "path": "patches/libjson.patch", "content": "diff -rup libjson2/JSONOptions.h libjson/JSONOptions.h\n--- libjson2/JSONOptions.h\t2012-03-10 01:41:30.000000000 -0800\n+++ libjson/JSONOptions.h\t2013-05-24 17:44:29.000000000 -0700\n@@ -11,7 +11,7 @@\n * JSON_LIBRARY must be declared if libjson is compiled as a static or dynamic \n * library. This exposes a C-style interface, but none of the inner workings of libjson\n */\n-#define JSON_LIBRARY\n+// #define JSON_LIBRARY\n \n \n /*\n@@ -94,7 +94,7 @@\n * JSON_BINARY is used to support binary, which is base64 encoded and decoded by libjson,\n * if this option is not turned off, no base64 support is included\n */\n-#define JSON_BINARY\n+// #define JSON_BINARY\n \n \n /*\n@@ -102,7 +102,7 @@\n * and decoding. This may be useful if you want to obfuscate your json, or send binary data over\n * a network\n */\n-#define JSON_EXPOSE_BASE64\n+// #define JSON_EXPOSE_BASE64\n \n \n /*\nOnly in libjson: JSONOptions.h.orig\nOnly in libjson: Objects_static\ndiff -rup libjson2/Source/JSONDefs.h libjson/Source/JSONDefs.h\n--- libjson2/Source/JSONDefs.h\t2012-02-23 09:30:04.000000000 -0800\n+++ libjson/Source/JSONDefs.h\t2013-05-24 17:44:29.000000000 -0700\n@@ -146,23 +146,6 @@ typedef void (*json_free_t)(void *);\n typedef void * (*json_realloc_t)(void *, unsigned long);\n #endif\n \n-#ifdef JSON_DEBUG\n- #ifdef NDEBUG\n-\t #ifdef __GNUC__\n-\t\t #warning, Have JSON_DEBUG on in a release build\n-\t #else\n-\t\t #error, Have JSON_DEBUG on in a release build\n-\t #endif\n- #endif\n-#else\n-\t#ifndef NDEBUG\n-\t #ifdef __GNUC__\n-\t\t #warning, Release build of libjson, but NDEBUG is not on\n-\t #else\n-\t\t #error, Release build of libjson, but NDEBUG is not on\n-\t #endif\n-\t#endif\n-#endif\n \n #ifdef JSON_UNIT_TEST\n #define JSON_PRIVATE public:\ndiff -rup libjson2/Source/JSONSharedString.h libjson/Source/JSONSharedString.h\n--- libjson2/Source/JSONSharedString.h\t2012-02-21 04:30:32.000000000 -0800\n+++ libjson/Source/JSONSharedString.h\t2013-05-24 17:56:07.000000000 -0700\n@@ -38,8 +38,9 @@ public:\n \n \tstruct iterator;\n \t struct const_iterator {\n-\t\tconst_iterator(const json_char * p, const json_shared_string * pa) : parent(pa), it(p){}\n-\t \n+\t\t//const_iterator(const json_char * p, const json_shared_string * pa) : parent(pa), it(p){}\n+\t const_iterator(const json_char * p, const json_shared_string * pa) : it(p){}\n+\n \t\t inline const_iterator& operator ++(void) json_nothrow { ++it; return *this; }\n \t\t inline const_iterator& operator --(void) json_nothrow { --it; return *this; }\n \t\t inline const_iterator& operator +=(long i) json_nothrow { it += i; return *this; }\n@@ -84,15 +85,15 @@ public:\n \t\t inline const_iterator & operator =(const const_iterator & orig) json_nothrow { it = orig.it; return *this; }\n \t\t const_iterator (const const_iterator & orig) json_nothrow : it(orig.it) {}\n \t private:\n-\t\t const json_shared_string * parent;\n+ // const json_shared_string * parent; // creates annoying warning\n \t\t const json_char * it;\n \t\t friend class json_shared_string;\n \t\t friend struct iterator;\n \t };\n-\t \n+\n \t struct iterator {\n \t\titerator(const json_char * p, const json_shared_string * pa) : parent(pa), it(p){}\n-\t \n+\n \t\t inline iterator& operator ++(void) json_nothrow { ++it; return *this; }\n \t\t inline iterator& operator --(void) json_nothrow { --it; return *this; }\n \t\t inline iterator& operator +=(long i) json_nothrow { it += i; return *this; }\n@@ -146,22 +147,22 @@ public:\n \n \n \tinline json_shared_string::iterator begin(void){\n-\t\titerator res = iterator(data(), this); \n+\t\titerator res = iterator(data(), this);\n \t\treturn res;\n \t}\n \tinline json_shared_string::iterator end(void){\n-\t\titerator res = iterator(data() + len, this); \n+\t\titerator res = iterator(data() + len, this);\n \t\treturn res;\n \t}\n \tinline json_shared_string::const_iterator begin(void) const {\n-\t\tconst_iterator res = const_iterator(data(), this); \n+\t\tconst_iterator res = const_iterator(data(), this);\n \t\treturn res;\n \t}\n \tinline json_shared_string::const_iterator end(void) const {\n-\t\tconst_iterator res = const_iterator(data() + len, this); \n+\t\tconst_iterator res = const_iterator(data() + len, this);\n \t\treturn res;\n \t}\n-\t\n+\n \n \tinline json_string::iterator std_begin(void){\n \t\treturn _str -> mystring.begin() + offset;\n@@ -169,36 +170,36 @@ public:\n \tinline json_string::iterator std_end(void){\n \t\treturn std_begin() + len;\n \t}\n-\t\n+\n \tinline json_string::const_iterator std_begin(void) const{\n \t\treturn _str -> mystring.begin() + offset;\n \t}\n \tinline json_string::const_iterator std_end(void) const{\n \t\treturn std_begin() + len;\n \t}\n-\t\n+\n \tinline json_shared_string(void) : offset(0), len(0), _str(new(json_malloc<json_shared_string_internal>(1)) json_shared_string_internal(json_global(EMPTY_JSON_STRING))) {}\n-\t\n+\n \tinline json_shared_string(const json_string & str) : offset(0), len(str.length()), _str(new(json_malloc<json_shared_string_internal>(1)) json_shared_string_internal(str)) {}\n-\t\n+\n \tinline json_shared_string(const json_shared_string & str, size_t _offset, size_t _len) : _str(str._str), offset(str.offset + _offset), len(_len) {\n \t\t++_str -> refCount;\n \t}\n-\t\n+\n \tinline json_shared_string(const json_shared_string & str, size_t _offset) : _str(str._str), offset(str.offset + _offset), len(str.len - _offset) {\n \t\t++_str -> refCount;\n \t}\n-\t\n+\n \tinline json_shared_string(const iterator & s, const iterator & e) : _str(s.parent -> _str), offset(s.it - s.parent -> _str -> mystring.data()), len(e.it - s.it){\n \t\t++_str -> refCount;\n \t}\n-\t\n+\n \tinline ~json_shared_string(void){\n \t\tderef();\n \t}\n-\t\n+\n \tinline bool empty(void) const { return len == 0; }\n-\t\n+\n \tsize_t find(json_char ch, size_t pos = 0) const {\n \t\tif (_str -> refCount == 1) return _str -> mystring.find(ch, pos);\n \t\tjson_string::const_iterator e = std_end();\n@@ -207,7 +208,7 @@ public:\n \t\t}\n \t\treturn json_string::npos;\n \t}\n-\t\n+\n \tinline json_char & operator[] (size_t loc){\n \t\treturn _str -> mystring[loc + offset];\n \t}\n@@ -218,21 +219,21 @@ public:\n \tinline size_t length() const { return len; }\n \tinline const json_char * c_str() const { return toString().c_str(); }\n \tinline const json_char * data() const { return _str -> mystring.data() + offset; }\n-\t\n+\n \tinline bool operator != (const json_shared_string & other) const {\n \t\tif ((other._str == _str) && (other.len == len) && (other.offset == offset)) return false;\n \t\treturn other.toString() != toString();\n \t}\n-\t\n+\n \tinline bool operator == (const json_shared_string & other) const {\n \t\tif ((other._str == _str) && (other.len == len) && (other.offset == offset)) return true;\n \t\treturn other.toString() == toString();\n \t}\n-\t\n+\n \tinline bool operator == (const json_string & other) const {\n \t\treturn other == toString();\n \t}\n-\t\n+\n \tjson_string & toString(void) const {\n \t\t//gonna have to do a real substring now anyway, so do it completely\n \t\tif (_str -> refCount == 1){\n@@ -246,8 +247,8 @@ public:\n \t\toffset = 0;\n \t\treturn _str -> mystring;\n \t}\n-\t\n-\t\n+\n+\n \tinline void assign(const json_shared_string & other, size_t _offset, size_t _len){\n \t\tif (other._str != _str){\n \t\t\tderef();\n@@ -257,11 +258,11 @@ public:\n \t\toffset = other.offset + _offset;\n \t\tlen = _len;\n \t}\n-\t\n+\n \tjson_shared_string(const json_shared_string & other) : _str(other._str), offset(other.offset), len(other.len){\n \t\t++_str -> refCount;\n \t}\n-\t\n+\n \tjson_shared_string & operator =(const json_shared_string & other){\n \t\tif (other._str != _str){\n \t\t\tderef();\n@@ -272,13 +273,13 @@ public:\n \t\tlen = other.len;\n \t\treturn *this;\n \t}\n-\t\n+\n \tjson_shared_string & operator += (const json_char c){\n \t\ttoString() += c;\n \t\t++len;\n \t\treturn *this;\n \t}\n-\t\n+\n \t//when doing a plus equal of another string, see if it shares the string and starts where this one left off, in which case just increase len\n JSON_PRIVATE\n \tstruct json_shared_string_internal {\nOnly in libjson/Source: JSONSharedString.h.orig\nOnly in libjson/Source: JSONSharedString.h.rej\nOnly in libjson: libjson.a\ndiff -rup libjson2/makefile libjson/makefile\n--- libjson2/makefile\t2012-02-23 14:34:36.000000000 -0800\n+++ libjson/makefile\t2013-05-24 17:44:29.000000000 -0700\n@@ -236,7 +236,7 @@ $(lib_target): banner installdirs $(addp\n endif\n \n # Compile object files\n-$(objdir)/%.o: $(srcdir)/%.cpp\n+$(objdir)/%.o: $(srcdir)/%.cpp installdirs\n \t$(CXX) $< -o $@ $(CXXFLAGS)\n \n ifeq ($(SHARED),1)\n" }, { "path": "patches/opencv_apple_rpath.patch", "content": "564a565\n> if(APPLE)\n571c572\n< INSTALL_NAME_DIR lib\n---\n> INSTALL_NAME_DIR \"@rpath\"\n573c574,581\n< \n---\n> else()\n> set_target_properties(${the_module} PROPERTIES\n> OUTPUT_NAME \"${the_module}${OPENCV_DLLVERSION}\"\n> DEBUG_POSTFIX \"${OPENCV_DEBUG_POSTFIX}\"\n> ARCHIVE_OUTPUT_DIRECTORY ${LIBRARY_OUTPUT_PATH}\n> RUNTIME_OUTPUT_DIRECTORY ${EXECUTABLE_OUTPUT_PATH}\n> INSTALL_NAME_DIR lib)\n> endif()\n" }, { "path": "patches/tcmalloc.patch", "content": "Index: src/base/linux_syscall_support.h\n===================================================================\n--- src/base/linux_syscall_support.h\t(revision 175)\n+++ src/base/linux_syscall_support.h\t(working copy)\n@@ -243,14 +243,13 @@\n long ru_nivcsw;\n };\n \n-struct siginfo;\n #if defined(__i386__) || defined(__arm__) || defined(__PPC__)\n \n /* include/asm-{arm,i386,mips,ppc}/signal.h */\n struct kernel_old_sigaction {\n union {\n void (*sa_handler_)(int);\n- void (*sa_sigaction_)(int, struct siginfo *, void *);\n+ void (*sa_sigaction_)(int, siginfo_t *, void *);\n };\n unsigned long sa_mask;\n unsigned long sa_flags;\n@@ -287,13 +286,13 @@\n unsigned long sa_flags;\n union {\n void (*sa_handler_)(int);\n- void (*sa_sigaction_)(int, struct siginfo *, void *);\n+ void (*sa_sigaction_)(int, siginfo_t *, void *);\n };\n struct kernel_sigset_t sa_mask;\n #else\n union {\n void (*sa_handler_)(int);\n- void (*sa_sigaction_)(int, struct siginfo *, void *);\n+ void (*sa_sigaction_)(int, siginfo_t *, void *);\n };\n unsigned long sa_flags;\n void (*sa_restorer)(void);\n" }, { "path": "patches/zookeeper/Makefile", "content": "all:\n\tcd src/c; make\n\ninstall:\n\tcd src/c; make install\n" }, { "path": "patches/zookeeper/configure", "content": "#!/bin/bash\nant compile_jute\ncd src/c\n./configure $@\n" }, { "path": "scripts/add_line_to_eof.sh", "content": "#!/bin/bash\nfor f in `find src \\( -name \"*.cpp\" -or -name \"*.hpp\" \\)`; do\nlastline=`tail -n 1 $f`\nlen=$((${#lastline}))\nif [ $len -ne 0 ]; then\n echo $f\n echo -e \"\" >> $f\nfi\ndone\n" }, { "path": "scripts/binary_list.txt", "content": "toolkits/clustering/kmeans\ntoolkits/graph_analytics/undirected_triangle_count\ntoolkits/graph_analytics/directed_triangle_count\ntoolkits/graph_analytics/pagerank\ntoolkits/graph_analytics/kcore\ntoolkits/graph_analytics/format_convert\ntoolkits/graph_analytics/sssp\ntoolkits/graph_analytics/simple_coloring\ntoolkits/collaborative_filtering/als\ntoolkits/collaborative_filtering/sparse_als\ntoolkits/collaborative_filtering/wals\ntoolkits/collaborative_filtering/sgd\ntoolkits/collaborative_filtering/biassgd\ntoolkits/collaborative_filtering/svdpp\ntoolkits/collaborative_filtering/svd\ntoolkits/collaborative_filtering/nmf\ntoolkits/collaborative_filtering/make_synthetic_als_data\ntoolkits/graphical_models/loopybp_denoise\ntoolkits/graphical_models/structured_prediction\ntoolkits/graphical_models/synthetic_image_data\ntookits/topic_modeling/lda_sequential_cgs\ntookits/topic_modeling/cgs_lda\ntookits/topic_modeling/cgs_lda_mimno_experimental\n" }, { "path": "scripts/build_linux_static.sh", "content": "if [ ! -d src ]; then\n echo \"Run from the graphlab root folder\"\n exit\nfi\n./configure -D MARCH=x86-64 -D MTUNE=generic -D COMPILER_FLAGS=\"-static-libgcc\\ -static-libstdc++\"\nscripts/compile_static_release.sh\n\n# is this a openmpi or a mpich2 build?\nISOPENMPI=0\nif grep -q mpi_cxx config.log\nthen\n rootdirname=\"graphlab_openmpi\"\n unstrippeddirname=\"graphlab_openmpi_unstripped\"\n ISOPENMPI=1\nelif grep -q mpich config.log\nthen\n rootdirname=\"graphlab_mpich2\"\n unstrippeddirname=\"graphlab_mpich2_unstripped\"\nelse\n echo \"Unable to detect MPI type\"\n exit\nfi\n\n\n# now package a binary release\nrm -rf ./$rootdirname\nrm -rf ./$unstrippeddirname\nmkdir $rootdirname\nmkdir $unstrippeddirname\nmkdir $rootdirname/gldeps\nmkdir $unstrippeddirname/gldeps\n\nfor file in `cat scripts/binary_list.txt`\ndo\n dname=`dirname $file`\n fname=`basename $file`\n\n deps=$(ldd release/$file | awk 'BEGIN{ORS=\" \"}$1 \\\n ~/^\\//{print $1}$3~/^\\//{print $3}' \\\n | sed 's/,$/\\n/')\n\n for dep in $deps\n do\n depname=`basename $dep`\n # definitely exclude jvm\n if [[ $depname == \"libjvm.so\" ]]; then\n continue\n fi\n if [ ! -f \"$rootdirname/gldeps/$depname\" ]; then\n echo \"Copying $dep\"\n cp \"$dep\" \"$rootdirname/gldeps/\"\n cp \"$dep\" \"$unstrippeddirname/gldeps/\"\n fi\n done\n\n mkdir -p $rootdirname/$dname\n cp release/$file $rootdirname/$dname/\n #strip it\n strip $rootdirname/$dname/$fname\n #package the script\n cp scripts/linux_run_script_template.sh $rootdirname/$dname/$fname.sh\n\n #repeat for unstripped\n mkdir -p $unstrippeddirname/$dname\n cp release/$file $unstrippeddirname/$dname/\n #package the script\n cp scripts/linux_run_script_template.sh $unstrippeddirname/$dname/$fname.sh\ndone\n\n#package all the rest of the stuff\n#copy the license\nmkdir $rootdirname/license\ncp license/LICENSE.txt $rootdirname/license/\n\nmkdir $unstrippeddirname/license\ncp license/LICENSE.txt $unstrippeddirname/license/\n\n#copy the README\ncp BINARY_README $rootdirname/README\ncp BINARY_README $unstrippeddirname/README\n\n# I am unable to get openmpi to work properly with the ld hack\n# since it appears to have complicated binary dependencies. \n# (it forks and launches some other daemon which has its own dependencies)\n# I will give up on this for now and try to get ABI compatibility.\n# it seems like 1.3 is compatible with 1.4 and 1.5 is compatbile with 1.6\nif [ ISOPENMPI -eq 1 ]; then\n rm $rootdirname/gldeps/libmpi.* $rootdirname/gldeps/libopen-*\n rm $unstrippeddirname/gldeps/libmpi.* $unstrippeddirname/gldeps/libopen-*\nfi\n\n#pack\ntar -cjvf $rootdirname.tar.bz2 $rootdirname\ntar -cjvf $unstrippeddirname.tar.bz2 $unstrippeddirname\n" }, { "path": "scripts/build_linux_static_no_jvm.sh", "content": "if [ ! -d src ]; then\n echo \"Run from the graphlab root folder\"\n exit\nfi\n./configure -D MARCH=x86-64 -D MTUNE=generic --no_jvm -D COMPILER_FLAGS=\"-static-libgcc\\ -static-libstdc++\"\nscripts/compile_static_release.sh\n\n# is this a openmpi or a mpich2 build?\nISOPENMPI=0\nif grep -q mpi_cxx config.log\nthen\n rootdirname=\"graphlab_openmpi_no_jvm\"\n unstrippeddirname=\"graphlab_openmpi_unstripped_no_jvm\"\n ISOPENMPI=1\nelif grep -q mpich config.log\nthen\n rootdirname=\"graphlab_mpich2_no_jvm\"\n unstrippeddirname=\"graphlab_mpich2_unstripped_no_jvm\"\nelse\n echo \"Unable to detect MPI type\"\n exit\nfi\n\n\n# now package a binary release\nrm -rf ./$rootdirname\nrm -rf ./$unstrippeddirname\nmkdir $rootdirname\nmkdir $unstrippeddirname\nmkdir $rootdirname/gldeps\nmkdir $unstrippeddirname/gldeps\n\nfor file in `cat scripts/binary_list.txt`\ndo\n dname=`dirname $file`\n fname=`basename $file`\n\n deps=$(ldd release/$file | awk 'BEGIN{ORS=\" \"}$1 \\\n ~/^\\//{print $1}$3~/^\\//{print $3}' \\\n | sed 's/,$/\\n/')\n\n for dep in $deps\n do\n depname=`basename $dep`\n # definitely exclude jvm\n if [[ $depname == \"libjvm.so\" ]]; then\n continue\n fi\n if [ ! -f \"$rootdirname/gldeps/$depname\" ]; then\n echo \"Copying $dep\"\n cp \"$dep\" \"$rootdirname/gldeps/\"\n cp \"$dep\" \"$unstrippeddirname/gldeps/\"\n fi\n done\n\n mkdir -p $rootdirname/$dname\n cp release/$file $rootdirname/$dname/\n #strip it\n strip $rootdirname/$dname/$fname\n #package the script\n cp scripts/linux_run_script_template.sh $rootdirname/$dname/$fname.sh\n\n #repeat for unstripped\n mkdir -p $unstrippeddirname/$dname\n cp release/$file $unstrippeddirname/$dname/\n #package the script\n cp scripts/linux_run_script_template.sh $unstrippeddirname/$dname/$fname.sh\ndone\n\n#package all the rest of the stuff\n#copy the license\nmkdir $rootdirname/license\ncp license/LICENSE.txt $rootdirname/license/\n\nmkdir $unstrippeddirname/license\ncp license/LICENSE.txt $unstrippeddirname/license/\n\n#copy the README\ncp BINARY_README $rootdirname/README\ncp BINARY_README $unstrippeddirname/README\n\n# I am unable to get openmpi to work properly with the ld hack\n# since it appears to have complicated binary dependencies. \n# (it forks and launches some other daemon which has its own dependencies)\n# I will give up on this for now and try to get ABI compatibility.\n# it seems like 1.3 is compatible with 1.4 and 1.5 is compatbile with 1.6\nif [ ISOPENMPI -eq 1 ]; then\n rm $rootdirname/gldeps/libmpi.* $rootdirname/gldeps/libopen-*\n rm $unstrippeddirname/gldeps/libmpi.* $unstrippeddirname/gldeps/libopen-*\nfi\n\n#pack\ntar -cjvf $rootdirname.tar.bz2 $rootdirname\ntar -cjvf $unstrippeddirname.tar.bz2 $unstrippeddirname\n" }, { "path": "scripts/build_linux_static_no_jvm_no_mpi.sh", "content": "if [ ! -d src ]; then\n echo \"Run from the graphlab root folder\"\n exit\nfi\n./configure -D MARCH=x86-64 -D MTUNE=generic --no_jvm -D NO_MPI:BOOL=true -D COMPILER_FLAGS:STRING=\"-static-libgcc\\ -static-libstdc++\" \nscripts/compile_static_release.sh $@\n\n# is this a openmpi or a mpich2 build?\nrootdirname=\"graphlab_no_jvm_no_mpi\"\nunstrippeddirname=\"graphlab_unstripped_no_jvm_no_mpi\"\nISOPENMPI=0\n\n\n# now package a binary release\nrm -rf ./$rootdirname\nrm -rf ./$unstrippeddirname\nmkdir $rootdirname\nmkdir $unstrippeddirname\nmkdir $rootdirname/gldeps\nmkdir $unstrippeddirname/gldeps\n\ntmp=$@\nif test $# -lt 1 ; then\n tmp=`cat scripts/binary_list.txt`\nfi\n\n\nfor file in $tmp\ndo\n dname=`dirname $file`\n fname=`basename $file`\n\n deps=$(ldd release/$file | awk 'BEGIN{ORS=\" \"}$1 \\\n ~/^\\//{print $1}$3~/^\\//{print $3}' \\\n | sed 's/,$/\\n/')\n\n for dep in $deps\n do\n depname=`basename $dep`\n # definitely exclude jvm\n if [ ! -f \"$rootdirname/gldeps/$depname\" ]; then\n echo \"Copying $dep\"\n cp \"$dep\" \"$rootdirname/gldeps/\"\n cp \"$dep\" \"$unstrippeddirname/gldeps/\"\n fi\n done\n\n mkdir -p $rootdirname/$dname\n cp release/$file $rootdirname/$dname/\n #strip it\n strip $rootdirname/$dname/$fname\n #package the script\n cp scripts/linux_run_script_template.sh $rootdirname/$dname/$fname.sh\n\n #repeat for unstripped\n mkdir -p $unstrippeddirname/$dname\n cp release/$file $unstrippeddirname/$dname/\n #package the script\n cp scripts/linux_run_script_template.sh $unstrippeddirname/$dname/$fname.sh\ndone\n\n#package all the rest of the stuff\n#copy the license\nmkdir $rootdirname/license\ncp license/LICENSE.txt $rootdirname/license/\n\nmkdir $unstrippeddirname/license\ncp license/LICENSE.txt $unstrippeddirname/license/\n\n#copy the README\ncp BINARY_README $rootdirname/README\ncp BINARY_README $unstrippeddirname/README\n\n#pack\ntar -cjvf $rootdirname.tar.bz2 $rootdirname\ntar -cjvf $unstrippeddirname.tar.bz2 $unstrippeddirname\n" }, { "path": "scripts/build_osx_static.sh", "content": "if [ ! -d src ]; then\n echo \"Run from the graphlab root folder\"\n exit\nfi\n\n\n./configure --no_jvm -D NO_MPI:BOOL=true -D COMPILER_FLAGS=\"-mmacosx-version-min=10.7\" -D MARCH=x86-64 -D MTUNE=generic -D HAS_CRC32:BOOL=false\nscripts/compile_static_release.sh\n\necho \"Packaging binary release...\"\n\n# now package a binary release\n# for whatever reason the mac binaries are quite small... \n# stripping not necessary\nrootdirname=\"graphlab_mac\"\nrm -rf ./$rootdirname\nmkdir $rootdirname\nfor file in `cat scripts/binary_list.txt`\ndo\n dname=`dirname $file`\n mkdir -p $rootdirname/$dname\n cp release/$file $rootdirname/$dname/\ndone\n\n#package all the rest of the stuff\n#copy the license\nmkdir $rootdirname/license\ncp license/LICENSE.txt $rootdirname/license/\n\n#copy the README\ncp BINARY_README $rootdirname/README\n\necho \"Binary release packaged in $rootdirname\"\ntar -cjvf $rootdirname.tar.bz2 $rootdirname\n" }, { "path": "scripts/compile_static_release.sh", "content": "#!/bin/bash\nif [ ! -d release ]; then\n echo \"Run from the graphlab root folder after ./configure\"\nelse\n cd release\n make external_dependencies\n cd ..\n rm -f deps/local/lib/libboost*.so deps/local/lib/libhdfs*.so deps/local/lib/libtcmalloc*.so deps/local/lib/libevent*.so deps/local/lib/libproto*.so\n rm -f deps/local/lib/libboost*.dylib deps/local/lib/libhdfs*.dylib deps/local/lib/libtcmalloc*.dylib deps/local/lib/libevent*.dylib\n cd release\n\ntmp=$@\nif test $# -lt 1 ; then\n tmp=`cat ../scripts/binary_list.txt`\nfi\necho $tmp\nfor file in $tmp\ndo\n pushd .\n dname=`dirname $file`\n fname=`basename $file`\n cd $dname\n make -j4 $fname\n popd\ndone\nfi\n" }, { "path": "scripts/ec2/benchmark_ec2.sh", "content": "#!/bin/bash\n\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n# \n# http://www.apache.org/licenses/LICENSE-2.0\n# \n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# This script is an example benchmarking of GraphLab for EC2\n# for testing scaling\n# (C) GraphLab Inc. 2013\n# Please send any questions or bug reports to graphlabapi@groups.google.com\n# Written by Danny Bickson\n\n\n############################################################################\n# CONFIGURATION\n############################################################################\nMAX_SLAVES=3 # configure the maximum number of slaves\nMAX_RETRY=3 # configure the number of experiemnt repeats\nPAGERANK=1 # if 1, runs pagerank\nSVD=1 # if 1, runs svd\nALS=1 # if 1, runs als\n\n#It is recommended to define the below two variables for easier setup\n#uncomment the below two lines once you set them up\n#export AWS_ACCESS_KEY_ID=[ Your access key ]\n#export AWS_SECRET_ACCESS_KEY=[ Your access key secret ]\n######################################################################\n\n# clean old running instances, if any\necho \"y\" | ./gl-ec2 -i ~/.ssh/amazonec2.pem -k amazonec2 destroy hpctest \n# launch ec2 cc2.8xlarge image\n./gl-ec2 -i ~/.ssh/amazonec2.pem -k amazonec2 -a hpc -s $MAX_SLAVES -t cc2.8xlarge launch hpctest \n# update the GraphLab version to be the latest, recompile, and update slaves\n./gl-ec2 -i ~/.ssh/amazonec2.pem -k amazonec2 update hpctest \n\n# run pagerank benchmarks\nif [ $PAGERANK -eq 1 ]; then\nfor i in `seq 0 1 $MAX_SLAVES`\ndo\n echo \"Running Pagerank\"\n for j in `seq 0 1 $MAX_RETRY`\n do\n ./gl-ec2 -i ~/.ssh/amazonec2.pem -k amazonec2 -s $i pagerank_demo hpctest \n done\ndone\nfi\n\n# run SVD benchmarks\nif [ $SVD -eq 1 ]; then\nfor i in `seq 0 1 $MAX_SLAVES`\ndo\n echo \"Running SVD\"\n for j in `seq 0 1 $MAX_RETRY`\n do\n ./gl-ec2 -i ~/.ssh/amazonec2.pem -k amazonec2 -s $i svd_demo hpctest \n done\ndone\nfi\n\n# run ALS benchmarks\nif [ $ALS -eq 1 ]; then\nfor i in `seq 0 1 $MAX_SLAVES`\ndo\n echo \"Running ALS\"\n for j in `seq 0 1 $MAX_RETRY`\n do\n if [ $first_time -eq 1 ]; then\n ./gl-ec2 -i ~/.ssh/amazonec2.pem -k amazonec2 -s $i als_demo hpctest \n fi\n done\ndone\nfi\n\n# clean everything\necho \"y\" | ./gl-ec2 -i ~/.ssh/amazonec2.pem -k amazonec2 destroy hpctest \n" }, { "path": "scripts/ec2/gl-ec2", "content": "#!/bin/sh\n\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n# \n# http://www.apache.org/licenses/LICENSE-2.0\n# \n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ncd \"`dirname $0`\"\nPYTHONPATH=\"./third_party/boto-2.4.1.zip/boto-2.4.1:$PYTHONPATH\" python ./gl_ec2.py $@\n" }, { "path": "scripts/ec2/gl_ec2.py", "content": "#ls!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n# \n# http://www.apache.org/licenses/LICENSE-2.0\n# \n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport boto\nimport logging\nimport os\nimport random\nimport shutil\nimport subprocess\nimport sys\nimport tempfile\nimport time\nimport urllib2\nimport stat\nfrom optparse import OptionParser\nfrom sys import stderr\nfrom boto.ec2.blockdevicemapping import BlockDeviceMapping, EBSBlockDeviceType\n\n# A static URL from which to figure out the latest GraphLab EC2 AMI\nSTD_AMI_URL = \"https://s3.amazonaws.com/GraphLabGit/graphlab2-std\"\nHVM_AMI_URL = \"https://s3.amazonaws.com/GraphLabGit/graphlab2-hvm\"\n\ncompilation_threads = 4\n\n# Configure and parse our command-line arguments\ndef parse_args():\n parser = OptionParser(usage=\"gl-ec2 [options] <action> <cluster_name>\"\n + \"\\n\\n<action> can be: launch, destroy, login, stop, start, start-hadoop, stop-hadoop, check-hadoop, get-master, attach-ebs, detach-ebs, als_demo, svd_demo, pagerank_demo, update, update-dbg\",\n add_help_option=False)\n parser.add_option(\"-h\", \"--help\", action=\"help\",\n help=\"Show this help message and exit\")\n parser.add_option(\"-s\", \"--slaves\", type=\"int\", default=1,\n help=\"Number of slaves to launch (default: 1)\")\n parser.add_option(\"-w\", \"--wait\", type=\"int\", default=120,\n help=\"Seconds to wait for nodes to start (default: 120)\")\n parser.add_option(\"-k\", \"--key-pair\",\n help=\"The name of the ssh identitiy key\")\n parser.add_option(\"-i\", \"--identity-file\", \n help=\"SSH private key file to use for logging into instances\")\n parser.add_option(\"-t\", \"--instance-type\", default=\"m1.xlarge\",\n help=\"Type of instance to launch (default: m1.xlarge). \" +\n \"WARNING: must be 64-bit; small instances won't work\")\n parser.add_option(\"-m\", \"--master-instance-type\", default=\"\",\n help=\"Master instance type (leave empty for same as instance-type)\")\n parser.add_option(\"-r\", \"--region\", default=\"us-west-2\",\n help=\"EC2 region zone to launch instances in\")\n parser.add_option(\"-z\", \"--zone\", default=\"\",\n help=\"Availability zone to launch instances in\")\n parser.add_option(\"-a\", \"--ami\", default=\"std\",\n help=\"Amazon Machine Image ID to use, or 'hpc' to use ami for high performance instances\" +\n \"(default: std)\")\n parser.add_option(\"-D\", metavar=\"[ADDRESS:]PORT\", dest=\"proxy_port\", \n help=\"Use SSH dynamic port forwarding to create a SOCKS proxy at \" +\n \"the given local address (for use with login)\")\n parser.add_option(\"--resume\", action=\"store_true\", default=False,\n help=\"Resume installation on a previously launched cluster \" +\n \"(for debugging)\")\n parser.add_option(\"--ebs-vol-size\", metavar=\"SIZE\", type=\"int\", default=0,\n help=\"Attach a new EBS volume of size SIZE (in GB) to each node as \" +\n \"/vol. The volumes will be deleted when the instances terminate. \" +\n \"Only possible on EBS-backed AMIs.\")\n parser.add_option(\"--ebs-vol-id\", default=\"\", \n help=\"Attach an existing EBS volume to the master node at /dev/sdh.\" +\n \"This is a required argument for action = attach-ebs or detach-ebs.\" +\n \"Only possible on EBS-backed AMIs.\")\n parser.add_option(\"--swap\", metavar=\"SWAP\", type=\"int\", default=1024,\n help=\"Swap space to set up per node, in MB (default: 1024)\")\n parser.add_option(\"--spot-price\", metavar=\"PRICE\", type=\"float\",\n help=\"If specified, launch slaves as spot instances with the given \" +\n \"maximum price (in dollars)\")\n (opts, args) = parser.parse_args()\n if len(args) != 2:\n parser.print_help()\n sys.exit(1)\n (action, cluster_name) = args\n if opts.identity_file == None and action in ['launch', 'login', 'start-hadoop', 'stop-hadoop', 'check-hadoop', 'als_demo', 'svd_demo','pagerank_demo', 'update', 'update-dbg']:\n print >> stderr, (\"ERROR: The -i or --identity-file argument is \" +\n \"required for \" + action)\n sys.exit(1)\n private_key_mode = str(oct(os.stat(opts.identity_file)[stat.ST_MODE])[-3:])\n if private_key_mode != \"400\" :\n print >> stderr, (\"ERROR: permissions of private key file \" +opts.identity_file+\n \" should be 400\")\n sys.exit(1)\n \n if os.getenv('AWS_ACCESS_KEY_ID') == None:\n print >> stderr, (\"ERROR: The environment variable AWS_ACCESS_KEY_ID \" +\n \"must be set\")\n sys.exit(1)\n if os.getenv('AWS_SECRET_ACCESS_KEY') == None:\n print >> stderr, (\"ERROR: The environment variable AWS_SECRET_ACCESS_KEY \" +\n \"must be set\")\n sys.exit(1)\n\n if opts.instance_type == \"m1.xlarge\":\n compilation_threads = 4\n\n return (opts, action, cluster_name)\n\n\n# Get the EC2 security group of the given name, creating it if it doesn't exist\ndef get_or_make_group(conn, name):\n groups = conn.get_all_security_groups()\n group = [g for g in groups if g.name == name]\n if len(group) > 0:\n return group[0]\n else:\n print \"Creating security group \" + name\n return conn.create_security_group(name, \"GraphLab EC2 group\")\n\n\n# Wait for a set of launched instances to exit the \"pending\" state\n# (i.e. either to start running or to fail and be terminated)\ndef wait_for_instances(conn, instances):\n while True:\n for i in instances:\n i.update()\n if len([i for i in instances if i.state == 'pending']) > 0:\n time.sleep(5)\n else:\n return\n\n\n# Check whether a given EC2 instance object is in a state we consider active,\n# i.e. not terminating or terminated. We count both stopping and stopped as\n# active since we can restart stopped clusters.\ndef is_active(instance):\n return (instance.state in ['pending', 'running', 'stopping', 'stopped'])\n\n\n# Launch a cluster of the given name, by setting up its security groups,\n# and then starting new instances in them.\n# Returns a tuple of EC2 reservation objects for the master, slave\n# and zookeeper instances (in that order).\n# Fails if there already instances running in the cluster's groups.\ndef launch_cluster(conn, opts, cluster_name):\n print \"Setting up security groups...\"\n master_group = get_or_make_group(conn, cluster_name + \"-master\")\n slave_group = get_or_make_group(conn, cluster_name + \"-slaves\")\n zoo_group = get_or_make_group(conn, cluster_name + \"-zoo\")\n # master_group = get_or_make_group(conn, cluster_name)\n # slave_group = get_or_make_group(conn, cluster_name)\n # zoo_group = get_or_make_group(conn, cluster_name)\n\n\n if master_group.rules == []: # Group was just now created\n master_group.authorize(src_group=master_group)\n master_group.authorize(src_group=slave_group)\n master_group.authorize(src_group=zoo_group)\n master_group.authorize('tcp', 22, 22, '0.0.0.0/0')\n master_group.authorize('tcp', 0, 65535, '0.0.0.0/0')\n master_group.authorize('udp', 0, 65535, '0.0.0.0/0')\n master_group.authorize('tcp', 8080, 8081, '0.0.0.0/0')\n master_group.authorize('tcp', 50030, 50030, '0.0.0.0/0')\n master_group.authorize('tcp', 50070, 50070, '0.0.0.0/0')\n master_group.authorize('tcp', 60070, 60070, '0.0.0.0/0')\n master_group.authorize('tcp', 38090, 38090, '0.0.0.0/0')\n if slave_group.rules == []: # Group was just now created\n slave_group.authorize(src_group=master_group)\n slave_group.authorize(src_group=slave_group)\n slave_group.authorize(src_group=zoo_group)\n slave_group.authorize('tcp', 0, 65535, '0.0.0.0/0')\n slave_group.authorize('udp', 0, 65535, '0.0.0.0/0')\n slave_group.authorize('tcp', 22, 22, '0.0.0.0/0')\n slave_group.authorize('tcp', 8080, 8081, '0.0.0.0/0')\n slave_group.authorize('tcp', 50060, 50060, '0.0.0.0/0')\n slave_group.authorize('tcp', 50075, 50075, '0.0.0.0/0')\n slave_group.authorize('tcp', 60060, 60060, '0.0.0.0/0')\n slave_group.authorize('tcp', 60075, 60075, '0.0.0.0/0')\n if zoo_group.rules == []: # Group was just now created\n zoo_group.authorize(src_group=master_group)\n zoo_group.authorize(src_group=slave_group)\n zoo_group.authorize(src_group=zoo_group)\n zoo_group.authorize('tcp', 22, 22, '0.0.0.0/0')\n zoo_group.authorize('tcp', 2181, 2181, '0.0.0.0/0')\n zoo_group.authorize('tcp', 2888, 2888, '0.0.0.0/0')\n zoo_group.authorize('tcp', 3888, 3888, '0.0.0.0/0')\n\n # Check if instances are already running in our groups\n print \"Checking for running cluster...\"\n reservations = conn.get_all_instances()\n for res in reservations:\n group_names = [g.id for g in res.groups]\n if master_group.name in group_names or slave_group.name in group_names or zoo_group.name in group_names:\n active = [i for i in res.instances if is_active(i)]\n if len(active) > 0:\n print >> stderr, (\"ERROR: There are already instances running in \" +\n \"group %s, %s or %s\" % (master_group.name, slave_group.name, zoo_group.name))\n sys.exit(1)\n\n if opts.ami == \"std\":\n try:\n opts.ami = urllib2.urlopen(STD_AMI_URL).read().strip()\n print \"GraphLab AMI for Standard Instances: \" + opts.ami\n except:\n print >> stderr, \"Could not read \" + STD_AMI_URL\n elif opts.ami == \"hpc\" :\n try:\n opts.ami = urllib2.urlopen(HVM_AMI_URL).read().strip()\n print \"GraphLab AMI for HPC Instances: \" + opts.ami\n compilation_threads = 8\n except:\n print >> stderr, \"Could not read \" + HVM_AMI_URL\n \n print \"Launching instances...\"\n try:\n image = conn.get_all_images(image_ids=[opts.ami])[0]\n except:\n print >> stderr, \"Could not find AMI \" + opts.ami\n sys.exit(1)\n\n # Create block device mapping so that we can add an EBS volume if asked to\n block_map = BlockDeviceMapping()\n if opts.ebs_vol_size > 0:\n device = EBSBlockDeviceType()\n device.size = opts.ebs_vol_size\n device.delete_on_termination = True\n block_map[\"/dev/sdv\"] = device\n\n # Launch slaves\n if opts.spot_price != None:\n # Launch spot instances with the requested price\n print (\"Requesting %d slaves as spot instances with price $%.3f\" %\n (opts.slaves, opts.spot_price))\n slave_reqs = conn.request_spot_instances(\n price = opts.spot_price,\n image_id = opts.ami,\n launch_group = \"launch-group-%s\" % cluster_name,\n placement = opts.zone,\n count = opts.slaves,\n key_name = opts.key_pair,\n security_groups = [slave_group],\n instance_type = opts.instance_type,\n block_device_map = block_map)\n my_req_ids = [req.id for req in slave_reqs]\n print \"Waiting for spot instances to be granted...\"\n while True:\n time.sleep(10)\n reqs = conn.get_all_spot_instance_requests()\n id_to_req = {}\n for r in reqs:\n id_to_req[r.id] = r\n active = 0\n instance_ids = []\n for i in my_req_ids:\n if id_to_req[i].state == \"active\":\n active += 1\n instance_ids.append(id_to_req[i].instance_id)\n if active == opts.slaves:\n print \"All %d slaves granted\" % opts.slaves\n reservations = conn.get_all_instances(instance_ids)\n slave_nodes = []\n for r in reservations:\n slave_nodes += r.instances\n break\n else:\n print \"%d of %d slaves granted, waiting longer\" % (active, opts.slaves)\n else:\n # Launch non-spot instances\n slave_res = image.run(key_name = opts.key_pair,\n security_groups = [slave_group],\n instance_type = opts.instance_type,\n placement = opts.zone,\n min_count = opts.slaves,\n max_count = opts.slaves,\n block_device_map = block_map)\n slave_nodes = slave_res.instances\n print \"Launched slaves, regid = \" + slave_res.id\n\n # # Launch masters\n master_type = opts.master_instance_type\n if master_type == \"\":\n master_type = opts.instance_type\n master_res = image.run(key_name = opts.key_pair,\n security_groups = [master_group],\n instance_type = master_type,\n placement = opts.zone,\n min_count = 1,\n max_count = 1,\n block_device_map = block_map)\n master_nodes = master_res.instances\n print \"Launched master, regid = \" + master_res.id\n\n zoo_nodes = []\n\n # Return all the instances\n return (master_nodes, slave_nodes, zoo_nodes)\n\n\n# Get the EC2 instances in an existing cluster if available.\n# Returns a tuple of lists of EC2 instance objects for the masters,\n# slaves and zookeeper nodes (in that order).\ndef get_existing_cluster(conn, opts, cluster_name):\n print \"Searching for existing cluster \" + cluster_name + \"...\"\n reservations = conn.get_all_instances()\n master_nodes = []\n slave_nodes = []\n zoo_nodes = []\n for res in reservations:\n active = [i for i in res.instances if is_active(i)]\n if len(active) > 0:\n print \"Acitve: \", active\n group_names = list(set(g.name for g in i.groups for i in res.instances)) #DB: bug fix as explained here: https://spark-project.atlassian.net/browse/SPARK-749\n print \"Group names: \", group_names \n if group_names == [cluster_name + \"-master\"]:\n master_nodes += res.instances\n elif group_names == [cluster_name + \"-slaves\"]:\n slave_nodes += res.instances\n elif group_names == [cluster_name + \"-zoo\"]:\n zoo_nodes += res.instances\n if master_nodes != [] and slave_nodes != []:\n print (\"Found %d master(s), %d slaves, %d ZooKeeper nodes\" %\n (len(master_nodes), len(slave_nodes), len(zoo_nodes)))\n return (master_nodes, slave_nodes, zoo_nodes)\n else:\n if master_nodes == [] and slave_nodes != []:\n print \"ERROR: Could not find master in group \" + cluster_name + \"-master\"\n elif master_nodes != [] and slave_nodes == []:\n print \"ERROR: Could not find slaves in group \" + cluster_name + \"-slaves\"\n else:\n print \"ERROR: Could not find any existing cluster\"\n sys.exit(1)\n\ndef get_internal_ips(conn, opts, cluster_name):\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster(conn, opts, cluster_name)\n hosts = [master_nodes[0].private_dns_name]\n for slave in slave_nodes:\n hosts.append(slave.private_dns_name)\n return hosts\n\ndef attach_ebs(conn, opts, cluster_name):\n if opts.ebs_vol_id==\"\": \n print \"ERROR: Please specify --ebs-vol-id\"\n return False\n\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster(conn, opts, cluster_name)\n masterid = master_nodes[0].id\n return conn.attach_volume(opts.ebs_vol_id, masterid, \"/dev/sdh\")\n\ndef detach_ebs(conn, opts, cluster_name):\n if opts.ebs_vol_id==\"\": \n print \"ERROR: Please specify --ebs-vol-id\"\n return False\n else:\n return conn.detach_volume(opts.ebs_vol_id)\n\n\n\n\n\n# Deploy configuration files and run setup scripts on a newly launched\n# or started EC2 cluster.\ndef setup_cluster(conn, master_nodes, slave_nodes, zoo_nodes, opts, cluster_name, deploy_ssh_key):\n # print \"Deploying files to master...\"\n # deploy_files(conn, \"deploy.generic\", opts, master_nodes, slave_nodes, zoo_nodes)\n master = master_nodes[0].public_dns_name\n if deploy_ssh_key:\n print \"Copying SSH key %s to master node %s...\" % (opts.identity_file,master)\n ssh(master, opts, 'sudo mkdir -p /root/.ssh; mkdir tmp')\n scp(master, opts, opts.identity_file, 'tmp/id_rsa')\n ssh(master, opts, 'sudo mv tmp/id_rsa ~/.ssh/')\n config = open(\"config\", \"w\")\n config.write(\"StrictHostKeyChecking no\\nBatchMode yes\\n\")\n config.close()\n scp(master, opts, \"config\", \".ssh/config\")\n for i in slave_nodes:\n ip = i.public_dns_name \n print \"Copying SSH key %s to slave node %s...\" % (opts.identity_file,ip)\n ssh(ip, opts, 'sudo mkdir -p /root/.ssh; mkdir tmp')\n scp(ip, opts, opts.identity_file, 'tmp/id_rsa')\n ssh(ip, opts, 'sudo mv tmp/id_rsa ~/.ssh/')\n scp(ip, opts, \"config\", \".ssh/config\")\n print \"Copy machines hostfile to master...\"\n hosts = get_internal_ips(conn, opts, cluster_name)\n hostfile = open(\"machines\", \"w\")\n for ip in hosts:\n hostfile.write(\"%s\\n\" % ip)\n hostfile.close()\n scp(master, opts, \"machines\", '~/machines')\n\n print \"Running setup on master...\"\n # ssh(master, opts, \"chmod u+x mesos-ec2/setup\")\n # ssh(master, opts, \"mesos-ec2/setup %s %s %s %s\" %\n # (\"generic\", \"none\", \"master\", opts.swap))\n print \"Done!\"\n print \"The master ip is : \" + master\n\n\n# Wait for a whole cluster (masters, slaves and ZooKeeper) to start up\ndef wait_for_cluster(conn, wait_secs, master_nodes, slave_nodes, zoo_nodes):\n print \"Waiting for instances to start up...\"\n time.sleep(5)\n wait_for_instances(conn, master_nodes)\n wait_for_instances(conn, slave_nodes)\n if zoo_nodes != []:\n wait_for_instances(conn, zoo_nodes)\n print \"Waiting %d more seconds...\" % wait_secs\n time.sleep(wait_secs)\n\n\n# Get number of local disks available for a given EC2 instance type.\ndef get_num_disks(instance_type):\n # From http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?InstanceStorage.html\n disks_by_instance = {\n \"m1.small\": 1,\n \"m1.large\": 2,\n \"m1.xlarge\": 4,\n \"t1.micro\": 1,\n \"c1.medium\": 1,\n \"c1.xlarge\": 4,\n \"m2.xlarge\": 1,\n \"m2.2xlarge\": 1,\n \"m2.4xlarge\": 2,\n \"cc1.4xlarge\": 2,\n \"cc2.8xlarge\": 4,\n \"cg1.4xlarge\": 2\n }\n if instance_type in disks_by_instance:\n return disks_by_instance[instance_type]\n else:\n print >> stderr, (\"WARNING: Don't know number of disks on instance type %s; assuming 1\"\n % instance_type)\n return 1\n\n\n# Deploy the configuration file templates in a given local directory to\n# a cluster, filling in any template parameters with information about the\n# cluster (e.g. lists of masters and slaves). Files are only deployed to\n# the first master instance in the cluster, and we expect the setup\n# script to be run on that instance to copy them to other nodes.\ndef deploy_files(conn, root_dir, opts, master_nodes, slave_nodes, zoo_nodes):\n active_master = master_nodes[0].public_dns_name\n\n num_disks = get_num_disks(opts.instance_type)\n hdfs_data_dirs = \"/mnt/ephemeral-hdfs/data\"\n mapred_local_dirs = \"/mnt/hadoop/mrlocal\"\n if num_disks > 1:\n for i in range(2, num_disks + 1):\n hdfs_data_dirs += \",/mnt%d/ephemeral-hdfs/data\" % i\n mapred_local_dirs += \",/mnt%d/hadoop/mrlocal\" % i\n\n if zoo_nodes != []:\n zoo_list = '\\n'.join([i.public_dns_name for i in zoo_nodes])\n cluster_url = \"zoo://\" + \",\".join(\n [\"%s:2181/mesos\" % i.public_dns_name for i in zoo_nodes])\n else:\n zoo_list = \"NONE\"\n cluster_url = \"%s:5050\" % active_master\n\n template_vars = {\n \"master_list\": '\\n'.join([i.public_dns_name for i in master_nodes]),\n \"active_master\": active_master,\n \"slave_list\": '\\n'.join([i.public_dns_name for i in slave_nodes]),\n \"zoo_list\": zoo_list,\n \"cluster_url\": cluster_url,\n \"hdfs_data_dirs\": hdfs_data_dirs,\n \"mapred_local_dirs\": mapred_local_dirs\n }\n\n # Create a temp directory in which we will place all the files to be\n # deployed after we substitue template parameters in them\n tmp_dir = tempfile.mkdtemp()\n for path, dirs, files in os.walk(root_dir):\n if path.find(\".svn\") == -1:\n dest_dir = os.path.join('/', path[len(root_dir):])\n local_dir = tmp_dir + dest_dir\n if not os.path.exists(local_dir):\n os.makedirs(local_dir)\n for filename in files:\n if filename[0] not in '#.~' and filename[-1] != '~':\n dest_file = os.path.join(dest_dir, filename)\n local_file = tmp_dir + dest_file\n with open(os.path.join(path, filename)) as src:\n with open(local_file, \"w\") as dest:\n text = src.read()\n for key in template_vars:\n text = text.replace(\"{{\" + key + \"}}\", template_vars[key])\n dest.write(text)\n dest.close()\n # rsync the whole directory over to the master machine\n command = ((\"rsync -rv -e 'ssh -o StrictHostKeyChecking=no -i %s' \" + \n \"'%s/' 'ubuntu@%s:/'\") % (opts.identity_file, tmp_dir, active_master))\n subprocess.check_call(command, shell=True)\n # Remove the temp directory we created above\n shutil.rmtree(tmp_dir)\n\n\n# Copy a file to a given host through scp, throwing an exception if scp fails\ndef scp(host, opts, local_file, dest_file):\n subprocess.check_call(\n \"scp -q -o StrictHostKeyChecking=no -i %s '%s' 'ubuntu@%s:%s'\" %\n (opts.identity_file, local_file, host, dest_file), shell=True)\n\n\n# Run a command on a host through ssh, throwing an exception if ssh fails\ndef ssh(host, opts, command):\n subprocess.check_call(\n \"ssh -t -o StrictHostKeyChecking=no -i %s ubuntu@%s '%s'\" %\n (opts.identity_file, host, command), shell=True)\n\n\ndef main():\n (opts, action, cluster_name) = parse_args()\n conn = boto.ec2.connect_to_region(opts.region)\n\n # Select an AZ at random if it was not specified.\n if opts.zone == \"\":\n opts.zone = random.choice(conn.get_all_zones()).name\n\n if action == \"launch\":\n if opts.resume:\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster(\n conn, opts, cluster_name)\n else:\n (master_nodes, slave_nodes, zoo_nodes) = launch_cluster(\n conn, opts, cluster_name)\n wait_for_cluster(conn, opts.wait, master_nodes, slave_nodes, zoo_nodes)\n setup_cluster(conn, master_nodes, slave_nodes, zoo_nodes, opts, cluster_name, True)\n\n elif action == \"destroy\":\n response = raw_input(\"Are you sure you want to destroy the cluster \" +\n cluster_name + \"?\\nALL DATA ON ALL NODES WILL BE LOST!!\\n\" +\n \"Destroy cluster \" + cluster_name + \" (y/N): \")\n if response == \"y\":\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster(\n conn, opts, cluster_name)\n print \"Terminating master...\"\n for inst in master_nodes:\n inst.terminate()\n print \"Terminating slaves...\"\n for inst in slave_nodes:\n inst.terminate()\n if zoo_nodes != []:\n print \"Terminating zoo...\"\n for inst in zoo_nodes:\n inst.terminate()\n\n elif action == \"login\":\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster(\n conn, opts, cluster_name)\n master = master_nodes[0].public_dns_name\n print \"Logging into master \" + master + \"...\"\n proxy_opt = \"\"\n if opts.proxy_port != None:\n proxy_opt = \"-D \" + opts.proxy_port\n subprocess.check_call(\"ssh -o StrictHostKeyChecking=no -i %s %s ubuntu@%s\" %\n (opts.identity_file, proxy_opt, master), shell=True)\n\n elif action == \"start-hadoop\":\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster(\n conn, opts, cluster_name)\n master = master_nodes[0].public_dns_name\n print \"Staring hadoop on master \" + master + \"...\"\n proxy_opt = \"\"\n if opts.proxy_port != None:\n proxy_opt = \"-D \" + opts.proxy_port\n subprocess.check_call(\"\"\"ssh -o StrictHostKeyChecking=no -i %s %s ubuntu@%s \\\"export PATH=$PATH:/opt/hadoop-1.2.1/bin;\n export CLASSPATH=$CLASSPATH:.:\\`hadoop classpath\\`;\n export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-amd64/;\n alias mpiexec='mpiexec.openmpi -hostfile ~/machines -x CLASSPATH -x JAVA_HOME'; /home/ubuntu/graphlab/scripts/ec2_tools/setup-hadoop\\\"\"\"\" % (opts.identity_file, proxy_opt, master), shell=True)\n\n elif action == \"check-hadoop\":\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster(\n conn, opts, cluster_name)\n master = master_nodes[0].public_dns_name\n print \"Checking hadoop on master \" + master + \"...\"\n proxy_opt = \"\"\n if opts.proxy_port != None:\n proxy_opt = \"-D \" + opts.proxy_port\n subprocess.check_call(\"\"\"ssh -o StrictHostKeyChecking=no -i %s %s ubuntu@%s \\\"export PATH=$PATH:/opt/hadoop-1.2.1/bin;\n export CLASSPATH=$CLASSPATH:.:\\`hadoop classpath\\`;\n export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-amd64/;\n jps\\\"\"\"\" % (opts.identity_file, proxy_opt, master), shell=True)\n\n elif action == \"stop-hadoop\":\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster(\n conn, opts, cluster_name)\n master = master_nodes[0].public_dns_name\n print \"Staring hadoop on master \" + master + \"...\"\n proxy_opt = \"\"\n if opts.proxy_port != None:\n proxy_opt = \"-D \" + opts.proxy_port\n subprocess.check_call(\"\"\"ssh -o StrictHostKeyChecking=no -i %s %s ubuntu@%s \\\"export PATH=$PATH:/opt/hadoop-1.2.1/bin;\n export CLASSPATH=$CLASSPATH:.:\\`hadoop classpath\\`;\n export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-amd64/;\n alias mpiexec='mpiexec -hostfile ~/machines -x CLASSPATH'; /home/ubuntu/graphlab/deps/hadoop/src/hadoop/bin/stop-all.sh\\\"\"\"\" % (opts.identity_file, proxy_opt, master), shell=True)\n\n elif action == \"als_demo\":\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster( conn, opts, cluster_name)\n master = master_nodes[0].public_dns_name\n print \"Running ALS demo on master \" + master + \"...\"\n proxy_opt = \"\"\n download_dataset = \"rm -fR smallnetflix; mkdir smallnetflix; cd smallnetflix/; wget -q http://graphlab.org/wp-content/uploads/2013/07/smallnetflix_mm.validate.gz; wget http://graphlab.org/wp-content/uploads/2013/07/smallnetflix_mm.train_.gz; gunzip *.gz; mv smallnetflix_mm.train_ smallnetflix_mm.train;cd ..;\"\n if opts.proxy_port != None:\n proxy_opt = \"-D \" + opts.proxy_port\n subprocess.check_call(\"\"\"ssh -o StrictHostKeyChecking=no -i %s %s ubuntu@%s \\\"\n cd graphlab/release/toolkits/collaborative_filtering/;\n %s\n mpiexec.openmpi -hostfile ~/machines -n %d /home/ubuntu/graphlab/release/toolkits/collaborative_filtering/als --matrix /home/ubuntu/graphlab/release/toolkits/collaborative_filtering/smallnetflix/ --max_iter=5 --ncpus=%d --predictions=out_predictions --minval=1 --maxval=5 --D=100;\n \\\"\"\"\" % (opts.identity_file, proxy_opt, master, (\"\" if opts.resume else download_dataset), opts.slaves+1,compilation_threads), shell=True)\n elif action == \"pagerank_demo\":\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster( conn, opts, cluster_name)\n master = master_nodes[0].public_dns_name\n print \"Running pagerank demo on master \" + master + \"...\"\n proxy_opt = \"\"\n download_dataset = \"rm -fR livejournal; mkdir livejournal; cd livejournal/; wget -q http://snap.stanford.edu/data/soc-LiveJournal1.txt.gz; gunzip *.gz; cd ..;\"\n if opts.proxy_port != None:\n proxy_opt = \"-D \" + opts.proxy_port\n subprocess.check_call(\"\"\"ssh -o StrictHostKeyChecking=no -i %s %s ubuntu@%s \\\"\n cd /home/ubuntu/graphlab/release/toolkits/graph_analytics/;\n %s\n mpiexec.openmpi -hostfile ~/machines -n %d /home/ubuntu/graphlab/release/toolkits/graph_analytics/pagerank --graph=/home/ubuntu/graphlab/release/toolkits/graph_analytics/livejournal/ --format=tsv --ncpus=%d --iterations=5 ;\n \\\"\"\"\" % (opts.identity_file, proxy_opt, master,(\"\" if opts.resume else download_dataset), opts.slaves+1,compilation_threads), shell=True)\n elif action == \"svd_demo\":\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster( conn, opts, cluster_name)\n master = master_nodes[0].public_dns_name\n print \"Running SVD demo on master \" + master + \"...\"\n proxy_opt = \"\"\n download_dataset = \"rm -fR livejournal; mkdir livejournal; cd livejournal/; wget -q http://snap.stanford.edu/data/soc-LiveJournal1.txt.gz; gunzip *.gz; cd ..;\"\n if opts.proxy_port != None:\n proxy_opt = \"-D \" + opts.proxy_port\n subprocess.check_call(\"\"\"ssh -o StrictHostKeyChecking=no -i %s %s ubuntu@%s \\\"\n cd graphlab/release/toolkits/collaborative_filtering/;\n %s \n mpiexec.openmpi -hostfile ~/machines -n %d /home/ubuntu/graphlab/release/toolkits/collaborative_filtering/svd --matrix /home/ubuntu/graphlab/release/toolkits/collaborative_filtering/livejournal --rows=4847572 --cols=4847571 --nsv=2 --nv=7 --max_iter=3 --tol=1e-2 --binary=true --save_vectors=1 --ncpus=%d --input_file_offset=0 --ortho_repeats=1 ;\n \\\"\"\"\" % (opts.identity_file, proxy_opt, master, (\"\" if opts.resume else download_dataset), opts.slaves+1, compilation_threads), shell=True)\n\n\n \n elif action == \"update\":\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster(\n conn, opts, cluster_name)\n master = master_nodes[0].public_dns_name\n print \"Running software update on master \" + master + \"...\"\n proxy_opt = \"\"\n if opts.proxy_port != None:\n proxy_opt = \"-D \" + opts.proxy_port\n scp(master, opts, \"machines\", '~/machines')\n subprocess.check_call(\"\"\"ssh -o StrictHostKeyChecking=no -i %s %s ubuntu@%s \\\"export PATH=$PATH:/bin/hadoop-1.2.1/bin/;\n export CLASSPATH=$CLASSPATH:.:`/bin/hadoop-1.2.1/bin/hadoop classpath`;\n export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-amd64/;\n alias mpiexec='mpiexec -hostfile ~/machines -x CLASSPATH'; \n sudo chmod -R a+rx /home/ubuntu/graphlab/deps/hadoop/; #DB: ugly, but sovles libhdfs bug\n cd graphlab/;\n git pull;\n ./configure; \n cd release/toolkits/collaborative_filtering/; \n make -j %d; \n cd ../graph_analytics/;\n make -j %d;\n cd ~/graphlab/release/toolkits; \n bash -x ~/graphlab/scripts/mpirsync\n \\\"\"\"\" % (opts.identity_file, proxy_opt, master, compilation_threads, compilation_threads), shell=True)\n\n elif action == \"update-dbg\":\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster(\n conn, opts, cluster_name)\n master = master_nodes[0].public_dns_name\n print \"Running software update on master \" + master + \"...\"\n proxy_opt = \"\"\n if opts.proxy_port != None:\n proxy_opt = \"-D \" + opts.proxy_port\n subprocess.check_call(\"\"\"ssh -o StrictHostKeyChecking=no -i %s %s ubuntu@%s \\\"\n sudo apt-get install gdb; \n cd graphlab/;\n hg pull; hg update; ./configure; cd debug; make; cd ~/graphlab/debug/toolkits; ~/graphlab/scripts/mpirsync\n \\\"\"\"\" % (opts.identity_file, proxy_opt, master), shell=True)\n\n elif action == \"get-master\":\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster(conn, opts, cluster_name)\n print master_nodes[0].public_dns_name\n \n\n elif action == \"stop\":\n response = raw_input(\"Are you sure you want to stop the cluster \" +\n cluster_name + \"?\\nDATA ON EPHEMERAL DISKS WILL BE LOST, \" +\n \"BUT THE CLUSTER WILL KEEP USING SPACE ON\\n\" + \n \"AMAZON EBS IF IT IS EBS-BACKED!!\\n\" +\n \"Stop cluster \" + cluster_name + \" (y/N): \")\n if response == \"y\":\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster(\n conn, opts, cluster_name)\n print \"Stopping master...\"\n for inst in master_nodes:\n if inst.state not in [\"shutting-down\", \"terminated\"]:\n inst.stop()\n print \"Stopping slaves...\"\n for inst in slave_nodes:\n if inst.state not in [\"shutting-down\", \"terminated\"]:\n inst.stop()\n if zoo_nodes != []:\n print \"Stopping zoo...\"\n for inst in zoo_nodes:\n if inst.state not in [\"shutting-down\", \"terminated\"]:\n inst.stop()\n\n elif action == \"start\":\n (master_nodes, slave_nodes, zoo_nodes) = get_existing_cluster(\n conn, opts, cluster_name)\n print \"Starting slaves...\"\n for inst in slave_nodes:\n if inst.state not in [\"shutting-down\", \"terminated\"]:\n inst.start()\n print \"Starting master...\"\n for inst in master_nodes:\n if inst.state not in [\"shutting-down\", \"terminated\"]:\n inst.start()\n if zoo_nodes != []:\n print \"Starting zoo...\"\n for inst in zoo_nodes:\n if inst.state not in [\"shutting-down\", \"terminated\"]:\n inst.start()\n wait_for_cluster(conn, opts.wait, master_nodes, slave_nodes, zoo_nodes)\n setup_cluster(conn, master_nodes, slave_nodes, zoo_nodes, opts, cluster_name, False)\n\n elif action == \"attach-ebs\":\n success = attach_ebs(conn, opts, cluster_name)\n if success:\n print \"ebs has been attached to masternode at /dev/sdh.\"\n\n elif action == \"detach-ebs\":\n success = detach_ebs(conn, opts, cluster_name)\n if success:\n print \"ebs has been detached.\"\n\n\n else:\n print >> stderr, \"Invalid action: %s\" % action\n sys.exit(1)\n\n\nif __name__ == \"__main__\":\n logging.basicConfig()\n main()\n" }, { "path": "scripts/ec2/readme", "content": "*The new ec2 aims to simplify the procedure of launching EC2 nodes with ready-to-go GraphLab environment.\n**The scripts are adapted from Spark's ec2 script. This document is also a variant of Spark's EC2 Script document at https://github.com/mesos/spark/wiki/EC2-Scripts\n\n\nBefore you start:\nCreate an Amazon EC2 key pair for yourself. This can be done by logging into your Amazon Web Services account through the AWS console, clicking Key Pairs on the left sidebar, and creating and downloading a key. Make sure that you set the permissions for the private key file to 600 (i.e. only you can read and write it) so that ssh will work.\n\nWhenever you want to use the gl-ec2 script, set the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to your Amazon EC2 access key ID and secret access key. These can be obtained from the AWS homepage by clicking Account > Security Credentials > Access Credentials.\n\n\n\nLaunching a Cluster\nGo into the scripts/ec2 directory in the release of GraphLab you downloaded.\nRun ./gl-ec2 -k <keypair> -i <key-file> -s <num-slaves> launch <cluster-name>, where <keypair> is the name of your EC2 key pair (that you gave it when you created it), <key-file> is the private key file for your key pair, <num-slaves> is the number of slave nodes to launch (try 1 at first), and <cluster-name> is the name to give to your cluster.\nYou can also run ./gl-ec2 --help to see more usage options.\n\nThe following options are worth pointing out:\n--ami={\"std\", \"hpc\", AMIID} can be used to specify the GraphLab AMI. The default is \"std\" for a standard cluster image, and \"hpc\" is for HPC cluster image. You can also specify your own AMIID. Notice that the ami you choose should be compatible with the instance type you use (see below). \"std\" and \"hpc\" refers to the latest ami we have. The ami id is stored on our s3 bucket named graphlabv2-ami. We can update the ami pointer there.\n\n--instance-type=<INSTANCE_TYPE> can be used to specify an EC2 instance type to use. The default type is m1.large (which has 2 cores and 7.5 GB RAM). Refer to the Amazon pages about EC2 instance types and EC2 pricing for information about other instance types. If you choose \"hpc\" as your ami above, you need to use cc type instance.\n\n--zone=<EC2_ZONE> can be used to specify an EC2 availability zone to launch instances in. Sometimes, you will get an error because there is not enough capacity in one zone, and you should try to launch in another. This happens mostly with the m1.large instance types; extra-large (both m1.xlarge and c1.xlarge) instances tend to be more available.\n\n--ebs-vol-size=GB will attach an EBS volume with a given amount of space to each node so that you can have a persistent HDFS cluster on your nodes across cluster restarts (see below).\nIf one of your launches fails due to e.g. not having the right permissions on your private key file, you can run launch with the --resume option to restart the setup process on an existing cluster.\n\n--ebs-vol-id=<EBS_VOL_ID> can be used to specify an ebs volume to be attached or detached. The availability zone of the volume must be the same as your instances.\n\n\nHere are a few common use cases:\n\n1. Start a 32 nodes (31 slaves + 1 master) standard cluster named \"test\":\n\n./gl-ec2 -k graphlabkey -i ~/.ssh/graphlab.pem -s 31 launch test\n\nOr Start a 32 nodes hpc cluster named \"test-hpc\" :\n\n./gl-ec2 -k graphlabkey -i ~/.ssh/graphlab.pem -s 31 --ami hpc --instance-type cc1.4xlarge launch test-hpc\n\n2. You can see the ip address of the master node you just created by:\n./gl-ec2 -k graphlabkey -i ~/.ssh/graphlab.pem get-master test\n\n3. You can attach the ebs volume whose id is AAA to the master node by:\n./gl-ec2 -k graphlabkey -i ~/.ssh/graphlab.pem --ebs-vol-id AAA attach-ebs test\n\n4. Login the master node by:\n./gl-ec2 -k graphlabkey -i ~/.ssh/graphlab.pem login test\n\n5. After you are done, destroy the cluster:\n./gl-ec2 -k graphlabkey -i ~/.ssh/graphlab.pem destroy test\n" }, { "path": "scripts/ec2_tools/scatter", "content": "#!/bin/bash\n\nsrc_path=$(hostname):$1\ndest_path=$2\n\necho \"Copying $src_path to $dest_path\"\nmpiexec.openmpi -hostfile ~/machines -nolocal -pernode scp -r $src_path $dest_path \n\n" }, { "path": "scripts/ec2_tools/setup-hadoop", "content": "#!/bin/bash\n\nif [ ! -e ~/machines ]; then\necho \"A list of machines must be provided in ~/machines\"\necho \"Exiting...\"\nexit 1\nfi\n\n#db: clean ssh known_hosts cache in case ip/name changed\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n sudo rm -fR ~/.ssh/known_hosts\n\nnamenode=`head -n 1 ~/machines` # first node is the master\necho \"Setting up config\"\necho -e \\\n \"export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-amd64/\\n\" > /bin/hadoop-1.2.1/conf/hadoop-env.sh\ncat /bin/hadoop-1.2.1/conf/hadoop-env.sh\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n scp $namenode:/bin/hadoop-1.2.1/conf/hadoop-env.sh /bin/hadoop-1.2.1/conf/hadoop-env.sh\n \necho \"Setting up namenode information.\"\necho \" namenode: \" $namenode\n# mpiexec.openmpi -hostfile ~/machines -pernode \\\n# ~/bin/set-namenode.sh $namenode /bin/hadoop-1.2.1/conf/core-site.xml\n\necho -e \\\n '<?xml version=\"1.0\"?>\\n'\\\n '<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>\\n'\\\n '<configuration>\\n'\\\n ' <property>\\n'\\\n ' <name>fs.default.name</name>\\n'\\\n ' <value>hdfs://'$namenode'/</value>\\n'\\\n ' </property>\\n'\\\n ' <property>\\n'\\\n ' <name>hadoop.tmp.dir</name>\\n'\\\n ' <value>/mnt/hadoop/tmp</value>\\n'\\\n ' </property>\\n'\\\n '</configuration>\\n' > /bin/hadoop-1.2.1/conf/core-site.xml\ncat /bin/hadoop-1.2.1/conf/core-site.xml\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n scp $namenode:/bin/hadoop-1.2.1/conf/core-site.xml /bin/hadoop-1.2.1/conf/core-site.xml\n\n\necho -e \\\n '<?xml version=\"1.0\"?>\\n' \\\n '<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>\\n' \\\n '<configuration>\\n' \\\n ' <property>\\n' \\\n ' <name>mapred.job.tracker</name>\\n' \\\n ' <value>'$namenode:19001'</value>\\n' \\\n ' </property>\\n' \\\n ' <property>\\n' \\\n ' <name>mapred.local.dir</name>\\n' \\\n ' <value>/mnt/hadoop/mapred</value>\\n' \\\n ' </property>\\n' \\\n ' <property> \\n' \\\n ' <name>mapred.tasktracker.map.tasks.maximum</name>\\n' \\\n ' <value>32</value> \\n' \\\n ' </property> \\n' \\\n ' <property> \\n' \\\n ' <name>mapred.tasktracker.reduce.tasks.maximum</name>\\n' \\\n ' <value>32</value> \\n' \\\n ' </property> \\n' \\\n ' <property> \\n' \\\n ' <name>mapred.reduce.tasks</name>\\n' \\\n ' <value>4</value> \\n' \\\n ' </property> \\n' \\\n ' <property> \\n' \\\n ' <name>mapred.child.java.bins</name>\\n' \\\n ' <value>-Xmx1500m</value> \\n' \\\n ' </property> \\n' \\\n '</configuration>\\n' > /bin/hadoop-1.2.1/conf/mapred-site.xml\ncat /bin/hadoop-1.2.1/conf/mapred-site.xml\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n scp $namenode:/bin/hadoop-1.2.1/conf/mapred-site.xml /bin/hadoop-1.2.1/conf/mapred-site.xml\n\n\n\necho -e \\\n '<?xml version=\"1.0\"?>\\n' \\\n '<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>\\n' \\\n '<configuration>\\n' \\\n ' <property>\\n' \\\n ' <name>dfs.name.dir</name>\\n' \\\n ' <value>/mnt/hadoop/nn</value>\\n' \\\n ' </property>\\n' \\\n ' <property>\\n' \\\n ' <name>dfs.data.dir</name>\\n' \\\n ' <value>/mnt/hadoop/hdfs</value>\\n' \\\n ' </property>\\n' \\\n ' <property> \\n' \\\n ' <name>dfs.replication</name>\\n' \\\n ' <value>3</value> \\n' \\\n ' </property> \\n' \\\n '</configuration>\\n' > /bin/hadoop-1.2.1/conf/hdfs-site.xml\ncat /bin/hadoop-1.2.1/conf/hdfs-site.xml\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n scp $namenode:/bin/hadoop-1.2.1/conf/hdfs-site.xml /bin/hadoop-1.2.1/conf/hdfs-site.xml\n\n\necho $namenode > /bin/hadoop-1.2.1/conf/masters\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n scp $namenode:/bin/hadoop-1.2.1/conf/masters /bin/hadoop-1.2.1/conf/masters\n\n\n\n\necho \"Creating data directories in /mnt\"\necho \" /mnt/tmp \"\necho \" /mnt/hadoop \"\n\n\n# mpiexec.openmpi -hostfile ~/machines -pernode \\\n# sudo rm -rf /mnt/hadoop/hdfs\n# mpiexec.openmpi -hostfile ~/machines -pernode \\\n# sudo rm -rf /mnt/hadoop/nn\n\n#DB: clean old hadoop files in case they are there\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n sudo rm -fR /mnt/hadoop/ \nmpiexec.openmpi -hostfile ~/machines -pernode \\\n sudo mkdir /mnt/hadoop/\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n sudo mkdir /mnt/hadoop/hdfs\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n sudo mkdir /mnt/hadoop/mapred\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n sudo mkdir /mnt/hadoop/tmp\n# mpiexec.openmpi -hostfile ~/machines -pernode \\\n# sudo mkdir /mnt/hadoop/nn\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n sudo chown -R ubuntu:users /mnt/hadoop/\n\n\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n sudo rm -fR /mnt/tmp/\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n sudo mkdir /mnt/tmp\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n sudo chown -R ubuntu:users /mnt/tmp\n\necho \"Machines file: \"\ncat ~/machines\nwhile read line\ndo\n echo \"Setting host name to $line\"\n mpiexec.openmpi -host $line -pernode sudo hostname $line\ndone < ~/machines\n\nexport JAVA_HOME=/usr/lib/jvm/java-6-openjdk-amd64/jre/\n\n/bin/hadoop-1.2.1/bin/hadoop namenode -format\n\n\necho \"Starting the dfs:\"\n/bin/hadoop-1.2.1/bin/start-dfs.sh\necho \"Starting map reduce:\"\n/bin/hadoop-1.2.1/bin/start-mapred.sh\n\n\n" }, { "path": "scripts/ec2_tools/setup-torque", "content": "#!/bin/bash\n\nif [ ! -e ~/machines ]; then\necho \"A list of machines must be provided in ~/machines\"\necho \"Exiting...\"\nexit 1\nfi\n\nnamenode=$(hostname)\n\necho \"Install Torque\"\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n sudo apt-get --yes install torque-server torque-scheduler \\\n torque-client torque-common torque-mom \n\necho \"Configuring Server\"\nsudo bash -c \"echo $namenode > /etc/torque/server_name\"\nsudo cp ~/machines /var/spool/torque/server_priv/nodes\nsudo qterm\nsudo pbs_server\n\necho \"Configuring MOM on remote machines\"\necho \"\\$pbsserver $namenode\" > /tmp/config_mom\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n scp $namenode:/tmp/config_mom /tmp/config\nmpiexec.openmpi -hostfile ~/machines -pernode \\\n sudo cp /tmp/config /var/spool/torque/mom_priv/.\n\n\nmpiexec.openmpi -hostfile ~/machines -pernode sudo momctl -s\nmpiexec.openmpi -hostfile ~/machines -pernode sudo pbs_mom\n\necho \"Configuring queue manager\"\nsudo qmgr -c 'create queue batch'\nsudo qmgr -c 'set queue batch queue_type = Execution'\nsudo qmgr -c 'set queue batch resources_default.nodes = 1'\nsudo qmgr -c 'set queue batch enabled = True'\nsudo qmgr -c 'set queue batch started = True'\nsudo qmgr -c 'set server scheduling = True'\nsudo qmgr -c 'set server default_queue = batch'\nsudo qmgr -c 'set server log_events = 511'\n\n\n\n\n" }, { "path": "scripts/install_graphlab.sh", "content": "#!/bin/bash\n\ngit clone https://code.google.com/p/graphlabapi/ \ncd graphlab\n./configure | tee install_configure_log.txt\ncd release\nmake -j2 | tee ../v2_debug_log.txt\n\n" }, { "path": "scripts/license_prepend.sh", "content": "DIR=\"$( cd -P \"$( dirname \"${BASH_SOURCE[0]}\" )\" && pwd )\"\necho $DIR\nif ! grep -q Apache $1\nthen\necho $1\ncat $DIR/../license/LICENSE_prepend.txt $1 > /tmp/out\nmv /tmp/out $1\nfi\n" }, { "path": "scripts/linux_run_script_no_jvm_template.sh", "content": "#!/bin/bash\n# This script will run the program in the same location and with the same\n# name as this script (without the .sh). Passing it the same set of command\n# line options.\n\n# It also parses the following environment variables\n# JAVA_HOME\n# Either JAVA_HOME or JVM_SO_PATH must be set.\n# This must point to the Java home directory.\n# For instance: /usr/lib/jvm/java-6-openjdk\n# This was tested with Oracle's implementation of Java (sun-jdk or open-jdk).\n\n# JVM_SO_PATH\n# Either JAVA_HOME or JVM_SO_PATH must be set.\n# The script will expect to find libjvm.so in \n# $JAVA_HOME/jre/lib/amd64/client/libjvm.so or\n# $JAVA_HOME/jre/lib/amd64/server/libjvm.so\n# If libjvm.so is not in either locations, the script will fail. In which \n# case, you should set this variable to the directory containing libjvm.so.\n\n# USE_SYSTEM_LIBS\n# Optional. If set to 1, the system's glibc (and other system dependencies) \n# will be used instead of the provided versions. \n\n\n\n\n# get the program name. By convention we will make it so that the \n# script's name is the same as the program name. But with a \".sh\" at the end\nPROG=$0\n#strip the \".sh\" at the end of the script name\nPROG=${PROG%.sh}\n\nPROGDIR=`dirname $0`\n\nJVM_PATH=\"\"\nHAS_HADOOP=0\n\nif [ -z $USE_SYSTEM_LIBS ]; then\n USE_SYSTEM_LIBS=0\nfi\n\nif [ \"$USE_SYSTEM_LIBS\" -eq \"1\" ]; then\n echo \"Using system libs.\"\n #using system libs\n $PROG $*\nelse\n # now. where do I find the dependency directory?\n # lets for now... assume that the directory organization must be\n # /gldeps\n # /toolkits/blah\n # /toolkits/otherblah \n DEPDIR=\"$PROGDIR/../../gldeps\"\n\n LIBPATH=$DEPDIR\n if [ ! -z \"$JVM_PATH\" ]; then\n LIBPATH=$LIBPATH:$JVM_PATH\n fi\n\n if [ ! -z \"$LD_LIBRARY_PATH\" ]; then\n LIBPATH=$LIBPATH:$LD_LIBRARY_PATH\n fi\n\n $DEPDIR/ld-linux-x86-64.so.2 --library-path $LIBPATH $PROG $*\nfi\n\n" }, { "path": "scripts/linux_run_script_template.sh", "content": "#!/bin/bash\n# This script will run the program in the same location and with the same\n# name as this script (without the .sh). Passing it the same set of command\n# line options.\n\n# It also parses the following environment variables\n# JAVA_HOME\n# Either JAVA_HOME or JVM_SO_PATH must be set.\n# This must point to the Java home directory.\n# For instance: /usr/lib/jvm/java-6-openjdk\n# This was tested with Oracle's implementation of Java (sun-jdk or open-jdk).\n\n# JVM_SO_PATH\n# Either JAVA_HOME or JVM_SO_PATH must be set.\n# The script will expect to find libjvm.so in \n# $JAVA_HOME/jre/lib/amd64/client/libjvm.so or\n# $JAVA_HOME/jre/lib/amd64/server/libjvm.so\n# If libjvm.so is not in either locations, the script will fail. In which \n# case, you should set this variable to the directory containing libjvm.so.\n\n# USE_SYSTEM_LIBS\n# Optional. If set to 1, the system's glibc (and other system dependencies) \n# will be used instead of the provided versions. \n\n\n\n\n# get the program name. By convention we will make it so that the \n# script's name is the same as the program name. But with a \".sh\" at the end\nPROG=$0\n#strip the \".sh\" at the end of the script name\nPROG=${PROG%.sh}\n\nPROGDIR=`dirname $0`\n\n# ok... now we build the command line.\n# If JVM_SO_PATH is set it takes priority.\n\nJVM_PATH=\"\"\nif [ ! -z \"$JVM_SO_PATH\" ]; then\n JVM_PATH=$JVM_SO_PATH\n if [ -a \"$JVM_SO_PATH/libjvm.so\" ]; then\n echo 'libjvm.so not found in $JVM_SO_PATH/libjvm.so'\n echo \"We are going to try to run anyway. This may not work correctly.\"\n fi\nelif [ ! -z \"$JAVA_HOME\" ]; then\n if [ -a \"$JAVA_HOME/jre/lib/amd64/client/libjvm.so\" ]; then\n JVM_PATH=\"$JAVA_HOME/jre/lib/amd64/client\"\n elif [ -a \"$JAVA_HOME/jre/lib/amd64/server/libjvm.so\" ]; then\n JVM_PATH=\"$JAVA_HOME/jre/lib/amd64/server\"\n else\n echo 'libjvm.so not found in either $JAVA_HOME/jre/lib/amd64/server'\n echo 'or $JAVA_HOME/jre/lib/amd64/client'\n echo \"We are going to try to run anyway. This may not work correctly.\"\n fi\nelse\n echo \"Neither JVM_PATH or JAVA_HOME is set.\"\n echo \"We are going to try to run anyway. This may not work correctly.\"\nfi\n\n# probe for hadoop command\nHAS_HADOOP=1\nhadoop classpath > /dev/null 2>&1 || {\n echo \"hadoop command not found. HDFS unavailable\"\n HAS_HADOOP=0\n}\n\nif [ -z $USE_SYSTEM_LIBS ]; then\n USE_SYSTEM_LIBS=0\nfi\n\nif [ \"$USE_SYSTEM_LIBS\" -eq \"1\" ]; then\n echo \"Using system libs.\"\n #using system libs\n if [ $HAS_HADOOP -eq 1 ]; then\n env LD_LIBRARY_PATH=$JVM_PATH:$LD_LIBRARY_PATH CLASSPATH=`hadoop classpath` $PROG $*\n else\n env LD_LIBRARY_PATH=$JVM_PATH:$LD_LIBRARY_PATH $PROG $*\n fi\nelse\n # now. where do I find the dependency directory?\n # lets for now... assume that the directory organization must be\n # /gldeps\n # /toolkits/blah\n # /toolkits/otherblah \n DEPDIR=\"$PROGDIR/../../gldeps\"\n\n LIBPATH=$DEPDIR\n if [ ! -z \"$JVM_PATH\" ]; then\n LIBPATH=$LIBPATH:$JVM_PATH\n fi\n\n if [ ! -z \"$LD_LIBRARY_PATH\" ]; then\n LIBPATH=$LIBPATH:$LD_LIBRARY_PATH\n fi\n\n if [ $HAS_HADOOP -eq 1 ]; then\n env CLASSPATH=`hadoop classpath` $DEPDIR/ld-linux-x86-64.so.2 --library-path $LIBPATH $PROG $*\n else\n $DEPDIR/ld-linux-x86-64.so.2 --library-path $LIBPATH $PROG $*\n fi\nfi\n\n" }, { "path": "scripts/make_all_docs.sh", "content": "doxygen\ndoxygen Doxyfile_internal \n" }, { "path": "scripts/make_dist.sh", "content": "#!/bin/bash\n\nmajor_version=2.1\n\necho \"THIS MUST BE RUN IN GRAPHLAB HOME\"\n\n## JOEY: WHY ARE WE REMOVING THE FOLDER AND THEN USING RSYNC?\nrm -fR dist/graphlabapi\nmkdir -p dist/graphlabapi\nrsync -vv -al --delete --delete-excluded \\\n --exclude=/debug --exclude=/release --exclude=/profile --exclude=/apps \\\n --exclude=.hg --exclude=/matlab \\\n --exclude=/dist --exclude=/deps --exclude=*~ --exclude=*.orig --exclude=/configure.deps \\\n --exclude /make_dist --exclude /BINARY_README * dist/graphlabapi/.\n\nmkdir dist/graphlabapi/apps\ncp dist/graphlabapi/demoapps/CMakeLists.txt dist/graphlabapi/apps/\nversion=`hg summary | grep parent | sed 's/parent: //g' | sed 's/:.*//g'`\nversion=\"v${major_version}.$version\"\necho \"Version: $version\"\n\n\ncd dist\ntar -vz \\\n -cf graphlabapi_${version}.tar.gz \\\n graphlabapi\ncd ..\n\nls -al dist | tail -n 1\n\n" }, { "path": "scripts/mpi_redirect_stdout.sh", "content": "#!/bin/bash\n\nif [ ! -z \"$PMI_RANK\" ]; then\nRANK=$PMI_RANK\nelif [ ! -z \"$OMPI_COMM_WORLD_RANK\" ]; then\nRANK=$OMPI_COMM_WORLD_RANK\nelse\necho \"Unable to figure out MPI Rank!\"\nexit 1\nfi\n#echo $RANK\n$* 2>&1 | tee out.$RANK \n" }, { "path": "scripts/mpirsync", "content": "#!/bin/bash\n\nsrc_path=$(hostname):$PWD\ndest_path=$PWD\nmpiexec.openmpi -hostfile ~/machines -nolocal -pernode mkdir -p $dest_path\nmpiexec.openmpi -hostfile ~/machines -nolocal -pernode rsync -e 'ssh -o StrictHostKeyChecking=no -i /home/ubuntu/.ssh/id_rsa' -avz --exclude '*.make' --exclude '*.cmake' --exclude '*.internal' --exclude '*.includecache' --exclude '*.o' $src_path/ $dest_path\n" }, { "path": "scripts/rpcexec.py", "content": "#!/usr/bin/python\nimport sys\nimport os\nimport string\nimport subprocess\nimport time\n\n\"\"\"\nUsage: rpcexec -n n_to_start -f [hostsfile] [program] [options]\nTo start local only: rpcexec [program] [options]\n\"\"\"\n\ndef escape(s):\n s = string.replace(s, '\"', '\\\\\"')\n s = string.replace(s, \"'\", \"\\\\'\")\n return s\n#enddef\n\n# gui: if xterm should run\n# machines: a vector of all the machines\n# port: a vector of the port number for ssh to connect to. must be same length as machines\n# machineid: The machineid to generate\n# prog: program to run\n# opts: options for the program\ndef get_ssh_cmd(gui, machines, port, machineid, prog, opts):\n\n allmachines = '\"' + string.join(machines, ',') + '\"'\n\n # construct the command line\n cwd = os.getcwd()\n if (gui):\n sshcmd = 'ssh -X -Y -n -q '\n else:\n sshcmd = 'ssh -n -q '\n #endif\n\n guicmd = ''\n if (gui):\n guicmd = 'xterm -geometry 120x60 -e '\n #endif\n\n if (machines[i] == \"localhost\" or machines[i].startswith(\"127.\")):\n cmd = 'env SPAWNNODES=%s SPAWNID=%d %s %s' % (allmachines,i, prog, opts)\n elif (port[i] == 22):\n cmd = sshcmd + '%s \"cd %s ; env SPAWNNODES=%s SPAWNID=%d %s %s %s\"' % \\\n (machines[machineid], escape(cwd), escape(allmachines),machineid, \\\n guicmd, escape(prog), escape(opts))\n else:\n cmd = sshcmd + '-oPort=%d %s \"cd %s ; env SPAWNNODES=%s SPAWNID=%d %s %s %s\"' % \\\n (port[machineid], machines[machineid], escape(cwd), escape(allmachines), \\\n machineid, guicmd, escape(prog), escape(opts))\n #endif\n return cmd\n#enddef\n\n\n\ndef get_screen_cmd(gui, machines, port, machineid, prog, opts):\n\n allmachines = '\"' + string.join(machines, ',') + '\"'\n\n # construct the command line\n cwd = os.getcwd()\n sshcmd = 'ssh -t '\n #endif\n\n guicmd = ''\n\n if (machines[i] == \"localhost\" or machines[i].startswith(\"127.\")):\n cmd = ['export SPAWNNODES=%s SPAWNID=%d ; %s %s' % (allmachines,i, prog, opts)]\n elif (port[i] == 22):\n cmd = [sshcmd + '%s \"cd %s ; export SPAWNNODES=%s SPAWNID=%d; %s %s %s ; bash -il\"' % \\\n (machines[machineid], escape(cwd), escape(allmachines),machineid, \\\n guicmd, escape(prog), escape(opts))]\n else:\n cmd = [sshcmd + '-oPort=%d %s \"cd %s ; export SPAWNNODES=%s SPAWNID=%d; %s %s %s ; bash -il\"' % \\\n (port[machineid], machines[machineid], escape(cwd), escape(allmachines), \\\n machineid, guicmd, escape(prog), escape(opts))]\n #endif\n return cmd\n#enddef\n\n\ndef shell_popen(cmd):\n print cmd\n return subprocess.Popen(cmd, shell=True)\n#endif\n\ndef shell_wait_native(cmd):\n print cmd\n pid = subprocess.Popen(cmd, shell=True)\n os.waitpid(pid.pid, 0)\n #time.sleep(0.5)\n#endif\n\n\nnmachines = 0\nhostsfile = ''\nprog = ''\nopts = ''\ngui = 0\ninscreen = 0\nscreenname = ''\nprinthelp = 0\ni = 1\nwhile(i < len(sys.argv)):\n if sys.argv[i] == '-h' or sys.argv[i] == '--help':\n printhelp = 1\n break\n elif sys.argv[i] == '-n':\n nmachines = int(sys.argv[i+1])\n i = i + 2\n elif sys.argv[i] == '-f':\n hostsfile = sys.argv[i+1]\n i = i + 2\n elif sys.argv[i] == '-g':\n gui = 1\n i = i + 1\n elif sys.argv[i] == '-s':\n inscreen = 1\n screenname = sys.argv[i+1]\n i = i + 2\n else:\n prog = sys.argv[i]\n if (len(sys.argv) > i+1):\n opts = string.join(sys.argv[(i+1):])\n #endif\n break\n #endif\n#endwhile\nif inscreen and gui:\n print (\"-s and -g are mutually exclusive\")\n exit(0)\n#endif\n\nif (printhelp):\n print\n print(\"Usage: rpcexec -n [n_to_start] -f [hostsfile] [program] [options]\")\n print(\"To start local only: rpcexec [program] [options]\")\n print(\"Optional Arguments:\")\n print(\"-g: Launch the command within Xterm on all machines. \")\n print(\"-s [screenname] : Launch a screen session and launch the\")\n print(\" commands in each window in each window. Any ssh connections\")\n print(\" are preserved on termination of the program with environment\")\n print(\" properly set up for subsequent executions\")\n print(\"\")\n print(\"Note: -s [screenname] and -g are mutually exclusive\")\n \n exit(0)\n#endif\n\nif (nmachines == 0 and hostsfile == ''):\n cmd = 'env SPAWNNODES=localhost SPAWNID=0 %s %s' % (prog, opts)\n p = shell_popen(cmd)\n os.waitpid(p.pid, 0)\n exit(0)\n#endif\nprint('Starting ' + str(nmachines) + ' machines')\nprint('Hosts file: ' + hostsfile)\nprint('Command Line to run: ' + prog + ' ' + opts)\n\n\n\n\n\n# open the hosts file and read the machines\ntry:\n f = open(hostsfile, 'r')\nexcept:\n print\n print(\"Unable to open hosts file\")\n print\n exit(0)\n#endtry\n\nmachines = [''] * nmachines\nport = [22] * nmachines\nfor i in range(nmachines):\n try:\n machines[i] = string.strip(f.readline())\n colonsplit = string.split(machines[i], ':')\n if (len(colonsplit) == 2):\n machines[i] = string.strip(colonsplit[0])\n port[i] = int(colonsplit[1])\n #endif\n except:\n print\n print(\"Unable to read line \" + str(i+1) + \" of hosts file\")\n print\n exit(0)\n#endfor\nf.close()\n\n# the commands to run to start for each node\ncmd = [None] * nmachines\nfor i in range(nmachines):\n if (inscreen == 0):\n cmd[i] = get_ssh_cmd(gui, machines, port, i, prog, opts)\n else:\n cmd[i] = get_screen_cmd(gui, machines, port, i, prog, opts)\n print cmd[i]\n #endif\n#endfor\n\nif (inscreen == 0):\n # now issue the ssh commands\n procs = [None] * nmachines\n for i in range(nmachines):\n procs[i] = shell_popen(cmd[i])\n #endfor\n \n for i in range(nmachines):\n os.waitpid(procs[i].pid, 0)\n #endfor\nelse:\n # create a new empty screen with the screen name\n shell_wait_native(\"screen -h 10000 -d -m -S \" + screenname)\n shell_wait_native(\"screen -h 10000 -x %s -p 0 -X title %s\" % (screenname, machines[0][0:8]))\n\n # start a bunch of empty screens\n for i in range(nmachines - 1):\n shell_wait_native(\"screen -x %s -X screen -t %s\" % (screenname, machines[i+1][0:8]))\n #endfor\n # set the titles in each one and run the program\n # we stripe it across windows so if there are ssh commands they will \n # have time to finish running first\n for j in range(2):\n for i in range(nmachines):\n if (len(cmd[i]) > j and cmd[i][j] != None):\n shell_wait_native(\"screen -x %s -p %d -X stuff %s\" % (screenname, i, \"'\"+cmd[i][j]+\"\\n'\"))\n #endif\n #endfor\n #endfor\n#endif\n" }, { "path": "scripts/test_dist.sh", "content": "#!/bin/sh\n\n#script for auto test graphlab distribution, written by danny bickson\nif [ $# -ne 1 ]; then\n echo \"Usage: $0 <release number>\"\nfi\nrm -fR /tmp/graphlabapi*\ncp dist/graphlabapi_v1_$1.tar.gz /tmp/\ncd /tmp/\n\ntar xvzf graphlabapi_v1_$1.tar.gz\ncd /tmp/graphlabapi\n./configure --bootstrap --yes\ncd release\nmake -j8\n\ncd tests\n./runtests.sh\n" }, { "path": "src/CMakeLists.txt", "content": "project(GraphLab)\n\nsubdirs(graphlab)\n\n" }, { "path": "src/graphlab/CMakeLists.txt", "content": "project(GraphLab)\n\n# link_libraries(${Boost_LIBRARIES})\n\n\nsubdirs(\n# aggregation\n engine\n graph\n options\n parallel\n scheduler\n util\n serialization\n logger\n ui\n# jni\n )\n\n\n\n#build the graphlab library\nadd_library(graphlab STATIC\n options/command_line_options.cpp\n options/options_map.cpp\n util/timer.cpp\n util/generics/any.cpp\n util/hdfs.cpp\n logger/logger.cpp\n logger/backtrace.cpp\n parallel/pthread_tools.cpp\n # parallel/qthread_tools.cpp\n parallel/thread_pool.cpp\n parallel/fiber_control.cpp\n parallel/fiber_group.cpp\n util/random.cpp\n scheduler/scheduler_list.cpp\n scheduler/fifo_scheduler.cpp\n scheduler/priority_scheduler.cpp\n scheduler/sweep_scheduler.cpp\n scheduler/queued_fifo_scheduler.cpp\n util/net_util.cpp\n util/safe_circular_char_buffer.cpp\n util/fs_util.cpp\n util/memory_info.cpp\n util/tracepoint.cpp\n util/mpi_tools.cpp\n util/web_util.cpp\n util/inplace_lf_queue.cpp\n zookeeper/zookeeper_common.cpp\n zookeeper/key_value.cpp\n zookeeper/server_list.cpp\n rpc/dc_tcp_comm.cpp\n rpc/circular_char_buffer.cpp\n rpc/dc_stream_receive.cpp\n rpc/dc_buffered_stream_send2.cpp\n rpc/dc.cpp\n rpc/request_reply_handler.cpp\n rpc/dc_init_from_env.cpp\n rpc/dc_init_from_mpi.cpp\n rpc/dc_init_from_zookeeper.cpp\n rpc/async_consensus.cpp\n rpc/fiber_async_consensus.cpp\n rpc/distributed_event_log.cpp\n rpc/delta_dht.cpp\n rpc/thread_local_send_buffer.cpp\n ui/mongoose/mongoose.cpp\n ui/metrics_server.cpp\n rpc/get_current_process_hash.cpp\n )\nrequires_core_deps(graphlab)\n\n\n# if(Sctp-FOUND)\n# set_property(TARGET graphlab APPEND PROPERTY COMPILE_FLAGS -DHAS_SCTP)\n# target_link_libraries(graphlab sctp)\n# endif()\n\n\n\n\nINSTALL(TARGETS \n graphlab ARCHIVE DESTINATION lib)\n\n\n\n\n" }, { "path": "src/graphlab/aggregation/CMakeLists.txt", "content": "project(GraphLab)\n\n\n\n\n" }, { "path": "src/graphlab/aggregation/aggregation_includes.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n \n#include <graphlab/aggregation/iaggregator.hpp>\n" }, { "path": "src/graphlab/aggregation/distributed_aggregator.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_DISTRIBUTED_AGGREGATOR\n#define GRAPHLAB_DISTRIBUTED_AGGREGATOR\n\n#ifndef __NO_OPENMP__\n#include <omp.h>\n#endif\n\n#include <map>\n#include <set>\n#include <string>\n#include <vector>\n#include <graphlab/rpc/dc_dist_object.hpp>\n#include <graphlab/vertex_program/icontext.hpp>\n#include <graphlab/graph/distributed_graph.hpp>\n#include <graphlab/util/generics/conditional_addition_wrapper.hpp>\n#include <graphlab/util/generics/test_function_or_functor_type.hpp>\n\n#include <graphlab/util/generics/any.hpp>\n#include <graphlab/util/timer.hpp>\n#include <graphlab/util/mutable_queue.hpp>\n#include <graphlab/logger/assertions.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n\n /**\n * \\internal\n * Implements a distributed aggregator interface which can be plugged\n * into the engine. This class includes management of periodic aggregators.\n * \n * Essentially, the engine should ideally pass-through all calls to\n * - add_vertex_aggregator()\n * - add_edge_aggregator()\n * - aggregate_now()\n * - aggregate_periodic()\n * \n * On engine start(), the engine should call aggregate_all_periodic() \n * to ensure all periodic aggregators are called once prior to vertex program\n * execution. After which, the start() function should be called to prepare\n * the state of the schedule. At termination of the engine, the stop()\n * function should be called to reset the state of the aggregator.\n * \n * During engine execution, two modes of operations are permitted: \n * synchronous, and asynchronous. In a synchronous mode of execution,\n * the tick_synchronous() function should be called periodically by \n * exactly one thread on each machine, at the same time. In an asynchronous\n * mode of execution, tick_asynchronous() should be called periodically\n * on each machine by some arbitrary thread. This polls the state of the \n * schedule and activates aggregation jobs which are ready. \n * \n * tick_synchronous() and tick_asynchronous() should not be used \n * simultaneously within the same engine execution . For details on their \n * usage, see their respective documentation.\n * \n */\n template<typename Graph, typename IContext>\n class distributed_aggregator {\n public:\n typedef Graph graph_type;\n typedef typename graph_type::local_edge_list_type local_edge_list_type;\n typedef typename graph_type::local_edge_type local_edge_type;\n typedef typename graph_type::edge_type edge_type;\n typedef typename graph_type::local_vertex_type local_vertex_type;\n typedef typename graph_type::vertex_type vertex_type ;\n typedef IContext icontext_type;\n\n dc_dist_object<distributed_aggregator> rmi;\n graph_type& graph;\n icontext_type* context;\n \n private:\n \n /**\n * \\internal\n * A base class which contains a \"type-free\" specification of the\n * reduction operation, thus allowing the aggregation to be performs at\n * runtime with no other type information whatsoever.\n */\n struct imap_reduce_base {\n /** \\brief makes a copy of the current map reduce spec without copying \n * accumulator data */\n virtual imap_reduce_base* clone_empty() const = 0;\n \n /** \\brief Performs a map operation on the given vertex adding to the\n * internal accumulator */\n virtual void perform_map_vertex(icontext_type&, vertex_type&) = 0;\n \n /** \\brief Performs a map operation on the given edge adding to the\n * internal accumulator */\n virtual void perform_map_edge(icontext_type&, edge_type&) = 0;\n \n /** \\brief Returns true if the accumulation is over vertices. \n Returns false if it is over edges.*/\n virtual bool is_vertex_map() const = 0; \n \n /** \\brief Returns the accumulator stored in an any. \n (by some magic, any's can be serialized) */\n virtual any get_accumulator() const = 0;\n \n /** \\brief Combines accumulators using a second accumulator \n stored in an any (as returned by get_accumulator).\n Must be thread safe.*/\n virtual void add_accumulator_any(any& other) = 0;\n\n /** \\brief Sets the value of the accumulator\n from an any (as returned by get_accumulator).\n Must be thread safe.*/\n virtual void set_accumulator_any(any& other) = 0;\n\n \n /** \\brief Combines accumulators using a second accumulator \n stored in a second imap_reduce_base class). Must be\n thread safe. */\n virtual void add_accumulator(imap_reduce_base* other) = 0;\n \n /** \\brief Resets the accumulator */\n virtual void clear_accumulator() = 0;\n \n /** \\brief Calls the finalize operation on internal accumulator */\n virtual void finalize(icontext_type&) = 0;\n\n virtual ~imap_reduce_base() { }\n };\n \n template <typename ReductionType>\n struct default_map_types{\n typedef ReductionType (*vertex_map_type)(icontext_type&, const vertex_type&);\n typedef ReductionType (*edge_map_type)(icontext_type&, const edge_type&);\n };\n\n /**\n * \\internal\n * A templated implementation of the imap_reduce_base above.\n * \\tparam ReductionType The reduction type. (The type the map function\n * returns)\n */\n template <typename ReductionType, \n typename VertexMapperType,\n typename EdgeMapperType,\n typename FinalizerType>\n struct map_reduce_type : public imap_reduce_base {\n conditional_addition_wrapper<ReductionType> acc;\n VertexMapperType map_vtx_function;\n EdgeMapperType map_edge_function;\n FinalizerType finalize_function;\n \n bool vertex_map;\n mutex lock;\n \n /**\n * \\brief Constructor which constructs a vertex reduction\n */\n map_reduce_type(VertexMapperType map_vtx_function,\n FinalizerType finalize_function)\n : map_vtx_function(map_vtx_function),\n finalize_function(finalize_function), vertex_map(true) { }\n\n /**\n * \\brief Constructor which constructs an edge reduction. The last bool\n * is unused and allows for disambiguation between the two constructors\n */\n map_reduce_type(EdgeMapperType map_edge_function,\n FinalizerType finalize_function,\n bool)\n : map_edge_function(map_edge_function),\n finalize_function(finalize_function), vertex_map(false) { }\n\n\n void perform_map_vertex(icontext_type& context, vertex_type& vertex) {\n /** \n * A compiler error on this line is typically due to the\n * aggregator map function not having the correct type. \n *\n * Verify that the map function has the following form:\n *\n * ReductionType mapfun(icontext_type& context, const vertex_type& vertex);\n *\n * It is also possible the accumulator type \n */\n ReductionType temp = map_vtx_function(context, vertex);\n /**\n * A compiler error on this line is typically due to the\n * accumulator (ReductionType of the map function not having an\n * operator+=. Ensure that the following is available:\n *\n * ReductionType& operator+=(ReductionType& lvalue, \n * const ReductionType& rvalue);\n */\n acc += temp;\n } // end of perform_map_vertex\n \n void perform_map_edge(icontext_type& context, edge_type& edge) {\n /** \n * A compiler error on this line is typically due to the\n * aggregator map function not having the correct type. \n *\n * Verify that the map function has the following form:\n *\n * ReductionType mapfun(icontext_type& context, const edge_type& vertex);\n *\n * It is also possible the accumulator type \n */\n ReductionType temp = map_edge_function(context, edge);\n /**\n * A compiler error on this line is typically due to the\n * accumulator (ReductionType of the map function not having an\n * operator+=. Ensure that the following is available:\n *\n * ReductionType& operator+=(ReductionType& lvalue, \n * const ReductionType& rvalue);\n */\n acc += temp; \n } // end of perform_map_edge\n \n bool is_vertex_map() const {\n return vertex_map;\n }\n \n any get_accumulator() const {\n return any(acc);\n }\n \n void add_accumulator_any(any& other) {\n lock.lock();\n acc += other.as<conditional_addition_wrapper<ReductionType> >();\n lock.unlock();\n }\n\n void set_accumulator_any(any& other) {\n lock.lock();\n acc = other.as<conditional_addition_wrapper<ReductionType> >();\n lock.unlock();\n }\n\n\n void add_accumulator(imap_reduce_base* other) {\n lock.lock();\n acc += dynamic_cast<map_reduce_type*>(other)->acc;\n lock.unlock();\n }\n\n void clear_accumulator() {\n acc.clear();\n }\n\n void finalize(icontext_type& context) {\n finalize_function(context, acc.value);\n }\n \n imap_reduce_base* clone_empty() const {\n map_reduce_type* copy;\n if (is_vertex_map()) {\n copy = new map_reduce_type(map_vtx_function,\n finalize_function);\n }\n else {\n copy = new map_reduce_type(map_edge_function,\n finalize_function,\n true);\n }\n return copy;\n }\n };\n \n\n std::map<std::string, imap_reduce_base*> aggregators;\n std::map<std::string, float> aggregate_period;\n\n struct async_aggregator_state {\n /// Performs reduction of all local threads. On machine 0, also\n /// accumulates for all machines.\n imap_reduce_base* root_reducer;\n /// Accumulator used for each thread\n std::vector<imap_reduce_base*> per_thread_aggregation;\n /// Count down the completion of the local machine threads\n atomic<int> local_count_down;\n /// Count down the completion of machines. Used only on machine 0\n atomic<int> distributed_count_down;\n };\n std::map<std::string, async_aggregator_state> async_state;\n\n float start_time;\n \n /* annoyingly the mutable queue is a max heap when I need a min-heap\n * to track the next thing to activate. So we need to keep \n * negative priorities... */\n mutable_queue<std::string, float> schedule;\n mutex schedule_lock;\n size_t ncpus;\n\n template <typename ReductionType, typename F>\n static void test_vertex_mapper_type(std::string key = \"\") {\n bool test_result = test_function_or_const_functor_2<F,\n ReductionType(icontext_type&,\n const vertex_type&),\n ReductionType,\n icontext_type&,\n const vertex_type&>::value;\n if (!test_result) {\n std::stringstream strm;\n strm << \"\\n\";\n if (key.empty()) {\n strm << \"Vertex Map Function does not pass strict runtime type checks. \\n\";\n }\n else {\n strm << \"Map Function in Vertex Aggregator \" << key\n << \" does not pass strict runtime type checks. \\n\";\n }\n if (boost::is_function<typename boost::remove_pointer<F>::type>::value) {\n strm\n << \"Function prototype should be \\n\" \n << \"\\t ReductionType f(icontext_type&, const vertex_type&)\\n\";\n }\n else {\n strm << \"Functor's operator() prototype should be \\n\"\n << \"\\t ReductionType operator()(icontext_type&, const vertex_type&) const\\n\";\n }\n strm << \"If you are not intentionally violating the abstraction,\"\n << \" we recommend fixing your function for safety reasons\";\n strm.flush();\n logstream(LOG_WARNING) << strm.str() << std::endl;\n }\n }\n\n template <typename ReductionType, typename F>\n static void test_edge_mapper_type(std::string key = \"\") {\n bool test_result = test_function_or_const_functor_2<F,\n ReductionType(icontext_type&,\n const edge_type&),\n ReductionType,\n icontext_type&,\n const edge_type&>::value;\n\n if (!test_result) {\n std::stringstream strm;\n strm << \"\\n\";\n if (key.empty()) {\n strm << \"Edge Map Function does not pass strict runtime type checks. \\n\";\n }\n else {\n strm << \"Map Function in Edge Aggregator \" << key\n << \" does not pass strict runtime type checks. \\n\";\n }\n if (boost::is_function<typename boost::remove_pointer<F>::type>::value) {\n strm << \"Function prototype should be \\n\"\n << \"\\t ReductionType f(icontext_type&, const edge_type&)\\n\";\n }\n else {\n strm << \"Functor's operator() prototype should be \"\n << \"\\t ReductionType operator()(icontext_type&, const edge_type&) const\\n\";\n }\n strm << \"If you are not intentionally violating the abstraction,\"\n << \" we recommend fixing your function for safety reasons\";\n logstream(LOG_WARNING) << strm.str() << std::endl;\n }\n }\n \n public:\n\n \n distributed_aggregator(distributed_control& dc, \n graph_type& graph, \n icontext_type* context):\n rmi(dc, this), graph(graph), \n context(context), ncpus(0) { }\n\n /**\n * \\copydoc graphlab::iengine::add_vertex_aggregator\n */\n template <typename ReductionType, \n typename VertexMapperType, \n typename FinalizerType>\n bool add_vertex_aggregator(const std::string& key,\n VertexMapperType map_function,\n FinalizerType finalize_function) {\n if (key.length() == 0) return false;\n if (aggregators.count(key) == 0) {\n\n if (rmi.procid() == 0) {\n // do a runtime type check\n test_vertex_mapper_type<ReductionType, VertexMapperType>(key);\n }\n \n aggregators[key] = new map_reduce_type<ReductionType,\n VertexMapperType,\n typename default_map_types<ReductionType>::edge_map_type,\n FinalizerType>(map_function, \n finalize_function);\n return true;\n }\n else {\n // aggregator already exists. fail \n return false;\n }\n }\n \n#if defined(__cplusplus) && __cplusplus >= 201103L\n /**\n * \\brief An overload of add_vertex_aggregator for C++11 which does not\n * require the user to provide the reduction type.\n *\n * This function is available only if the compiler has C++11 support.\n * Specifically, it uses C++11's decltype operation to infer the\n * reduction type, thus eliminating the need for the function\n */\n template <typename VertexMapperType, \n typename FinalizerType>\n bool add_vertex_aggregator(const std::string& key,\n VertexMapperType map_function,\n FinalizerType finalize_function) {\n //typedef decltype(map_function(*context,graph.vertex(0))) ReductionType;\n typedef decltype(map_function(*context, graph.vertex(0))) ReductionType;\n if (key.length() == 0) return false;\n if (aggregators.count(key) == 0) {\n aggregators[key] = new map_reduce_type<ReductionType,\n VertexMapperType,\n typename default_map_types<ReductionType>::edge_map_type,\n FinalizerType>(map_function, \n finalize_function);\n return true;\n }\n else {\n // aggregator already exists. fail \n return false;\n }\n }\n#endif\n\n /**\n * \\copydoc graphlab::iengine::add_edge_aggregator\n */\n template <typename ReductionType,\n typename EdgeMapperType,\n typename FinalizerType>\n bool add_edge_aggregator(const std::string& key,\n EdgeMapperType map_function,\n FinalizerType finalize_function) {\n if (key.length() == 0) return false;\n if (aggregators.count(key) == 0) {\n if (rmi.procid() == 0) {\n // do a runtime type check\n test_edge_mapper_type<ReductionType, EdgeMapperType>(key);\n }\n aggregators[key] = new map_reduce_type<ReductionType, \n typename default_map_types<ReductionType>::vertex_map_type,\n EdgeMapperType, \n FinalizerType>(map_function, \n finalize_function, \n true);\n return true;\n }\n else {\n // aggregator already exists. fail \n return false;\n }\n }\n \n#if defined(__cplusplus) && __cplusplus >= 201103L\n /**\n * \\brief An overload of add_edge_aggregator for C++11 which does not\n * require the user to provide the reduction type.\n *\n * This function is available only if the compiler has C++11 support.\n * Specifically, it uses C++11's decltype operation to infer the\n * reduction type, thus eliminating the need for the function\n * call to be templatized over the reduction type. \n */\n template <typename EdgeMapperType,\n typename FinalizerType>\n bool add_edge_aggregator(const std::string& key,\n EdgeMapperType map_function,\n FinalizerType finalize_function) {\n // an edge_type is actually hard to get\n typedef decltype(map_function(*context, edge_type(graph.l_vertex(0).in_edges()[0]) )) ReductionType;\n if (key.length() == 0) return false;\n if (aggregators.count(key) == 0) {\n aggregators[key] = new map_reduce_type<ReductionType, \n typename default_map_types<ReductionType>::vertex_map_type,\n EdgeMapperType, \n FinalizerType>(map_function, \n finalize_function, \n true);\n return true;\n }\n else {\n // aggregator already exists. fail \n return false;\n }\n }\n#endif\n \n /**\n * \\copydoc graphlab::iengine::aggregate_now\n */\n bool aggregate_now(const std::string& key) {\n ASSERT_MSG(graph.is_finalized(), \"Graph must be finalized\");\n if (aggregators.count(key) == 0) {\n ASSERT_MSG(false, \"Requested aggregator %s not found\", key.c_str());\n return false;\n }\n \n imap_reduce_base* mr = aggregators[key];\n mr->clear_accumulator();\n // ok. now we perform reduction on local data in parallel\n#ifdef _OPENMP\n#pragma omp parallel\n#endif\n {\n imap_reduce_base* localmr = mr->clone_empty();\n if (localmr->is_vertex_map()) {\n#ifdef _OPENMP\n #pragma omp for\n#endif\n for (int i = 0; i < (int)graph.num_local_vertices(); ++i) {\n local_vertex_type lvertex = graph.l_vertex(i);\n if (lvertex.owner() == rmi.procid()) {\n vertex_type vertex(lvertex);\n localmr->perform_map_vertex(*context, vertex);\n }\n }\n }\n else {\n#ifdef _OPENMP\n #pragma omp for\n#endif\n for (int i = 0; i < (int)graph.num_local_vertices(); ++i) {\n foreach(local_edge_type e, graph.l_vertex(i).in_edges()) {\n edge_type edge(e);\n localmr->perform_map_edge(*context, edge);\n }\n }\n }\n#ifdef _OPENMP\n #pragma omp critical\n#endif\n {\n mr->add_accumulator(localmr);\n }\n delete localmr;\n }\n \n std::vector<any> gathervec(rmi.numprocs());\n gathervec[rmi.procid()] = mr->get_accumulator();\n \n rmi.gather(gathervec, 0);\n \n if (rmi.procid() == 0) {\n // machine 0 aggregates the accumulators\n // sums them together and broadcasts it\n for (procid_t i = 1; i < rmi.numprocs(); ++i) {\n mr->add_accumulator_any(gathervec[i]);\n }\n any val = mr->get_accumulator();\n rmi.broadcast(val, true);\n }\n else {\n // all other machines wait for the broadcast value\n any val;\n rmi.broadcast(val, false);\n mr->set_accumulator_any(val);\n }\n mr->finalize(*context);\n mr->clear_accumulator();\n gathervec.clear();\n return true;\n }\n \n \n /**\n * \\copydoc graphlab::iengine::aggregate_periodic\n */\n bool aggregate_periodic(const std::string& key, float seconds) {\n rmi.barrier();\n if (seconds < 0) return false;\n if (aggregators.count(key) == 0) return false;\n else aggregate_period[key] = seconds;\n return true;\n }\n \n /**\n * Performs aggregation on all keys registered with a period.\n * May be used on engine start() to ensure all periodic \n * aggregators are executed before engine execution.\n */\n void aggregate_all_periodic() {\n typename std::map<std::string, float>::iterator iter =\n aggregate_period.begin();\n while (iter != aggregate_period.end()) { \n aggregate_now(iter->first);\n ++iter;\n }\n }\n \n \n /**\n * Must be called on engine start. Initializes the internal scheduler.\n * Must be called on all machines simultaneously.\n * ncpus is really only important for the asynchronous implementation.\n * It must be equal to the number of engine threads.\n *\n * \\param [in] cpus Number of engine threads used. This is only necessary\n * if the asynchronous form is used.\n */\n void start(size_t ncpus = 0) {\n rmi.barrier();\n schedule.clear();\n start_time = timer::approx_time_seconds();\n typename std::map<std::string, float>::iterator iter =\n aggregate_period.begin();\n while (iter != aggregate_period.end()) {\n // schedule is a max heap. To treat it like a min heap\n // I need to insert negative keys\n schedule.push(iter->first, -iter->second);\n ++iter;\n }\n this->ncpus = ncpus;\n\n // now initialize the asyncronous reduction states\n if(ncpus > 0) {\n iter = aggregate_period.begin();\n while (iter != aggregate_period.end()) {\n async_state[iter->first].local_count_down = (int)ncpus;\n async_state[iter->first].distributed_count_down =\n (int)rmi.numprocs();\n \n async_state[iter->first].per_thread_aggregation.resize(ncpus);\n for (size_t i = 0; i < ncpus; ++i) {\n async_state[iter->first].per_thread_aggregation[i] =\n aggregators[iter->first]->clone_empty();\n }\n async_state[iter->first].root_reducer =\n aggregators[iter->first]->clone_empty();\n ++iter;\n }\n }\n }\n \n \n /**\n * If asynchronous aggregation is desired, this function is\n * to be called periodically on each machine. This polls the schedule to\n * check if there is an aggregator which needs to be activated. If there\n * is an aggregator to be started, this function will return a non empty\n * string. This function is thread reentrant and each activated aggregator\n * will only return a non empty string call to one call to\n * tick_asynchronous() on each machine.\n * \n * If an empty is returned, the asynchronous engine\n * must ensure that all threads (ncpus per machine) must eventually\n * call tick_asynchronous_compute(cpuid, key) where key is the return string.\n */ \n std::string tick_asynchronous() {\n // if we fail to acquire the lock, go ahead\n if (!schedule_lock.try_lock()) return \"\";\n \n // see if there is a key to run\n float curtime = timer::approx_time_seconds() - start_time;\n std::string key;\n bool has_entry = false;\n if (!schedule.empty() && -schedule.top().second <= curtime) {\n key = schedule.top().first;\n has_entry = true;\n schedule.pop();\n }\n schedule_lock.unlock();\n\n // no key to run. return false\n if (has_entry == false) return \"\";\n else return key;\n // ok. we have a key to run, construct the local reducers\n }\n\n \n /**\n * Once tick_asynchronous() returns a key, all threads in the engine\n * should call tick_asynchronous_compute() with a matching key.\n * This function will perform the computation for the key in question\n * and send the accumulated result back to machine 0 when done\n */\n void tick_asynchronous_compute(size_t cpuid, const std::string& key) {\n // acquire and check the async_aggregator_state\n typename std::map<std::string, async_aggregator_state>::iterator iter =\n async_state.find(key);\n ASSERT_MSG(iter != async_state.end(), \"Key %s not found\", key.c_str());\n ASSERT_GT(iter->second.per_thread_aggregation.size(), cpuid);\n \n imap_reduce_base* localmr = iter->second.per_thread_aggregation[cpuid];\n // perform the reduction using the local mr\n if (localmr->is_vertex_map()) {\n for (int i = cpuid;i < (int)graph.num_local_vertices(); i+=ncpus) {\n local_vertex_type lvertex = graph.l_vertex(i);\n if (lvertex.owner() == rmi.procid()) {\n vertex_type vertex(lvertex);\n localmr->perform_map_vertex(*context, vertex);\n }\n }\n } else {\n for (int i = cpuid;i < (int)graph.num_local_vertices(); i+=ncpus) {\n foreach(local_edge_type e, graph.l_vertex(i).in_edges()) {\n edge_type edge(e);\n localmr->perform_map_edge(*context, edge);\n }\n }\n }\n iter->second.root_reducer->add_accumulator(localmr);\n int countdown_val = iter->second.local_count_down.dec();\n\n ASSERT_LT(countdown_val, ncpus);\n ASSERT_GE(countdown_val, 0);\n if (countdown_val == 0) {\n // reset the async_state to pristine condition.\n // - clear all thread reducers since we got all we need from them\n // - clear all the local root reducer except for machine 0 (and after\n // we read the accumulator from them.\n // - reset the counters\n for (size_t i = 0;\n i < iter->second.per_thread_aggregation.size(); ++i) {\n iter->second.per_thread_aggregation[i]->clear_accumulator();\n }\n iter->second.local_count_down = ncpus;\n \n if (rmi.procid() != 0) {\n // ok we need to signal back to the the root to perform finalization\n // read the accumulator\n any acc = iter->second.root_reducer->get_accumulator();\n iter->second.root_reducer->clear_accumulator();\n rmi.remote_call(0, &distributed_aggregator::rpc_key_merge,\n key, acc);\n }\n else {\n decrement_distributed_counter(key);\n }\n }\n }\n\n /**\n * RPC Call called by other machines with their accumulator for the key.\n * This function will merge the accumulator and perform finalization\n * when all accumulators are received\n */\n void rpc_key_merge(const std::string& key, any& acc) {\n // acquire and check the async_aggregator_state \n typename std::map<std::string, async_aggregator_state>::iterator iter =\n async_state.find(key);\n ASSERT_MSG(iter != async_state.end(), \"Key %s not found\", key.c_str());\n iter->second.root_reducer->add_accumulator_any(acc);\n decrement_distributed_counter(key);\n }\n\n /**\n * Called whenever one machine finishes all of its local accumulation.\n * When the counter determines that all machine's accumulators have been\n * received, this function performs finalization and prepares and\n * broadcasts the next scheduled time for the key.\n */\n void decrement_distributed_counter(const std::string& key) {\n // must be master machine\n ASSERT_EQ(rmi.procid(), 0);\n // acquire and check the async_aggregator_state \n typename std::map<std::string, async_aggregator_state>::iterator iter =\n async_state.find(key);\n ASSERT_MSG(iter != async_state.end(), \"Key %s not found\", key.c_str());\n int countdown_val = iter->second.distributed_count_down.dec();\n logstream(LOG_INFO) << \"Distributed Aggregation of \" << key << \". \"\n << countdown_val << \" remaining.\" << std::endl;\n\n ASSERT_LE(countdown_val, rmi.numprocs());\n ASSERT_GE(countdown_val, 0);\n if (countdown_val == 0) {\n logstream(LOG_INFO) << \"Aggregate completion of \" << key << std::endl;\n any acc_val = iter->second.root_reducer->get_accumulator();\n // set distributed count down again for the second phase:\n // waiting for everyone to finish finalization\n iter->second.distributed_count_down = rmi.numprocs();\n for (procid_t i = 1;i < rmi.numprocs(); ++i) {\n rmi.remote_call(i, &distributed_aggregator::rpc_perform_finalize,\n key, acc_val);\n }\n iter->second.root_reducer->finalize(*context);\n iter->second.root_reducer->clear_accumulator();\n decrement_finalize_counter(key);\n }\n }\n\n /**\n * Called from the root machine to all machines to perform finalization\n * on the key\n */\n void rpc_perform_finalize(const std::string& key, any& acc_val) {\n ASSERT_NE(rmi.procid(), 0);\n typename std::map<std::string, async_aggregator_state>::iterator iter =\n async_state.find(key);\n ASSERT_MSG(iter != async_state.end(), \"Key %s not found\", key.c_str());\n \n iter->second.root_reducer->set_accumulator_any(acc_val);\n iter->second.root_reducer->finalize(*context);\n iter->second.root_reducer->clear_accumulator();\n // reply to the root machine\n rmi.remote_call(0, &distributed_aggregator::decrement_finalize_counter,\n key);\n }\n\n\n void decrement_finalize_counter(const std::string& key) {\n typename std::map<std::string, async_aggregator_state>::iterator iter =\n async_state.find(key);\n ASSERT_MSG(iter != async_state.end(), \"Key %s not found\", key.c_str());\n int countdown_val = iter->second.distributed_count_down.dec();\n if (countdown_val == 0) {\n // done! all finalization is complete.\n // reset the counter\n iter->second.distributed_count_down = rmi.numprocs();\n // when is the next time we start. \n // time is as an offset to start_time\n float next_time = timer::approx_time_seconds() + \n aggregate_period[key] - start_time;\n logstream(LOG_INFO) << rmi.procid() << \"Reschedule of \" << key\n << \" at \" << next_time << std::endl;\n rpc_schedule_key(key, next_time);\n for (procid_t i = 1;i < rmi.numprocs(); ++i) {\n rmi.remote_call(i, &distributed_aggregator::rpc_schedule_key,\n key, next_time);\n }\n }\n }\n\n /**\n * Called to schedule the next trigger time for the key\n */\n void rpc_schedule_key(const std::string& key, float next_time) {\n schedule_lock.lock();\n schedule.push(key, -next_time);\n schedule_lock.unlock();\n }\n\n \n /**\n * If synchronous aggregation is desired, this function is\n * To be called simultaneously by one thread on each machine. \n * This polls the schedule to see if there\n * is an aggregator which needs to be activated. If there is an aggregator \n * to be started, this function will perform aggregation.\n */ \n void tick_synchronous() {\n // if timer has exceeded our top key\n float curtime = timer::approx_time_seconds() - start_time;\n rmi.broadcast(curtime, rmi.procid() == 0);\n // note that we do not call approx_time_seconds everytime\n // this ensures that each key will only be run at most once.\n // each time tick_synchronous is called.\n std::vector<std::pair<std::string, float> > next_schedule;\n while(!schedule.empty() && -schedule.top().second <= curtime) {\n std::string key = schedule.top().first;\n aggregate_now(key);\n schedule.pop();\n // when is the next time we start. \n // time is as an offset to start_time\n float next_time = (timer::approx_time_seconds() + \n aggregate_period[key] - start_time);\n rmi.broadcast(next_time, rmi.procid() == 0);\n next_schedule.push_back(std::make_pair(key, -next_time));\n }\n\n for (size_t i = 0;i < next_schedule.size(); ++i) {\n schedule.push(next_schedule[i].first, next_schedule[i].second);\n }\n }\n\n /**\n * Must be called on engine stop. Clears the internal scheduler\n * And resets all incomplete states.\n */\n void stop() {\n schedule.clear();\n // clear the aggregators\n {\n typename std::map<std::string, imap_reduce_base*>::iterator iter =\n aggregators.begin();\n while (iter != aggregators.end()) {\n iter->second->clear_accumulator();\n ++iter;\n }\n }\n // clear the asynchronous state\n {\n typename std::map<std::string, async_aggregator_state>::iterator\n iter = async_state.begin();\n while (iter != async_state.end()) {\n delete iter->second.root_reducer;\n for (size_t i = 0;\n i < iter->second.per_thread_aggregation.size();\n ++i) {\n delete iter->second.per_thread_aggregation[i];\n }\n iter->second.per_thread_aggregation.clear();\n ++iter;\n }\n async_state.clear();\n }\n }\n\n\n std::set<std::string> get_all_periodic_keys() const {\n typename std::map<std::string, float>::const_iterator iter =\n aggregate_period.begin();\n std::set<std::string> ret;\n while (iter != aggregate_period.end()) {\n ret.insert(iter->first);\n ++iter;\n }\n return ret;\n }\n \n \n \n \n template <typename ResultType, typename MapFunctionType>\n ResultType map_reduce_vertices(MapFunctionType mapfunction) {\n ASSERT_MSG(graph.is_finalized(), \"Graph must be finalized\");\n\n if (rmi.procid() == 0) {\n // do a runtime type check\n test_vertex_mapper_type<ResultType, MapFunctionType>();\n }\n \n rmi.barrier();\n bool global_result_set = false;\n ResultType global_result = ResultType();\n#ifdef _OPENMP\n#pragma omp parallel\n#endif\n {\n bool result_set = false;\n ResultType result = ResultType();\n#ifdef _OPENMP\n #pragma omp for\n#endif\n for (int i = 0; i < (int)graph.num_local_vertices(); ++i) {\n if (graph.l_vertex(i).owner() == rmi.procid()) {\n if (!result_set) {\n vertex_type vtx(graph.l_vertex(i));\n result = mapfunction(*context, vtx);\n result_set = true;\n }\n else if (result_set){\n vertex_type vtx(graph.l_vertex(i));\n result += mapfunction(*context, vtx);\n }\n }\n }\n#ifdef _OPENMP\n #pragma omp critical\n#endif\n {\n if (result_set) {\n if (!global_result_set) {\n global_result = result;\n global_result_set = true;\n }\n else {\n global_result += result;\n }\n }\n }\n }\n conditional_addition_wrapper<ResultType> wrapper(global_result, global_result_set);\n rmi.all_reduce(wrapper);\n return wrapper.value;\n }\n\n\n \n template <typename ResultType, typename MapFunctionType>\n ResultType map_reduce_edges(MapFunctionType mapfunction) {\n ASSERT_MSG(graph.is_finalized(), \"Graph must be finalized\");\n \n if (rmi.procid() == 0) {\n // do a runtime type check\n test_edge_mapper_type<ResultType, MapFunctionType>();\n }\n \n rmi.barrier();\n bool global_result_set = false;\n ResultType global_result = ResultType();\n#ifdef _OPENMP\n#pragma omp parallel\n#endif\n {\n bool result_set = false;\n ResultType result = ResultType();\n#ifdef _OPENMP\n #pragma omp for\n#endif\n for (int i = 0; i < (int)graph.num_local_vertices(); ++i) {\n foreach(const local_edge_type& e, graph.l_vertex(i).in_edges()) {\n if (!result_set) {\n edge_type edge(e);\n result = mapfunction(*context, edge);\n result_set = true;\n }\n else if (result_set){\n edge_type edge(e);\n result += mapfunction(*context, edge);\n }\n }\n }\n#ifdef _OPENMP\n #pragma omp critical\n#endif\n {\n if (result_set) {\n if (!global_result_set) {\n global_result = result;\n global_result_set = true;\n }\n else {\n global_result += result;\n }\n }\n }\n }\n\n conditional_addition_wrapper<ResultType> wrapper(global_result, global_result_set);\n rmi.all_reduce(wrapper);\n return wrapper.value;\n }\n\n template <typename TransformType>\n void transform_vertices(TransformType transform_functor) {\n ASSERT_MSG(graph.is_finalized(), \"Graph must be finalized\");\n rmi.barrier();\n#ifdef _OPENMP\n #pragma omp parallel for\n#endif\n for (int i = 0; i < (int)graph.num_local_vertices(); ++i) {\n if (graph.l_vertex(i).owner() == rmi.procid()) {\n vertex_type vtx(graph.l_vertex(i));\n transform_functor(*context, vtx);\n }\n }\n rmi.barrier();\n graph.synchronize();\n }\n\n\n template <typename TransformType>\n void transform_edges(TransformType transform_functor) {\n ASSERT_MSG(graph.is_finalized(), \"Graph must be finalized\");\n rmi.barrier();\n#ifdef _OPENMP\n #pragma omp parallel for\n#endif\n for (int i = 0; i < (int)graph.num_local_vertices(); ++i) {\n foreach(const local_edge_type& e, graph.l_vertex(i).in_edges()) {\n edge_type edge(e);\n transform_functor(*context, edge);\n }\n }\n rmi.barrier();\n }\n \n \n \n \n \n ~distributed_aggregator() {\n delete context;\n }\n }; \n\n\n}; // end of graphlab namespace\n#include <graphlab/macros_undef.hpp>\n\n#endif\n" }, { "path": "src/graphlab/docs/faq.dox", "content": "/**\n \n\\page FAQ FAQ\n\n##I am trying to run GraphLab distributed using files from HDFS as input. However, I am getting screens full of errors\n\nYou may need to set the CLASSPATH environment variable.\nInstead of running:\n\n\\verbatim\nmpiexec -n N ... graphlab_program ...\n\\endverbatim\n\nTry running \n\n\\verbatim\nmpiexec -n N ... env CLASSPATH=`hadoop classpath` graphlab_program ...\n\\endverbatim\n\n##I am trying to run GraphLab distributed, but it seems to be failing to find my graph input files. I am not using HDFS.\n\nYou need to make sure that all machines have access to the graph files at exactly the same paths.\ni.e. Either you need to have an NFS file share, or a distributed file system, or you need to\ncopy all graph files to all machines.\n\n\n\n\n */\n" }, { "path": "src/graphlab/docs/overview.dox", "content": "/**\n\n \\defgroup engines GraphLab Engines\n\\defgroup util GraphLab Utility Classes and Functions\n\\defgroup rpc GraphLab RPC\n\\defgroup random Random Number Generators\n\\defgroup group_serialization Serialization\n\\defgroup toolkits GraphLab Toolkits\n\\defgroup httpserver Metrics Reporting Webserver\n\\defgroup warp Warp System\n\\mainpage \n \n\n The GraphLab project started in 2009 to develop a new parallel\n computation abstraction tailored to machine learning. GraphLab 1.0\n represents our first shared memoy design which, through the addition\n of several matrix factorization toolkits, started to grow a community of users.\n\n In the last couple of years, we have focused our development effort\n on the distributed environment. Unfortunately, it took nearly a year\n to figure out that distributing the GraphLab 1 abstraction was\n excessively complicated and is unable to scale up to power-law\n graphs commonly seen in the real world.\n\n In GraphLab 2.1, we completely redesign of the GraphLab 1 framework\n for the distributed environment. The implementation is distributed\n by design and a \"shared-memory\" execution is essentially running a\n distributed system on a cluster of 1 machine. \n \n And in this new release of GraphLab 2.2, we introduce the new \\ref warp\n which through the use of fine-grained user-mode threading, introduces a new\n API which brings about a major increase in useability, and will allow us to \n provide new capabilities more easily in the future.\n\n There are two starting points where one may begin using GraphLab. \\li \\ref\n toolkits \"Toolkits\" You can lookup the toolkit documentation here if you have\n a computation task which is already implemented by one of our toolkits. \\li\n \\ref using_graphlab \"GraphLab C++ Tutorial\" If you have a computation task\n which is not implemented by our toolkits, you could try implementing\n yourself! For now a certain degree of C++ knowledge is required. \n\n The new GraphLab 2.2 \\ref warp is available for experimentation. A\n \\ref using_warp tutorial is provided, and we are are looking for feedback \n to continue extending and improving the Warp system. Performance tuning is \n also underway.\n\n Software Stack\n =============\n \\image html software_stack.png \n \\image html system_overview.png \n\n\n*/\n" }, { "path": "src/graphlab/docs/using.dox", "content": "/**\n \\page using_graphlab Basic GraphLab Tutorial\n\n In this example, we would implement a simple PageRank application \n from scratch, demonstrating all the core GraphLab concepts from loading a \n graph to performing computation and saving the results. \n \n The implementation philosophy of the GraphLab API is to expose an\n MPI-like SPMD (Single Program Multiple Data) Interface.\n That is to say, we try to enforce the illusion that all machines \n are running the same operations in lock-step.\n\n For instance, a GraphLab program in pseudo code may look like:\n\n \\verbatim\n main() {\n ...\n Load Graph from file using parsing_function; \n global variable RESULT = map reduce on graph vertices using map_function;\n transform graph vertices using transform_function;\n ...\n create an asynchronous engine an attach it to the graph;\n\n engine.start();\n\n save Graph using saver() object;\n }\n \\endverbatim\n\n In the distributed environment, each of these operations are run in lock step.\n However, each individual operation may have significant complexity (perhaps\n even running asynchronously). To support this illusion requires the user to\n implement a number of external functions / classes. For instance, in the \n above pseudo-code, the user needs to implement a \\c map_function, a \n \\c transform_function , etc.\n\n While GraphLab's RPC implementation permits the implementation of much more\n complex computation/communication interleaving behavior, we discourage it\n and we encourage users to use the aggregate \"SPMD\"-like operations as much\n as possible. Indeed, none of the toolkit applications we implemented require\n any more than these operations.\n As we understand the abstraction needs of the community better,\n we can continue to expand on the scope of these operations.\n \n\n The tutorial is divided into the following sections:\n - \\subpage using_graphlab_create_project \n - \\subpage using_graphlab_empty_app \n - \\subpage using_graphlab_distributed_graph \n - \\subpage using_graphlab_distributed_graph_load_data \n - \\subpage using_graphlab_distributed_graph_vertex_program \n - \\subpage using_scheduling \n - \\subpage using_saving_answers \n - \\subpage using_conclusion\n\n \\page using_graphlab_create_project 1: Creating a GraphLab project\n\n To create a GraphLab project, simply create a sub-directory in the\n graphlab/apps/ folder with your project name. For instance,\n graphlab/apps/my_first_app. Within the sub-directory, create a text file\n called CMakeLists.txt with the following contents\n\n \\verbatim\n project(My_Project)\n add_graphlab_executable(my_first_app my_first_app.cpp)\n \\endverbatim\n\n The project name \"My_Project\" is an arbitrary name used to identify your\n application. <tt>add_graphlab_executable</tt> \n is a CMake macro that will compile a program called <tt>my_first_app</tt>\n using the CPP file <tt>my_first_app.cpp</tt>, and linking in all\n GraphLab libraries and dependencies.\n\n If your program needs multiple CPP files simply append to the list.\n For instance:\n \\verbatim\n add_graphlab_executable(my_first_app my_first_app.cpp tools.cpp stuff.cpp)\n \\endverbatim\n will compile and link 3 cpp files together into a single program.\n\n For more complex uses of CMake, see the Cmake documentation \n <a href=http://www.cmake.org/cmake/help/documentation.html> here</a>.\n\n In the \\ref using_graphlab_empty_app \"next section\", we will implement \"Hello World\".\n\n \\page using_graphlab_empty_app 2: Hello World in GraphLab\n\n To use GraphLab,\n \\code\n #include <graphlab.hpp>\n \\endcode\n\n All of GraphLab lives in the <code>graphlab</code> namespace. You may use\n \\code\n using namespace graphlab;\n \\endcode\n if you wish, but we \n <a href=http://www.parashift.com/c++-faq-lite/coding-standards.html#faq-27.5>\n recommend against it</a>.\n\n Your main function should begin and end with:\n \\code\n int main(int argc, char** argv) {\n graphlab::mpi_tools::init(argc, argv);\n graphlab::distributed_control dc;\n \n ... main body ...\n graphlab::mpi_tools::finalize();\n }\n \\endcode\n \n <code>dc</code> is the distributed communication layer which is needed by \n a number of the core GraphLab objects, whether you are running distributed \n or not.\n\n Place the following code in <tt>my_first_app.cpp</tt>. To create the \n program run the configure script, than run \"make\" in the debug/ release/ \n build folders. The program when executed, will print \"Hello World!\".\n \\code\n #include <graphlab.hpp>\n int main(int argc, char** argv) {\n graphlab::mpi_tools::init(argc, argv);\n graphlab::distributed_control dc;\n \n dc.cout() << \"Hello World!\\n\";\n\n graphlab::mpi_tools::finalize();\n }\n \\endcode\n\n \\ref graphlab::distributed_control::cout \"dc.cout()\" provides a wrapper around \n standard <tt>std::cout</tt>, but wraps it in a way that when used\n in a distributed environment, only one copy will print, even though all \n machines execute it. To try that run the following:\n \\verbatim\n mpiexec -n 4 ./my_first_app\n \\endverbatim\n This should run 4 instances of \"my_first_app\" all on the local machine.\n However, only one \"Hello World!\" will be printed.\n\n \\note The \\ref graphlab::distributed_control \"distributed_control\" object is the\n core RPC implementation and has many other capabilities. See \\ref RPC for\n details.\n\n In the \\ref using_graphlab_distributed_graph \"next section\", \n we will see how to define a distributed graph.\n\n \\page using_graphlab_distributed_graph 3: Defining a Graph\n \n The datastructure which surrounds much of GraphLab's computation capabilities\n is the \\ref graphlab::distributed_graph \"distributed_graph\".\n The Distributed Graph is a directed graph datastructure comprising of\n vertices and directed edges, but with no duplicated edges allowed. i.e. there\n can be only one edge from vertex A to vertex B, and one edge from vertex B\n to vertex A. An arbitrary user data type can be associated with each vertex\n and each edge as long as the data type is \\ref sec_serializable. \n\n \\section using_graphlab_distributed_graph_vdata Vertex Data\n\n Since we are writing PageRank, we will first we define a struct describing a\n web page. This will be the contents of the vertex. This struct here holds a\n name of the webpage, as well as the resultant PageRank. A constructor which\n assigns a name is provided for later convenience. Observe that we also\n defined a default constructor as this is \\b required for it to be used in the\n graph.\n \\code\n struct web_page {\n std::string pagename;\n double pagerank;\n web_page():pagerank(0.0) { }\n explicit web_page(std::string name):pagename(name),pagerank(0.0){ }\n };\n \\endcode\n\n To make this \\ref sec_serializable, we need to define a \\c save and \\c load \n member function. The \\c save function simply writes the \\c pagename and \n \\c pagerank fields into the output archive object. The \\c load function \n performs the reverse. Care should be made to ensure that the \\c save and\n \\c load functions are symmetric.\n\n \\code\n struct web_page {\n std::string pagename;\n double pagerank;\n web_page():pagerank(0.0) { }\n explicit web_page(std::string name):pagename(name),pagerank(0.0){ }\n\n void save(graphlab::oarchive& oarc) const {\n oarc << pagename << pagerank;\n }\n\n void load(graphlab::iarchive& iarc) {\n iarc >> pagename >> pagerank;\n }\n };\n \\endcode\n\n \\section using_graphlab_distributed_graph_edata Edge Data\n\n Since we do not need any information to be stored on the edges of the graph,\n we will just use the graphlab::empty data type which will ensure that\n the edge data does not take up any memory.\n\n\n \\section using_graphlab_distributed_graph_defining_graph Defining the Graph\n\n The graphlab::distributed_graph data type takes two template arguments:\n \\li \\c VertexData The type of data to be stored on each vertex\n \\li \\c EdgeData The type of data to be stored on each edge\n\n \n For convenience, we define the type of the graph using a typedef:\n \\code\n typedef graphlab::distributed_graph<web_page, graphlab::empty> graph_type;\n \\endcode\n\n \\section using_graphlab_distributed_graph_putting_together Putting It Together\n\n At this point, our code looks like this:\n \\code\n #include <string>\n #include <graphlab.hpp>\n\n struct web_page {\n std::string pagename;\n double pagerank;\n web_page():pagerank(0.0) { }\n explicit web_page(std::string name):pagename(name),pagerank(0.0){ }\n\n void save(graphlab::oarchive& oarc) const {\n oarc << pagename << pagerank;\n }\n\n void load(graphlab::iarchive& iarc) {\n iarc >> pagename >> pagerank;\n }\n };\n \n typedef graphlab::distributed_graph<web_page, graphlab::empty> graph_type;\n\n int main(int argc, char** argv) {\n graphlab::mpi_tools::init(argc, argv);\n graphlab::distributed_control dc;\n \n dc.cout() << \"Hello World!\\n\";\n\n graphlab::mpi_tools::finalize();\n }\n \\endcode\n\n We have constructed the datatypes required for the graph to operate.\n In the \\ref using_graphlab_distributed_graph_load_data \"next section\",\n we will fill out the graph using some synthetic data.\n\n \\page using_graphlab_distributed_graph_load_data 4: Loading Graph Data\n\n The distributed_graph requires each vertex to have a numeric ID of type\n graphlab::vertex_id_type : at a moment a 32-bit integer (this will be \n lengthened to 64-bits or greater in the near future so you should not\n depend on it being 32-bits). Vertices do not need to be consecutively\n numbered. The ID corresponding to \n <tt>(graphlab::vertex_id_type)(-1)</tt> (or the maximum integer value) is\n reserved for internal use and should not be assigned.\n \n \n To load graph data from a file, we need to implement a line parser for the\n distributed_graph's \n \\ref graphlab::distributed_graph::load(std::string path, line_parser_type line_parser) load()\n function.\n \n The \\c load() load works in a simple straight-forward way. It assumes \n that each line in the file is \"independent\"; i.e. the order in which lines\n in the file appear do not matter. Each line is then passed into the user\n provided line-parsing function which then proceeds to add vertices or edges\n to the graph.\n\n\n For instance, we could describe the following input file for\n our pagerank task.\n\n \\verbatim\n 1 a.com 4 10\n 4 b.org 10\n 10 c.edu 11 1\n 11 d.gov 10\n \\endverbatim\n\n Where each line contains first an ID for the page, then the name of\n the page, and finally a list of all the IDs the page links to.\n Thus describing the following graph:\n \n \\image html example_webgraph.gif\n \n We can then implement the following line parser:\n \\code\n bool line_parser(graph_type& graph, \n const std::string& filename, \n const std::string& textline) {\n std::stringstream strm(textline);\n graphlab::vertex_id_type vid;\n std::string pagename;\n // first entry in the line is a vertex ID\n strm >> vid;\n strm >> pagename;\n // insert this web page\n graph.add_vertex(vid, web_page(pagename));\n // while there are elements in the line, continue to read until we fail\n while(1){\n graphlab::vertex_id_type other_vid;\n strm >> other_vid;\n if (strm.fail()) \n return true;\n graph.add_edge(vid, other_vid);\n }\n return true;\n }\n \\endcode\n\n To load this file, we simply construct a graph,\n\n \\code\n graph_type graph(dc);\n graph.load(\"graph.txt\", line_parser);\n \\endcode\n\n The key behind the \\c load() function is that its actual behavior is to\n load <b>all files which begin with the name provided</b>. In other words,\n if the graph file is cut into many smaller pieces such as <tt>graph.txt.1\n graph.txt.2, graph.txt.3</tt>, etc, the system will load all the files\n matching \\c graph.txt*, and possibly in parallel (if running in a distributed\n environment, it is important to ensure that all machines can access the\n same set of files). Furthermore, the \\c load() function automatically\n supports HDFS loading and obeys the same rules. Finally, if a filename\n ends with the \\c .gz extension, it is automatically treated as a gzip \n compressed file and will be automatically decompressed for reading.\n\n \\code\n graph.load(\"hdfs:///hdfsnamenode/data/graph\", line_parser);\n \\endcode\n \n will load all files on the name node \\c hdfsnamenode, and matching the pattern\n \\c /data/graph*.\n\n Multiple calls may be made to \\c load() to load different sets of files: \n each call may use a different line parser. The only requirement is that\n each edge and each vertex be added no more than once.\n\n Once all graph data is loaded, a call to \n \\code\n graph.finalize();\n \\endcode\n is necessary to commit the graph structure. This will reorganize the graph\n datastructures for optimal run-time access.\n\n At this point, your code will look like this:\n\n \\code\n #include <string>\n #include <graphlab.hpp>\n\n struct web_page {\n std::string pagename;\n double pagerank;\n web_page():pagerank(0.0) { }\n explicit web_page(std::string name):pagename(name),pagerank(0.0){ }\n\n void save(graphlab::oarchive& oarc) const {\n oarc << pagename << pagerank;\n }\n\n void load(graphlab::iarchive& iarc) {\n iarc >> pagename >> pagerank;\n }\n };\n \n typedef graphlab::distributed_graph<web_page, graphlab::empty> graph_type;\n\n\n bool line_parser(graph_type& graph, \n const std::string& filename, \n const std::string& textline) {\n std::stringstream strm(textline);\n graphlab::vertex_id_type vid;\n std::string pagename;\n // first entry in the line is a vertex ID\n strm >> vid;\n strm >> pagename;\n // insert this web page\n graph.add_vertex(vid, web_page(pagename));\n // while there are elements in the line, continue to read until we fail\n while(1){\n graphlab::vertex_id_type other_vid;\n strm >> other_vid;\n if (strm.fail())\n return true;\n graph.add_edge(vid, other_vid);\n }\n return true;\n }\n\n int main(int argc, char** argv) {\n graphlab::mpi_tools::init(argc, argv);\n graphlab::distributed_control dc;\n \n graph_type graph(dc);\n graph.load(\"graph.txt\", line_parser);\n\n graphlab::mpi_tools::finalize();\n }\n \\endcode\n\n \\note The stringstream is somewhat slow and is not the fastest way to \n parse a string. Significant performance gains can be made through the use of\n C parsing or perhaps even boost::spirit.\n\n \\section load_data_other_topics Other Topics\n\n The distributed graph provides several built-in formats which can be used\n to save/load graph structure. See \n \\ref graphlab::distributed_graph::save_format() \"distributed_graph::save_format()\"\n and\n \\ref graphlab::distributed_graph::load_format() \"distributed_graph::load_format()\"\n for more details.\n\n The distributed_graph takes as a second option, a graphlab::graphlab_options\n datastructure which contains runtime options that can affect the behavior\n and performance of GraphLab. See \\ref graphlab::distributed_graph::distributed_graph() \"the constructor\"\n for more details.\n\n GraphLab provides a convenient command line parser in\n graphlab::command_line_options (really, a wrapper around\n boost::program_options simpler features). The parser is easy to use and\n automatically exposes GraphLab's runtime options on the command line. \n\n\n \\page using_graphlab_distributed_graph_vertex_program 5: Writing the PageRank Vertex Program\n\n Finally, we get to writing the PageRank vertex program itself.\n A detailed description the pagerank update can be found on wikipedia \n <a href=http://en.wikipedia.org/wiki/PageRank>here</a>.\n \n In pseudo-code:\n\n \\verbatim\n To compute PageRank of page P:\n acc = 0;\n For Each In-page Q:\n acc += Q.pagerank / Q.num_out_links\n End\n P.pagerank = 0.85 * acc + 0.15\n \\endverbatim\n \n We need to map the pagerank pseudo code to the GraphLab vertex program.\n Note: an alternate way of presenting the same algorithm in terms of probabilities is:\n \\verbatim\n P.pagerank2 = 0.85 * acc + 0.15 / n\n \\endverbatim\n where n is the total number of graph nodes. Both formulations are equivalent since one can easily verify that\n \\verbatim\n P.pagerank2 * n = P.pagerank\n \\endverbatim\n In GraphLab, we choose the first implementation, since dealing with graphs with billions of nodes will lead to numerical errors\n and pagerank values which are very close to zero. \n \n \\section using_graphlab_vertex_program PageRank Vertex Program\n\n A \"vertex program\" can be thought of as a little program which is executed on\n a vertex in the graph. The vertex program is short lived: it performs the following\n 3 phases of execution: each phase providing it access to a different section of\n the neighborhood of the vertex, then is destroyed. \n \\li A \\b gather phase where \\ref graphlab::ivertex_program::gather() \"gather()\" function\n in the vertex program is called on each edge on the vertex's adjacent edges.\n returning a value with each gather.\n \\li An \\b apply phase where the values returned by the gather's are summed\n together and given to the \\ref graphlab::ivertex_program::apply() \"apply()\"\n function in the vertex program.\n \\li A \\b scatter phase where \\ref graphlab::ivertex_program::scatter() \"scatter()\"\n function in the vertex program is once again called on each edge on the\n vertex's adjacent edges.\n \n See graphlab::ivertex_program for detailed documentation on the behavior of\n the vertex program.\n\n It is simplest to just demonstrate the PageRank vertex program in code: \n\n \\code\nclass pagerank_program :\n public graphlab::ivertex_program<graph_type, double>,\n public graphlab::IS_POD_TYPE {\npublic:\n // we are going to gather on all the in-edges\n edge_dir_type gather_edges(icontext_type& context,\n const vertex_type& vertex) const {\n return graphlab::IN_EDGES;\n }\n\n // for each in-edge gather the weighted sum of the edge.\n double gather(icontext_type& context, const vertex_type& vertex,\n edge_type& edge) const {\n return edge.source().data().pagerank / edge.source().num_out_edges();\n }\n \n // Use the total rank of adjacent pages to update this page \n void apply(icontext_type& context, vertex_type& vertex,\n const gather_type& total) {\n double newval = total * 0.85 + 0.15;\n vertex.data().pagerank = newval;\n }\n \n // No scatter needed. Return NO_EDGES \n edge_dir_type scatter_edges(icontext_type& context,\n const vertex_type& vertex) const {\n return graphlab::NO_EDGES;\n }\n}; \n \\endcode\n\n The \\c pagerank_program inherits from graphlab::ivertex_program which is\n itself templatized over the type of the graph (\\c graph_type) and the type\n returned by the \\c gather() obersion. \n\n The pagerank_program must also be \\ref sec_serializable. Since this program\n does not contain any data elements, it is a POD type and is sufficient to\n inherit from graphlab::IS_POD_TYPE.\n\n We will now explain each function.\n\n## <b>gather_edges(icontext_type& context, const vertex_type& vertex)</b>\n The \\ref graphlab::ivertex_program::gather_edges() \"gather_edges()\" function\n returns the collection of edges to gather. It may return graphlab::IN_EDGES,\n graphlab::OUT_EDGES, graphlab::NO_EDGES, or graphlab::ALL_EDGES. The PageRank\n update uses only in pages, thus we return graphlab::IN_EDGES.\n\n The gather_edges() function is also passed a \\c context object which provides\n additional access to the execution environment: such as obtaining the number\n of edges in the graph (\\ref graphlab::icontext::num_edges() \"num_edges()\"),\n the ability to immediately stop execution (\\ref graphlab::icontext::stop() \"stop()\")\n among others.\n\n It is also passed a reference to the vertex the current vertex_program is\n executing on through the \\c vertex argument. Through \\c vertex, the function\n can read the data on the vertex as well as obtain other meta-data such as\n the number of in-edges of the current vertex (see graphlab::distributed_graph::vertex_type).\n\n## <b>gather(icontext_type& context, const vertex_type& vertex, edge_type& edge)</b>\n According to the PageRank equation, we must compute a weighted sum of the \n in-pages. The \\ref graphlab::ivertex_program::gather() \"gather()\" function \n is thus executed on each in-edge in of the current vertex, returning the edge's\n contribution to the \"acc\" value. \n\n To compute the current edge's contribution to the weight, we use the \\c edge\n argument which provides direct access to the data on the edge, the source vertex\n of the edge, and the destination vertex. (see graphlab::distributed_graph::edge_type)\n \n \\note While \\c gather() technically has a non-const reference to the source and \n target vertex data through \\c edge.source() and \\c edge.target(), it should not modify\n them. The data on the edge (accessible through \\c edge.data()) is modifiable however.\n\n Once the contribution is computed, we simply return it since the result\n from all \\c gather() calls are summed up by the execution engine (using only the += \n operator). The data type of the accumulation is a \\b double, \n and this must be provided in the second template argument of the ivertex_program\n the pagerank_program inheritted from.\n\n## <b>apply(icontext_type& context, vertex_type& vertex, const gather_type& total)</b>\n The returned values from each gather are implicitly summed up (in parallel)\n behind the scenes as passed to the apply() function in the \\c total argument.\n Observe that now the \\c vertex parameter is modifiable, and we use\n this to write the new pagerank to the current vertex.\n\n \n## <b>scatter_edges(icontext_type& context, const vertex_type& vertex)</b> \n Scatter is similar to gather: it is executed on each edge of\n the vertex, but does not accumulate any values. \n Since the pagerank computation does not require a scatter operation we simply\n have scatter_edges return graphlab::NO_EDGES.\n \n\n\n \\section using_graphlab_vertex_program_running Running the Vertex Program\n\n To run the above vertex program on all vertices in the graph \\b once, \n we simply construct an engine in main() (after finalizing the graph)\n\n \\code\n graphlab::omni_engine<pagerank_program> engine(dc, graph, \"sync\");\n engine.signal_all();\n engine.start();\n \\endcode\n\n The \\ref graphlab::omni_engine \"omni_engine\" is a engine wrapper that allows\n you to easily select between a synchronous engine or an asynchronous engine.\n In this case, we select a synchronous engine. Passing \"async\" will select an\n asynchronous engine. \n\n \\note If graphlab::command_line_options are used, it can be passed as an \n additional 4th argument to the constructor. This will allow the engine type\n to be modified at runtime.\n\n The \\ref graphlab::omni_engine::signal_all \"signal_all()\" function, as the\n name suggests, signals all the vertices in the graph to run. \n \\ref graphlab::omni_engine::start() \"start()\" will begin execution of all\n signaled vertices. Since the synchronous engine is selected all vertices will\n perform the gather/apply/scatter operations in lock-step. If the asynchronous\n engine is selected, the vertices will run asynchronously, but a distributed\n locking procedure is used internally to ensure data consistency.\n\n Each vertex in the graph will run exactly once. We could of course embed\n the signal+start operations in a loop to run multiple rounds, but that would \n be inefficient. In the \\ref using_scheduling \"next section\" we will learn how\n to signal vertices inside a vertex program.\n\n \\page using_scheduling 6: Runtime Scheduling\n\n During engine execution, the engine maintains (in a distributed fashion),\n a schedule of vertex programs to run. The \n \\ref graphlab::omni_engine::signal_all \"engine.signal_all()\"\n function call essentially injects a list of all vertices in the graph into\n the engine's scheduler. In this section, we will see how the schedule\n can be modified during engine execution and how that could be used (to great\n effect in some cases) to accelerate convergence of your program.\n\n \\section using_scheduling_self Self Scheduling Example\n\n The simplest form of \"dynamic\" scheduling is to repeat each vertex's execution\n for a certain fixed number of iterations (say 10). \n To do that, we add a \n \"counter\" to the data on each vertex by modifying the \\c web_page struct:\n\n \\code\n struct web_page {\n std::string pagename;\n double pagerank;\n int counter;\n\n web_page():pagerank(0.0),counter(0) { }\n explicit web_page(std::string name):pagename(name),\n pagerank(0.0),counter(0){ }\n\n void save(graphlab::oarchive& oarc) const {\n oarc << pagename << pagerank << counter;\n }\n\n void load(graphlab::iarchive& iarc) {\n iarc >> pagename >> pagerank >> counter;\n }\n };\n \\endcode\n\n Observe that the constructors were modified to initialize the counters at 0,\n and the save/load functions must now also save the counter variable.\n\n To achieve self-scheduling, we simply modify the \\c apply() function in the\n \\c pagerank_program to increment the counter, and signal the current vertex\n if the counter has not reached 10.\n \\code\n // Use the total rank of adjacent pages to update this page \n void apply(icontext_type& context, vertex_type& vertex,\n const gather_type& total) {\n double newval = total * 0.85 + 0.15;\n vertex.data().pagerank = newval;\n ++vertex.data().counter;\n if (vertex.data().counter < 10) context.signal(vertex);\n // of course, instead of simply \"10\" here, this could be comparing\n // against a global variable set by a command line option.\n }\n \\endcode\n The \\ref graphlab::icontext::signal() \"context.signal(vertex)\" call\n inserts the current vertex into the scheduler. The guarantee provided by the\n \\c signal() call is that:\n <b> The vertex signaled will be eventually executed some time after completion\n of the \\c signal() function call. </b>\n \n If used together with the synchronous engine, this program will perform\n exactly the equivalent of the traditional \"matrix-multiplication-like\"\n PageRank iteration. \n\n \\section using_scheduling_dynamic Dynamic Scheduling Example\n\n Alternatively, we could take a more \"contextual\" approach to scheduling.\n Considering that PageRank is a numeric procedure performed on a large graph,\n it is not unreasonable to believe that some parts of the graph will \n converge before other parts of the graph. We could therefore \n save computation if we only recompute vertices which may change by large\n amounts.\n\n To implement this, we consider the pagerank_program implemented earlier.\n We will not make modifications to the gather phases, but we will change the\n apply phase and introduce a scatter phase. The goal is to achieve the following:\n \n \\li If the current vertex's PageRank does not change much, no additional\n action is performed.\n \\li However, if the current vertex's PageRank changed by greater than some\n threshold (1E-3), we will \\c signal() all out-pages to recompute their\n PageRank value.\n\n \\code\nclass dynamic_pagerank_program :\n public graphlab::ivertex_program<graph_type, double>,\n public graphlab::IS_POD_TYPE {\nprivate:\n // a variable local to this program\n bool perform_scatter;\npublic:\n // no changes to gather_edges and gather\n edge_dir_type gather_edges(icontext_type& context,\n const vertex_type& vertex) const {\n return graphlab::IN_EDGES;\n }\n double gather(icontext_type& context, const vertex_type& vertex,\n edge_type& edge) const {\n return edge.source().data().pagerank / edge.source().num_out_edges();\n }\n \n\n // Use the total rank of adjacent pages to update this page \n void apply(icontext_type& context, vertex_type& vertex,\n const gather_type& total) {\n double newval = total * 0.85 + 0.15;\n double prevval = vertex.data().pagerank;\n vertex.data().pagerank = newval;\n perform_scatter = (std::fabs(prevval - newval) > 1E-3);\n }\n \n // The scatter edges depend on whether the pagerank has converged \n edge_dir_type scatter_edges(icontext_type& context,\n const vertex_type& vertex) const {\n if (perform_scatter) return graphlab::OUT_EDGES;\n else return graphlab::NO_EDGES;\n }\n\n void scatter(icontext_type& context, const vertex_type& vertex,\n edge_type& edge) const {\n context.signal(edge.target());\n }\n}; \n \\endcode\n\n Firstly, we observe that we introduced a private variable \\c perform_scatter\n to the program. This variable is short-lived and is local to this particular\n execution of the pagerank program. In the \\c apply() function, we compute the\n change to the current vertex's pagerank, and if it is above a certain threshold,\n we set the \\c perform_scatter variable to true. \n \n This next influences the \n behavior of the \\c scatter_edges() function. If \\c perform_scatter is false,\n (i.e. insufficient change was made to the current PageRank), we do not perform a scatter.\n However, if sufficient change was made, \\c scatter_edges() will return\n graphlab::OUT_EDGES which will cause the \\c scatter() function to be executed\n on all out-going edges of the current vertex. The \\c scatter() function then\n simply schedules/signals the destination vertex, requesting it to be executed\n in the future, picking up the large change made to the current vertex.\n\n If ran using the synchronous engine, you will observe that the time spent within\n each synchronous iteration decreases, as the number of \"signalled\" vertices\n in each iteration decreases over time. This also works well in the asynchronous\n setting where powerful dynamic schedulers are used to control the order\n of execution.\n\n \\note If you dig further into the documentation you will see that the \n signalling operation can itself be used as a messaging primitive to carry a\n message to a destination vertex. This allows GraphLab v2.1 to in some sense,\n also include \"Pregel\" as part of the implementation. Furthermore, the\n message could define a \"message priority\" which can be used in conjunction with\n the priority-queue based dynamic schedulers to obtain greater control over\n the order of execution in the asynchronous engine.\n\n In the \\ref using_saving_answers \"next section\", we will see how to save \n output of the system.\n\n \\page using_saving_answers 7: Saving Results\n\n Saving the graph requires us to implement a graph writer class comprising of\n two functions: \\c save_vertex() and \\c save_edge().\n\n \\code\n class graph_writer {\n public:\n std::string save_vertex(graph_type::vertex_type v) { return \"\"; }\n std::string save_edge(graph_type::edge_type e) { return \"\"; }\n };\n \\endcode\n\n The \\c save_vertex() and \\c save_edge() functions are respectively\n called on each vertex/edge in the graph. These functions return a string\n which is then directly written to the output file.\n\n For instance, to save an output file comprising of <tt>[webpage] [pagerank]</tt>\n lines, we may implement the following:\n\n \\code\n class graph_writer {\n public:\n std::string save_vertex(graph_type::vertex_type v) {\n std::stringstream strm;\n // remember the \\n at the end! This will provide a line break\n // after each page.\n strm << v.data().pagename << \"\\t\" << v.data().pagerank << \"\\n\";\n return strm.str();\n }\n std::string save_edge(graph_type::edge_type e) { return \"\"; }\n };\n \\endcode\n\n Since we are not interested in the edges, the \\c save_edge() function\n simply returns an empty string.\n\n \\note The stringstream is somewhat slow and is not the fastest way to \n write a string. Performance gains can be made through the use of\n C string operations.\n\n Then to write the graph, we will call\n \\code\n graph.save(\"output\",\n graph_writer(),\n false, // set to true if each output file is to be gzipped\n true, // whether vertices are saved\n false); // whether edges are saved\n \\endcode\n\n This will save a sequence of files named <tt> output_1_of_N, output_2_of_N ... \n </tt> where N is some integer. Concatenating all the files together will produce\n the combined output. If the gzip option is set, each of the files will have a\n \\c .gz suffix and \\c gunzip must be used to decompres the file for reading.\n\n If the output path is located on HDFS, for instance:\n\\verbatim\nhdfs:///namenode/data/output\n\\endverbatim\nThe result will be saved to the HDFS cluster with the given namename,\n in the subdirectory /data with the filenames\n<tt> output_1_of_N, output_2_of_N ... </tt>.\n\nThere are several other \"built-in\" saving formats which can be accessed through\nthe ref graphlab::distributed_graph::save_format() \"graph.save_format()\"function.\n\nThe \\ref using_conclusion \"next section\" is a brief conclusion.\n\n\\page using_conclusion 8: Conclusion\n\nThis completes the core of the GraphLab tutorial. We have went through an overview\nof \n\\li How to start a GraphLab project\n\\li How to read a graph from disk/HDFS\n\\li How to write a vertex program\n\\li Dynamic Scheduling in a vertex program\n\\li How to save a graph to disk/HDFS\n\nThere are many \nmore features which we are unable to introduce through the course of\nthis tutorial which we hope you will be able to discover by exploring the \ndocumentation. \n\nSome really useful tools that we would like to bring to your attention are:\n<ul>\n<li> Perform MapReduce over the vertices or the edges in the graph.\n <ul>\n <li> graphlab::distributed_graph::map_reduce_vertices()\n <li> graphlab::distributed_graph::map_reduce_edges()\n </ul>\n<li> Perform MapReduce over the vertices or edges in the graph, while being\nprovided a context in the Map function, thus allowing finer grained control\nover signalling.\n <ul>\n <li> graphlab::iengine::map_reduce_vertices()\n <li> graphlab::iengine::map_reduce_edges()\n </ul>\n<li> Make a modification to all the vertices or edges in the graph\n <ul>\n <li> graphlab::distributed_graph::transform_vertices()\n <li> graphlab::distributed_graph::transform_edges()\n </ul>\n<li> Make a modification to all the vertices or edges in the graph, while being\nprovided a context in the Map function, thus allowing finer grained control\nover signalling.\n <ul>\n <li> graphlab::iengine::transform_vertices()\n <li> graphlab::iengine::transform_edges()\n </ul>\n<li> Register a MapReduce operation which performs periodically while\na GraphLab engine is running thus allowing for global state.\n <ul>\n <li> graphlab::iengine::add_vertex_aggregator()\n <li> graphlab::iengine::add_edge_aggregator()\n <li> graphlab::iengine::aggregate_now()\n <li> graphlab::iengine::aggregate_periodic()\n </ul>\n</ul>\n\nInterleaving these operations together with GraphLab vertex_programs allow\nfor a huge amount of flexibility, allowing for a large number of algorithms to\nbe implemented easily, and efficiently.\n\n\n*/\n" }, { "path": "src/graphlab/docs/using_warp.dox", "content": "/**\n\\page using_warp GraphLab Warp System Tutorial\n\nThis tutorial demonstrate the latest feature of GraphLab -- the Warp System.\nWe encourage you to complete the basic graphlab tutorial before advancing.\n\nThe design of the warp system aims to improve the programming interface of\nGraphLab by simplyfing the procedure of writing the vertex program without\nsacrificing performance. The basic design of the Warp system lies in use of \nfine-grained user-mode threading to hide communication latency of blocking \ncalls; and as such expose a more intuitive and easy to use API Interface.\n\nWe begin with a simple synthetic example. Say, we have a graph with an \ninteger on each vertex, and we would like to compute on each vertex, the total\nvalue of its neighbors.\n\n\\code\nstruct vertex_data {\n int value;\n int neighbor_total;\n};\n\ntypedef graphlab::distributed_graph<vertex_data, empty> graph_type;\n\\endcode\n\nThe Warp System is included by including the single header\n\n\\code\n#include <graphlab/warp.hpp>\n\\endcode\n\nOne of the key functions the warp engine provides, is a parfor over all \nvertices, excuting a single function on all vertices.\n\n\\code\n// Runs the compute_neighborhood_total function on all vertices in the graph\ngraphlab::warp::parfor_all_vertices(graph, compute_neighborhood_total);\n\nvoid compute_neighborhood_total(graph_type::vertex_type vertex) {\n ...\n}\n\\endcode\n\nNow, within the compute_neighborhood_total function, we would like to \ncompute the sum of the \"value\" field of all neighboring vertices, and assign\nit to the \"neighbor_total\" value of the current vertex. However, the \nparfor_all_vertices function requires that compute_neighborhood_total only\ntakes a single argument: the vertex, so how do we get the value of the\nneighbors? We use a call to a warp function called warp::map_reduce_neighborhood(),\nwhich allows us to compute an aggregation over the neighborhood of the graph\n\n\\code\nint gather_value(graph_type::edge_type edge\n graph_type::vertex_type other /* the other vertex*/ ) {\n return other.value;\n}\n\nvoid combine(int& a, const int& b) {\n a += b;\n}\n\nvoid compute_neighborhood_total(graph_type::vertex_type vertex) {\n vertex.value = graphlab::warp::map_reduce_neighborhood(vertex, ALL_EDGES, gather_value, combine); \n}\n\\endcode\n\nThe warp::map_reduce_neighborhood() function calls the gather_value function\non all adjacent edges of the graph. The return value of the gather_value \nfunction is then combined using the combine function, and the result returned.\nHere, we explicitly defined a combine function, but there is a default combiner \nprovided which simply uses +=. As such, in this case, the combiner is in fact, optional,\nand the following will work just fine:\n\n\\code\nvoid compute_neighborhood_total(graph_type::vertex_type vertex) {\n vertex.value = graphlab::warp::map_reduce_neighborhood(vertex, ALL_EDGES, gather_value); \n}\n\\endcode\n\nThe trick to the Warp System is that in the distributed setting, \nwarp::map_reduce_neighborhood() can be a distributed call. However, through \nthe use of fine-grained threading (fibers), we can hide the cost of the \ndistributed latency by creating thousands of fibers to evaluate the parfor.\n\nIn addition to the warp::parfor_all_vertices() function and the warp::map_reduce_neighborhood()\n function we demonstrated, the entire Warp system comprises of \nan asynchronous \\ref graphlab::warp_engine \"warp engine\", which operates similarly\nto the asynchronous engine, but allows you to implement an arbitrary function\nrather than the restricted Gather-Apply-Scatter model, and 4 basic functions.\n\n- \\ref graphlab::warp::parfor_all_vertices() \"warp::parfor_all_vertices()\"\nprovides a simple parallel for loop over all vertices in the graph, or in a\ngiven set of vertices. \n\n- \\ref graphlab::warp::map_reduce_neighborhood() \"warp::map_reduce_neighborhood()\"\nallows a map-reduce aggregation of the neighborhood of a vertex to be performed.\n\n- \\ref graphlab::warp::transform_neighborhood() \"warp::transform_neighborhood()\"\nallows a parallel transformation of the neighborhood of a vertex to be performed.\n\n- \\ref graphlab::warp::broadcast_neighborhood() \"warp::broadcast_neighborhood()\"\nallows a parallel transformation of the neighborhood of a vertex to be performed \nand also provides a warp_engine context.\n\nIn this example, we would demonstrate how PageRank can be implemented using\nthe Warp System in two ways, using the warp::parfor_all_vertices() , \nand using the WarpEngine. the same PageRank application using the\nWarp System. See \\ref warp for detailed documentation on\nthe behavior of the Warp System.\n\n- \\subpage using_warp_graph_functions \n- \\subpage using_warp_graph_vertex_program \n\n\n\\page using_warp_graph_functions PageRank with Warp Parfor\n\n The warp functions provides great flexibility to write simple parallel function on graphs.\n For example, an PageRank program which computes asynchronous \n sweeps over all vertices, can be written only using warp functions. \n\n given a graph with a float on each vertex:\n\n \\code\n typedef graphlab::distributed_graph<float, graphlab::empty> graph_type;\n \\endcode\n\n We use the the parfor_all_vertices to run a pagerank function on all vertices\n \\code\n int main(int argc, char** argv) {\n ...\n for (int i = 0; i < NUM_ITER; ++i) {\n // runs the pagerank function on all the vertices in the graph.\n graphlab::warp::parfor_all_vertices(graph, pagerank); \n }\n ...\n }\n\n\n void pagerank(graph_type::vertex_type vertex) {\n ...\n }\n \\endcode\n\n\n The pagerank function then simply has to use a warp::map_reduce_neighborhood()\n call to compute the weighted sum of then neighborhood's PageRank value.\n Note that here, we use the default combiner (a += operation) to merge the results\n from the pagerank_map function.\n\n \\code\n float pagerank_map(graph_type::edge_type edge, graph_type::vertex_type other) {\n return other.data() / other.num_out_edges();\n }\n\n void pagerank(graph_type::vertex_type vertex) {\n // computes an aggregate over the neighborhood using map_reduce_neighborhood\n vertex.data() = 0.15 + 0.85 * graphlab::warp::map_reduce_neighborhood(vertex,\n IN_EDGES,\n pagerank_map);\n }\n\n \\endcode\n\n Using c++11 lambda we can further simplify the pagerank function.\n \\code\n void pagerank(graph_type::vertex_type vertex) {\n // computes an aggregate over the neighborhood using map_reduce_neighborhood\n vertex.data() = 0.15 + 0.85 * graphlab::warp::map_reduce_neighborhood(vertex,\n IN_EDGES,\n [](graph_type::edge_type edge, \n graph_type::vertex_type other) { \n return other.data() / other.num_out_edges();\n });\n }\n \\endcode\n It is important that the C++11 lambda represent\n a regular function pointer, and must not capture a closure. (i.e. you cannot \n use [=] or [&]).\n\n If additional parameters must be passed to the mapper or combiner functions,\n an optional \"extra argument\" can be specified.\n\n \\code\n float pagerank_map(graph_type::edge_type edge, \n graph_type::vertex_type other,\n const float& weight) {\n return 0.85 * other.data() / other.num_out_edges();\n }\n\n void combiner(float& a, const float& b, const float& other) {\n a += b;\n }\n\n void pagerank(graph_type::vertex_type vertex) {\n // computes an aggregate over the neighborhood using map_reduce_neighborhood\n vertex.data() = 0.15 + graphlab::warp::map_reduce_neighborhood(vertex,\n IN_EDGES,\n float(0.85), // this argument will show up as \n // the 3rd argument in pagerank_map\n pagerank_map,\n combiner);\n }\n \\endcode\n\n\\page using_warp_graph_vertex_program PageRank with the Warp Engine\n We use the Warp system to provide a dynamic asynchronous engine, similar in \n nature to the \\ref asynchronous_engine \"asynchronous engine\", but without\n the Gather-Apply-Scatter limitations. Instead you simply specify an \n update function. Which is of the type\n \\code\n void update_function(engine_type::context& context,\n graph_type::vertex_type vertex) {\n }\n \\endcode\n\n Within the update function, All blocking warp functions such as \n warp::map_reduce_neighborhood(),\n warp::transform_neighborhood() and\n warp::broadcast_neighborhood() \n can be used to make changes to the graph data, and to \n schedule other vertices for computation.\n\n \\section using_warp_graph_vertex_program_updatefn Pagerank Update Function \n\n Given a graph with a float on each vertex:\n\n \\code\n typedef graphlab::distributed_graph<float, graphlab::empty> graph_type;\n \\endcode \n\n we first define the engine type\n\n\n \\code\n typedef graphlab::warp::warp_engine<graph_type> engine_type;\n \\endcode \n\n Now PageRank can be written using the Warp Engine, by defining an update\n function:\n\n \\code\n\n float pagerank_map(graph_type::edge_type edge, graph_type::vertex_type other) {\n return other.data() / other.num_out_edges();\n }\n \n void signal_neighbor(engine_type::context& context,\n graph_type::edge_type edge, graph_type::vertex_type other) {\n context.signal(other);\n }\n\n\n void pagerank_update_function(engine_type::context& context,\n graph_type::vertex_type vertex) {\n\n // save the old pagerank value\n float oldval = vertex.data();\n\n // compute the new pagerank using blocking warp function\n vertex.data() = 0.15 + 0.85 *graphlab::warp::map_reduce_neighborhood(vertex,\n IN_EDGES,\n pagerank_map);\n\n // Schedule out edges if we exceed tolerance.\n if (std::fabs(oldval - vertex.data()) > TOLERANCE) {\n graphlab::warp::broadcast_neighborhood(context,\n vertex,\n OUT_EDGES,\n signal_neighbor);\n }\n }\n \\endcode\n\n The broadcast_neighborhood call simply runs the provided function\n (signal_neighbor), on all selected edges (OUT_EDGES) in this case.\n the warp::broadcast_neighborhood() function requires the context to allow the signal_neighbor\n function to perform dynamic scheduling. warp::map_reduce_neighborhood()\n essentially accomplishes the same as the \"Gather\" function in the GAS model,\n and warp::broadcast_neighborhood() essentially accomplishes the same role as \n as the Scatter, allowing you to make modifications to the edges on the graph.\n\n As you can see, the warp engine let you write much simpler vertex update\n function comparing to the vertex_program used in previous versions of\n engines.\n\n\n \\section using_warp_graph_vertex_program_running Running the Update Function \n To run the above vertex program on all vertices in the graph \\b once, \n we simply construct an engine in main() (after finalizing the graph)\n\n \\code\n int main(int argc, char** argv) {\n ...\n\n graphlab::warp::warp_engine<graph_type> engine(dc, graph);\n // sets the update function to use\n engine.set_update_function(pagerank_update_function);\n // signals all vertices to run. warp::warp_engine::signal_vset()\n // can also be used to signal a subset of vertices\n engine.signal_all();\n // run the engine until scheduler is empty.\n engine.start();\n\n ...\n }\n \\endcode\n\n Just like the asynchronous engine, there are numerous options for the \n scheduler type, and scheduler capabilities, which we willl not go into great detail here.\n*/\n" }, { "path": "src/graphlab/engine/CMakeLists.txt", "content": "project(GraphLab)\n\n# subdirs(callback)\n\n" }, { "path": "src/graphlab/engine/async_consistent_engine.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n\n#ifndef GRAPHLAB_ASYNC_CONSISTENT_ENGINE\n#define GRAPHLAB_ASYNC_CONSISTENT_ENGINE\n\n#include <deque>\n#include <boost/bind.hpp>\n\n#include <graphlab/scheduler/ischeduler.hpp>\n#include <graphlab/scheduler/scheduler_factory.hpp>\n#include <graphlab/scheduler/get_message_priority.hpp>\n#include <graphlab/vertex_program/ivertex_program.hpp>\n#include <graphlab/vertex_program/icontext.hpp>\n#include <graphlab/vertex_program/context.hpp>\n#include <graphlab/engine/iengine.hpp>\n#include <graphlab/engine/execution_status.hpp>\n#include <graphlab/options/graphlab_options.hpp>\n#include <graphlab/rpc/dc_dist_object.hpp>\n#include <graphlab/engine/distributed_chandy_misra.hpp>\n#include <graphlab/engine/message_array.hpp>\n\n#include <graphlab/util/tracepoint.hpp>\n#include <graphlab/util/memory_info.hpp>\n#include <graphlab/util/generics/conditional_addition_wrapper.hpp>\n#include <graphlab/rpc/distributed_event_log.hpp>\n#include <graphlab/parallel/fiber_group.hpp>\n#include <graphlab/parallel/fiber_control.hpp>\n#include <graphlab/rpc/fiber_async_consensus.hpp>\n#include <graphlab/aggregation/distributed_aggregator.hpp>\n#include <graphlab/parallel/fiber_remote_request.hpp>\n#include <graphlab/macros_def.hpp>\n\n\n\nnamespace graphlab {\n\n\n /**\n * \\ingroup engines\n *\n * \\brief The asynchronous consistent engine executed vertex programs\n * asynchronously and can ensure mutual exclusion such that adjacent vertices\n * are never executed simultaneously. The default mode is \"factorized\"\n * consistency in which only individual gathers/applys/\n * scatters are guaranteed to be consistent, but this can be strengthened to\n * provide full mutual exclusion.\n *\n *\n * \\tparam VertexProgram\n * The user defined vertex program type which should implement the\n * \\ref graphlab::ivertex_program interface.\n *\n * ### Execution Semantics\n *\n * On start() the \\ref graphlab::ivertex_program::init function is invoked\n * on all vertex programs in parallel to initialize the vertex program,\n * vertex data, and possibly signal vertices.\n *\n * After which, the engine spawns a collection of threads where each thread\n * individually performs the following tasks:\n * \\li Extract a message from the scheduler.\n * \\li Perform distributed lock acquisition on the vertex which is supposed\n * to receive the message. The lock system enforces that no neighboring\n * vertex is executing at the same time. The implementation is based\n * on the Chandy-Misra solution to the dining philosophers problem.\n * (Chandy, K.M.; Misra, J. (1984). The Drinking Philosophers Problem.\n * ACM Trans. Program. Lang. Syst)\n * \\li Once lock acquisition is complete,\n * \\ref graphlab::ivertex_program::init is called on the vertex\n * program. As an optimization, any messages sent to this vertex\n * before completion of lock acquisition is merged into original message\n * extracted from the scheduler.\n * \\li Execute the gather on the vertex program by invoking\n * the user defined \\ref graphlab::ivertex_program::gather function\n * on the edge direction returned by the\n * \\ref graphlab::ivertex_program::gather_edges function. The gather\n * functions can modify edge data but cannot modify the vertex\n * program or vertex data and can be executed on multiple\n * edges in parallel.\n * * \\li Execute the apply function on the vertex-program by\n * invoking the user defined \\ref graphlab::ivertex_program::apply\n * function passing the sum of the gather functions. If \\ref\n * graphlab::ivertex_program::gather_edges returns no edges then\n * the default gather value is passed to apply. The apply function\n * can modify the vertex program and vertex data.\n * \\li Execute the scatter on the vertex program by invoking\n * the user defined \\ref graphlab::ivertex_program::scatter function\n * on the edge direction returned by the\n * \\ref graphlab::ivertex_program::scatter_edges function. The scatter\n * functions can modify edge data but cannot modify the vertex\n * program or vertex data and can be executed on multiple\n * edges in parallel.\n * \\li Release all locks acquired in the lock acquisition stage,\n * and repeat until the scheduler is empty.\n *\n * The engine threads multiplexes the above procedure through a secondary\n * internal queue, allowing an arbitrary large number of vertices to\n * begin processing at the same time.\n *\n * ### Construction\n *\n * The asynchronous consistent engine is constructed by passing in a\n * \\ref graphlab::distributed_control object which manages coordination\n * between engine threads and a \\ref graphlab::distributed_graph object\n * which is the graph on which the engine should be run. The graph should\n * already be populated and cannot change after the engine is constructed.\n * In the distributed setting all program instances (running on each machine)\n * should construct an instance of the engine at the same time.\n *\n * Computation is initiated by signaling vertices using either\n * \\ref graphlab::async_consistent_engine::signal or\n * \\ref graphlab::async_consistent_engine::signal_all. In either case all\n * machines should invoke signal or signal all at the same time. Finally,\n * computation is initiated by calling the\n * \\ref graphlab::async_consistent_engine::start function.\n *\n * ### Example Usage\n *\n * The following is a simple example demonstrating how to use the engine:\n * \\code\n * #include <graphlab.hpp>\n *\n * struct vertex_data {\n * // code\n * };\n * struct edge_data {\n * // code\n * };\n * typedef graphlab::distributed_graph<vertex_data, edge_data> graph_type;\n * typedef float gather_type;\n * struct pagerank_vprog :\n * public graphlab::ivertex_program<graph_type, gather_type> {\n * // code\n * };\n *\n * int main(int argc, char** argv) {\n * // Initialize control plain using mpi\n * graphlab::mpi_tools::init(argc, argv);\n * graphlab::distributed_control dc;\n * // Parse command line options\n * graphlab::command_line_options clopts(\"PageRank algorithm.\");\n * std::string graph_dir;\n * clopts.attach_option(\"graph\", &graph_dir, graph_dir,\n * \"The graph file.\");\n * if(!clopts.parse(argc, argv)) {\n * std::cout << \"Error in parsing arguments.\" << std::endl;\n * return EXIT_FAILURE;\n * }\n * graph_type graph(dc, clopts);\n * graph.load_structure(graph_dir, \"tsv\");\n * graph.finalize();\n * std::cout << \"#vertices: \" << graph.num_vertices()\n * << \" #edges:\" << graph.num_edges() << std::endl;\n * graphlab::async_consistent_engine<pagerank_vprog> engine(dc, graph, clopts);\n * engine.signal_all();\n * engine.start();\n * std::cout << \"Runtime: \" << engine.elapsed_seconds();\n * graphlab::mpi_tools::finalize();\n * }\n * \\endcode\n *\n * \\see graphlab::omni_engine\n * \\see graphlab::synchronous_engine\n *\n * <a name=engineopts>Engine Options</a>\n * =========================\n * The asynchronous engine supports several engine options which can\n * be set as command line arguments using \\c --engine_opts :\n *\n * \\li \\b timeout (default: infinity) Maximum time in seconds the engine will\n * run for. The actual runtime may be marginally greater as the engine\n * waits for all threads and processes to flush all active tasks before\n * returning.\n * \\li \\b factorized (default: true) Set to true to weaken the consistency\n * model to factorized consistency where only individual gather/apply/scatter\n * calls are guaranteed to be locally consistent. Can produce massive\n * increases in throughput at a consistency penalty.\n * \\li \\b nfibers (default: 10000) Number of fibers to use\n * \\li \\b stacksize (default: 16384) Stacksize of each fiber.\n */\n template<typename VertexProgram>\n class async_consistent_engine: public iengine<VertexProgram> {\n\n public:\n /**\n * \\brief The user defined vertex program type. Equivalent to the\n * VertexProgram template argument.\n *\n * The user defined vertex program type which should implement the\n * \\ref graphlab::ivertex_program interface.\n */\n typedef VertexProgram vertex_program_type;\n\n /**\n * \\brief The user defined type returned by the gather function.\n *\n * The gather type is defined in the \\ref graphlab::ivertex_program\n * interface and is the value returned by the\n * \\ref graphlab::ivertex_program::gather function. The\n * gather type must have an <code>operator+=(const gather_type&\n * other)</code> function and must be \\ref sec_serializable.\n */\n typedef typename VertexProgram::gather_type gather_type;\n\n /**\n * \\brief The user defined message type used to signal neighboring\n * vertex programs.\n *\n * The message type is defined in the \\ref graphlab::ivertex_program\n * interface and used in the call to \\ref graphlab::icontext::signal.\n * The message type must have an\n * <code>operator+=(const gather_type& other)</code> function and\n * must be \\ref sec_serializable.\n */\n typedef typename VertexProgram::message_type message_type;\n\n /**\n * \\brief The type of data associated with each vertex in the graph\n *\n * The vertex data type must be \\ref sec_serializable.\n */\n typedef typename VertexProgram::vertex_data_type vertex_data_type;\n\n /**\n * \\brief The type of data associated with each edge in the graph\n *\n * The edge data type must be \\ref sec_serializable.\n */\n typedef typename VertexProgram::edge_data_type edge_data_type;\n\n /**\n * \\brief The type of graph supported by this vertex program\n *\n * See graphlab::distributed_graph\n */\n typedef typename VertexProgram::graph_type graph_type;\n\n /**\n * \\brief The type used to represent a vertex in the graph.\n * See \\ref graphlab::distributed_graph::vertex_type for details\n *\n * The vertex type contains the function\n * \\ref graphlab::distributed_graph::vertex_type::data which\n * returns a reference to the vertex data as well as other functions\n * like \\ref graphlab::distributed_graph::vertex_type::num_in_edges\n * which returns the number of in edges.\n *\n */\n typedef typename graph_type::vertex_type vertex_type;\n\n /**\n * \\brief The type used to represent an edge in the graph.\n * See \\ref graphlab::distributed_graph::edge_type for details.\n *\n * The edge type contains the function\n * \\ref graphlab::distributed_graph::edge_type::data which returns a\n * reference to the edge data. In addition the edge type contains\n * the function \\ref graphlab::distributed_graph::edge_type::source and\n * \\ref graphlab::distributed_graph::edge_type::target.\n *\n */\n typedef typename graph_type::edge_type edge_type;\n\n /**\n * \\brief The type of the callback interface passed by the engine to vertex\n * programs. See \\ref graphlab::icontext for details.\n *\n * The context callback is passed to the vertex program functions and is\n * used to signal other vertices, get the current iteration, and access\n * information about the engine.\n */\n typedef icontext<graph_type, gather_type, message_type> icontext_type;\n\n private:\n /// \\internal \\brief The base type of all schedulers\n message_array<message_type> messages;\n\n /** \\internal\n * \\brief The true type of the callback context interface which\n * implements icontext. \\see graphlab::icontext graphlab::context\n */\n typedef context<async_consistent_engine> context_type;\n\n // context needs access to internal functions\n friend class context<async_consistent_engine>;\n\n /// \\internal \\brief The type used to refer to vertices in the local graph\n typedef typename graph_type::local_vertex_type local_vertex_type;\n /// \\internal \\brief The type used to refer to edges in the local graph\n typedef typename graph_type::local_edge_type local_edge_type;\n /// \\internal \\brief The type used to refer to vertex IDs in the local graph\n typedef typename graph_type::lvid_type lvid_type;\n\n /// \\internal \\brief The type of the current engine instantiation\n typedef async_consistent_engine<VertexProgram> engine_type;\n\n typedef conditional_addition_wrapper<gather_type> conditional_gather_type;\n \n /// The RPC interface\n dc_dist_object<async_consistent_engine<VertexProgram> > rmi;\n\n /// A reference to the active graph\n graph_type& graph;\n\n /// A pointer to the lock implementation\n distributed_chandy_misra<graph_type>* cmlocks;\n\n /// Per vertex data locks\n std::vector<simple_spinlock> vertexlocks;\n\n /// Total update function completion time\n std::vector<double> total_completion_time;\n\n /**\n * \\brief This optional vector contains caches of previous gather\n * contributions for each machine.\n *\n * Caching is done locally and therefore a high-degree vertex may\n * have multiple caches (one per machine).\n */\n std::vector<gather_type> gather_cache;\n\n /**\n * \\brief A bit indicating if the local gather for that vertex is\n * available.\n */\n dense_bitset has_cache;\n\n bool use_cache;\n\n /// Engine threads.\n fiber_group thrgroup;\n\n //! The scheduler\n ischeduler* scheduler_ptr;\n\n typedef typename iengine<VertexProgram>::aggregator_type aggregator_type;\n aggregator_type aggregator;\n\n /// Number of kernel threads\n size_t ncpus;\n /// Size of each fiber stack\n size_t stacksize;\n /// Number of fibers\n size_t nfibers;\n /// set to true if engine is started\n bool started;\n\n bool track_task_time;\n /// A pointer to the distributed consensus object\n fiber_async_consensus* consensus;\n\n /**\n * Used only by the locking subsystem.\n * to allow the fiber to go to sleep when waiting for the locks to\n * be ready.\n */\n struct vertex_fiber_cm_handle {\n mutex lock;\n bool philosopher_ready;\n size_t fiber_handle;\n };\n std::vector<vertex_fiber_cm_handle*> cm_handles;\n\n dense_bitset program_running;\n dense_bitset hasnext;\n\n // Various counters.\n atomic<uint64_t> programs_executed;\n\n timer launch_timer;\n\n /// Defaults to (-1), defines a timeout\n size_t timed_termination;\n \n /// engine option. Sets to true if factorized consistency is used\n bool factorized_consistency;\n\n bool endgame_mode;\n\n /// Time when engine is started\n float engine_start_time;\n\n /// True when a force stop is triggered (possibly via a timeout)\n bool force_stop;\n\n graphlab_options opts_copy; // local copy of options to pass to\n // scheduler construction\n\n execution_status::status_enum termination_reason;\n\n std::vector<mutex> aggregation_lock;\n std::vector<std::deque<std::string> > aggregation_queue;\n public:\n\n /**\n * Constructs an asynchronous consistent distributed engine.\n * The number of threads to create are read from\n * \\ref graphlab_options::get_ncpus \"opts.get_ncpus()\". The scheduler to\n * construct is read from\n * \\ref graphlab_options::get_scheduler_type() \"opts.get_scheduler_type()\".\n * The default scheduler\n * is the queued_fifo scheduler. For details on the scheduler types\n * \\see scheduler_types\n *\n * See the <a href=#engineopts> main class documentation</a> for the\n * available engine options.\n *\n * \\param dc Distributed controller to associate with\n * \\param graph The graph to schedule over. The graph must be fully\n * constructed and finalized.\n * \\param opts A graphlab::graphlab_options object containing options and\n * parameters for the scheduler and the engine.\n */\n async_consistent_engine(distributed_control &dc,\n graph_type& graph,\n const graphlab_options& opts = graphlab_options()) :\n rmi(dc, this), graph(graph), scheduler_ptr(NULL),\n aggregator(dc, graph, new context_type(*this, graph)), started(false),\n engine_start_time(timer::approx_time_seconds()), force_stop(false) {\n rmi.barrier();\n\n nfibers = 10000;\n stacksize = 16384;\n use_cache = false;\n factorized_consistency = true;\n track_task_time = false;\n timed_termination = (size_t)(-1);\n termination_reason = execution_status::UNSET;\n set_options(opts);\n init();\n total_completion_time.resize(fiber_control::get_instance().num_workers());\n init();\n rmi.barrier();\n }\n\n private:\n\n /**\n * \\internal\n * Configures the engine with the provided options.\n * The number of threads to create are read from\n * opts::get_ncpus(). The scheduler to construct is read from\n * graphlab_options::get_scheduler_type(). The default scheduler\n * is the queued_fifo scheduler. For details on the scheduler types\n * \\see scheduler_types\n */\n void set_options(const graphlab_options& opts) {\n rmi.barrier();\n ncpus = opts.get_ncpus();\n ASSERT_GT(ncpus, 0);\n aggregation_lock.resize(opts.get_ncpus());\n aggregation_queue.resize(opts.get_ncpus());\n std::vector<std::string> keys = opts.get_engine_args().get_option_keys();\n foreach(std::string opt, keys) {\n if (opt == \"timeout\") {\n opts.get_engine_args().get_option(\"timeout\", timed_termination);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: timeout = \" << timed_termination << std::endl;\n } else if (opt == \"factorized\") {\n opts.get_engine_args().get_option(\"factorized\", factorized_consistency);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: factorized = \" << factorized_consistency << std::endl;\n } else if (opt == \"nfibers\") {\n opts.get_engine_args().get_option(\"nfibers\", nfibers);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: nfibers = \" << nfibers << std::endl;\n } else if (opt == \"track_task_time\") {\n opts.get_engine_args().get_option(\"track_task_time\", track_task_time);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: track_task_time = \" << track_task_time<< std::endl;\n }else if (opt == \"stacksize\") {\n opts.get_engine_args().get_option(\"stacksize\", stacksize);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: stacksize= \" << stacksize << std::endl;\n } else if (opt == \"use_cache\") {\n opts.get_engine_args().get_option(\"use_cache\", use_cache);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: use_cache = \" << use_cache << std::endl;\n } else {\n logstream(LOG_FATAL) << \"Unexpected Engine Option: \" << opt << std::endl;\n }\n }\n opts_copy = opts;\n // set a default scheduler if none\n if (opts_copy.get_scheduler_type() == \"\") {\n opts_copy.set_scheduler_type(\"queued_fifo\");\n }\n\n // construct scheduler passing in the copy of the options from set_options\n scheduler_ptr = scheduler_factory::\n new_scheduler(graph.num_local_vertices(),\n opts_copy);\n rmi.barrier();\n\n // create initial fork arrangement based on the alternate vid mapping\n if (factorized_consistency == false) {\n cmlocks = new distributed_chandy_misra<graph_type>(rmi.dc(), graph,\n boost::bind(&engine_type::lock_ready, this, _1));\n \n }\n else {\n cmlocks = NULL;\n }\n\n // construct the termination consensus object\n consensus = new fiber_async_consensus(rmi.dc(), nfibers);\n }\n\n /**\n * \\internal\n * Initializes the engine with respect to the associated graph.\n * This call will initialize all internal and scheduling datastructures.\n * This function must be called prior to any signal function.\n */\n void init() {\n // construct all the required datastructures\n // deinitialize performs the reverse\n graph.finalize();\n scheduler_ptr->set_num_vertices(graph.num_local_vertices());\n messages.resize(graph.num_local_vertices());\n vertexlocks.resize(graph.num_local_vertices());\n program_running.resize(graph.num_local_vertices());\n hasnext.resize(graph.num_local_vertices());\n if (use_cache) {\n gather_cache.resize(graph.num_local_vertices(), gather_type());\n has_cache.resize(graph.num_local_vertices());\n has_cache.clear();\n }\n if (!factorized_consistency) {\n cm_handles.resize(graph.num_local_vertices());\n }\n rmi.barrier();\n }\n\n\n\n public:\n ~async_consistent_engine() {\n delete consensus;\n delete cmlocks;\n delete scheduler_ptr;\n }\n\n\n\n\n // documentation inherited from iengine\n size_t num_updates() const {\n return programs_executed.value;\n }\n\n\n\n\n\n // documentation inherited from iengine\n float elapsed_seconds() const {\n return timer::approx_time_seconds() - engine_start_time;\n }\n\n\n /**\n * \\brief Not meaningful for the asynchronous engine. Returns -1.\n */\n int iteration() const { return -1; }\n\n\n/**************************************************************************\n * Signaling Interface *\n **************************************************************************/\n\n private:\n\n /**\n * \\internal\n * This is used to receive a message forwarded from another machine\n */\n void rpc_signal(vertex_id_type vid,\n const message_type& message) {\n if (force_stop) return;\n const lvid_type local_vid = graph.local_vid(vid);\n double priority;\n messages.add(local_vid, message, &priority);\n scheduler_ptr->schedule(local_vid, priority);\n consensus->cancel();\n }\n\n /**\n * \\internal\n * \\brief Signals a vertex with an optional message\n *\n * Signals a vertex, and schedules it to be executed in the future.\n * must be called on a vertex accessible by the current machine.\n */\n void internal_signal(const vertex_type& vtx,\n const message_type& message = message_type()) {\n if (force_stop) return;\n if (started) {\n const typename graph_type::vertex_record& rec = graph.l_get_vertex_record(vtx.local_id());\n const procid_t owner = rec.owner;\n if (endgame_mode) {\n // fast signal. push to the remote machine immediately\n if (owner != rmi.procid()) {\n const vertex_id_type vid = rec.gvid;\n rmi.remote_call(owner, &engine_type::rpc_signal, vid, message);\n }\n else {\n double priority;\n messages.add(vtx.local_id(), message, &priority);\n scheduler_ptr->schedule(vtx.local_id(), priority);\n consensus->cancel();\n }\n }\n else {\n\n double priority;\n messages.add(vtx.local_id(), message, &priority);\n scheduler_ptr->schedule(vtx.local_id(), priority);\n consensus->cancel();\n }\n }\n else {\n double priority;\n messages.add(vtx.local_id(), message, &priority);\n scheduler_ptr->schedule(vtx.local_id(), priority);\n consensus->cancel();\n }\n } // end of schedule\n\n\n /**\n * \\internal\n * \\brief Signals a vertex with an optional message\n *\n * Signals a global vid, and schedules it to be executed in the future.\n * If current machine does not contain the vertex, it is ignored.\n */\n void internal_signal_gvid(vertex_id_type gvid,\n const message_type& message = message_type()) {\n if (force_stop) return;\n if (graph.is_master(gvid)) {\n internal_signal(graph.vertex(gvid), message);\n } else {\n procid_t proc = graph.master(gvid);\n rmi.remote_call(proc, &async_consistent_engine::internal_signal_gvid,\n gvid, message);\n }\n } \n\n\n void rpc_internal_stop() {\n force_stop = true;\n termination_reason = execution_status::FORCED_ABORT;\n }\n\n /**\n * \\brief Force engine to terminate immediately.\n *\n * This function is used to stop the engine execution by forcing\n * immediate termination.\n */\n void internal_stop() {\n for (procid_t i = 0;i < rmi.numprocs(); ++i) {\n rmi.remote_call(i, &async_consistent_engine::rpc_internal_stop);\n }\n }\n\n\n\n /**\n * \\brief Post a to a previous gather for a give vertex.\n *\n * This function is called by the \\ref graphlab::context.\n *\n * @param [in] vertex The vertex to which to post a change in the sum\n * @param [in] delta The change in that sum\n */\n void internal_post_delta(const vertex_type& vertex,\n const gather_type& delta) {\n if(use_cache) {\n const lvid_type lvid = vertex.local_id();\n vertexlocks[lvid].lock();\n if( has_cache.get(lvid) ) {\n gather_cache[lvid] += delta;\n } else {\n // You cannot add a delta to an empty cache. A complete\n // gather must have been run.\n // gather_cache[lvid] = delta;\n // has_cache.set_bit(lvid);\n }\n vertexlocks[lvid].unlock();\n }\n }\n\n /**\n * \\brief Clear the cached gather for a vertex if one is\n * available.\n *\n * This function is called by the \\ref graphlab::context.\n *\n * @param [in] vertex the vertex for which to clear the cache\n */\n void internal_clear_gather_cache(const vertex_type& vertex) {\n const lvid_type lvid = vertex.local_id();\n if(use_cache && has_cache.get(lvid)) {\n vertexlocks[lvid].lock();\n gather_cache[lvid] = gather_type();\n has_cache.clear_bit(lvid);\n vertexlocks[lvid].unlock();\n }\n\n }\n\n public:\n\n\n\n void signal(vertex_id_type gvid,\n const message_type& message = message_type()) {\n rmi.barrier();\n internal_signal_gvid(gvid, message);\n rmi.barrier();\n }\n\n\n void signal_all(const message_type& message = message_type(),\n const std::string& order = \"shuffle\") {\n vertex_set vset = graph.complete_set();\n signal_vset(vset, message, order);\n } // end of schedule all\n\n void signal_vset(const vertex_set& vset,\n const message_type& message = message_type(),\n const std::string& order = \"shuffle\") {\n logstream(LOG_DEBUG) << rmi.procid() << \": Schedule All\" << std::endl;\n // allocate a vector with all the local owned vertices\n // and schedule all of them.\n std::vector<vertex_id_type> vtxs;\n vtxs.reserve(graph.num_local_own_vertices());\n for(lvid_type lvid = 0;\n lvid < graph.get_local_graph().num_vertices();\n ++lvid) {\n if (graph.l_vertex(lvid).owner() == rmi.procid() &&\n vset.l_contains(lvid)) {\n vtxs.push_back(lvid);\n }\n }\n\n if(order == \"shuffle\") {\n graphlab::random::shuffle(vtxs.begin(), vtxs.end());\n }\n foreach(lvid_type lvid, vtxs) {\n double priority;\n messages.add(lvid, message, &priority);\n scheduler_ptr->schedule(lvid, priority);\n }\n rmi.barrier();\n }\n\n\n private: \n\n /**\n * Gets a task from the scheduler and the associated message\n */\n sched_status::status_enum get_next_sched_task( size_t threadid,\n lvid_type& lvid,\n message_type& msg) {\n while (1) {\n sched_status::status_enum stat = \n scheduler_ptr->get_next(threadid % ncpus, lvid);\n if (stat == sched_status::NEW_TASK) {\n if (messages.get(lvid, msg)) return stat;\n else continue;\n }\n return stat;\n }\n }\n\n void set_endgame_mode() {\n if (!endgame_mode) logstream(LOG_EMPH) << \"Endgame mode\\n\";\n endgame_mode = true;\n rmi.dc().set_fast_track_requests(true);\n } \n\n /**\n * \\internal\n * Called when get_a_task returns no internal task not a scheduler task.\n * This rechecks the status of the internal task queue and the scheduler\n * inside a consensus critical section.\n */\n bool try_to_quit(size_t threadid,\n bool& has_sched_msg,\n lvid_type& sched_lvid,\n message_type &msg) {\n if (timer::approx_time_seconds() - engine_start_time > timed_termination) {\n termination_reason = execution_status::TIMEOUT;\n force_stop = true;\n }\n fiber_control::yield();\n logstream(LOG_DEBUG) << rmi.procid() << \"-\" << threadid << \": \" << \"Termination Attempt \" << std::endl;\n has_sched_msg = false;\n consensus->begin_done_critical_section(threadid);\n sched_status::status_enum stat = \n get_next_sched_task(threadid, sched_lvid, msg);\n if (stat == sched_status::EMPTY || force_stop) {\n logstream(LOG_DEBUG) << rmi.procid() << \"-\" << threadid << \": \"\n << \"\\tTermination Double Checked\" << std::endl;\n\n if (!endgame_mode) logstream(LOG_EMPH) << \"Endgame mode\\n\";\n endgame_mode = true;\n // put everyone in endgame\n for (procid_t i = 0;i < rmi.dc().numprocs(); ++i) {\n rmi.remote_call(i, &async_consistent_engine::set_endgame_mode);\n } \n bool ret = consensus->end_done_critical_section(threadid);\n if (ret == false) {\n logstream(LOG_DEBUG) << rmi.procid() << \"-\" << threadid << \": \"\n << \"\\tCancelled\" << std::endl;\n } else {\n logstream(LOG_DEBUG) << rmi.procid() << \"-\" << threadid << \": \"\n << \"\\tDying\" << \" (\" << fiber_control::get_tid() << \")\" << std::endl;\n }\n return ret;\n } else {\n logstream(LOG_DEBUG) << rmi.procid() << \"-\" << threadid << \": \"\n << \"\\tCancelled by Scheduler Task\" << std::endl;\n consensus->cancel_critical_section(threadid);\n has_sched_msg = true;\n return false;\n }\n } // end of try to quit\n\n\n /**\n * \\internal\n * When all distributed locks are acquired, this function is called\n * from the chandy misra implementation on the master vertex.\n * Here, we perform initialization\n * of the task and switch the vertex to a gathering state\n */\n void lock_ready(lvid_type lvid) {\n cm_handles[lvid]->lock.lock();\n cm_handles[lvid]->philosopher_ready = true;\n fiber_control::schedule_tid(cm_handles[lvid]->fiber_handle);\n cm_handles[lvid]->lock.unlock();\n }\n\n\n conditional_gather_type perform_gather(vertex_id_type vid,\n vertex_program_type& vprog_) {\n vertex_program_type vprog = vprog_;\n lvid_type lvid = graph.local_vid(vid);\n local_vertex_type local_vertex(graph.l_vertex(lvid));\n vertex_type vertex(local_vertex);\n context_type context(*this, graph);\n edge_dir_type gather_dir = vprog.gather_edges(context, vertex);\n conditional_gather_type accum;\n\n //check against the cache\n if( use_cache && has_cache.get(lvid) ) {\n accum.set(gather_cache[lvid]);\n return accum;\n }\n // do in edges\n if(gather_dir == IN_EDGES || gather_dir == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.in_edges()) {\n edge_type edge(local_edge);\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n vertexlocks[std::min(a,b)].lock();\n vertexlocks[std::max(a,b)].lock();\n accum += vprog.gather(context, vertex, edge);\n vertexlocks[a].unlock();\n vertexlocks[b].unlock();\n }\n } \n // do out edges\n if(gather_dir == OUT_EDGES || gather_dir == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.out_edges()) {\n edge_type edge(local_edge);\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n vertexlocks[std::min(a,b)].lock();\n vertexlocks[std::max(a,b)].lock();\n accum += vprog.gather(context, vertex, edge);\n vertexlocks[a].unlock();\n vertexlocks[b].unlock();\n }\n } \n if (use_cache) {\n gather_cache[lvid] = accum.value; has_cache.set_bit(lvid);\n }\n return accum;\n }\n\n\n void perform_scatter_local(lvid_type lvid,\n vertex_program_type& vprog) {\n local_vertex_type local_vertex(graph.l_vertex(lvid));\n vertex_type vertex(local_vertex);\n context_type context(*this, graph);\n edge_dir_type scatter_dir = vprog.scatter_edges(context, vertex);\n if(scatter_dir == IN_EDGES || scatter_dir == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.in_edges()) {\n edge_type edge(local_edge);\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n vertexlocks[std::min(a,b)].lock();\n vertexlocks[std::max(a,b)].lock();\n vprog.scatter(context, vertex, edge);\n vertexlocks[a].unlock();\n vertexlocks[b].unlock();\n }\n } \n if(scatter_dir == OUT_EDGES || scatter_dir == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.out_edges()) {\n edge_type edge(local_edge);\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n vertexlocks[std::min(a,b)].lock();\n vertexlocks[std::max(a,b)].lock();\n vprog.scatter(context, vertex, edge);\n vertexlocks[a].unlock();\n vertexlocks[b].unlock();\n }\n } \n\n // release locks\n if (!factorized_consistency) {\n cmlocks->philosopher_stops_eating_per_replica(lvid);\n }\n }\n\n\n void perform_scatter(vertex_id_type vid,\n vertex_program_type& vprog_,\n const vertex_data_type& newdata) {\n vertex_program_type vprog = vprog_;\n lvid_type lvid = graph.local_vid(vid);\n vertexlocks[lvid].lock();\n graph.l_vertex(lvid).data() = newdata;\n vertexlocks[lvid].unlock();\n perform_scatter_local(lvid, vprog);\n }\n\n\n // make sure I am the only person running.\n // if returns false, the message has been dropped into the message array.\n // quit\n bool get_exclusive_access_to_vertex(const lvid_type lvid,\n const message_type& msg) {\n vertexlocks[lvid].lock();\n bool someone_else_running = program_running.set_bit(lvid);\n if (someone_else_running) {\n // bad. someone else is here.\n // drop it into the message array\n messages.add(lvid, msg);\n hasnext.set_bit(lvid);\n } \n vertexlocks[lvid].unlock();\n return !someone_else_running;\n }\n\n\n\n // make sure I am the only person running.\n // if returns false, the message has been dropped into the message array.\n // quit\n void release_exclusive_access_to_vertex(const lvid_type lvid) {\n vertexlocks[lvid].lock();\n // someone left a next message for me\n // reschedule it at high priority\n if (hasnext.get(lvid)) {\n scheduler_ptr->schedule(lvid, 10000.0);\n consensus->cancel();\n hasnext.clear_bit(lvid);\n }\n program_running.clear_bit(lvid);\n vertexlocks[lvid].unlock();\n }\n\n\n /**\n * \\internal\n * Called when the scheduler returns a vertex to run.\n * If this function is called with vertex locks acquired, prelocked\n * should be true. Otherwise it should be false.\n */\n void eval_sched_task(const lvid_type lvid,\n const message_type& msg) {\n const typename graph_type::vertex_record& rec = graph.l_get_vertex_record(lvid);\n vertex_id_type vid = rec.gvid;\n char task_time_data[sizeof(timer)];\n timer* task_time;\n if (track_task_time) {\n // placement new to create the timer\n task_time = reinterpret_cast<timer*>(task_time_data);\n new (task_time) timer();\n }\n // if this is another machine's forward it\n if (rec.owner != rmi.procid()) {\n rmi.remote_call(rec.owner, &engine_type::rpc_signal, vid, msg);\n return;\n }\n // I have to run this myself\n \n if (!get_exclusive_access_to_vertex(lvid, msg)) return;\n\n /**************************************************************************/\n /* Acquire Locks */\n /**************************************************************************/\n if (!factorized_consistency) {\n // begin lock acquisition\n cm_handles[lvid] = new vertex_fiber_cm_handle;\n cm_handles[lvid]->philosopher_ready = false;\n cm_handles[lvid]->fiber_handle = fiber_control::get_tid();\n cmlocks->make_philosopher_hungry(lvid);\n cm_handles[lvid]->lock.lock();\n while (!cm_handles[lvid]->philosopher_ready) {\n fiber_control::deschedule_self(&(cm_handles[lvid]->lock.m_mut));\n cm_handles[lvid]->lock.lock();\n }\n cm_handles[lvid]->lock.unlock();\n }\n\n /**************************************************************************/\n /* Begin Program */\n /**************************************************************************/\n context_type context(*this, graph);\n vertex_program_type vprog = vertex_program_type();\n local_vertex_type local_vertex(graph.l_vertex(lvid));\n vertex_type vertex(local_vertex);\n\n /**************************************************************************/\n /* init phase */\n /**************************************************************************/\n vprog.init(context, vertex, msg);\n\n /**************************************************************************/\n /* Gather Phase */\n /**************************************************************************/\n conditional_gather_type gather_result;\n std::vector<request_future<conditional_gather_type> > gather_futures;\n foreach(procid_t mirror, local_vertex.mirrors()) {\n gather_futures.push_back(\n object_fiber_remote_request(rmi, \n mirror, \n &async_consistent_engine::perform_gather, \n vid,\n vprog));\n }\n gather_result += perform_gather(vid, vprog);\n\n for(size_t i = 0;i < gather_futures.size(); ++i) {\n gather_result += gather_futures[i]();\n }\n\n /**************************************************************************/\n /* apply phase */\n /**************************************************************************/\n vertexlocks[lvid].lock();\n vprog.apply(context, vertex, gather_result.value); \n vertexlocks[lvid].unlock();\n\n\n /**************************************************************************/\n /* scatter phase */\n /**************************************************************************/\n\n // should I wait for the scatter? nah... but in case you want to\n // the code is commented below\n /*foreach(procid_t mirror, local_vertex.mirrors()) {\n rmi.remote_call(mirror, \n &async_consistent_engine::perform_scatter, \n vid,\n vprog,\n local_vertex.data());\n }*/\n\n std::vector<request_future<void> > scatter_futures;\n foreach(procid_t mirror, local_vertex.mirrors()) {\n scatter_futures.push_back(\n object_fiber_remote_request(rmi, \n mirror, \n &async_consistent_engine::perform_scatter, \n vid,\n vprog,\n local_vertex.data()));\n }\n perform_scatter_local(lvid, vprog);\n for(size_t i = 0;i < scatter_futures.size(); ++i) \n scatter_futures[i]();\n\n /************************************************************************/\n /* Release Locks */\n /************************************************************************/\n // the scatter is used to release the chandy misra\n // here I cleanup\n if (!factorized_consistency) {\n delete cm_handles[lvid];\n cm_handles[lvid] = NULL;\n }\n release_exclusive_access_to_vertex(lvid);\n if (track_task_time) {\n total_completion_time[fiber_control::get_worker_id()] += \n task_time->current_time();\n task_time->~timer();\n }\n programs_executed.inc(); \n }\n\n\n /**\n * \\internal\n * Per thread main loop\n */\n void thread_start(size_t threadid) {\n bool has_sched_msg = false;\n std::vector<std::vector<lvid_type> > internal_lvid;\n lvid_type sched_lvid;\n\n message_type msg;\n float last_aggregator_check = timer::approx_time_seconds();\n timer ti; ti.start();\n while(1) {\n if (timer::approx_time_seconds() != last_aggregator_check && !endgame_mode) {\n last_aggregator_check = timer::approx_time_seconds();\n std::string key = aggregator.tick_asynchronous();\n if (key != \"\") {\n for (size_t i = 0;i < aggregation_lock.size(); ++i) {\n aggregation_lock[i].lock();\n aggregation_queue[i].push_back(key);\n aggregation_lock[i].unlock();\n }\n }\n }\n\n // test the aggregator\n while(!aggregation_queue[fiber_control::get_worker_id()].empty()) {\n size_t wid = fiber_control::get_worker_id();\n ASSERT_LT(wid, ncpus);\n aggregation_lock[wid].lock();\n std::string key = aggregation_queue[wid].front();\n aggregation_queue[wid].pop_front();\n aggregation_lock[wid].unlock();\n aggregator.tick_asynchronous_compute(wid, key);\n }\n\n sched_status::status_enum stat = get_next_sched_task(threadid, sched_lvid, msg);\n\n\n has_sched_msg = stat != sched_status::EMPTY;\n if (stat != sched_status::EMPTY) {\n eval_sched_task(sched_lvid, msg);\n if (endgame_mode) rmi.dc().flush();\n }\n else if (!try_to_quit(threadid, has_sched_msg, sched_lvid, msg)) {\n /*\n * We failed to obtain a task, try to quit\n */\n if (has_sched_msg) {\n eval_sched_task(sched_lvid, msg);\n }\n } else { \n break; \n }\n\n if (fiber_control::worker_has_priority_fibers_on_queue()) {\n fiber_control::yield();\n }\n }\n } // end of thread start\n\n/**************************************************************************\n * Main engine start() *\n **************************************************************************/\n\n public:\n\n\n /**\n * \\brief Start the engine execution.\n *\n * This function starts the engine and does not\n * return until the scheduler has no tasks remaining.\n *\n * \\return the reason for termination\n */\n execution_status::status_enum start() {\n bool old_fasttrack = rmi.dc().set_fast_track_requests(false);\n logstream(LOG_INFO) << \"Spawning \" << nfibers << \" threads\" << std::endl;\n ASSERT_TRUE(scheduler_ptr != NULL);\n consensus->reset();\n\n // now. It is of critical importance that we match the number of \n // actual workers\n \n\n // start the aggregator\n aggregator.start(ncpus);\n aggregator.aggregate_all_periodic();\n\n started = true;\n\n rmi.barrier();\n size_t allocatedmem = memory_info::allocated_bytes();\n rmi.all_reduce(allocatedmem);\n\n engine_start_time = timer::approx_time_seconds();\n force_stop = false;\n endgame_mode = false;\n programs_executed = 0;\n launch_timer.start();\n\n termination_reason = execution_status::RUNNING;\n if (rmi.procid() == 0) {\n logstream(LOG_INFO) << \"Total Allocated Bytes: \" << allocatedmem << std::endl;\n }\n thrgroup.set_stacksize(stacksize);\n \n size_t effncpus = std::min(ncpus, fiber_control::get_instance().num_workers());\n for (size_t i = 0; i < nfibers ; ++i) {\n thrgroup.launch(boost::bind(&engine_type::thread_start, this, i), \n i % effncpus);\n }\n thrgroup.join();\n aggregator.stop();\n // if termination reason was not changed, then it must be depletion\n if (termination_reason == execution_status::RUNNING) {\n termination_reason = execution_status::TASK_DEPLETION;\n }\n\n size_t ctasks = programs_executed.value;\n rmi.all_reduce(ctasks);\n programs_executed.value = ctasks;\n\n rmi.cout() << \"Completed Tasks: \" << programs_executed.value << std::endl;\n\n\n size_t numjoins = messages.num_joins();\n rmi.all_reduce(numjoins);\n rmi.cout() << \"Schedule Joins: \" << numjoins << std::endl;\n\n size_t numadds = messages.num_adds();\n rmi.all_reduce(numadds);\n rmi.cout() << \"Schedule Adds: \" << numadds << std::endl;\n\n if (track_task_time) {\n double total_task_time = 0;\n for (size_t i = 0;i < total_completion_time.size(); ++i) {\n total_task_time += total_completion_time[i];\n }\n rmi.all_reduce(total_task_time);\n rmi.cerr() << \"Average Task Completion Time = \" \n << total_task_time / programs_executed.value << std::endl;\n }\n\n\n ASSERT_TRUE(scheduler_ptr->empty());\n started = false;\n\n rmi.dc().set_fast_track_requests(old_fasttrack);\n return termination_reason;\n } // end of start\n\n\n public:\n aggregator_type* get_aggregator() { return &aggregator; }\n\n }; // end of class\n} // namespace\n\n#include <graphlab/macros_undef.hpp>\n\n#endif // GRAPHLAB_DISTRIBUTED_ENGINE_HPP\n\n" }, { "path": "src/graphlab/engine/distributed_chandy_misra.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_DISTRIBUTED_CHANDY_MISRA_HPP\n#define GRAPHLAB_DISTRIBUTED_CHANDY_MISRA_HPP\n#include <vector>\n#include <graphlab/rpc/dc_dist_object.hpp>\n#include <graphlab/rpc/distributed_event_log.hpp>\n#include <graphlab/logger/assertions.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n\n/**\n *\n * \\internal\n */\ntemplate <typename GraphType>\nclass distributed_chandy_misra {\n public:\n typedef typename GraphType::local_vertex_type local_vertex_type;\n typedef typename GraphType::local_edge_type local_edge_type;\n \n typedef typename GraphType::vertex_id_type vertex_id_type;\n typedef typename GraphType::lvid_type lvid_type;\n\n typedef distributed_chandy_misra<GraphType> dcm_type;\n dc_dist_object<dcm_type> rmi;\n GraphType& graph;\n\n boost::function<void(lvid_type)> callback;\n boost::function<void(lvid_type)> hors_doeuvre_callback;\n /*\n * Each \"fork\" is one character.\n * bit 0: owner. if 0 is src. if 1 is target\n * bit 1: clean = 0, dirty = 1\n * bit 2: owner 0 request\n * bit 3: owner 1 request\n */\n std::vector<unsigned char> forkset;\n enum { OWNER_BIT = 1,\n DIRTY_BIT = 2,\n REQUEST_0 = 4,\n REQUEST_1 = 8 };\n enum {OWNER_SOURCE = 0, OWNER_TARGET = 1};\n inline unsigned char request_bit(bool owner) {\n return owner ? REQUEST_1 : REQUEST_0;\n }\n\n\n enum {\n COLLISIONS = 0,\n CANCELLATIONS = 1,\n ACCEPTED_CANCELLATIONS = 2\n };\n \n struct philosopher {\n vertex_id_type num_edges;\n vertex_id_type forks_acquired;\n simple_spinlock lock;\n unsigned char state;\n unsigned char counter;\n bool cancellation_sent;\n bool lockid;\n };\n std::vector<philosopher> philosopherset;\n atomic<size_t> clean_fork_count;\n \n /*\n * Possible values for the philosopher state\n */\n enum {\n THINKING = 0,\n HUNGRY = 1,\n HORS_DOEUVRE = 2, \n EATING = 3\n };\n\n /** Places a request for the fork. Requires fork to be locked */\n inline void request_for_fork(size_t forkid, bool nextowner) {\n __sync_fetch_and_or(&forkset[forkid], request_bit(nextowner));\n }\n\n inline bool fork_owner(size_t forkid) {\n return forkset[forkid] & OWNER_BIT;\n }\n\n inline bool fork_dirty(size_t forkid) {\n return !!(forkset[forkid] & DIRTY_BIT);\n }\n\n inline void dirty_fork(size_t forkid) {\n if ((forkset[forkid] & DIRTY_BIT) == 0) clean_fork_count.dec();\n __sync_fetch_and_or(&forkset[forkid], DIRTY_BIT);\n }\n\n\n void compute_initial_fork_arrangement() {\n for (lvid_type i = 0;i < graph.num_local_vertices(); ++i) {\n local_vertex_type lvertex(graph.l_vertex(i));\n philosopherset[i].num_edges = lvertex.num_in_edges() +\n lvertex.num_out_edges();\n philosopherset[i].state = THINKING;\n philosopherset[i].forks_acquired = 0;\n philosopherset[i].counter = 0;\n philosopherset[i].cancellation_sent = false;\n philosopherset[i].lockid = false;\n }\n for (lvid_type i = 0;i < graph.num_local_vertices(); ++i) {\n local_vertex_type lvertex(graph.l_vertex(i));\n foreach(local_edge_type edge, lvertex.in_edges()) {\n if (edge.source().global_id() > edge.target().global_id()) {\n forkset[edge.id()] = DIRTY_BIT | OWNER_TARGET;\n philosopherset[edge.target().id()].forks_acquired++;\n }\n else {\n forkset[edge.id()] = DIRTY_BIT | OWNER_SOURCE;\n philosopherset[edge.source().id()].forks_acquired++;\n }\n }\n }\n }\n\n /**\n * We already have v1, we want to acquire v2.\n * When this function returns, both v1 and v2 locks are acquired\n */\n void try_acquire_edge_with_backoff(lvid_type v1,\n lvid_type v2) {\n if (v1 < v2) {\n philosopherset[v2].lock.lock();\n }\n else if (!philosopherset[v2].lock.try_lock()) {\n philosopherset[v1].lock.unlock();\n philosopherset[v2].lock.lock();\n philosopherset[v1].lock.lock();\n }\n }\n \n/****************************************************************************\n * Tries to move a requested fork\n *\n * Pseudocode:\n * If current owner is hungry and fork is clean\n * Ignore\n * ElseIf current owner is Thinking\n * Relinquish fork immediately and clear the request flag\n * ElseIf current owner is hors_doeuvre and fork is clean\n * Ignore\n * ElseIf current owner is hors_doeuvre and fork is dirty\n * Send cancellation message\n * Set cancelsent\n * End\n * Return true if changes were made\n ***************************************************************************/\n inline bool advance_fork_state_on_lock(size_t forkid,\n lvid_type source,\n lvid_type target) {\n unsigned char currentowner = forkset[forkid] & OWNER_BIT;\n if (currentowner == OWNER_SOURCE) {\n // if the current owner is not eating, and the\n // fork is dirty and other side has placed a request\n if (philosopherset[source].state != EATING &&\n (forkset[forkid] & DIRTY_BIT) &&\n (forkset[forkid] & REQUEST_1)) {\n\n if (philosopherset[source].state != HORS_DOEUVRE) {\n // change the owner and clean the fork)\n forkset[forkid] = OWNER_TARGET;\n clean_fork_count.inc();\n if (philosopherset[source].state == HUNGRY) {\n forkset[forkid] |= REQUEST_0;\n }\n philosopherset[source].forks_acquired--;\n philosopherset[target].forks_acquired++;\n return true;\n }\n else if (philosopherset[source].cancellation_sent == false) {\n //PERMANENT_ACCUMULATE_DIST_EVENT(eventlog, CANCELLATIONS, 1);\n philosopherset[source].cancellation_sent = true;\n bool lockid = philosopherset[source].lockid;\n philosopherset[source].lock.unlock();\n philosopherset[target].lock.unlock();\n issue_cancellation_request_unlocked(source, lockid);\n philosopherset[std::min(source, target)].lock.lock();\n philosopherset[std::max(source, target)].lock.lock();\n }\n }\n }\n else {\n // if the current owner is not eating, and the\n // fork is dirty and other side has placed a request\n if (philosopherset[target].state != EATING &&\n (forkset[forkid] & DIRTY_BIT) &&\n (forkset[forkid] & REQUEST_0)) {\n // change the owner and clean the fork)\n if (philosopherset[target].state != HORS_DOEUVRE) {\n forkset[forkid] = OWNER_SOURCE;\n clean_fork_count.inc();\n if (philosopherset[target].state == HUNGRY) {\n forkset[forkid] |= REQUEST_1;\n }\n philosopherset[source].forks_acquired++;\n philosopherset[target].forks_acquired--;\n return true;\n }\n else if (philosopherset[target].cancellation_sent == false) {\n //PERMANENT_ACCUMULATE_DIST_EVENT(eventlog, CANCELLATIONS, 1);\n philosopherset[target].cancellation_sent = true;\n bool lockid = philosopherset[target].lockid;\n philosopherset[source].lock.unlock();\n philosopherset[target].lock.unlock();\n issue_cancellation_request_unlocked(target, lockid);\n philosopherset[std::min(source, target)].lock.lock();\n philosopherset[std::max(source, target)].lock.lock();\n }\n }\n }\n //PERMANENT_ACCUMULATE_DIST_EVENT(eventlog, COLLISIONS, 1);\n return false;\n }\n\n\n/****************************************************************************\n * Performs a cancellation on a vertex.\n * \n * If lockIds do not match, ignore\n * If counter == 0 ignore\n * Otherwise, counter++ and reply cancellation accept.\n * Unfortunately, I cannot perform a local call here even if I am the\n * owner since this may produce a lock cycle. Irregardless of whether\n * the owner is local or not, this must be performed by a remote call\n ***************************************************************************/\n\n void cancellation_request_unlocked(lvid_type lvid, procid_t requestor, bool lockid) {\n philosopherset[lvid].lock.lock();\n \n if (philosopherset[lvid].lockid == lockid) {\n if (philosopherset[lvid].counter > 0) {\n /*ASSERT_TRUE(philosopherset[lvid].state == HORS_DOEUVRE || \n philosopherset[lvid].state == HUNGRY);*/\n ++philosopherset[lvid].counter;\n bool lockid = philosopherset[lvid].lockid;\n //PERMANENT_ACCUMULATE_DIST_EVENT(eventlog, ACCEPTED_CANCELLATIONS, 1);\n vertex_id_type gvid = graph.global_vid(lvid);\n logstream(LOG_DEBUG) << rmi.procid() <<\n \": Cancellation accepted on \" << gvid <<\n \"(\" << (int)philosopherset[lvid].counter << \")\" << std::endl;\n philosopherset[lvid].lock.unlock();\n \n if (requestor != rmi.procid()) {\n unsigned char pkey = rmi.dc().set_sequentialization_key(gvid % 254 + 1);\n rmi.remote_call(requestor,\n &dcm_type::rpc_cancellation_accept,\n gvid,\n lockid);\n rmi.dc().set_sequentialization_key(pkey);\n }\n else {\n cancellation_accept_unlocked(lvid, lockid);\n }\n }\n else {\n philosopherset[lvid].lock.unlock();\n logstream(LOG_DEBUG) << rmi.procid() <<\n \": Cancellation on \" << graph.global_vid(lvid) <<\n \" denied due to lock completion\" << std::endl;\n }\n }\n else {\n philosopherset[lvid].lock.unlock();\n logstream(LOG_DEBUG) << rmi.procid() <<\n \": Cancellation on \" << graph.global_vid(lvid) <<\n \" denied to invalid lock ID\" << std::endl;\n }\n \n }\n\n void rpc_cancellation_request(vertex_id_type gvid, procid_t requestor, bool lockid) {\n lvid_type lvid = graph.local_vid(gvid);\n cancellation_request_unlocked(lvid, requestor, lockid);\n }\n\n void issue_cancellation_request_unlocked(lvid_type lvid, bool lockid) {\n // signal the master\n logstream(LOG_DEBUG) << rmi.procid() <<\n \": Requesting cancellation on \" << graph.global_vid(lvid) << std::endl;\n local_vertex_type lvertex(graph.l_vertex(lvid));\n \n if (lvertex.owner() == rmi.procid()) {\n cancellation_request_unlocked(lvid, rmi.procid(), lockid);\n }\n else {\n unsigned char pkey = rmi.dc().set_sequentialization_key(lvertex.global_id() % 254 + 1);\n rmi.remote_call(lvertex.owner(),\n &dcm_type::rpc_cancellation_request,\n lvertex.global_id(),\n rmi.procid(), \n lockid);\n rmi.dc().set_sequentialization_key(pkey);\n\n }\n }\n\n\n/****************************************************************************\n * Accepts a cancellation on a vertex.\n *\n * Pseudocode\n * Change back to Hungry\n * Releases all dirty forks\n ****************************************************************************/\n\n void rpc_cancellation_accept(vertex_id_type gvid, bool lockid) {\n lvid_type lvid = graph.local_vid(gvid);\n cancellation_accept_unlocked(lvid, lockid);\n }\n\n void cancellation_accept_unlocked(lvid_type p_id, bool lockid) {\n std::vector<lvid_type> retval;\n philosopherset[p_id].lock.lock();\n //philosopher is now hungry!\n /*ASSERT_EQ (lockid, philosopherset[p_id].lockid);\n ASSERT_EQ((int)philosopherset[p_id].state, (int)HORS_DOEUVRE); */\n philosopherset[p_id].state = HUNGRY;\n philosopherset[p_id].cancellation_sent = false;\n \n local_vertex_type lvertex(graph.l_vertex(p_id));\n logstream(LOG_DEBUG) << rmi.procid() <<\n \": Cancellation accept received on \" << lvertex.global_id() << \" \" <<\n philosopherset[p_id].state << std::endl;\n\n // for each fork I own, try to give it away\n foreach(local_edge_type edge, lvertex.in_edges()) {\n try_acquire_edge_with_backoff(edge.target().id(), edge.source().id());\n if (philosopherset[p_id].state == HUNGRY) {\n //std::cout << \"\\t\" << graph.edge_id(edge) << \": \" << edge.source() << \"->\" << edge.target() << std::endl;\n lvid_type other = edge.source().id();\n size_t edgeid = edge.id();\n if (fork_owner(edgeid) == OWNER_TARGET && fork_dirty(edgeid)) {\n \n if (advance_fork_state_on_lock(edgeid, edge.source().id(), edge.target().id()) &&\n philosopherset[other].state == HUNGRY &&\n philosopherset[other].forks_acquired == philosopherset[other].num_edges) {\n philosopherset[other].state = HORS_DOEUVRE;\n philosopherset[other].cancellation_sent = false;\n // signal eating on other\n retval.push_back(other);\n }\n }\n philosopherset[edge.source().id()].lock.unlock();\n }\n else {\n philosopherset[edge.source().id()].lock.unlock();\n break;\n }\n \n }\n //std::cout << \"out edges: \" << std::endl;\n foreach(local_edge_type edge, lvertex.out_edges()) {\n //std::cout << \"\\t\" << graph.edge_id(edge) << \": \" << edge.source() << \"->\" << edge.target() << std::endl;\n try_acquire_edge_with_backoff(edge.source().id(), edge.target().id());\n if (philosopherset[p_id].state == HUNGRY) {\n lvid_type other = edge.target().id();\n size_t edgeid = edge.id();\n if (fork_owner(edgeid) == OWNER_SOURCE && fork_dirty(edgeid)) {\n if (advance_fork_state_on_lock(edgeid, edge.source().id(), edge.target().id()) &&\n philosopherset[other].state == HUNGRY &&\n philosopherset[other].forks_acquired == philosopherset[other].num_edges) {\n philosopherset[other].state = HORS_DOEUVRE;\n philosopherset[other].cancellation_sent = false;\n // signal eating on other\n retval.push_back(other);\n }\n }\n philosopherset[edge.target().id()].lock.unlock();\n }\n else { \n philosopherset[edge.target().id()].lock.unlock();\n break;\n }\n }\n \n if (philosopherset[p_id].state == HUNGRY &&\n philosopherset[p_id].forks_acquired == philosopherset[p_id].num_edges) {\n philosopherset[p_id].cancellation_sent = false;\n philosopherset[p_id].state = HORS_DOEUVRE;\n retval.push_back(p_id);\n }\n \n philosopherset[p_id].lock.unlock();\n foreach(lvid_type lvid, retval) {\n enter_hors_doeuvre_unlocked(lvid);\n }\n\n }\n \n/****************************************************************************\n * Make Philosopher Hungry.\n *\n * Pseudocode:\n * Set Philosopher to Hungry\n * For all edges adjacent to v with forks it does not own:\n * Send request for fork to neighboring vertex\n *\n * Conditions:\n * Must be Thinking\n * New lock ID must not be the same as the old lock ID\n *\n * Possible Immediate Transitions:\n * Current vertex may enter HORS_DOEUVRE\n ***************************************************************************/\n void rpc_make_philosopher_hungry(vertex_id_type gvid, bool newlockid) {\n lvid_type lvid = graph.local_vid(gvid);\n logstream(LOG_DEBUG) << rmi.procid() <<\n \": Local HUNGRY Philosopher \" << gvid << std::endl;\n philosopherset[lvid].lock.lock();\n\n //ASSERT_EQ((int)philosopherset[lvid].state, (int)THINKING);\n philosopherset[lvid].state = HUNGRY;\n\n// ASSERT_NE(philosopherset[lvid].lockid, newlockid);\n philosopherset[lvid].lockid = newlockid;\n\n philosopherset[lvid].lock.unlock();\n\n local_philosopher_grabs_forks(lvid);\n }\n\n void local_philosopher_grabs_forks(lvid_type p_id) {\n philosopherset[p_id].lock.lock();\n local_vertex_type lvertex(graph.l_vertex(p_id));\n //philosopher is now hungry!\n// now try to get all the forks. lock one edge at a time\n // using the backoff strategy\n //std::cout << \"vertex \" << p_id << std::endl;\n //std::cout << \"in edges: \" << std::endl;\n foreach(local_edge_type edge, lvertex.in_edges()) {\n try_acquire_edge_with_backoff(edge.target().id(), edge.source().id());\n if (philosopherset[p_id].state == HUNGRY) {\n\n //std::cout << \"\\t\" << graph.edge_id(edge) << \": \" << edge.source() << \"->\" << edge.target() << std::endl;\n size_t edgeid = edge.id();\n // if fork is owned by other edge, try to take it\n if (fork_owner(edgeid) == OWNER_SOURCE) {\n request_for_fork(edgeid, OWNER_TARGET);\n advance_fork_state_on_lock(edgeid, edge.source().id(), edge.target().id());\n }\n philosopherset[edge.source().id()].lock.unlock();\n }\n else {\n philosopherset[edge.source().id()].lock.unlock();\n break;\n }\n }\n //std::cout << \"out edges: \" << std::endl;\n foreach(local_edge_type edge, lvertex.out_edges()) {\n //std::cout << \"\\t\" << graph.edge_id(edge) << \": \" << edge.source() << \"->\" << edge.target() << std::endl;\n try_acquire_edge_with_backoff(edge.source().id(), edge.target().id());\n if (philosopherset[p_id].state == HUNGRY) {\n size_t edgeid = edge.id();\n\n // if fork is owned by other edge, try to take it\n if (fork_owner(edgeid) == OWNER_TARGET) {\n request_for_fork(edgeid, OWNER_SOURCE);\n advance_fork_state_on_lock(edgeid, edge.source().id(), edge.target().id());\n }\n philosopherset[edge.target().id()].lock.unlock();\n }\n else {\n philosopherset[edge.target().id()].lock.unlock();\n break;\n }\n }\n\n bool enter_hors = false;\n if (philosopherset[p_id].state == HUNGRY &&\n philosopherset[p_id].forks_acquired == philosopherset[p_id].num_edges) {\n philosopherset[p_id].state = HORS_DOEUVRE;\n philosopherset[p_id].cancellation_sent = false;\n enter_hors = true;\n }\n philosopherset[p_id].lock.unlock();\n if (enter_hors) enter_hors_doeuvre_unlocked(p_id);\n }\n\n/************************************************************************\n *\n * Called when a vertex may be ready to enter hors dourre\n * Locks must be maintained. HORS_DOEUVRE must be set prior\n * to entering this function .\n *\n ***********************************************************************/\n void enter_hors_doeuvre_unlocked(lvid_type p_id) {\n // if I got all forks I can eat\n logstream(LOG_DEBUG) << rmi.procid() <<\n \": Local HORS_DOEUVRE Philosopher \" << graph.global_vid(p_id) << std::endl;\n // signal the master\n local_vertex_type lvertex(graph.l_vertex(p_id));\n\n if (lvertex.owner() == rmi.procid()) {\n signal_ready_unlocked(p_id, philosopherset[p_id].lockid);\n }\n else {\n unsigned char pkey = rmi.dc().set_sequentialization_key(lvertex.global_id() % 254 + 1);\n if (hors_doeuvre_callback != NULL) hors_doeuvre_callback(p_id);\n rmi.remote_call(lvertex.owner(),\n &dcm_type::rpc_signal_ready,\n lvertex.global_id(), philosopherset[p_id].lockid);\n rmi.dc().set_sequentialization_key(pkey);\n }\n }\n\n/************************************************************************\n *\n * Called when a vertex enters HORS_DOEUVRE. Locks must be maintained.\n * \n * Conditions:\n * vertex must be in HUNGRY or HORS_DOEUVRE\n * lock IDs must match\n *\n * Possible Immediate Transitions\n * If counter == 0, transit to EATING\n ***********************************************************************/\n\n void signal_ready_unlocked(lvid_type lvid, bool lockid) {\n philosopherset[lvid].lock.lock();\n if(!(philosopherset[lvid].state == (int)HUNGRY ||\n philosopherset[lvid].state == (int)HORS_DOEUVRE)) {\n logstream(LOG_ERROR) << rmi.procid() <<\n \": Bad signal ready state!!!! : \" << (int)philosopherset[lvid].state << std::endl;\n logstream(LOG_ERROR) << rmi.procid() <<\n \" Lock IDs : \" << (int)philosopherset[lvid].lockid << \" \" << (int)lockid << std::endl;\n logstream(LOG_ERROR) << rmi.procid() <<\n \": BAD Global HORS_DOEUVRE \" << graph.global_vid(lvid)\n << \"(\" << (int)philosopherset[lvid].counter << \")\" << std::endl;\n \n /* ASSERT_TRUE(philosopherset[lvid].state == (int)HUNGRY ||\n philosopherset[lvid].state == (int)HORS_DOEUVRE);*/\n }\n \n// ASSERT_EQ(philosopherset[lvid].lockid, lockid);\n philosopherset[lvid].counter--;\n\n logstream(LOG_DEBUG) << rmi.procid() <<\n \": Global HORS_DOEUVRE \" << graph.global_vid(lvid)\n << \"(\" << (int)philosopherset[lvid].counter << \")\" << \" \" << (int)(philosopherset[lvid].state) << std::endl;\n \n if(philosopherset[lvid].counter == 0) {\n philosopherset[lvid].lock.unlock();\n // broadcast EATING\n local_vertex_type lvertex(graph.l_vertex(lvid));\n unsigned char pkey = rmi.dc().set_sequentialization_key(lvertex.global_id() % 254 + 1);\n rmi.remote_call(lvertex.mirrors().begin(), lvertex.mirrors().end(),\n &dcm_type::rpc_set_eating, lvertex.global_id(), lockid);\n set_eating(lvid, lockid);\n rmi.dc().set_sequentialization_key(pkey);\n }\n else {\n philosopherset[lvid].lock.unlock();\n }\n }\n\n\n void rpc_signal_ready(vertex_id_type gvid, bool lockid) {\n lvid_type lvid = graph.local_vid(gvid);\n signal_ready_unlocked(lvid, lockid);\n }\n\n void set_eating(lvid_type lvid, bool lockid) {\n philosopherset[lvid].lock.lock();\n \n logstream(LOG_DEBUG) << rmi.procid() <<\n \": EATING \" << graph.global_vid(lvid)\n << \"(\" << (int)philosopherset[lvid].counter << \")\" << std::endl;\n \n// ASSERT_EQ((int)philosopherset[lvid].state, (int)HORS_DOEUVRE);\n// ASSERT_EQ(philosopherset[lvid].lockid, lockid);\n philosopherset[lvid].state = EATING;\n philosopherset[lvid].cancellation_sent = false;\n philosopherset[lvid].lock.unlock();\n if (graph.l_vertex(lvid).owner() == rmi.procid()) {\n logstream(LOG_DEBUG) << rmi.procid() <<\n \": CALLBACK \" << graph.global_vid(lvid) << std::endl;\n\n callback(lvid);\n }\n }\n\n void rpc_set_eating(vertex_id_type gvid, bool lockid) {\n\n logstream(LOG_DEBUG) << rmi.procid() <<\n \": Receive Set EATING \" << gvid << std::endl;\n \n lvid_type lvid = graph.local_vid(gvid);\n set_eating(lvid, lockid);\n }\n/************************************************************************\n *\n * Called when a vertex stops eating\n *\n ***********************************************************************/\n\n\n\n inline bool advance_fork_state_on_unlock(size_t forkid,\n lvid_type source,\n lvid_type target) {\n\n unsigned char currentowner = forkset[forkid] & OWNER_BIT;\n if (currentowner == OWNER_SOURCE) {\n // if the current owner is not eating, and the\n // fork is dirty and other side has placed a request\n if ((forkset[forkid] & DIRTY_BIT) &&\n (forkset[forkid] & REQUEST_1)) {\n // change the owner and clean the fork)\n // keep my request bit if any\n clean_fork_count.inc();\n forkset[forkid] = OWNER_TARGET;\n philosopherset[source].forks_acquired--;\n philosopherset[target].forks_acquired++;\n return true;\n }\n }\n else {\n // if the current owner is not eating, and the\n // fork is dirty and other side has placed a request\n if ((forkset[forkid] & DIRTY_BIT) &&\n (forkset[forkid] & REQUEST_0)) {\n // change the owner and clean the fork)\n // keep my request bit if any\n clean_fork_count.inc();\n forkset[forkid] = OWNER_SOURCE;\n philosopherset[source].forks_acquired++;\n philosopherset[target].forks_acquired--;\n return true;\n }\n }\n return false;\n }\n\n\n \n\n \n void local_philosopher_stops_eating(lvid_type p_id) {\n std::vector<lvid_type> retval;\n philosopherset[p_id].lock.lock();\n if (philosopherset[p_id].state != EATING) {\n std::cout << rmi.procid() << \": \" << p_id << \"FAILED!! Cannot Stop Eating!\" << std::endl;\n// ASSERT_EQ((int)philosopherset[p_id].state, (int)EATING);\n }\n \n local_vertex_type lvertex(graph.l_vertex(p_id));\n // now forks are dirty\n foreach(local_edge_type edge, lvertex.in_edges()) {\n dirty_fork(edge.id());\n }\n \n foreach(local_edge_type edge, lvertex.out_edges()) {\n dirty_fork(edge.id());\n }\n \n \n philosopherset[p_id].state = THINKING;\n philosopherset[p_id].counter = 0;\n\n // now forks are dirty\n foreach(local_edge_type edge, lvertex.in_edges()) {\n try_acquire_edge_with_backoff(edge.target().id(), edge.source().id());\n lvid_type other = edge.source().id();\n if (philosopherset[p_id].state == THINKING) {\n size_t edgeid = edge.id();\n advance_fork_state_on_unlock(edgeid, edge.source().id(), edge.target().id());\n if (philosopherset[other].state == HUNGRY &&\n philosopherset[other].forks_acquired ==\n philosopherset[other].num_edges) {\n philosopherset[other].state = HORS_DOEUVRE;\n philosopherset[other].cancellation_sent = false;\n // signal eating on other\n retval.push_back(other);\n }\n philosopherset[other].lock.unlock();\n }\n else {\n philosopherset[other].lock.unlock();\n break;\n }\n }\n\n foreach(local_edge_type edge, lvertex.out_edges()) {\n try_acquire_edge_with_backoff(edge.source().id(), edge.target().id());\n lvid_type other = edge.target().id();\n if (philosopherset[p_id].state == THINKING) {\n size_t edgeid = edge.id();\n advance_fork_state_on_unlock(edgeid, edge.source().id(), edge.target().id());\n if (philosopherset[other].state == HUNGRY &&\n philosopherset[other].forks_acquired ==\n philosopherset[other].num_edges) {\n philosopherset[other].state = HORS_DOEUVRE;\n philosopherset[other].cancellation_sent = false;\n // signal eating on other\n retval.push_back(other);\n }\n philosopherset[other].lock.unlock();\n }\n else {\n philosopherset[other].lock.unlock();\n break;\n }\n }\n\n philosopherset[p_id].lock.unlock();\n foreach(lvid_type lvid, retval) {\n enter_hors_doeuvre_unlocked(lvid);\n }\n }\n\n void rpc_philosopher_stops_eating(vertex_id_type gvid) {\n logstream(LOG_DEBUG) << rmi.procid() << \": Receive STOP eating on \" << gvid << std::endl;\n local_philosopher_stops_eating(graph.local_vid(gvid));\n }\n\n public:\n inline distributed_chandy_misra(distributed_control &dc,\n GraphType &graph,\n boost::function<void(lvid_type)> callback,\n boost::function<void(lvid_type)> hors_doeuvre_callback = NULL\n ):\n rmi(dc, this),\n graph(graph),\n callback(callback),\n hors_doeuvre_callback(hors_doeuvre_callback){\n forkset.resize(graph.num_local_edges(), 0);\n philosopherset.resize(graph.num_local_vertices());\n compute_initial_fork_arrangement();\n\n rmi.barrier();\n }\n\n size_t num_clean_forks() const {\n return clean_fork_count.value;\n }\n\n void initialize_master_philosopher_as_hungry_locked(lvid_type p_id,\n bool lockid) {\n philosopherset[p_id].lockid = lockid;\n philosopherset[p_id].state = HUNGRY;\n philosopherset[p_id].counter = graph.l_vertex(p_id).num_mirrors() + 1;\n }\n \n void make_philosopher_hungry(lvid_type p_id) {\n local_vertex_type lvertex(graph.l_vertex(p_id)); \n// ASSERT_EQ(rec.get_owner(), rmi.procid());\n philosopherset[p_id].lock.lock();\n// ASSERT_EQ((int)philosopherset[p_id].state, (int)THINKING);\n bool newlockid = !philosopherset[p_id].lockid;\n initialize_master_philosopher_as_hungry_locked(p_id, newlockid);\n \n logstream(LOG_DEBUG) << rmi.procid() <<\n \": Global HUNGRY \" << lvertex.global_id()\n << \"(\" << (int)philosopherset[p_id].counter << \")\" << std::endl;\n \n philosopherset[p_id].lock.unlock();\n \n unsigned char pkey = rmi.dc().set_sequentialization_key(lvertex.global_id() % 254 + 1);\n rmi.remote_call(lvertex.mirrors().begin(), lvertex.mirrors().end(),\n &dcm_type::rpc_make_philosopher_hungry, lvertex.global_id(), newlockid);\n rmi.dc().set_sequentialization_key(pkey);\n local_philosopher_grabs_forks(p_id);\n }\n \n \n \n void make_philosopher_hungry_per_replica(lvid_type p_id) {\n local_vertex_type lvertex(graph.l_vertex(p_id)); \n philosopherset[p_id].lock.lock();\n// ASSERT_EQ((int)philosopherset[p_id].state, (int)THINKING);\n\n if (lvertex.owner() == rmi.procid()) {\n bool newlockid = !philosopherset[p_id].lockid;\n initialize_master_philosopher_as_hungry_locked(p_id, newlockid);\n \n logstream(LOG_DEBUG) << rmi.procid() <<\n \": Global HUNGRY \" << lvertex.global_id()\n << \"(\" << (int)philosopherset[p_id].counter << \")\" << std::endl;\n }\n else {\n bool newlockid = !philosopherset[p_id].lockid;\n philosopherset[p_id].lockid = newlockid;\n philosopherset[p_id].state = HUNGRY;\n }\n philosopherset[p_id].lock.unlock();\n local_philosopher_grabs_forks(p_id);\n }\n \n \n void philosopher_stops_eating(lvid_type p_id) {\n local_vertex_type lvertex(graph.l_vertex(p_id)); \n\n logstream(LOG_DEBUG) << rmi.procid() <<\n \": Global STOP Eating \" << lvertex.global_id() << std::endl;\n\n philosopherset[p_id].lock.lock();\n// ASSERT_EQ(philosopherset[p_id].state, (int)EATING);\n philosopherset[p_id].counter = 0;\n philosopherset[p_id].lock.unlock();\n unsigned char pkey = rmi.dc().set_sequentialization_key(lvertex.global_id() % 254 + 1);\n rmi.remote_call(lvertex.mirrors().begin(), lvertex.mirrors().end(),\n &dcm_type::rpc_philosopher_stops_eating, lvertex.global_id());\n rmi.dc().set_sequentialization_key(pkey);\n local_philosopher_stops_eating(p_id);\n }\n\n void philosopher_stops_eating_per_replica(lvid_type p_id) {\n logstream(LOG_DEBUG) << rmi.procid() <<\n \": Global STOP Eating \" << graph.global_vid(p_id) << std::endl;\n\n// ASSERT_EQ(philosopherset[p_id].state, (int)EATING);\n \n local_philosopher_stops_eating(p_id);\n }\n\n\n void no_locks_consistency_check() {\n // make sure all forks are dirty\n for (size_t i = 0;i < forkset.size(); ++i) ASSERT_TRUE(fork_dirty(i));\n // all philosophers are THINKING\n for (size_t i = 0;i < philosopherset.size(); ++i) ASSERT_TRUE(philosopherset[i].state == THINKING);\n }\n\n void print_out() {\n \n boost::unordered_set<size_t> eidset1;\n boost::unordered_set<size_t> eidset2;\n for (lvid_type v = 0; v < graph.num_local_vertices(); ++v) {\n local_vertex_type lvertex(graph.l_vertex(v));\n foreach(local_edge_type edge, lvertex.in_edges()) {\n size_t edgeid = edge.id();\n ASSERT_TRUE(eidset1.find(edgeid) == eidset1.end());\n eidset1.insert(edgeid);\n }\n foreach(local_edge_type edge, lvertex.out_edges()) {\n size_t edgeid = edge.id();\n ASSERT_TRUE(eidset2.find(edgeid) == eidset2.end());\n eidset2.insert(edgeid);\n }\n }\n ASSERT_EQ(eidset1.size(), eidset2.size());\n eidset1.clear(); eidset2.clear();\n complete_consistency_check();\n \n std::cout << \"Philosophers\\n\";\n std::cout << \"------------\\n\";\n for (lvid_type v = 0; v < graph.num_local_vertices(); ++v) {\n local_vertex_type lvertex(graph.l_vertex(v));\n std::cout << graph.global_vid(v) << \": \" << (int)philosopherset[v].state << \" \" <<\n philosopherset[v].forks_acquired << \" \" << philosopherset[v].num_edges << \" \";\n if (philosopherset[v].forks_acquired == philosopherset[v].num_edges) std::cout << \"---------------!\";\n std::cout << \"\\n\";\n std::cout << \"\\tin: \";\n foreach(local_edge_type edge, lvertex.in_edges()) {\n size_t edgeid = edge.id();\n if (fork_dirty(forkset[edgeid])) std::cout << edgeid << \":\" << (int)forkset[edgeid] << \" \";\n }\n std::cout << \"\\n\\tout: \";\n foreach(local_edge_type edge, lvertex.out_edges()) {\n size_t edgeid = edge.id();\n if (fork_dirty(forkset[edgeid])) std::cout << edgeid << \":\" << (int)forkset[edgeid] << \" \";\n }\n std::cout << \"\\n\";\n }\n }\n \n void complete_consistency_check() {\n for (lvid_type v = 0; v < graph.num_local_vertices(); ++v) {\n local_vertex_type lvertex(graph.l_vertex(v));\n // count the number of forks I own\n size_t numowned = 0;\n size_t numowned_clean = 0;\n foreach(local_edge_type edge, lvertex.in_edges()) {\n size_t edgeid = edge.id();\n if (fork_owner(edgeid) == OWNER_TARGET) {\n numowned++;\n if (!fork_dirty(edgeid)) numowned_clean++;\n }\n }\n foreach(local_edge_type edge, lvertex.out_edges()) {\n size_t edgeid = edge.id();\n if (fork_owner(edgeid) == OWNER_SOURCE) {\n numowned++;\n if (!fork_dirty(edgeid)) numowned_clean++;\n }\n }\n\n ASSERT_EQ(philosopherset[v].forks_acquired, numowned);\n if (philosopherset[v].state == THINKING) {\n ASSERT_EQ(numowned_clean, 0);\n }\n else if (philosopherset[v].state == HUNGRY) {\n ASSERT_NE(philosopherset[v].num_edges, philosopherset[v].forks_acquired);\n // any fork I am unable to acquire. Must be clean, and the other person\n // must be eating or hungry\n foreach(local_edge_type edge, lvertex.in_edges()) {\n size_t edgeid = edge.id();\n // not owned\n if (fork_owner(edgeid) == OWNER_SOURCE) {\n if (philosopherset[edge.source().id()].state != EATING) {\n if (fork_dirty(edgeid)) {\n std::cout << (int)(forkset[edgeid]) << \" \"\n << (int)philosopherset[edge.source().id()].state\n << \"->\" << (int)philosopherset[edge.target().id()].state\n << std::endl;\n ASSERT_FALSE(fork_dirty(edgeid));\n }\n }\n ASSERT_NE(philosopherset[edge.source().id()].state, (int)THINKING);\n }\n }\n foreach(local_edge_type edge, lvertex.out_edges()) {\n size_t edgeid = edge.id();\n if (fork_owner(edgeid) == OWNER_TARGET) {\n if (philosopherset[edge.target().id()].state != EATING) {\n if (fork_dirty(edgeid)) {\n std::cout << (int)(forkset[edgeid]) << \" \"\n << (int)philosopherset[edge.source().id()].state\n << \"->\"\n << (int)philosopherset[edge.target().id()].state\n << std::endl;\n ASSERT_FALSE(fork_dirty(edgeid));\n }\n }\n ASSERT_NE(philosopherset[edge.target().id()].state, (int)THINKING);\n }\n }\n\n }\n else if (philosopherset[v].state == EATING) {\n ASSERT_EQ(philosopherset[v].forks_acquired, philosopherset[v].num_edges);\n }\n }\n }\n};\n\n}\n\n#include <graphlab/macros_undef.hpp>\n\n#endif\n" }, { "path": "src/graphlab/engine/engine_includes.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n\n#include <graphlab/engine/iengine.hpp>\n#include <graphlab/engine/synchronous_engine.hpp>\n#include <graphlab/engine/async_consistent_engine.hpp>\n#include <graphlab/engine/omni_engine.hpp>\n\n#include <graphlab/engine/execution_status.hpp>\n\n//#include <graphlab/engine/asynchronous_engine.hpp>\n//#include <graphlab/engine/engine_factory.hpp>\n//#include <graphlab/engine/engine_options.hpp>\n\n\n" }, { "path": "src/graphlab/engine/execution_status.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_EXECUTION_STATUS_HPP\n#define GRAPHLAB_EXECUTION_STATUS_HPP\n\nnamespace graphlab {\n\n /**\n * \\brief the reasons for execution completion.\n *\n * Because there are several reasons why the graphlab engine might\n * terminate the exec_status value is returned from the start\n * function after completing execution. \n *\n */\n struct execution_status {\n enum status_enum {\n UNSET, /** The default termination reason */ \n RUNNING, /** The engine is currently running */\n TASK_DEPLETION, /**<Execution completed successfully due to\n task depletion */ \n TIMEOUT, /**< The execution completed after timing\n out */\n \n FORCED_ABORT, /**< the engine was stopped by calling force\n abort */\n \n EXCEPTION /**< the engine was stopped by an exception */\n }; // end of enum\n \n // Convenience function.\n static std::string to_string(status_enum es) {\n switch(es) {\n case UNSET: return \"engine not run!\";\n case RUNNING: return \"engine is still running!\"; \n case TASK_DEPLETION: return \"task depletion (natural)\";\n case TIMEOUT: return \"timeout\";\n case FORCED_ABORT: return \"forced abort\";\n case EXCEPTION: return \"exception\";\n default: return \"unknown\";\n };\n } // end of to_string\n };\n\n\n\n}; // end of namespace graphlab\n#endif\n\n\n\n" }, { "path": "src/graphlab/engine/iengine.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n/**\n * Also contains code that is Copyright 2011 Yahoo! Inc. All rights\n * reserved. \n *\n * Contributed under the iCLA for:\n * Joseph Gonzalez (jegonzal@yahoo-inc.com) \n *\n */\n\n#ifndef GRAPHLAB_IENGINE_HPP\n#define GRAPHLAB_IENGINE_HPP\n\n#include <boost/bind.hpp>\n#include <boost/functional.hpp>\n\n#include <graphlab/vertex_program/icontext.hpp>\n#include <graphlab/engine/execution_status.hpp>\n#include <graphlab/options/graphlab_options.hpp>\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/aggregation/distributed_aggregator.hpp>\n#include <graphlab/vertex_program/op_plus_eq_concept.hpp>\n#include <graphlab/graph/vertex_set.hpp>\n\n\n#if defined(__cplusplus) && __cplusplus >= 201103L\n// for whatever reason boost concept is broken under C++11. \n// Temporary workaround. TOFIX\n#undef BOOST_CONCEPT_ASSERT\n#define BOOST_CONCEPT_ASSERT(unused)\n#endif\n\nnamespace graphlab {\n \n\n /**\n * \\ingroup engine\n *\n * \\brief The abstract interface of a GraphLab engine. \n * \n * A GraphLab engine is responsible for executing vertex programs in\n * parallel on one or more machines. GraphLab has a collection of\n * different engines with different guarantees on how\n * vertex-programs are executed. However each engine must implement\n * the iengine interface to allow them to be used \"interchangeably.\"\n *\n * In addition to executing vertex programs GraphLab engines also\n * expose a synchronous aggregation framework. This allows users to\n * attach \"map-reduce\" style jobs that are run periodically on all\n * edges or vertices while GraphLab programs are actively running.\n *\n * Example Usage\n * =================\n *\n * One can use the iengine interface to select between different\n * engines at runtime:\n *\n * \\code\n * iengine<pagerank>* engine_ptr = NULL;\n * if(cmdline_arg == \"synchronous\") {\n * engine_ptr = new synchronous_engine<pagerank>(dc, graph, cmdopts); \n * } else {\n * engine_ptr = new async_consistent_engine<pagerank>(dc, graph, cmdopts); \n * }\n * // Attach an aggregator\n * engine_ptr->add_edge_aggregator<float>(\"edge_map\", \n * edge_map_fun, finalize_fun);\n * // Make it run every 3 seconds\n * engine_ptr->aggregate_periodic(\"edge_map\");\n * // Signal all vertices\n * engine_ptr->signal_all();\n * // Run the engine\n * engine_ptr->start();\n * // do something interesting\n * delete engine_ptr; engine_ptr = NULL;\n * \\endcode \n *\n * @tparam VertexProgram The user defined vertex program which should extend the\n * \\ref ivertex_program interface.\n */\n template<typename VertexProgram>\n class iengine {\n public:\n /**\n * \\brief The user defined vertex program type which should extend\n * ivertex_program.\n */\n typedef VertexProgram vertex_program_type;\n\n /**\n * \\cond GRAPHLAB_INTERNAL\n * \\brief GraphLab Requires that vertex programs be default\n * constructible.\n *\n * \\code\n * class vertex_program {\n * public:\n * vertex_program() { }\n * }; \n * \\endcode\n */\n BOOST_CONCEPT_ASSERT((boost::DefaultConstructible<vertex_program_type>));\n /// \\endcond\n\n\n\n /**\n * \\cond GRAPHLAB_INTERNAL\n *\n * \\brief GraphLab requires that the vertex programx type be\n * Serializable. See \\ref sec_serializable for detials.\n */\n BOOST_CONCEPT_ASSERT((graphlab::Serializable<vertex_program_type>));\n /// \\endcond\n\n\n /**\n * \\brief The user defined message type which is defined in\n * ivertex_program::message_type. \n *\n */\n typedef typename vertex_program_type::message_type message_type;\n\n /**\n * \\brief The graph type which is defined in\n * ivertex_program::graph_type and will typically be\n * \\ref distributed_graph.\n */\n typedef typename vertex_program_type::graph_type graph_type;\n\n /**\n * \\brief The vertex identifier type defined in \n * \\ref graphlab::vertex_id_type.\n */\n typedef typename graph_type::vertex_id_type vertex_id_type; \n\n /**\n * \\brief the vertex object type which contains a reference to the\n * vertex data and is defined in the iengine::graph_type \n * (see for example \\ref distributed_graph::vertex_type).\n */\n typedef typename graph_type::vertex_type vertex_type;\n\n /**\n * \\brief the edge object type which contains a reference to the\n * edge data and is defined in the iengine::graph_type (see for\n * example \\ref distributed_graph::edge_type).\n */\n typedef typename graph_type::edge_type edge_type;\n\n /**\n * \\brief The context type which is passed into vertex programs as\n * a callback to the engine. \n *\n * Most engines use the \\ref graphlab::context implementation.\n */\n typedef typename vertex_program_type::icontext_type icontext_type;\n\n /**\n * \\brief The type of the distributed aggregator used by each engine to\n * implement distributed aggregation.\n */ \n typedef distributed_aggregator<graph_type, icontext_type> aggregator_type;\n\n\n /**\n * \\internal\n * \\brief Virtual destructor required for inheritance\n */ \n virtual ~iengine() {};\n \n /**\n * \\brief Start the engine execution.\n *\n * Behavior details depend on the engine implementation. See the\n * implementation documentation for specifics.\n * \n * @return the reason for termination\n */\n virtual execution_status::status_enum start() = 0;\n \n /**\n * \\brief Compute the total number of updates (calls to apply)\n * executed since start was last invoked.\n *\n * \\return Total number of updates\n */\n virtual size_t num_updates() const = 0;\n\n /**\n * \\brief Get the elapsed time in seconds since start was last\n * called.\n * \n * \\return elapsed time in seconds\n */\n virtual float elapsed_seconds() const = 0;\n\n /**\n * \\brief get the current iteration number. This is not defined\n * for all engines in which case -1 is returned.\n *\n * \\return the current iteration or -1 if not supported.\n */\n virtual int iteration() const { return -1; }\n\n \n /**\n * \\brief Signals single a vertex with an optional message.\n * \n * This function sends a message to particular vertex which will\n * receive that message on start. The signal function must be\n * invoked on all machines simultaneously. For example:\n *\n * \\code\n * graphlab::synchronous_engine<vprog> engine(dc, graph, opts);\n * engine.signal(0); // signal vertex zero\n * \\endcode\n *\n * and _not_:\n *\n * \\code\n * graphlab::synchronous_engine<vprog> engine(dc, graph, opts);\n * if(dc.procid() == 0) engine.signal(0); // signal vertex zero\n * \\endcode\n *\n * Since signal is executed synchronously on all machines it\n * should only be used to schedule a small set of vertices. The\n * preferred method to signal a large set of vertices (e.g., all\n * vertices that are a certain type) is to use either the vertex\n * program init function or the aggregation framework. For\n * example to signal all vertices that have a particular value one\n * could write:\n *\n * \\code\n * struct bipartite_opt : \n * public graphlab::ivertex_program<graph_type, gather_type> {\n * // The user defined init function\n * void init(icontext_type& context, vertex_type& vertex) {\n * // Signal myself if I am a certain type\n * if(vertex.data().on_left) context.signal(vertex);\n * }\n * // other vastly more interesting code\n * };\n * \\endcode\n *\n * @param [in] vid the vertex id to signal\n * @param [in] message the message to send to that vertex. The\n * default message is sent if no message is provided. \n * (See ivertex_program::message_type for details about the\n * message_type). \n */\n virtual void signal(vertex_id_type vertex,\n const message_type& message = message_type()) = 0;\n \n /**\n * \\brief Signal all vertices with a particular message.\n * \n * This function sends the same message to all vertices which will\n * receive that message on start. The signal_all function must be\n * invoked on all machines simultaneously. For example:\n *\n * \\code\n * graphlab::synchronous_engine<vprog> engine(dc, graph, opts);\n * engine.signal_all(); // signal all vertices\n * \\endcode\n *\n * and _not_:\n *\n * \\code\n * graphlab::synchronous_engine<vprog> engine(dc, graph, opts);\n * if(dc.procid() == 0) engine.signal_all(); // signal vertex zero\n * \\endcode\n *\n * The signal_all function is the most common way to send messages\n * to the engine. For example in the pagerank application we want\n * all vertices to be active on the first round. Therefore we\n * would write:\n *\n * \\code\n * graphlab::synchronous_engine<pagerank> engine(dc, graph, opts);\n * engine.signal_all();\n * engine.start();\n * \\endcode\n *\n * @param [in] message the message to send to all vertices. The\n * default message is sent if no message is provided\n * (See ivertex_program::message_type for details about the\n * message_type). \n */\n virtual void signal_all(const message_type& message = message_type(),\n const std::string& order = \"shuffle\") = 0;\n \n /**\n * \\brief Signal a set of vertices with a particular message.\n * \n * This function sends the same message to a set of vertices which will\n * receive that message on start. The signal_vset function must be\n * invoked on all machines simultaneously. For example:\n *\n * \\code\n * graphlab::synchronous_engine<vprog> engine(dc, graph, opts);\n * engine.signal_vset(vset); // signal a subset of vertices\n * \\endcode\n *\n * signal_all() is conceptually equivalent to:\n *\n * \\code\n * engine.signal_vset(graph.complete_set());\n * \\endcode\n *\n * @param [in] vset The set of vertices to signal \n * @param [in] message the message to send to all vertices. The\n * default message is sent if no message is provided\n * (See ivertex_program::message_type for details about the\n * message_type). \n */\n virtual void signal_vset(const vertex_set& vset,\n const message_type& message = message_type(),\n const std::string& order = \"shuffle\") = 0;\n\n\n /** \n * \\brief Creates a vertex aggregator. Returns true on success.\n * Returns false if an aggregator of the same name already\n * exists.\n *\n * Creates a vertex aggregator associated to a particular key.\n * The map_function is called over every vertex in the graph, and the\n * return value of the map is summed. The finalize_function is then called\n * on the result of the reduction. The finalize_function is called on\n * all machines. The map_function should only read the graph data,\n * and should not make any modifications.\n *\n * ### Basic Usage \n * For instance, if the graph has float vertex data, and float edge data:\n * \\code\n * typedef graphlab::distributed_graph<float, float> graph_type;\n * \\endcode\n *\n * An aggregator can be constructed to compute the absolute sum of all the\n * vertex data. To do this, we define two functions.\n * \\code\n * float absolute_vertex_data(engine_type::icontext_type& context,\n * graph_type::vertex_type vertex) {\n * return std::fabs(vertex.data());\n * }\n *\n * void print_finalize(engine_type::icontext_type& context, \n * float total) {\n * std::cout << total << \"\\n\";\n * }\n * \\endcode\n * \n * Next, we define the aggregator in the engine by calling \n * add_vertex_aggregator(). We must assign it a unique\n * name which will be used to reference this particular aggregate\n * operation. We shall call it \"absolute_vertex_sum\".\n * \\code\n * engine.add_vertex_aggregator<float>(\"absolute_vertex_sum\",\n * absolute_vertex_data, \n * print_finalize);\n * \\endcode\n *\n * When executed, the engine execute <code>absolute_vertex_data()</code>\n * on each vertex in the graph. <code>absolute_vertex_data()</code> \n * reads the vertex data, and returns its absolute value. All return \n * values are then summing them together using the float's += operator.\n * The final result is than passed to the <code>print_finalize</code>\n * function. The template argument <code><float></code> is necessary to\n * provide information about the return type of\n * <code>absolute_vertex_data</code>.\n * \n *\n * This aggregator can be run immediately by calling \n * aggregate_now() with the name of the aggregator.\n * \\code\n * engine.aggregate_now(\"absolute_vertex_sum\");\n * \\endcode\n *\n * Or can be arranged to run periodically together with the engine \n * execution (in this example, every 1.5 seconds).\n * \\code\n * engine.aggregate_periodic(\"absolute_vertex_sum\", 1.5);\n * \\endcode\n * \n * Note that since finalize is called on <b>all machines</b>, multiple\n * copies of the total will be printed. If only one copy is desired,\n * see \\ref graphlab::icontext::cout() \"context.cout()\" or to get \n * the actual process ID using \n * \\ref graphlab::icontext::procid() \"context.procid()\"\n *\n * In practice, the reduction type can be any arbitrary user-defined type\n * as long as a += operator is defined. This permits great flexibility\n * in the type of operations the aggregator can perform.\n *\n * ### Details\n * The add_vertex_aggregator() function is also templatized over both\n * function types and there is no strong enforcement of the exact argument\n * types of the map function and the reduce function. For instance, in the\n * above example, the following print_finalize() variants may also be\n * accepted.\n *\n * \\code\n * void print_finalize(engine_type::icontext_type& context, double total) {\n * std::cout << total << \"\\n\";\n * }\n *\n * void print_finalize(engine_type::icontext_type& context, float& total) {\n * std::cout << total << \"\\n\";\n * }\n *\n * void print_finalize(engine_type::icontext_type& context, const float& total) {\n * std::cout << total << \"\\n\";\n * }\n * \\endcode\n * In particlar, the last variation may be useful for performance reasons\n * if the reduction type is large.\n *\n * ### Distributed Behavior\n * To obtain consistent distributed behavior in the distributed setting,\n * we designed the aggregator to minimize the amount of asymmetry among \n * the machines. In particular, the finalize operation is guaranteed to be\n * called on all machines. This therefore permits global variables to be\n * modified on finalize since all machines are ensured to be eventually\n * consistent. \n *\n * For instance, in the above example, print_finalize could\n * store the result in a global variable:\n * \\code\n * void print_finalize(engine_type::icontext_type& context, float total) {\n * GLOBAL_TOTAL = total;\n * }\n * \\endcode \n * which will make it accessible to all other running update functions.\n *\n * \\tparam ReductionType The output of the map function. Must have\n * operator+= defined, and must be \\ref sec_serializable.\n * \\tparam VertexMapperType The type of the map function. \n * Not generally needed.\n * Can be inferred by the compiler.\n * \\tparam FinalizerType The type of the finalize function. \n * Not generally needed.\n * Can be inferred by the compiler.\n *\n * \\param [in] map_function The Map function to use. Must take an\n * \\param [in] key The name of this aggregator. Must be unique.\n * \\ref icontext_type& as its first argument, and\n * a \\ref vertex_type, or a reference to a \n * \\ref vertex_type as its second argument.\n * Returns a ReductionType which must be summable\n * and \\ref sec_serializable .\n * \\param [in] finalize_function The Finalize function to use. Must take\n * an \\ref icontext_type& as its first\n * argument and a ReductionType, or a\n * reference to a ReductionType as its second\n * argument.\n */\n template <typename ReductionType,\n typename VertexMapType,\n typename FinalizerType>\n bool add_vertex_aggregator(const std::string& key,\n VertexMapType map_function,\n FinalizerType finalize_function) {\n BOOST_CONCEPT_ASSERT((graphlab::Serializable<ReductionType>));\n BOOST_CONCEPT_ASSERT((graphlab::OpPlusEq<ReductionType>));\n\n aggregator_type* aggregator = get_aggregator();\n if(aggregator == NULL) {\n logstream(LOG_FATAL) << \"Aggregation not supported by this engine!\" \n << std::endl;\n return false; // does not return\n }\n return aggregator->template add_vertex_aggregator<ReductionType>(key, map_function, \n finalize_function);\n } // end of add vertex aggregator\n\n#if defined(__cplusplus) && __cplusplus >= 201103L\n /**\n * \\brief An overload of add_vertex_aggregator for C++11 which does not\n * require the user to provide the reduction type.\n *\n * This function is available only if the compiler has C++11 support.\n * Specifically, it uses C++11's decltype operation to infer the\n * reduction type, thus eliminating the need for the function\n * call to be templatized over the reduction type. For instance,\n * in the add_vertex_aggregator() example, it allows the following\n * code to be written:\n * \\code\n * engine.add_vertex_aggregator(\"absolute_vertex_sum\",\n * absolute_vertex_data, \n * print_finalize);\n * \\endcode\n *\n * \\tparam VertexMapperType The type of the map function. \n * Not generally needed.\n * Can be inferred by the compiler.\n * \\tparam FinalizerType The type of the finalize function. \n * Not generally needed.\n * Can be inferred by the compiler.\n *\n * \\param [in] key The name of this aggregator. Must be unique.\n * \\param [in] map_function The Map function to use. Must take an\n * \\ref icontext_type& as its first argument, and\n * a \\ref vertex_type, or a reference to a \n * \\ref vertex_type as its second argument.\n * Returns a ReductionType which must be summable\n * and \\ref sec_serializable .\n * \\param [in] finalize_function The Finalize function to use. Must take\n * an \\ref icontext_type& as its first\n * argument and a ReductionType, or a\n * reference to a ReductionType as its second\n * argument.\n */\n template <typename VertexMapType,\n typename FinalizerType>\n bool add_vertex_aggregator(const std::string& key,\n VertexMapType map_function,\n FinalizerType finalize_function) {\n aggregator_type* aggregator = get_aggregator();\n if(aggregator == NULL) {\n logstream(LOG_FATAL) << \"Aggregation not supported by this engine!\" \n << std::endl;\n return false; // does not return\n }\n return aggregator->add_vertex_aggregator(key, map_function, \n finalize_function);\n } // end of add vertex aggregator\n\n#endif\n \n\n /** \n * \\brief Creates an edge aggregator. Returns true on success.\n * Returns false if an aggregator of the same name already\n * exists.\n *\n * Creates a edge aggregator associated to a particular key.\n * The map_function is called over every edge in the graph, and the\n * return value of the map is summed. The finalize_function is then called\n * on the result of the reduction. The finalize_function is called on\n * all machines. The map_function should only read the graph data,\n * and should not make any modifications.\n\n *\n * ### Basic Usage \n * For instance, if the graph has float vertex data, and float edge data:\n * \\code\n * typedef graphlab::distributed_graph<float, float> graph_type;\n * \\endcode\n *\n * An aggregator can be constructed to compute the absolute sum of all the\n * edge data. To do this, we define two functions.\n * \\code\n * float absolute_edge_data(engine_type::icontext_type& context,\n * graph_type::edge_type edge) {\n * return std::fabs(edge.data());\n * }\n *\n * void print_finalize(engine_type::icontext_type& context, float total) {\n * std::cout << total << \"\\n\";\n * }\n * \\endcode\n * \n * Next, we define the aggregator in the engine by calling \n * add_edge_aggregator(). We must assign it a unique\n * name which will be used to reference this particular aggregate\n * operation. We shall call it \"absolute_edge_sum\".\n * \\code\n * engine.add_edge_aggregator<float>(\"absolute_edge_sum\",\n * absolute_edge_data, \n * print_finalize);\n * \\endcode\n *\n *\n * When executed, the engine execute <code>absolute_edge_data()</code>\n * on each edge in the graph. <code>absolute_edge_data()</code> \n * reads the edge data, and returns its absolute value. All return \n * values are then summing them together using the float's += operator.\n * The final result is than passed to the <code>print_finalize</code>\n * function. The template argument <code><float></code> is necessary to\n * provide information about the return type of\n * <code>absolute_edge_data</code>.\n * \n *\n * This aggregator can be run immediately by calling \n * aggregate_now() with the name of the aggregator.\n * \\code\n * engine.aggregate_now(\"absolute_edge_sum\");\n * \\endcode\n *\n * Or can be arranged to run periodically together with the engine \n * execution (in this example, every 1.5 seconds).\n * \\code\n * engine.aggregate_periodic(\"absolute_edge_sum\", 1.5);\n * \\endcode\n * \n * Note that since finalize is called on <b>all machines</b>, multiple\n * copies of the total will be printed. If only one copy is desired,\n * see \\ref graphlab::icontext::cout() \"context.cout()\" or to get \n * the actual process ID using \n * \\ref graphlab::icontext::procid() \"context.procid()\"\n *\n * ### Details\n * The add_edge_aggregator() function is also templatized over both\n * function types and there is no strong enforcement of the exact argument\n * types of the map function and the reduce function. For instance, in the\n * above example, the following print_finalize() variants may also be\n * accepted.\n *\n * \\code\n * void print_finalize(engine_type::icontext_type& context, double total) {\n * std::cout << total << \"\\n\";\n * }\n *\n * void print_finalize(engine_type::icontext_type& context, float& total) {\n * std::cout << total << \"\\n\";\n * }\n *\n * void print_finalize(engine_type::icontext_type& context, const float& total) {\n * std::cout << total << \"\\n\";\n * }\n * \\endcode\n * In particlar, the last variation may be useful for performance reasons\n * if the reduction type is large.\n *\n * ### Distributed Behavior\n * To obtain consistent distributed behavior in the distributed setting,\n * we designed the aggregator to minimize the amount of asymmetry among \n * the machines. In particular, the finalize operation is guaranteed to be\n * called on all machines. This therefore permits global variables to be\n * modified on finalize since all machines are ensured to be eventually\n * consistent. \n *\n * For instance, in the above example, print_finalize could\n * store the result in a global variable:\n * \\code\n * void print_finalize(engine_type::icontext_type& context, float total) {\n * GLOBAL_TOTAL = total;\n * }\n * \\endcode \n * which will make it accessible to all other running update functions.\n *\n * \\tparam ReductionType The output of the map function. Must have\n * operator+= defined, and must be \\ref sec_serializable.\n * \\tparam EdgeMapperType The type of the map function. \n * Not generally needed.\n * Can be inferred by the compiler.\n * \\tparam FinalizerType The type of the finalize function. \n * Not generally needed.\n * Can be inferred by the compiler.\n *\n * \\param [in] key The name of this aggregator. Must be unique.\n * \\param [in] map_function The Map function to use. Must take an\n * \\ref icontext_type& as its first argument, and\n * a \\ref edge_type, or a reference to a \n * \\ref edge_type as its second argument.\n * Returns a ReductionType which must be summable\n * and \\ref sec_serializable .\n * \\param [in] finalize_function The Finalize function to use. Must take\n * an \\ref icontext_type& as its first\n * argument and a ReductionType, or a\n * reference to a ReductionType as its second\n * argument.\n */\n \n template <typename ReductionType,\n typename EdgeMapType,\n typename FinalizerType>\n bool add_edge_aggregator(const std::string& key,\n EdgeMapType map_function,\n FinalizerType finalize_function) {\n BOOST_CONCEPT_ASSERT((graphlab::Serializable<ReductionType>));\n BOOST_CONCEPT_ASSERT((graphlab::OpPlusEq<ReductionType>));\n aggregator_type* aggregator = get_aggregator();\n if(aggregator == NULL) {\n logstream(LOG_FATAL) << \"Aggregation not supported by this engine!\"\n << std::endl;\n return false; // does not return\n }\n return aggregator->template add_edge_aggregator<ReductionType>\n (key, map_function, finalize_function);\n } // end of add edge aggregator\n\n\n#if defined(__cplusplus) && __cplusplus >= 201103L\n\n /**\n * \\brief An overload of add_edge_aggregator for C++11 which does not\n * require the user to provide the reduction type.\n *\n * This function is available only if the compiler has C++11 support.\n * Specifically, it uses C++11's decltype operation to infer the\n * reduction type, thus eliminating the need for the function\n * call to be templatized over the reduction type. For instance,\n * in the add_edge_aggregator() example, it allows the following\n * code to be written:\n * \\code\n * engine.add_edge_aggregator(\"absolute_edge_sum\",\n * absolute_edge_data, \n * print_finalize);\n * \\endcode\n *\n * \\tparam EdgeMapperType The type of the map function. \n * Not generally needed.\n * Can be inferred by the compiler.\n * \\tparam FinalizerType The type of the finalize function. \n * Not generally needed.\n * Can be inferred by the compiler.\n *\n * \\param [in] key The name of this aggregator. Must be unique.\n * \\param [in] map_function The Map function to use. Must take an\n * \\ref icontext_type& as its first argument, and\n * a \\ref vertex_type, or a reference to a \n * \\ref vertex_type as its second argument.\n * Returns a ReductionType which must be summable\n * and \\ref sec_serializable .\n * \\param [in] finalize_function The Finalize function to use. Must take\n * an \\ref icontext_type& as its first\n * argument and a ReductionType, or a\n * reference to a ReductionType as its second\n * argument.\n */\n template <typename EdgeMapType,\n typename FinalizerType>\n bool add_edge_aggregator(const std::string& key,\n EdgeMapType map_function,\n FinalizerType finalize_function) {\n aggregator_type* aggregator = get_aggregator();\n if(aggregator == NULL) {\n logstream(LOG_FATAL) << \"Aggregation not supported by this engine!\" \n << std::endl;\n return false; // does not return\n }\n return aggregator->add_edge_aggregator(key, map_function, \n finalize_function);\n } // end of add edge aggregator\n#endif\n\n /**\n * \\brief Performs an immediate aggregation on a key\n *\n * Performs an immediate aggregation on a key. All machines must\n * call this simultaneously. If the key is not found,\n * false is returned. Otherwise returns true on success.\n *\n * For instance, the following code will run the aggregator\n * with the name \"absolute_vertex_sum\" immediately.\n * \\code\n * engine.aggregate_now(\"absolute_vertex_sum\");\n * \\endcode\n *\n * \\param[in] key Key to aggregate now. Must be a key\n * previously created by add_vertex_aggregator()\n * or add_edge_aggregator().\n * \\return False if key not found, True on success.\n */\n bool aggregate_now(const std::string& key) {\n aggregator_type* aggregator = get_aggregator();\n if(aggregator == NULL) {\n logstream(LOG_FATAL) << \"Aggregation not supported by this engine!\" \n << std::endl;\n return false; // does not return\n }\n return aggregator->aggregate_now(key);\n } // end of aggregate_now\n\n\n /**\n * \\brief Performs a map-reduce operation on each vertex in the \n * graph returning the result.\n * \n * Given a map function, map_reduce_vertices() call the map function on all\n * vertices in the graph. The return values are then summed together and the\n * final result returned. The map function should only read the vertex data\n * and should not make any modifications. map_reduce_vertices() must be\n * called on all machines simultaneously.\n *\n * ### Basic Usage \n * For instance, if the graph has float vertex data, and float edge data:\n * \\code\n * typedef graphlab::distributed_graph<float, float> graph_type;\n * \\endcode\n * \n * To compute an absolute sum over all the vertex data, we would write\n * a function which reads in each a vertex, and returns the absolute\n * value of the data on the vertex.\n * \\code\n * float absolute_vertex_data(engine_type::icontext_type& context,\n * graph_type::vertex_type vertex) {\n * return std::fabs(vertex.data());\n * }\n * \\endcode\n * After which calling:\n * \\code\n * float sum = engine.map_reduce_vertices<float>(absolute_vertex_data);\n * \\endcode\n * will call the <code>absolute_vertex_data()</code> function\n * on each vertex in the graph. <code>absolute_vertex_data()</code>\n * reads the value of the vertex and returns the absolute result.\n * This return values are then summed together and returned. \n * All machines see the same result.\n *\n * The template argument <code><float></code> is needed to inform\n * the compiler regarding the return type of the mapfunction.\n *\n * ### Signalling\n * Another common use for the map_reduce_vertices() function is \n * in signalling. Since the map function is passed a context, it\n * can be used to perform signalling of vertices for execution\n * during a later \\ref start() \"engine.start()\" call.\n *\n * For instance, the following code will signal all vertices\n * with value >= 1\n * \\code\n * graphlab::empty signal_vertices(engine_type::icontext_type& context,\n * graph_type::vertex_type vertex) {\n * if (vertex.data() >= 1) context.signal(vertex);\n * return graphlab::empty()\n * }\n * \\endcode\n * Note that in this case, we are not interested in a reduction\n * operation, and thus we return a graphlab::empty object.\n * Calling:\n * \\code\n * engine.map_reduce_vertices<graphlab::empty>(signal_vertices);\n * \\endcode\n * will run <code>signal_vertices()</code> on all vertices,\n * signalling all vertices with value <= 1\n *\n * ### Relations\n * The map function has the same structure as that in\n * add_vertex_aggregator() and may be reused in an aggregator.\n * This function is also very similar to \n * graphlab::distributed_graph::map_reduce_vertices()\n * with the difference that this takes a context and thus\n * can be used to perform signalling.\n * Finally transform_vertices() can be used to perform a similar\n * but may also make modifications to graph data.\n *\n * \\tparam ReductionType The output of the map function. Must have\n * operator+= defined, and must be \\ref sec_serializable.\n * \\tparam VertexMapperType The type of the map function. \n * Not generally needed.\n * Can be inferred by the compiler.\n * \\param mapfunction The map function to use. Must take an\n * \\ref icontext_type& as its first argument, and\n * a \\ref vertex_type, or a reference to a \n * \\ref vertex_type as its second argument.\n * Returns a ReductionType which must be summable\n * and \\ref sec_serializable .\n */\n template <typename ReductionType, typename VertexMapperType>\n ReductionType map_reduce_vertices(VertexMapperType mapfunction) {\n aggregator_type* aggregator = get_aggregator();\n BOOST_CONCEPT_ASSERT((graphlab::Serializable<ReductionType>));\n BOOST_CONCEPT_ASSERT((graphlab::OpPlusEq<ReductionType>));\n\n if(aggregator == NULL) {\n logstream(LOG_FATAL) << \"Aggregation not supported by this engine!\"\n << std::endl;\n return ReductionType(); // does not return\n }\n return aggregator->template map_reduce_vertices<ReductionType>(mapfunction); \n }\n\n /**\n * \\brief Performs a map-reduce operation on each edge in the \n * graph returning the result.\n * \n * Given a map function, map_reduce_edges() call the map function on all\n * edges in the graph. The return values are then summed together and the\n * final result returned. The map function should only read data\n * and should not make any modifications. map_reduce_edges() must be\n * called on all machines simultaneously.\n *\n * ### Basic Usage \n * For instance, if the graph has float vertex data, and float edge data:\n * \\code\n * typedef graphlab::distributed_graph<float, float> graph_type;\n * \\endcode\n * \n * To compute an absolute sum over all the edge data, we would write\n * a function which reads in each a edge, and returns the absolute\n * value of the data on the edge.\n * \\code\n * float absolute_edge_data(engine_type::icontext_type& context,\n * graph_type::edge_type edge) {\n * return std::fabs(edge.data());\n * }\n * \\endcode\n * After which calling:\n * \\code\n * float sum = engine.map_reduce_edges<float>(absolute_edge_data);\n * \\endcode\n * will call the <code>absolute_edge_data()</code> function\n * on each edge in the graph. <code>absolute_edge_data()</code>\n * reads the value of the edge and returns the absolute result.\n * This return values are then summed together and returned. \n * All machines see the same result.\n *\n * The template argument <code><float></code> is needed to inform\n * the compiler regarding the return type of the mapfunction.\n *\n * ### Signalling\n * Another common use for the map_reduce_edges() function is \n * in signalling. Since the map function is passed a context, it\n * can be used to perform signalling of edges for execution\n * during a later \\ref start() \"engine.start()\" call.\n *\n * For instance, the following code will signal the source\n * vertex of each edge.\n * \\code\n * graphlab::empty signal_source(engine_type::icontext_type& context,\n * graph_type::edge_type edge) {\n * context.signal(edge.source());\n * return graphlab::empty()\n * }\n * \\endcode\n * Note that in this case, we are not interested in a reduction\n * operation, and thus we return a graphlab::empty object.\n * Calling:\n * \\code\n * engine.map_reduce_edges<graphlab::empty>(signal_source);\n * \\endcode\n * will run <code>signal_source()</code> on all edges,\n * signalling all source vertices.\n *\n * ### Relations\n * The map function has the same structure as that in\n * add_edge_aggregator() and may be reused in an aggregator.\n * This function is also very similar to \n * graphlab::distributed_graph::map_reduce_edges()\n * with the difference that this takes a context and thus\n * can be used to perform signalling.\n * Finally transform_edges() can be used to perform a similar\n * but may also make modifications to graph data.\n *\n * \\tparam ReductionType The output of the map function. Must have\n * operator+= defined, and must be \\ref sec_serializable.\n * \\tparam EdgeMapperType The type of the map function. \n * Not generally needed.\n * Can be inferred by the compiler.\n * \\param mapfunction The map function to use. Must take an\n * \\ref icontext_type& as its first argument, and\n * a \\ref edge_type, or a reference to a \n * \\ref edge_type as its second argument.\n * Returns a ReductionType which must be summable\n * and \\ref sec_serializable .\n */\n template <typename ReductionType, typename EdgeMapperType>\n ReductionType map_reduce_edges(EdgeMapperType mapfunction) {\n aggregator_type* aggregator = get_aggregator();\n BOOST_CONCEPT_ASSERT((graphlab::Serializable<ReductionType>));\n BOOST_CONCEPT_ASSERT((graphlab::OpPlusEq<ReductionType>));\n if(aggregator == NULL) {\n logstream(LOG_FATAL) << \"Aggregation not supported by this engine!\" \n << std::endl;\n return ReductionType(); // does not return\n }\n return aggregator->template map_reduce_edges<ReductionType>(mapfunction); \n }\n \n \n /**\n * \\brief Performs a transformation operation on each vertex in the graph.\n *\n * Given a mapfunction, transform_vertices() calls mapfunction on \n * every vertex in graph. The map function may make modifications\n * to the data on the vertex. transform_vertices() must be called by all\n * machines simultaneously.\n *\n * ### Basic Usage \n * For instance, if the graph has integer vertex data, and integer edge\n * data: \n * \\code\n * typedef graphlab::distributed_graph<size_t, size_t> graph_type;\n * \\endcode\n * \n * To set each vertex value to be the number of out-going edges,\n * we may write the following function: \n * \\code\n * void set_vertex_value(engine_type::icontext_type& context,\n * graph_type::vertex_type vertex) {\n * vertex.data() = vertex.num_out_edges();\n * }\n * \\endcode\n *\n * Calling transform_vertices():\n * \\code\n * engine.transform_vertices(set_vertex_value);\n * \\endcode\n * will run the <code>set_vertex_value()</code> function\n * on each vertex in the graph, setting its new value. \n *\n * ### Signalling\n * Since the mapfunction is provided with a context, the mapfunction\n * can also be used to perform signalling. For instance, the \n * <code>set_vertex_value</code> function above may be modified to set \n * the value of the vertex, but to also signal the vertex if\n * it has more than 5 outgoing edges.\n *\n * \\code\n * void set_vertex_value(engine_type::icontext_type& context,\n * graph_type::vertex_type vertex) {\n * vertex.data() = vertex.num_out_edges();\n * if (vertex.num_out_edges() > 5) context.signal(vertex);\n * }\n * \\endcode\n *\n * However, if the purpose of the function is to only signal\n * without making modifications, map_reduce_vertices() will be\n * more efficient as this function will additionally perform\n * distributed synchronization of modified data.\n *\n * ### Relations\n * map_reduce_vertices() provide similar signalling functionality, \n * but should not make modifications to graph data. \n * graphlab::distributed_graph::transform_vertices() provide\n * the same graph modification capabilities, but without a context\n * and thus cannot perform signalling.\n *\n * \\tparam VertexMapperType The type of the map function. \n * Not generally needed.\n * Can be inferred by the compiler.\n * \\param mapfunction The map function to use. Must take an\n * \\ref icontext_type& as its first argument, and\n * a \\ref vertex_type, or a reference to a \n * \\ref vertex_type as its second argument.\n * Returns void.\n */ \n template <typename VertexMapperType>\n void transform_vertices(VertexMapperType mapfunction) {\n aggregator_type* aggregator = get_aggregator();\n if(aggregator == NULL) {\n logstream(LOG_FATAL) << \"Aggregation not supported by this engine!\"\n << std::endl;\n return; // does not return\n }\n aggregator->transform_vertices(mapfunction); \n }\n\n /**\n * \\brief Performs a transformation operation on each edge in the graph.\n *\n * Given a mapfunction, transform_edges() calls mapfunction on \n * every edge in graph. The map function may make modifications\n * to the data on the edge. transform_edges() must be called on\n * all machines simultaneously.\n *\n * ### Basic Usage \n * For instance, if the graph has integer vertex data, and integer edge\n * data: \n * \\code\n * typedef graphlab::distributed_graph<size_t, size_t> graph_type;\n * \\endcode\n * \n * To set each edge value to be the number of out-going edges\n * of the target vertex, we may write the following:\n * \\code\n * void set_edge_value(engine_type::icontext_type& context,\n * graph_type::edge_type edge) {\n * edge.data() = edge.target().num_out_edges();\n * }\n * \\endcode\n *\n * Calling transform_edges():\n * \\code\n * engine.transform_edges(set_edge_value);\n * \\endcode\n * will run the <code>set_edge_value()</code> function\n * on each edge in the graph, setting its new value. \n *\n * ### Signalling\n * Since the mapfunction is provided with a context, the mapfunction\n * can also be used to perform signalling. For instance, the \n * <code>set_edge_value</code> function above may be modified to set \n * the value of the edge, but to also signal the target vertex. \n *\n * \\code\n * void set_edge_value(engine_type::icontext_type& context,\n * graph_type::edge_type edge) {\n * edge.data() = edge.target().num_out_edges();\n * context.signal(edge.target());\n * }\n * \\endcode\n *\n * However, if the purpose of the function is to only signal\n * without making modifications, map_reduce_edges() will be\n * more efficient as this function will additionally perform\n * distributed synchronization of modified data.\n *\n * ### Relations\n * map_reduce_edges() provide similar signalling functionality, \n * but should not make modifications to graph data. \n * graphlab::distributed_graph::transform_edges() provide\n * the same graph modification capabilities, but without a context\n * and thus cannot perform signalling.\n *\n * \\tparam EdgeMapperType The type of the map function. \n * Not generally needed.\n * Can be inferred by the compiler.\n * \\param mapfunction The map function to use. Must take an\n * \\ref icontext_type& as its first argument, and\n * a \\ref edge_type, or a reference to a \n * \\ref edge_type as its second argument.\n * Returns void.\n */ \n template <typename EdgeMapperType>\n void transform_edges(EdgeMapperType mapfunction) {\n aggregator_type* aggregator = get_aggregator();\n if(aggregator == NULL) {\n logstream(LOG_FATAL) << \"Aggregation not supported by this engine!\" \n << std::endl;\n return; // does not return\n }\n aggregator->transform_edges(mapfunction); \n }\n \n /**\n * \\brief Requests that a particular aggregation key\n * be recomputed periodically when the engine is running.\n *\n * Requests that the aggregator with a given key be aggregated\n * every certain number of seconds when the engine is running.\n * Note that the period is prescriptive: in practice the actual\n * period will be larger than the requested period. \n * Seconds must be >= 0;\n *\n * For instance, the following code will schedule the aggregator\n * with the name \"absolute_vertex_sum\" to run every 1.5 seconds.\n * \\code\n * engine.aggregate_periodic(\"absolute_vertex_sum\", 1.5);\n * \\endcode\n *\n * \\param [in] key Key to schedule. Must be a key\n * previously created by add_vertex_aggregator()\n * or add_edge_aggregator().\n * \\param [in] seconds How frequently to schedule. Must be >=\n * 0. seconds == 0 will ensure that this key is continously\n * recomputed.\n * \n * All machines must call simultaneously.\n * \\return Returns true if key is found and seconds >= 0,\n * and false otherwise.\n */\n bool aggregate_periodic(const std::string& key, float seconds) {\n aggregator_type* aggregator = get_aggregator();\n if(aggregator == NULL) {\n logstream(LOG_FATAL) << \"Aggregation not supported by this engine!\" \n << std::endl;\n return false; // does not return \n }\n return aggregator->aggregate_periodic(key, seconds);\n } // end of aggregate_periodic\n\n\n\n /**\n * \\cond GRAPHLAB_INTERNAL\n * \\internal\n * \\brief This is used by iengine to get the \n * \\ref distributed_aggregator from the derived class to support\n * the local templated aggregator interface. \n *\n * \\return a pointer to the distributed aggregator for that\n * engine. If no aggregator is available or aggregation is not\n * supported then return NULL.\n */\n virtual aggregator_type* get_aggregator() = 0;\n /// \\endcond\n }; // end of iengine interface\n\n} // end of namespace graphlab\n\n#endif\n\n" }, { "path": "src/graphlab/engine/message_array.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_MESSAGE_ARRAY_HPP\n#define GRAPHLAB_MESSAGE_ARRAY_HPP\n\n\n#include <vector>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/parallel/atomic.hpp>\n#include <graphlab/scheduler/get_message_priority.hpp>\nnamespace graphlab {\n\n /**\n * \\TODO DOCUMENT THIS CLASS\n */ \n \n template<typename ValueType>\n class message_array {\n public:\n typedef ValueType value_type;\n\n private: \n struct message_box {\n value_type value;\n bool empty;\n\n message_box() : empty(true) { }\n /** returns false if element is already present */\n inline bool add(const value_type& other, double& priority) {\n if (empty) {\n value = other;\n empty = false;\n priority = scheduler_impl::get_message_priority(value);\n return true;\n } else {\n value += other;\n priority = scheduler_impl::get_message_priority(value);\n return false;\n }\n }\n \n void clear() {\n value = value_type();\n empty = true;\n }\n \n }; \n\n std::vector<message_box> message_vector;\n // lock array\n simple_spinlock lock_array[65536];\n size_t joincounter[65536];\n size_t addcounter[65536];\n\n /** Not assignable */\n void operator=(const message_array& other) { }\n\n static size_t get_lock_idx(size_t i) {\n return i % 65536;\n }\n public:\n /** Initialize the per vertex task set */\n message_array(size_t num_vertices = 0) :\n message_vector(num_vertices) { \n for (size_t i = 0; i < 65536; ++i) {\n joincounter[i] = 0; \n addcounter[i] = 0;\n }\n }\n\n /**\n * Resizes the number of elements this message vector can hold\n */\n void resize(size_t num_vertices) {\n message_vector.resize(num_vertices);\n }\n\n /** Add a message to the set returning false if a message is already\n present. */\n bool add(const size_t idx, \n const value_type& val,\n double* message_priority = NULL) {\n double priority;\n size_t lockidx = get_lock_idx(idx);\n lock_array[lockidx].lock();\n bool ret = message_vector[idx].add(val, priority);\n joincounter[lockidx] += !ret;\n addcounter[lockidx]++;\n lock_array[lockidx].unlock();\n if (message_priority) (*message_priority) = priority;\n return ret;\n } \n\n /** Returns the current message stored at idx and \n * clears the message.\n * Returns true on success and false if there is no message\n * stored at the index.\n */\n bool get(const size_t idx,\n value_type& ret_val) {\n bool has_val = false;\n size_t lockidx = get_lock_idx(idx);\n lock_array[lockidx].lock();\n if (!message_vector[idx].empty) {\n ret_val = message_vector[idx].value;\n message_vector[idx].clear();\n has_val = true;\n }\n lock_array[lockidx].unlock();\n return has_val;\n }\n\n /** Returns the current message stored at idx. \n * Returns true on success and false if there is no message\n * stored at the index.\n * Does not change the contents of the message\n */\n bool peek(const size_t idx,\n value_type& ret_val) {\n bool has_val = false;\n size_t lockidx = get_lock_idx(idx);\n lock_array[lockidx].lock();\n if (!message_vector[idx].empty) {\n ret_val = message_vector[idx].value;\n has_val = true;\n }\n lock_array[lockidx].unlock();\n return has_val;\n }\n \n\n /// clears the message at a particular idx\n void clear(const size_t idx) { \n size_t lockidx = get_lock_idx(idx);\n lock_array[lockidx].lock();\n message_vector[idx].clear(); \n lock_array[lockidx].unlock();\n }\n\n /// Returns true if the message at position idx is empty\n bool empty(const size_t idx) const {\n return message_vector[idx].empty;\n }\n\n bool empty() const {\n for (size_t i = 0;i < message_vector.size(); ++i) {\n if (!message_vector[i].empty) return false;\n }\n return true;\n }\n\n /// Returns the length of the message vector\n size_t size() const { \n return message_vector.size(); \n }\n \n size_t num_joins() const { \n size_t total_joins = 0;\n for (size_t i = 0; i < 65536; ++i) {\n total_joins += joincounter[i];\n }\n return total_joins;\n }\n\n\n size_t num_adds() const { \n size_t total_adds = 0;\n for (size_t i = 0; i < 65536; ++i) {\n total_adds += addcounter[i];\n }\n return total_adds;\n }\n\n /// not thread safe. Clears all contents\n void clear() {\n for (size_t i = 0; i < message_vector.size(); ++i) clear(i);\n }\n\n \n }; // end of vertex map\n\n}; // end of namespace graphlab\n\n#undef VALUE_PENDING\n\n#endif\n\n" }, { "path": "src/graphlab/engine/omni_engine.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_OMNI_ENGINE_HPP\n#define GRAPHLAB_OMNI_ENGINE_HPP\n\n\n\n#include <graphlab/options/graphlab_options.hpp>\n\n#include <graphlab/engine/iengine.hpp>\n#include <graphlab/engine/synchronous_engine.hpp>\n#include <graphlab/engine/async_consistent_engine.hpp>\n\nnamespace graphlab {\n\n\n /**\n * \\ingroup engines\n * \n * \\brief The omni engine encapsulates all the GraphLab engines\n * allowing the user to select which engine to use at runtime. \n *\n * The actual engine type is set as a string argument to the\n * constructor of the omni_engine. Forexample:\n *\n * \\code\n * std::string exec_model = \"synchronous\";\n * // do something to determine the exec_model (possibly command\n * // line processing)\n * // Create the engine\n * graphlab::omni_engine<pagerank_vprog> engine(dc, graph, opts, exec_model);\n * \\endcode\n *\n * The specific engine type can be overriden by command line\n * arguments (engine_opts=\"type=<type>\"):\n *\n * \\code\n * graphlab::omni_engine<pagerank_vprog> engine(dc, graph, opts, \"synchronous\");\n * \\endcode\n *\n * then calling the progam with the command line options:\n * \n \\verbatim\n %> mpiexec -n 16 ./pagerank --engine_opts=\"type=synchronous\"\n \\endverbatim\n * \n * The currently supproted types are:\n * \n * \\li \"synchronous\" or \"sync\": uses the synchronous engine \n * (\\ref synchronous_engine)\n * \\li \"asynchronous\" or \"async\": uses the asynchronous engine\n * (\\ref async_consistent_engine)\n*\n * \\see graphlab::synchronous_engine\n * \\see graphlab::async_consistent_engine\n *\n */\n template<typename VertexProgram>\n class omni_engine : public iengine<VertexProgram> {\n public:\n /** \\brief The type of the iengine */\n typedef iengine<VertexProgram> iengine_type;\n\n /**\n * \\brief The user defined vertex program type which should extend\n * ivertex_program.\n */\n typedef VertexProgram vertex_program_type;\n\n /**\n * \\brief The user defined message type which is defined in\n * ivertex_program::message_type. \n *\n */\n typedef typename vertex_program_type::message_type message_type;\n\n /**\n * \\brief The graph type which is defined in\n * ivertex_program::graph_type and will typically be\n * \\ref distributed_graph.\n */\n typedef typename vertex_program_type::graph_type graph_type;\n\n\n /**\n * \\brief The user defined type returned by the gather function.\n *\n * The gather type is defined in the \\ref graphlab::ivertex_program\n * interface and is the value returned by the\n * \\ref graphlab::ivertex_program::gather function. The\n * gather type must have an <code>operator+=(const gather_type&\n * other)</code> function and must be \\ref sec_serializable.\n */\n typedef typename VertexProgram::gather_type gather_type;\n\n /**\n * \\brief The vertex identifier type defined in \n * \\ref graphlab::vertex_id_type.\n */\n typedef typename graph_type::vertex_id_type vertex_id_type; \n\n /**\n * \\brief The type of the distributed aggregator used by each engine to\n * implement distributed aggregation.\n */ \n typedef typename iengine_type::aggregator_type aggregator_type; \n\n /**\n * \\brief the type of synchronous engine\n */\n typedef synchronous_engine<VertexProgram> synchronous_engine_type;\n\n /**\n * \\brief the type of asynchronous engine\n */\n typedef async_consistent_engine<VertexProgram> async_consistent_engine_type;\n\n\n\n private:\n\n /**\n * \\brief A pointer to the actual engine in use.\n */\n iengine_type* engine_ptr;\n\n /**\n * \\brief omni engines are not default constructible\n */\n omni_engine() { }\n\n /**\n * \\brief omni engines are not copyable\n */\n omni_engine(const omni_engine& other ) { }\n\n\n public:\n\n /**\n * \\brief Construct an omni engine for a given graph with the\n * default_engine_type unless the engine options contain an\n * alternative type.\n *\n * \\param [in] dc a distributed control object that is used to\n * connect this engine with it's counter parts on other machines.\n * \\param [in,out] graph the graph object that this engine will\n * transform.\n * \\param [in] options the command line options which are used to\n * configure the engine. Note that the engine option \"type\" can\n * be used to select the engine to use (synchronous or\n * asynchronous).\n * \\param [in] default_engine_type The user must specify what\n * engine type to use if no command line option is given.\n */\n omni_engine(distributed_control& dc, graph_type& graph,\n const std::string& default_engine_type,\n const graphlab_options& options = graphlab_options()) :\n engine_ptr(NULL) {\n graphlab_options new_options = options;\n std::string engine_type = default_engine_type;\n options_map& engine_options = new_options.get_engine_args();\n if(engine_options.get_option(\"type\", engine_type)) {\n // the engine option was set so use it instead\n // clear from the options map\n engine_options.options.erase(\"type\");\n }\n // Process the engine types\n if(engine_type == \"sync\" || engine_type == \"synchronous\") {\n logstream(LOG_INFO) << \"Using the Synchronous engine.\" << std::endl;\n engine_ptr = new synchronous_engine_type(dc, graph, new_options);\n } else if(engine_type == \"async\" || engine_type == \"asynchronous\") {\n logstream(LOG_INFO) << \"Using the Asynchronous engine.\" << std::endl;\n engine_ptr = new async_consistent_engine_type(dc, graph, new_options);\n } else {\n logstream(LOG_FATAL) << \"Invalid engine type: \" << engine_type << std::endl;\n }\n } // end of constructor\n\n /**\n * \\brief Destroy the internal engine destroying all vertex\n * programs associated with this engine.\n */\n ~omni_engine() {\n if(engine_ptr != NULL) {\n delete engine_ptr; engine_ptr = NULL;\n }\n } // end of destructor\n\n execution_status::status_enum start( ) { return engine_ptr->start(); }\n\n size_t num_updates() const { return engine_ptr->num_updates(); }\n float elapsed_seconds() const { return engine_ptr->elapsed_seconds(); }\n int iteration() const { return engine_ptr->iteration(); }\n void signal(vertex_id_type vertex,\n const message_type& message = message_type()) {\n engine_ptr->signal(vertex, message);\n }\n void signal_all(const message_type& message = message_type(),\n const std::string& order = \"shuffle\") {\n engine_ptr->signal_all(message, order);\n }\n void signal_vset(const vertex_set& vset,\n const message_type& message = message_type(),\n const std::string& order = \"shuffle\") {\n engine_ptr->signal_vset(vset, message, order);\n }\n\n\n aggregator_type* get_aggregator() { return engine_ptr->get_aggregator(); }\n\n\n }; // end of omni_engine\n\n\n\n\n}; // end of namespace graphlab\n\n#endif\n\n" }, { "path": "src/graphlab/engine/synchronous_engine.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n#ifndef GRAPHLAB_SYNCHRONOUS_ENGINE_HPP\n#define GRAPHLAB_SYNCHRONOUS_ENGINE_HPP\n\n#include <deque>\n#include <boost/bind.hpp>\n\n#include <graphlab/engine/iengine.hpp>\n\n#include <graphlab/vertex_program/ivertex_program.hpp>\n#include <graphlab/vertex_program/icontext.hpp>\n#include <graphlab/vertex_program/context.hpp>\n\n#include <graphlab/engine/execution_status.hpp>\n#include <graphlab/options/graphlab_options.hpp>\n\n\n\n\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/parallel/fiber_barrier.hpp>\n#include <graphlab/util/tracepoint.hpp>\n#include <graphlab/util/memory_info.hpp>\n\n#include <graphlab/rpc/dc_dist_object.hpp>\n#include <graphlab/rpc/distributed_event_log.hpp>\n#include <graphlab/rpc/fiber_buffered_exchange.hpp>\n\n\n\n\n\n\n#include <graphlab/macros_def.hpp>\n\nnamespace graphlab {\n\n\n /**\n * \\ingroup engines\n *\n * \\brief The synchronous engine executes all active vertex program\n * synchronously in a sequence of super-step (iterations) in both the\n * shared and distributed memory settings.\n *\n * \\tparam VertexProgram The user defined vertex program which\n * should implement the \\ref graphlab::ivertex_program interface.\n *\n *\n * ### Execution Semantics\n *\n * On start() the \\ref graphlab::ivertex_program::init function is invoked\n * on all vertex programs in parallel to initialize the vertex program,\n * vertex data, and possibly signal vertices.\n * The engine then proceeds to execute a sequence of\n * super-steps (iterations) each of which is further decomposed into a\n * sequence of minor-steps which are also executed synchronously:\n * \\li Receive all incoming messages (signals) by invoking the\n * \\ref graphlab::ivertex_program::init function on all\n * vertex-programs that have incoming messages. If a\n * vertex-program does not have any incoming messages then it is\n * not active during this super-step.\n * \\li Execute all gathers for active vertex programs by invoking\n * the user defined \\ref graphlab::ivertex_program::gather function\n * on the edge direction returned by the\n * \\ref graphlab::ivertex_program::gather_edges function. The gather\n * functions can modify edge data but cannot modify the vertex\n * program or vertex data and therefore can be executed on multiple\n * edges in parallel. The gather type is used to accumulate (sum)\n * the result of the gather function calls.\n * \\li Execute all apply functions for active vertex-programs by\n * invoking the user defined \\ref graphlab::ivertex_program::apply\n * function passing the sum of the gather functions. If \\ref\n * graphlab::ivertex_program::gather_edges returns no edges then\n * the default gather value is passed to apply. The apply function\n * can modify the vertex program and vertex data.\n * \\li Execute all scatters for active vertex programs by invoking\n * the user defined \\ref graphlab::ivertex_program::scatter function\n * on the edge direction returned by the\n * \\ref graphlab::ivertex_program::scatter_edges function. The scatter\n * functions can modify edge data but cannot modify the vertex\n * program or vertex data and therefore can be executed on multiple\n * edges in parallel.\n *\n * ### Construction\n *\n * The synchronous engine is constructed by passing in a\n * \\ref graphlab::distributed_control object which manages coordination\n * between engine threads and a \\ref graphlab::distributed_graph object\n * which is the graph on which the engine should be run. The graph should\n * already be populated and cannot change after the engine is constructed.\n * In the distributed setting all program instances (running on each machine)\n * should construct an instance of the engine at the same time.\n *\n * Computation is initiated by signaling vertices using either\n * \\ref graphlab::synchronous_engine::signal or\n * \\ref graphlab::synchronous_engine::signal_all. In either case all\n * machines should invoke signal or signal all at the same time. Finally,\n * computation is initiated by calling the\n * \\ref graphlab::synchronous_engine::start function.\n *\n * ### Example Usage\n *\n * The following is a simple example demonstrating how to use the engine:\n * \\code\n * #include <graphlab.hpp>\n *\n * struct vertex_data {\n * // code\n * };\n * struct edge_data {\n * // code\n * };\n * typedef graphlab::distributed_graph<vertex_data, edge_data> graph_type;\n * typedef float gather_type;\n * struct pagerank_vprog :\n * public graphlab::ivertex_program<graph_type, gather_type> {\n * // code\n * };\n *\n * int main(int argc, char** argv) {\n * // Initialize control plain using mpi\n * graphlab::mpi_tools::init(argc, argv);\n * graphlab::distributed_control dc;\n * // Parse command line options\n * graphlab::command_line_options clopts(\"PageRank algorithm.\");\n * std::string graph_dir;\n * clopts.attach_option(\"graph\", &graph_dir, graph_dir,\n * \"The graph file.\");\n * if(!clopts.parse(argc, argv)) {\n * std::cout << \"Error in parsing arguments.\" << std::endl;\n * return EXIT_FAILURE;\n * }\n * graph_type graph(dc, clopts);\n * graph.load_structure(graph_dir, \"tsv\");\n * graph.finalize();\n * std::cout << \"#vertices: \" << graph.num_vertices()\n * << \" #edges:\" << graph.num_edges() << std::endl;\n * graphlab::synchronous_engine<pagerank_vprog> engine(dc, graph, clopts);\n * engine.signal_all();\n * engine.start();\n * std::cout << \"Runtime: \" << engine.elapsed_seconds();\n * graphlab::mpi_tools::finalize();\n * }\n * \\endcode\n *\n *\n *\n * <a name=engineopts>Engine Options</a>\n * =====================\n * The synchronous engine supports several engine options which can\n * be set as command line arguments using \\c --engine_opts :\n *\n * \\li <b>max_iterations</b>: (default: infinity) The maximum number\n * of iterations (super-steps) to run.\n *\n * \\li <b>timeout</b>: (default: infinity) The maximum time in\n * seconds that the engine may run. When the time runs out the\n * current iteration is completed and then the engine terminates.\n *\n * \\li <b>use_cache</b>: (default: false) This is used to enable\n * caching. When caching is enabled the gather phase is skipped for\n * vertices that already have a cached value. To use caching the\n * vertex program must either clear (\\ref icontext::clear_gather_cache)\n * or update (\\ref icontext::post_delta) the cache values of\n * neighboring vertices during the scatter phase.\n *\n * \\li \\b snapshot_interval If set to a positive value, a snapshot\n * is taken every this number of iterations. If set to 0, a snapshot\n * is taken before the first iteration. If set to a negative value,\n * no snapshots are taken. Defaults to -1. A snapshot is a binary\n * dump of the graph.\n *\n * \\li \\b snapshot_path If snapshot_interval is set to a value >=0,\n * this option must be specified and should contain a target basename\n * for the snapshot. The path including folder and file prefix in\n * which the snapshots should be saved.\n *\n * \\see graphlab::omni_engine\n * \\see graphlab::async_consistent_engine\n * \\see graphlab::semi_synchronous_engine\n */\n template<typename VertexProgram>\n class synchronous_engine :\n public iengine<VertexProgram> {\n\n public:\n /**\n * \\brief The user defined vertex program type. Equivalent to the\n * VertexProgram template argument.\n *\n * The user defined vertex program type which should implement the\n * \\ref graphlab::ivertex_program interface.\n */\n typedef VertexProgram vertex_program_type;\n\n /**\n * \\brief The user defined type returned by the gather function.\n *\n * The gather type is defined in the \\ref graphlab::ivertex_program\n * interface and is the value returned by the\n * \\ref graphlab::ivertex_program::gather function. The\n * gather type must have an <code>operator+=(const gather_type&\n * other)</code> function and must be \\ref sec_serializable.\n */\n typedef typename VertexProgram::gather_type gather_type;\n\n\n /**\n * \\brief The user defined message type used to signal neighboring\n * vertex programs.\n *\n * The message type is defined in the \\ref graphlab::ivertex_program\n * interface and used in the call to \\ref graphlab::icontext::signal.\n * The message type must have an\n * <code>operator+=(const gather_type& other)</code> function and\n * must be \\ref sec_serializable.\n */\n typedef typename VertexProgram::message_type message_type;\n\n /**\n * \\brief The type of data associated with each vertex in the graph\n *\n * The vertex data type must be \\ref sec_serializable.\n */\n typedef typename VertexProgram::vertex_data_type vertex_data_type;\n\n /**\n * \\brief The type of data associated with each edge in the graph\n *\n * The edge data type must be \\ref sec_serializable.\n */\n typedef typename VertexProgram::edge_data_type edge_data_type;\n\n /**\n * \\brief The type of graph supported by this vertex program\n *\n * See graphlab::distributed_graph\n */\n typedef typename VertexProgram::graph_type graph_type;\n\n /**\n * \\brief The type used to represent a vertex in the graph.\n * See \\ref graphlab::distributed_graph::vertex_type for details\n *\n * The vertex type contains the function\n * \\ref graphlab::distributed_graph::vertex_type::data which\n * returns a reference to the vertex data as well as other functions\n * like \\ref graphlab::distributed_graph::vertex_type::num_in_edges\n * which returns the number of in edges.\n *\n */\n typedef typename graph_type::vertex_type vertex_type;\n\n /**\n * \\brief The type used to represent an edge in the graph.\n * See \\ref graphlab::distributed_graph::edge_type for details.\n *\n * The edge type contains the function\n * \\ref graphlab::distributed_graph::edge_type::data which returns a\n * reference to the edge data. In addition the edge type contains\n * the function \\ref graphlab::distributed_graph::edge_type::source and\n * \\ref graphlab::distributed_graph::edge_type::target.\n *\n */\n typedef typename graph_type::edge_type edge_type;\n\n /**\n * \\brief The type of the callback interface passed by the engine to vertex\n * programs. See \\ref graphlab::icontext for details.\n *\n * The context callback is passed to the vertex program functions and is\n * used to signal other vertices, get the current iteration, and access\n * information about the engine.\n */\n typedef icontext<graph_type, gather_type, message_type> icontext_type;\n\n private:\n\n /**\n * \\brief Local vertex type used by the engine for fast indexing\n */\n typedef typename graph_type::local_vertex_type local_vertex_type;\n\n /**\n * \\brief Local edge type used by the engine for fast indexing\n */\n typedef typename graph_type::local_edge_type local_edge_type;\n\n /**\n * \\brief Local vertex id type used by the engine for fast indexing\n */\n typedef typename graph_type::lvid_type lvid_type;\n\n std::vector<double> per_thread_compute_time;\n /**\n * \\brief The actual instance of the context type used by this engine.\n */\n typedef context<synchronous_engine> context_type;\n friend class context<synchronous_engine>;\n\n\n /**\n * \\brief The type of the distributed aggregator inherited from iengine\n */\n typedef typename iengine<vertex_program_type>::aggregator_type aggregator_type;\n\n /**\n * \\brief The object used to communicate with remote copies of the\n * synchronous engine.\n */\n dc_dist_object< synchronous_engine<VertexProgram> > rmi;\n\n /**\n * \\brief A reference to the distributed graph on which this\n * synchronous engine is running.\n */\n graph_type& graph;\n\n /**\n * \\brief The number of CPUs used.\n */\n size_t ncpus;\n\n /**\n * \\brief The local worker threads used by this engine\n */\n fiber_group threads;\n\n /**\n * \\brief A thread barrier that is used to control the threads in the\n * thread pool.\n */\n fiber_barrier thread_barrier;\n\n /**\n * \\brief The maximum number of super-steps (iterations) to run\n * before terminating. If the max iterations is reached the\n * engine will terminate if their are no messages remaining.\n */\n size_t max_iterations;\n\n\n /* \n * \\brief When caching is enabled the gather phase is skipped for\n * vertices that already have a cached value. To use caching the\n * vertex program must either clear (\\ref icontext::clear_gather_cache)\n * or update (\\ref icontext::post_delta) the cache values of\n * neighboring vertices during the scatter phase.\n */\n bool use_cache;\n\n /**\n * \\brief A snapshot is taken every this number of iterations.\n * If snapshot_interval == 0, a snapshot is only taken before the first\n * iteration. If snapshot_interval < 0, no snapshots are taken.\n */\n int snapshot_interval;\n\n /// \\brief The target base name the snapshot is saved in.\n std::string snapshot_path;\n\n /**\n * \\brief A counter that tracks the current iteration number since\n * start was last invoked.\n */\n size_t iteration_counter;\n\n /**\n * \\brief The time in seconds at which the engine started.\n */\n float start_time;\n\n /**\n * \\brief The timeout time in seconds\n */\n float timeout;\n\n /**\n * \\brief Schedules all vertices every iteration\n */\n bool sched_allv;\n\n /**\n * \\brief Used to stop the engine prematurely\n */\n bool force_abort;\n\n /**\n * \\brief The vertex locks protect access to vertex specific\n * data-structures including\n * \\ref graphlab::synchronous_engine::gather_accum\n * and \\ref graphlab::synchronous_engine::messages.\n */\n std::vector<simple_spinlock> vlocks;\n\n\n /**\n * \\brief The elocks protect individual edges during gather and\n * scatter. Technically there is a potential race since gather\n * and scatter can modify edge values and can overlap. The edge\n * lock ensures that only one gather or scatter occurs on an edge\n * at a time.\n */\n std::vector<simple_spinlock> elocks;\n\n\n\n /**\n * \\brief The vertex programs associated with each vertex on this\n * machine.\n */\n std::vector<vertex_program_type> vertex_programs;\n\n /**\n * \\brief Vector of messages associated with each vertex.\n */\n std::vector<message_type> messages;\n\n /**\n * \\brief Bit indicating whether a message is present for each vertex.\n */\n dense_bitset has_message;\n\n\n /**\n * \\brief Gather accumulator used for each master vertex to merge\n * the result of all the machine specific accumulators (or\n * caches).\n *\n * The gather accumulator can be accessed by multiple threads at\n * once and therefore must be guarded by a vertex locks in\n * \\ref graphlab::synchronous_engine::vlocks\n */\n std::vector<gather_type> gather_accum;\n\n /**\n * \\brief Bit indicating if the gather has accumulator contains any\n * values.\n *\n * While dense bitsets are thread safe the value of this bit must\n * change concurrently with the\n * \\ref graphlab::synchronous_engine::gather_accum and therefore is\n * set while holding the lock in\n * \\ref graphlab::synchronous_engine::vlocks.\n */\n dense_bitset has_gather_accum;\n\n\n /**\n * \\brief This optional vector contains caches of previous gather\n * contributions for each machine.\n *\n * Caching is done locally and therefore a high-degree vertex may\n * have multiple caches (one per machine).\n */\n std::vector<gather_type> gather_cache;\n\n /**\n * \\brief A bit indicating if the local gather for that vertex is\n * available.\n */\n dense_bitset has_cache;\n\n /**\n * \\brief A bit (for master vertices) indicating if that vertex is active\n * (received a message on this iteration).\n */\n dense_bitset active_superstep;\n\n /**\n * \\brief The number of local vertices (masters) that are active on this\n * iteration.\n */\n atomic<size_t> num_active_vertices;\n\n /**\n * \\brief A bit indicating (for all vertices) whether to\n * participate in the current minor-step (gather or scatter).\n */\n dense_bitset active_minorstep;\n\n /**\n * \\brief A counter measuring the number of applys that have been completed\n */\n atomic<size_t> completed_applys;\n\n\n /**\n * \\brief The shared counter used coordinate operations between\n * threads.\n */\n atomic<size_t> shared_lvid_counter;\n\n\n /**\n * \\brief The pair type used to synchronize vertex programs across machines.\n */\n typedef std::pair<vertex_id_type, vertex_program_type> vid_prog_pair_type;\n\n /**\n * \\brief The type of the exchange used to synchronize vertex programs\n */\n typedef fiber_buffered_exchange<vid_prog_pair_type> vprog_exchange_type;\n\n /**\n * \\brief The distributed exchange used to synchronize changes to\n * vertex programs.\n */\n vprog_exchange_type vprog_exchange;\n\n /**\n * \\brief The pair type used to synchronize vertex across across machines.\n */\n typedef std::pair<vertex_id_type, vertex_data_type> vid_vdata_pair_type;\n\n /**\n * \\brief The type of the exchange used to synchronize vertex data\n */\n typedef fiber_buffered_exchange<vid_vdata_pair_type> vdata_exchange_type;\n\n /**\n * \\brief The distributed exchange used to synchronize changes to\n * vertex programs.\n */\n vdata_exchange_type vdata_exchange;\n\n /**\n * \\brief The pair type used to synchronize the results of the gather phase\n */\n typedef std::pair<vertex_id_type, gather_type> vid_gather_pair_type;\n\n /**\n * \\brief The type of the exchange used to synchronize gather\n * accumulators\n */\n typedef fiber_buffered_exchange<vid_gather_pair_type> gather_exchange_type;\n\n /**\n * \\brief The distributed exchange used to synchronize gather\n * accumulators.\n */\n gather_exchange_type gather_exchange;\n\n /**\n * \\brief The pair type used to synchronize messages\n */\n typedef std::pair<vertex_id_type, message_type> vid_message_pair_type;\n\n /**\n * \\brief The type of the exchange used to synchronize messages\n */\n typedef fiber_buffered_exchange<vid_message_pair_type> message_exchange_type;\n\n /**\n * \\brief The distributed exchange used to synchronize messages\n */\n message_exchange_type message_exchange;\n\n\n /**\n * \\brief The distributed aggregator used to manage background\n * aggregation.\n */\n aggregator_type aggregator;\n\n DECLARE_EVENT(EVENT_APPLIES);\n DECLARE_EVENT(EVENT_GATHERS);\n DECLARE_EVENT(EVENT_SCATTERS);\n DECLARE_EVENT(EVENT_ACTIVE_CPUS);\n public:\n\n /**\n * \\brief Construct a synchronous engine for a given graph and options.\n *\n * The synchronous engine should be constructed after the graph\n * has been loaded (e.g., \\ref graphlab::distributed_graph::load)\n * and the graphlab options have been set\n * (e.g., \\ref graphlab::command_line_options).\n *\n * In the distributed engine the synchronous engine must be called\n * on all machines at the same time (in the same order) passing\n * the \\ref graphlab::distributed_control object. Upon\n * construction the synchronous engine allocates several\n * data-structures to store messages, gather accumulants, and\n * vertex programs and therefore may require considerable memory.\n *\n * The number of threads to create are read from\n * \\ref graphlab_options::get_ncpus \"opts.get_ncpus()\".\n *\n * See the <a href=\"#engineopts\">main class documentation</a>\n * for details on the available options.\n *\n * @param [in] dc Distributed controller to associate with\n * @param [in,out] graph A reference to the graph object that this\n * engine will modify. The graph must be fully constructed and\n * finalized.\n * @param [in] opts A graphlab::graphlab_options object specifying engine\n * parameters. This is typically constructed using\n * \\ref graphlab::command_line_options.\n */\n synchronous_engine(distributed_control& dc, graph_type& graph,\n const graphlab_options& opts = graphlab_options());\n\n\n /**\n * \\brief Start execution of the synchronous engine.\n *\n * The start function begins computation and does not return until\n * there are no remaining messages or until max_iterations has\n * been reached.\n *\n * The start() function modifies the data graph through the vertex\n * programs and so upon return the data graph should contain the\n * result of the computation.\n *\n * @return The reason for termination\n */\n execution_status::status_enum start();\n\n // documentation inherited from iengine\n size_t num_updates() const;\n\n // documentation inherited from iengine\n void signal(vertex_id_type vid,\n const message_type& message = message_type());\n\n // documentation inherited from iengine\n void signal_all(const message_type& message = message_type(),\n const std::string& order = \"shuffle\");\n\n void signal_vset(const vertex_set& vset,\n const message_type& message = message_type(),\n const std::string& order = \"shuffle\");\n\n\n // documentation inherited from iengine\n float elapsed_seconds() const;\n\n /**\n * \\brief Get the current iteration number since start was last\n * invoked.\n *\n * \\return the current iteration\n */\n int iteration() const;\n\n\n /**\n * \\brief Compute the total memory used by the entire distributed\n * system.\n *\n * @return The total memory used in bytes.\n */\n size_t total_memory_usage() const;\n\n /**\n * \\brief Get a pointer to the distributed aggregator object.\n *\n * This is currently used by the \\ref graphlab::iengine interface to\n * implement the calls to aggregation.\n *\n * @return a pointer to the local aggregator.\n */\n aggregator_type* get_aggregator();\n\n /**\n * \\brief Initialize the engine and allocate datastructures for vertex, and lock,\n * clear all the messages.\n */\n void init();\n\n\n private:\n\n\n /**\n * \\brief Resize the datastructures to fit the graph size (in case of dynamic graph). Keep all the messages\n * and caches.\n */\n void resize();\n\n /**\n * \\brief This internal stop function is called by the \\ref graphlab::context to\n * terminate execution of the engine.\n */\n void internal_stop();\n\n /**\n * \\brief This function is called remote by the rpc to force the\n * engine to stop.\n */\n void rpc_stop();\n\n /**\n * \\brief Signal a vertex.\n *\n * This function is called by the \\ref graphlab::context.\n *\n * @param [in] vertex the vertex to signal\n * @param [in] message the message to send to that vertex.\n */\n void internal_signal(const vertex_type& vertex,\n const message_type& message = message_type());\n\n /**\n * \\brief Called by the context to signal an arbitrary vertex.\n *\n * @param [in] gvid the global vertex id of the vertex to signal\n * @param [in] message the message to send to that vertex.\n */\n void internal_signal_gvid(vertex_id_type gvid,\n const message_type& message = message_type());\n\n /**\n * \\brief This function tests if this machine is the master of\n * gvid and signals if successful.\n */\n void internal_signal_rpc(vertex_id_type gvid,\n const message_type& message = message_type());\n\n\n /**\n * \\brief Post a to a previous gather for a give vertex.\n *\n * This function is called by the \\ref graphlab::context.\n *\n * @param [in] vertex The vertex to which to post a change in the sum\n * @param [in] delta The change in that sum\n */\n void internal_post_delta(const vertex_type& vertex,\n const gather_type& delta);\n\n /**\n * \\brief Clear the cached gather for a vertex if one is\n * available.\n *\n * This function is called by the \\ref graphlab::context.\n *\n * @param [in] vertex the vertex for which to clear the cache\n */\n void internal_clear_gather_cache(const vertex_type& vertex);\n\n\n // Program Steps ==========================================================\n\n\n void thread_launch_wrapped_event_counter(boost::function<void(void)> fn) {\n INCREMENT_EVENT(EVENT_ACTIVE_CPUS, 1);\n fn();\n DECREMENT_EVENT(EVENT_ACTIVE_CPUS, 1);\n }\n\n /**\n * \\brief Executes ncpus copies of a member function each with a\n * unique consecutive id (thread id).\n *\n * This function is used by the main loop to execute each of the\n * stages in parallel.\n *\n * The member function must have the type:\n *\n * \\code\n * void synchronous_engine::member_fun(size_t threadid);\n * \\endcode\n *\n * This function runs an rmi barrier after termination\n *\n * @tparam the type of the member function.\n * @param [in] member_fun the function to call.\n */\n template<typename MemberFunction>\n void run_synchronous(MemberFunction member_fun) {\n shared_lvid_counter = 0;\n if (ncpus <= 1) {\n INCREMENT_EVENT(EVENT_ACTIVE_CPUS, 1);\n }\n // launch the initialization threads\n for(size_t i = 0; i < ncpus; ++i) {\n fiber_control::affinity_type affinity;\n affinity.clear(); affinity.set_bit(i);\n boost::function<void(void)> invoke = boost::bind(member_fun, this, i);\n threads.launch(boost::bind(\n &synchronous_engine::thread_launch_wrapped_event_counter,\n this,\n invoke), affinity);\n }\n // Wait for all threads to finish\n threads.join();\n rmi.barrier();\n if (ncpus <= 1) {\n DECREMENT_EVENT(EVENT_ACTIVE_CPUS, 1);\n }\n } // end of run_synchronous\n\n // /**\n // * \\brief Initialize all vertex programs by invoking\n // * \\ref graphlab::ivertex_program::init on all vertices.\n // *\n // * @param thread_id the thread to run this as which determines\n // * which vertices to process.\n // */\n // void initialize_vertex_programs(size_t thread_id);\n\n /**\n * \\brief Synchronize all message data.\n *\n * @param thread_id the thread to run this as which determines\n * which vertices to process.\n */\n void exchange_messages(size_t thread_id);\n\n\n /**\n * \\brief Invoke the \\ref graphlab::ivertex_program::init function\n * on all vertex programs that have inbound messages.\n *\n * @param thread_id the thread to run this as which determines\n * which vertices to process.\n */\n void receive_messages(size_t thread_id);\n\n\n /**\n * \\brief Execute the \\ref graphlab::ivertex_program::gather function on all\n * vertices that received messages for the edges specified by the\n * \\ref graphlab::ivertex_program::gather_edges.\n *\n * @param thread_id the thread to run this as which determines\n * which vertices to process.\n */\n void execute_gathers(size_t thread_id);\n\n\n\n\n /**\n * \\brief Execute the \\ref graphlab::ivertex_program::apply function on all\n * all vertices that received messages in this super-step (active).\n *\n * @param thread_id the thread to run this as which determines\n * which vertices to process.\n */\n void execute_applys(size_t thread_id);\n\n /**\n * \\brief Execute the \\ref graphlab::ivertex_program::scatter function on all\n * vertices that received messages for the edges specified by the\n * \\ref graphlab::ivertex_program::scatter_edges.\n *\n * @param thread_id the thread to run this as which determines\n * which vertices to process.\n */\n void execute_scatters(size_t thread_id);\n\n // Data Synchronization ===================================================\n /**\n * \\brief Send the vertex program for the local vertex id to all\n * of its mirrors.\n *\n * @param [in] lvid the vertex to sync. This muster must be the\n * master of that vertex.\n */\n void sync_vertex_program(lvid_type lvid, size_t thread_id);\n\n /**\n * \\brief Receive all incoming vertex programs and update the\n * local mirrors.\n *\n * This function returns when there are no more incoming vertex\n * programs and should be called after a flush of the vertex\n * program exchange.\n */\n void recv_vertex_programs();\n\n /**\n * \\brief Send the vertex data for the local vertex id to all of\n * its mirrors.\n *\n * @param [in] lvid the vertex to sync. This machine must be the master\n * of that vertex.\n */\n void sync_vertex_data(lvid_type lvid, size_t thread_id);\n\n /**\n * \\brief Receive all incoming vertex data and update the local\n * mirrors.\n *\n * This function returns when there are no more incoming vertex\n * data and should be called after a flush of the vertex data\n * exchange.\n */\n void recv_vertex_data();\n\n /**\n * \\brief Send the gather value for the vertex id to its master.\n *\n * @param [in] lvid the vertex to send the gather value to\n * @param [in] accum the locally computed gather value.\n */\n void sync_gather(lvid_type lvid, const gather_type& accum,\n size_t thread_id);\n\n\n /**\n * \\brief Receive the gather values from the buffered exchange.\n *\n * This function returns when there is nothing left in the\n * buffered exchange and should be called after the buffered\n * exchange has been flushed\n */\n void recv_gathers();\n\n /**\n * \\brief Send the accumulated message for the local vertex to its\n * master.\n *\n * @param [in] lvid the vertex to send\n */\n void sync_message(lvid_type lvid, const size_t thread_id);\n\n /**\n * \\brief Receive the messages from the buffered exchange.\n *\n * This function returns when there is nothing left in the\n * buffered exchange and should be called after the buffered\n * exchange has been flushed\n */\n void recv_messages();\n\n\n }; // end of class synchronous engine\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n /**\n * Constructs an synchronous distributed engine.\n * The number of threads to create are read from\n * opts::get_ncpus().\n *\n * Valid engine options (graphlab_options::get_engine_args()):\n * \\arg \\c max_iterations Sets the maximum number of iterations the\n * engine will run for.\n * \\arg \\c use_cache If set to true, partial gathers are cached.\n * See \\ref gather_caching to understand the behavior of the\n * gather caching model and how it may be used to accelerate program\n * performance.\n *\n * \\param dc Distributed controller to associate with\n * \\param graph The graph to schedule over. The graph must be fully\n * constructed and finalized.\n * \\param opts A graphlab_options object containing options and parameters\n * for the engine.\n */\n template<typename VertexProgram>\n synchronous_engine<VertexProgram>::\n synchronous_engine(distributed_control &dc,\n graph_type& graph,\n const graphlab_options& opts) :\n rmi(dc, this), graph(graph),\n ncpus(opts.get_ncpus()),\n threads(2*1024*1024 /* 2MB stack per fiber*/),\n thread_barrier(opts.get_ncpus()),\n max_iterations(-1), snapshot_interval(-1), iteration_counter(0),\n timeout(0), sched_allv(false),\n vprog_exchange(dc),\n vdata_exchange(dc),\n gather_exchange(dc),\n message_exchange(dc),\n aggregator(dc, graph, new context_type(*this, graph)) {\n // Process any additional options\n std::vector<std::string> keys = opts.get_engine_args().get_option_keys();\n per_thread_compute_time.resize(opts.get_ncpus());\n use_cache = false;\n foreach(std::string opt, keys) {\n if (opt == \"max_iterations\") {\n opts.get_engine_args().get_option(\"max_iterations\", max_iterations);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: max_iterations = \"\n << max_iterations << std::endl;\n } else if (opt == \"timeout\") {\n opts.get_engine_args().get_option(\"timeout\", timeout);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: timeout = \"\n << timeout << std::endl;\n } else if (opt == \"use_cache\") {\n opts.get_engine_args().get_option(\"use_cache\", use_cache);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: use_cache = \"\n << use_cache << std::endl;\n } else if (opt == \"snapshot_interval\") {\n opts.get_engine_args().get_option(\"snapshot_interval\", snapshot_interval);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: snapshot_interval = \"\n << snapshot_interval << std::endl;\n } else if (opt == \"snapshot_path\") {\n opts.get_engine_args().get_option(\"snapshot_path\", snapshot_path);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: snapshot_path = \"\n << snapshot_path << std::endl;\n } else if (opt == \"sched_allv\") {\n opts.get_engine_args().get_option(\"sched_allv\", sched_allv);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: sched_allv = \"\n << sched_allv << std::endl;\n } else {\n logstream(LOG_FATAL) << \"Unexpected Engine Option: \" << opt << std::endl;\n }\n }\n\n if (snapshot_interval >= 0 && snapshot_path.length() == 0) {\n logstream(LOG_FATAL)\n << \"Snapshot interval specified, but no snapshot path\" << std::endl;\n }\n INITIALIZE_EVENT_LOG(dc);\n ADD_CUMULATIVE_EVENT(EVENT_APPLIES, \"Applies\", \"Calls\");\n ADD_CUMULATIVE_EVENT(EVENT_GATHERS , \"Gathers\", \"Calls\");\n ADD_CUMULATIVE_EVENT(EVENT_SCATTERS , \"Scatters\", \"Calls\");\n ADD_INSTANTANEOUS_EVENT(EVENT_ACTIVE_CPUS, \"Active Threads\", \"Threads\");\n graph.finalize();\n init();\n } // end of synchronous engine\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>:: init() {\n resize();\n // Clear up\n force_abort = false;\n iteration_counter = 0;\n completed_applys = 0;\n has_message.clear();\n has_gather_accum.clear();\n has_cache.clear();\n active_superstep.clear();\n active_minorstep.clear();\n }\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>:: resize() {\n memory_info::log_usage(\"Before Engine Initialization\");\n // Allocate vertex locks and vertex programs\n vlocks.resize(graph.num_local_vertices());\n vertex_programs.resize(graph.num_local_vertices());\n // allocate the edge locks\n //elocks.resize(graph.num_local_edges());\n // Allocate messages and message bitset\n messages.resize(graph.num_local_vertices(), message_type());\n has_message.resize(graph.num_local_vertices());\n // Allocate gather accumulators and accumulator bitset\n gather_accum.resize(graph.num_local_vertices(), gather_type());\n has_gather_accum.resize(graph.num_local_vertices());\n\n // If caching is used then allocate cache data-structures\n if (use_cache) {\n gather_cache.resize(graph.num_local_vertices(), gather_type());\n has_cache.resize(graph.num_local_vertices());\n }\n // Allocate bitset to track active vertices on each bitset.\n active_superstep.resize(graph.num_local_vertices());\n active_minorstep.resize(graph.num_local_vertices());\n\n // Print memory usage after initialization\n memory_info::log_usage(\"After Engine Initialization\");\n }\n\n\n template<typename VertexProgram>\n typename synchronous_engine<VertexProgram>::aggregator_type*\n synchronous_engine<VertexProgram>::get_aggregator() {\n return &aggregator;\n } // end of get_aggregator\n\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::internal_stop() {\n for (size_t i = 0; i < rmi.numprocs(); ++i)\n rmi.remote_call(i, &synchronous_engine<VertexProgram>::rpc_stop);\n } // end of internal_stop\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::rpc_stop() {\n force_abort = true;\n } // end of rpc_stop\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n signal(vertex_id_type gvid, const message_type& message) {\n if (vlocks.size() != graph.num_local_vertices())\n resize();\n rmi.barrier();\n internal_signal_rpc(gvid, message);\n rmi.barrier();\n } // end of signal\n\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n signal_all(const message_type& message, const std::string& order) {\n if (vlocks.size() != graph.num_local_vertices())\n resize();\n for(lvid_type lvid = 0; lvid < graph.num_local_vertices(); ++lvid) {\n if(graph.l_is_master(lvid)) {\n internal_signal(vertex_type(graph.l_vertex(lvid)), message);\n }\n }\n } // end of signal all\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n signal_vset(const vertex_set& vset,\n const message_type& message, const std::string& order) {\n if (vlocks.size() != graph.num_local_vertices())\n resize();\n for(lvid_type lvid = 0; lvid < graph.num_local_vertices(); ++lvid) {\n if(graph.l_is_master(lvid) && vset.l_contains(lvid)) {\n internal_signal(vertex_type(graph.l_vertex(lvid)), message);\n }\n }\n } // end of signal all\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n internal_signal(const vertex_type& vertex,\n const message_type& message) {\n const lvid_type lvid = vertex.local_id();\n vlocks[lvid].lock();\n if( has_message.get(lvid) ) {\n messages[lvid] += message;\n } else {\n messages[lvid] = message;\n has_message.set_bit(lvid);\n }\n vlocks[lvid].unlock();\n } // end of internal_signal\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n internal_signal_gvid(vertex_id_type gvid, const message_type& message) {\n procid_t proc = graph.master(gvid);\n if(proc == rmi.procid()) internal_signal_rpc(gvid, message);\n else rmi.remote_call(proc, \n &synchronous_engine<VertexProgram>::internal_signal_rpc,\n gvid, message);\n } \n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n internal_signal_rpc(vertex_id_type gvid,\n const message_type& message) {\n if (graph.is_master(gvid)) {\n internal_signal(graph.vertex(gvid), message);\n }\n } // end of internal_signal_rpc\n\n\n\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n internal_post_delta(const vertex_type& vertex, const gather_type& delta) {\n const bool caching_enabled = !gather_cache.empty();\n if(caching_enabled) {\n const lvid_type lvid = vertex.local_id();\n vlocks[lvid].lock();\n if( has_cache.get(lvid) ) {\n gather_cache[lvid] += delta;\n } else {\n // You cannot add a delta to an empty cache. A complete\n // gather must have been run.\n // gather_cache[lvid] = delta;\n // has_cache.set_bit(lvid);\n }\n vlocks[lvid].unlock();\n }\n } // end of post_delta\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n internal_clear_gather_cache(const vertex_type& vertex) {\n const bool caching_enabled = !gather_cache.empty();\n const lvid_type lvid = vertex.local_id();\n if(caching_enabled && has_cache.get(lvid)) {\n vlocks[lvid].lock();\n gather_cache[lvid] = gather_type();\n has_cache.clear_bit(lvid);\n vlocks[lvid].unlock();\n }\n } // end of clear_gather_cache\n\n\n\n\n template<typename VertexProgram>\n size_t synchronous_engine<VertexProgram>::\n num_updates() const { return completed_applys.value; }\n\n template<typename VertexProgram>\n float synchronous_engine<VertexProgram>::\n elapsed_seconds() const { return timer::approx_time_seconds() - start_time; }\n\n template<typename VertexProgram>\n int synchronous_engine<VertexProgram>::\n iteration() const { return iteration_counter; }\n\n\n\n template<typename VertexProgram>\n size_t synchronous_engine<VertexProgram>::total_memory_usage() const {\n size_t allocated_memory = memory_info::allocated_bytes();\n rmi.all_reduce(allocated_memory);\n return allocated_memory;\n } // compute the total memory usage of the GraphLab system\n\n\n template<typename VertexProgram> execution_status::status_enum\n synchronous_engine<VertexProgram>::start() {\n if (vlocks.size() != graph.num_local_vertices())\n resize();\n completed_applys = 0;\n rmi.barrier();\n\n // Initialization code ==================================================\n // Reset event log counters?\n // Start the timer\n graphlab::timer timer; timer.start();\n start_time = timer::approx_time_seconds();\n iteration_counter = 0;\n force_abort = false;\n execution_status::status_enum termination_reason =\n execution_status::UNSET;\n // if (perform_init_vtx_program) {\n // // Initialize all vertex programs\n // run_synchronous( &synchronous_engine::initialize_vertex_programs );\n // }\n aggregator.start();\n rmi.barrier();\n\n if (snapshot_interval == 0) {\n graph.save_binary(snapshot_path);\n }\n\n float last_print = -5;\n if (rmi.procid() == 0) {\n logstream(LOG_EMPH) << \"Iteration counter will only output every 5 seconds.\"\n << std::endl;\n }\n // Program Main loop ====================================================\n while(iteration_counter < max_iterations && !force_abort ) {\n\n // Check first to see if we are out of time\n if(timeout != 0 && timeout < elapsed_seconds()) {\n termination_reason = execution_status::TIMEOUT;\n break;\n }\n\n bool print_this_round = (elapsed_seconds() - last_print) >= 5;\n\n if(rmi.procid() == 0 && print_this_round) {\n logstream(LOG_EMPH)\n << rmi.procid() << \": Starting iteration: \" << iteration_counter\n << std::endl;\n last_print = elapsed_seconds();\n }\n // Reset Active vertices ----------------------------------------------\n // Clear the active super-step and minor-step bits which will\n // be set upon receiving messages\n active_superstep.clear(); active_minorstep.clear();\n has_gather_accum.clear();\n rmi.barrier();\n\n // Exchange Messages --------------------------------------------------\n // Exchange any messages in the local message vectors\n // if (rmi.procid() == 0) std::cout << \"Exchange messages...\" << std::endl;\n run_synchronous( &synchronous_engine::exchange_messages );\n /**\n * Post conditions:\n * 1) only master vertices have messages\n */\n\n // Receive Messages ---------------------------------------------------\n // Receive messages to master vertices and then synchronize\n // vertex programs with mirrors if gather is required\n //\n\n // if (rmi.procid() == 0) std::cout << \"Receive messages...\" << std::endl;\n num_active_vertices = 0;\n run_synchronous( &synchronous_engine::receive_messages );\n if (sched_allv) {\n active_minorstep.fill();\n }\n has_message.clear();\n /**\n * Post conditions:\n * 1) there are no messages remaining\n * 2) All masters that received messages have their\n * active_superstep bit set\n * 3) All masters and mirrors that are to participate in the\n * next gather phases have their active_minorstep bit\n * set.\n * 4) num_active_vertices is the number of vertices that\n * received messages.\n */\n\n // Check termination condition ---------------------------------------\n size_t total_active_vertices = num_active_vertices;\n rmi.all_reduce(total_active_vertices);\n if (rmi.procid() == 0 && print_this_round)\n logstream(LOG_EMPH)\n << \"\\tActive vertices: \" << total_active_vertices << std::endl;\n if(total_active_vertices == 0 ) {\n termination_reason = execution_status::TASK_DEPLETION;\n break;\n }\n\n\n // Execute gather operations-------------------------------------------\n // Execute the gather operation for all vertices that are active\n // in this minor-step (active-minorstep bit set).\n // if (rmi.procid() == 0) std::cout << \"Gathering...\" << std::endl;\n run_synchronous( &synchronous_engine::execute_gathers );\n // Clear the minor step bit since only super-step vertices\n // (only master vertices are required to participate in the\n // apply step)\n active_minorstep.clear(); // rmi.barrier();\n /**\n * Post conditions:\n * 1) gather_accum for all master vertices contains the\n * result of all the gathers (even if they are drawn from\n * cache)\n * 2) No minor-step bits are set\n */\n\n // Execute Apply Operations -------------------------------------------\n // Run the apply function on all active vertices\n // if (rmi.procid() == 0) std::cout << \"Applying...\" << std::endl;\n run_synchronous( &synchronous_engine::execute_applys );\n /**\n * Post conditions:\n * 1) any changes to the vertex data have been synchronized\n * with all mirrors.\n * 2) all gather accumulators have been cleared\n * 3) If a vertex program is participating in the scatter\n * phase its minor-step bit has been set to active (both\n * masters and mirrors) and the vertex program has been\n * synchronized with the mirrors.\n */\n\n\n // Execute Scatter Operations -----------------------------------------\n // Execute each of the scatters on all minor-step active vertices.\n run_synchronous( &synchronous_engine::execute_scatters );\n /**\n * Post conditions:\n * 1) NONE\n */\n if(rmi.procid() == 0 && print_this_round)\n logstream(LOG_EMPH) << \"\\t Running Aggregators\" << std::endl;\n // probe the aggregator\n aggregator.tick_synchronous();\n\n ++iteration_counter;\n\n if (snapshot_interval > 0 && iteration_counter % snapshot_interval == 0) {\n graph.save_binary(snapshot_path);\n }\n }\n\n if (rmi.procid() == 0) {\n logstream(LOG_EMPH) << iteration_counter\n << \" iterations completed.\" << std::endl;\n }\n // Final barrier to ensure that all engines terminate at the same time\n double total_compute_time = 0;\n for (size_t i = 0;i < per_thread_compute_time.size(); ++i) {\n total_compute_time += per_thread_compute_time[i];\n }\n std::vector<double> all_compute_time_vec(rmi.numprocs());\n all_compute_time_vec[rmi.procid()] = total_compute_time;\n rmi.all_gather(all_compute_time_vec);\n\n size_t global_completed = completed_applys;\n rmi.all_reduce(global_completed);\n completed_applys = global_completed;\n rmi.cout() << \"Updates: \" << completed_applys.value << \"\\n\";\n if (rmi.procid() == 0) {\n logstream(LOG_INFO) << \"Compute Balance: \";\n for (size_t i = 0;i < all_compute_time_vec.size(); ++i) {\n logstream(LOG_INFO) << all_compute_time_vec[i] << \" \";\n }\n logstream(LOG_INFO) << std::endl;\n }\n rmi.full_barrier();\n // Stop the aggregator\n aggregator.stop();\n // return the final reason for termination\n return termination_reason;\n } // end of start\n\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n exchange_messages(const size_t thread_id) {\n context_type context(*this, graph);\n const size_t TRY_RECV_MOD = 100;\n size_t vcount = 0;\n fixed_dense_bitset<8 * sizeof(size_t)> local_bitset; // a word-size = 64 bit\n while (1) {\n // increment by a word at a time\n lvid_type lvid_block_start =\n shared_lvid_counter.inc_ret_last(8 * sizeof(size_t));\n if (lvid_block_start >= graph.num_local_vertices()) break;\n // get the bit field from has_message\n size_t lvid_bit_block = has_message.containing_word(lvid_block_start);\n if (lvid_bit_block == 0) continue;\n // initialize a word sized bitfield\n local_bitset.clear();\n local_bitset.initialize_from_mem(&lvid_bit_block, sizeof(size_t));\n foreach(size_t lvid_block_offset, local_bitset) {\n lvid_type lvid = lvid_block_start + lvid_block_offset;\n if (lvid >= graph.num_local_vertices()) break;\n // if the vertex is not local and has a message send the\n // message and clear the bit\n if(!graph.l_is_master(lvid)) {\n sync_message(lvid, thread_id);\n has_message.clear_bit(lvid);\n // clear the message to save memory\n messages[lvid] = message_type();\n }\n if(++vcount % TRY_RECV_MOD == 0) recv_messages();\n }\n } // end of loop over vertices to send messages\n message_exchange.partial_flush();\n // Finish sending and receiving all messages\n thread_barrier.wait();\n if(thread_id == 0) message_exchange.flush();\n thread_barrier.wait();\n recv_messages();\n } // end of exchange_messages\n\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n receive_messages(const size_t thread_id) {\n context_type context(*this, graph);\n const size_t TRY_RECV_MOD = 100;\n size_t vcount = 0;\n size_t nactive_inc = 0;\n fixed_dense_bitset<8 * sizeof(size_t)> local_bitset; // a word-size = 64 bit\n\n while (1) {\n // increment by a word at a time\n lvid_type lvid_block_start =\n shared_lvid_counter.inc_ret_last(8 * sizeof(size_t));\n if (lvid_block_start >= graph.num_local_vertices()) break;\n // get the bit field from has_message\n size_t lvid_bit_block = has_message.containing_word(lvid_block_start);\n if (lvid_bit_block == 0) continue;\n // initialize a word sized bitfield\n local_bitset.clear();\n local_bitset.initialize_from_mem(&lvid_bit_block, sizeof(size_t));\n\n foreach(size_t lvid_block_offset, local_bitset) {\n lvid_type lvid = lvid_block_start + lvid_block_offset;\n if (lvid >= graph.num_local_vertices()) break;\n\n // if this is the master of lvid and we have a message\n if(graph.l_is_master(lvid)) {\n // The vertex becomes active for this superstep\n active_superstep.set_bit(lvid);\n ++nactive_inc;\n // Pass the message to the vertex program\n vertex_type vertex = vertex_type(graph.l_vertex(lvid));\n vertex_programs[lvid].init(context, vertex, messages[lvid]);\n // clear the message to save memory\n messages[lvid] = message_type();\n if (sched_allv) continue;\n // Determine if the gather should be run\n const vertex_program_type& const_vprog = vertex_programs[lvid];\n const vertex_type const_vertex = vertex;\n if(const_vprog.gather_edges(context, const_vertex) !=\n graphlab::NO_EDGES) {\n active_minorstep.set_bit(lvid);\n sync_vertex_program(lvid, thread_id);\n }\n }\n if(++vcount % TRY_RECV_MOD == 0) recv_vertex_programs();\n }\n }\n\n num_active_vertices += nactive_inc;\n vprog_exchange.partial_flush();\n // Flush the buffer and finish receiving any remaining vertex\n // programs.\n thread_barrier.wait();\n if(thread_id == 0) {\n vprog_exchange.flush();\n }\n thread_barrier.wait();\n\n recv_vertex_programs();\n\n } // end of receive messages\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n execute_gathers(const size_t thread_id) {\n context_type context(*this, graph);\n const size_t TRY_RECV_MOD = 1000;\n size_t vcount = 0;\n const bool caching_enabled = !gather_cache.empty();\n timer ti;\n\n fixed_dense_bitset<8 * sizeof(size_t)> local_bitset; // a word-size = 64 bit\n\n while (1) {\n // increment by a word at a time\n lvid_type lvid_block_start =\n shared_lvid_counter.inc_ret_last(8 * sizeof(size_t));\n if (lvid_block_start >= graph.num_local_vertices()) break;\n // get the bit field from has_message\n size_t lvid_bit_block = active_minorstep.containing_word(lvid_block_start);\n if (lvid_bit_block == 0) continue;\n // initialize a word sized bitfield\n local_bitset.clear();\n local_bitset.initialize_from_mem(&lvid_bit_block, sizeof(size_t));\n\n foreach(size_t lvid_block_offset, local_bitset) {\n lvid_type lvid = lvid_block_start + lvid_block_offset;\n if (lvid >= graph.num_local_vertices()) break;\n\n bool accum_is_set = false;\n gather_type accum = gather_type();\n // if caching is enabled and we have a cache entry then use\n // that as the accum\n if( caching_enabled && has_cache.get(lvid) ) {\n accum = gather_cache[lvid];\n accum_is_set = true;\n } else {\n // recompute the local contribution to the gather\n const vertex_program_type& vprog = vertex_programs[lvid];\n local_vertex_type local_vertex = graph.l_vertex(lvid);\n const vertex_type vertex(local_vertex);\n const edge_dir_type gather_dir = vprog.gather_edges(context, vertex);\n // Loop over in edges\n size_t edges_touched = 0;\n vprog.pre_local_gather(accum);\n if(gather_dir == IN_EDGES || gather_dir == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.in_edges()) {\n edge_type edge(local_edge);\n // elocks[local_edge.id()].lock();\n if(accum_is_set) { // \\todo hint likely\n accum += vprog.gather(context, vertex, edge);\n } else {\n accum = vprog.gather(context, vertex, edge);\n accum_is_set = true;\n }\n ++edges_touched;\n // elocks[local_edge.id()].unlock();\n }\n } // end of if in_edges/all_edges\n // Loop over out edges\n if(gather_dir == OUT_EDGES || gather_dir == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.out_edges()) {\n edge_type edge(local_edge);\n // elocks[local_edge.id()].lock();\n if(accum_is_set) { // \\todo hint likely\n accum += vprog.gather(context, vertex, edge);\n } else {\n accum = vprog.gather(context, vertex, edge);\n accum_is_set = true;\n }\n // elocks[local_edge.id()].unlock();\n ++edges_touched;\n }\n INCREMENT_EVENT(EVENT_GATHERS, edges_touched);\n } // end of if out_edges/all_edges\n vprog.post_local_gather(accum);\n // If caching is enabled then save the accumulator to the\n // cache for future iterations. Note that it is possible\n // that the accumulator was never set in which case we are\n // effectively \"zeroing out\" the cache.\n if(caching_enabled && accum_is_set) {\n gather_cache[lvid] = accum; has_cache.set_bit(lvid);\n } // end of if caching enabled\n }\n // If the accum contains a value for the local gather we put\n // that estimate in the gather exchange.\n if(accum_is_set) sync_gather(lvid, accum, thread_id);\n if(!graph.l_is_master(lvid)) {\n // if this is not the master clear the vertex program\n vertex_programs[lvid] = vertex_program_type();\n }\n\n // try to recv gathers if there are any in the buffer\n if(++vcount % TRY_RECV_MOD == 0) recv_gathers();\n }\n } // end of loop over vertices to compute gather accumulators\n per_thread_compute_time[thread_id] += ti.current_time();\n gather_exchange.partial_flush();\n // Finish sending and receiving all gather operations\n thread_barrier.wait();\n if(thread_id == 0) gather_exchange.flush();\n thread_barrier.wait();\n recv_gathers();\n } // end of execute_gathers\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n execute_applys(const size_t thread_id) {\n context_type context(*this, graph);\n const size_t TRY_RECV_MOD = 1000;\n size_t vcount = 0;\n timer ti;\n\n fixed_dense_bitset<8 * sizeof(size_t)> local_bitset; // allocate a word size = 64bits\n while (1) {\n // increment by a word at a time\n lvid_type lvid_block_start =\n shared_lvid_counter.inc_ret_last(8 * sizeof(size_t));\n if (lvid_block_start >= graph.num_local_vertices()) break;\n // get the bit field from has_message\n size_t lvid_bit_block = active_superstep.containing_word(lvid_block_start);\n if (lvid_bit_block == 0) continue;\n // initialize a word sized bitfield\n local_bitset.clear();\n local_bitset.initialize_from_mem(&lvid_bit_block, sizeof(size_t));\n foreach(size_t lvid_block_offset, local_bitset) {\n lvid_type lvid = lvid_block_start + lvid_block_offset;\n if (lvid >= graph.num_local_vertices()) break;\n\n // Only master vertices can be active in a super-step\n ASSERT_TRUE(graph.l_is_master(lvid));\n vertex_type vertex(graph.l_vertex(lvid));\n // Get the local accumulator. Note that it is possible that\n // the gather_accum was not set during the gather.\n const gather_type& accum = gather_accum[lvid];\n INCREMENT_EVENT(EVENT_APPLIES, 1);\n vertex_programs[lvid].apply(context, vertex, accum);\n // record an apply as a completed task\n ++completed_applys;\n // Clear the accumulator to save some memory\n gather_accum[lvid] = gather_type();\n // synchronize the changed vertex data with all mirrors\n sync_vertex_data(lvid, thread_id);\n // determine if a scatter operation is needed\n const vertex_program_type& const_vprog = vertex_programs[lvid];\n const vertex_type const_vertex = vertex;\n if(const_vprog.scatter_edges(context, const_vertex) !=\n graphlab::NO_EDGES) {\n active_minorstep.set_bit(lvid);\n sync_vertex_program(lvid, thread_id);\n } else { // we are done so clear the vertex program\n vertex_programs[lvid] = vertex_program_type();\n }\n // try to receive vertex data\n if(++vcount % TRY_RECV_MOD == 0) {\n recv_vertex_programs();\n recv_vertex_data();\n }\n }\n } // end of loop over vertices to run apply\n\n per_thread_compute_time[thread_id] += ti.current_time();\n vprog_exchange.partial_flush();\n vdata_exchange.partial_flush();\n // Finish sending and receiving all changes due to apply operations\n thread_barrier.wait();\n if(thread_id == 0) { \n vprog_exchange.flush(); vdata_exchange.flush(); \n }\n thread_barrier.wait();\n recv_vertex_programs();\n recv_vertex_data();\n } // end of execute_applys\n\n\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n execute_scatters(const size_t thread_id) {\n context_type context(*this, graph);\n timer ti;\n fixed_dense_bitset<8 * sizeof(size_t)> local_bitset; // allocate a word size = 64 bits\n while (1) {\n // increment by a word at a time\n lvid_type lvid_block_start =\n shared_lvid_counter.inc_ret_last(8 * sizeof(size_t));\n if (lvid_block_start >= graph.num_local_vertices()) break;\n // get the bit field from has_message\n size_t lvid_bit_block = active_minorstep.containing_word(lvid_block_start);\n if (lvid_bit_block == 0) continue;\n // initialize a word sized bitfield\n local_bitset.clear();\n local_bitset.initialize_from_mem(&lvid_bit_block, sizeof(size_t));\n foreach(size_t lvid_block_offset, local_bitset) {\n lvid_type lvid = lvid_block_start + lvid_block_offset;\n if (lvid >= graph.num_local_vertices()) break;\n\n const vertex_program_type& vprog = vertex_programs[lvid];\n local_vertex_type local_vertex = graph.l_vertex(lvid);\n const vertex_type vertex(local_vertex);\n const edge_dir_type scatter_dir = vprog.scatter_edges(context, vertex);\n\t\t\t\tsize_t edges_touched = 0;\n // Loop over in edges\n if(scatter_dir == IN_EDGES || scatter_dir == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.in_edges()) {\n edge_type edge(local_edge);\n // elocks[local_edge.id()].lock();\n vprog.scatter(context, vertex, edge);\n // elocks[local_edge.id()].unlock();\n }\n\t\t\t\t\t++edges_touched;\n } // end of if in_edges/all_edges\n // Loop over out edges\n if(scatter_dir == OUT_EDGES || scatter_dir == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.out_edges()) {\n edge_type edge(local_edge);\n // elocks[local_edge.id()].lock();\n vprog.scatter(context, vertex, edge);\n // elocks[local_edge.id()].unlock();\n }\n\t\t\t\t\t++edges_touched;\n } // end of if out_edges/all_edges\n\t\t\t\tINCREMENT_EVENT(EVENT_SCATTERS, edges_touched);\n // Clear the vertex program\n vertex_programs[lvid] = vertex_program_type();\n } // end of if active on this minor step\n } // end of loop over vertices to complete scatter operation\n\n per_thread_compute_time[thread_id] += ti.current_time();\n } // end of execute_scatters\n\n\n\n // Data Synchronization ===================================================\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n sync_vertex_program(lvid_type lvid, const size_t thread_id) {\n ASSERT_TRUE(graph.l_is_master(lvid));\n const vertex_id_type vid = graph.global_vid(lvid);\n local_vertex_type vertex = graph.l_vertex(lvid);\n foreach(const procid_t& mirror, vertex.mirrors()) {\n vprog_exchange.send(mirror,\n std::make_pair(vid, vertex_programs[lvid]));\n }\n } // end of sync_vertex_program\n\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n recv_vertex_programs() {\n typename vprog_exchange_type::recv_buffer_type recv_buffer;\n while(vprog_exchange.recv(recv_buffer)) {\n for (size_t i = 0;i < recv_buffer.size(); ++i) {\n typename vprog_exchange_type::buffer_type& buffer = recv_buffer[i].buffer;\n foreach(const vid_prog_pair_type& pair, buffer) {\n const lvid_type lvid = graph.local_vid(pair.first);\n // ASSERT_FALSE(graph.l_is_master(lvid));\n vertex_programs[lvid] = pair.second;\n active_minorstep.set_bit(lvid);\n }\n }\n }\n } // end of recv vertex programs\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n sync_vertex_data(lvid_type lvid, const size_t thread_id) {\n ASSERT_TRUE(graph.l_is_master(lvid));\n const vertex_id_type vid = graph.global_vid(lvid);\n local_vertex_type vertex = graph.l_vertex(lvid);\n foreach(const procid_t& mirror, vertex.mirrors()) {\n vdata_exchange.send(mirror, std::make_pair(vid, vertex.data()));\n }\n } // end of sync_vertex_data\n\n\n\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n recv_vertex_data() {\n typename vdata_exchange_type::recv_buffer_type recv_buffer;\n while(vdata_exchange.recv(recv_buffer)) {\n for (size_t i = 0;i < recv_buffer.size(); ++i) {\n typename vdata_exchange_type::buffer_type& buffer = recv_buffer[i].buffer;\n foreach(const vid_vdata_pair_type& pair, buffer) {\n const lvid_type lvid = graph.local_vid(pair.first);\n ASSERT_FALSE(graph.l_is_master(lvid));\n graph.l_vertex(lvid).data() = pair.second;\n }\n }\n }\n } // end of recv vertex data\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n sync_gather(lvid_type lvid, const gather_type& accum, const size_t thread_id) {\n if(graph.l_is_master(lvid)) {\n vlocks[lvid].lock();\n if(has_gather_accum.get(lvid)) {\n gather_accum[lvid] += accum;\n } else {\n gather_accum[lvid] = accum;\n has_gather_accum.set_bit(lvid);\n }\n vlocks[lvid].unlock();\n } else {\n const procid_t master = graph.l_master(lvid);\n const vertex_id_type vid = graph.global_vid(lvid);\n gather_exchange.send(master, std::make_pair(vid, accum));\n }\n } // end of sync_gather\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n recv_gathers() {\n typename gather_exchange_type::recv_buffer_type recv_buffer;\n while(gather_exchange.recv(recv_buffer)) {\n for (size_t i = 0;i < recv_buffer.size(); ++i) {\n typename gather_exchange_type::buffer_type& buffer = recv_buffer[i].buffer;\n foreach(const vid_gather_pair_type& pair, buffer) {\n const lvid_type lvid = graph.local_vid(pair.first);\n const gather_type& accum = pair.second;\n ASSERT_TRUE(graph.l_is_master(lvid));\n vlocks[lvid].lock();\n if( has_gather_accum.get(lvid) ) {\n gather_accum[lvid] += accum;\n } else {\n gather_accum[lvid] = accum;\n has_gather_accum.set_bit(lvid);\n }\n vlocks[lvid].unlock();\n }\n }\n }\n } // end of recv_gather\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n sync_message(lvid_type lvid, const size_t thread_id) {\n ASSERT_FALSE(graph.l_is_master(lvid));\n const procid_t master = graph.l_master(lvid);\n const vertex_id_type vid = graph.global_vid(lvid);\n message_exchange.send(master, std::make_pair(vid, messages[lvid]));\n } // end of send_message\n\n\n\n\n template<typename VertexProgram>\n void synchronous_engine<VertexProgram>::\n recv_messages() {\n typename message_exchange_type::recv_buffer_type recv_buffer;\n while(message_exchange.recv(recv_buffer)) {\n for (size_t i = 0;i < recv_buffer.size(); ++i) {\n typename message_exchange_type::buffer_type& buffer = recv_buffer[i].buffer;\n foreach(const vid_message_pair_type& pair, buffer) {\n const lvid_type lvid = graph.local_vid(pair.first);\n ASSERT_TRUE(graph.l_is_master(lvid));\n vlocks[lvid].lock();\n if( has_message.get(lvid) ) {\n messages[lvid] += pair.second;\n } else {\n messages[lvid] = pair.second;\n has_message.set_bit(lvid);\n }\n vlocks[lvid].unlock();\n }\n }\n }\n } // end of recv_messages\n\n\n\n\n\n\n\n\n\n\n\n}; // namespace\n\n\n#include <graphlab/macros_undef.hpp>\n\n#endif\n\n" }, { "path": "src/graphlab/engine/warp_engine.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n\n#ifndef GRAPHLAB_WARP_ENGINE\n#define GRAPHLAB_WARP_ENGINE\n\n#include <deque>\n#include <boost/bind.hpp>\n\n#include <graphlab/scheduler/ischeduler.hpp>\n#include <graphlab/scheduler/scheduler_factory.hpp>\n#include <graphlab/scheduler/get_message_priority.hpp>\n#include <graphlab/engine/iengine.hpp>\n#include <graphlab/engine/execution_status.hpp>\n#include <graphlab/options/graphlab_options.hpp>\n#include <graphlab/rpc/dc_dist_object.hpp>\n#include <graphlab/engine/distributed_chandy_misra.hpp>\n#include <graphlab/engine/message_array.hpp>\n#include <graphlab/serialization/serialize_to_from_string.hpp>\n#include <graphlab/util/tracepoint.hpp>\n#include <graphlab/util/memory_info.hpp>\n#include <graphlab/util/generics/conditional_addition_wrapper.hpp>\n#include <graphlab/rpc/distributed_event_log.hpp>\n#include <graphlab/parallel/fiber_group.hpp>\n#include <graphlab/parallel/fiber_control.hpp>\n#include <graphlab/rpc/fiber_async_consensus.hpp>\n#include <graphlab/aggregation/distributed_aggregator.hpp>\n#include <graphlab/parallel/fiber_remote_request.hpp>\n#include <graphlab/macros_def.hpp>\n\n\n\nnamespace graphlab {\n\nnamespace warp {\n\n /**\n * \\ingroup warp\n *\n * \\brief The warp engine executed update functions\n * asynchronously and can ensure mutual exclusion such that adjacent vertices\n * are never executed simultaneously. The default mode is \"factorized\"\n * consistency in which only individual gathers/applys/\n * scatters are guaranteed to be consistent, but this can be strengthened to\n * provide full mutual exclusion.\n *\n * ### Execution Semantics\n * The update function is a simple user defined function of the type\n *\n * \\code\n * void update_function(engine_type::context& context,\n * graph_type::vertex_type vertex) {\n * }\n * \\endcode\n *\n * Based on a scheduler, update functions are executed on each scheduled \n * vertex. All computation is performed from within fine-grained threads\n * called fibers, which allows to create thousands of such fibers, thus\n * hiding distributed communication latency.\n *\n * Within the update function, All blocking warp functions such as \n * \\ref graphlab::warp::map_reduce_neighborhood, \n * \\ref graphlab::warp::transform_neighborhood, and \n * \\ref graphlab::warp::broadcast_neighborhood\n * can be used to make changes to the graph data, and to schedule other \n * vertices for computation.\n *\n * The engine stops when the scheduler is empty.\n *\n * ### Construction\n *\n * The warp engine is constructed by passing in a\n * \\ref graphlab::distributed_control object which manages coordination\n * between engine threads and a \\ref graphlab::distributed_graph object\n * which is the graph on which the engine should be run. \n * \n * Computation is initiated by signaling vertices using either\n * \\ref graphlab::warp_engine::signal or\n * \\ref graphlab::warp_engine::signal_all. In either case all\n * machines should invoke signal or signal all at the same time. Finally,\n * computation is initiated by calling the\n * \\ref graphlab::warp_engine::start function.\n *\n * \\see warp::map_reduce_neighborhood()\n * \\see warp::transform_neighborhood()\n * \\see warp::broadcast_neighborhood()\n *\n * ### Example Usage\n *\n * The following is a simple example demonstrating how to use the engine:\n * \\code\n * #include <graphlab.hpp>\n *\n * struct vertex_data {\n * // code\n * };\n * struct edge_data {\n * // code\n * };\n * typedef graphlab::distributed_graph<vertex_data, edge_data> graph_type;\n * typedef graphlab::warp_engine<graph_type> engine_type;\n *\n * void pagerank(engine_type::context& context,\n * graph_type::vertex_type vertex) {\n * ... \n * } \n *\n *\n * int main(int argc, char** argv) {\n * // Initialize control plain using mpi\n * graphlab::mpi_tools::init(argc, argv);\n * graphlab::distributed_control dc;\n * // Parse command line options\n * graphlab::command_line_options clopts(\"PageRank algorithm.\");\n * std::string graph_dir;\n * clopts.attach_option(\"graph\", graph_dir,\n * \"The graph file.\");\n * if(!clopts.parse(argc, argv)) {\n * std::cout << \"Error in parsing arguments.\" << std::endl;\n * return EXIT_FAILURE;\n * }\n * graph_type graph(dc, clopts);\n * graph.load_structure(graph_dir, \"tsv\");\n * graph.finalize();\n * std::cout << \"#vertices: \" << graph.num_vertices()\n * << \" #edges:\" << graph.num_edges() << std::endl;\n * engine_type engine(dc, graph, clopts);\n * engine.set_update_function(pagerank);\n * engine.signal_all();\n * engine.start();\n * std::cout << \"Runtime: \" << engine.elapsed_seconds();\n * graphlab::mpi_tools::finalize();\n * }\n * \\endcode\n *\n *\n * <a name=engineopts>Engine Options</a>\n * =========================\n * The warp engine supports several engine options which can\n * be set as command line arguments using \\c --engine_opts :\n *\n * \\li \\b timeout (default: infinity) Maximum time in seconds the engine will\n * run for. The actual runtime may be marginally greater as the engine\n * waits for all threads and processes to flush all active tasks before\n * returning.\n * \\li \\b factorized (default: true) Set to true to weaken the consistency\n * model to factorized consistency where only individual gather/apply/scatter\n * calls are guaranteed to be locally consistent. Can produce massive\n * increases in throughput at a consistency penalty.\n * \\li \\b nfibers (default: 10000) Number of fibers to use\n * \\li \\b stacksize (default: 16384) Stacksize of each fiber.\n */\n template <typename GraphType, typename MessageType = graphlab::empty>\n class warp_engine {\n\n public:\n /**\n * \\brief The user defined message type used to signal neighboring\n * vertex programs.\n */\n typedef MessageType message_type;\n\n /**\n * The type of the graph associated with this engine.\n */\n typedef GraphType graph_type;\n\n /**\n * \\brief The type of data associated with each vertex in the graph\n *\n * The vertex data type must be \\ref sec_serializable.\n */\n typedef typename graph_type::vertex_data_type vertex_data_type;\n\n /**\n * \\brief The type of data associated with each edge in the graph\n *\n * The edge data type must be \\ref sec_serializable.\n */\n typedef typename graph_type::edge_data_type edge_data_type;\n\n\n /**\n * \\brief The type used to represent a vertex in the graph.\n * See \\ref graphlab::distributed_graph::vertex_type for details\n *\n * The vertex type contains the function\n * \\ref graphlab::distributed_graph::vertex_type::data which\n * returns a reference to the vertex data as well as other functions\n * like \\ref graphlab::distributed_graph::vertex_type::num_in_edges\n * which returns the number of in edges.\n *\n */\n typedef typename graph_type::vertex_type vertex_type;\n\n /**\n * \\brief The type used to represent an edge in the graph.\n * See \\ref graphlab::distributed_graph::edge_type for details.\n *\n * The edge type contains the function\n * \\ref graphlab::distributed_graph::edge_type::data which returns a\n * reference to the edge data. In addition the edge type contains\n * the function \\ref graphlab::distributed_graph::edge_type::source and\n * \\ref graphlab::distributed_graph::edge_type::target.\n *\n */\n typedef typename graph_type::edge_type edge_type;\n\n\n\n\n struct context {\n typedef warp_engine engine_type;\n typedef typename engine_type::graph_type graph_type;\n typedef typename graph_type::vertex_type vertex_type;\n typedef typename graph_type::edge_type edge_type;\n typedef typename graph_type::local_vertex_type local_vertex_type;\n\n warp_engine& engine;\n graph_type& graph;\n std::string original_value;\n vertex_type vtx;\n bool vtx_set;\n\n context(warp_engine& engine, graph_type& graph, \n vertex_type vtx):\n engine(engine), \n graph(graph), \n vtx(vtx),\n vtx_set(true) { \n set_synchronized();\n }\n \n\n context(warp_engine& engine, graph_type& graph):\n engine(engine), \n graph(graph), \n vtx(graph, 0),\n vtx_set(false) { \n }\n \n /**\n * \\brief Get the total number of vertices in the graph.\n *\n * \\return the total number of vertices in the entire graph.\n */\n size_t num_vertices() const { return graph.num_vertices(); }\n\n /**\n * \\brief Get the number of edges in the graph.\n *\n * Each direction counts as a separate edge.\n *\n * \\return the total number of edges in the entire graph.\n */ \n size_t num_edges() const { return graph.num_edges(); }\n\n /**\n * \\brief Get the id of this process.\n *\n * The procid is a number between 0 and \n * \\ref graphlab::icontext::num_procs\n * \n * \\warning Each process may have many threads\n *\n * @return the process of this machine.\n */\n size_t procid() const { return graph.procid(); }\n\n /**\n * \\brief Get the number of processes in the current execution.\n *\n * This is typically the number of mpi jobs created:\n * \\code\n * %> mpiexec -n 16 ./pagerank\n * \\endcode\n * would imply that num_procs() returns 16.\n *\n * @return the number of processes in the current execution\n */\n size_t num_procs() const { return graph.numprocs(); }\n\n /**\n * \\brief Returns a standard output object (like cout)\n * which only prints once even when running distributed.\n * \n * This returns a C++ standard output stream object\n * which maps directly to std::cout on machine with \n * process ID 0, and to empty output streamss\n * on all other processes. Calling,\n * \\code\n * context.cout() << \"Hello World!\";\n * \\endcode\n * will therefore only print if the code is run on machine 0.\n * This is useful in the finalize operation in aggregators.\n */\n std::ostream& cout() const {\n return graph.dc().cout();\n }\n\n /**\n * \\brief Returns a standard error object (like cerr)\n * which only prints once even when running distributed.\n * \n * This returns a C++ standard output stream object\n * which maps directly to std::cerr on machine with \n * process ID 0, and to empty output streamss\n * on all other processes. Calling,\n * \\code\n * context.cerr() << \"Hello World!\";\n * \\endcode\n * will therefore only print if the code is run on machine 0.\n * This is useful in the finalize operation in aggregators.\n */\n\n std::ostream& cerr() const {\n return graph.dc().cerr();\n }\n\n /**\n * \\brief Get the elapsed time in seconds since start was called.\n * \n * \\return runtine in seconds\n */ \n float elapsed_seconds() const { return engine.elapsed_seconds(); }\n\n /**\n * \\brief Return the current interation number (if supported).\n *\n * \\return the current interation number if support or -1\n * otherwise.\n */\n int iteration() const { return -1; }\n\n /**\n * \\brief Signal the engine to stop executing additional update\n * functions.\n *\n * \\warning The execution engine will stop *eventually* and\n * additional update functions may be executed prior to when the\n * engine stops. For-example the synchronous engine (see \\ref\n * synchronous_engine) will complete the current super-step before\n * terminating.\n */\n void stop() { engine.internal_stop(); }\n\n /**\n * \\brief Signal a vertex with a particular message.\n *\n * This function is an essential part of the GraphLab abstraction\n * and is used to encode iterative computation. Typically a vertex\n * program will signal neighboring vertices during the scatter\n * phase. A vertex program may choose to signal neighbors on when\n * changes made during the previous phases break invariants or warrant\n * future computation on neighboring vertices.\n * \n * The signal function takes two arguments. The first is mandatory\n * and specifies which vertex to signal. The second argument is\n * optional and is used to send a message. If no message is\n * provided then the default message is used.\n *\n * \\param vertex [in] The vertex to send the message to\n * \\param message [in] The message to send, defaults to message_type(). \n */\n void signal(const vertex_type& vertex, \n const message_type& message = message_type()) {\n engine.internal_signal(vertex, message);\n }\n\n\n\n /**\n * \\brief Signal an arbitrary vertex ID with a particular message.\n *\n * This function is an essential part of the GraphLab abstraction\n * and is used to encode iterative computation. Typically a vertex\n * program will signal neighboring vertices during the scatter\n * phase. A vertex program may choose to signal neighbors on when\n * changes made during the previous phases break invariants or warrant\n * future computation on neighboring vertices.\n * \n * The signal function takes two arguments. The first is mandatory\n * and specifies which vertex to signal. The second argument is\n * optional and is used to send a message. If no message is\n * provided then the default message is used.\n *\n * \\param vertex [in] The vertex to send the message to\n * \\param message [in] The message to send, defaults to message_type(). \n */\n void signal(vertex_id_type gvid, \n const message_type& message = message_type()) {\n engine.internal_signal_gvid(gvid, message);\n }\n\n\n /**\n * \\internal\n * \\brief Flags that this vertex was synchronized.\n */\n void set_synchronized() {\n if (vtx_set && graph.l_is_master(vtx.local_id())) {\n original_value = serialize_to_string(vtx.data());\n }\n }\n\n /**\n * \\brief Synchronizes all copies of this vertex\n * \n * If the current vertex value has changed, copy the vertex value to\n * all mirrors. This is for advanced use!\n * Under most circumstances you should not need to use \n * this function directly.\n */\n void synchronize() {\n if (vtx_set && graph.l_is_master(vtx.local_id())) {\n std::string new_value = serialize_to_string(vtx.data());\n if (original_value != new_value) {\n // synchronize this vertex's value\n engine.synchronize_one_vertex_wait(vtx);\n }\n std::swap(original_value, new_value);\n }\n }\n };\n\n /**\n * The type of the context.\n */\n typedef context context_type;\n\n /// \\internal\n typedef context icontext_type;\n\n /// The type of the update function\n typedef boost::function<void(context_type&, vertex_type)> update_function_type;\n\n private:\n /// \\internal \\brief The base type of all schedulers\n message_array<message_type> messages;\n\n // context needs access to internal functions\n friend struct context;\n\n /// \\internal \\brief The type used to refer to vertices in the local graph\n typedef typename graph_type::local_vertex_type local_vertex_type;\n /// \\internal \\brief The type used to refer to edges in the local graph\n typedef typename graph_type::local_edge_type local_edge_type;\n /// \\internal \\brief The type used to refer to vertex IDs in the local graph\n typedef typename graph_type::lvid_type lvid_type;\n\n /// \\internal \\brief The type of the current engine instantiation\n typedef warp_engine engine_type;\n\n \n /// The RPC interface\n dc_dist_object<warp_engine> rmi;\n\n /// A reference to the active graph\n graph_type& graph;\n\n /// A pointer to the lock implementation\n distributed_chandy_misra<graph_type>* cmlocks;\n\n /// Per vertex data locks\n std::vector<simple_spinlock> vertexlocks;\n\n\n /**\n * \\brief A bit indicating if the local gather for that vertex is\n * available.\n */\n dense_bitset has_cache;\n\n /// Engine threads.\n fiber_group thrgroup;\n\n //! The scheduler\n ischeduler* scheduler_ptr;\n\n typedef distributed_aggregator<graph_type, context_type> aggregator_type;\n aggregator_type aggregator;\n\n /// Number of kernel threads\n size_t ncpus;\n /// Size of each fiber stack\n size_t stacksize;\n /// Number of fibers\n size_t nfibers;\n /// set to true if engine is started\n bool started;\n /// A pointer to the distributed consensus object\n fiber_async_consensus* consensus;\n\n /**\n * Used only by the locking subsystem.\n * to allow the fiber to go to sleep when waiting for the locks to\n * be ready.\n */\n struct vertex_fiber_cm_handle {\n mutex lock;\n bool philosopher_ready;\n size_t fiber_handle;\n };\n std::vector<vertex_fiber_cm_handle*> cm_handles;\n\n dense_bitset program_running;\n dense_bitset hasnext;\n\n // Various counters.\n atomic<uint64_t> programs_executed;\n\n timer launch_timer;\n\n /// Defaults to (-1), defines a timeout\n size_t timed_termination;\n \n /// engine option. Sets to true if factorized consistency is used\n bool factorized_consistency;\n\n bool endgame_mode;\n\n /// Time when engine is started\n float engine_start_time;\n\n /// True when a force stop is triggered (possibly via a timeout)\n bool force_stop;\n\n graphlab_options opts_copy; // local copy of options to pass to\n // scheduler construction\n\n execution_status::status_enum termination_reason;\n\n std::vector<mutex> aggregation_lock;\n std::vector<std::deque<std::string> > aggregation_queue;\n\n update_function_type update_fn;\n public:\n\n /**\n * Constructs an asynchronous consistent distributed engine.\n * The number of threads to create are read from\n * \\ref graphlab_options::get_ncpus \"opts.get_ncpus()\". The scheduler to\n * construct is read from\n * \\ref graphlab_options::get_scheduler_type() \"opts.get_scheduler_type()\".\n * The default scheduler\n * is the queued_fifo scheduler. For details on the scheduler types\n * \\see scheduler_types\n *\n * See the <a href=#engineopts> main class documentation</a> for the\n * available engine options.\n *\n * \\param dc Distributed controller to associate with\n * \\param graph The graph to schedule over. The graph must be fully\n * constructed and finalized.\n * \\param opts A graphlab::graphlab_options object containing options and\n * parameters for the scheduler and the engine.\n */\n warp_engine(distributed_control &dc,\n graph_type& graph,\n const graphlab_options& opts = graphlab_options()) :\n rmi(dc, this), graph(graph), scheduler_ptr(NULL),\n aggregator(dc, graph, new context_type(*this, graph)), started(false),\n engine_start_time(timer::approx_time_seconds()), force_stop(false) {\n rmi.barrier();\n\n nfibers = 10000;\n stacksize = 16384;\n factorized_consistency = true;\n update_fn = NULL;\n timed_termination = (size_t)(-1);\n termination_reason = execution_status::UNSET;\n set_options(opts);\n initialize();\n rmi.barrier();\n }\n\n /** \\internal\n * For the warp engine to find the remote instances of this class\n */\n size_t get_rpc_obj_id() {\n return rmi.get_obj_id();\n }\n\n private:\n\n /**\n * \\internal\n * Configures the engine with the provided options.\n * The number of threads to create are read from\n * opts::get_ncpus(). The scheduler to construct is read from\n * graphlab_options::get_scheduler_type(). The default scheduler\n * is the queued_fifo scheduler. For details on the scheduler types\n * \\see scheduler_types\n */\n void set_options(const graphlab_options& opts) {\n rmi.barrier();\n ncpus = opts.get_ncpus();\n ASSERT_GT(ncpus, 0);\n aggregation_lock.resize(opts.get_ncpus());\n aggregation_queue.resize(opts.get_ncpus());\n std::vector<std::string> keys = opts.get_engine_args().get_option_keys();\n foreach(std::string opt, keys) {\n if (opt == \"timeout\") {\n opts.get_engine_args().get_option(\"timeout\", timed_termination);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: timeout = \" << timed_termination << std::endl;\n } else if (opt == \"factorized\") {\n opts.get_engine_args().get_option(\"factorized\", factorized_consistency);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: factorized = \" << factorized_consistency << std::endl;\n } else if (opt == \"nfibers\") {\n opts.get_engine_args().get_option(\"nfibers\", nfibers);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: nfibers = \" << nfibers << std::endl;\n } else if (opt == \"stacksize\") {\n opts.get_engine_args().get_option(\"stacksize\", stacksize);\n if (rmi.procid() == 0)\n logstream(LOG_EMPH) << \"Engine Option: stacksize= \" << stacksize << std::endl;\n } else {\n logstream(LOG_FATAL) << \"Unexpected Engine Option: \" << opt << std::endl;\n }\n }\n opts_copy = opts;\n // set a default scheduler if none\n if (opts_copy.get_scheduler_type() == \"\") {\n opts_copy.set_scheduler_type(\"queued_fifo\");\n }\n\n // construct scheduler passing in the copy of the options from set_options\n scheduler_ptr = scheduler_factory::\n new_scheduler(graph.num_local_vertices(),\n opts_copy);\n rmi.barrier();\n\n // create initial fork arrangement based on the alternate vid mapping\n if (factorized_consistency == false) {\n cmlocks = new distributed_chandy_misra<graph_type>(rmi.dc(), graph,\n boost::bind(&engine_type::lock_ready, this, _1));\n \n }\n else {\n cmlocks = NULL;\n }\n\n // construct the termination consensus object\n consensus = new fiber_async_consensus(rmi.dc(), nfibers);\n }\n\n /**\n * \\internal\n * Initializes the engine with respect to the associated graph.\n * This call will initialize all internal and scheduling datastructures.\n * This function must be called prior to any signal function.\n */\n void initialize() {\n // construct all the required datastructures\n // deinitialize performs the reverse\n graph.finalize();\n scheduler_ptr->set_num_vertices(graph.num_local_vertices());\n messages.resize(graph.num_local_vertices());\n vertexlocks.resize(graph.num_local_vertices());\n program_running.resize(graph.num_local_vertices());\n hasnext.resize(graph.num_local_vertices());\n \n if (!factorized_consistency) {\n cm_handles.resize(graph.num_local_vertices());\n }\n rmi.barrier();\n }\n\n\n\n public:\n ~warp_engine() {\n delete consensus;\n delete cmlocks;\n delete scheduler_ptr;\n }\n\n\n /**\n * Sets the update function to use for execution.\n * The update function must be of the type void(context_type&, vertex_type),\n * but more generally, may be a \n * boost::function<void(context_type&, vertex_type)>\n */\n void set_update_function(update_function_type update_function) {\n update_fn = update_function;\n }\n\n\n /**\n * \\brief Compute the total number of updates (calls to apply)\n * executed since start was last invoked.\n *\n * \\return Total number of updates\n */\n size_t num_updates() const {\n return programs_executed.value;\n }\n\n\n\n\n\n /**\n * \\brief Get the elapsed time in seconds since start was last\n * called.\n * \n * \\return elapsed time in seconds\n */\n float elapsed_seconds() const {\n return timer::approx_time_seconds() - engine_start_time;\n }\n\n\n /**\n * \\brief Not meaningful for the asynchronous engine. Returns -1.\n */\n int iteration() const { return -1; }\n\n\n/**************************************************************************\n * Signaling Interface *\n **************************************************************************/\n\n private:\n\n /**\n * \\internal\n * This is used to receive a message forwarded from another machine\n */\n void rpc_signal(vertex_id_type vid,\n const message_type& message) {\n if (force_stop) return;\n const lvid_type local_vid = graph.local_vid(vid);\n double priority;\n messages.add(local_vid, message, &priority);\n scheduler_ptr->schedule(local_vid, priority);\n consensus->cancel();\n }\n\n /**\n * \\internal\n * \\brief Signals a vertex with an optional message\n *\n * Signals a vertex, and schedules it to be executed in the future.\n * must be called on a vertex accessible by the current machine.\n */\n void internal_signal(const vertex_type& vtx,\n const message_type& message = message_type()) {\n if (force_stop) return;\n if (started) {\n const typename graph_type::vertex_record& rec = graph.l_get_vertex_record(vtx.local_id());\n const procid_t owner = rec.owner;\n if (endgame_mode) {\n // fast signal. push to the remote machine immediately\n if (owner != rmi.procid()) {\n const vertex_id_type vid = rec.gvid;\n rmi.remote_call(owner, &engine_type::rpc_signal, vid, message);\n }\n else {\n double priority;\n messages.add(vtx.local_id(), message, &priority);\n scheduler_ptr->schedule(vtx.local_id(), priority);\n consensus->cancel();\n }\n }\n else {\n\n double priority;\n messages.add(vtx.local_id(), message, &priority);\n scheduler_ptr->schedule(vtx.local_id(), priority);\n consensus->cancel();\n }\n }\n else {\n double priority;\n messages.add(vtx.local_id(), message, &priority);\n scheduler_ptr->schedule(vtx.local_id(), priority);\n consensus->cancel();\n }\n } // end of schedule\n\n\n /**\n * \\internal\n * \\brief Signals a vertex with an optional message\n *\n * Signals a global vid, and schedules it to be executed in the future.\n * If current machine does not contain the vertex, it is ignored.\n */\n void internal_signal_gvid(vertex_id_type gvid,\n const message_type& message = message_type()) {\n if (force_stop) return;\n if (graph.is_master(gvid)) {\n internal_signal(graph.vertex(gvid), message);\n } else {\n procid_t proc = graph.master(gvid);\n rmi.remote_call(proc, &warp_engine::internal_signal,\n gvid, message);\n }\n } \n\n\n\n void rpc_internal_stop() {\n force_stop = true;\n termination_reason = execution_status::FORCED_ABORT;\n }\n\n /**\n * \\brief Force engine to terminate immediately.\n *\n * This function is used to stop the engine execution by forcing\n * immediate termination.\n */\n void internal_stop() {\n for (procid_t i = 0;i < rmi.numprocs(); ++i) {\n rmi.remote_call(i, &warp_engine::rpc_internal_stop);\n }\n }\n\n public:\n\n\n /**\n * \\brief Signals single a vertex with an optional message.\n * \n * This function sends a message to particular vertex which will\n * receive that message on start. The signal function must be\n * invoked on all machines simultaneously. For example:\n *\n * \\code\n * graphlab::warp_engine<graph_type> engine(dc, graph, opts);\n * engine.signal(0); // signal vertex zero\n * \\endcode\n *\n * and _not_:\n *\n * \\code\n * graphlab::warp_engine<graph_type> engine(dc, graph, opts);\n * if(dc.procid() == 0) engine.signal(0); // signal vertex zero\n * \\endcode\n *\n * Since signal is executed synchronously on all machines it\n * should only be used to schedule a small set of vertices. The\n * preferred method to signal a large set of vertices (e.g., all\n * vertices that are a certain type) is to use either the vertex\n * program init function or the aggregation framework. For\n * example to signal all vertices that have a particular value one\n * could write:\n *\n * \\code\n * struct bipartite_opt : \n * public graphlab::ivertex_program<graph_type, gather_type> {\n * // The user defined init function\n * void init(icontext_type& context, vertex_type& vertex) {\n * // Signal myself if I am a certain type\n * if(vertex.data().on_left) context.signal(vertex);\n * }\n * // other vastly more interesting code\n * };\n * \\endcode\n *\n * @param [in] vid the vertex id to signal\n * @param [in] message the message to send to that vertex. The\n * default message is sent if no message is provided. \n * (See ivertex_program::message_type for details about the\n * message_type). \n */\n void signal(vertex_id_type gvid,\n const message_type& message = message_type()) {\n rmi.barrier();\n internal_signal_gvid(gvid, message);\n rmi.barrier();\n }\n\n /**\n * \\brief Signal all vertices with a particular message.\n * \n * This function sends the same message to all vertices which will\n * receive that message on start. The signal_all function must be\n * invoked on all machines simultaneously. For example:\n *\n * \\code\n * graphlab::warp_engine<graph_type> engine(dc, graph, opts);\n * engine.signal_all(); // signal all vertices\n * \\endcode\n *\n * and _not_:\n *\n * \\code\n * graphlab::warp_engine<graph_type> engine(dc, graph, opts);\n * if(dc.procid() == 0) engine.signal_all(); // signal vertex zero\n * \\endcode\n *\n * The signal_all function is the most common way to send messages\n * to the engine. For example in the pagerank application we want\n * all vertices to be active on the first round. Therefore we\n * would write:\n *\n * \\code\n * graphlab::warp_engine<graph_type> engine(dc, graph, opts);\n * engine.signal_all();\n * engine.start();\n * \\endcode\n *\n * @param [in] message the message to send to all vertices. The\n * default message is sent if no message is provided\n * (See ivertex_program::message_type for details about the\n * message_type). \n */\n void signal_all(const message_type& message = message_type(),\n const std::string& order = \"shuffle\") {\n vertex_set vset = graph.complete_set();\n signal_vset(vset, message, order);\n } // end of schedule all\n\n\n /**\n * \\brief Signal a set of vertices with a particular message.\n * \n * This function sends the same message to a set of vertices which will\n * receive that message on start. The signal_vset function must be\n * invoked on all machines simultaneously. For example:\n *\n * \\code\n * graphlab::warp_engine<graph_type> engine(dc, graph, opts);\n * engine.signal_vset(vset); // signal a subset of vertices\n * \\endcode\n *\n * signal_all() is conceptually equivalent to:\n *\n * \\code\n * engine.signal_vset(graph.complete_set());\n * \\endcode\n *\n * @param [in] vset The set of vertices to signal \n * @param [in] message the message to send to all vertices. The\n * default message is sent if no message is provided\n * (See ivertex_program::message_type for details about the\n * message_type). \n */\n void signal_vset(const vertex_set& vset,\n const message_type& message = message_type(),\n const std::string& order = \"shuffle\") {\n logstream(LOG_DEBUG) << rmi.procid() << \": Schedule All\" << std::endl;\n // allocate a vector with all the local owned vertices\n // and schedule all of them.\n std::vector<vertex_id_type> vtxs;\n vtxs.reserve(graph.num_local_own_vertices());\n for(lvid_type lvid = 0;\n lvid < graph.get_local_graph().num_vertices();\n ++lvid) {\n if (graph.l_vertex(lvid).owner() == rmi.procid() &&\n vset.l_contains(lvid)) {\n vtxs.push_back(lvid);\n }\n }\n\n if(order == \"shuffle\") {\n graphlab::random::shuffle(vtxs.begin(), vtxs.end());\n }\n foreach(lvid_type lvid, vtxs) {\n double priority;\n messages.add(lvid, message, &priority);\n scheduler_ptr->schedule(lvid, priority);\n }\n rmi.barrier();\n }\n\n\n private: \n\n /**\n * Gets a task from the scheduler and the associated message\n */\n sched_status::status_enum get_next_sched_task(size_t threadid,\n lvid_type& lvid,\n message_type& msg) {\n while (1) {\n sched_status::status_enum stat = \n scheduler_ptr->get_next(threadid % ncpus, lvid);\n if (stat == sched_status::NEW_TASK) {\n if (messages.get(lvid, msg)) return stat;\n else continue;\n }\n return stat;\n }\n }\n\n void set_endgame_mode() {\n if (!endgame_mode) logstream(LOG_EMPH) << \"Endgame mode\\n\";\n endgame_mode = true;\n rmi.dc().set_fast_track_requests(true);\n } \n\n /**\n * \\internal\n * Called when get_a_task returns no internal task not a scheduler task.\n * This rechecks the status of the internal task queue and the scheduler\n * inside a consensus critical section.\n */\n bool try_to_quit(size_t threadid,\n bool& has_sched_msg,\n lvid_type& sched_lvid,\n message_type &msg) {\n if (timer::approx_time_seconds() - engine_start_time > timed_termination) {\n termination_reason = execution_status::TIMEOUT;\n force_stop = true;\n }\n logstream(LOG_DEBUG) << rmi.procid() << \"-\" << threadid << \": \" << \"Termination Attempt \" << std::endl;\n has_sched_msg = false;\n fiber_control::yield();\n consensus->begin_done_critical_section(threadid);\n sched_status::status_enum stat = \n get_next_sched_task(threadid, sched_lvid, msg);\n if (stat == sched_status::EMPTY || force_stop) {\n logstream(LOG_DEBUG) << rmi.procid() << \"-\" << threadid << \": \"\n << \"\\tTermination Double Checked\" << std::endl;\n\n if (!endgame_mode) logstream(LOG_EMPH) << \"Endgame mode\\n\";\n endgame_mode = true;\n // put everyone in endgame\n for (procid_t i = 0;i < rmi.dc().numprocs(); ++i) {\n rmi.remote_call(i, &warp_engine::set_endgame_mode);\n } \n bool ret = consensus->end_done_critical_section(threadid);\n if (ret == false) {\n logstream(LOG_DEBUG) << rmi.procid() << \"-\" << threadid << \": \"\n << \"\\tCancelled\" << std::endl;\n } else {\n logstream(LOG_DEBUG) << rmi.procid() << \"-\" << threadid << \": \"\n << \"\\tDying\" << \" (\" << fiber_control::get_tid() << \")\" << std::endl;\n }\n return ret;\n } else {\n logstream(LOG_DEBUG) << rmi.procid() << \"-\" << threadid << \": \"\n << \"\\tCancelled by Scheduler Task\" << std::endl;\n consensus->cancel_critical_section(threadid);\n has_sched_msg = true;\n return false;\n }\n } // end of try to quit\n\n\n /**\n * \\internal\n * When all distributed locks are acquired, this function is called\n * from the chandy misra implementation on the master vertex.\n * Here, we perform initialization\n * of the task and switch the vertex to a gathering state\n */\n void lock_ready(lvid_type lvid) {\n cm_handles[lvid]->lock.lock();\n cm_handles[lvid]->philosopher_ready = true;\n fiber_control::schedule_tid(cm_handles[lvid]->fiber_handle);\n cm_handles[lvid]->lock.unlock();\n }\n\n\n // make sure I am the only person running.\n // if returns false, the message has been dropped into the message array.\n // quit\n bool get_exclusive_access_to_vertex(const lvid_type lvid,\n const message_type& msg) {\n vertexlocks[lvid].lock();\n bool someone_else_running = program_running.set_bit(lvid);\n if (someone_else_running) {\n // bad. someone else is here.\n // drop it into the message array\n messages.add(lvid, msg);\n hasnext.set_bit(lvid);\n } \n vertexlocks[lvid].unlock();\n return !someone_else_running;\n }\n\n\n\n // make sure I am the only person running.\n // if returns false, the message has been dropped into the message array.\n // quit\n void release_exclusive_access_to_vertex(const lvid_type lvid) {\n vertexlocks[lvid].lock();\n // someone left a next message for me\n // reschedule it at high priority\n if (hasnext.get(lvid)) {\n scheduler_ptr->schedule(lvid, 10000.0);\n consensus->cancel();\n hasnext.clear_bit(lvid);\n }\n program_running.clear_bit(lvid);\n vertexlocks[lvid].unlock();\n }\n\n void update_vertex_value(vertex_id_type vid,\n vertex_data_type& vdata) {\n local_vertex_type lvtx(graph.l_vertex(graph.local_vid(vid)));\n lvtx.data() = vdata;\n }\n\n void synchronize_one_vertex(vertex_type vtx) {\n local_vertex_type lvtx(vtx);\n foreach(procid_t mirror, lvtx.mirrors()) {\n rmi.remote_call(mirror, &warp_engine::update_vertex_value, vtx.id(), vtx.data());\n }\n }\n\n\n void synchronize_one_vertex_wait(vertex_type vtx) {\n local_vertex_type lvtx(vtx);\n std::vector<request_future<void> > futures;\n foreach(procid_t mirror, lvtx.mirrors()) {\n futures.push_back(object_fiber_remote_request(rmi, \n mirror, \n &warp_engine::update_vertex_value, \n vtx.id(), \n vtx.data()));\n }\n for (size_t i = 0;i < futures.size(); ++i) {\n futures[i]();\n }\n }\n\n /**\n * \\internal\n * Called when the scheduler returns a vertex to run.\n * If this function is called with vertex locks acquired, prelocked\n * should be true. Otherwise it should be false.\n */\n void eval_sched_task(const lvid_type lvid,\n const message_type& msg) {\n const typename graph_type::vertex_record& rec = graph.l_get_vertex_record(lvid);\n vertex_id_type vid = rec.gvid;\n // if this is another machine's forward it\n if (rec.owner != rmi.procid()) {\n rmi.remote_call(rec.owner, &engine_type::rpc_signal, vid, msg);\n return;\n }\n // I have to run this myself\n \n if (!get_exclusive_access_to_vertex(lvid, msg)) return;\n\n /**************************************************************************/\n /* Acquire Locks */\n /**************************************************************************/\n if (!factorized_consistency) {\n // begin lock acquisition\n cm_handles[lvid] = new vertex_fiber_cm_handle;\n cm_handles[lvid]->philosopher_ready = false;\n cm_handles[lvid]->fiber_handle = fiber_control::get_tid();\n cmlocks->make_philosopher_hungry(lvid);\n cm_handles[lvid]->lock.lock();\n while (!cm_handles[lvid]->philosopher_ready) {\n fiber_control::deschedule_self(&(cm_handles[lvid]->lock.m_mut));\n cm_handles[lvid]->lock.lock();\n }\n cm_handles[lvid]->lock.unlock();\n }\n\n \n\n local_vertex_type l_vtx(graph.l_vertex(lvid));\n local_vertex_type vtx(l_vtx);\n\n\n context ctx(*this, graph, vtx);\n update_fn(ctx, vtx);\n ctx.synchronize();\n /************************************************************************/\n /* Release Locks */\n /************************************************************************/\n // cleanup\n if (!factorized_consistency) {\n cmlocks->philosopher_stops_eating(lvid);\n delete cm_handles[lvid];\n cm_handles[lvid] = NULL;\n }\n release_exclusive_access_to_vertex(lvid);\n programs_executed.inc(); \n }\n\n\n /**\n * \\internal\n * Per thread main loop\n */\n void thread_start(size_t threadid) {\n bool has_sched_msg = false;\n std::vector<std::vector<lvid_type> > internal_lvid;\n lvid_type sched_lvid;\n\n message_type msg;\n float last_aggregator_check = timer::approx_time_seconds();\n timer ti; ti.start();\n while(1) {\n if (timer::approx_time_seconds() != last_aggregator_check && !endgame_mode) {\n last_aggregator_check = timer::approx_time_seconds();\n std::string key = aggregator.tick_asynchronous();\n if (key != \"\") {\n for (size_t i = 0;i < aggregation_lock.size(); ++i) {\n aggregation_lock[i].lock();\n aggregation_queue[i].push_back(key);\n aggregation_lock[i].unlock();\n }\n }\n }\n\n // test the aggregator\n while(!aggregation_queue[fiber_control::get_worker_id()].empty()) {\n size_t wid = fiber_control::get_worker_id();\n ASSERT_LT(wid, ncpus);\n aggregation_lock[wid].lock();\n std::string key = aggregation_queue[wid].front();\n aggregation_queue[wid].pop_front();\n aggregation_lock[wid].unlock();\n aggregator.tick_asynchronous_compute(wid, key);\n }\n\n sched_status::status_enum stat = get_next_sched_task(threadid, sched_lvid, msg);\n\n\n has_sched_msg = stat != sched_status::EMPTY;\n if (stat != sched_status::EMPTY) {\n eval_sched_task(sched_lvid, msg);\n if (endgame_mode) rmi.dc().flush();\n }\n else if (!try_to_quit(threadid, has_sched_msg, sched_lvid, msg)) {\n /*\n * We failed to obtain a task, try to quit\n */\n if (has_sched_msg) {\n eval_sched_task(sched_lvid, msg);\n }\n } else { \n break; \n }\n if (fiber_control::worker_has_priority_fibers_on_queue()) fiber_control::yield();\n }\n } // end of thread start\n\n/**************************************************************************\n * Main engine start() *\n **************************************************************************/\n\n public:\n\n\n /**\n * \\brief Start the engine execution.\n *\n * This function starts the engine and does not\n * return until the scheduler has no tasks remaining.\n *\n * \\return the reason for termination\n */\n execution_status::status_enum start() {\n bool old_fasttrack = rmi.dc().set_fast_track_requests(false);\n logstream(LOG_INFO) << \"Spawning \" << nfibers << \" threads\" << std::endl;\n ASSERT_TRUE(scheduler_ptr != NULL);\n consensus->reset();\n\n // now. It is of critical importance that we match the number of \n // actual workers\n \n\n // start the aggregator\n aggregator.start(ncpus);\n aggregator.aggregate_all_periodic();\n\n started = true;\n\n rmi.barrier();\n size_t allocatedmem = memory_info::allocated_bytes();\n rmi.all_reduce(allocatedmem);\n\n engine_start_time = timer::approx_time_seconds();\n force_stop = false;\n endgame_mode = false;\n programs_executed = 0;\n launch_timer.start();\n\n termination_reason = execution_status::RUNNING;\n if (rmi.procid() == 0) {\n logstream(LOG_INFO) << \"Total Allocated Bytes: \" << allocatedmem << std::endl;\n }\n fiber_group::affinity_type affinity;\n affinity.clear();\n for (size_t i = 0; i < ncpus; ++i) {\n affinity.set_bit(i);\n }\n thrgroup.set_affinity(affinity);\n thrgroup.set_stacksize(stacksize);\n\n for (size_t i = 0; i < nfibers ; ++i) {\n thrgroup.launch(boost::bind(&engine_type::thread_start, this, i));\n }\n thrgroup.join();\n aggregator.stop();\n // if termination reason was not changed, then it must be depletion\n if (termination_reason == execution_status::RUNNING) {\n termination_reason = execution_status::TASK_DEPLETION;\n }\n\n size_t ctasks = programs_executed.value;\n rmi.all_reduce(ctasks);\n programs_executed.value = ctasks;\n\n rmi.cout() << \"Completed Tasks: \" << programs_executed.value << std::endl;\n\n\n size_t numjoins = messages.num_joins();\n rmi.all_reduce(numjoins);\n rmi.cout() << \"Schedule Joins: \" << numjoins << std::endl;\n\n size_t numadds = messages.num_adds();\n rmi.all_reduce(numadds);\n rmi.cout() << \"Schedule Adds: \" << numadds << std::endl;\n\n\n ASSERT_TRUE(scheduler_ptr->empty());\n started = false;\n\n rmi.dc().set_fast_track_requests(old_fasttrack);\n return termination_reason;\n } // end of start\n\n\n public:\n aggregator_type* get_aggregator() { return &aggregator; }\n\n }; // end of class\n\n} // namespace warp\n} // namespace\n\n#include <graphlab/macros_undef.hpp>\n#include <graphlab/engine/warp_graph_broadcast.hpp>\n#include <graphlab/engine/warp_graph_mapreduce.hpp>\n#include <graphlab/engine/warp_graph_transform.hpp>\n#endif \n\n" }, { "path": "src/graphlab/engine/warp_graph_broadcast.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_WARP_GRAPH_BROADCAST_HPP\n#define GRAPHLAB_WARP_GRAPH_BROADCAST_HPP\n\n#include <boost/bind.hpp>\n#include <graphlab/util/generics/conditional_combiner_wrapper.hpp>\n#include <graphlab/parallel/fiber_group.hpp>\n#include <graphlab/parallel/fiber_control.hpp>\n#include <graphlab/parallel/fiber_remote_request.hpp>\n#include <graphlab/logger/assertions.hpp>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n\nnamespace warp {\n\nnamespace warp_impl {\n\ntemplate <typename EngineType, typename GraphType>\nstruct broadcast_neighborhood_impl {\n\n typedef typename EngineType::context_type context_type;\n typedef typename GraphType::vertex_type vertex_type;\n typedef typename GraphType::vertex_data_type vertex_data_type;\n typedef typename GraphType::edge_type edge_type;\n typedef typename GraphType::local_vertex_type local_vertex_type;\n typedef typename GraphType::local_edge_type local_edge_type;\n typedef typename GraphType::vertex_record vertex_record;\n\n\n/**************************************************************************/\n/* */\n/* Basic MapReduce Neighborhood Implementation */\n/* */\n/**************************************************************************/\n/*\n * The master calls basic_mapreduce_neighborhood.\n * Which then issues calls to basic_local_mapper on each machine with a replica.\n */\n\n static void basic_local_broadcast_neighborhood(context_type& context, \n edge_dir_type edge_direction,\n void(*broadcast_fn)(context_type& context,\n edge_type edge,\n vertex_type other),\n vertex_id_type vid) {\n GraphType& graph(context.graph);\n lvid_type lvid = graph.local_vid(vid);\n local_vertex_type local_vertex(context.graph.l_vertex(lvid));\n \n if(edge_direction == IN_EDGES || edge_direction == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.in_edges()) {\n edge_type edge(local_edge);\n vertex_type other(local_edge.source());\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n graph.get_lock_manager()[std::min(a,b)].lock();\n graph.get_lock_manager()[std::max(a,b)].lock();\n broadcast_fn(context, edge, other);\n graph.get_lock_manager()[a].unlock();\n graph.get_lock_manager()[b].unlock();\n }\n } \n // do out edges\n if(edge_direction == OUT_EDGES || edge_direction == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.out_edges()) {\n edge_type edge(local_edge);\n vertex_type other(local_edge.target());\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n graph.get_lock_manager()[std::min(a,b)].lock();\n graph.get_lock_manager()[std::max(a,b)].lock();\n broadcast_fn(context, edge, other);\n graph.get_lock_manager()[a].unlock();\n graph.get_lock_manager()[b].unlock();\n }\n } \n }\n\n\n static void basic_local_broadcast_neighborhood_from_remote(std::pair<size_t, size_t> objid,\n edge_dir_type edge_direction,\n size_t broadcast_ptr,\n vertex_id_type vid,\n vertex_data_type& vdata) {\n EngineType* engine = reinterpret_cast<EngineType*>(distributed_control::get_instance()->get_registered_object(objid.first));\n GraphType* graph = reinterpret_cast<GraphType*>(distributed_control::get_instance()->get_registered_object(objid.second));\n vertex_type vertex(graph->l_vertex(graph->local_vid(vid)));\n context_type context(*engine, *graph, vertex);\n vertex.data() = vdata;\n // cast the mappers and combiners back into their pointer types\n void(*broadcast_fn)(context_type&, edge_type edge, vertex_type other) = \n reinterpret_cast<void(*)(context_type&, edge_type, vertex_type)>(broadcast_ptr);\n basic_local_broadcast_neighborhood(\n context,\n edge_direction,\n broadcast_fn,\n vid);\n }\n\n static void basic_broadcast_neighborhood(context_type& context,\n typename GraphType::vertex_type current,\n edge_dir_type edge_direction,\n void(*broadcast_fn)(context_type& context, edge_type edge, vertex_type other)) {\n // get a reference to the graph\n GraphType& graph = current.graph_ref;\n // get the object ID of the graph\n std::pair<size_t, size_t> objid(context.engine.get_rpc_obj_id(), graph.get_rpc_obj_id());\n typename GraphType::vertex_record vrecord = graph.l_get_vertex_record(current.local_id());\n\n // make sure we are running on a master vertex\n ASSERT_EQ(vrecord.owner, distributed_control::get_instance_procid());\n \n // create num-mirrors worth of requests\n std::vector<request_future<void > > requests(vrecord.num_mirrors());\n \n size_t ctr = 0;\n foreach(procid_t proc, vrecord.mirrors()) {\n // issue the communication\n requests[ctr] = fiber_remote_request(proc, \n basic_local_broadcast_neighborhood_from_remote,\n objid,\n edge_direction,\n reinterpret_cast<size_t>(broadcast_fn),\n current.id(),\n current.data());\n ++ctr;\n }\n // compute the local tasks\n basic_local_broadcast_neighborhood(context,\n edge_direction, \n broadcast_fn, \n current.id());\n // now, wait for everyone\n for (size_t i = 0;i < requests.size(); ++i) {\n requests[i]();\n }\n }\n};\n\n\ntemplate <typename EngineType, typename GraphType, typename ExtraArg>\nstruct broadcast_neighborhood_impl2 {\n\n typedef typename EngineType::context_type context_type;\n typedef typename GraphType::vertex_type vertex_type;\n typedef typename GraphType::vertex_data_type vertex_data_type;\n typedef typename GraphType::edge_type edge_type;\n typedef typename GraphType::local_vertex_type local_vertex_type;\n typedef typename GraphType::local_edge_type local_edge_type;\n typedef typename GraphType::vertex_record vertex_record;\n\n/**************************************************************************/\n/* */\n/* Extended MapReduce Neighborhood Implementation */\n/* */\n/**************************************************************************/\n/*\n * The master calls extended_mapreduce_neighborhood.\n * Which then issues calls to extended_local_mapper on each machine with a replica.\n * The extended mapreduce neighborhood allows the mapper and combiner to take\n * an optional argument\n */\n\n\n static void extended_local_broadcast_neighborhood(context_type& context,\n edge_dir_type edge_direction,\n void(*broadcast_fn)(context_type& context,\n edge_type edge,\n vertex_type other,\n const ExtraArg extra),\n vertex_id_type vid,\n const ExtraArg extra) {\n GraphType& graph(context.graph);\n lvid_type lvid = graph.local_vid(vid);\n local_vertex_type local_vertex(graph.l_vertex(lvid));\n \n if(edge_direction == IN_EDGES || edge_direction == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.in_edges()) {\n edge_type edge(local_edge);\n vertex_type other(local_edge.source());\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n graph.get_lock_manager()[std::min(a,b)].lock();\n graph.get_lock_manager()[std::max(a,b)].lock();\n broadcast_fn(context, edge, other, extra);\n graph.get_lock_manager()[a].unlock();\n graph.get_lock_manager()[b].unlock();\n }\n } \n // do out edges\n if(edge_direction == OUT_EDGES || edge_direction == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.out_edges()) {\n edge_type edge(local_edge);\n vertex_type other(local_edge.target());\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n graph.get_lock_manager()[std::min(a,b)].lock();\n graph.get_lock_manager()[std::max(a,b)].lock();\n broadcast_fn(context, edge, other, extra);\n graph.get_lock_manager()[a].unlock();\n graph.get_lock_manager()[b].unlock();\n }\n } \n }\n\n\n static void extended_local_broadcast_neighborhood_from_remote(std::pair<size_t, size_t> objid,\n edge_dir_type edge_direction,\n size_t broadcast_ptr,\n vertex_id_type vid,\n vertex_data_type& vdata,\n const ExtraArg extra) {\n\n EngineType* engine = reinterpret_cast<EngineType*>(distributed_control::get_instance()->get_registered_object(objid.first));\n GraphType* graph = reinterpret_cast<GraphType*>(distributed_control::get_instance()->get_registered_object(objid.second));\n vertex_type vertex(graph->l_vertex(graph->local_vid(vid)));\n context_type context(*engine, *graph, vertex);\n vertex.data() = vdata;\n // cast the mappers and combiners back into their pointer types\n void(*broadcast_fn)(context_type&, edge_type edge, vertex_type other, const ExtraArg) = \n reinterpret_cast<void(*)(context_type&, edge_type, vertex_type, const ExtraArg)>(broadcast_ptr);\n extended_local_broadcast_neighborhood(\n context,\n edge_direction,\n broadcast_fn,\n vid,\n extra);\n }\n\n static void extended_broadcast_neighborhood(context_type& context,\n typename GraphType::vertex_type current,\n edge_dir_type edge_direction,\n void(*broadcast_fn)(context_type& context, edge_type edge, vertex_type other, const ExtraArg extra),\n const ExtraArg extra) {\n // get a reference to the graph\n GraphType& graph = current.graph_ref;\n // get the object ID of the graph\n std::pair<size_t, size_t> objid(context.engine.get_rpc_obj_id(), graph.get_rpc_obj_id());\n typename GraphType::vertex_record vrecord = graph.l_get_vertex_record(current.local_id());\n\n // make sure we are running on a master vertex\n ASSERT_EQ(vrecord.owner, distributed_control::get_instance_procid());\n \n // create num-mirrors worth of requests\n std::vector<request_future<void> > requests(vrecord.num_mirrors());\n \n size_t ctr = 0;\n foreach(procid_t proc, vrecord.mirrors()) {\n // issue the communication\n requests[ctr] = fiber_remote_request(proc, \n extended_local_broadcast_neighborhood_from_remote,\n objid,\n edge_direction,\n reinterpret_cast<size_t>(broadcast_fn),\n current.id(),\n current.data(),\n extra);\n ++ctr;\n }\n // compute the local tasks\n extended_local_broadcast_neighborhood(context, \n edge_direction, \n broadcast_fn, \n current.id(),\n extra);\n // now, wait for everyone\n for (size_t i = 0;i < requests.size(); ++i) {\n requests[i]();\n }\n }\n\n\n};\n\n} // namespace warp::warp_impl\n\n\n/**\n * \\ingroup warp\n *\n * the broadcast_neighborhood function allows a parallel transformation of the\n * neighborhood of a vertex to be performed and also provides a warp_engine\n * context. . This is a blocking operation, and\n * will not return until the distributed computation is complete. When run\n * inside a fiber, to hide latency, the system will automatically context\n * switch to evaluate some other fiber which is ready to run. This function \n * is functionally similar to the transform_neighborhood function, but requires\n * a \\ref graphlab::warp_engine \"warp_engine\" \n * context to be provided. The warp_engine context will also be passed on to\n * the transform function.\n *\n * Abstractly, the computation accomplishes the following:\n *\n * \\code\n * foreach(edge in neighborhood of current vertex) {\n * transform_fn(context, edge, opposite vertex)\n * }\n * \\endcode\n *\n * \\attention It is important that the transform_fn should only make\n * modifications to the edge data, and not the data on either of the vertices.\n *\n * \\attention Unlike the transform_neighborhood function, this call actually\n * performs synchronization, so the value of both vertex endpoints are\n * correct. \n *\n * Here is an example which schedules all vertices on out edges.\n * \n * \\code\n * void schedule(engine_type::context& context,\n * graph_type::edge_type edge, \n * graph_type::vertex_type other) {\n * context.signal(other);\n * }\n *\n *\n * void update_function(engine_type::context& context,\n * graph_type::vertex_type vertex) {\n * warp::broadcast_neighborhood(context,\n * vertex,\n * OUT_EDGES,\n * schedule);\n * }\n *\n * \\endcode\n *\n *\n * An overload is provided which allows you to pass an additional arbitrary\n * argument to the broadcast.\n *\n * \\param current The vertex to map reduce the neighborhood over\n * \\param edge_direction To run over all IN_EDGES, OUT_EDGES or ALL_EDGES\n * \\param transform_fn The transform function to run on all the selected edges.\n * \n * \\see warp_engine\n * \\see warp::parfor_all_vertices\n * \\see warp::map_reduce_neighborhood\n * \\see warp::transform_neighborhood\n */\ntemplate <typename ContextType, typename VertexType>\nvoid broadcast_neighborhood(ContextType& context,\n VertexType current,\n edge_dir_type edge_direction,\n void (*broadcast_fn)(ContextType& context,\n typename VertexType::graph_type::edge_type edge,\n VertexType other)) {\n warp_impl::\n broadcast_neighborhood_impl<typename ContextType::engine_type, typename VertexType::graph_type>::\n basic_broadcast_neighborhood(context, current, edge_direction, \n broadcast_fn);\n context.set_synchronized();\n}\n\n\n\n/**\n * \\ingroup warp\n *\n * the broadcast_neighborhood function allows a parallel transformation of the\n * neighborhood of a vertex to be performed and also provides a warp_engine\n * context. . This is a blocking operation, and\n * will not return until the distributed computation is complete. When run\n * inside a fiber, to hide latency, the system will automatically context\n * switch to evaluate some other fiber which is ready to run. This function \n * is functionally similar to the transform_neighborhood function, but requires\n * a \\ref graphlab::warp_engine \"warp_engine\" \n * context to be provided. The \\ref graphlab::warp_engine::context \n * \"warp_engine context\" will also be passed on to\n * the transform function.\n *\n * This is the more general overload of the broadcast_neighborhood function\n * which allows an additional arbitrary extra argument to be passed along\n * to the transform function\n *\n * Abstractly, the computation accomplishes the following:\n *\n * \\code\n * foreach(edge in neighborhood of current vertex) {\n * transform_fn(context, edge, opposite vertex, extraarg)\n * }\n * \\endcode\n *\n * \\attention It is important that the transform_fn should only make\n * modifications to the edge data, and not the data on either of the vertices.\n *\n * \\attention Unlike the transform_neighborhood function, this call actually\n * performs synchronization, so the value of both vertex endpoints are\n * correct. \n *\n * Here is an example which schedules all vertices on out edges with a \n * particular value\n * \\code\n * void schedule(engine_type::context& context,\n * graph_type::edge_type edge, \n * graph_type::vertex_type other,\n * int value) {\n * if (edge.data() == value) context.signal(other);\n * }\n *\n *\n * void update_function(engine_type::context& context,\n * graph_type::vertex_type vertex) {\n * // schedule all neighbors connected to an out edge \n * // with value 10\n * warp::broadcast_neighborhood(context,\n * vertex,\n * OUT_EDGES,\n * int(10),\n * schedule));\n * }\n *\n * \\endcode\n *\n * \\param current The vertex to map reduce the neighborhood over\n * \\param edge_direction To run over all IN_EDGES, OUT_EDGES or ALL_EDGES\n * \\param extra An additional argument to be passed to the broadcast\n * \\param transform_fn The transform function to run on all the selected edges.\n * \n * \\see warp_engine\n * \\see warp::parfor_all_vertices()\n * \\see warp::map_reduce_neighborhood()\n * \\see warp::transform_neighborhood()\n */\ntemplate <typename ContextType, typename ExtraArg, typename VertexType>\nvoid broadcast_neighborhood(ContextType& context,\n VertexType current,\n edge_dir_type edge_direction,\n void(*broadcast_fn)(ContextType& context,\n typename VertexType::graph_type::edge_type edge,\n VertexType other,\n const ExtraArg extra),\n const ExtraArg extra) {\n warp_impl::\n broadcast_neighborhood_impl2<typename ContextType::engine_type, typename VertexType::graph_type, ExtraArg>::\n extended_broadcast_neighborhood(context, current, edge_direction, broadcast_fn, extra);\n context.set_synchronized();\n}\n\n\n\n\n} // namespace warp\n\n} // namespace graphlab\n\n#include <graphlab/macros_undef.hpp>\n#endif\n" }, { "path": "src/graphlab/engine/warp_graph_mapreduce.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_WARP_GRAPH_MAP_REDUCE_HPP\n#define GRAPHLAB_WARP_GRAPH_MAP_REDUCE_HPP\n\n#include <boost/bind.hpp>\n#include <graphlab/util/generics/conditional_combiner_wrapper.hpp>\n#include <graphlab/parallel/fiber_group.hpp>\n#include <graphlab/parallel/fiber_control.hpp>\n#include <graphlab/parallel/fiber_remote_request.hpp>\n#include <graphlab/logger/assertions.hpp>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n\nnamespace warp {\n\nnamespace warp_impl {\n\n/**\n * The default combiner used for combining mapped results from\n * warp::map_reduce_neighborhood(); merges self with other using operator +=. \n */\ntemplate <typename T>\nvoid default_combiner(T& self, const T& other) {\n self += other;\n}\n\n/**\n * The default combiner used for combining mapped results from\n * warp::map_reduce_neighborhood() which\n * takes an unused argument; merges self with other using operator +=. \n */\ntemplate <typename T, typename ExtraArgs>\nvoid extended_default_combiner(T& self, const T& other, const ExtraArgs& unused) {\n self += other;\n}\n\n\ntemplate <typename RetType, typename GraphType>\nstruct map_reduce_neighborhood_impl {\n\n typedef typename GraphType::vertex_type vertex_type;\n typedef typename GraphType::edge_type edge_type;\n typedef typename GraphType::local_vertex_type local_vertex_type;\n typedef typename GraphType::local_edge_type local_edge_type;\n typedef typename GraphType::vertex_record vertex_record;\n\n\n/**************************************************************************/\n/* */\n/* Basic MapReduce Neighborhood Implementation */\n/* */\n/**************************************************************************/\n/*\n * The master calls basic_mapreduce_neighborhood.\n * Which then issues calls to basic_local_mapper on each machine with a replica.\n */\n\n static conditional_combiner_wrapper<RetType> basic_local_mapper(GraphType& graph,\n edge_dir_type edge_direction,\n RetType (*mapper)(edge_type edge, vertex_type other),\n void (*combiner)(RetType&, const RetType&),\n vertex_id_type vid) {\n lvid_type lvid = graph.local_vid(vid);\n local_vertex_type local_vertex(graph.l_vertex(lvid));\n \n conditional_combiner_wrapper<RetType> accum(combiner);\n if(edge_direction == IN_EDGES || edge_direction == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.in_edges()) {\n edge_type edge(local_edge);\n vertex_type other(local_edge.source());\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n graph.get_lock_manager()[std::min(a,b)].lock();\n graph.get_lock_manager()[std::max(a,b)].lock();\n accum += mapper(edge, other);\n graph.get_lock_manager()[a].unlock();\n graph.get_lock_manager()[b].unlock();\n }\n } \n // do out edges\n if(edge_direction == OUT_EDGES || edge_direction == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.out_edges()) {\n edge_type edge(local_edge);\n vertex_type other(local_edge.target());\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n graph.get_lock_manager()[std::min(a,b)].lock();\n graph.get_lock_manager()[std::max(a,b)].lock();\n accum += mapper(edge, other);\n graph.get_lock_manager()[a].unlock();\n graph.get_lock_manager()[b].unlock();\n }\n } \n return accum;\n }\n\n\n static conditional_combiner_wrapper<RetType> basic_local_mapper_from_remote(size_t objid,\n edge_dir_type edge_direction,\n size_t mapper_ptr,\n size_t combiner_ptr,\n vertex_id_type vid) {\n // cast the mappers and combiners back into their pointer types\n RetType (*mapper)(edge_type edge, vertex_type other) = \n reinterpret_cast<RetType(*)(edge_type, vertex_type)>(mapper_ptr);\n void (*combiner)(RetType&, const RetType&) = \n reinterpret_cast<void (*)(RetType&, const RetType&)>(combiner_ptr);\n return basic_local_mapper(\n *reinterpret_cast<GraphType*>(distributed_control::get_instance()->get_registered_object(objid)),\n edge_direction,\n mapper,\n combiner,\n vid);\n }\n\n static RetType basic_map_reduce_neighborhood(typename GraphType::vertex_type current,\n edge_dir_type edge_direction,\n RetType (*mapper)(edge_type edge,\n vertex_type other),\n void (*combiner)(RetType& self, \n const RetType& other)) {\n // get a reference to the graph\n GraphType& graph = current.graph_ref;\n // get the object ID of the graph\n size_t objid = graph.get_rpc_obj_id();\n typename GraphType::vertex_record vrecord = graph.l_get_vertex_record(current.local_id());\n\n // make sure we are running on a master vertex\n ASSERT_EQ(vrecord.owner, distributed_control::get_instance_procid());\n \n // create num-mirrors worth of requests\n std::vector<request_future<conditional_combiner_wrapper<RetType> > > requests(vrecord.num_mirrors());\n \n size_t ctr = 0;\n foreach(procid_t proc, vrecord.mirrors()) {\n // issue the communication\n requests[ctr] = fiber_remote_request(proc, \n map_reduce_neighborhood_impl<RetType, GraphType>::basic_local_mapper_from_remote,\n objid,\n edge_direction,\n reinterpret_cast<size_t>(mapper),\n reinterpret_cast<size_t>(combiner),\n current.id());\n ++ctr;\n }\n // compute the local tasks\n conditional_combiner_wrapper<RetType> accum = basic_local_mapper(graph, \n edge_direction, \n mapper, \n combiner,\n current.id());\n accum.set_combiner(combiner);\n // now, wait for everyone\n for (size_t i = 0;i < requests.size(); ++i) {\n accum += requests[i]();\n }\n return accum.value;\n }\n\n};\n\n/**************************************************************************/\n/* */\n/* Extended MapReduce Neighborhood Implementation */\n/* */\n/**************************************************************************/\n/*\n * The master calls extended_mapreduce_neighborhood.\n * Which then issues calls to extended_local_mapper on each machine with a replica.\n * The extended mapreduce neighborhood allows the mapper and combiner to take\n * an optional argument\n */\n\ntemplate <typename RetType, typename GraphType, typename ExtraArg>\nstruct map_reduce_neighborhood_impl2 {\n\n typedef typename GraphType::vertex_type vertex_type;\n typedef typename GraphType::edge_type edge_type;\n typedef typename GraphType::local_vertex_type local_vertex_type;\n typedef typename GraphType::local_edge_type local_edge_type;\n typedef typename GraphType::vertex_record vertex_record;\n\n static conditional_combiner_wrapper<RetType> extended_local_mapper(GraphType& graph,\n edge_dir_type edge_direction,\n RetType (*mapper)(edge_type edge, vertex_type other, const ExtraArg),\n void (*combiner)(RetType&, const RetType&, const ExtraArg),\n vertex_id_type vid,\n const ExtraArg extra) {\n\n lvid_type lvid = graph.local_vid(vid);\n local_vertex_type local_vertex(graph.l_vertex(lvid));\n \n conditional_combiner_wrapper<RetType> accum(boost::bind(combiner, _1, _2, extra));\n if(edge_direction == IN_EDGES || edge_direction == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.in_edges()) {\n edge_type edge(local_edge);\n vertex_type other(local_edge.source());\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n\n graph.get_lock_manager()[std::min(a,b)].lock();\n graph.get_lock_manager()[std::max(a,b)].lock();\n accum += mapper(edge, other, extra);\n graph.get_lock_manager()[a].unlock();\n graph.get_lock_manager()[b].unlock();\n }\n } \n // do out edges\n if(edge_direction == OUT_EDGES || edge_direction == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.out_edges()) {\n edge_type edge(local_edge);\n vertex_type other(local_edge.target());\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n\n graph.get_lock_manager()[std::min(a,b)].lock();\n graph.get_lock_manager()[std::max(a,b)].lock();\n accum += mapper(edge, other, extra);\n graph.get_lock_manager()[a].unlock();\n graph.get_lock_manager()[b].unlock();\n }\n } \n return accum;\n }\n\n static conditional_combiner_wrapper<RetType> extended_local_mapper_from_remote(size_t objid,\n edge_dir_type edge_direction,\n size_t mapper_ptr,\n size_t combiner_ptr,\n vertex_id_type vid,\n const ExtraArg extra) {\n // cast the mappers and combiners back into their pointer types\n RetType (*mapper)(edge_type edge, vertex_type other, const ExtraArg) = \n reinterpret_cast<RetType(*)(edge_type, vertex_type, const ExtraArg)>(mapper_ptr);\n void (*combiner)(RetType&, const RetType&, const ExtraArg) = \n reinterpret_cast<void (*)(RetType&, const RetType&, const ExtraArg)>(combiner_ptr);\n return extended_local_mapper(\n *reinterpret_cast<GraphType*>(distributed_control::get_instance()->get_registered_object(objid)),\n edge_direction,\n mapper,\n combiner,\n vid,\n extra);\n }\n\n static RetType extended_map_reduce_neighborhood(typename GraphType::vertex_type current,\n edge_dir_type edge_direction,\n const ExtraArg extra,\n RetType (*mapper)(edge_type edge,\n vertex_type other,\n const ExtraArg extra),\n void (*combiner)(RetType& self, \n const RetType& other,\n const ExtraArg extra)) {\n // get a reference to the graph\n GraphType& graph = current.graph_ref;\n typename GraphType::vertex_record vrecord = graph.l_get_vertex_record(current.local_id());\n size_t objid = graph.get_rpc_obj_id();\n\n // make sure we are running on a master vertex\n ASSERT_EQ(vrecord.owner, distributed_control::get_instance_procid());\n \n // create num-mirrors worth of requests\n std::vector<request_future<conditional_combiner_wrapper<RetType> > > requests(vrecord.num_mirrors());\n \n size_t ctr = 0;\n foreach(procid_t proc, vrecord.mirrors()) {\n // issue the communication\n requests[ctr] = fiber_remote_request(proc, \n map_reduce_neighborhood_impl2::extended_local_mapper_from_remote,\n objid,\n edge_direction,\n reinterpret_cast<size_t>(mapper),\n reinterpret_cast<size_t>(combiner),\n current.id(),\n extra);\n ++ctr;\n }\n // compute the local tasks\n conditional_combiner_wrapper<RetType> accum = \n extended_local_mapper(graph, edge_direction, mapper, \n combiner, current.id(), extra);\n\n accum.set_combiner(boost::bind(combiner, _1, _2, boost::ref(extra)));\n // now, wait for everyone\n for (size_t i = 0;i < requests.size(); ++i) {\n accum += requests[i]();\n }\n return accum.value;\n }\n\n};\n\n} // namespace warp::warp_impl\n\n/**\n * \\ingroup warp\n *\n * This Warp function allows a map-reduce aggregation of the neighborhood of a\n * vertex to be performed. This is a blocking operation, and will not return\n * until the distributed computation is complete. When run inside a fiber, to\n * hide latency, the system will automatically context switch to evaluate some\n * other fiber which is ready to run. \n *\n * Abstractly, the computation accomplishes the following:\n *\n * \\code\n * ResultType result()\n * foreach(edge in neighborhood of current vertex) {\n * combiner(result, mapper(edge, opposite vertex))\n * }\n * return result\n * \\endcode\n *\n * \\attention This call does not accomplish synchronization, thus \n * modifications to the current vertex will not be reflected during\n * the call. In other words, inside the mapper function, only the values on\n * edge.data() and other.data() will be valid. The value of the vertex\n * on the \"self\" end of the edge will not reflect changes you made to the vertex\n * immediately before calling warp::map_reduce_neighborhood(). Use the overload\n * of map_reduce_neighborhood (below) if you want to pass on additional\n * information to the mapper.\n *\n * Here is an example which implements the PageRank computation, using\n * the parfor_all_vertices function to create a parallel for loop using fibers.\n * \\code\n * float pagerank_map(graph_type::edge_type edge, graph_type::vertex_type other) {\n * return other.data() / other.num_out_edges();\n * }\n *\n * // the function arguments of the combiner must match the return type of the\n * // map function.\n * void pagerank_combine(float& a, const float& b) {\n * a += b;\n * }\n *\n * void pagerank(graph_type::vertex_type vertex) {\n * // computes an aggregate over the neighborhood using map_reduce_neighborhood\n * // The pagerank_map function will be executed over every in-edge of the graph,\n * // and the result of each map is combined using the pagerank_combine \n * // function. The pagerank_combine is not strictly necessary here since the\n * // default combine behavior is to use += anyway.\n * vertex.data() = 0.15 + 0.85 * warp::map_reduce_neighborhood(vertex,\n * IN_EDGES,\n * pagerank_map,\n * pagerank_combine);\n * }\n *\n * ...\n * // runs the pagerank function on all the vertices in the graph.\n * parfor_all_vertices(graph, pagerank); \n * \\endcode\n *\n * An overload is provided which allows you to pass an additional arbitrary\n * argument to the mappers and combiners.\n *\n *\n * \\param current The vertex to map reduce the neighborhood over\n * \\param edge_direction To run over all IN_EDGES, OUT_EDGES or ALL_EDGES\n * \\param mapper The map function that will be executed. Must be a function pointer.\n * \\param combiner The combine function that will be executed. Must be a function pointer.\n * Optional. Defaults to using \"+=\" on the output of the mapper\n * \n * \\return The result of the neighborhood map reduce operation. The return\n * type matches the return type of the mapper.\n *\n * \\see warp_engine\n * \\see warp::parfor_all_vertices()\n * \\see warp::transform_neighborhood()\n * \\see warp::broadcast_neighborhood()\n */\ntemplate <typename RetType, typename VertexType>\nRetType map_reduce_neighborhood(VertexType current,\n edge_dir_type edge_direction,\n RetType (*mapper)(typename VertexType::graph_type::edge_type edge,\n VertexType other),\n void (*combiner)(RetType& self, \n const RetType& other) = warp_impl::default_combiner<RetType>) {\n return warp_impl::\n map_reduce_neighborhood_impl<RetType, \n typename VertexType::graph_type>::\n basic_map_reduce_neighborhood(current, edge_direction, \n mapper, combiner);\n}\n\n\n\n/**\n * \\ingroup warp\n *\n * This Warp function allows a map-reduce aggregation of the neighborhood of a\n * vertex to be performed. This is a blocking operation, and will not return\n * until the distributed computation is complete. When run inside a fiber, to\n * hide latency, the system will automatically context switch to evaluate some\n * other fiber which is ready to run. \n *\n * This is the more general overload of the map_reduce_neighborhood function\n * which allows an additional arbitrary extra argument to be passed along\n * to the mapper and combiner functions.\n * \n * Abstractly, the computation accomplishes the following:\n *\n * \\code\n * ResultType result()\n * foreach(edge in neighborhood of current vertex) {\n * combiner(result, mapper(edge, opposite vertex, extraarg), extraarg)\n * }\n * return result\n * \\endcode\n *\n * Here is an example which implements the PageRank computation, using\n * the parfor_all_vertices function to create a parallel for loop using fibers.\n * We demonstrate the additional argument by passing on the 0.85 scaling value to\n * be computed in the map.\n * \\code\n * float pagerank_map(graph_type::edge_type edge, \n * graph_type::vertex_type other, const float scale) {\n * // the scale value here will match the last argument passed into\n * // the map_reduce_neighborhood call. In this case, it is fixed to\n * // 0.85.\n * return scale * other.data() / other.num_out_edges();\n * }\n *\n * // the function arguments of the combiner must match the return type of the\n * // map function.\n * void pagerank_combine(float& a, const float& b, const float _scale_unused) {\n * a += b;\n * }\n *\n * void pagerank(graph_type::vertex_type vertex) {\n * // computes an aggregate over the neighborhood using map_reduce_neighborhood\n * // The pagerank_map function will be executed over every in-edge of the graph,\n * // and the result of each map is combined using the pagerank_combine \n * // function. The pagerank_combine is not strictly necessary here since the\n * // default combine behavior is to use += anyway.\n * vertex.data() = 0.15 + warp::map_reduce_neighborhood(vertex,\n * IN_EDGES,\n * float(0.85),\n * pagerank_map,\n * pagerank_combine);\n * }\n *\n * ...\n * // runs the pagerank function on all the vertices in the graph.\n * parfor_all_vertices(graph, pagerank); \n * \\endcode\n *\n *\n * \\param current The vertex to map reduce the neighborhood over\n * \\param edge_direction To run over all IN_EDGES, OUT_EDGES or ALL_EDGES\n * \\param extra An additional argument to be passed to the mapper and combiner\n * functions. \n * \\param mapper The map function that will be executed. Must be a\n * function pointer.\n * \\param combiner The combine function that will be executed. Must be a\n * function pointer. Optional. Defaults to using \"+=\" on the output of the\n * mapper \\return The result of the neighborhood map reduce operation. The\n * return type matches the return type of the mapper.\n *\n * \\see warp_engine\n * \\see warp::parfor_all_vertices()\n * \\see warp::transform_neighborhood()\n * \\see warp::broadcast_neighborhood()\n */\ntemplate <typename RetType, typename ExtraArg, typename VertexType>\nRetType map_reduce_neighborhood(VertexType current,\n edge_dir_type edge_direction,\n const ExtraArg extra,\n RetType (*mapper)(typename VertexType::graph_type::edge_type edge,\n VertexType other,\n const ExtraArg extra),\n void (*combiner)(RetType& self, \n const RetType& other,\n const ExtraArg extra) = warp_impl::extended_default_combiner<RetType, ExtraArg>) {\n return warp_impl::\n map_reduce_neighborhood_impl2<RetType, typename VertexType::graph_type, ExtraArg>::\n extended_map_reduce_neighborhood(current, edge_direction, \n extra, \n mapper, combiner);\n}\n\n\n\n\n} // namespace warp\n\n} // namespace graphlab\n\n#include <graphlab/macros_undef.hpp>\n#endif\n" }, { "path": "src/graphlab/engine/warp_graph_transform.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_WARP_GRAPH_TRANSFORM_HPP\n#define GRAPHLAB_WARP_GRAPH_TRANSFORM_HPP\n\n#include <boost/bind.hpp>\n#include <graphlab/util/generics/conditional_combiner_wrapper.hpp>\n#include <graphlab/parallel/fiber_group.hpp>\n#include <graphlab/parallel/fiber_control.hpp>\n#include <graphlab/parallel/fiber_remote_request.hpp>\n#include <graphlab/logger/assertions.hpp>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n\nnamespace warp {\n\nnamespace warp_impl {\n\ntemplate <typename GraphType>\nstruct transform_neighborhood_impl {\n\n typedef typename GraphType::vertex_type vertex_type;\n typedef typename GraphType::edge_type edge_type;\n typedef typename GraphType::local_vertex_type local_vertex_type;\n typedef typename GraphType::local_edge_type local_edge_type;\n typedef typename GraphType::vertex_record vertex_record;\n\n\n/**************************************************************************/\n/* */\n/* Basic MapReduce Neighborhood Implementation */\n/* */\n/**************************************************************************/\n/*\n * The master calls basic_mapreduce_neighborhood.\n * Which then issues calls to basic_local_mapper on each machine with a replica.\n */\n\n static void basic_local_transform_neighborhood(GraphType& graph,\n edge_dir_type edge_direction,\n void(*transform_fn)(edge_type edge,\n vertex_type other),\n vertex_id_type vid) {\n lvid_type lvid = graph.local_vid(vid);\n local_vertex_type local_vertex(graph.l_vertex(lvid));\n \n if(edge_direction == IN_EDGES || edge_direction == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.in_edges()) {\n edge_type edge(local_edge);\n vertex_type other(local_edge.source());\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n graph.get_lock_manager()[std::min(a,b)].lock();\n graph.get_lock_manager()[std::max(a,b)].lock();\n transform_fn(edge, other);\n graph.get_lock_manager()[a].unlock();\n graph.get_lock_manager()[b].unlock();\n }\n } \n // do out edges\n if(edge_direction == OUT_EDGES || edge_direction == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.out_edges()) {\n edge_type edge(local_edge);\n vertex_type other(local_edge.target());\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n graph.get_lock_manager()[std::min(a,b)].lock();\n graph.get_lock_manager()[std::max(a,b)].lock();\n transform_fn(edge, other);\n graph.get_lock_manager()[a].unlock();\n graph.get_lock_manager()[b].unlock();\n }\n } \n }\n\n\n static void basic_local_transform_neighborhood_from_remote(size_t objid,\n edge_dir_type edge_direction,\n size_t transform_ptr,\n vertex_id_type vid) {\n // cast the mappers and combiners back into their pointer types\n void(*transform_fn)(edge_type edge, vertex_type other) = \n reinterpret_cast<void(*)(edge_type, vertex_type)>(transform_ptr);\n basic_local_transform_neighborhood(\n *reinterpret_cast<GraphType*>(distributed_control::get_instance()->get_registered_object(objid)),\n edge_direction,\n transform_fn,\n vid);\n }\n\n static void basic_transform_neighborhood(typename GraphType::vertex_type current,\n edge_dir_type edge_direction,\n void(*transform_fn)(edge_type edge, vertex_type other)) {\n // get a reference to the graph\n GraphType& graph = current.graph_ref;\n // get the object ID of the graph\n size_t objid = graph.get_rpc_obj_id();\n typename GraphType::vertex_record vrecord = graph.l_get_vertex_record(current.local_id());\n\n // make sure we are running on a master vertex\n ASSERT_EQ(vrecord.owner, distributed_control::get_instance_procid());\n \n // create num-mirrors worth of requests\n std::vector<request_future<void > > requests(vrecord.num_mirrors());\n \n size_t ctr = 0;\n foreach(procid_t proc, vrecord.mirrors()) {\n // issue the communication\n requests[ctr] = fiber_remote_request(proc, \n transform_neighborhood_impl<GraphType>::basic_local_transform_neighborhood_from_remote,\n objid,\n edge_direction,\n reinterpret_cast<size_t>(transform_fn),\n current.id());\n ++ctr;\n }\n // compute the local tasks\n basic_local_transform_neighborhood(graph, \n edge_direction, \n transform_fn, \n current.id());\n // now, wait for everyone\n for (size_t i = 0;i < requests.size(); ++i) {\n requests[i]();\n }\n }\n};\n\n\ntemplate <typename GraphType, typename ExtraArg>\nstruct transform_neighborhood_impl2 {\n\n typedef typename GraphType::vertex_type vertex_type;\n typedef typename GraphType::edge_type edge_type;\n typedef typename GraphType::local_vertex_type local_vertex_type;\n typedef typename GraphType::local_edge_type local_edge_type;\n typedef typename GraphType::vertex_record vertex_record;\n\n/**************************************************************************/\n/* */\n/* Extended MapReduce Neighborhood Implementation */\n/* */\n/**************************************************************************/\n/*\n * The master calls extended_mapreduce_neighborhood.\n * Which then issues calls to extended_local_mapper on each machine with a replica.\n * The extended mapreduce neighborhood allows the mapper and combiner to take\n * an optional argument\n */\n\n\n static void extended_local_transform_neighborhood(GraphType& graph,\n edge_dir_type edge_direction,\n void(*transform_fn)(edge_type edge,\n vertex_type other,\n const ExtraArg extra),\n vertex_id_type vid,\n const ExtraArg extra) {\n lvid_type lvid = graph.local_vid(vid);\n local_vertex_type local_vertex(graph.l_vertex(lvid));\n \n if(edge_direction == IN_EDGES || edge_direction == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.in_edges()) {\n edge_type edge(local_edge);\n vertex_type other(local_edge.source());\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n graph.get_lock_manager()[std::min(a,b)].lock();\n graph.get_lock_manager()[std::max(a,b)].lock();\n transform_fn(edge, other, extra);\n graph.get_lock_manager()[a].unlock();\n graph.get_lock_manager()[b].unlock();\n }\n } \n // do out edges\n if(edge_direction == OUT_EDGES || edge_direction == ALL_EDGES) {\n foreach(local_edge_type local_edge, local_vertex.out_edges()) {\n edge_type edge(local_edge);\n vertex_type other(local_edge.target());\n lvid_type a = edge.source().local_id(), b = edge.target().local_id();\n graph.get_lock_manager()[std::min(a,b)].lock();\n graph.get_lock_manager()[std::max(a,b)].lock();\n transform_fn(edge, other, extra);\n graph.get_lock_manager()[a].unlock();\n graph.get_lock_manager()[b].unlock();\n }\n } \n }\n\n\n static void extended_local_transform_neighborhood_from_remote(size_t objid,\n edge_dir_type edge_direction,\n size_t transform_ptr,\n vertex_id_type vid,\n const ExtraArg extra) {\n // cast the mappers and combiners back into their pointer types\n void(*transform_fn)(edge_type edge, vertex_type other, const ExtraArg) = \n reinterpret_cast<void(*)(edge_type, vertex_type, const ExtraArg)>(transform_ptr);\n extended_local_transform_neighborhood(\n *reinterpret_cast<GraphType*>(distributed_control::get_instance()->get_registered_object(objid)),\n edge_direction,\n transform_fn,\n vid,\n extra);\n }\n\n static void extended_transform_neighborhood(typename GraphType::vertex_type current,\n edge_dir_type edge_direction,\n void(*transform_fn)(edge_type edge, vertex_type other, const ExtraArg extra),\n const ExtraArg extra) {\n // get a reference to the graph\n GraphType& graph = current.graph_ref;\n // get the object ID of the graph\n size_t objid = graph.get_rpc_obj_id();\n typename GraphType::vertex_record vrecord = graph.l_get_vertex_record(current.local_id());\n\n // make sure we are running on a master vertex\n ASSERT_EQ(vrecord.owner, distributed_control::get_instance_procid());\n \n // create num-mirrors worth of requests\n std::vector<request_future<void> > requests(vrecord.num_mirrors());\n \n size_t ctr = 0;\n foreach(procid_t proc, vrecord.mirrors()) {\n // issue the communication\n requests[ctr] = fiber_remote_request(proc, \n transform_neighborhood_impl2<GraphType, ExtraArg>::extended_local_transform_neighborhood_from_remote,\n objid,\n edge_direction,\n reinterpret_cast<size_t>(transform_fn),\n current.id(),\n extra);\n ++ctr;\n }\n // compute the local tasks\n extended_local_transform_neighborhood(graph, \n edge_direction, \n transform_fn, \n current.id(),\n extra);\n // now, wait for everyone\n for (size_t i = 0;i < requests.size(); ++i) {\n requests[i]();\n }\n }\n\n\n};\n\n} // namespace warp::warp_impl\n\n/**\n * \\ingroup warp\n * \n * The transform_neighborhood() function allows a parallel transformation of the\n * neighborhood of a vertex to be performed. This is a blocking operation, and\n * will not return until the distributed computation is complete. When run\n * inside a fiber, to hide latency, the system will automatically context\n * switch to evaluate some other fiber which is ready to run. \n *\n * Abstractly, the computation accomplishes the following:\n *\n * \\code\n * foreach(edge in neighborhood of current vertex) {\n * transform_fn(edge, opposite vertex)\n * }\n * \\endcode\n *\n * \\attention It is important that the transform_fn should only make modifications to the\n * edge data, and not the data on the other vertex.\n *\n * \\attention This call does not accomplish synchronization, thus \n * modifications to the current vertex will not be reflected during\n * the call. In other words, inside the mapper function, only the values on\n * edge.data() and other.data() will be valid. The value of the vertex\n * on the \"self\" end of the edge will not reflect changes you made to the vertex\n * immediately before calling transform_neighborhood(). Use the overload of \n * transform_neighborhood (below) if you want to pass on additional information to the\n * mapper.\n *\n * Here is an example which clears all the in edge values of some set of vertices.\n * \\code\n * void clear_value(graph_type::edge_type edge, graph_type::vertex_type other) {\n * edge.data() = 0;\n * }\n *\n *\n * void clear_in_edges(graph_type::vertex_type vertex) {\n * warp::transform_neighborhood(vertex,\n * IN_EDGES,\n * clear_value);\n * }\n *\n * ...\n * warp::parfor_all_vertices(graph, clear_in_edges, some_vset); \n * \\endcode\n *\n *\n * An overload is provided which allows you to pass an additional arbitrary\n * argument to the transform\n *\n * \\param current The vertex to map reduce the neighborhood over\n * \\param edge_direction To run over all IN_EDGES, OUT_EDGES or ALL_EDGES\n * \\param transform_fn The transform function to run on all the selected edges.\n * \n * \\see warp_engine\n * \\see warp::parfor_all_vertices()\n * \\see warp::map_reduce_neighborhood()\n * \\see warp::broadcast_neighborhood()\n */\ntemplate <typename VertexType>\nvoid transform_neighborhood(VertexType current,\n edge_dir_type edge_direction,\n void (*transform_fn)(typename VertexType::graph_type::edge_type edge,\n VertexType other)) {\n warp_impl::\n transform_neighborhood_impl<typename VertexType::graph_type>::\n basic_transform_neighborhood(current, edge_direction, \n transform_fn);\n}\n\n\n/**\n * \\ingroup warp\n *\n * The transform_neighborhood function allows a parallel transformation of the\n * neighborhood of a vertex to be performed. This is a blocking operation, and\n * will not return until the distributed computation is complete. When run\n * inside a fiber, to hide latency, the system will automatically context\n * switch to evaluate some other fiber which is ready to run. \n *\n * This is the more general overload of the \n * warp::transform_neighborhood() function\n * which allows an additional arbitrary extra argument to be passed along\n * to the mapper and combiner functions.\n *\n * Abstractly, the computation accomplishes the following:\n *\n * \\code\n * foreach(edge in neighborhood of current vertex) {\n * transform_fn(edge, extraarg, opposite vertex)\n * }\n * \\endcode\n *\n * \\attention It is important that the transform_fn should only make modifications to the\n * edge data, and not the data on the other vertex.\n *\n * \\attention This call does not accomplish synchronization, thus \n * modifications to the current vertex will not be reflected during\n * the call. In other words, inside the mapper function, only the values on\n * edge.data() and other.data() will be valid. The value of the vertex\n * on the \"self\" end of the edge will not reflect changes you made to the vertex\n * immediately before calling warp::transform_neighborhood() . Use the overload of \n * warp::transform_neighborhood() if you want to pass on additional information to the\n * mapper.\n *\n * Here is an example which set all the in edge values of some set of vertices\n * to the source vertex's value.\n * \\code\n * void set_value(graph_type::edge_type edge, graph_type::vertex_type other,\n * int value) {\n * edge.data() = value;\n * }\n *\n *\n * void set_in_edges(graph_type::vertex_type vertex) {\n * warp::transform_neighborhood(vertex,\n * IN_EDGES,\n * clear_value,\n * vertex.data(), // say this is an integer\n * );\n * }\n *\n * ...\n * warp::parfor_all_vertices(graph, set_in_edges, some_vset); \n * \\endcode\n *\n *\n * \\param current The vertex to map reduce the neighborhood over\n * \\param edge_direction To run over all IN_EDGES, OUT_EDGES or ALL_EDGES\n * \\param extra An additional argument to be passed to the mapper and combiner functions.\n * \\param transform_fn The transform function to run on all the selected edges.\n * \n * \\see warp_engine\n * \\see warp::parfor_all_vertices()\n * \\see warp::map_reduce_neighborhood()\n * \\see warp::broadcast_neighborhood()\n */\ntemplate <typename ExtraArg, typename VertexType>\nvoid transform_neighborhood(VertexType current,\n edge_dir_type edge_direction,\n void(*transform_fn)(typename VertexType::graph_type::edge_type edge,\n VertexType other,\n const ExtraArg extra),\n const ExtraArg extra) {\n warp_impl::\n transform_neighborhood_impl2<typename VertexType::graph_type, ExtraArg>::\n extended_transform_neighborhood(current, edge_direction, \n transform_fn,\n extra);\n}\n\n\n\n\n} // namespace warp\n\n} // namespace graphlab\n\n#include <graphlab/macros_undef.hpp>\n#endif\n" }, { "path": "src/graphlab/engine/warp_parfor_all_vertices.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_WARP_PARFOR_ALL_VERTICES_HPP\n#define GRAPHLAB_WARP_PARFOR_ALL_VERTICES_HPP\n#include <boost/function.hpp>\n#include <graphlab/parallel/fiber_group.hpp>\n#include <graphlab/parallel/atomic.hpp>\n#include <graphlab/graph/vertex_set.hpp>\n#include <graphlab/rpc/dc.hpp>\nnamespace graphlab {\nnamespace warp {\n\nnamespace warp_impl {\n\n\n/*\n * Actual Parfor implementation.\n * Holds a reference to all the arguments.\n * Each fiber increments the atomic counter and runs the fn on it)\n */\ntemplate <typename GraphType>\nstruct parfor_all_vertices_impl{\n\n GraphType& graph; \n boost::function<void(typename GraphType::vertex_type)> fn;\n vertex_set& vset;\n atomic<size_t> ctr;\n\n parfor_all_vertices_impl(GraphType& graph,\n boost::function<void(typename GraphType::vertex_type)> fn,\n vertex_set& vset): graph(graph),fn(fn),vset(vset),ctr(0) { }\n\n void run_fiber() {\n while (1) {\n size_t lvid = ctr.inc_ret_last();\n if (lvid >= graph.num_local_vertices() || !vset.l_contains(lvid)) break;\n typename GraphType::local_vertex_type l_vertex = graph.l_vertex(lvid);\n if (l_vertex.owned()) {\n typename GraphType::vertex_type vertex(l_vertex);\n fn(vertex);\n }\n } \n }\n};\n\n} // namespace warp_impl\n\n\n/**\n * \\ingroup warp\n *\n * This Warp function provides a simple parallel for loop over all vertices\n * in the graph, or in a given set of vertices. A large number of light-weight \n * threads called fibers are used to run the user specified function,\n * allowing the user to make what normally will be blocking calls, on \n * the neighborhood of each vertex.\n *\n * \\code\n * float pagerank_map(graph_type::edge_type edge, graph_type::vertex_type other) {\n * return other.data() / other.num_out_edges();\n * }\n *\n * void pagerank(graph_type::vertex_type vertex) {\n * // computes an aggregate over the neighborhood using map_reduce_neighborhood\n * vertex.data() = 0.15 + 0.85 * warp::map_reduce_neighborhood(vertex,\n * IN_EDGES,\n * pagerank_map);\n * }\n *\n * ...\n * // runs the pagerank function on all the vertices in the graph.\n * parfor_all_vertices(graph, pagerank); \n * \\endcode\n *\n * \\param graph A reference to the graph object\n * \\param fn A function to run on each vertex. Has the prototype void(GraphType::vertex_type). Can be a boost::function\n * \\param vset A set of vertices to run on\n * \\param nfibers Number of fiber threads to use. Defaults to 10000\n * \\param stacksize Size of each fiber stack in bytes. Defaults to 16384 bytes\n *\n * \\see graphlab::warp::map_reduce_neighborhood()\n * \\see graphlab::warp::warp_graph_transform()\n */\ntemplate <typename GraphType, typename FunctionType>\nvoid parfor_all_vertices(GraphType& graph,\n FunctionType fn,\n vertex_set vset = GraphType::complete_set(),\n size_t nfibers = 10000,\n size_t stacksize = 16384) {\n distributed_control::get_instance()->barrier();\n bool old_fast_track = distributed_control::get_instance()->set_fast_track_requests(false);\n fiber_group group;\n group.set_stacksize(stacksize);\n warp_impl::parfor_all_vertices_impl<GraphType> parfor(graph, fn, vset);\n \n for (size_t i = 0;i < nfibers; ++i) {\n group.launch(boost::bind(&warp_impl::parfor_all_vertices_impl<GraphType>::run_fiber, &parfor));\n }\n group.join();\n distributed_control::get_instance()->barrier();\n distributed_control::get_instance()->set_fast_track_requests(old_fast_track);\n graph.synchronize(vset);\n}\n\n} // namespace warp\n} // namespace graphlab\n#endif\n" }, { "path": "src/graphlab/graph/CMakeLists.txt", "content": "project(GraphLab)\n\n" }, { "path": "src/graphlab/graph/builtin_parsers.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n#ifndef GRAPHLAB_GRAPH_BUILTIN_PARSERS_HPP\n#define GRAPHLAB_GRAPH_BUILTIN_PARSERS_HPP\n\n#include <string>\n#include <sstream>\n#include <iostream>\n\n#if defined(__cplusplus) && __cplusplus >= 201103L\n// do not include spirit\n#else\n#include <boost/config/warning_disable.hpp>\n#include <boost/spirit/include/qi.hpp>\n#include <boost/spirit/include/phoenix_core.hpp>\n#include <boost/spirit/include/phoenix_operator.hpp>\n#include <boost/spirit/include/phoenix_stl.hpp>\n#endif\n\n#include <graphlab/util/stl_util.hpp>\n#include <graphlab/logger/logger.hpp>\n#include <graphlab/serialization/serialization_includes.hpp>\n\n\n\n\nnamespace graphlab {\n\n namespace builtin_parsers {\n \n /**\n * \\brief Parse files in the Stanford Network Analysis Package format.\n *\n * example:\n *\n * # some comment\n * # another comment\n * 1 2\n * 3 4\n * 1 4\n *\n */\n template <typename Graph>\n bool snap_parser(Graph& graph, const std::string& srcfilename,\n const std::string& str) {\n if (str.empty()) return true;\n else if (str[0] == '#') {\n std::cout << str << std::endl;\n } else {\n size_t source, target;\n char* targetptr;\n source = strtoul(str.c_str(), &targetptr, 10);\n if (targetptr == NULL) return false;\n target = strtoul(targetptr, NULL, 10);\n if(source != target) graph.add_edge(source, target);\n }\n return true;\n } // end of snap parser\n\n /**\n * \\brief Parse files in the standard tsv format\n *\n * This is identical to the SNAP format but does not allow comments.\n *\n */\n template <typename Graph>\n bool tsv_parser(Graph& graph, const std::string& srcfilename,\n const std::string& str) {\n if (str.empty()) return true;\n size_t source, target;\n char* targetptr;\n source = strtoul(str.c_str(), &targetptr, 10);\n if (targetptr == NULL) return false;\n target = strtoul(targetptr, NULL, 10);\n if(source != target) graph.add_edge(source, target);\n return true;\n } // end of tsv parser\n\n\n template <typename Graph>\n bool csv_parser(Graph& graph, \n const std::string& filename, \n const std::string& textline) {\n if (textline.empty()) return true;\n size_t split = textline.find_first_of(\",\");\n if (split == std::string::npos) return true;\n else {\n std::string t = textline;\n t[split] = 0;\n graph.add_edge(strtoul(t.c_str(), NULL, 10), \n strtoul(t.c_str() + split + 1, NULL, 10));\n return true;\n }\n }\n\n\n#if defined(__cplusplus) && __cplusplus >= 201103L\n // The spirit parser seems to have issues when compiling under\n // C++11. Temporary workaround with a hard coded parser. TOFIX\n template <typename Graph>\n bool adj_parser(Graph& graph, const std::string& srcfilename,\n const std::string& line) {\n // If the line is empty simply skip it\n if(line.empty()) return true;\n std::stringstream strm(line);\n vertex_id_type source; \n size_t n;\n strm >> source;\n if (strm.fail()) return false;\n strm >> n;\n if (strm.fail()) return true;\n\n size_t nadded = 0;\n while (strm.good()) {\n vertex_id_type target;\n strm >> target;\n if (strm.fail()) break;\n if (source != target) graph.add_edge(source, target);\n ++nadded;\n } \n if (n != nadded) return false;\n return true;\n } // end of adj parser\n\n#else\n\n template <typename Graph>\n bool adj_parser(Graph& graph, const std::string& srcfilename,\n const std::string& line) {\n // If the line is empty simply skip it\n if(line.empty()) return true;\n // We use the boost spirit parser which requires (too) many separate\n // namespaces so to make things clear we shorten them here.\n namespace qi = boost::spirit::qi;\n namespace ascii = boost::spirit::ascii;\n namespace phoenix = boost::phoenix;\n vertex_id_type source(-1);\n vertex_id_type ntargets(-1);\n std::vector<vertex_id_type> targets;\n const bool success = qi::phrase_parse\n (line.begin(), line.end(), \n // Begin grammar\n (\n qi::ulong_[phoenix::ref(source) = qi::_1] >> -qi::char_(\",\") >>\n qi::ulong_[phoenix::ref(ntargets) = qi::_1] >> -qi::char_(\",\") >>\n *(qi::ulong_[phoenix::push_back(phoenix::ref(targets), qi::_1)] % -qi::char_(\",\"))\n )\n ,\n // End grammar\n ascii::space); \n // Test to see if the boost parser was able to parse the line\n if(!success || ntargets != targets.size()) {\n logstream(LOG_ERROR) << \"Parse error in vertex prior parser.\" << std::endl;\n return false;\n }\n for(size_t i = 0; i < targets.size(); ++i) {\n if(source != targets[i]) graph.add_edge(source, targets[i]);\n }\n return true;\n } // end of adj parser\n#endif\n\n template <typename Graph>\n struct tsv_writer{\n typedef typename Graph::vertex_type vertex_type;\n typedef typename Graph::edge_type edge_type;\n std::string save_vertex(vertex_type) { return \"\"; }\n std::string save_edge(edge_type e) {\n return tostr(e.source().id()) + \"\\t\" + tostr(e.target().id()) + \"\\n\";\n }\n };\n\n\n\n \n template <typename Graph>\n struct graphjrl_writer{\n typedef typename Graph::vertex_type vertex_type;\n typedef typename Graph::edge_type edge_type;\n\n /**\n * \\internal\n * Replaces \\255 with \\255\\1\n * Replaces \\\\n with \\255\\0\n */\n static std::string escape_newline(charstream& strm) {\n size_t ctr = 0;\n char *ptr = strm->c_str();\n size_t strmlen = strm->size();\n for (size_t i = 0;i < strmlen; ++i) {\n ctr += (ptr[i] == (char)255 || ptr[i] == '\\n');\n }\n\n std::string ret(ctr + strmlen, 0);\n\n size_t target = 0;\n for (size_t i = 0;i < strmlen; ++i, ++ptr) {\n if ((*ptr) == (char)255) {\n ret[target] = (char)255;\n ret[target + 1] = 1;\n target += 2;\n }\n else if ((*ptr) == '\\n') {\n ret[target] = (char)255;\n ret[target + 1] = 0;\n target += 2;\n }\n else {\n ret[target] = (*ptr);\n ++target;\n }\n }\n assert(target == ctr + strmlen);\n return ret;\n }\n\n /**\n * \\internal\n * Replaces \\255\\1 with \\255\n * Replaces \\255\\0 with \\\\n\n */\n static std::string unescape_newline(const std::string& str) {\n size_t ctr = 0;\n // count the number of escapes\n for (size_t i = 0;i < str.length(); ++i) {\n ctr += (str[i] == (char)255);\n }\n // real length is string length - escapes\n std::string ret(str.size() - ctr, 0);\n\n const char escapemap[2] = {'\\n', (char)255};\n \n size_t target = 0;\n for (size_t i = 0;i < str.length(); ++i, ++target) {\n if (str[i] == (char)255) {\n // escape character\n ++i;\n ASSERT_MSG(str[i] == 0 || str[i] == 1,\n \"Malformed escape sequence in graphjrl file.\");\n ret[target] = escapemap[(int)str[i]];\n }\n else {\n ret[target] = str[i];\n }\n }\n return ret;\n }\n \n std::string save_vertex(vertex_type v) {\n charstream strm(128);\n oarchive oarc(strm);\n oarc << char(0) << v.id() << v.data();\n strm.flush();\n return escape_newline(strm) + \"\\n\";\n }\n \n std::string save_edge(edge_type e) {\n charstream strm(128);\n oarchive oarc(strm);\n oarc << char(1) << e.source().id() << e.target().id() << e.data();\n strm.flush();\n return escape_newline(strm) + \"\\n\";\n }\n };\n\n\n\n template <typename Graph>\n bool graphjrl_parser(Graph& graph, const std::string& srcfilename,\n const std::string& str) {\n std::string unescapedstr = graphjrl_writer<Graph>::unescape_newline(str);\n boost::iostreams::stream<boost::iostreams::array_source>\n istrm(unescapedstr.c_str(), unescapedstr.length());\n iarchive iarc(istrm);\n \n char entrytype;\n iarc >> entrytype;\n if (entrytype == 0) {\n typename Graph::vertex_id_type vid;\n typename Graph::vertex_data_type vdata;\n iarc >> vid >> vdata;\n graph.add_vertex(vid, vdata);\n }\n else if (entrytype == 1) {\n typename Graph::vertex_id_type srcvid, destvid;\n typename Graph::edge_data_type edata;\n iarc >> srcvid >> destvid >> edata;\n graph.add_edge(srcvid, destvid, edata);\n }\n else {\n return false;\n }\n return true;\n }\n \n } // namespace builtin_parsers\n} // namespace graphlab\n\n#endif\n" }, { "path": "src/graphlab/graph/distributed_graph.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_DISTRIBUTED_GRAPH_HPP\n#define GRAPHLAB_DISTRIBUTED_GRAPH_HPP\n\n#ifndef __NO_OPENMP__\n#include <omp.h>\n#endif\n\n#include <cmath>\n\n#include <string>\n#include <list>\n#include <vector>\n#include <set>\n#include <map>\n#include <graphlab/util/dense_bitset.hpp>\n\n\n#include <queue>\n#include <algorithm>\n#include <functional>\n#include <fstream>\n#include <iostream>\n#include <sstream>\n\n#include <boost/functional.hpp>\n#include <boost/algorithm/string/predicate.hpp>\n#include <boost/iostreams/stream.hpp>\n#include <boost/iostreams/filtering_streambuf.hpp>\n\n#include <boost/iostreams/filtering_stream.hpp>\n#include <boost/iostreams/copy.hpp>\n#include <boost/iostreams/filter/gzip.hpp>\n#include <boost/filesystem.hpp>\n#include <boost/concept/requires.hpp>\n\n\n#include <graphlab/logger/logger.hpp>\n#include <graphlab/logger/assertions.hpp>\n\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_dist_object.hpp>\n#include <graphlab/rpc/buffered_exchange.hpp>\n#include <graphlab/util/random.hpp>\n#include <graphlab/util/branch_hints.hpp>\n#include <graphlab/util/generics/conditional_addition_wrapper.hpp>\n\n#include <graphlab/options/graphlab_options.hpp>\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/vertex_program/op_plus_eq_concept.hpp>\n\n#include <graphlab/graph/local_graph.hpp>\n#include <graphlab/graph/dynamic_local_graph.hpp>\n\n#include <graphlab/graph/graph_gather_apply.hpp>\n#include <graphlab/graph/ingress/distributed_ingress_base.hpp>\n#include <graphlab/graph/ingress/distributed_oblivious_ingress.hpp>\n#include <graphlab/graph/ingress/distributed_hdrf_ingress.hpp>\n#include <graphlab/graph/ingress/distributed_random_ingress.hpp>\n#include <graphlab/graph/ingress/distributed_identity_ingress.hpp>\n\n#include <graphlab/graph/ingress/sharding_constraint.hpp>\n#include <graphlab/graph/ingress/distributed_constrained_random_ingress.hpp>\n\n#include <graphlab/graph/graph_hash.hpp>\n\n#include <graphlab/util/hopscotch_map.hpp>\n\n#include <graphlab/util/fs_util.hpp>\n#include <graphlab/util/hdfs.hpp>\n\n\n#include <graphlab/graph/builtin_parsers.hpp>\n#include <graphlab/graph/vertex_set.hpp>\n\n#include <graphlab/macros_def.hpp>\nnamespace tests {\nclass distributed_graph_test;\n}\nnamespace graphlab {\n\n /** \\brief A directed graph datastructure which is distributed across\n * multiple machines.\n *\n * This class implements a distributed directed graph datastructure where\n * vertices and edges may contain arbitrary user-defined datatypes as\n * templatized by the VertexData and EdgeData template parameters.\n *\n * ### Initialization\n *\n * To declare a distributed graph you write:\n * \\code typedef\n * graphlab::distributed_graph<vdata, edata> graph_type;\n * graph_type graph(dc, clopts);\n * \\endcode\n * where <code>vdata</code> is the type of data to be\n * stored on vertices, and <code>edata</code> is the type of data to be\n * stored on edges. The constructor must be called simultaneously on all\n * machines. <code>dc</code> is a graphlab::distributed_control object that\n * must be constructed at the start of the program, and clopts is a\n * graphlab::graphlab_options object that is used to pass graph\n * construction runtime options to the graph. See the code examples for\n * further details.\n *\n * Each vertex is uniquely identified by an unsigned numeric ID of the type\n * graphlab::vertex_id_type. Vertex IDs need not be sequential. However, the\n * ID corresponding to <code>(vertex_id_type)(-1)</code> is reserved. (This\n * is the largest possible ID, corresponding to 0xFFFFFFFF when using 32-bit\n * IDs).\n *\n * Edges are not numbered, but are uniquely identified by its source->target\n * pair. In other words, there can only be two edges between any pair of\n * vertices, the edge going in the forward direction, and the edge going in\n * the backward direction.\n *\n * ### Construction\n *\n * The distributed graph can be constructed in two different ways. The\n * first, and the preferred method, is to construct the graph from files\n * located on a shared filesystem (NFS mounts for instance) , or from files\n * on HDFS (HDFS support must be compiled).\n *\n * To construct from files, the load_format() function provides built-in\n * parsers to construct the graph structure from various graph file formats\n * on disk or HDFS. Alternatively, the\n * \\ref load(const std::string& path, line_parser_type line_parser) \"load()\"\n * function provides generalized parsing capabilities\n * allowing you to construct from your own defined file format.\n * Alternatively, load_binary() may be used to perform an extremely rapid\n * load of a graph previously saved with save_binary(). The caveat being that\n * the number of machines used to save the graph must match the number of\n * machines used to load the graph.\n *\n * The second construction strategy is to call the add_vertex() and\n * add_edge() functions directly. These functions are parallel reentrant, and\n * are also distributed. Each vertex and each edge should be added no more\n * than once across all machines.\n *\n * add_vertex() calls are not strictly required since add_edge(i, j) will\n * implicitly construct vertices i and j. The data on these vertices\n * will be default constructed.\n *\n * ### Finalization\n * After all vertices and edges are inserted into the graph\n * via either load from file functions or direct calls to add_vertex() and\n * add_edge(), for the graph to the useable, it must be finalized.\n *\n * This is performed by calling \\code graph.finalize(); \\endcode on all\n * machines simultaneously. None of the load* functions perform finalization\n * so multiple load operations could be performed (reading from different\n * file groups) before finalization.\n *\n * The finalize() operation partitions the graph and synchronizes all\n * internal graph datastructures. After this point, all graph computation\n * operations such as engine, map_reduce and transform operations will\n * function.\n *\n * ### Partitioning Strategies\n *\n * The graph is partitioned across the machines using a \"vertex separator\"\n * strategy where edges are assigned to machines, while vertices may span\n * multiple machines. There are three partitioning strategies implemented.\n * These can be selected by setting --graph_opts=\"ingress=[partition_method]\"\n * on the command line.\n * \\li \\c \"random\" The most naive and the fastest partitioner. Random places\n * edges on machines.\n * \\li \\c \"oblivious\" Runs at roughly half the speed of random. Machines\n * indepedently partitions the segment of the graph it\n * read. Improves partitioning quality and will reduce\n * runtime memory consumption.\n *\n * \\li \\c \"grid\" Runs at rouphly the same speed of random. Randomly places\n * edges on machines with a grid constraint.\n * This obtains quality partition, close to oblivious,\n * but currently only works with perfect square number of machines.\n *\n * \\li \\c \"pds\" Runs at roughly the speed of random. Randomly places\n * edges on machines with a sparser constraint generated by\n * perfect difference set. This obtains the highest quality partition,\n * reducing runtime memory consumption significantly, without load-time penalty.\n * Currently only works with p^2+p+1 number of machines (p prime).\n *\t\n * \\li \\c \"hdrf\" Runs at roughly the speed of oblivious.\n *\t HDRF provides the smallest average replication factor with close to optimal load balance.\n *\t\t The HDRF algorithm is extensively described in the following publication:\n *\t\t F. Petroni, L. Querzoni, K. Daudjee, S. Kamali and G. Iacoboni: \n *\t\t \"HDRF: Stream-Based Partitioning for Power-Law Graphs\". \n *\t\t CIKM, 2015.\n *\n * ### Referencing Vertices / Edges Many GraphLab operations will pass around\n * vertex_type and edge_type objects. These objects are light-weight copyable\n * opaque references to vertices and edges in the distributed graph. The\n * vertex_type object provides capabilities such as:\n * \\li \\c vertex_type::id() Returns the ID of the vertex\n * \\li \\c vertex_type::num_in_edges() Returns the number of in edges\n * \\li \\c vertex_type::num_out_edges() Returns the number of out edges\n * \\li \\c vertex_type::data() Returns a <b>reference</b> to the data on\n * the vertex\n *\n * No traversal operations are currently provided and there there is no\n * single method to return a list of adjacent edges to the vertex.\n *\n * The edge_type object has similar capabilities:\n * \\li \\c edge_type::data() Returns a <b>reference</b> to the data on the edge\n * \\li \\c edge_type::source() Returns a \\ref vertex_type of the source vertex\n * \\li \\c edge_type::target() Returns a \\ref vertex_type of the target vertex\n *\n * This permits the use of <code>edge.source().data()</code> for instance, to\n * obtain the vertex data on the source vertex.\n *\n * See the documentation for \\ref vertex_type and \\ref edge_type for further\n * details.\n *\n * Due to the distributed nature of the graph, There is at the moment, no way\n * to obtain a reference to arbitrary vertices or edges. The only way to\n * obtain a reference to vertices or edges, is if one is passed to you via a\n * callback (for instance in map_reduce_vertices() / map_reduce_edges() or in\n * an update function). To manipulate the graph at a more fine-grained level\n * will require a more intimate understanding of the underlying distributed\n * graph representation.\n *\n * ### Saving the graph\n * After computation is complete, the graph structure can be saved\n * via save_format() which provides built-in writers to write various\n * graph formats to disk or HDFS. Alternatively,\n * \\ref save(const std::string& prefix, writer writer, bool gzip, bool save_vertex, bool save_edge, size_t files_per_machine) \"save()\"\n * provides generalized writing capabilities allowing you to write\n * your own graph output to disk or HDFS.\n *\n * ### Distributed Representation\n * The graph is partitioned over machines using vertex separators.\n * In other words, each edge is assigned to a unique machine while\n * vertices are allowed to span multiple machines.\n *\n * The image below demonstrates the procedure. The example graph\n * on the left is to be separated among 4 machines where the cuts\n * are denoted by the dotted red lines. After partitioning,\n * (the image on the right), each vertex along the cut\n * is now separated among multiple machines. For instance, the\n * central vertex spans 4 different machines.\n *\n * \\image html partition_fig.gif\n *\n * Each vertex which span multiple machines, has a <b>master</b>\n * machine (a black vertex), and all other instances of the vertex\n * are called <b>mirrors</b>. For instance, we observe that the central\n * vertex spans 4 machines, where machine 3 holds the <b>master</b>\n * copy, while all remaining machines hold <b>mirrored</b> copies.\n *\n * This concept of vertex separators allow us to easily manage large\n * power-law graphs where vertices may have extremely high degrees,\n * since the adjacency information for even the high degree vertices\n * can be separated across multiple machines.\n *\n * ### Internal Representation\n * \\warning This is only useful if you plan to make use of the graph\n * in ways which exceed the provided abstractions.\n *\n * Each machine maintains its local section of the graph in a\n * graphlab::local_graph object. The local_graph object assigns\n * each vertex a sequential vertex ID called the local vertex ID.\n * A hash table is used to provide a mapping between the local vertex IDs\n * and their corresponding global vertex IDs. Additionally, each local\n * vertex is associated with a \\ref vertex_record which provides information\n * about global ID of the vertex, the machine which holds the master instance\n * of the vertex, as well as a list of all machines holding a mirror\n * of the vertex.\n *\n * To support traversal of the local graph, two additional types, the\n * \\ref local_vertex_type and the \\ref local_edge_type is provided which\n * provide references to vertices and edges on the local graph. These behave\n * similarly to the \\ref vertex_type and \\ref edge_type types and have\n * similar functionality. However, since these reference the local graph,\n * there is substantially more flexility. In particular, the function\n * l_vertex() may be used to obtain a reference to a local vertex from a\n * local vertex ID. Also unlike the \\ref vertex_type , the \\ref\n * local_vertex_type support traversal operations such as returning a list of\n * all in_edges (local_vertex_type::in_edges()). However, the list only\n * contains the edges which are local to the current machine. See\n * \\ref local_vertex_type and \\ref local_edge_type for more details.\n *\n *\n * \\tparam VertexData Type of data stored on vertices. Must be\n * Copyable, Default Constructable, Copy\n * Constructable and \\ref sec_serializable.\n * \\tparam EdgeData Type of data stored on edges. Must be\n * Copyable, Default Constructable, Copy\n * Constructable and \\ref sec_serializable.\n */\n template<typename VertexData, typename EdgeData>\n class distributed_graph {\n public:\n\n /// The type of the vertex data stored in the graph\n typedef VertexData vertex_data_type;\n\n /**\n * \\cond GRAPHLAB_INTERNAL\n * \\brief GraphLab Requires that vertex data be default\n * constructible. That is you must have a public member:\n *\n * \\code\n * class vertex_data {\n * public:\n * vertex_data() { }\n * }; //\n * \\endcode\n */\n BOOST_CONCEPT_ASSERT((boost::DefaultConstructible<VertexData>));\n /// \\endcond\n\n /**\n * \\cond GRAPHLAB_INTERNAL\n * \\brief GraphLab Requires that vertex data be default\n * Serializable.\n */\n BOOST_CONCEPT_ASSERT((graphlab::Serializable<VertexData>));\n /// \\endcond\n\n\n /// The type of the edge data stored in the graph\n typedef EdgeData edge_data_type;\n\n /**\n * \\cond GRAPHLAB_INTERNAL\n * \\brief GraphLab Requires that edge data be default\n * constructible. That is you must have a public member:\n *\n * \\code\n * class edge_data {\n * public:\n * edge_data() { }\n * }; //\n * \\endcode\n */\n BOOST_CONCEPT_ASSERT((boost::DefaultConstructible<EdgeData>));\n /// \\endcond\n\n /**\n * \\cond GRAPHLAB_INTERNAL\n * \\brief GraphLab Requires that edge data be default\n * Serializable.\n */\n BOOST_CONCEPT_ASSERT((graphlab::Serializable<EdgeData>));\n /// \\endcond\n\n\n /**\n The line parse is any function (or functor) that has the form:\n\n <code>\n bool line_parser(distributed_graph& graph, const std::string& filename,\n const std::string& textline);\n </code>\n\n the line parser returns true if the line is parsed successfully and\n calls graph.add_vertex(...) or graph.add_edge(...)\n\n See \\ref graphlab::distributed_graph::load(std::string path, line_parser_type line_parser) \"load()\"\n for details.\n */\n typedef boost::function<bool(distributed_graph&, const std::string&,\n const std::string&)> line_parser_type;\n\n\n typedef fixed_dense_bitset<RPC_MAX_N_PROCS> mirror_type;\n\n /// The type of the local graph used to store the graph data\n#ifdef USE_DYNAMIC_LOCAL_GRAPH\n typedef graphlab::dynamic_local_graph<VertexData, EdgeData> local_graph_type;\n#else \n typedef graphlab::local_graph<VertexData, EdgeData> local_graph_type;\n#endif\n typedef graphlab::distributed_graph<VertexData, EdgeData> graph_type;\n\n typedef std::vector<simple_spinlock> lock_manager_type;\n\n friend class distributed_ingress_base<VertexData, EdgeData>;\n\n\n // Make friends with graph operation classes \n template <typename Graph, typename GatherType>\n friend class graph_gather_apply;\n\n\n // Make friends with Ingress classes\n friend class distributed_random_ingress<VertexData, EdgeData>;\n friend class distributed_identity_ingress<VertexData, EdgeData>;\n friend class distributed_oblivious_ingress<VertexData, EdgeData>;\n friend class distributed_hdrf_ingress<VertexData, EdgeData>;\n friend class distributed_constrained_random_ingress<VertexData, EdgeData>;\n\n typedef graphlab::vertex_id_type vertex_id_type;\n typedef graphlab::lvid_type lvid_type;\n typedef graphlab::edge_id_type edge_id_type;\n\n struct vertex_type;\n typedef bool edge_list_type;\n class edge_type;\n\n struct local_vertex_type;\n struct local_edge_list_type;\n class local_edge_type;\n\n /**\n * \\brief Vertex object which provides access to the vertex data\n * and information about the vertex.\n *\n * The vertex_type object may be copied and has very little internal\n * state. It behaves as a reference to location of the vertex\n * in the internal graph representation. While vertex_type may be copied\n * it must not outlive the underlying graph.\n */\n struct vertex_type {\n typedef distributed_graph graph_type;\n distributed_graph& graph_ref;\n lvid_type lvid;\n\n /// \\cond GRAPHLAB_INTERNAL\n /** \\brief Constructs a vertex_type object with local vid\n * lvid. This function should not be used directly. Use\n * distributed_graph::vertex() or distributed_graph::l_vertex()\n *\n * \\param graph_ref A reference to the parent graph object\n * \\param lvid The local VID of the vertex to be accessed\n */\n vertex_type(distributed_graph& graph_ref, lvid_type lvid):\n graph_ref(graph_ref), lvid(lvid) { }\n /// \\endcond\n\n /// \\brief Compares two vertex_type's for equality. Returns true\n // if they reference the same vertex and false otherwise.\n bool operator==(vertex_type& v) const {\n return lvid == v.lvid;\n }\n\n /// \\brief Returns a constant reference to the data on the vertex\n const vertex_data_type& data() const {\n return graph_ref.get_local_graph().vertex_data(lvid);\n }\n\n /// \\brief Returns a mutable reference to the data on the vertex\n vertex_data_type& data() {\n return graph_ref.get_local_graph().vertex_data(lvid);\n }\n\n /// \\brief Returns the number of in edges of the vertex\n size_t num_in_edges() const {\n return graph_ref.l_get_vertex_record(lvid).num_in_edges;\n }\n\n /// \\brief Returns the number of out edges of the vertex\n size_t num_out_edges() const {\n return graph_ref.l_get_vertex_record(lvid).num_out_edges;\n }\n\n /// \\brief Returns the vertex ID of the vertex\n vertex_id_type id() const {\n return graph_ref.global_vid(lvid);\n }\n\n /// \\cond GRAPHLAB_INTERNAL\n\n /// \\brief Returns a list of in edges (not implemented)\n edge_list_type in_edges() __attribute__ ((noreturn)) {\n ASSERT_TRUE(false);\n }\n\n /// \\brief Returns a list of out edges (not implemented)\n edge_list_type out_edges() __attribute__ ((noreturn)) {\n ASSERT_TRUE(false);\n }\n /// \\endcond\n\n /**\n * \\brief Returns the local ID of the vertex\n */\n lvid_type local_id() const {\n return lvid;\n }\n\n };\n\n\n /**\n * \\brief The edge represents an edge in the graph and provide\n * access to the data associated with that edge as well as the\n * source and target distributed::vertex_type objects.\n *\n * An edge object may be copied and has very little internal state\n * and essentially only a reference to the location of the edge\n * information in the underlying graph. Therefore edge objects\n * can be copied but must not outlive the underlying graph.\n */\n class edge_type {\n private:\n /** \\brief An internal reference to the underlying graph */\n distributed_graph& graph_ref;\n /** \\brief The edge in the local graph */\n typename local_graph_type::edge_type edge;\n\n /**\n * \\internal\n * \\brief Constructs a edge_type object from a edge_type\n * object of the graphlab::local_graph.\n * lvid. This function should not be used directly.\n *\n * \\param graph_ref A reference to the parent graph object\n * \\param edge The local graph's edge_type to access\n */\n edge_type(distributed_graph& graph_ref,\n typename local_graph_type::edge_type edge):\n graph_ref(graph_ref), edge(edge) { }\n friend class distributed_graph;\n public:\n\n /** \n * \\internal\n * Unimplemented default constructor to help with\n * various type inference needs\n */\n edge_type();\n\n /**\n * \\brief Returns the source vertex of the edge.\n * This function returns a vertex_object by value and as a\n * consequence it is possible to use the resulting vertex object\n * to access and *modify* the associated vertex data.\n *\n * Modification of vertex data obtained through an edge object\n * is *usually not safe* and can lead to data corruption.\n *\n * \\return The vertex object representing the source vertex.\n */\n vertex_type source() const {\n return vertex_type(graph_ref, edge.source().id());\n }\n\n /**\n * \\brief Returns the target vertex of the edge.\n *\n * This function returns a vertex_object by value and as a\n * consequence it is possible to use the resulting vertex object\n * to access and *modify* the associated vertex data.\n *\n * Modification of vertex data obtained through an edge object\n * is *usually not safe* and can lead to data corruption.\n *\n * \\return The vertex object representing the target vertex.\n */\n vertex_type target() const {\n return vertex_type(graph_ref, edge.target().id());\n }\n\n /**\n * \\brief Returns a constant reference to the data on the edge\n */\n const edge_data_type& data() const { return edge.data(); }\n\n /**\n * \\brief Returns a mutable reference to the data on the edge\n */\n edge_data_type& data() { return edge.data(); }\n\n }; // end of edge_type\n\n\n\n\n\n // CONSTRUCTORS ==========================================================>\n\n /**\n * Constructs a distributed graph. All machines must call this constructor\n * simultaneously.\n *\n * Value graph options are:\n * \\li \\c ingress The graph partitioning method to use. May be \"random\"\n * \"grid\" or \"pds\". The methods have roughly the same runtime \n * complexity, but the increasing partition qaulity. \"grid\" \n * requires number of machine P be able to layout as a n*m = P \n * grid with ( |m-n| <= 2). \"pds\" uses requires P = p^2+p+1 where \n * p is a prime number.\n *\n * \\li \\c userecent An optimization that can decrease memory utilization\n * of oblivious and batch quite significantly (especially\n * when there are a large number of machines) at a small\n * partitioning penalty. Defaults to 0. Set to 1 to\n * enable.\n * \\li \\c bufsize The batch size used by the batch ingress method.\n * Defaults to 50,000. Increasing this number will\n * decrease partitioning time with a penalty to partitioning\n * quality.\n *\n * \\param [in] dc Distributed controller to associate with\n * \\param [in] opts A graphlab::graphlab_options object specifying engine\n * parameters. This is typically constructed using\n * \\ref graphlab::command_line_options.\n */\n distributed_graph(distributed_control& dc,\n const graphlab_options& opts = graphlab_options()) :\n rpc(dc, this), finalized(false), vid2lvid(),\n nverts(0), nedges(0), local_own_nverts(0), nreplicas(0),\n ingress_ptr(NULL), \n#ifdef _OPENMP\n vertex_exchange(dc, omp_get_max_threads()), \n#else\n vertex_exchange(dc), \n#endif\n vset_exchange(dc), parallel_ingress(true) {\n rpc.barrier();\n set_options(opts);\n }\n\n ~distributed_graph() {\n delete ingress_ptr; ingress_ptr = NULL;\n }\n\n\n lock_manager_type& get_lock_manager() {\n return lock_manager;\n }\n private:\n void set_options(const graphlab_options& opts) {\n size_t bufsize = 50000;\n bool usehash = false;\n bool userecent = false;\n std::string ingress_method = \"\";\n std::vector<std::string> keys = opts.get_graph_args().get_option_keys();\n foreach(std::string opt, keys) {\n if (opt == \"ingress\") {\n opts.get_graph_args().get_option(\"ingress\", ingress_method);\n if (rpc.procid() == 0)\n logstream(LOG_EMPH) << \"Graph Option: ingress = \"\n << ingress_method << std::endl;\n } else if (opt == \"parallel_ingress\") {\n opts.get_graph_args().get_option(\"parallel_ingress\", parallel_ingress);\n if (!parallel_ingress && rpc.procid() == 0)\n logstream(LOG_EMPH) << \"Disable parallel ingress. Graph will be streamed through one node.\"\n << std::endl;\n }\n /**\n * These options below are deprecated.\n */\n else if (opt == \"bufsize\") {\n opts.get_graph_args().get_option(\"bufsize\", bufsize);\n if (rpc.procid() == 0)\n logstream(LOG_EMPH) << \"Graph Option: bufsize = \"\n << bufsize << std::endl;\n } else if (opt == \"usehash\") {\n opts.get_graph_args().get_option(\"usehash\", usehash);\n if (rpc.procid() == 0)\n logstream(LOG_EMPH) << \"Graph Option: usehash = \"\n << usehash << std::endl;\n } else if (opt == \"userecent\") {\n opts.get_graph_args().get_option(\"userecent\", userecent);\n if (rpc.procid() == 0)\n logstream(LOG_EMPH) << \"Graph Option: userecent = \"\n << userecent << std::endl;\n } else {\n logstream(LOG_ERROR) << \"Unexpected Graph Option: \" << opt << std::endl;\n }\n }\n set_ingress_method(ingress_method, bufsize, usehash, userecent);\n }\n\n public:\n\n\n\n // METHODS ===============================================================>\n bool is_dynamic() const {\n return local_graph.is_dynamic();\n }\n \n /**\n * \\brief Commits the graph structure. Once a graph is finalized it may\n * no longer be modified. Must be called on all machines simultaneously.\n *\n * Finalize is used to complete graph ingress by resolving vertex\n * ownship and completing local data structures. Once a graph is finalized\n * its structure may not be modified. Repeated calls to finalize() do\n * nothing.\n */\n void finalize() {\n#ifndef USE_DYNAMIC_LOCAL_GRAPH\n if (finalized) return;\n#endif\n ASSERT_NE(ingress_ptr, NULL);\n logstream(LOG_INFO) << \"Distributed graph: enter finalize\" << std::endl;\n ingress_ptr->finalize();\n lock_manager.resize(num_local_vertices());\n rpc.barrier(); \n\n finalized = true;\n }\n\n /// \\brief Returns true if the graph is finalized.\n bool is_finalized() {\n return finalized;\n }\n\n /** \\brief Get the number of vertices */\n size_t num_vertices() const { return nverts; }\n\n /** \\brief Get the number of edges */\n size_t num_edges() const { return nedges; }\n\n /** \\brief converts a vertex ID to a vertex object. This function should\n * not be used without a deep understanding of the distributed graph\n * representation.\n *\n * This functions converts a global vertex ID to a vertex_type object.\n * The global vertex ID must exist on this machine or assertion failures\n * will be produced.\n */\n vertex_type vertex(vertex_id_type vid) {\n return vertex_type(*this, local_vid(vid));\n }\n\n /// \\cond GRAPHLAB_INTERNAL\n /** \\brief Get a list of all in edges of a given vertex ID. Not Implemented */\n edge_list_type in_edges(const vertex_id_type vid) const\n __attribute__((noreturn)) {\n // Not implemented.\n logstream(LOG_WARNING) << \"in_edges not implemented. \" << std::endl;\n ASSERT_TRUE(false);\n }\n\n /** Get a list of all out edges of a given vertex ID. Not Implemented */\n edge_list_type out_edges(const vertex_id_type vid) const\n __attribute__((noreturn)) {\n // Not implemented.\n logstream(LOG_WARNING) << \"in_edges not implemented. \" << std::endl;\n ASSERT_TRUE(false);\n }\n /// \\endcond\n\n\n\n /**\n * \\brief Returns the number of in edges of a given global vertex ID. This\n * function should not be used without a deep understanding of the\n * distributed graph representation.\n *\n * Returns the number of in edges of a given vertex ID. Equivalent to\n * vertex(vid).num_in_edges(). The global vertex ID must exist on this\n * machine or assertion failures will be produced.\n */\n size_t num_in_edges(const vertex_id_type vid) const {\n return get_vertex_record(vid).num_in_edges;\n }\n\n\n /**\n * \\brief Returns the number of out edges of a given global vertex ID. This\n * function should not be used without a deep understanding of the\n * distributed graph representation.\n *\n * Returns the number of out edges of a given vertex ID. Equivalent to\n * vertex(vid).num_out_edges(). The global vertex ID must exist on this\n * machine or assertion failures will be produced.\n */\n size_t num_out_edges(const vertex_id_type vid) const {\n return get_vertex_record(vid).num_out_edges;\n }\n\n\n /**\n * Defines the strategy to use when duplicate vertices are inserted.\n * The default behavior is that an arbitrary vertex data is picked.\n * This allows you to define a combining strategy.\n */\n void set_duplicate_vertex_strategy(boost::function<void(vertex_data_type&,\n const vertex_data_type&)>\n combine_strategy) {\n ingress_ptr->set_duplicate_vertex_strategy(combine_strategy);\n }\n\n /**\n * \\brief Creates a vertex containing the vertex data.\n *\n * Creates a vertex with a particular vertex ID and containing a\n * particular vertex data. Vertex IDs need not be sequential, and\n * may arbitrarily span the unsigned integer range of vertex_id_type\n * with the exception of (vertex_id_type)(-1), or corresponding to\n * 0xFFFFFFFF on 32-bit vertex IDs.\n *\n * This function is parallel and distributed. i.e. It does not matter which\n * machine, or which thread on which machines calls add_vertex() for a\n * particular ID.\n *\n * However, each vertex may only be added exactly once.\n *\n * Returns true if successful, returns false if a vertex with id (-1) \n * was added.\n */\n bool add_vertex(const vertex_id_type& vid,\n const VertexData& vdata = VertexData() ) {\n#ifndef USE_DYNAMIC_LOCAL_GRAPH\n if(finalized) {\n logstream(LOG_FATAL)\n << \"\\n\\tAttempting to add a vertex to a finalized graph.\"\n << \"\\n\\tVertices cannot be added to a graph after finalization.\"\n << std::endl;\n }\n#else\n finalized = false;\n#endif\n if(vid == vertex_id_type(-1)) {\n logstream(LOG_ERROR)\n << \"\\n\\tAdding a vertex with id -1 is not allowed.\"\n << \"\\n\\tThe -1 vertex id is reserved for internal use.\"\n << std::endl;\n return false;\n }\n ASSERT_NE(ingress_ptr, NULL);\n ingress_ptr->add_vertex(vid, vdata);\n return true;\n }\n\n\n /**\n * \\brief Creates an edge connecting vertex source, and vertex target().\n *\n * Creates a edge connecting two vertex IDs.\n *\n * This function is parallel and distributed. i.e. It does not matter which\n * machine, or which thread on which machines calls add_edge() for a\n * particular ID.\n *\n * However, each edge direction may only be added exactly once. i.e.\n * if edge 5->6 is added already, no other calls to add edge 5->6 should be\n * made.\n *\n * Returns true on success. Returns false if it is a self-edge, or if\n * we are trying to create a vertex with ID (vertex_id_type)(-1).\n */\n bool add_edge(vertex_id_type source, vertex_id_type target,\n const EdgeData& edata = EdgeData()) {\n\n#ifndef USE_DYNAMIC_LOCAL_GRAPH\n if(finalized) {\n logstream(LOG_FATAL)\n << \"\\n\\tAttempting to add an edge to a finalized graph.\"\n << \"\\n\\tEdges cannot be added to a graph after finalization.\"\n << std::endl;\n }\n#else \n finalized = false;\n#endif\n if(source == vertex_id_type(-1)) {\n logstream(LOG_ERROR)\n << \"\\n\\tThe source vertex with id vertex_id_type(-1)\\n\"\n << \"\\tor unsigned value \" << vertex_id_type(-1) << \" in edge \\n\"\n << \"\\t(\" << source << \"->\" << target << \") is not allowed.\\n\"\n << \"\\tThe -1 vertex id is reserved for internal use.\"\n << std::endl;\n return false;\n }\n if(target == vertex_id_type(-1)) {\n logstream(LOG_ERROR)\n << \"\\n\\tThe target vertex with id vertex_id_type(-1)\\n\"\n << \"\\tor unsigned value \" << vertex_id_type(-1) << \" in edge \\n\"\n << \"\\t(\" << source << \"->\" << target << \") is not allowed.\\n\"\n << \"\\tThe -1 vertex id is reserved for internal use.\"\n << std::endl;\n return false;\n }\n if(source == target) {\n logstream(LOG_ERROR)\n << \"\\n\\tTrying to add self edge (\" << source << \"->\" << target << \").\"\n << \"\\n\\tSelf edges are not allowed.\"\n << std::endl;\n return false;\n }\n ASSERT_NE(ingress_ptr, NULL);\n\n ingress_ptr->add_edge(source, target, edata);\n return true;\n }\n\n\n /**\n * \\brief Performs a map-reduce operation on each vertex in the\n * graph returning the result.\n *\n * Given a map function, map_reduce_vertices() call the map function on all\n * vertices in the graph. The return values are then summed together and the\n * final result returned. The map function should only read the vertex data\n * and should not make any modifications. map_reduce_vertices() must be\n * called on all machines simultaneously.\n *\n * ### Basic Usage\n * For instance, if the graph has float vertex data, and float edge data:\n * \\code\n * typedef graphlab::distributed_graph<float, float> graph_type;\n * \\endcode\n *\n * To compute an absolute sum over all the vertex data, we would write\n * a function which reads in each a vertex, and returns the absolute\n * value of the data on the vertex.\n * \\code\n * float absolute_vertex_data(const graph_type::vertex_type& vertex) {\n * return std::fabs(vertex.data());\n * }\n * \\endcode\n * After which calling:\n * \\code\n * float sum = graph.map_reduce_vertices<float>(absolute_vertex_data);\n * \\endcode\n * will call the <code>absolute_vertex_data()</code> function\n * on each vertex in the graph. <code>absolute_vertex_data()</code>\n * reads the value of the vertex and returns the absolute result.\n * This return values are then summed together and returned.\n * All machines see the same result.\n *\n * The template argument <code><float></code> is needed to inform\n * the compiler regarding the return type of the mapfunction.\n *\n * The optional argument vset can be used to restrict he set of vertices\n * map-reduced over.\n *\n * ### Relations\n * This function is similar to\n * graphlab::iengine::map_reduce_vertices()\n * with the difference that this does not take a context\n * and thus cannot influence engine signalling.\n * transform_vertices() can be used to perform a similar\n * but may also make modifications to graph data.\n *\n * \\tparam ReductionType The output of the map function. Must have\n * operator+= defined, and must be \\ref sec_serializable.\n * \\tparam VertexMapperType The type of the map function.\n * Not generally needed.\n * Can be inferred by the compiler.\n * \\param mapfunction The map function to use. Must take\n * a \\ref vertex_type, or a reference to a\n * \\ref vertex_type as its only argument.\n * Returns a ReductionType which must be summable\n * and \\ref sec_serializable .\n * \\param vset The set of vertices to map reduce over. Optional. Defaults to\n * complete_set()\n */\n template <typename ReductionType, typename MapFunctionType>\n ReductionType map_reduce_vertices(MapFunctionType mapfunction,\n const vertex_set& vset = complete_set()) {\n BOOST_CONCEPT_ASSERT((graphlab::Serializable<ReductionType>));\n BOOST_CONCEPT_ASSERT((graphlab::OpPlusEq<ReductionType>));\n if(!finalized) {\n logstream(LOG_FATAL)\n << \"\\n\\tAttempting to run graph.map_reduce_vertices(...) \"\n << \"\\n\\tbefore calling graph.finalize().\"\n << std::endl;\n }\n\n rpc.barrier();\n bool global_result_set = false;\n ReductionType global_result = ReductionType();\n#ifdef _OPENMP\n#pragma omp parallel\n#endif\n {\n bool result_set = false;\n ReductionType result = ReductionType();\n#ifdef _OPENMP\n #pragma omp for\n#endif\n for (int i = 0; i < (int)local_graph.num_vertices(); ++i) {\n if (lvid2record[i].owner == rpc.procid() &&\n vset.l_contains((lvid_type)i)) {\n if (!result_set) {\n const vertex_type vtx(l_vertex(i));\n result = mapfunction(vtx);\n result_set = true;\n }\n else if (result_set){\n const vertex_type vtx(l_vertex(i));\n const ReductionType tmp = mapfunction(vtx);\n result += tmp;\n }\n }\n }\n#ifdef _OPENMP\n #pragma omp critical\n#endif\n {\n if (result_set) {\n if (!global_result_set) {\n global_result = result;\n global_result_set = true;\n }\n else {\n global_result += result;\n }\n }\n }\n }\n conditional_addition_wrapper<ReductionType>\n wrapper(global_result, global_result_set);\n rpc.all_reduce(wrapper);\n return wrapper.value;\n } // end of map_reduce_vertices\n\n /**\n * \\brief Performs a map-reduce operation on each edge in the\n * graph returning the result.\n *\n * Given a map function, map_reduce_edges() call the map function on all\n * edges in the graph. The return values are then summed together and the\n * final result returned. The map function should only read data\n * and should not make any modifications. map_reduce_edges() must be\n * called on all machines simultaneously.\n *\n * ### Basic Usage\n * For instance, if the graph has float vertex data, and float edge data:\n * \\code\n * typedef graphlab::distributed_graph<float, float> graph_type;\n * \\endcode\n *\n * To compute an absolute sum over all the edge data, we would write\n * a function which reads in each a edge, and returns the absolute\n * value of the data on the edge.\n * \\code\n * float absolute_edge_datac(const graph_type::edge_type& edge) {\n * return std::fabs(edge.data());\n * }\n * \\endcode\n * After which calling:\n * \\code\n * float sum = graph.map_reduce_edges<float>(absolute_edge_data);\n * \\endcode\n * will call the <code>absolute_edge_data()</code> function\n * on each edge in the graph. <code>absolute_edge_data()</code>\n * reads the value of the edge and returns the absolute result.\n * This return values are then summed together and returned.\n * All machines see the same result.\n *\n * The template argument <code><float></code> is needed to inform\n * the compiler regarding the return type of the mapfunction.\n *\n * The two optional arguments vset and edir can be used to restrict the\n * set of edges which are map-reduced over.\n *\n * ### Relations\n * This function similar to\n * graphlab::distributed_graph::map_reduce_edges()\n * with the difference that this does not take a context\n * and thus cannot influence engine signalling.\n * Finally transform_edges() can be used to perform a similar\n * but may also make modifications to graph data.\n *\n * \\tparam ReductionType The output of the map function. Must have\n * operator+= defined, and must be \\ref sec_serializable.\n * \\tparam EdgeMapperType The type of the map function.\n * Not generally needed.\n * Can be inferred by the compiler.\n * \\param mapfunction The map function to use. Must take\n * a \\ref edge_type, or a reference to a\n * \\ref edge_type as its only argument.\n * Returns a ReductionType which must be summable\n * and \\ref sec_serializable .\n * \\param vset A set of vertices. Combines with\n * edir to identify the set of edges. For instance, if\n * edir == IN_EDGES, map_reduce_edges will map over all in edges\n * of the vertices in vset. Optional. Defaults to complete_set().\n * \\param edir An edge direction. Combines with vset to identify the set\n * of edges to map over. For instance, if\n * edir == IN_EDGES, map_reduce_edges will map over all in edges\n * of the vertices in vset. Optional. Defaults to IN_EDGES.\n */\n template <typename ReductionType, typename MapFunctionType>\n ReductionType map_reduce_edges(MapFunctionType mapfunction,\n const vertex_set& vset = complete_set(),\n edge_dir_type edir = IN_EDGES) {\n BOOST_CONCEPT_ASSERT((graphlab::Serializable<ReductionType>));\n BOOST_CONCEPT_ASSERT((graphlab::OpPlusEq<ReductionType>));\n if(!finalized) {\n logstream(LOG_FATAL)\n << \"\\n\\tAttempting to run graph.map_reduce_vertices(...)\"\n << \"\\n\\tbefore calling graph.finalize().\"\n << std::endl;\n }\n\n rpc.barrier();\n bool global_result_set = false;\n ReductionType global_result = ReductionType();\n#ifdef _OPENMP\n#pragma omp parallel\n#endif\n {\n bool result_set = false;\n ReductionType result = ReductionType();\n#ifdef _OPENMP\n #pragma omp for\n#endif\n for (int i = 0; i < (int)local_graph.num_vertices(); ++i) {\n if (vset.l_contains((lvid_type)i)) {\n if (edir == IN_EDGES || edir == ALL_EDGES) {\n foreach(const local_edge_type& e, l_vertex(i).in_edges()) {\n if (!result_set) {\n edge_type edge(e);\n result = mapfunction(edge);\n result_set = true;\n }\n else if (result_set){\n edge_type edge(e);\n const ReductionType tmp = mapfunction(edge);\n result += tmp;\n }\n }\n }\n if (edir == OUT_EDGES || edir == ALL_EDGES) {\n foreach(const local_edge_type& e, l_vertex(i).out_edges()) {\n if (!result_set) {\n edge_type edge(e);\n result = mapfunction(edge);\n result_set = true;\n }\n else if (result_set){\n edge_type edge(e);\n const ReductionType tmp = mapfunction(edge);\n result += tmp;\n }\n }\n }\n }\n }\n#ifdef _OPENMP\n #pragma omp critical\n#endif\n {\n if (result_set) {\n if (!global_result_set) {\n global_result = result;\n global_result_set = true;\n }\n else {\n global_result += result;\n }\n }\n }\n }\n\n conditional_addition_wrapper<ReductionType>\n wrapper(global_result, global_result_set);\n rpc.all_reduce(wrapper);\n return wrapper.value;\n } // end of map_reduce_edges\n\n\n\n /**\n * \\brief Performs a fold operation on each vertex in the\n * graph returning the result.\n *\n * Given a fold function, fold_vertices() call the fold function on all\n * vertices in the graph, passing around a aggregator variable.\n * The return values are then summed together across machines using the\n * combiner function and the final result returned. The fold function should\n * only read the vertex data and should not make any modifications.\n * fold_vertices() must be called on all machines simultaneously.\n *\n * ### Basic Usage\n * For instance, if the graph has float vertex data, and float edge data:\n * \\code\n * typedef graphlab::distributed_graph<float, float> graph_type;\n * \\endcode\n *\n * To compute an absolute sum over all the vertex data, we would write\n * a function which reads in each a vertex, and returns the absolute\n * value of the data on the vertex.\n * \\code\n * void absolute_vertex_data(const graph_type::vertex_type& vertex, float& total) {\n * total += std::fabs(vertex.data());\n * }\n * \\endcode\n * After which calling:\n * \\code\n * float sum = graph.fold_vertices<float>(absolute_vertex_data);\n * \\endcode\n * will call the <code>absolute_vertex_data()</code> function\n * on each vertex in the graph. <code>absolute_vertex_data()</code>\n * reads the value of the vertex and returns the absolute result.\n * This return values are then summed together and returned.\n * All machines see the same result.\n *\n * The template argument <code><float></code> is needed to inform\n * the compiler regarding the return type of the fold function.\n *\n * The optional argument vset can be used to restrict he set of vertices\n * map-reduced over.\n *\n * Unlike map_reduce_vertices, this function exposes to a certain extent,\n * the internals of the parallelism structure since the fold is used within\n * a thread, but across threads/machines operator+= is used. The behavior\n * of the foldfunction, or the behavior of the return type should not make\n * assumptions on the undocumented behavior of this function (such as \n * when the fold is used, and when += is used).\n *\n * ### Relations\n * This function is similar to\n * map_reduce_vertices()\n * with the difference that this uses a fold and is hence more efficient\n * for large aggregation objects.\n * transform_vertices() can be used to perform a similar\n * but may also make modifications to graph data.\n *\n * \\tparam ReductionType The output of the map function. Must have\n * operator+= defined, and must be \\ref sec_serializable.\n * \\tparam VertexFoldType The type of the fold function.\n * Not generally needed.\n * Can be inferred by the compiler.\n * \\param foldfunction The fold function to use. Must take\n * a \\ref vertex_type, or a reference to a\n * \\ref vertex_type as its first argument, and a \n * reference to a ReductionType in its second argument.\n * \\param vset The set of vertices to fold reduce over. Optional. Defaults to\n * complete_set()\n */\n template <typename ReductionType, typename VertexFoldType>\n ReductionType fold_vertices(VertexFoldType foldfunction,\n const vertex_set& vset = complete_set()) {\n BOOST_CONCEPT_ASSERT((graphlab::Serializable<ReductionType>));\n BOOST_CONCEPT_ASSERT((graphlab::OpPlusEq<ReductionType>));\n if(!finalized) {\n logstream(LOG_FATAL)\n << \"\\n\\tAttempting to run graph.map_reduce_vertices(...) \"\n << \"\\n\\tbefore calling graph.finalize().\"\n << std::endl;\n }\n\n rpc.barrier();\n bool global_result_set = false;\n ReductionType global_result = ReductionType();\n#ifdef _OPENMP\n#pragma omp parallel\n#endif\n {\n ReductionType result = ReductionType();\n#ifdef _OPENMP\n #pragma omp for\n#endif\n for (int i = 0; i < (int)local_graph.num_vertices(); ++i) {\n if (lvid2record[i].owner == rpc.procid() &&\n vset.l_contains((lvid_type)i)) {\n const vertex_type vtx(l_vertex(i));\n foldfunction(vtx, result);\n }\n }\n#ifdef _OPENMP\n #pragma omp critical\n#endif\n {\n if (!global_result_set) {\n global_result = result;\n global_result_set = true;\n }\n else {\n global_result += result;\n }\n }\n }\n conditional_addition_wrapper<ReductionType>\n wrapper(global_result, global_result_set);\n rpc.all_reduce(wrapper);\n return wrapper.value;\n } \n\n\n\n /**\n * \\brief Performs a fold operation on each edge in the\n * graph returning the result.\n *\n * Given a fold function, fold_edges() call the fold function on all\n * edges in the graph passing an aggregator. \n * The return values are then summed together across machines and \n * final result returned. The fold function should only read data\n * and should not make any modifications. fold_edges() must be\n * called on all machines simultaneously.\n *\n * ### Basic Usage\n * For instance, if the graph has float vertex data, and float edge data:\n * \\code\n * typedef graphlab::distributed_graph<float, float> graph_type;\n * \\endcode\n *\n * To compute an absolute sum over all the edge data, we would write\n * a function which reads in each a edge, and returns the absolute\n * value of the data on the edge.\n * \\code\n * void absolute_edge_data(const graph_type::edge_type& edge, float& acc) {\n * acc += std::fabs(edge.data());\n * }\n * \\endcode\n * After which calling:\n * \\code\n * float sum = graph.fold_edges<float>(absolute_edge_data);\n * \\endcode\n * will call the <code>absolute_edge_data()</code> function\n * on each edge in the graph. <code>absolute_edge_data()</code>\n * reads the value of the edge and returns the absolute result.\n * This return values are then summed together and returned.\n * All machines see the same result.\n *\n * The template argument <code><float></code> is needed to inform\n * the compiler regarding the return type of the foldfunction.\n *\n * The two optional arguments vset and edir can be used to restrict the\n * set of edges which are map-reduced over.\n *\n * ### Relations\n * This function similar to\n * graphlab::distributed_graph::map_reduce_edges()\n * with the difference that this uses a fold and is hence more efficient\n * for large aggregation objects.\n * Finally transform_edges() can be used to perform a similar\n * but may also make modifications to graph data.\n *\n * \\tparam ReductionType The output of the map function. Must have\n * operator+= defined, and must be \\ref sec_serializable.\n * \\tparam EdgeFoldType The type of the Fold function.\n * Not generally needed.\n * Can be inferred by the compiler.\n * \\param fold function The map function to use. Must take\n * a \\ref edge_type, or a reference to a\n * \\ref edge_type as its first argument, and\n * a reference to a ReductionType in its second argument.\n * \\param vset A set of vertices. Combines with\n * edir to identify the set of edges. For instance, if\n * edir == IN_EDGES, map_reduce_edges will map over all in edges\n * of the vertices in vset. Optional. Defaults to complete_set().\n * \\param edir An edge direction. Combines with vset to identify the set\n * of edges to map over. For instance, if\n * edir == IN_EDGES, map_reduce_edges will map over all in edges\n * of the vertices in vset. Optional. Defaults to IN_EDGES.\n */\n template <typename ReductionType, typename FoldFunctionType>\n ReductionType fold_edges(FoldFunctionType foldfunction,\n const vertex_set& vset = complete_set(),\n edge_dir_type edir = IN_EDGES) {\n BOOST_CONCEPT_ASSERT((graphlab::Serializable<ReductionType>));\n BOOST_CONCEPT_ASSERT((graphlab::OpPlusEq<ReductionType>));\n if(!finalized) {\n logstream(LOG_FATAL)\n << \"\\n\\tAttempting to run graph.map_reduce_vertices(...)\"\n << \"\\n\\tbefore calling graph.finalize().\"\n << std::endl;\n }\n\n rpc.barrier();\n bool global_result_set = false;\n ReductionType global_result = ReductionType();\n#ifdef _OPENMP\n#pragma omp parallel\n#endif\n {\n ReductionType result = ReductionType();\n#ifdef _OPENMP\n #pragma omp for\n#endif\n for (int i = 0; i < (int)local_graph.num_vertices(); ++i) {\n if (vset.l_contains((lvid_type)i)) {\n if (edir == IN_EDGES || edir == ALL_EDGES) {\n foreach(const local_edge_type& e, l_vertex(i).in_edges()) {\n edge_type edge(e);\n foldfunction(edge, result);\n }\n }\n if (edir == OUT_EDGES || edir == ALL_EDGES) {\n foreach(const local_edge_type& e, l_vertex(i).out_edges()) {\n edge_type edge(e);\n foldfunction(edge, result);\n }\n }\n }\n }\n#ifdef _OPENMP\n #pragma omp critical\n#endif\n {\n if (!global_result_set) {\n global_result = result;\n global_result_set = true;\n }\n else {\n global_result += result;\n }\n }\n }\n\n conditional_addition_wrapper<ReductionType>\n wrapper(global_result, global_result_set);\n rpc.all_reduce(wrapper);\n return wrapper.value;\n } // end of map_reduce_edges\n\n\n\n /**\n * \\brief Performs a transformation operation on each vertex in the graph.\n *\n * Given a mapfunction, transform_vertices() calls mapfunction on\n * every vertex in graph. The map function may make modifications\n * to the data on the vertex. transform_vertices() must be called by all\n * machines simultaneously.\n *\n * The optional vset argument may be used to restrict the set of vertices\n * operated upon.\n *\n * ### Basic Usage\n * For instance, if the graph has integer vertex data, and integer edge\n * data:\n * \\code\n * typedef graphlab::distributed_graph<size_t, size_t> graph_type;\n * \\endcode\n *\n * To set each vertex value to be the number of out-going edges,\n * we may write the following function:\n * \\code\n * void set_vertex_value(graph_type::vertex_type& vertex)i {\n * vertex.data() = vertex.num_out_edges();\n * }\n * \\endcode\n *\n * Calling transform_vertices():\n * \\code\n * graph.transform_vertices(set_vertex_value);\n * \\endcode\n * will run the <code>set_vertex_value()</code> function\n * on each vertex in the graph, setting its new value.\n *\n * ### Relations\n * map_reduce_vertices() provide similar signalling functionality,\n * but should not make modifications to graph data.\n * graphlab::iengine::transform_vertices() provide\n * the same graph modification capabilities, but with a context\n * and thus can perform signalling.\n *\n * \\tparam VertexMapperType The type of the map function.\n * Not generally needed.\n * Can be inferred by the compiler.\n * \\param mapfunction The map function to use. Must take an\n * \\ref icontext_type& as its first argument, and\n * a \\ref vertex_type, or a reference to a\n * \\ref vertex_type as its second argument.\n * Returns void.\n * \\param vset The set of vertices to transform. Optional. Defaults to\n * complete_set()\n */\n template <typename TransformType>\n void transform_vertices(TransformType transform_functor,\n const vertex_set vset = complete_set()) {\n if(!finalized) {\n logstream(LOG_FATAL)\n << \"\\n\\tAttempting to call graph.transform_vertices(...)\"\n << \"\\n\\tbefore finalizing the graph.\"\n << std::endl;\n }\n\n rpc.barrier();\n#ifdef _OPENMP\n #pragma omp parallel for\n#endif\n for (int i = 0; i < (int)local_graph.num_vertices(); ++i) {\n if (lvid2record[i].owner == rpc.procid() &&\n vset.l_contains((lvid_type)i)) {\n vertex_type vtx(l_vertex(i));\n transform_functor(vtx);\n }\n }\n rpc.barrier();\n synchronize();\n }\n\n /**\n * \\brief Performs a transformation operation on each edge in the graph.\n *\n * Given a mapfunction, transform_edges() calls mapfunction on\n * every edge in graph. The map function may make modifications\n * to the data on the edge. transform_edges() must be called on\n * all machines simultaneously.\n *\n * ### Basic Usage\n * For instance, if the graph has integer vertex data, and integer edge\n * data:\n * \\code\n * typedef graphlab::distributed_graph<size_t, size_t> graph_type;\n * \\endcode\n *\n * To set each edge value to be the number of out-going edges\n * of the target vertex, we may write the following:\n * \\code\n * void set_edge_value(graph_type::edge_type& edge) {\n * edge.data() = edge.target().num_out_edges();\n * }\n * \\endcode\n *\n * Calling transform_edges():\n * \\code\n * graph.transform_edges(set_edge_value);\n * \\endcode\n * will run the <code>set_edge_value()</code> function\n * on each edge in the graph, setting its new value.\n *\n * The two optional arguments vset and edir may be used to restrict\n * the set of edges operated upon.\n *\n * ### Relations\n * map_reduce_edges() provide similar signalling functionality,\n * but should not make modifications to graph data.\n * graphlab::iengine::transform_edges() provide\n * the same graph modification capabilities, but with a context\n * and thus can perform signalling.\n *\n * \\tparam EdgeMapperType The type of the map function.\n * Not generally needed.\n * Can be inferred by the compiler.\n * \\param mapfunction The map function to use. Must take an\n * \\ref icontext_type& as its first argument, and\n * a \\ref edge_type, or a reference to a\n * \\ref edge_type as its second argument.\n * Returns void.\n * \\param vset A set of vertices. Combines with\n * edir to identify the set of edges. For instance, if\n * edir == IN_EDGES, map_reduce_edges will map over all in edges\n * of the vertices in vset. Optional. Defaults to complete_set().\n * \\param edir An edge direction. Combines with vset to identify the set\n * of edges to map over. For instance, if\n * edir == IN_EDGES, map_reduce_edges will map over all in edges\n * of the vertices in vset. Optional. Defaults to IN_EDGES.\n */\n template <typename TransformType>\n void transform_edges(TransformType transform_functor,\n const vertex_set& vset = complete_set(),\n edge_dir_type edir = IN_EDGES) {\n if(!finalized) {\n logstream(LOG_FATAL)\n << \"\\n\\tAttempting to call graph.transform_edges(...)\"\n << \"\\n\\tbefore finalizing the graph.\"\n << std::endl;\n }\n rpc.barrier();\n#ifdef _OPENMP\n #pragma omp parallel for\n#endif\n for (int i = 0; i < (int)local_graph.num_vertices(); ++i) {\n if (vset.l_contains((lvid_type)i)) {\n if (edir == IN_EDGES || edir == ALL_EDGES) {\n foreach(const local_edge_type& e, l_vertex(i).in_edges()) {\n edge_type edge(e);\n transform_functor(edge);\n }\n }\n if (edir == OUT_EDGES || edir == ALL_EDGES) {\n foreach(const local_edge_type& e, l_vertex(i).out_edges()) {\n edge_type edge(e);\n transform_functor(edge);\n }\n }\n }\n }\n rpc.barrier();\n }\n\n // disable documentation for parallel_for stuff. These are difficult\n // to use properly by the user\n /// \\cond GRAPHLAB_INTERNAL\n /**\n * \\internal\n * parallel_for_vertices will partition the set of vertices among the\n * vector of accfunctions. Each accfunction is then executed sequentially\n * on the set of vertices it was assigned.\n *\n * \\param accfunction must be a void function which takes a single\n * vertex_type argument. It may be a functor and contain state.\n * The function need not be reentrant as it is only called sequentially\n */\n template <typename VertexFunctorType>\n void parallel_for_vertices(std::vector<VertexFunctorType>& accfunction) {\n ASSERT_TRUE(finalized);\n rpc.barrier();\n int numaccfunctions = (int)accfunction.size();\n ASSERT_GE(numaccfunctions, 1);\n#ifdef _OPENMP\n #pragma omp parallel for\n#endif\n for (int i = 0; i < (int)accfunction.size(); ++i) {\n for (int j = i;j < (int)local_graph.num_vertices(); j+=numaccfunctions) {\n if (lvid2record[j].owner == rpc.procid()) {\n accfunction[i](vertex_type(l_vertex(j)));\n }\n }\n }\n rpc.barrier();\n }\n\n\n /**\n * \\internal\n * parallel_for_edges will partition the set of edges among the\n * vector of accfunctions. Each accfunction is then executed sequentially\n * on the set of edges it was assigned.\n *\n * \\param accfunction must be a void function which takes a single\n * edge_type argument. It may be a functor and contain state.\n * The function need not be reentrant as it is only called sequentially\n */\n template <typename EdgeFunctorType>\n void parallel_for_edges(std::vector<EdgeFunctorType>& accfunction) {\n ASSERT_TRUE(finalized);\n rpc.barrier();\n int numaccfunctions = (int)accfunction.size();\n ASSERT_GE(numaccfunctions, 1);\n#ifdef _OPENMP\n #pragma omp parallel for\n#endif\n for (int i = 0; i < (int)accfunction.size(); ++i) {\n for (int j = i;j < (int)local_graph.num_vertices(); j+=numaccfunctions) {\n foreach(const local_edge_type& e, l_vertex(j).in_edges()) {\n accfunction[i](edge_type(e));\n }\n }\n }\n rpc.barrier();\n }\n\n\n\n /** \\brief Load the graph from an archive */\n void load(iarchive& arc) {\n // read the vertices\n arc >> nverts\n >> nedges\n >> local_own_nverts\n >> nreplicas\n >> vid2lvid\n >> lvid2record\n >> local_graph;\n finalized = true;\n // check the graph condition\n } // end of load\n\n\n\n /** \\brief Save the graph to an archive */\n void save(oarchive& arc) const {\n if(!finalized) {\n logstream(LOG_FATAL)\n << \"\\n\\tAttempting to save a graph before calling graph.finalize().\"\n << std::endl;\n }\n // Write the number of edges and vertices\n arc << nverts\n << nedges\n << local_own_nverts\n << nreplicas\n << vid2lvid\n << lvid2record\n << local_graph;\n } // end of save\n\n /// \\endcond\n\n /// \\brief Clears and resets the graph, releasing all memory used.\n void clear () {\n foreach (vertex_record& vrec, lvid2record)\n vrec.clear();\n lvid2record.clear();\n vid2lvid.clear();\n local_graph.clear();\n finalized=false;\n nverts = nedges = local_own_nverts = nreplicas = 0;\n }\n\n\n /** \\brief Load a distributed graph from a native binary format\n * previously saved with save_binary(). This function must be called\n * simultaneously on all machines.\n *\n * This function loads a sequence of files numbered\n * \\li [prefix].0.gz\n * \\li [prefix].1.gz\n * \\li [prefix].2.gz\n * \\li etc.\n *\n * These files must be previously saved using save_binary(), and\n * must be saved <b>using the same number of machines</b>.\n * This function uses the graphlab serialization system, so\n * the user must ensure that the vertex data and edge data\n * serialization formats have not changed since the graph was saved.\n *\n * A graph loaded using load_binary() is already finalized and\n * structure modifications are not permitted after loading.\n *\n * Return true on success and false on failure if the file cannot be loaded.\n */\n bool load_binary(const std::string& prefix) {\n rpc.full_barrier();\n std::string fname = prefix + tostr(rpc.procid()) + \".bin\";\n\n logstream(LOG_INFO) << \"Load graph from \" << fname << std::endl;\n if(boost::starts_with(fname, \"hdfs://\")) {\n graphlab::hdfs hdfs;\n graphlab::hdfs::fstream in_file(hdfs, fname);\n boost::iostreams::filtering_stream<boost::iostreams::input> fin;\n fin.push(boost::iostreams::gzip_decompressor());\n fin.push(in_file);\n\n if(!fin.good()) {\n logstream(LOG_ERROR) << \"\\n\\tError opening file: \" << fname << std::endl;\n return false;\n }\n iarchive iarc(fin);\n iarc >> *this;\n fin.pop();\n fin.pop();\n in_file.close();\n } else {\n std::ifstream in_file(fname.c_str(),\n std::ios_base::in | std::ios_base::binary);\n if(!in_file.good()) {\n logstream(LOG_ERROR) << \"\\n\\tError opening file: \" << fname << std::endl;\n return false;\n }\n boost::iostreams::filtering_stream<boost::iostreams::input> fin;\n fin.push(boost::iostreams::gzip_decompressor());\n fin.push(in_file);\n iarchive iarc(fin);\n iarc >> *this;\n fin.pop();\n fin.pop();\n in_file.close();\n }\n logstream(LOG_INFO) << \"Finish loading graph from \" << fname << std::endl;\n rpc.full_barrier();\n return true;\n } // end of load\n\n\n /** \\brief Saves a distributed graph to a native binary format\n * which can be loaded with load_binary(). This function must be called\n * simultaneously on all machines.\n *\n * This function saves a sequence of files numbered\n * \\li [prefix].0.gz\n * \\li [prefix].1.gz\n * \\li [prefix].2.gz\n * \\li etc.\n *\n * This files can be loaded with load_binary() using the <b> same number\n * of machines</b>.\n * This function uses the graphlab serialization system, so\n * the vertex data and edge data serialization formats must not\n * change between the use of save_binary() and load_binary().\n *\n * If the graph is not alreasy finalized before save_binary() is called,\n * this function will finalize the graph.\n *\n * Returns true on success, and false if the graph cannot be loaded from\n * the specified file.\n */\n bool save_binary(const std::string& prefix) {\n rpc.full_barrier();\n finalize();\n timer savetime; savetime.start();\n std::string fname = prefix + tostr(rpc.procid()) + \".bin\";\n logstream(LOG_INFO) << \"Save graph to \" << fname << std::endl;\n if(boost::starts_with(fname, \"hdfs://\")) {\n graphlab::hdfs hdfs;\n graphlab::hdfs::fstream out_file(hdfs, fname, true);\n boost::iostreams::filtering_stream<boost::iostreams::output> fout;\n fout.push(boost::iostreams::gzip_compressor());\n fout.push(out_file);\n if (!fout.good()) {\n logstream(LOG_ERROR) << \"\\n\\tError opening file: \" << fname << std::endl;\n return false;\n }\n oarchive oarc(fout);\n oarc << *this;\n fout.pop();\n fout.pop();\n out_file.close();\n } else {\n std::ofstream out_file(fname.c_str(),\n std::ios_base::out | std::ios_base::binary);\n if (!out_file.good()) {\n logstream(LOG_ERROR) << \"\\n\\tError opening file: \" << fname << std::endl;\n return false;\n }\n boost::iostreams::filtering_stream<boost::iostreams::output> fout;\n fout.push(boost::iostreams::gzip_compressor());\n fout.push(out_file);\n oarchive oarc(fout);\n oarc << *this;\n fout.pop();\n fout.pop();\n out_file.close();\n }\n logstream(LOG_INFO) << \"Finish saving graph to \" << fname << std::endl\n << \"Finished saving binary graph: \"\n << savetime.current_time() << std::endl;\n rpc.full_barrier();\n return true;\n } // end of save\n\n\n /**\n * \\brief Saves the graph to the filesystem using a provided Writer object.\n * Like \\ref save(const std::string& prefix, writer writer, bool gzip, bool save_vertex, bool save_edge, size_t files_per_machine) \"save()\"\n * but only saves to local filesystem.\n */\n template<typename Writer>\n void save_to_posixfs(const std::string& prefix, Writer writer,\n bool gzip = true,\n bool save_vertex = true,\n bool save_edge = true,\n size_t files_per_machine = 4) {\n typedef boost::function<void(vertex_type)> vertex_function_type;\n typedef boost::function<void(edge_type)> edge_function_type;\n typedef std::ofstream base_fstream_type;\n typedef boost::iostreams::filtering_stream<boost::iostreams::output>\n boost_fstream_type;\n rpc.full_barrier();\n finalize();\n // figure out the filenames\n std::vector<std::string> graph_files;\n std::vector<base_fstream_type*> outstreams;\n std::vector<boost_fstream_type*> booststreams;\n graph_files.resize(files_per_machine);\n for(size_t i = 0; i < files_per_machine; ++i) {\n //graph_files[i] = prefix + \"_\" + tostr(1 + i + rpc.procid() * files_per_machine)\n graph_files[i] = prefix + \".\" + tostr(1 + i + rpc.procid() * files_per_machine)\n + \"_of_\" + tostr(rpc.numprocs() * files_per_machine);\n if (gzip) graph_files[i] += \".gz\";\n }\n\n // create the vector of callbacks\n std::vector<vertex_function_type> vertex_callbacks(graph_files.size());\n std::vector<edge_function_type> edge_callbacks(graph_files.size());\n\n for(size_t i = 0; i < graph_files.size(); ++i) {\n logstream(LOG_INFO) << \"Saving to file: \" << graph_files[i] << std::endl;\n // open the stream\n base_fstream_type* out_file =\n new base_fstream_type(graph_files[i].c_str(),\n std::ios_base::out | std::ios_base::binary);\n // attach gzip if the file is gzip\n boost_fstream_type* fout = new boost_fstream_type;\n // Using gzip filter\n if (gzip) fout->push(boost::iostreams::gzip_compressor());\n fout->push(*out_file);\n\n outstreams.push_back(out_file);\n booststreams.push_back(fout);\n // construct the callback for the parallel for\n typedef distributed_graph<vertex_data_type, edge_data_type> graph_type;\n vertex_callbacks[i] =\n boost::bind(&graph_type::template save_vertex_to_stream<boost_fstream_type, Writer>,\n this, _1, boost::ref(*fout), boost::ref(writer));\n edge_callbacks[i] =\n boost::bind(&graph_type::template save_edge_to_stream<boost_fstream_type, Writer>,\n this, _1, boost::ref(*fout), boost::ref(writer));\n }\n\n if (save_vertex) parallel_for_vertices(vertex_callbacks);\n if (save_edge) parallel_for_edges(edge_callbacks);\n\n // cleanup\n for(size_t i = 0; i < graph_files.size(); ++i) {\n booststreams[i]->pop();\n if (gzip) booststreams[i]->pop();\n delete booststreams[i];\n delete outstreams[i];\n }\n vertex_callbacks.clear();\n edge_callbacks.clear();\n outstreams.clear();\n booststreams.clear();\n rpc.full_barrier();\n } // end of save to posixfs\n\n\n\n /**\n * \\brief Saves the graph to HDFS using a provided Writer object.\n * Like \\ref save(const std::string& prefix, writer writer, bool gzip, bool save_vertex, bool save_edge, size_t files_per_machine) \"save()\"\n * but only saves to HDFS.\n */\n template<typename Writer>\n void save_to_hdfs(const std::string& prefix, Writer writer,\n bool gzip = true,\n bool save_vertex = true,\n bool save_edge = true,\n size_t files_per_machine = 4) {\n typedef boost::function<void(vertex_type)> vertex_function_type;\n typedef boost::function<void(edge_type)> edge_function_type;\n typedef graphlab::hdfs::fstream base_fstream_type;\n typedef boost::iostreams::filtering_stream<boost::iostreams::output>\n boost_fstream_type;\n rpc.full_barrier();\n finalize();\n // figure out the filenames\n std::vector<std::string> graph_files;\n std::vector<base_fstream_type*> outstreams;\n std::vector<boost_fstream_type*> booststreams;\n graph_files.resize(files_per_machine);\n for(size_t i = 0; i < files_per_machine; ++i) {\n graph_files[i] = prefix + \"_\" + tostr(1 + i + rpc.procid() * files_per_machine)\n + \"_of_\" + tostr(rpc.numprocs() * files_per_machine);\n if (gzip) graph_files[i] += \".gz\";\n }\n\n if(!hdfs::has_hadoop()) {\n logstream(LOG_FATAL)\n << \"\\n\\tAttempting to save a graph to HDFS but GraphLab\"\n << \"\\n\\twas built without HDFS.\"\n << std::endl;\n }\n hdfs& hdfs = hdfs::get_hdfs();\n\n // create the vector of callbacks\n\n std::vector<vertex_function_type> vertex_callbacks(graph_files.size());\n std::vector<edge_function_type> edge_callbacks(graph_files.size());\n\n for(size_t i = 0; i < graph_files.size(); ++i) {\n logstream(LOG_INFO) << \"Saving to file: \" << graph_files[i] << std::endl;\n // open the stream\n base_fstream_type* out_file = new base_fstream_type(hdfs,\n graph_files[i],\n true);\n // attach gzip if the file is gzip\n boost_fstream_type* fout = new boost_fstream_type;\n // Using gzip filter\n if (gzip) fout->push(boost::iostreams::gzip_compressor());\n fout->push(*out_file);\n\n outstreams.push_back(out_file);\n booststreams.push_back(fout);\n // construct the callback for the parallel for\n typedef distributed_graph<vertex_data_type, edge_data_type> graph_type;\n vertex_callbacks[i] =\n boost::bind(&graph_type::template save_vertex_to_stream<boost_fstream_type, Writer>,\n this, _1, boost::ref(*fout), writer);\n edge_callbacks[i] =\n boost::bind(&graph_type::template save_edge_to_stream<boost_fstream_type, Writer>,\n this, _1, boost::ref(*fout), writer);\n }\n\n if (save_vertex) parallel_for_vertices(vertex_callbacks);\n if (save_edge) parallel_for_edges(edge_callbacks);\n\n // cleanup\n for(size_t i = 0; i < graph_files.size(); ++i) {\n booststreams[i]->pop();\n if (gzip) booststreams[i]->pop();\n delete booststreams[i];\n delete outstreams[i];\n }\n vertex_callbacks.clear();\n edge_callbacks.clear();\n outstreams.clear();\n booststreams.clear();\n rpc.full_barrier();\n } // end of save to hdfs\n\n\n\n /**\n * \\brief Saves the graph to the filesystem or to HDFS using\n * a user provided Writer object. This function should be called on\n * all machines simultaneously.\n *\n * This function saves the current graph to disk using a user provided\n * Writer object. The writer object must implement two functions:\n * \\code\n * std::string Writer::save_vertex(graph_type::vertex_type v);\n * std::string Writer::save_edge(graph_type::edge_type e);\n * \\endcode\n *\n * The <code>save_vertex()</code> function will be called on each vertex\n * on the graph, and the output of the function is written to file.\n * Similarly, the <code>save_edge()</code> function is called on each edge\n * in the graph and the output written to file.\n *\n * For instance, a simple Writer object which saves a file containing\n * a list of edges will be:\n * \\code\n * struct edge_list_writer {\n * std::string save_vertex(vertex_type) { return \"\"; }\n * std::string save_edge(edge_type e) {\n * char c[128];\n * sprintf(c, \"%u\\t%u\\n\", e.source().id(), e.target().id());\n * return c;\n * }\n * };\n * \\endcode\n * The save_edge() function is called on each edge in the graph. It then\n * constructs a string containing \"[source] \\\\t [target] \\\\n\" and returns\n * the string.\n *\n * This can also be used to data in human readable format. For instance,\n * if the vertex data type is a floating point number (say a PageRank\n * value), to save a list of vertices and their corresponding PageRanks,\n * the following writer could be implemented:\n * \\code\n * struct pagerank_writer {\n * std::string save_vertex(vertex_type v) {\n * char c[128];\n * sprintf(c, \"%u\\t%f\\n\", v.id(), v.data());\n * return c;\n * }\n * std::string save_edge(edge_type) {}\n * };\n * \\endcode\n * \\note Note that these is not an example a reliable parser since sprintf\n * may break if the size of vertex_id_type changes\n *\n * The output files will be written in\n * \\li [prefix]_1_of_16.gz\n * \\li [prefix]_2_of_16.gz\n * \\li [prefix].3_of_16.gz\n * \\li etc.\n *\n * To accelerate the saving process, multiple files are be written\n * per machine in parallel. If the gzip option is not set, the \".gz\" suffix\n * is not added.\n *\n * For instance, if there are 4 machines, running:\n * \\code\n * save(\"test_graph\", pagerank_writer);\n * \\endcode\n * Will create the files\n * \\li test_graph_1_of_16.gz\n * \\li test_graph_2_of_16.gz\n * \\li ...\n * \\li test_graph_16_of_16.gz\n *\n * If HDFS support is compiled in, this function can save to HDFS by\n * adding \"hdfs://\" to the prefix.\n *\n * For instance, if there are 4 machines, running:\n * \\code\n * save(\"hdfs:///hdfs_server/data/test_graph\", pagerank_writer);\n * \\endcode\n * Will create on the HDFS server, the files\n * \\li /data/test_graph_1_of_16.gz\n * \\li /data/test_graph_2_of_16.gz\n * \\li ...\n * \\li /data/test_graph_16_of_16.gz\n *\n * \\tparam Writer The writer object type. This is generally inferred by the\n * compiler and need not be specified.\n *\n * \\param prefix The file prefix to save the output graph files. The output\n * files will be numbered [prefix].0 , [prefix].1 , etc.\n * If prefix begins with \"hdfs://\", the output is written to\n * HDFS\n * \\param writer The writer object to use.\n * \\param gzip If gzip compression should be used. If set, all files will be\n * appended with the .gz suffix. Defaults to true.\n * \\param save_vertex If vertices should be saved. Defaults to true.\n * \\param save_edges If edges should be saved. Defaults to true.\n * \\param files_per_machine Number of files to write simultaneously in\n * parallel per machine. Defaults to 4.\n */\n template<typename Writer>\n void save(const std::string& prefix, Writer writer,\n bool gzip = true, bool save_vertex = true, bool save_edge = true,\n size_t files_per_machine = 4) {\n if(boost::starts_with(prefix, \"hdfs://\")) {\n save_to_hdfs(prefix, writer, gzip, save_vertex, save_edge, files_per_machine);\n } else {\n save_to_posixfs(prefix, writer, gzip, save_vertex, save_edge, files_per_machine);\n }\n } // end of save\n\n\n\n /**\n * \\brief Saves the graph in the specified format. This function should be\n * called on all machines simultaneously.\n *\n * The output files will be written in\n * \\li [prefix].0.gz\n * \\li [prefix].1.gz\n * \\li [prefix].2.gz\n * \\li etc.\n *\n * To accelerate the saving process, multiple files are be written\n * per machine in parallel. If the gzip option is not set, the \".gz\" suffix\n * is not added.\n *\n * For instance, if there are 4 machines, running:\n * \\code\n * save_format(\"test_graph\", \"tsv\");\n * \\endcode\n * Will create the files\n * \\li test_graph_0.gz\n * \\li test_graph_1.gz\n * \\li ...\n * \\li test_graph_15.gz\n *\n * The supported formats are described in \\ref graph_formats.\n *\n * \\param prefix The file prefix to save the output graph files. The output\n * files will be numbered [prefix].0 , [prefix].1 , etc.\n * If prefix begins with \"hdfs://\", the output is written to\n * HDFS.\n * \\param format The file format to save in.\n * Either \"tsv\", \"snap\", \"graphjrl\" or \"bin\".\n * \\param gzip If gzip compression should be used. If set, all files will be\n * appended with the .gz suffix. Defaults to true. Ignored\n * if format == \"bin\".\n * \\param files_per_machine Number of files to write simultaneously in\n * parallel per machine. Defaults to 4. Ignored if\n * format == \"bin\".\n */\n void save_format(const std::string& prefix, const std::string& format,\n bool gzip = true, size_t files_per_machine = 4) {\n if (format == \"snap\" || format == \"tsv\") {\n save(prefix, builtin_parsers::tsv_writer<distributed_graph>(),\n gzip, false, true, files_per_machine);\n } else if (format == \"graphjrl\") {\n save(prefix, builtin_parsers::graphjrl_writer<distributed_graph>(),\n gzip, true, true, files_per_machine);\n } else if (format == \"bin\") {\n save_binary(prefix);\n } else if (format == \"bintsv4\") {\n save_direct(prefix, gzip, &graph_type::save_bintsv4_to_stream);\n } else {\n logstream(LOG_FATAL)\n << \"Unrecognized Format \\\"\" << format << \"\\\"!\" << std::endl;\n return;\n }\n } // end of save structure\n\n\n\n\n /**\n * \\brief Load a graph from a collection of files in stored on\n * the filesystem using the user defined line parser. Like\n * \\ref load(const std::string& path, line_parser_type line_parser)\n * but only loads from the filesystem.\n */\n void load_from_posixfs(std::string prefix,\n line_parser_type line_parser) {\n std::string directory_name; std::string original_path(prefix);\n boost::filesystem::path path(prefix);\n std::string search_prefix;\n if (boost::filesystem::is_directory(path)) {\n // if this is a directory\n // force a \"/\" at the end of the path\n // make sure to check that the path is non-empty. (you do not\n // want to make the empty path \"\" the root path \"/\" )\n directory_name = path.native();\n }\n else {\n directory_name = path.parent_path().native();\n search_prefix = path.filename().native();\n directory_name = (directory_name.empty() ? \".\" : directory_name);\n }\n std::vector<std::string> graph_files;\n fs_util::list_files_with_prefix(directory_name, search_prefix, graph_files);\n if (graph_files.size() == 0) {\n logstream(LOG_WARNING) << \"No files found matching \" << original_path << std::endl;\n }\n\n#ifdef _OPENMP\n#pragma omp parallel for\n#endif\n for(size_t i = 0; i < graph_files.size(); ++i) {\n if ((parallel_ingress && (i % rpc.numprocs() == rpc.procid()))\n || (!parallel_ingress && (rpc.procid() == 0))) {\n logstream(LOG_EMPH) << \"Loading graph from file: \" << graph_files[i] << std::endl;\n // is it a gzip file ?\n const bool gzip = boost::ends_with(graph_files[i], \".gz\");\n // open the stream\n std::ifstream in_file(graph_files[i].c_str(),\n std::ios_base::in | std::ios_base::binary);\n // attach gzip if the file is gzip\n boost::iostreams::filtering_stream<boost::iostreams::input> fin;\n // Using gzip filter\n if (gzip) fin.push(boost::iostreams::gzip_decompressor());\n fin.push(in_file);\n const bool success = load_from_stream(graph_files[i], fin, line_parser);\n if(!success) {\n logstream(LOG_FATAL)\n << \"\\n\\tError parsing file: \" << graph_files[i] << std::endl;\n }\n fin.pop();\n if (gzip) fin.pop();\n }\n }\n rpc.full_barrier();\n } // end of load from posixfs\n\n /**\n * \\brief Load a graph from a collection of files in stored on\n * the HDFS using the user defined line parser. Like\n * \\ref load(const std::string& path, line_parser_type line_parser)\n * but only loads from HDFS.\n */\n void load_from_hdfs(std::string prefix, line_parser_type line_parser) {\n // force a \"/\" at the end of the path\n // make sure to check that the path is non-empty. (you do not\n // want to make the empty path \"\" the root path \"/\" )\n std::string path = prefix;\n if (path.length() > 0 && path[path.length() - 1] != '/') path = path + \"/\";\n if(!hdfs::has_hadoop()) {\n logstream(LOG_FATAL)\n << \"\\n\\tAttempting to load a graph from HDFS but GraphLab\"\n << \"\\n\\twas built without HDFS.\"\n << std::endl;\n }\n hdfs& hdfs = hdfs::get_hdfs();\n std::vector<std::string> graph_files;\n graph_files = hdfs.list_files(path);\n if (graph_files.size() == 0) {\n logstream(LOG_WARNING) << \"No files found matching \" << prefix << std::endl;\n }\n#ifdef _OPENMP\n#pragma omp parallel for\n#endif\n for(size_t i = 0; i < graph_files.size(); ++i) {\n if ((parallel_ingress && (i % rpc.numprocs() == rpc.procid())) ||\n (!parallel_ingress && (rpc.procid() == 0))) {\n logstream(LOG_EMPH) << \"Loading graph from file: \" << graph_files[i] << std::endl;\n // is it a gzip file ?\n const bool gzip = boost::ends_with(graph_files[i], \".gz\");\n // open the stream\n graphlab::hdfs::fstream in_file(hdfs, graph_files[i]);\n boost::iostreams::filtering_stream<boost::iostreams::input> fin;\n if(gzip) fin.push(boost::iostreams::gzip_decompressor());\n fin.push(in_file);\n const bool success = load_from_stream(graph_files[i], fin, line_parser);\n if(!success) {\n logstream(LOG_FATAL)\n << \"\\n\\tError parsing file: \" << graph_files[i] << std::endl;\n }\n fin.pop();\n if (gzip) fin.pop();\n }\n }\n rpc.full_barrier();\n } // end of load from hdfs\n\n\n /**\n * \\brief Load a the graph from a given path using a user defined\n * line parser. This function should be called on all machines\n * simultaneously.\n *\n * This functions loads all files in the filesystem or on HDFS matching\n * the pattern \"[prefix]*\".\n *\n * Examples:\n *\n * <b> prefix = \"webgraph.txt\" </b>\n *\n * will load the file webgraph.txt if such a file exists. It will also\n * load all files in the current directory which begins with \"webgraph.txt\".\n * For instance, webgraph.txt.0, webgraph.txt.1, etc.\n *\n * <b>prefix = \"graph/data\"</b>\n *\n * will load all files in the \"graph\" directory which begin with \"data\"\n *\n * <b> prefix = \"hdfs:///hdfs_server/graph/data\" </b>\n *\n * will load all files from the HDFS server in the \"/graph/\" directory\n * which begin with \"data\".\n *\n * If files have the \".gz\" suffix, it is automatically decompressed.\n *\n * The line_parser is a user defined function matching the following\n * prototype:\n *\n * \\code\n * bool parser(graph_type& graph,\n * const std::string& filename,\n * const std::string& line);\n * \\endcode\n *\n * The load() function will call the parser one line at a time, and the\n * paser function should process the line and call add_vertex / add_edge\n * functions in the graph. It should return true on success, and false\n * on failure. Since the parsing may be parallelized,\n * the parser should treat each line independently\n * and not depend on a sequential pass through a file.\n *\n * For instance, if the graph is in a simple edge list format, a parser\n * could be:\n * \\code\n * bool edge_list_parser(graph_type& graph,\n * const std::string& filename,\n * const std::string& line) {\n * if (line.empty()) return true;\n * vertex_id_type source, target;\n * if (sscanf(line.c_str(), \"%u %u\", source, target) < 2) {\n * // parsed less than 2 objects, failure.\n * return false;\n * }\n * else {\n * graph.add_edge(source, target);\n * return true;\n * }\n * }\n * \\endcode\n * \\note Note that this is not an example a reliable parser since sscanf\n * may break if the size of vertex_id_type changes\n *\n * \\param prefix The file prefix to read from. All files matching\n * the pattern \"[prefix]*\" are loaded. If prefix begins with\n * \"hdfs://\" the files are read from hdfs.\n * \\param line_parser A user defined parsing function\n */\n void load(std::string prefix, line_parser_type line_parser) {\n rpc.full_barrier();\n if (prefix.length() == 0) return;\n if(boost::starts_with(prefix, \"hdfs://\")) {\n load_from_hdfs(prefix, line_parser);\n } else {\n load_from_posixfs(prefix, line_parser);\n }\n rpc.full_barrier();\n } // end of load\n\n /**\n * \\brief Constructs a synthetic power law graph. Must be called on\n * all machines simultaneously.\n *\n * This function constructs a synthetic out-degree power law of \"nverts\"\n * vertices with a particular alpha parameter.\n * In other words, the probability that a vertex has out-degree \\f$d\\f$,\n * is given by:\n *\n * \\f[ P(d) \\propto d^{-\\alpha} \\f]\n *\n * By default, the out-degree distribution of each vertex\n * will have power-law distribution, but the in-degrees will be nearly\n * uniform. This can be reversed by setting the second argument \"in_degree\"\n * to true.\n *\n * \\param nverts Number of vertices to generate\n * \\param in_degree If set to true, the graph will have power-law in-degree.\n * Defaults to false.\n * \\param alpha The alpha parameter in the power law distribution. Defaults\n * to 2.1\n * \\param truncate Limits the maximum degree of any vertex. (thus generating\n * a truncated power-law distribution). Necessary\n * for large number of vertices (hundreds of millions)\n * since this function allocates a PDF vector of\n * \"nverts\" to sample from.\n */\n void load_synthetic_powerlaw(size_t nverts, bool in_degree = false,\n double alpha = 2.1, size_t truncate = (size_t)(-1)) {\n rpc.full_barrier();\n std::vector<double> prob(std::min(nverts, truncate), 0);\n logstream(LOG_INFO) << \"constructing pdf\" << std::endl;\n for(size_t i = 0; i < prob.size(); ++i)\n prob[i] = std::pow(double(i+1), -alpha);\n logstream(LOG_INFO) << \"constructing cdf\" << std::endl;\n random::pdf2cdf(prob);\n logstream(LOG_INFO) << \"Building graph\" << std::endl;\n size_t target_index = rpc.procid();\n size_t addedvtx = 0;\n\n // A large prime number\n const size_t HASH_OFFSET = 2654435761;\n for(size_t source = rpc.procid(); source < nverts;\n source += rpc.numprocs()) {\n const size_t out_degree = random::multinomial_cdf(prob) + 1;\n for(size_t i = 0; i < out_degree; ++i) {\n target_index = (target_index + HASH_OFFSET) % nverts;\n while (source == target_index) {\n target_index = (target_index + HASH_OFFSET) % nverts;\n }\n if(in_degree) add_edge(target_index, source);\n else add_edge(source, target_index);\n }\n ++addedvtx;\n if (addedvtx % 10000000 == 0) {\n logstream(LOG_EMPH) << addedvtx << \" inserted\\n\";\n }\n }\n rpc.full_barrier();\n } // end of load random powerlaw\n\n\n /**\n * \\brief load a graph with a standard format. Must be called on all\n * machines simultaneously.\n *\n * The supported graph formats are described in \\ref graph_formats.\n */\n void load_format(const std::string& path, const std::string& format) {\n line_parser_type line_parser;\n if (format == \"snap\") {\n line_parser = builtin_parsers::snap_parser<distributed_graph>;\n load(path, line_parser);\n } else if (format == \"adj\") {\n line_parser = builtin_parsers::adj_parser<distributed_graph>;\n load(path, line_parser);\n } else if (format == \"tsv\") {\n line_parser = builtin_parsers::tsv_parser<distributed_graph>;\n load(path, line_parser);\n } else if (format == \"csv\") {\n line_parser = builtin_parsers::csv_parser<distributed_graph>;\n load(path, line_parser);\n } else if (format == \"graphjrl\") {\n line_parser = builtin_parsers::graphjrl_parser<distributed_graph>;\n load(path, line_parser);\n } else if (format == \"bintsv4\") {\n load_direct(path,&graph_type::load_bintsv4_from_stream);\n } else if (format == \"bin\") {\n load_binary(path);\n } else {\n logstream(LOG_ERROR)\n << \"Unrecognized Format \\\"\" << format << \"\\\"!\" << std::endl;\n return;\n }\n } // end of load\n\n\n/****************************************************************************\n * Vertex Set Functions *\n * ---------------------- *\n * Manages operations involving sets of vertices *\n ****************************************************************************/\n\n /**\n * \\brief Retuns an empty set of vertices\n */\n static vertex_set empty_set() {\n return vertex_set(false);\n }\n\n /**\n * \\brief Retuns a full set of vertices\n */\n static vertex_set complete_set() {\n return vertex_set(true);\n }\n\n ///\n vertex_set neighbors(const vertex_set& cur,\n edge_dir_type edir) {\n // foreach master bit which is set, set its corresponding mirror\n // synchronize master to mirrors\n vertex_set ret(empty_set());\n ret.make_explicit(*this);\n\n foreach(size_t lvid, cur.get_lvid_bitset(*this)) {\n if (edir == IN_EDGES || edir == ALL_EDGES) {\n foreach(local_edge_type e, l_vertex(lvid).in_edges()) {\n ret.set_lvid_unsync(e.source().id());\n }\n }\n if (edir == OUT_EDGES || edir == ALL_EDGES) {\n foreach(local_edge_type e, l_vertex(lvid).out_edges()) {\n ret.set_lvid_unsync(e.target().id());\n }\n }\n }\n ret.synchronize_mirrors_to_master_or(*this, vset_exchange);\n ret.synchronize_master_to_mirrors(*this, vset_exchange);\n return ret;\n }\n\n\n /**\n * \\brief Constructs a vertex set from a predicate operation which\n * is executed on each vertex.\n *\n * This function selects a subset of vertices on which the predicate\n * evaluates to true.\n For instance if vertices contain an integer, the following\n * code will construct a set of vertices containing only vertices with data\n * which are a multiple of 2.\n *\n * \\code\n * bool is_multiple_of_2(const graph_type::vertex_type& vertex) {\n * return vertex.data() % 2 == 0;\n * }\n * vertex_set even_vertices = graph.select(is_multiple_of_2);\n * \\endcode\n *\n * select() also takes a second argument which restricts the set of vertices\n * queried. For instance,\n * \\code\n * bool is_multiple_of_3(const graph_type::vertex_type& vertex) {\n * return vertex.data() % 3 == 0;\n * }\n * vertex_set div_6_vertices = graph.select(is_multiple_of_3, even_vertices);\n * \\endcode\n * will select from the set of even vertices, all vertices which are also\n * divisible by 3. The resultant set is therefore the set of all vertices\n * which are divisible by 6.\n *\n * \\param select_functor A function/functor which takes a\n * const vertex_type& argument and returns a boolean\n * denoting of the vertex is to be included in the\n * returned set\n * \\param vset Optional. The set of vertices to evaluate the selection on.\n * Defaults to complete_set()\n */\n template <typename FunctionType>\n vertex_set select(FunctionType select_functor,\n const vertex_set& vset = complete_set()) {\n vertex_set ret(empty_set());\n\n ret.make_explicit(*this);\n#ifdef _OPENMP\n #pragma omp for\n#endif\n for (int i = 0; i < (int)local_graph.num_vertices(); ++i) {\n if (lvid2record[i].owner == rpc.procid() &&\n vset.l_contains((lvid_type)i)) {\n const vertex_type vtx(l_vertex(i));\n if (select_functor(vtx)) ret.set_lvid(i);\n }\n }\n ret.synchronize_master_to_mirrors(*this, vset_exchange);\n return ret;\n }\n\n void sync_vertex_set_master_to_mirrors(vertex_set& vset) {\n vset.synchronize_master_to_mirrors(*this, vset_exchange);\n }\n\n /**\n * \\brief Returns the number of vertices in a vertex set.\n *\n * This function must be called on all machines and returns the number of\n * vertices contained in the vertex set.\n *\n * For instance:\n * \\code\n * graph.vertex_set_size(graph.complete_set());\n * \\endcode\n * will always evaluate to graph.num_vertices();\n */\n size_t vertex_set_size(const vertex_set& vset) {\n size_t count = 0;\n for (int i = 0; i < (int)local_graph.num_vertices(); ++i) {\n count += (lvid2record[i].owner == rpc.procid() &&\n vset.l_contains((lvid_type)i));\n }\n rpc.all_reduce(count);\n return count;\n }\n\n\n /**\n * \\brief Returns true if the vertex set is empty\n *\n * This function must be called on all machines and returns\n * true if the vertex set is empty\n */\n bool vertex_set_empty(const vertex_set& vset) {\n if (vset.lazy) return !vset.is_complete_set;\n\n size_t count = vset.get_lvid_bitset(*this).empty();\n rpc.all_reduce(count);\n return count == rpc.numprocs();\n }\n/****************************************************************************\n * Internal Functions *\n * ---------------------- *\n * These functions functions and types provide internal access to the *\n * underlying graph representation. They should not be used unless you *\n * *really* know what you are doing. *\n ****************************************************************************/\n\n\n /**\n * \\internal\n * The vertex record stores information associated with each\n * vertex on this proc\n */\n struct vertex_record {\n /// The official owning processor for this vertex\n procid_t owner;\n /// The local vid of this vertex on this proc\n vertex_id_type gvid;\n /// The number of in edges\n vertex_id_type num_in_edges, num_out_edges;\n /** The set of proc that mirror this vertex. The owner should\n NOT be in this set.*/\n mirror_type _mirrors;\n vertex_record() :\n owner(-1), gvid(-1), num_in_edges(0), num_out_edges(0) { }\n vertex_record(const vertex_id_type& vid) :\n owner(-1), gvid(vid), num_in_edges(0), num_out_edges(0) { }\n procid_t get_owner () const { return owner; }\n const mirror_type& mirrors() const { return _mirrors; }\n size_t num_mirrors() const { return _mirrors.popcount(); }\n\n void clear() {\n _mirrors.clear();\n }\n\n void load(iarchive& arc) {\n clear();\n arc >> owner\n >> gvid\n >> num_in_edges\n >> num_out_edges\n >> _mirrors;\n }\n\n void save(oarchive& arc) const {\n arc << owner\n << gvid\n << num_in_edges\n << num_out_edges\n << _mirrors;\n } // end of save\n\n bool operator==(const vertex_record& other) const {\n return (\n (owner == other.owner) &&\n (gvid == other.gvid) &&\n (num_in_edges == other.num_in_edges) &&\n (num_out_edges == other.num_out_edges) && \n (_mirrors == other._mirrors)\n );\n }\n }; // end of vertex_record\n\n\n\n\n /** \\internal\n * \\brief converts a local vertex ID to a local vertex object\n */\n local_vertex_type l_vertex(lvid_type vid) {\n return local_vertex_type(*this, vid);\n }\n\n /** \\internal\n *\\brief Get the Total number of vertex replicas in the graph */\n size_t num_replicas() const { return nreplicas; }\n\n /** \\internal\n *\\brief Get the number of vertices local to this proc */\n size_t num_local_vertices() const { return local_graph.num_vertices(); }\n\n /** \\internal\n *\\brief Get the number of edges local to this proc */\n size_t num_local_edges() const { return local_graph.num_edges(); }\n\n /** \\internal\n *\\brief Get the number of vertices owned by this proc */\n size_t num_local_own_vertices() const { return local_own_nverts; }\n\n /** \\internal\n *\\brief Convert a global vid to a local vid */\n lvid_type local_vid (const vertex_id_type vid) const {\n // typename boost::unordered_map<vertex_id_type, lvid_type>::\n // const_iterator iter = vid2lvid.find(vid);\n typename hopscotch_map_type::const_iterator iter = vid2lvid.find(vid);\n return iter->second;\n } // end of local_vertex_id\n\n /** \\internal\n *\\brief Convert a local vid to a global vid */\n vertex_id_type global_vid(const lvid_type lvid) const {\n ASSERT_LT(lvid, lvid2record.size());\n return lvid2record[lvid].gvid;\n } // end of global_vertex_id\n\n\n /** \\internal\n * \\brief Returns true if the local graph as an instance of (master or mirror)\n * of the vertex ID.\n */\n bool contains_vertex(const vertex_id_type vid) const {\n return vid2lvid.find(vid) != vid2lvid.end();\n }\n /**\n * \\internal\n * \\brief Returns an edge list of all in edges of a local vertex ID\n * on the local graph\n *\n * Equivalent to l_vertex(lvid).in_edges()\n */\n local_edge_list_type l_in_edges(const lvid_type lvid) {\n return local_edge_list_type(*this, local_graph.in_edges(lvid));\n }\n\n /**\n * \\internal\n * \\brief Returns the number of in edges of a local vertex ID\n * on the local graph\n *\n * Equivalent to l_vertex(lvid).num in_edges()\n */\n size_t l_num_in_edges(const lvid_type lvid) const {\n return local_graph.num_in_edges(lvid);\n }\n\n /**\n * \\internal\n * \\brief Returns an edge list of all out edges of a local vertex ID\n * on the local graph\n *\n * Equivalent to l_vertex(lvid).out_edges()\n */\n local_edge_list_type l_out_edges(const lvid_type lvid) {\n return local_edge_list_type(*this, local_graph.out_edges(lvid));\n }\n\n /**\n * \\internal\n * \\brief Returns the number of out edges of a local vertex ID\n * on the local graph\n *\n * Equivalent to l_vertex(lvid).num out_edges()\n */\n size_t l_num_out_edges(const lvid_type lvid) const {\n return local_graph.num_out_edges(lvid);\n }\n\n procid_t procid() const {\n return rpc.procid();\n }\n\n\n procid_t numprocs() const {\n return rpc.numprocs();\n }\n\n distributed_control& dc() {\n return rpc.dc();\n }\n\n\n\n /** \\internal\n * \\brief Returns the internal vertex record of a given global vertex ID\n */\n const vertex_record& get_vertex_record(vertex_id_type vid) const {\n // typename boost::unordered_map<vertex_id_type, lvid_type>::\n // const_iterator iter = vid2lvid.find(vid);\n typename hopscotch_map_type::const_iterator iter = vid2lvid.find(vid);\n ASSERT_TRUE(iter != vid2lvid.end());\n return lvid2record[iter->second];\n }\n\n /** \\internal\n * \\brief Returns the internal vertex record of a given local vertex ID\n */\n vertex_record& l_get_vertex_record(lvid_type lvid) {\n ASSERT_LT(lvid, lvid2record.size());\n return lvid2record[lvid];\n }\n\n /** \\internal\n * \\brief Returns the internal vertex record of a given local vertex ID\n */\n const vertex_record& l_get_vertex_record(lvid_type lvid) const {\n ASSERT_LT(lvid, lvid2record.size());\n return lvid2record[lvid];\n }\n\n /** \\internal\n * \\brief Returns true if the provided global vertex ID is a\n * master vertex on this machine and false otherwise.\n */\n bool is_master(vertex_id_type vid) const {\n const procid_t owning_proc = graph_hash::hash_vertex(vid) % rpc.numprocs();\n return (owning_proc == rpc.procid());\n }\n\n\n procid_t master(vertex_id_type vid) const {\n const procid_t owning_proc = graph_hash::hash_vertex(vid) % rpc.numprocs();\n return owning_proc;\n }\n\n /** \\internal\n * \\brief Returns true if the provided local vertex ID is a master vertex.\n * Returns false otherwise.\n */\n bool l_is_master(lvid_type lvid) const {\n ASSERT_LT(lvid, lvid2record.size());\n return lvid2record[lvid].owner == rpc.procid();\n }\n\n /** \\internal\n * \\brief Returns the master procid for vertex lvid.\n */\n procid_t l_master(lvid_type lvid) const {\n ASSERT_LT(lvid, lvid2record.size());\n return lvid2record[lvid].owner;\n }\n\n\n /** \\internal\n * \\brief Returns a reference to the internal graph representation\n */\n local_graph_type& get_local_graph() {\n return local_graph;\n }\n\n /** \\internal\n * \\brief Returns a const reference to the internal graph representation\n */\n const local_graph_type& get_local_graph() const {\n return local_graph;\n }\n\n\n\n\n /** \\internal\n * This function synchronizes the master vertex data with all the mirrors.\n * This function must be called simultaneously by all machines\n */\n void synchronize(const vertex_set& vset = complete_set()) {\n typedef std::pair<vertex_id_type, vertex_data_type> pair_type;\n\n procid_t sending_proc;\n // Loop over all the local vertex records\n\n\n#ifdef _OPENMP\n#pragma omp parallel for\n#endif\n for(lvid_type lvid = 0; lvid < lvid2record.size(); ++lvid) {\n typename buffered_exchange<pair_type>::buffer_type recv_buffer;\n const vertex_record& record = lvid2record[lvid];\n // if this machine is the owner of a record then send the\n // vertex data to all mirrors\n if(record.owner == rpc.procid() && vset.l_contains(lvid)) {\n foreach(size_t proc, record.mirrors()) {\n const pair_type pair(record.gvid, local_graph.vertex_data(lvid));\n#ifdef _OPENMP\n vertex_exchange.send(proc, pair, omp_get_thread_num());\n#else\n vertex_exchange.send(proc, pair);\n#endif\n }\n }\n // Receive any vertex data and update local mirrors\n while(vertex_exchange.recv(sending_proc, recv_buffer, true)) {\n foreach(const pair_type& pair, recv_buffer) {\n vertex(pair.first).data() = pair.second;\n }\n recv_buffer.clear();\n }\n }\n\n\n typename buffered_exchange<pair_type>::buffer_type recv_buffer;\n vertex_exchange.flush();\n while(vertex_exchange.recv(sending_proc, recv_buffer)) {\n foreach(const pair_type& pair, recv_buffer) {\n vertex(pair.first).data() = pair.second;\n }\n recv_buffer.clear();\n }\n ASSERT_TRUE(vertex_exchange.empty());\n } // end of synchronize\n\n\n\n\n\n\n /** \\internal\n * vertex type while provides access to local graph vertices.\n */\n struct local_vertex_type {\n distributed_graph& graph_ref;\n lvid_type lvid;\n\n local_vertex_type(distributed_graph& graph_ref, lvid_type lvid):\n graph_ref(graph_ref), lvid(lvid) { }\n\n /// \\brief Can be casted from local_vertex_type using an explicit cast\n explicit local_vertex_type(vertex_type v) :graph_ref(v.graph_ref),lvid(v.lvid) { }\n /// \\brief Can be casted to vertex_type using an explicit cast\n operator vertex_type() const {\n return vertex_type(graph_ref, lvid);\n }\n\n bool operator==(local_vertex_type& v) const {\n return lvid == v.lvid;\n }\n\n /// \\brief Returns a reference to the data on the local vertex\n const vertex_data_type& data() const {\n return graph_ref.get_local_graph().vertex_data(lvid);\n }\n\n /// \\brief Returns a reference to the data on the local vertex\n vertex_data_type& data() {\n return graph_ref.get_local_graph().vertex_data(lvid);\n }\n\n /** \\brief Returns the number of in edges on the\n * local graph of this local vertex\n */\n size_t num_in_edges() const {\n return graph_ref.get_local_graph().num_in_edges(lvid);\n }\n\n /** \\brief Returns the number of in edges on the\n * local graph of this local vertex\n */\n size_t num_out_edges() const {\n return graph_ref.get_local_graph().num_out_edges(lvid);\n }\n\n /// \\brief Returns the local ID of this local vertex\n lvid_type id() const {\n return lvid;\n }\n\n /// \\brief Returns the global ID of this local vertex\n vertex_id_type global_id() const {\n return graph_ref.global_vid(lvid);\n }\n\n /** \\brief Returns a list of all in edges on the\n * local graph of this local vertex\n */\n local_edge_list_type in_edges() {\n return graph_ref.l_in_edges(lvid);\n }\n\n /** \\brief Returns a list of all out edges on the\n * local graph of this local vertex\n */\n local_edge_list_type out_edges() {\n return graph_ref.l_out_edges(lvid);\n }\n\n /** \\brief Returns the owner of this local vertex\n */\n procid_t owner() const {\n return graph_ref.l_get_vertex_record(lvid).owner;\n }\n\n /** \\brief Returns the owner of this local vertex\n */\n bool owned() const {\n return graph_ref.l_get_vertex_record(lvid).owner == graph_ref.procid();\n }\n\n /** \\brief Returns the number of in_edges of this vertex\n * on the global graph\n */\n size_t global_num_in_edges() const {\n return graph_ref.l_get_vertex_record(lvid).num_in_edges;\n }\n\n\n /** \\brief Returns the number of out_edges of this vertex\n * on the global graph\n */\n size_t global_num_out_edges() const {\n return graph_ref.l_get_vertex_record(lvid).num_out_edges;\n }\n\n\n /** \\brief Returns the set of mirrors of this vertex\n */\n const mirror_type& mirrors() const {\n return graph_ref.l_get_vertex_record(lvid)._mirrors;\n }\n\n size_t num_mirrors() const {\n return graph_ref.l_get_vertex_record(lvid).num_mirrors();\n }\n\n /** \\brief Returns the vertex record of this\n * this local vertex\n */\n vertex_record& get_vertex_record() {\n return graph_ref.l_get_vertex_record(lvid);\n }\n };\n\n\n /** \\internal\n * edge type which provides access to local graph edges */\n class local_edge_type {\n private:\n distributed_graph& graph_ref;\n typename local_graph_type::edge_type e;\n public:\n local_edge_type(distributed_graph& graph_ref,\n typename local_graph_type::edge_type e):\n graph_ref(graph_ref), e(e) { }\n\n /// \\brief Can be converted from edge_type via an explicit cast\n explicit local_edge_type(edge_type ge) :graph_ref(ge.graph_ref),e(ge.e) { }\n\n /// \\brief Can be casted to edge_type using an explicit cast\n operator edge_type() const {\n return edge_type(graph_ref, e);\n }\n\n /// \\brief Returns the source local vertex of the edge\n local_vertex_type source() const { return local_vertex_type(graph_ref, e.source().id()); }\n\n /// \\brief Returns the target local vertex of the edge\n local_vertex_type target() const { return local_vertex_type(graph_ref, e.target().id()); }\n\n\n\n /// \\brief Returns a constant reference to the data on the vertex\n const edge_data_type& data() const { return e.data(); }\n\n /// \\brief Returns a reference to the data on the vertex\n edge_data_type& data() { return e.data(); }\n\n /// \\brief Returns the internal ID of this edge\n edge_id_type id() const { return e.id(); }\n };\n\n /** \\internal\n * \\brief A functor which converts local_graph_type::edge_type to\n * local_edge_type\n */\n struct make_local_edge_type_functor {\n typedef typename local_graph_type::edge_type argument_type;\n typedef local_edge_type result_type;\n distributed_graph& graph_ref;\n make_local_edge_type_functor(distributed_graph& graph_ref):\n graph_ref(graph_ref) { }\n result_type operator() (const argument_type et) const {\n return local_edge_type(graph_ref, et);\n }\n };\n\n\n /** \\internal\n * \\brief A list of edges. Used by l_in_edges() and l_out_edges()\n */\n struct local_edge_list_type {\n make_local_edge_type_functor me_functor;\n typename local_graph_type::edge_list_type elist;\n\n typedef boost::transform_iterator<make_local_edge_type_functor,\n typename local_graph_type::edge_list_type::iterator> iterator;\n typedef iterator const_iterator;\n\n local_edge_list_type(distributed_graph& graph_ref,\n typename local_graph_type::edge_list_type elist) :\n me_functor(graph_ref), elist(elist) { }\n /// \\brief Returns the number of edges in the list\n size_t size() const { return elist.size(); }\n\n /// \\brief Random access to the list elements\n local_edge_type operator[](size_t i) const { return me_functor(elist[i]); }\n\n /** \\brief Returns an iterator to the beginning of the list.\n *\n * Returns an iterator to the beginning of the list. \\see end()\n * The iterator_type is local_edge_list_type::iterator.\n *\n * \\code\n * local_edge_list_type::iterator iter = elist.begin();\n * while(iter != elist.end()) {\n * ... [do stuff] ...\n * ++iter;\n * }\n * \\endcode\n *\n */\n iterator begin() const { return\n boost::make_transform_iterator(elist.begin(), me_functor); }\n\n /** \\brief Returns an iterator to the end of the list.\n *\n * Returns an iterator to the end of the list. \\see begin()\n * The iterator_type is local_edge_list_type::iterator.\n *\n * \\code\n * local_edge_list_type::iterator iter = elist.begin();\n * while(iter != elist.end()) {\n * ... [do stuff] ...\n * ++iter;\n * }\n * \\endcode\n *\n */\n iterator end() const { return\n boost::make_transform_iterator(elist.end(), me_functor); }\n\n /// \\brief Returns true if the list is empty\n bool empty() const { return elist.empty(); }\n };\n\n\n private:\n\n // PRIVATE DATA MEMBERS ===================================================>\n /** The rpc interface for this class */\n mutable dc_dist_object<distributed_graph> rpc;\n\n public:\n\n // For the warp engine to find the remote instances of this class\n size_t get_rpc_obj_id() {\n return rpc.get_obj_id();\n }\n\n private:\n bool finalized;\n\n /** The local graph data */\n local_graph_type local_graph;\n\n /** The map from global vertex ids to vertex records */\n std::vector<vertex_record> lvid2record;\n\n // boost::unordered_map<vertex_id_type, lvid_type> vid2lvid;\n /** The map from global vertex ids back to local vertex ids */\n typedef hopscotch_map<vertex_id_type, lvid_type> hopscotch_map_type;\n typedef hopscotch_map_type vid2lvid_map_type;\n\n hopscotch_map_type vid2lvid;\n\n\n /** The global number of vertices and edges */\n size_t nverts, nedges;\n\n /** The number of vertices owned by this proc */\n size_t local_own_nverts;\n\n /** The global number of vertex replica */\n size_t nreplicas;\n\n /** pointer to the distributed ingress object*/\n distributed_ingress_base<VertexData, EdgeData>* ingress_ptr;\n\n /** Buffered Exchange used by synchronize() */\n buffered_exchange<std::pair<vertex_id_type, vertex_data_type> > vertex_exchange;\n\n /** Buffered Exchange used by vertex sets */\n buffered_exchange<vertex_id_type> vset_exchange;\n\n /** Command option to disable parallel ingress. Used for simulating single node ingress */\n bool parallel_ingress;\n\n\n lock_manager_type lock_manager;\n\n void set_ingress_method(const std::string& method,\n size_t bufsize = 50000, bool usehash = false, bool userecent = false) {\n if(ingress_ptr != NULL) { delete ingress_ptr; ingress_ptr = NULL; }\n if (method == \"oblivious\") {\n if (rpc.procid() == 0) logstream(LOG_EMPH) << \"Use oblivious ingress, usehash: \" << usehash\n << \", userecent: \" << userecent << std::endl;\n ingress_ptr = new distributed_oblivious_ingress<VertexData, EdgeData>(rpc.dc(), *this, usehash, userecent);\n } else if (method == \"hdrf\") {\n if (rpc.procid() == 0) logstream(LOG_EMPH) << \"Use hdrf oblivious ingress, usehash: \" << usehash\n << \", userecent: \" << userecent << std::endl;\n ingress_ptr = new distributed_hdrf_ingress<VertexData, EdgeData>(rpc.dc(), *this, usehash, userecent);\n } else if (method == \"random\") {\n if (rpc.procid() == 0)logstream(LOG_EMPH) << \"Use random ingress\" << std::endl;\n ingress_ptr = new distributed_random_ingress<VertexData, EdgeData>(rpc.dc(), *this); \n } else if (method == \"grid\") {\n if (rpc.procid() == 0)logstream(LOG_EMPH) << \"Use grid ingress\" << std::endl;\n ingress_ptr = new distributed_constrained_random_ingress<VertexData, EdgeData>(rpc.dc(), *this, \"grid\");\n } else if (method == \"pds\") {\n if (rpc.procid() == 0)logstream(LOG_EMPH) << \"Use pds ingress\" << std::endl;\n ingress_ptr = new distributed_constrained_random_ingress<VertexData, EdgeData>(rpc.dc(), *this, \"pds\");\n } else {\n // use default ingress method if none is specified\n std::string ingress_auto=\"\";\n size_t num_shards = rpc.numprocs();\n int nrow, ncol, p;\n if (sharding_constraint::is_pds_compatible(num_shards, p)) {\n ingress_auto=\"pds\";\n ingress_ptr = new distributed_constrained_random_ingress<VertexData, EdgeData>(rpc.dc(), *this, \"pds\");\n } else if (sharding_constraint::is_grid_compatible(num_shards, nrow, ncol)) {\n ingress_auto=\"grid\";\n ingress_ptr = new distributed_constrained_random_ingress<VertexData, EdgeData>(rpc.dc(), *this, \"grid\");\n } else {\n ingress_auto=\"oblivious\";\n ingress_ptr = new distributed_oblivious_ingress<VertexData, EdgeData>(rpc.dc(), *this, usehash, userecent);\n }\n if (rpc.procid() == 0)logstream(LOG_EMPH) << \"Automatically determine ingress method: \" << ingress_auto << std::endl;\n }\n // batch ingress is deprecated\n // if (method == \"batch\") {\n // logstream(LOG_EMPH) << \"Use batch ingress, bufsize: \" << bufsize\n // << \", usehash: \" << usehash << \", userecent\" << userecent << std::endl;\n // ingress_ptr = new distributed_batch_ingress<VertexData, EdgeData>(rpc.dc(), *this,\n // bufsize, usehash, userecent);\n // } else \n } // end of set ingress method\n\n\n /**\n \\internal\n This internal function is used to load a single line from an input stream\n */\n template<typename Fstream>\n bool load_from_stream(std::string filename, Fstream& fin,\n line_parser_type& line_parser) {\n size_t linecount = 0;\n timer ti; ti.start();\n while(fin.good() && !fin.eof()) {\n std::string line;\n std::getline(fin, line);\n if(line.empty()) continue;\n if(fin.fail()) break;\n const bool success = line_parser(*this, filename, line);\n if (!success) {\n logstream(LOG_WARNING)\n << \"Error parsing line \" << linecount << \" in \"\n << filename << \": \" << std::endl\n << \"\\t\\\"\" << line << \"\\\"\" << std::endl;\n return false;\n }\n ++linecount;\n if (ti.current_time() > 5.0) {\n logstream(LOG_INFO) << linecount << \" Lines read\" << std::endl;\n ti.start();\n }\n }\n return true;\n } // end of load from stream\n\n\n template<typename Fstream, typename Writer>\n void save_vertex_to_stream(vertex_type& vertex, Fstream& fout, Writer writer) {\n fout << writer.save_vertex(vertex);\n } // end of save_vertex_to_stream\n\n\n template<typename Fstream, typename Writer>\n void save_edge_to_stream(edge_type& edge, Fstream& fout, Writer writer) {\n std::string ret = writer.save_edge(edge);\n fout << ret;\n } // end of save_edge_to_stream\n\n\n void save_bintsv4_to_stream(std::ostream& out) {\n for (int i = 0; i < (int)local_graph.num_vertices(); ++i) {\n uint32_t src = l_vertex(i).global_id();\n foreach(local_edge_type e, l_vertex(i).out_edges()) {\n uint32_t dest = e.target().global_id();\n out.write(reinterpret_cast<char*>(&src), 4);\n out.write(reinterpret_cast<char*>(&dest), 4);\n }\n if (l_vertex(i).owner() == rpc.procid()) {\n vertex_type gv = vertex_type(l_vertex(i));\n // store disconnected vertices if I am the master of the vertex\n if (gv.num_in_edges() == 0 && gv.num_out_edges() == 0) {\n out.write(reinterpret_cast<char*>(&src), 4);\n uint32_t dest = (uint32_t)(-1);\n out.write(reinterpret_cast<char*>(&dest), 4);\n }\n }\n }\n }\n\n bool load_bintsv4_from_stream(std::istream& in) {\n while(in.good()) {\n uint32_t src, dest;\n in.read(reinterpret_cast<char*>(&src), 4);\n in.read(reinterpret_cast<char*>(&dest), 4);\n if (in.fail()) break;\n if (dest == (uint32_t)(-1)) {\n add_vertex(src);\n }\n else {\n add_edge(src, dest);\n }\n }\n return true;\n }\n\n\n /** \\brief Saves a distributed graph using a direct ostream saving function\n *\n * This function saves a sequence of files numbered\n * \\li [prefix]_0\n * \\li [prefix]_1\n * \\li [prefix]_2\n * \\li etc.\n *\n * This files can be loaded with direct_stream_load().\n */\n void save_direct(const std::string& prefix, bool gzip,\n boost::function<void (graph_type*, std::ostream&)> saver) {\n rpc.full_barrier();\n finalize();\n timer savetime; savetime.start();\n std::string fname = prefix + \"_\" + tostr(rpc.procid() + 1) + \"_of_\" +\n tostr(rpc.numprocs());\n if (gzip) fname = fname + \".gz\";\n logstream(LOG_INFO) << \"Save graph to \" << fname << std::endl;\n if(boost::starts_with(fname, \"hdfs://\")) {\n graphlab::hdfs hdfs;\n graphlab::hdfs::fstream out_file(hdfs, fname, true);\n boost::iostreams::filtering_stream<boost::iostreams::output> fout;\n if (gzip) fout.push(boost::iostreams::gzip_compressor());\n fout.push(out_file);\n if (!fout.good()) {\n logstream(LOG_FATAL) << \"\\n\\tError opening file: \" << fname << std::endl;\n exit(-1);\n }\n saver(this, boost::ref(fout));\n fout.pop();\n if (gzip) fout.pop();\n out_file.close();\n } else {\n std::ofstream out_file(fname.c_str(),\n std::ios_base::out | std::ios_base::binary);\n if (!out_file.good()) {\n logstream(LOG_FATAL) << \"\\n\\tError opening file: \" << fname << std::endl;\n exit(-1);\n }\n boost::iostreams::filtering_stream<boost::iostreams::output> fout;\n if (gzip) fout.push(boost::iostreams::gzip_compressor());\n fout.push(out_file);\n saver(this, boost::ref(fout));\n fout.pop();\n if (gzip) fout.pop();\n out_file.close();\n }\n logstream(LOG_INFO) << \"Finish saving graph to \" << fname << std::endl\n << \"Finished saving bintsv4 graph: \"\n << savetime.current_time() << std::endl;\n rpc.full_barrier();\n } // end of save\n\n\n\n /**\n * \\brief Load a graph from a collection of files in stored on\n * the filesystem using the user defined line parser. Like\n * \\ref load(const std::string& path, line_parser_type line_parser)\n * but only loads from the filesystem.\n */\n void load_direct_from_posixfs(std::string prefix,\n boost::function<bool (graph_type*, std::istream&)> parser) {\n std::string directory_name; std::string original_path(prefix);\n boost::filesystem::path path(prefix);\n std::string search_prefix;\n if (boost::filesystem::is_directory(path)) {\n // if this is a directory\n // force a \"/\" at the end of the path\n // make sure to check that the path is non-empty. (you do not\n // want to make the empty path \"\" the root path \"/\" )\n directory_name = path.native();\n }\n else {\n directory_name = path.parent_path().native();\n search_prefix = path.filename().native();\n directory_name = (directory_name.empty() ? \".\" : directory_name);\n }\n std::vector<std::string> graph_files;\n fs_util::list_files_with_prefix(directory_name, search_prefix, graph_files);\n if (graph_files.size() == 0) {\n logstream(LOG_WARNING) << \"No files found matching \" << original_path << std::endl;\n }\n for(size_t i = 0; i < graph_files.size(); ++i) {\n if (i % rpc.numprocs() == rpc.procid()) {\n logstream(LOG_EMPH) << \"Loading graph from file: \" << graph_files[i] << std::endl;\n // is it a gzip file ?\n const bool gzip = boost::ends_with(graph_files[i], \".gz\");\n // open the stream\n std::ifstream in_file(graph_files[i].c_str(),\n std::ios_base::in | std::ios_base::binary);\n // attach gzip if the file is gzip\n boost::iostreams::filtering_stream<boost::iostreams::input> fin;\n // Using gzip filter\n if (gzip) fin.push(boost::iostreams::gzip_decompressor());\n fin.push(in_file);\n const bool success = parser(this, boost::ref(fin));\n if(!success) {\n logstream(LOG_FATAL)\n << \"\\n\\tError parsing file: \" << graph_files[i] << std::endl;\n }\n fin.pop();\n if (gzip) fin.pop();\n }\n }\n rpc.full_barrier();\n }\n\n /**\n * \\brief Load a graph from a collection of files in stored on\n * the HDFS using the user defined line parser. Like\n * \\ref load(const std::string& path, line_parser_type line_parser)\n * but only loads from HDFS.\n */\n void load_direct_from_hdfs(std::string prefix,\n boost::function<bool (graph_type*, std::istream&)> parser) {\n // force a \"/\" at the end of the path\n // make sure to check that the path is non-empty. (you do not\n // want to make the empty path \"\" the root path \"/\" )\n std::string path = prefix;\n if (path.length() > 0 && path[path.length() - 1] != '/') path = path + \"/\";\n if(!hdfs::has_hadoop()) {\n logstream(LOG_FATAL)\n << \"\\n\\tAttempting to load a graph from HDFS but GraphLab\"\n << \"\\n\\twas built without HDFS.\"\n << std::endl;\n }\n hdfs& hdfs = hdfs::get_hdfs();\n std::vector<std::string> graph_files;\n graph_files = hdfs.list_files(path);\n if (graph_files.size() == 0) {\n logstream(LOG_WARNING) << \"No files found matching \" << prefix << std::endl;\n }\n for(size_t i = 0; i < graph_files.size(); ++i) {\n if (i % rpc.numprocs() == rpc.procid()) {\n logstream(LOG_EMPH) << \"Loading graph from file: \" << graph_files[i] << std::endl;\n // is it a gzip file ?\n const bool gzip = boost::ends_with(graph_files[i], \".gz\");\n // open the stream\n graphlab::hdfs::fstream in_file(hdfs, graph_files[i]);\n boost::iostreams::filtering_stream<boost::iostreams::input> fin;\n if(gzip) fin.push(boost::iostreams::gzip_decompressor());\n fin.push(in_file);\n const bool success = parser(this, boost::ref(fin));\n if(!success) {\n logstream(LOG_FATAL)\n << \"\\n\\tError parsing file: \" << graph_files[i] << std::endl;\n }\n fin.pop();\n if (gzip) fin.pop();\n }\n }\n rpc.full_barrier();\n }\n\n void load_direct(std::string prefix,\n boost::function<bool (graph_type*, std::istream&)> parser) {\n rpc.full_barrier();\n if(boost::starts_with(prefix, \"hdfs://\")) {\n load_direct_from_hdfs(prefix, parser);\n } else {\n load_direct_from_posixfs(prefix, parser);\n }\n rpc.full_barrier();\n } // end of load\n\n friend class tests::distributed_graph_test;\n }; // End of graph\n} // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n#endif\n" }, { "path": "src/graphlab/graph/dynamic_local_graph.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_DYNAMIC_LOCAL_GRAPH_HPP\n#define GRAPHLAB_DYNAMIC_LOCAL_GRAPH_HPP\n\n\n#include <cmath>\n\n#include <string>\n#include <list>\n#include <vector>\n#include <set>\n#include <map>\n\n#include <queue>\n#include <algorithm>\n#include <functional>\n#include <fstream>\n\n#include <boost/bind.hpp>\n#include <boost/unordered_set.hpp>\n#include <boost/type_traits.hpp>\n#include <boost/typeof/typeof.hpp>\n#include <boost/iterator/transform_iterator.hpp>\n#include <boost/iterator/counting_iterator.hpp>\n#include <boost/iterator/zip_iterator.hpp>\n#include <boost/range/iterator_range.hpp>\n\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/graph/local_edge_buffer.hpp>\n#include <graphlab/util/random.hpp>\n#include <graphlab/util/generics/shuffle.hpp>\n#include <graphlab/util/generics/counting_sort.hpp>\n#include <graphlab/util/generics/dynamic_csr_storage.hpp>\n#include <graphlab/parallel/atomic.hpp>\n\n#include <graphlab/logger/logger.hpp>\n#include <graphlab/logger/assertions.hpp>\n\n#include <graphlab/serialization/iarchive.hpp>\n#include <graphlab/serialization/oarchive.hpp>\n\n#include <graphlab/util/random.hpp>\n#include <graphlab/macros_def.hpp>\n\n\nnamespace graphlab {\n template<typename VertexData, typename EdgeData>\n class dynamic_local_graph {\n public:\n\n /** The type of the vertex data stored in the local_graph. */\n typedef VertexData vertex_data_type;\n\n /** The type of the edge data stored in the local_graph. */\n typedef EdgeData edge_data_type;\n\n typedef graphlab::vertex_id_type vertex_id_type;\n typedef graphlab::edge_id_type edge_id_type;\n\n private:\n class edge_iterator;\n\n public:\n typedef boost::iterator_range<edge_iterator> edge_list_type;\n\n /** Vertex object which provides access to the vertex data\n * and information about it.\n */\n class vertex_type;\n\n /** Edge object which provides access to the edge data\n * and information about it.\n */\n class edge_type;\n\n public:\n\n // CONSTRUCTORS ============================================================>\n /** Create an empty local_graph. */\n dynamic_local_graph() { }\n\n /** Create a local_graph with nverts vertices. */\n dynamic_local_graph(size_t nverts) :\n vertices(nverts) {}\n\n // METHODS =================================================================>\n\n static bool is_dynamic() {\n return true;\n }\n\n /**\n * \\brief Resets the local_graph state.\n */\n void clear() {\n vertices.clear();\n edges.clear();\n _csc_storage.clear();\n _csr_storage.clear();\n std::vector<VertexData>().swap(vertices);\n std::vector<EdgeData>().swap(edges);\n edge_buffer.clear();\n }\n\n /** \\brief Get the number of vertices */\n size_t num_vertices() const {\n return vertices.size();\n } // end of num vertices\n\n /** \\brief Get the number of edges */\n size_t num_edges() const {\n return edges.size();\n } // end of num edges\n\n\n /**\n * \\brief Creates a vertex containing the vertex data and returns the id\n * of the new vertex id. Vertex ids are assigned in increasing order with\n * the first vertex having id 0.\n */\n void add_vertex(lvid_type vid, const VertexData& vdata = VertexData() ) {\n if(vid >= vertices.size()) {\n // Enable capacity doubling if resizing beyond capacity\n if(vid >= vertices.capacity()) {\n const size_t new_size = std::max(2 * vertices.capacity(),\n size_t(vid));\n vertices.reserve(new_size);\n }\n vertices.resize(vid+1);\n }\n vertices[vid] = vdata;\n } // End of add vertex;\n\n void reserve(size_t num_vertices) {\n ASSERT_GE(num_vertices, vertices.size());\n vertices.reserve(num_vertices);\n }\n\n /**\n * \\brief Add additional vertices up to provided num_vertices. This will\n * fail if resizing down.\n */\n void resize(size_t num_vertices ) {\n ASSERT_GE(num_vertices, vertices.size());\n vertices.resize(num_vertices);\n } // End of resize\n\n void reserve_edge_space(size_t n) {\n edge_buffer.reserve_edge_space(n);\n }\n /**\n * \\brief Creates an edge connecting vertex source to vertex target. Any\n * existing data will be cleared. Should not be called after finalization.\n */\n edge_id_type add_edge(lvid_type source, lvid_type target,\n const EdgeData& edata = EdgeData()) {\n if(source == target) {\n logstream(LOG_FATAL)\n << \"Attempting to add self edge (\" << source << \" -> \" << target << \"). \"\n << \"This operation is not permitted in GraphLab!\" << std::endl;\n ASSERT_MSG(source != target, \"Attempting to add self edge!\");\n }\n\n if(source >= vertices.size() || target >= vertices.size())\n add_vertex(std::max(source, target));\n\n // Add the edge to the set of edge data (this copies the edata)\n edge_buffer.add_edge(source, target, edata);\n\n // This is not the final edge_id, so we always return 0.\n return 0;\n } // End of add edge\n\n /**\n * \\brief Add edges in block.\n */\n void add_edges(const std::vector<lvid_type>& src_arr,\n const std::vector<lvid_type>& dst_arr,\n const std::vector<EdgeData>& edata_arr) {\n ASSERT_TRUE((src_arr.size() == dst_arr.size())\n && (src_arr.size() == edata_arr.size()));\n\n for (size_t i = 0; i < src_arr.size(); ++i) {\n lvid_type source = src_arr[i];\n lvid_type target = dst_arr[i];\n if ( source >= vertices.size()\n || target >= vertices.size() ) {\n logstream(LOG_FATAL)\n << \"Attempting add_edge (\" << source\n << \" -> \" << target\n << \") when there are only \" << vertices.size()\n << \" vertices\" << std::endl;\n ASSERT_MSG(source < vertices.size(), \"Invalid source vertex!\");\n ASSERT_MSG(target < vertices.size(), \"Invalid target vertex!\");\n }\n\n if(source == target) {\n logstream(LOG_FATAL)\n << \"Attempting to add self edge (\" << source << \" -> \" << target << \"). \"\n << \"This operation is not permitted in GraphLab!\" << std::endl;\n ASSERT_MSG(source != target, \"Attempting to add self edge!\");\n }\n }\n edge_buffer.add_block_edges(src_arr, dst_arr, edata_arr);\n } // End of add block edges\n\n\n /** \\brief Returns a vertex of given ID. */\n vertex_type vertex(lvid_type vid) {\n ASSERT_LT(vid, vertices.size());\n return vertex_type(*this, vid);\n }\n\n /** \\brief Returns a vertex of given ID. */\n const vertex_type vertex(lvid_type vid) const {\n ASSERT_LT(vid, vertices.size());\n return vertex_type(*this, vid);\n }\n\n /** \\brief Returns a reference to the data stored on the vertex v. */\n VertexData& vertex_data(lvid_type v) {\n ASSERT_LT(v, vertices.size());\n return vertices[v];\n } // end of data(v)\n\n /** \\brief Returns a constant reference to the data stored on the vertex v. */\n const VertexData& vertex_data(lvid_type v) const {\n ASSERT_LT(v, vertices.size());\n return vertices[v];\n } // end of data(v)\n\n /**\n * \\brief Finalize the local_graph data structure by\n * sorting edges to maximize the efficiency of graphlab.\n * This function takes O(|V|log(degree)) time and will\n * fail if there are any duplicate edges.\n * Detail implementation depends on the type of graph_storage.\n * This is also automatically invoked by the engine at start.\n */\n void finalize() {\n\n graphlab::timer mytimer; mytimer.start();\n#ifdef DEBUG_GRAPH\n logstream(LOG_DEBUG) << \"Graph2 finalize starts.\" << std::endl;\n#endif\n std::vector<edge_id_type> src_permute;\n std::vector<edge_id_type> dest_permute;\n std::vector<edge_id_type> src_counting_prefix_sum;\n std::vector<edge_id_type> dest_counting_prefix_sum;\n\n#ifdef DEBUG_GRAPH\n logstream(LOG_DEBUG) << \"Graph2 finalize: Sort by source vertex\" << std::endl;\n#endif\n counting_sort(edge_buffer.source_arr, dest_permute, &src_counting_prefix_sum);\n#ifdef DEBUG_GRAPH\n logstream(LOG_DEBUG) << \"Graph2 finalize: Sort by dest id\" << std::endl;\n#endif\n counting_sort(edge_buffer.target_arr, src_permute, &dest_counting_prefix_sum);\n\n std::vector< std::pair<lvid_type, edge_id_type> > csr_values;\n std::vector< std::pair<lvid_type, edge_id_type> > csc_values;\n\n csr_values.reserve(dest_permute.size());\n edge_id_type begineid = edges.size();\n for (size_t i = 0; i < dest_permute.size(); ++i) {\n csr_values.push_back(std::pair<lvid_type, edge_id_type> (edge_buffer.target_arr[dest_permute[i]],\n begineid + dest_permute[i]));\n }\n csc_values.reserve(src_permute.size());\n\n for (size_t i = 0; i < src_permute.size(); ++i) {\n csc_values.push_back(std::pair<lvid_type, edge_id_type> (edge_buffer.source_arr[src_permute[i]],\n begineid + src_permute[i]));\n }\n ASSERT_EQ(csc_values.size(), csr_values.size());\n\n // fast path with first time insertion.\n if (edges.size() == 0) {\n edges.swap(edge_buffer.data);\n edge_buffer.clear();\n // warp into csr csc storage.\n _csr_storage.wrap(src_counting_prefix_sum, csr_values);\n _csc_storage.wrap(dest_counting_prefix_sum, csc_values);\n } else {\n // insert edge data\n edges.reserve(edges.size() + edge_buffer.size());\n edges.insert(edges.end(), edge_buffer.data.begin(), edge_buffer.data.end());\n std::vector<EdgeData>().swap(edge_buffer.data);\n edge_buffer.clear();\n size_t begin, end;\n for (size_t i = 0; i < src_counting_prefix_sum.size(); ++i) {\n begin = src_counting_prefix_sum[i];\n end = (i==src_counting_prefix_sum.size()-1)\n ? csr_values.size()\n : src_counting_prefix_sum[i+1];\n if (end > begin) {\n _csr_storage.insert(i, csr_values.begin()+begin, csr_values.begin()+end);\n }\n }\n for (size_t i = 0; i < dest_counting_prefix_sum.size(); ++i) {\n begin = dest_counting_prefix_sum[i];\n end = (i==dest_counting_prefix_sum.size()-1)\n ? csc_values.size()\n : dest_counting_prefix_sum[i+1];\n if (end > begin) {\n _csc_storage.insert(i, csc_values.begin()+begin, csc_values.begin()+end);\n }\n }\n _csr_storage.repack();\n _csc_storage.repack();\n }\n ASSERT_EQ(_csr_storage.num_values(), _csc_storage.num_values());\n ASSERT_EQ(_csr_storage.num_values(), edges.size());\n\n#ifdef DEBUG_GRAPH\n logstream(LOG_DEBUG) << \"End of finalize.\" << std::endl;\n#endif\n logstream(LOG_INFO) << \"Graph finalized in \" << mytimer.current_time()\n << \" secs\" << std::endl;\n\n#ifdef DEBUG_GRAPH\n _csr_storage.meminfo(std::cerr);\n _csc_storage.meminfo(std::cerr);\n#endif\n } // End of finalize\n\n\n /** \\brief Load the local_graph from an archive */\n void load(iarchive& arc) {\n clear();\n // read the vertices\n arc >> vertices\n >> edges\n >> _csr_storage\n >> _csc_storage;\n } // end of load\n\n /** \\brief Save the local_graph to an archive */\n void save(oarchive& arc) const {\n // Write the number of edges and vertices\n arc << vertices\n << edges\n << _csr_storage\n << _csc_storage;\n } // end of save\n\n /** swap two graphs */\n void swap(dynamic_local_graph& other) {\n std::swap(vertices, other.vertices);\n std::swap(edges, other.edges);\n std::swap(_csr_storage, other._csr_storage);\n std::swap(_csc_storage, other._csc_storage);\n } // end of swap\n\n\n /** \\brief Load the local_graph from a file */\n void load(const std::string& filename) {\n std::ifstream fin(filename.c_str());\n iarchive iarc(fin);\n iarc >> *this;\n fin.close();\n } // end of load\n\n /**\n * \\brief save the local_graph to the file given by the filename\n */\n void save(const std::string& filename) const {\n std::ofstream fout(filename.c_str());\n oarchive oarc(fout);\n oarc << *this;\n fout.close();\n } // end of save\n\n /**\n * \\brief save the adjacency structure to a text file.\n *\n * Save the adjacency structure as a text file in:\n * src_Id, dest_Id \\n\n * src_Id, dest_Id \\n\n * format.\n */\n void save_adjacency(const std::string& filename) const {\n std::ofstream fout(filename.c_str());\n ASSERT_TRUE(fout.good());\n\n for (size_t i = 0; i < num_vertices(); ++i) {\n vertex_type v(i);\n edge_list_type ls = v.out_edges();\n foreach(edge_type e, ls) {\n fout << (lvid_type)i << \", \" << e.target().id() << \"\\n\";\n ASSERT_TRUE(fout.good());\n }\n }\n fout.close();\n }\n\n /****************************************************************************\n * Internal Functions *\n * ---------------------- *\n * These functions functions and types provide internal access to the *\n * underlying local_graph representation. They should not be used unless you *\n * *really* know what you are doing. *\n ****************************************************************************/\n /**\n * \\internal\n * \\brief Returns the number of in edges of the vertex with the given id. */\n size_t num_in_edges(const lvid_type v) const {\n return _csc_storage.begin(v).pdistance_to(_csc_storage.end(v));\n }\n\n /**\n * \\internal\n * \\brief Returns the number of in edges of the vertex with the given id. */\n size_t num_out_edges(const lvid_type v) const {\n return _csr_storage.begin(v).pdistance_to(_csr_storage.end(v));\n }\n\n /**\n * \\internal\n * \\brief Returns a list of in edges of the vertex with the given id. */\n edge_list_type in_edges(lvid_type v) {\n edge_iterator begin = edge_iterator(*this, edge_iterator::CSC,\n _csc_storage.begin(v), v);\n edge_iterator end = edge_iterator(*this, edge_iterator::CSC,\n _csc_storage.end(v), v);\n return boost::make_iterator_range(begin, end);\n }\n\n /**\n * \\internal\n * \\brief Returns a list of out edges of the vertex with the given id. */\n edge_list_type out_edges(lvid_type v) {\n edge_iterator begin = edge_iterator(*this, edge_iterator::CSR,\n _csr_storage.begin(v), v);\n edge_iterator end = edge_iterator(*this, edge_iterator::CSR,\n _csr_storage.end(v), v);\n return boost::make_iterator_range(begin, end);\n }\n\n /**\n * \\internal\n * \\brief Returns edge data of edge_type e\n * */\n EdgeData& edge_data(edge_id_type eid) {\n ASSERT_LT(eid, num_edges());\n return edges[eid];\n }\n /**\n * \\internal\n * \\brief Returns const edge data of edge_type e\n * */\n const EdgeData& edge_data(edge_id_type eid) const {\n ASSERT_LT(eid, num_edges());\n return edges[eid];\n }\n\n /**\n * \\internal\n * \\brief Returns the estimated memory footprint of the local_graph. */\n size_t estimate_sizeof() const {\n const size_t vlist_size = sizeof(vertices) +\n sizeof(VertexData) * vertices.capacity();\n size_t elist_size = _csr_storage.estimate_sizeof()\n + _csc_storage.estimate_sizeof()\n + sizeof(edges) + sizeof(EdgeData)*edges.capacity();\n size_t ebuffer_size = edge_buffer.estimate_sizeof();\n return vlist_size + elist_size + ebuffer_size;\n }\n\n /** \\internal\n * \\brief For debug purpose, returns the largest vertex id in the edge_buffer\n */\n const lvid_type maxlvid() const {\n if (edge_buffer.size()) {\n lvid_type max(0);\n foreach(lvid_type i, edge_buffer.source_arr)\n max = std::max(max, i);\n foreach(lvid_type i, edge_buffer.target_arr)\n max = std::max(max, i);\n return max;\n } else {\n return lvid_type(-1);\n }\n }\n\n private:\n /**\n * \\internal\n * CSR/CSC storage types\n */\n typedef dynamic_csr_storage<std::pair<lvid_type, edge_id_type>, edge_id_type> csr_type;\n\n typedef typename csr_type::iterator csr_edge_iterator;\n\n // PRIVATE DATA MEMBERS ===================================================>\n //\n /** The vertex data is simply a vector of vertex data */\n std::vector<VertexData> vertices;\n\n /** Stores the edge data and edge relationships. */\n csr_type _csr_storage;\n csr_type _csc_storage;\n std::vector<EdgeData> edges;\n\n /** The edge data is a vector of edges where each edge stores its\n source, destination, and data. Used for temporary storage. The\n data is transferred into CSR+CSC representation in\n Finalize. This will be cleared after finalized.*/\n local_edge_buffer<VertexData, EdgeData> edge_buffer;\n\n /**************************************************************************/\n /* */\n /* declare friends */\n /* */\n /**************************************************************************/\n friend class local_graph_test;\n }; // End of class dynamic_local_graph\n\n\n template<typename VertexData, typename EdgeData>\n std::ostream& operator<<(std::ostream& out,\n const dynamic_local_graph<VertexData, EdgeData>& local_graph) {\n for(lvid_type vid = 0; vid < local_graph.num_vertices(); ++vid) {\n foreach(edge_id_type eid, local_graph.out_edge_ids(vid))\n out << vid << \", \" << local_graph.target(eid) << '\\n';\n }\n return out;\n }\n} // end of namespace graphlab\n\n\n/////////////////////// Implementation of Helper Class ////////////////////////////\n\nnamespace graphlab {\n template<typename VertexData, typename EdgeData>\n class dynamic_local_graph<VertexData, EdgeData>::vertex_type {\n public:\n vertex_type(dynamic_local_graph& lgraph_ref, lvid_type vid):lgraph_ref(lgraph_ref),vid(vid) { }\n\n /// \\brief Returns a constant reference to the data on the vertex.\n const vertex_data_type& data() const {\n return lgraph_ref.vertex_data(vid);\n }\n /// \\brief Returns a reference to the data on the vertex.\n vertex_data_type& data() {\n return lgraph_ref.vertex_data(vid);\n }\n /// \\brief Returns the number of in edges of the vertex.\n size_t num_in_edges() const {\n return lgraph_ref.num_in_edges(vid);\n }\n /// \\brief Returns the number of out edges of the vertex.\n size_t num_out_edges() const {\n return lgraph_ref.num_out_edges(vid);\n }\n /// \\brief Returns the ID of the vertex.\n lvid_type id() const {\n return vid;\n }\n /// \\brief Returns a list of in edges.\n edge_list_type in_edges() {\n return lgraph_ref.in_edges(vid);\n }\n /// \\brief Returns a list of out edges.\n edge_list_type out_edges() {\n return lgraph_ref.out_edges(vid);\n }\n private:\n dynamic_local_graph& lgraph_ref;\n lvid_type vid;\n };\n\n template<typename VertexData, typename EdgeData>\n class dynamic_local_graph<VertexData, EdgeData>::edge_type {\n public:\n edge_type(dynamic_local_graph& lgraph_ref, lvid_type _source, lvid_type _target, edge_id_type _eid) :\n lgraph_ref(lgraph_ref), _source(_source), _target(_target), _eid(_eid) { }\n\n /// \\brief Returns a constant reference to the data on the edge.\n const edge_data_type& data() const {\n return lgraph_ref.edge_data(_eid);\n }\n /// \\brief Returns a reference to the data on the edge.\n edge_data_type& data() {\n return lgraph_ref.edge_data(_eid);\n }\n /// \\brief Returns the source vertex of the edge.\n vertex_type source() const {\n return vertex_type(lgraph_ref, _source);\n }\n /// \\brief Returns the target vertex of the edge.\n vertex_type target() const {\n return vertex_type(lgraph_ref, _target);\n }\n /// \\brief Returns the internal ID of this edge\n edge_id_type id() const { return _eid; }\n\n private:\n dynamic_local_graph& lgraph_ref;\n lvid_type _source;\n lvid_type _target;\n edge_id_type _eid;\n };\n\n template<typename VertexData, typename EdgeData>\n class dynamic_local_graph<VertexData, EdgeData>::edge_iterator :\n public boost::iterator_facade < edge_iterator,\n edge_type,\n boost::random_access_traversal_tag,\n edge_type> {\n public:\n enum list_type {CSR, CSC};\n\n edge_iterator(dynamic_local_graph& lgraph_ref, list_type _type,\n csr_edge_iterator _iter, lvid_type _vid)\n : lgraph_ref(lgraph_ref), _type(_type), _iter(_iter), _vid(_vid) {}\n\n private:\n friend class boost::iterator_core_access;\n\n void increment() {\n ++_iter;\n }\n bool equal(const edge_iterator& other) const\n {\n ASSERT_EQ(_type, other._type);\n return _iter == other._iter;\n }\n edge_type dereference() const {\n return make_value();\n }\n void advance(int n) {\n _iter += n;\n }\n ptrdiff_t distance_to(const edge_iterator& other) const {\n return (other._iter - _iter);\n }\n private:\n edge_type make_value() const {\n typename csr_edge_iterator::reference ref = *_iter;\n switch (_type) {\n case CSC: {\n return edge_type(lgraph_ref, ref.first, _vid, ref.second);\n }\n case CSR: {\n return edge_type(lgraph_ref, _vid, ref.first, ref.second);\n }\n default: return edge_type(lgraph_ref, -1, -1, -1);\n }\n }\n dynamic_local_graph& lgraph_ref;\n const list_type _type;\n csr_edge_iterator _iter;\n const lvid_type _vid;\n }; // end of edge_iterator\n\n} // end of namespace\n\n\nnamespace std {\n /**\n * Swap two graphs\n */\n template<typename VertexData, typename EdgeData>\n inline void swap(graphlab::dynamic_local_graph<VertexData,EdgeData>& a,\n graphlab::dynamic_local_graph<VertexData,EdgeData>& b) {\n a.swap(b);\n } // end of swap\n}; // end of namespace std\n\n\n#include <graphlab/macros_undef.hpp>\n#endif\n // Insert edges into finalized graph\n// if (finalized) {\n// graphlab::timer mytimer; mytimer.start();\n// #ifdef DEBUG_GRAPH\n// logstream(LOG_INFO) << \"Insert edges into finalized graph...\" << std::endl;\n// #endif\n// // insert adjacency into csr/csc\n// for (size_t i = 0; i < edge_buffer.size(); ++i) {\n// edge_id_type eid = edges.size() + i;\n// _csr_storage.insert(edge_buffer.source_arr[i],\n// std::pair<lvid_type, edge_id_type>(\n// edge_buffer.target_arr[i], eid));\n// _csc_storage.insert(edge_buffer.target_arr[i],\n// std::pair<lvid_type,edge_id_type>(\n// edge_buffer.source_arr[i], eid));\n// }\n//\n// // insert edge data\n// edges.reserve(edges.size() + edge_buffer.size());\n// edges.insert(edges.end(), edge_buffer.data.begin(), edge_buffer.data.end());\n// std::vector<EdgeData>().swap(edge_buffer.data);\n//\n// edge_buffer.clear();\n//\n// #ifdef DEBGU_GRAPH\n// logstream(LOG_DEBUG) << \"Finish finalization.\" << std::endl;\n// #endif\n// logstream(LOG_INFO) << \"Graph finalized in \" << mytimer.current_time()\n// << \" secs\" << std::endl;\n// _csr_storage.meminfo(std::cerr);\n// _csc_storage.meminfo(std::cerr);\n// return;\n// }\n" }, { "path": "src/graphlab/graph/graph_basic_types.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_GRAPH_BASIC_TYPES\n#define GRAPHLAB_GRAPH_BASIC_TYPES\n\n#include <stdint.h>\n\nnamespace graphlab {\n\n\n\n\n#ifdef USE_VID32\n /// Identifier type of a vertex which is globally consistent. Guaranteed to be integral\n typedef uint32_t vertex_id_type;\n#else\n typedef uint64_t vertex_id_type;\n#endif\n\n /// Identifier type of a vertex which is only locally consistent. Guaranteed to be integral\n typedef vertex_id_type lvid_type;\n\n /**\n * Identifier type of an edge which is only locally\n * consistent. Guaranteed to be integral and consecutive.\n */\n typedef lvid_type edge_id_type;\n\n /**\n * \\brief The set of edges that are traversed during gather and scatter\n * operations.\n */\n enum edge_dir_type {\n /**\n * \\brief No edges implies that no edges are processed during the\n * corresponding gather or scatter phase, essentially skipping\n * that phase.\n */\n NO_EDGES = 0,\n /**\n * \\brief In edges implies that only whose target is the center\n * vertex are processed during gather or scatter.\n */\n IN_EDGES = 1,\n /**\n * \\brief Out edges implies that only whose source is the center\n * vertex are processed during gather or scatter.\n */\n OUT_EDGES = 2 ,\n /**\n * \\brief All edges implies that all adges adjacent to a the\n * center vertex are processed on gather or scatter. Note that\n * some neighbors may be encountered twice if there is both an in\n * and out edge to that neighbor.\n */\n ALL_EDGES = 3};\n} // end of namespace graphlab\n\n#endif\n" }, { "path": "src/graphlab/graph/graph_formats.dox", "content": "/**\n\n\\page graph_formats Graph File Formats\n\nWe build in support for 3 common portable graph file formats (tsv, snap, adj),\none GraphLab specific portable format (bintsv4) as well 2 GraphLab specific\nnon-portable formats (graphjrl, bin).\n\n\\section graph_portable_formats Portable Formats\nAll portable graph file formats supported are unable to store graph data,\nbut can only store graph structure. The formats currently with built-in support\nare \"tsv\", \"snap\", \"adj\" and \"bintsv4\", described below. Graphs of this format\ncan be saved / loaded using graphlab::distributed_graph::save_format()\nand graphlab::distributed_graph::load_format() functions. \n\n\"tsv\", \"snap\" and \"adj\" are text formats and are human readable.\n\n\"bintsv4\" is a binary format.\n\n\n\\subsection graph_tsv_format tsv (edge list)\nThe TSV format is a simple edge-list between vertices where each line in \nthe file is a [src ID] [target ID] pair separated by whitespace.\n\nFor instance, the following graph:\n\\image html graph_format_example.gif\n\ncan be stored as:\n\\verbatim\n1 2\n1 5\n7 5\n5 7\n7 1\n\\endverbatim\n\nNote that vertex IDs do not need to be consecutive, and edges may appear\nin any arbitrary order. Furthermore, the graph specification requires that\nvertex IDs are 32-bit integers ranging from 0 to (2^32 - 2). The ID (2^32 - 1)\nis reserved.\n\nEmpty lines in the file are permissible, but no other symbols are permitted.\n\nObserve that the TSV format cannot store vertices with no edges.\n \n\\subsection graph_snap_format snap (edge list)\nThe SNAP file format is supported to simplify the use of datasets from \nthe <a href=http://snap.stanford.edu/data/>Stanford Large Network Dataset Collection</a>\n\nThe format is identical to \\ref TSV TSV with one minor difference: lines\nbeginning with \"#\" are treated as comments and ignored.\n\nFor instance, the following graph:\n\\image html graph_format_example.gif\n\ncan be stored as:\n\\verbatim\n# example graph\n# vertices: 4 edges: 5\n1 2\n1 5\n7 5\n5 7\n7 1\n\\endverbatim\n\nNote that while the file includes a vertex and an edge count in this example,\nthey are treated as comments and ignored.\n\nObserve that the SNAP format cannot store vertices with no edges.\n\n\\subsection graph_adj_format adj (adjacency list) \nThe Adjacency list file format stores on each line, a source vertex, followed\nby a list of all target vertices: each line has the following format:\n\n\\verbatim\n[vertex ID] [number of target vertices] [target ID 1] [target ID 2] [target ID 3] ...\n\\endverbatim\n\nThis format is more compact to store, and in practice will partition better\nin the distributed setting (since edges have a natural grouping).\nFurthermore, this format is capable of storing disconnected vertices.\n\nFor instance, the following graph:\n\\image html graph_format_example.gif\n\ncan be stored as:\n\\verbatim\n1 2 2 5\n7 2 7 1\n5 1 7\n\\endverbatim\n\nWe may include the line\n\\verbatim\n2 0\n\\endverbatim\nto identify that vertex 2 has no out-edges.\nHowever, this is optional since vertex 2 will be created when the edge\n<tt>1->2</tt> is created. such lines are only necessary for truly disconnected\nvertices.\n\n\n\\subsection graph_bintsv4_format bintsv4 (binary edge list)\nThe bintsv4 format is a binary storage format. The graph is represented\nas a sequence of 8 byte blocks: \n\n\\verbatim\n---------------------------------\n| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |\n---------------------------------\n| src VID | dest VID |\n---------------------------------\n\\endverbatim\n\nWhere each block stores a pair of 32 bit unsigned integer values in x86 \nlittle endian format. \nEach block represents an edge src -> dest. Vertex IDs cannot take on the value\n2^32-1\n\nDisconnected vertices are stored as:\n\n\\verbatim\n---------------------------------\n| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |\n---------------------------------\n| VID | 2^32 - 1 |\n---------------------------------\n\\endverbatim\n\n\n\n\\section graph_nonportable_formats Non-Portable Formats\nThe non-portable formats store all information in the graph including the\ngraph data. These formats are convenient and in the case of the \"bin\" format\nsignificantly faster to load. However, they are also brittle in that \nchanges to your vertex/edge data serialization will render any saved files\nunreadable. Also, we may make changes to the serialization binary format\nat any point. These formats should be treated as fast, temporary storage\nmethods and must not be used for long-term archival. \n\n\\subsection graph_format_graphjrl graphjrl (Graph Journal) \n\nThe GraphJRL format serializes each vertex and edge onto a <tt>\\n</tt> \nterminated line, ensuring to escape any <tt>\\n</tt> characters that occur\nwithin the serialized data.\n\nThis format is not human readable and should be treated as temporary storage.\nUnlike the \"bin\" format below, graphs saved in this format do not require\nthe same number of machines to load the graph. i.e. graphs saved using 8 \nmachines can be loaded using any arbitrary number of machines.\n\n\n\\subsection graph_format_bin bin (Distributed Graph Binary)\nThis format is simply a direct serialization of all Distributed Graph\ndatastructures. The graph is finalized before saving, and thus do not need\nto be finalized after loading. This is the most efficient storage\nof the distributed graph, requiring the least loading time.\n\nHowever, the disadvantage of the \"bin\" format is that it requires exactly the \nsame number of machines to load the graph as there was when saving the graph.\nIn other words, if 8 machines were used to save the graph, it must be loaded\nusing exactly 8 machines. \n*/\n" }, { "path": "src/graphlab/graph/graph_gather_apply.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n#ifndef GRAPHLAB_GRAPH_GATHER_APPLY_HPP\n#define GRAPHLAB_GRAPH_GATHER_APPLY_HPP\n\n#include <deque>\n#include <boost/bind.hpp>\n#include <graphlab/options/graphlab_options.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/graph/vertex_set.hpp>\n#include <graphlab/util/memory_info.hpp>\n#include <graphlab/parallel/thread_pool.hpp>\n\n#include <graphlab/rpc/dc_dist_object.hpp>\n#include <graphlab/rpc/buffered_exchange.hpp>\n\n#include <graphlab/macros_def.hpp>\n\nnamespace graphlab {\n template<typename Graph, typename GatherType>\n class graph_gather_apply {\n public:\n /**\n * \\brief The type of graph supported by this vertex program\n *\n * See graphlab::distributed_graph\n */\n typedef Graph graph_type;\n\n /**\n * \\brief Graph related types\n */\n typedef typename graph_type::vertex_type vertex_type;\n typedef typename graph_type::vertex_data_type vertex_data_type;\n typedef typename graph_type::mirror_type mirror_type;\n typedef typename graph_type::local_vertex_type local_vertex_type;\n typedef typename graph_type::local_edge_type local_edge_type;\n typedef typename graph_type::lvid_type lvid_type;\n\n\n /**\n * \\brief The result type of the gather operation.\n */\n typedef GatherType gather_type; \n\n /**\n * \\brief The type of the gather function. \n */\n typedef typename boost::function<gather_type (lvid_type, graph_type&)> gather_fun_type;\n\n /**\n * \\brief The gather operation which will be called on each vertex (master and mirrors) and send to the master vertex.\n */\n gather_fun_type gather_fun;\n\n /**\n * \\brief The type of the apply function. \n */\n typedef typename boost::function<void (lvid_type, const gather_type& accum, graph_type&)> apply_fun_type;\n\n /**\n * \\brief The apply operation which will be called on each vertex (master and mirrors) with the result of gather.\n */\n apply_fun_type apply_fun;\n\n private:\n\n /**\n * \\brief The object used to communicate with remote copies of the\n * synchronous engine.\n */\n dc_dist_object< graph_gather_apply<Graph,GatherType> > rmi;\n\n /**\n * \\brief A reference to the distributed graph on which this\n * synchronous engine is running.\n */\n graph_type& graph;\n\n /**\n * \\brief The local worker threads used by this engine\n */\n thread_pool threads;\n\n /**\n * \\brief A thread barrier that is used to control the threads in the\n * thread pool.\n */\n graphlab::barrier thread_barrier;\n\n /**\n * \\brief The shared counter used coordinate operations between\n * threads.\n */\n atomic<size_t> shared_lvid_counter;\n\n /**\n * \\brief The vertex locks protect access to vertex specific data-structrues including\n * \\ref graphlab::graph_gather_apply::gather_accum.\n */\n std::vector<simple_spinlock> vlocks;\n\n /**\n * \\brief Bit indicating if the gather has accumulator contains any values.\n *\n * While dense bitsets are thread safe the value of this bit must change concurrently with \n * the \\ref graphlab::graph_gather_apply and therefore is set while holding the lock in \n * \\ref graphlab::graph_gather_apply::vlocks\n */\n dense_bitset has_gather_accum;\n \n\n /**\n * \\brief Gather accumulator used for each master vertex to merge the result of all the machine\n * specific accumulators.\n *\n * The gather accumulator can be accessed by multiple threads at once and therefore must be guarded \n * by a vertex locks in \\ref graphlab::graph_gather_apply::vlocks\n */\n std::vector<gather_type> gather_accum;\n\n\n /**\n * \\brief The pair type used to synchronize the results of the gather phase\n */\n typedef std::pair<vertex_id_type, gather_type> vid_gather_pair_type;\n\n /**\n * \\brief The type of the exchange used to synchronize gather\n * accumulators\n */\n typedef buffered_exchange<vid_gather_pair_type> gather_exchange_type;\n\n /**\n * \\brief The distributed exchange used to synchronize gather\n * accumulators.\n */\n gather_exchange_type gather_exchange;\n\n\n public:\n\n /**\n * \\brief Construct a graph gather_apply operation with a given graph and \n * gather apply functions.\n *\n *\n * In the distributed engine the synchronous engine must be called\n * on all machines at the same time (in the same order) passing\n * the \\ref graphlab::distributed_control object. Upon\n * construction the synchronous engine allocates several\n * data-structures to store messages, gather accumulants, and\n * vertex programs and therefore may require considerable memory.\n *\n * The number of threads to create are read from\n * \\ref graphlab_options::get_ncpus \"opts.get_ncpus()\".\n *\n * See the <a href=\"#engineopts\">main class documentation</a>\n * for details on the available options.\n *\n * @param [in] dc Distributed controller to associate with\n * @param [in,out] graph A reference to the graph object that this\n * engine will modify. The graph must be fully constructed and\n * finalized.\n * @param [in] opts A graphlab::graphlab_options object specifying engine\n * parameters. This is typically constructed using\n * \\ref graphlab::command_line_options.\n */\n graph_gather_apply(graph_type& graph,\n gather_fun_type gather_fun,\n apply_fun_type apply_fun,\n const graphlab_options& opts = graphlab_options());\n\n\n /**\n * \\brief Start execution of the synchronous engine.\n *\n * The start function begins computation and does not return until\n * there are no remaining messages or until max_iterations has\n * been reached.\n *\n * The start() function modifies the data graph through the vertex\n * programs and so upon return the data graph should contain the\n * result of the computation.\n *\n * @return The reason for termination\n */\n void exec(const vertex_set& vset = vertex_set(true));\n\n private:\n // Program Steps ==========================================================\n /**\n * \\brief Executes ncpus copies of a member function each with a\n * unique consecutive id (thread id).\n *\n * This function is used by the main loop to execute each of the\n * stages in parallel.\n *\n * The member function must have the type:\n *\n * \\code\n * void graph_gather_apply::member_fun(size_t threadid, const vertex_set& vset);\n * \\endcode\n *\n * This function runs an rmi barrier after termination\n *\n * @tparam the type of the member function.\n * @param [in] member_fun the function to call.\n */\n template<typename MemberFunction>\n void run_synchronous(MemberFunction member_fun, const vertex_set& vset) {\n shared_lvid_counter = 0;\n if (threads.size() <= 1) {\n (this->*(member_fun))(0, vset);\n }\n else {\n // launch the initialization threads\n for(size_t i = 0; i < threads.size(); ++i) {\n boost::function<void(void)> invoke = boost::bind(member_fun, this, i, vset);\n threads.launch(invoke, i);\n }\n }\n // Wait for all threads to finish\n threads.join();\n rmi.barrier();\n } // end of run_synchronous\n\n /**\n * \\brief Execute the \\ref graphlab::graph_gather_apply::gather_fun function on all\n * vertices in the vset. The result of the gather will be send to the master proc of the vertex. \n *\n * The accumulators are stored in \\ref graphlab::graph_gather_apply::gather_accum if the \n * the vertex is pre-allocated in the local graph of the proc. Otherwise, it wil be stored in\n * \\ref graphlab::graph_gather_apply::temporary_gather_map and be merged in the graph.\n * \n * @param thread_id the thread to run this as which determines\n * @param vset the vertex set specifying the set of vertex to run the gather operation. \n */\n void execute_gathers(const size_t thread_id, const vertex_set& vset);\n\n\n /**\n * \\brief Scatter the gather accumulator from master to the mirrors.\n *\n * @param thread_id the thread to run this as which determines\n * @param vset the vertex set specifying the set of vertex to run the gather operation. \n */\n void execute_scatters(const size_t thread_id, const vertex_set& vset);\n\n /**\n * \\brief Execute the \\ref graphlab::graph_gahter_apply::apply_fun function on all\n * all vertices (masters and mirrors) with the synrhonized gather accumulators.\n *\n * @param thread_id the thread to run this as which determines\n * @param vset the vertex set specifying the set of vertex to run the gather operation. \n */\n void execute_applys(const size_t thread_id, const vertex_set& vset);\n\n\n // Data Synchronization ===================================================\n /**\n * \\brief Send the gather value for the vertex id to its master.\n *\n * @param [in] lvid the vertex to send the gather value to\n * @param [in] accum the locally computed gather value.\n */\n void sync_gather(lvid_type lvid, const gather_type& accum, size_t thread_id);\n\n /**\n * \\brief Receive the gather values from the buffered exchange.\n *\n * This function returns when there is nothing left in the\n * buffered exchange and should be called after the buffered\n * exchange has been flushed\n */\n void recv_gathers(const bool try_to_recv = false);\n\n\n /**\n * \\brief Send the gather values from master to mirrors.\n */\n void scatter_gather(lvid_type lvid, const gather_type& accum, size_t thread_id);\n }; // end of class synchronous engine\n\n\n /**\n * Constructs an synchronous distributed engine.\n * The number of threads to create are read from\n * opts::get_ncpus().\n *\n * Valid engine options (graphlab_options::get_engine_args()):\n * \\arg \\c max_iterations Sets the maximum number of iterations the\n * engine will run for.\n * \\arg \\c use_cache If set to true, partial gathers are cached.\n * See \\ref gather_caching to understand the behavior of the\n * gather caching model and how it may be used to accelerate program\n * performance.\n *\n * \\param dc Distributed controller to associate with\n * \\param graph The graph to schedule over. The graph must be fully\n * constructed and finalized.\n * \\param opts A graphlab_options object containing options and parameters\n * for the engine.\n */\n template<typename Graph, typename GatherType>\n graph_gather_apply<Graph,GatherType>::graph_gather_apply(\n graph_type& graph,\n gather_fun_type gather_fun,\n apply_fun_type apply_fun,\n const graphlab_options& opts) :\n gather_fun(gather_fun), apply_fun(apply_fun), rmi(graph.dc(), this), graph(graph),\n threads(opts.get_ncpus()),\n thread_barrier(opts.get_ncpus()),\n gather_exchange(graph.dc(), opts.get_ncpus()) { } \n\n\n template<typename Graph, typename GatherType> \n void graph_gather_apply<Graph,GatherType>::exec(const vertex_set& vset) {\n if (vset.lazy && !vset.is_complete_set)\n return;\n\n gather_accum.clear();\n // Allocate vertex locks and vertex programs\n vlocks.resize(graph.num_local_vertices());\n // Allocate gather accumulators and accumulator bitset\n gather_accum.resize(graph.num_local_vertices(), gather_type());\n has_gather_accum.resize(graph.num_local_vertices());\n has_gather_accum.clear();\n rmi.barrier();\n\n // Execute gather operations-------------------------------------------\n // Execute the gather operation for all vertices that are active\n // in this minor-step (active-minorstep bit set).\n // if (rmi.procid() == 0) std::cout << \"Gathering...\" << std::endl;\n run_synchronous(&graph_gather_apply::execute_gathers, vset);\n\n // Execute the gather operation for all vertices that are active\n // in this minor-step (active-minorstep bit set).\n // if (rmi.procid() == 0) std::cout << \"Gathering...\" << std::endl;\n run_synchronous(&graph_gather_apply::execute_scatters, vset);\n\n\n // Execute Apply Operations -------------------------------------------\n // Run the apply function on all active vertices\n // if (rmi.procid() == 0) std::cout << \"Applying...\" << std::endl;\n run_synchronous(&graph_gather_apply::execute_applys, vset);\n\n /**\n * Post conditions:\n * 1) any changes to the vertex data have been synchronized\n * with all mirrors.\n * 2) all gather accumulators have been cleared\n */\n\n // Final barrier to ensure that all engines terminate at the same time\n rmi.full_barrier();\n } // end of start\n\n\n template<typename Graph, typename GatherType>\n void graph_gather_apply<Graph, GatherType>::\n execute_gathers(const size_t thread_id, const vertex_set& vset) {\n const bool TRY_TO_RECV = true;\n const size_t TRY_RECV_MOD = 1000;\n size_t vcount = 0;\n timer ti;\n\n fixed_dense_bitset<8 * sizeof(size_t)> local_bitset;\n while (1) {\n // increment by a word at a time\n lvid_type lvid_block_start =\n shared_lvid_counter.inc_ret_last(8 * sizeof(size_t));\n if (lvid_block_start >= graph.num_local_vertices()) break;\n\n local_bitset.clear();\n\n if (vset.lazy) {\n ASSERT_TRUE(vset.is_complete_set);\n local_bitset.fill();\n } else {\n // get the bit field from has_message\n size_t lvid_bit_block = vset.localvset.containing_word(lvid_block_start);\n if (lvid_bit_block == 0) continue;\n // initialize a word sized bitfield\n local_bitset.initialize_from_mem(&lvid_bit_block, sizeof(size_t));\n }\n\n foreach(size_t lvid_block_offset, local_bitset) {\n lvid_type lvid = lvid_block_start + lvid_block_offset;\n if (lvid >= graph.num_local_vertices()) break;\n\n // std::cout << \"proc \" << rmi.procid() << \" gather on lvid \" << lvid << std::endl;\n gather_type accum = gather_fun(lvid, graph);\n\n // If the accum contains a value for the local gather we put\n // that estimate in the gather exchange.\n sync_gather(lvid, accum, thread_id);\n\n // try to recv gathers if there are any in the buffer\n if(++vcount % TRY_RECV_MOD == 0) recv_gathers(TRY_TO_RECV);\n }\n } // end of loop over vertices to compute gather accumulators\n gather_exchange.partial_flush(thread_id);\n // Finish sending and receiving all gather operations\n thread_barrier.wait();\n if(thread_id == 0) gather_exchange.flush();\n thread_barrier.wait();\n recv_gathers();\n } // end of execute_gathers\n \n template<typename Graph, typename GatherType>\n void graph_gather_apply<Graph, GatherType>::\n execute_scatters(const size_t thread_id, const vertex_set& vset) {\n const bool TRY_TO_RECV = true;\n const size_t TRY_RECV_MOD = 1000;\n size_t vcount = 0;\n timer ti;\n\n fixed_dense_bitset<8 * sizeof(size_t)> local_bitset;\n while (1) {\n // increment by a word at a time\n lvid_type lvid_block_start =\n shared_lvid_counter.inc_ret_last(8 * sizeof(size_t));\n if (lvid_block_start >= graph.num_local_vertices()) break;\n\n local_bitset.clear();\n\n if (vset.lazy) {\n ASSERT_TRUE(vset.is_complete_set);\n local_bitset.fill();\n } else {\n // get the bit field from has_message\n size_t lvid_bit_block = vset.localvset.containing_word(lvid_block_start);\n if (lvid_bit_block == 0) continue;\n // initialize a word sized bitfield\n local_bitset.initialize_from_mem(&lvid_bit_block, sizeof(size_t));\n }\n\n foreach(size_t lvid_block_offset, local_bitset) {\n lvid_type lvid = lvid_block_start + lvid_block_offset;\n if (lvid >= graph.num_local_vertices()) break;\n\n if (graph.l_is_master(lvid)) {\n const gather_type& accum = gather_accum[lvid];\n apply_fun(lvid, accum, graph);\n scatter_gather(lvid, accum, thread_id);\n // try to recv gathers if there are any in the buffer\n if(++vcount % TRY_RECV_MOD == 0) recv_gathers(TRY_TO_RECV);\n }\n }\n } // end of loop over vertices to compute gather accumulators\n gather_exchange.partial_flush(thread_id);\n // Finish sending and receiving all gather operations\n thread_barrier.wait();\n if(thread_id == 0) gather_exchange.flush();\n thread_barrier.wait();\n recv_gathers();\n } // end of execute_gathers\n\n\n\n template<typename Graph, typename GatherType>\n void graph_gather_apply<Graph,GatherType>::\n execute_applys(const size_t thread_id, const vertex_set& vset) {\n fixed_dense_bitset<8 * sizeof(size_t)> local_bitset;\n while (1) {\n // increment by a word at a time\n lvid_type lvid_block_start = shared_lvid_counter.inc_ret_last(8 * sizeof(size_t));\n if (lvid_block_start >= graph.num_local_vertices()) break;\n\n if (vset.lazy) {\n ASSERT_TRUE(vset.is_complete_set);\n local_bitset.fill();\n } else {\n // get the bit field from has_message\n size_t lvid_bit_block = vset.localvset.containing_word(lvid_block_start);\n if (lvid_bit_block == 0) continue;\n // initialize a word sized bitfield\n local_bitset.initialize_from_mem(&lvid_bit_block, sizeof(size_t));\n }\n\n foreach(size_t lvid_block_offset, local_bitset) {\n lvid_type lvid = lvid_block_start + lvid_block_offset;\n if (lvid >= graph.num_local_vertices()) break;\n\n if (graph.l_is_master(lvid))\n continue;\n\n // Master and mirror vertices both should perform the apply step. \n // vertex_type vertex(graph.l_vertex(lvid));\n\n // Get the local accumulator. Note that it is possible that\n // the gather_accum was not set during the gather.\n const gather_type& accum = gather_accum[lvid];\n\n apply_fun(lvid, accum, graph);\n }\n } // end of loop over vertices to run apply\n } // end of execute_applys\n\n\n // Data Synchronization ===================================================\n template<typename Graph, typename GatherType>\n void graph_gather_apply<Graph,GatherType>::\n sync_gather(lvid_type lvid, const gather_type& accum, const size_t thread_id) {\n if(graph.l_is_master(lvid)) {\n vlocks[lvid].lock();\n if(has_gather_accum.get(lvid)) {\n gather_accum[lvid] += accum;\n } else {\n gather_accum[lvid] = accum;\n has_gather_accum.set_bit(lvid);\n }\n vlocks[lvid].unlock();\n } else {\n const procid_t master = graph.l_master(lvid);\n const vertex_id_type vid = graph.global_vid(lvid);\n gather_exchange.send(master, std::make_pair(vid, accum), thread_id);\n }\n } // end of sync_gather\n\n template<typename Graph, typename GatherType>\n void graph_gather_apply<Graph, GatherType>::\n scatter_gather(lvid_type lvid, const gather_type& accum, const size_t thread_id) {\n ASSERT_TRUE(graph.l_is_master(lvid));\n const vertex_id_type vid = graph.global_vid(lvid);\n local_vertex_type vertex = graph.l_vertex(lvid);\n foreach(const procid_t& mirror, vertex.mirrors()) {\n gather_exchange.send(mirror, std::make_pair(vid, accum), thread_id);\n }\n } // end of sync_gather\n\n template<typename Graph, typename GatherType>\n void graph_gather_apply<Graph,GatherType>::\n recv_gathers(const bool try_to_recv) {\n procid_t procid(-1);\n typename gather_exchange_type::buffer_type buffer;\n while(gather_exchange.recv(procid, buffer, try_to_recv)) {\n foreach(const vid_gather_pair_type& pair, buffer) {\n ASSERT_TRUE(graph.vid2lvid.find(pair.first) != graph.vid2lvid.end());\n const lvid_type lvid = graph.local_vid(pair.first);\n const gather_type& accum = pair.second;\n vlocks[lvid].lock();\n if( has_gather_accum.get(lvid) ) {\n gather_accum[lvid] += accum;\n } else {\n gather_accum[lvid] = accum;\n has_gather_accum.set_bit(lvid);\n }\n vlocks[lvid].unlock();\n }\n }\n } // end of recv_gather\n}; // namespace\n#include <graphlab/macros_undef.hpp>\n#endif\n\n\n// Remove the capability of merging flying gather accumulators.\n\n\n /**\n * \\brief Temporary map storing the gather accumulators that is not preallocated in the \n * \\ref graphlab::graph_gather_apply::gather_accum. The key of the map is the global vertex id.\n *\n * This map can be accessed by multiple threads at once and therefore must be guarded by a \n * lock \\ref graphlab::graph_gather_apply::tmp_gather_map_lock.\n */\n // boost::unordered_map<vertex_id_type, gather_type> tmp_gather_map;\n\n\n /**\n * \\brief Lock that protects access to the temporaroy gather accumulator map \\ref graphlab::graph_gather_apply::tmp_gather_map. \n */\n // mutex tmp_gather_map_lock;\n\n // /**\n // * \\brief Merge the gather accumulators in \\ref graphlab::graph_gather_apply::tmp_gather_accum \n // * in to \\ref graphlab::graph_gather_apply::gahter_accum. Resize the local graph and the associated \n // * data structures (\\ref graphlab::distributed_graph::vid2lvid, \n // * \\ref graphlab::distributed_graph::lvid2record)with the new vertex.\n // */\n // void merge_temporary_gather_map();\n\n // template<typename Graph, typename GatherType>\n // void graph_gather_apply<Graph, GatherType>::\n // merge_temporary_gather_map() {\n // // merge in the tmp_gather_map\n // typename boost::unordered_map<vertex_id_type, gather_type>::const_iterator \n // it = tmp_gather_map.begin();\n\n // // resize the graph\n // size_t new_size = graph.num_local_vertices() + tmp_gather_map.size();\n // gather_accum.resize(new_size);\n // graph.lvid2record.resize(new_size);\n // graph.local_graph.resize(new_size);\n // for (; it != tmp_gather_map.end(); ++it) {\n // lvid_type lvid = graph.vid2lvid.size();\n\n // // update graph\n // graph.vid2lvid[it->first] = lvid;\n // graph.lvid2record[lvid].gvid = it->first;\n // graph.lvid2record[lvid].owner = rmi.procid();\n\n // // update gather accum vector\n // gather_accum[lvid] = it->second;\n // }\n // tmp_gather_map.clear();\n // }\n\n\n // // Merge in the temporaroy gather map-------------------------------------------\n // // with the side effect of resizing the graph and associate vertex datastructures.\n // merge_temporary_gather_map();\n\n\n // if the vid does not exist on the proc, put in a temporary map and merge in later\n // if (graph.vid2lvid.find(pair.first) == graph.vid2lvid.end()) {\n // tmp_gather_map_lock.lock();\n // if (tmp_gather_map.find(pair.first) == tmp_gather_map.end()) {\n // tmp_gather_map[pair.first] = pair.second;\n // } else {\n // tmp_gather_map[pair.first] += pair.second;\n // }\n // tmp_gather_map_lock.unlock();\n // } else {\n\n" }, { "path": "src/graphlab/graph/graph_hash.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n#ifndef GRAPHLAB_GRAPH_HASH_UTIL\n#define GRAPHLAB_GRAPH_HASH_UTIL\n\n#include <boost/function.hpp>\n#include <boost/functional/hash.hpp>\n#include <graphlab/util/integer_mix.hpp>\n\nnamespace graphlab {\n namespace graph_hash {\n /** \\brief Returns the hashed value of a vertex. */\n inline static size_t hash_vertex (const vertex_id_type vid) { \n return integer_mix(vid);\n }\n\n /** \\brief Returns the hashed value of an edge. */\n inline static size_t hash_edge (const std::pair<vertex_id_type, vertex_id_type>& e, const uint32_t seed = 5) {\n // a bunch of random numbers\n#if (__SIZEOF_PTRDIFF_T__ == 8)\n static const size_t a[8] = {0x6306AA9DFC13C8E7,\n 0xA8CD7FBCA2A9FFD4,\n 0x40D341EB597ECDDC,\n 0x99CFA1168AF8DA7E,\n 0x7C55BCC3AF531D42,\n 0x1BC49DB0842A21DD,\n 0x2181F03B1DEE299F,\n 0xD524D92CBFEC63E9};\n#else\n static const size_t a[8] = {0xFC13C8E7,\n 0xA2A9FFD4,\n 0x597ECDDC,\n 0x8AF8DA7E,\n 0xAF531D42,\n 0x842A21DD,\n 0x1DEE299F,\n 0xBFEC63E9};\n#endif\n vertex_id_type src = e.first;\n vertex_id_type dst = e.second;\n return (integer_mix(src^a[seed%8]))^(integer_mix(dst^a[(seed+1)%8]));\n }\n } // end of graph_hash namespace\n} // end of graphlab namespace\n#endif\n" }, { "path": "src/graphlab/graph/graph_includes.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_GRAPH_INCLUDES\n#define GRAPHLAB_GRAPH_INCLUDES\n\n#include <graphlab/graph/distributed_graph.hpp>\n#include <graphlab/graph/vertex_set.hpp>\n#endif\n\n\n\n\n" }, { "path": "src/graphlab/graph/graph_ops.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n/**\n * \\file graph_ops.hpp\n *\n * This file supports basic graph io operations to simplify reading\n * and writing adjacency structures from files.\n *\n */\n\n#ifndef GRAPHLAB_GRAPH_OPS_HPP\n#define GRAPHLAB_GRAPH_OPS_HPP\n\n\n\n#include <iostream>\n#include <fstream>\n#include <string>\n\n#include <boost/algorithm/string/predicate.hpp>\n#include <graphlab/graph/distributed_graph.hpp>\n\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n \n\n namespace graph_ops {\n \n \n /**\n * builds a topological_sort of the graph returning it in topsort. \n * \n * \\param[out] topsort Resultant topological sort of the graph vertices.\n *\n * function will return false if graph is not acyclic.\n */\n template <typename VertexType, typename EdgeType>\n bool topological_sort(const distributed_graph<VertexType, EdgeType>& graph, \n std::vector<vertex_id_type>& topsort) {\n typedef distributed_graph<VertexType, EdgeType> graph_type;\n topsort.clear();\n topsort.reserve(graph.num_vertices());\n std::vector<size_t> indeg;\n indeg.resize(graph.num_vertices());\n std::queue<vertex_id_type> q;\n for (size_t i = 0;i < graph.num_vertices(); ++i) {\n indeg[i] = graph.get_in_edges(i).size();\n if (indeg[i] == 0) {\n q.push(i);\n }\n }\n \n while (!q.empty()) {\n vertex_id_type v = q.front();\n q.pop();\n topsort.push_back(v);\n foreach(typename graph_type::edge_type edge, graph.get_out_edges(v)) {\n vertex_id_type destv = edge.target();\n --indeg[destv];\n if (indeg[destv] == 0) {\n q.push(destv);\n }\n }\n }\n if (q.empty() && topsort.size() != graph.num_vertices()) {\n return false;\n }\n return true;\n } // end of topological sort\n\n\n template <typename VertexType, typename EdgeType>\n size_t num_neighbors(const distributed_graph<VertexType, EdgeType>& graph, \n vertex_id_type& vid) {\n typedef distributed_graph<VertexType, EdgeType> graph_type;\n typename graph_type::edge_list_type in_edges = graph.in_edges(vid); \n typename graph_type::edge_list_type out_edges = graph.out_edges(vid);\n typename graph_type::edge_list_type::const_iterator i = in_edges.begin();\n typename graph_type::edge_list_type::const_iterator j = out_edges.begin();\n size_t count = 0; \n for( ; i != in_edges.end() && j != out_edges.end(); ++count) {\n if(i->source() == j->target()) { \n ++i; ++j; \n } else if(i->source() < j->target()) { \n ++i; \n } else { \n ++j; \n }\n }\n for( ; i != in_edges.end(); ++i, ++count);\n for( ; j != out_edges.end(); ++j, ++count);\n return count;\n } // end of num_neighbors\n\n\n\n template <typename VertexType, typename EdgeType>\n void neighbors(const distributed_graph<VertexType, EdgeType>& graph, \n const vertex_id_type vid, \n std::vector<vertex_id_type>& neighbors ) {\n typedef distributed_graph<VertexType, EdgeType> graph_type;\n typename graph_type::edge_list_type in_edges = graph.in_edges(vid); \n typename graph_type::edge_list_type out_edges = graph.out_edges(vid);\n typename graph_type::edge_list_type::const_iterator i = in_edges.begin();\n typename graph_type::edge_list_type::const_iterator j = out_edges.begin();\n while(i != in_edges.end() && j != out_edges.end()) {\n if(i->source() == j->target()) { \n neighbors.push_back(i->source()); \n ++i; ++j; \n } else if(i->source() < j->target()) {\n neighbors.push_back(i->source()); \n ++i; \n } else { \n neighbors.push_back(j->target()); \n ++j; \n } \n }\n for( ; i != in_edges.end(); ++i) neighbors.push_back(i->source());\n for( ; j != out_edges.end(); ++j) neighbors.push_back(j->target());\n } // end of neighbors\n\n\n\n\n \n\n\n \n template <typename VertexType, typename EdgeType>\n bool save_metis_structure(const std::string& filename,\n const distributed_graph<VertexType, EdgeType>& graph) { \n typedef distributed_graph<VertexType, EdgeType> graph_type;\n typedef typename graph_type::edge_type edge_type;\n typedef typename graph_type::edge_list_type edge_list_type;\n \n std::ofstream fout(filename.c_str());\n if(!fout.good()) return false;\n // Count the number of actual edges\n size_t nedges = 0;\n for(vertex_id_type i = 0; i < graph.num_vertices(); ++i)\n nedges += num_neighbors(graph, i);\n fout << graph.num_vertices() << ' ' << (nedges/2) << '\\n';\n // Save the adjacency structure\n std::vector<vertex_id_type> neighbor_set;\n for(vertex_id_type i = 0; i < graph.num_vertices(); ++i) {\n neighbors(graph, i, neighbor_set);\n for(size_t j = 0; j < neighbor_set.size(); ++j) {\n fout << (neighbor_set[j] + 1);\n if(j + 1 < neighbor_set.size()) fout << ' ';\n }\n fout << '\\n';\n }\n fout.close();\n return true;\n } // end of save metis\n\n\n\n\n\n template <typename VertexType, typename EdgeType>\n bool save_edge_list_structure(const std::string& filename,\n const distributed_graph<VertexType, EdgeType>& graph) { \n typedef distributed_graph<VertexType, EdgeType> graph_type;\n typedef typename graph_type::edge_type edge_type;\n typedef typename graph_type::edge_list_type edge_list_type;\n\n std::ofstream fout(filename.c_str());\n if(!fout.good()) return false;\n for(vertex_id_type i = 0; i < graph.num_vertices(); ++i) \n foreach(edge_type edge, graph.out_edges(i)) \n fout << edge.source() << '\\t' << edge.target() << '\\n'; \n fout.close();\n return true;\n } // end of save metis\n\n\n\n\n template <typename VertexType, typename EdgeType>\n bool save_zoltan_hypergraph_structure(const std::string& filename,\n const distributed_graph<VertexType, EdgeType>& graph) { \n typedef distributed_graph<VertexType, EdgeType> graph_type;\n typedef typename graph_type::edge_type edge_type;\n typedef typename graph_type::edge_list_type edge_list_type;\n\n std::ofstream fout(filename.c_str());\n if(!fout.good()) return false;\n\n // ok. I need to uniquely number each edge.\n // how?\n boost::unordered_map<std::pair<vertex_id_type, \n vertex_id_type>, size_t> edgetoid;\n size_t curid = 0;\n for(vertex_id_type i = 0; i < graph.num_vertices(); ++i) {\n foreach(const typename graph_type::edge_type& edge, graph.in_edges(i)) {\n std::pair<vertex_id_type, vertex_id_type> e = \n std::make_pair(edge.source(), edge.target());\n if (e.first > e.second) std::swap(e.first, e.second);\n if (edgetoid.find(e) == edgetoid.end()) {\n edgetoid[e] = curid;\n ++curid;\n }\n }\n foreach(const typename graph_type::edge_type& edge, graph.out_edges(i)) {\n std::pair<vertex_id_type, vertex_id_type> e = \n std::make_pair(edge.source(), edge.target());\n if (e.first > e.second) std::swap(e.first, e.second);\n if (edgetoid.find(e) == edgetoid.end()) {\n edgetoid[e] = curid;\n ++curid;\n }\n }\n }\n\n size_t numedges = curid;\n // each edge is a vertex, each vertex is an edge\n // a pin is total adjacency of a hyper edge\n fout << numedges << \"\\n\\n\";\n for (size_t i = 0;i < numedges; ++i) {\n fout << i+1 << \"\\n\";\n }\n fout << \"\\n\";\n fout << graph.num_vertices() << \"\\n\\n\";\n \n fout << numedges * 2 << \"\\n\\n\";\n // loop over the \"hyperedge\" and write out the edges it is adjacent to\n for(vertex_id_type i = 0; i < graph.num_vertices(); ++i) {\n boost::unordered_set<size_t> adjedges;\n foreach(const typename graph_type::edge_type& edge, graph.in_edges(i)) {\n std::pair<vertex_id_type, vertex_id_type> e = \n std::make_pair(edge.source(), edge.target());\n if (e.first > e.second) std::swap(e.first, e.second);\n adjedges.insert(edgetoid[e]);\n }\n foreach(const typename graph_type::edge_type& edge, graph.out_edges(i)) {\n std::pair<vertex_id_type, vertex_id_type> e = \n std::make_pair(edge.source(), edge.target());\n if (e.first > e.second) std::swap(e.first, e.second);\n adjedges.insert(edgetoid[e]);\n }\n // write\n std::vector<size_t> adjedgesvec;\n std::copy(adjedges.begin(), adjedges.end(), \n std::inserter(adjedgesvec, adjedgesvec.end()));\n fout << i+1 << \" \" << adjedgesvec.size() << \"\\t\"; \n for (size_t j = 0;j < adjedgesvec.size(); ++j) {\n fout << adjedgesvec[j] + 1;\n if (j < adjedgesvec.size() - 1) fout << \"\\t\";\n }\n fout << \"\\n\";\n }\n fout.close();\n return true;\n } // end of save_zoltan_hypergraph_structure\n\n\n\n }; // end of graph ops\n}; // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n#endif\n\n\n\n\n\n" }, { "path": "src/graphlab/graph/graph_storage_deprecated.hpp", "content": "/** \n * Copyright (c) 2011 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n/* *\n * Author: Haijie Gu (haijieg@cs.cmu.edu)\n * Date: 04/08/2013\n *\n * CSR+CSC implementation of a graph storage.\n * */\n\n#ifndef GRAPHLAB_GRAPH_STORAGE_HPP\n#define GRAPHLAB_GRAPH_STORAGE_HPP\n#ifndef __NO_OPENMP__\n#include <omp.h>\n#endif\n\n#include <cmath>\n#include <string>\n#include <list>\n#include <vector>\n#include <set>\n#include <map>\n\n#include <queue>\n#include <algorithm>\n#include <functional>\n\n#include <boost/version.hpp>\n#include <boost/bind.hpp>\n#include <boost/unordered_set.hpp>\n#include <boost/iterator/zip_iterator.hpp>\n#include <boost/iterator/counting_iterator.hpp>\n#include <boost/iterator/transform_iterator.hpp>\n#include <boost/iterator/iterator_facade.hpp>\n\n#include <graphlab/graph/local_edge_buffer.hpp>\n#include <graphlab/graph/graph_basic_types.hpp>\n\n#include <graphlab/logger/logger.hpp>\n#include <graphlab/logger/assertions.hpp>\n\n#include <graphlab/serialization/iarchive.hpp>\n#include <graphlab/serialization/oarchive.hpp>\n\n#include <graphlab/util/random.hpp>\n#include <graphlab/util/generics/shuffle.hpp>\n#include <graphlab/util/generics/counting_sort.hpp>\n#include <graphlab/util/generics/vector_zip.hpp>\n#include <graphlab/util/generics/csr_storage.hpp>\n\n#include <graphlab/parallel/atomic.hpp>\n\n#include <graphlab/macros_def.hpp>\n\nnamespace graphlab {\n \n template<typename VertexData, typename EdgeData>\n class graph_storage {\n public:\n typedef graphlab::lvid_type lvid_type;\n typedef graphlab::edge_id_type edge_id_type;\n\n /** The type of the edge data stored in the graph. */\n typedef EdgeData edge_data_type;\n\n /** The type of the vertex data stored in the graph. */\n typedef VertexData vertex_data_type;\n\n /** \n * \\internal\n * CSR/CSC storage types\n */\n typedef csr_storage<lvid_type, edge_id_type> csr_type;\n\n typedef csr_storage<std::pair<lvid_type, edge_id_type>, edge_id_type> csc_type; \n\n /* ----------------------------------------------------------------------------- */\n /* helper data field and structures: edge_data_list, class edge, class edge_list */\n /* ----------------------------------------------------------------------------- */\n class edge_type {\n public:\n edge_type () : _source(-1), _target(-1), _eid(-1) {}\n edge_type(lvid_type _source, lvid_type _target, edge_id_type _eid)\n : _source(_source), _target(_target), _eid(_eid) { }\n\n inline bool is_empty() { return _eid == -1; } \n lvid_type source() const { return _source; }\n lvid_type target() const { return _target; }\n edge_id_type id() const { return _eid; }\n\n private:\n lvid_type _source;\n lvid_type _target; \n edge_id_type _eid;\n };\n\n typedef boost::tuple<typename csr_type::iterator,\n boost::counting_iterator<edge_id_type> > csr_iterator_tuple;\n\n typedef boost::zip_iterator<csr_iterator_tuple> csr_zip_iterator;\n\n csr_zip_iterator make_csr_zip_iterator(typename csr_type::iterator iter,\n edge_id_type beginid) {\n return csr_zip_iterator(\n csr_iterator_tuple(iter, boost::counting_iterator<edge_id_type>(beginid)));\n }\n \n struct make_edge_type_csr_functor {\n typedef typename csr_zip_iterator::value_type const& argument_type;\n typedef edge_type result_type;\n\n make_edge_type_csr_functor() : sourceid(-1) { } \n\n make_edge_type_csr_functor(lvid_type sourceid) : sourceid(sourceid) { }\n\n result_type operator() (argument_type arg) const {\n lvid_type destid = arg.template get<0>();\n edge_id_type eid = arg.template get<1>();\n return edge_type(sourceid, destid, eid);\n }\n lvid_type sourceid;\n };\n\n struct make_edge_type_csc_functor {\n typedef typename csc_type::value_type& argument_type;\n typedef edge_type result_type;\n make_edge_type_csc_functor() : destid(-1) {}\n\n make_edge_type_csc_functor(lvid_type destid) : destid(destid) { }\n\n result_type operator() (argument_type arg) const {\n return edge_type(arg.first, destid, arg.second);\n }\n lvid_type destid;\n };\n\n typedef boost::transform_iterator<make_edge_type_csr_functor,\n csr_zip_iterator> csr_edge_iterator;\n typedef boost::transform_iterator<make_edge_type_csc_functor,\n typename csc_type::iterator> csc_edge_iterator;\n\n class edge_iterator : \n public boost::iterator_facade <\n edge_iterator,\n edge_type,\n boost::random_access_traversal_tag,\n edge_type> {\n public:\n edge_iterator() : _type(UNDEFINED) {}\n edge_iterator(csc_edge_iterator iter) : _type(CSC), csc_iter(iter) {}\n edge_iterator(csr_edge_iterator iter) : _type(CSR), csr_iter(iter) {}\n\n private:\n friend class boost::iterator_core_access;\n\n void increment() {\n switch (_type) {\n case CSC: ++csc_iter; break;\n case CSR: ++csr_iter; break;\n default: return;\n }\n }\n bool equal(const edge_iterator& other) const\n {\n ASSERT_EQ(_type, other._type);\n switch (_type) {\n case CSC: return csc_iter == other.csc_iter;\n case CSR: return csr_iter == other.csr_iter;\n default: return true;\n }\n }\n edge_type dereference() const { \n switch (_type) {\n case CSC: return *csc_iter;\n case CSR: return *csr_iter;\n default: return edge_type();\n }\n }\n void decrement() {\n switch (_type) {\n case CSC: --csc_iter; break;\n case CSR: --csr_iter; break;\n default: return;\n }\n }\n void advance(int n) {\n switch (_type) {\n case CSC: csc_iter+=n; break;\n case CSR: csr_iter+=n; break;\n default: return;\n }\n } \n int distance_to(const edge_iterator& other) const {\n switch (_type) {\n case CSC: return other.csc_iter - csc_iter;\n case CSR: return other.csr_iter - csr_iter;\n default: return 0;\n }\n }\n private:\n enum list_type {CSR, CSC, UNDEFINED}; \n list_type _type;\n csc_edge_iterator csc_iter;\n csr_edge_iterator csr_iter;\n }; // end of edge_iterator\n\n class edge_list {\n public:\n edge_list(edge_iterator _begin, edge_iterator _end) :\n _begin(_begin), _end(_end) {}\n\n typedef edge_iterator iterator;\n typedef iterator const_iterator;\n\n inline size_t size() const { \n return _end - _begin;\n }\n\n inline edge_type operator[](size_t i) const {\n return *(_begin+i);\n }\n\n bool is_empty() const { return size() == 0; }\n\n iterator begin() const { return _begin; }\n\n iterator end() const { return _end; }\n\n private:\n edge_iterator _begin;\n edge_iterator _end;\n };\n\n public:\n // CONSTRUCTORS ============================================================>\n graph_storage() { }\n\n // METHODS =================================================================>\n /** \\brief Returns the number of edges in the graph. */\n size_t num_edges() const { return _edata_storage.size(); }\n\n /** \\brief Returns the number of vertices in the graph. */\n size_t num_vertices() const { return _csr_storage.num_keys(); }\n\n /** \\brief Returns the number of in edges of the vertex. */\n size_t num_in_edges (const lvid_type v) const {\n return (_csc_storage.end(v) - _csc_storage.begin(v));\n }\n\n /** \\brief Returns the number of out edges of the vertex. */\n size_t num_out_edges (const lvid_type v) const {\n return (_csr_storage.end(v) - _csr_storage.begin(v));\n }\n\n /** \\brief Returns a list of in edges of a vertex. */\n edge_list in_edges(const lvid_type v) {\n make_edge_type_csc_functor functor(v);\n // make_edge_type_csc_functor functor;\n csc_edge_iterator begin =\n boost::make_transform_iterator (_csc_storage.begin(v), functor);\n csc_edge_iterator end =\n boost::make_transform_iterator (_csc_storage.end(v), functor);\n return edge_list ( edge_iterator(begin), edge_iterator(end));\n }\n\n /** \\brief Returns a list of out edges of a vertex. */\n edge_list out_edges(const lvid_type v) {\n make_edge_type_csr_functor functor(v);\n csr_zip_iterator beginiter = \n make_csr_zip_iterator(_csr_storage.begin(v), \n _csr_storage.begin(v)-_csr_storage.begin(0));\n csr_zip_iterator enditer = \n make_csr_zip_iterator(_csr_storage.end(v), \n _csr_storage.end(v)-_csr_storage.begin(0));\n\n csr_edge_iterator begin = boost::make_transform_iterator (\n beginiter, functor);\n csr_edge_iterator end = boost::make_transform_iterator (\n enditer, functor);\n return edge_list ( edge_iterator(begin), edge_iterator(end));\n }\n\n /** \\brief Returns edge data of edge_type e*/\n EdgeData& edge_data(edge_id_type eid) {\n ASSERT_LT(eid, num_edges());\n return _edata_storage[eid]; \n }\n\n const EdgeData& edge_data(edge_id_type eid) const {\n ASSERT_LT(eid, num_edges());\n return _edata_storage[eid]; \n }\n\n /** \\brief Finalize the graph storage. \n * Construct the CSC, CSR, by sorting edges to maximize the\n * efficiency of graphlab. \n * This function takes O(|V|log(degree)) time and will \n * fail if there are any duplicate edges.\n */\n void finalize(local_edge_buffer<VertexData, EdgeData> &edges) {\n#ifdef DEBUG_GRAPH\n logstream(LOG_DEBUG) << \"Graph2 finalize starts.\" << std::endl;\n#endif\n std::vector<edge_id_type> permute;\n std::vector<edge_id_type> src_counting_prefix_sum;\n std::vector<edge_id_type> dest_counting_prefix_sum;\n \n#ifdef DEBUG_GRAPH\n logstream(LOG_DEBUG) << \"Graph2 finalize: Sort by source vertex\" << std::endl;\n#endif\n // Sort edges by source;\n // Begin of counting sort.\n counting_sort(edges.source_arr, permute, &src_counting_prefix_sum);\n\n // Inplace permute of edge_data, edge_src, edge_target array.\n#ifdef DEBUG_GRAPH\n logstream(LOG_DEBUG) << \"Graph2 finalize: Inplace permute by source id\" << std::endl;\n#endif\n lvid_type swap_src; lvid_type swap_target; EdgeData swap_data;\n for (size_t i = 0; i < permute.size(); ++i) {\n if (i != permute[i]) {\n // Reserve the ith entry;\n size_t j = i;\n swap_data = edges.data[i];\n swap_src = edges.source_arr[i];\n swap_target = edges.target_arr[i];\n // Begin swap cycle:\n while (j != permute[j]) {\n size_t next = permute[j];\n if (next != i) {\n edges.data[j] = edges.data[next];\n edges.source_arr[j] = edges.source_arr[next];\n edges.target_arr[j] = edges.target_arr[next];\n permute[j] = j;\n j = next;\n } else {\n // end of cycle\n edges.data[j] = swap_data;\n edges.source_arr[j] = swap_src;\n edges.target_arr[j] = swap_target;\n permute[j] = j;\n break;\n }\n }\n }\n }\n // Unfortunately the zip iterator does not work as expected. Need more debugging...\n // typedef \n // boost::tuple< std::vector<lvid_type>::iterator,\n // std::vector<lvid_type>::iterator,\n // typename std::vector<EdgeData>::iterator> iterator_tuple;\n\n // typedef boost::zip_iterator<iterator_tuple> zip_iterator;\n\n // inplace_shuffle( zip_iterator(iterator_tuple(\n // edges.source_arr.begin(),\n // edges.target_arr.begin(),\n // edges.data.begin())),\n // zip_iterator(iterator_tuple(\n // edges.source_arr.end(),\n // edges.target_arr.end(),\n // edges.data.end())),\n // permute);\n#ifdef DEBUG_GRAPH\n logstream(LOG_DEBUG) << \"Graph2 finalize: Sort by dest id\" << std::endl;\n#endif\n counting_sort(edges.target_arr, permute, &dest_counting_prefix_sum); \n // Shuffle source array\n#ifdef DEBUG_GRAPH\n logstream(LOG_DEBUG) << \"Graph2 finalize: Outofplace permute by dest id\" << std::endl;\n#endif\n\n outofplace_shuffle(edges.source_arr, permute);\n // Use inplace shuffle to reduce peak memory footprint:\n // inplace_shuffle(edges.source_arr, permute);\n // counting_sort(edges.target_arr, permute);\n\n // warp into csr csc storage.\n _csr_storage.wrap(src_counting_prefix_sum, edges.target_arr);\n std::vector<std::pair<lvid_type, edge_id_type> > csc_value = vector_zip(edges.source_arr, permute);\n _csc_storage.wrap(dest_counting_prefix_sum, csc_value); \n _edata_storage.swap(edges.data);\n ASSERT_EQ(_csr_storage.num_values(), _csc_storage.num_values());\n ASSERT_EQ(_csr_storage.num_values(), _edata_storage.size());\n#ifdef DEBGU_GRAPH\n logstream(LOG_DEBUG) << \"End of finalize.\" << std::endl;\n#endif\n } // end of finalize.\n\n\n /** \\brief Reset the storage. */\n void clear() {\n _csr_storage.clear();\n _csc_storage.clear();\n }\n\n size_t estimate_sizeof() const {\n return _csr_storage.estimate_sizeof() + _csc_storage.estimate_sizeof();\n } // end of estimate_sizeof\n\n\n // ------------- Private data storage ----------------\n private:\n csr_type _csr_storage;\n csc_type _csc_storage;\n std::vector<EdgeData> _edata_storage;\n\n public:\n /** \\brief Load the graph from an archive */\n void load(iarchive& arc) {\n clear();\n arc >> _csr_storage\n >> _csc_storage\n >> _edata_storage;\n }\n\n /** \\brief Save the graph to an archive */\n void save(oarchive& arc) const {\n arc << _csr_storage\n << _csc_storage\n << _edata_storage;\n }\n\n /** swap two graph storage*/\n void swap(graph_storage& other) {\n _csr_storage.swap(other._csr_storage);\n _csc_storage.swap(other._csc_storage);\n _edata_storage.swap(other._edata_storage);\n }\n };// End of graph store;\n}// End of namespace;\n\nnamespace std {\n template<typename VertexData, typename EdgeData>\n inline void swap(graphlab::graph_storage<VertexData,EdgeData>& a, \n graphlab::graph_storage<VertexData,EdgeData>& b) {\n a.swap(b);\n } // end of swap\n}; // end of std namespace\n#include <graphlab/macros_undef.hpp>\n#endif\n" }, { "path": "src/graphlab/graph/graph_vertex_join.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_GRAPH_JOIN_HPP\n#define GRAPHLAB_GRAPH_JOIN_HPP\n#include <utility>\n#include <boost/unordered_map.hpp>\n#include <graphlab/util/hopscotch_map.hpp>\n#include <graphlab/graph/distributed_graph.hpp>\n#include <graphlab/rpc/dc_dist_object.hpp>\nnamespace graphlab {\n\n\n/**\n * \\brief Provides the ability to pass information between vertices of two \n * different graphs\n *\n * \\tparam LeftGraph Type of the left graph \n * \\tparam RightGraph Type of the right graph \n\n * The graph_vertex_join class allows information to be passed between\n * vertices of two different graphs. \n * \n * Given two graphs <code>g1</code> and <code>g2</code>, possibly of different \n * types:\n * \n * \\code\n * typedef distributed_graph<VData1, EData1> graph_1_type;\n * typedef distributed_graph<VData2, EData2> graph_2_type;\n * graph_1_type g1;\n * graph_2_type g2;\n * \\endcode\n *\n * A graph_vertex_join object can be created:\n * \\code\n * graph_vertex_type<graph_1_type, graph_2_type> vjoin(dc, g1, g2);\n * \\endcode\n *\n * The first argument is the distributed control object. The second argument\n * shall be referred to as the graph on the \"left\" side of the join, and the \n * third argument shall be referred to as the graph on the \"right\" side of the\n * join.\n *\n * The join operates by having each vertex in both graph emit an integer key.\n * Vertices with the same key are then combined into the same group. The \n * semantics of the key depends on the join operation to be performed.\n * Right now, the only join operation supported is the Left Injective Join\n * and the Right Injective Join (see below).\n *\n*\n *\n * ## Left Injective Join\n * For the left injective join, vertices in the same graph must emit distinct\n * unique keys. i.e. Each vertex in <code>g1</code> must emit a key which is \n * different from all other vertices in <code>g1</code>. Vertices on the right\n * graph are then matched with vertices on the left graph with the same key.\n * The join operation is then allowed to modify vertices on the left graph \n * given the data on the vertices of the right graph.\n * \n * To emit the keys:\n * \\code\n * vjoin.prepare_injective_join(left_emit_key, right_emit_key);\n * \\endcode\n * left_emit_key and right_emit_key are functions (or lambda) with the following \n * prototype:\n * \\code\n * size_t left_emit_key(const graph_1_type::vertex_type& vertex);\n * size_t right_emit_key(const graph_2_type::vertex_type& vertex);\n * \\endcode\n * They essentially take as a constant argument, the vertex of their respective\n * graphs, and return an integer key. If the key has value (-1) it does not \n * participate in the join.\n * After keys are emitted and prepared with prepare_join, to perform a left\n * injective join:\n * \\code\n * vjoin.left_injective_join(join_op);\n * \\endcode\n * Where join_op is a function with the following prototype:\n * \\code\n * void join_op(graph_1_type::vertex_type& left_vertex, \n * const graph_2_type::vertex_data_type right_vertex_data);\n * \\endcode\n * Note the asymmetry in the arguments: the left vertex is passed as a \n * vertex_type, while for the right vertex, only the vertex data is accessible.\n * The function may make modifications on the left vertex.\n * \n * The left_injective_join() function must be called by all machines.\n * As a result, it may be used from within an engine's \n * \\ref graphlab::iengine::add_vertex_aggregator aggregator's finalize\n * function.\n *\n * ### Left Injective Join Example\n * I have two graphs with identical structure. The left graph has data\n * \\code\n * struct left_vertex_data {\n * size_t user_id;\n * std::string user_name;\n * std::string email_address;\n * // ... serializers omitted ...\n * }\n * typedef distributed_graph<left_vertex_data, some_edge_data> left_graph_type;\n * \\endcode\n * However, when the left graph was constructed, there was no email address\n * information conveniently available, and thus was left blank.\n *\n * And the right graph has vertex data: \n * \\code\n * struct right_vertex_data {\n * size_t user_id;\n * std::string email_address;\n * // ... serializers omitted ...\n * }\n * typedef distributed_graph<right_vertex_data, some_edge_data> right_graph_type;\n * \\endcode\n * which was loaded from some other source, and contains all the email address\n * information.\n *\n * I create emit functions for both graphs :\n * \\code\n * size_t emit_user_id_field_left(const left_graph_type::vertex_type& vtype) {\n * return vtype.data().user_id;\n * }\n * size_t emit_user_id_field_left(const right_graph_type::vertex_type& vtype) {\n * return vtype.data().user_id;\n * }\n * \\endcode\n *\n * Create a join object and prepare the join:\n * \\code\n * graph_vertex_join<left_graph_type, right_graph_type> vjoin(dc, \n * left_graph, \n * right_graph);\n * vjoin.prepare_injective_join(emit_user_id_field_left, \n * emit_user_id_field_right);\n * \\endcode\n *\n * To copy the email address field from the right graph to the left graph:\n * \\code\n * void join_email_address(left_graph_type::vertex_type& left_vertex,\n * const right_vertex_data& rvtx) {\n * left_vertex.data().email_address = rvtx.email_address; \n * }\n * \n * vjoin.left_injective_join(join_email_address);\n * \\endcode\n * \n * ## Right Injective Join\n * The right injective join is similar to the left injective join, but\n * with types reversed.\n */\ntemplate <typename LeftGraph, typename RightGraph> \nclass graph_vertex_join {\n public:\n /// Type of the left graph\n typedef LeftGraph left_graph_type;\n /// Type of the right graph\n typedef RightGraph right_graph_type;\n /// Vertex Type of the left graph\n typedef typename right_graph_type::vertex_type left_vertex_type;\n /// Vertex Type of the right graph\n typedef typename left_graph_type::vertex_type right_vertex_type;\n /// Local Vertex Type of the left graph\n typedef typename right_graph_type::local_vertex_type left_local_vertex_type;\n /// Local Vertex Type of the right graph\n typedef typename left_graph_type::local_vertex_type right_local_vertex_type;\n /// Vertex Data Type of the left graph\n typedef typename right_graph_type::vertex_data_type left_data_type;\n /// Vertex Data Type of the right graph\n typedef typename left_graph_type::vertex_data_type right_data_type;\n\n dc_dist_object<graph_vertex_join<LeftGraph, RightGraph> > rmi;\n\n private:\n /// Reference to the left graph\n left_graph_type& left_graph;\n /// Reference to the right graph\n right_graph_type& right_graph;\n \n struct injective_join_index {\n std::vector<size_t> vtx_to_key;\n hopscotch_map<size_t, vertex_id_type> key_to_vtx;\n // we use -1 here to indicate that the vertex is not participating\n std::vector<procid_t> opposing_join_proc;\n };\n\n injective_join_index left_inj_index, right_inj_index;\n\n public:\n graph_vertex_join(distributed_control& dc,\n left_graph_type& left,\n right_graph_type& right): \n rmi(dc, this), left_graph(left), right_graph(right) { }\n\n\n /**\n * \\brief Initializes the join by associating each vertex with a key\n *\n * \\tparam LeftEmitKey Type of the left_emit_key parameter. It should\n * not be necessary to specify this. C++ type inference should be able\n * to infer this automatically.\n * \\tparam RightEmitKey Type of the right_emit_key parameter. It should\n * not be necessary to specify this. C++ type inference should be able\n * to infer this automatically.\n *\n * \\param left_emit_key A function which takes a vertex_type from the\n * left graph and emits an integral key value. \n * Can be a lambda, of the prototype:\n * size_t left_emit_key(const LeftGraph::vertex_type& vertex);\n * \\param right_emit_key A function which takes a vertex_type from the\n * right graph and emits an integral key value. \n * Can be a lambda, of the prototype:\n * size_t right_emit_key(const RightGraph::vertex_type& vertex);\n *\n * The semantics of the key depend on the actual join operation performed.\n * This function must be called by all machines.\n *\n * left_emit_key and right_emit_key are functions (or lambda) with the \n * following prototype:\n * \\code\n * size_t left_emit_key(const graph_1_type::vertex_type& vertex);\n * size_t right_emit_key(const graph_2_type::vertex_type& vertex);\n * \\endcode\n * They essentially take as a constant argument, the vertex of their \n * respective graphs, and return an integer key. If a vertex emits the key\n * (size_t)(-1) it does not participate in the join.\n *\n * prepare_injective_join() only needs to be called once. After which an \n * arbitrary number of left_injective_join() and right_injective_join() \n * calls may be made.\n *\n * If after a join, a new join is to be performed on the same graph using\n * new data, or new emit functions, prepare_injective_join() can be called\n * again to recompute the join. \n */\n template <typename LeftEmitKey, typename RightEmitKey>\n void prepare_injective_join(LeftEmitKey left_emit_key, \n RightEmitKey right_emit_key) {\n typedef std::pair<size_t, vertex_id_type> key_vertex_pair;\n // Basically, what we are trying to do is to figure out, for each vertex\n // on one side of the graph, which vertices for the other graph\n // (and on on which machines) emitted the same key.\n //\n // The target datastructure is:\n // vtx_to_key[vtx]: The key for each vertex\n // opposing_join_proc[vtx]: Machines which hold a vertex on the opposing\n // graph which emitted the same key\n // key_to_vtx[key] Mapping of keys to vertices. \n \n // resize the left index\n // resize the right index\n\n reset_and_fill_injective_index(left_inj_index, \n left_graph, \n left_emit_key, \"left graph\"); \n\n reset_and_fill_injective_index(right_inj_index, \n right_graph, \n right_emit_key, \"right graph\"); \n rmi.barrier(); \n // now, we need cross join across all machines to figure out the \n // opposing join proc\n // we need to do this twice. Once for left, and once for right. \n compute_injective_join();\n }\n\n /**\n * \\brief Performs an injective join from the right graph to the left graph.\n * \n * \\tparam JoinOp The type of the joinop function. It should\n * not be necessary to specify this. C++ type inference should be able\n * to infer this automatically.\n *\n * \\param join_op The joining function. May be a function pointer or a \n * lambda matching the prototype\n * void join_op(LeftGraph::vertex_type& left_vertex, \n * const RightGraph::vertex_data_type right_vertex_data);\n * \n * prepare_injective_join() must be called before hand.\n * All machines must call this function. join_op will be called on each\n * left vertex with the data on a right vertex which emitted the same key\n * in prepare_injective_join(). The join_op function is allowed to modify\n * the vertex data on the left graph.\n */\n template <typename JoinOp>\n void left_injective_join(JoinOp join_op) {\n injective_join(left_inj_index, left_graph,\n right_inj_index, right_graph, \n join_op);\n }\n\n\n /**\n * \\brief Performs an injective join from the left graph to the right graph.\n * \n * \\tparam JoinOp The type of the joinop function. It should\n * not be necessary to specify this. C++ type inference should be able\n * to infer this automatically.\n *\n * \\param join_op The joining function. May be a function pointer or a \n * lambda matching the prototype\n * void join_op(RightGraph::vertex_type& right_vertex, \n * const LeftGraph::vertex_data_type left_vertex_data);\n * \n * prepare_injective_join() must be called before hand.\n * All machines must call this function. join_op will be called on each\n * rght vertex with the data on a left vertex which emitted the same key\n * in prepare_injective_join(). The join_op function is allowed to modify\n * the vertex data on the right graph.\n */\n template <typename JoinOp>\n void right_injective_join(JoinOp join_op) {\n injective_join(right_inj_index, right_graph,\n left_inj_index, left_graph, \n join_op);\n }\n\n private:\n template <typename Graph, typename EmitKey>\n void reset_and_fill_injective_index(injective_join_index& idx,\n Graph& graph,\n EmitKey& emit_key,\n const char* message) {\n // clear the data\n idx.vtx_to_key.resize(graph.num_local_vertices());\n idx.key_to_vtx.clear(); \n idx.opposing_join_proc.resize(graph.num_local_vertices(), (procid_t)(-1));\n // loop through vertices, get the key and fill vtx_to_key and key_to_vtx\n for(lvid_type v = 0; v < graph.num_local_vertices(); ++v) {\n typename Graph::local_vertex_type lv = graph.l_vertex(v);\n if (lv.owned()) {\n typename Graph::vertex_type vtx(lv);\n size_t key = emit_key(vtx);\n idx.vtx_to_key[v] = key;\n if (key != (size_t)(-1)) {\n if (idx.key_to_vtx.count(key) > 0) {\n logstream(LOG_ERROR) << \"Duplicate key in \" << message << std::endl;\n logstream(LOG_ERROR) << \"Duplicate keys not permitted\" << std::endl;\n throw \"Duplicate Key in Join\";\n }\n idx.key_to_vtx.insert(std::make_pair(key, v));\n }\n }\n }\n }\n\n void compute_injective_join() {\n std::vector<std::vector<size_t> > left_keys = \n get_procs_with_keys(left_inj_index.vtx_to_key, left_graph);\n std::vector<std::vector<size_t> > right_keys = \n get_procs_with_keys(right_inj_index.vtx_to_key, right_graph);\n // now. for each key on the right, I need to figure out which proc it\n // belongs in. and vice versa. This is actually kind of annoying.\n // but since it is one-to-one, I only need to make a hash map of one side.\n hopscotch_map<size_t, procid_t> left_key_to_procs;\n\n // construct a hash table of keys to procs\n // clear frequently to use less memory\n for (size_t p = 0; p < left_keys.size(); ++p) {\n for (size_t i = 0; i < left_keys[p].size(); ++i) {\n ASSERT_MSG(left_key_to_procs.count(left_keys[p][i]) == 0,\n \"Duplicate keys not permitted for left graph keys in injective join\");\n left_key_to_procs.insert(std::make_pair(left_keys[p][i], p));\n }\n std::vector<size_t>().swap(left_keys[p]);\n }\n left_keys.clear();\n \n std::vector<\n std::vector<\n std::pair<size_t, procid_t> > > left_match(rmi.numprocs());\n std::vector<\n std::vector<\n std::pair<size_t, procid_t> > > right_match(rmi.numprocs());\n\n // now for each key on the right, find the matching key on the left\n for (size_t p = 0; p < right_keys.size(); ++p) {\n for (size_t i = 0; i < right_keys[p].size(); ++i) {\n size_t key = right_keys[p][i];\n hopscotch_map<size_t, procid_t>::iterator iter =\n left_key_to_procs.find(key);\n if (iter != left_key_to_procs.end()) {\n ASSERT_MSG(iter->second != (procid_t)(-1),\n \"Duplicate keys not permitted for right graph keys in injective join\");\n // we have a match\n procid_t left_proc = iter->second;\n procid_t right_proc = p;\n // now. left has to be told about right and right\n // has to be told about left\n left_match[left_proc].push_back(std::make_pair(key, right_proc));\n right_match[right_proc].push_back(std::make_pair(key, left_proc));\n // set the map entry to -1 \n // so we know if it is ever reused\n iter->second = (procid_t)(-1); \n }\n }\n std::vector<size_t>().swap(right_keys[p]);\n }\n right_keys.clear();\n\n rmi.all_to_all(left_match);\n rmi.all_to_all(right_match);\n // fill in the index\n // go through the left match and set up the opposing index to based\n // on the match result\n#ifdef _OPENMP\n#pragma omp parallel for\n#endif\n for (size_t p = 0;p < left_match.size(); ++p) {\n for (size_t i = 0;i < left_match[p].size(); ++i) {\n // search for the key in the left index\n hopscotch_map<size_t, vertex_id_type>::const_iterator iter = \n left_inj_index.key_to_vtx.find(left_match[p][i].first);\n ASSERT_TRUE(iter != left_inj_index.key_to_vtx.end());\n // fill in the match\n left_inj_index.opposing_join_proc[iter->second] = left_match[p][i].second;\n }\n }\n left_match.clear();\n // repeat for the right match\n#ifdef _OPENMP\n#pragma omp parallel for\n#endif\n for (size_t p = 0;p < right_match.size(); ++p) {\n for (size_t i = 0;i < right_match[p].size(); ++i) {\n // search for the key in the right index\n hopscotch_map<size_t, vertex_id_type>::const_iterator iter = \n right_inj_index.key_to_vtx.find(right_match[p][i].first);\n ASSERT_TRUE(iter != right_inj_index.key_to_vtx.end());\n // fill in the match\n right_inj_index.opposing_join_proc[iter->second] = right_match[p][i].second;\n }\n }\n right_match.clear();\n // ok done.\n }\n\n // each key is assigned to a controlling machine, who receives\n // the partial list of keys every other machine owns.\n template <typename Graph>\n std::vector<std::vector<size_t> > \n get_procs_with_keys(const std::vector<size_t>& local_key_list, Graph& g) {\n // this machine will get all keys from each processor where\n // key = procid mod numprocs\n std::vector<std::vector<size_t> > procs_with_keys(rmi.numprocs());\n for (size_t i = 0; i < local_key_list.size(); ++i) {\n if (g.l_vertex(i).owned() && local_key_list[i] != (size_t)(-1)) {\n procid_t target_procid = local_key_list[i] % rmi.numprocs();\n procs_with_keys[target_procid].push_back(local_key_list[i]);\n }\n }\n rmi.all_to_all(procs_with_keys);\n return procs_with_keys;\n }\n\n template <typename TargetGraph, typename SourceGraph, typename JoinOp>\n void injective_join(injective_join_index& target,\n TargetGraph& target_graph,\n injective_join_index& source,\n SourceGraph& source_graph,\n JoinOp joinop) {\n // build up the exchange structure.\n // move right vertex data to left\n std::vector<\n std::vector<\n std::pair<size_t, typename SourceGraph::vertex_data_type> > > \n source_data(rmi.numprocs());\n\n for (size_t i = 0; i < source.opposing_join_proc.size(); ++i) {\n if (source_graph.l_vertex(i).owned()) {\n procid_t target_proc = source.opposing_join_proc[i];\n if (target_proc >= 0 && target_proc < rmi.numprocs()) {\n source_data[target_proc].push_back(\n std::make_pair(source.vtx_to_key[i],\n source_graph.l_vertex(i).data()));\n }\n }\n }\n // exchange\n rmi.all_to_all(source_data);\n // ok. now join against left\n#ifdef _OPENMP\n#pragma omp parallel for\n#endif\n for (size_t p = 0;p < source_data.size(); ++p) {\n for (size_t i = 0;i < source_data[p].size(); ++i) {\n // find the target vertex with the matching key\n hopscotch_map<size_t, vertex_id_type>::const_iterator iter = \n target.key_to_vtx.find(source_data[p][i].first);\n ASSERT_TRUE(iter != target.key_to_vtx.end());\n // found it!\n typename TargetGraph::local_vertex_type \n lvtx = target_graph.l_vertex(iter->second);\n typename TargetGraph::vertex_type vtx(lvtx);\n joinop(vtx, source_data[p][i].second);\n }\n }\n target_graph.synchronize();\n }\n};\n\n} // namespace graphlab\n\n#endif\n" }, { "path": "src/graphlab/graph/ingress/distributed_batch_ingress.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_DISTRIBUTED_BATCH_INGRESS_HPP\n#define GRAPHLAB_DISTRIBUTED_BATCH_INGRESS_HPP\n\n#include <boost/unordered_set.hpp>\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/graph/graph_hash.hpp>\n#include <graphlab/graph/ingress/distributed_ingress_base.hpp>\n#include <graphlab/graph/distributed_graph.hpp>\n#include <graphlab/rpc/buffered_exchange.hpp>\n#include <graphlab/rpc/distributed_event_log.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n#include <graphlab/macros_def.hpp>\n\nnamespace graphlab {\n template<typename VertexData, typename EdgeData>\n class distributed_graph;\n\n template<typename VertexData, typename EdgeData>\n class distributed_batch_ingress : \n public distributed_ingress_base<VertexData, EdgeData> {\n public:\n typedef distributed_graph<VertexData, EdgeData> graph_type;\n /// The type of the vertex data stored in the graph \n typedef VertexData vertex_data_type;\n /// The type of the edge data stored in the graph \n typedef EdgeData edge_data_type;\n\n typedef typename graph_type::vertex_record vertex_record;\n typedef typename graph_type::mirror_type mirror_type;\n\n dc_dist_object<distributed_batch_ingress> rpc;\n typedef distributed_ingress_base<VertexData, EdgeData> base_type;\n\n mutex local_graph_lock;\n mutex lvid2record_lock;\n\n typedef fixed_dense_bitset<RPC_MAX_N_PROCS> bin_counts_type;\n\n /** Type of the degree hash table: \n * a map from vertex id to a bitset of length num_procs. */\n typedef typename boost::unordered_map<vertex_id_type, bin_counts_type> \n dht_degree_table_type;\n\n /** distributed hash table stored on local machine */ \n boost::unordered_map<vertex_id_type, bin_counts_type > dht_degree_table;\n\n /** The map from vertex id to its DHT entry.\n * Must be called with a readlock acquired on dht_degree_table_lock. */\n size_t vid_to_dht_entry_with_readlock(vertex_id_type vid) {\n if (dht_degree_table.count(vid) == 0) {\n dht_degree_table_lock.unlock();\n dht_degree_table_lock.writelock();\n dht_degree_table[vid].clear(); \n dht_degree_table_lock.unlock();\n dht_degree_table_lock.readlock();\n }\n return vid;\n }\n rwlock dht_degree_table_lock;\n\n /** Local minibatch buffer */\n /** Number of edges in the current buffer. */\n size_t num_edges; \n /** Capacity of the local buffer. */\n size_t bufsize; \n std::vector<std::pair<vertex_id_type, vertex_id_type> > edgesend;\n std::vector<EdgeData> edatasend;\n mutex edgesend_lock;\n /** A set of vertex in the current batch requiring query the DHT. */\n std::vector<boost::unordered_set<vertex_id_type> > query_set;\n /** The map from proc_id to num_edges on that proc */\n std::vector<size_t> proc_num_edges;\n\n DECLARE_TRACER(batch_ingress_add_edge);\n DECLARE_TRACER(batch_ingress_add_edges);\n DECLARE_TRACER(batch_ingress_compute_assignments);\n DECLARE_TRACER(batch_ingress_request_degree_table);\n DECLARE_TRACER(batch_ingress_get_degree_table);\n DECLARE_TRACER(batch_ingress_update_degree_table);\n\n /** Ingress tratis. */\n bool usehash;\n bool userecent; \n\n public:\n distributed_batch_ingress(distributed_control& dc, graph_type& graph, \n size_t bufsize = 50000, bool usehash = false, bool userecent = false) :\n base_type(dc, graph), rpc(dc, this), \n num_edges(0), bufsize(bufsize), query_set(dc.numprocs()),\n proc_num_edges(dc.numprocs()), usehash(usehash), userecent(userecent) { \n rpc.barrier(); \n\n INITIALIZE_TRACER(batch_ingress_add_edge, \"Time spent in add edge\");\n INITIALIZE_TRACER(batch_ingress_add_edges, \"Time spent in add block edges\" );\n INITIALIZE_TRACER(batch_ingress_compute_assignments, \"Time spent in compute assignment\");\n INITIALIZE_TRACER(batch_ingress_request_degree_table, \"Time spent in requesting assignment\");\n INITIALIZE_TRACER(batch_ingress_get_degree_table, \"Time spent in retrieve degree table\");\n INITIALIZE_TRACER(batch_ingress_update_degree_table, \"Time spent in update degree table\");\n }\n\n /** Adds an edge to the batch ingress buffer, and updates the query set. */\n void add_edge(vertex_id_type source, vertex_id_type target, const EdgeData& edata) {\n BEGIN_TRACEPOINT(batch_ingress_add_edge);\n edgesend_lock.lock();\n ASSERT_LT(edgesend.size(), bufsize);\n edgesend.push_back(std::make_pair(source, target)); \n edatasend.push_back(edata); \n query_set[graph_hash::hash_vertex(source) % rpc.numprocs()].insert(source);\n query_set[graph_hash::hash_vertex(target) % rpc.numprocs()].insert(target);\n ++num_edges;\n edgesend_lock.unlock();\n END_TRACEPOINT(batch_ingress_add_edge);\n if (is_full()) flush();\n } // end of add_edge\n\n /** Flush the buffer and call base finalize. */; \n void finalize() { \n rpc.full_barrier();\n flush(); \n rpc.full_barrier();\n base_type::finalize();\n } // end of finalize\n\n\n private:\n\n // HELPER ROUTINES =======================================================> \n /** Add edges in block to the local current graph. */\n void add_edges(const std::vector<vertex_id_type>& source_arr, \n const std::vector<vertex_id_type>& target_arr, \n const std::vector<EdgeData>& edata_arr) {\n\n BEGIN_TRACEPOINT(batch_ingress_add_edges);\n ASSERT_TRUE((source_arr.size() == target_arr.size())\n && (source_arr.size() == edata_arr.size())); \n if (source_arr.size() == 0) return;\n\n std::vector<lvid_type> local_source_arr; \n local_source_arr.reserve(source_arr.size());\n std::vector<lvid_type> local_target_arr;\n local_target_arr.reserve(source_arr.size());\n\n // The map from vertex_id to its degree on this proc.\n std::vector<std::vector<vertex_id_type> > local_degree_count(rpc.numprocs());\n\n lvid_type max_lvid = 0;\n\n lvid2record_lock.lock();\n // Lock and update the lvid2record.\n for (size_t i = 0; i < source_arr.size(); ++i) {\n vertex_id_type source = source_arr[i];\n vertex_id_type target = target_arr[i]; \n lvid_type lvid_source(-1);\n lvid_type lvid_target(-1);\n // typedef typename boost::unordered_map<vertex_id_type, lvid_type>::iterator \n // vid2lvid_iter;\n typedef typename hopscotch_map<vertex_id_type, lvid_type>::iterator\n vid2lvid_iter;\n vid2lvid_iter iter;\n\n iter = base_type::graph.vid2lvid.find(source);\n if (iter == base_type::graph.vid2lvid.end()) {\n lvid_source = base_type::graph.vid2lvid.size();\n base_type::graph.vid2lvid[source]=lvid_source;\n base_type::graph.lvid2record.push_back(vertex_record(source));\n } else {\n lvid_source = iter->second;\n }\n\n iter = base_type::graph.vid2lvid.find(target);\n if (iter == base_type::graph.vid2lvid.end()) {\n lvid_target = base_type::graph.vid2lvid.size();\n base_type::graph.vid2lvid[target]=lvid_target;\n base_type::graph.lvid2record.push_back(vertex_record(target));\n } else {\n lvid_target = iter->second;\n }\n\n local_source_arr.push_back(lvid_source);\n local_target_arr.push_back(lvid_target);\n max_lvid = std::max(std::max(lvid_source, lvid_target), \n max_lvid);\n\n local_degree_count[graph_hash::hash_vertex(source) % rpc.numprocs()].push_back(source);\n local_degree_count[graph_hash::hash_vertex(target) % rpc.numprocs()].push_back(target);\n }\n lvid2record_lock.unlock();\n\n // Send out local_degree count.\n for (size_t i = 0; i < rpc.numprocs(); ++i) {\n if (i != rpc.procid()) {\n rpc.remote_call(i, \n &distributed_batch_ingress::block_add_degree_counts, \n rpc.procid(),\n local_degree_count[i]);\n } else {\n block_add_degree_counts(rpc.procid(), local_degree_count[i]);\n }\n local_degree_count[i].clear();\n }\n\n // Lock and add edges to local graph.\n local_graph_lock.lock();\n if (max_lvid >= base_type::graph.local_graph.num_vertices()) {\n //std::cout << rpc.procid() << \": \" << max_lvid << std::endl;\n base_type::graph.local_graph.resize(max_lvid + 1);\n }\n base_type::graph.local_graph.add_edges(local_source_arr, \n local_target_arr, edata_arr);\n local_graph_lock.unlock();\n \n END_TRACEPOINT(batch_ingress_add_edges);\n } // end of add edges\n\n /** Updates the local part of the distributed table. */\n void block_add_degree_counts (procid_t pid, std::vector<vertex_id_type>& whohas) {\n BEGIN_TRACEPOINT(batch_ingress_update_degree_table);\n dht_degree_table_lock.readlock();\n foreach (vertex_id_type& vid, whohas) {\n size_t idx = vid_to_dht_entry_with_readlock(vid);\n dht_degree_table[idx].set_bit_unsync(pid);\n }\n dht_degree_table_lock.unlock();\n END_TRACEPOINT(batch_ingress_update_degree_table);\n }\n\n /** Returns the degree counts by querying the distributed table. */\n dht_degree_table_type \n block_get_degree_table(const boost::unordered_set<vertex_id_type>& vid_query) {\n BEGIN_TRACEPOINT(batch_ingress_get_degree_table);\n dht_degree_table_type answer;\n dht_degree_table_lock.readlock();\n foreach (vertex_id_type qvid, vid_query) {\n answer[qvid] = dht_degree_table[vid_to_dht_entry_with_readlock(qvid)]; \n }\n dht_degree_table_lock.unlock();\n END_TRACEPOINT(batch_ingress_get_degree_table);\n return answer;\n } // end of block get degree table\n\n\n /** Assign edges in the buffer greedily using the recent query of DHT. */\n void assign_edges(std::vector<std::vector<vertex_id_type> >& proc_src,\n std::vector<std::vector<vertex_id_type> >& proc_dst,\n std::vector<std::vector<EdgeData> >& proc_edata) {\n ASSERT_EQ(num_edges, edgesend.size());\n\n edgesend_lock.lock();\n \n if (num_edges == 0) {\n edgesend_lock.unlock();\n return;\n }\n BEGIN_TRACEPOINT(batch_ingress_request_degree_table);\n std::vector<dht_degree_table_type> degree_table(rpc.numprocs());\n \n // Query the DHT.\n for (size_t i = 0; i < rpc.numprocs(); ++i) {\n if (i == rpc.procid()) {\n degree_table[i] = block_get_degree_table(query_set[i]);\n } else {\n degree_table[i] = \n rpc.remote_request(i, \n &distributed_batch_ingress::block_get_degree_table,\n query_set[i]);\n }\n query_set[i].clear();\n }\n END_TRACEPOINT(batch_ingress_request_degree_table);\n\n // Make assigment.\n for (size_t i = 0; i < num_edges; ++i) {\n std::pair<vertex_id_type, vertex_id_type>& e = \n edgesend[i];\n\n BEGIN_TRACEPOINT(batch_ingress_compute_assignments);\n size_t src_proc = graph_hash::hash_vertex(e.first) % rpc.numprocs();\n size_t dst_proc = graph_hash::hash_vertex(e.second) % rpc.numprocs();\n bin_counts_type& src_degree = degree_table[src_proc][e.first];\n bin_counts_type& dst_degree = degree_table[dst_proc][e.second];\n procid_t proc = base_type::edge_decision.edge_to_proc_greedy(e.first, e.second, \n src_degree, dst_degree, proc_num_edges, usehash, userecent);\n END_TRACEPOINT(batch_ingress_compute_assignments);\n\n ASSERT_LT(proc, proc_src.size());\n proc_src[proc].push_back(e.first);\n proc_dst[proc].push_back(e.second);\n proc_edata[proc].push_back(edatasend[i]);\n }\n\n // Clear the sending buffer.\n edgesend.clear();\n edatasend.clear();\n edgesend_lock.unlock();\n } // end assign edge\n\n /** Flushes all edges in the buffer. */\n void flush() {\n std::vector< std::vector<vertex_id_type> > proc_src(rpc.numprocs());\n std::vector< std::vector<vertex_id_type> > proc_dst(rpc.numprocs());\n std::vector< std::vector<EdgeData> > proc_edata(rpc.numprocs());\n assign_edges(proc_src, proc_dst, proc_edata);\n for (size_t i = 0; i < proc_src.size(); ++i) {\n if (proc_src[i].size() == 0) \n continue;\n if (i == rpc.procid()) {\n add_edges(proc_src[i], proc_dst[i], proc_edata[i]);\n num_edges -= proc_src[i].size();\n } else {\n rpc.remote_call(i, &distributed_batch_ingress::add_edges,\n proc_src[i], proc_dst[i], proc_edata[i]);\n num_edges -= proc_src[i].size();\n } // end if\n } // end for\n } // end flush\n\n /** Returns the number of edges in the buffer. */\n size_t size() { return num_edges; }\n\n /** Returns whether the buffer is full. */\n bool is_full() { return size() >= bufsize; }\n }; // end of distributed_batch_ingress\n\n}; // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n\n#endif\n" }, { "path": "src/graphlab/graph/ingress/distributed_constrained_batch_ingress.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_DISTRIBUTED_CONSTRAINED_BATCH_INGRESS_HPP\n#define GRAPHLAB_DISTRIBUTED_CONSTRAINED_BATCH_INGRESS_HPP\n\n#include <boost/unordered_set.hpp>\n#include <graphlab/graph/distributed_graph.hpp>\n#include <graphlab/graph/graph_hash.hpp>\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/graph/ingress/distributed_ingress_base.hpp>\n#include <graphlab/rpc/buffered_exchange.hpp>\n#include <graphlab/rpc/distributed_event_log.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n#include <graphlab/graph/ingress/sharding_constraint.hpp>\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n template<typename VertexData, typename EdgeData>\n class distributed_graph;\n\n template<typename VertexData, typename EdgeData>\n class distributed_constrained_batch_ingress : \n public distributed_ingress_base<VertexData, EdgeData> {\n public:\n typedef distributed_graph<VertexData, EdgeData> graph_type;\n /// The type of the vertex data stored in the graph \n typedef VertexData vertex_data_type;\n /// The type of the edge data stored in the graph \n typedef EdgeData edge_data_type;\n\n typedef typename graph_type::vertex_record vertex_record;\n typedef typename graph_type::mirror_type mirror_type;\n\n dc_dist_object<distributed_constrained_batch_ingress> rpc;\n typedef distributed_ingress_base<VertexData, EdgeData> base_type;\n\n mutex local_graph_lock;\n mutex lvid2record_lock;\n\n typedef fixed_dense_bitset<RPC_MAX_N_PROCS> bin_counts_type;\n\n /** Type of the degree hash table: \n * a map from vertex id to a bitset of length num_procs. */\n typedef typename boost::unordered_map<vertex_id_type, bin_counts_type> \n dht_degree_table_type;\n\n /** distributed hash table stored on local machine */ \n std::vector<bin_counts_type > dht_degree_table;\n\n /** The map from vertex id to its DHT entry.\n * Must be called with a readlock acquired on dht_degree_table_lock. */\n size_t vid_to_dht_entry_with_readlock(vertex_id_type vid) {\n size_t idx = (vid - rpc.procid()) / rpc.numprocs();\n if (dht_degree_table.size() <= idx) {\n dht_degree_table_lock.unlock();\n dht_degree_table_lock.writelock();\n \n if (dht_degree_table.size() <= idx) {\n size_t newsize = std::max(dht_degree_table.size() * 2, idx + 1);\n dht_degree_table.resize(newsize);\n }\n dht_degree_table_lock.unlock();\n dht_degree_table_lock.readlock();\n }\n return idx;\n }\n rwlock dht_degree_table_lock;\n\n /** Local minibatch buffer */\n /** Number of edges in the current buffer. */\n size_t num_edges; \n /** Capacity of the local buffer. */\n size_t bufsize; \n std::vector<std::pair<vertex_id_type, vertex_id_type> > edgesend;\n std::vector<EdgeData> edatasend;\n mutex edgesend_lock;\n /** A set of vertex in the current batch requiring query the DHT. */\n std::vector<boost::unordered_set<vertex_id_type> > query_set;\n /** The map from proc_id to num_edges on that proc */\n std::vector<size_t> proc_num_edges;\n\n /** Ingress tratis. */\n bool usehash;\n bool userecent; \n\n sharding_constraint* constraint;\n boost::hash<vertex_id_type> hashvid;\n\n public:\n distributed_constrained_batch_ingress(distributed_control& dc, graph_type& graph, \n size_t bufsize = 50000, bool usehash = false, bool userecent = false) :\n base_type(dc, graph), rpc(dc, this), \n num_edges(0), bufsize(bufsize), query_set(dc.numprocs()),\n proc_num_edges(dc.numprocs()), usehash(usehash), userecent(userecent) { \n constraint = new sharding_constraint(dc.numprocs(), \"grid\"); \n rpc.barrier(); \n }\n ~distributed_constrained_batch_ingress() { \n delete constraint;\n }\n\n /** Adds an edge to the batch ingress buffer, and updates the query set. */\n void add_edge(vertex_id_type source, vertex_id_type target, const EdgeData& edata) {\n edgesend_lock.lock();\n ASSERT_LT(edgesend.size(), bufsize);\n edgesend.push_back(std::make_pair(source, target)); \n edatasend.push_back(edata); \n query_set[graph_hash::hash_vertex(source) % rpc.numprocs()].insert(source);\n query_set[graph_hash::hash_vertex(target) % rpc.numprocs()].insert(target);\n ++num_edges;\n edgesend_lock.unlock();\n if (is_full()) flush();\n } // end of add_edge\n\n /** Flush the buffer and call base finalize. */; \n void finalize() { \n rpc.full_barrier();\n flush(); \n rpc.full_barrier();\n base_type::finalize();\n } // end of finalize\n\n\n private:\n\n // HELPER ROUTINES =======================================================> \n /** Add edges in block to the local current graph. */\n void add_edges(const std::vector<vertex_id_type>& source_arr, \n const std::vector<vertex_id_type>& target_arr, \n const std::vector<EdgeData>& edata_arr) {\n\n ASSERT_TRUE((source_arr.size() == target_arr.size())\n && (source_arr.size() == edata_arr.size())); \n if (source_arr.size() == 0) return;\n\n std::vector<lvid_type> local_source_arr; \n local_source_arr.reserve(source_arr.size());\n std::vector<lvid_type> local_target_arr;\n local_target_arr.reserve(source_arr.size());\n\n // The map from vertex_id to its degree on this proc.\n std::vector<std::vector<vertex_id_type> > local_degree_count(rpc.numprocs());\n\n lvid_type max_lvid = 0;\n\n lvid2record_lock.lock();\n // Lock and update the lvid2record.\n for (size_t i = 0; i < source_arr.size(); ++i) {\n vertex_id_type source = source_arr[i];\n vertex_id_type target = target_arr[i]; \n lvid_type lvid_source(-1);\n lvid_type lvid_target(-1);\n // typedef typename boost::unordered_map<vertex_id_type, lvid_type>::iterator \n // vid2lvid_iter;\n typedef typename cuckoo_map_pow2<vertex_id_type, lvid_type, 3, uint32_t>::iterator\n vid2lvid_iter;\n vid2lvid_iter iter;\n\n iter = base_type::graph.vid2lvid.find(source);\n if (iter == base_type::graph.vid2lvid.end()) {\n lvid_source = base_type::graph.vid2lvid.size();\n base_type::graph.vid2lvid[source]=lvid_source;\n base_type::graph.lvid2record.push_back(vertex_record(source));\n } else {\n lvid_source = iter->second;\n }\n\n iter = base_type::graph.vid2lvid.find(target);\n if (iter == base_type::graph.vid2lvid.end()) {\n lvid_target = base_type::graph.vid2lvid.size();\n base_type::graph.vid2lvid[target]=lvid_target;\n base_type::graph.lvid2record.push_back(vertex_record(target));\n } else {\n lvid_target = iter->second;\n }\n\n local_source_arr.push_back(lvid_source);\n local_target_arr.push_back(lvid_target);\n max_lvid = std::max(std::max(lvid_source, lvid_target), \n max_lvid);\n\n local_degree_count[graph_hash::hash_vertex(source) % rpc.numprocs()].push_back(source);\n local_degree_count[graph_hash::hash_vertex(target) % rpc.numprocs()].push_back(target);\n }\n lvid2record_lock.unlock();\n\n // Send out local_degree count.\n for (size_t i = 0; i < rpc.numprocs(); ++i) {\n if (i != rpc.procid()) {\n rpc.remote_call(i, \n &distributed_constrained_batch_ingress::block_add_degree_counts, \n rpc.procid(),\n local_degree_count[i]);\n } else {\n block_add_degree_counts(rpc.procid(), local_degree_count[i]);\n }\n local_degree_count[i].clear();\n }\n\n // Lock and add edges to local graph.\n local_graph_lock.lock();\n if (max_lvid >= base_type::graph.local_graph.num_vertices()) {\n //std::cout << rpc.procid() << \": \" << max_lvid << std::endl;\n base_type::graph.local_graph.resize(max_lvid + 1);\n }\n base_type::graph.local_graph.add_edges(local_source_arr, \n local_target_arr, edata_arr);\n local_graph_lock.unlock();\n \n } // end of add edges\n\n /** Updates the local part of the distributed table. */\n void block_add_degree_counts (procid_t pid, std::vector<vertex_id_type>& whohas) {\n dht_degree_table_lock.readlock();\n foreach (vertex_id_type& vid, whohas) {\n size_t idx = vid_to_dht_entry_with_readlock(vid);\n dht_degree_table[idx].set_bit_unsync(pid);\n }\n dht_degree_table_lock.unlock();\n }\n\n /** Returns the degree counts by querying the distributed table. */\n dht_degree_table_type \n block_get_degree_table(const boost::unordered_set<vertex_id_type>& vid_query) {\n dht_degree_table_type answer;\n dht_degree_table_lock.readlock();\n foreach (vertex_id_type qvid, vid_query) {\n answer[qvid] = dht_degree_table[vid_to_dht_entry_with_readlock(qvid)]; \n }\n dht_degree_table_lock.unlock();\n return answer;\n } // end of block get degree table\n\n\n /** Assign edges in the buffer greedily using the recent query of DHT. */\n void assign_edges(std::vector<std::vector<vertex_id_type> >& proc_src,\n std::vector<std::vector<vertex_id_type> >& proc_dst,\n std::vector<std::vector<EdgeData> >& proc_edata) {\n ASSERT_EQ(num_edges, edgesend.size());\n\n edgesend_lock.lock();\n \n if (num_edges == 0) {\n edgesend_lock.unlock();\n return;\n }\n std::vector<dht_degree_table_type> degree_table(rpc.numprocs());\n \n // Query the DHT.\n for (size_t i = 0; i < rpc.numprocs(); ++i) {\n if (i == rpc.procid()) {\n degree_table[i] = block_get_degree_table(query_set[i]);\n } else {\n degree_table[i] = \n rpc.remote_request(i, \n &distributed_constrained_batch_ingress::block_get_degree_table,\n query_set[i]);\n }\n query_set[i].clear();\n }\n\n // Make assigment.\n for (size_t i = 0; i < num_edges; ++i) {\n std::pair<vertex_id_type, vertex_id_type>& e = \n edgesend[i];\n\n size_t src_proc = graph_hash::hash_vertex(e.first) % rpc.numprocs();\n size_t dst_proc = graph_hash::hash_vertex(e.second) % rpc.numprocs();\n bin_counts_type& src_degree = degree_table[src_proc][e.first];\n bin_counts_type& dst_degree = degree_table[dst_proc][e.second];\n\n const std::vector<procid_t>& candidates = \n constraint->get_joint_neighbors(get_master(e.first), get_master(e.second));\n\n procid_t proc = base_type::edge_decision.edge_to_proc_greedy(e.first, e.second, \n src_degree, dst_degree, candidates, proc_num_edges, usehash, userecent);\n\n ASSERT_LT(proc, proc_src.size());\n proc_src[proc].push_back(e.first);\n proc_dst[proc].push_back(e.second);\n proc_edata[proc].push_back(edatasend[i]);\n }\n\n // Clear the sending buffer.\n edgesend.clear();\n edatasend.clear();\n edgesend_lock.unlock();\n } // end assign edge\n\n /** Flushes all edges in the buffer. */\n void flush() {\n std::vector< std::vector<vertex_id_type> > proc_src(rpc.numprocs());\n std::vector< std::vector<vertex_id_type> > proc_dst(rpc.numprocs());\n std::vector< std::vector<EdgeData> > proc_edata(rpc.numprocs());\n assign_edges(proc_src, proc_dst, proc_edata);\n for (size_t i = 0; i < proc_src.size(); ++i) {\n if (proc_src[i].size() == 0) \n continue;\n if (i == rpc.procid()) {\n add_edges(proc_src[i], proc_dst[i], proc_edata[i]);\n num_edges -= proc_src[i].size();\n } else {\n rpc.remote_call(i, &distributed_constrained_batch_ingress::add_edges,\n proc_src[i], proc_dst[i], proc_edata[i]);\n num_edges -= proc_src[i].size();\n } // end if\n } // end for\n } // end flush\n\n /** Returns the number of edges in the buffer. */\n size_t size() { return num_edges; }\n\n /** Returns whether the buffer is full. */\n bool is_full() { return size() >= bufsize; }\n\n /** Return the master shard of the vertex. */\n procid_t get_master (vertex_id_type vid) {\n return hashvid(vid) % base_type::rpc.numprocs();\n }\n\n\n }; // end of distributed_constrained_batch_ingress\n\n\n}; // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n\n#endif\n" }, { "path": "src/graphlab/graph/ingress/distributed_constrained_oblivious_ingress.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_DISTRIBUTED_CONSTRAINED_OBLIVIOUS_INGRESS_HPP\n#define GRAPHLAB_DISTRIBUTED_CONSTRAINED_OBLIVIOUS_INGRESS_HPP\n\n\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/graph/ingress/idistributed_ingress.hpp>\n#include <graphlab/graph/ingress/distributed_ingress_base.hpp>\n#include <graphlab/graph/ingress/ingress_edge_decision.hpp>\n#include <graphlab/graph/distributed_graph.hpp>\n#include <graphlab/rpc/buffered_exchange.hpp>\n#include <graphlab/rpc/distributed_event_log.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n#include <graphlab/util/cuckoo_map_pow2.hpp>\n#include <graphlab/graph/ingress/sharding_constraint.hpp>\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n template<typename VertexData, typename EdgeData>\n class distributed_graph;\n\n /**\n * \\brief Ingress object assigning edges using randoming hash function.\n */\n template<typename VertexData, typename EdgeData>\n class distributed_constrained_oblivious_ingress: \n public distributed_ingress_base<VertexData, EdgeData> {\n public:\n typedef distributed_graph<VertexData, EdgeData> graph_type;\n /// The type of the vertex data stored in the graph \n typedef VertexData vertex_data_type;\n /// The type of the edge data stored in the graph \n typedef EdgeData edge_data_type;\n \n typedef typename graph_type::vertex_record vertex_record;\n typedef typename graph_type::mirror_type mirror_type;\n\n typedef distributed_ingress_base<VertexData, EdgeData> base_type;\n // typedef typename boost::unordered_map<vertex_id_type, std::vector<size_t> > degree_hash_table_type;\n typedef fixed_dense_bitset<RPC_MAX_N_PROCS> bin_counts_type; \n\n /** Type of the degree hash table: \n * a map from vertex id to a bitset of length num_procs. */\n typedef cuckoo_map_pow2<vertex_id_type, bin_counts_type,3,uint32_t> degree_hash_table_type;\n degree_hash_table_type dht;\n\n /** Array of number of edges on each proc. */\n std::vector<size_t> proc_num_edges;\n\n /** Ingress tratis. */\n bool usehash;\n bool userecent;\n\n sharding_constraint* constraint;\n boost::hash<vertex_id_type> hashvid;\n\n public:\n distributed_constrained_oblivious_ingress(distributed_control& dc, graph_type& graph, bool usehash = false, bool userecent = false) :\n base_type(dc, graph),\n dht(-1),proc_num_edges(dc.numprocs()), usehash(usehash), userecent(userecent) { \n constraint = new sharding_constraint(dc.numprocs(), \"grid\"); \n }\n\n ~distributed_constrained_oblivious_ingress() { \n delete constraint;\n }\n\n /** Add an edge to the ingress object using oblivious greedy assignment. */\n void add_edge(vertex_id_type source, vertex_id_type target,\n const EdgeData& edata) {\n dht[source]; dht[target];\n const std::vector<procid_t>& candidates = \n constraint->get_joint_neighbors(get_master(source), get_master(target));\n const procid_t owning_proc = \n base_type::edge_decision.edge_to_proc_greedy(source, target, dht[source], dht[target], candidates, proc_num_edges, usehash, userecent);\n typedef typename base_type::edge_buffer_record edge_buffer_record;\n edge_buffer_record record(source, target, edata);\n base_type::edge_exchange.send(owning_proc, record);\n } // end of add edge\n\n virtual void finalize() {\n dht.clear();\n distributed_ingress_base<VertexData, EdgeData>::finalize(); \n }\n\n private:\n procid_t get_master (vertex_id_type vid) {\n return hashvid(vid) % base_type::rpc.numprocs();\n }\n\n\n }; // end of distributed_ob_ingress\n\n}; // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n\n#endif\n" }, { "path": "src/graphlab/graph/ingress/distributed_constrained_random_ingress.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_DISTRIBUTED_CONSTRAINED_RANDOM_INGRESS_HPP\n#define GRAPHLAB_DISTRIBUTED_CONSTRAINED_RANDOM_INGRESS_HPP\n\n#include <boost/functional/hash.hpp>\n\n#include <graphlab/rpc/buffered_exchange.hpp>\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/graph/ingress/distributed_ingress_base.hpp>\n#include <graphlab/graph/distributed_graph.hpp>\n#include <graphlab/graph/ingress/sharding_constraint.hpp>\n#include <graphlab/graph/ingress/ingress_edge_decision.hpp>\n\n\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n template<typename VertexData, typename EdgeData>\n class distributed_graph;\n\n /**\n * \\brief Ingress object assigning edges using randoming hash function.\n */\n template<typename VertexData, typename EdgeData>\n class distributed_constrained_random_ingress : \n public distributed_ingress_base<VertexData, EdgeData> {\n public:\n typedef distributed_graph<VertexData, EdgeData> graph_type;\n /// The type of the vertex data stored in the graph \n typedef VertexData vertex_data_type;\n /// The type of the edge data stored in the graph \n typedef EdgeData edge_data_type;\n\n\n typedef distributed_ingress_base<VertexData, EdgeData> base_type;\n\n sharding_constraint* constraint;\n boost::hash<vertex_id_type> hashvid;\n\n public:\n distributed_constrained_random_ingress(distributed_control& dc, graph_type& graph,\n const std::string& method) :\n base_type(dc, graph) {\n constraint = new sharding_constraint(dc.numprocs(), method);\n } // end of constructor\n\n ~distributed_constrained_random_ingress() { \n delete constraint;\n }\n\n /** Add an edge to the ingress object using random assignment. */\n void add_edge(vertex_id_type source, vertex_id_type target,\n const EdgeData& edata) {\n typedef typename base_type::edge_buffer_record edge_buffer_record;\n\n const std::vector<procid_t>& candidates = constraint->get_joint_neighbors(graph_hash::hash_vertex(source) % base_type::rpc.numprocs(),\n graph_hash::hash_vertex(target) % base_type::rpc.numprocs());\n\n const procid_t owning_proc = \n base_type::edge_decision.edge_to_proc_random(source, target, candidates);\n\n\n const edge_buffer_record record(source, target, edata);\n#ifdef _OPENMP\n base_type::edge_exchange.send(owning_proc, record, omp_get_thread_num());\n#else\n base_type::edge_exchange.send(owning_proc, record);\n#endif\n } // end of add edge\n }; // end of distributed_constrained_random_ingress\n}; // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n\n#endif\n" }, { "path": "src/graphlab/graph/ingress/distributed_hdrf_ingress.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_DISTRIBUTED_HDRF_INGRESS_HPP\n#define GRAPHLAB_DISTRIBUTED_HDRF_INGRESS_HPP\n\n\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/graph/ingress/distributed_ingress_base.hpp>\n#include <graphlab/graph/ingress/ingress_edge_decision.hpp>\n#include <graphlab/graph/distributed_graph.hpp>\n#include <graphlab/rpc/buffered_exchange.hpp>\n#include <graphlab/rpc/distributed_event_log.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n#include <graphlab/util/cuckoo_map_pow2.hpp>\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n template<typename VertexData, typename EdgeData>\n class distributed_graph;\n\n template<typename VertexData, typename EdgeData>\n class distributed_hdrf_ingress: \n public distributed_ingress_base<VertexData, EdgeData> {\n public:\n typedef distributed_graph<VertexData, EdgeData> graph_type;\n /// The type of the vertex data stored in the graph \n typedef VertexData vertex_data_type;\n /// The type of the edge data stored in the graph \n typedef EdgeData edge_data_type;\n \n typedef typename graph_type::vertex_record vertex_record;\n typedef typename graph_type::mirror_type mirror_type;\n\n typedef distributed_ingress_base<VertexData, EdgeData> base_type;\n typedef fixed_dense_bitset<RPC_MAX_N_PROCS> bin_counts_type; \n\n /** Type of the replica degree hash table: \n * a map from vertex id to a bitset of length num_procs.\n\t */\n typedef cuckoo_map_pow2<vertex_id_type, bin_counts_type,3,uint32_t> degree_hash_table_type;\n degree_hash_table_type dht;\n \n /** Type of the vertex degree hash table:\n * a map from vertex id to a bitset of length num_procs.\n\t */\n typedef cuckoo_map_pow2<vertex_id_type, size_t,3,uint32_t> true_degree_hash_table_type;\n true_degree_hash_table_type degree_dht;\n\n /** Array of number of edges on each proc. */\n std::vector<size_t> proc_num_edges;\n\n /** Ingress tratis. */\n bool usehash;\n bool userecent;\n\n public:\n distributed_hdrf_ingress(distributed_control& dc, graph_type& graph, bool usehash = false, bool userecent = false) :\n base_type(dc, graph),\n dht(-1),degree_dht(-1),proc_num_edges(dc.numprocs()), usehash(usehash), userecent(userecent) {\n\n //INITIALIZE_TRACER(ob_ingress_compute_assignments, \"Time spent in compute assignment\");\n }\n\n ~distributed_hdrf_ingress() { }\n\n /** Add an edge to the ingress object using hdrf greedy assignment. */\n void add_edge(vertex_id_type source, vertex_id_type target,\n const EdgeData& edata) {\n dht[source]; dht[target];\n degree_dht[source]; degree_dht[target];\n\n const procid_t owning_proc = \n base_type::edge_decision.edge_to_proc_hdrf(source, target, dht[source], dht[target], degree_dht[source], degree_dht[target], proc_num_edges, usehash, userecent);\n\n typedef typename base_type::edge_buffer_record edge_buffer_record;\n edge_buffer_record record(source, target, edata);\n base_type::edge_exchange.send(owning_proc, record);\n } // end of add edge\n\n virtual void finalize() {\n dht.clear();\n degree_dht.clear();\n distributed_ingress_base<VertexData, EdgeData>::finalize();\n \n size_t count = 0;\n for(std::vector<size_t>::iterator it = proc_num_edges.begin(); it != proc_num_edges.end(); ++it) {\n count = count + *it;\n }\n \n logstream(LOG_EMPH) << \"TOTAL PROCESSED ELEMENTS: \" << count << std::endl;\n \n }\n\n }; // end of distributed_ob_ingress\n\n}; // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n\n#endif\n" }, { "path": "src/graphlab/graph/ingress/distributed_identity_ingress.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_DISTRIBUTED_IDENTITY_INGRESS_HPP\n#define GRAPHLAB_DISTRIBUTED_IDENTITY_INGRESS_HPP\n\n#include <boost/functional/hash.hpp>\n\n#include <graphlab/rpc/buffered_exchange.hpp>\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/graph/ingress/distributed_ingress_base.hpp>\n#include <graphlab/graph/distributed_graph.hpp>\n\n\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n template<typename VertexData, typename EdgeData>\n class distributed_graph;\n\n /**\n * \\brief Ingress object assigning edges to the loading machine itself.\n */\n template<typename VertexData, typename EdgeData>\n class distributed_identity_ingress : \n public distributed_ingress_base<VertexData, EdgeData> {\n public:\n typedef distributed_graph<VertexData, EdgeData> graph_type;\n /// The type of the vertex data stored in the graph \n typedef VertexData vertex_data_type;\n /// The type of the edge data stored in the graph \n typedef EdgeData edge_data_type;\n\n typedef distributed_ingress_base<VertexData, EdgeData> base_type;\n\n public:\n distributed_identity_ingress(distributed_control& dc, graph_type& graph) :\n base_type(dc, graph) {\n } // end of constructor\n\n ~distributed_identity_ingress() { }\n\n /** Add an edge to the ingress object and assign the edge to itself. */\n void add_edge(vertex_id_type source, vertex_id_type target,\n const EdgeData& edata) {\n typedef typename base_type::edge_buffer_record edge_buffer_record;\n const procid_t owning_proc = base_type::rpc.procid();\n const edge_buffer_record record(source, target, edata);\n base_type::edge_exchange.send(owning_proc, record);\n } // end of add edge\n }; // end of distributed_identity_ingress\n}; // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n\n#endif\n" }, { "path": "src/graphlab/graph/ingress/distributed_ingress_base.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n * use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_DISTRIBUTED_INGRESS_BASE_HPP\n#define GRAPHLAB_DISTRIBUTED_INGRESS_BASE_HPP\n\n#include <graphlab/graph/distributed_graph.hpp>\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/graph/graph_hash.hpp>\n#include <graphlab/graph/ingress/ingress_edge_decision.hpp>\n#include <graphlab/graph/graph_gather_apply.hpp>\n#include <graphlab/util/memory_info.hpp>\n#include <graphlab/util/hopscotch_map.hpp>\n#include <graphlab/rpc/buffered_exchange.hpp>\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n\n /**\n * \\brief Implementation of the basic ingress functionality.\n */\n template <typename VertexType, typename EdgeType> \n class distributed_graph;\n\n template<typename VertexData, typename EdgeData>\n class distributed_ingress_base {\n public:\n typedef distributed_graph<VertexData, EdgeData> graph_type;\n /// The type of the vertex data stored in the graph \n typedef VertexData vertex_data_type;\n /// The type of the edge data stored in the graph \n typedef EdgeData edge_data_type;\n \n typedef typename graph_type::vertex_record vertex_record;\n typedef typename graph_type::mirror_type mirror_type;\n\n \n /// The rpc interface for this object\n dc_dist_object<distributed_ingress_base> rpc;\n /// The underlying distributed graph object that is being loaded\n graph_type& graph;\n\n /// Temporary buffers used to store vertex data on ingress\n struct vertex_buffer_record {\n vertex_id_type vid;\n vertex_data_type vdata;\n vertex_buffer_record(vertex_id_type vid = -1,\n vertex_data_type vdata = vertex_data_type()) :\n vid(vid), vdata(vdata) { }\n void load(iarchive& arc) { arc >> vid >> vdata; }\n void save(oarchive& arc) const { arc << vid << vdata; }\n }; \n buffered_exchange<vertex_buffer_record> vertex_exchange;\n\n /// Temporar buffers used to store edge data on ingress\n struct edge_buffer_record {\n vertex_id_type source, target;\n edge_data_type edata;\n edge_buffer_record(const vertex_id_type& source = vertex_id_type(-1), \n const vertex_id_type& target = vertex_id_type(-1), \n const edge_data_type& edata = edge_data_type()) :\n source(source), target(target), edata(edata) { }\n void load(iarchive& arc) { arc >> source >> target >> edata; }\n void save(oarchive& arc) const { arc << source << target << edata; }\n };\n buffered_exchange<edge_buffer_record> edge_exchange;\n\n /// Detail vertex record for the second pass coordination. \n struct vertex_negotiator_record {\n mirror_type mirrors;\n vertex_id_type num_in_edges, num_out_edges;\n bool has_data;\n vertex_data_type vdata;\n vertex_negotiator_record() : num_in_edges(0), num_out_edges(0), has_data(false) { }\n\n void load(iarchive& arc) { \n arc >> num_in_edges >> num_out_edges >> mirrors >> has_data >> vdata;\n }\n void save(oarchive& arc) const { \n arc << num_in_edges << num_out_edges << mirrors << has_data << vdata;\n }\n\n vertex_negotiator_record operator+=(const vertex_negotiator_record& v2) {\n num_in_edges += v2.num_in_edges;\n num_out_edges += v2.num_out_edges;\n mirrors |= v2.mirrors;\n if (v2.has_data) {\n vdata = v2.vdata;\n }\n return *this;\n }\n };\n\n /// Ingress decision object for computing the edge destination. \n ingress_edge_decision<VertexData, EdgeData> edge_decision;\n\n public:\n distributed_ingress_base(distributed_control& dc, graph_type& graph) :\n rpc(dc, this), graph(graph), \n#ifdef _OPENMP\n vertex_exchange(dc, omp_get_max_threads()), \n edge_exchange(dc, omp_get_max_threads()),\n#else\n vertex_exchange(dc), edge_exchange(dc),\n#endif\n edge_decision(dc) {\n rpc.barrier();\n } // end of constructor\n\n virtual ~distributed_ingress_base() { }\n\n /** \\brief Add an edge to the ingress object. */\n virtual void add_edge(vertex_id_type source, vertex_id_type target,\n const EdgeData& edata) {\n const procid_t owning_proc = \n edge_decision.edge_to_proc_random(source, target, rpc.numprocs());\n const edge_buffer_record record(source, target, edata);\n#ifdef _OPENMP\n edge_exchange.send(owning_proc, record, omp_get_thread_num());\n#else\n edge_exchange.send(owning_proc, record);\n#endif\n } // end of add edge\n\n\n /** \\brief Add an vertex to the ingress object. */\n virtual void add_vertex(vertex_id_type vid, const VertexData& vdata) { \n const procid_t owning_proc = graph_hash::hash_vertex(vid) % rpc.numprocs();\n const vertex_buffer_record record(vid, vdata);\n#ifdef _OPENMP\n vertex_exchange.send(owning_proc, record, omp_get_thread_num());\n#else\n vertex_exchange.send(owning_proc, record);\n#endif\n } // end of add vertex\n\n\n void set_duplicate_vertex_strategy(\n boost::function<void(vertex_data_type&,\n const vertex_data_type&)> combine_strategy) {\n vertex_combine_strategy = combine_strategy;\n }\n\n\n\n /** \\brief Finalize completes the local graph data structure \n * and the vertex record information. \n *\n * \\internal\n * The finalization goes through 5 steps:\n *\n * 1. Construct local graph using the received edges, during which\n * the vid2lvid map is built.\n *\n * 2. Construct lvid2record map (of empty entries) using the received vertices. \n *\n * 3. Complete lvid2record map by exchanging the vertex_info. \n *\n * 4. Exchange the negotiation records, including singletons. (Local graph \n * handling singletons). \n *\n * 5. Exchange global graph statistics.\n */\n virtual void finalize() {\n\n rpc.full_barrier();\n\n bool first_time_finalize = false;\n /**\n * Fast pass for first time finalization. \n */\n if (graph.is_dynamic()) {\n size_t nverts = graph.num_local_vertices();\n rpc.all_reduce(nverts);\n first_time_finalize = (nverts == 0);\n } else {\n first_time_finalize = false;\n }\n\n\n if (rpc.procid() == 0) {\n logstream(LOG_EMPH) << \"Finalizing Graph...\" << std::endl;\n }\n\n typedef typename hopscotch_map<vertex_id_type, lvid_type>::value_type\n vid2lvid_pair_type;\n\n typedef typename buffered_exchange<edge_buffer_record>::buffer_type \n edge_buffer_type;\n\n typedef typename buffered_exchange<vertex_buffer_record>::buffer_type \n vertex_buffer_type;\n\n /**\n * \\internal\n * Buffer storage for new vertices to the local graph.\n */\n typedef typename graph_type::hopscotch_map_type vid2lvid_map_type;\n vid2lvid_map_type vid2lvid_buffer;\n\n /**\n * \\internal\n * The begining id assinged to the first new vertex.\n */\n const lvid_type lvid_start = graph.vid2lvid.size();\n\n /**\n * \\internal\n * Bit field incidate the vertex that is updated during the ingress. \n */\n dense_bitset updated_lvids(graph.vid2lvid.size());\n\n /**************************************************************************/\n /* */\n /* Flush any additional data */\n /* */\n /**************************************************************************/\n edge_exchange.flush(); vertex_exchange.flush(); \n\n /**\n * Fast pass for redundant finalization with no graph changes. \n */\n {\n size_t changed_size = edge_exchange.size() + vertex_exchange.size();\n rpc.all_reduce(changed_size);\n if (changed_size == 0) {\n logstream(LOG_INFO) << \"Skipping Graph Finalization because no changes happened...\" << std::endl;\n return;\n }\n }\n\n if(rpc.procid() == 0) \n memory_info::log_usage(\"Post Flush\");\n\n \n /**************************************************************************/\n /* */\n /* Construct local graph */\n /* */\n /**************************************************************************/\n { // Add all the edges to the local graph\n logstream(LOG_INFO) << \"Graph Finalize: constructing local graph\" << std::endl;\n const size_t nedges = edge_exchange.size()+1;\n graph.local_graph.reserve_edge_space(nedges + 1); \n edge_buffer_type edge_buffer;\n procid_t proc;\n while(edge_exchange.recv(proc, edge_buffer)) {\n foreach(const edge_buffer_record& rec, edge_buffer) {\n // Get the source_vlid;\n lvid_type source_lvid(-1);\n if(graph.vid2lvid.find(rec.source) == graph.vid2lvid.end()) {\n if (vid2lvid_buffer.find(rec.source) == vid2lvid_buffer.end()) {\n source_lvid = lvid_start + vid2lvid_buffer.size();\n vid2lvid_buffer[rec.source] = source_lvid;\n } else {\n source_lvid = vid2lvid_buffer[rec.source];\n }\n } else {\n source_lvid = graph.vid2lvid[rec.source];\n updated_lvids.set_bit(source_lvid);\n }\n // Get the target_lvid;\n lvid_type target_lvid(-1);\n if(graph.vid2lvid.find(rec.target) == graph.vid2lvid.end()) {\n if (vid2lvid_buffer.find(rec.target) == vid2lvid_buffer.end()) {\n target_lvid = lvid_start + vid2lvid_buffer.size();\n vid2lvid_buffer[rec.target] = target_lvid;\n } else {\n target_lvid = vid2lvid_buffer[rec.target];\n }\n } else {\n target_lvid = graph.vid2lvid[rec.target];\n updated_lvids.set_bit(target_lvid);\n }\n graph.local_graph.add_edge(source_lvid, target_lvid, rec.edata);\n // std::cout << \"add edge \" << rec.source << \"\\t\" << rec.target << std::endl;\n } // end of loop over add edges\n } // end for loop over buffers\n edge_exchange.clear();\n\n ASSERT_EQ(graph.vid2lvid.size() + vid2lvid_buffer.size(), graph.local_graph.num_vertices());\n if(rpc.procid() == 0) {\n memory_info::log_usage(\"Finished populating local graph.\");\n }\n\n // Finalize local graph\n logstream(LOG_INFO) << \"Graph Finalize: finalizing local graph.\" \n << std::endl;\n graph.local_graph.finalize();\n logstream(LOG_INFO) << \"Local graph info: \" << std::endl\n << \"\\t nverts: \" << graph.local_graph.num_vertices()\n << std::endl\n << \"\\t nedges: \" << graph.local_graph.num_edges()\n << std::endl;\n \n if(rpc.procid() == 0) {\n memory_info::log_usage(\"Finished finalizing local graph.\"); \n // debug\n // std::cout << graph.local_graph << std::endl;\n }\n }\n\n /**************************************************************************/\n /* */\n /* Receive and add vertex data to masters */\n /* */\n /**************************************************************************/\n // Setup the map containing all the vertices being negotiated by this machine\n { // Receive any vertex data sent by other machines\n vertex_buffer_type vertex_buffer; procid_t sending_proc(-1);\n while(vertex_exchange.recv(sending_proc, vertex_buffer)) {\n foreach(const vertex_buffer_record& rec, vertex_buffer) {\n lvid_type lvid(-1);\n if (graph.vid2lvid.find(rec.vid) == graph.vid2lvid.end()) {\n if (vid2lvid_buffer.find(rec.vid) == vid2lvid_buffer.end()) {\n lvid = lvid_start + vid2lvid_buffer.size();\n vid2lvid_buffer[rec.vid] = lvid;\n } else {\n lvid = vid2lvid_buffer[rec.vid];\n }\n } else {\n lvid = graph.vid2lvid[rec.vid];\n updated_lvids.set_bit(lvid);\n }\n if (vertex_combine_strategy && lvid < graph.num_local_vertices()) {\n vertex_combine_strategy(graph.l_vertex(lvid).data(), rec.vdata);\n } else {\n graph.local_graph.add_vertex(lvid, rec.vdata);\n }\n }\n }\n vertex_exchange.clear();\n if(rpc.procid() == 0) \n memory_info::log_usage(\"Finished adding vertex data\");\n } // end of loop to populate vrecmap\n\n\n\n /**************************************************************************/\n /* */\n /* assign vertex data and allocate vertex (meta)data space */\n /* */\n /**************************************************************************/\n { // Determine masters for all negotiated vertices\n const size_t local_nverts = graph.vid2lvid.size() + vid2lvid_buffer.size();\n graph.lvid2record.reserve(local_nverts);\n graph.lvid2record.resize(local_nverts);\n graph.local_graph.resize(local_nverts);\n foreach(const vid2lvid_pair_type& pair, vid2lvid_buffer) {\n vertex_record& vrec = graph.lvid2record[pair.second];\n vrec.gvid = pair.first;\n vrec.owner = graph_hash::hash_vertex(pair.first) % rpc.numprocs();\n }\n ASSERT_EQ(local_nverts, graph.local_graph.num_vertices());\n ASSERT_EQ(graph.lvid2record.size(), graph.local_graph.num_vertices());\n if(rpc.procid() == 0) \n memory_info::log_usage(\"Finihsed allocating lvid2record\");\n }\n\n /**************************************************************************/\n /* */\n /* Master handshake */\n /* */\n /**************************************************************************/\n {\n#ifdef _OPENMP\n buffered_exchange<vertex_id_type> vid_buffer(rpc.dc(), omp_get_max_threads());\n#else\n buffered_exchange<vertex_id_type> vid_buffer(rpc.dc());\n#endif\n\n#ifdef _OPENMP\n#pragma omp parallel for\n#endif\n // send not owned vids to their master\n for (lvid_type i = lvid_start; i < graph.lvid2record.size(); ++i) {\n procid_t master = graph.lvid2record[i].owner;\n if (master != rpc.procid())\n#ifdef _OPENMP\n vid_buffer.send(master, graph.lvid2record[i].gvid, omp_get_thread_num());\n#else\n vid_buffer.send(master, graph.lvid2record[i].gvid);\n#endif\n }\n vid_buffer.flush();\n rpc.barrier();\n\n // receive all vids owned by me\n mutex flying_vids_lock;\n boost::unordered_map<vertex_id_type, mirror_type> flying_vids;\n#ifdef _OPENMP\n#pragma omp parallel\n#endif\n {\n typename buffered_exchange<vertex_id_type>::buffer_type buffer;\n procid_t recvid;\n while(vid_buffer.recv(recvid, buffer)) {\n foreach(const vertex_id_type vid, buffer) {\n if (graph.vid2lvid.find(vid) == graph.vid2lvid.end()) {\n if (vid2lvid_buffer.find(vid) == vid2lvid_buffer.end()) {\n flying_vids_lock.lock();\n mirror_type& mirrors = flying_vids[vid];\n flying_vids_lock.unlock();\n mirrors.set_bit(recvid);\n } else {\n lvid_type lvid = vid2lvid_buffer[vid];\n graph.lvid2record[lvid]._mirrors.set_bit(recvid);\n }\n } else {\n lvid_type lvid = graph.vid2lvid[vid];\n graph.lvid2record[lvid]._mirrors.set_bit(recvid);\n updated_lvids.set_bit(lvid);\n }\n }\n }\n }\n\n vid_buffer.clear();\n // reallocate spaces for the flying vertices. \n size_t vsize_old = graph.lvid2record.size();\n size_t vsize_new = vsize_old + flying_vids.size();\n graph.lvid2record.resize(vsize_new);\n graph.local_graph.resize(vsize_new);\n for (typename boost::unordered_map<vertex_id_type, mirror_type>::iterator it = flying_vids.begin();\n it != flying_vids.end(); ++it) {\n lvid_type lvid = lvid_start + vid2lvid_buffer.size();\n vertex_id_type gvid = it->first; \n graph.lvid2record[lvid].owner = rpc.procid();\n graph.lvid2record[lvid].gvid = gvid;\n graph.lvid2record[lvid]._mirrors= it->second;\n vid2lvid_buffer[gvid] = lvid;\n // std::cout << \"proc \" << rpc.procid() << \" recevies flying vertex \" << gvid << std::endl;\n }\n } // end of master handshake\n\n /**************************************************************************/\n /* */\n /* Merge in vid2lvid_buffer */\n /* */\n /**************************************************************************/\n {\n if (graph.vid2lvid.size() == 0) {\n graph.vid2lvid.swap(vid2lvid_buffer);\n } else {\n graph.vid2lvid.rehash(graph.vid2lvid.size() + vid2lvid_buffer.size());\n foreach (const typename vid2lvid_map_type::value_type& pair, vid2lvid_buffer) {\n graph.vid2lvid.insert(pair);\n }\n vid2lvid_buffer.clear();\n // vid2lvid_buffer.swap(vid2lvid_map_type(-1));\n }\n }\n\n\n /**************************************************************************/\n /* */\n /* synchronize vertex data and meta information */\n /* */\n /**************************************************************************/\n {\n // construct the vertex set of changed vertices\n \n // Fast pass for first time finalize;\n vertex_set changed_vset(true);\n\n // Compute the vertices that needs synchronization \n if (!first_time_finalize) {\n vertex_set changed_vset = vertex_set(false);\n changed_vset.make_explicit(graph);\n updated_lvids.resize(graph.num_local_vertices());\n for (lvid_type i = lvid_start; i < graph.num_local_vertices(); ++i) {\n updated_lvids.set_bit(i);\n }\n changed_vset.localvset = updated_lvids; \n buffered_exchange<vertex_id_type> vset_exchange(rpc.dc());\n // sync vset with all mirrors\n changed_vset.synchronize_mirrors_to_master_or(graph, vset_exchange);\n changed_vset.synchronize_master_to_mirrors(graph, vset_exchange);\n }\n\n graphlab::graph_gather_apply<graph_type, vertex_negotiator_record> \n vrecord_sync_gas(graph, \n boost::bind(&distributed_ingress_base::finalize_gather, this, _1, _2), \n boost::bind(&distributed_ingress_base::finalize_apply, this, _1, _2, _3));\n vrecord_sync_gas.exec(changed_vset);\n\n if(rpc.procid() == 0) \n memory_info::log_usage(\"Finished synchronizing vertex (meta)data\");\n }\n\n exchange_global_info();\n } // end of finalize\n\n\n /* Exchange graph statistics among all nodes and compute\n * global statistics for the distributed graph. */\n void exchange_global_info () {\n // Count the number of vertices owned locally\n graph.local_own_nverts = 0;\n foreach(const vertex_record& record, graph.lvid2record)\n if(record.owner == rpc.procid()) ++graph.local_own_nverts;\n\n // Finalize global graph statistics. \n logstream(LOG_INFO)\n << \"Graph Finalize: exchange global statistics \" << std::endl;\n\n // Compute edge counts\n std::vector<size_t> swap_counts(rpc.numprocs());\n swap_counts[rpc.procid()] = graph.num_local_edges();\n rpc.all_gather(swap_counts);\n graph.nedges = 0;\n foreach(size_t count, swap_counts) graph.nedges += count;\n\n\n // compute vertex count\n swap_counts[rpc.procid()] = graph.num_local_own_vertices();\n rpc.all_gather(swap_counts);\n graph.nverts = 0;\n foreach(size_t count, swap_counts) graph.nverts += count;\n\n // compute replicas\n swap_counts[rpc.procid()] = graph.num_local_vertices();\n rpc.all_gather(swap_counts);\n graph.nreplicas = 0;\n foreach(size_t count, swap_counts) graph.nreplicas += count;\n\n\n if (rpc.procid() == 0) {\n logstream(LOG_EMPH) << \"Graph info: \" \n << \"\\n\\t nverts: \" << graph.num_vertices()\n << \"\\n\\t nedges: \" << graph.num_edges()\n << \"\\n\\t nreplicas: \" << graph.nreplicas\n << \"\\n\\t replication factor: \" << (double)graph.nreplicas/graph.num_vertices()\n << std::endl;\n }\n }\n\n\n private:\n boost::function<void(vertex_data_type&, const vertex_data_type&)> vertex_combine_strategy;\n\n /**\n * \\brief Gather the vertex distributed meta data.\n */\n vertex_negotiator_record finalize_gather(lvid_type& lvid, graph_type& graph) {\n vertex_negotiator_record accum;\n accum.num_in_edges = graph.local_graph.num_in_edges(lvid);\n accum.num_out_edges = graph.local_graph.num_out_edges(lvid);\n if (graph.l_is_master(lvid)) {\n accum.has_data = true;\n accum.vdata = graph.l_vertex(lvid).data();\n accum.mirrors = graph.lvid2record[lvid]._mirrors;\n } \n return accum;\n }\n\n /**\n * \\brief Update the vertex datastructures with the gathered vertex metadata. \n */\n void finalize_apply(lvid_type lvid, const vertex_negotiator_record& accum, graph_type& graph) {\n typename graph_type::vertex_record& vrec = graph.lvid2record[lvid];\n vrec.num_in_edges = accum.num_in_edges;\n vrec.num_out_edges = accum.num_out_edges;\n graph.l_vertex(lvid).data() = accum.vdata;\n vrec._mirrors = accum.mirrors;\n }\n }; // end of distributed_ingress_base\n}; // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n\n#endif\n" }, { "path": "src/graphlab/graph/ingress/distributed_oblivious_ingress.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_DISTRIBUTED_OBLIVIOUS_INGRESS_HPP\n#define GRAPHLAB_DISTRIBUTED_OBLIVIOUS_INGRESS_HPP\n\n\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/graph/ingress/distributed_ingress_base.hpp>\n#include <graphlab/graph/ingress/ingress_edge_decision.hpp>\n#include <graphlab/graph/distributed_graph.hpp>\n#include <graphlab/rpc/buffered_exchange.hpp>\n#include <graphlab/rpc/distributed_event_log.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n#include <graphlab/util/cuckoo_map_pow2.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n template<typename VertexData, typename EdgeData>\n class distributed_graph;\n\n /**\n * \\brief Ingress object assigning edges using randoming hash function.\n */\n template<typename VertexData, typename EdgeData>\n class distributed_oblivious_ingress: \n public distributed_ingress_base<VertexData, EdgeData> {\n public:\n typedef distributed_graph<VertexData, EdgeData> graph_type;\n /// The type of the vertex data stored in the graph \n typedef VertexData vertex_data_type;\n /// The type of the edge data stored in the graph \n typedef EdgeData edge_data_type;\n \n typedef typename graph_type::vertex_record vertex_record;\n typedef typename graph_type::mirror_type mirror_type;\n\n typedef distributed_ingress_base<VertexData, EdgeData> base_type;\n // typedef typename boost::unordered_map<vertex_id_type, std::vector<size_t> > degree_hash_table_type;\n typedef fixed_dense_bitset<RPC_MAX_N_PROCS> bin_counts_type; \n\n /** Type of the degree hash table: \n * a map from vertex id to a bitset of length num_procs. */\n typedef cuckoo_map_pow2<vertex_id_type, bin_counts_type,3,uint32_t> degree_hash_table_type;\n degree_hash_table_type dht;\n\n /** Array of number of edges on each proc. */\n std::vector<size_t> proc_num_edges;\n simple_spinlock obliv_lock;\n \n /** Ingress traits. */\n bool usehash;\n bool userecent;\n\n public:\n distributed_oblivious_ingress(distributed_control& dc, graph_type& graph, bool usehash = false, bool userecent = false) :\n base_type(dc, graph),\n dht(-1),proc_num_edges(dc.numprocs()), usehash(usehash), userecent(userecent) { \n\n //INITIALIZE_TRACER(ob_ingress_compute_assignments, \"Time spent in compute assignment\");\n }\n\n ~distributed_oblivious_ingress() { }\n\n /** Add an edge to the ingress object using oblivious greedy assignment. */\n void add_edge(vertex_id_type source, vertex_id_type target,\n const EdgeData& edata) {\n obliv_lock.lock();\n dht[source]; dht[target];\n const procid_t owning_proc = \n base_type::edge_decision.edge_to_proc_greedy(source, target, dht[source], dht[target], proc_num_edges, usehash, userecent);\n obliv_lock.unlock();\n\n typedef typename base_type::edge_buffer_record edge_buffer_record;\n edge_buffer_record record(source, target, edata);\n#ifdef _OPENMP\n base_type::edge_exchange.send(owning_proc, record, omp_get_thread_num());\n#else \n base_type::edge_exchange.send(owning_proc, record);\n#endif\n } // end of add edge\n\n virtual void finalize() {\n dht.clear();\n distributed_ingress_base<VertexData, EdgeData>::finalize(); \n \n }\n\n }; // end of distributed_ob_ingress\n\n}; // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n\n#endif\n" }, { "path": "src/graphlab/graph/ingress/distributed_random_ingress.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_DISTRIBUTED_RANDOM_INGRESS_HPP\n#define GRAPHLAB_DISTRIBUTED_RANDOM_INGRESS_HPP\n\n#include <boost/functional/hash.hpp>\n\n#include <graphlab/rpc/buffered_exchange.hpp>\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/graph/ingress/distributed_ingress_base.hpp>\n#include <graphlab/graph/distributed_graph.hpp>\n\n\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n template<typename VertexData, typename EdgeData>\n class distributed_graph;\n\n /**\n * \\brief Ingress object assigning edges using randoming hash function.\n */\n template<typename VertexData, typename EdgeData>\n class distributed_random_ingress : \n public distributed_ingress_base<VertexData, EdgeData> {\n public:\n typedef distributed_graph<VertexData, EdgeData> graph_type;\n /// The type of the vertex data stored in the graph \n typedef VertexData vertex_data_type;\n /// The type of the edge data stored in the graph \n typedef EdgeData edge_data_type;\n\n\n typedef distributed_ingress_base<VertexData, EdgeData> base_type;\n \n public:\n distributed_random_ingress(distributed_control& dc, graph_type& graph) :\n base_type(dc, graph) {\n } // end of constructor\n\n ~distributed_random_ingress() { }\n\n /** Add an edge to the ingress object using random assignment. */\n void add_edge(vertex_id_type source, vertex_id_type target,\n const EdgeData& edata) {\n typedef typename base_type::edge_buffer_record edge_buffer_record;\n const procid_t owning_proc = base_type::edge_decision.edge_to_proc_random(source, target, base_type::rpc.numprocs());\n const edge_buffer_record record(source, target, edata);\n base_type::edge_exchange.send(owning_proc, record);\n } // end of add edge\n }; // end of distributed_random_ingress\n}; // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n\n#endif\n" }, { "path": "src/graphlab/graph/ingress/ingress_edge_decision.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n#ifndef GRAPHLAB_DISTRIBUTED_INGRESS_EDGE_DECISION_HPP\n#define GRAPHLAB_DISTRIBUTED_INGRESS_EDGE_DECISION_HPP\n\n#include <graphlab/graph/distributed_graph.hpp>\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/graph/graph_hash.hpp>\n#include <graphlab/rpc/distributed_event_log.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n#include <boost/random/uniform_int_distribution.hpp>\n\nnamespace graphlab {\n template<typename VertexData, typename EdgeData>\n class distributed_graph;\n \n template<typename VertexData, typename EdgeData>\n class ingress_edge_decision {\n\n public:\n typedef graphlab::vertex_id_type vertex_id_type;\n typedef distributed_graph<VertexData, EdgeData> graph_type;\n typedef fixed_dense_bitset<RPC_MAX_N_PROCS> bin_counts_type; \n\n public:\n /** \\brief A decision object for computing the edge assingment. */\n ingress_edge_decision(distributed_control& dc) { }\n\n /** Random assign (source, target) to a machine p in {0, ... numprocs-1} */\n procid_t edge_to_proc_random (const vertex_id_type source, \n const vertex_id_type target,\n size_t numprocs) {\n typedef std::pair<vertex_id_type, vertex_id_type> edge_pair_type;\n const edge_pair_type edge_pair(std::min(source, target), \n std::max(source, target));\n return graph_hash::hash_edge(edge_pair) % (numprocs);\n };\n\n /** Random assign (source, target) to a machine p in a list of candidates */\n procid_t edge_to_proc_random (const vertex_id_type source, \n const vertex_id_type target,\n const std::vector<procid_t> & candidates) {\n typedef std::pair<vertex_id_type, vertex_id_type> edge_pair_type;\n const edge_pair_type edge_pair(std::min(source, target), \n std::max(source, target));\n\n return candidates[graph_hash::hash_edge(edge_pair) % (candidates.size())];\n };\n\n\n /** Greedy assign (source, target) to a machine using: \n * bitset<MAX_MACHINE> src_degree : the degree presence of source over machines\n * bitset<MAX_MACHINE> dst_degree : the degree presence of target over machines\n * vector<size_t> proc_num_edges : the edge counts over machines\n * */\n procid_t edge_to_proc_greedy (const vertex_id_type source, \n const vertex_id_type target,\n bin_counts_type& src_degree,\n bin_counts_type& dst_degree,\n std::vector<size_t>& proc_num_edges,\n bool usehash = false,\n bool userecent = false) {\n size_t numprocs = proc_num_edges.size();\n\n // Compute the score of each proc.\n procid_t best_proc = -1; \n double maxscore = 0.0;\n double epsilon = 1.0; \n std::vector<double> proc_score(numprocs); \n size_t minedges = *std::min_element(proc_num_edges.begin(), proc_num_edges.end());\n size_t maxedges = *std::max_element(proc_num_edges.begin(), proc_num_edges.end());\n\n for (size_t i = 0; i < numprocs; ++i) {\n size_t sd = src_degree.get(i) + (usehash && (source % numprocs == i));\n size_t td = dst_degree.get(i) + (usehash && (target % numprocs == i));\n double bal = (maxedges - proc_num_edges[i])/(epsilon + maxedges - minedges);\n proc_score[i] = bal + ((sd > 0) + (td > 0));\n }\n maxscore = *std::max_element(proc_score.begin(), proc_score.end());\n\n std::vector<procid_t> top_procs; \n for (size_t i = 0; i < numprocs; ++i)\n if (std::fabs(proc_score[i] - maxscore) < 1e-5)\n top_procs.push_back(i);\n\n // Hash the edge to one of the best procs.\n typedef std::pair<vertex_id_type, vertex_id_type> edge_pair_type;\n const edge_pair_type edge_pair(std::min(source, target), \n std::max(source, target));\n best_proc = top_procs[graph_hash::hash_edge(edge_pair) % top_procs.size()];\n\n ASSERT_LT(best_proc, numprocs);\n if (userecent) {\n src_degree.clear();\n dst_degree.clear();\n }\n src_degree.set_bit(best_proc);\n dst_degree.set_bit(best_proc);\n ++proc_num_edges[best_proc];\n return best_proc;\n };\n\n /** Greedy assign (source, target) to a machine using: \n * bitset<MAX_MACHINE> src_degree : the degree presence of source over machines\n * bitset<MAX_MACHINE> dst_degree : the degree presence of target over machines\n * vector<size_t> proc_num_edges : the edge counts over machines\n * */\n procid_t edge_to_proc_greedy (const vertex_id_type source, \n const vertex_id_type target,\n bin_counts_type& src_degree,\n bin_counts_type& dst_degree,\n std::vector<procid_t>& candidates,\n std::vector<size_t>& proc_num_edges,\n bool usehash = false,\n bool userecent = false\n ) {\n size_t numprocs = proc_num_edges.size();\n\n // Compute the score of each proc.\n procid_t best_proc = -1; \n double maxscore = 0.0;\n double epsilon = 1.0; \n std::vector<double> proc_score(candidates.size()); \n size_t minedges = *std::min_element(proc_num_edges.begin(), proc_num_edges.end());\n size_t maxedges = *std::max_element(proc_num_edges.begin(), proc_num_edges.end());\n\n for (size_t j = 0; j < candidates.size(); ++j) {\n size_t i = candidates[j];\n size_t sd = src_degree.get(i) + (usehash && (source % numprocs == i));\n size_t td = dst_degree.get(i) + (usehash && (target % numprocs == i));\n double bal = (maxedges - proc_num_edges[i])/(epsilon + maxedges - minedges);\n proc_score[j] = bal + ((sd > 0) + (td > 0));\n }\n maxscore = *std::max_element(proc_score.begin(), proc_score.end());\n\n std::vector<procid_t> top_procs; \n for (size_t j = 0; j < candidates.size(); ++j)\n if (std::fabs(proc_score[j] - maxscore) < 1e-5)\n top_procs.push_back(candidates[j]);\n\n // Hash the edge to one of the best procs.\n typedef std::pair<vertex_id_type, vertex_id_type> edge_pair_type;\n const edge_pair_type edge_pair(std::min(source, target), \n std::max(source, target));\n best_proc = top_procs[graph_hash::hash_edge(edge_pair) % top_procs.size()];\n\n ASSERT_LT(best_proc, numprocs);\n if (userecent) {\n src_degree.clear();\n dst_degree.clear();\n }\n src_degree.set_bit(best_proc);\n dst_degree.set_bit(best_proc);\n ++proc_num_edges[best_proc];\n return best_proc;\n };\n \n /** HDRF greedy assign (source, target) to a machine using: \n * bitset<MAX_MACHINE> src_degree : the degree presence of source over machines\n * bitset<MAX_MACHINE> dst_degree : the degree presence of target over machines\n * size_t src_true_degree : the degree of source vertex over machines\n * size_t dst_true_degree : the degree of target vertex over machines\n * vector<size_t> proc_num_edges : the edge counts over machines\n *\n * author : Fabio Petroni [www.fabiopetroni.com]\n\t * Giorgio Iacoboni [g.iacoboni@gmail.com]\n *\n * Based on the publication:\t\n * F. Petroni, L. Querzoni, K. Daudjee, S. Kamali and G. Iacoboni: \n * \"HDRF: Stream-Based Partitioning for Power-Law Graphs\". \n * CIKM, 2015.\n * */\n procid_t edge_to_proc_hdrf (const vertex_id_type source, \n const vertex_id_type target,\n bin_counts_type& src_degree,\n bin_counts_type& dst_degree,\n size_t& src_true_degree,\n size_t& dst_true_degree,\n std::vector<size_t>& proc_num_edges,\n bool usehash = false,\n bool userecent = false) {\n \n size_t numprocs = proc_num_edges.size();\n \n size_t degree_u = src_true_degree;\n degree_u = degree_u +1;\n size_t degree_v = dst_true_degree;\n degree_v = degree_v +1;\n size_t SUM = degree_u + degree_v;\n double fu = degree_u;\n fu /= SUM;\n double fv = degree_v;\n fv /= SUM;\n \n // Compute the score of each proc.\n procid_t best_proc = -1; \n double maxscore = 0.0;\n double epsilon = 1.0; \n std::vector<double> proc_score(numprocs); \n size_t minedges = *std::min_element(proc_num_edges.begin(), proc_num_edges.end());\n size_t maxedges = *std::max_element(proc_num_edges.begin(), proc_num_edges.end());\n \n for (size_t i = 0; i < numprocs; ++i) {\n\t\t double new_sd = 0;\n\t\t double new_td = 0;\n\t\t size_t sd = src_degree.get(i) + (usehash && (source % numprocs == i));\n\t\t size_t td = dst_degree.get(i) + (usehash && (target % numprocs == i));\n\t\t if (sd > 0){\n\t\t new_sd = 1+(1-fu);\n\t\t }\n\t\t if (td > 0){\n\t\t new_td = 1+(1-fv);\n\t\t }\n double bal = (maxedges - proc_num_edges[i])/(epsilon + maxedges - minedges);\n\n proc_score[i] = bal + new_sd + new_td;\n }\n \n maxscore = *std::max_element(proc_score.begin(), proc_score.end());\n \n std::vector<procid_t> top_procs; \n for (size_t i = 0; i < numprocs; ++i)\n if (std::fabs(proc_score[i] - maxscore) < 1e-5)\n top_procs.push_back(i);\n \n // Hash the edge to one of the best procs.\n typedef std::pair<vertex_id_type, vertex_id_type> edge_pair_type;\n const edge_pair_type edge_pair(std::min(source, target), std::max(source, target));\n best_proc = top_procs[graph_hash::hash_edge(edge_pair) % top_procs.size()];\n \n ASSERT_LT(best_proc, numprocs);\n if (userecent) {\n src_degree.clear();\n dst_degree.clear();\n }\n src_degree.set_bit(best_proc);\n dst_degree.set_bit(best_proc);\n ++proc_num_edges[best_proc];\n ++src_true_degree;\n ++dst_true_degree;\n return best_proc;\n };\n };// end of ingress_edge_decision\n}\n\n#endif\n" }, { "path": "src/graphlab/graph/ingress/sharding_constraint.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_DISTRIBUTED_SHARDING_CONSTRAINT_HPP\n#define GRAPHLAB_DISTRIBUTED_SHARDING_CONSTRAINT_HPP\n\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/util/generate_pds.hpp>\n#include <algorithm>\n#include <vector>\n\n\n/**\n * This class defines the dependencies among the shards when using\n * a constrained partitioning algorithm.\n *\n * In constrained partitioning, vertices are assgined to a master shard\n * using hash function on the vids. Each shard S masters a partition of \n * vertices: V_s. \n *\n * Let Ai be the set of shards that Shard i depends on. Then the partitioning\n * algorithm can only put edges with either ends in V_si into Ai. For example,\n * Shard i is the master of vertex u, and Shard j is the master of vertex v,\n * then edge u->v must be placed into Ai \\intersect Aj.\n *\n * This class currently has two implementations of the shard constraints. One\n * construction is based on a grid, and the other is based on perfect difference set.\n * Both algorithms guarentees that Ai \\intersect Aj is non-empty.\n * \n * \\note: grid methods requires the number of shards to be a perfect square number. pds\n * requires the number of shards to be p^2 + p + 1 where p is a prime number.\n * \n */\nnamespace graphlab {\n class sharding_constraint {\n size_t nshards;\n std::vector<std::vector<procid_t> > constraint_graph;\n\n std::vector<std::vector<std::vector<procid_t> > > joint_nbr_cache;\n public:\n /// Test if the provided num_shards can be used for grid construction: \n // n == nrow*ncol && (abs(nrow-ncol) <= 2)\n static bool is_grid_compatible(size_t num_shards, int& nrow, int& ncol) {\n double approx_sqrt = sqrt(num_shards);\n nrow = floor(approx_sqrt);\n for (ncol = nrow; ncol <= nrow + 2; ++ncol) {\n if (ncol * nrow == (int)num_shards) {\n return true;\n }\n }\n return false;\n }\n\n static bool is_pds_compatible(size_t num_shards, int& p) {\n p = floor(sqrt(num_shards-1));\n return (p>0 && ((p*p+p+1) == (int)num_shards));\n }\n\n public:\n sharding_constraint(size_t num_shards, std::string method) {\n nshards = num_shards;\n // ignore the method input for now, only construct grid graph. \n // assuming nshards is perfect square\n if (method == \"grid\") {\n make_grid_constraint();\n } else if (method == \"pds\") {\n make_pds_constraint();\n } else {\n logstream(LOG_FATAL) << \"Unknown sharding constraint method: \" << method << std::endl;\n }\n\n joint_nbr_cache.resize(num_shards);\n for (size_t i = 0; i < num_shards; ++i) {\n joint_nbr_cache[i].resize(num_shards);\n for (size_t j = 0; j < num_shards; ++j) {\n compute_neighbors(i, j, joint_nbr_cache[i][j]);\n ASSERT_GT(joint_nbr_cache[i][j].size(), 0);\n }\n }\n }\n\n bool get_neighbors (procid_t shard, std::vector<procid_t>& neighbors) {\n ASSERT_LT(shard, nshards);\n neighbors.clear();\n std::vector<procid_t>& ls = constraint_graph[shard];\n for (size_t i = 0; i < ls.size(); ++i)\n neighbors.push_back(ls[i]);\n return true;\n }\n\n \n const std::vector<procid_t>& get_joint_neighbors (procid_t shardi, procid_t shardj) {\n return joint_nbr_cache[shardi][shardj];\n }\n\n private:\n void make_grid_constraint() {\n int ncols, nrows;\n if (!is_grid_compatible(nshards, nrows, ncols)) {\n logstream(LOG_FATAL) << \"Num shards: \" << nshards << \" cannot be used for grid ingress.\" << std::endl;\n };\n\n for (size_t i = 0; i < nshards; i++) {\n std::vector<procid_t> adjlist;\n // add self\n adjlist.push_back(i);\n\n // add the row of i\n size_t rowbegin = (i/ncols) * ncols;\n for (size_t j = rowbegin; j < rowbegin + ncols; ++j)\n if (i != j) adjlist.push_back(j); \n\n // add the col of i\n for (size_t j = i % ncols; j < nshards; j+=ncols)\n if (i != j) adjlist.push_back(j); \n\n std::sort(adjlist.begin(), adjlist.end());\n constraint_graph.push_back(adjlist);\n }\n }\n\n void make_pds_constraint() {\n int p = 0;\n if (!is_pds_compatible(nshards, p)) {\n logstream(LOG_FATAL) << \"Num shards: \" << nshards << \" cannot be used for pdsingress.\" << std::endl;\n };\n pds pds_generator;\n std::vector<size_t> results;\n if (p == 1) {\n results.push_back(0);\n results.push_back(2);\n } else {\n results = pds_generator.get_pds(p);\n }\n for (size_t i = 0; i < nshards; i++) {\n std::vector<procid_t> adjlist;\n for (size_t j = 0; j < results.size(); j++) {\n adjlist.push_back( (results[j] + i) % nshards);\n }\n std::sort(adjlist.begin(), adjlist.end());\n constraint_graph.push_back(adjlist);\n }\n }\n\n\n bool compute_neighbors(procid_t shardi, procid_t shardj, std::vector<procid_t>& neighbors) {\n ASSERT_EQ(neighbors.size(), 0);\n ASSERT_LT(shardi, nshards);\n ASSERT_LT(shardj, nshards);\n // if (shardi == shardj) {\n // neighbors.push_back(shardi);\n // return true;\n // }\n\n std::vector<procid_t>& ls1 = constraint_graph[shardi];\n std::vector<procid_t>& ls2 = constraint_graph[shardj];\n neighbors.clear();\n size_t i = 0;\n size_t j = 0;\n while (i < ls1.size() && j < ls2.size()) {\n if (ls1[i] == ls2[j]) {\n neighbors.push_back(ls1[i]);\n ++i; ++j;\n } else if (ls1[i] < ls2[j]) {\n ++i;\n } else {\n ++j;\n }\n }\n return true;\n }\n\n }; // end of sharding_constraint\n}; // end of namespace graphlab\n#endif\n" }, { "path": "src/graphlab/graph/local_edge_buffer.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_LOCAL_EDGE_BUFFER\n#define GRAPHLAB_LOCAL_EDGE_BUFFER\n\n#include <vector>\n#include <graphlab/graph/graph_basic_types.hpp>\n\nnamespace graphlab { \n\n template<typename VertexData, typename EdgeData>\n // Edge class for temporary storage. Will be finalized into the CSR+CSC form.\n class local_edge_buffer {\n public:\n std::vector<EdgeData> data;\n std::vector<lvid_type> source_arr;\n std::vector<lvid_type> target_arr;\n public:\n local_edge_buffer() {}\n void reserve_edge_space(size_t n) {\n data.reserve(n);\n source_arr.reserve(n);\n target_arr.reserve(n);\n }\n // \\brief Add an edge to the temporary storage.\n void add_edge(lvid_type source, lvid_type target, EdgeData _data) {\n data.push_back(_data);\n source_arr.push_back(source);\n target_arr.push_back(target);\n }\n // \\brief Add edges in block to the temporary storage.\n void add_block_edges(const std::vector<lvid_type>& src_arr, \n const std::vector<lvid_type>& dst_arr, \n const std::vector<EdgeData>& edata_arr) {\n data.insert(data.end(), edata_arr.begin(), edata_arr.end());\n source_arr.insert(source_arr.end(), src_arr.begin(), src_arr.end());\n target_arr.insert(target_arr.end(), dst_arr.begin(), dst_arr.end());\n }\n // \\brief Remove all contents in the storage. \n void clear() {\n std::vector<EdgeData>().swap(data);\n std::vector<lvid_type>().swap(source_arr);\n std::vector<lvid_type>().swap(target_arr);\n }\n // \\brief Return the size of the storage.\n size_t size() const {\n return source_arr.size();\n }\n // \\brief Return the estimated memory footprint used.\n size_t estimate_sizeof() const {\n return data.capacity()*sizeof(EdgeData) + \n source_arr.capacity()*sizeof(lvid_type)*2 + \n sizeof(data) + sizeof(source_arr)*2 + sizeof(local_edge_buffer);\n }\n }; // end of class local_edge_buffer.\n} // end of namespace\n#endif\n" }, { "path": "src/graphlab/graph/local_graph.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_LOCAL_GRAPH_HPP\n#define GRAPHLAB_LOCAL_GRAPH_HPP\n\n#include <cmath>\n\n#include <string>\n#include <list>\n#include <vector>\n#include <set>\n#include <map>\n\n#include <queue>\n#include <algorithm>\n#include <functional>\n#include <fstream>\n\n#include <boost/bind.hpp>\n#include <boost/unordered_set.hpp>\n#include <boost/type_traits.hpp>\n#include <boost/typeof/typeof.hpp>\n#include <boost/iterator/transform_iterator.hpp>\n#include <boost/iterator/counting_iterator.hpp>\n#include <boost/iterator/zip_iterator.hpp>\n#include <boost/range/iterator_range.hpp>\n\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/graph/local_edge_buffer.hpp>\n#include <graphlab/util/random.hpp>\n#include <graphlab/util/generics/shuffle.hpp>\n#include <graphlab/util/generics/counting_sort.hpp>\n#include <graphlab/util/generics/vector_zip.hpp>\n#include <graphlab/util/generics/csr_storage.hpp>\n#include <graphlab/parallel/atomic.hpp>\n\n#include <graphlab/logger/logger.hpp>\n#include <graphlab/logger/assertions.hpp>\n\n#include <graphlab/serialization/iarchive.hpp>\n#include <graphlab/serialization/oarchive.hpp>\n\n#include <graphlab/util/random.hpp>\n#include <graphlab/macros_def.hpp>\n\nnamespace graphlab { \n\n template<typename VertexData, typename EdgeData>\n class local_graph {\n public:\n \n /** The type of the vertex data stored in the local_graph. */\n typedef VertexData vertex_data_type;\n\n /** The type of the edge data stored in the local_graph. */\n typedef EdgeData edge_data_type;\n\n typedef graphlab::vertex_id_type vertex_id_type;\n typedef graphlab::edge_id_type edge_id_type;\n\n class edge_type;\n class vertex_type;\n\n private:\n class edge_iterator;\n\n public:\n typedef boost::iterator_range<edge_iterator> edge_list_type;\n\n /** Vertex object which provides access to the vertex data\n * and information about it.\n */ \n class vertex_type {\n public:\n vertex_type(local_graph& lgraph_ref, lvid_type vid):lgraph_ref(lgraph_ref),vid(vid) { }\n\n /// \\brief Returns a constant reference to the data on the vertex.\n const vertex_data_type& data() const {\n return lgraph_ref.vertex_data(vid);\n }\n /// \\brief Returns a reference to the data on the vertex.\n vertex_data_type& data() {\n return lgraph_ref.vertex_data(vid);\n }\n /// \\brief Returns the number of in edges of the vertex.\n size_t num_in_edges() const {\n return lgraph_ref.num_in_edges(vid);\n }\n /// \\brief Returns the number of out edges of the vertex.\n size_t num_out_edges() const {\n return lgraph_ref.num_out_edges(vid);\n }\n /// \\brief Returns the ID of the vertex. \n lvid_type id() const {\n return vid;\n }\n /// \\brief Returns a list of in edges.\n edge_list_type in_edges() {\n return lgraph_ref.in_edges(vid);\n }\n /// \\brief Returns a list of out edges.\n edge_list_type out_edges() {\n return lgraph_ref.out_edges(vid);\n }\n private:\n local_graph& lgraph_ref;\n lvid_type vid;\n };\n /** Edge object which provides access to the edge data\n * and information about it.\n */\n class edge_type {\n public:\n edge_type(local_graph& lgraph_ref, lvid_type _source, lvid_type _target, edge_id_type _eid) : \n lgraph_ref(lgraph_ref), _source(_source), _target(_target), _eid(_eid) { }\n\n /// \\brief Returns a constant reference to the data on the edge.\n const edge_data_type& data() const {\n return lgraph_ref.edge_data(_eid);\n }\n /// \\brief Returns a reference to the data on the edge.\n edge_data_type& data() {\n return lgraph_ref.edge_data(_eid);\n }\n /// \\brief Returns the source vertex of the edge.\n vertex_type source() const {\n return vertex_type(lgraph_ref, _source);\n }\n /// \\brief Returns the target vertex of the edge.\n vertex_type target() const {\n return vertex_type(lgraph_ref, _target);\n }\n /// \\brief Returns the internal ID of this edge\n edge_id_type id() const { return _eid; }\n\n private:\n local_graph& lgraph_ref;\n lvid_type _source;\n lvid_type _target;\n edge_id_type _eid;\n };\n\n public:\n\n // CONSTRUCTORS ============================================================>\n \n /** Create an empty local_graph. */\n local_graph() : finalized(false) { }\n\n /** Create a local_graph with nverts vertices. */\n local_graph(size_t nverts) :\n vertices(nverts),\n finalized(false) { }\n\n // METHODS =================================================================>\n \n static bool is_dynamic() {\n return false;\n }\n\n /**\n * \\brief Resets the local_graph state.\n */\n void clear() {\n finalized = false;\n vertices.clear();\n edges.clear();\n _csc_storage.clear();\n _csr_storage.clear();\n std::vector<VertexData>().swap(vertices);\n std::vector<EdgeData>().swap(edges);\n edge_buffer.clear();\n }\n\n /**\n * \\brief Finalize the local_graph data structure by\n * sorting edges to maximize the efficiency of graphlab. \n * This function takes O(|V|log(degree)) time and will \n * fail if there are any duplicate edges.\n * Detail implementation depends on the type of graph_storage.\n * This is also automatically invoked by the engine at start.\n */\n void finalize() { \n if(finalized) return;\n graphlab::timer mytimer; mytimer.start();\n#ifdef DEBUG_GRAPH\n logstream(LOG_DEBUG) << \"Graph2 finalize starts.\" << std::endl;\n#endif\n std::vector<edge_id_type> permute;\n std::vector<edge_id_type> src_counting_prefix_sum;\n std::vector<edge_id_type> dest_counting_prefix_sum;\n \n#ifdef DEBUG_GRAPH\n logstream(LOG_DEBUG) << \"Graph2 finalize: Sort by source vertex\" << std::endl;\n#endif\n // Sort edges by source;\n // Begin of counting sort.\n counting_sort(edge_buffer.source_arr, permute, &src_counting_prefix_sum);\n\n // Inplace permute of edge_data, edge_src, edge_target array.\n#ifdef DEBUG_GRAPH\n logstream(LOG_DEBUG) << \"Graph2 finalize: Inplace permute by source id\" << std::endl;\n#endif\n lvid_type swap_src; lvid_type swap_target; EdgeData swap_data;\n for (size_t i = 0; i < permute.size(); ++i) {\n if (i != permute[i]) {\n // Reserve the ith entry;\n size_t j = i;\n swap_data = edge_buffer.data[i];\n swap_src = edge_buffer.source_arr[i];\n swap_target = edge_buffer.target_arr[i];\n // Begin swap cycle:\n while (j != permute[j]) {\n size_t next = permute[j];\n if (next != i) {\n edge_buffer.data[j] = edge_buffer.data[next];\n edge_buffer.source_arr[j] = edge_buffer.source_arr[next];\n edge_buffer.target_arr[j] = edge_buffer.target_arr[next];\n permute[j] = j;\n j = next;\n } else {\n // end of cycle\n edge_buffer.data[j] = swap_data;\n edge_buffer.source_arr[j] = swap_src;\n edge_buffer.target_arr[j] = swap_target;\n permute[j] = j;\n break;\n }\n }\n }\n }\n#ifdef DEBUG_GRAPH\n logstream(LOG_DEBUG) << \"Graph2 finalize: Sort by dest id\" << std::endl;\n#endif\n counting_sort(edge_buffer.target_arr, permute, &dest_counting_prefix_sum); \n // Shuffle source array\n#ifdef DEBUG_GRAPH\n logstream(LOG_DEBUG) << \"Graph2 finalize: Outofplace permute by dest id\" << std::endl;\n#endif\n\n outofplace_shuffle(edge_buffer.source_arr, permute);\n // Use inplace shuffle to reduce peak memory footprint:\n // inplace_shuffle(edge_buffer.source_arr, permute);\n // counting_sort(edge_buffer.target_arr, permute);\n\n // warp into csr csc storage.\n _csr_storage.wrap(src_counting_prefix_sum, edge_buffer.target_arr);\n std::vector<std::pair<lvid_type, edge_id_type> > csc_value = vector_zip(edge_buffer.source_arr, permute);\n //ASSERT_EQ(csc_value.size(), edge_buffer.size());\n _csc_storage.wrap(dest_counting_prefix_sum, csc_value); \n edges.swap(edge_buffer.data);\n ASSERT_EQ(_csr_storage.num_values(), _csc_storage.num_values());\n ASSERT_EQ(_csr_storage.num_values(), edges.size());\n#ifdef DEBGU_GRAPH\n logstream(LOG_DEBUG) << \"End of finalize.\" << std::endl;\n#endif\n\n logstream(LOG_INFO) << \"Graph finalized in \" << mytimer.current_time() \n << \" secs\" << std::endl;\n finalized = true;\n } // End of finalize\n\n /** \\brief Get the number of vertices */\n size_t num_vertices() const {\n return vertices.size();\n } // end of num vertices\n\n /** \\brief Get the number of edges */\n size_t num_edges() const {\n return edges.size();\n } // end of num edges\n\n /** \n * \\brief Creates a vertex containing the vertex data and returns the id\n * of the new vertex id. Vertex ids are assigned in increasing order with\n * the first vertex having id 0.\n */\n void add_vertex(lvid_type vid, \n const VertexData& vdata = VertexData() ) {\n if(vid >= vertices.size()) {\n // Enable capacity doubling if resizing beyond capacity\n if(vid >= vertices.capacity()) {\n const size_t new_size = std::max(2 * vertices.capacity(), \n size_t(vid));\n vertices.reserve(new_size);\n }\n vertices.resize(vid+1);\n }\n vertices[vid] = vdata; \n } // End of add vertex;\n\n void reserve(size_t num_vertices) {\n ASSERT_GE(num_vertices, vertices.size());\n vertices.reserve(num_vertices);\n }\n\n /** \n * \\brief Add additional vertices up to provided num_vertices. This will\n * fail if resizing down.\n */\n void resize(size_t num_vertices ) {\n ASSERT_GE(num_vertices, vertices.size());\n vertices.resize(num_vertices);\n } // End of resize\n\n void reserve_edge_space(size_t n) {\n edge_buffer.reserve_edge_space(n);\n }\n /**\n * \\brief Creates an edge connecting vertex source to vertex target. Any\n * existing data will be cleared. Should not be called after finalization.\n */\n edge_id_type add_edge(lvid_type source, lvid_type target, \n const EdgeData& edata = EdgeData()) {\n if (finalized) {\n logstream(LOG_FATAL)\n << \"Attempting add edge to a finalized local_graph.\" << std::endl;\n ASSERT_MSG(false, \"Add edge to a finalized local_graph.\");\n }\n\n if(source == target) {\n logstream(LOG_FATAL) \n << \"Attempting to add self edge (\" << source << \" -> \" << target << \"). \"\n << \"This operation is not permitted in GraphLab!\" << std::endl;\n ASSERT_MSG(source != target, \"Attempting to add self edge!\");\n }\n\n if(source >= vertices.size() || target >= vertices.size()) \n add_vertex(std::max(source, target));\n\n // Add the edge to the set of edge data (this copies the edata)\n edge_buffer.add_edge(source, target, edata);\n\n // This is not the final edge_id, so we always return 0. \n return 0;\n } // End of add edge\n \n /**\n * \\brief Add edges in block.\n */\n void add_edges(const std::vector<lvid_type>& src_arr, \n const std::vector<lvid_type>& dst_arr, \n const std::vector<EdgeData>& edata_arr) {\n ASSERT_TRUE((src_arr.size() == dst_arr.size())\n && (src_arr.size() == edata_arr.size()));\n if (finalized) {\n logstream(LOG_FATAL)\n << \"Attempting add edges to a finalized local_graph.\" << std::endl;\n }\n\n for (size_t i = 0; i < src_arr.size(); ++i) {\n lvid_type source = src_arr[i];\n lvid_type target = dst_arr[i];\n if ( source >= vertices.size() \n || target >= vertices.size() ) {\n logstream(LOG_FATAL) \n << \"Attempting add_edge (\" << source\n << \" -> \" << target\n << \") when there are only \" << vertices.size() \n << \" vertices\" << std::endl;\n ASSERT_MSG(source < vertices.size(), \"Invalid source vertex!\");\n ASSERT_MSG(target < vertices.size(), \"Invalid target vertex!\");\n }\n\n if(source == target) {\n logstream(LOG_FATAL) \n << \"Attempting to add self edge (\" << source << \" -> \" << target << \"). \"\n << \"This operation is not permitted in GraphLab!\" << std::endl;\n ASSERT_MSG(source != target, \"Attempting to add self edge!\");\n }\n }\n edge_buffer.add_block_edges(src_arr, dst_arr, edata_arr);\n } // End of add block edges\n\n\n /** \\brief Returns a vertex of given ID. */\n vertex_type vertex(lvid_type vid) {\n ASSERT_LT(vid, vertices.size());\n return vertex_type(*this, vid);\n }\n\n /** \\brief Returns a vertex of given ID. */\n const vertex_type vertex(lvid_type vid) const {\n ASSERT_LT(vid, vertices.size());\n return vertex_type(*this, vid);\n }\n\n /** \\brief Returns a reference to the data stored on the vertex v. */\n VertexData& vertex_data(lvid_type v) {\n ASSERT_LT(v, vertices.size());\n return vertices[v];\n } // end of data(v)\n\n /** \\brief Returns a constant reference to the data stored on the vertex v. */\n const VertexData& vertex_data(lvid_type v) const {\n ASSERT_LT(v, vertices.size());\n return vertices[v];\n } // end of data(v)\n\n /** \\brief Load the local_graph from an archive */\n void load(iarchive& arc) {\n clear(); \n // read the vertices\n arc >> vertices\n >> edges \n >> _csr_storage\n >> _csc_storage\n >> finalized;\n } // end of load\n\n /** \\brief Save the local_graph to an archive */\n void save(oarchive& arc) const {\n // Write the number of edges and vertices\n arc << vertices\n << edges\n << _csr_storage \n << _csc_storage\n << finalized;\n } // end of save\n \n /** swap two graphs */\n void swap(local_graph& other) {\n finalized = other.finalized;\n std::swap(vertices, other.vertices);\n std::swap(edges, other.edges);\n std::swap(_csr_storage, other._csr_storage);\n std::swap(_csc_storage, other._csc_storage);\n std::swap(finalized, other.finalized);\n } // end of swap\n\n\n /** \\brief Load the local_graph from a file */\n void load(const std::string& filename) {\n std::ifstream fin(filename.c_str());\n iarchive iarc(fin);\n iarc >> *this;\n fin.close();\n } // end of load\n\n /**\n * \\brief save the local_graph to the file given by the filename\n */ \n void save(const std::string& filename) const {\n std::ofstream fout(filename.c_str());\n oarchive oarc(fout);\n oarc << *this;\n fout.close();\n } // end of save\n\n /**\n * \\brief save the adjacency structure to a text file.\n *\n * Save the adjacency structure as a text file in:\n * src_Id, dest_Id \\n\n * src_Id, dest_Id \\n\n * format.\n */\n void save_adjacency(const std::string& filename) const {\n std::ofstream fout(filename.c_str());\n ASSERT_TRUE(fout.good());\n\n for (size_t i = 0; i < num_vertices(); ++i) {\n vertex_type v(i);\n edge_list_type ls = v.out_edges();\n foreach(edge_type e, ls) {\n fout << (lvid_type)i << \", \" << e.target().id() << \"\\n\";\n ASSERT_TRUE(fout.good());\n }\n }\n fout.close();\n }\n\n /****************************************************************************\n * Internal Functions *\n * ---------------------- *\n * These functions functions and types provide internal access to the *\n * underlying local_graph representation. They should not be used unless you *\n * *really* know what you are doing. *\n ****************************************************************************/\n /** \n * \\internal\n * \\brief Returns the number of in edges of the vertex with the given id. */\n size_t num_in_edges(const lvid_type v) const {\n ASSERT_TRUE(finalized);\n return (_csc_storage.end(v) - _csc_storage.begin(v));\n }\n\n /** \n * \\internal\n * \\brief Returns the number of in edges of the vertex with the given id. */\n size_t num_out_edges(const lvid_type v) const {\n ASSERT_TRUE(finalized);\n return (_csr_storage.end(v) - _csr_storage.begin(v));\n }\n\n /** \n * \\internal\n * \\brief Returns a list of in edges of the vertex with the given id. */\n edge_list_type in_edges(lvid_type v) {\n edge_iterator begin = edge_iterator(*this, _csc_storage.begin(v), v);\n edge_iterator end = edge_iterator(*this, _csc_storage.end(v), v);\n return boost::make_iterator_range(begin, end);\n }\n\n /** \n * \\internal\n * \\brief Returns a list of out edges of the vertex with the given id. */\n edge_list_type out_edges(lvid_type v) {\n\n csr_type::iterator base_begin = _csr_storage.begin(v);\n csr_type::iterator base_end = _csr_storage.end(v);\n\n edge_id_type begin_eid = base_begin - _csr_storage.begin(0); \n edge_id_type end_eid = base_end - _csr_storage.begin(0); \n\n boost::counting_iterator<edge_id_type> counter_begin(begin_eid);\n boost::counting_iterator<edge_id_type> counter_end(end_eid);\n\n edge_iterator begin = \n edge_iterator(*this,\n csr_edge_iterator(csr_iterator_tuple(base_begin, counter_begin)), v);\n\n edge_iterator end = \n edge_iterator(*this,\n csr_edge_iterator(csr_iterator_tuple(base_end, counter_end)), v);\n\n return boost::make_iterator_range(begin, end);\n }\n\n /** \n * \\internal\n * \\brief Returns edge data of edge_type e\n * */\n EdgeData& edge_data(edge_id_type eid) {\n ASSERT_LT(eid, num_edges());\n return edges[eid]; \n }\n /** \n * \\internal\n * \\brief Returns const edge data of edge_type e\n * */\n const EdgeData& edge_data(edge_id_type eid) const {\n ASSERT_LT(eid, num_edges());\n return edges[eid]; \n }\n\n /** \n * \\internal\n * \\brief Returns the estimated memory footprint of the local_graph. */\n size_t estimate_sizeof() const {\n const size_t vlist_size = sizeof(vertices) + \n sizeof(VertexData) * vertices.capacity();\n size_t elist_size = _csr_storage.estimate_sizeof() \n + _csc_storage.estimate_sizeof()\n + sizeof(edges) + sizeof(EdgeData)*edges.capacity();\n size_t ebuffer_size = edge_buffer.estimate_sizeof();\n // std::cerr << \"local_graph: tmplist size: \" << (double)elist_size/(1024*1024)\n // << \" gstoreage size: \" << (double)store_size/(1024*1024)\n // << \" vdata list size: \" << (double)vlist_size/(1024*1024)\n // << std::endl;\n return vlist_size + elist_size + ebuffer_size;\n }\n\n\n /** \\internal\n * \\brief For debug purpose, returns the largest vertex id in the edge_buffer\n */ \n const lvid_type maxlvid() const {\n if (edge_buffer.size()) {\n lvid_type max(0);\n foreach(lvid_type i, edge_buffer.source_arr)\n max = std::max(max, i); \n foreach(lvid_type i, edge_buffer.target_arr)\n max = std::max(max, i); \n return max;\n } else {\n return lvid_type(-1);\n }\n }\n \n private: \n /** \n * \\internal\n * CSR/CSC storage types\n */\n typedef csr_storage<lvid_type, edge_id_type> csr_type;\n typedef csr_storage<std::pair<lvid_type, edge_id_type>, edge_id_type> csc_type; \n\n typedef boost::tuple<csr_type::iterator,\n boost::counting_iterator<edge_id_type>\n > csr_iterator_tuple;\n\n typedef boost::zip_iterator<csr_iterator_tuple> csr_edge_iterator;\n typedef csc_type::iterator csc_edge_iterator;\n\n class edge_iterator : \n public boost::iterator_facade <\n edge_iterator,\n edge_type,\n boost::random_access_traversal_tag,\n edge_type> {\n public:\n edge_iterator(local_graph& lgraph_ref, \n csc_edge_iterator iter, lvid_type sourceid) \n : lgraph_ref(lgraph_ref), _type(CSC), csc_iter(iter), vid(sourceid) {}\n edge_iterator(local_graph& lgraph_ref,\n csr_edge_iterator iter, lvid_type destid) \n : lgraph_ref(lgraph_ref), _type(CSR), csr_iter(iter), vid(destid) {}\n\n private:\n friend class boost::iterator_core_access;\n\n void increment() {\n switch (_type) {\n case CSC: ++csc_iter; break;\n case CSR: ++csr_iter; break;\n default: return;\n }\n }\n bool equal(const edge_iterator& other) const\n {\n ASSERT_EQ(_type, other._type);\n switch (_type) {\n case CSC: return csc_iter == other.csc_iter;\n case CSR: return csr_iter == other.csr_iter;\n default: return true;\n }\n }\n edge_type dereference() const { \n return make_value();\n }\n void decrement() {\n switch (_type) {\n case CSC: --csc_iter; break;\n case CSR: --csr_iter; break;\n default: return;\n }\n }\n void advance(int n) {\n switch (_type) {\n case CSC: csc_iter+=n; break;\n case CSR: csr_iter+=n; break;\n default: return;\n }\n } \n ptrdiff_t distance_to(const edge_iterator& other) const {\n switch (_type) {\n case CSC: return other.csc_iter - csc_iter;\n case CSR: return other.csr_iter - csr_iter;\n default: return 0;\n }\n }\n private:\n edge_type make_value() const {\n switch (_type) {\n case CSC: {\n typename csc_edge_iterator::reference val\n = *csc_iter;\n return edge_type(lgraph_ref, val.first, vid, val.second);\n }\n case CSR: {\n typename csr_edge_iterator::reference val\n = *csr_iter;\n return edge_type(lgraph_ref,\n vid,\n val.template get<0>(),\n val.template get<1>());\n }\n default: return edge_type(lgraph_ref, -1, -1, -1);\n }\n }\n enum list_type {CSR, CSC}; \n local_graph& lgraph_ref;\n const list_type _type;\n csc_edge_iterator csc_iter;\n csr_edge_iterator csr_iter;\n const lvid_type vid;\n }; // end of edge_iterator\n\n\n /**************************************************************************/\n /* */\n /* PRIVATE DATA MEMBERS */\n /* */\n /**************************************************************************/\n /** The vertex data is simply a vector of vertex data */\n std::vector<VertexData> vertices;\n\n /** Stores the edge data and edge relationships. */\n csr_type _csr_storage;\n csc_type _csc_storage;\n std::vector<EdgeData> edges;\n\n /** The edge data is a vector of edges where each edge stores its\n source, destination, and data. Used for temporary storage. The\n data is transferred into CSR+CSC representation in\n Finalize. This will be cleared after finalized.*/\n local_edge_buffer<VertexData, EdgeData> edge_buffer;\n \n /** Mark whether the local_graph is finalized. Graph finalization is a\n costly procedure but it can also dramatically improve\n performance. */\n bool finalized;\n\n\n /**************************************************************************/\n /* */\n /* declare friends */\n /* */\n /**************************************************************************/\n friend class local_graph_test; \n }; // End of class local_graph\n\n\n template<typename VertexData, typename EdgeData>\n std::ostream& operator<<(std::ostream& out,\n local_graph<VertexData, EdgeData>& g) {\n typedef typename local_graph<VertexData, EdgeData>::edge_type edge_type;\n for(lvid_type vid = 0; vid < g.num_vertices(); ++vid) {\n foreach(const edge_type& e, g.out_edges(vid))\n out << e.source().id() << \", \" << e.target().id() << '\\n';\n }\n return out;\n }\n} // end of namespace graphlab\n\n\nnamespace std {\n /**\n * Swap two graphs\n */\n template<typename VertexData, typename EdgeData>\n inline void swap(graphlab::local_graph<VertexData,EdgeData>& a,\n graphlab::local_graph<VertexData,EdgeData>& b) {\n a.swap(b);\n } // end of swap\n\n}; // end of namespace std\n\n#include <graphlab/macros_undef.hpp>\n#endif\n" }, { "path": "src/graphlab/graph/local_graph_ops.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n/**\n * \\file graph_ops.hpp\n *\n * This file supports basic graph io operations to simplify reading\n * and writing adjacency structures from files.\n *\n */\n\n#ifndef GRAPHLAB_LOCAL_GRAPH_OPS_HPP\n#define GRAPHLAB_LOCAL_GRAPH_OPS_HPP\n\n\n\n#include <iostream>\n#include <fstream>\n#include <string>\n\n#include <boost/algorithm/string/predicate.hpp>\n#include <graphlab/graph/local_graph.hpp>\n\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n \n\n namespace local_graph_ops {\n \n \n /**\n * builds a topological_sort of the graph returning it in topsort. \n * \n * \\param[out] topsort Resultant topological sort of the graph vertices.\n *\n * function will return false if graph is not acyclic.\n */\n template <typename VertexType, typename EdgeType>\n bool topological_sort(const graphlab::local_graph<VertexType, EdgeType>& graph,\n std::vector<vertex_id_type>& topsort) {\n typedef graphlab::local_graph<VertexType, EdgeType> graph_type;\n topsort.clear();\n topsort.reserve(graph.num_vertices());\n std::vector<size_t> indeg;\n indeg.resize(graph.num_vertices());\n std::queue<vertex_id_type> q;\n for (size_t i = 0;i < graph.num_vertices(); ++i) {\n indeg[i] = graph.get_in_edges(i).size();\n if (indeg[i] == 0) {\n q.push(i);\n }\n }\n \n while (!q.empty()) {\n vertex_id_type v = q.front();\n q.pop();\n topsort.push_back(v);\n foreach(typename graph_type::edge_type edge, graph.get_out_edges(v)) {\n vertex_id_type destv = edge.target();\n --indeg[destv];\n if (indeg[destv] == 0) {\n q.push(destv);\n }\n }\n }\n if (q.empty() && topsort.size() != graph.num_vertices()) {\n return false;\n }\n return true;\n } // end of topological sort\n\n\n template <typename VertexType, typename EdgeType>\n size_t num_neighbors(const graphlab::local_graph<VertexType, EdgeType>& graph,\n vertex_id_type& vid) {\n typedef graphlab::local_graph<VertexType, EdgeType> graph_type;\n typename graph_type::edge_list_type in_edges = graph.in_edges(vid); \n typename graph_type::edge_list_type out_edges = graph.out_edges(vid);\n typename graph_type::edge_list_type::const_iterator i = in_edges.begin();\n typename graph_type::edge_list_type::const_iterator j = out_edges.begin();\n size_t count = 0; \n for( ; i != in_edges.end() && j != out_edges.end(); ++count) {\n if(i->source() == j->target()) { \n ++i; ++j; \n } else if(i->source() < j->target()) { \n ++i; \n } else { \n ++j; \n }\n }\n for( ; i != in_edges.end(); ++i, ++count);\n for( ; j != out_edges.end(); ++j, ++count);\n return count;\n } // end of num_neighbors\n\n\n\n template <typename VertexType, typename EdgeType>\n void neighbors(const graphlab::local_graph<VertexType, EdgeType>& graph,\n const vertex_id_type vid, \n std::vector<vertex_id_type>& neighbors ) {\n typedef graphlab::local_graph<VertexType, EdgeType> graph_type;\n typename graph_type::edge_list_type in_edges = graph.in_edges(vid); \n typename graph_type::edge_list_type out_edges = graph.out_edges(vid);\n typename graph_type::edge_list_type::const_iterator i = in_edges.begin();\n typename graph_type::edge_list_type::const_iterator j = out_edges.begin();\n while(i != in_edges.end() && j != out_edges.end()) {\n if(i->source() == j->target()) { \n neighbors.push_back(i->source()); \n ++i; ++j; \n } else if(i->source() < j->target()) {\n neighbors.push_back(i->source()); \n ++i; \n } else { \n neighbors.push_back(j->target()); \n ++j; \n } \n }\n for( ; i != in_edges.end(); ++i) neighbors.push_back(i->source());\n for( ; j != out_edges.end(); ++j) neighbors.push_back(j->target());\n } // end of neighbors\n\n\n\n\n \n\n\n \n template <typename VertexType, typename EdgeType>\n bool save_metis_structure(const std::string& filename,\n const graphlab::local_graph<VertexType, EdgeType>& graph) {\n typedef graphlab::local_graph<VertexType, EdgeType> graph_type;\n typedef typename graph_type::edge_type edge_type;\n typedef typename graph_type::edge_list_type edge_list_type;\n \n std::ofstream fout(filename.c_str());\n if(!fout.good()) return false;\n // Count the number of actual edges\n size_t nedges = 0;\n for(vertex_id_type i = 0; i < graph.num_vertices(); ++i)\n nedges += num_neighbors(graph, i);\n fout << graph.num_vertices() << ' ' << (nedges/2) << '\\n';\n // Save the adjacency structure\n std::vector<vertex_id_type> neighbor_set;\n for(vertex_id_type i = 0; i < graph.num_vertices(); ++i) {\n neighbors(graph, i, neighbor_set);\n for(size_t j = 0; j < neighbor_set.size(); ++j) {\n fout << (neighbor_set[j] + 1);\n if(j + 1 < neighbor_set.size()) fout << ' ';\n }\n fout << '\\n';\n }\n fout.close();\n return true;\n } // end of save metis\n\n\n\n\n\n template <typename VertexType, typename EdgeType>\n bool save_edge_list_structure(const std::string& filename,\n const graphlab::local_graph<VertexType, EdgeType>& graph) {\n typedef graphlab::local_graph<VertexType, EdgeType> graph_type;\n typedef typename graph_type::edge_type edge_type;\n typedef typename graph_type::edge_list_type edge_list_type;\n\n std::ofstream fout(filename.c_str());\n if(!fout.good()) return false;\n for(vertex_id_type i = 0; i < graph.num_vertices(); ++i) \n foreach(edge_type edge, graph.out_edges(i)) \n fout << edge.source() << '\\t' << edge.target() << '\\n'; \n fout.close();\n return true;\n } // end of save metis\n\n\n\n\n template <typename VertexType, typename EdgeType>\n bool save_zoltan_hypergraph_structure(const std::string& filename,\n const graphlab::local_graph<VertexType, EdgeType>& graph) {\n typedef graphlab::local_graph<VertexType, EdgeType> graph_type;\n typedef typename graph_type::edge_type edge_type;\n typedef typename graph_type::edge_list_type edge_list_type;\n\n std::ofstream fout(filename.c_str());\n if(!fout.good()) return false;\n\n // ok. I need to uniquely number each edge.\n // how?\n boost::unordered_map<std::pair<vertex_id_type, \n vertex_id_type>, size_t> edgetoid;\n size_t curid = 0;\n for(vertex_id_type i = 0; i < graph.num_vertices(); ++i) {\n foreach(const typename graph_type::edge_type& edge, graph.in_edges(i)) {\n std::pair<vertex_id_type, vertex_id_type> e = \n std::make_pair(edge.source(), edge.target());\n if (e.first > e.second) std::swap(e.first, e.second);\n if (edgetoid.find(e) == edgetoid.end()) {\n edgetoid[e] = curid;\n ++curid;\n }\n }\n foreach(const typename graph_type::edge_type& edge, graph.out_edges(i)) {\n std::pair<vertex_id_type, vertex_id_type> e = \n std::make_pair(edge.source(), edge.target());\n if (e.first > e.second) std::swap(e.first, e.second);\n if (edgetoid.find(e) == edgetoid.end()) {\n edgetoid[e] = curid;\n ++curid;\n }\n }\n }\n\n size_t numedges = curid;\n // each edge is a vertex, each vertex is an edge\n // a pin is total adjacency of a hyper edge\n fout << numedges << \"\\n\\n\";\n for (size_t i = 0;i < numedges; ++i) {\n fout << i+1 << \"\\n\";\n }\n fout << \"\\n\";\n fout << graph.num_vertices() << \"\\n\\n\";\n \n fout << numedges * 2 << \"\\n\\n\";\n // loop over the \"hyperedge\" and write out the edges it is adjacent to\n for(vertex_id_type i = 0; i < graph.num_vertices(); ++i) {\n boost::unordered_set<size_t> adjedges;\n foreach(const typename graph_type::edge_type& edge, graph.in_edges(i)) {\n std::pair<vertex_id_type, vertex_id_type> e = \n std::make_pair(edge.source(), edge.target());\n if (e.first > e.second) std::swap(e.first, e.second);\n adjedges.insert(edgetoid[e]);\n }\n foreach(const typename graph_type::edge_type& edge, graph.out_edges(i)) {\n std::pair<vertex_id_type, vertex_id_type> e = \n std::make_pair(edge.source(), edge.target());\n if (e.first > e.second) std::swap(e.first, e.second);\n adjedges.insert(edgetoid[e]);\n }\n // write\n std::vector<size_t> adjedgesvec;\n std::copy(adjedges.begin(), adjedges.end(), \n std::inserter(adjedgesvec, adjedgesvec.end()));\n fout << i+1 << \" \" << adjedgesvec.size() << \"\\t\"; \n for (size_t j = 0;j < adjedgesvec.size(); ++j) {\n fout << adjedgesvec[j] + 1;\n if (j < adjedgesvec.size() - 1) fout << \"\\t\";\n }\n fout << \"\\n\";\n }\n fout.close();\n return true;\n } // end of save_zoltan_hypergraph_structure\n\n\n\n }; // end of graph ops\n}; // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n#endif\n\n\n\n\n\n" }, { "path": "src/graphlab/graph/vertex_set.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n#ifndef GRAPHLAB_GRAPH_VERTEX_SET_HPP\n#define GRAPHLAB_GRAPH_VERTEX_SET_HPP\n\n#include <graphlab/util/dense_bitset.hpp>\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/rpc/buffered_exchange.hpp>\n#include <graphlab/macros_def.hpp>\n\nnamespace graphlab {\n\n/**\n * \\brief Describes a set of vertices\n *\n * The vertex_set describes a set of vertices upon which\n * union / intersection / difference can be performed.\n * These sets can then be passed into graph aggregate operations\n * such as distributed_graph::map_reduce_vertices to perform aggregates\n * over \\b subsets of vertices or edges. Engines also permit signalling of\n * sets of vertices through\n * \\ref graphlab::iengine::signal_all(const vertex_set& vset, const message_type& message, const std::string& order) \"signal_all()\".\n *\n * \\ref distributed_graph::complete_set() and \\ref distributed_graph::empty_set()\n * provide two convenient functions to obtain a full or an empty set of\n * vertices.\n * \\code\n * vertex_set all = graph.complete_set();\n * vertex_set empty = graph.empty_set();\n * \\endcode\n *\n * \\ref distributed_graph::select() can be used to obtain a restriction of the\n * set of vertices. For instance if vertices contain an integer, the following\n * code will construct a set of vertices containing only vertices with data\n * which are a multiple of 2.\n *\n * \\code\n * bool is_multiple_of_2(const graph_type::vertex_type& vertex) {\n * return vertex.data() % 2 == 0;\n * }\n * vertex_set even_vertices = graph.select(is_multiple_of_2);\n * \\endcode\n * For more details see \\ref distributed_graph::select()\n *\n * The size of the vertex set can only be queried through the graph using\n * \\ref distributed_graph::vertex_set_size();\n *\n */\nclass vertex_set {\n public:\n /**\n * Used only if \\ref lazy is false.\n * If \\ref lazy is false, this must be the same size as the graph's\n * graphlab::distributed_graph::num_local_vertices().\n * The invariant is that the bit value of each mirror vertex must be the\n * same value as the bit value on their corresponding master vertices.\n */\n mutable dense_bitset localvset;\n\n /**\n * Used only if \\ref lazy is set.\n * If is_complete_set is true, this set describes the set of all vertices.\n * If is_complete set is false, this set describes the empty set.\n */\n bool is_complete_set;\n\n /**\n * If set, the localvset is empty and not used.\n * instead, \\ref is_complete_set will define the set of vertices.\n */\n mutable bool lazy;\n\n\n /**\n * \\internal\n * \\brief Returns a const reference to the underlying bitset.\n */\n template <typename DGraphType>\n const dense_bitset& get_lvid_bitset(const DGraphType& dgraph) const {\n if (lazy) make_explicit(dgraph);\n return localvset;\n }\n\n\n /**\n * \\internal\n * Sets a bit in the bitset without local threading\n * synchronization. vertex set must be made explicit. This call does not\n * perform remote synchronization and addititional distributed\n * synchronization calls must be made to restore datastructure invariants.\n */\n inline void set_lvid_unsync(lvid_type lvid) {\n ASSERT_FALSE(lazy);\n localvset.set_bit_unsync(lvid);\n }\n\n\n /**\n * \\internal\n * Sets a bit in the bitset with local threading\n * synchronization. vertex set must be made explicit. This call does not\n * perform remote synchronization and addititional distributed\n * synchronization calls must be made to restore datastructure invariants.\n */\n inline void set_lvid(lvid_type lvid) {\n ASSERT_FALSE(lazy);\n localvset.set_bit(lvid);\n }\n\n /**\n * \\internal\n * Makes the internal representation explicit by clearing the lazy flag\n * and filling the bitset.\n */\n template <typename DGraphType>\n void make_explicit(const DGraphType& dgraph) const {\n if (lazy) {\n localvset.resize(dgraph.num_local_vertices());\n if (is_complete_set) {\n localvset.fill();\n }\n else {\n localvset.clear();\n }\n lazy = false;\n }\n }\n\n /**\n * \\internal\n * Copies the master state to each mirror.\n * Restores the datastructure invariants.\n */\n template <typename DGraphType>\n void synchronize_master_to_mirrors(DGraphType& dgraph,\n buffered_exchange<vertex_id_type>& exchange) {\n if (lazy) {\n make_explicit(dgraph);\n return;\n }\n foreach(size_t lvid, localvset) {\n typename DGraphType::local_vertex_type lvtx = dgraph.l_vertex(lvid);\n if (lvtx.owned()) {\n // send to mirrors\n vertex_id_type gvid = lvtx.global_id();\n foreach(size_t proc, lvtx.mirrors()) {\n exchange.send(proc, gvid);\n }\n }\n else {\n localvset.clear_bit_unsync(lvid);\n }\n }\n exchange.flush();\n\n typename buffered_exchange<vertex_id_type>::buffer_type recv_buffer;\n procid_t sending_proc;\n\n while(exchange.recv(sending_proc, recv_buffer)) {\n foreach(vertex_id_type gvid, recv_buffer) {\n localvset.set_bit_unsync(dgraph.vertex(gvid).local_id());\n }\n recv_buffer.clear();\n }\n exchange.barrier();\n }\n\n\n /**\n * \\internal\n * Let the master state be the logical OR of the mirror states.\n */\n template <typename DGraphType>\n void synchronize_mirrors_to_master_or(DGraphType& dgraph,\n buffered_exchange<vertex_id_type>& exchange) {\n if (lazy) {\n make_explicit(dgraph);\n return;\n }\n foreach(size_t lvid, localvset) {\n typename DGraphType::local_vertex_type lvtx = dgraph.l_vertex(lvid);\n if (!lvtx.owned()) {\n // send to master\n vertex_id_type gvid = lvtx.global_id();\n exchange.send(lvtx.owner(), gvid);\n }\n }\n exchange.flush();\n\n typename buffered_exchange<vertex_id_type>::buffer_type recv_buffer;\n procid_t sending_proc;\n\n while(exchange.recv(sending_proc, recv_buffer)) {\n foreach(vertex_id_type gvid, recv_buffer) {\n localvset.set_bit_unsync(dgraph.vertex(gvid).local_id());\n }\n recv_buffer.clear();\n }\n exchange.barrier();\n }\n\n template <typename VertexType, typename EdgeType>\n friend class distributed_graph;\n\n public:\n /// default constructor which constructs an empty set.\n vertex_set():is_complete_set(false), lazy(true){}\n\n\n /** Constructs a completely empty, or a completely full vertex set\n * \\param complete If set to true, creates a set of all vertices.\n * If set to false, creates an empty set.\n */\n explicit vertex_set(bool complete):is_complete_set(complete),lazy(true){}\n\n /// copy constructor\n inline vertex_set(const vertex_set& other):\n localvset(other.localvset),\n is_complete_set(other.is_complete_set),\n lazy(other.lazy) {}\n\n /// copyable\n inline vertex_set& operator=(const vertex_set& other) {\n localvset = other.localvset;\n is_complete_set = other.is_complete_set;\n lazy = other.lazy;\n return *this;\n }\n\n /**\n * \\internal\n * Queries if a local vertex ID is contained within the vertex set\n */\n inline bool l_contains(lvid_type lvid) const {\n if (lazy) return is_complete_set;\n if (lvid < localvset.size()) {\n return localvset.get(lvid);\n }\n else {\n return false;\n }\n }\n\n /**\n * \\brief Takes the set intersection of two vertex sets.\n *\n * \\code\n * vertex_set intersection_result = a & b;\n * \\endcode\n * A vertex is in \\c intersection_result if and only if the vertex is in\n * \\b both set \\c a and set \\c b.\n */\n inline vertex_set operator&(const vertex_set& other) const {\n vertex_set ret = (*this);\n ret &= other;\n return ret;\n }\n\n /**\n * \\brief Takes the set union of two vertex sets.\n *\n * \\code\n * vertex_set union_result = a | b;\n * \\endcode\n * A vertex is in \\c union_result if and only if the vertex is in\n * \\b either of set \\c a and set \\c b.\n *\n */\n inline vertex_set operator|(const vertex_set& other) const {\n vertex_set ret = (*this);\n ret |= other;\n return ret;\n }\n\n\n /**\n * \\brief Takes the set difference of two vertex sets.\n *\n * \\code\n * vertex_set difference_result = a - b;\n * \\endcode\n * A vertex is in \\c difference_result if and only if the vertex is in\n * set a and not in set b.\n *\n * Equivalent to:\n *\n * \\code\n * vertex_set inv_b = ~b;\n * vertex_set difference_result = a & inv_b;\n * \\endcode\n */\n inline vertex_set operator-(const vertex_set& other) const {\n vertex_set ret = (*this);\n ret -= other;\n return ret;\n }\n\n\n /**\n * \\brief Takes the set intersection of the current vertex set with another\n * vertex set.\n *\n * \\code\n * a &= b;\n * \\endcode\n * A vertex is in the resultant \\c a if and only if the vertex was in\n * \\b both set \\c a and set \\c b.\n */\n inline vertex_set& operator&=(const vertex_set& other) {\n if (lazy) {\n if (is_complete_set) (*this) = other;\n else (*this) = vertex_set(false);\n }\n else if (other.lazy) {\n if (other.is_complete_set) /* no op */;\n else (*this) = vertex_set(false);\n }\n else {\n localvset &= other.localvset;\n }\n return *this;\n }\n\n /**\n * \\brief Takes the set union of the current vertex set with another\n * vertex set.\n *\n * \\code\n * a |= b;\n * \\endcode\n * A vertex is in the resultant \\c a if and only if the vertex was in\n * \\b either set \\c a and set \\c b.\n */\n inline vertex_set& operator|=(const vertex_set& other) {\n if (lazy) {\n if (is_complete_set) (*this) = vertex_set(true);\n else (*this) = other;\n }\n else if (other.lazy) {\n if (other.is_complete_set) (*this) = vertex_set(true);\n else /* no op */;\n }\n else {\n localvset |= other.localvset;\n }\n return *this;\n }\n\n /**\n * \\brief Takes the set difference of the current vertex set with another\n * vertex set.\n *\n * \\code\n * a -= b;\n * \\endcode\n * A vertex is in the resultant \\c a if and only if the vertex was in\n * set \\c a but not in set \\c b.\n *\n * Conceptually equivalent to\n * \\code\n * a &= ~b;\n * \\endcode\n */\n inline vertex_set& operator-=(const vertex_set& other) {\n if (lazy) {\n if (is_complete_set) (*this) = ~other;\n else (*this) = vertex_set(false);\n }\n else if (other.lazy) {\n if (other.is_complete_set) (*this) = vertex_set(false);\n else /* no op */;\n }\n else {\n localvset -= other.localvset;\n }\n return *this;\n }\n\n /**\n * \\brief Returns the inverse of the current set.\n *\n * \\code\n * vertex_set inv_b = ~b;\n * \\endcode\n * A vertex is in \\c inv_b if and only if it is not in \\c b\n */\n inline vertex_set operator~() const {\n vertex_set ret(*this);\n ret.invert();\n return ret;\n }\n\n /**\n * \\brief Inverts the current set in-place.\n *\n * \\code\n * b.invert();\n * \\endcode\n * A vertex is in the result \\c b if and only if it is not in \\c b\n */\n inline void invert() {\n if (lazy) {\n is_complete_set = !is_complete_set;\n }\n else {\n localvset.invert();\n }\n }\n\n\n};\n\n\n} // namespace graphlab\n#include <graphlab/macros_undef.hpp>\n#endif\n" }, { "path": "src/graphlab/jni/CMakeLists.txt", "content": "project(GraphLab)\n# NOTE: do not link tcmalloc! Does not like Java.\nadd_jni_library(graphlabjni\n org_graphlab_Updater.cpp\n org_graphlab_Aggregator.cpp\n org_graphlab_Context.cpp\n org_graphlab_Core.cpp\n java_any.cpp)\n\nif(JNI_REALLY_FOUND)\n # copy to ${SRC}/extapis/java_jni/lib when done\n get_target_property(graphlabjni_location graphlabjni LOCATION)\n add_custom_command(\n TARGET graphlabjni\n POST_BUILD\n COMMAND cp ${graphlabjni_location}\n ${CMAKE_SOURCE_DIR}/extapis/java_jni/lib/\n COMMENT \"Copying graphlabjni to extapis/java_jni/lib\")\n \n # if ant is found, build Java classes\n if(ANT_FOUND)\n add_custom_command(\n TARGET graphlabjni\n POST_BUILD\n COMMAND cd ${CMAKE_SOURCE_DIR}/extapis/java_jni && ant\n COMMENT \"Building Java classes using Ant.\")\n else()\n message(STATUS\n \"Ant not detected. Java classes will not be built.\") \n endif()\n\nendif()" }, { "path": "src/graphlab/jni/java_any.cpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n \n/**\n * @file java_any.cpp\n * @author Jiunn Haur Lim <jiunnhal@cmu.edu>\n */ \n \n#include <graphlab.hpp>\n#include \"java_any.hpp\"\n#include \"org_graphlab_Core.hpp\"\n#include \"org_graphlab_Updater.hpp\"\n\nusing namespace graphlab;\n\n//---------------------------------------------------------------\n// java_any instance members\n//---------------------------------------------------------------\n\njava_any::java_any(JNIEnv *env, jobject &obj){\n // create a new ref so that it doesn't get garbage collected\n mobj = env->NewGlobalRef(obj);\n}\n\njava_any::java_any() : mobj(NULL){}\n\njobject &java_any::obj() {\n return mobj;\n}\n\nconst jobject &java_any::obj() const {\n return mobj;\n}\n\njava_any::java_any(const java_any& other){\n \n // other doesn't have an existing ref\n if (NULL == other.mobj){\n this->mobj = NULL;\n return;\n }\n \n // create a new ref\n JNIEnv *env = proxy_updater::core::get_jni_env();\n this->mobj = env->NewGlobalRef(other.mobj);\n \n}\n\njava_any &java_any::operator=(const java_any& other){\n \n // prevent self assignment\n if (this == &other) return *this;\n \n JNIEnv *env = proxy_updater::core::get_jni_env();\n jobject obj = NULL;\n \n // if other has a java object, create a new ref\n if (NULL != other.mobj){\n obj = env->NewGlobalRef(other.mobj);\n }\n \n // if this has a java object, delete ref\n if (NULL != this->mobj){\n env->DeleteGlobalRef(this->mobj);\n }\n \n // assign!\n this->mobj = obj;\n \n return *this;\n \n}\n\nvoid java_any::set_obj(jobject obj){\n\n JNIEnv *env = proxy_updater::core::get_jni_env();\n \n if (NULL != mobj){\n // delete current ref\n env->DeleteGlobalRef(mobj);\n mobj = NULL;\n }\n \n if (NULL != obj){\n mobj = env->NewGlobalRef(obj);\n }\n \n}\n\njava_any::~java_any(){\n if (NULL == mobj) return;\n // delete reference to allow garbage collection\n JNIEnv *env = proxy_updater::core::get_jni_env();\n env->DeleteGlobalRef(mobj);\n mobj = NULL;\n}\n\nbool java_any::handle_exception(JNIEnv *env) const {\n \n // check for exception\n jthrowable exc = env->ExceptionOccurred();\n if (!exc) return false;\n\n env->ExceptionDescribe();\n env->ExceptionClear();\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"thrown from C code.\");\n \n return true;\n \n}" }, { "path": "src/graphlab/jni/java_any.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n \n/**\n * @file java_any.hpp\n * @author Jiunn Haur Lim <jiunnhal@cmu.edu>\n */\n \n#ifndef JAVA_ANY_HPP\n#define JAVA_ANY_HPP\n\n#include <jni.h>\n\nnamespace graphlab {\n\n /**\n * Generic wrapper for Java objects (jobject). It creates a NewGlobalRef on\n * the jobject in the default constructor, and deletes the GlobalRef in the\n * destructor. An assignment operator is also provided to deal with creating\n * and deleting edges. Subclasses should provide a copy constructor, and there\n * are two scenarios: NewGlobalRef during copy, or object clone during copy.\n */\n class java_any {\n \n private:\n \n /** Java object */\n jobject mobj;\n \n public:\n \n /**\n * Constructor for java_any.\n * Initializes this object with the associated Java object.\n * @param[in] env JNI environment - used to create a new reference\n * to obj\n * @param[in] obj Java object. This constructor will create a new\n * reference to the object to prevent garbage\n * collection.\n */\n java_any(JNIEnv *env, jobject &obj);\n \n /** The default constructor does nothing. mobj is initialized to NULL. */\n java_any();\n \n /**\n * Copy constructor\n * If `other` has a reference to a java object, increases the reference count.\n * Child classes may wish to override this to implement clone behavior.\n */\n java_any(const java_any& other); \n \n /**\n * Copy assignment operator for java_any.\n * If \\c other has a \\c mobj, creates a new reference to it.\n */\n java_any &operator=(const java_any &other);\n \n /**\n * Retrieves the associated Java object\n */\n jobject &obj();\n const jobject &obj() const;\n \n /**\n * Deletes the reference to the Java object so that it may be garbage\n * collected.\n */\n ~java_any();\n \n protected:\n \n /**\n * Deletes the current ref (if any) and creates a new ref if `obj` is not null.\n * @param[in] obj replaces current object ref\n */\n void set_obj(jobject obj);\n \n /**\n * Checks for and rethrows Java exceptions.\n * @param[in] env JNI environment\n * @return true if exception was found; false otherwise\n */\n bool handle_exception(JNIEnv *env) const;\n \n };\n\n};\n\n#endif" }, { "path": "src/graphlab/jni/org_graphlab_Aggregator.cpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \"org_graphlab_Aggregator.hpp\"\n\nusing namespace graphlab;\n\n//---------------------------------------------------------------\n// proxy_aggregator static members\n//---------------------------------------------------------------\n\njmethodID proxy_aggregator::java_exec = 0;\njmethodID proxy_aggregator::java_add = 0;\njmethodID proxy_aggregator::java_finalize = 0;\njmethodID proxy_aggregator::java_clone = 0;\n\nJNIEXPORT void JNICALL\n Java_org_graphlab_Aggregator_initNative\n (JNIEnv *env, jclass clazz){\n\n // Aggregator#exec\n proxy_aggregator::java_exec =\n env->GetMethodID(clazz, \"exec\", \"(JLorg/graphlab/data/Vertex;)V\");\n \n // Aggregator#add\n proxy_aggregator::java_add =\n env->GetMethodID(clazz, \"add\", \"(Lorg/graphlab/Aggregator;)V\");\n \n // Aggregator#finalize\n proxy_aggregator::java_finalize = \n env->GetMethodID(clazz, \"finalize\", \"(J)V\");\n \n // Aggregator#clone\n proxy_aggregator::java_clone = \n env->GetMethodID(clazz, \"clone\", \"()Lorg/graphlab/Aggregator;\");\n \n}\n\n//---------------------------------------------------------------\n// proxy_aggregator instance members\n//---------------------------------------------------------------\n\nproxy_aggregator::\n proxy_aggregator(JNIEnv *env, jobject &obj)\n : java_any(env, obj){}\n\nproxy_aggregator::proxy_aggregator()\n : java_any(){}\n\nproxy_aggregator::\n proxy_aggregator(const proxy_aggregator& other) {\n\n // other doesn't have an existing ref\n if (NULL == other.obj()){\n set_obj(NULL);\n return;\n }\n \n // clone the java object\n JNIEnv *env = core::get_jni_env();\n set_obj(env->CallObjectMethod(other.obj(), java_clone));\n \n}\n\nproxy_aggregator &proxy_aggregator::operator=(const proxy_aggregator& other){\n \n if (this == &other) return *this;\n java_any::operator=(other);\n return *this;\n \n}\n\nproxy_aggregator::~proxy_aggregator(){}\n\n//---------------------------------------------------------------\n// proxy_aggregator instance members - the update function\n//---------------------------------------------------------------\n\nvoid proxy_aggregator::operator()(icontext_type& context){\n \n // forward call to org.graphlab.Aggregator#exec\n JNIEnv *env = core::get_jni_env();\n env->CallVoidMethod (obj(), java_exec,\n &context,\n context.vertex_data().obj());\n handle_exception(env);\n\n}\n\n//---------------------------------------------------------------\n// proxy_aggregator instance members - the add function\n//---------------------------------------------------------------\n\nvoid proxy_aggregator::operator+=(const proxy_aggregator& other) {\n \n // forward call to org.graphlab.Aggregator#add\n JNIEnv *env = core::get_jni_env();\n env->CallVoidMethod (obj(), java_add, other.obj());\n handle_exception(env);\n\n}\n\n//---------------------------------------------------------------\n// proxy_aggregator instance members - the finalize function\n//---------------------------------------------------------------\n\nvoid proxy_aggregator::finalize(iglobal_context& context){\n\n // forward call to org.graphlab.Aggregator#finalize\n JNIEnv *env = core::get_jni_env();\n env->CallVoidMethod (obj(), java_finalize, &context);\n handle_exception(env);\n\n}" }, { "path": "src/graphlab/jni/org_graphlab_Aggregator.h", "content": "/* DO NOT EDIT THIS FILE - it is machine generated */\n#include <jni.h>\n/* Header for class org_graphlab_Aggregator */\n\n#ifndef _Included_org_graphlab_Aggregator\n#define _Included_org_graphlab_Aggregator\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n/*\n * Class: org_graphlab_Aggregator\n * Method: initNative\n * Signature: ()V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Aggregator_initNative\n (JNIEnv *, jclass);\n\n#ifdef __cplusplus\n}\n#endif\n#endif\n" }, { "path": "src/graphlab/jni/org_graphlab_Aggregator.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n/**\n * @file org_graphlab_Aggregator.hpp\n * @author Jiunn Haur Lim <jiunnhal@cmu.edu>\n */\n\n#ifndef ORG_GRAPHLAB_AGGREGATOR_HPP\n#define ORG_GRAPHLAB_AGGREGATOR_HPP\n\n#include <graphlab.hpp>\n#include \"java_any.hpp\"\n#include \"org_graphlab_Core.hpp\"\n#include \"org_graphlab_Updater.hpp\"\n\n#include \"org_graphlab_Aggregator.h\"\n\nnamespace graphlab {\n \n /**\n * Proxy aggregator.\n * Mirrors and forwards calls to the corresponding Java aggregator.\n * The constructor creates a new reference to the Java object (so that it\n * doesn't get garbage collected.) The destructor will delete the reference\n * to allow the corresponding Java object to be garbaged collected. The copy\n * constructor clones the Java object.\n */\n class proxy_aggregator : \n public iaggregator<proxy_graph, proxy_updater, proxy_aggregator>,\n public java_any {\n \n private:\n \n typedef proxy_updater::core core;\n \n public:\n \n /**\n * Method ID of org.graphlab.Aggregator#exec.\n */\n static jmethodID java_exec;\n \n /**\n * Method ID of org.graphlab.Aggregator#add.\n */\n static jmethodID java_add;\n \n /**\n * Method ID of org.graphlab.Aggregator#finalize.\n */\n static jmethodID java_finalize;\n \n /**\n * Method ID of org.graphlab.Aggregator#clone\n */\n static jmethodID java_clone;\n \n /**\n * Constructor for proxy aggregator.\n * Initializes this object with the associated Java org.graphlab.Updater\n * object.\n * @param[in] env JNI environment - used to create a new reference\n * to javaUpdater.\n * @param[in] java_aggregator Java org.graphlab.Aggregator object. This constructor\n * will create a new reference to the object to prevent\n * garbage collection.\n */\n proxy_aggregator(JNIEnv *env, jobject &java_aggregator);\n \n /** The default constructor does nothing */\n proxy_aggregator();\n \n /**\n * Copy constructor for proxy_aggregator.\n * If \\c other has a \\c mobj, creates a new reference to it.\n */\n proxy_aggregator(const proxy_aggregator& other);\n \n /**\n * Copy assignment operator for proxy_aggregator.\n * If \\c other has a \\c mobj, creates a new reference to it.\n */\n proxy_aggregator &operator=(const proxy_aggregator &other);\n \n /**\n * Deletes the reference to the Java object so that it may be garbage\n * collected.\n */\n ~proxy_aggregator();\n \n void operator()(icontext_type& context);\n \n void operator+=(const proxy_aggregator& other);\n \n void finalize(iglobal_context& context);\n \n };\n\n};\n\n#endif" }, { "path": "src/graphlab/jni/org_graphlab_Context.cpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n \n/**\n * @file org_graphlab_Context.cpp\n * Implementations of native methods in \\c org.graphlab.Context. Refer to\n * corresponding Javadoc.\n * @author Jiunn Haur Lim <jiunnhal@cmu.edu>\n */\n\n#include \"org_graphlab_Context.hpp\"\n#include \"org_graphlab_Updater.hpp\"\n\nusing namespace graphlab;\n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n JNIEXPORT void JNICALL\n Java_org_graphlab_Context_schedule\n ( JNIEnv *env,\n jobject obj,\n jlong context_ptr,\n jobject updater,\n jint vertex_id){\n\n // convert longs to pointers\n proxy_updater::context *context = (proxy_updater::context *) context_ptr;\n context->schedule(vertex_id, proxy_updater(env, updater));\n \n }\n\n#ifdef __cplusplus\n}\n#endif" }, { "path": "src/graphlab/jni/org_graphlab_Context.h", "content": "/* DO NOT EDIT THIS FILE - it is machine generated */\n#include <jni.h>\n/* Header for class org_graphlab_Context */\n\n#ifndef _Included_org_graphlab_Context\n#define _Included_org_graphlab_Context\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n/*\n * Class: org_graphlab_Context\n * Method: schedule\n * Signature: (JLorg/graphlab/Updater;I)V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Context_schedule\n (JNIEnv *, jobject, jlong, jobject, jint);\n\n#ifdef __cplusplus\n}\n#endif\n#endif\n" }, { "path": "src/graphlab/jni/org_graphlab_Context.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n/**\n * @file org_graphlab_Context.hpp\n * \\c javah will generate \\c org_graphlab_Context.h from the native methods\n * defined in \\c org.graphlab.Context (and so will overwrite the file every time).\n * Define any additional classes/structs/typedefs in this hpp file.\n * @author Jiunn Haur Lim <jiunnhal@cmu.edu>\n */\n\n#ifndef ORG_GRAPHLAB_CONTEXT_HPP\n#define ORG_GRAPHLAB_CONTEXT_HPP\n\n#include <graphlab.hpp>\n#include \"org_graphlab_Context.h\"\n\n// define any additional classes/structs/typedefs here\n\n#endif" }, { "path": "src/graphlab/jni/org_graphlab_Core.cpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n/**\n * @file org_graphlab_Core.cpp\n *\n * Contains the JNI interface for org.graphlab.Core. In general, applications\n * will keep their graphs in the Java layer and access the engine through the\n * JNI. This wrapper provides a proxy graph for the engine to manipulate and\n * forwards update calls to the Java layer. To learn how to use this interface,\n * refer to the org.graphlab.Core class and to the examples.\n *\n * @author Jiunn Haur Lim <jiunnhal@cmu.edu>\n */\n\n#include <wordexp.h>\n#include \"org_graphlab_Core.hpp\"\n#include \"org_graphlab_Updater.hpp\"\n#include \"org_graphlab_Aggregator.hpp\"\n\nusing namespace graphlab;\n\n//---------------------------------------------------------------\n// jni_core static members\n//---------------------------------------------------------------\n\ntemplate<typename G, typename U>\nJavaVM* jni_core<G, U>::mjvm = NULL;\n\ntemplate<typename G, typename U>\nconst size_t jni_core<G, U>::ENV_ID = 1;\n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n//---------------------------------------------------------------\n// static helper functions\n//---------------------------------------------------------------\n\n static jlong createCore (JNIEnv *env, jobject obj, int argc, char **argv){\n \n // configure log level\n global_logger().set_log_level(LOG_DEBUG);\n global_logger().set_log_to_console(false);\n\n // set jvm, if we don't have it already\n if (NULL == proxy_updater::core::get_jvm()){\n JavaVM* jvm = NULL;\n env->GetJavaVM(&jvm);\n proxy_updater::core::set_jvm(jvm);\n }\n \n // store env for this thread\n thread::get_local(proxy_updater::core::ENV_ID) = env;\n \n // allocate and configure core\n proxy_updater::core *jni_core = new proxy_updater::core(env, obj);\n if (NULL != argv){\n (*jni_core)().parse_options(argc, argv);\n }\n \n // return address of jni_core\n return (long) jni_core;\n \n }\n \n//---------------------------------------------------------------\n// JNI functions\n//---------------------------------------------------------------\n\n JNIEXPORT jlong JNICALL\n Java_org_graphlab_Core_createCore__\n (JNIEnv *env, jobject obj){\n return createCore(env, obj, 0, NULL);\n }\n \n JNIEXPORT jlong JNICALL\n Java_org_graphlab_Core_createCore__Ljava_lang_String_2\n (JNIEnv *env, jobject obj, jstring command_line_args){\n \n // convert jstring to c string\n const char *cstr = NULL;\n cstr = env->GetStringUTFChars(command_line_args, NULL);\n if (NULL == cstr) {\n return 0; /* OutOfMemoryError already thrown */\n }\n \n // prepend with dummy name\n char buffer[1024];\n snprintf(buffer, 1024, \"x %s\", cstr);\n env->ReleaseStringUTFChars(command_line_args, cstr);\n \n // split string\n wordexp_t we_result;\n if (0 != wordexp(buffer, &we_result, 0)) return 0;\n \n // create core\n jlong ptr = createCore(env, obj, we_result.we_wordc, we_result.we_wordv);\n \n // cleanup\n wordfree(&we_result);\n return ptr;\n \n }\n \n JNIEXPORT void JNICALL\n Java_org_graphlab_Core_destroyCore\n (JNIEnv *env, jobject obj, jlong ptr){\n \n if (NULL == env || 0 == ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"ptr must not be null.\");\n return;\n }\n \n proxy_updater::core *jni_core = (proxy_updater::core *) ptr;\n \n // cleanup core\n delete jni_core;\n \n }\n\n JNIEXPORT void JNICALL\n Java_org_graphlab_Core_resizeGraph\n (JNIEnv *env, jobject obj, jlong ptr, jint count){\n \n if (NULL == env || 0 == ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"ptr must not be null.\");\n return;\n }\n \n proxy_updater::core *jni_core = (proxy_updater::core *) ptr;\n (*jni_core)().graph().resize(count);\n \n }\n \n JNIEXPORT void JNICALL\n Java_org_graphlab_Core_addVertex\n (JNIEnv *env, jobject obj, jlong ptr,\n jobject app_vertex, jint vertex_id){\n \n if (NULL == env || 0 == ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"ptr must not be null.\");\n return;\n }\n \n proxy_updater::core *jni_core = (proxy_updater::core *) ptr;\n \n // add to graph\n (*jni_core)().graph()\n .add_vertex(vertex_id, proxy_vertex(env, app_vertex));\n \n }\n \n JNIEXPORT void JNICALL\n Java_org_graphlab_Core_addEdge\n (JNIEnv *env, jobject obj, jlong ptr,\n jint source, jint target, jobject app_edge){\n \n if (NULL == env || 0 == ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"ptr must not be null.\");\n return;\n }\n \n proxy_updater::core *jni_core = (proxy_updater::core *) ptr;\n \n // add to graph\n (*jni_core)().graph().add_edge(source, target, proxy_edge(env, app_edge));\n \n }\n \n JNIEXPORT jdouble JNICALL\n Java_org_graphlab_Core_start\n (JNIEnv *env, jobject obj, jlong ptr){\n \n if (NULL == env || 0 == ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"ptr must not be null.\");\n return 0;\n }\n \n proxy_updater::core *jni_core = (proxy_updater::core *) ptr;\n (*jni_core)().engine().get_options().print();\n \n double runtime = (*jni_core)().start();\n return runtime;\n \n }\n \n JNIEXPORT jlong JNICALL\n Java_org_graphlab_Core_lastUpdateCount\n (JNIEnv *env, jobject obj, jlong ptr){\n \n if (NULL == env || 0 == ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"ptr must not be null.\");\n return 0;\n }\n \n proxy_updater::core *jni_core = (proxy_updater::core *) ptr;\n return (*jni_core)().engine().last_update_count();\n \n }\n\n JNIEXPORT void JNICALL\n Java_org_graphlab_Core_addGlobalConst\n (JNIEnv *env, jobject obj, jlong ptr, jstring key, jobject to_store){\n \n if (NULL == env || 0 == ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"ptr must not be null.\");\n return;\n }\n \n proxy_updater::core *jni_core = (proxy_updater::core *) ptr;\n \n // convert jstring to c string\n const char * key_str = env->GetStringUTFChars(key, NULL);\n \n (*jni_core)().add_global_const(std::string(key_str), java_any(env, to_store));\n \n // free memory\n env->ReleaseStringUTFChars(key, key_str);\n \n return;\n \n }\n \n JNIEXPORT void JNICALL\n Java_org_graphlab_Core_addGlobal\n (JNIEnv *env, jobject obj, jlong ptr, jstring key, jobject to_store){\n \n if (NULL == env || 0 == ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"ptr must not be null.\");\n return;\n }\n \n proxy_updater::core *jni_core = (proxy_updater::core *) ptr;\n \n // convert jstring to c string\n const char * key_str = env->GetStringUTFChars(key, NULL);\n \n java_any a = java_any(env, to_store);\n (*jni_core)().add_global(std::string(key_str), a);\n \n // free memory\n env->ReleaseStringUTFChars(key, key_str);\n \n return;\n \n }\n \n JNIEXPORT void JNICALL\n Java_org_graphlab_Core_setGlobal\n (JNIEnv *env, jobject obj, jlong ptr, jstring key, jobject to_store){\n \n if (NULL == env || 0 == ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"ptr must not be null.\");\n return;\n }\n \n proxy_updater::core *jni_core = (proxy_updater::core *) ptr;\n \n // convert jstring to c string\n const char * key_str = env->GetStringUTFChars(key, NULL);\n \n java_any a = java_any(env, to_store);\n (*jni_core)().set_global(std::string(key_str), a);\n \n // free memory\n env->ReleaseStringUTFChars(key, key_str);\n \n return;\n \n }\n \n\n JNIEXPORT jobject JNICALL\n Java_org_graphlab_Core_getGlobal\n (JNIEnv *env, jobject obj, jlong ptr, jstring key){\n \n if (NULL == env || 0 == ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"ptr must not be null.\");\n return NULL;\n }\n \n proxy_updater::core *jni_core = (proxy_updater::core *) ptr;\n \n // convert jstring to c string\n const char * key_str = env->GetStringUTFChars(key, NULL);\n \n java_any stored = (*jni_core)().get_global<java_any>(std::string(key_str));\n \n // free memory\n env->ReleaseStringUTFChars(key, key_str);\n \n return env->NewLocalRef(stored.obj());\n \n }\n\n JNIEXPORT void JNICALL\n Java_org_graphlab_Core_setNCpus\n (JNIEnv * env, jobject obj, jlong ptr, jlong ncpus) {\n \n if (NULL == env || 0 == ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"ptr must not be null.\");\n return;\n }\n \n proxy_updater::core *jni_core = (proxy_updater::core *) ptr;\n (*jni_core)().set_ncpus(ncpus);\n \n }\n\n JNIEXPORT void JNICALL\n Java_org_graphlab_Core_setSchedulerType\n (JNIEnv * env, jobject obj, jlong ptr, jstring scheduler_str) {\n \n if (NULL == env || 0 == ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"ptr must not be null.\");\n return;\n }\n \n const char *str = env->GetStringUTFChars(scheduler_str, NULL);\n if (NULL == str) return; // OutOfMemoryError already thrown\n \n proxy_updater::core *jni_core = (proxy_updater::core *) ptr;\n (*jni_core)().set_scheduler_type(std::string(str));\n env->ReleaseStringUTFChars(scheduler_str, str);\n \n }\n\n JNIEXPORT void JNICALL\n Java_org_graphlab_Core_setScopeType\n (JNIEnv * env, jobject obj, jlong ptr, jstring scope_str) {\n \n if (NULL == env || 0 == ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"ptr must not be null.\");\n return;\n }\n \n const char *str = env->GetStringUTFChars(scope_str, NULL);\n if (NULL == str) return; // OutOfMemoryError already thrown\n \n proxy_updater::core *jni_core = (proxy_updater::core *) ptr;\n (*jni_core)().set_scope_type(std::string(str));\n env->ReleaseStringUTFChars(scope_str, str);\n \n }\n \n JNIEXPORT void JNICALL\n Java_org_graphlab_Core_schedule\n (JNIEnv * env, jobject obj,\n jlong core_ptr, jobject updater, jint vertex_id){\n \n if (NULL == env || 0 == core_ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"core_ptr must not be null.\");\n return;\n }\n\n // get objects from pointers\n proxy_updater::core *jni_core = (proxy_updater::core *) core_ptr;\n\n // schedule vertex\n (*jni_core)().schedule(vertex_id, proxy_updater(env, updater));\n \n }\n \n JNIEXPORT void JNICALL \n Java_org_graphlab_Core_scheduleAll\n (JNIEnv * env, jobject obj,\n jlong core_ptr, jobject updater) {\n\n if (NULL == env || 0 == core_ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"core_ptr and updater_ptr must not be null.\");\n return;\n }\n\n // get objects from pointers\n proxy_updater::core *jni_core = (proxy_updater::core *) core_ptr;\n\n // schedule vertex\n (*jni_core)().schedule_all(proxy_updater(env, updater));\n\n }\n \n JNIEXPORT void JNICALL\n Java_org_graphlab_Core_addAggregator\n (JNIEnv * env, jobject obj,\n jlong core_ptr, jstring key, jobject aggregator,\n jlong frequency){\n \n if (NULL == env || 0 == core_ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"core_ptr and updater_ptr must not be null.\");\n return;\n }\n\n // get objects from pointers\n proxy_updater::core *jni_core = (proxy_updater::core *) core_ptr;\n\n // add aggregator\n const char * key_str = env->GetStringUTFChars(key, NULL);\n (*jni_core)().add_aggregator(std::string(key_str),\n proxy_aggregator(env, aggregator),\n frequency);\n env->ReleaseStringUTFChars(key, key_str);\n \n }\n \n JNIEXPORT void JNICALL\n Java_org_graphlab_Core_aggregateNow\n (JNIEnv * env, jobject obj,\n jlong core_ptr, jstring key){\n \n if (NULL == env || 0 == core_ptr){\n proxy_updater::core::throw_exception(\n env,\n \"java/lang/IllegalArgumentException\",\n \"core_ptr and updater_ptr must not be null.\");\n return;\n }\n\n // get objects from pointers\n proxy_updater::core *jni_core = (proxy_updater::core *) core_ptr;\n\n // add aggregator\n const char * key_str = env->GetStringUTFChars(key, NULL);\n (*jni_core)().aggregate_now(std::string(key_str));\n env->ReleaseStringUTFChars(key, key_str);\n \n }\n\n#ifdef __cplusplus\n}\n#endif\n" }, { "path": "src/graphlab/jni/org_graphlab_Core.h", "content": "/* DO NOT EDIT THIS FILE - it is machine generated */\n#include <jni.h>\n/* Header for class org_graphlab_Core */\n\n#ifndef _Included_org_graphlab_Core\n#define _Included_org_graphlab_Core\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n/*\n * Class: org_graphlab_Core\n * Method: createCore\n * Signature: ()J\n */\nJNIEXPORT jlong JNICALL Java_org_graphlab_Core_createCore__\n (JNIEnv *, jobject);\n\n/*\n * Class: org_graphlab_Core\n * Method: createCore\n * Signature: (Ljava/lang/String;)J\n */\nJNIEXPORT jlong JNICALL Java_org_graphlab_Core_createCore__Ljava_lang_String_2\n (JNIEnv *, jobject, jstring);\n\n/*\n * Class: org_graphlab_Core\n * Method: destroyCore\n * Signature: (J)V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Core_destroyCore\n (JNIEnv *, jobject, jlong);\n\n/*\n * Class: org_graphlab_Core\n * Method: resizeGraph\n * Signature: (JI)V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Core_resizeGraph\n (JNIEnv *, jobject, jlong, jint);\n\n/*\n * Class: org_graphlab_Core\n * Method: addVertex\n * Signature: (JLorg/graphlab/data/Vertex;)I\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Core_addVertex\n (JNIEnv *, jobject, jlong, jobject, jint);\n\n/*\n * Class: org_graphlab_Core\n * Method: addEdge\n * Signature: (JII)V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Core_addEdge\n (JNIEnv *, jobject, jlong, jint, jint, jobject);\n\n/*\n * Class: org_graphlab_Core\n * Method: schedule\n * Signature: (JLorg/graphlab/Updater;I)V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Core_schedule\n (JNIEnv *, jobject, jlong, jobject, jint);\n\n/*\n * Class: org_graphlab_Core\n * Method: scheduleAll\n * Signature: (JLorg/graphlab/Updater;)V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Core_scheduleAll\n (JNIEnv *, jobject, jlong, jobject);\n\n/*\n * Class: org_graphlab_Core\n * Method: start\n * Signature: (J)D\n */\nJNIEXPORT jdouble JNICALL Java_org_graphlab_Core_start\n (JNIEnv *, jobject, jlong);\n\n/*\n * Class: org_graphlab_Core\n * Method: lastUpdateCount\n * Signature: (J)J\n */\nJNIEXPORT jlong JNICALL Java_org_graphlab_Core_lastUpdateCount\n (JNIEnv *, jobject, jlong);\n\n/*\n * Class: org_graphlab_Core\n * Method: addGlobalConst\n * Signature: (JLjava/lang/String;Ljava/lang/Object;)V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Core_addGlobalConst\n (JNIEnv *, jobject, jlong, jstring, jobject);\n\n/*\n * Class: org_graphlab_Core\n * Method: addGlobal\n * Signature: (JLjava/lang/String;Ljava/lang/Object;)V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Core_addGlobal\n (JNIEnv *, jobject, jlong, jstring, jobject);\n\n/*\n * Class: org_graphlab_Core\n * Method: getGlobal\n * Signature: (JLjava/lang/String;)Ljava/lang/Object;\n */\nJNIEXPORT jobject JNICALL Java_org_graphlab_Core_getGlobal\n (JNIEnv *, jobject, jlong, jstring);\n\n/*\n * Class: org_graphlab_Core\n * Method: setGlobal\n * Signature: (JLjava/lang/String;Ljava/lang/Object;)V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Core_setGlobal\n (JNIEnv *, jobject, jlong, jstring, jobject);\n\n/*\n * Class: org_graphlab_Core\n * Method: addAggregator\n * Signature: (JLjava/lang/String;Lorg/graphlab/Aggregator;J)V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Core_addAggregator\n (JNIEnv *, jobject, jlong, jstring, jobject, jlong);\n\n/*\n * Class: org_graphlab_Core\n * Method: aggregateNow\n * Signature: (JLjava/lang/String;)V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Core_aggregateNow\n (JNIEnv *, jobject, jlong, jstring);\n\n/*\n * Class: org_graphlab_Core\n * Method: setNCpus\n * Signature: (JJ)V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Core_setNCpus\n (JNIEnv *, jobject, jlong, jlong);\n\n/*\n * Class: org_graphlab_Core\n * Method: setSchedulerType\n * Signature: (JLjava/lang/String;)V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Core_setSchedulerType\n (JNIEnv *, jobject, jlong, jstring);\n\n/*\n * Class: org_graphlab_Core\n * Method: setScopeType\n * Signature: (JLjava/lang/String;)V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Core_setScopeType\n (JNIEnv *, jobject, jlong, jstring);\n\n#ifdef __cplusplus\n}\n#endif\n#endif\n" }, { "path": "src/graphlab/jni/org_graphlab_Core.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n \n/**\n * @file org_graphlab_Core.hpp\n * \\c javah will generate \\c org_graphlab_Core.h from the native methods\n * defined in \\c org.graphlab.Context (and so will overwrite the file every time).\n * Define any additional classes/structs/typedefs in this hpp file.\n * @author Jiunn Haur Lim <jiunnhal@cmu.edu>\n */\n\n#ifndef ORG_GRAPHLAB_CORE_HPP\n#define ORG_GRAPHLAB_CORE_HPP\n\n#include <execinfo.h>\n#include <graphlab.hpp>\n\n#include \"java_any.hpp\"\n#include \"org_graphlab_Core.h\"\n\nnamespace graphlab {\n\n /**\n * Wrapper for graphlab::core.\n * Contains the core, a reference to the Java core object (so that it\n * doesn't get garbage collected), and other utility functions for dealing\n * with the JVM.\n */\n template <typename Graph, typename UpdateFunctor>\n class jni_core : public java_any {\n \n public:\n \n /** ID of pointer to JNI environment in thread local store */\n static const size_t ENV_ID;\n \n private:\n \n typedef core<Graph, UpdateFunctor> core_type;\n \n /** graphlab::core object - the soul that this body wraps around */\n core_type *mcore;\n \n /** Java virtual machine reference - set only once for each process */\n static JavaVM *mjvm;\n \n public:\n\n /**\n * Creates a new graphlab core and a new reference to the associated\n * Java org.graphlab.Core object (so that it doesn't get garbage collected.)\n * @param[in] env JNI environment, which will be used to create the\n * reference to the Java object.\n * @param[in] obj associated org.graphlab.Core object.\n */\n jni_core (JNIEnv *env, jobject &obj) : java_any (env, obj) {\n this->mcore = new core_type();\n }\n \n /**\n * Gets the real graphlab core that this method wraps around\n * @return graphlab::core\n */\n core_type &operator()(){\n return *mcore;\n }\n \n /**\n * Deallocates the graphlab core. Parent constructor will delete jobject\n * reference.\n */\n ~jni_core(){\n delete mcore;\n }\n \n /**\n * Saves a reference to the Java Virtual Machine.\n * @param[in] jvm pointer to the Java Virtual Machine\n */\n static void set_jvm (JavaVM *jvm){\n mjvm = jvm;\n }\n \n /**\n * Gets a reference to the Java Virtual Machine.\n * @return pointer to the Java Virtual Machine\n */\n static JavaVM *get_jvm (){\n return mjvm;\n }\n \n /**\n * Detaches the current thread from the JVM.\n * If a pointer to the JNI environment cannot be found in the thread-local\n * store, that means that this thread has already been detached, and the\n * function will return immediately. Otherwise, the thread is detached and\n * the pointer to the JNI environment is removed from the thread-local\n * store.\n */\n static void detach_from_jvm() {\n \n // if the current thread is still attached, detach it\n if (thread::contains(ENV_ID)) {\n int res = mjvm->DetachCurrentThread();\n assert(res >= 0);\n thread::get_local(ENV_ID) = NULL;\n }\n \n }\n \n static void dump_backtrace(int sig){\n \n void *array[10];\n size_t size;\n \n // get void*'s for all entries on the stack\n size = backtrace(array, 10);\n \n // print out all the frames to stderr\n backtrace_symbols_fd(array, size, 2);\n \n }\n \n /** Convenience method for throwing Java exceptions */\n static void throw_exception(JNIEnv* env,\n const char *exception,\n const char *message){\n jclass exc = env->FindClass(exception);\n if (NULL == exc) return;\n env->ThrowNew(exc, message);\n }\n \n /**\n * Retrieves the JNI environment for the current thread.\n * If a pointer to the JNI environment can be found in the thread-local\n * store, returns immediately; otherwise, that means that the current\n * thread has not been attached to the JVM yet. In that case, this \n * function will attach the current thread to the JVM and save the\n * associated JNI environment to the thread-local storage.\n * @return JNI environment associated with the current thread.\n */\n static JNIEnv *get_jni_env (){\n \n JNIEnv *jenv = NULL;\n \n // if current thread is not already on the JVM, attach it \n if (!thread::contains(ENV_ID)) {\n \n int res = mjvm->AttachCurrentThread((void **)&jenv, NULL);\n assert(res >= 0);\n \n // store JNI environment in thread-local storage\n thread::get_local(ENV_ID) = jenv;\n thread::set_thread_destroy_callback(detach_from_jvm);\n \n }\n \n // return the environment associated with the current thread\n return thread::get_local(ENV_ID).as<JNIEnv *>();\n \n }\n \n };\n\n}\n\n#endif\n" }, { "path": "src/graphlab/jni/org_graphlab_Updater.cpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \"org_graphlab_Updater.hpp\"\n\nusing namespace graphlab;\n\n//---------------------------------------------------------------\n// proxy_updater static members\n//---------------------------------------------------------------\n\njmethodID proxy_updater::java_update = 0;\njmethodID proxy_updater::java_add = 0;\njmethodID proxy_updater::java_priority = 0;\njmethodID proxy_updater::java_clone = 0;\njmethodID proxy_updater::java_is_factorizable = 0;\njmethodID proxy_updater::java_gather_edges = 0;\njmethodID proxy_updater::java_scatter_edges = 0;\njmethodID proxy_updater::java_consistency = 0;\njmethodID proxy_updater::java_gather_consistency = 0;\njmethodID proxy_updater::java_scatter_consistency = 0;\njmethodID proxy_updater::java_init_gather = 0;\njmethodID proxy_updater::java_gather = 0;\njmethodID proxy_updater::java_merge = 0;\njmethodID proxy_updater::java_apply = 0;\njmethodID proxy_updater::java_scatter = 0;\n\nJNIEXPORT void JNICALL\n Java_org_graphlab_Updater_initNative\n (JNIEnv *env, jclass clazz){\n\n proxy_updater::java_update =\n env->GetMethodID(clazz, \"update\", \"(JLorg/graphlab/data/Vertex;)V\");\n\n proxy_updater::java_add =\n env->GetMethodID(clazz, \"add\", \"(Lorg/graphlab/Updater;)V\");\n\n proxy_updater::java_priority = \n env->GetMethodID(clazz, \"priority\", \"()D\");\n\n proxy_updater::java_clone = \n env->GetMethodID(clazz, \"clone\", \"()Lorg/graphlab/Updater;\");\n\n proxy_updater::java_is_factorizable =\n env->GetMethodID(clazz, \"isFactorizable\", \"()Z\");\n\n proxy_updater::java_gather_edges =\n env->GetMethodID(clazz, \"gatherEdges\", \"()I\");\n\n proxy_updater::java_scatter_edges =\n env->GetMethodID(clazz, \"scatterEdges\", \"()I\");\n\n proxy_updater::java_consistency =\n env->GetMethodID(clazz, \"consistency\", \"()I\");\n\n proxy_updater::java_gather_consistency =\n env->GetMethodID(clazz, \"gatherConsistency\", \"()I\");\n\n proxy_updater::java_scatter_consistency =\n env->GetMethodID(clazz, \"scatterConsistency\", \"()I\");\n\n proxy_updater::java_init_gather = \n env->GetMethodID(clazz, \"initGather\", \"()V\");\n\n proxy_updater::java_gather =\n env->GetMethodID(clazz, \"gather\", \"(Ljava/lang/Object;)V\");\n\n proxy_updater::java_merge = \n env->GetMethodID(clazz, \"merge\", \"(Lorg/graphlab/Updater;)V\");\n\n proxy_updater::java_apply =\n env->GetMethodID(clazz, \"apply\", \"(Lorg/graphlab/data/Vertex;)V\");\n\n proxy_updater::java_scatter = \n env->GetMethodID(clazz, \"scatter\",\n \"(JLjava/lang/Object;)V\");\n \n}\n\n//---------------------------------------------------------------\n// proxy_updater instance members\n//---------------------------------------------------------------\n\nproxy_updater::\n proxy_updater(JNIEnv *env, jobject &obj)\n : java_any(env, obj){}\n\nproxy_updater::proxy_updater(){}\n\nproxy_updater::\n proxy_updater(const proxy_updater& other){\n\n // other doesn't have an existing ref\n if (NULL == other.obj()){\n set_obj(NULL);\n return;\n }\n \n // clone the java object\n JNIEnv *env = core::get_jni_env();\n set_obj(env->CallObjectMethod(other.obj(), java_clone));\n \n}\n\nproxy_updater &proxy_updater::operator=(const proxy_updater& other){\n \n if (this == &other) return *this;\n java_any::operator=(other);\n return *this;\n \n}\n\nproxy_updater::~proxy_updater(){}\n\n//---------------------------------------------------------------\n// proxy_updater instance members - the update function\n//---------------------------------------------------------------\n\nvoid proxy_updater::operator()(icontext_type& context){\n \n jobject vertex = context.const_vertex_data().obj();\n if (NULL == vertex) return; // BUG?\n \n // forward call to org.graphlab.Updater#update\n JNIEnv *env = core::get_jni_env();\n env->CallVoidMethod (obj(), java_update,\n &context,\n vertex);\n handle_exception(env);\n\n}\n\n//---------------------------------------------------------------\n// proxy_updater instance members - the add function\n//---------------------------------------------------------------\n\nvoid proxy_updater::operator+=(const proxy_updater& other) const {\n\n // forward call to org.graphlab.Updater#add\n JNIEnv *env = core::get_jni_env();\n env->CallVoidMethod (obj(), java_add, other.obj());\n handle_exception(env);\n\n}\n\nbool proxy_updater::is_factorizable() const {\n JNIEnv *env = core::get_jni_env();\n bool factorizable = env->CallBooleanMethod(obj(), java_is_factorizable);\n handle_exception(env);\n return factorizable;\n}\n\nedge_set proxy_updater::gather_edges() const {\n JNIEnv *env = core::get_jni_env();\n int e = env->CallIntMethod(obj(), java_gather_edges);\n handle_exception(env);\n switch(e){\n case 0: return IN_EDGES;\n case 1: return OUT_EDGES;\n case 2: return ALL_EDGES;\n default: return NO_EDGES;\n }\n}\n\nedge_set proxy_updater::scatter_edges() const {\n JNIEnv *env = core::get_jni_env();\n int e = env->CallIntMethod(obj(), java_scatter_edges);\n handle_exception(env);\n switch(e){\n case 0: return IN_EDGES;\n case 1: return OUT_EDGES;\n case 2: return ALL_EDGES;\n default: return NO_EDGES;\n }\n}\n\nconsistency_model proxy_updater::consistency() const {\n JNIEnv *env = core::get_jni_env();\n int c = env->CallIntMethod(obj(), java_consistency);\n handle_exception(env);\n switch(c){\n case 0: return NULL_CONSISTENCY;\n case 1: return VERTEX_CONSISTENCY;\n case 2: return EDGE_CONSISTENCY;\n case 3: return FULL_CONSISTENCY;\n default: return DEFAULT_CONSISTENCY;\n }\n}\n\nconsistency_model proxy_updater::gather_consistency() const {\n JNIEnv *env = core::get_jni_env();\n int c = env->CallIntMethod(obj(), java_gather_consistency);\n handle_exception(env);\n switch(c){\n case 0: return NULL_CONSISTENCY;\n case 1: return VERTEX_CONSISTENCY;\n case 2: return EDGE_CONSISTENCY;\n case 3: return FULL_CONSISTENCY;\n default: return DEFAULT_CONSISTENCY;\n }\n}\n\nconsistency_model proxy_updater::scatter_consistency() const {\n JNIEnv *env = core::get_jni_env();\n int c = env->CallIntMethod(obj(), java_scatter_consistency);\n handle_exception(env);\n switch(c){\n case 0: return NULL_CONSISTENCY;\n case 1: return VERTEX_CONSISTENCY;\n case 2: return EDGE_CONSISTENCY;\n case 3: return FULL_CONSISTENCY;\n default: return DEFAULT_CONSISTENCY;\n }\n}\n\nvoid proxy_updater::init_gather(icontext_type& context) {\n JNIEnv *env = core::get_jni_env();\n env->CallVoidMethod(obj(), java_init_gather);\n handle_exception(env);\n}\n\nvoid proxy_updater::gather(icontext_type& context, const edge_type& edge){\n JNIEnv *env = core::get_jni_env();\n env->CallVoidMethod(obj(), java_gather, context.const_edge_data(edge).obj());\n handle_exception(env);\n}\n \nvoid proxy_updater::merge(const update_functor_type& other){\n JNIEnv *env = core::get_jni_env();\n env->CallVoidMethod(obj(), java_merge, other.obj());\n handle_exception(env);\n}\n\nvoid proxy_updater::apply(icontext_type& context){\n jobject vertex = context.const_vertex_data().obj();\n if (NULL == vertex) return; // BUG?\n JNIEnv *env = core::get_jni_env();\n env->CallVoidMethod(obj(), java_apply, vertex);\n handle_exception(env);\n}\n \nvoid proxy_updater::scatter(icontext_type& context, const edge_type& edge){\n JNIEnv *env = core::get_jni_env();\n env->CallVoidMethod(obj(), java_scatter,\n &context, context.const_edge_data(edge).obj());\n handle_exception(env);\n}\n" }, { "path": "src/graphlab/jni/org_graphlab_Updater.h", "content": "/* DO NOT EDIT THIS FILE - it is machine generated */\n#include <jni.h>\n/* Header for class org_graphlab_Updater */\n\n#ifndef _Included_org_graphlab_Updater\n#define _Included_org_graphlab_Updater\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n#undef org_graphlab_Updater_IN_EDGES\n#define org_graphlab_Updater_IN_EDGES 0L\n#undef org_graphlab_Updater_OUT_EDGES\n#define org_graphlab_Updater_OUT_EDGES 1L\n#undef org_graphlab_Updater_ALL_EDGES\n#define org_graphlab_Updater_ALL_EDGES 2L\n#undef org_graphlab_Updater_NO_EDGES\n#define org_graphlab_Updater_NO_EDGES 3L\n#undef org_graphlab_Updater_NULL_CONSISTENCY\n#define org_graphlab_Updater_NULL_CONSISTENCY 0L\n#undef org_graphlab_Updater_VERTEX_CONSISTENCY\n#define org_graphlab_Updater_VERTEX_CONSISTENCY 1L\n#undef org_graphlab_Updater_EDGE_CONSISTENCY\n#define org_graphlab_Updater_EDGE_CONSISTENCY 2L\n#undef org_graphlab_Updater_FULL_CONSISTENCY\n#define org_graphlab_Updater_FULL_CONSISTENCY 3L\n#undef org_graphlab_Updater_DEFAULT_CONSISTENCY\n#define org_graphlab_Updater_DEFAULT_CONSISTENCY 4L\n/*\n * Class: org_graphlab_Updater\n * Method: initNative\n * Signature: ()V\n */\nJNIEXPORT void JNICALL Java_org_graphlab_Updater_initNative\n (JNIEnv *, jclass);\n\n#ifdef __cplusplus\n}\n#endif\n#endif\n" }, { "path": "src/graphlab/jni/org_graphlab_Updater.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n/**\n * @file org_graphlab_Updater.hpp\n * @author Jiunn Haur Lim <jiunnhal@cmu.edu>\n */\n\n#ifndef ORG_GRAPHLAB_UPDATER_HPP\n#define ORG_GRAPHLAB_UPDATER_HPP\n\n#include <graphlab.hpp>\n#include \"java_any.hpp\"\n#include \"org_graphlab_Core.hpp\"\n#include \"org_graphlab_Updater.h\"\n\nnamespace graphlab {\n\n /** Proxy edge */\n class proxy_edge : public java_any {\n public:\n /**\n * Creates a new proxy_edge and a new reference to the associated\n * Java Edge object (so that it doesn't get garbage collected.)\n * @param[in] env JNI environment, which will be used to create the\n * reference to the Java object.\n * @param[in] obj associated org.graphlab.Core object.\n */\n proxy_edge (JNIEnv *env, jobject &obj) : java_any (env, obj) {}\n };\n \n /** Proxy vertex */\n class proxy_vertex : public java_any {\n public:\n proxy_vertex () : java_any () {}\n /**\n * Creates a new proxy_vertex and a new reference to the associated\n * Java Vertex object (so that it doesn't get garbage collected.)\n * @param[in] env JNI environment, which will be used to create the\n * reference to the Java object.\n * @param[in] obj associated org.graphlab.Core object.\n */\n proxy_vertex (JNIEnv *env, jobject &obj) : java_any (env, obj) {}\n };\n \n /** Proxy graph */\n typedef graph<proxy_vertex, proxy_edge> proxy_graph;\n \n /**\n * Proxy updater.\n * Mirrors and forwards update calls to the corresponding Java updater.\n * The constructor creates a new reference to the Java object (so that it\n * doesn't get garbage collected.) The destructor will delete the reference\n * to allow the corresponding Java object to be garbaged collected. The copy\n * constructor clones the Java object.\n *\n * Note that multiple proxy_updaters may correspond to the same\n * org.graphlab.Updater object.\n */\n class proxy_updater : \n public iupdate_functor<proxy_graph, proxy_updater>,\n public java_any {\n \n public:\n \n /** jni_core type that uses the proxy graph and the proxy updater */\n typedef jni_core<proxy_graph, proxy_updater> core;\n\n /** context type that uses the proxy graph and the proxy updater */\n typedef iupdate_functor<proxy_graph, proxy_updater>::icontext_type context;\n \n /** Method ID of org.graphlab.Updater#update */\n static jmethodID java_update;\n \n /** Method ID of org.graphlab.Updater#add */\n static jmethodID java_add;\n \n /** Method ID of org.graphlab.Updater#priority */\n static jmethodID java_priority;\n \n /** Method ID of org.graphlab.Updater#clone */\n static jmethodID java_clone;\n \n /** Method ID of org.graphlab.Updater#isFactorizable */\n static jmethodID java_is_factorizable;\n \n /** Method ID of org.graphlab.Updater#gatherEdges */\n static jmethodID java_gather_edges;\n \n /** Method ID of org.graphlab.Updater#scatterEdges */\n static jmethodID java_scatter_edges;\n \n /** Method ID of org.graphlab.Updater#consistency */\n static jmethodID java_consistency;\n \n /** Method ID of org.graphlab.Updater#gatherConsistency */\n static jmethodID java_gather_consistency;\n \n /** Method ID of org.graphlab.Updater#scatterConsistency */\n static jmethodID java_scatter_consistency;\n \n /** Method ID of org.graphlab.Updater#initGather */\n static jmethodID java_init_gather;\n \n /** Method ID of org.graphlab.Updater#gather */\n static jmethodID java_gather;\n \n /** Method ID of org.graphlab.Updater#merge */\n static jmethodID java_merge;\n \n /** Method ID of org.graphlab.Updater#apply */\n static jmethodID java_apply;\n \n /** Method ID of org.graphlab.Updater#scatter */\n static jmethodID java_scatter;\n \n /**\n * Constructor for proxy updater.\n * Initializes this object with the associated Java org.graphlab.Updater\n * object.\n * @param[in] env JNI environment - used to create a new reference\n * to javaUpdater.\n * @param[in] java_updater Java org.graphlab.Updater object. This constructor\n * will create a new reference to the object to prevent\n * garbage collection.\n */\n proxy_updater(JNIEnv *env, jobject &java_updater);\n \n /** The default constructor does nothing */\n proxy_updater();\n \n /**\n * Copy constructor for proxy_updater.\n * If \\c other has a \\c mjava_updater, creates a new reference to it.\n */\n proxy_updater(const proxy_updater& other);\n \n /**\n * Copy assignment operator for proxy_updater.\n * If \\c other has a \\c mjava_updater, creates a new reference to it.\n */\n proxy_updater &operator=(const proxy_updater &other);\n \n /**\n * Deletes the reference to the Java object so that it may be garbage\n * collected.\n */\n ~proxy_updater();\n \n void operator()(icontext_type& context);\n void operator+=(const update_functor_type& other) const;\n bool is_factorizable() const;\n edge_set gather_edges() const;\n edge_set scatter_edges() const;\n consistency_model consistency() const;\n consistency_model gather_consistency() const;\n consistency_model scatter_consistency() const;\n void init_gather(icontext_type& context);\n void gather(icontext_type& context, const edge_type& edge);\n void merge(const update_functor_type& other);\n void apply(icontext_type& context);\n void scatter(icontext_type& context, const edge_type& edge);\n \n };\n \n}\n\n#endif\n" }, { "path": "src/graphlab/logger/CMakeLists.txt", "content": "project(GraphLab)\n" }, { "path": "src/graphlab/logger/assertions.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n// Copyright (c) 2005, Google Inc.\n// All rights reserved.\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following disclaimer\n// in the documentation and/or other materials provided with the\n// distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived from\n// this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n// ---\n// This file contains #include information about logging-related stuff.\n// Pretty much everybody needs to #include this file so that they can\n// log various happenings.\n//\n#ifndef _ASSERTIONS_H_\n#define _ASSERTIONS_H_\n\n#include <stdarg.h>\n#include <stdlib.h>\n#include <stdio.h>\n#ifdef HAVE_UNISTD_H\n#include <unistd.h> // for write()\n#endif\n#include <string.h> // for strlen(), strcmp()\n#include <assert.h>\n#include <errno.h> // for errno\n#include <sstream>\n#include <cassert>\n#include <graphlab/logger/logger.hpp>\n#include <graphlab/logger/fail_method.hpp>\n#include <graphlab/logger/backtrace.hpp>\n\n#include <boost/typeof/typeof.hpp>\n\nextern void __print_back_trace();\n\n// On some systems (like freebsd), we can't call write() at all in a\n// global constructor, perhaps because errno hasn't been set up.\n// Calling the write syscall is safer (it doesn't set errno), so we\n// prefer that. Note we don't care about errno for logging: we just\n// do logging on a best-effort basis.\n#define WRITE_TO_STDERR(buf, len) (logbuf(LOG_FATAL, buf, len))\n\n// CHECK dies with a fatal error if condition is not true. It is *not*\n// controlled by NDEBUG, so the check will be executed regardless of\n// compilation mode. Therefore, it is safe to do things like:\n// CHECK(fp->Write(x) == 4)\n#define CHECK(condition) \\\n do { \\\n if (__builtin_expect(!(condition), 0)) { \\\n logstream(LOG_ERROR) \\\n << \"Check failed: \" << #condition << std::endl; \\\n __print_back_trace(); \\\n GRAPHLAB_LOGGER_FAIL_METHOD(\"assertion failure\"); \\\n } \\\n } while(0)\n\n\n// This prints errno as well. errno is the posix defined last error\n// number. See errno.h\n#define PCHECK(condition) \\\n do { \\\n if (__builtin_expect(!(condition), 0)) { \\\n const int _PCHECK_err_no_ = errno; \\\n logstream(LOG_ERROR) \\\n << \"Check failed: \" << #condition << \": \" \\\n << strerror(err_no) << std::endl; \\\n __print_back_trace(); \\\n GRAPHLAB_LOGGER_FAIL_METHOD(\"assertion failure\"); \\\n } \\\n } while(0)\n\n// Helper macro for binary operators; prints the two values on error\n// Don't use this macro directly in your code, use CHECK_EQ et al below\n\n// WARNING: These don't compile correctly if one of the arguments is a pointer\n// and the other is NULL. To work around this, simply static_cast NULL to the\n// type of the desired pointer.\n#if defined(__cplusplus) && __cplusplus >= 201103L\n#define CHECK_OP(op, val1, val2) \\\n do { \\\n const auto _CHECK_OP_v1_ = val1; \\\n const auto _CHECK_OP_v2_ = val2; \\\n if (__builtin_expect(!((_CHECK_OP_v1_) op \\\n (decltype(val1))(_CHECK_OP_v2_)), 0)) { \\\n logstream(LOG_ERROR) \\\n << \"Check failed: \" \\\n << #val1 << #op << #val2 \\\n << \" [\" \\\n << _CHECK_OP_v1_ \\\n << ' ' << #op << ' ' \\\n << _CHECK_OP_v2_ << \"]\" << std::endl; \\\n __print_back_trace(); \\\n GRAPHLAB_LOGGER_FAIL_METHOD(\"assertion failure\"); \\\n } \\\n } while(0)\n#else\n#define CHECK_OP(op, val1, val2) \\\n do { \\\n const typeof(val1) _CHECK_OP_v1_ = (typeof(val1))val1; \\\n const typeof(val2) _CHECK_OP_v2_ = (typeof(val2))val2; \\\n if (__builtin_expect(!((_CHECK_OP_v1_) op \\\n (typeof(val1))(_CHECK_OP_v2_)), 0)) { \\\n logstream(LOG_ERROR) \\\n << \"Check failed: \" \\\n << #val1 << #op << #val2 \\\n << \" [\" \\\n << _CHECK_OP_v1_ \\\n << ' ' << #op << ' ' \\\n << _CHECK_OP_v2_ << \"]\" << std::endl; \\\n __print_back_trace(); \\\n GRAPHLAB_LOGGER_FAIL_METHOD(\"assertion failure\"); \\\n } \\\n } while(0)\n#endif\n\n#define CHECK_EQ(val1, val2) CHECK_OP(==, val1, val2)\n#define CHECK_NE(val1, val2) CHECK_OP(!=, val1, val2)\n#define CHECK_LE(val1, val2) CHECK_OP(<=, val1, val2)\n#define CHECK_LT(val1, val2) CHECK_OP(< , val1, val2)\n#define CHECK_GE(val1, val2) CHECK_OP(>=, val1, val2)\n#define CHECK_GT(val1, val2) CHECK_OP(> , val1, val2)\n\n// Synonyms for CHECK_* that are used in some unittests.\n#define EXPECT_EQ(val1, val2) CHECK_EQ(val1, val2)\n#define EXPECT_NE(val1, val2) CHECK_NE(val1, val2)\n#define EXPECT_LE(val1, val2) CHECK_LE(val1, val2)\n#define EXPECT_LT(val1, val2) CHECK_LT(val1, val2)\n#define EXPECT_GE(val1, val2) CHECK_GE(val1, val2)\n#define EXPECT_GT(val1, val2) CHECK_GT(val1, val2)\n#define ASSERT_EQ(val1, val2) EXPECT_EQ(val1, val2)\n#define ASSERT_NE(val1, val2) EXPECT_NE(val1, val2)\n#define ASSERT_LE(val1, val2) EXPECT_LE(val1, val2)\n#define ASSERT_LT(val1, val2) EXPECT_LT(val1, val2)\n#define ASSERT_GE(val1, val2) EXPECT_GE(val1, val2)\n#define ASSERT_GT(val1, val2) EXPECT_GT(val1, val2)\n// As are these variants.\n#define EXPECT_TRUE(cond) CHECK(cond)\n#define EXPECT_FALSE(cond) CHECK(!(cond))\n#define EXPECT_STREQ(a, b) CHECK(strcmp(a, b) == 0)\n#define ASSERT_TRUE(cond) EXPECT_TRUE(cond)\n#define ASSERT_FALSE(cond) EXPECT_FALSE(cond)\n#define ASSERT_STREQ(a, b) EXPECT_STREQ(a, b)\n\n\n#define ASSERT_MSG(condition, fmt, ...) \\\n do { \\\n if (__builtin_expect(!(condition), 0)) { \\\n logstream(LOG_ERROR) \\\n << \"Check failed: \" << #condition << \":\\n\"; \\\n logger(LOG_ERROR, fmt, ##__VA_ARGS__); \\\n __print_back_trace(); \\\n GRAPHLAB_LOGGER_FAIL_METHOD(\"assertion failure\"); \\\n } \\\n } while(0)\n\n// Used for (libc) functions that return -1 and set errno\n#define CHECK_ERR(invocation) PCHECK((invocation) != -1)\n\n// A few more checks that only happen in debug mode\n#ifdef NDEBUG\n#define DCHECK_EQ(val1, val2)\n#define DCHECK_NE(val1, val2)\n#define DCHECK_LE(val1, val2)\n#define DCHECK_LT(val1, val2)\n#define DCHECK_GE(val1, val2)\n#define DCHECK_GT(val1, val2)\n#define DASSERT_TRUE(cond)\n#define DASSERT_FALSE(cond)\n#define DASSERT_EQ(val1, val2)\n#define DASSERT_NE(val1, val2)\n#define DASSERT_LE(val1, val2)\n#define DASSERT_LT(val1, val2)\n#define DASSERT_GE(val1, val2)\n#define DASSERT_GT(val1, val2)\n\n#define DASSERT_MSG(condition, fmt, ...)\n\n#else\n#define DCHECK_EQ(val1, val2) CHECK_EQ(val1, val2)\n#define DCHECK_NE(val1, val2) CHECK_NE(val1, val2)\n#define DCHECK_LE(val1, val2) CHECK_LE(val1, val2)\n#define DCHECK_LT(val1, val2) CHECK_LT(val1, val2)\n#define DCHECK_GE(val1, val2) CHECK_GE(val1, val2)\n#define DCHECK_GT(val1, val2) CHECK_GT(val1, val2)\n#define DASSERT_TRUE(cond) ASSERT_TRUE(cond)\n#define DASSERT_FALSE(cond) ASSERT_FALSE(cond)\n#define DASSERT_EQ(val1, val2) ASSERT_EQ(val1, val2)\n#define DASSERT_NE(val1, val2) ASSERT_NE(val1, val2)\n#define DASSERT_LE(val1, val2) ASSERT_LE(val1, val2)\n#define DASSERT_LT(val1, val2) ASSERT_LT(val1, val2)\n#define DASSERT_GE(val1, val2) ASSERT_GE(val1, val2)\n#define DASSERT_GT(val1, val2) ASSERT_GT(val1, val2)\n\n\n#define DASSERT_MSG(condition, fmt, ...) \\\n do { \\\n if (__builtin_expect(!(condition), 0)) { \\\n logstream(LOG_ERROR) \\\n << \"Check failed: \" << #condition << \":\\n\"; \\\n logger(LOG_ERROR, fmt, ##__VA_ARGS__); \\\n __print_back_trace(); \\\n GRAPHLAB_LOGGER_FAIL_METHOD(\"assertion failure\"); \\\n } \\\n } while(0)\n\n#endif\n\n\n#ifdef ERROR\n#undef ERROR // may conflict with ERROR macro on windows\n#endif\n\n#endif // _LOGGING_H_\n\n" }, { "path": "src/graphlab/logger/assertions.hpp.orig", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n// Copyright (c) 2005, Google Inc.\n// All rights reserved.\n// \n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n// \n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following disclaimer\n// in the documentation and/or other materials provided with the\n// distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived from\n// this software without specific prior written permission.\n// \n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n// ---\n// This file contains #include information about logging-related stuff.\n// Pretty much everybody needs to #include this file so that they can\n// log various happenings.\n//\n#ifndef _ASSERTIONS_H_\n#define _ASSERTIONS_H_\n\n#include <stdarg.h>\n#include <stdlib.h>\n#include <stdio.h>\n#ifdef HAVE_UNISTD_H\n#include <unistd.h> // for write()\n#endif\n#include <string.h> // for strlen(), strcmp()\n#include <assert.h>\n#include <errno.h> // for errno\n#include <sstream>\n#include <cassert>\n#include <graphlab/logger/logger.hpp>\n\n#include <boost/typeof/typeof.hpp>\n\nextern void __print_back_trace();\n\n// On some systems (like freebsd), we can't call write() at all in a\n// global constructor, perhaps because errno hasn't been set up.\n// Calling the write syscall is safer (it doesn't set errno), so we\n// prefer that. Note we don't care about errno for logging: we just\n// do logging on a best-effort basis.\n#define WRITE_TO_STDERR(buf, len) (logbuf(LOG_FATAL, buf, len))\n\n// CHECK dies with a fatal error if condition is not true. It is *not*\n// controlled by NDEBUG, so the check will be executed regardless of\n// compilation mode. Therefore, it is safe to do things like:\n// CHECK(fp->Write(x) == 4)\n#define CHECK(condition) \\\n do { \\\n if (__builtin_expect(!(condition), 0)) { \\\n logstream(LOG_ERROR) \\\n << \"Check failed: \" << #condition << std::endl; \\\n __print_back_trace(); \\\n throw(\"assertion failure\"); \\\n } \\\n } while(0)\n\n\n// This prints errno as well. errno is the posix defined last error\n// number. See errno.h\n#define PCHECK(condition) \\\n do { \\\n if (__builtin_expect(!(condition), 0)) { \\\n const int _PCHECK_err_no_ = errno; \\\n logstream(LOG_ERROR) \\\n << \"Check failed: \" << #condition << \": \" \\\n << strerror(err_no) << std::endl; \\\n __print_back_trace(); \\\n throw(\"assertion failure\"); \\\n } \\\n } while(0)\n\n// Helper macro for binary operators; prints the two values on error\n// Don't use this macro directly in your code, use CHECK_EQ et al below\n\n// WARNING: These don't compile correctly if one of the arguments is a pointer\n// and the other is NULL. To work around this, simply static_cast NULL to the\n// type of the desired pointer.\n#if defined(__cplusplus) && __cplusplus >= 201103L\n#define CHECK_OP(op, val1, val2) \\\n do { \\\n const auto _CHECK_OP_v1_ = val1; \\\n const auto _CHECK_OP_v2_ = val2; \\\n if (__builtin_expect(!((_CHECK_OP_v1_) op \\\n (decltype(val1))(_CHECK_OP_v2_)), 0)) { \\\n logstream(LOG_ERROR) \\\n << \"Check failed: \" \\\n << #val1 << #op << #val2 \\\n << \" [\" \\\n << _CHECK_OP_v1_ \\\n << ' ' << #op << ' ' \\\n << _CHECK_OP_v2_ << \"]\" << std::endl; \\\n __print_back_trace(); \\\n throw(\"assertion failure\"); \\\n } \\\n } while(0)\n#else\n#define CHECK_OP(op, val1, val2) \\\n do { \\\n const typeof(val1) _CHECK_OP_v1_ = (typeof(val1))val1; \\\n const typeof(val2) _CHECK_OP_v2_ = (typeof(val2))val2; \\\n if (__builtin_expect(!((_CHECK_OP_v1_) op \\\n (typeof(val1))(_CHECK_OP_v2_)), 0)) { \\\n logstream(LOG_ERROR) \\\n << \"Check failed: \" \\\n << #val1 << #op << #val2 \\\n << \" [\" \\\n << _CHECK_OP_v1_ \\\n << ' ' << #op << ' ' \\\n << _CHECK_OP_v2_ << \"]\" << std::endl; \\\n __print_back_trace(); \\\n throw(\"assertion failure\"); \\\n } \\\n } while(0)\n#endif\n\n#define CHECK_EQ(val1, val2) CHECK_OP(==, val1, val2)\n#define CHECK_NE(val1, val2) CHECK_OP(!=, val1, val2)\n#define CHECK_LE(val1, val2) CHECK_OP(<=, val1, val2)\n#define CHECK_LT(val1, val2) CHECK_OP(< , val1, val2)\n#define CHECK_GE(val1, val2) CHECK_OP(>=, val1, val2)\n#define CHECK_GT(val1, val2) CHECK_OP(> , val1, val2)\n\n// Synonyms for CHECK_* that are used in some unittests.\n#define EXPECT_EQ(val1, val2) CHECK_EQ(val1, val2)\n#define EXPECT_NE(val1, val2) CHECK_NE(val1, val2)\n#define EXPECT_LE(val1, val2) CHECK_LE(val1, val2)\n#define EXPECT_LT(val1, val2) CHECK_LT(val1, val2)\n#define EXPECT_GE(val1, val2) CHECK_GE(val1, val2)\n#define EXPECT_GT(val1, val2) CHECK_GT(val1, val2)\n#define ASSERT_EQ(val1, val2) EXPECT_EQ(val1, val2)\n#define ASSERT_NE(val1, val2) EXPECT_NE(val1, val2)\n#define ASSERT_LE(val1, val2) EXPECT_LE(val1, val2)\n#define ASSERT_LT(val1, val2) EXPECT_LT(val1, val2)\n#define ASSERT_GE(val1, val2) EXPECT_GE(val1, val2)\n#define ASSERT_GT(val1, val2) EXPECT_GT(val1, val2)\n// As are these variants.\n#define EXPECT_TRUE(cond) CHECK(cond)\n#define EXPECT_FALSE(cond) CHECK(!(cond))\n#define EXPECT_STREQ(a, b) CHECK(strcmp(a, b) == 0)\n#define ASSERT_TRUE(cond) EXPECT_TRUE(cond)\n#define ASSERT_FALSE(cond) EXPECT_FALSE(cond)\n#define ASSERT_STREQ(a, b) EXPECT_STREQ(a, b)\n\n\n#define ASSERT_MSG(condition, fmt, ...) \\\n do { \\\n if (__builtin_expect(!(condition), 0)) { \\\n logstream(LOG_ERROR) \\\n << \"Check failed: \" << #condition << \":\\n\"; \\\n logger(LOG_ERROR, fmt, ##__VA_ARGS__); \\\n __print_back_trace(); \\\n throw(\"assertion failure\"); \\\n } \\\n } while(0)\n\n// Used for (libc) functions that return -1 and set errno\n#define CHECK_ERR(invocation) PCHECK((invocation) != -1)\n\n// A few more checks that only happen in debug mode\n#ifdef NDEBUG\n#define DCHECK_EQ(val1, val2)\n#define DCHECK_NE(val1, val2)\n#define DCHECK_LE(val1, val2)\n#define DCHECK_LT(val1, val2)\n#define DCHECK_GE(val1, val2)\n#define DCHECK_GT(val1, val2)\n#define DASSERT_TRUE(cond)\n#define DASSERT_FALSE(cond)\n#define DASSERT_MSG(condition, fmt, ...)\n\n#else\n#define DCHECK_EQ(val1, val2) CHECK_EQ(val1, val2)\n#define DCHECK_NE(val1, val2) CHECK_NE(val1, val2)\n#define DCHECK_LE(val1, val2) CHECK_LE(val1, val2)\n#define DCHECK_LT(val1, val2) CHECK_LT(val1, val2)\n#define DCHECK_GE(val1, val2) CHECK_GE(val1, val2)\n#define DCHECK_GT(val1, val2) CHECK_GT(val1, val2)\n#define DASSERT_TRUE(cond) ASSERT_TRUE(cond)\n#define DASSERT_FALSE(cond) ASSERT_FALSE(cond)\n#define DASSERT_MSG(condition, fmt, ...) \\\n do { \\\n if (__builtin_expect(!(condition), 0)) { \\\n logstream(LOG_ERROR) \\\n << \"Check failed: \" << #condition << \":\\n\"; \\\n logger(LOG_ERROR, fmt, ##__VA_ARGS__); \\\n __print_back_trace(); \\\n throw(\"assertion failure\"); \\\n } \\\n } while(0)\n\n#endif\n\n\n#ifdef ERROR\n#undef ERROR // may conflict with ERROR macro on windows\n#endif\n\n#endif // _LOGGING_H_\n\n" }, { "path": "src/graphlab/logger/backtrace.cpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <execinfo.h>\n#include <iostream>\n#include <stdlib.h>\n#include <stdio.h>\n#include <string.h>\n#include <unistd.h>\n#include <cxxabi.h>\n#include <pthread.h>\n\n/** Code from http://mykospark.net/2009/09/runtime-backtrace-in-c-with-name-demangling/ */\nstd::string demangle(const char* symbol) {\n size_t size;\n int status;\n char temp[1024];\n char* demangled;\n //first, try to demangle a c++ name\n if (1 == sscanf(symbol, \"%*[^(]%*[^_]%127[^)+]\", temp)) {\n if (NULL != (demangled = abi::__cxa_demangle(temp, NULL, &size, &status))) {\n std::string result(demangled);\n free(demangled);\n return result;\n }\n }\n //if that didn't work, try to get a regular c symbol\n if (1 == sscanf(symbol, \"%127s\", temp)) {\n return temp;\n }\n\n //if all else fails, just return the symbol\n return symbol;\n}\n\n\nstatic pthread_mutex_t back_trace_file_lock = PTHREAD_MUTEX_INITIALIZER;\nstatic size_t write_count = 0;\nstatic bool write_error = 0;\nstatic int backtrace_file_number = 0;\n\n\nextern void __set_back_trace_file_number(int number) {\n backtrace_file_number = number;\n}\n\n/* Obtain a backtrace and print it to ofile. */\nvoid __print_back_trace() {\n void *array[1024];\n size_t size, i;\n char **strings;\n\n pthread_mutex_lock(&back_trace_file_lock);\n\n if (write_error) {\n pthread_mutex_unlock(&back_trace_file_lock);\n return;\n }\n char filename[1024];\n sprintf(filename, \"backtrace.%d\", backtrace_file_number);\n\n FILE* ofile = NULL;\n if (write_count == 0) {\n ofile = fopen(filename, \"w\");\n }\n else {\n ofile = fopen(filename, \"a\");\n }\n // if unable to open the file for output\n if (ofile == NULL) {\n // print an error, set the error flag so we don't ever print it again\n fprintf(stderr, \"Unable to open output backtrace file.\\n\");\n write_error = 1;\n pthread_mutex_unlock(&back_trace_file_lock);\n return;\n }\n ++write_count;\n\n size = backtrace(array, 1024);\n strings = backtrace_symbols(array, size);\n\n fprintf(ofile, \"Pointers\\n\");\n fprintf(ofile, \"------------\\n\");\n for (i = 0; i < size; ++i) {\n fprintf(ofile, \"%p\\n\", array[i]);\n }\n\n\n fprintf(ofile, \"Raw\\n\");\n fprintf(ofile, \"------------\\n\");\n for (i = 0; i < size; ++i) {\n fprintf(ofile, \"%s\\n\", strings[i]);\n }\n fprintf(ofile, \"\\nDemangled\\n\");\n fprintf(ofile, \"------------\\n\");\n\n for (i = 0; i < size; ++i) {\n std::string ret = demangle(strings[i]);\n fprintf(ofile, \"%s\\n\", ret.c_str());\n }\n free(strings);\n\n fprintf(ofile, \"-------------------------------------------------------\\n\");\n fprintf(ofile, \"\\n\\n\");\n\n fclose(ofile);\n pthread_mutex_unlock(&back_trace_file_lock);\n}\n\n" }, { "path": "src/graphlab/logger/backtrace.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_BACKTRACE_HPP\n#define GRAPHLAB_BACKTRACE_HPP\n\nextern void __set_back_trace_file_number(int number);\nextern void __print_back_trace();\n\n#endif\n" }, { "path": "src/graphlab/logger/fail_method.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_LOGGER_FAIL_METHOD\n\n#ifdef GRAPHLAB_LOGGER_THROW_ON_FAILURE\n#define GRAPHLAB_LOGGER_FAIL_METHOD(str) throw(str)\n#else\n#define GRAPHLAB_LOGGER_FAIL_METHOD(str) abort()\n#endif\n\n#endif\n" }, { "path": "src/graphlab/logger/logger.cpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <graphlab/logger/logger.hpp>\n#include <cstdarg>\n#include <cstdio>\n#include <cstring>\n#include <fstream>\n#include <iostream>\n#include <pthread.h>\n#include <graphlab/logger/backtrace.hpp>\n\nfile_logger& global_logger() {\n static file_logger l;\n return l;\n}\n\n\n\n\nvoid streambuffdestructor(void* v){\n logger_impl::streambuff_tls_entry* t =\n reinterpret_cast<logger_impl::streambuff_tls_entry*>(v);\n delete t;\n}\n\nconst char* messages[] = { \"DEBUG: \",\n \"DEBUG: \",\n \"INFO: \",\n \"INFO: \",\n \"WARNING: \",\n \"ERROR: \",\n \"FATAL: \"};\n\n\nfile_logger::file_logger() {\n log_file = \"\";\n log_to_console = true;\n log_level = LOG_EMPH;\n pthread_mutex_init(&mut, NULL);\n pthread_key_create(&streambuffkey, streambuffdestructor);\n}\n\nfile_logger::~file_logger() {\n if (fout.good()) {\n fout.flush();\n fout.close();\n }\n\n pthread_mutex_destroy(&mut);\n}\n\nbool file_logger::set_log_file(std::string file) {\n // close the file if it is open\n if (fout.good()) {\n fout.flush();\n fout.close();\n log_file = \"\";\n }\n // if file is not an empty string, open the new file\n if (file.length() > 0) {\n fout.open(file.c_str());\n if (fout.fail()) return false;\n log_file = file;\n }\n return true;\n}\n\n\n\n#define RESET 0\n#define BRIGHT 1\n#define DIM 2\n#define UNDERLINE 3\n#define BLINK 4\n#define REVERSE 7\n#define HIDDEN 8\n\n#define BLACK 0\n#define RED 1\n#define GREEN 2\n#define YELLOW 3\n#define BLUE 4\n#define MAGENTA 5\n#define CYAN 6\n#define WHITE 7\n\nvoid textcolor(FILE* handle, int attr, int fg)\n{\n char command[13];\n /* Command is the control command to the terminal */\n sprintf(command, \"%c[%d;%dm\", 0x1B, attr, fg + 30);\n fprintf(handle, \"%s\", command);\n}\n\nvoid reset_color(FILE* handle)\n{\n char command[20];\n /* Command is the control command to the terminal */\n sprintf(command, \"%c[0m\", 0x1B);\n fprintf(handle, \"%s\", command);\n}\n\n\n\nvoid file_logger::_log(int lineloglevel,const char* file,const char* function,\n int line,const char* fmt, va_list ap ){\n // if the logger level fits\n if (lineloglevel >= log_level){\n // get just the filename. this line found on a forum on line.\n // claims to be from google.\n file = ((strrchr(file, '/') ? : file- 1) + 1);\n\n char str[1024];\n\n // write the actual header\n int byteswritten = snprintf(str,1024, \"%s%s(%s:%d): \",\n messages[lineloglevel],file,function,line);\n // write the actual logger\n\n byteswritten += vsnprintf(str + byteswritten,1024 - byteswritten,fmt,ap);\n\n str[byteswritten] = '\\n';\n str[byteswritten+1] = 0;\n // write the output\n if (fout.good()) {\n pthread_mutex_lock(&mut);\n fout << str;;\n pthread_mutex_unlock(&mut);\n }\n if (log_to_console) {\n#ifdef COLOROUTPUT\n if (lineloglevel == LOG_FATAL) {\n textcolor(stderr, BRIGHT, RED);\n }\n else if (lineloglevel == LOG_ERROR) {\n textcolor(stderr, BRIGHT, RED);\n }\n else if (lineloglevel == LOG_WARNING) {\n textcolor(stderr, BRIGHT, MAGENTA);\n }\n else if (lineloglevel == LOG_EMPH) {\n textcolor(stderr, BRIGHT, GREEN);\n }\n#endif\n std::cerr << str;;\n#ifdef COLOROUTPUT\n reset_color(stderr);\n#endif\n }\n }\n}\n\n\n\nvoid file_logger::_logbuf(int lineloglevel,const char* file,const char* function,\n int line,const char* buf, int len) {\n // if the logger level fits\n if (lineloglevel >= log_level){\n // get just the filename. this line found on a forum on line.\n // claims to be from google.\n file = ((strrchr(file, '/') ? : file- 1) + 1);\n\n // length of the 'head' of the string\n size_t headerlen = snprintf(NULL,0,\"%s%s(%s:%d): \",\n messages[lineloglevel],file,function,line);\n\n if (headerlen> 2047) {\n std::cerr << \"Header length exceed buffer length!\";\n }\n else {\n char str[2048];\n const char *newline=\"\\n\";\n // write the actual header\n int byteswritten = snprintf(str,2047,\"%s%s(%s:%d): \",\n messages[lineloglevel],file,function,line);\n _lograw(lineloglevel,str, byteswritten);\n _lograw(lineloglevel,buf, len);\n _lograw(lineloglevel,newline, (int)strlen(newline));\n }\n }\n}\n\nvoid file_logger::_lograw(int lineloglevel, const char* buf, int len) {\n if (fout.good()) {\n pthread_mutex_lock(&mut);\n fout.write(buf,len);\n pthread_mutex_unlock(&mut);\n }\n if (log_to_console) {\n#ifdef COLOROUTPUT\n pthread_mutex_lock(&mut);\n if (lineloglevel == LOG_FATAL) {\n textcolor(stderr, BRIGHT, RED);\n }\n else if (lineloglevel == LOG_ERROR) {\n textcolor(stderr, BRIGHT, RED);\n }\n else if (lineloglevel == LOG_WARNING) {\n textcolor(stderr, BRIGHT, MAGENTA);\n }\n else if (lineloglevel == LOG_DEBUG) {\n textcolor(stderr, BRIGHT, YELLOW);\n }\n else if (lineloglevel == LOG_EMPH) {\n textcolor(stderr, BRIGHT, GREEN);\n }\n#endif\n std::cerr.write(buf,len);\n#ifdef COLOROUTPUT\n\n pthread_mutex_unlock(&mut);\n reset_color(stderr);\n#endif\n }\n}\n\nfile_logger& file_logger::start_stream(int lineloglevel,const char* file,\n const char* function, int line, bool do_start) {\n // get the stream buffer\n logger_impl::streambuff_tls_entry* streambufentry =\n reinterpret_cast<logger_impl::streambuff_tls_entry*>(\n pthread_getspecific(streambuffkey));\n // create the key if it doesn't exist\n if (streambufentry == NULL) {\n streambufentry = new logger_impl::streambuff_tls_entry;\n pthread_setspecific(streambuffkey, streambufentry);\n }\n std::stringstream& streambuffer = streambufentry->streambuffer;\n bool& streamactive = streambufentry->streamactive;\n\n if (lineloglevel >= log_level){\n // get the stream buffer\n // if do not start the stream, just quit\n if (do_start == false) {\n streamactive = false;\n return *this;\n }\n\n file = ((strrchr(file, '/') ? : file- 1) + 1);\n\n if (streambuffer.str().length() == 0) {\n streambuffer << messages[lineloglevel] << file\n << \"(\" << function << \":\" <<line<<\"): \";\n }\n streamactive = true;\n streamloglevel = lineloglevel;\n } else {\n streamactive = false;\n }\n return *this;\n}\n\n" }, { "path": "src/graphlab/logger/logger.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n/**\n * @file logger.hpp\n * Usage:\n * First include logger.hpp. To logger, use the logger() function\n * There are 2 output levels. A \"soft\" output level which is\n * set by calling global_logger.set_log_level(), as well as a \"hard\" output\n * level OUTPUTLEVEL which is set in the source code (logger.h).\n *\n * when you call \"logger()\" with a loglevel and if the loglevel is greater than\n * both of the output levels, the string will be written.\n * written to a logger file. Otherwise, logger() has no effect.\n *\n * The difference between the hard level and the soft level is that the\n * soft level can be changed at runtime, while the hard level optimizes away\n * logging calls at compile time.\n */\n\n#ifndef GRAPHLAB_LOG_LOG_HPP\n#define GRAPHLAB_LOG_LOG_HPP\n#include <fstream>\n#include <sstream>\n#include <cstdlib>\n#include <cassert>\n#include <cstring>\n#include <cstdarg>\n#include <pthread.h>\n#include <graphlab/util/timer.hpp>\n#include <graphlab/logger/fail_method.hpp>\n#include <graphlab/logger/backtrace.hpp>\n\n/**\n * \\def LOG_FATAL\n * Used for fatal and probably irrecoverable conditions\n * \\def LOG_ERROR\n * Used for errors which are recoverable within the scope of the function\n * \\def LOG_WARNING\n * Logs interesting conditions which are probably not fatal\n * \\def LOG_EMPH\n * Outputs as LOG_INFO, but in LOG_WARNING colors. Useful for\n * outputting information you want to emphasize.\n * \\def LOG_INFO\n * Used for providing general useful information\n * \\def LOG_DEBUG\n * Debugging purposes only\n * \\def LOG_EVERYTHING\n * Log everything \n */\n// sgr - needed additional debug levels. I can undo this change if \n// necessary. although it seems to me that log levels should count \n// up and saturate so the messages label array can always be used.\n#define LOG_NONE 7\n#define LOG_FATAL 6\n#define LOG_ERROR 5\n#define LOG_WARNING 4\n#define LOG_EMPH 3\n#define LOG_INFO 2\n#define LOG_DEBUG 1\n#define LOG_EVERYTHING 0 // technically unsigned int\n\n/**\n * \\def OUTPUTLEVEL\n * The minimum level to logger at\n * \\def LOG_NONE\n * OUTPUTLEVEL to LOG_NONE to disable logging\n */\n\n#ifndef OUTPUTLEVEL\n#define OUTPUTLEVEL LOG_DEBUG\n#endif\n/// If set, logs to screen will be printed in color\n#define COLOROUTPUT\n\n\n/**\n * \\def logger(lvl,fmt,...)\n * extracts the filename, line number\n * and function name and calls _log. It will be optimized\n * away if LOG_NONE is set\n * This relies on a few compiler macros. As far as I know, these\n * macros are pretty standard among most other C++ compilers.\n */\n#if OUTPUTLEVEL == LOG_NONE\n// totally disable logging\n#define logger(lvl,fmt,...)\n#define logbuf(lvl,fmt,...)\n#define logstream(lvl) if(0) null_stream()\n\n#define logger_once(lvl,fmt,...)\n#define logstream_once(lvl) if(0) null_stream()\n\n#define logger_ontick(sec,lvl,fmt,...)\n#define logstream_ontick(sec, lvl) if(0) null_stream()\n\n#else\n\n#define logger(lvl,fmt,...) \\\n (log_dispatch<(lvl >= OUTPUTLEVEL)>::exec(lvl,__FILE__, __func__ ,__LINE__,fmt,##__VA_ARGS__))\n\n\n#define logbuf(lvl,buf,len) \\\n (log_dispatch<(lvl >= OUTPUTLEVEL)>::exec(lvl,__FILE__, \\\n __func__ ,__LINE__,buf,len))\n\n#define logstream(lvl) \\\n if(lvl >= global_logger().get_log_level()) (log_stream_dispatch<(lvl >= OUTPUTLEVEL)>::exec(lvl,__FILE__, __func__ ,__LINE__) )\n\n#define logger_once(lvl,fmt,...) \\\n{ \\\n static bool __printed__ = false; \\\n if (!__printed__) { \\\n __printed__ = true; \\\n (log_dispatch<(lvl >= OUTPUTLEVEL)>::exec(lvl,__FILE__, __func__ ,__LINE__,fmt,##__VA_ARGS__)); \\\n } \\\n}\n\n#define logstream_once(lvl) \\\n(*({ \\\n static bool __printed__ = false; \\\n bool __prev_printed__ = __printed__; \\\n if (!__printed__) __printed__ = true; \\\n &(log_stream_dispatch<(lvl >= OUTPUTLEVEL)>::exec(lvl,__FILE__, __func__ ,__LINE__, !__prev_printed__) ); \\\n}))\n\n#define logger_ontick(sec,lvl,fmt,...) \\\n{ \\\n static float last_print = -sec - 1; \\\n float curtime = graphlab::timer::approx_time_seconds(); \\\n if (last_print + sec <= curtime) { \\\n last_print = curtime; \\\n (log_dispatch<(lvl >= OUTPUTLEVEL)>::exec(lvl,__FILE__, __func__ ,__LINE__,fmt,##__VA_ARGS__)); \\\n } \\\n}\n\n#define logstream_ontick(sec,lvl) \\\n(*({ \\\n static float last_print = -sec - 1; \\\n float curtime = graphlab::timer::approx_time_seconds(); \\\n bool print_now = false; \\\n if (last_print + sec <= curtime) { \\\n last_print = curtime; \\\n print_now = true; \\\n } \\\n &(log_stream_dispatch<(lvl >= OUTPUTLEVEL)>::exec(lvl,__FILE__, __func__ ,__LINE__, print_now) ); \\\n}))\n\n\n#endif\n\nnamespace logger_impl {\nstruct streambuff_tls_entry {\n std::stringstream streambuffer;\n bool streamactive;\n};\n}\n\n\nextern void __print_back_trace();\n\n/**\n logging class.\n This writes to a file, and/or the system console.\n*/\nclass file_logger{\n public:\n /** Default constructor. By default, log_to_console is on,\n there is no logger file, and logger level is set to LOG_EMPH\n */\n file_logger();\n\n ~file_logger(); /// destructor. flushes and closes the current logger file\n\n /** Closes the current logger file if one exists.\n if 'file' is not an empty string, it will be opened and\n all subsequent logger output will be written into 'file'.\n Any existing content of 'file' will be cleared.\n Return true on success and false on failure.\n */\n bool set_log_file(std::string file);\n\n /// If consolelog is true, subsequent logger output will be written to stderr\n void set_log_to_console(bool consolelog) {\n log_to_console = consolelog;\n }\n\n /// Returns the current logger file.\n std::string get_log_file(void) {\n return log_file;\n }\n\n /// Returns true if output is being written to stderr\n bool get_log_to_console() {\n return log_to_console;\n }\n\n /// Returns the current logger level\n int get_log_level() {\n return log_level;\n }\n\n file_logger& start_stream(int lineloglevel,const char* file,const char* function, int line, bool do_start = true);\n\n template <typename T>\n file_logger& operator<<(T a) {\n // get the stream buffer\n logger_impl::streambuff_tls_entry* streambufentry = reinterpret_cast<logger_impl::streambuff_tls_entry*>(\n pthread_getspecific(streambuffkey));\n if (streambufentry != NULL) {\n std::stringstream& streambuffer = streambufentry->streambuffer;\n bool& streamactive = streambufentry->streamactive;\n\n if (streamactive) streambuffer << a;\n }\n return *this;\n }\n\n inline file_logger& operator<<(const char* a) {\n // get the stream buffer\n logger_impl::streambuff_tls_entry* streambufentry = reinterpret_cast<logger_impl::streambuff_tls_entry*>(\n pthread_getspecific(streambuffkey));\n if (streambufentry != NULL) {\n std::stringstream& streambuffer = streambufentry->streambuffer;\n bool& streamactive = streambufentry->streamactive;\n\n if (streamactive) {\n streambuffer << a;\n if (a[strlen(a)-1] == '\\n') {\n stream_flush();\n }\n }\n }\n return *this;\n }\n\n inline file_logger& operator<<(std::ostream& (*f)(std::ostream&)){\n // get the stream buffer\n logger_impl::streambuff_tls_entry* streambufentry = reinterpret_cast<logger_impl::streambuff_tls_entry*>(\n pthread_getspecific(streambuffkey));\n if (streambufentry != NULL) {\n std::stringstream& streambuffer = streambufentry->streambuffer;\n bool& streamactive = streambufentry->streamactive;\n\n typedef std::ostream& (*endltype)(std::ostream&);\n if (streamactive) {\n if (endltype(f) == endltype(std::endl)) {\n streambuffer << \"\\n\";\n stream_flush();\n if(streamloglevel == LOG_FATAL) {\n __print_back_trace();\n GRAPHLAB_LOGGER_FAIL_METHOD(\"LOG_FATAL encountered\");\n }\n }\n }\n }\n return *this;\n }\n\n\n\n /** Sets the current logger level. All logging commands below the current\n logger level will not be written. */\n void set_log_level(int new_log_level) {\n log_level = new_log_level;\n }\n\n /**\n * logs the message if loglevel>=OUTPUTLEVEL\n * This function should not be used directly. Use logger()\n *\n * @param loglevel Type of message \\see LOG_DEBUG LOG_INFO LOG_WARNING LOG_ERROR LOG_FATAL\n * @param file File where the logger call originated\n * @param function Function where the logger call originated\n * @param line Line number where the logger call originated\n * @param fmt printf format string\n * @param arg var args. The parameters that match the format string\n */\n void _log(int loglevel,const char* file,const char* function,\n int line,const char* fmt, va_list arg );\n\n\n void _logbuf(int loglevel,const char* file,const char* function,\n int line, const char* buf, int len);\n\n void _lograw(int loglevel, const char* buf, int len);\n\n inline void stream_flush() {\n // get the stream buffer\n logger_impl::streambuff_tls_entry* streambufentry = reinterpret_cast<logger_impl::streambuff_tls_entry*>(\n pthread_getspecific(streambuffkey));\n if (streambufentry != NULL) {\n std::stringstream& streambuffer = streambufentry->streambuffer;\n\n streambuffer.flush();\n _lograw(streamloglevel,\n streambuffer.str().c_str(),\n (int)(streambuffer.str().length()));\n streambuffer.str(\"\");\n }\n }\n private:\n std::ofstream fout;\n std::string log_file;\n\n pthread_key_t streambuffkey;\n\n int streamloglevel;\n pthread_mutex_t mut;\n\n bool log_to_console;\n int log_level;\n\n};\n\n\nfile_logger& global_logger();\n\n/**\nWrapper to generate 0 code if the output level is lower than the log level\n*/\ntemplate <bool dostuff>\nstruct log_dispatch {};\n\ntemplate <>\nstruct log_dispatch<true> {\n inline static void exec(int loglevel,const char* file,const char* function,\n int line,const char* fmt, ... ) {\n va_list argp;\n va_start(argp, fmt);\n global_logger()._log(loglevel, file, function, line, fmt, argp);\n va_end(argp);\n if(loglevel == LOG_FATAL) {\n __print_back_trace();\n GRAPHLAB_LOGGER_FAIL_METHOD(\"LOG_FATAL encountered\");\n }\n }\n};\n\ntemplate <>\nstruct log_dispatch<false> {\n inline static void exec(int loglevel,const char* file,const char* function,\n int line,const char* fmt, ... ) {}\n};\n\n\nstruct null_stream {\n template<typename T>\n inline null_stream operator<<(T t) { return null_stream(); }\n inline null_stream operator<<(const char* a) { return null_stream(); }\n inline null_stream operator<<(std::ostream& (*f)(std::ostream&)) { return null_stream(); }\n};\n\n\ntemplate <bool dostuff>\nstruct log_stream_dispatch {};\n\ntemplate <>\nstruct log_stream_dispatch<true> {\n inline static file_logger& exec(int lineloglevel,const char* file,const char* function, int line, bool do_start = true) {\n return global_logger().start_stream(lineloglevel, file, function, line, do_start);\n }\n};\n\ntemplate <>\nstruct log_stream_dispatch<false> {\n inline static null_stream exec(int lineloglevel,const char* file,const char* function, int line, bool do_start = true) {\n return null_stream();\n }\n};\n\nvoid textcolor(FILE* handle, int attr, int fg);\nvoid reset_color(FILE* handle);\n\n#endif\n\n" }, { "path": "src/graphlab/logger/logger_includes.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n#include <graphlab/logger/assertions.hpp>\n#include <graphlab/logger/logger.hpp>\n\n" }, { "path": "src/graphlab/macros_def.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <boost/foreach.hpp>\n#include <stdint.h>\n\n// if GNUC is available, this checks if the file which included\n// macros_def.hpp is the same file which included macros_undef.hpp\n#ifdef __GNUC__\n#define GRAPHLAB_MACROS_INC_LEVEL __INCLUDE_LEVEL__\n#endif\n\n\n// prevent this file from being included before other graphlab headers\n#ifdef GRAPHLAB_MACROS\n#error \"Repeated include of <macros_def.hpp>. This probably means that macros_def.hpp was not the last include, or some header file failed to include <macros_undef.hpp>\"\n#endif\n\n#define GRAPHLAB_MACROS\n\n/** A macro to disallow the copy constructor and operator= functions\n This should be used in the private: declarations for a class */\n#define DISALLOW_COPY_AND_ASSIGN(TypeName) \\\nTypeName(const TypeName&); \\\nvoid operator=(const TypeName&);\n\n\n\n// Shortcut macro definitions\n//! see http://www.boost.org/doc/html/foreach.html \n#define foreach BOOST_FOREACH\n\n#define rev_foreach BOOST_REVERSE_FOREACH\n\n\n\n\n" }, { "path": "src/graphlab/macros_undef.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifdef __GNUC__\n#if (GRAPHLAB_MACROS_INC_LEVEL != __INCLUDE_LEVEL__)\n #error \"A <macros_def.hpp> was not paired with a <macros_undef.hpp>\" \n#endif\n#undef GRAPHLAB_MACROS_INC_LEVEL\n#endif\n\n\n#undef GRAPHLAB_MACROS\n#undef DISALLOW_COPY_AND_ASSIGN\n#undef foreach\n#undef rev_foreach\n\n" }, { "path": "src/graphlab/options/CMakeLists.txt", "content": "project(GraphLab)\n\n\n\n\n" }, { "path": "src/graphlab/options/command_line_options.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n#include <graphlab/options/command_line_options.hpp>\n#include <graphlab/scheduler/scheduler_list.hpp>\n\n\nnamespace boost { \n\n template<>\n std::string lexical_cast< std::string>(const std::vector<int>& vec) {\n return graphlab_vec_to_string(vec);\n }\n\n template<>\n std::string lexical_cast<std::string>(const std::vector<uint32_t>& vec) {\n return graphlab_vec_to_string(vec);\n }\n\n template<>\n std::string lexical_cast<std::string>(const std::vector<uint64_t>& vec) {\n return graphlab_vec_to_string(vec);\n }\n\n template<>\n std::string lexical_cast< std::string >(const std::vector<double>& vec) {\n return graphlab_vec_to_string(vec);\n }\n\n template<>\n std::string lexical_cast< std::string>(const std::vector<float>& vec) {\n return graphlab_vec_to_string(vec);\n }\n\n template<>\n std::string lexical_cast< std::string>(const std::vector<std::string>& vec) {\n return graphlab_vec_to_string(vec);\n }\n};\n\n\nnamespace graphlab {\n \nstatic const char* engine_help_string = \n#include <graphlab/options/engine_help.txt>\n;\n\nstatic const char* graph_help_string = \n#include <graphlab/options/graph_help.txt>\n;\n\n\n\n bool command_line_options::parse(int argc, const char* const* argv,\n bool allow_unregistered) {\n namespace boost_po = boost::program_options;\n \n size_t ncpus(get_ncpus());\n std::string engine_opts_string;\n std::string schedulertype(get_scheduler_type());\n std::string scheduler_opts_string = \"\";\n std::string graph_opts_string = \"\";\n\n if(!suppress_graphlab_options) {\n // Set the program options\n desc.add_options()\n (\"ncpus\",\n boost_po::value<size_t>(&(ncpus))->\n default_value(ncpus),\n \"Number of cpus to use per machine. Defaults to (#cores - 2)\")\n (\"scheduler\",\n boost_po::value<std::string>(&(schedulertype))->\n default_value(schedulertype),\n (std::string(\"Supported schedulers are: \"\n + get_scheduler_names_str() +\n \". Too see options for each scheduler, run the program with the option\"\n \" ---schedhelp=[scheduler_name]\").c_str()))\n (\"engine_opts\",\n boost_po::value<std::string>(&(engine_opts_string))->\n default_value(engine_opts_string),\n \"string of engine options i.e., \\\"timeout=100\\\"\")\n (\"graph_opts\",\n boost_po::value<std::string>(&(graph_opts_string))->\n default_value(graph_opts_string),\n \"String of graph options i.e., \\\"ingress=random\\\"\")\n (\"scheduler_opts\",\n boost_po::value<std::string>(&(scheduler_opts_string))->\n default_value(scheduler_opts_string),\n \"String of scheduler options i.e., \\\"strict=true\\\"\")\n (\"engine_help\",\n boost_po::value<std::string>()->implicit_value(\"\"),\n \"Display help for engine options.\")\n (\"graph_help\",\n boost_po::value<std::string>()->implicit_value(\"\"),\n \"Display help for the distributed graph.\")\n (\"scheduler_help\",\n boost_po::value<std::string>()->implicit_value(\"\"),\n \"Display help for schedulers.\");\n }\n // Parse the arguments\n try {\n std::vector<std::string> arguments;\n std::copy(argv + 1, argv + argc + !argc, \n std::inserter(arguments, arguments.end()));\n\n boost_po::command_line_parser parser(arguments);\n parser.options(desc);\n if (allow_unregistered) parser.allow_unregistered();\n if (num_positional) parser.positional(pos_opts);\n boost_po::parsed_options parsed = parser.run();\n if (allow_unregistered) {\n unrecognized_options = \n boost_po::collect_unrecognized(parsed.options, \n boost_po::include_positional);\n } else {\n unrecognized_options.clear();\n }\n boost_po::store(parsed, vm);\n boost_po::notify(vm);\n } catch( boost_po::error error) {\n std::cout << \"Invalid syntax:\\n\"\n << \"\\t\" << error.what()\n << \"\\n\\n\" << std::endl\n << \"Description:\"\n << std::endl;\n print_description();\n return false;\n }\n if(vm.count(\"help\")) {\n print_description();\n return false;\n }\n if (vm.count(\"scheduler_help\")) {\n std::string schedname = vm[\"scheduler_help\"].as<std::string>();\n if (schedname != \"\") {\n print_scheduler_info(schedname, std::cout);\n } else {\n std::vector<std::string> schednames = get_scheduler_names();\n for(size_t i = 0;i < schednames.size(); ++i) {\n print_scheduler_info(schednames[i], std::cout);\n }\n }\n return false;\n } \n if (vm.count(\"engine_help\")) {\n std::cout << engine_help_string; \n return false;\n }\n if (vm.count(\"graph_help\")) {\n std::cout << graph_help_string; \n return false;\n } \n set_ncpus(ncpus);\n\n set_scheduler_type(schedulertype);\n\n get_scheduler_args().parse_string(scheduler_opts_string);\n get_engine_args().parse_string(engine_opts_string);\n get_graph_args().parse_string(graph_opts_string);\n return true;\n } // end of parse\n\n\n bool command_line_options::is_set(const std::string& option) {\n return vm.count(option);\n }\n\n void command_line_options::add_positional(const std::string& str) {\n num_positional++;\n pos_opts.add(str.c_str(), 1);\n }\n}\n\n" }, { "path": "src/graphlab/options/command_line_options.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_COMMAND_LINE_OPTIONS\n#define GRAPHLAB_COMMAND_LINE_OPTIONS\n\n#include <string>\n#include <vector>\n\n\n#include <boost/program_options.hpp>\n\n\n#include <graphlab/options/graphlab_options.hpp>\n\n\nnamespace boost {\n /**\n \\ingroup util\n Converts a vector of any stream output-able type to a string\n */\n template<typename T>\n std::string graphlab_vec_to_string(const std::vector<T>& vec) {\n std::stringstream strm;\n strm << \"{\" ;\n for(size_t i = 0; i < vec.size(); ++i) {\n strm << vec[i];\n if(i < vec.size() - 1) strm << \", \";\n }\n strm << \"}\";\n return strm.str();\n }\n \n /**\n \\ingroup util\n Provides lexical cast from vector<int> to string.\n Converts a vector of 1,2,3 to the string \"{1, 2, 3}\"\n */ \n template<>\n std::string lexical_cast< std::string>(const std::vector<int>& vec);\n\n /**\n \\ingroup util\n Provides lexical cast from vector<int> to string.\n Converts a vector of 1,2,3 to the string \"{1, 2, 3}\"\n */ \n template<>\n std::string lexical_cast< std::string>(const std::vector<uint32_t>& vec);\n\n /**\n \\ingroup util\n Provides lexical cast from vector<size_t> to string.\n Converts a vector of 1,2,3 to the string \"{1, 2, 3}\"\n */\n template<>\n std::string lexical_cast<std::string>(const std::vector<uint64_t>& vec);\n\n \n /**\n \\ingroup util\n Provides lexical cast from vector<double> to string.\n Converts a vector of 1.1,2.2,3.3 to the string \"{1.1, 2.2, 3.3}\"\n */\n template<>\n std::string lexical_cast< std::string >(const std::vector<double>& vec);\n \n /**\n \\ingroup util\n Provides lexical cast from vector<float> to string.\n Converts a vector of 1.1,2.2,3.3 to the string \"{1.1, 2.2, 3.3}\"\n */\n template<>\n std::string lexical_cast< std::string>(const std::vector<float>& vec);\n \n /**\n \\ingroup util\n Provides lexical cast from vector<string> to string.\n Converts a vector of \"hello\", \"world\" to the string \"{hello, world}\"\n */\n template<>\n std::string lexical_cast< std::string>(const std::vector<std::string>& vec);\n\n}; // end of namespace boost\n\n\nnamespace graphlab {\n /**\n * \\ingroup util \n *\n * \\brief The GraphLab command line options class helps parse basic\n * command line options for the GraphLab framework as well as user\n * applications.\n *\n * Early in the development of GraphLab we realized that a lot of\n * time was spent writing code to parse the many GraphLab options as\n * well as each of the applications options. In many cases we were\n * using the boost::program_options library which while very\n * powerful can also be fairly complicated. \n *\n * As a consequence, we developed a simple command line options\n * object that parses the standard argv options capturing GraphLab\n * specific options and also processing users options. GraphLab\n * command line tools to enable user applications to benefit from\n * sophisticated and still easy to use command line parsing.\n *\n * The command_line_options data-structure is built on top of the\n * boost::program_options library. We have tried to retain much of\n * the functionality of the boost::program_options library while\n * hiding some of the less \"friendly\" template meta-programming\n * \"features\".\n *\n * Here is an example of how the library is used:\n *\n * \\code\n * int main(int argc, char** argv) {\n *\n * std::string filename;\n * size_t dimensions = 20;\n * double bound = 1E-5;\n * bool use_x = false;\n * std::vector<size_t> nsamples(1,10000);\n * \n * // Parse command line options\n * graphlab::command_line_options clopts(\"Welcome to a the HelloWorld\");\n * clopts.attach_option(\"file\", filename, \"The input filename (required)\");\n * clopts.add_positional(\"file\");\n * clopts.attach_option(\"dim\", dimensions,\n * \"the dimension of the grid\");\n * clopts.attach_option(\"bound\", bound,\n * \"The termination bound\");\n * clopts.attach_option(\"usex\", use_x,\n * \"Use algorithm x\");\n * clopts.attach_option(\"nsamples\", nsamples,\n * \"A vector of the number of samples\"); \n *\n * if(!clopts.parse(argc, argv)) return EXIT_FAILURE;\n * \n * if(!clopts.is_set(\"file\")) {\n * std::cout << \"Input file not provided\" << std::endl;\n * clopts.print_description();\n * return EXIT_FAILURE;\n * }\n * }\n * \\endcode\n *\n */\n class command_line_options : public graphlab_options {\n\n boost::program_options::options_description desc;\n boost::program_options::positional_options_description \n pos_opts;\n size_t num_positional;\n boost::program_options::variables_map vm;\n \n bool suppress_graphlab_options;\n \n std::vector<std::string> unrecognized_options;\n\n public:\n\n /**\n * \\brief Construct a command options object with basic settings. \n *\n * \\param [in] desc_str The description of the program that is\n * printed when --help is invoked (in addition to all the options\n * and their descriptions).\n *\n * \\param [in] suppress_graphlab_options If set to true the\n * standard GraphLab options are not parsed and the help screen.\n * only presents the users options. This is useful in cases where\n * command line options are needed outside of GraphLab binary\n * (e.g., simple utilities).\n */\n command_line_options(std::string desc_str,\n bool suppress_graphlab_options = false) : \n desc(desc_str), num_positional(0),\n suppress_graphlab_options(suppress_graphlab_options) { \n // Add documentation for help\n namespace boost_po = boost::program_options; \n desc.add_options()(\"help\", \"Print this help message.\");\n } // End constructor\n\n\n\n \n /// Print the same message that is printed when the --help command\n /// line argument is provided.\n inline void print_description() const { std::cout << desc << std::endl; }\n\n\n /**\n * \\brief This function should be called AFTER all the options\n * have been seen (including positionals). The parse function\n * reads the standard command line arguments and fills in the\n * attached variables. If there is an error in the syntax or\n * parsing fails the parse routine will print the error and return\n * false.\n *\n * If allow_unregistered is set to true, will permit unrecognized options\n */\n bool parse(int argc, const char* const* argv, \n bool allow_unregistered = false);\n\n /** \n * \\brief The is set function is used to test if the user provided\n * the option. The option string should match one of the attached\n * options.\n */\n bool is_set(const std::string& option);\n\n\n /**\n * If allow_unregistered flag is set on parse\n * this will contain the list of unrecognized options\n */\n inline std::vector<std::string> unrecognized() const {\n return unrecognized_options;\n }\n\n /**\n * \\brief attach a user defined option to the command line options\n * parser.\n *\n * The attach option command is used to attach a user defined\n * option to the command line options parser.\n *\n * \\param [in] option The name of the command line flag for that\n * option.\n *\n * \\param [in,out] ret_var A reference to an \"arbitrary\" type\n * which can be any of the basic types (char, int, size_t, float,\n * double, bool, string...) or an std::vector of basic types. It\n * is important that the ret_cont point to a memory block that\n * will exist when parse is invoked. The default value is read\n * from the ret_cont\n * \n * \\param [in] description Used to describe the option when --help is\n * called or when print_description is invoked.\n */\n template<typename T>\n void attach_option(const std::string& option,\n T& ret_var,\n const std::string& description) {\n namespace boost_po = boost::program_options;\n desc.add_options()\n (option.c_str(), boost_po::value<T>(&ret_var)->default_value(ret_var), \n description.c_str());\n } // end of attach_option\n\n\n // /**\n // \\brief attach a user defined option to the command line options\n // parser.\n \n // The attach option command is used to attach a user defined option\n // to the command line options parser.\n \n // \\param option The name of the command line flag for that option.\n\n // \\param ret_cont A pointer to an \"arbitrary\" type which can be any\n // of the basic types (char, int, size_t, float,\n // double, bool, string...) or an std::vector of\n // basic types. It is important that the ret_cont\n // point to a memory block that will exist when parse\n // is invoked.\n\n // \\param default_value The default value of the parameter if the\n // user does not provide this parameter on the\n // command line.\n\n // \\param description Used to describe the option when --help \n // is called or when print_description is invoked.\n // */\n // template<typename T>\n // void attach_option(const std::string& option,\n // T* ret_cont,\n // const T& default_value, \n // const std::string& description) {\n // namespace boost_po = boost::program_options;\n // assert(ret_cont != NULL);\n // desc.add_options()\n // (option.c_str(),\n // boost_po::value<T>(ret_cont)->default_value(default_value),\n // description.c_str());\n // }\n \n /** \n * \\brief This function adds the option as a positional argument.\n * A positional argument does not require --option and instead is\n * read based on its location. Each add_positional call adds to\n * the next position. \n */\n void add_positional(const std::string& str);\n\n \n }; // end class command line options\n\n\n \n\n\n\n\n}; // end namespace graphlab\n\n\n#endif\n\n" }, { "path": "src/graphlab/options/engine_help.txt", "content": "\"Synchronous Engine (sync)\\n\"\n\"=========================\\n\"\n\"The synchronous engine executes all active vertex program\\n\"\n\"synchronously in a sequence of super-step (iterations) in both the\\n\"\n\"shared and distributed memory settings.\\n\"\n\"\\n\"\n\"max_iterations: (default: infinity) The maximum number\\n\"\n\"of iterations (super-steps) to run.\\n\"\n\"\\n\"\n\"timeout: (default: infinity) The maximum time in\\n\"\n\"seconds that the engine may run. When the time runs out the\\n\"\n\"current iteration is completed and then the engine terminates.\\n\"\n\"\\n\"\n\"use_cache: (default: false) This is used to enable\\n\"\n\"caching. The update function must be written in a specific way\\n\"\n\"to take advantage of this. See the documentation for details.\\n\"\n\"\\n\"\n\"snapshot_interval: (default: -1) If set to a positive value, a snapshot\\n\"\n\"is taken every this number of iterations. If set to 0, a snapshot\\n\"\n\"is taken before the first iteration. If set to a negative value,\\n\"\n\"no snapshots are taken. A snapshot is a binary dump of the graph.\\n\"\n\"\\n\"\n\"snapshot_path: If snapshot_interval is set to a value >=0,\\n\"\n\"this option must be specified and should contain a target basename \\n\"\n\"for the snapshot. The path including folder and file prefix in \\n\"\n\"which the snapshots should be saved.\\n\"\n\"\\n\"\n\"\\n\"\n\"Asynchronous Engine (async)\\n\"\n\"===========================\\n\"\n\"The asynchronous consistent engine executed vertex programs\\n\"\n\"asynchronously and can ensure mutual exclusion such that adjacent vertices\\n\"\n\"do not run simultaneously\\n\"\n\"timeout: (default: infinity) Maximum time in seconds the engine will\\n\"\n\"run for. The actual runtime may be marginally greater as the engine\\n\"\n\"waits for all threads and processes to flush all active tasks before\\n\"\n\"returning.\\n\"\n\"factorized: (default: true) Set to true to weaken the consistency\\n\"\n\"model to factorized consistency where only individual gather/apply/scatter\\n\"\n\"calls are guaranteed to be locally consistent. Can produce massive\\n\"\n\"increases in throughput at a consistency penalty.\\n\"\n\"nfibers: (default: 3000) Number of fibers to use\\n\"\n\"stacksize: (default: 16384) Stacksize of each fiber.\\n\"\n\n\"Warp Engine \\n\"\n\"===========================\\n\"\n\"The warp engine executes update functions under the warp system\\n\"\n\"asynchronously and can ensure mutual exclusion such that adjacent vertices\\n\"\n\"do not run simultaneously\\n\"\n\"timeout: (default: infinity) Maximum time in seconds the engine will\\n\"\n\"run for. The actual runtime may be marginally greater as the engine\\n\"\n\"waits for all threads and processes to flush all active tasks before\\n\"\n\"returning.\\n\"\n\"factorized: (default: true) Set to true to weaken the consistency\\n\"\n\"model to factorized consistency where only individual gather/apply/scatter\\n\"\n\"calls are guaranteed to be locally consistent. Can produce massive\\n\"\n\"increases in throughput at a consistency penalty.\\n\"\n\"nfibers: (default: 3000) Number of fibers to use\\n\"\n\"stacksize: (default: 16384) Stacksize of each fiber.\\n\"\n\n" }, { "path": "src/graphlab/options/graph_help.txt", "content": "\"Graph Options\\n\"\n\"==============\\n\"\n\"ingress: The graph partitioning method to use. May be \\\"random\\\",\\n\"\n\"\\\"grid\\\", \\\"pds\\\", \\\"oblivious\\\" or \\\"hdrf\\\". The methods are in\"\n\"increasing complexity. \\\"random\\\" is the simplest and produces the \\n\"\n\"worst partitions, while \\\"hdrf\\\" takes the longest, but produces\\n\"\n\"a significantly better result.\\n\"\n\"\\n\"\n\"userecent: An optimization that can decrease memory utilization\\n\"\n\"of oblivious significantly at a small\\n\"\n\"partitioning penalty. Defaults to 0. Set to 1 to \\n\"\n\"enable.\\n\"\n\"\\n\"\n" }, { "path": "src/graphlab/options/graphlab_options.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n/**\n * Also contains code that is Copyright 2011 Yahoo! Inc. All rights\n * reserved. \n *\n * Contributed under the iCLA for:\n * Joseph Gonzalez (jegonzal@yahoo-inc.com) \n *\n */\n\n\n\n#ifndef GRAPHLAB_GRAPHLAB_OPTIONS_HPP\n#define GRAPHLAB_GRAPHLAB_OPTIONS_HPP\n\n#ifndef __NO_OPENMP__\n#include <omp.h>\n#endif\n\n#include <graphlab/options/options_map.hpp>\n \n#include <graphlab/parallel/pthread_tools.hpp>\nnamespace graphlab {\n\n\n /**\n * The engine options class is really a simple struct that contains\n * the basic options needed to create an engine. These options\n * include:\n \n <ul>\n\n <li> size_t ncpus: The number of cpus (threads) to use for this\n engine. </li>\n\n <li> std::string engine_type: The type of engine to use. Currently\n we support {async, synchronous}. </li>\n\n <li> std::string scheduler_type: The type of scheduler to user.\n Currently we support a wide range of schedulers: {synchronous,\n fifo, priority, sampling, splash, sweep, multiqueue_fifo,\n multiqueue_priority, set, clustered_priority, round_robin,\n chromatic} </li>\n\n <li> size_t splash_size: The size parameter for the splash\n scheduler. </li>\n </ul>\n */\n class graphlab_options {\n public:\n //! The number of cpus\n size_t ncpus;\n \n //! The type of scheduler to use\n std::string scheduler_type;\n \n //! additional arguments to the engine\n options_map engine_args;\n \n //! additional arguments to the scheduler\n options_map scheduler_args;\n\n //! Options for the graph\n options_map graph_args;\n\n graphlab_options() :\n ncpus(thread::cpu_count() > 2 ? (thread::cpu_count() - 2) : 2) {\n // Grab all the compiler flags \n /* \\todo: Add these back at some point\n #ifdef COMPILEFLAGS\n #define QUOTEME_(x) #x\n #define QUOTEME(x) QUOTEME_(x)\n compile_flags = QUOTEME(COMPILEFLAGS);\n #undef QUOTEME\n #undef QUOTEME_\n #endif \n */\n } // end of constructor\n\n\n\n\n virtual ~graphlab_options() {}\n\n\n //! Set the number of cpus\n void set_ncpus(size_t n)\n {\n#ifndef __NO_OPENMP__\n ncpus = n;\n omp_set_num_threads(ncpus);\n#else\n ncpus = n;\n#endif\n }\n\n //! Get the number of cpus\n size_t get_ncpus() const { return ncpus; }\n\n void set_scheduler_type(const std::string& stype) {\n //! \\todo: ADD CHECKING\n scheduler_type = stype;\n } \n\n\n //! Get the type of scheduler\n const std::string& get_scheduler_type() const {\n return scheduler_type;\n }\n\n //! Get the engine arguments\n const options_map& get_engine_args() const {\n return engine_args;\n }\n\n //! Get the engine arguments\n options_map& get_engine_args() {\n return engine_args;\n }\n\n const options_map& get_graph_args() const {\n return graph_args;\n }\n\n options_map& get_graph_args() {\n return graph_args;\n } \n\n\n const options_map& get_scheduler_args() const {\n return scheduler_args;\n }\n\n options_map& get_scheduler_args() {\n return scheduler_args;\n }\n\n /**\n * Display the current engine options\n */\n virtual void print() const {\n std::cout << \"GraphLab Options -------------------\\n\" \n << \"ncpus: \" << ncpus << \"\\n\"\n << \"scheduler: \" << scheduler_type << \"\\n\";\n std::cout << \"\\n\";\n std::cout << \"Scheduler Options: \\n\";\n std::cout << scheduler_args;\n std::cout << \"Graph Options: \\n\";\n std::cout << graph_args;\n std::cout << \"Engine Options: \\n\";\n std::cout << engine_args;\n std::cout << std::endl;\n }\n\n\n\n };\n\n\n \n}\n#endif\n\n" }, { "path": "src/graphlab/options/options_includes.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n/**\n * Also contains code that is Copyright 2011 Yahoo! Inc. All rights\n * reserved. \n *\n * Contributed under the iCLA for:\n * Joseph Gonzalez (jegonzal@yahoo-inc.com) \n *\n */\n\n\n\n#include <graphlab/options/command_line_options.hpp>\n#include <graphlab/options/options_map.hpp>\n#include <graphlab/options/graphlab_options.hpp>\n\n" }, { "path": "src/graphlab/options/options_map.cpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n/**\n * Also contains code that is Copyright 2011 Yahoo! Inc. All rights\n * reserved. \n *\n * Contributed under the iCLA for:\n * Joseph Gonzalez (jegonzal@yahoo-inc.com) \n *\n */\n\n\n#include <string>\n#include <sstream>\n#include <iostream>\n#include <iomanip>\n\n#include <graphlab/options/options_map.hpp>\n\nnamespace graphlab {\n \n \n void options_map::parse_string(std::string arguments) {\n std::pair<std::string, options_map> ret;\n // Break the string appart\n if(!arguments.empty()) {\n std::replace(arguments.begin(), arguments.end(), ',', ' ');\n std::replace(arguments.begin(), arguments.end(), ';', ' '); \n std::stringstream arg_strm(arguments);\n bool ret = parse_options(arg_strm);\n if (ret == false) {\n logstream(LOG_FATAL) << \"Malformed option. Failed to parse \\\"\" \n << arguments << \"\\\"\" << std::endl;\n }\n } \n }\n\n\n\n std::ostream& operator<<(std::ostream& out,\n const graphlab::options_map& opts) {\n // save the format flags\n std::ios_base::fmtflags fmt = out.flags();\n \n std::map<std::string,\n graphlab::options_map::option_values>::const_iterator\n i = opts.options.begin();\n while(i != opts.options.end()) { \n //out.setf(std::ios::left);\n out << std::setw(18) << std::left << i->first; \n out << std::setw(2) << \"= \"; \n //out.setf(std::ios::right);\n out << i->second.strval;\n out << std::endl;\n ++i;\n }\n // reset the format flags\n out.flags(fmt);\n out << std::endl;\n return out;\n }\n\n}; // end of namespace graphlab\n\n" }, { "path": "src/graphlab/options/options_map.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n/**\n * Also contains code that is Copyright 2011 Yahoo! Inc. All rights\n * reserved. \n *\n * Contributed under the iCLA for:\n * Joseph Gonzalez (jegonzal@yahoo-inc.com) \n *\n */\n\n\n#ifndef GRAPHLAB_OPTIONS_MAP_HPP\n#define GRAPHLAB_OPTIONS_MAP_HPP\n#include <map>\n#include <sstream>\n#include <ostream>\n#include <istream>\n#include <boost/lexical_cast.hpp>\n#include <graphlab/logger/logger.hpp>\n#include <graphlab/util/stl_util.hpp>\n#include <graphlab/util/generics/robust_cast.hpp>\n\nnamespace graphlab {\n\n /**\n options data structure. Defines a collection of key->value pairs\n where the key is a string, and the value is an arbitrary data\n type. The options_map class will invisibly cast between string,\n integer and double data types.\n */\n class options_map {\n public:\n \n options_map() {};\n\n explicit options_map(std::string &s) {\n parse_string(s);\n };\n\n /**\n * Add an option -> value pair where value is a string.\n * Don't use. set_option() prefered.\n */\n inline void set_option_str(const std::string &opt,\n const std::string &val) {\n options[opt].strval = val;\n try {\n options[opt].intval = boost::lexical_cast<int>(val);\n } catch(boost::bad_lexical_cast& error) {options[opt].intval = 0; }\n try {\n options[opt].dblval = boost::lexical_cast<double>(val);\n } catch(boost::bad_lexical_cast& error) { options[opt].dblval = 0.0; }\n\n if (val == \"true\" || val == \"TRUE\" || \n val == \"yes\" || val == \"YES\" || val == \"1\") options[opt].boolval = true;\n }\n\n template <typename T>\n void set_option(const std::string& opt, const T& val) {\n if (boost::is_convertible<T, std::string>::value) {\n set_option_str(opt, robust_cast<std::string>(val));\n } else {\n options[opt].strval = robust_cast<std::string>(val);\n options[opt].intval = robust_cast<int>(val);\n options[opt].dblval = robust_cast<double>(val);\n options[opt].boolval = robust_cast<bool>(val);\n }\n }\n\n /**\n * Test if the option has been created\n */\n inline bool is_set(const std::string& opt) const { \n return options.find(opt) != options.end();\n }\n\n\n /**\n * Reads a string option\n */\n inline bool get_option(const std::string& opt, std::string& val) const {\n std::map<std::string, option_values>::const_iterator i = options.find(opt);\n if (i == options.end()) return false;\n val = i->second.strval;\n return true;\n }\n\n /**\n * Reads a string option\n */\n inline bool get_option(const std::string& opt, bool& val) const {\n std::map<std::string, option_values>::const_iterator i = options.find(opt);\n if (i == options.end()) return false;\n val = i->second.boolval;\n return true;\n }\n\n\n /**\n * Reads a integer option\n */\n template <typename IntType>\n inline bool get_option(const std::string& opt, IntType& val) const {\n std::map<std::string, option_values>::const_iterator i = options.find(opt);\n if (i == options.end()) return false;\n val = i->second.intval;\n return true;\n }\n\n /**\n * Reads a float option\n */\n inline bool get_option(const std::string& opt, float& val) const {\n std::map<std::string, option_values>::const_iterator i = options.find(opt);\n if (i == options.end()) return false;\n val = i->second.dblval;\n return true;\n }\n\n /**\n * Reads a double option\n */\n inline bool get_option(const std::string& opt, double& val) const {\n std::map<std::string, option_values>::const_iterator i = options.find(opt);\n if (i == options.end()) return false;\n val = i->second.dblval;\n return true;\n }\n\n\n /**\n * Erases an option\n */\n inline void erase_option(const std::string &opt) {\n options.erase(opt);\n }\n\n /**\n * Clears all options\n */\n void clear_options() {\n options.clear();\n }\n\n \n /**\n * Parses an option stream of the form \"a=b c=d ...\"\n */\n inline bool parse_options(std::istream& s) {\n options.clear();\n std::string opt, value;\n // read till the equal\n while(s.good()) {\n getline(s, opt, '=');\n if (s.bad() || s.eof()) return false;\n \n getline(s, value, ' ');\n if (s.bad()) return false;\n set_option_str(trim(opt), trim(value));\n }\n return true;\n }\n\n /// The internal storage of the options\n struct option_values{\n std::string strval;\n int intval;\n double dblval;\n bool boolval;\n option_values () : intval(0), dblval(0), boolval(false) { }\n };\n\n\n std::vector<std::string> get_option_keys() const {\n std::map<std::string, option_values>::const_iterator iter = options.begin();\n std::vector<std::string> ret;\n while (iter != options.end()) {\n ret.push_back(iter->first);\n ++iter;\n }\n return ret;\n }\n \n /**\n * Parse a comma delimited series of key1=value1,key2=value2 \n */\n void parse_string(std::string arguments);\n\n std::map<std::string, option_values> options;\n\n };\n\n\n std::ostream& operator<<(std::ostream& out,\n const graphlab::options_map& opts);\n\n\n} // end of graphlab namespace\n\n\n\n\n#endif\n\n" }, { "path": "src/graphlab/parallel/CMakeLists.txt", "content": "project(GraphLab)\n\n\n" }, { "path": "src/graphlab/parallel/atomic.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_ATOMIC_HPP\n#define GRAPHLAB_ATOMIC_HPP\n\n#include <stdint.h>\n\n#include <boost/type_traits/is_integral.hpp>\n\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/parallel/atomic_ops.hpp>\n\nnamespace graphlab {\nnamespace graphlab_impl {\n template<typename T, bool IsIntegral>\n class atomic_impl {};\n /**\n * \\internal \n * \\brief atomic object \n * A templated class for creating atomic numbers.\n */\n template<typename T>\n class atomic_impl <T, true>: public IS_POD_TYPE {\n public:\n //! The current value of the atomic number\n volatile T value;\n\n //! Creates an atomic number with value \"value\"\n atomic_impl(const T& value = T()) : value(value) { }\n \n //! Performs an atomic increment by 1, returning the new value\n T inc() { return __sync_add_and_fetch(&value, 1); }\n\n //! Performs an atomic decrement by 1, returning the new value\n T dec() { return __sync_sub_and_fetch(&value, 1); }\n \n //! Lvalue implicit cast\n operator T() const { return value; }\n\n //! Performs an atomic increment by 1, returning the new value\n T operator++() { return inc(); }\n\n //! Performs an atomic decrement by 1, returning the new value\n T operator--() { return dec(); }\n \n //! Performs an atomic increment by 'val', returning the new value\n T inc(const T val) { return __sync_add_and_fetch(&value, val); }\n \n //! Performs an atomic decrement by 'val', returning the new value\n T dec(const T val) { return __sync_sub_and_fetch(&value, val); }\n \n //! Performs an atomic increment by 'val', returning the new value\n T operator+=(const T val) { return inc(val); }\n\n //! Performs an atomic decrement by 'val', returning the new value\n T operator-=(const T val) { return dec(val); }\n\n //! Performs an atomic increment by 1, returning the old value\n T inc_ret_last() { return __sync_fetch_and_add(&value, 1); }\n \n //! Performs an atomic decrement by 1, returning the old value\n T dec_ret_last() { return __sync_fetch_and_sub(&value, 1); }\n\n //! Performs an atomic increment by 1, returning the old value\n T operator++(int) { return inc_ret_last(); }\n\n //! Performs an atomic decrement by 1, returning the old value\n T operator--(int) { return dec_ret_last(); }\n\n //! Performs an atomic increment by 'val', returning the old value\n T inc_ret_last(const T val) { return __sync_fetch_and_add(&value, val); }\n \n //! Performs an atomic decrement by 'val', returning the new value\n T dec_ret_last(const T val) { return __sync_fetch_and_sub(&value, val); }\n\n //! Performs an atomic exchange with 'val', returning the previous value\n T exchange(const T val) { return __sync_lock_test_and_set(&value, val); }\n };\n \n // specialization for floats and doubles\n template<typename T>\n class atomic_impl <T, false>: public IS_POD_TYPE {\n public:\n //! The current value of the atomic number\n volatile T value;\n\n //! Creates an atomic number with value \"value\"\n atomic_impl(const T& value = T()) : value(value) { }\n \n //! Performs an atomic increment by 1, returning the new value\n T inc() { return inc(1); }\n\n //! Performs an atomic decrement by 1, returning the new value\n T dec() { return dec(1); }\n \n //! Lvalue implicit cast\n operator T() const { return value; }\n\n //! Performs an atomic increment by 1, returning the new value\n T operator++() { return inc(); }\n\n //! Performs an atomic decrement by 1, returning the new value\n T operator--() { return dec(); }\n \n //! Performs an atomic increment by 'val', returning the new value\n T inc(const T val) { \n T prev_value;\n T new_value;\n do {\n prev_value = value;\n new_value = prev_value + val;\n } while(!atomic_compare_and_swap(value, prev_value, new_value));\n return new_value; \n }\n \n //! Performs an atomic decrement by 'val', returning the new value\n T dec(const T val) { \n T prev_value;\n T new_value;\n do {\n prev_value = value;\n new_value = prev_value - val;\n } while(!atomic_compare_and_swap(value, prev_value, new_value));\n return new_value; \n }\n \n //! Performs an atomic increment by 'val', returning the new value\n T operator+=(const T val) { return inc(val); }\n\n //! Performs an atomic decrement by 'val', returning the new value\n T operator-=(const T val) { return dec(val); }\n\n //! Performs an atomic increment by 1, returning the old value\n T inc_ret_last() { return inc_ret_last(1); }\n \n //! Performs an atomic decrement by 1, returning the old value\n T dec_ret_last() { return dec_ret_last(1); }\n\n //! Performs an atomic increment by 1, returning the old value\n T operator++(int) { return inc_ret_last(); }\n\n //! Performs an atomic decrement by 1, returning the old value\n T operator--(int) { return dec_ret_last(); }\n\n //! Performs an atomic increment by 'val', returning the old value\n T inc_ret_last(const T val) { \n T prev_value;\n T new_value;\n do {\n prev_value = value;\n new_value = prev_value + val;\n } while(!atomic_compare_and_swap(value, prev_value, new_value));\n return prev_value; \n }\n \n //! Performs an atomic decrement by 'val', returning the new value\n T dec_ret_last(const T val) { \n T prev_value;\n T new_value;\n do {\n prev_value = value;\n new_value = prev_value - val;\n } while(!atomic_compare_and_swap(value, prev_value, new_value));\n return prev_value; \n }\n\n //! Performs an atomic exchange with 'val', returning the previous value\n T exchange(const T val) { return __sync_lock_test_and_set(&value, val); }\n };\n} // namespace graphlab_impl\n\ntemplate <typename T>\nclass atomic: public graphlab_impl::atomic_impl<T, boost::is_integral<T>::value> { \n public:\n //! Creates an atomic number with value \"value\"\n atomic(const T& value = T()): \n graphlab_impl::atomic_impl<T, boost::is_integral<T>::value>(value) { }\n \n};\n\n} // namespace graphlab\n#endif\n\n" }, { "path": "src/graphlab/parallel/atomic_add_vector2_empty_specialization.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n/**\n * Also contains code that is Copyright 2011 Yahoo! Inc. All rights\n * reserved.\n *\n * Contributed under the iCLA for:\n * Joseph Gonzalez (jegonzal@yahoo-inc.com)\n *\n */\n\n\n\n#ifndef GRAPHLAB_ATOMIC_ADD_VECTOR2_EMPTY_SPECIALIZATION_HPP\n#define GRAPHLAB_ATOMIC_ADD_VECTOR2_EMPTY_SPECIALIZATION_HPP\n\n\n#include <vector>\n\n\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/util/lock_free_pool.hpp>\n#include <graphlab/util/empty.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n#include <graphlab/parallel/atomic_add_vector2.hpp>\n\n\nnamespace graphlab {\n\n /**\n * \\TODO DOCUMENT THIS CLASS\n */\n\n template<>\n class atomic_add_vector2<graphlab::empty> {\n public:\n typedef graphlab::empty value_type;\n\n\n private:\n\n dense_bitset atomic_box_vec;\n\n\n /** Not assignable */\n void operator=(const atomic_add_vector2& other) { }\n\n\n public:\n /** Initialize the per vertex task set */\n atomic_add_vector2(size_t num_vertices = 0) {\n resize(num_vertices);\n atomic_box_vec.clear();\n }\n\n /**\n * Resize the internal locks for a different graph\n */\n void resize(size_t num_vertices) {\n atomic_box_vec.resize(num_vertices);\n atomic_box_vec.clear();\n }\n\n /** Add a task to the set returning false if the task was already\n present. */\n bool add(const size_t& idx,\n const value_type& val) {\n return !atomic_box_vec.set_bit(idx);\n } // end of add task to set\n\n\n // /** Add a task to the set returning false if the task was already\n // present. */\n // bool add_unsafe(const size_t& idx,\n // const value_type& val) {\n // ASSERT_LT(idx, atomic_box_vec.size());\n // return atomic_box_vec[idx].set_unsafe(pool, val, joincounter);\n // } // end of add task to set\n\n\n bool add(const size_t& idx,\n const value_type& val,\n value_type& new_value) {\n return !atomic_box_vec.set_bit(idx);\n } // end of add task to set\n\n\n bool test_and_get(const size_t& idx,\n value_type& ret_val) {\n return atomic_box_vec.clear_bit(idx);\n }\n\n bool peek(const size_t& idx,\n value_type& ret_val) {\n return atomic_box_vec.get(idx);\n }\n\n bool empty(const size_t& idx) const {\n return !atomic_box_vec.get(idx);\n }\n\n size_t size() const {\n return atomic_box_vec.size();\n }\n\n size_t num_joins() const {\n return 0;\n }\n\n\n void clear() {\n atomic_box_vec.clear();\n }\n\n void clear(size_t i) { atomic_box_vec.clear_bit(i);}\n\n }; // end of vertex map\n\n}; // end of namespace graphlab\n\n#undef VALUE_PENDING\n\n#endif\n\n" }, { "path": "src/graphlab/parallel/atomic_ops.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_ATOMIC_OPS_HPP\n#define GRAPHLAB_ATOMIC_OPS_HPP\n\n#include <stdint.h>\n\n\nnamespace graphlab {\n /**\n * \\ingroup util\n atomic instruction that is equivalent to the following:\n \\code\n if (a==oldval) { \n a = newval; \n return true; \n }\n else {\n return false;\n }\n \\endcode\n */\n template<typename T>\n bool atomic_compare_and_swap(T& a, T oldval, T newval) {\n return __sync_bool_compare_and_swap(&a, oldval, newval);\n };\n\n /**\n * \\ingroup util\n atomic instruction that is equivalent to the following:\n \\code\n if (a==oldval) { \n a = newval; \n return true; \n }\n else {\n return false;\n }\n \\endcode\n */\n template<typename T>\n bool atomic_compare_and_swap(volatile T& a, \n T oldval, \n T newval) {\n return __sync_bool_compare_and_swap(&a, oldval, newval);\n };\n\n /**\n * \\ingroup util\n atomic instruction that is equivalent to the following:\n \\code\n if (a==oldval) { \n a = newval; \n return true; \n }\n else {\n return false;\n }\n \\endcode\n */\n template <>\n inline bool atomic_compare_and_swap(volatile double& a, \n double oldval, \n double newval) {\n volatile uint64_t* a_ptr = reinterpret_cast<volatile uint64_t*>(&a);\n const uint64_t* oldval_ptr = reinterpret_cast<const uint64_t*>(&oldval);\n const uint64_t* newval_ptr = reinterpret_cast<const uint64_t*>(&newval);\n return __sync_bool_compare_and_swap(a_ptr, *oldval_ptr, *newval_ptr);\n };\n\n /**\n * \\ingroup util\n atomic instruction that is equivalent to the following:\n \\code\n if (a==oldval) { \n a = newval; \n return true; \n }\n else {\n return false;\n }\n \\endcode\n */\n template <>\n inline bool atomic_compare_and_swap(volatile float& a, \n float oldval, \n float newval) {\n volatile uint32_t* a_ptr = reinterpret_cast<volatile uint32_t*>(&a);\n const uint32_t* oldval_ptr = reinterpret_cast<const uint32_t*>(&oldval);\n const uint32_t* newval_ptr = reinterpret_cast<const uint32_t*>(&newval);\n return __sync_bool_compare_and_swap(a_ptr, *oldval_ptr, *newval_ptr);\n };\n\n /** \n * \\ingroup util\n * \\brief Atomically exchanges the values of a and b.\n * \\warning This is not a full atomic exchange. Read of a,\n * and the write of b into a is atomic. But the write into b is not.\n */\n template<typename T>\n void atomic_exchange(T& a, T& b) {\n b = __sync_lock_test_and_set(&a, b);\n };\n\n /** \n * \\ingroup util\n * \\brief Atomically exchanges the values of a and b.\n * \\warning This is not a full atomic exchange. Read of a,\n * and the write of b into a is atomic. But the write into b is not.\n */\n template<typename T>\n void atomic_exchange(volatile T& a, T& b) {\n b = __sync_lock_test_and_set(&a, b);\n };\n\n /** \n * \\ingroup util\n * \\brief Atomically sets a to the newval, returning the old value\n */\n template<typename T>\n T fetch_and_store(T& a, const T& newval) {\n return __sync_lock_test_and_set(&a, newval);\n };\n\n}\n#endif\n\n" }, { "path": "src/graphlab/parallel/cache_line_pad.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n/**\n * Also contains code that is Copyright 2011 Yahoo! Inc. All rights\n * reserved. \n *\n * Contributed under the iCLA for:\n * Joseph Gonzalez (jegonzal@yahoo-inc.com) \n *\n */\n\n\n\n\n\n#ifndef GRAPHLAB_CACHE_LINE_PAD\n#define GRAPHLAB_CACHE_LINE_PAD\n\nnamespace graphlab {\n /**\n * Used to prevent false cache sharing by padding T\n */\n template <typename T> struct cache_line_pad {\n T value;\n char pad[64 - (sizeof(T) % 64)]; \n cache_line_pad(const T& value = T()) : value(value) { }\n T& operator=(const T& other) { return value = other; } \n operator T() const { return value; }\n }; // end of cache_line_pad\n\n}; // end of namespace\n\n\n\n\n\n#endif\n" }, { "path": "src/graphlab/parallel/deferred_rwlock.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef DEFERRED_RWLOCK_HPP\n#define DEFERRED_RWLOCK_HPP\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/parallel/queued_rwlock.hpp>\n#include <graphlab/logger/assertions.hpp>\nnamespace graphlab {\nclass deferred_rwlock{\n public:\n\n struct request{\n char lockclass : 2;\n __attribute__((may_alias)) uint64_t id : 62; \n request* next;\n };\n private:\n request* head;\n request* tail;\n uint16_t reader_count;\n bool writer;\n simple_spinlock lock;\n public:\n\n deferred_rwlock(): head(NULL),\n tail(NULL), reader_count(0),writer(false) { }\n\n // debugging purposes only\n inline size_t get_reader_count() {\n __sync_synchronize();\n return reader_count;\n }\n\n // debugging purposes only\n inline bool has_waiters() {\n return head != NULL || tail != NULL;\n }\n\n inline void insert_queue(request *I) {\n if (head == NULL) {\n head = I;\n tail = I;\n }\n else {\n tail->next = I;\n tail = I;\n }\n }\n inline void insert_queue_head(request *I) {\n if (head == NULL) {\n head = I;\n tail = I;\n }\n else {\n I->next = head;\n head = I;\n }\n }\n \n inline bool writelock_priority(request *I) {\n I->next = NULL;\n I->lockclass = QUEUED_RW_LOCK_REQUEST_WRITE;\n lock.lock();\n if (reader_count == 0 && writer == false) {\n // fastpath\n writer = true;\n lock.unlock();\n return true;\n }\n else {\n insert_queue_head(I);\n lock.unlock();\n return false;\n }\n }\n \n inline bool writelock(request *I) {\n I->next = NULL;\n I->lockclass = QUEUED_RW_LOCK_REQUEST_WRITE;\n lock.lock();\n if (reader_count == 0 && writer == false) {\n // fastpath\n writer = true;\n lock.unlock();\n return true;\n }\n else {\n insert_queue(I);\n lock.unlock();\n return false;\n }\n }\n\n // completes the write lock on the head. lock must be acquired\n // head must be a write lock\n inline void complete_wrlock() {\n // ASSERT_EQ(reader_count.value, 0);\n head = head->next;\n if (head == NULL) tail = NULL;\n writer = true;\n }\n\n // completes the read lock on the head. lock must be acquired\n // head must be a read lock\n inline size_t complete_rdlock(request* &released) {\n released = head;\n size_t numcompleted = 1;\n head = head->next;\n request* readertail = released;\n while (head != NULL && head->lockclass == QUEUED_RW_LOCK_REQUEST_READ) {\n readertail = head;\n head = head->next;\n numcompleted++;\n }\n reader_count += numcompleted;\n if (head == NULL) tail = NULL;\n \n // now released is the head to a reader list\n // and head is the head of a writer list\n // I want to go through the writer list and extract all the readers\n // this essentially \n // splits the list into two sections, one containing only readers, and \n // one containing only writers.\n // (reader biased locking)\n if (head != NULL) {\n request* latestwriter = head;\n request* cur = head->next;\n while (1) {\n if (cur->lockclass == QUEUED_RW_LOCK_REQUEST_WRITE) {\n latestwriter = cur;\n }\n else {\n readertail->next = cur;\n readertail = cur;\n reader_count++;\n numcompleted++;\n latestwriter->next = cur->next;\n }\n if (cur == tail) break;\n cur=cur->next;\n }\n }\n return numcompleted;\n }\n \n inline size_t wrunlock(request* &released) {\n released = NULL;\n lock.lock();\n writer = false;\n size_t ret = 0;\n if (head != NULL) {\n if (head->lockclass == QUEUED_RW_LOCK_REQUEST_READ) {\n ret = complete_rdlock(released);\n if (ret == 2) assert(released->next != NULL);\n }\n else {\n writer = true;\n released = head;\n complete_wrlock();\n ret = 1;\n }\n }\n lock.unlock();\n return ret;\n }\n\n inline size_t readlock(request *I, request* &released) {\n released = NULL;\n size_t ret = 0;\n I->next = NULL;\n I->lockclass = QUEUED_RW_LOCK_REQUEST_READ;\n lock.lock();\n // there are readers and no one is writing\n if (head == NULL && writer == false) {\n // fast path\n ++reader_count;\n lock.unlock();\n released = I;\n return 1;\n }\n else {\n // slow path. Insert into queue\n insert_queue(I);\n if (head->lockclass == QUEUED_RW_LOCK_REQUEST_READ && writer == false) {\n ret = complete_rdlock(released);\n }\n lock.unlock();\n return ret;\n }\n }\n\n inline size_t readlock_priority(request *I, request* &released) {\n released = NULL;\n size_t ret = 0;\n I->next = NULL;\n I->lockclass = QUEUED_RW_LOCK_REQUEST_READ;\n lock.lock();\n // there are readers and no one is writing\n if (head == NULL && writer == false) {\n // fast path\n ++reader_count;\n lock.unlock();\n released = I;\n return 1;\n }\n else {\n // slow path. Insert into queue\n insert_queue_head(I);\n if (head->lockclass == QUEUED_RW_LOCK_REQUEST_READ && writer == false) {\n ret = complete_rdlock(released);\n }\n lock.unlock();\n return ret;\n }\n }\n\n inline size_t rdunlock(request* &released) {\n released = NULL;\n lock.lock();\n --reader_count;\n if (reader_count == 0) {\n size_t ret = 0;\n if (head != NULL) {\n if (head->lockclass == QUEUED_RW_LOCK_REQUEST_READ) {\n ret = complete_rdlock(released);\n }\n else {\n writer = true;\n released = head;\n complete_wrlock();\n ret = 1;\n }\n }\n lock.unlock();\n return ret;\n }\n else {\n lock.unlock();\n return 0;\n }\n }\n};\n\n}\n#endif\n\n" }, { "path": "src/graphlab/parallel/fiber_barrier.hpp", "content": "#ifndef GRAPHLAB_PARALLEL_FIBER_BARRIER_HPP\n#define GRAPHLAB_PARALLEL_FIBER_BARRIER_HPP\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/parallel/fiber_control.hpp>\nnamespace graphlab {\n class fiber_barrier {\n private:\n graphlab::mutex mutex;\n graphlab::conditional conditional;\n mutable int needed;\n mutable int called; \n \n mutable bool barrier_sense;\n mutable bool barrier_release;\n bool alive;\n\n mutable std::vector<size_t> fiber_handles;\n\n // not copyconstructible\n fiber_barrier(const fiber_barrier&) { }\n\n\n // not copyable\n void operator=(const fiber_barrier& m) { }\n\n public:\n /// Construct a barrier which will only fall when numthreads enter\n fiber_barrier(size_t numthreads) {\n needed = numthreads;\n called = 0;\n barrier_sense = false;\n barrier_release = true;\n alive = true;\n fiber_handles.resize(needed);\n }\n\n void resize_unsafe(size_t numthreads) {\n needed = numthreads;\n fiber_handles.resize(needed);\n }\n \n /// Wait on the barrier until numthreads has called wait\n inline void wait() const {\n if (!alive) return;\n mutex.lock();\n // set waiting;\n fiber_handles[called] = fiber_control::get_tid();\n called++;\n bool listening_on = barrier_sense;\n if (called == needed) {\n // if I have reached the required limit, wait up. Set waiting\n // to 0 to make sure everyone wakes up\n std::vector<size_t> to_wake = fiber_handles;\n called = 0;\n barrier_release = barrier_sense;\n barrier_sense = !barrier_sense;\n // clear all waiting, wake everyone. (less 1 since current thread\n // is already awake)\n for (size_t i = 0;i < to_wake.size() - 1; ++i) {\n fiber_control::schedule_tid(to_wake[i]);\n } \n } else {\n // while no one has broadcasted, sleep\n while(barrier_release != listening_on && alive) {\n fiber_control::deschedule_self(&mutex.m_mut);\n mutex.lock();\n }\n }\n mutex.unlock();\n }\n }; // end of conditional\n \n}\n#endif \n" }, { "path": "src/graphlab/parallel/fiber_conditional.hpp", "content": "#ifndef GRAPHLAB_FIBER_CONDITIONAL_HPP\n#define GRAPHLAB_FIBER_CONDITIONAL_HPP\n#include <vector>\n#include <queue>\n#include <graphlab/parallel/fiber_control.hpp>\nnamespace graphlab {\n\n/**\n * \\ingroup util\n * Wrapper around pthread's condition variable, that can work with both\n * fibers and threads simultaneously, but at a cost of much greater memory \n * requirements.\n *\n * Limitations.\n * - Does not support timed wait\n * - threads and fibers are not queued perfectly. fibers are preferentially \n * signaled.\n *\n * Before you use, see \\ref parallel_object_intricacies.\n */\nclass fiber_conditional {\n private:\n mutable pthread_cond_t m_cond;\n mutex lock;\n mutable std::queue<size_t> fibers; // used to hold the fibers that are waiting here\n\n // not copyable\n void operator=(const fiber_conditional& m) { }\n\n public:\n fiber_conditional() {\n int error = pthread_cond_init(&m_cond, NULL);\n ASSERT_TRUE(!error);\n }\n\n /** Copy constructor which does not copy. Do not use!\n Required for compatibility with some STL implementations (LLVM).\n which use the copy constructor for vector resize, \n rather than the standard constructor. */\n fiber_conditional(const fiber_conditional &) {\n int error = pthread_cond_init(&m_cond, NULL);\n ASSERT_TRUE(!error);\n }\n\n\n\n /// Waits on condition. The mutex must already be acquired. Caller\n /// must be careful about spurious wakes.\n inline void wait(const mutex& mut) const {\n size_t tid = fiber_control::get_tid();\n if (tid > 0) {\n lock.lock();\n fibers.push(tid);\n lock.unlock();\n fiber_control::deschedule_self(&mut.m_mut);\n mut.lock();\n } else {\n int error = pthread_cond_wait(&m_cond, &mut.m_mut);\n ASSERT_TRUE(!error);\n }\n }\n /// Signals one waiting thread to wake up\n inline void signal() const {\n if (!fibers.empty()) {\n lock.lock();\n if (!fibers.empty()) {\n size_t tid = fibers.front();\n fibers.pop();\n lock.unlock();\n fiber_control::schedule_tid(tid);\n return;\n }\n lock.unlock();\n }\n int error = pthread_cond_signal(&m_cond);\n ASSERT_TRUE(!error);\n }\n /// Wakes up all waiting threads\n inline void broadcast() const {\n lock.lock();\n while (!fibers.empty()) {\n size_t tid = fibers.front();\n fibers.pop();\n fiber_control::schedule_tid(tid);\n }\n lock.unlock();\n int error = pthread_cond_broadcast(&m_cond);\n ASSERT_TRUE(!error);\n }\n\n ~fiber_conditional() {\n ASSERT_EQ(fibers.size(), 0);\n int error = pthread_cond_destroy(&m_cond);\n ASSERT_TRUE(!error);\n }\n}; \n}\n#endif\n" }, { "path": "src/graphlab/parallel/fiber_control.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <boost/bind.hpp>\n#include <graphlab/util/random.hpp>\n#include <graphlab/parallel/fiber_control.hpp>\n#include <graphlab/logger/assertions.hpp>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/macros_def.hpp>\n//#include <valgrind/valgrind.h>\nnamespace graphlab {\n\nbool fiber_control::tls_created = false;\nbool fiber_control::instance_created = false;\nsize_t fiber_control::instance_construct_params_nworkers = 0;\nsize_t fiber_control::instance_construct_params_affinity_base = 0;\npthread_key_t fiber_control::tlskey;\n\nfiber_control::affinity_type fiber_control::all_affinity() {\n affinity_type ret;\n ret.fill();\n return ret;\n}\n\nfiber_control::fiber_control(size_t nworkers, \n size_t affinity_base)\n :nworkers(nworkers),\n affinity_base(affinity_base),\n stop_workers(false),\n flsdeleter(NULL) {\n // initialize the thread local storage keys\n if (!tls_created) {\n pthread_key_create(&tlskey, fiber_control::tls_deleter);\n tls_created = true;\n }\n\n // set up the queues.\n schedule.resize(nworkers);\n for (size_t i = 0;i < nworkers; ++i) {\n schedule[i].waiting = false;\n schedule[i].nwaiting = 0;\n schedule[i].affinity_queue = new inplace_lf_queue2<fiber>;\n schedule[i].priority_queue = new inplace_lf_queue2<fiber>;\n schedule[i].popped_affinity_queue = NULL;\n schedule[i].popped_priority_queue = NULL;\n }\n // launch the workers\n for (size_t i = 0;i < nworkers; ++i) {\n workers.launch(boost::bind(&fiber_control::worker_init, this, i), \n affinity_base + i);\n }\n}\n\nfiber_control::~fiber_control() {\n join();\n stop_workers = true;\n for (size_t i = 0;i < nworkers; ++i) {\n schedule[i].active_lock.lock();\n schedule[i].active_cond.broadcast();\n schedule[i].active_lock.unlock();\n delete schedule[i].affinity_queue;\n delete schedule[i].priority_queue;\n }\n workers.join();\n\n\n\n pthread_key_delete(tlskey);\n}\n\n\nvoid fiber_control::tls_deleter(void* f) {\n fiber_control::tls* t = (fiber_control::tls*)(f);\n delete t;\n}\n\nvoid fiber_control::create_tls_ptr() {\n pthread_setspecific(tlskey, (void*)(new fiber_control::tls));\n}\n\n\nfiber_control::tls* fiber_control::get_tls_ptr() {\n if (tls_created == false) return NULL;\n else return (fiber_control::tls*) pthread_getspecific(tlskey);\n}\n\nfiber_control::fiber* fiber_control::get_active_fiber() {\n tls* t = get_tls_ptr();\n if (t != NULL) return t->cur_fiber;\n else return NULL;\n}\n\n\n\n\nvoid fiber_control::active_queue_insert_tail(size_t workerid, fiber_control::fiber* value) {\n if (value->scheduleable) {\n// printf(\"%ld: Scheduling %ld on %ld\\n\", get_worker_id(), value->id, workerid);\n schedule[workerid].affinity_queue->enqueue(value);\n ++schedule[workerid].nwaiting;\n if (schedule[workerid].waiting) {\n schedule[workerid].active_lock.lock();\n schedule[workerid].active_cond.signal();\n schedule[workerid].active_lock.unlock();\n }\n }\n}\n\n\nvoid fiber_control::active_queue_insert_head(size_t workerid, fiber_control::fiber* value) {\n if (value->scheduleable) {\n// printf(\"%ld: Scheduling %ld on %ld\\n\", get_worker_id(), value->id, workerid);\n schedule[workerid].priority_queue->enqueue(value);\n ++schedule[workerid].nwaiting;\n if (schedule[workerid].waiting) {\n schedule[workerid].active_lock.lock();\n schedule[workerid].active_cond.signal();\n schedule[workerid].active_lock.unlock();\n }\n }\n}\n\nfiber_control::fiber* fiber_control::try_pop_queue(inplace_lf_queue2<fiber>& lfqueue,\n fiber*& popped_queue) {\n fiber_control::fiber* ret = NULL;\n // if there is stuff in the popped queue, pop it.\n if (popped_queue == NULL) {\n popped_queue = lfqueue.dequeue_all();\n }\n\n if (popped_queue != NULL) {\n ret = popped_queue;\n do {\n popped_queue = ret->next;\n asm volatile(\"pause\\n\": : :\"memory\");\n } while(popped_queue == NULL);\n // we have reached the end of the queue. clear the popped queue\n // and return\n if (popped_queue == lfqueue.end_of_dequeue_list()) {\n popped_queue = NULL;\n }\n }\n return ret;\n}\n\nfiber_control::fiber* fiber_control::active_queue_remove(size_t workerid) {\n fiber_control::fiber* ret = NULL;\n thread_schedule& curts = schedule[workerid];\n ret = try_pop_queue(*curts.priority_queue, curts.popped_priority_queue);\n if (ret == NULL) {\n ret = try_pop_queue(*curts.affinity_queue , curts.popped_affinity_queue);\n }\n if (ret) {\n // printf(\"%ld: Running %ld\\n\", get_worker_id(), ret->id);\n }\n return ret;\n}\n\nvoid fiber_control::exit() {\n distributed_control* dc = distributed_control::get_instance();\n if (dc) dc->flush();\n fiber* fib = get_active_fiber();\n if (fib != NULL) {\n // add to garbage.\n fib->terminate = true;\n yield(); // never returns\n ASSERT_MSG(false, \"Impossible Condition. Dead Fiber woke up\");\n } else {\n ASSERT_MSG(false, \"Calling fiber exit not from a fiber\");\n }\n}\n\nstatic timer flush_timer;\nmutex flush_lock;\n\nvoid fiber_control::worker_init(size_t workerid) {\n /*\n * This is the \"root\" stack for each worker.\n * When there are active user threads associated with this worker, \n * it will switch directly between the fibers.\n * But, when the worker has no other fiber to run, it will return to this\n * stack and and wait in a condition variable\n */\n // create a root context\n create_tls_ptr();\n // set up the tls structure\n tls* t = get_tls_ptr();\n t->prev_fiber = NULL;\n t->cur_fiber = NULL;\n t->garbage = NULL;\n t->workerid = workerid;\n t->parent = this;\n\n schedule[workerid].waiting = true;\n schedule[workerid].active_lock.lock();\n while(!stop_workers) {\n // get a fiber to run\n fiber* next_fib = t->parent->active_queue_remove(workerid);\n if (next_fib != NULL) {\n // if there is a fiber. yield to it\n schedule[workerid].active_lock.unlock();\n schedule[workerid].waiting = false;\n active_workers.inc();\n yield_to(next_fib);\n if (flush_timer.current_time() > 0.0001 && flush_lock.try_lock()) {\n distributed_control* dc = distributed_control::get_instance();\n if (dc) dc->flush_soon();\n flush_timer.start();\n flush_lock.unlock();\n }\n active_workers.dec();\n schedule[workerid].waiting = true;\n schedule[workerid].active_lock.lock();\n } else {\n // if there is no fiber. wait.\n schedule[workerid].active_cond.wait(schedule[workerid].active_lock);\n }\n }\n schedule[workerid].active_lock.unlock();\n}\n\nstruct trampoline_args {\n boost::function<void(void)> fn;\n};\n\n// the trampoline to call the user function. This function never returns\nvoid fiber_control::trampoline(intptr_t _args) {\n // we may have launched to here by switching in from another fiber.\n // we will need to clean up the previous fiber\n tls* t = get_tls_ptr();\n if (t->prev_fiber) t->parent->reschedule_fiber(t->workerid, t->prev_fiber);\n t->prev_fiber = NULL;\n\n trampoline_args* args = reinterpret_cast<trampoline_args*>(_args);\n try {\n args->fn();\n } catch (...) {\n }\n delete args;\n fiber_control::exit();\n}\n\nsize_t fiber_control::launch(boost::function<void(void)> fn, \n size_t stacksize, \n affinity_type affinity) {\n ASSERT_GT(affinity.popcount(), 0);\n size_t b = 0;\n ASSERT_TRUE(affinity.first_bit(b));\n // make sure there is always a worker I can work on\n ASSERT_LT(b, nworkers);\n\n // allocate a stack\n fiber* fib = new fiber;\n fib->parent = this;\n fib->stack = malloc(stacksize);\n fib->id = fiber_id_counter.inc();\n foreach(size_t b, affinity) {\n if (b < nworkers) fib->affinity_array.push_back((unsigned char)b);\n else break;\n }\n ASSERT_GT(fib->affinity_array.size(), 0);\n fib->affinity = affinity;\n //VALGRIND_STACK_REGISTER(fib->stack, (char*)fib->stack + stacksize);\n fib->fls = NULL;\n fib->next = NULL;\n fib->deschedule_lock = NULL;\n fib->terminate = false;\n fib->descheduled = false;\n fib->scheduleable = true;\n // construct the initial context\n trampoline_args* args = new trampoline_args;\n args->fn = fn;\n fib->initial_trampoline_args = (intptr_t)(args);\n // stack grows downwards.\n fib->context = boost::context::make_fcontext((char*)fib->stack + stacksize,\n stacksize,\n trampoline);\n fibers_active.inc();\n\n // find a place to put the thread\n size_t choice = pick_fiber_worker(fib);\n active_queue_insert_tail(choice, fib);\n return reinterpret_cast<size_t>(fib);\n}\n\nsize_t fiber_control::pick_fiber_worker(fiber* fib) {\n // first try to use the original worker if possible\n size_t choice = get_worker_id();\n if (choice == (size_t)(-1) || fib->affinity.get(choice) == 0) {\n //choice rejected, pick randomly from the available choices\n // if there is only one affinity option, return it\n if (fib->affinity_array.size() == 1) {\n choice = fib->affinity_array[0];\n } else {\n size_t ra = graphlab::random::fast_uniform<size_t>(0,fib->affinity_array.size() - 1);\n std::swap(fib->affinity_array[ra], fib->affinity_array[0]);\n choice = fib->affinity_array[0];\n }\n }\n return choice;\n}\n\nvoid fiber_control::yield_to(fiber* next_fib) {\n // the core scheduling logic\n tls* t = get_tls_ptr();\n \n// if (next_fib) {\n// if (t->cur_fiber) {\n// printf(\"%ld: yield to: %ld from %ld\\n\", get_worker_id(), next_fib->id, t->cur_fiber->id);\n// } else {\n// printf(\"%ld: yield to: %ld\\n\", get_worker_id(), next_fib->id);\n// }\n// } \n if (next_fib != NULL) {\n // reset the priority flag\n next_fib->priority = false;\n // current fiber moves to previous\n // next fiber move to current\n t->prev_fiber = t->cur_fiber;\n t->cur_fiber = next_fib;\n if (t->prev_fiber != NULL) {\n // context switch to fib outside the lock\n boost::context::jump_fcontext(t->prev_fiber->context,\n t->cur_fiber->context,\n t->cur_fiber->initial_trampoline_args);\n } else {\n boost::context::jump_fcontext(&t->base_context,\n t->cur_fiber->context,\n t->cur_fiber->initial_trampoline_args);\n }\n } else {\n // ok. there isn't anything to schedule to\n // am I meant to be terminated? or descheduled?\n if (t->cur_fiber &&\n (t->cur_fiber->terminate || t->cur_fiber->descheduled) ) {\n // yup. killing current fiber\n // context switch back to basecontext which will\n // do the cleanup\n //\n // current fiber moves to previous\n // next fiber (base context) move to current\n // (as identifibed by cur_fiber = NULL)\n t->prev_fiber = t->cur_fiber;\n t->cur_fiber = NULL;\n boost::context::jump_fcontext(t->prev_fiber->context,\n &t->base_context,\n 0);\n } else {\n // nothing to do, and not terminating...\n // then don't yield!\n return;\n }\n }\n // reread the tls pointer because we may have woken up in a different thread\n t = get_tls_ptr();\n // reschedule the previous fiber\n if (t->prev_fiber) reschedule_fiber(t->workerid, t->prev_fiber);\n t->prev_fiber = NULL;\n\n // if distributed_controller alive\n distributed_control* dc = distributed_control::get_instance();\n if (dc && t->workerid < dc->num_handler_threads()) {\n dc->handle_incoming_calls(t->workerid, dc->num_handler_threads());\n }\n}\n\nvoid fiber_control::reschedule_fiber(size_t workerid, fiber* fib) {\n fib->lock.lock();\n if (!fib->terminate && !fib->descheduled) {\n fib->lock.unlock();\n // we reschedule it\n // Re-lock the queue\n //printf(\"%ld: Reinserting %ld\\n\", get_worker_id(), fib->id);\n if (!fib->priority) active_queue_insert_tail(workerid, fib);\n else active_queue_insert_head(workerid, fib);\n } else if (fib->descheduled) {\n // unflag descheduled and unset scheduleable\n fib->descheduled = false;\n fib->scheduleable = false;\n if (fib->deschedule_lock) pthread_mutex_unlock(fib->deschedule_lock);\n fib->deschedule_lock = NULL;\n //printf(\"%ld: Descheduling complete %ld\\n\", get_worker_id(), fib->id);\n fib->lock.unlock();\n } else if (fib->terminate) {\n fib->lock.unlock();\n // previous fiber is dead. destroy it\n free(fib->stack);\n //VALGRIND_STACK_DEREGISTER(fib->stack);\n // delete the fiber local storage if any\n if (fib->fls && flsdeleter) flsdeleter(fib->fls);\n delete fib;\n // if we are out of threads, signal the join\n if (fibers_active.dec() == 0) {\n join_lock.lock();\n join_cond.signal();\n join_lock.unlock();\n }\n } else {\n // impossible condition\n assert(false);\n }\n}\n\nvoid fiber_control::yield() {\n // the core scheduling logic\n tls* t = get_tls_ptr();\n if (t == NULL) return;\n // remove some other work to do.\n fiber_control* parentgroup = t->parent;\n size_t workerid = t->workerid;\n fiber* next_fib = parentgroup->active_queue_remove(workerid);\n t->parent->yield_to(next_fib);\n}\n\n\nvoid fiber_control::fast_yield() {\n yield();\n}\n\n\nvoid fiber_control::join() {\n join_lock.lock();\n while(fibers_active.value > 0) {\n join_cond.wait(join_lock);\n }\n join_lock.unlock();\n}\n\nsize_t fiber_control::get_tid() {\n fiber_control::tls* tls = get_tls_ptr();\n if (tls != NULL) return reinterpret_cast<size_t>(tls->cur_fiber);\n else return (size_t)(0);\n}\n\n\nbool fiber_control::in_fiber() {\n return get_tls_ptr() != NULL;\n}\n\nvoid fiber_control::deschedule_self(pthread_mutex_t* lock) {\n fiber* fib = get_tls_ptr()->cur_fiber;\n fib->lock.lock();\n assert(fib->descheduled == false);\n assert(fib->scheduleable == true);\n fib->deschedule_lock = lock;\n fib->descheduled = true;\n //printf(\"%ld: Descheduling requested %ld\\n\", get_worker_id(), fib->id);\n fib->lock.unlock();\n yield();\n}\n\nbool fiber_control::worker_has_priority_fibers_on_queue() {\n tls* t = get_tls_ptr();\n if (t == NULL) return false;\n fiber_control* parentgroup = t->parent;\n size_t workerid = t->workerid;\n return !parentgroup->schedule[workerid].priority_queue->empty();\n}\n\nbool fiber_control::worker_has_fibers_on_queue() {\n tls* t = get_tls_ptr();\n if (t == NULL) return false;\n fiber_control* parentgroup = t->parent;\n size_t workerid = t->workerid;\n return !parentgroup->schedule[workerid].priority_queue->empty() ||\n !parentgroup->schedule[workerid].affinity_queue->empty();\n}\n\nsize_t fiber_control::get_worker_id() {\n fiber_control::tls* tls = get_tls_ptr();\n if (tls != NULL) return tls->workerid;\n else return (size_t)(-1);\n}\n\nvoid fiber_control::schedule_tid(size_t tid, bool priority) {\n fiber* fib = reinterpret_cast<fiber*>(tid);\n fib->lock.lock();\n // we MUST get here only after the thread was completely descheduled\n // or no deschedule operation has happened yet.\n assert(fib->descheduled == false);\n fib->descheduled = false;\n if (fib->scheduleable == false) {\n // if this thread was descheduled completely. Reschedule it.\n //printf(\"%ld: Scheduling requested %ld\\n\", get_worker_id(), fib->id);\n fib->scheduleable = true;\n fib->priority = priority;\n fib->lock.unlock();\n size_t choice = fib->parent->pick_fiber_worker(fib);\n fib->parent->reschedule_fiber(choice, fib);\n } else {\n //printf(\"%ld: Scheduling requested of running thread %ld\\n\", get_worker_id(), fib->id);\n fib->lock.unlock();\n }\n}\n\n\nvoid fiber_control::set_tls_deleter(void (*deleter)(void*)) {\n flsdeleter = deleter;\n}\n\nvoid* fiber_control::get_tls() {\n fiber_control::tls* f = get_tls_ptr();\n if (f != NULL) {\n return f->cur_fiber->fls;\n } else {\n // cannot get TLS of a non-fiber\n ASSERT_MSG(false, \"Trying to get a fiber TLS from a non-fiber\");\n return NULL;\n }\n}\n\nvoid fiber_control::set_tls(void* tls) {\n fiber_control::tls* f = get_tls_ptr();\n if (f != NULL) {\n f->cur_fiber->fls = tls;\n } else {\n // cannot get TLS of a non-fiber\n ASSERT_MSG(false, \"Trying to get a fiber TLS from a non-fiber\");\n }\n}\n\n\nvoid fiber_control::instance_set_parameters(size_t nworkers = 0,\n size_t affinity_base = 0) {\n instance_construct_params_nworkers = nworkers;\n instance_construct_params_affinity_base = affinity_base;\n}\n\nfiber_control& fiber_control::get_instance() {\n fiber_control::instance_created = true;\n // set sane defaults\n if (instance_construct_params_nworkers == 0) {\n instance_construct_params_nworkers = thread::cpu_count();\n }\n static fiber_control singleton(instance_construct_params_nworkers, \n instance_construct_params_affinity_base);\n return singleton;\n}\n\n}\n" }, { "path": "src/graphlab/parallel/fiber_control.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_FIBER_CONTROL_HPP\n#define GRAPHLAB_FIBER_CONTROL_HPP\n\n#include <stdint.h>\n#include <cstdlib>\n#include <boost/context/all.hpp>\n#include <boost/function.hpp>\n#include <boost/lockfree/queue.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n#include <graphlab/util/inplace_lf_queue2.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/parallel/atomic.hpp>\nnamespace graphlab {\n\n/**\n * The master controller for the user mode threading system\n */\nclass fiber_control {\n public:\n\n typedef fixed_dense_bitset<64> affinity_type;\n static affinity_type all_affinity();\n\n struct fiber {\n simple_spinlock lock;\n fiber_control* parent;\n boost::context::fcontext_t* context;\n void* stack;\n size_t id;\n affinity_type affinity;\n std::vector<unsigned char> affinity_array;\n void* fls; // fiber local storage\n fiber* next;\n intptr_t initial_trampoline_args;\n pthread_mutex_t* deschedule_lock; // if descheduled is set, we will\n // atomically deschedule and unlock\n // this mutex\n bool descheduled; // flag. set if this fiber is to be descheduled.\n // This is a temporary flag, and is only used to notify\n // the context switch to deschedule this thread.\n // lock must be acquired for this to be modified\n\n bool terminate; // flag. set if this fiber is to be destroyed.\n // This is a temporary flag, and is only used to notify\n // the context switch to destroy this thread.\n\n bool scheduleable; // Managed by the queue management routines.\n // Set if the fiber is inside the scheduling queue\n // or is running in a thread.\n // lock must be acquired for this to be modified.\n bool priority; // flag. If set, rescheduling this fiber\n // will cause it to be placed at the head of the queue\n };\n\n\n private:\n size_t nworkers;\n size_t affinity_base;\n atomic<size_t> fiber_id_counter;\n atomic<size_t> fibers_active;\n atomic<size_t> active_workers;\n mutex join_lock;\n conditional join_cond;\n\n bool stop_workers;\n\n // The scheduler is a simple queue. One for each worker\n struct thread_schedule {\n thread_schedule():waiting(false) { }\n mutex active_lock;\n conditional active_cond;\n volatile bool waiting;\n size_t nwaiting;\n // a queue of fibers to evaluate before those in the thread_queue\n inplace_lf_queue2<fiber>* affinity_queue;\n fiber* popped_affinity_queue;\n\n inplace_lf_queue2<fiber>* priority_queue;\n fiber* popped_priority_queue;\n };\n std::vector<thread_schedule> schedule;\n\n thread_group workers;\n\n\n // locks must be acquired outside the call\n void active_queue_insert_head(size_t workerid, fiber* value);\n void active_queue_insert_tail(size_t workerid, fiber* value);\n void active_queue_insert_tail(fiber* value);\n fiber* active_queue_remove(size_t workerid);\n\n // a thread local storage for the worker to point to a fiber\n static bool tls_created;\n struct tls {\n fiber_control* parent;\n fiber* prev_fiber; // the fiber we context switch from\n fiber* cur_fiber; // the fiber we are context switching to\n fiber* garbage; // A fiber to delete after the context switch\n size_t workerid;\n boost::context::fcontext_t base_context;\n };\n\n static pthread_key_t tlskey; // points to the tls structure above\n static void tls_deleter(void* tls);\n\n /// internal function to create the TLS for the worker threads\n static void create_tls_ptr();\n /// internal function to read the TLS for the worker threads\n static tls* get_tls_ptr();\n /// Returns the current fiber scheduled on this worker thread\n static fiber* get_active_fiber();\n\n /// Gets a fiber from the lock-free / popped pair\n fiber* try_pop_queue(inplace_lf_queue2<fiber>& lfqueue, fiber*& popped_queue);\n /// The function that each worker thread starts off running\n void worker_init(size_t workerid);\n\n void reschedule_fiber(size_t workerid, fiber* pfib);\n void yield_to(fiber* next_fib);\n static void trampoline(intptr_t _args);\n\n void (*flsdeleter)(void*);\n\n size_t pick_fiber_worker(fiber* fib);\n\n // delete copy constructor\n fiber_control(fiber_control&) {};\n \n public:\n\n /// Private constructor\n fiber_control(size_t nworkers, size_t affinity_base);\n\n ~fiber_control();\n\n /** the basic launch function\n * Returns a fiber ID. IDs are not sequential.\n * \\note The ID is really a pointer to a fiber_control::fiber object.\n */\n size_t launch(boost::function<void (void)> fn, \n size_t stacksize = 8192, \n affinity_type worker_affinity = all_affinity());\n\n\n /**\n * Waits for all functions to join\n */\n void join();\n\n\n /**\n * Returns the number of workers\n */\n size_t num_workers() {\n return nworkers;\n }\n\n /**\n * Returns the number of threads that have yet to join\n */\n inline size_t num_threads() {\n return fibers_active.value;\n }\n\n /**\n * Returns the total number threads ever created\n */\n inline size_t total_threads_created() {\n return fiber_id_counter.value;\n }\n /**\n * Sets the TLS deletion function. The deletion function will be called\n * on every non-NULL TLS value.\n */\n void set_tls_deleter(void (*deleter)(void*));\n\n /**\n * Gets the TLS value. Defaults to NULL.\n * Note that this function will only work within a fiber.\n */\n static void* get_tls();\n /**\n * Sets the TLS value.\n * Note that this function will only work within a fiber.\n * If the value is not NULL, and the deletion function is set by\n * set_tls_deleter(), the deleter will be called on the value on\n * fiber termination.\n */\n static void set_tls(void* value);\n\n /**\n * Kills the current fiber.\n * Note that this function will only work within a fiber.\n * Implodes dramatically if called from outside a fiber.\n */\n static void exit();\n\n /**\n * Yields to another fiber.\n * Note that this function will only work within a fiber.\n * If called from outside a fiber, returns immediately.\n */\n static void yield();\n\n\n /**\n * Yields to another fiber of the same affinity.\n * Note that this function will only work within a fiber.\n * If called from outside a fiber, returns immediately.\n */\n static void fast_yield();\n\n\n /**\n * Returns true if the current worker has other fiber waiting on its queue\n */\n static bool worker_has_fibers_on_queue();\n\n\n /**\n * Returns true if the current worker has other priority fibers waiting on \n * its queue\n */\n static bool worker_has_priority_fibers_on_queue();\n\n\n /// True if the singleton instance was created\n static bool instance_created; \n static size_t instance_construct_params_nworkers; \n static size_t instance_construct_params_affinity_base;\n\n /**\n * Sets the fiber control construction parameters.\n * Fails with an assertion failure if the instance has already been created.\n * Must be called prior to any other calls to get_instance()\n * \\param nworkers Number of worker threads to spawn. If set to 0,\n * the number of workers will be automatically determined\n * based on the number of cores the system has.\n * \\param affinity_base First worker will have CPU affinity equal to \n * affinity_base. Second will be affinity_base + 1, etc.\n * Defaults to 0.\n */\n static void instance_set_parameters(size_t nworkers,\n size_t affinity_base);\n\n /**\n * Gets a reference to the main fiber control singleton\n */\n static fiber_control& get_instance();\n\n /**\n * Returns the current fiber handle.\n * Note that fiber handles are not sequential, and are really a\n * pointer to an internal datastructure.\n * If called from within a fiber, returns a non-zero value.\n * If called out outsize a fiber, returns 0.\n */\n static size_t get_tid();\n\n\n /**\n * Returns true if the calling thread is in a fiber, false otherwise.\n */\n static bool in_fiber();\n\n /**\n * Returns the worker managing the current fiber.\n * Worker IDs are sequential.\n * If called from outside a fiber, returns (size_t)(-1)\n */\n static size_t get_worker_id();\n\n\n /**\n * Atomically deschedules the current thread and unlocks the mutex.\n *\n * deschedule_self() and schedule_tid() must be managed carefully\n * to avoid race conditions. i.e. schedule_tid() happending before\n * deschedule_self().\n *\n * To support this correctly, the descheduling must be paired together\n * with a mutex.\n *\n * For instance, to use this to implement a promise.\n * \\code\n * // descheduling fiber\n * pthread_mutex_lock(&lock);\n * if ( ... promise not ready ...) {\n * deschedule_self(&lock);\n * } else {\n * pthread_mutex_unlock(&lock);\n * }\n * ... use the promise ...\n * \\endcode\n *\n *\n * The promise execution thread then must do the following\n * \\code\n * ... tid contains the fiber ID to wake when promise is done\n * ... compute promise...\n * pthread_mutex_lock(&lock); // same lock as above\n * ... set promise completion...\n * schedule_tid(tid); // wake up the fiber\n * pthread_mutex_unlock(&lock);\n * \\endcode\n */\n static void deschedule_self(pthread_mutex_t* lock);\n\n /**\n * Schedules a fiber for execution.\n * If this fiber was previously descheduled by\n * deschedule_self(), the fiber is scheduled for execution.\n * Otherwise, nothing happens. Some care must be taken to avoid race\n * conditions. See the deschedule_self() function for details.\n * This thread by default will be stuck at the head of queue\n * and will wake up quickly.\n *\n * \\param priority If true, thread will be placed at the head\n * of the scheduler. If false, it will be placed at the tail\n * of the scheduler\n */\n static void schedule_tid(size_t tid, bool priority = true);\n};\n\n}\n\n#endif\n\n" }, { "path": "src/graphlab/parallel/fiber_group.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include <boost/bind.hpp>\n#include <graphlab/parallel/fiber_group.hpp>\n#include <graphlab/logger/assertions.hpp>\nnamespace graphlab {\n\nvoid fiber_group::invoke(const boost::function<void (void)>& spawn_function, \n fiber_group* group) {\n spawn_function();\n group->decrement_running_counter();\n}\n\n\nvoid fiber_group::launch(const boost::function<void (void)> &spawn_function) {\n launch(spawn_function, affinity);\n}\n\n\nvoid fiber_group::launch(const boost::function<void (void)> &spawn_function,\n affinity_type worker_affinity) {\n increment_running_counter();\n fiber_control::get_instance().launch(boost::bind(invoke, spawn_function, this), \n stacksize,\n worker_affinity); \n}\n\nvoid fiber_group::launch(const boost::function<void (void)> &spawn_function,\n size_t worker_affinity) {\n increment_running_counter();\n fiber_group::affinity_type affinity;\n affinity.set_bit(worker_affinity);\n fiber_control::get_instance().launch(boost::bind(invoke, spawn_function, this), \n stacksize,\n affinity); \n}\n\n\nvoid fiber_group::join() {\n join_lock.lock();\n // no one else is waiting\n ASSERT_EQ(join_waiting, false);\n // otherwise, we need to wait\n join_waiting = true;\n while(threads_running.value != 0) {\n join_cond.wait(join_lock);\n }\n join_waiting = false;\n join_lock.unlock();\n}\n\n} // namespace graphlab\n\n" }, { "path": "src/graphlab/parallel/fiber_group.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_FIBER_GROUP_HPP\n#define GRAPHLAB_FIBER_GROUP_HPP\n#include <graphlab/parallel/fiber_control.hpp>\n#include <graphlab/parallel/atomic.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\nnamespace graphlab {\n\n/**\n * Defines a group of fibers. Analogous to the thread_group, but is meant\n * to run only little user-mode threads. It is important that fibers never \n * block, since there is no way to context switch out from a blocked fiber.\n * The fiber_group uses the fiber_control singleton instance to manage its\n * fibers.\n */\nclass fiber_group {\n public:\n typedef fiber_control::affinity_type affinity_type;\n\n private:\n size_t stacksize;\n affinity_type affinity;\n atomic<size_t> threads_running;\n mutex join_lock;\n // to be triggered once the threads_running counter becomes 0\n conditional join_cond; \n // set to true if someone is waiting on a join()\n bool join_waiting;\n\n inline void increment_running_counter() {\n threads_running.inc();\n }\n\n inline void decrement_running_counter() {\n // now, a bit of care is needed here\n size_t r = threads_running.dec();\n if (r == 0) {\n join_lock.lock();\n if (join_waiting) {\n join_cond.signal();\n }\n join_lock.unlock();\n }\n }\n\n // wraps the call so that we can do the appropriate termination\n static void invoke(const boost::function<void (void)>& spawn_function, \n fiber_group* group);\n\n public:\n\n\n fiber_group(size_t stacksize = 8192, \n affinity_type affinity = fiber_control::all_affinity()) : \n stacksize(stacksize), \n affinity(affinity),\n join_waiting(false) { }\n\n\n /**\n * Sets the stacksize of each fiber.\n * Only takes effect for threads launched after this.\n */\n inline void set_stacksize(size_t new_stacksize) {\n stacksize = new_stacksize;\n }\n\n\n /**\n * Sets the affinity for each fiber.\n * Only takes effect for threads launched after this.\n */\n inline void set_affinity(affinity_type new_affinity) {\n affinity = new_affinity;\n }\n\n /**\n * Launch a single thread which calls spawn_function.\n */\n void launch(const boost::function<void (void)> &spawn_function);\n \n\n\n /**\n * Launch a single thread which calls spawn_function with worker affinity.\n */\n void launch(const boost::function<void (void)> &spawn_function, \n affinity_type worker_affinity);\n\n\n /**\n * Launch a single thread which calls spawn_function with a single \n * thread affinity\n */\n void launch(const boost::function<void (void)> &spawn_function,\n size_t worker_affinity);\n\n /** Waits for all threads to complete execution. const char* exceptions\n * thrown by threads are forwarded to the join() function.\n */\n void join();\n\n /// Returns the number of running threads.\n inline size_t running_threads() {\n return threads_running;\n }\n //\n //! Destructor. Waits for all threads to complete execution\n inline ~fiber_group(){ join(); }\n\n};\n\n} // namespace graphlab \n#endif\n" }, { "path": "src/graphlab/parallel/fiber_remote_request.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_FIBER_RPC_FUTURE_HPP\n#define GRAPHLAB_FIBER_RPC_FUTURE_HPP\n#include <graphlab/rpc/request_future.hpp>\n#include <graphlab/rpc/request_reply_handler.hpp>\n#include <graphlab/parallel/fiber_control.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\nnamespace graphlab {\n\n/**\n * A implementation of the ireply_container interface\n * that will wait for rpc requests, but if the request is issued from within \n * a fiber, will deschedule the fiber.\n */\nstruct fiber_reply_container: public dc_impl::ireply_container {\n dc_impl::blob val;\n mutex lock;\n conditional cond;\n // if wait is in a fiber, this will contain the ID of the fiber to wake up\n // If 0, the wait is not in a fiber.\n size_t waiting_tid;\n // true when the blob is assigned\n bool valready;\n\n fiber_reply_container():waiting_tid(0),valready(false) { }\n\n ~fiber_reply_container() {\n val.free();\n }\n\n void wait() {\n if (fiber_control::in_fiber()) { \n // if I am in a fiber, use the deschedule mechanism \n lock.lock();\n waiting_tid = fiber_control::get_tid();\n while(!valready) {\n // set the waiting tid value\n // deschedule myself. This will deschedule the fiber\n // and unlock the lock atomically\n fiber_control::deschedule_self(&lock.m_mut);\n // unlock the condition variable, this does not re-lock the lock\n lock.lock();\n }\n lock.unlock();\n } else {\n // Otherwise use the condition variable\n waiting_tid = 0;\n lock.lock();\n while(!valready) cond.wait(lock);\n lock.unlock();\n }\n }\n\n void receive(procid_t source, dc_impl::blob b) {\n lock.lock();\n val = b;\n valready = true;\n if (waiting_tid) {\n // it is a fiber. wake it up.\n fiber_control::schedule_tid(waiting_tid);\n } else {\n // not in fiber. This is just a condition signal\n cond.signal();\n }\n lock.unlock();\n }\n bool ready() const {\n return valready;\n }\n\n dc_impl::blob& get_blob() {\n return val;\n }\n};\n\n\n#if DOXYGEN_DOCUMENTATION\n\n\n/**\n * \\brief Performs a nonblocking RPC call to the target machine\n * to run the provided function pointer which has an expected return value.\n *\n * fiber_remote_request() calls the function \"fn\" on a target remote machine.\n * Provided arguments are serialized and sent to the target.\n * Therefore, all arguments are necessarily transmitted by value.\n * If the target function has a return value, it is sent back to calling\n * machine. fiber_remote_request() returns immediately a \\ref\n * graphlab::request_future object which will allow you wait for the return\n * value.\n *\n * fiber_remote_request() has an identical interface to \n * \\ref graphlab::distributed_control::future_remote_request() , but has the \n * additional capability that if a \\ref graphlab::request_future::wait() is \n * called on the request while within a fiber, it deschedules the fiber and\n * context switches, returning only when the future is ready. This allows\n * the future to be used from within a fiber.\n *\n * Since this function is not a member of the distributed_control class,\n * it uses the function \\ref distributed_control::get_instance() to obtain\n * the last instance of the distribute_control class created. This should be\n * sufficient for most use cases.\n *\n * \\ref graphlab::object_fiber_remote_request is the version of this function\n * for remotely calling class member functions.\n *\n * Example:\n * \\code\n * // A print function is defined\n * int add_one(int i) {\n * return i + 1;\n * }\n *\n * ... ...\n * // call the add_one function on machine 1\n * int i = 10;\n * graphlab::request_future<int> ret = fiber_remote_request(1, add_one, i);\n * // this is safe to do within a fiber as it will not halt other fibers.\n * int result = ret();\n * // result will be 11\n * \\endcode\n *\n * \\see graphlab::distributed_control::remote_request\n * graphlab::distributed_control::future_remote_request\n * graphlab::object_fiber_remote_request\n *\n * \\param targetmachine The ID of the machine to run the function on\n * \\param fn The function to run on the target machine. Must be a pointer to\n * member function in the owning object.\n * \\param ... The arguments to send to Fn. Arguments must be serializable.\n * and must be castable to the target types.\n *\n * \\returns Returns a future templated around the same type as the return \n * value of the called function\n */\n request_future<RetVal> fiber_remote_request(procid_t targetmachine, Fn fn, ...);\n\n\n\n/**\n * \\brief Performs a nonblocking RPC call to the target machine\n * to run the provided function pointer which has an expected return value.\n *\n * object_fiber_remote_request() calls the function \"fn\" on a target remote machine.\n * Provided arguments are serialized and sent to the target.\n * Therefore, all arguments are necessarily transmitted by value.\n * If the target function has a return value, it is sent back to calling\n * machine. object_fiber_remote_request() returns immediately a \\ref\n * graphlab::request_future object which will allow you wait for the return\n * value.\n *\n * object_fiber_remote_request() has an identical interface to \n * \\ref graphlab::dc_dist_object::future_remote_request() , but has the \n * additional capability that if a \\ref graphlab::request_future::wait() is \n * called on the request while within a fiber, it deschedules the fiber and\n * context switches, returning only when the future is ready. This allows\n * the future to be used from within a fiber.\n *\n * Since this function is not a member of the \\ref dc_dist_object class,\n * it needs to be provided a reference to the owning object's dc_dist_object.\n *\n * \\ref graphlab::fiber_remote_request is the version of this function\n * for remotely calling global functions.\n *\n * Example:\n * \\code\n * // A print function is defined in the distributed object\n * class distributed_obj_example {\n * graphlab::dc_dist_object<distributed_obj_example> rmi;\n * ... initialization and constructor ...\n * private:\n * int add_one(int i) {\n * return i + 1;\n * }\n * public:\n * int add_one_from_machine_1(int i) {\n * // calls the add_one function on machine 1 with the argument i\n * // this call returns immediately\n * graphlab::request_future<int> future =\n * object_future_remote_request(rmi, 1, &distributed_obj_example::add_one, i);\n *\n * // ... we can do other stuff here\n * // then when we want the answer\n * // this is safe to do within a fiber as it will not halt other fibers.\n * int result = future();\n * return result;\n * }\n * }\n * \\endcode\n *\n * \\see graphlab::dc_dist_object::remote_request\n * graphlab::dc_dist_object::future_remote_request\n * graphlab::fiber_remote_request\n *\n * \\param rmiobj The dc_dist_object to use to send the request.\n * \\param targetmachine The ID of the machine to run the function on\n * \\param fn The function to run on the target machine. Must be a pointer to\n * member function in the owning object.\n * \\param ... The arguments to send to Fn. Arguments must be serializable.\n * and must be castable to the target types.\n *\n * \\returns Returns a future templated around the same type as the return \n * value of the called function\n */\n request_future<RetVal> object_fiber_remote_request(dc_dist_object<T> rmiobj,\n procid_t targetmachine, \n Fn fn, ...);\n\n\n\n#endif\n\n\n\n#include <boost/preprocessor.hpp>\n#include <graphlab/rpc/function_arg_types_def.hpp>\n\n#define GENARGS(Z,N,_) BOOST_PP_CAT(T, N) BOOST_PP_CAT(i, N)\n#define GENI(Z,N,_) BOOST_PP_CAT(i, N)\n#define GENT(Z,N,_) BOOST_PP_CAT(T, N)\n#define GENARC(Z,N,_) arc << BOOST_PP_CAT(i, N);\n\n#define REQUEST_INTERFACE_GENERATOR(Z,N,ARGS) \\\ntemplate<typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\n BOOST_PP_TUPLE_ELEM(1,0,ARGS) (procid_t target, \\\n F remote_function BOOST_PP_COMMA_IF(N) \\\n BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n request_future<__GLRPC_FRESULT> reply(new fiber_reply_container); \\\n distributed_control* dc = distributed_control::get_instance(); \\\n ASSERT_NE(dc, NULL); \\\n dc->custom_remote_request(target, reply.get_handle(), STANDARD_CALL, remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENI ,_) ); \\\n return reply; \\\n} \n\nBOOST_PP_REPEAT(7, REQUEST_INTERFACE_GENERATOR, (request_future<__GLRPC_FRESULT> fiber_remote_request) )\n\n#include <graphlab/rpc/function_arg_types_undef.hpp>\n\n#include <graphlab/rpc/mem_function_arg_types_def.hpp>\n#define OBJECT_REQUEST_INTERFACE_GENERATOR(Z,N,ARGS) \\\ntemplate<typename RMI, typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\n BOOST_PP_TUPLE_ELEM(1,0,ARGS) (RMI& rmi, \\\n procid_t target, \\\n F remote_function BOOST_PP_COMMA_IF(N) \\\n BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n request_future<__GLRPC_FRESULT> reply(new fiber_reply_container); \\\n rmi.custom_remote_request(target, reply.get_handle(), STANDARD_CALL, remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENI ,_) ); \\\n return reply; \\\n} \n\n\n\n /*\n Generates the interface functions. 3rd argument is a tuple (interface name, issue name, flags)\n */\nBOOST_PP_REPEAT(7, OBJECT_REQUEST_INTERFACE_GENERATOR, (request_future<__GLRPC_FRESULT> object_fiber_remote_request) )\n\n#include <graphlab/rpc/mem_function_arg_types_undef.hpp>\n\n#undef OBJECT_REQUEST_INTERFACE_GENERATOR\n#undef REQUEST_INTERFACE_GENERATOR\n#undef GENARC\n#undef GENT\n#undef GENI\n#undef GENARGS\n\n} // namespace graphlab\n\n#endif\n" }, { "path": "src/graphlab/parallel/lockfree_push_back.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_PARALLEL_LOCKFREE_PUSHBACK_HPP\n#define GRAPHLAB_PARALLEL_LOCKFREE_PUSHBACK_HPP\n#include <graphlab/parallel/atomic.hpp>\n\nnamespace graphlab {\n\nnamespace lockfree_push_back_impl {\n struct idx_ref {\n idx_ref(): reference_count(0), idx(0) { }\n idx_ref(size_t idx): reference_count(0), idx(idx) { }\n \n volatile int reference_count;\n atomic<size_t> idx;\n enum {\n MAX_REF = 65536\n };\n \n inline void inc_ref() {\n while (1) {\n int curref = reference_count;\n if ((curref & MAX_REF) == 0 &&\n atomic_compare_and_swap(reference_count, curref, curref + 1)) {\n break;\n }\n } \n }\n\n inline void wait_till_no_ref() {\n while((reference_count & (MAX_REF - 1)) != 0);\n }\n \n inline void dec_ref() {\n __sync_fetch_and_sub(&reference_count, 1);\n }\n\n inline void flag_ref() {\n __sync_fetch_and_xor(&reference_count, MAX_REF);\n }\n \n inline size_t inc_idx() {\n return idx.inc_ret_last();\n }\n\n inline size_t inc_idx(size_t n) {\n return idx.inc_ret_last(n);\n }\n };\n} // lockfree_push_back_impl\n \n/**\n * Provides a lock free way to insert elements to the end\n * of a container. Container must provide 3 functions.\n * - T& operator[](size_t idx)\n * - void resize(size_t len)\n * - size_t size()\n *\n * resize(n) must guarantee that size() >= n.\n * T& operator[](size_t idx) must succeed for idx < size() and must be \n * safely executeable in parallel.\n * size() must be safely executeable in parallel with resize().\n */\ntemplate <typename Container, typename T = typename Container::value_type>\nclass lockfree_push_back {\n private:\n Container& container;\n lockfree_push_back_impl::idx_ref cur;\n mutex mut;\n float scalefactor;\n public:\n lockfree_push_back(Container& container, size_t startidx, float scalefactor = 2):\n container(container),cur(startidx), scalefactor(scalefactor) { }\n\n size_t size() const {\n return cur.idx.value;\n }\n\n void set_size(size_t s) {\n cur.idx.value = s;\n }\n\n template <typename Iterator>\n size_t push_back(Iterator begin, Iterator end) {\n size_t numel = std::distance(begin, end);\n size_t putpos = cur.inc_idx(numel);\n size_t endidx = putpos + numel;\n while(1) {\n cur.inc_ref();\n if (endidx <= container.size()) {\n while(putpos < endidx) {\n container[putpos] = (*begin);\n ++putpos; ++begin;\n }\n cur.dec_ref();\n break;\n }\n else {\n cur.dec_ref();\n\n if (mut.try_lock()) {\n // ok. we need to resize\n // flag the reference and wait till there are no more references\n cur.flag_ref();\n cur.wait_till_no_ref();\n // we are exclusive here. resize\n if (endidx > container.size()) {\n container.resize(std::max<size_t>(endidx, container.size() * scalefactor));\n }\n while(putpos < endidx) {\n container[putpos] = (*begin);\n ++putpos; ++begin;\n }\n cur.flag_ref();\n mut.unlock();\n break;\n }\n }\n }\n return putpos;\n }\n \n bool query(size_t item, T& value) {\n bool ret = false;\n cur.inc_ref();\n if (item < cur.idx) {\n value = container[item];\n ret = true;\n }\n cur.dec_ref();\n return ret;\n }\n\n T* query(size_t item) {\n T* ret = NULL;\n cur.inc_ref();\n if (item < cur.idx) {\n ret = &(container[item]);\n }\n cur.dec_ref();\n return ret;\n }\n\n bool query_unsafe(size_t item, T& value) {\n bool ret = false;\n if (item < cur.idx) {\n value = container[item];\n ret = true;\n }\n return ret;\n }\n\n T* query_unsafe(size_t item) {\n T* ret = NULL;\n if (item < cur.idx) {\n ret = &(container[item]);\n }\n return ret;\n }\n\n\n size_t push_back(const T& t) {\n size_t putpos = cur.inc_idx();\n while(1) {\n cur.inc_ref();\n if (putpos < container.size()) {\n container[putpos] = t;\n cur.dec_ref();\n break;\n }\n else {\n cur.dec_ref();\n\n if (mut.try_lock()) {\n // ok. we need to resize\n // flag the reference and wait till there are no more references\n cur.flag_ref();\n cur.wait_till_no_ref();\n // we are exclusive here. resize\n if (putpos >= container.size()) {\n container.resize(std::max<size_t>(putpos + 1, container.size() * scalefactor));\n }\n container[putpos] = t;\n cur.flag_ref();\n mut.unlock();\n break;\n }\n }\n }\n return putpos;\n }\n};\n\n} // namespace graphlab\n#endif\n" }, { "path": "src/graphlab/parallel/mutex.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_MUTEX_HPP\n#define GRAPHLAB_MUTEX_HPP\n\n\n#include <pthread.h>\n#include <graphlab/logger/assertions.hpp>\n\n\nnamespace graphlab {\n\n /**\n * \\ingroup util\n *\n * Simple wrapper around pthread's mutex.\n * Before you use, see \\ref parallel_object_intricacies.\n */\n class mutex {\n public:\n // mutable not actually needed\n mutable pthread_mutex_t m_mut;\n /// constructs a mutex\n mutex() {\n int error = pthread_mutex_init(&m_mut, NULL);\n ASSERT_TRUE(!error);\n }\n /** Copy constructor which does not copy. Do not use!\n Required for compatibility with some STL implementations (LLVM).\n which use the copy constructor for vector resize,\n rather than the standard constructor. */\n mutex(const mutex&) {\n int error = pthread_mutex_init(&m_mut, NULL);\n ASSERT_TRUE(!error);\n }\n\n ~mutex(){\n int error = pthread_mutex_destroy( &m_mut );\n ASSERT_TRUE(!error);\n }\n\n // not copyable\n void operator=(const mutex& m) { }\n\n /// Acquires a lock on the mutex\n inline void lock() const {\n int error = pthread_mutex_lock( &m_mut );\n // if (error) std::cout << \"mutex.lock() error: \" << error << std::endl;\n ASSERT_TRUE(!error);\n }\n /// Releases a lock on the mutex\n inline void unlock() const {\n int error = pthread_mutex_unlock( &m_mut );\n ASSERT_TRUE(!error);\n }\n /// Non-blocking attempt to acquire a lock on the mutex\n inline bool try_lock() const {\n return pthread_mutex_trylock( &m_mut ) == 0;\n }\n friend class conditional;\n }; // End of Mutex\n\n\n\n\n /**\n * \\ingroup util\n *\n * Simple wrapper around pthread's recursive mutex.\n * Before you use, see \\ref parallel_object_intricacies.\n */\n class recursive_mutex {\n public:\n // mutable not actually needed\n mutable pthread_mutex_t m_mut;\n /// constructs a mutex\n recursive_mutex() {\n pthread_mutexattr_t attr;\n int error = pthread_mutexattr_init(&attr);\n ASSERT_TRUE(!error);\n error = pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE);\n ASSERT_TRUE(!error);\n error = pthread_mutex_init(&m_mut, &attr);\n ASSERT_TRUE(!error);\n pthread_mutexattr_destroy(&attr);\n }\n /** Copy constructor which does not copy. Do not use!\n Required for compatibility with some STL implementations (LLVM).\n which use the copy constructor for vector resize,\n rather than the standard constructor. */\n recursive_mutex(const recursive_mutex&) {\n pthread_mutexattr_t attr;\n int error = pthread_mutexattr_init(&attr);\n ASSERT_TRUE(!error);\n error = pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE);\n ASSERT_TRUE(!error);\n error = pthread_mutex_init(&m_mut, &attr);\n ASSERT_TRUE(!error);\n pthread_mutexattr_destroy(&attr);\n }\n\n ~recursive_mutex(){\n int error = pthread_mutex_destroy( &m_mut );\n ASSERT_TRUE(!error);\n }\n\n // not copyable\n void operator=(const recursive_mutex& m) { }\n\n /// Acquires a lock on the mutex\n inline void lock() const {\n int error = pthread_mutex_lock( &m_mut );\n // if (error) std::cout << \"mutex.lock() error: \" << error << std::endl;\n ASSERT_TRUE(!error);\n }\n /// Releases a lock on the mutex\n inline void unlock() const {\n int error = pthread_mutex_unlock( &m_mut );\n ASSERT_TRUE(!error);\n }\n /// Non-blocking attempt to acquire a lock on the mutex\n inline bool try_lock() const {\n return pthread_mutex_trylock( &m_mut ) == 0;\n }\n friend class conditional;\n }; // End of Mutex\n\n\n\n\n} // end of graphlab namespace\n\n\n#endif\n" }, { "path": "src/graphlab/parallel/parallel_includes.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n// #include <atomic.hpp>\n// #include <graphlab/parallel/pthread_tools.hpp>\n\n" }, { "path": "src/graphlab/parallel/pthread_tools.cpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <boost/bind.hpp>\n#include <graphlab/macros_def.hpp>\n\nnamespace graphlab { \n\n // Some magic to ensure that keys are created at program startup =========>\n void destroy_tls_data(void* ptr);\n struct thread_keys {\n pthread_key_t GRAPHLAB_TSD_ID;\n thread_keys() : GRAPHLAB_TSD_ID(0) { \n pthread_key_create(&GRAPHLAB_TSD_ID,\n destroy_tls_data);\n }\n };\n // This function is to be called prior to any thread starting\n // execution to ensure that the static member keys is constructed\n // prior to any threads launching\n static pthread_key_t get_tsd_id() {\n static thread_keys keys;\n return keys.GRAPHLAB_TSD_ID;\n }\n // This forces get_tsd_id to be called prior to main.\n static pthread_key_t __unused_init_keys__(get_tsd_id());\n \n // the combination of the two mechanisms above will force the\n // thread local store to be initialized\n // 1: before main\n // 2: before any other global variables which spawn threads\n \n // END MAGIC =============================================================>\n\n// -----------------------------------------------------------------\n// Thread Object Static Members \n// -----------------------------------------------------------------\n \n\n /**\n * Create thread specific data\n */\n thread::tls_data* create_tls_data(size_t thread_id = 0) {\n // Require that the data not yet exist\n assert(pthread_getspecific(get_tsd_id()) == NULL);\n // Create the data\n thread::tls_data* data =\n new thread::tls_data(thread_id);\n assert(data != NULL);\n // Set the data\n pthread_setspecific(get_tsd_id(), data);\n // Return the associated tsd\n return data;\n } // end create the thread specific data\n\n /**\n * This function tries to get the thread specific data. If no\n * thread specific data has been associated with the thread than it\n * is created.\n */\n thread::tls_data& thread::get_tls_data() {\n // get the tsd\n tls_data* tsd =\n reinterpret_cast<tls_data*>\n (pthread_getspecific(get_tsd_id()));\n // If no tsd be has been associated, create one\n if(tsd == NULL) tsd = create_tls_data();\n assert(tsd != NULL);\n return *tsd;\n } // end of get thread specific data\n\n \n /**\n * Create thread specific data\n */\n void destroy_tls_data(void* ptr) {\n thread::tls_data* tsd =\n reinterpret_cast<thread::tls_data*>(ptr);\n if(tsd != NULL) {\n delete tsd;\n }\n } // end destroy the thread specific data\n\n \n\n\n //! Little helper function used to launch threads\n void* thread::invoke(void *_args) {\n void* retval = NULL;\n thread::invoke_args* args = static_cast<thread::invoke_args*>(_args);\n // Create the graphlab thread specific data\n create_tls_data(args->m_thread_id); \n //! Run the users thread code\n try {\n args->spawn_routine();\n }\n catch (const char* msg) {\n retval = (void*)msg;\n }\n //! Delete the arguments \n delete args;\n \n //! Properly kill the thread\n thread_destroy_callback();\n return retval;\n } // end of invoke\n\n \n\n \n\n /**\n * This static method joins the invoking thread with the other\n * thread object. This thread will not return from the join\n * routine until the other thread complets it run.\n */\n void thread::join(thread& other) {\n void *status = NULL;\n // joint the first element\n int error = 0;\n if(other.active()) {\n error = pthread_join( other.m_p_thread, &status);\n if (status != NULL) {\n const char* strstatus = (const char*) status;\n throw strstatus;\n }\n }\n if(error) {\n std::cout << \"Major error in join\" << std::endl;\n std::cout << \"pthread_join() returned error \" << error << std::endl;\n exit(EXIT_FAILURE);\n }\n } // end of join\n\n\n /**\n * Return the number processing units (individual cores) on this\n * system\n */\n size_t thread::cpu_count() {\n#if defined __linux__\n char* jobsStr = getenv(\"GRAPHLAB_THREADS_PER_WORKER\");\n if (jobsStr) {\n int nThreads = atoi(jobsStr);\n if ( nThreads < 2 ) return 2;\n else return nThreads;\n }\n else {\n return sysconf(_SC_NPROCESSORS_CONF);\n }\n#elif defined(__MACH__) && defined(_SC_NPROCESSORS_ONLN)\n return sysconf (_SC_NPROCESSORS_ONLN);\n#elif defined(__MACH__) && defined(HW_NCPU)\n int ncpus = 1;\n size_t len = sizeof(ncpus);\n sysctl((int[2]) {CTL_HW, HW_NCPU}, 2, &ncpus, &len, NULL, 0);\n return ncpus;\n#else\n return 0;\n#endif\n } // end of cpu count\n \n /**\n * Allow defining a callback when thread is destroyed.\n * This is needed at least from Java JNI, where we have to detach\n * thread from JVM before it dies.\n */\n void (*__thr_callback)() = NULL;\n\n void thread::thread_destroy_callback() {\n if (__thr_callback != NULL) __thr_callback();\n }\n \n void thread::set_thread_destroy_callback(void (*callback)()) {\n __thr_callback = callback;\n }\n\n\n// -----------------------------------------------------------------\n// Thread Object Public Members \n// -----------------------------------------------------------------\n\n \n void thread::launch(const boost::function<void (void)> &spawn_routine) {\n get_tsd_id();\n ASSERT_FALSE(thread_started);\n // fill in the thread attributes\n pthread_attr_t attr;\n int error = 0;\n error = pthread_attr_init(&attr);\n ASSERT_TRUE(!error);\n error = pthread_attr_setstacksize(&attr, m_stack_size);\n ASSERT_TRUE(!error);\n error = pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);\n ASSERT_TRUE(!error); \n error =\n pthread_create(&m_p_thread, \n &attr, \n invoke, \n static_cast<void*>(new invoke_args(m_thread_id, \n spawn_routine)) );\n thread_started = true;\n if(error) {\n std::cout << \"Major error in thread_group.launch (pthread_create). Error: \" \n << error << std::endl;\n exit(EXIT_FAILURE);\n }\n // destroy the attribute object\n error = pthread_attr_destroy(&attr);\n ASSERT_TRUE(!error);\n }\n \n void thread::launch(const boost::function<void (void)> &spawn_routine, \n size_t cpu_id){\n get_tsd_id();\n // if this is not a linux based system simply invoke start and\n // return;\n#ifndef __linux__\n launch(spawn_routine);\n return;\n#else\n ASSERT_FALSE(thread_started);\n if (cpu_id == size_t(-1)) {\n launch(spawn_routine);\n return;\n }\n if (cpu_count() > 0) {\n cpu_id = cpu_id % cpu_count();\n }\n else {\n // unknown CPU count\n launch(spawn_routine);\n return;\n }\n \n // fill in the thread attributes\n pthread_attr_t attr;\n int error = 0;\n error = pthread_attr_init(&attr);\n ASSERT_TRUE(!error);\n error = pthread_attr_setstacksize(&attr, m_stack_size);\n ASSERT_TRUE(!error);\n error = pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);\n ASSERT_TRUE(!error);\n\n#ifdef HAS_SET_AFFINITY\n // Set Processor Affinity masks (linux only)\n cpu_set_t cpu_set;\n CPU_ZERO(&cpu_set);\n CPU_SET(cpu_id % CPU_SETSIZE, &cpu_set);\n\n pthread_attr_setaffinity_np(&attr, sizeof(cpu_set), &cpu_set);\n#endif\n \n // Launch the thread\n error = pthread_create(&m_p_thread, \n &attr, \n invoke,\n static_cast<void*>(new invoke_args(m_thread_id, \n spawn_routine)));\n thread_started = true;\n if(error) {\n std::cout << \"Major error in thread_group.launch\" << std::endl;\n std::cout << \"pthread_create() returned error \" << error << std::endl;\n exit(EXIT_FAILURE);\n }\n \n \n \n // destroy the attribute object\n error = pthread_attr_destroy(&attr);\n ASSERT_TRUE(!error);\n#endif\n }\n \n // -----------------------------------------------------------------\n // Thread Group Object Public Members \n // -----------------------------------------------------------------\n // thread group exception forwarding is a little more complicated\n // because it has to be able to catch it on a bunch of threads\n\n void thread_group::invoke(boost::function<void (void)> spawn_function,\n thread_group *group) {\n const char* retval = NULL;\n try {\n spawn_function();\n }\n catch (const char* c) {\n // signal the thread group to join this thread\n retval = c;\n }\n group->mut.lock();\n group->joinqueue.push(std::make_pair(pthread_self(), retval));\n group->cond.signal();\n group->mut.unlock();\n\n }\n \n\n void thread_group::launch(const boost::function<void (void)> &spawn_function) {\n // Create a thread object and launch it. \n // We do not need to keep a copy of the thread around\n thread local_thread(m_thread_counter++);\n mut.lock();\n threads_running++;\n mut.unlock();\n local_thread.launch(boost::bind(thread_group::invoke, spawn_function, this));\n } \n\n\n void thread_group::launch(const boost::function<void (void)> &spawn_function, \n size_t cpu_id) {\n if (cpu_id == size_t(-1)) {\n launch(spawn_function);\n return;\n }\n // Create a thread object\n thread local_thread(m_thread_counter++);\n mut.lock();\n threads_running++;\n mut.unlock();\n local_thread.launch(boost::bind(thread_group::invoke, spawn_function, this), \n cpu_id);\n } // end of launch\n\n void thread_group::join() {\n mut.lock();\n while(threads_running > 0) {\n // if no threads are joining. wait\n while (joinqueue.empty()) cond.wait(mut); \n // a thread is joining\n std::pair<pthread_t, const char*> joining_thread = joinqueue.front();\n joinqueue.pop();\n threads_running--;\n // Reset the thread counter after killing all threads\n if(threads_running == 0) m_thread_counter = 0;\n // unlock here since I might be in join for a little while\n mut.unlock();\n void *unusedstatus = NULL;\n pthread_join(joining_thread.first, &unusedstatus);\n // if there is a return value\n // throw it. It is safe to throw here since I have the mutex unlocked.\n if (joining_thread.second) {\n throw(joining_thread.second);\n }\n mut.lock();\n }\n mut.unlock(); \n } // end of join\n\n\n} // end of namespace graphlab\n\n" }, { "path": "src/graphlab/parallel/pthread_tools.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_PTHREAD_TOOLS_HPP\n#define GRAPHLAB_PTHREAD_TOOLS_HPP\n\n\n#include <cstdlib>\n#include <pthread.h>\n#include <semaphore.h>\n#include <sched.h>\n#include <signal.h>\n#include <sys/time.h>\n#include <vector>\n#include <list>\n#include <queue>\n#include <iostream>\n#include <boost/function.hpp>\n#include <graphlab/logger/assertions.hpp>\n#include <graphlab/parallel/atomic_ops.hpp>\n#include <graphlab/util/generics/any.hpp>\n#include <graphlab/util/branch_hints.hpp>\n#include <boost/unordered_map.hpp>\n#undef _POSIX_SPIN_LOCKS\n#define _POSIX_SPIN_LOCKS -1\n\n\n#include <graphlab/parallel/mutex.hpp>\n\n\n\n\nnamespace graphlab {\n\n\n\n\n\n#if _POSIX_SPIN_LOCKS >= 0\n /**\n * \\ingroup util\n *\n * Wrapper around pthread's spinlock.\n *\n * Before you use, see \\ref parallel_object_intricacies.\n */\n class spinlock {\n private:\n // mutable not actually needed\n mutable pthread_spinlock_t m_spin;\n public:\n /// constructs a spinlock\n spinlock () {\n int error = pthread_spin_init(&m_spin, PTHREAD_PROCESS_PRIVATE);\n ASSERT_TRUE(!error);\n }\n \n /** Copy constructor which does not copy. Do not use!\n Required for compatibility with some STL implementations (LLVM).\n which use the copy constructor for vector resize, \n rather than the standard constructor. */\n spinlock(const spinlock&) {\n int error = pthread_spin_init(&m_spin, PTHREAD_PROCESS_PRIVATE);\n ASSERT_TRUE(!error);\n }\n \n // not copyable\n void operator=(const spinlock& m) { }\n\n\n /// Acquires a lock on the spinlock\n inline void lock() const { \n int error = pthread_spin_lock( &m_spin );\n ASSERT_TRUE(!error);\n }\n /// Releases a lock on the spinlock\n inline void unlock() const {\n int error = pthread_spin_unlock( &m_spin );\n ASSERT_TRUE(!error);\n }\n /// Non-blocking attempt to acquire a lock on the spinlock\n inline bool try_lock() const {\n return pthread_spin_trylock( &m_spin ) == 0;\n }\n ~spinlock(){\n int error = pthread_spin_destroy( &m_spin );\n ASSERT_TRUE(!error);\n }\n friend class conditional;\n }; // End of spinlock\n#define SPINLOCK_SUPPORTED 1\n#else\n //! if spinlock not supported, it is typedef it to a mutex.\n typedef mutex spinlock;\n#define SPINLOCK_SUPPORTED 0\n#endif\n\n \n /**\n * \\ingroup util\n *If pthread spinlock is not implemented, \n * this provides a simple alternate spin lock implementation.\n *\n * Before you use, see \\ref parallel_object_intricacies.\n */\n class simple_spinlock {\n private:\n // mutable not actually needed\n mutable volatile char spinner;\n public:\n /// constructs a spinlock\n simple_spinlock () {\n spinner = 0;\n }\n \n /** Copy constructor which does not copy. Do not use!\n Required for compatibility with some STL implementations (LLVM).\n which use the copy constructor for vector resize, \n rather than the standard constructor. */\n simple_spinlock(const simple_spinlock&) {\n spinner = 0;\n }\n \n // not copyable\n void operator=(const simple_spinlock& m) { }\n\n \n /// Acquires a lock on the spinlock\n inline void lock() const { \n while(spinner == 1 || __sync_lock_test_and_set(&spinner, 1));\n }\n /// Releases a lock on the spinlock\n inline void unlock() const {\n __sync_synchronize();\n spinner = 0;\n }\n /// Non-blocking attempt to acquire a lock on the spinlock\n inline bool try_lock() const {\n return (__sync_lock_test_and_set(&spinner, 1) == 0);\n }\n ~simple_spinlock(){\n ASSERT_TRUE(spinner == 0);\n }\n };\n \n\n /**\n * \\ingroup util\n *If pthread spinlock is not implemented, \n * this provides a simple alternate spin lock implementation.\n *\n * Before you use, see \\ref parallel_object_intricacies.\n */\n class padded_simple_spinlock {\n private:\n // mutable not actually needed\n mutable volatile char spinner;\n // char padding[63];\n public:\n /// constructs a spinlock\n padded_simple_spinlock () {\n spinner = 0;\n }\n \n /** Copy constructor which does not copy. Do not use!\n Required for compatibility with some STL implementations (LLVM).\n which use the copy constructor for vector resize, \n rather than the standard constructor. */\n padded_simple_spinlock(const padded_simple_spinlock&) {\n spinner = 0;\n }\n \n // not copyable\n void operator=(const padded_simple_spinlock& m) { }\n\n \n /// Acquires a lock on the spinlock\n inline void lock() const { \n while(spinner == 1 || __sync_lock_test_and_set(&spinner, 1));\n }\n /// Releases a lock on the spinlock\n inline void unlock() const {\n __sync_synchronize();\n spinner = 0;\n }\n /// Non-blocking attempt to acquire a lock on the spinlock\n inline bool try_lock() const {\n return (__sync_lock_test_and_set(&spinner, 1) == 0);\n }\n ~padded_simple_spinlock(){\n ASSERT_TRUE(spinner == 0);\n }\n };\n \n\n\n\n /**\n * \\ingroup util\n * Wrapper around pthread's condition variable\n *\n * Before you use, see \\ref parallel_object_intricacies.\n */\n class conditional {\n private:\n mutable pthread_cond_t m_cond;\n\n public:\n conditional() {\n int error = pthread_cond_init(&m_cond, NULL);\n ASSERT_TRUE(!error);\n }\n \n /** Copy constructor which does not copy. Do not use!\n Required for compatibility with some STL implementations (LLVM).\n which use the copy constructor for vector resize, \n rather than the standard constructor. */\n conditional(const conditional &) {\n int error = pthread_cond_init(&m_cond, NULL);\n ASSERT_TRUE(!error);\n }\n \n // not copyable\n void operator=(const conditional& m) { }\n\n \n /// Waits on condition. The mutex must already be acquired. Caller\n /// must be careful about spurious wakes.\n inline void wait(const mutex& mut) const {\n int error = pthread_cond_wait(&m_cond, &mut.m_mut);\n ASSERT_TRUE(!error);\n }\n /// Like wait() but with a time limit of \"sec\" seconds\n inline int timedwait(const mutex& mut, size_t sec) const {\n struct timespec timeout;\n struct timeval tv;\n struct timezone tz;\n gettimeofday(&tv, &tz);\n timeout.tv_nsec = tv.tv_usec * 1000;\n timeout.tv_sec = tv.tv_sec + (time_t)sec;\n return pthread_cond_timedwait(&m_cond, &mut.m_mut, &timeout);\n }\n /// Like wait() but with a time limit of \"ms\" milliseconds\n inline int timedwait_ms(const mutex& mut, size_t ms) const {\n struct timespec timeout;\n struct timeval tv;\n gettimeofday(&tv, NULL);\n // convert ms to s and ns\n size_t s = ms / 1000;\n ms = ms % 1000;\n size_t ns = ms * 1000000;\n // convert timeval to timespec\n timeout.tv_nsec = tv.tv_usec * 1000;\n timeout.tv_sec = tv.tv_sec;\n \n // add the time\n timeout.tv_nsec += (suseconds_t)ns;\n timeout.tv_sec += (time_t)s;\n // shift the nsec to sec if overflow\n if (timeout.tv_nsec > 1000000000) {\n timeout.tv_sec ++;\n timeout.tv_nsec -= 1000000000;\n }\n return pthread_cond_timedwait(&m_cond, &mut.m_mut, &timeout);\n }\n /// Like wait() but with a time limit of \"ns\" nanoseconds\n inline int timedwait_ns(const mutex& mut, size_t ns) const {\n struct timespec timeout;\n struct timeval tv;\n gettimeofday(&tv, NULL);\n assert(ns > 0);\n // convert ns to s and ns\n size_t s = ns / 1000000;\n ns = ns % 1000000;\n\n // convert timeval to timespec\n timeout.tv_nsec = tv.tv_usec * 1000;\n timeout.tv_sec = tv.tv_sec;\n \n // add the time\n timeout.tv_nsec += (suseconds_t)ns;\n timeout.tv_sec += (time_t)s;\n // shift the nsec to sec if overflow\n if (timeout.tv_nsec > 1000000000) {\n timeout.tv_sec ++;\n timeout.tv_nsec -= 1000000000;\n }\n return pthread_cond_timedwait(&m_cond, &mut.m_mut, &timeout);\n }\n /// Signals one waiting thread to wake up\n inline void signal() const {\n int error = pthread_cond_signal(&m_cond);\n ASSERT_TRUE(!error);\n }\n /// Wakes up all waiting threads\n inline void broadcast() const {\n int error = pthread_cond_broadcast(&m_cond);\n ASSERT_TRUE(!error);\n }\n ~conditional() {\n int error = pthread_cond_destroy(&m_cond);\n ASSERT_TRUE(!error);\n }\n }; // End conditional\n\n\n#ifdef __APPLE__\n /**\n * Custom implementation of a semaphore.\n *\n * Before you use, see \\ref parallel_object_intricacies.\n */\n class semaphore {\n private:\n conditional cond;\n mutex mut;\n mutable volatile size_t semvalue;\n mutable volatile size_t waitercount;\n\n public:\n semaphore() {\n semvalue = 0;\n waitercount = 0;\n }\n /** Copy constructor which does not copy. Do not use!\n Required for compatibility with some STL implementations (LLVM).\n which use the copy constructor for vector resize, \n rather than the standard constructor. */\n semaphore(const semaphore&) {\n semvalue = 0;\n waitercount = 0;\n }\n \n // not copyable\n void operator=(const semaphore& m) { }\n\n inline void post() const {\n mut.lock();\n if (waitercount > 0) {\n cond.signal();\n }\n semvalue++;\n mut.unlock();\n }\n inline void wait() const {\n mut.lock();\n waitercount++;\n while (semvalue == 0) {\n cond.wait(mut);\n }\n waitercount--;\n semvalue--;\n mut.unlock();\n }\n ~semaphore() {\n ASSERT_TRUE(waitercount == 0);\n ASSERT_TRUE(semvalue == 0);\n }\n }; // End semaphore\n#else\n /**\n * Wrapper around pthread's semaphore\n *\n * Before you use, see \\ref parallel_object_intricacies.\n */\n class semaphore {\n private:\n mutable sem_t m_sem;\n\n public:\n semaphore() {\n int error = sem_init(&m_sem, 0,0);\n ASSERT_TRUE(!error);\n }\n \n /** Copy constructor with does not copy. Do not use!\n Required for compatibility with some STL implementations (LLVM).\n which use the copy constructor for vector resize, \n rather than the standard constructor. */\n semaphore(const semaphore&) {\n int error = sem_init(&m_sem, 0,0);\n ASSERT_TRUE(!error);\n }\n \n // not copyable\n void operator=(const semaphore& m) { }\n\n inline void post() const {\n int error = sem_post(&m_sem);\n ASSERT_TRUE(!error);\n }\n inline void wait() const {\n int error = sem_wait(&m_sem);\n ASSERT_TRUE(!error);\n }\n ~semaphore() {\n int error = sem_destroy(&m_sem);\n ASSERT_TRUE(!error);\n }\n }; // End semaphore\n#endif\n \n\n#define atomic_xadd(P, V) __sync_fetch_and_add((P), (V))\n#define cmpxchg(P, O, N) __sync_val_compare_and_swap((P), (O), (N))\n#define atomic_inc(P) __sync_add_and_fetch((P), 1)\n#define atomic_add(P, V) __sync_add_and_fetch((P), (V))\n#define atomic_set_bit(P, V) __sync_or_and_fetch((P), 1<<(V))\n#define cpu_relax() asm volatile(\"pause\\n\": : :\"memory\")\n\n /**\n * \\class spinrwlock\n * rwlock built around \"spinning\"\n * source adapted from http://locklessinc.com/articles/locks/\n * \"Scalable Reader-Writer Synchronization for Shared-Memory Multiprocessors\"\n * John Mellor-Crummey and Michael Scott\n */\n class spinrwlock {\n\n union rwticket {\n unsigned u;\n unsigned short us;\n __extension__ struct {\n unsigned char write;\n unsigned char read;\n unsigned char users;\n } s;\n };\n mutable bool writing;\n mutable volatile rwticket l;\n public:\n spinrwlock() {\n memset(const_cast<rwticket*>(&l), 0, sizeof(rwticket));\n }\n inline void writelock() const {\n unsigned me = atomic_xadd(&l.u, (1<<16));\n unsigned char val = (unsigned char)(me >> 16);\n \n while (val != l.s.write) asm volatile(\"pause\\n\": : :\"memory\");\n writing = true;\n }\n\n inline void wrunlock() const{\n rwticket t = *const_cast<rwticket*>(&l);\n\n t.s.write++;\n t.s.read++;\n \n *(volatile unsigned short *) (&l) = t.us;\n writing = false;\n __asm(\"mfence\");\n }\n\n inline void readlock() const {\n unsigned me = atomic_xadd(&l.u, (1<<16));\n unsigned char val = (unsigned char)(me >> 16);\n \n while (val != l.s.read) asm volatile(\"pause\\n\": : :\"memory\");\n l.s.read++;\n }\n\n inline void rdunlock() const {\n atomic_inc(&l.s.write);\n }\n \n inline void unlock() const {\n if (!writing) rdunlock();\n else wrunlock();\n }\n };\n\n\n\n#define RW_WAIT_BIT 0\n#define RW_WRITE_BIT 1\n#define RW_READ_BIT 2\n\n#define RW_WAIT 1\n#define RW_WRITE 2\n#define RW_READ 4\n\n struct spinrwlock2 {\n mutable unsigned int l;\n\n spinrwlock2():l(0) {}\n void writelock() const {\n while (1) {\n unsigned state = l;\n\n /* No readers or writers? */\n if (state < RW_WRITE)\n {\n /* Turn off RW_WAIT, and turn on RW_WRITE */\n if (cmpxchg(&l, state, RW_WRITE) == state) return;\n\n /* Someone else got there... time to wait */\n state = l;\n }\n\n /* Turn on writer wait bit */\n if (!(state & RW_WAIT)) atomic_set_bit(&l, RW_WAIT_BIT);\n\n /* Wait until can try to take the lock */\n while (l > RW_WAIT) cpu_relax();\n }\n }\n\n void wrunlock() const {\n atomic_add(&l, -RW_WRITE);\n }\n\n void readlock() const {\n while (1) {\n /* A writer exists? */\n while (l & (RW_WAIT | RW_WRITE)) cpu_relax();\n\n /* Try to get read lock */\n if (!(atomic_xadd(&l, RW_READ) & (RW_WAIT | RW_WRITE))) return;\n\n /* Undo */\n atomic_add(&l, -RW_READ);\n }\n }\n\n void rdunlock() const {\n atomic_add(&l, -RW_READ);\n }\n };\n\n#undef atomic_xadd\n#undef cmpxchg\n#undef atomic_inc\n#undef atomic_set_bit\n#undef atomic_add\n#undef RW_WAIT_BIT \n#undef RW_WRITE_BIT \n#undef RW_READ_BIT\n#undef RW_WAIT\n#undef RW_WRITE \n#undef RW_READ\n\n\n /**\n * \\class rwlock\n * Wrapper around pthread's rwlock\n *\n * Before you use, see \\ref parallel_object_intricacies.\n */\n class rwlock {\n private:\n mutable pthread_rwlock_t m_rwlock;\n public:\n rwlock() {\n int error = pthread_rwlock_init(&m_rwlock, NULL);\n ASSERT_TRUE(!error);\n }\n ~rwlock() {\n int error = pthread_rwlock_destroy(&m_rwlock);\n ASSERT_TRUE(!error);\n }\n \n // not copyable\n void operator=(const rwlock& m) { }\n \n /** \n * \\todo: Remove! \n *\n * Copy constructor which does not copy. Do not use! Required for\n * compatibility with some STL implementations (LLVM). which use\n * the copy constructor for vector resize, rather than the\n * standard constructor. */\n rwlock(const rwlock &) {\n int error = pthread_rwlock_init(&m_rwlock, NULL);\n ASSERT_TRUE(!error);\n }\n\n inline void readlock() const {\n pthread_rwlock_rdlock(&m_rwlock);\n //ASSERT_TRUE(!error);\n }\n inline void writelock() const {\n pthread_rwlock_wrlock(&m_rwlock);\n //ASSERT_TRUE(!error);\n }\n inline bool try_readlock() const {\n return pthread_rwlock_tryrdlock(&m_rwlock) == 0;\n }\n inline bool try_writelock() const {\n return pthread_rwlock_trywrlock(&m_rwlock) == 0;\n }\n inline void unlock() const {\n pthread_rwlock_unlock(&m_rwlock);\n //ASSERT_TRUE(!error);\n }\n inline void rdunlock() const {\n unlock();\n }\n inline void wrunlock() const {\n unlock();\n }\n }; // End rwlock\n\n\n\n\n\n /**\n * \\ingroup util\n * This is a simple sense-reversing barrier implementation.\n * In addition to standard barrier functionality, this also\n * provides a \"cancel\" function which can be used to destroy\n * the barrier, releasing all threads stuck in the barrier.\n *\n * Before you use, see \\ref parallel_object_intricacies.\n */\n class cancellable_barrier {\n private:\n graphlab::mutex mutex;\n graphlab::conditional conditional;\n mutable int needed;\n mutable int called; \n \n mutable bool barrier_sense;\n mutable bool barrier_release;\n bool alive;\n\n // not copyconstructible\n cancellable_barrier(const cancellable_barrier&) { }\n\n\n public:\n /// Construct a barrier which will only fall when numthreads enter\n cancellable_barrier(size_t numthreads) {\n needed = numthreads;\n called = 0;\n barrier_sense = false;\n barrier_release = true;\n alive = true;\n }\n\n // not copyable\n void operator=(const cancellable_barrier& m) { }\n\n void resize_unsafe(size_t numthreads) {\n needed = numthreads;\n }\n \n /**\n * \\warning: This barrier is safely NOT reusable with this cancel\n * definition\n */\n inline void cancel() {\n alive = false;\n conditional.broadcast();\n }\n /// Wait on the barrier until numthreads has called wait\n inline void wait() const {\n if (!alive) return;\n mutex.lock();\n // set waiting;\n called++;\n bool listening_on = barrier_sense;\n if (called == needed) {\n // if I have reached the required limit, wait up. Set waiting\n // to 0 to make sure everyone wakes up\n called = 0;\n barrier_release = barrier_sense;\n barrier_sense = !barrier_sense;\n // clear all waiting\n conditional.broadcast();\n } else {\n // while no one has broadcasted, sleep\n while(barrier_release != listening_on && alive) conditional.wait(mutex);\n }\n mutex.unlock();\n }\n }; // end of conditional\n \n\n\n /**\n * \\class barrier\n * Wrapper around pthread's barrier\n *\n * Before you use, see \\ref parallel_object_intricacies.\n */\n#ifdef __linux__\n /**\n * \\ingroup util\n * Wrapper around pthread's barrier\n */\n class barrier {\n private:\n mutable pthread_barrier_t m_barrier;\n // not copyconstructable\n barrier(const barrier&) { }\n public:\n /// Construct a barrier which will only fall when numthreads enter\n barrier(size_t numthreads) { \n pthread_barrier_init(&m_barrier, NULL, (unsigned)numthreads); } \n // not copyable\n void operator=(const barrier& m) { }\n void resize_unsafe(size_t numthreads) {\n pthread_barrier_destroy(&m_barrier);\n pthread_barrier_init(&m_barrier, NULL, (unsigned)numthreads);\n }\n ~barrier() { pthread_barrier_destroy(&m_barrier); }\n /// Wait on the barrier until numthreads has called wait\n inline void wait() const { pthread_barrier_wait(&m_barrier); }\n };\n\n#else \n /* In some systems, pthread_barrier is not available.\n */\n typedef cancellable_barrier barrier;\n#endif\n\n\n\n inline void prefetch_range(void *addr, size_t len) {\n char *cp;\n char *end = (char*)(addr) + len;\n\n for (cp = (char*)(addr); cp < end; cp += 64) __builtin_prefetch(cp, 0); \n }\n inline void prefetch_range_write(void *addr, size_t len) {\n char *cp;\n char *end = (char*)(addr) + len;\n\n for (cp = (char*)(addr); cp < end; cp += 64) __builtin_prefetch(cp, 1);\n }\n\n\n\n\n\n\n\n\n\n /**\n * \\ingroup util\n * A collection of routines for creating and managing threads.\n *\n * The thread object performs limited exception forwarding.\n * exception throws within a thread of type const char* will be caught\n * and forwarded to the join() function.\n * If the call to join() is wrapped by a try-catch block, the exception\n * will be caught safely and thread cleanup will be completed properly.\n */\n class thread {\n public:\n\n /**\n * This class contains the data unique to each thread. All threads\n * are gauranteed to have an associated graphlab thread_specific\n * data. The thread object is copyable. \n */ \n class tls_data {\n public:\n inline tls_data(size_t thread_id) : thread_id_(thread_id) { }\n inline size_t thread_id() { return thread_id_; }\n inline void set_thread_id(size_t t) { thread_id_ = t; }\n any& operator[](const size_t& id) { return local_data[id]; }\n bool contains(const size_t& id) const {\n return local_data.find(id) != local_data.end();\n }\n size_t erase(const size_t& id) {\n return local_data.erase(id);\n }\n private:\n size_t thread_id_;\n boost::unordered_map<size_t, any> local_data;\n }; // end of thread specific data\n\n\n\n /// Static helper routines\n // ===============================================================\n\n /**\n * Get the thread specific data associated with this thread\n */\n static tls_data& get_tls_data();\n \n /** Get the id of the calling thread. This will typically be the\n index in the thread group. Between 0 to ncpus. */\n static inline size_t thread_id() { return get_tls_data().thread_id(); }\n \n /** Set the id of the calling thread. This will typically be the\n index in the thread group. Between 0 to ncpus. */\n static inline void set_thread_id(size_t t) { get_tls_data().set_thread_id(t); }\n\n /**\n * Get a reference to an any object\n */\n static inline any& get_local(const size_t& id) {\n return get_tls_data()[id];\n }\n\n /**\n * Check to see if there is an entry in the local map\n */\n static inline bool contains(const size_t& id) {\n return get_tls_data().contains(id);\n }\n \n /**\n * Removes the entry from the local map.\n * @return number of elements erased.\n */\n static inline size_t erase(const size_t& id){\n return get_tls_data().erase(id);\n }\n \n /**\n * This static method joins the invoking thread with the other\n * thread object. This thread will not return from the join\n * routine until the other thread complets it run.\n */\n static void join(thread& other);\n \n // Called just before thread exits. Can be used\n // to do special cleanup... (need for Java JNI)\n static void thread_destroy_callback();\n static void set_thread_destroy_callback(void (*callback)());\n\n \n /**\n * Return the number processing units (individual cores) on this\n * system\n */\n static size_t cpu_count();\n\n\n private:\n \n struct invoke_args{\n size_t m_thread_id;\n boost::function<void(void)> spawn_routine; \n invoke_args(size_t m_thread_id, const boost::function<void(void)> &spawn_routine)\n : m_thread_id(m_thread_id), spawn_routine(spawn_routine) { };\n };\n \n //! Little helper function used to launch threads\n static void* invoke(void *_args); \n \n public:\n \n /**\n * Creates a thread with a user-defined associated thread ID\n */\n inline thread(size_t thread_id = 0) : \n m_stack_size(0), \n m_p_thread(0),\n m_thread_id(thread_id),\n thread_started(false){\n // Calculate the stack size in in bytes;\n const int BYTES_PER_MB = 1048576; \n const int DEFAULT_SIZE_IN_MB = 8;\n m_stack_size = DEFAULT_SIZE_IN_MB * BYTES_PER_MB;\n }\n\n /**\n * execute this function to spawn a new thread running spawn_function\n * routine \n */\n void launch(const boost::function<void (void)> &spawn_routine);\n\n /**\n * Same as launch() except that you can specify a CPU on which to\n * run the thread. This only currently supported in Linux and if\n * invoked on a non Linux based system this will be equivalent to\n * start().\n */\n void launch(const boost::function<void (void)> &spawn_routine, size_t cpu_id);\n\n\n /**\n * Join the calling thread with this thread.\n * const char* exceptions\n * thrown by the thread is forwarded to the join() function.\n */\n inline void join() {\n if(this == NULL) {\n std::cout << \"Failure on join()\" << std::endl;\n exit(EXIT_FAILURE);\n }\n join(*this);\n }\n\n /// Returns true if the thread is still running\n inline bool active() const {\n return thread_started;\n }\n \n inline ~thread() { }\n\n /// Returns the pthread thread id\n inline pthread_t pthreadid() {\n return m_p_thread;\n }\n private:\n \n \n //! The size of the internal stack for this thread\n size_t m_stack_size;\n \n //! The internal pthread object\n pthread_t m_p_thread;\n \n //! the threads id\n size_t m_thread_id;\n \n bool thread_started;\n }; // End of class thread\n\n \n\n\n\n /**\n * \\ingroup util\n * Manages a collection of threads.\n *\n * The thread_group object performs limited exception forwarding.\n * exception throws within a thread of type const char* will be caught\n * and forwarded to the join() function.\n * If the call to join() is wrapped by a try-catch block, the exception\n * will be caught safely and thread cleanup will be completed properly.\n *\n * If multiple threads are running in the thread-group, the master should\n * test if running_threads() is > 0, and retry the join().\n */\n class thread_group {\n private:\n size_t m_thread_counter;\n size_t threads_running;\n mutex mut;\n conditional cond;\n std::queue<std::pair<pthread_t, const char*> > joinqueue;\n // not implemented\n thread_group& operator=(const thread_group &thrgrp);\n thread_group(const thread_group&);\n static void invoke(boost::function<void (void)> spawn_function, thread_group *group);\n public:\n /** \n * Initializes a thread group. \n */\n thread_group() : m_thread_counter(0), threads_running(0) { }\n\n /** \n * Launch a single thread which calls spawn_function No CPU affinity is\n * set so which core it runs on is up to the OS Scheduler\n */\n void launch(const boost::function<void (void)> &spawn_function);\n\n /**\n * Launch a single thread which calls spawn_function Also sets CPU\n * Affinity\n */\n void launch(const boost::function<void (void)> &spawn_function, size_t cpu_id);\n\n /** Waits for all threads to complete execution. const char* exceptions\n thrown by threads are forwarded to the join() function.\n */\n void join();\n \n /// Returns the number of running threads.\n inline size_t running_threads() {\n return threads_running;\n }\n //! Destructor. Waits for all threads to complete execution\n inline ~thread_group(){ join(); }\n\n }; // End of thread group\n\n\n /// Runs f in a new thread. convenience function for creating a new thread quickly.\n inline thread launch_in_new_thread(const boost::function<void (void)> &f, \n size_t cpuid = size_t(-1)) {\n thread thr;\n if (cpuid != size_t(-1)) thr.launch(f, cpuid);\n else thr.launch(f);\n return thr;\n }\n\n /// an integer value padded to 64 bytes\n struct padded_integer {\n size_t val;\n char __pad__[64 - sizeof(size_t)];\n };\n}; // End Namespace\n\n#endif\n\n" }, { "path": "src/graphlab/parallel/queued_rwlock.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef QUEUED_RWLOCK_HPP\n#define QUEUED_RWLOCK_HPP\n\n\nnamespace graphlab {\n \n \n#define QUEUED_RW_LOCK_REQUEST_READ 0\n#define QUEUED_RW_LOCK_REQUEST_WRITE 1\n#define QUEUED_RW_LOCK_REQUEST_NONE 2\n\n/**\n * Fair rw-lock with local-only spinning implemented and\n * modified from \n * Scalable Reader-Writer Synchronization for Shared-Memory Multiprocessors.\n * John M. Mellor-Crummey and Michael L. Scott\n */\nclass queued_rw_lock{\n public:\n \n union state_union {\n volatile uint32_t stateu;\n struct {\n volatile uint16_t successor_class;\n volatile bool blocked;\n } state;\n };\n \n struct request{\n void* id; \n volatile request* volatile next;\n volatile state_union s;\n volatile char lockclass;\n };\n private:\n request* volatile tail;\n atomic<size_t> reader_count;\n request* volatile next_writer;\n public:\n queued_rw_lock(): tail(NULL), reader_count(0), next_writer(NULL) { }\n \n inline void writelock(request *I) {\n I->lockclass = QUEUED_RW_LOCK_REQUEST_WRITE;\n I->next = NULL;\n I->s.stateu = 0;\n I->s.state.blocked = true;\n I->s.state.successor_class = QUEUED_RW_LOCK_REQUEST_NONE;\n __sync_synchronize();\n request* predecessor = __sync_lock_test_and_set(&tail, I);\n\n if (predecessor == NULL) {\n next_writer = I;\n __sync_synchronize();\n if (reader_count.value == 0) {\n if (__sync_lock_test_and_set(&next_writer, (request*)NULL) == I) {\n I->s.state.blocked = false;\n }\n }\n }\n else {\n predecessor->s.state.successor_class = QUEUED_RW_LOCK_REQUEST_WRITE;\n __sync_synchronize(); \n predecessor->next = I;\n }\n // while I->blocked. continue\n volatile state_union& is = I->s;\n while (is.state.blocked) sched_yield();\n assert(reader_count.value == 0);\n }\n\n inline void wrunlock(request *I) {\n __sync_synchronize(); \n if (I->next != NULL || !__sync_bool_compare_and_swap(&tail, I, (request*)NULL)) {\n // wait\n while(I->next == NULL) sched_yield();\n __sync_synchronize(); \n \n if (I->next->lockclass == QUEUED_RW_LOCK_REQUEST_READ) {\n reader_count.inc();\n }\n I->next->s.state.blocked = false;\n }\n }\n\n inline void readlock(request *I) {\n I->lockclass =QUEUED_RW_LOCK_REQUEST_READ;\n I->next = NULL;\n I->s.stateu = 0;\n I->s.state.successor_class = QUEUED_RW_LOCK_REQUEST_NONE;\n I->s.state.blocked = true;\n __sync_synchronize(); \n request* predecessor = __sync_lock_test_and_set(&tail, I);\n if (predecessor == NULL) {\n reader_count.inc();\n I->s.state.blocked = false;\n }\n else {\n \n state_union tempold, tempnew;\n tempold.state.blocked = true;\n tempold.state.successor_class = QUEUED_RW_LOCK_REQUEST_NONE;\n tempnew.state.blocked = true;\n tempnew.state.successor_class = QUEUED_RW_LOCK_REQUEST_READ;\n __sync_synchronize();\n if (predecessor->lockclass == QUEUED_RW_LOCK_REQUEST_WRITE ||\n atomic_compare_and_swap(predecessor->s.stateu,\n tempold.stateu,\n tempnew.stateu)) {\n \n predecessor->next = I;\n // wait\n __sync_synchronize(); \n volatile state_union& is = I->s;\n while(is.state.blocked) sched_yield();\n }\n else {\n reader_count.inc();\n predecessor->next = I;\n __sync_synchronize();\n I->s.state.blocked = false;\n }\n }\n __sync_synchronize();\n if (I->s.state.successor_class == QUEUED_RW_LOCK_REQUEST_READ) {\n \n // wait\n while(I->next == NULL) sched_yield();\n reader_count.inc();\n I->next->s.state.blocked = false;\n }\n }\n\n inline void rdunlock(request *I) {\n __sync_synchronize();\n if (I->next != NULL || !__sync_bool_compare_and_swap(&tail, I, (request*)NULL)) {\n while(I->next == NULL) sched_yield();\n if (I->s.state.successor_class == QUEUED_RW_LOCK_REQUEST_WRITE) {\n next_writer = (request*)(I->next);\n __sync_synchronize();\n }\n }\n if (reader_count.dec() == 0) {\n __sync_synchronize(); \n request * w = __sync_lock_test_and_set(&next_writer, (request*)NULL);\n if (w != NULL) {\n w->s.state.blocked = false;\n __sync_synchronize(); \n }\n }\n }\n};\n\n}\n#endif\n\n" }, { "path": "src/graphlab/parallel/thread_pool.cpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n#include <graphlab/parallel/thread_pool.hpp>\n#include <graphlab/logger/assertions.hpp>\n\nnamespace graphlab {\n\n\n thread_pool::thread_pool(size_t nthreads, bool affinity) {\n waiting_on_join = false;\n tasks_inserted = 0;\n tasks_completed = 0;\n cpu_affinity = affinity;\n pool_size = nthreads;\n spawn_thread_group();\n } // end of thread_pool\n\n\n void thread_pool::resize(size_t nthreads) {\n // if the current pool size does not equal the requested number of\n // threads shut the pool down and startup with correct number of\n // threads. \\todo: If the pool size is too small just add\n // additional threads rather than destroying the pool\n if(nthreads != pool_size) {\n pool_size = nthreads;\n\n // stop the queue from blocking\n spawn_queue.stop_blocking();\n \n // join the threads in the thread group\n while(true) {\n try {\n threads.join(); break;\n } catch (const char* error_str) {\n // this should not be possible!\n logstream(LOG_FATAL) \n << \"Unexpected exception caught in thread pool destructor: \" \n << error_str << std::endl;\n }\n }\n spawn_queue.start_blocking();\n spawn_thread_group();\n }\n } // end of set_nthreads\n\n\n size_t thread_pool::size() const { return pool_size; }\n\n\n /**\n Creates the thread group\n */\n void thread_pool::spawn_thread_group() {\n size_t ncpus = thread::cpu_count();\n // start all the threads if CPU affinity is set\n for (size_t i = 0;i < pool_size; ++i) {\n if (cpu_affinity) {\n threads.launch(boost::bind(&thread_pool::wait_for_task, this), i % ncpus);\n }\n else {\n threads.launch(boost::bind(&thread_pool::wait_for_task, this));\n }\n }\n } // end of spawn_thread_group\n\n\n void thread_pool::destroy_all_threads() {\n // wait for all execution to complete\n spawn_queue.wait_until_empty();\n // kill the queue\n spawn_queue.stop_blocking();\n \n // join the threads in the thread group\n while(1) {\n try {\n threads.join();\n break;\n }\n catch (const char* c) {\n // this should not be possible!\n logstream(LOG_FATAL) \n << \"Unexpected exception caught in thread pool destructor: \" \n << c << std::endl;\n ASSERT_TRUE(false);\n }\n }\n } // end of destroy_all_threads\n\n void thread_pool::set_cpu_affinity(bool affinity) {\n if (affinity != cpu_affinity) {\n cpu_affinity = affinity;\n // stop the queue from blocking\n spawn_queue.stop_blocking();\n \n // join the threads in the thread group\n while(1) {\n try {\n threads.join(); break;\n } catch (const char* c) {\n // this should not be possible!\n logstream(LOG_FATAL) \n << \"Unexpected exception caught in thread pool destructor: \" \n << c << std::endl;\n // ASSERT_TRUE(false); // unnecessary\n }\n }\n spawn_queue.start_blocking();\n spawn_thread_group();\n }\n } // end of set_cpu_affinity\n\n \n void thread_pool::wait_for_task() {\n while(1) {\n std::pair<std::pair<boost::function<void (void)>, int>, bool> queue_entry;\n // pop from the queue\n queue_entry = spawn_queue.dequeue();\n if (queue_entry.second) {\n // try to run the function. remember to put it in a try catch\n try {\n int virtual_thread_id = queue_entry.first.second;\n size_t cur_thread_id = thread::thread_id();\n if (virtual_thread_id != -1) {\n thread::set_thread_id(virtual_thread_id);\n }\n queue_entry.first.first();\n thread::set_thread_id(cur_thread_id);\n } catch(const char* ex) {\n // if an exception was raised, put it in the exception queue\n mut.lock();\n exception_queue.push(ex);\n event_condition.signal();\n mut.unlock();\n }\n \n mut.lock();\n tasks_completed++;\n // the waiting on join flag just prevents me from \n // signaling every time completed == inserted. Which could be very\n // very often\n if (waiting_on_join && \n tasks_completed == tasks_inserted) event_condition.signal();\n mut.unlock();\n }\n else {\n // quit if the queue is dead\n break;\n }\n }\n } // end of wait_for_task\n\n void thread_pool::launch(const boost::function<void (void)> &spawn_function, \n int thread_id) {\n mut.lock();\n tasks_inserted++;\n spawn_queue.enqueue(std::make_pair(spawn_function, thread_id));\n mut.unlock();\n }\n\n void thread_pool::join() {\n std::pair<bool, bool> eventret;\n // first we wait for the queue to empty\n spawn_queue.wait_until_empty();\n \n mut.lock();\n waiting_on_join = true;\n while(1) {\n // check the exception queue. \n if (!exception_queue.empty()) {\n // pop an exception\n const char* ex = exception_queue.front();\n exception_queue.pop();\n // unlock and throw the event\n waiting_on_join = false;\n mut.unlock();\n throw(ex);\n }\n // nothing to throw, check if all tasks were completed\n if (tasks_completed == tasks_inserted) {\n // yup\n break;\n }\n event_condition.wait(mut);\n }\n waiting_on_join = false;\n mut.unlock();\n }\n\n\n thread_pool::~thread_pool() {\n destroy_all_threads();\n }\n\n}\n" }, { "path": "src/graphlab/parallel/thread_pool.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n\n\n#ifndef GRAPHLAB_THREAD_POOL_HPP\n#define GRAPHLAB_THREAD_POOL_HPP\n\n#include <boost/bind.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/util/blocking_queue.hpp>\n\nnamespace graphlab {\n\n /**\n * \\ingroup util\n * Manages a pool of threads.\n * \n * The interface is nearly identical to the \\ref thread_group. \n * The key difference is internal behavior. The thread pool preallocates a\n * collection of threads which it keeps asleep. When tasks are issued \n * through the \"launch\" function, threads are woken up to perform the\n * tasks. \n *\n * The thread_pool object performs limited exception forwarding.\n * exception throws within a thread of type const char* will be caught\n * and forwarded to the join() function.\n * If the call to join() is wrapped by a try-catch block, the exception\n * will be caught safely and thread cleanup will be completed properly.\n *\n * If multiple threads are running in the thread-group, the master should\n * test if running_threads() is > 0, and retry the join().\n *\n */\n class thread_pool {\n private:\n\n thread_group threads;\n blocking_queue<std::pair<boost::function<void (void)>, int> > spawn_queue;\n size_t pool_size;\n \n // protects the exception queue, and the task counters\n mutex mut;\n conditional event_condition; // to wake up the joining thread\n std::queue<const char*> exception_queue;\n size_t tasks_inserted;\n size_t tasks_completed;\n bool waiting_on_join; // true if a thread is waiting in join\n\n bool cpu_affinity;\n // not implemented\n thread_pool& operator=(const thread_pool &thrgrp);\n thread_pool(const thread_pool&);\n \n /**\n Called by each thread. Loops around a queue of tasks.\n */\n void wait_for_task();\n\n /**\n Creates all the threads in the thread pool.\n Resets the task and exception queue\n */\n void spawn_thread_group();\n \n /**\n Destroys the thread pool.\n Also destroys the task queue\n */\n void destroy_all_threads();\n public:\n \n /* Initializes a thread pool with nthreads. \n * If affinity is set, the nthreads will by default stripe across \n * the available cores on the system. \n */\n thread_pool(size_t nthreads = 2, bool affinity = false);\n \n /**\n * Set the number of threads in the queue\n */\n void resize(size_t nthreads);\n \n /**\n * Get the number of threads\n */\n size_t size() const;\n\n\n /**\n * Changes the CPU affinity. Note that pthread does not provide\n * a way to change CPU affinity on a currently started thread.\n * This function therefore waits for all threads in the pool\n * to finish their current task, and destroy all the threads. Then\n * new threads are created with the new affinity setting.\n */\n void set_cpu_affinity(bool affinity);\n \n /**\n Gets the CPU affinity.\n */\n bool get_cpu_affinity() { return cpu_affinity; };\n \n /** \n * Launch a single thread which calls spawn_function. If affinity\n * is set on construction of the thread_pool, the thread handling the\n * function will be locked on to one particular CPU.\n *\n * If virtual_threadid is set, the target thread will appear to have\n * thread ID equal to the requested thread ID\n */\n void launch(const boost::function<void (void)> &spawn_function, \n int virtual_threadid = -1);\n \n \n /** Waits for all threads to become free. const char* exceptions\n thrown by threads are forwarded to the join() function.\n Once this function returns normally, the queue is empty.\n \n Note that this function may not return if producers continually insert\n tasks through launch. \n */\n void join();\n \n //! Destructor. Cleans up all threads\n ~thread_pool();\n };\n \n}\n#endif\n" }, { "path": "src/graphlab/rpc/CMakeLists.txt", "content": "project(GraphLab)\n\n\n" }, { "path": "src/graphlab/rpc/async_consensus.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <graphlab/rpc/async_consensus.hpp>\n\nnamespace graphlab {\n async_consensus::async_consensus(distributed_control &dc,\n size_t required_threads_in_done,\n const dc_impl::dc_dist_object_base *attach)\n :rmi(dc, this), attachedobj(attach),\n last_calls_sent(0), last_calls_received(0),\n numactive(required_threads_in_done),\n ncpus(required_threads_in_done),\n done(false),\n trying_to_sleep(0),\n critical(ncpus, 0),\n sleeping(ncpus, 0),\n hastoken(dc.procid() == 0),\n cond(ncpus){\n\n cur_token.total_calls_sent = 0;\n cur_token.total_calls_received = 0;\n cur_token.last_change = (procid_t)(rmi.numprocs() - 1);\n }\n\n void async_consensus::reset() {\n last_calls_sent = 0;\n last_calls_received = 0;\n numactive = ncpus;\n done = false;\n trying_to_sleep = false;\n critical = std::vector<char>(ncpus, 0);\n sleeping = std::vector<char>(ncpus, 0);\n hastoken = (rmi.procid() == 0);\n cur_token.total_calls_sent = 0;\n cur_token.total_calls_received = 0;\n cur_token.last_change = (procid_t)(rmi.numprocs() - 1);\n }\n\n void async_consensus::force_done() {\n m.lock();\n done = true;\n m.unlock();\n cancel();\n }\n\n void async_consensus::begin_done_critical_section(size_t cpuid) {\n trying_to_sleep.inc();\n critical[cpuid] = true;\n m.lock();\n }\n\n\n void async_consensus::cancel_critical_section(size_t cpuid) {\n m.unlock();\n critical[cpuid] = false;\n trying_to_sleep.dec();\n }\n \n bool async_consensus::end_done_critical_section(size_t cpuid) {\n // if done flag is set, quit immediately\n if (done) {\n m.unlock();\n critical[cpuid] = false;\n trying_to_sleep.dec();\n return true;\n }\n /*\n Assertion: Since numactive is decremented only within \n a critical section, and is incremented only within the same critical \n section. Therefore numactive is a valid counter of the number of threads \n outside of this critical section. \n */\n --numactive;\n \n /*\n Assertion: If numactive is ever 0 at this point, the algorithm is done.\n WLOG, let the current thread which just decremented numactive be thread 0\n \n Since there is only 1 active thread (0), there must be no threads \n performing insertions, and are no othe threads which are waking up.\n All threads must therefore be sleeping in cond.wait().\n */\n if (numactive == 0) {\n logstream(LOG_INFO) << rmi.procid() << \": Termination Possible\" << std::endl;\n if (hastoken) pass_the_token();\n }\n sleeping[cpuid] = true;\n while(1) {\n // here we are protected by the mutex again.\n \n // woken up by someone else. leave the \n // terminator\n if (sleeping[cpuid] == false || done) {\n break;\n }\n cond[cpuid].wait(m);\n }\n m.unlock();\n critical[cpuid] = false;\n trying_to_sleep.dec();\n return done;\n }\n \n void async_consensus::cancel() {\n /*\n Assertion: numactive > 0 if there is work to do.\n If there are threads trying to sleep, lets wake them up\n */\n if (trying_to_sleep > 0 || numactive < ncpus) {\n m.lock();\n size_t oldnumactive = numactive;\n // once I acquire this lock, all threads must be\n // in the following states\n // 1: still running and has not reached begin_critical_section()\n // 2: is sleeping in cond.wait()\n // 3: has called begin_critical_section() but has not acquired\n // the mutex\n // In the case of 1,3: These threads will perform one more sweep\n // of their task queues. Therefore they will see any new job if available\n // in the case of 2: numactive must be < ncpus since numactive\n // is mutex protected. Then I can wake them up by\n // clearing their sleeping flags and broadcasting.\n if (numactive < ncpus) {\n // this is safe. Note that it is done from within \n // the critical section.\n for (size_t i = 0;i < ncpus; ++i) {\n numactive += sleeping[i];\n if (sleeping[i]) {\n sleeping[i] = 0;\n cond[i].signal();\n }\n }\n if (oldnumactive == 0 && !done) {\n logstream(LOG_INFO) << rmi.procid() << \": Waking\" << std::endl;\n }\n\n }\n m.unlock();\n }\n }\n\n void async_consensus::cancel_one(size_t cpuhint) {\n if (critical[cpuhint]) {\n m.lock();\n size_t oldnumactive = numactive;\n // see new_job() for detailed comments\n if (sleeping[cpuhint]) {\n numactive += sleeping[cpuhint];\n sleeping[cpuhint] = 0;\n if (oldnumactive == 0 && !done) {\n logstream(LOG_INFO) << rmi.procid() << \": Waking\" << std::endl;\n }\n cond[cpuhint].signal();\n }\n m.unlock();\n }\n }\n\n void async_consensus::receive_the_token(token &tok) {\n m.lock();\n // save the token\n hastoken = true;\n cur_token = tok;\n // if I am waiting on done, pass the token.\n logstream(LOG_INFO) << rmi.procid() << \": Token Received\" << std::endl;\n if (numactive == 0) {\n pass_the_token();\n }\n m.unlock();\n }\n\n void async_consensus::pass_the_token() {\n // note that this function does not acquire the token lock\n // the caller must acquire it \n assert(hastoken);\n // first check if we are done\n if (cur_token.last_change == rmi.procid() && \n cur_token.total_calls_received == cur_token.total_calls_sent) {\n logstream(LOG_INFO) << \"Completed Token: \" \n << cur_token.total_calls_received << \" \" \n << cur_token.total_calls_sent << std::endl;\n // we have completed a loop around!\n // broadcast a completion\n for (procid_t i = 0;i < rmi.numprocs(); ++i) {\n if (i != rmi.procid()) {\n rmi.control_call(i,\n &async_consensus::force_done);\n }\n }\n // set the complete flag\n // we can't call consensus() since it will deadlock\n done = true;\n // this is the same code as cancel(), but we can't call cancel \n // since we are holding on to a lock\n if (numactive < ncpus) {\n // this is safe. Note that it is done from within \n // the critical section.\n for (size_t i = 0;i < ncpus; ++i) {\n numactive += sleeping[i];\n if (sleeping[i]) {\n sleeping[i] = 0;\n cond[i].signal();\n }\n }\n }\n\n }\n else {\n // update the token\n size_t callsrecv;\n size_t callssent;\n \n if (attachedobj) {\n callsrecv = attachedobj->calls_received();\n callssent = attachedobj->calls_sent();\n }\n else {\n callsrecv = rmi.dc().calls_received();\n callssent = rmi.dc().calls_sent();\n }\n\n if (callssent != last_calls_sent ||\n callsrecv != last_calls_received) {\n cur_token.total_calls_sent += callssent - last_calls_sent;\n cur_token.total_calls_received += callsrecv - last_calls_received;\n cur_token.last_change = rmi.procid();\n }\n //std::cout << \"Sending token: (\" << cur_token.total_calls_sent\n //<< \", \" << cur_token.total_calls_received << \")\" << std::endl;\n\n last_calls_sent = callssent;\n last_calls_received = callsrecv;\n // send it along.\n hastoken = false;\n logstream(LOG_INFO) << \"Passing Token \" << rmi.procid() << \"-->\" \n << (rmi.procid() + 1) % rmi.numprocs() << \": \"\n << cur_token.total_calls_received << \" \" \n << cur_token.total_calls_sent << std::endl;\n\n rmi.control_call((procid_t)((rmi.procid() + 1) % rmi.numprocs()),\n &async_consensus::receive_the_token,\n cur_token);\n }\n }\n}\n\n\n" }, { "path": "src/graphlab/rpc/async_consensus.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef ASYNC_TERMINATOR_HPP\n#define ASYNC_TERMINATOR_HPP\n\n#include <graphlab/parallel/pthread_tools.hpp>\n\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_dist_object_base.hpp>\n#include <graphlab/rpc/dc_dist_object.hpp>\n\n\nnamespace graphlab {\n /**\n * \\ingroup rpc\n * \\brief This implements a distributed consensus algorithm which waits\n * for global completion of all computation/RPC events on a given object.\n *\n * The use case is as follows:\n * \n * A collection of threads on a collection of distributed machines, each\n * running the following\n * \n * \\code\n * while (work to be done) {\n * Do_stuff\n * }\n * \\endcode\n * \n * However, <tt>Do_stuff</tt> will include RPC calls which may introduce\n * work to other threads/machines.\n * Figuring out when termination is possible is complex. For instance RPC calls \n * could be in-flight and not yet received. This async_consensus class \n * implements a solution built around the algorithm in\n * <i>Misra, J.: Detecting Termination of Distributed Computations Using Markers, SIGOPS, 1983</i>\n * extended to handle the mixed parallelism (distributed with threading) case.\n * \n * The main loop of the user has to be modified to:\n * \n * \\code\n * done = false;\n * while (!done) {\n * Do_stuff\n * // if locally, I see that there is no work to be done\n * // we begin the consensus checking\n * if (no work to be done) {\n * // begin the critical section and try again\n * consensus.begin_done_critical_section();\n * // if still no work to be done\n * if (no work to be done) {\n * done = consensus.end_done_critical_section();\n * }\n * else {\n * consensus.cancel_critical_section();\n * }\n * }\n * }\n * \n * \\endcode\n * \n * Additionally, incoming RPC calls which create work must ensure there are\n * active threads which are capable of processing the work. An easy solution \n * will be to simply cancel_one(). Other more optimized solutions\n * include keeping a counter of the number of active threads, and only calling\n * cancel() or cancel_one() if all threads are asleep. (Note that the optimized\n * solution does require some care to ensure dead-lock free execution).\n *\n * The async_consensus works with regular kernel threads. See \n * \\ref graphlab::fiber_async_consensus for a version which works with\n * fibers.\n *\n * \\see graphlab::fiber_async_consensus\n */\n class async_consensus {\n public:\n /** \\brief Constructs an asynchronous consensus object\n *\n * The consensus procedure waits till all threads have no work to do and are \n * waiting in consensus, and there is no communication going on which\n * could wake up a thread. The consensus object is therefore associated\n * with a communication context, either a graphlab::dc_dist_object,\n * or the global context (the root distributed_control).\n * \n * \\param dc The distributed control object to use for communication\n * \\param required_threads_in_done local consensus is achieved if this many\n * threads are waiting for consensus locally.\n * \\param attach The context to associate with. If NULL, we associate with\n * the global context. \n */\n async_consensus(distributed_control &dc, size_t required_threads_in_done = 1,\n const dc_impl::dc_dist_object_base* attach = NULL);\n\n\n /**\n * \\brief A thread enters the critical section by calling\n * this function. \n * \n * After which it should check its termination \n * condition locally. If termination condition\n * is still fulfilled, end_done_critical_section() should be called.\n * Otherwise cancel_critical_section() should be called.\n *\n * \\param cpuid Thread ID of the thread.\n */\n void begin_done_critical_section(size_t cpuid);\n\n /**\n * \\brief Leaves the critical section because termination condition\n * is not fullfilled.\n *\n * See begin_done_critical_section()\n * \\param cpuid Thread ID of the thread.\n */\n void cancel_critical_section(size_t cpuid);\n\n /**\n * \\brief Thread must call this function if termination condition\n * is fullfilled while in the critical section. \n *\n * See begin_done_critical_section()\n * \n * \\param cpuid Thread ID of the thread.\n * \\returns True if global consensus is achieved. False otherwise. \n */\n bool end_done_critical_section(size_t cpuid);\n\n /**\n * \\brief Forces the consensus to be set\n */\n void force_done();\n \n \n /// \\brief Wakes up all local threads waiting in done()\n void cancel();\n\n /// \\brief Wakes up a specific thread waiting in done()\n void cancel_one(size_t cpuid);\n\n /// \\brief Returns true if consensus is achieved.\n bool is_done() const {\n return done;\n }\n /** \\brief Resets the consensus object. Must be called simultaneously by\n * exactly one thread on each machine.\n * This function is not safe to call while consensus is being achieved.\n */\n void reset();\n \n private:\n\n /**\n * The token object that is passed around the machines.\n * It counts the total number of RPC calls that has been sent\n * or received, as well as the machine which last changed the value.\n * When the token goes one full round with no change, consensus is\n * achieved.\n */\n struct token {\n size_t total_calls_sent;\n size_t total_calls_received;\n procid_t last_change;\n void save(oarchive &oarc) const {\n oarc << total_calls_sent << total_calls_received << last_change;\n }\n\n void load(iarchive &iarc) {\n iarc >> total_calls_sent >> total_calls_received >> last_change;\n }\n };\n\n \n dc_dist_object<async_consensus> rmi;\n const dc_impl::dc_dist_object_base* attachedobj;\n \n size_t last_calls_sent;\n size_t last_calls_received;\n\n \n \n /// counts the number of threads which are not sleeping\n /// protected by the mutex\n size_t numactive; \n \n /// Total number of CPUs\n size_t ncpus;\n \n /// once flag is set, the terminator is invalid, and all threads\n /// should leave\n bool done;\n \n /// set if abort() is called\n // bool forced_abort; \n \n /// Number of threads which have called\n /// begin_critical_section(), and have not left end_critical_section()\n /// This is an atomic counter and is not protected.\n atomic<size_t> trying_to_sleep;\n \n /// critical[i] is set if thread i has called \n /// begin_critical_section(), but has not left end_critical_section()\n /// sum of critical should be the same as trying_to_sleep\n std::vector<char> critical;\n \n /// sleeping[i] is set if threads[i] is in cond.wait()\n std::vector<char> sleeping;\n \n \n bool hastoken;\n /// If I have the token, the value of the token\n token cur_token;\n\n mutex m;\n std::vector<conditional> cond;\n \n\n void receive_the_token(token &tok);\n void pass_the_token();\n };\n\n}\n#endif\n\n\n\n" }, { "path": "src/graphlab/rpc/buffered_exchange.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_BUFFERED_EXCHANGE_HPP\n#define GRAPHLAB_BUFFERED_EXCHANGE_HPP\n\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/parallel/fiber_control.hpp>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_dist_object.hpp>\n#include <graphlab/util/mpi_tools.hpp>\n\n\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n\n /**\n * \\ingroup rpc\n *\n * The buffered exchange provides high performance exchange of bulk data \n * between machines. The basic usage is simple:\n *\n * For instance, if we are doing bulk exchanges of integers:\n * \\code\n * buffered_exchange<int> exchange(dc, numthreads);\n * .. In parallel .. {\n * exchange.send([target machine], [value to send to target], [thread id])\n * exchange.partial_flush([thread id]);\n * }\n *\n * .. now in 1 thread ..\n * exchange.flush()\n *\n * .. In parallel .. {\n * procid_t proc;\n * buffered_exchange<int>::buffer_type buffer; // (an array of integers)\n * while(exchange.recv(proc, buffer)) {\n * process array of integers (buffeer) which were sent by proc\n * }\n * }\n * \\endcode\n *\n * \\note The buffered exchange sends data in the background, so recv can be\n * called even before the flush calls.\n *\n * \\see graphlab::fiber_buffered_exchange\n */\n template<typename T>\n class buffered_exchange {\n public:\n typedef std::vector<T> buffer_type;\n\n private:\n struct buffer_record {\n procid_t proc;\n buffer_type buffer;\n buffer_record() : proc(-1) { }\n }; // end of buffer record\n\n\n\n /** The rpc interface for this class */\n mutable dc_dist_object<buffered_exchange> rpc;\n\n std::deque< buffer_record > recv_buffers;\n mutex recv_lock;\n\n\n struct send_record {\n oarchive* oarc;\n size_t numinserts;\n };\n\n std::vector<send_record> send_buffers;\n std::vector< mutex > send_locks;\n const size_t num_threads;\n const size_t max_buffer_size;\n\n\n // typedef boost::function<void (const T& tref)> handler_type;\n // handler_type recv_handler;\n\n public:\n /**\n * Constructs a buffered exchange object.\n *\n * \\ref dc The master distributed_control object\n * \\ref num_threads The number of threads to support. This is essentially\n * the number of fine-grained locks to use. This does not\n * need to match the total number of threads used during \n * the exchange process, but there are performance / contention\n * advantages if this matches.\n * \\ref max_buffer_size The size of the per thread and per target send buffer.\n */\n buffered_exchange(distributed_control& dc,\n const size_t num_threads = 1,\n const size_t max_buffer_size = DEFAULT_BUFFERED_EXCHANGE_SIZE) :\n rpc(dc, this),\n send_buffers(num_threads * dc.numprocs()),\n send_locks(num_threads * dc.numprocs()),\n num_threads(num_threads),\n max_buffer_size(max_buffer_size) {\n //\n for (size_t i = 0;i < send_buffers.size(); ++i) {\n // initialize the split call\n send_buffers[i].oarc = rpc.split_call_begin(&buffered_exchange::rpc_recv);\n send_buffers[i].numinserts = 0;\n // begin by writing the src proc.\n (*(send_buffers[i].oarc)) << rpc.procid();\n }\n rpc.barrier();\n }\n\n\n ~buffered_exchange() {\n // clear the send buffers\n for (size_t i = 0;i < send_buffers.size(); ++i) {\n rpc.split_call_cancel(send_buffers[i].oarc);\n }\n }\n // buffered_exchange(distributed_control& dc, handler_type recv_handler,\n // size_t buffer_size = 1000) :\n // rpc(dc, this), send_buffers(dc.numprocs()), send_locks(dc.numprocs()),\n // max_buffer_size(buffer_size), recv_handler(recv_handler) { rpc.barrier(); }\n\n\n /**\n * Sends a value to a target machine.\n * Use the send buffer owned by thread_id.\n */\n void send(const procid_t proc, const T& value, const size_t thread_id = 0) {\n ASSERT_LT(proc, rpc.numprocs());\n ASSERT_LT(thread_id, num_threads);\n const size_t index = thread_id * rpc.numprocs() + proc;\n ASSERT_LT(index, send_locks.size());\n send_locks[index].lock();\n\n (*(send_buffers[index].oarc)) << value;\n ++send_buffers[index].numinserts;\n\n if(send_buffers[index].oarc->off >= max_buffer_size) {\n oarchive* prevarc = swap_buffer(index);\n send_locks[index].unlock();\n // complete the send\n rpc.split_call_end(proc, prevarc);\n } else {\n send_locks[index].unlock();\n }\n } // end of send\n\n /**\n * Flushes the send buffer owned owned by thread_id.\n */\n void partial_flush(size_t thread_id) {\n for(procid_t proc = 0; proc < rpc.numprocs(); ++proc) {\n const size_t index = thread_id * rpc.numprocs() + proc;\n ASSERT_LT(proc, rpc.numprocs());\n if (send_buffers[index].numinserts > 0) {\n send_locks[index].lock();\n oarchive* prevarc = swap_buffer(index);\n send_locks[index].unlock();\n // complete the send\n rpc.split_call_end(proc, prevarc);\n rpc.dc().flush_soon(proc);\n }\n }\n }\n\n /**\n * Flushes all send buffers. Must be called only on one thread.\n * Will not return until all machines call flush.\n */\n void flush() {\n for(size_t i = 0; i < send_buffers.size(); ++i) {\n const procid_t proc = i % rpc.numprocs();\n ASSERT_LT(proc, rpc.numprocs());\n send_locks[i].lock();\n if (send_buffers[i].numinserts > 0) {\n oarchive* prevarc = swap_buffer(i);\n // complete the send\n rpc.split_call_end(proc, prevarc);\n }\n send_locks[i].unlock();\n }\n rpc.dc().flush_soon();\n rpc.full_barrier();\n } // end of flush\n\n\n\n /**\n * Returns a collection of T sent by ret_proc.\n *\n * \\param ret_proc On successful return, will contain a valid procid indicating\n * that the values in the buffer were sent by that process.\n * \\param ret_buffer A sequence of values sent by ret_proc\n * \\param try_lock If true, will not acquire the lock if the lock is \n * contended. Useful for polling the receive buffer\n * while sending is occuring.\n * \\return True on success and there are values in the buffer. \n * False if the receive buffer is empty. Or if try_lock is set,\n * False may also indicate the buffer lock is being contended.\n */\n bool recv(procid_t& ret_proc, buffer_type& ret_buffer,\n const bool try_lock = false) {\n fiber_control::fast_yield();\n dc_impl::blob read_buffer;\n bool has_lock = false;\n if(try_lock) {\n if (recv_buffers.empty()) return false;\n has_lock = recv_lock.try_lock();\n } else {\n recv_lock.lock();\n has_lock = true;\n }\n bool success = false;\n if(has_lock) {\n if(!recv_buffers.empty()) {\n success = true;\n buffer_record& rec = recv_buffers.front();\n // read the record\n ret_proc = rec.proc;\n ret_buffer.swap(rec.buffer);\n ASSERT_LT(ret_proc, rpc.numprocs());\n recv_buffers.pop_front();\n }\n recv_lock.unlock();\n }\n\n return success;\n } // end of recv\n\n\n\n /**\n * Returns the number of elements available for receiving.\n */\n size_t size() const {\n typedef typename std::deque< buffer_record >::const_iterator iterator;\n recv_lock.lock();\n size_t count = 0;\n foreach(const buffer_record& rec, recv_buffers) {\n count += rec.buffer.size();\n }\n recv_lock.unlock();\n return count;\n } // end of size\n\n /**\n * Returns true if there are no elements available for receiving.\n */\n bool empty() const { return recv_buffers.empty(); }\n\n void clear() { }\n\n void barrier() { rpc.barrier(); }\n private:\n void rpc_recv(size_t len, wild_pointer w) {\n buffer_type tmp;\n iarchive iarc(reinterpret_cast<const char*>(w.ptr), len);\n // first desrialize the source process\n procid_t src_proc; iarc >> src_proc;\n ASSERT_LT(src_proc, rpc.numprocs());\n // create an iarchive which just points to the last size_t bytes\n // to get the number of elements\n iarchive numel_iarc(reinterpret_cast<const char*>(w.ptr) + len - sizeof(size_t),\n sizeof(size_t));\n size_t numel = 0; \n numel_iarc.read(reinterpret_cast<char*>(&numel), sizeof(size_t));\n //std::cout << \"Receiving: \" << numel << \"\\n\";\n tmp.resize(numel);\n for (size_t i = 0;i < numel; ++i) {\n iarc >> tmp[i];\n }\n\n recv_lock.lock();\n recv_buffers.push_back(buffer_record());\n buffer_record& rec = recv_buffers.back();\n rec.proc = src_proc;\n rec.buffer.swap(tmp);\n recv_lock.unlock();\n } // end of rpc rcv\n\n\n // create a new buffer for send_buffer[index], returning the old buffer\n oarchive* swap_buffer(size_t index) {\n oarchive* swaparc = rpc.split_call_begin(&buffered_exchange::rpc_recv);\n std::swap(send_buffers[index].oarc, swaparc);\n // write the length at the end of the buffere are returning\n (*swaparc).write(reinterpret_cast<char*>(&send_buffers[index].numinserts), sizeof(size_t));\n\n //std::cout << \"Sending : \" << (send_buffers[index].numinserts)<< \"\\n\";\n // reset the insertion count\n send_buffers[index].numinserts = 0;\n // write the current procid into the new buffer\n (*(send_buffers[index].oarc)) << rpc.procid();\n return swaparc;\n }\n\n\n }; // end of buffered exchange\n\n\n}; // end of graphlab namespace\n#include <graphlab/macros_undef.hpp>\n\n#endif\n\n\n" }, { "path": "src/graphlab/rpc/caching_dht.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n\n/*\n \\author Yucheng Low (ylow), Joseph Gonzalez (jegonzal)\n An implementation of a distributed integer -> integer map with caching\n capabilities. \n\n*/\n\n#ifndef GRAPHLAB_CACHING_DHT_HPP\n#define GRAPHLAB_CACHING_DHT_HPP\n#include <boost/unordered_map.hpp>\n#include <boost/intrusive/list.hpp>\n#include <boost/functional/hash.hpp>\n\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/util/synchronized_unordered_map.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n\nnamespace graphlab {\n\n namespace dc_impl {\n /**\n * \\internal \n * \\ingroup rpc\n A cache entry for the caching_dht. \n Boost intrusive is used to provide the LRU capabilities here\n */\n template<typename KeyType, typename ValueType>\n class lru_list {\n public:\n\n KeyType key; /// the key assiciated with this cache entry\n ValueType value; /// the value assiciated with this cache entry\n typedef boost::intrusive::list_member_hook<\n boost::intrusive::link_mode<boost::intrusive::auto_unlink> >\n lru_member_hook_type;\n\n lru_member_hook_type member_hook_;\n ~lru_list() { }\n explicit lru_list(const KeyType& k = KeyType(), const ValueType &v = ValueType()) : key(k), value(v) {\n }\n };\n\n } // namespace dc_impl\n\n /**\n * \\internal\n * \\ingroup rpc\n This implements a limited distributed key -> value map with caching capabilities\n It is up to the user to determine cache invalidation policies. User explicitly\n calls the invalidate() function to clear local cache entries\n */\n template<typename KeyType, typename ValueType>\n class caching_dht{\n public:\n\n typedef dc_impl::lru_list<KeyType, ValueType> lru_entry_type;\n /// datatype of the data map\n typedef boost::unordered_map<KeyType, ValueType> map_type;\n /// datatype of the local cache map\n typedef boost::unordered_map<KeyType, lru_entry_type* > cache_type;\n\n\n typedef boost::intrusive::member_hook<lru_entry_type,\n typename lru_entry_type::lru_member_hook_type,\n &lru_entry_type::member_hook_> MemberOption;\n /// datatype of the intrusive LRU list embedded in the cache map\n typedef boost::intrusive::list<lru_entry_type, \n MemberOption, \n boost::intrusive::constant_time_size<false> > lru_list_type;\n\n\n private:\n mutable dc_dist_object<caching_dht<KeyType, ValueType> > rpc;\n \n mutex datalock;\n map_type data; /// The actual table data that is distributed\n \n mutex cachelock; /// lock for the cache datastructures\n mutable cache_type cache; /// The cache table\n mutable lru_list_type lruage; /// THe LRU linked list associated with the cache\n\n\n procid_t numprocs; /// NUmber of processors\n size_t maxcache; /// Maximum cache size allowed\n\n mutable size_t reqs;\n mutable size_t misses;\n\n boost::hash<KeyType> hasher;\n\n public:\n\n /// Constructor. Creates the integer map.\n caching_dht(distributed_control &dc, \n size_t max_cache_size = 1024):rpc(dc, this),data(11) {\n cache.rehash(max_cache_size);\n maxcache = max_cache_size;\n logger(LOG_INFO, \"%d Creating distributed_hash_table. Cache Limit = %d\", \n dc.procid(), maxcache);\n reqs = 0;\n misses = 0;\n }\n\n\n ~caching_dht() {\n data.clear();\n typename cache_type::iterator i = cache.begin();\n while (i != cache.end()) {\n delete i->second;\n ++i;\n }\n cache.clear();\n }\n \n \n /// Sets the key to the value\n void set(const KeyType& key, const ValueType &newval) {\n size_t hashvalue = hasher(key);\n size_t owningmachine = hashvalue % rpc.dc().numprocs();\n if (owningmachine == rpc.dc().procid()) {\n datalock.lock();\n data[key] = newval;\n datalock.unlock();\n } else {\n rpc.remote_call(owningmachine, \n &caching_dht<KeyType,ValueType>::set, \n key,\n newval);\n update_cache(key, newval);\n }\n }\n \n\n /** Gets the value associated with the key. returns true on success.. */\n std::pair<bool, ValueType> get(const KeyType &key) const {\n // figure out who owns the key\n size_t hashvalue = hasher(key);\n size_t owningmachine = hashvalue % rpc.dc().numprocs();\n \n std::pair<bool, ValueType> ret;\n // if I own the key, get it from the map table\n if (owningmachine == rpc.dc().procid()) {\n datalock.lock();\n typename map_type::const_iterator iter = data.find(key); \n if (iter == data.end()) {\n ret.first = false;\n } else {\n ret.first = true;\n ret.second = iter->second;\n }\n datalock.unlock();\n } else {\n ret = rpc.remote_request(owningmachine, \n &caching_dht<KeyType,ValueType>::get, \n key);\n if (ret.first) update_cache(key, ret.second);\n else invalidate(key);\n }\n return ret;\n }\n\n\n /** Gets the value associated with the key, reading from cache if available\n Note that the cache may be out of date. */\n std::pair<bool, ValueType> get_cached(const KeyType &key) const {\n // if this is to my current machine, just get it and don't go to cache\n size_t hashvalue = hasher(key);\n size_t owningmachine = hashvalue % rpc.dc().numprocs();\n if (owningmachine == rpc.dc().procid()) return get(key);\n \n \n reqs++;\n cachelock.lock();\n // check if it is in the cache\n typename cache_type::iterator i = cache.find(key);\n if (i == cache.end()) {\n // nope. not in cache. Call the regular get\n cachelock.unlock();\n misses++;\n return get(key);\n }\n else {\n // yup. in cache. return the value\n std::pair<bool, ValueType> ret;\n ret.first = true;\n ret.second = i->second->value;\n // shift the cache entry to the head of the LRU list\n lruage.erase(lru_list_type::s_iterator_to(*(i->second)));\n lruage.push_front(*(i->second));\n cachelock.unlock();\n return ret;\n }\n }\n\n /// Invalidates the cache entry associated with this key\n void invalidate(const KeyType &key) const{\n cachelock.lock();\n // is the key I am invalidating in the cache?\n typename cache_type::iterator i = cache.find(key);\n if (i != cache.end()) {\n // drop it from the lru list\n delete i->second;\n cache.erase(i);\n }\n cachelock.unlock();\n }\n\n\n double cache_miss_rate() {\n return double(misses) / double(reqs);\n }\n\n size_t num_gets() const {\n return reqs;\n }\n size_t num_misses() const {\n return misses;\n }\n\n size_t cache_size() const {\n return cache.size();\n }\n\n private:\n\n \n\n /// Updates the cache with this new value\n void update_cache(const KeyType &key, const ValueType &val) const{\n cachelock.lock();\n typename cache_type::iterator i = cache.find(key);\n // create a new entry\n if (i == cache.end()) {\n cachelock.unlock();\n // if we are out of room, remove the lru entry\n if (cache.size() >= maxcache) remove_lru();\n cachelock.lock();\n // insert the element, remember the iterator so we can push it\n // straight to the LRU list\n std::pair<typename cache_type::iterator, bool> ret = cache.insert(std::make_pair(key, new lru_entry_type(key, val)));\n if (ret.second) lruage.push_front(*(ret.first->second));\n } else {\n // modify entry in place\n i->second->value = val;\n // swap to front of list\n //boost::swap_nodes(lru_list_type::s_iterator_to(i->second), lruage.begin());\n lruage.erase(lru_list_type::s_iterator_to(*(i->second)));\n lruage.push_front(*(i->second));\n }\n cachelock.unlock();\n }\n\n /// Removes the least recently used element from the cache\n void remove_lru() const{\n cachelock.lock();\n KeyType keytoerase = lruage.back().key;\n // is the key I am invalidating in the cache?\n typename cache_type::iterator i = cache.find(keytoerase);\n if (i != cache.end()) {\n // drop it from the lru list\n delete i->second;\n cache.erase(i);\n }\n cachelock.unlock();\n }\n\n };\n\n}\n#endif\n\n" }, { "path": "src/graphlab/rpc/circular_char_buffer.cpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <cstdlib>\n#include <cstring>\n#include <graphlab/logger/assertions.hpp>\n#include <graphlab/rpc/circular_char_buffer.hpp>\n\nnamespace graphlab {\n \n circular_char_buffer::circular_char_buffer(std::streamsize initial) {\n initial = std::max<size_t>((size_t)initial, 4);\n // allocate something to start with\n buffer = (char*)malloc(initial);\n head = 0;\n tail = 0;\n bufsize = initial;\n len = 0;\n }\n\n circular_char_buffer::circular_char_buffer(const circular_char_buffer &src) {\n // allocate minimum of 4 bytes\n bufsize = std::max<size_t>(src.size(), 4);\n buffer = (char*)malloc(bufsize);\n \n // copy the buffer in src\n src.peek(buffer, src.size());\n // set the lengths\n len = src.size();\n tail = src.size();\n head = 0;\n if (tail == bufsize) tail = 0;\n }\n \n circular_char_buffer& \n circular_char_buffer::operator=(const circular_char_buffer& src) {\n // reset head and tail\n clear();\n // make sure we have enough room\n reserve(src.size());\n src.peek(buffer, src.size());\n len = src.size();\n tail = src.size();\n if (tail == bufsize) tail = 0;\n return *this;\n }\n \n void circular_char_buffer::reserve(std::streamsize s) {\n // minimum of 4 bytes. disallow reserve of 0\n if (s <= 4) s = 4;\n // do nothing if s is smaller than the current buffer size\n if (s <= bufsize) return;\n \n // do a reallocation\n buffer = (char*)realloc((void*)buffer, s);\n\n // now, we need to be careful to reposition the head and tail\n // there are 2 cases\n \n // case 1: no loop around,\n // Easiest case. do nothing. just update bufsize\n // case 2: we have a loop around,\n // copy the left side of the loop around to the end. \n if (tail >= head) {\n bufsize = s;\n }\n else {\n // how much excess space do we have now?\n size_t excess = (size_t)s - bufsize;\n // move up to excess bytes to the end \n size_t movebytes = std::min<size_t>(tail, excess);\n memcpy(buffer + bufsize, buffer, movebytes); \n // move the remaining bytes to the start of the buffer\n memmove(buffer, buffer+movebytes, tail - movebytes);\n // update buftail\n // if movebytes == tail, then tail has been wiped out\n // and it is no longer a looparound\n bufsize = s; \n tail = (head + len) % bufsize;\n\n }\n consistency_check();\n }\n\n void circular_char_buffer::squeeze() {\n // squeeze to a minimum of 4 bytes\n if (bufsize <= 4) return;\n // 2 cases\n // case 1: no loop around\n // case 2: loop around. Easiest solution is to allocate a new buffer\n if (tail >= head) {\n if (head > 0) memmove(buffer, buffer+head, len);\n std::streamsize efflen = std::max(len + 1, std::streamsize(4));\n buffer = (char*)realloc((void*)buffer, efflen);\n head = 0;\n tail = len;\n bufsize = efflen;\n }\n else {\n // allocate a new buffer\n std::streamsize efflen = std::max(len + 1, std::streamsize(4));\n char *newbuf = (char*)malloc(efflen);\n // read into this buffer\n peek(newbuf, len);\n // free the old buffer\n free(buffer);\n buffer = newbuf;\n head = 0;\n tail = len;\n bufsize = efflen;\n }\n consistency_check();\n }\n\n\n void circular_char_buffer::align() {\n // squeeze to a minimum of 4 bytes\n if (bufsize <= 4) return;\n // 2 cases\n // case 1: no loop around\n // case 2: loop around. Easiest solution is to allocate a new buffer\n if (tail >= head) {\n if (head > 0) memmove(buffer, buffer+head, len);\n head = 0;\n tail = len;\n }\n else {\n // allocate a new buffer\n char *newbuf = (char*)malloc(bufsize);\n // read into this buffer\n peek(newbuf, len);\n // free the old buffer\n free(buffer);\n buffer = newbuf;\n head = 0;\n tail = len;\n }\n }\n\n bool circular_char_buffer::align_requires_alloc() {\n // squeeze to a minimum of 4 bytes\n if (bufsize <= 4) return false;\n // 2 cases\n // case 1: no loop around\n // case 2: loop around. Easiest solution is to allocate a new buffer\n if (tail >= head) {\n return false;\n }\n else {\n return true;\n }\n }\n std::streamsize circular_char_buffer::peek(char* c, \n std::streamsize clen) const {\n std::streamsize readlen = std::min(clen, len);\n // eliminate the case where head == tail, but buffer is empty\n if (readlen == 0) return 0;\n \n // first copy from head to end of buffer\n std::streamsize firstcopy = std::min(bufsize - head, readlen);\n memcpy(c, buffer+head, firstcopy);\n if (firstcopy == readlen) return readlen;\n // second copy from beginning of buffer to tail\n std::streamsize secondcopy = std::min(tail, readlen - firstcopy);\n memcpy(c+firstcopy, buffer, secondcopy);\n consistency_check();\n\n return readlen;\n\n }\n\n std::streamsize circular_char_buffer::skip(std::streamsize clen) {\n std::streamsize readlen = std::min(clen, len);\n head += readlen;\n if (head >= bufsize) head -= bufsize;\n len -= readlen;\n consistency_check(); \n return readlen;\n }\n\n std::streamsize circular_char_buffer::read(char* c, \n std::streamsize clen) {\n if (len == 0) return -1;\n std::streamsize readlen = peek(c, clen);\n skip(readlen);\n consistency_check(); \n\n return readlen;\n }\n\n std::streamsize circular_char_buffer::peek(std::string &c, \n std::streamsize clen) const{\n c.clear();\n c.resize(clen);\n return peek(const_cast<char*>(c.c_str()), clen);\n }\n\n std::streamsize circular_char_buffer::read(std::string &c, \n std::streamsize clen) {\n c.clear();\n c.resize(clen);\n return read(const_cast<char*>(c.c_str()), clen);\n }\n\n\n std::streamsize circular_char_buffer::write(const char* c, \n std::streamsize clen) {\n // if we do not have enough capacity.\n // make sure we have enough capacity\n reserve(clen + len + 1);\n len += clen;\n \n std::streamsize firstcopy = std::min(clen, bufsize - tail);\n memcpy(buffer + tail, c, firstcopy);\n tail += firstcopy;\n if (tail == bufsize) tail = 0;\n if (firstcopy == clen) {\n consistency_check(); \n return clen;\n }\n \n std::streamsize secondcopy = clen - firstcopy;\n memcpy(buffer, c + firstcopy, secondcopy);\n tail += secondcopy;\n consistency_check(); \n return clen;\n }\n\n void circular_char_buffer::clear() {\n head = 0; tail = 0; len = 0;\n }\n\n circular_char_buffer::~circular_char_buffer() {\n free(buffer);\n }\n\n std::streamsize circular_char_buffer::introspective_read(char* &s) {\n if (len == 0) {\n s = NULL;\n return 0;\n }\n s = buffer + head;\n // how much we do read?\n // we can go up to the end of the buffer, or until a looparound\n // case 1: no looparound\n // case 2: looparound\n std::streamsize readlen = 0;\n if (tail > head) {\n readlen = tail - head;\n }\n else {\n readlen = bufsize - head;\n }\n skip(readlen);\n return readlen;\n }\n\n std::streamsize circular_char_buffer::introspective_read(char* &s, std::streamsize clen) {\n if (len == 0) {\n s = NULL;\n return 0;\n }\n s = buffer + head;\n // how much we do read?\n // we can go up to the end of the buffer, or until a looparound\n // case 1: no looparound\n // case 2: looparound\n std::streamsize readlen = 0;\n if (tail > head) {\n readlen = tail - head;\n }\n else {\n readlen = bufsize - head;\n }\n if (clen < readlen) readlen = clen;\n\n skip(readlen);\n return readlen;\n }\n\n\n std::streamsize circular_char_buffer::introspective_write(char* &s) {\n s = buffer + tail;\n if (tail >= head) {\n // case 1. no looparound. \n return bufsize - tail - (head==0);\n }\n else {\n // case 2 looparound\n return head - tail - 1;\n }\n }\n \n void circular_char_buffer::advance_write(std::streamsize bytes) {\n tail += bytes;\n if (tail >= bufsize) tail -= bufsize;\n len += bytes;\n consistency_check();\n }\n \n};\n\n" }, { "path": "src/graphlab/rpc/circular_char_buffer.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_CIRCULAR_CHAR_BUFFER_HPP\n#define GRAPHLAB_CIRCULAR_CHAR_BUFFER_HPP\n#include <string>\n#include <iostream> \n#include <graphlab/logger/assertions.hpp>\n#include <boost/iostreams/stream.hpp>\n#include <boost/iostreams/categories.hpp>\nnamespace graphlab {\n \n /**\n * \\ingroup rpc\n * \\internal\n A self-resizing circular buffer of characters\n */\n class circular_char_buffer {\n public:\n \n /// Creates a circular buffer with some initial capacity\n circular_char_buffer(std::streamsize initialsize = 1024);\n /// copy constructor\n circular_char_buffer(const circular_char_buffer &src);\n /// assignment operator\n circular_char_buffer& operator=(const circular_char_buffer& src);\n \n /// destructors\n ~circular_char_buffer();\n \n /// writes len bytes into the buffer\n std::streamsize write(const char* c, std::streamsize clen);\n \n /** tries to peek up to 'len' bytes from the buffer.\n the actual number of bytes read will be returned.\n This is a non-destructive operation */\n std::streamsize peek(char* c, std::streamsize clen) const;\n \n /** reads up to 'len' bytes from the buffer.\n the actual number of bytes read will be returned.\n This is a destructive operation */\n std::streamsize read(char* c, std::streamsize clen);\n\n \n /** tries to peek up to 'len' bytes from the buffer.\n the actual number of bytes read will be returned.\n This is a non-destructive operation. string overload of peek() */\n std::streamsize peek(std::string &s, std::streamsize clen) const;\n \n /** reads up to 'len' bytes from the buffer. the actual number of\n bytes read will be returned. This is a destructive\n operation. string overload of read() */\n std::streamsize read(std::string &s, std::streamsize clen); \n \n /** skip some number of input bytes. Returns the number of bytes\n actually skipped*/\n std::streamsize skip(std::streamsize clen);\n \n /** reserves at least s bytes of capacity. Tries to perform as few\n memory copies as possible. No change is made if s is smaller\n than the current capacity. */\n void reserve(std::streamsize s);\n \n /** Squeezes out all empty capacity in the buffer so that\n the capacity is the same as the length of the buffer */\n void squeeze();\n\n \n /** Rotates the buffer so that the head is at index 0.\n buffer reserved size is preserved*/\n void align();\n\n\n /** Returns true if realignment requires a reallocation */\n bool align_requires_alloc();\n \n /**\n * Returns a pointer (through s) and a length of the read. This\n * pointer is a direct pointer into the internal buffer of this\n * datastructure. The pointer is valid as long as no other\n * operations are performed on this structure. The length of the\n * introspective_read may be less than the actual length of the\n * buffer. Multiple calls to introspective_read may be necessary\n * to read all data in the buffer. If the function returns 0, the\n * buffer is empty.\n */\n std::streamsize introspective_read(char* &s);\n \n /**\n * Returns a pointer (through s) and a length of the read. This\n * pointer is a direct pointer into the internal buffer of this\n * datastructure. The pointer is valid as long as no other\n * operations are performed on this structure. The length of the\n * introspective_read may be less than the number of bytes\n * requested. Multiple calls to introspective_read may be\n * necessary to read all data in the buffer. If the function\n * returns 0, the buffer is empty.\n */\n std::streamsize introspective_read(char* &s, std::streamsize clen);\n \n /**\n Returns a pointer to the next empty region of the buffer. The\n return value is the maximum contigious length writable. When\n writes complete, advance_write should be used to integrate the\n written bytes\n */\n std::streamsize introspective_write(char* &s);\n \n void advance_write(std::streamsize bytes);\n \n inline void consistency_check() const {\n /* ASSERT_GE(head, 0); ASSERT_GE(tail, 0);\n ASSERT_LT(head, bufsize); ASSERT_LE(tail, bufsize);\n if (tail > head) ASSERT_EQ(tail - head, len);\n else if (head < tail) ASSERT_EQ(head + bufsize - tail, len);\n else if (head == tail) ASSERT_EQ(len, 0); */\n }\n \n /** clears the stream */\n void clear();\n \n /** Gets the number of characters in the stream */\n inline std::streamsize size() const {\n return len;\n }\n \n /** Gets the size of the buffer. \n \\note: The useable space is reserved_size() - 1 */\n inline std::streamsize reserved_size() const {\n return bufsize;\n }\n private:\n \n inline bool buffer_full() const {\n return len == bufsize;\n }\n \n char* buffer;\n /** \n * points to the head of the queue. \n * Reader reads from here\n */\n std::streamsize head; \n \n /** \n * points to one past the end of the queue. \n * writer writes to here. if tail == head, buffer must be empty\n */\n std::streamsize tail; \n std::streamsize bufsize; // current size of the buffer\n std::streamsize len; // number of bytes stored in the buffer\n };\n\n /**\n A boost source device which can attach to a circular buffer\n */\n struct circular_char_buffer_source {\n circular_char_buffer_source(circular_char_buffer &buf, \n size_t maxlen = size_t(-1)):buf(buf), maxlen(maxlen) { }\n \n circular_char_buffer &buf;\n size_t maxlen;\n typedef char char_type;\n struct category : public boost::iostreams::source_tag { };\n\n /** to satisfy the optimally buffered tag. Since this is an\n in-memory buffer. the optimal buffer size (for any wrapping \n stream) is 0. */\n inline std::streamsize optimal_buffer_size() const { return 0; }\n\n inline std::streamsize read(char* s, std::streamsize n) {\n if ((size_t)(n) > maxlen) n = (std::streamsize)maxlen;\n maxlen -= (size_t)n;\n if (n == 0) return -1;\n else return buf.read(s, n);\n }\n };\n\n /**\n A boost sink device which can attach to a circular buffer\n */\n struct circular_char_buffer_sink {\n circular_char_buffer_sink(circular_char_buffer &buf):buf(buf) { }\n \n circular_char_buffer &buf;\n typedef char char_type;\n struct category: public boost::iostreams::sink_tag,\n public boost::iostreams::multichar_tag { };\n\n /** to satisfy the optimally buffered tag. Since this is an\n in-memory buffer. the optimal buffer size is 0. */\n inline std::streamsize optimal_buffer_size() const { return 0; }\n\n inline std::streamsize write(const char* s, std::streamsize n) {\n return buf.write(s, n);\n }\n };\n\n struct circular_char_buffer_device {\n circular_char_buffer_device(circular_char_buffer &buf):buf(buf) { }\n \n circular_char_buffer &buf;\n typedef char char_type;\n struct category : public boost::iostreams::bidirectional_device_tag,\n public boost::iostreams::optimally_buffered_tag{ };\n\n /** to satisfy the optimally buffered tag. Since this is an\n in-memory buffer. the optimal buffer size is 0. */\n inline std::streamsize optimal_buffer_size() const { return 0; }\n\n inline std::streamsize write(const char* s, std::streamsize n) {\n return buf.write(s, n);\n }\n \n inline std::streamsize read(char* s, std::streamsize n) {\n return buf.read(s, n);\n }\n };\n\n}\n#endif\n\n" }, { "path": "src/graphlab/rpc/circular_iovec_buffer.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_RPC_CIRCULAR_IOVEC_BUFFER_HPP\n#define GRAPHLAB_RPC_CIRCULAR_IOVEC_BUFFER_HPP\n#include <vector>\n#include <sys/socket.h>\n\nnamespace graphlab{\nnamespace dc_impl {\n\n/**\n * \\ingroup rpc\n * \\internal\n * A circular buffer which maintains a parallel sequence of iovecs.\n * One sequence is basic iovecs\n * The other sequence is used for storing the original unomidifed pointers\n * This is minimally checked. length must be a power of 2\n */\nstruct circular_iovec_buffer {\n inline circular_iovec_buffer(size_t len = 4096) {\n v.resize(4096);\n parallel_v.resize(4096);\n head = 0;\n tail = 0;\n numel = 0;\n }\n\n inline bool empty() const {\n return numel == 0;\n }\n\n size_t size() const {\n return numel;\n }\n\n\n void reserve(size_t _n) {\n if (_n <= v.size()) return;\n size_t originalsize = v.size();\n size_t n = v.size();\n // must be a power of 2\n while (n <= _n) n *= 2;\n\n v.resize(n);\n parallel_v.resize(n);\n if (head >= tail && numel > 0) {\n // there is a loop around\n // we need to fix the shift\n size_t newtail = originalsize;\n for (size_t i = 0;i < tail; ++i) {\n v[newtail] = v[i];\n parallel_v[newtail] = parallel_v[i];\n ++newtail;\n }\n tail = newtail;\n }\n }\n\n inline void write(const std::vector<iovec>& other, size_t nwrite) {\n reserve(numel + nwrite);\n for (size_t i = 0;i < nwrite; ++i) {\n v[tail] = other[i];\n parallel_v[tail] = other[i];\n tail = (tail + 1) & (v.size() - 1);\n }\n numel += nwrite;\n\n }\n\n /**\n * Writes an entry into the buffer, resizing the buffer if necessary.\n * This buffer will take over all iovec pointers and free them when done\n */\n inline void write(const iovec &entry) {\n if (numel == v.size()) {\n reserve(2 * numel);\n }\n\n v[tail] = entry;\n parallel_v[tail] = entry;\n tail = (tail + 1) & (v.size() - 1); ++numel;\n }\n\n\n /**\n * Writes an entry into the buffer, resizing the buffer if necessary.\n * This buffer will take over all iovec pointers and free them when done.\n * This version of write allows the iovec that is sent to be different from the\n * iovec that is freed. (for instance, what is sent could be subarray of\n * what is to be freed.\n */\n inline void write(const iovec &entry, const iovec& actual_ptr_entry) {\n if (numel == v.size()) {\n reserve(2 * numel);\n }\n\n v[tail] = actual_ptr_entry;\n parallel_v[tail] = entry;\n tail = (tail + 1) & (v.size() - 1); ++numel;\n }\n\n\n /**\n * Erases a single iovec from the head and free the pointer\n */\n inline void erase_from_head_and_free() {\n free(v[head].iov_base);\n head = (head + 1) & (v.size() - 1);\n --numel;\n }\n\n /**\n * Fills a msghdr for unsent data.\n */\n void fill_msghdr(struct msghdr& data) {\n data.msg_iov = &(parallel_v[head]);\n if (head < tail) {\n data.msg_iovlen = tail - head;\n }\n else {\n data.msg_iovlen = v.size() - head;\n }\n data.msg_iovlen = std::min<size_t>(IOV_MAX, data.msg_iovlen);\n }\n\n /**\n * Advances the head as if some amount of data was sent.\n */\n void sent(size_t len) {\n while(len > 0) {\n size_t curv_sent_len = std::min(len, parallel_v[head].iov_len);\n parallel_v[head].iov_len -= curv_sent_len;\n parallel_v[head].iov_base = (char*)(parallel_v[head].iov_base) + curv_sent_len;\n len -= curv_sent_len;\n if (parallel_v[head].iov_len == 0) {\n erase_from_head_and_free();\n }\n }\n }\n\n std::vector<struct iovec> v;\n std::vector<struct iovec> parallel_v;\n size_t head;\n size_t tail;\n size_t numel;\n};\n\n}\n}\n\n#endif\n" }, { "path": "src/graphlab/rpc/dc.cpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <unistd.h>\n#include <sys/param.h>\n#include <stdlib.h>\n#include <sys/types.h>\n#include <sys/socket.h>\n#include <ifaddrs.h>\n#include <netinet/in.h>\n\n#include <map>\n#include <sstream>\n\n#include <boost/unordered_map.hpp>\n#include <boost/bind.hpp>\n//#include <graphlab/logger/assertions.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/util/stl_util.hpp>\n#include <graphlab/util/net_util.hpp>\n#include <graphlab/util/mpi_tools.hpp>\n\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_tcp_comm.hpp>\n//#include <graphlab/rpc/dc_sctp_comm.hpp>\n#include <graphlab/rpc/dc_buffered_stream_send2.hpp>\n#include <graphlab/rpc/dc_stream_receive.hpp>\n#include <graphlab/rpc/request_reply_handler.hpp>\n#include <graphlab/rpc/dc_services.hpp>\n\n#include <graphlab/rpc/dc_init_from_env.hpp>\n#include <graphlab/rpc/dc_init_from_mpi.hpp>\n#include <graphlab/rpc/dc_init_from_zookeeper.hpp>\n\n\nnamespace graphlab {\n\nnamespace dc_impl {\n\n\nbool thrlocal_sequentialization_key_initialized = false;\npthread_key_t thrlocal_sequentialization_key;\n\nbool thrlocal_send_buffer_key_initialized = false;\npthread_key_t thrlocal_send_buffer_key;\n\nvoid thrlocal_send_buffer_key_deleter(void* p) {\n if (p != NULL) {\n thread_local_buffer* buf = (thread_local_buffer*)(p);\n if (buf != NULL) {\n delete buf;\n } \n }\n}\n\n} // namespace dc_impl\n\n\n\nprocid_t distributed_control::last_dc_procid = 0;\ndistributed_control* distributed_control::last_dc = NULL;\n\nprocid_t distributed_control::get_instance_procid() {\n return last_dc_procid;\n}\n\ndistributed_control* distributed_control::get_instance() {\n return last_dc;\n}\n\n\n\n\nunsigned char distributed_control::set_sequentialization_key(unsigned char newkey) {\n size_t oldval = reinterpret_cast<size_t>(pthread_getspecific(dc_impl::thrlocal_sequentialization_key));\n size_t newval = newkey;\n pthread_setspecific(dc_impl::thrlocal_sequentialization_key, reinterpret_cast<void*>(newval));\n assert(oldval < 256);\n return (unsigned char)oldval;\n}\n\nunsigned char distributed_control::new_sequentialization_key() {\n size_t oldval = reinterpret_cast<size_t>(pthread_getspecific(dc_impl::thrlocal_sequentialization_key));\n size_t newval = (oldval + 1) % 256;\n pthread_setspecific(dc_impl::thrlocal_sequentialization_key, reinterpret_cast<void*>(newval));\n assert(oldval < 256);\n return (unsigned char)oldval;\n}\n\nunsigned char distributed_control::get_sequentialization_key() {\n size_t oldval = reinterpret_cast<size_t>(pthread_getspecific(dc_impl::thrlocal_sequentialization_key));\n assert(oldval < 256);\n return (unsigned char)oldval;\n}\n\n\ndistributed_control::distributed_control() {\n dc_init_param initparam;\n if (init_param_from_env(initparam)) {\n logstream(LOG_INFO) << \"Distributed Control Initialized from Environment\" << std::endl;\n } else if (init_param_from_zookeeper(initparam)) {\n logstream(LOG_INFO) << \"Distributed Control Initialized from Zookeeper\" << std::endl;\n } else if (mpi_tools::initialized() && init_param_from_mpi(initparam)) {\n logstream(LOG_INFO) << \"Distributed Control Initialized from MPI\" << std::endl;\n }\n else {\n logstream(LOG_INFO) << \"Shared Memory Execution\" << std::endl;\n // get a port and socket\n std::pair<size_t, int> port_and_sock = get_free_tcp_port();\n size_t port = port_and_sock.first;\n int sock = port_and_sock.second;\n\n initparam.machines.push_back(std::string(\"localhost:\") + tostr(port));\n initparam.curmachineid = 0;\n initparam.initstring = std::string(\" __sockhandle__=\") + tostr(sock) + \" \";\n initparam.numhandlerthreads = RPC_DEFAULT_NUMHANDLERTHREADS;\n initparam.commtype = RPC_DEFAULT_COMMTYPE;\n }\n init(initparam.machines,\n initparam.initstring,\n initparam.curmachineid,\n initparam.numhandlerthreads,\n initparam.commtype);\n INITIALIZE_TRACER(dc_receive_queuing, \"dc: time spent on enqueue\");\n INITIALIZE_TRACER(dc_receive_multiplexing, \"dc: time spent exploding a chunk\");\n INITIALIZE_TRACER(dc_call_dispatch, \"dc: time spent issuing RPC calls\");\n}\n\ndistributed_control::distributed_control(dc_init_param initparam) {\n init(initparam.machines,\n initparam.initstring,\n initparam.curmachineid,\n initparam.numhandlerthreads,\n initparam.commtype);\n INITIALIZE_TRACER(dc_receive_queuing, \"dc: time spent on enqueue\");\n INITIALIZE_TRACER(dc_receive_multiplexing, \"dc: time spent exploding a chunk\");\n INITIALIZE_TRACER(dc_call_dispatch, \"dc: time spent issuing RPC calls\");\n}\n\n\ndistributed_control::~distributed_control() {\n // detach the instance\n last_dc = NULL;\n last_dc_procid = 0;\n distributed_services->full_barrier();\n logstream(LOG_INFO) << \"Shutting down distributed control \" << std::endl;\n FREE_CALLBACK_EVENT(EVENT_NETWORK_BYTES);\n FREE_CALLBACK_EVENT(EVENT_RPC_CALLS);\n // call all deletion callbacks\n for (size_t i = 0; i < deletion_callbacks.size(); ++i) {\n deletion_callbacks[i]();\n }\n\n size_t bytessent = bytes_sent();\n for (size_t i = 0;i < senders.size(); ++i) {\n senders[i]->flush();\n }\n\n comm->close();\n\n for (size_t i = 0;i < senders.size(); ++i) {\n delete senders[i];\n }\n senders.clear();\n\n pthread_key_delete(dc_impl::thrlocal_sequentialization_key);\n pthread_key_delete(dc_impl::thrlocal_send_buffer_key);\n\n size_t bytesreceived = bytes_received();\n for (size_t i = 0;i < receivers.size(); ++i) {\n receivers[i]->shutdown();\n delete receivers[i];\n }\n receivers.clear();\n // shutdown function call handlers\n for (size_t i = 0;i < fcallqueue.size(); ++i) fcallqueue[i].stop_blocking();\n fcallhandlers.join();\n logstream(LOG_INFO) << \"Bytes Sent: \" << bytessent << std::endl;\n logstream(LOG_INFO) << \"Calls Sent: \" << calls_sent() << std::endl;\n logstream(LOG_INFO) << \"Network Sent: \" << network_bytes_sent() << std::endl;\n logstream(LOG_INFO) << \"Bytes Received: \" << bytesreceived << std::endl;\n logstream(LOG_INFO) << \"Calls Received: \" << calls_received() << std::endl;\n\n delete comm;\n\n}\n\n\nvoid distributed_control::exec_function_call(procid_t source,\n unsigned char packet_type_mask,\n const char* data,\n const size_t len) {\n BEGIN_TRACEPOINT(dc_call_dispatch);\n // extract the dispatch function\n iarchive arc(data, len);\n size_t f;\n arc >> f;\n // a regular funcion call\n dc_impl::dispatch_type dispatch = (dc_impl::dispatch_type)f;\n dispatch(*this, source, packet_type_mask, data + arc.off, len - arc.off);\n if ((packet_type_mask & CONTROL_PACKET) == 0) inc_calls_received(source);\n END_TRACEPOINT(dc_call_dispatch);\n}\n\nunsigned char distributed_control::get_block_sequentialization_key(fcallqueue_entry& fcallblock) {\n unsigned char seq_key = 0;\n char* data = fcallblock.chunk_src;\n size_t remaininglen = fcallblock.chunk_len;\n // loop through all the messages\n while(remaininglen > 0) {\n ASSERT_GE(remaininglen, sizeof(dc_impl::packet_hdr));\n dc_impl::packet_hdr hdr = *reinterpret_cast<dc_impl::packet_hdr*>(data);\n ASSERT_LE(hdr.len, remaininglen);\n if (hdr.sequentialization_key != 0) {\n seq_key = hdr.sequentialization_key;\n break;\n }\n data += sizeof(dc_impl::packet_hdr) + hdr.len;\n remaininglen -= sizeof(dc_impl::packet_hdr) + hdr.len;\n }\n return seq_key;\n}\n\nvoid distributed_control::deferred_function_call_chunk(char* buf, size_t len, procid_t src) {\n BEGIN_TRACEPOINT(dc_receive_queuing);\n fcallqueue_entry* fc = new fcallqueue_entry;\n fc->chunk_src = buf;\n fc->chunk_len = len;\n fc->chunk_ref_counter = NULL;\n fc->is_chunk = true;\n fc->source = src;\n fcallqueue_length.inc();\n\n#ifdef RPC_BLOCK_STRIPING\n static size_t __idx;\n // approximate balancing\n size_t idx = __idx++ % fcallqueue.size();\n fcallqueue[idx].enqueue(fc, !fcall_handler_blockers.get(idx));\n#else\n idx = src % fcallqueue.size();\n fcallqueue[idx].enqueue(fc, !fcall_handler_blockers.get(idx));\n#endif\n/*\n if (get_block_sequentialization_key(*fc) > 0) {\n fcallqueue[src % fcallqueue.size()].enqueue(fc);\n } else {\n const uint32_t prod = \n random::fast_uniform(uint32_t(0), \n uint32_t(fcallqueue.size() * fcallqueue.size() - 1));\n const uint32_t r1 = prod / fcallqueue.size();\n const uint32_t r2 = prod % fcallqueue.size();\n uint32_t idx = (fcallqueue[r1].size() < fcallqueue[r2].size()) ? r1 : r2; \n fcallqueue[idx].enqueue(fc);\n } */\n\n// const uint32_t prod = \n// random::fast_uniform(uint32_t(0), \n// uint32_t(fcallqueue.size() * fcallqueue.size() - 1));\n// const uint32_t r1 = prod / fcallqueue.size();\n// const uint32_t r2 = prod % fcallqueue.size();\n// uint32_t idx = (fcallqueue[r1].size() < fcallqueue[r2].size()) ? r1 : r2; \n// fcallqueue[idx].enqueue(fc);\n END_TRACEPOINT(dc_receive_queuing);\n}\n\n\nvoid distributed_control::process_fcall_block(fcallqueue_entry &fcallblock) {\n if (fcallblock.is_chunk == false) {\n for (size_t i = 0;i < fcallblock.calls.size(); ++i) {\n fcallqueue_length.dec();\n exec_function_call(fcallblock.source, fcallblock.calls[i].packet_mask,\n fcallblock.calls[i].data, fcallblock.calls[i].len);\n }\n if (fcallblock.chunk_ref_counter != NULL) {\n if (fcallblock.chunk_ref_counter->dec(fcallblock.calls.size()) == 0) {\n delete fcallblock.chunk_ref_counter;\n free(fcallblock.chunk_src);\n }\n }\n }\n#ifdef RPC_DO_NOT_BREAK_BLOCKS\n else {\n fcallqueue_length.dec();\n\n //parse the data in fcallblock.data\n char* data = fcallblock.chunk_src;\n size_t remaininglen = fcallblock.chunk_len;\n //PERMANENT_ACCUMULATE_DIST_EVENT(eventlog, BYTES_EVENT, remaininglen);\n while(remaininglen > 0) {\n ASSERT_GE(remaininglen, sizeof(dc_impl::packet_hdr));\n dc_impl::packet_hdr hdr = *reinterpret_cast<dc_impl::packet_hdr*>(data);\n ASSERT_LE(hdr.len, remaininglen);\n\n if (!(hdr.packet_type_mask & CONTROL_PACKET)) {\n global_bytes_received[hdr.src].inc(hdr.len);\n }\n\n exec_function_call(fcallblock.source, hdr.packet_type_mask,\n data + sizeof(dc_impl::packet_hdr),\n hdr.len);\n data += sizeof(dc_impl::packet_hdr) + hdr.len;\n remaininglen -= sizeof(dc_impl::packet_hdr) + hdr.len;\n }\n free(fcallblock.chunk_src);\n }\n#else\n else {\n fcallqueue_length.dec();\n BEGIN_TRACEPOINT(dc_receive_multiplexing);\n fcallqueue_entry* queuebufs[fcallqueue.size()];\n atomic<size_t>* refctr = new atomic<size_t>(0);\n\n fcallqueue_entry immediate_queue;\n\n immediate_queue.chunk_src = fcallblock.chunk_src;\n immediate_queue.chunk_ref_counter = refctr;\n immediate_queue.chunk_len = 0;\n immediate_queue.source = fcallblock.source;\n immediate_queue.is_chunk = false;\n\n for (size_t i = 0;i < fcallqueue.size(); ++i) {\n queuebufs[i] = new fcallqueue_entry;\n queuebufs[i]->chunk_src = fcallblock.chunk_src;\n queuebufs[i]->chunk_ref_counter = refctr;\n queuebufs[i]->chunk_len = 0;\n queuebufs[i]->source = fcallblock.source;\n queuebufs[i]->is_chunk = false;\n }\n\n //parse the data in fcallblock.data\n char* data = fcallblock.chunk_src;\n size_t remaininglen = fcallblock.chunk_len;\n //PERMANENT_ACCUMULATE_DIST_EVENT(eventlog, BYTES_EVENT, remaininglen);\n size_t stripe = 0;\n while(remaininglen > 0) {\n ASSERT_GE(remaininglen, sizeof(dc_impl::packet_hdr));\n dc_impl::packet_hdr hdr = *reinterpret_cast<dc_impl::packet_hdr*>(data);\n ASSERT_LE(hdr.len, remaininglen);\n\n refctr->value++;\n\n\n if ((hdr.packet_type_mask & CONTROL_PACKET)) {\n // control calls are handled immediately with priority.\n immediate_queue.calls.push_back(function_call_block(\n data + sizeof(dc_impl::packet_hdr),\n hdr.len,\n hdr.packet_type_mask));\n } else {\n global_bytes_received[hdr.src].inc(hdr.len);\n if (hdr.sequentialization_key == 0) {\n queuebufs[stripe]->calls.push_back(function_call_block(\n data + sizeof(dc_impl::packet_hdr),\n hdr.len,\n hdr.packet_type_mask));\n ++stripe;\n if (stripe == (fcallblock.source % fcallqueue.size())) ++stripe;\n if (stripe >= fcallqueue.size()) stripe -= fcallqueue.size();\n }\n else {\n size_t idx = (hdr.sequentialization_key % (fcallqueue.size()));\n queuebufs[idx]->calls.push_back(function_call_block(\n data + sizeof(dc_impl::packet_hdr),\n hdr.len,\n hdr.packet_type_mask));\n }\n }\n data += sizeof(dc_impl::packet_hdr) + hdr.len;\n remaininglen -= sizeof(dc_impl::packet_hdr) + hdr.len;\n }\n END_TRACEPOINT(dc_receive_multiplexing);\n BEGIN_TRACEPOINT(dc_receive_queuing);\n for (size_t i = 0;i < fcallqueue.size(); ++i) {\n if (queuebufs[i]->calls.size() > 0) {\n fcallqueue_length.inc(queuebufs[i]->calls.size());\n fcallqueue[i].enqueue(queuebufs[i]);\n }\n else {\n delete queuebufs[i];\n }\n }\n END_TRACEPOINT(dc_receive_queuing);\n if (immediate_queue.calls.size() > 0) process_fcall_block(immediate_queue);\n }\n#endif\n}\n\nvoid distributed_control::stop_handler_threads(size_t threadid,\n size_t total_threadid) {\n stop_handler_threads_no_wait(threadid, total_threadid);\n}\n\nvoid distributed_control::stop_handler_threads_no_wait(size_t threadid,\n size_t total_threadid) {\n for (size_t i = threadid;i < fcallqueue.size(); i += total_threadid) {\n fcall_handler_blockers.set_bit(i);\n }\n}\n\n\nvoid distributed_control::start_handler_threads(size_t threadid,\n size_t total_threadid) {\n for (size_t i = threadid;i < fcallqueue.size(); i += total_threadid) {\n fcall_handler_blockers.clear_bit(i);\n fcallqueue[i].broadcast();\n }\n}\n\nvoid distributed_control::handle_incoming_calls(size_t threadid,\n size_t total_threadid) {\n for (size_t i = threadid;i < fcallqueue.size(); i += total_threadid) {\n if (fcallqueue[i].empty_unsafe() == false) {\n std::deque<fcallqueue_entry*> q;\n fcallqueue[i].swap(q);\n while (!q.empty()) {\n fcallqueue_entry* entry;\n entry = q.front();\n q.pop_front();\n\n process_fcall_block(*entry);\n delete entry;\n }\n }\n }\n}\n\nvoid distributed_control::fcallhandler_loop(size_t id) {\n // pop an element off the queue\n// float t = timer::approx_time_seconds();\n fcall_handler_active[id].inc();\n while(fcallqueue[id].is_alive()) {\n fcallqueue[id].wait_for_data();\n std::deque<fcallqueue_entry*> q;\n fcallqueue[id].swap(q);\n while (!q.empty()) {\n fcallqueue_entry* entry;\n entry = q.front();\n q.pop_front();\n\n process_fcall_block(*entry);\n delete entry;\n }\n // std::cerr << \"Handler \" << id << \" died.\" << std::endl;\n }\n fcall_handler_active[id].dec();\n}\n\n\nstd::map<std::string, std::string>\n distributed_control::parse_options(std::string initstring) {\n std::map<std::string, std::string> options;\n std::replace(initstring.begin(), initstring.end(), ',', ' ');\n std::replace(initstring.begin(), initstring.end(), ';', ' ');\n std::string opt, value;\n // read till the equal\n std::stringstream s(initstring);\n while(s.good()) {\n getline(s, opt, '=');\n if (s.bad() || s.eof()) break;\n getline(s, value, ' ');\n if (s.bad()) break;\n options[trim(opt)] = trim(value);\n }\n return options;\n}\n\nvoid distributed_control::init(const std::vector<std::string> &machines,\n const std::string &initstring,\n procid_t curmachineid,\n size_t numhandlerthreads,\n dc_comm_type commtype) {\n\n if (numhandlerthreads == RPC_DEFAULT_NUMHANDLERTHREADS) {\n // autoconfigure\n if (thread::cpu_count() > 2) numhandlerthreads = thread::cpu_count() - 2;\n else numhandlerthreads = 2;\n }\n ASSERT_MSG(machines.size() <= RPC_MAX_N_PROCS,\n \"Number of processes exceeded hard limit of %d\", RPC_MAX_N_PROCS);\n\n // initialize thread local storage\n if (dc_impl::thrlocal_sequentialization_key_initialized == false) {\n dc_impl::thrlocal_sequentialization_key_initialized = true;\n int err = pthread_key_create(&dc_impl::thrlocal_sequentialization_key, NULL);\n ASSERT_EQ(err, 0);\n }\n\n if (dc_impl::thrlocal_send_buffer_key_initialized == false) {\n dc_impl::thrlocal_send_buffer_key = true;\n int err = pthread_key_create(&dc_impl::thrlocal_send_buffer_key, dc_impl::thrlocal_send_buffer_key_deleter);\n ASSERT_EQ(err, 0);\n }\n\n //-------- Initialize the full barrier ---------\n full_barrier_in_effect = false;\n procs_complete.resize(machines.size());\n //-----------------------------------------------\n\n // initialize the counters\n\n global_calls_sent.resize(machines.size());\n global_calls_received.resize(machines.size());\n global_bytes_received.resize(machines.size());\n fcallqueue.resize(numhandlerthreads);\n\n // options\n set_fast_track_requests(true);\n\n // parse the initstring\n std::map<std::string,std::string> options = parse_options(initstring);\n\n if (commtype == TCP_COMM) {\n comm = new dc_impl::dc_tcp_comm();\n } else {\n ASSERT_MSG(false, \"Unexpected value for comm type\");\n }\n for (procid_t i = 0; i < machines.size(); ++i) {\n receivers.push_back(new dc_impl::dc_stream_receive(this, i));\n senders.push_back(new dc_impl::dc_buffered_stream_send2(this, comm, i));\n }\n // create the handler threads\n // store the threads in the threadgroup\n fcall_handler_active.resize(numhandlerthreads);\n fcall_handler_blockers.resize(numhandlerthreads);\n fcallhandlers.set_stacksize(256*1024); // 256K\n for (size_t i = 0;i < numhandlerthreads; ++i) {\n fiber_control::affinity_type affinity;\n affinity.clear();\n affinity.set_bit(i);\n fcallhandlers.launch(boost::bind(&distributed_control::fcallhandler_loop,\n this, i), affinity);\n }\n\n\n // set the local proc values\n localprocid = curmachineid;\n localnumprocs = machines.size();\n\n\n // construct the services\n distributed_services = new dc_services(*this);\n // start the machines\n\n // improves reliability of initialization\n#ifdef HAS_MPI\n if (mpi_tools::initialized()) MPI_Barrier(MPI_COMM_WORLD);\n#endif\n\n comm->init(machines, options, curmachineid,\n receivers, senders);\n logstream(LOG_INFO) << \"TCP Communication layer constructed.\" << std::endl;\n if (localprocid == 0) {\n logstream(LOG_EMPH) << \"Cluster of \" << machines.size() << \" instances created.\" << std::endl;\n // check for duplicate IP addresses\n std::map<std::string, size_t> ipaddresses;\n for (size_t i = 0; i < machines.size(); ++i ){\n size_t pos = machines[i].find(\":\");\n ASSERT_NE(pos, std::string::npos);\n std::string address = machines[i].substr(0, pos);\n ipaddresses[address]++;\n }\n bool hasduplicate = false;\n std::map<std::string, size_t>::const_iterator iter = ipaddresses.begin();\n while (iter != ipaddresses.end()) {\n if (iter->second > 1) {\n hasduplicate = true;\n logstream(LOG_WARNING) << \"Duplicate IP address: \" << iter->first << std::endl;\n }\n ++iter;\n }\n if (hasduplicate) {\n logstream(LOG_WARNING) << \"For maximum performance, GraphLab strongly prefers running just one process per machine.\" << std::endl;\n }\n }\n\n\n // improves reliability of initialization\n#ifdef HAS_MPI\n if (mpi_tools::initialized()) MPI_Barrier(MPI_COMM_WORLD);\n#endif\n\n // set the value of the last_dc for the get_instance function\n last_dc = this;\n // set the static variable for the get_instance_procid() function\n last_dc_procid = localprocid;\n\n barrier();\n // initialize the empty stream\n nullstrm.open(boost::iostreams::null_sink());\n\n // initialize the event log\n\n INITIALIZE_EVENT_LOG(*this);\n ADD_CUMULATIVE_CALLBACK_EVENT(EVENT_NETWORK_BYTES, \"Network Utilization\",\n \"MB\", boost::bind(&distributed_control::network_megabytes_sent, this));\n ADD_CUMULATIVE_CALLBACK_EVENT(EVENT_RPC_CALLS, \"RPC Calls\",\n \"Calls\", boost::bind(&distributed_control::calls_sent, this));\n}\n\n\n\nvoid distributed_control::barrier() {\n distributed_services->barrier();\n}\n\nvoid distributed_control::flush() {\n for (procid_t i = 0;i < senders.size(); ++i) {\n senders[i]->flush();\n }\n}\n\nvoid distributed_control::flush(procid_t p) {\n senders[p]->flush();\n}\n\nvoid distributed_control::flush_soon() {\n for (procid_t i = 0;i < senders.size(); ++i) {\n senders[i]->flush_soon();\n }\n}\n\n\nvoid distributed_control::flush_soon(procid_t p) {\n senders[p]->flush_soon();\n}\n/*****************************************************************************\n Implementation of Full Barrier\n*****************************************************************************/\n/* It is unfortunate but this is copy paste code from dc_dist_object.hpp\n I thought for a long time how to implement this without copy pasting and\n I can't think of a simple enough solution.\n\n Part of the issue is that the \"context\" concept was not built into to the\n RPC system to begin with and is currently folded in through the dc_dist_object system.\n As a result, the global context becomes very hard to define properly.\n Including a dc_dist_object as a member only resolves the high level contexts\n such as barrier, broadcast, etc which do not require intrusive access into\n deeper information about the context. The full barrier however, requires deep\n information about the context which cannot be resolved easily.\n*/\n\n/**\nThis barrier ensures globally across all machines that\nall calls issued prior to this barrier are completed before\nreturning. This function could return prematurely if\nother threads are still issuing function calls since we\ncannot differentiate between calls issued before the barrier\nand calls issued while the barrier is being evaluated.\n*/\nvoid distributed_control::full_barrier() {\n // gather a sum of all the calls issued to machine 0\n std::vector<size_t> calls_sent_to_target(numprocs(), 0);\n for (size_t i = 0;i < numprocs(); ++i) {\n calls_sent_to_target[i] = global_calls_sent[i].value;\n }\n\n // tell node 0 how many calls there are\n std::vector<std::vector<size_t> > all_calls_sent(numprocs());\n all_calls_sent[procid()] = calls_sent_to_target;\n all_gather(all_calls_sent, true);\n\n // get the number of calls I am supposed to receive from each machine\n calls_to_receive.clear(); calls_to_receive.resize(numprocs(), 0);\n for (size_t i = 0;i < numprocs(); ++i) {\n calls_to_receive[i] += all_calls_sent[i][procid()];\n// std::cout << \"Expecting \" << calls_to_receive[i] << \" calls from \" << i << std::endl;\n }\n // clear the counters\n num_proc_recvs_incomplete.value = numprocs();\n procs_complete.clear();\n // activate the full barrier\n full_barrier_in_effect = true;\n __asm(\"mfence\");\n // begin one pass to set all which are already completed\n for (procid_t i = 0;i < numprocs(); ++i) {\n if (global_calls_received[i].value >= calls_to_receive[i]) {\n if (procs_complete.set_bit(i) == false) {\n num_proc_recvs_incomplete.dec();\n }\n }\n }\n\n full_barrier_lock.lock();\n while (num_proc_recvs_incomplete.value > 0) full_barrier_cond.wait(full_barrier_lock);\n full_barrier_lock.unlock();\n full_barrier_in_effect = false;\n barrier();\n// for (size_t i = 0; i < numprocs(); ++i) {\n// std::cout << \"Received \" << global_calls_received[i].value << \" from \" << i << std::endl;\n// }\n}\n\n\n} //namespace graphlab\n\n" }, { "path": "src/graphlab/rpc/dc.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_DC_HPP\n#define GRAPHLAB_DC_HPP\n#include <iostream>\n#include <boost/iostreams/stream.hpp>\n#include <boost/function.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/parallel/fiber_group.hpp>\n#include <graphlab/parallel/fiber_conditional.hpp>\n#include <graphlab/util/resizing_array_sink.hpp>\n#include <graphlab/util/fiber_blocking_queue.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n#include <graphlab/serialization/serialization_includes.hpp>\n\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n\n#include <graphlab/rpc/dc_receive.hpp>\n#include <graphlab/rpc/dc_send.hpp>\n#include <graphlab/rpc/dc_comm_base.hpp>\n#include <graphlab/rpc/dc_dist_object_base.hpp>\n\n#include <graphlab/rpc/is_rpc_call.hpp>\n#include <graphlab/rpc/function_call_issue.hpp>\n#include <graphlab/rpc/function_broadcast_issue.hpp>\n#include <graphlab/rpc/request_issue.hpp>\n#include <graphlab/rpc/request_reply_handler.hpp>\n#include <graphlab/rpc/function_ret_type.hpp>\n#include <graphlab/rpc/dc_compile_parameters.hpp>\n#include <graphlab/rpc/thread_local_send_buffer.hpp>\n#include <graphlab/util/tracepoint.hpp>\n#include <graphlab/rpc/distributed_event_log.hpp>\n#include <boost/preprocessor.hpp>\n#include <graphlab/rpc/function_arg_types_def.hpp>\n\n\nnamespace graphlab {\n\n\n/**\n * \\ingroup rpc\n * \\brief Distributed control constructor parameters.\n *\n * Provides the communication layer with a list of ip addresses and\n * port numbers which enumerate all the machines to establish connections\n * with.\n *\n * You should not need to this. The default constructor in\n * graphlab::distributed_control does it for you.\n * See \\ref RPC for usage details.\n */\nstruct dc_init_param{\n /** A vector containing a list of hostnames/ipaddresses and port numbers\n * of all machines participating in this RPC program.\n * for instance:\n * \\code\n * machines.push_back(\"127.0.0.1:10000\");\n * machines.push_back(\"127.0.0.1:10001\");\n * \\endcode\n */\n std::vector<std::string> machines;\n\n /** Additional construction options of the form\n \"key1=value1,key2=value2\".\n\n There are no available options at this time.\n\n Internal options which should not be used\n \\li \\b __socket__=NUMBER Forces TCP comm to use this socket number for its\n listening socket instead of creating a new one.\n The socket must already be bound to the listening\n port.\n */\n std::string initstring;\n\n /** The index of this machine into the machines vector */\n procid_t curmachineid;\n /** Number of background RPC handling threads to create */\n size_t numhandlerthreads;\n /** The communication method. */\n dc_comm_type commtype;\n\n /**\n * Constructs a dc_init_param object.\n * \\param numhandlerthreads Optional Argument. The number of handler\n * threads to create. Defaults to\n * \\ref RPC_DEFAULT_NUMHANDLERTHREADS\n * \\param commtype The Communication type. The only accepted value now is\n * TCP_COMM\n */\n dc_init_param(size_t numhandlerthreads = RPC_DEFAULT_NUMHANDLERTHREADS,\n dc_comm_type commtype = RPC_DEFAULT_COMMTYPE):\n numhandlerthreads(numhandlerthreads),\n commtype(commtype) {\n }\n};\n\n\n\n// forward declarations\nclass dc_services;\n\nnamespace dc_impl {\n class dc_buffered_stream_send2;\n class dc_stream_receive;\n}\n\n/**\n * \\ingroup rpc\n * \\brief The distributed control object is primary means of communication\n * between the distributed GraphLab processes.\n *\n * The distributed_control object provides asynchronous, multi-threaded\n * Remote Procedure Call (RPC) services to allow distributed GraphLab\n * processes to communicate with each other. Currently, the only\n * communication method implemented is TCP/IP.\n * There are several ways of setting up the communication layer, but the most\n * reliable, and the preferred method, is to \"bootstrap\" using MPI. See your\n * local MPI documentation for details on how to launch MPI jobs.\n *\n * To construct a distributed_control object, the simplest method is to just\n * invoke the default constructor.\n *\n * \\code\n * // initialize MPI\n * mpi_tools::init(argc, argv);\n * // construct distributed control object\n * graphlab::distributed_control dc;\n * \\endcode\n *\n * After which all distributed control services will operate correctly.\n *\n * Each process is assigned a sequential process ID at starting at 0.\n * i.e. The first process will have a process ID of 0, the second process\n * will have an ID of 1, etc. distributed_control::procid() can be used to\n * obtain the current machine's process ID, and distributed_control::numprocs()\n * can be used to obtain the total number of processes.\n *\n * The primary functions used to communicate between processes are\n * distributed_control::remote_call() and\n * distributed_control::remote_request(). These functions are thread-safe and\n * can be called very rapidly as they only write into a local buffer.\n * Communication is handled by a background thread. On the remote side,\n * RPC calls are handled in parallel by a thread pool, and thus may be\n * parallelized arbitrarily. Operations such as\n * distributed_control::full_barrier(), or the sequentialization key\n * can be used to get finer grained control over order of execution on the\n * remote machine.\n *\n * A few other additional helper functions are also provided to support\n * \"synchronous\" modes of communication. These functions are not thread-safe\n * and can only be called on one thread per machine. These functions block\n * until all machines call the same function. For instance, if gather() is\n * called on one machine, it will not return until all machines call gather().\n *\n * \\li distributed_control::barrier()\n * \\li distributed_control::full_barrier()\n * \\li distributed_control::broadcast()\n * \\li distributed_control::all_reduce()\n * \\li distributed_control::all_reduce2()\n * \\li distributed_control::gather()\n * \\li distributed_control::all_gather()\n *\n * \\note These synchronous operations are modeled after some MPI collective\n * operations. However, these operations here are not particularly optimized\n * and will generally be slower than their MPI counterparts. However, the\n * implementations here are much easier to use, relying extensively on\n * serialization to simplify communication.\n *\n * To support Object Oriented Programming like methodologies, we allow the\n * creation of <b>Distributed Objects</b> through graphlab::dc_dist_object.\n * dc_dist_object allows a class to construct its own local copy of\n * a distributed_control object allowing instances of the class to communicate\n * with each other across the network.\n *\n * See \\ref RPC for usage examples.\n */\nclass distributed_control{\n public:\n /** \\internal\n * Each element of the function call queue is a data/len pair */\n struct function_call_block{\n function_call_block() {}\n\n function_call_block(char* data, size_t len,\n unsigned char packet_mask):\n data(data), len(len), packet_mask(packet_mask){}\n\n char* data;\n size_t len;\n unsigned char packet_mask;\n };\n private:\n /// initialize receiver threads. private form of the constructor\n void init(const std::vector<std::string> &machines,\n const std::string &initstring,\n procid_t curmachineid,\n size_t numhandlerthreads,\n dc_comm_type commtype = RPC_DEFAULT_COMMTYPE);\n\n /// a pointer to the communications subsystem\n dc_impl::dc_comm_base* comm;\n\n /// senders and receivers to all machines\n std::vector<dc_impl::dc_receive*> receivers;\n std::vector<dc_impl::dc_send*> senders;\n\n /// A thread group of function call handlers\n fiber_group fcallhandlers;\n std::vector<atomic<size_t> > fcall_handler_active;\n dense_bitset fcall_handler_blockers;\n\n struct fcallqueue_entry {\n std::vector<function_call_block> calls;\n char* chunk_src;\n size_t chunk_len;\n atomic<size_t>* chunk_ref_counter;\n procid_t source;\n bool is_chunk;\n };\n /// a queue of functions to be executed\n std::vector<fiber_blocking_queue<fcallqueue_entry*> > fcallqueue;\n // number of blocks waiting to be deserialized + the number of\n // incomplete function calls\n atomic<size_t> fcallqueue_length;\n\n /// object registrations;\n std::vector<void*> registered_objects;\n std::vector<dc_impl::dc_dist_object_base*> registered_rmi_instance;\n\n /// For convenience, we provide a instance of dc_services\n dc_services* distributed_services;\n\n /// ID of the local machine\n procid_t localprocid;\n\n\n /// Number of machines\n procid_t localnumprocs;\n\n std::vector<atomic<size_t> > global_calls_sent;\n std::vector<atomic<size_t> > global_calls_received;\n\n std::vector<atomic<size_t> > global_bytes_received;\n\n std::vector<boost::function<void(void)> > deletion_callbacks;\n\n template <typename T> friend class dc_dist_object;\n friend class dc_impl::dc_stream_receive;\n friend class dc_impl::dc_buffered_stream_send2;\n friend struct dc_impl::thread_local_buffer;\n\n /// disable the operator= by placing it in private\n distributed_control& operator=(const distributed_control& dc) { return *this; }\n\n\n std::map<std::string, std::string> parse_options(std::string initstring);\n\n volatile inline size_t num_registered_objects() {\n return registered_objects.size();\n }\n\n\n\n DECLARE_TRACER(dc_receive_queuing);\n DECLARE_TRACER(dc_receive_multiplexing);\n DECLARE_TRACER(dc_call_dispatch);\n\n DECLARE_EVENT(EVENT_NETWORK_BYTES);\n DECLARE_EVENT(EVENT_RPC_CALLS);\n public:\n\n /**\n * Default constructor. Automatically tries to read the initialization\n * from environment variables, or from MPI (if MPI is initialized).\n */\n distributed_control();\n\n /**\n * Passes custom constructed initialization parameters in\n * \\ref dc_init_param\n *\n * Though dc_init_param can be obtained from environment variables using\n * dc_init_from_env() or from MPI using dc_init_from_mpi(),\n * using the default constructor is prefered.\n */\n explicit distributed_control(dc_init_param initparam);\n\n ~distributed_control();\n\n\n // The procid of the last distributed_control object created\n // this is quite legacy stuff when we technically permitted multiple DC\n // objects. Now, a lot of the system is built around the assumption\n // of a singleton DC.\n static procid_t last_dc_procid;\n // a pointer to the last distributed_control object created\n // this is quite legacy stuff when we technically permitted multiple DC\n // objects. Now, a lot of the system is built around the assumption\n // of a singleton DC.\n static distributed_control* last_dc;\n\n /**\n * Gets the procid of the last distributed_control instance created.\n * If there is no distributed_control instance, this returns 0.\n * For instance, this returns the current machine's procid if there is only\n * one distributed_control.\n */\n static procid_t get_instance_procid();\n\n inline size_t num_handler_threads() const {\n return fcallqueue.size();\n }\n\n /**\n * Gets a pointer to the last distributed_control instance created.\n * If there is no distributed_control instance, this returns NULL.\n */\n static distributed_control* get_instance();\n\n /// returns the id of the current process\n inline procid_t procid() const {\n return localprocid;\n }\n\n /// returns the number of processes in total.\n inline procid_t numprocs() const {\n return localnumprocs;\n }\n\n\n bool use_fast_track_requests;\n\n /// Sets the fast track status, returning the previous value\n bool set_fast_track_requests(bool val) {\n bool ret = use_fast_track_requests;\n use_fast_track_requests = val;\n return ret;\n }\n\n /// Returns true if we should fast track all request messages\n bool fast_track_requests() {\n return use_fast_track_requests;\n }\n\n /**\n * \\brief Registers a callback which will be called on deletion of the\n * distributed_control object.\n *\n * This function is useful for distributed static variables which may\n * be only be deleted after main().\n */\n void register_deletion_callback(boost::function<void(void)> deleter) {\n deletion_callbacks.push_back(deleter);\n }\n\n /**\n \\brief Sets the sequentialization key to a new value, returning the\n previous value.\n\n All RPC calls made using the same key value (as long as the key is non-zero)\n will sequentialize. RPC calls made while the key value is 0 can be\n run in parallel in arbitrary order.\n\n \\code\n oldval = distributed_control::set_sequentialization_key(new_key);\n // ...\n // ... do stuff\n // ...\n set_sequentialization_key(oldval);\n \\endcode\n\n The key value is <b>thread-local</b> thus setting the key value in\n one thread does not affect the key value in another thread.\n */\n static unsigned char set_sequentialization_key(unsigned char newkey);\n\n /**\n \\brief Creates a new sequentialization key, returning the old value.\n\n All RPC calls made using the same key value (as long as the key is non-zero)\n will sequentialize. RPC calls made while the key value is 0 can be run in\n parallel in arbitrary order. However, since new_sequentialization_key() uses\n a very naive key selection system, we recommend the use of\n set_sequentialization_key().\n\n User should\n \\code\n oldval = distributed_control::new_sequentialization_key();\n // ...\n // ... do stuff\n // ...\n set_sequentialization_key(oldval);\n \\endcode\n\n The key value is <b>thread-local</b> thus setting the key value in\n one thread does not affect the key value in another thread.\n */\n static unsigned char new_sequentialization_key();\n\n /** \\brief gets the current sequentialization key. This function is not\n * generally useful.\n */\n static unsigned char get_sequentialization_key();\n\n\n\n\n /*\n * The key RPC communication functions are all macro generated\n * and doxygen does not like them so much.\n * Here, we will block all of them out\n * and have another set of \"fake\" functions later on which are wrapped\n * with a #if 0 so C++ will ignore them.\n */\n\n /// \\cond GRAPHLAB_INTERNAL\n\n\n\n /*\n This generates the interface functions for the standard calls, basic calls\n The generated code looks like this:\n\n template<typename F , typename T0> void remote_call (procid_t target, F remote_function , const T0 &i0 )\n {\n ASSERT_LT(target, senders.size());\n dc_impl::remote_call_issue1 <F , T0> ::exec(senders[target],\n STANDARD_CALL,\n target,\n remote_function ,\n i0 );\n }\n The arguments passed to the RPC_INTERFACE_GENERATOR ARE: (interface name, issue processor name, flags)\n\n */\n #define GENARGS(Z,N,_) BOOST_PP_CAT(const T, N) BOOST_PP_CAT(&i, N)\n #define GENI(Z,N,_) BOOST_PP_CAT(i, N)\n #define GENT(Z,N,_) BOOST_PP_CAT(T, N)\n #define GENARC(Z,N,_) arc << BOOST_PP_CAT(i, N);\n\n #define RPC_INTERFACE_GENERATOR(Z,N,FNAME_AND_CALL) \\\n template<typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\n void BOOST_PP_TUPLE_ELEM(3,0,FNAME_AND_CALL) (procid_t target, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n ASSERT_LT(target, senders.size()); \\\n BOOST_PP_CAT( BOOST_PP_TUPLE_ELEM(3,1,FNAME_AND_CALL),N) \\\n <F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, T)> \\\n ::exec(senders[target], BOOST_PP_TUPLE_ELEM(3,2,FNAME_AND_CALL), target, remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENI ,_) ); \\\n } \\\n\n /*\n Generates the interface functions. 3rd argument is a tuple (interface name, issue name, flags)\n */\n BOOST_PP_REPEAT(6, RPC_INTERFACE_GENERATOR, (remote_call, dc_impl::remote_call_issue, STANDARD_CALL) )\n BOOST_PP_REPEAT(6, RPC_INTERFACE_GENERATOR, (reply_remote_call,dc_impl::remote_call_issue, STANDARD_CALL | FLUSH_PACKET) )\n BOOST_PP_REPEAT(6, RPC_INTERFACE_GENERATOR, (control_call, dc_impl::remote_call_issue, (STANDARD_CALL | CONTROL_PACKET)) )\n\n\n#define BROADCAST_INTERFACE_GENERATOR(Z,N,FNAME_AND_CALL) \\\n template<typename Iterator, typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\n void BOOST_PP_TUPLE_ELEM(3,0,FNAME_AND_CALL) (Iterator target_begin, Iterator target_end, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n if (target_begin == target_end) return; \\\n BOOST_PP_CAT( BOOST_PP_TUPLE_ELEM(3,1,FNAME_AND_CALL),N) \\\n <Iterator, F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, T)> \\\n ::exec(senders, BOOST_PP_TUPLE_ELEM(3,2,FNAME_AND_CALL), target_begin, target_end, remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENI ,_) ); \\\n } \\\n\n BOOST_PP_REPEAT(6, BROADCAST_INTERFACE_GENERATOR, (remote_call, dc_impl::remote_broadcast_issue, STANDARD_CALL) )\n\n\n #define CUSTOM_REQUEST_INTERFACE_GENERATOR(Z,N,ARGS) \\\n template<typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\n BOOST_PP_TUPLE_ELEM(2,0,ARGS) (procid_t target, size_t handle, unsigned char flags, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n BOOST_PP_CAT( BOOST_PP_TUPLE_ELEM(2,1,ARGS),N) \\\n <F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, T)> \\\n ::exec(senders[target], handle, flags, target, remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENI ,_) ); \\\n } \n\n\n #define FUTURE_REQUEST_INTERFACE_GENERATOR(Z,N,ARGS) \\\n template<typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\n BOOST_PP_TUPLE_ELEM(2,0,ARGS) (procid_t target, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n ASSERT_LT(target, senders.size()); \\\n request_future<__GLRPC_FRESULT> reply; \\\n custom_remote_request(target, reply.get_handle(), BOOST_PP_TUPLE_ELEM(2,1,ARGS), remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENI ,_) ); \\\n return reply; \\\n } \n\n\n #define REQUEST_INTERFACE_GENERATOR(Z,N,ARGS) \\\n template<typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\n BOOST_PP_TUPLE_ELEM(2,0,ARGS) (procid_t target, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n request_future<__GLRPC_FRESULT> reply; \\\n custom_remote_request(target, reply.get_handle(), BOOST_PP_TUPLE_ELEM(2,1,ARGS), remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENI ,_) ); \\\n return reply(); \\\n } \n\n\n /*\n Generates the interface functions. 3rd argument is a tuple (interface name, issue name, flags)\n */\n BOOST_PP_REPEAT(7, CUSTOM_REQUEST_INTERFACE_GENERATOR, (void custom_remote_request, dc_impl::remote_request_issue) )\n BOOST_PP_REPEAT(7, REQUEST_INTERFACE_GENERATOR, (typename dc_impl::function_ret_type<__GLRPC_FRESULT>::type remote_request, (STANDARD_CALL | FLUSH_PACKET)) )\n BOOST_PP_REPEAT(7, FUTURE_REQUEST_INTERFACE_GENERATOR, (request_future<__GLRPC_FRESULT> future_remote_request, (STANDARD_CALL)) )\n\n\n\n #undef RPC_INTERFACE_GENERATOR\n #undef BROADCAST_INTERFACE_GENERATOR\n #undef REQUEST_INTERFACE_GENERATOR\n #undef FUTURE_REQUEST_INTERFACE_GENERATOR\n #undef CUSTOM_REQUEST_INTERFACE_GENERATOR\n #undef GENARC\n #undef GENT\n #undef GENI\n #undef GENARGS\n /// \\endcond\n\n/*************************************************************************\n * Here begins the Doxygen fake functions block *\n *************************************************************************/\n\n#if DOXYGEN_DOCUMENTATION\n\n/**\n * \\brief Performs a non-blocking RPC call to the target machine\n * to run the provided function pointer.\n *\n * remote_call() calls the function \"fn\" on a target remote machine. Provided\n * arguments are serialized and sent to the target.\n * Therefore, all arguments are necessarily transmitted by value.\n * If the target function has a return value, the return value is lost.\n *\n * remote_call() is non-blocking and does not wait for the target machine\n * to complete execution of the function. Different remote_calls may be handled\n * by different threads on the target machine and thus the target function\n * should be made thread-safe.\n * Alternatively, see set_sequentialization_key()\n * to force sequentialization of groups of remote calls.\n *\n * If blocking operation is desired, remote_request() may be used.\n * Alternatively, a full_barrier() may also be used to wait for completion of\n * all incomplete RPC calls.\n *\n * Example:\n * \\code\n * // A print function is defined\n * void print(std::string s) {\n * std::cout << s << \"\\n\";\n * }\n *\n * ... ...\n * // call the print function on machine 1 to print \"hello\"\n * dc.remote_call(1, print, \"hello\");\n * \\endcode\n *\n *\n *\n * \\param targetmachine The ID of the machine to run the function on\n * \\param fn The function to run on the target machine\n * \\param ... The arguments to send to Fn. Arguments must be serializable.\n * and must be castable to the target types.\n */\n void remote_call(procid_t targetmachine, Fn fn, ...);\n\n\n\n/**\n * \\brief Performs a non-blocking RPC call to a collection of machines\n * to run the provided function pointer.\n *\n * This function calls the provided function pointer on a collection of\n * machines contained in the iterator range [begin, end).\n * Provided arguments are serialized and sent to the target.\n * Therefore, all arguments are necessarily transmitted by value.\n * If the target function has a return value, the return value is lost.\n *\n * This function is functionally equivalent to:\n *\n * \\code\n * while(machine_begin != machine_end) {\n * remote_call(*machine_begin, fn, ...);\n * ++machine_begin;\n * }\n * \\endcode\n *\n * However, this function makes some optimizations to ensure all arguments\n * are only serialized once instead of \\#calls times.\n *\n * This function is non-blocking and does not wait for the target machines\n * to complete execution of the function. Different remote_calls may be handled\n * by different threads on the target machines and thus the target function\n * should be made thread-safe. Alternatively, see set_sequentialization_key()\n * to force sequentialization of groups of remote_calls. A full_barrier()\n * may also be issued to wait for completion of all RPC calls issued prior\n * to the full barrier.\n *\n * Example:\n * \\code\n * // A print function is defined\n * void print(std::string s) {\n * std::cout << s << \"\\n\";\n * }\n *\n * ... ...\n * // call the print function on machine 1, 3 and 5 to print \"hello\"\n * std::vector<procid_t> procs;\n * procs.push_back(1); procs.push_back(3); procs.push_back(5);\n * dc.remote_call(procs.begin(), procs.end(), print, \"hello\");\n * \\endcode\n *\n *\n * \\param machine_begin The beginning of an iterator range containing a list\n * machines to call. Iterator::value_type must be\n * castable to procid_t.\n * \\param machine_end The end of an iterator range containing a list\n * machines to call. Iterator::value_type must be\n * castable to procid_t.\n * \\param fn The function to run on the target machine\n * \\param ... The arguments to send to Fn. Arguments must be serializable.\n * and must be castable to the target types.\n */\n void remote_call(Iterator machine_begin, Iterator machine_end, Fn fn, ...);\n\n\n/**\n * \\brief Performs a blocking RPC call to the target machine\n * to run the provided function pointer.\n *\n * remote_request() calls the function \"fn\" on a target remote machine. Provided\n * arguments are serialized and sent to the target.\n * Therefore, all arguments are necessarily transmitted by value.\n * If the target function has a return value, it is sent back to calling\n * machine.\n *\n * Unlike remote_call(), remote_request() is blocking and waits for the target\n * machine to complete execution of the function. However, different\n * remote_requests may be still be handled by different threads on the target\n * machine.\n *\n * Example:\n * \\code\n * // A print function is defined\n * int add_one(int i) {\n * return i + 1;\n * }\n *\n * ... ...\n * // call the add_one function on machine 1\n * int i = 10;\n * i = dc.remote_request(1, add_one, i);\n * // i will now be 11\n * \\endcode\n *\n * \\see graphlab::fiber_remote_request\n * graphlab::distributed_control::future_remote_request\n *\n * \\param targetmachine The ID of the machine to run the function on\n * \\param fn The function to run on the target machine\n * \\param ... The arguments to send to Fn. Arguments must be serializable.\n * and must be castable to the target types.\n *\n * \\returns Returns the same return type as the function fn\n */\n RetVal remote_request(procid_t targetmachine, Fn fn, ...);\n\n\n\n/**\n * \\brief Performs a non-blocking RPC call to the target machine\n * to run the provided function pointer.\n *\n * future_remote_request() calls the function \"fn\" on a target remote machine. Provided\n * arguments are serialized and sent to the target.\n * Therefore, all arguments are necessarily transmitted by value.\n * If the target function has a return value, it is sent back to calling\n * machine.\n *\n * future_remote_request() is like remote_request(), but is non-blocking.\n * Instead, it returns immediately a \\ref graphlab::request_future object\n * which will allow you wait for the return value.\n *\n * Example:\n * \\code\n * // A print function is defined\n * int add_one(int i) {\n * return i + 1;\n * }\n *\n * ... ...\n * // call the add_one function on machine 1\n * int i = 10;\n * graphlab::request_future<int> ret = dc.remote_request(1, add_one, i);\n * int result = ret();\n * // result will be 11\n * \\endcode\n *\n * \\see graphlab::fiber_remote_request\n * graphlab::distributed_control::remote_request\n *\n * \\param targetmachine The ID of the machine to run the function on\n * \\param fn The function to run on the target machine\n * \\param ... The arguments to send to Fn. Arguments must be serializable.\n * and must be castable to the target types.\n *\n * \\returns Returns the same return type as the function fn\n */\n request_future<RetVal> future_remote_request(procid_t targetmachine, Fn fn, ...);\n\n\n\n#endif\n/*************************************************************************\n * Here end the Doxygen fake functions block *\n *************************************************************************/\n\n\n private:\n /**\n *\n * l\n Immediately calls the function described by the data\n inside the buffer. This should not be called directly.\n */\n void exec_function_call(procid_t source, unsigned char packet_type_mask, const char* data, const size_t len);\n\n\n\n /**\n * \\internal\n * Called by handler threads to process the function call block\n */\n void process_fcall_block(fcallqueue_entry &fcallblock);\n\n\n /**\n * \\internal\n * Receive a collection of serialized function calls.\n * This function will take ownership of the pointer\n */\n void deferred_function_call_chunk(char* buf, size_t len, procid_t src);\n\n\n /**\n * \\internal\n * Gets the sequentialization key of a block if any.\n */\n unsigned char get_block_sequentialization_key(fcallqueue_entry& fcallblock);\n\n /**\n * \\internal\n This is called by the function handler threads\n */\n void fcallhandler_loop(size_t id);\n\n public:\n /// \\cond GRAPHLAB_INTERNAL\n /**\n * \\internal\n * Stops one group of handler threads and wait for them to complete.\n * May be used to allow external threads to take over RPC processing.\n *\n * \\param threadid Group number to stop\n * \\param total_threadid Number of groups\n */\n void stop_handler_threads(size_t threadid, size_t total_threadid);\n\n /**\n * \\internal\n * Stops one group of handler threads and returns immediately without\n * waiting for them to complete.\n * May be used to allow external threads to take over RPC processing.\n *\n * \\param threadid Group number to stop\n * \\param total_threadid Number of groups\n */\n void stop_handler_threads_no_wait(size_t threadid, size_t total_threadid);\n\n /**\n * \\internal\n * Performs RPC processing for a group of threads in lieu of the built-in\n * RPC threads. The group must be stopped before using stop_handler_threads\n *\n * \\param threadid Group number to handle\n * \\param total_threadid Number of groups\n */\n void handle_incoming_calls(size_t threadid, size_t total_threadid);\n\n\n /**\n * \\internal\n * Restarts internal RPC threads for a group.\n * The group must be stopped before using stop_handler_threads\n *\n * \\param threadid Group number to restart\n * \\param total_threadid Number of groups\n */\n void start_handler_threads(size_t threadid, size_t total_threadid);\n\n /// \\internal\n size_t recv_queue_length() const {\n return fcallqueue_length.value;\n }\n /// \\internal\n size_t send_queue_length() const {\n return comm->send_queue_length();\n }\n\n /// \\endcond\n\n private:\n inline void inc_calls_sent(procid_t procid) {\n //PERMANENT_ACCUMULATE_DIST_EVENT(eventlog, CALLS_EVENT, 1);\n global_calls_sent[procid].inc();\n }\n\n inline void inc_calls_received(procid_t procid) {\n\n if (!full_barrier_in_effect) {\n size_t t = global_calls_received[procid].inc();\n if (full_barrier_in_effect) {\n if (t == calls_to_receive[procid]) {\n // if it was me who set the bit\n if (procs_complete.set_bit(procid) == false) {\n // then decrement the incomplete count.\n // if it was me to decreased it to 0\n // lock and signal\n full_barrier_lock.lock();\n if (num_proc_recvs_incomplete.dec() == 0) {\n full_barrier_cond.signal();\n }\n full_barrier_lock.unlock();\n }\n }\n }\n }\n else {\n //check the proc I just incremented.\n // If I just exceeded the required size, I need\n // to decrement the full barrier counter\n if (global_calls_received[procid].inc() == calls_to_receive[procid]) {\n // if it was me who set the bit\n if (procs_complete.set_bit(procid) == false) {\n // then decrement the incomplete count.\n // if it was me to decreased it to 0\n // lock and signal\n full_barrier_lock.lock();\n if (num_proc_recvs_incomplete.dec() == 0) {\n full_barrier_cond.signal();\n }\n full_barrier_lock.unlock();\n }\n }\n }\n }\n\n public:\n /// \\brief Returns the total number of RPC calls made\n inline size_t calls_sent() const {\n size_t ctr = 0;\n for (size_t i = 0;i < numprocs(); ++i) {\n ctr += global_calls_sent[i].value;\n }\n return ctr;\n }\n\n /// \\brief Returns the total number of RPC calls made in millions\n inline double mega_calls_sent() const {\n size_t ctr = 0;\n for (size_t i = 0;i < numprocs(); ++i) {\n ctr += global_calls_sent[i].value;\n }\n return double(ctr)/(1024 * 1024);\n }\n\n\n\n /// \\brief Returns the total number of RPC calls received\n inline size_t calls_received() const {\n size_t ctr = 0;\n for (size_t i = 0;i < numprocs(); ++i) {\n ctr += global_calls_received[i].value;\n }\n return ctr;\n }\n\n /** \\brief Returns the total number of bytes sent excluding headers and other\n * control overhead. Also see network_bytes_sent()\n */\n inline size_t bytes_sent() const {\n size_t ret = 0;\n for (size_t i = 0;i < senders.size(); ++i) ret += senders[i]->bytes_sent();\n return ret;\n }\n\n /** \\brief Returns the total number of bytes sent including all headers\n * and other control overhead. Also see bytes_sent()\n */\n inline size_t network_bytes_sent() const {\n return comm->network_bytes_sent();\n }\n\n /** \\brief Returns the total number of megabytes sent including all headers\n * and other control overhead. Also see network_bytes_sent()\n */\n inline double network_megabytes_sent() const {\n return double(comm->network_bytes_sent()) / (1024 * 1024);\n }\n\n\n\n /** \\brief Returns the total number of bytes received excluding all headers\n * and other control overhead. Also see bytes_sent().\n */\n inline size_t bytes_received() const {\n size_t ret = 0;\n for (size_t i = 0;i < global_bytes_received.size(); ++i) {\n ret += global_bytes_received[i].value;\n }\n return ret;\n }\n\n /// \\cond GRAPHLAB_INTERNAL\n\n /// \\internal\n inline size_t register_object(void* v, dc_impl::dc_dist_object_base *rmiinstance) {\n ASSERT_NE(v, (void*)NULL);\n registered_objects.push_back(v);\n registered_rmi_instance.push_back(rmiinstance);\n return registered_objects.size() - 1;\n }\n\n /// \\internal\n inline void* get_registered_object(size_t id) {\n while(__builtin_expect((id >= num_registered_objects()), 0)) sched_yield();\n while (__builtin_expect(registered_objects[id] == NULL, 0)) sched_yield();\n return registered_objects[id];\n }\n\n /// \\internal\n inline dc_impl::dc_dist_object_base* get_rmi_instance(size_t id) {\n while(id >= num_registered_objects()) sched_yield();\n ASSERT_NE(registered_rmi_instance[id], (void*)NULL);\n return registered_rmi_instance[id];\n }\n\n /// \\internal\n inline void clear_registered_object(size_t id) {\n registered_objects[id] = (void*)NULL;\n registered_rmi_instance[id] = NULL;\n }\n\n inline void register_send_buffer(dc_impl::thread_local_buffer* buffer) {\n for (size_t i = 0;i < senders.size(); ++i) {\n senders[i]->register_send_buffer(buffer);\n }\n }\n\n inline void unregister_send_buffer(dc_impl::thread_local_buffer* buffer) {\n for (size_t i = 0;i < senders.size(); ++i) {\n senders[i]->unregister_send_buffer(buffer);\n }\n }\n\n /// \\endcond\n\n /**\n * \\brief Performs a local flush of all send buffers\n */\n void flush();\n\n /**\n * \\brief Performs a local flush of all send buffers\n */\n void flush(procid_t p);\n\n /**\n * \\brief Requests a flush of all send buffers to happen soon;\n */\n void flush_soon();\n\n /**\n * \\brief Requests a flush of one send buffers to happen soon;\n */\n void flush_soon(procid_t p);\n\n /**\n * \\brief Writes a string to the send buffer and flushes\n */\n inline void write_to_buffer(procid_t target, char* c, size_t len) {\n senders[target]->write_to_buffer(c, len);\n }\n\n\n /**\n * \\brief Sends an object to a target machine and blocks until the\n * target machine calls recv_from() to receive the object.\n *\n * This function sends a \\ref sec_serializable object \"t\" to the target\n * machine, but waits for the target machine to call recv_from()\n * before returning to receive the object before returning.\n *\n * Example:\n * \\code\n * int i;\n * if (dc.procid() == 0) {\n * i = 10;\n * // if I am machine 0, I send the value i = 10 to machine 1\n * dc.send_to(1, i);\n * } else if (dc.procid() == 1) {\n * // machine 1 receives the value of i from machine 0\n * dc.recv_from(0, i);\n * }\n * // at this point machines 0 and 1 have the value i = 10\n * \\endcode\n *\n * \\tparam U the type of object to send. This should be inferred by the\n * compiler.\n * \\param target The target machine to send to. Target machine must call\n * recv_from() before this call will return.\n * \\param t The object to send. It must be serializable. The type must\n * match the target machine's call to recv_from()\n * \\param control Optional parameter. Defaults to false. If set to true,\n * this will marked as control plane communication and will\n * not register in bytes_received() or bytes_sent(). This must\n * match the \"control\" parameter on the target machine's\n * recv_from() call.\n *\n * \\note Behavior is undefined if multiple threads on the same machine\n * call send_to simultaneously\n *\n */\n template <typename U>\n inline void send_to(procid_t target, U& t, bool control = false);\n\n /**\n * \\brief Waits to receives an object a source machine sent via send_to()\n *\n * This function waits to receives a \\ref sec_serializable object \"t\" from a\n * source machine. The source machine must send the object using\n * send_to(). The source machine will wait for the target machine's\n * recv_from() to complete before returning.\n *\n * Example:\n * \\code\n * int i;\n * if (dc.procid() == 0) {\n * i = 10;\n * // if I am machine 0, I send the value i = 10 to machine 1\n * dc.send_to(1, i);\n * } else if (dc.procid() == 1) {\n * // machine 1 receives the value of i from machine 0\n * dc.recv_from(0, i);\n * }\n * // at this point machines 0 and 1 have the value i = 10\n * \\endcode\n *\n * \\tparam U the type of object to receive. This should be inferred by the\n * compiler.\n * \\param source The target machine to receive from. This function will block\n * until data is received.\n * \\param t The object to receive. It must be serializable and the type\n * must match the source machine's call to send_to()\n * \\param control Optional parameter. Defaults to false. If set to true,\n * this will marked as control plane communication and will\n * not register in bytes_received() or bytes_sent(). This must\n * match the \"control\" parameter on the source machine's\n * send_to() call.\n *\n * \\note Behavior is undefined if multiple threads on the same machine\n * call recv_from simultaneously\n *\n */\n template <typename U>\n inline void recv_from(procid_t source, U& t, bool control = false);\n\n\n /**\n * \\brief This function allows one machine to broadcasts an object to all\n * machines.\n *\n * The originator calls broadcast with data provided in\n * in 'data' and originator set to true.\n * All other callers call with originator set to false.\n *\n * The originator will then return 'data'. All other machines\n * will receive the originator's transmission in the \"data\" parameter.\n *\n * This call is guaranteed to have barrier-like behavior. That is to say,\n * this call will block until all machines enter the broadcast function.\n *\n * Example:\n * \\code\n * int i;\n * if (procid() == 0) {\n * // if I am machine 0, I broadcast the value i = 10 to all machines\n * i = 10;\n * dc.broadcast(i, true);\n * } else {\n * // all other machines receive the broadcast value\n * dc.broadcast(i, false);\n * }\n * // at this point, all machines have i = 10\n * \\endcode\n *\n * \\note Behavior is undefined if more than one machine calls broadcast\n * with originator set to true.\n *\n * \\note Behavior is undefined if multiple threads on the same machine\n * call broadcast simultaneously\n *\n * \\param data If this is the originator, this will contain the object to\n * broadcast. Otherwise, this will be a reference to the object\n * receiving the broadcast.\n * \\param originator Set to true if this is the source of the broadcast.\n * Set to false otherwise.\n * \\param control Optional parameter. Defaults to false. If set to true,\n * this will marked as control plane communication and will\n * not register in bytes_received() or bytes_sent(). This must\n * be the same on all machines.\n */\n template <typename U>\n inline void broadcast(U& data, bool originator, bool control = false);\n\n /**\n * \\brief Collects information contributed by each machine onto\n * one machine.\n *\n * The goal is to collect some information from each machine onto a single\n * target machine (sendto). To accomplish this,\n * each machine constructs a vector of length numprocs(), and stores\n * the data to communicate in the procid()'th entry in the vector.\n * Then calling gather with the vector and the target machine will send\n * the contributed value to the target.\n * When the function returns, machine sendto will have the complete vector\n * where data[i] is the data contributed by machine i.\n *\n * Example:\n * \\code\n * // construct the vector of values\n * std::vector<int> values;\n * values.resize(dc.numprocs());\n *\n * // set my contributed value\n * values[dc.procid()] = dc.procid();\n * dc.gather(values, 0);\n * // at this point machine 0 will have a vector with length equal to the\n * // number of processes, and containing values [0, 1, 2, ...]\n * // All other machines value vector will be unchanged.\n * \\endcode\n *\n * \\note Behavior is undefined machines call gather with different values for\n * sendto\n *\n * \\note Behavior is undefined if multiple threads on the same machine\n * call gather simultaneously\n *\n * \\param data A vector of length equal to the number of processes. The\n * information to communicate is in the entry data[procid()]\n * \\param sendto Machine which will hold the complete vector at the end\n * of the operation. All machines must have the same value\n * for this parameter.\n * \\param control Optional parameter. Defaults to false. If set to true,\n * this will marked as control plane communication and will\n * not register in bytes_received() or bytes_sent(). This must\n * be the same on all machines.\n */\n template <typename U>\n inline void gather(std::vector<U>& data, procid_t sendto, bool control = false);\n\n /**\n * \\brief Sends some information contributed by each machine to all machines\n *\n * The goal is to have each machine broadcast a piece of information to all\n * machines. This is like gather(), but all machines have the complete vector\n * at the end. To accomplish this, each machine constructs a vector of\n * length numprocs(), and stores the data to communicate in the procid()'th\n * entry in the vector. Then calling all_gather with the vector will result\n * in all machines having a complete copy of the vector containing all\n * contributions (entry 0 from machine 0, entry 1 from machine 1, etc).\n *\n * Example:\n * \\code\n * // construct the vector of values\n * std::vector<int> values;\n * values.resize(dc.numprocs());\n *\n * // set my contributed value\n * values[dc.procid()] = dc.procid();\n * dc.all_gather(values);\n * // at this point all machine will have a vector with length equal to the\n * // number of processes, and containing values [0, 1, 2, ...]\n * \\endcode\n *\n * \\note Behavior is undefined if multiple threads on the same machine\n * call all_gather simultaneously\n *\n * \\param data A vector of length equal to the number of processes. The\n * information to communicate is in the entry data[procid()]\n * \\param control Optional parameter. Defaults to false. If set to true,\n * this will marked as control plane communication and will\n * not register in bytes_received() or bytes_sent(). This must\n * be the same on all machines.\n */\n template <typename U>\n inline void all_gather(std::vector<U>& data, bool control = false);\n\n\n /**\n * \\brief Combines a value contributed by each machine, making the result\n * available to all machines.\n *\n * Each machine calls all_reduce() with a object which is serializable\n * and has operator+= implemented. When all_reduce() returns, the \"data\"\n * variable will contain a value corresponding to adding up the objects\n * contributed by each machine.\n *\n * Example:\n * \\code\n * int i = 1;\n * dc.all_reduce(i);\n * // since each machine contributed the value \"1\",\n * // all machines will have i = numprocs() here.\n * \\endcode\n *\n * \\param data A piece of data to perform a reduction over.\n * The type must implement operator+=.\n * \\param control Optional parameter. Defaults to false. If set to true,\n * this will marked as control plane communication and will\n * not register in bytes_received() or bytes_sent(). This must\n * be the same on all machines.\n */\n template <typename U>\n inline void all_reduce(U& data, bool control = false);\n\n /**\n * \\brief Combines a value contributed by each machine, making the result\n * available to all machines.\n *\n * This function is equivalent to all_reduce(), but with an externally\n * defined PlusEqual function.\n *\n * Each machine calls all_reduce() with a object which is serializable\n * and a function \"plusequal\" which combines two instances of the object.\n * When all_reduce2() returns, the \"data\"\n * variable will contain a value corresponding to adding up the objects\n * contributed by each machine using the plusequal function.\n *\n * Where U is the type of the object, the plusequal function must be of\n * the form:\n * \\code\n * void plusequal(U& left, const U& right);\n * \\endcode\n * and must implement the equivalent of <code>left += right; </code>\n *\n * Example:\n * \\code\n * void int_plus_equal(int& a, const int& b) {\n * a+=b;\n * }\n *\n * int i = 1;\n * dc.all_reduce2(i, int_plus_equal);\n * // since each machine contributed the value \"1\",\n * // all machines will have i = numprocs() here.\n * \\endcode\n *\n * \\param data A piece of data to perform a reduction over.\n * \\param plusequal A plusequal function on the data. Must have the prototype\n * void plusequal(U&, const U&)\n * \\param control Optional parameter. Defaults to false. If set to true,\n * this will marked as control plane communication and will\n * not register in bytes_received() or bytes_sent(). This must\n * be the same on all machines.\n */\n template <typename U, typename PlusEqual>\n inline void all_reduce2(U& data, PlusEqual plusequal, bool control = false);\n\n\n /**\n \\brief A distributed barrier which waits for all machines to call the\n barrier() function before proceeding.\n\n A machine calling the barrier() will wait until every machine\n reaches this barrier before continuing. Only one thread from each machine\n should call the barrier.\n\n \\see full_barrier\n */\n void barrier();\n\n\n\n\n /*****************************************************************************\n Implementation of Full Barrier\n*****************************************************************************/\n /**\n * \\brief A distributed barrier which waits for all machines to call\n * the full_barrier() function before proceeding. Also waits for all\n * previously issued remote calls to complete.\n\n Similar to the barrier(), but provides additional guarantees that\n all calls issued prior to this barrier are completed before\n i returning.\n\n \\note This function could return prematurely if\n other threads are still issuing function calls since we\n cannot differentiate between calls issued before the barrier\n and calls issued while the barrier is being evaluated.\n Therefore, when used in a multithreaded scenario, the user must ensure\n that all other threads which may perform operations using this object\n are stopped before the full barrier is initated.\n\n \\see barrier\n */\n void full_barrier();\n\n\n /**\n * \\brief A wrapper on cout, that outputs only on machine 0\n */\n std::ostream& cout() const {\n if (procid() == 0) return std::cout;\n else return nullstrm;\n }\n\n /**\n * \\brief A wrapper on cerr, that outputs only on machine 0\n */\n std::ostream& cerr() const {\n if (procid() == 0) return std::cerr;\n else return nullstrm;\n }\n\n private:\n mutex full_barrier_lock;\n fiber_conditional full_barrier_cond;\n std::vector<size_t> calls_to_receive;\n // used to inform the counter that the full barrier\n // is in effect and all modifications to the calls_recv\n // counter will need to lock and signal\n volatile bool full_barrier_in_effect;\n\n /** number of 'source' processor counts which have\n not achieved the right recv count */\n atomic<size_t> num_proc_recvs_incomplete;\n\n /// Marked as 1 if the proc is complete\n dense_bitset procs_complete;\n ///\\internal\n mutable boost::iostreams::stream<boost::iostreams::null_sink> nullstrm;\n\n /*****************************************************************************\n Collection of Statistics\n*****************************************************************************/\n\n private:\n struct collected_statistics {\n size_t callssent;\n size_t bytessent;\n size_t network_bytessent;\n collected_statistics(): callssent(0), bytessent(0), network_bytessent(0) { }\n void save(oarchive &oarc) const {\n oarc << callssent << bytessent << network_bytessent;\n }\n void load(iarchive &iarc) {\n iarc >> callssent >> bytessent >> network_bytessent;\n }\n };\n public:\n /** Gather RPC statistics. All machines must call\n this function at the same time. However, only proc 0 will\n return values */\n std::map<std::string, size_t> gather_statistics();\n};\n\n\n\n\n} // namespace graphlab\n\n#define REGISTER_RPC(dc, f) dc.register_rpc<typeof(f)*, f>(std::string(BOOST_PP_STRINGIZE(f)))\n\n#include <graphlab/rpc/function_arg_types_undef.hpp>\n#include <graphlab/rpc/function_call_dispatch.hpp>\n#include <graphlab/rpc/request_dispatch.hpp>\n#include <graphlab/rpc/dc_dist_object.hpp>\n#include <graphlab/rpc/dc_services.hpp>\n\nnamespace graphlab {\n\ntemplate <typename U>\ninline void distributed_control::send_to(procid_t target, U& t, bool control) {\n distributed_services->send_to(target, t, control);\n}\n\ntemplate <typename U>\ninline void distributed_control::recv_from(procid_t source, U& t, bool control) {\n distributed_services->recv_from(source, t, control);\n}\n\ntemplate <typename U>\ninline void distributed_control::broadcast(U& data, bool originator, bool control) {\n distributed_services->broadcast(data, originator, control);\n}\n\ntemplate <typename U>\ninline void distributed_control::gather(std::vector<U>& data, procid_t sendto, bool control) {\n distributed_services->gather(data, sendto, control);\n}\n\ntemplate <typename U>\ninline void distributed_control::all_gather(std::vector<U>& data, bool control) {\n distributed_services->all_gather(data, control);\n}\n\ntemplate <typename U>\ninline void distributed_control::all_reduce(U& data, bool control) {\n distributed_services->all_reduce(data, control);\n}\n\n\ntemplate <typename U, typename PlusEqual>\ninline void distributed_control::all_reduce2(U& data, PlusEqual plusequal, bool control) {\n distributed_services->all_reduce2(data, plusequal, control);\n}\n\n\n\n\n}\n\n#include <graphlab/util/mpi_tools.hpp>\n#endif\n\n" }, { "path": "src/graphlab/rpc/dc_buffered_stream_send2.cpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <iostream>\n#include <boost/iostreams/stream.hpp>\n\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_buffered_stream_send2.hpp>\n#include <graphlab/util/branch_hints.hpp>\nnamespace graphlab {\nnamespace dc_impl {\n\n void dc_buffered_stream_send2::flush() {\n comm->trigger_send_timeout(target, true);\n }\n\n void dc_buffered_stream_send2::flush_soon() {\n comm->trigger_send_timeout(target, false);\n }\n\n\n\n inline size_t dc_buffered_stream_send2::bytes_sent() {\n size_t ret = total_bytes_sent;\n lock.lock();\n for (size_t i = 0;i < send_buffers.size(); ++i) {\n ret += send_buffers[i]->get_bytes_sent(target); \n }\n lock.unlock();\n return ret;\n }\n\n void dc_buffered_stream_send2::write_to_buffer(char* c, size_t len) {\n lock.lock();\n additional_flush_buffers.push_back(std::make_pair(c, len));\n lock.unlock();\n }\n\n void dc_buffered_stream_send2::register_send_buffer(thread_local_buffer* buffer) {\n lock.lock();\n send_buffers.push_back(buffer);\n to_send.resize(send_buffers.size());\n lock.unlock();\n }\n\n void dc_buffered_stream_send2::unregister_send_buffer(thread_local_buffer* buffer) {\n lock.lock();\n for (size_t i = 0;i < send_buffers.size(); ++i) {\n if (send_buffers[i] == buffer) {\n total_bytes_sent.inc(send_buffers[i]->get_bytes_sent(target));\n send_buffers.erase(send_buffers.begin() + i);\n break;\n }\n }\n to_send.resize(send_buffers.size());\n lock.unlock();\n }\n\n dc_buffered_stream_send2::~dc_buffered_stream_send2() {\n // unregister all the buffers.\n std::vector<thread_local_buffer*> all_buffers;\n for (size_t i = 0; i < all_buffers.size(); ++i) {\n unregister_send_buffer(all_buffers[i]);\n }\n }\n\n size_t dc_buffered_stream_send2::get_outgoing_data(circular_iovec_buffer& outdata) {\n lock.lock();\n size_t sendlen = 0;\n for (size_t i = 0;i < send_buffers.size(); ++i) {\n std::pair<buffer_elem*, buffer_elem*> bufs = send_buffers[i]->extract(target);\n if (bufs.first != NULL) {\n while(bufs.first != bufs.second) {\n buffer_elem* prev = bufs.first;\n iovec sendvec;\n sendvec.iov_base = bufs.first->buf;\n sendvec.iov_len = bufs.first->len;\n sendlen += sendvec.iov_len;\n outdata.write(sendvec);\n buffer_elem** next = &bufs.first->next;\n volatile buffer_elem** n = (volatile buffer_elem**)(next);\n while(__unlikely__((*n) == NULL)) {\n asm volatile(\"pause\\n\": : :\"memory\");\n }\n bufs.first = (buffer_elem*)(*n);\n delete prev;\n }\n }\n }\n for (size_t i = 0;i < additional_flush_buffers.size(); ++i) {\n iovec sendvec;\n sendvec.iov_base = additional_flush_buffers[i].first;\n sendvec.iov_len = additional_flush_buffers[i].second;\n sendlen += sendvec.iov_len;\n outdata.write(sendvec);\n }\n lock.unlock();\n return sendlen;\n }\n} // namespace dc_impl\n} // namespace graphlab\n\n\n" }, { "path": "src/graphlab/rpc/dc_buffered_stream_send2.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef DC_BUFFERED_STREAM_SEND2_HPP\n#define DC_BUFFERED_STREAM_SEND2_HPP\n#include <iostream>\n#include <boost/function.hpp>\n#include <boost/bind.hpp>\n#include <boost/type_traits/is_base_of.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/thread_local_send_buffer.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_comm_base.hpp>\n#include <graphlab/rpc/dc_send.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/util/inplace_lf_queue.hpp>\n#include <graphlab/logger/logger.hpp>\nnamespace graphlab {\nclass distributed_control;\n\nnamespace dc_impl {\n\n\n/**\n * \\internal\n \\ingroup rpc\nSender for the dc class.\n The job of the sender is to take as input data blocks of\n pieces which should be sent to a single destination socket.\n This can be thought of as a sending end of a multiplexor.\n This class performs buffered transmissions using an blocking\n queue with one call per queue entry.\n A seperate thread is used to transmit queue entries. Rudimentary\n write combining is used to decrease transmission overhead.\n This is typically the best performing sender.\n\n This can be enabled by passing \"buffered_queued_send=yes\"\n in the distributed control initstring.\n\n dc_buffered_stream_send22 is similar, but does not perform write combining.\n\n*/\n\nclass dc_buffered_stream_send2: public dc_send{\n public:\n dc_buffered_stream_send2(distributed_control* dc,\n dc_comm_base *comm,\n procid_t target) :\n dc(dc), comm(comm), target(target) { }\n\n ~dc_buffered_stream_send2();\n\n void register_send_buffer(thread_local_buffer* buffer);\n\n void unregister_send_buffer(thread_local_buffer* buffer);\n\n size_t get_outgoing_data(circular_iovec_buffer& outdata);\n\n inline size_t bytes_sent();\n\n void write_to_buffer(char* c, size_t len);\n\n void flush();\n\n void flush_soon();\n\n private:\n /// pointer to the owner\n distributed_control* dc;\n dc_comm_base *comm;\n procid_t target;\n atomic<size_t> total_bytes_sent;\n\n\n\n std::vector<thread_local_buffer*> send_buffers;\n // temporary array matched to the same length as send_buffers\n // to avoid repeated reallocation of this array when \n // get_outgoing_data is called\n std::vector<std::vector<std::pair<char*, size_t> > > to_send;\n\n std::vector<std::pair<char*, size_t> > additional_flush_buffers;\n mutex lock;\n};\n\n\n\n} // namespace dc_impl\n} // namespace graphlab\n#endif // DC_BUFFERED_STREAM_SEND_EXPQUEUE_HPP\n\n" }, { "path": "src/graphlab/rpc/dc_comm_base.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef DC_COMM_BASE_HPP\n#define DC_COMM_BASE_HPP\n#include <sys/socket.h>\n#include <vector>\n#include <string>\n#include <map>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/dc_receive.hpp>\n#include <graphlab/rpc/dc_send.hpp>\nnamespace graphlab {\nnamespace dc_impl { \n\n \n/**\n * \\ingroup rpc\n * \\internal\nThe base class of all comms implementations\n*/\nclass dc_comm_base {\n public:\n \n inline dc_comm_base() { };\n \n virtual size_t capabilities() const = 0;\n /**\n Parses initialization parameters. Most of these parameters are\n user provided, or provided on a higher level initialization system.\n It is entirely up to the comm implementation how these parameters to be treated.\n The descriptions here are largely prescriptive.\n All machines are called with the same initialization parameters (of course with the \n exception of curmachineid)\n\n The expected behavior is that \n this fuction should pause until all communication has been set up\n and returns the number of systems in the network.\n After which, all other remaining public functions (numprocs(), send(), etc)\n should operate normally. Every received message should immediate trigger the \n attached receiver\n \n machines: a vector of string over machine IDs. This is typically provided by the user\n or through some other initialization mechanism\n initstring: Additional parameters passed by the user\n curmachineid: The ID of the current machine. Will be size_t(-1) if this is not available.\n (Some comm protocols will negotiate this itself.)\n \n receiver: the receiving object\n */\n virtual void init(const std::vector<std::string> &machines,\n const std::map<std::string,std::string> &initopts,\n procid_t curmachineid,\n std::vector<dc_receive*> receiver,\n std::vector<dc_send*> sender) = 0;\n\n /// Must close all connections when this function is called\n virtual void close() = 0;\n \n virtual void trigger_send_timeout(procid_t target, bool urgent) = 0;\n \n virtual ~dc_comm_base() {}\n virtual procid_t numprocs() const = 0;\n \n virtual procid_t procid() const = 0;\n \n virtual size_t network_bytes_sent() const = 0;\n virtual size_t network_bytes_received() const = 0;\n virtual size_t send_queue_length() const = 0;\n\n};\n\n} // namespace dc_impl\n} // namespace graphlab\n#endif\n\n" }, { "path": "src/graphlab/rpc/dc_compile_parameters.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_DC_COMPILE_PARAMETERS_HPP\n#define GRAPHLAB_DC_COMPILE_PARAMETERS_HPP\n\n// do not change\n/**\n \\ingroup rpc\n \\def RPC_DEFAULT_COMMTYPE\n \\brief default communication method\n */\n#define RPC_DEFAULT_COMMTYPE TCP_COMM\n\n/**\n \\ingroup rpc\n \\def RPC_MAX_N_PROCS\n \\brief Maximum number of processes supported\n */\n#define RPC_MAX_N_PROCS 128\n\n/**\n * \\ingroup RPC\n * \\def RECEIVE_BUFFER_SIZE\n * The size of the receive buffer for each socket\n */\n#define RECEIVE_BUFFER_SIZE 131072\n\n/**************************************************************************/\n/* */\n/* Send Buffer Behavior Control */\n/* */\n/**************************************************************************/\n\n/*\n * The architecture of the sending subsystem is that there is 1 main send thread.\n * Which polls a collection of thread local queues.\n *\n * Each thread local send queues comprises of 1 queue for each target machine.\n * Each queue comprises of 2 parts:\n * - An array of \"full\" buffers\n * - One not-full buffer.\n */ \n\n/**\n * \\ingroup RPC\n * \\def SEND_POLL_TIMEOUT\n * The TCP sender polls the queues every so often to ensure\n * progress; This is the timeout value for the number of microseconds\n * between each poll.\n */\n#define SEND_POLL_TIMEOUT 10000\n\n\n/**\n * \\ingroup rpc\n * \\def INITIAL_BUFFER_SIZE\n * Each buffer is allocated to this size at the start\n */\n#define INITIAL_BUFFER_SIZE 65536\n\n\n/**\n * \\ingroup rpc\n * \\def FULL_BUFFER_SIZE_LIMIT\n * Once the buffer contents exceeds this, it becomes a full buffer.\n */\n#define FULL_BUFFER_SIZE_LIMIT 63000\n\n/**\n * \\ingroup RPC\n * \\def NUM_FULL_BUFFER_LIMIT \n * Number of full buffers in the send queue before a flush is explicitly called.\n */\n#define NUM_FULL_BUFFER_LIMIT 32 \n\n/**************************************************************************/\n/* */\n/* RPC Handling Control */\n/* */\n/**************************************************************************/\n\n/**\n \\ingroup rpc\n \\def RPC_DEFAULT_NUMHANDLERTHREADS\n \\brief default number of handler threads to spawn.\n */\n#define RPC_DEFAULT_NUMHANDLERTHREADS (size_t)(-1)\n\n/**\n * \\ingroup RPC\n * \\def RPC_DO_NOT_BREAK_BLOCKS\n *\n * If this option is turned on,\n * collections of messages recieved in a buffer \n * will all be executed by the same thread. \n * This decreases latency and increases throughput\n * but at a cost of parallelism.\n * Also, if turned on together with RPC_BLOCK_STRIPING,\n * the sequentialization key is ignored.\n */\n#define RPC_DO_NOT_BREAK_BLOCKS\n\n\n/**\n * \\ingroup RPC\n * \\def RPC_BLOCK_STRIPING\n * Incoming buffers are striped across threads \n * to be processed. If this is turned on together with\n * RPC_DO_NOT_BREAK_BLOCKS, the sequentialization key is\n * ignored.\n */\n#define RPC_BLOCK_STRIPING\n\n/**************************************************************************/\n/* */\n/* Miscellaneous */\n/* */\n/**************************************************************************/\n/**\n * \\ingroup RPC\n * \\def DEFAULT_BUFFERED_EXCHANGE_SIZE\n * maximum size of each buffer in the buffer exchange. Beyond this size,\n * a send is performed.\n */\n#define DEFAULT_BUFFERED_EXCHANGE_SIZE FULL_BUFFER_SIZE_LIMIT\n\n\n#endif\n" }, { "path": "src/graphlab/rpc/dc_dist_object.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <graphlab/rpc/dc.hpp>\n\n#ifndef GRAPHLAB_DC_DIST_OBJECT_HPP\n#define GRAPHLAB_DC_DIST_OBJECT_HPP\n#include <vector>\n#include <string>\n#include <set>\n#include <graphlab/parallel/atomic.hpp>\n#include <graphlab/parallel/fiber_conditional.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/dc_dist_object_base.hpp>\n#include <graphlab/rpc/object_request_issue.hpp>\n#include <graphlab/rpc/object_call_issue.hpp>\n#include <graphlab/rpc/object_broadcast_issue.hpp>\n#include <graphlab/rpc/function_ret_type.hpp>\n#include <graphlab/rpc/mem_function_arg_types_def.hpp>\n#include <graphlab/util/charstream.hpp>\n#include <boost/preprocessor.hpp>\n#include <graphlab/util/tracepoint.hpp>\n#include <graphlab/rpc/request_reply_handler.hpp>\n#include <graphlab/macros_def.hpp>\n\n#define BARRIER_BRANCH_FACTOR 128\n\n\nnamespace graphlab {\n\n\n/**\n\\ingroup rpc\n\\brief Provides a class with its own distributed communication context, allowing\ninstances of the class to communicate with other remote instances.\n\nThe philosophy behind the dc_dist_object is the concept of \"distributed\nobjects\". The idea is that the user should be able to write code:\n\n\\code\nvoid main() {\n // ... initialization of a distributed_control object dc ...\n\n distributed_vector vec(dc), vec2(dc);\n distributed_graph g(dc);\n}\n\\endcode\nwhere if run in a distributed setting, the \"vec\" variable, can behave as if it\nis a single distributed object, and automatically coordinate its operations\nacross the network; communicating with the other instances of \"vec\" on the\nother machines. Essentially, each object (vec, vec2 and g) constructs its own\nprivate communication context, which allows every machine's \"vec\" variable to\ncommunicate only with other machine's \"vec\" variable. And similarly for \"vec2\"\nand \"g\". This private communication context is provided by this dc_dist_object\nclass.\n\nTo construct a distributed object requires little work:\n\\code\nclass distributed_int_vector {\n private:\n // creates a local dc_dist_object context\n graphlab::dc_dist_object<distributed_int_vector> rmi;\n\n public:\n // context must be initialized on construction with the\n // root distributed_control object\n distributed_int_vector(distributed_control& dc): rmi(dc, this) {\n ... other initialization ...\n // make sure all machines finish constructing this object\n // before continuing\n rmi.barrier();\n }\n};\n\\endcode\n\nAfter which remote_call(), and remote_request() can be used to communicate\nacross the network with the same matching instance of the\ndistributed_int_vector.\n\nEach dc_dist_object maintains its own private communication context which\nis not influences by other communication contexts. In other words, the\n<code>rmi.barrier()</code>, and all other operations in each instance of the\ndistributed_int_vector are independent of each other. In particular, the\n<code>rmi.full_barrier()</code> only waits for completion of all RPC calls\nfrom within the current communication context.\n\nSee the examples in \\ref RPC for more usage examples.\n\n\\note While there is no real limit to the number of distributed\nobjects that can be created. However, each dc_dist_object does contain\na reasonably large amount of state, so frequent construction and deletion\nof objects is not recommended.\n*/\ntemplate <typename T>\nclass dc_dist_object : public dc_impl::dc_dist_object_base{\n private:\n distributed_control &dc_;\n size_t obj_id;\n size_t control_obj_id; // object id of this object\n T* owner;\n std::vector<atomic<size_t> > callsreceived;\n std::vector<atomic<size_t> > callssent;\n std::vector<atomic<size_t> > bytessent;\n // make operator= private\n dc_dist_object<T>& operator=(const dc_dist_object<T> &d) {return *this;}\n friend class distributed_control;\n\n\n DECLARE_TRACER(distobj_remote_call_time);\n\n\n public:\n\n /// \\cond GRAPHLAB_INTERNAL\n\n /// Should not be used by the user\n void inc_calls_received(procid_t p) {\n if (!full_barrier_in_effect) {\n size_t t = callsreceived[p].inc();\n if (full_barrier_in_effect) {\n if (t == calls_to_receive[p]) {\n // if it was me who set the bit\n if (procs_complete.set_bit(p) == false) {\n // then decrement the incomplete count.\n // if it was me to decreased it to 0\n // lock and signal\n full_barrier_lock.lock();\n if (num_proc_recvs_incomplete.dec() == 0) {\n full_barrier_cond.signal();\n }\n full_barrier_lock.unlock();\n }\n }\n }\n }\n else {\n //check the proc I just incremented.\n // If I just exceeded the required size, I need\n // to decrement the full barrier counter\n if (callsreceived[p].inc() == calls_to_receive[p]) {\n // if it was me who set the bit\n if (procs_complete.set_bit(p) == false) {\n // then decrement the incomplete count.\n // if it was me to decreased it to 0\n // lock and signal\n full_barrier_lock.lock();\n if (num_proc_recvs_incomplete.dec() == 0) {\n full_barrier_cond.signal();\n }\n full_barrier_lock.unlock();\n }\n }\n }\n }\n\n /// Should not be used by the user\n void inc_calls_sent(procid_t p) {\n callssent[p].inc();\n }\n\n /// Should not be used by the user\n void inc_bytes_sent(procid_t p, size_t bytes) {\n bytessent[p].inc(bytes);\n }\n\n /// Should not be used by the user\n size_t get_obj_id() const {\n return obj_id;\n }\n /// \\endcond GRAPHLAB_INTERNAL\n public:\n\n /**\n * \\brief Constructs a distributed object context.\n *\n * The constructor constructs a distributed object context which is\n * associated with the \"owner\" object.\n *\n * \\param dc_ The root distributed_control which provides the\n * communication control plane.\n * \\param owner The object to associate with\n */\n dc_dist_object(distributed_control &dc_, T* owner):\n dc_(dc_),owner(owner) {\n callssent.resize(dc_.numprocs());\n callsreceived.resize(dc_.numprocs());\n bytessent.resize(dc_.numprocs());\n //------ Initialize the matched send/recv ------\n recv_froms.resize(dc_.numprocs());\n //------ Initialize the gatherer ------\n gather_receive.resize(dc_.numprocs());\n\n\n //------- Initialize the Barrier ----------\n child_barrier_counter.value = 0;\n barrier_sense = 1;\n barrier_release = -1;\n\n\n // compute my children\n childbase = size_t(dc_.procid()) * BARRIER_BRANCH_FACTOR + 1;\n if (childbase >= dc_.numprocs()) {\n numchild = 0;\n }\n else {\n size_t maxchild = std::min<size_t>(dc_.numprocs(),\n childbase + BARRIER_BRANCH_FACTOR);\n numchild = (procid_t)(maxchild - childbase);\n }\n\n parent = (procid_t)((dc_.procid() - 1) / BARRIER_BRANCH_FACTOR) ;\n\n //-------- Initialize all gather --------------\n ab_child_barrier_counter.value = 0;\n ab_barrier_sense = 1;\n ab_barrier_release = -1;\n\n\n //-------- Initialize the full barrier ---------\n\n full_barrier_in_effect = false;\n procs_complete.resize(dc_.numprocs());\n\n // register\n obj_id = dc_.register_object(owner, this);\n control_obj_id = dc_.register_object(this, this);\n\n //-------- Initialize Tracer\n std::string name = typeid(T).name();\n INITIALIZE_TRACER(distobj_remote_call_time,\n std::string(\"dc_dist_object \") + name + \": remote_call time\");\n }\n\n /// \\brief The number of function calls received by this object\n size_t calls_received() const {\n size_t ctr = 0;\n for (size_t i = 0;i < numprocs(); ++i) {\n ctr += callsreceived[i].value;\n }\n return ctr;\n }\n\n /// \\brief The number of function calls sent from this object\n size_t calls_sent() const {\n size_t ctr = 0;\n for (size_t i = 0;i < numprocs(); ++i) {\n ctr += callssent[i].value;\n }\n return ctr;\n }\n\n /** \\brief The number of bytes sent from this object, excluding\n * headers and other control overhead.\n */\n size_t bytes_sent() const {\n size_t ctr = 0;\n for (size_t i = 0;i < numprocs(); ++i) {\n ctr += bytessent[i].value;\n }\n return ctr;\n }\n\n /// \\brief A reference to the underlying distributed_control object\n distributed_control& dc() {\n return dc_;\n }\n\n /// \\brief A const reference to the underlying distributed_control object\n const distributed_control& dc() const {\n return dc_;\n }\n\n /// \\brief The current process ID\n inline procid_t procid() const {\n return dc_.procid();\n }\n\n /// \\brief The number of processes in the distributed program.\n inline procid_t numprocs() const {\n return dc_.numprocs();\n }\n\n /**\n * \\brief A wrapper on cout, that outputs only on machine 0\n */\n std::ostream& cout() const {\n return dc_.cout();\n }\n\n /**\n * \\brief A wrapper on cerr, that outputs only on machine 0\n */\n std::ostream& cerr() const {\n return dc_.cout();\n }\n\n /// \\cond GRAPHLAB_INTERNAL\n\n /*\n This generates the interface functions for the standard calls, basic calls\n The function looks like this:\n \\code\n template<typename F , typename T0> void remote_call (procid_t target, F remote_function , T0 i0 )\n {\n ASSERT_LT(target, dc_.senders.size());\n if ((STANDARD_CALL & CONTROL_PACKET) == 0) inc_calls_sent(target);\n dc_impl::object_call_issue1 <T, F , T0> ::exec(dc_.senders[target],\n STANDARD_CALL,\n target,obj_id,\n remote_function ,\n i0 );\n }\n\n The argument to the RPC_INTERFACE_GENERATOR are:\n - the name of the rpc call (\"remote_call\" in the first one)\n - the name of the issueing processor (\"object_call_issue\")\n - The flags to set on the call (\"STANDARD_CALL\")\n\n The call can be issued with\n rmi.remote_call(target,\n &object_type::function_name,\n arg1,\n arg2...)\n \\endcode\n */\n #define GENARGS(Z,N,_) BOOST_PP_CAT(T, N) BOOST_PP_CAT(i, N)\n #define GENI(Z,N,_) BOOST_PP_CAT(i, N)\n #define GENT(Z,N,_) BOOST_PP_CAT(T, N)\n #define GENARC(Z,N,_) arc << BOOST_PP_CAT(i, N);\n\n #define RPC_INTERFACE_GENERATOR(Z,N,FNAME_AND_CALL) \\\n template<typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\n void BOOST_PP_TUPLE_ELEM(3,0,FNAME_AND_CALL) (procid_t target, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n ASSERT_LT(target, dc_.senders.size()); \\\n BEGIN_TRACEPOINT(distobj_remote_call_time); \\\n if ((BOOST_PP_TUPLE_ELEM(3,2,FNAME_AND_CALL) & CONTROL_PACKET) == 0) inc_calls_sent(target); \\\n BOOST_PP_CAT( BOOST_PP_TUPLE_ELEM(3,1,FNAME_AND_CALL),N) \\\n <T, F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, T)> \\\n ::exec(this, dc_.senders[target], BOOST_PP_TUPLE_ELEM(3,2,FNAME_AND_CALL), target,obj_id, remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENI ,_) ); \\\n END_TRACEPOINT(distobj_remote_call_time); \\\n } \\\n\n /*\n Generates the interface functions. 3rd argument is a tuple (interface name, issue name, flags)\n */\n BOOST_PP_REPEAT(7, RPC_INTERFACE_GENERATOR, (remote_call, dc_impl::object_call_issue, STANDARD_CALL) )\n BOOST_PP_REPEAT(7, RPC_INTERFACE_GENERATOR, (control_call,dc_impl::object_call_issue, (STANDARD_CALL | CONTROL_PACKET)) )\n\n /**\n * This generates a \"split call\". Where the header of the call message\n * is written to with split_call_begin, and the message actually sent with\n * split_call_end(). It is then up to the user to serialize the message arguments\n * into the oarchive returned. The split call can provide performance gains \n * when the contents of the message are large, since this allows the user to\n * control the serialization process. \n *\n * Example:\n * \\code\n * struct mystruct {\n * void function_to_call(size_t len, wild_pointer w) { \n * // w will contain all the serialized contents of ..stuff...\n * }\n *\n * void stuff() {\n * oarchive* oarc = rmi.split_call_begin(&mystruct::function_to_call);\n * (*oarc) << ... stuff...\n * rmi.split_call_end(1, // to machine 1\n * oarc);\n * \n * }\n * }\n * \\endcode\n */\n oarchive* split_call_begin(void (T::*remote_function)(size_t, wild_pointer)) {\n return dc_impl::object_split_call<T, void(T::*)(size_t, wild_pointer)>::split_call_begin(this, obj_id, remote_function);\n }\n\n /**\n * Sends a split call started by \\ref split_call_begin\n * See \\ref split_call_begin for details.\n */\n void split_call_end(procid_t target, oarchive* oarc) {\n inc_calls_sent(target);\n return dc_impl::object_split_call<T, void(T::*)(size_t, wild_pointer)>::split_call_end(this, oarc, dc_.senders[target],\n target, STANDARD_CALL);\n }\n\n /**\n * Cancels a split call began with split_call_begin\n */\n void split_call_cancel(oarchive* oarc) {\n return dc_impl::object_split_call<T, void(T::*)(size_t, wild_pointer)>::split_call_cancel(oarc);\n }\n\n\n\n #define BROADCAST_INTERFACE_GENERATOR(Z,N,FNAME_AND_CALL) \\\n template<typename Iterator, typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\n void BOOST_PP_TUPLE_ELEM(3,0,FNAME_AND_CALL) (Iterator target_begin, Iterator target_end, \\\n F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n if (target_begin == target_end) return; \\\n BEGIN_TRACEPOINT(distobj_remote_call_time); \\\n if ((BOOST_PP_TUPLE_ELEM(3,2,FNAME_AND_CALL) & CONTROL_PACKET) == 0) { \\\n Iterator iter = target_begin; \\\n while (iter != target_end){ \\\n inc_calls_sent(*iter); \\\n ++iter; \\\n } \\\n } \\\n BOOST_PP_CAT( BOOST_PP_TUPLE_ELEM(3,1,FNAME_AND_CALL),N) \\\n <Iterator, T, F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, T)> \\\n ::exec(this, dc_.senders, BOOST_PP_TUPLE_ELEM(3,2,FNAME_AND_CALL), target_begin, target_end,obj_id, remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENI ,_) ); \\\n END_TRACEPOINT(distobj_remote_call_time); \\\n }\n\n BOOST_PP_REPEAT(7, BROADCAST_INTERFACE_GENERATOR, (remote_call, dc_impl::object_broadcast_issue, STANDARD_CALL) )\n\n /*\n The generation procedure for requests are the same. The only\n difference is that the function name has to be changed a little to\n be identify the return type of the function, (typename\n dc_impl::function_ret_type<__GLRPC_FRESULT>) and the issuing\n processor is object_request_issue.\n\n The call can be issued with\n \\code\n ret = rmi.remote_request(target,\n &object_type::function_name,\n arg1,\n arg2...)\n \\endcode\n */\n#define CUSTOM_REQUEST_INTERFACE_GENERATOR(Z,N,ARGS) \\\n template<typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\n BOOST_PP_TUPLE_ELEM(2,0,ARGS) (procid_t target, size_t handle, unsigned char flags, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n ASSERT_LT(target, dc_.senders.size()); \\\n if ((flags & CONTROL_PACKET) == 0) inc_calls_sent(target); \\\n BOOST_PP_CAT( BOOST_PP_TUPLE_ELEM(2,1,ARGS),N) \\\n <T, F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, T)> \\\n ::exec(this, dc_.senders[target], handle, flags, target,obj_id, remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENI ,_) ); \\\n }\n\n\n#define FUTURE_REQUEST_INTERFACE_GENERATOR(Z,N,ARGS) \\\n template<typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\n BOOST_PP_TUPLE_ELEM(2,0,ARGS) (procid_t target, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n ASSERT_LT(target, dc_.senders.size()); \\\n request_future<__GLRPC_FRESULT> reply; \\\n custom_remote_request(target, reply.get_handle(), BOOST_PP_TUPLE_ELEM(2,1,ARGS), remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENI ,_) ); \\\n return reply; \\\n } \n\n #define REQUEST_INTERFACE_GENERATOR(Z,N,ARGS) \\\n template<typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\n BOOST_PP_TUPLE_ELEM(2,0,ARGS) (procid_t target, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n ASSERT_LT(target, dc_.senders.size()); \\\n request_future<__GLRPC_FRESULT> reply; \\\n custom_remote_request(target, reply.get_handle(),BOOST_PP_TUPLE_ELEM(2,1,ARGS), remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENI ,_) ); \\\n return reply(); \\\n }\n\n\n /*\n Generates the interface functions. 3rd argument is a tuple\n (interface name, issue name, flags)\n */\n BOOST_PP_REPEAT(6, CUSTOM_REQUEST_INTERFACE_GENERATOR, (void custom_remote_request, dc_impl::object_request_issue) )\n BOOST_PP_REPEAT(6, REQUEST_INTERFACE_GENERATOR, (typename dc_impl::function_ret_type<__GLRPC_FRESULT>::type remote_request, (STANDARD_CALL | FLUSH_PACKET)) )\n BOOST_PP_REPEAT(6, FUTURE_REQUEST_INTERFACE_GENERATOR, (request_future<__GLRPC_FRESULT> future_remote_request, (STANDARD_CALL)) )\n\n\n\n #undef RPC_INTERFACE_GENERATOR\n #undef BROADCAST_INTERFACE_GENERATOR\n #undef REQUEST_INTERFACE_GENERATOR\n #undef CUSTOM_REQUEST_INTERFACE_GENERATOR\n #undef FUTURE_REQUEST_INTERFACE_GENERATOR\n /* Now generate the interface functions which allow me to call this\n dc_dist_object directly The internal calls are similar to the ones\n above. The only difference is that is that instead of 'obj_id', the\n parameter passed to the issue processor is \"control_obj_id\" which\n identifies the current RMI class.\n */\n #define RPC_INTERFACE_GENERATOR(Z,N,FNAME_AND_CALL) \\\n template<typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\n void BOOST_PP_TUPLE_ELEM(3,0,FNAME_AND_CALL) (procid_t target, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n ASSERT_LT(target, dc_.senders.size()); \\\n if ((BOOST_PP_TUPLE_ELEM(3,2,FNAME_AND_CALL) & CONTROL_PACKET) == 0) inc_calls_sent(target); \\\n BOOST_PP_CAT( BOOST_PP_TUPLE_ELEM(3,1,FNAME_AND_CALL),N) \\\n <dc_dist_object<T>, F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, T)> \\\n ::exec(this, dc_.senders[target], BOOST_PP_TUPLE_ELEM(3,2,FNAME_AND_CALL), target,control_obj_id, remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENI ,_) ); \\\n } \\\n\n BOOST_PP_REPEAT(6, RPC_INTERFACE_GENERATOR, (internal_call,dc_impl::object_call_issue, STANDARD_CALL) )\n BOOST_PP_REPEAT(6, RPC_INTERFACE_GENERATOR, (internal_control_call,dc_impl::object_call_issue, (STANDARD_CALL | CONTROL_PACKET)) )\n\n\n #define REQUEST_INTERFACE_GENERATOR(Z,N,ARGS) \\\n template<typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\n BOOST_PP_TUPLE_ELEM(3,0,ARGS) (procid_t target, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n ASSERT_LT(target, dc_.senders.size()); \\\n request_future<__GLRPC_FRESULT> reply; \\\n if ((BOOST_PP_TUPLE_ELEM(3,2,ARGS) & CONTROL_PACKET) == 0) inc_calls_sent(target); \\\n BOOST_PP_CAT( BOOST_PP_TUPLE_ELEM(3,1,ARGS),N) \\\n <dc_dist_object<T>, F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, T)> \\\n ::exec(this, dc_.senders[target], reply.get_handle(), BOOST_PP_TUPLE_ELEM(3,2,ARGS), target,control_obj_id, remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENI ,_) ); \\\n return reply(); \\\n } \\\n\n /*\n Generates the interface functions. 3rd argument is a tuple (interface name, issue name, flags)\n */\n BOOST_PP_REPEAT(6, REQUEST_INTERFACE_GENERATOR, (typename dc_impl::function_ret_type<__GLRPC_FRESULT>::type internal_request, dc_impl::object_request_issue, (STANDARD_CALL)) )\n BOOST_PP_REPEAT(6, REQUEST_INTERFACE_GENERATOR, (typename dc_impl::function_ret_type<__GLRPC_FRESULT>::type internal_control_request, dc_impl::object_request_issue, (STANDARD_CALL | CONTROL_PACKET)) )\n\n\n #undef RPC_INTERFACE_GENERATOR\n #undef REQUEST_INTERFACE_GENERATOR\n #undef GENARC\n #undef GENT\n #undef GENI\n #undef GENARGS\n\n /// \\endcond\n\n#if DOXYGEN_DOCUMENTATION\n\n/**\n * \\brief Performs a non-blocking RPC call to the target machine\n * to run the provided function pointer.\n *\n * remote_call() calls the function \"fn\" on a target remote machine.\n * \"fn\" may be public, private or protected within the owner class; there are\n * no access restrictions. Provided arguments are serialized and sent to the\n * target. Therefore, all arguments are necessarily transmitted by value.\n * If the target function has a return value, the return value is lost.\n *\n * remote_call() is non-blocking and does not wait for the target machine\n * to complete execution of the function. Different remote_calls may be handled\n * by different threads on the target machine and thus the target function\n * should be made thread-safe.\n * Alternatively, see distributed_control::set_sequentialization_key()\n * to force sequentialization of groups of remote calls.\n *\n * If blocking operation is desired, remote_request() may be used.\n * Alternatively, a full_barrier() may also be used to wait for completion of\n * all incomplete RPC calls.\n *\n * Example:\n * \\code\n * // A print function is defined in the distributed object\n * class distributed_obj_example {\n * graphlab::dc_dist_object<distributed_obj_example> rmi;\n * ... initialization and constructor ...\n * private:\n * void print(std::string s) {\n * std::cout << s << \"\\n\";\n * }\n * public:\n * void print_on_machine_one(std::string s) {\n * // calls the print function on machine 1 with the argument \"s\"\n * rmi.remote_call(1, &distributed_obj_example::print, s);\n * }\n * }\n * \\endcode\n *\n * Note the syntax for obtaining a pointer to a member function.\n *\n * \\param targetmachine The ID of the machine to run the function on\n * \\param fn The function to run on the target machine. Must be a pointer to\n * member function in the owning object.\n * \\param ... The arguments to send to Fn. Arguments must be serializable.\n * and must be castable to the target types.\n */\n void remote_call(procid_t targetmachine, Fn fn, ...);\n\n\n\n/**\n * \\brief Performs a non-blocking RPC call to a collection of machines\n * to run the provided function pointer.\n *\n * This function calls the provided function pointer on a collection of\n * machines contained in the iterator range [begin, end).\n * Provided arguments are serialized and sent to the target.\n * Therefore, all arguments are necessarily transmitted by value.\n * If the target function has a return value, the return value is lost.\n *\n * This function is functionally equivalent to:\n *\n * \\code\n * while(machine_begin != machine_end) {\n * remote_call(*machine_begin, fn, ...);\n * ++machine_begin;\n * }\n * \\endcode\n *\n * However, this function makes some optimizations to ensure all arguments\n * are only serialized once instead of \\#calls times.\n *\n * This function is non-blocking and does not wait for the target machines\n * to complete execution of the function. Different remote_calls may be handled\n * by different threads on the target machines and thus the target function\n * should be made thread-safe. Alternatively, see\n * distributed_control::set_sequentialization_key() to force sequentialization\n * of groups of remote_calls. A full_barrier()\n * may also be issued to wait for completion of all RPC calls issued prior\n * to the full barrier.\n *\n * Example:\n * \\code\n * // A print function is defined in the distributed object\n * class distributed_obj_example {\n * graphlab::dc_dist_object<distributed_obj_example> rmi;\n * ... initialization and constructor ...\n * private:\n * void print(std::string s) {\n * std::cout << s << \"\\n\";\n * }\n * public:\n * void print_on_some_machines(std::string s) {\n * std::vector<procid_t> procs;\n * procs.push_back(1); procs.push_back(3); procs.push_back(5);\n *\n * // calls the print function on machine 1,3,5 with the argument \"s\"\n * rmi.remote_call(procs.begin(), procs.end(),\n * &distributed_obj_example::print, s);\n * }\n * }\n * \\endcode\n *\n *\n * \\param machine_begin The beginning of an iterator range containing a list\n * machines to call. Iterator::value_type must be\n * castable to procid_t.\n * \\param machine_end The end of an iterator range containing a list\n * machines to call. Iterator::value_type must be\n * castable to procid_t.\n * \\param fn The function to run on the target machine. Must be a pointer to\n * member function in the owning object.\n * \\param ... The arguments to send to Fn. Arguments must be serializable.\n * and must be castable to the target types.\n */\n void remote_call(Iterator machine_begin, Iterator machine_end, Fn fn, ...);\n\n\n/**\n * \\brief Performs a blocking RPC call to the target machine\n * to run the provided function pointer.\n *\n * remote_request() calls the function \"fn\" on a target remote machine. Provided\n * arguments are serialized and sent to the target.\n * Therefore, all arguments are necessarily transmitted by value.\n * If the target function has a return value, it is sent back to calling\n * machine.\n *\n * Unlike remote_call(), remote_request() is blocking and waits for the target\n * machine to complete execution of the function. However, different\n * remote_requests may be still be handled by different threads on the target\n * machine.\n *\n * Example:\n * \\code\n * // A print function is defined in the distributed object\n * class distributed_obj_example {\n * graphlab::dc_dist_object<distributed_obj_example> rmi;\n * ... initialization and constructor ...\n * private:\n * int add_one(int i) {\n * return i + 1;\n * }\n * public:\n * int add_one_from_machine_1(int i) {\n * // calls the add_one function on machine 1 with the argument i\n * return rmi.remote_request(1, &distributed_obj_example::add_one, i);\n * }\n * }\n * \\endcode\n *\n * \\see graphlab::object_fiber_remote_request\n * graphlab::dc_dist_object::future_remote_request\n *\n * \\param targetmachine The ID of the machine to run the function on\n * \\param fn The function to run on the target machine. Must be a pointer to\n * member function in the owning object.\n * \\param ... The arguments to send to Fn. Arguments must be serializable.\n * and must be castable to the target types.\n *\n * \\returns Returns the same return type as the function fn\n */\n RetVal remote_request(procid_t targetmachine, Fn fn, ...);\n\n\n\n/**\n * \\brief Performs a nonblocking RPC call to the target machine\n * to run the provided function pointer which has an expected return value.\n *\n * future_remote_request() calls the function \"fn\" on a target remote machine.\n * Provided arguments are serialized and sent to the target.\n * Therefore, all arguments are necessarily transmitted by value.\n * If the target function has a return value, it is sent back to calling\n * machine.\n *\n * future_remote_request() is like remote_request(), but is non-blocking.\n * Instead, it returns immediately a \\ref graphlab::request_future object\n * which will allow you wait for the return value.\n *\n * Example:\n * \\code\n * // A print function is defined in the distributed object\n * class distributed_obj_example {\n * graphlab::dc_dist_object<distributed_obj_example> rmi;\n * ... initialization and constructor ...\n * private:\n * int add_one(int i) {\n * return i + 1;\n * }\n * public:\n * int add_one_from_machine_1(int i) {\n * // calls the add_one function on machine 1 with the argument i\n * // this call returns immediately\n * graphlab::request_future<int> future =\n * rmi.future_remote_request(1, &distributed_obj_example::add_one, i);\n *\n * // ... we can do other stuff here\n * // then when we want the answer\n * int result = future();\n * return result;\n * }\n * }\n * \\endcode\n *\n * \\see graphlab::object_fiber_remote_request\n * graphlab::dc_dist_object::remote_request\n *\n * \\param targetmachine The ID of the machine to run the function on\n * \\param fn The function to run on the target machine. Must be a pointer to\n * member function in the owning object.\n * \\param ... The arguments to send to Fn. Arguments must be serializable.\n * and must be castable to the target types.\n *\n * \\returns Returns a future templated around the same type as the return \n * value of the called function\n */\n request_future<RetVal> future_remote_request(procid_t targetmachine, Fn fn, ...);\n\n\n\n#endif\n/*****************************************************************************\n Implementation of matched send_to / recv_from\n *****************************************************************************/\n\n\n private:\n std::vector<dc_impl::recv_from_struct> recv_froms;\n\n void block_and_wait_for_recv(size_t src,\n std::string& str,\n size_t tag) {\n recv_froms[src].lock.lock();\n recv_froms[src].data = str;\n recv_froms[src].tag = tag;\n recv_froms[src].hasdata = true;\n recv_froms[src].cond.signal();\n recv_froms[src].lock.unlock();\n }\n\n public:\n\n /**\n \\copydoc distributed_control::send_to()\n */\n template <typename U>\n void send_to(procid_t target, U& t, bool control = false) {\n std::stringstream strm;\n oarchive oarc(strm);\n oarc << t;\n strm.flush();\n dc_impl::basic_reply_container rt;\n // I shouldn't use a request to block here since\n // that will take up a thread on the remote side\n // so I simulate a request here.\n size_t rtptr = reinterpret_cast<size_t>(&rt);\n if (control == false) {\n internal_call(target, &dc_dist_object<T>::block_and_wait_for_recv,\n procid(), strm.str(), rtptr);\n }\n else {\n internal_control_call(target, &dc_dist_object<T>::block_and_wait_for_recv,\n procid(), strm.str(), rtptr);\n }\n // wait for reply\n rt.wait();\n\n if (control == false) inc_calls_sent(target);\n }\n\n\n /**\n \\copydoc distributed_control::recv_from()\n */\n template <typename U>\n void recv_from(procid_t source, U& t, bool control = false) {\n // wait on the condition variable until I have data\n dc_impl::recv_from_struct &recvstruct = recv_froms[source];\n recvstruct.lock.lock();\n while (recvstruct.hasdata == false) {\n recvstruct.cond.wait(recvstruct.lock);\n }\n\n // got the data. deserialize it\n std::stringstream strm(recvstruct.data);\n iarchive iarc(strm);\n iarc >> t;\n // clear the data\n std::string(\"\").swap(recvstruct.data);\n // remember the tag so we can unlock it before the remote call\n size_t tag = recvstruct.tag;\n // clear the has data flag\n recvstruct.hasdata = false;\n // unlock\n recvstruct.lock.unlock();\n if (control == false) {\n // remote call to release the sender. Use an empty blob\n dc_.control_call(source, request_reply_handler, tag, dc_impl::blob());\n // I have to increment the calls sent manually here\n // since the matched send/recv calls do not go through the\n // typical object calls. It goes through the DC, but I also want to charge\n // it to this object\n inc_calls_received(source);\n }\n else {\n dc_.control_call(source, request_reply_handler, tag, dc_impl::blob());\n }\n }\n\n\n\n/*****************************************************************************\n Implementation of Broadcast\n *****************************************************************************/\n\nprivate:\n\n std::string broadcast_receive;\n\n void set_broadcast_receive(const std::string &s) {\n broadcast_receive = s;\n }\n\n\n public:\n\n /// \\copydoc distributed_control::broadcast()\n template <typename U>\n void broadcast(U& data, bool originator, bool control = false) {\n if (originator) {\n // construct the data stream\n std::stringstream strm;\n oarchive oarc(strm);\n oarc << data;\n strm.flush();\n broadcast_receive = strm.str();\n if (control == false) {\n for (size_t i = 0;i < numprocs(); ++i) {\n if (i != procid()) {\n internal_request(i,\n &dc_dist_object<T>::set_broadcast_receive,\n broadcast_receive);\n }\n }\n }\n else {\n for (size_t i = 0;i < numprocs(); ++i) {\n if (i != procid()) {\n internal_control_request(i,\n &dc_dist_object<T>::set_broadcast_receive,\n broadcast_receive);\n }\n }\n }\n }\n\n // by the time originator gets here, all machines\n // will have received the data due to the broadcast_receive\n // set a barrier here.\n barrier();\n\n // all machines will now deserialize the data\n if (!originator) {\n std::stringstream strm(broadcast_receive);\n iarchive iarc(strm);\n iarc >> data;\n }\n barrier();\n }\n\n\n/*****************************************************************************\n Implementation of Gather, all_gather\n *****************************************************************************/\n\n private:\n std::vector<std::string> gather_receive;\n atomic<size_t> gatherid;\n\n void set_gather_receive(procid_t source, const std::string &s, size_t gid) {\n while(gatherid.value != gid) sched_yield();\n gather_receive[source] = s;\n }\n public:\n\n /// \\copydoc distributed_control::gather()\n template <typename U>\n void gather(std::vector<U>& data, procid_t sendto, bool control = false) {\n // if not root\n if (sendto != procid()) {\n std::stringstream strm( std::ios::out | std::ios::binary );\n oarchive oarc(strm);\n oarc << data[procid()];\n strm.flush();\n if (control == false) {\n internal_request(sendto,\n &dc_dist_object<T>::set_gather_receive,\n procid(),\n strm.str(),\n gatherid.value);\n }\n else {\n internal_control_request(sendto,\n &dc_dist_object<T>::set_gather_receive,\n procid(),\n strm.str(),\n gatherid.value);\n }\n }\n barrier();\n if (sendto == procid()) {\n // if I am the receiver\n for (procid_t i = 0; i < numprocs(); ++i) {\n if (i != procid()) {\n // receiving only from others\n std::stringstream strm(gather_receive[i],\n std::ios::in | std::ios::binary);\n assert(strm.good());\n iarchive iarc(strm);\n iarc >> data[i];\n }\n }\n }\n gatherid.inc();\n barrier();\n }\n\n/********************************************************************\n Implementation of all gather\n*********************************************************************/\n\n\n\n private:\n // ------- Sense reversing barrier data ----------\n /// The next value of the barrier. either +1 or -1\n int ab_barrier_sense;\n /// When this flag == the current barrier value. The barrier is complete\n int ab_barrier_release;\n /** when barrier sense is 1, barrier clears when\n * child_barrier_counter == numchild. When barrier sense is -1, barrier\n * clears when child_barrier_counter == 0;\n */\n atomic<int> ab_child_barrier_counter;\n /// condition variable and mutex protecting the barrier variables\n fiber_conditional ab_barrier_cond;\n mutex ab_barrier_mut;\n std::string ab_children_data[BARRIER_BRANCH_FACTOR];\n std::string ab_alldata;\n\n /**\n The child calls this function in the parent once the child enters the barrier\n */\n void __ab_child_to_parent_barrier_trigger(procid_t source, std::string collect) {\n ab_barrier_mut.lock();\n // assert childbase <= source <= childbase + BARRIER_BRANCH_FACTOR\n ASSERT_GE(source, childbase);\n ASSERT_LT(source, childbase + BARRIER_BRANCH_FACTOR);\n ab_children_data[source - childbase] = collect;\n ab_child_barrier_counter.inc(ab_barrier_sense);\n ab_barrier_cond.signal();\n ab_barrier_mut.unlock();\n }\n\n /**\n This is on the downward pass of the barrier. The parent calls this function\n to release all the children's barriers\n */\n void __ab_parent_to_child_barrier_release(int releaseval,\n std::string allstrings,\n int use_control_calls) {\n // send the release downwards\n // get my largest child\n logger(LOG_DEBUG, \"AB Barrier Release %d\", releaseval);\n ab_alldata = allstrings;\n for (procid_t i = 0;i < numchild; ++i) {\n if (use_control_calls) {\n internal_control_call((procid_t)(childbase + i),\n &dc_dist_object<T>::__ab_parent_to_child_barrier_release,\n releaseval,\n ab_alldata,\n use_control_calls);\n }\n else {\n internal_call((procid_t)(childbase + i),\n &dc_dist_object<T>::__ab_parent_to_child_barrier_release,\n releaseval,\n ab_alldata,\n use_control_calls);\n }\n }\n ab_barrier_mut.lock();\n ab_barrier_release = releaseval;\n ab_barrier_cond.signal();\n ab_barrier_mut.unlock();\n }\n\n\n public:\n\n /// \\copydoc distributed_control::all_gather()\n template <typename U>\n void all_gather(std::vector<U>& data, bool control = false) {\n if (numprocs() == 1) return;\n // get the string representation of the data\n charstream strm(128);\n oarchive oarc(strm);\n oarc << data[procid()];\n strm.flush();\n // upward message\n int ab_barrier_val = ab_barrier_sense;\n ab_barrier_mut.lock();\n // wait for all children to be done\n while(1) {\n if ((ab_barrier_sense == -1 && ab_child_barrier_counter.value == 0) ||\n (ab_barrier_sense == 1 && ab_child_barrier_counter.value == (int)(numchild))) {\n // flip the barrier sense\n ab_barrier_sense = -ab_barrier_sense;\n // call child to parent in parent\n ab_barrier_mut.unlock();\n if (procid() != 0) {\n // collect all my children data\n charstream strstrm(128);\n oarchive oarc2(strstrm);\n oarc2 << std::string(strm->c_str(), strm->size());\n for (procid_t i = 0;i < numchild; ++i) {\n strstrm.write(ab_children_data[i].c_str(), ab_children_data[i].length());\n }\n strstrm.flush();\n if (control) {\n internal_control_call(parent,\n &dc_dist_object<T>::__ab_child_to_parent_barrier_trigger,\n procid(),\n std::string(strstrm->c_str(), strstrm->size()));\n }\n else {\n internal_call(parent,\n &dc_dist_object<T>::__ab_child_to_parent_barrier_trigger,\n procid(),\n std::string(strstrm->c_str(), strstrm->size()));\n }\n }\n break;\n }\n ab_barrier_cond.wait(ab_barrier_mut);\n }\n\n\n logger(LOG_DEBUG, \"AB barrier phase 1 complete\");\n // I am root. send the barrier release downwards\n if (procid() == 0) {\n ab_barrier_release = ab_barrier_val;\n // build the downward data\n charstream strstrm(128);\n oarchive oarc2(strstrm);\n oarc2 << std::string(strm->c_str(), strm->size());\n for (procid_t i = 0;i < numchild; ++i) {\n strstrm.write(ab_children_data[i].c_str(), ab_children_data[i].length());\n }\n strstrm.flush();\n ab_alldata = std::string(strstrm->c_str(), strstrm->size());\n for (procid_t i = 0;i < numchild; ++i) {\n logger(LOG_DEBUG, \"Sending AB release to %d\", childbase + i);\n internal_control_call((procid_t)(childbase + i),\n &dc_dist_object<T>::__ab_parent_to_child_barrier_release,\n ab_barrier_val,\n ab_alldata,\n (int)control);\n\n }\n }\n // wait for the downward message releasing the barrier\n logger(LOG_DEBUG, \"AB barrier waiting for %d\", ab_barrier_val);\n ab_barrier_mut.lock();\n while(1) {\n if (ab_barrier_release == ab_barrier_val) break;\n ab_barrier_cond.wait(ab_barrier_mut);\n }\n // read the collected data and release the lock\n std::string local_ab_alldata = ab_alldata;\n ab_barrier_mut.unlock();\n\n logger(LOG_DEBUG, \"barrier phase 2 complete\");\n // now the data is a DFS search of a heap\n // I need to unpack it\n size_t heappos = 0;\n std::stringstream istrm(local_ab_alldata);\n iarchive iarc(istrm);\n\n for (size_t i = 0;i < numprocs(); ++i) {\n std::string s;\n iarc >> s;\n\n std::stringstream strm2(s);\n iarchive iarc2(strm2);\n iarc2 >> data[heappos];\n\n if (i + 1 == numprocs()) break;\n // advance heappos\n // leftbranch\n bool lefttraverseblock = false;\n while (1) {\n // can we continue going deaper down the left?\n size_t leftbranch = heappos * BARRIER_BRANCH_FACTOR + 1;\n if (lefttraverseblock == false && leftbranch < numprocs()) {\n heappos = leftbranch;\n break;\n }\n // ok. can't go down the left\n bool this_is_a_right_branch = (((heappos - 1) % BARRIER_BRANCH_FACTOR) == BARRIER_BRANCH_FACTOR - 1);\n // if we are a left branch, go to sibling\n if (this_is_a_right_branch == false) {\n size_t sibling = heappos + 1;\n if (sibling < numprocs()) {\n heappos = sibling;\n break;\n }\n }\n\n // we have finished this subtree, go back up to parent\n // and block the depth traversal on the next round\n // unless heappos is 0\n\n heappos = (heappos - 1) / BARRIER_BRANCH_FACTOR;\n lefttraverseblock = true;\n continue;\n // go to sibling\n }\n\n }\n }\n\n /// \\copydoc distributed_control::all_reduce2()\n template <typename U, typename PlusEqual>\n void all_reduce2(U& data, PlusEqual plusequal, bool control = false) {\n if (numprocs() == 1) return;\n // get the string representation of the data\n /* charstream strm(128);\n oarchive oarc(strm);\n oarc << data;\n strm.flush();*/\n // upward message\n int ab_barrier_val = ab_barrier_sense;\n ab_barrier_mut.lock();\n // wait for all children to be done\n while(1) {\n if ((ab_barrier_sense == -1 && ab_child_barrier_counter.value == 0) ||\n (ab_barrier_sense == 1 && ab_child_barrier_counter.value == (int)(numchild))) {\n // flip the barrier sense\n ab_barrier_sense = -ab_barrier_sense;\n // call child to parent in parent\n ab_barrier_mut.unlock();\n if (procid() != 0) {\n // accumulate my children data\n for (procid_t i = 0;i < numchild; ++i) {\n std::stringstream istrm(ab_children_data[i]);\n iarchive iarc(istrm);\n U tmp;\n iarc >> tmp;\n plusequal(data, tmp);\n }\n // upward message\n charstream ostrm(128);\n oarchive oarc(ostrm);\n oarc << data;\n ostrm.flush();\n if (control) {\n internal_control_call(parent,\n &dc_dist_object<T>::__ab_child_to_parent_barrier_trigger,\n procid(),\n std::string(ostrm->c_str(), ostrm->size()));\n }\n else {\n internal_call(parent,\n &dc_dist_object<T>::__ab_child_to_parent_barrier_trigger,\n procid(),\n std::string(ostrm->c_str(), ostrm->size()));\n }\n }\n break;\n }\n ab_barrier_cond.wait(ab_barrier_mut);\n }\n\n\n logger(LOG_DEBUG, \"AB barrier phase 1 complete\");\n // I am root. send the barrier release downwards\n if (procid() == 0) {\n ab_barrier_release = ab_barrier_val;\n for (procid_t i = 0;i < numchild; ++i) {\n std::stringstream istrm(ab_children_data[i]);\n iarchive iarc(istrm);\n U tmp;\n iarc >> tmp;\n plusequal(data, tmp);\n }\n // build the downward data\n charstream ostrm(128);\n oarchive oarc(ostrm);\n oarc << data;\n ostrm.flush();\n ab_alldata = std::string(ostrm->c_str(), ostrm->size());\n for (procid_t i = 0;i < numchild; ++i) {\n internal_control_call((procid_t)(childbase + i),\n &dc_dist_object<T>::__ab_parent_to_child_barrier_release,\n ab_barrier_val,\n ab_alldata,\n (int)control);\n\n }\n }\n // wait for the downward message releasing the barrier\n logger(LOG_DEBUG, \"AB barrier waiting for %d\", ab_barrier_val);\n ab_barrier_mut.lock();\n while(1) {\n if (ab_barrier_release == ab_barrier_val) break;\n ab_barrier_cond.wait(ab_barrier_mut);\n }\n\n if (procid() != 0) {\n // read the collected data and release the lock\n std::string local_ab_alldata = ab_alldata;\n ab_barrier_mut.unlock();\n\n logger(LOG_DEBUG, \"barrier phase 2 complete\");\n\n std::stringstream istrm(local_ab_alldata);\n iarchive iarc(istrm);\n iarc >> data;\n }\n else {\n ab_barrier_mut.unlock();\n }\n }\n\n\n template <typename U>\n struct default_plus_equal {\n void operator()(U& u, const U& v) {\n u += v;\n }\n };\n\n /// \\copydoc distributed_control::all_reduce()\n template <typename U>\n void all_reduce(U& data, bool control = false) {\n all_reduce2(data, default_plus_equal<U>(), control);\n }\n\n////////////////////////////////////////////////////////////////////////////\n\n\n/*****************************************************************************\n Implementation of All Scatter\n *****************************************************************************/\n\n template <typename U>\n void all_to_all(std::vector<U>& data, bool control = false) {\n ASSERT_EQ(data.size(), numprocs());\n for (size_t i = 0;i < data.size(); ++i) {\n if (i != procid()) {\n std::stringstream strm( std::ios::out | std::ios::binary );\n oarchive oarc(strm);\n oarc << data[i];\n strm.flush();\n if (control == false) {\n internal_call(i,\n &dc_dist_object<T>::set_gather_receive,\n procid(),\n strm.str(),\n gatherid.value);\n }\n else {\n internal_control_call(i,\n &dc_dist_object<T>::set_gather_receive,\n procid(),\n strm.str(),\n gatherid.value);\n }\n }\n }\n full_barrier();\n for (size_t i = 0; i < data.size(); ++i) {\n if (i != procid()) {\n std::stringstream strm(gather_receive[i],\n std::ios::in | std::ios::binary);\n assert(strm.good());\n iarchive iarc(strm);\n iarc >> data[i];\n }\n }\n gatherid.inc();\n barrier();\n }\n\n\n/*****************************************************************************\n Implementation of Barrier\n *****************************************************************************/\n\n\n\n private:\n // ------- Sense reversing barrier data ----------\n /// The next value of the barrier. either +1 or -1\n int barrier_sense;\n /// When this flag == the current barrier value. The barrier is complete\n int barrier_release;\n /** when barrier sense is 1, barrier clears when\n * child_barrier_counter == numchild. When barrier sense is -1, barrier\n * clears when child_barrier_counter == 0;\n */\n atomic<int> child_barrier_counter;\n /// condition variable and mutex protecting the barrier variables\n fiber_conditional barrier_cond;\n mutex barrier_mut;\n procid_t parent; /// parent node\n size_t childbase; /// id of my first child\n procid_t numchild; /// number of children\n\n\n\n\n /**\n The child calls this function in the parent once the child enters the barrier\n */\n void __child_to_parent_barrier_trigger(procid_t source) {\n barrier_mut.lock();\n // assert childbase <= source <= childbase + BARRIER_BRANCH_FACTOR\n ASSERT_GE(source, childbase);\n ASSERT_LT(source, childbase + BARRIER_BRANCH_FACTOR);\n child_barrier_counter.inc(barrier_sense);\n barrier_cond.signal();\n barrier_mut.unlock();\n }\n\n /**\n This is on the downward pass of the barrier. The parent calls this function\n to release all the children's barriers\n */\n void __parent_to_child_barrier_release(int releaseval) {\n // send the release downwards\n // get my largest child\n logger(LOG_DEBUG, \"Barrier Release %d\", releaseval);\n for (procid_t i = 0;i < numchild; ++i) {\n internal_control_call((procid_t)(childbase + i),\n &dc_dist_object<T>::__parent_to_child_barrier_release,\n releaseval);\n\n }\n barrier_mut.lock();\n barrier_release = releaseval;\n barrier_cond.signal();\n barrier_mut.unlock();\n }\n\n\n public:\n\n /// \\copydoc distributed_control::barrier()\n void barrier() {\n // upward message\n int barrier_val = barrier_sense;\n barrier_mut.lock();\n // wait for all children to be done\n while(1) {\n if ((barrier_sense == -1 && child_barrier_counter.value == 0) ||\n (barrier_sense == 1 && child_barrier_counter.value == (int)(numchild))) {\n // flip the barrier sense\n barrier_sense = -barrier_sense;\n // call child to parent in parent\n barrier_mut.unlock();\n if (procid() != 0) {\n internal_control_call(parent,\n &dc_dist_object<T>::__child_to_parent_barrier_trigger,\n procid());\n }\n break;\n }\n barrier_cond.wait(barrier_mut);\n }\n\n\n logger(LOG_DEBUG, \"barrier phase 1 complete\");\n // I am root. send the barrier release downwards\n if (procid() == 0) {\n barrier_release = barrier_val;\n\n for (procid_t i = 0;i < numchild; ++i) {\n internal_control_call((procid_t)(childbase + i),\n &dc_dist_object<T>::__parent_to_child_barrier_release,\n barrier_val);\n\n }\n }\n // wait for the downward message releasing the barrier\n logger(LOG_DEBUG, \"barrier waiting for %d\", barrier_val);\n barrier_mut.lock();\n while(1) {\n if (barrier_release == barrier_val) break;\n barrier_cond.wait(barrier_mut);\n }\n barrier_mut.unlock();\n\n logger(LOG_DEBUG, \"barrier phase 2 complete\");\n }\n\n\n /*****************************************************************************\n Implementation of Full Barrier\n*****************************************************************************/\n private:\n mutex full_barrier_lock;\n fiber_conditional full_barrier_cond;\n std::vector<size_t> calls_to_receive;\n // used to inform the counter that the full barrier\n // is in effect and all modifications to the calls_recv\n // counter will need to lock and signal\n volatile bool full_barrier_in_effect;\n\n /** number of 'source' processor counts which have\n not achieved the right recv count */\n atomic<size_t> num_proc_recvs_incomplete;\n\n /// Marked as 1 if the proc is complete\n dense_bitset procs_complete;\n\n public:\n\n /// \\copydoc distributed_control::full_barrier()\n void full_barrier() {\n // gather a sum of all the calls issued to machine 0\n std::vector<size_t> calls_sent_to_target(numprocs(), 0);\n for (size_t i = 0;i < numprocs(); ++i) {\n calls_sent_to_target[i] = callssent[i].value;\n }\n\n // tell node 0 how many calls there are\n std::vector<std::vector<size_t> > all_calls_sent(numprocs());\n all_calls_sent[procid()] = calls_sent_to_target;\n all_gather(all_calls_sent, true);\n\n // get the number of calls I am supposed to receive from each machine\n calls_to_receive.clear(); calls_to_receive.resize(numprocs(), 0);\n for (size_t i = 0;i < numprocs(); ++i) {\n calls_to_receive[i] += all_calls_sent[i][procid()];\n }\n // clear the counters\n num_proc_recvs_incomplete.value = numprocs();\n procs_complete.clear();\n // activate the full barrier\n full_barrier_in_effect = true;\n __asm(\"mfence\");\n // begin one pass to set all which are already completed\n for (size_t i = 0;i < numprocs(); ++i) {\n if (callsreceived[i].value >= calls_to_receive[i]) {\n if (procs_complete.set_bit(i) == false) {\n num_proc_recvs_incomplete.dec();\n }\n } else {\n logstream(LOG_DEBUG) << \"Expecting \" << calls_to_receive[i] \n << \" calls from \" << i << \" but only \" \n << callsreceived[i].value << \"received.\" << std::endl;\n }\n }\n\n full_barrier_lock.lock();\n while (num_proc_recvs_incomplete.value > 0) {\n logstream(LOG_DEBUG) << \"Calls Incomplete. Waiting.\" << std::endl;\n full_barrier_cond.wait(full_barrier_lock);\n }\n full_barrier_lock.unlock();\n full_barrier_in_effect = false;\n// for (size_t i = 0; i < numprocs(); ++i) {\n// std::cout << \"Received \" << global_calls_received[i].value << \" from \" << i << std::endl;\n// }\n barrier();\n }\n\n /* -------------------- Implementation of Gather Statistics -----------------*/\n private:\n struct collected_statistics {\n size_t callssent;\n size_t bytessent;\n collected_statistics(): callssent(0), bytessent(0) { }\n void save(oarchive &oarc) const {\n oarc << callssent << bytessent;\n }\n void load(iarchive &iarc) {\n iarc >> callssent >> bytessent;\n }\n };\n public:\n /** Gather RPC statistics. All machines must call\n this function at the same time. However, only proc 0 will\n return values */\n std::map<std::string, size_t> gather_statistics() {\n std::map<std::string, size_t> ret;\n\n std::vector<collected_statistics> stats(numprocs());\n stats[procid()].callssent = calls_sent();\n stats[procid()].bytessent = bytes_sent();\n logstream(LOG_INFO) << procid() << \": calls_sent: \";\n for (size_t i = 0;i < numprocs(); ++i) {\n logstream(LOG_INFO) << callssent[i].value << \", \";\n }\n logstream(LOG_INFO) << std::endl;\n logstream(LOG_INFO) << procid() << \": calls_recv: \";\n for (size_t i = 0;i < numprocs(); ++i) {\n logstream(LOG_INFO) << callsreceived[i].value << \", \";\n }\n logstream(LOG_INFO) << std::endl;\n\n\n gather(stats, 0, true);\n if (procid() == 0) {\n collected_statistics cs;\n for (size_t i = 0;i < numprocs(); ++i) {\n cs.callssent += stats[i].callssent;\n cs.bytessent += stats[i].bytessent;\n }\n ret[\"total_calls_sent\"] = cs.callssent;\n ret[\"total_bytes_sent\"] = cs.bytessent;\n }\n return ret;\n }\n};\n\n#include <graphlab/macros_undef.hpp>\n#include <graphlab/rpc/mem_function_arg_types_undef.hpp>\n#undef BARRIER_BRANCH_FACTOR\n}// namespace graphlab\n#endif\n\n" }, { "path": "src/graphlab/rpc/dc_dist_object_base.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_DC_DIST_OBJECT_BASE_HPP\n#define GRAPHLAB_DC_DIST_OBJECT_BASE_HPP\n#include <vector>\n#include <graphlab/rpc/dc_internal_types.hpp>\nnamespace graphlab {\n\nnamespace dc_impl {\n/**\n * \\ingroup rpc\n * \\internal\nProvides an interface for extracting and updating counters from dc_dist_objects\n*/\nclass dc_dist_object_base{\n public:\n\n virtual ~dc_dist_object_base() { } \n\n /// Increment the number of calls sent from this object\n virtual void inc_calls_sent(procid_t source) = 0;\n /// Increment the number of calls received by this object\n virtual void inc_calls_received(procid_t dest) = 0;\n \n /// Increment the number of bytes sent from this object\n virtual void inc_bytes_sent(procid_t target, size_t bytes) = 0;\n\n /// Return the number of calls received by this object\n virtual size_t calls_received() const = 0;\n /// Return the number of calls sent from this object\n virtual size_t calls_sent() const = 0;\n};\n\n}\n}\n\n#endif\n\n" }, { "path": "src/graphlab/rpc/dc_init_from_env.cpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <cstdio>\n#include <cstdlib>\n#include <string>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_init_from_env.hpp>\n#include <graphlab/util/stl_util.hpp>\n#include <graphlab/logger/logger.hpp>\nnamespace graphlab {\n\nbool init_param_from_env(dc_init_param& param) {\n char* nodeid = getenv(\"SPAWNID\");\n if (nodeid == NULL) {\n return false;\n }\n param.curmachineid = atoi(nodeid);\n\n char* nodes = getenv(\"SPAWNNODES\");\n std::string nodesstr = nodes;\n if (nodes == NULL) {\n return false;\n }\n\n param.machines = strsplit(nodesstr, \",\");\n for (size_t i = 0;i < param.machines.size(); ++i) {\n param.machines[i] = param.machines[i] + \":\" + tostr(10000 + i);\n }\n // set defaults\n param.numhandlerthreads = RPC_DEFAULT_NUMHANDLERTHREADS;\n param.commtype = RPC_DEFAULT_COMMTYPE;\n return true;\n}\n\n} // namespace graphlab\n\n" }, { "path": "src/graphlab/rpc/dc_init_from_env.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_DC_INIT_FROM_ENV_HPP\n#define GRAPHLAB_DC_INIT_FROM_ENV_HPP\n#include <graphlab/rpc/dc.hpp>\nnamespace graphlab {\n /** \n * \\ingroup rpc\n * initializes parameters from environment. Returns true on success */\n bool init_param_from_env(dc_init_param& param);\n}\n\n#endif // GRAPHLAB_DC_INIT_FROM_ENV_HPP\n\n\n\n" }, { "path": "src/graphlab/rpc/dc_init_from_mpi.cpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <iostream>\n#include <cstdio>\n#include <cstdlib>\n#include <string>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_init_from_mpi.hpp>\n#include <graphlab/util/stl_util.hpp>\n#include <graphlab/util/net_util.hpp>\n#include <graphlab/logger/logger.hpp>\n\n#ifdef HAS_MPI\n#include <graphlab/util/mpi_tools.hpp>\n#endif\nnamespace graphlab {\n\nbool init_param_from_mpi(dc_init_param& param,dc_comm_type commtype) {\n#ifdef HAS_MPI\n ASSERT_MSG(commtype == TCP_COMM, \"MPI initialization only supports TCP at the moment\");\n // Look for a free port to use. \n std::pair<size_t, int> port_and_sock = get_free_tcp_port();\n size_t port = port_and_sock.first;\n int sock = port_and_sock.second;\n \n std::string ipaddr = \n get_local_ip_as_str(mpi_tools::rank() == 0 /* print stuff only if I am master */);\n ipaddr = ipaddr + \":\" + tostr(port);\n // now do an allgather\n logstream(LOG_INFO) << \"Will Listen on: \" << ipaddr << std::endl;\n std::vector<std::string> machines;\n mpi_tools::all_gather(ipaddr, param.machines);\n // set defaults\n param.curmachineid = (procid_t)(mpi_tools::rank());\n\n param.numhandlerthreads = RPC_DEFAULT_NUMHANDLERTHREADS;\n param.commtype = commtype;\n param.initstring = param.initstring + std::string(\" __sockhandle__=\") + tostr(sock) + \" \";\n return true;\n#else\n std::cerr << \"MPI Support not compiled!\" << std::endl;\n exit(0);\n#endif\n}\n\n} // namespace graphlab\n\n\n" }, { "path": "src/graphlab/rpc/dc_init_from_mpi.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_DC_INIT_FROM_MPI_HPP\n#define GRAPHLAB_DC_INIT_FROM_MPI_HPP\n#include <graphlab/rpc/dc.hpp>\nnamespace graphlab {\n /**\n * \\ingroup rpc \n * initializes parameters from MPI. Returns true on success\n MPI must be initialized before calling this function */\n bool init_param_from_mpi(dc_init_param& param, dc_comm_type commtype = TCP_COMM);\n}\n\n#endif // GRAPHLAB_DC_INIT_FROM_MPI_HPP\n\n\n" }, { "path": "src/graphlab/rpc/dc_init_from_zookeeper.cpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <cstdio>\n#include <cstdlib>\n#include <string>\n#include <vector>\n#include <algorithm>\n#include <boost/bind.hpp>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/zookeeper/server_list.hpp>\n#include <graphlab/util/stl_util.hpp>\n#include <graphlab/util/net_util.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/logger/logger.hpp>\nnamespace graphlab {\n\nvoid zk_callback(zookeeper::server_list* slist,\n std::string name_space,\n std::vector<std::string> servers,\n std::vector<std::string>& result,\n size_t num_to_watch_for,\n mutex& result_lock,\n conditional& result_cond) {\n if (servers.size() == num_to_watch_for) {\n result_lock.lock();\n result = servers;\n slist->stop_watching(\"graphlab\");\n result_cond.signal();\n result_lock.unlock();\n }\n}\n\n\nbool init_param_from_zookeeper(dc_init_param& param) {\n char* zk_hosts = getenv(\"ZK_SERVERS\");\n char* zk_jobname = getenv(\"ZK_JOBNAME\");\n char* zk_numnodes = getenv(\"ZK_NUMNODES\");\n if (zk_hosts == NULL || zk_jobname == NULL || zk_numnodes == NULL) {\n return false;\n }\n\n std::vector<std::string> zk_hosts_list = strsplit(zk_hosts, \",\");\n\n // number of nodes to wait for\n size_t numnodes = atoi(zk_numnodes);\n ASSERT_GE(numnodes, 1);\n logstream(LOG_EMPH) << \"Using Zookeeper for Initialization. Waiting for \"\n << numnodes << \" to join\" << std::endl;\n\n // generate a unique identifier for this server\n\n std::pair<size_t, int> port_and_sock = get_free_tcp_port();\n size_t port = port_and_sock.first;\n int sock = port_and_sock.second;\n std::string ipaddr = get_local_ip_as_str(true);\n ipaddr = ipaddr + \":\" + tostr(port);\n logstream(LOG_INFO) << \"Will Listen on: \" << ipaddr << std::endl;\n\n // get an ip address\n zookeeper::server_list server_list(zk_hosts_list,\n zk_jobname,\n ipaddr);\n\n // final server list goes here\n std::vector<std::string> received_servers;\n // locks to product the final server list\n mutex lock;\n conditional cond;\n\n // construct the watch to watch for changes on zookeeper\n server_list.set_callback(boost::bind(zk_callback,\n _1,\n _2,\n _3,\n boost::ref(received_servers),\n numnodes,\n boost::ref(lock),\n boost::ref(cond)));\n\n server_list.join(\"graphlab\");\n\n lock.lock();\n received_servers = server_list.watch_changes(\"graphlab\");\n // wait until I get all the servers\n // TODO: add a timeout\n while(received_servers.size() < numnodes) cond.wait(lock);\n lock.unlock();\n\n // done!\n // now make sure that everyone sees the server list in the same order\n\n ASSERT_EQ(received_servers.size(), numnodes);\n std::sort(received_servers.begin(), received_servers.end());\n\n // now fill the parameter list\n param.machines = received_servers;\n param.curmachineid = std::find(received_servers.begin(), received_servers.end(),\n ipaddr) - received_servers.begin();\n ASSERT_LT(param.curmachineid, received_servers.size());\n param.numhandlerthreads = RPC_DEFAULT_NUMHANDLERTHREADS;\n param.commtype = RPC_DEFAULT_COMMTYPE;\n param.initstring = param.initstring + std::string(\" __sockhandle__=\") + tostr(sock) + \" \";\n // detach from the server list\n // now, this takes advantage of the Zookeeper feature that\n // every machine sees all changes in the same order.\n // i.e. At some point, everyone would have seen a complete server list.\n // Once that happens, everyone can leave.\n server_list.set_callback(NULL);\n server_list.leave(\"graphlab\");\n\n return true;\n}\n\n} // namespace graphlab\n\n" }, { "path": "src/graphlab/rpc/dc_init_from_zookeeper.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_DC_INIT_FROM_ZOOKEEPER_HPP\n#define GRAPHLAB_DC_INIT_FROM_ZOOKEEPER_HPP\n#include <graphlab/rpc/dc.hpp>\nnamespace graphlab {\n /**\n * \\ingroup rpc\n * initializes parameters from ZooKeeper. Returns true on success.\n * To initialize from Zookeeper, the following environment variables must be set\n *\n * ZK_SERVERS: A comma separated list of zookeeper servers. Port\n * number must be included.\n * ZK_JOBNAME: The name of the job to use. This must be unique to the cluster.\n * i.e. no other job with the same name must run at the same time\n * ZK_NUMNODES: The number of processes to wait for\n *\n */\n bool init_param_from_zookeeper(dc_init_param& param);\n}\n\n#endif // GRAPHLAB_DC_INIT_FROM_ZOOKEEPER_HPP\n\n" }, { "path": "src/graphlab/rpc/dc_internal_types.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef DC_INTERNAL_TYPES_HPP\n#define DC_INTERNAL_TYPES_HPP\n#include <boost/function.hpp>\n#include <boost/unordered_map.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/util/resizing_array_sink.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/serialization/serialization_includes.hpp>\nnamespace graphlab {\nclass distributed_control;\n\nnamespace dc_impl { \n\n/** \n * \\internal\n * \\ingroup rpc\n * The type of the callback function used by the communications\nclasses when data is received*/\ntypedef void (*comm_recv_callback_type)(void* tag, procid_t src, \n const char* buf, size_t len);\n\n/**\n * \\internal\n * \\ingroup rpc\n * The type of the local function call dispatcher.\n * \\see dispatch_type2\n */\ntypedef void (*dispatch_type)(distributed_control& dc, procid_t, unsigned char, const char* data, size_t len);\n\n/**\n *\\internal\n * \\ingroup rpc\n * A second type of the local function call dispatcher.\n * Currently only used by POD calls. TODO: to move all other call\n * systems to use dispatch2.\n * \\see dispatch_type\n */\ntypedef void (*dispatch_type2)(distributed_control& dc, procid_t, unsigned char, const char* data, size_t len);\n\n\ntypedef boost::unordered_map<std::string, dispatch_type> dispatch_map_type;\n\n// commm capabilities\nconst size_t COMM_STREAM = 1;\nconst size_t COMM_DATAGRAM = 0;\n\n/**\n * \\internal\n * \\ingroup rpc\n * The header form of each packet */\nstruct packet_hdr {\n uint32_t len; /// length of the packet\n procid_t src; /// source machine\n unsigned char packet_type_mask; /// the types are in dc_packet_mask.hpp\n unsigned char sequentialization_key;\n};\n\ntypedef uint32_t block_header_type;\n\n/**\n * \\internal\n * \\ingroup rpc\n * special handling for the only pointer datatype \nwe natively support serialization for. Basically,\nwe must delete it. if charstring_free is called on a\nchar*, it will be deleted. Otherwise it will not do anything*/\ntemplate <typename T> \ninline void charstring_free(T& t) { }\n\n/**\n * \\internal\n * \\ingroup rpc\n */\ntemplate <>\ninline void charstring_free<char*>(char* &c){\n delete [] c;\n};\n\n\n\n/**\n * \\internal\n * \\ingroup rpc\n * \n * The data needed to receive the matched send / recvs */\nstruct recv_from_struct {\n inline recv_from_struct():tag(0), hasdata(false) { }\n \n std::string data;\n size_t tag;\n mutex lock;\n conditional cond;\n bool hasdata;\n \n};\n\n/**\n * \\internal\n * \\ingroup rpc\n * Used for termination detection\n */\nstruct terminator_token {\n terminator_token():calls_sent(0),calls_recv(0),terminate(false) { }\n terminator_token(size_t sent, size_t recv):calls_sent(sent),\n calls_recv(recv),terminate(false) { }\n size_t calls_sent;\n size_t calls_recv;\n bool terminate;\n};\n\n\n\n/**\n * Used to maintain a linked list of buffers.\n */\nstruct buffer_elem {\n char* buf;\n size_t len;\n buffer_elem* next;\n};\n\n}\n}\n\nSERIALIZABLE_POD(graphlab::dc_impl::terminator_token);\n#endif\n\n" }, { "path": "src/graphlab/rpc/dc_packet_mask.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef DC_PACKET_MASK_HPP\n#define DC_PACKET_MASK_HPP\nnamespace graphlab {\n // --------- Packet header types --------------\n\n /**\n * \\internal\n * \\ingroup rpc\n * Used for regular calls which go into a thread pool\n * for evaluation\n */\n const unsigned char STANDARD_CALL = 1;\n\n /**\n * \\internal\n \\ingroup rpc\n * \n If control packet flag is set, this packet \n does not increment any counters.\n */\n const unsigned char CONTROL_PACKET = 16; \n\n /**\n * \\internal\n * \\ingroup rpc\n * \n * Used to identify that after sending this \n * packet, a flush is required\n */\n const unsigned char FLUSH_PACKET = 64;\n}\n#endif\n\n" }, { "path": "src/graphlab/rpc/dc_receive.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef DC_RECEIVE_HPP\n#define DC_RECEIVE_HPP\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/parallel/atomic.hpp>\nnamespace graphlab {\nnamespace dc_impl {\n \n/**\n\\ingroup rpc\n\\internal\nBase class of the data receiving class.\nThis class forms the receiving side of a \"multiplexer\"\nData entering from a single socket will be passed to this\nfunction through the incoming_data function call.\n\nThis class must understand the packet header and issue the right\ncalls in the owning dc. \n*/\nclass dc_receive {\n public:\n dc_receive() { };\n virtual ~dc_receive() { };\n\n /**\n gets a buffer. The buffer length is returned in retbuflength\n This will be used for receiving data.\n If get_buffer() or advance_buffer() is called,\n incoming_data will never be called.\n */\n virtual char* get_buffer(size_t& retbuflength) = 0;\n \n /**\n Commits a buffer obtained using get_buffer.\n c will be the result of a previous call to get_buffer() or advance_buffer()\n This function should commit a range of bytes starting of c,\n up to 'wrotelength' bytes. A new empty buffer should be returned\n and the size is returned in retbuflength\n */\n virtual char* advance_buffer(char* c, size_t wrotelength, \n size_t& retbuflength) = 0;\n \n\n /**\n * Last call sent to any instance of dc_receive.\n * If the sender multithreads, the sending thread must shut down.\n */\n virtual void shutdown() = 0;\n};\n\n\n} // namespace dc_impl\n} // namespace graphlab\n#endif\n\n" }, { "path": "src/graphlab/rpc/dc_send.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef DC_SEND_HPP\n#define DC_SEND_HPP\n#include <sys/types.h>\n#include <sys/socket.h>\n\n#include <iostream>\n#include <graphlab/rpc/circular_iovec_buffer.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/thread_local_send_buffer.hpp>\n#include <graphlab/rpc/dc_types.hpp>\nnamespace graphlab {\nnamespace dc_impl {\n\n/**\n\\ingroup rpc\n\\internal\nBase class of the data sending class.\nThis class forms the sending side of a \"multiplexer\"\nsend_data() will be called with a packet mask as well as a\ncharacter stream containing the contents of the packet.\nThe class should accumulate the data in an iovec structure\nand relinquish it on get_outgoing_data()\n*/\nclass dc_send{\n public:\n dc_send() { }\n \n virtual ~dc_send() { }\n\n virtual void register_send_buffer(thread_local_buffer* buffer) = 0;\n virtual void unregister_send_buffer(thread_local_buffer* buffer) = 0;\n\n /**\n Bytes sent must be incremented BEFORE the data is transmitted.\n Packets marked CONTROL_PACKET should not be counted\n */\n virtual size_t bytes_sent() = 0;\n \n /**\n * flushes immediately\n */ \n virtual void flush() = 0;\n\n /**\n * Requests a flush as soon as possible\n */\n virtual void flush_soon() = 0;\n\n\n /**\n * Writes a string to an internal buffer to be flushed later.\n * This is a \"slow path\" to be used only when the thread local buffer\n * is not available.\n */\n virtual void write_to_buffer(char* c, size_t len) = 0;\n\n virtual size_t set_option(std::string opt, size_t val) {\n return 0;\n }\n\n /**\n * Returns length if there is data, 0 otherwise. This function\n * must be reentrant, but it is guaranteed that only one thread will\n * call this function at anytime.\n */\n virtual size_t get_outgoing_data(circular_iovec_buffer& outdata) = 0;\n\n\n /**\n * Utility function: writes a packet header into an archive.\n * but returns an offset to the location of the length entry allowing it to\n * be filled in later.\n */\n inline static size_t write_packet_header(oarchive& oarc, \n procid_t src, \n unsigned char packet_type_mask, \n unsigned char sequentialization_key) {\n size_t base = oarc.off;\n oarc.advance(sizeof(packet_hdr));\n packet_hdr* hdr = reinterpret_cast<packet_hdr*>(oarc.buf + (oarc.off - sizeof(packet_hdr)));\n hdr->len = 0;\n hdr->src = src;\n hdr->packet_type_mask = packet_type_mask;\n hdr->sequentialization_key = sequentialization_key;\n return base;\n }\n};\n \n\n} // namespace dc_impl\n} // namespace graphlab\n#endif\n\n" }, { "path": "src/graphlab/rpc/dc_services.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <graphlab/rpc/dc_dist_object.hpp>\n#ifndef GRAPHLAB_DC_SERVICES_HPP\n#define GRAPHLAB_DC_SERVICES_HPP\n#include <graphlab/parallel/pthread_tools.hpp>\n\n\n\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n\n /**\n \\internal\n \\ingroup rpc\n Creates a new context for MPI-like global global operations.\n Where all machines create an instance of dc_services at the same time,\n operations performed by the new dc_services instance will not interfere\n and will run in parallel with other contexts. i.e. If I have two\n distributed dc_services instances, one instance can \n perform a barrier while another instance performs a broadcast() at the same \n time.\n */\n class dc_services {\n private:\n dc_dist_object<dc_services> rmi;\n \n public:\n dc_services(distributed_control &dc):rmi(dc, this) { }\n \n /// Returns the underlying dc_dist_object \n dc_dist_object<dc_services>& rmi_instance() {\n return rmi;\n }\n\n /// Returns the underlying dc_dist_object \n const dc_dist_object<dc_services>& rmi_instance() const {\n return rmi;\n }\n \n /**\n \\copydoc distributed_control::send_to()\n */\n template <typename U>\n inline void send_to(procid_t target, U& t, bool control = false) {\n rmi.send_to(target, t, control);\n }\n \n /**\n \\copydoc distributed_control::recv_from()\n */\n template <typename U>\n inline void recv_from(procid_t source, U& t, bool control = false) {\n rmi.recv_from(source, t, control);\n }\n\n /**\n \\copydoc distributed_control::broadcast()\n */\n template <typename U>\n inline void broadcast(U& data, bool originator, bool control = false) { \n rmi.broadcast(data, originator, control);\n }\n\n /**\n \\copydoc distributed_control::gather()\n */\n template <typename U>\n inline void gather(std::vector<U>& data, procid_t sendto, bool control = false) {\n rmi.gather(data, sendto, control);\n }\n\n /**\n \\copydoc distributed_control::all_gather()\n */\n template <typename U>\n inline void all_gather(std::vector<U>& data, bool control = false) {\n rmi.all_gather(data, control);\n }\n\n /**\n \\copydoc distributed_control::all_reduce()\n */\n template <typename U>\n inline void all_reduce(U& data, bool control = false) {\n rmi.all_reduce(data, control);\n }\n\n /// \\copydoc distributed_control::all_reduce2()\n template <typename U, typename PlusEqual>\n void all_reduce2(U& data, PlusEqual plusequal, bool control = false) {\n rmi.all_reduce2(data, plusequal, control);\n }\n\n /// \\copydoc distributed_control::barrier()\n inline void barrier() {\n rmi.barrier();\n }\n \n \n /// \\copydoc distributed_control::full_barrier()\n inline void full_barrier() {\n rmi.full_barrier();\n }\n \n \n\n };\n\n\n} // end of namespace graphlab\n\n\n#include <graphlab/macros_undef.hpp>\n#endif\n\n" }, { "path": "src/graphlab/rpc/dc_stream_receive.cpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n#include <iostream>\n#include <algorithm>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/dc_stream_receive.hpp>\n\n//#define DC_RECEIVE_DEBUG\nnamespace graphlab {\nnamespace dc_impl {\n\n\n\nchar* dc_stream_receive::get_buffer(size_t& retbuflength) {\n retbuflength = write_buffer_len - write_buffer_written;\n return writebuffer + write_buffer_written;\n}\n\n\nchar* dc_stream_receive::advance_buffer(char* c, size_t wrotelength, \n size_t& retbuflength) {\n // find the last complete message we have read\n write_buffer_written += wrotelength;\n if (write_buffer_written >= sizeof(packet_hdr)) {\n size_t offset = 0;\n packet_hdr* hdr = reinterpret_cast<packet_hdr*>(writebuffer);\n // keep pushing the header until I reach a point where there is insufficient\n // room to read a header, or the message is not large enough\n while(offset + sizeof(packet_hdr) <= write_buffer_written &&\n offset + hdr->len + sizeof(packet_hdr) <= write_buffer_written) {\n offset += hdr->len + sizeof(packet_hdr);\n hdr = reinterpret_cast<packet_hdr*>(writebuffer + offset);\n }\n\n if (offset > 0) {\n // ok. everything before the offset is good\n // since we are going to give this buffer away, we need to prepare a new buffer\n\n // allocate whatever it is going to take to hold next message\n // have we read the incomplete message's header?\n size_t incomplete_message_len = 0;\n if (offset + sizeof(packet_hdr) <= write_buffer_written) incomplete_message_len = hdr->len;\n\n size_t new_buflen = std::max<size_t>(sizeof(packet_hdr) + incomplete_message_len, RECEIVE_BUFFER_SIZE);\n char* new_writebuffer = (char*)malloc(new_buflen);\n\n if (write_buffer_len - offset > 0) {\n // copy over to the new buffer everything we will not use\n memcpy(new_writebuffer, writebuffer + offset, write_buffer_written - offset);\n }\n // if we reach here, we have an available block\n // give away the buffer to dc\n dc->deferred_function_call_chunk(writebuffer, offset, associated_proc);\n writebuffer = new_writebuffer;\n write_buffer_written -= offset;\n write_buffer_len = new_buflen;\n } else {\n // nothing ready yet\n // do we have enough room though?\n if (hdr->len + sizeof(packet_hdr) > write_buffer_len) {\n size_t newlen = hdr->len + sizeof(packet_hdr);\n writebuffer = (char*)realloc(writebuffer, newlen);\n write_buffer_len = newlen;\n }\n }\n }\n return get_buffer(retbuflength);\n}\n\n\n \nvoid dc_stream_receive::shutdown() { }\n\n} // namespace dc_impl\n} // namespace graphlab\n\n" }, { "path": "src/graphlab/rpc/dc_stream_receive.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef DC_STREAM_RECEIVE_HPP\n#define DC_STREAM_RECEIVE_HPP\n#include <boost/type_traits/is_base_of.hpp>\n#include <graphlab/rpc/circular_char_buffer.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_compile_parameters.hpp>\n#include <graphlab/rpc/dc_receive.hpp>\n#include <graphlab/parallel/atomic.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/logger/logger.hpp>\nnamespace graphlab {\nclass distributed_control;\n\nnamespace dc_impl {\n\n/**\n * \\internal\n \\ingroup rpc\n Receiver processor for the dc class.\n The job of the receiver is to take as input a byte stream\n (as received from the socket) and cut it up into meaningful chunks.\n This can be thought of as a receiving end of a multiplexor.\n \n This is the default unbuffered receiver.\n*/\nclass dc_stream_receive: public dc_receive{\n public:\n \n dc_stream_receive(distributed_control* dc, procid_t associated_proc): \n writebuffer(NULL), write_buffer_written(0), dc(dc), \n associated_proc(associated_proc) { \n writebuffer = (char*)malloc(RECEIVE_BUFFER_SIZE);\n write_buffer_len = RECEIVE_BUFFER_SIZE;\n }\n\n private:\n\n char* writebuffer;\n size_t write_buffer_written;\n size_t write_buffer_len;\n \n /// pointer to the owner\n distributed_control* dc;\n\n procid_t associated_proc;\n \n void shutdown();\n\n \n char* get_buffer(size_t& retbuflength);\n \n\n char* advance_buffer(char* c, size_t wrotelength, \n size_t& retbuflength);\n \n};\n\n\n} // namespace dc_impl\n} // namespace graphlab\n#endif\n\n" }, { "path": "src/graphlab/rpc/dc_tcp_comm.cpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon UniversityIP addresses.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <sys/socket.h>\n#include <netinet/in.h>\n#include <arpa/inet.h>\n#include <netdb.h>\n#include <unistd.h>\n#include <fcntl.h>\n#include <signal.h>\n#include <netinet/tcp.h>\n#include <ifaddrs.h>\n#include <poll.h>\n\n#include <limits>\n#include <vector>\n#include <string>\n#include <map>\n\n#include <boost/lexical_cast.hpp>\n#include <boost/bind.hpp>\n#include <graphlab/logger/logger.hpp>\n#include <graphlab/rpc/dc_tcp_comm.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/get_current_process_hash.cpp>\n#define compile_barrier() asm volatile(\"\": : :\"memory\")\n\n#include <graphlab/macros_def.hpp>\n\n// prefix mangling if not Mac\n#ifndef __APPLE__\n#include <graphlab/rpc/evwrapdef.h>\n#endif\n#include <event2/event.h>\n#include <event2/thread.h>\n\n//#define COMM_DEBUG\nnamespace graphlab {\n\n namespace dc_impl {\n\n void dc_tcp_comm::init(const std::vector<std::string> &machines,\n const std::map<std::string,std::string> &initopts,\n procid_t curmachineid,\n std::vector<dc_receive*> receiver_,\n std::vector<dc_send*> sender_) {\n\n curid = curmachineid;\n ASSERT_LT(machines.size(), std::numeric_limits<procid_t>::max());\n nprocs = (procid_t)(machines.size());\n receiver = receiver_;\n sender = sender_;\n\n // insert machines into the address map\n all_addrs.resize(nprocs);\n portnums.resize(nprocs);\n assert(triggered_timeouts.size() >= nprocs);\n triggered_timeouts.clear();\n // fill all the socks\n sock.resize(nprocs);\n for (size_t i = 0;i < nprocs; ++i) {\n sock[i].id = i;\n sock[i].owner = this;\n sock[i].outsock = -1;\n sock[i].insock = -1;\n sock[i].inevent = NULL;\n sock[i].outevent = NULL;\n sock[i].wouldblock = false;\n sock[i].data.msg_name = NULL;\n sock[i].data.msg_namelen = 0;\n sock[i].data.msg_control = NULL;\n sock[i].data.msg_controllen = 0;\n sock[i].data.msg_flags = 0;\n sock[i].data.msg_iovlen = 0;\n sock[i].data.msg_iov = NULL;\n }\n\n program_md5 = get_current_process_hash();\n ASSERT_EQ(program_md5.length(), 32);\n // parse the machines list, and extract the relevant address information\n for (size_t i = 0;i < machines.size(); ++i) {\n // extract the port number\n size_t pos = machines[i].find(\":\");\n ASSERT_NE(pos, std::string::npos);\n std::string address = machines[i].substr(0, pos);\n size_t port = boost::lexical_cast<size_t>(machines[i].substr(pos+1));\n\n struct hostent* ent = gethostbyname(address.c_str());\n ASSERT_EQ(ent->h_length, 4);\n uint32_t addr = *reinterpret_cast<uint32_t*>(ent->h_addr_list[0]);\n\n all_addrs[i] = addr;\n ASSERT_LT(port, 65536);\n portnums[i] = (uint16_t)(port);\n }\n network_bytessent = 0;\n buffered_len = 0;\n // if sock handle is set\n std::map<std::string, std::string>::const_iterator iter =\n initopts.find(\"__sockhandle__\");\n if (iter != initopts.end()) {\n open_listening(atoi(iter->second.c_str()));\n } else {\n open_listening();\n }\n // to improve the \"synchronous\" nature of the connection setup,\n // the last machine will do this in reverse order.\n // To wait for all machines to connect to it, before it\n // tries to connect to all machines.\n // The last machine therefore essentially acts as the \"barrier\" leader\n if (curid != nprocs - 1) {\n // not the last machine.\n // Connect to everyone, EXCEPT the last machine\n // and wait for all incoming connections\n for(procid_t i = 0;i < nprocs - 1; ++i) connect(i);\n\n // wait for p - 1 incoming connections\n insock_lock.lock();\n while(1) {\n if (num_in_connected() == sock.size() - 1) break;\n insock_cond.wait(insock_lock);\n }\n insock_lock.unlock();\n\n // connect to the last machine\n connect(nprocs - 1);\n\n // wait for the last connection\n insock_lock.lock();\n while(1) {\n if (num_in_connected() == sock.size()) break;\n insock_cond.wait(insock_lock);\n }\n insock_lock.unlock();\n } else {\n // I am the last machine\n ASSERT_EQ(curid, nprocs - 1);\n // wait for all incoming connections before connecting to everyone\n // connect to myself\n connect(nprocs - 1);\n insock_lock.lock();\n while(1) {\n if (num_in_connected() == sock.size()) break;\n insock_cond.wait(insock_lock);\n }\n insock_lock.unlock();\n // now, when I know that machines 0 to #procs - 2 have\n // all established a connection to each other\n // connect to everyone. This is essentially equivalent to the\n // barrier release message\n for(size_t i = 0;i < nprocs; ++i) connect(i);\n }\n // everyone is connected.\n // Construct the eventbase\n construct_events();\n // we reserve the last 2 cores for communication\n inthreads.launch(boost::bind(&dc_tcp_comm::receive_loop, this, inevbase), thread::cpu_count() - 2);\n outthreads.launch(boost::bind(&dc_tcp_comm::send_loop, this, outevbase), thread::cpu_count() - 1);\n is_closed = false;\n }\n\n void dc_tcp_comm::construct_events() {\n int ret = evthread_use_pthreads();\n if (ret < 0) logstream(LOG_FATAL) << \"Unable to initialize libevent with pthread support!\" << std::endl;\n // number of evs to create.\n outevbase = event_base_new();\n if (!outevbase) logstream(LOG_FATAL) << \"Unable to construct libevent base\" << std::endl;\n send_all_timeout.owner = this;\n send_all_timeout.send_all = true;\n send_triggered_timeout.owner = this;\n send_triggered_timeout.send_all = false;\n send_all_event = event_new(outevbase, -1, EV_TIMEOUT | EV_PERSIST, on_send_event, &(send_all_timeout));\n assert(send_all_event != NULL);\n struct timeval t = {SEND_POLL_TIMEOUT / 1000000, SEND_POLL_TIMEOUT % 1000000} ;\n event_add(send_all_event, &t);\n send_triggered_event = event_new(outevbase, -1, EV_TIMEOUT | EV_PERSIST, on_send_event, &(send_triggered_timeout));\n assert(send_triggered_event != NULL);\n\n inevbase = event_base_new();\n if (!inevbase) logstream(LOG_FATAL) << \"Unable to construct libevent base\" << std::endl;\n\n\n //register all event objects\n for (size_t i = 0;i < sock.size(); ++i) {\n sock[i].inevent = event_new(inevbase, sock[i].insock, EV_READ | EV_PERSIST | EV_ET,\n on_receive_event, &(sock[i]));\n if (sock[i].inevent == NULL) {\n logstream(LOG_FATAL) << \"Unable to register socket read event\" << std::endl;\n }\n\n sock[i].outevent = event_new(outevbase, sock[i].outsock, EV_WRITE | EV_PERSIST | EV_ET,\n on_send_event, &(sock[i]));\n if (sock[i].outevent == NULL) {\n logstream(LOG_FATAL) << \"Unable to register socket write event\" << std::endl;\n }\n\n event_add(sock[i].inevent, NULL);\n //struct timeval t = {0, 10};\n event_add(sock[i].outevent, NULL);\n }\n }\n\n size_t dc_tcp_comm::num_in_connected() const {\n size_t connected = 0;\n for (size_t i = 0;i < sock.size(); ++i) {\n connected += (sock[i].insock != -1);\n }\n return connected;\n }\n\n void dc_tcp_comm::trigger_send_timeout(procid_t target, bool urgent) {\n if (!urgent) {\n if (sock[target].wouldblock == false &&\n triggered_timeouts.get(target) == false) {\n triggered_timeouts.set_bit(target);\n event_active(send_triggered_event, EV_TIMEOUT, 1);\n }\n }\n else {\n process_sock(&(sock[target]));\n }\n }\n\n\n void dc_tcp_comm::close() {\n if (is_closed) return;\n logstream(LOG_INFO) << \"Closing listening socket\" << std::endl;\n // close the listening socket\n if (listensock > 0) {\n ::close(listensock);\n listensock = -1;\n }\n // shutdown the listening thread\n listenthread.join();\n\n // clear the outevent loop\n event_base_loopbreak(outevbase);\n outthreads.join();\n for (size_t i = 0;i < sock.size(); ++i) {\n event_free(sock[i].outevent);\n }\n event_free(send_triggered_event);\n event_free(send_all_event);\n event_base_free(outevbase);\n\n\n logstream(LOG_INFO) << \"Closing outgoing sockets\" << std::endl;\n // close all outgoing sockets\n for (size_t i = 0;i < sock.size(); ++i) {\n if (sock[i].outsock > 0) {\n ::close(sock[i].outsock);\n sock[i].outsock = -1;\n }\n }\n\n // clear the inevent loop\n event_base_loopbreak(inevbase);\n inthreads.join();\n for (size_t i = 0;i < sock.size(); ++i) {\n event_free(sock[i].inevent);\n }\n event_base_free(inevbase);\n\n\n logstream(LOG_INFO) << \"Closing incoming sockets\" << std::endl;\n // close all incoming sockets\n for (size_t i = 0;i < sock.size(); ++i) {\n if (sock[i].insock > 0) {\n ::close(sock[i].insock);\n sock[i].insock = -1;\n }\n }\n is_closed = true;\n }\n\n\n bool dc_tcp_comm::send_till_block(socket_info& sockinfo) {\n sockinfo.wouldblock = false;\n // while there is still data to be sent\n BEGIN_TRACEPOINT(tcp_send_call);\n while(!sockinfo.outvec.empty()) {\n sockinfo.outvec.fill_msghdr(sockinfo.data);\n ssize_t ret = sendmsg(sockinfo.outsock, &sockinfo.data, 0);\n if (ret < 0) {\n END_TRACEPOINT(tcp_send_call);\n if (errno == EWOULDBLOCK || errno == EAGAIN) {\n sockinfo.wouldblock = true;\n return false;\n }\n else {\n logstream(LOG_FATAL) << \"send error: \" << strerror(errno) << std::endl;\n return false;\n }\n }\n\n#ifdef COMM_DEBUG\n logstream(LOG_INFO) << ret << \" bytes --> \" << sockinfo.id << std::endl;\n#endif\n network_bytessent.inc(ret);\n sockinfo.outvec.sent(ret);\n }\n END_TRACEPOINT(tcp_send_call);\n return true;\n }\n\n int dc_tcp_comm::sendtosock(int sockfd, const char* buf, size_t len) {\n size_t numsent = 0;\n BEGIN_TRACEPOINT(tcp_send_call);\n while (numsent < len) {\n ssize_t ret = ::send(sockfd, buf + numsent, len - numsent, 0);\n if (ret < 0) {\n logstream(LOG_ERROR) << \"send error: \" << strerror(errno) << std::endl;\n END_TRACEPOINT(tcp_send_call);\n return errno;\n }\n numsent += ret;\n }\n END_TRACEPOINT(tcp_send_call);\n return 0;\n }\n\n void dc_tcp_comm::set_tcp_no_delay(int fd) {\n int flag = 1;\n int result = setsockopt(fd, /* socket affected */\n IPPROTO_TCP, /* set option at TCP level */\n TCP_NODELAY, /* name of option */\n (char *) &flag,\n sizeof(int));\n if (result < 0) {\n logstream(LOG_WARNING)\n << \"Unable to disable Nagle. Performance may be signifantly reduced\"\n << std::endl;\n }\n // set nonblocking\n }\n\n void dc_tcp_comm::set_non_blocking(int fd) {\n int flag = fcntl(fd, F_GETFL);\n if (flag < 0) {\n logstream(LOG_FATAL) << \"Unable to get socket flags\" << std::endl;\n }\n flag |= O_NONBLOCK;\n if (fcntl(fd, F_SETFL, flag) < 0) {\n logstream(LOG_FATAL) << \"Unable to set socket as non-blocking\" << std::endl;\n }\n\n }\n\n\n void dc_tcp_comm::new_socket(int newsock, sockaddr_in* otheraddr,\n procid_t id) {\n // figure out the address of the incoming connection\n uint32_t addr = *reinterpret_cast<uint32_t*>(&(otheraddr->sin_addr));\n // locate the incoming address in the list\n logstream(LOG_INFO) << \"Incoming connection from \"\n << inet_ntoa(otheraddr->sin_addr) << std::endl;\n ASSERT_LT(id, all_addrs.size());\n ASSERT_EQ(all_addrs[id], addr);\n insock_lock.lock();\n ASSERT_EQ(sock[id].insock, -1);\n sock[id].insock = newsock;\n insock_cond.signal();\n insock_lock.unlock();\n logstream(LOG_INFO) << \"Proc \" << procid() << \" accepted connection \"\n << \"from machine \" << id << std::endl;\n }\n\n\n\n void dc_tcp_comm::open_listening(int sockhandle) {\n // open listening socket\n if (sockhandle == 0) {\n listensock = socket(AF_INET, SOCK_STREAM, 0);\n // uninteresting boiler plate. Set the port number and socket type\n sockaddr_in my_addr;\n my_addr.sin_family = AF_INET;\n my_addr.sin_port = htons(portnums[curid]);\n my_addr.sin_addr.s_addr = INADDR_ANY;\n memset(&(my_addr.sin_zero), '\\0', 8);\n logstream(LOG_INFO) << \"Proc \" << procid() << \" Bind on \"\n << portnums[curid] << \"\\n\";\n if (bind(listensock, (sockaddr*)&my_addr, sizeof(my_addr)) < 0)\n {\n logstream(LOG_FATAL) << \"bind: \" << strerror(errno) << \"\\n\";\n ASSERT_TRUE(0);\n }\n }\n else {\n listensock = sockhandle;\n }\n logstream(LOG_INFO) << \"Proc \" << procid()\n << \" listening on \" << portnums[curid] << \"\\n\";\n ASSERT_EQ(0, listen(listensock, 128));\n // spawn a thread which loops around accept\n listenthread.launch(boost::bind(&dc_tcp_comm::accept_handler, this));\n } // end of open_listening\n\n void dc_tcp_comm::connect(size_t target) {\n if (sock[target].outsock != -1) {\n return;\n } else {\n int newsock = socket(AF_INET, SOCK_STREAM, 0);\n set_tcp_no_delay(newsock);\n sockaddr_in serv_addr;\n serv_addr.sin_family = AF_INET;\n // set the target port\n serv_addr.sin_port = htons(portnums[target]);\n // set the target address\n serv_addr.sin_addr = *(struct in_addr*)&(all_addrs[target]);\n memset(&(serv_addr.sin_zero), '\\0', 8);\n // Connect!\n logstream(LOG_INFO) << \"Trying to connect from \"\n << curid << \" -> \" << target\n << \" on port \" << portnums[target] << \"\\n\";\n logger(LOG_INFO, \"Destination IP = %s\", inet_ntoa(serv_addr.sin_addr));\n // retry 10 times at 1 second intervals\n bool success = false;\n for (size_t i = 0;i < 10; ++i) {\n if (::connect(newsock, (sockaddr*)&serv_addr, sizeof(serv_addr)) < 0) {\n logstream(LOG_INFO)\n << \"connect \" << curid << \" to \" << target << \": \"\n << strerror(errno) << \". Retrying...\\n\";\n timer::sleep(1);\n // posix says that\n /* If connect() fails, the state of the socket is unspecified.\n Conforming applications should close the file descriptor and\n create a new socket before attempting to reconnect. */\n ::close(newsock);\n newsock = socket(AF_INET, SOCK_STREAM, 0);\n set_tcp_no_delay(newsock);\n } else {\n // send the initial message\n initial_message msg; \n msg.id = curid;\n memcpy(msg.md5, program_md5.c_str(), 32);\n sendtosock(newsock, reinterpret_cast<char*>(&msg), sizeof(initial_message));\n set_non_blocking(newsock);\n success = true;\n break;\n }\n }\n if (!success) {\n logstream(LOG_FATAL) << \"Failed to establish connection\" << std::endl;\n }\n // remember the socket\n sock[target].outsock = newsock;\n logstream(LOG_INFO) << \"connection from \" << curid << \" to \" << target\n << \" established.\" << std::endl;\n }\n } // end of connect\n\n\n\n\n\n////////////////////////////////////////////////////////////////////////////\n// These stuff run in seperate threads //\n////////////////////////////////////////////////////////////////////////////\n\n // waits for incoming connections\n void dc_tcp_comm::accept_handler() {\n pollfd pf;\n pf.fd = listensock;\n pf.events = POLLIN;\n pf.revents = 0;\n size_t numsocks_connected = 0;\n logstream(LOG_INFO) << \"Listening thread launched.\" << std::endl;\n while(numsocks_connected < sock.size()) {\n // wait for incoming event\n poll(&pf, 1, 1000);\n // if we have a POLLIN, we have an incoming socket request\n if (pf.revents & POLLIN) {\n logstream(LOG_INFO) << \"Accepting....\" << std::endl;\n // accept the socket\n sockaddr_in their_addr;\n socklen_t namelen = sizeof(sockaddr_in);\n int newsock = accept(listensock, (sockaddr*)&their_addr, &namelen);\n logstream(LOG_INFO) << \"Accepted\" << std::endl;\n if (newsock < 0) {\n break;\n }\n // set the socket options and inform the\n set_tcp_no_delay(newsock);\n // before accepting the socket, get the machine number\n initial_message remote_message;\n ssize_t msglen = 0;\n while(msglen != sizeof(initial_message)) {\n int retval = recv(newsock, (char*)(&remote_message) + msglen,\n sizeof(initial_message) - msglen, 0);\n if (retval < 0) {\n if (errno == EWOULDBLOCK || errno == EAGAIN) {\n continue;\n }\n else {\n logstream(LOG_FATAL) << \"error: \" << errno << \" receive error: \" << strerror(errno) << std::endl;\n }\n }\n else if (retval > 0) {\n msglen += retval;\n }\n else if (retval == 0) {\n std::cout << \"error: connection dropped.\" << std::endl;\n ::close(newsock);\n newsock = -1;\n break;\n }\n }\n if (newsock != -1) {\n // validate the md5 hash\n std::string other_md5 = std::string(remote_message.md5, 32);\n if (other_md5 != program_md5) {\n logstream(LOG_FATAL) << \"MD5 mismatch. \\n \"\n << \"\\tProcess \" << curid << \" has hash \" << program_md5 << \" \\n \"\n << \"\\tProcess \" << remote_message.id << \" has hash \" << other_md5 << \" \\n \"\n << \"\\tGraphLab requires all machines to run exactly the same binary.\" << std::endl;\n }\n // register the new socket\n set_non_blocking(newsock);\n new_socket(newsock, &their_addr, remote_message.id);\n ++numsocks_connected;\n }\n }\n if (listensock == -1) {\n // the owner has closed\n break;\n }\n }\n logstream(LOG_INFO) << \"Listening thread quitting\" << std::endl;\n } // end of run\n\n\n // libevent receive handler\n void on_receive_event(int fd, short ev, void* arg) {\n dc_tcp_comm::socket_info* sockinfo = (dc_tcp_comm::socket_info*)(arg);\n dc_tcp_comm* comm = sockinfo->owner;\n if (ev & EV_READ) {\n // get a direct pointer to my receiver\n dc_receive* receiver = comm->receiver[sockinfo->id];\n\n size_t buflength;\n char *c = receiver->get_buffer(buflength);\n while(1) {\n ssize_t msglen = recv(fd, c, buflength, 0);\n if (msglen < 0) {\n if (errno == EAGAIN || errno == EWOULDBLOCK) break;\n else {\n logstream(LOG_FATAL) << \"receive error: \" << strerror(errno) << std::endl;\n break;\n }\n }\n else if (msglen == 0) {\n // socket closed\n break;\n }\n else if (msglen > 0) {\n comm->network_bytesreceived.inc(msglen);\n #ifdef COMM_DEBUG\n logstream(LOG_INFO) << msglen << \" bytes <-- \"\n << sockinfo->id << std::endl;\n #endif\n c = receiver->advance_buffer(c, msglen, buflength);\n }\n }\n }\n }\n\n void dc_tcp_comm::receive_loop(struct event_base* ev) {\n logstream(LOG_INFO) << \"Receive loop Started\" << std::endl;\n int ret = event_base_dispatch(ev);\n if (ret != 0) {\n logstream(LOG_FATAL) << \"Receive loop Quit with \" << ret << std::endl;\n }\n else {\n logstream(LOG_INFO) << \"Receive loop Stopped\" << std::endl;\n }\n }\n\n\n void dc_tcp_comm::check_for_new_data(dc_tcp_comm::socket_info& sockinfo) {\n buffered_len.inc(sender[sockinfo.id]->get_outgoing_data(sockinfo.outvec));\n }\n\n\n inline void process_sock(dc_tcp_comm::socket_info* sockinfo) {\n if (sockinfo->m.try_lock()) {\n dc_tcp_comm* comm = sockinfo->owner;\n // get a direct pointer to my receiver\n if (sockinfo->wouldblock == false) {\n comm->check_for_new_data(*sockinfo);\n if (!sockinfo->outvec.empty()) {\n comm->send_till_block(*sockinfo);\n }\n }\n sockinfo->m.unlock();\n }\n }\n\n // libevent receive handler\n void on_send_event(int fd, short ev, void* arg) {\n if (ev & EV_WRITE) {\n dc_tcp_comm::socket_info* sockinfo = (dc_tcp_comm::socket_info*)(arg);\n sockinfo->wouldblock = false;\n process_sock(sockinfo);\n }\n else if (ev & EV_TIMEOUT) {\n dc_tcp_comm::timeout_event* te = (dc_tcp_comm::timeout_event*)(arg);\n dc_tcp_comm* comm = te->owner;\n if (te->send_all == false) {\n // this is a triggered event\n foreach(uint32_t i, comm->triggered_timeouts) {\n comm->triggered_timeouts.clear_bit(i);\n dc_tcp_comm::socket_info* sockinfo = &(comm->sock[i]);\n process_sock(sockinfo);\n }\n } else {\n // send all event\n for(uint32_t i = 0;i < comm->sock.size(); ++i) {\n dc_tcp_comm::socket_info* sockinfo = &(comm->sock[i]);\n process_sock(sockinfo);\n }\n }\n }\n }\n\n\n void dc_tcp_comm::send_loop(struct event_base* ev) {\n logstream(LOG_INFO) << \"Send loop Started\" << std::endl;\n int ret = event_base_dispatch(ev);\n if (ret != 0) {\n logstream(LOG_FATAL) << \"Send loop Quit with \" << ret << std::endl;\n }\n else {\n logstream(LOG_INFO) << \"Send loop Stopped\" << std::endl;\n }\n }\n }; // end of namespace dc_impl\n}; // end of namespace graphlab\n\n" }, { "path": "src/graphlab/rpc/dc_tcp_comm.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef DC_TCP_COMM_HPP\n#define DC_TCP_COMM_HPP\n\n#include <sys/socket.h>\n#include <netinet/in.h>\n\n#include <vector>\n#include <string>\n#include <map>\n\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/dc_comm_base.hpp>\n#include <graphlab/rpc/circular_iovec_buffer.hpp>\n#include <graphlab/util/tracepoint.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n\n#ifndef __APPLE__\n// prefix mangling if not Mac\n#include <graphlab/rpc/evwrapdef.h>\n#endif\n#include <event2/event.h>\nnamespace graphlab {\nnamespace dc_impl {\n\n\nvoid on_receive_event(int fd, short ev, void* arg);\nvoid on_send_event(int fd, short ev, void* arg);\n\n/**\n \\ingroup rpc\n \\internal\nTCP implementation of the communications subsystem.\nProvides a single object interface to sending/receiving data streams to\na collection of machines.\n*/\nclass dc_tcp_comm:public dc_comm_base {\n public:\n\n DECLARE_TRACER(tcp_send_call);\n\n inline dc_tcp_comm() {\n is_closed = true;\n INITIALIZE_TRACER(tcp_send_call, \"dc_tcp_comm: send syscall\");\n }\n\n size_t capabilities() const {\n return COMM_STREAM;\n }\n\n /**\n this fuction should pause until all communication has been set up\n and returns the number of systems in the network.\n After which, all other remaining public functions (numprocs(), send(), etc)\n should operate normally. Every received message should immediate trigger the\n attached receiver\n\n machines: a vector of strings where each string is of the form [IP]:[portnumber]\n initopts: unused\n curmachineid: The ID of the current machine. machines[curmachineid] will be\n the listening address of this machine\n\n recvcallback: A function pointer to the receiving function. This function must be thread-safe\n tag: An additional pointer passed to the receiving function.\n */\n void init(const std::vector<std::string> &machines,\n const std::map<std::string,std::string> &initopts,\n procid_t curmachineid,\n std::vector<dc_receive*> receiver,\n std::vector<dc_send*> senders);\n\n /** shuts down all sockets and cleans up */\n void close();\n\n ~dc_tcp_comm() {\n close();\n }\n\n inline bool channel_active(size_t target) const {\n return (sock[target].outsock != -1);\n }\n\n /**\n Returns the number of machines in the network.\n Only valid after call to init()\n */\n inline procid_t numprocs() const {\n return nprocs;\n }\n\n /**\n * Returns the current machine ID.\n * Only valid after call to init()\n */\n inline procid_t procid() const {\n return curid;\n }\n\n /**\n * Returns the total number of bytes sent\n */\n inline size_t network_bytes_sent() const {\n return network_bytessent.value;\n }\n\n /**\n * Returns the total number of bytes received\n */\n inline size_t network_bytes_received() const {\n return network_bytesreceived.value;\n }\n\n inline size_t send_queue_length() const {\n size_t a = network_bytessent.value;\n size_t b = buffered_len.value;\n return b - a;\n }\n\n /**\n Sends the string of length len to the target machine dest.\n Only valid after call to init();\n Establishes a connection if necessary\n */\n void send(size_t target, const char* buf, size_t len);\n\n void trigger_send_timeout(procid_t target, bool urgent);\n\n private:\n /// Sets TCP_NO_DELAY on the socket passed in fd\n void set_tcp_no_delay(int fd);\n\n void set_non_blocking(int fd);\n\n /// called when listener receives an incoming socket request\n void new_socket(int newsock, sockaddr_in* otheraddr, procid_t remotemachineid);\n\n\n /// The number of incoming connections established\n size_t num_in_connected() const;\n\n /** opens the listening sock and spawns a thread to listen on it.\n * Uses sockhandle if non-zero\n */\n void open_listening(int sockhandle = 0);\n\n\n /// constructs a connection to the target machine\n void connect(size_t target);\n\n /// wrapper around the standard send. but loops till the buffer is all sent\n int sendtosock(int sockfd, const char* buf, size_t len);\n\n\n procid_t curid; /// if od the current processor\n procid_t nprocs; /// number of processors\n bool is_closed; /// whether this socket is closed\n\n std::string program_md5; /// MD5 hash of current program\n\n\n /// all_addrs[i] will contain the IP address of machine i\n std::vector<uint32_t> all_addrs;\n std::map<uint32_t, procid_t> addr2id;\n std::vector<uint16_t> portnums;\n\n std::vector<dc_receive*> receiver;\n std::vector<dc_send*> sender;\n atomic<size_t> buffered_len;\n\n\n struct initial_message {\n procid_t id;\n char md5[32];\n };\n\n\n /// All information about stuff regarding a particular sock\n /// Passed to the receive handler\n struct socket_info{\n size_t id; /// which machine this is connected to\n dc_tcp_comm* owner; /// this object\n int outsock; /// FD of the outgoing socket\n int insock; /// FD of the incoming socket\n struct event* inevent; /// event object for incoming information\n struct event* outevent; /// event object for outgoing information\n bool wouldblock;\n mutex m;\n\n circular_iovec_buffer outvec; /// outgoing data\n struct msghdr data;\n };\n\n mutex insock_lock; /// locks the insock field in socket_info\n conditional insock_cond; /// triggered when the insock field in socket_info changes\n\n struct timeout_event {\n bool send_all;\n dc_tcp_comm* owner;\n };\n\n std::vector<socket_info> sock;\n\n /**\n * Sends as much of the buffer inside the sockinfo as possible\n * until the send call will block or all sends are complete.\n * Returns true when the buffer has been completely sent\n * If wouldblock returns true, the next call to send_till_block may block\n */\n void send_all(socket_info& sockinfo);\n bool send_till_block(socket_info& sockinfo);\n void check_for_new_data(socket_info& sockinfo);\n void construct_events();\n\n\n\n // counters\n atomic<size_t> network_bytessent;\n atomic<size_t> network_bytesreceived;\n\n //////////// Receiving Sockets //////////////////////\n thread_group inthreads;\n void receive_loop(struct event_base*);\n\n friend void process_sock(socket_info* sockinfo);\n friend void on_receive_event(int fd, short ev, void* arg);\n struct event_base* inevbase;\n\n\n //////////// Sending Sockets //////////////////////\n thread_group outthreads;\n void send_loop(struct event_base*);\n friend void on_send_event(int fd, short ev, void* arg);\n struct event_base* outevbase;\n struct event* send_triggered_event;\n struct event* send_all_event;\n timeout_event send_triggered_timeout;\n timeout_event send_all_timeout;\n\n fixed_dense_bitset<256> triggered_timeouts;\n //////////// Listening Sockets //////////////////////\n int listensock;\n thread listenthread;\n void accept_handler();\n};\n\nvoid process_sock(dc_tcp_comm::socket_info* sockinfo);\n\n} // namespace dc_impl\n} // namespace graphlab\n\n#ifndef __APPLE__\n// prefix mangling if not Mac\n#include <graphlab/rpc/evwrapundef.h>\n#endif\n\n#endif\n\n" }, { "path": "src/graphlab/rpc/dc_thread_get_send_buffer.hpp", "content": "#ifndef GRAPHLAB_RPC_DC_DEPENDENCY_SPLIT_HPP\n#define GRAPHLAB_RPC_DC_DEPENDENCY_SPLIT_HPP\n\n/*\n * This implements a bunch of internal functions which should really reside\n * as static functions in distributed_control. But \n *\n */\n#include <graphlab/serialization/oarchive.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/rpc/thread_local_send_buffer.hpp>\n\nnamespace graphlab {\nnamespace dc_impl {\nextern pthread_key_t thrlocal_send_buffer_key;\nextern pthread_key_t thrlocal_sequentialization_key;\n\n/**\n * \\internal\n * Obtains the thread local send buffer for a given target\n */\ninline oarchive* get_thread_local_buffer(procid_t target) {\n void* ptr = pthread_getspecific(thrlocal_send_buffer_key);\n thread_local_buffer* p = (thread_local_buffer*)(ptr);\n if (p == NULL) {\n p = new thread_local_buffer;\n pthread_setspecific(thrlocal_send_buffer_key, (void*)p);\n }\n return p->acquire(target);\n}\n\n/**\n * \\internal\n * Releases the thread local send buffer for the given target\n */\ninline void release_thread_local_buffer(procid_t target, \n bool do_not_count_bytes_sent) {\n void* ptr = pthread_getspecific(thrlocal_send_buffer_key);\n thread_local_buffer* p = (thread_local_buffer*)(ptr);\n p->release(target, do_not_count_bytes_sent);\n}\n\n/**\n * \\internal\n * Writes a sequence of bytes to the local send buffer\n */\ninline void write_thread_local_buffer(procid_t target, \n char* c,\n size_t len,\n bool do_not_count_bytes_sent) {\n void* ptr = pthread_getspecific(thrlocal_send_buffer_key);\n thread_local_buffer* p = (thread_local_buffer*)(ptr);\n p->write(target, c, len, do_not_count_bytes_sent);\n}\n\n\n\n/**\n * \\internal\n */\ninline void push_flush_thread_local_buffer() {\n void* ptr = pthread_getspecific(thrlocal_send_buffer_key);\n thread_local_buffer* p = (thread_local_buffer*)(ptr);\n if (p) p->push_flush();\n}\n\n/**\n * \\internal\n */\ninline void pull_flush_thread_local_buffer(procid_t proc) {\n void* ptr = pthread_getspecific(thrlocal_send_buffer_key);\n thread_local_buffer* p = (thread_local_buffer*)(ptr);\n if (p) p->pull_flush(proc);\n}\n\n/**\n * \\internal\n */\ninline void pull_flush_soon_thread_local_buffer(procid_t proc) {\n void* ptr = pthread_getspecific(thrlocal_send_buffer_key);\n thread_local_buffer* p = (thread_local_buffer*)(ptr);\n if (p) p->pull_flush_soon(proc);\n}\n\n\n\n/**\n * \\internal\n */\ninline void pull_flush_soon_thread_local_buffer() {\n void* ptr = pthread_getspecific(thrlocal_send_buffer_key);\n thread_local_buffer* p = (thread_local_buffer*)(ptr);\n if (p) p->pull_flush_soon();\n}\n\n\n\n/**\n * Gets the current procid.\n * This function really exists to split the dependency between this header and\n * dc.hpp\n */\ninline procid_t _get_procid() {\n void* ptr = pthread_getspecific(thrlocal_send_buffer_key);\n thread_local_buffer* p = (thread_local_buffer*)(ptr);\n if (p == NULL) {\n p = new thread_local_buffer;\n pthread_setspecific(thrlocal_send_buffer_key, (void*)p);\n }\n return p->procid;\n}\n\n/**\n * Get the current sequentialization key.\n * This function really exists to split the dependency between this header and\n * dc.hpp\n */\ninline procid_t _get_sequentialization_key() {\n size_t oldval = reinterpret_cast<size_t>(pthread_getspecific(dc_impl::thrlocal_sequentialization_key));\n return (unsigned char)oldval;\n}\n\n}\n}\n#endif\n" }, { "path": "src/graphlab/rpc/dc_types.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef DISTRIBUTED_CONTROL_TYPES_HPP\n#define DISTRIBUTED_CONTROL_TYPES_HPP\n#include <inttypes.h>\n#include <graphlab/serialization/iarchive.hpp>\nnamespace graphlab {\n /// The type used for numbering processors \\ingroup rpc\n typedef uint16_t procid_t;\n\n /**\n * \\internal\n * \\ingroup rpc\n * The underlying communication protocol\n */\n enum dc_comm_type {\n TCP_COMM, ///< TCP/IP\n SCTP_COMM ///< SCTP (limited support)\n };\n\n\n /**\n * \\internal\n * \\ingroup rpc\n * A pointer that points directly into\n * the middle of a deserialized buffer.\n */\n struct wild_pointer {\n const void* ptr;\n\n void load(iarchive& iarc) {\n assert(iarc.buf != NULL);\n ptr = reinterpret_cast<const void*>(iarc.buf + iarc.off);\n }\n };\n};\n#include <graphlab/rpc/dc_packet_mask.hpp>\n#endif\n\n" }, { "path": "src/graphlab/rpc/delta_dht.cpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n/**\n * Also contains code that is Copyright 2011 Yahoo! Inc. All rights\n * reserved. \n *\n * Contributed under the iCLA for:\n * Joseph Gonzalez (jegonzal@yahoo-inc.com) \n *\n */\n\n\n#include <boost/unordered_map.hpp>\n#include <graphlab/rpc/delta_dht.hpp>\n\n\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n namespace delta_dht_impl {\n\n typedef boost::unordered_map<const void*, icache*> cache_map_type;\n \n void destroy_tls_data(void* ptr) {\n cache_map_type* cache_map_ptr = static_cast<cache_map_type*>(ptr);\n if(cache_map_ptr != NULL) { \n cache_map_type& cache_map = *cache_map_ptr;\n typedef cache_map_type::value_type pair_type;\n foreach(pair_type& pair, cache_map) {\n if(pair.second != NULL) { \n delete pair.second; \n pair.second = NULL;\n }\n }\n delete cache_map_ptr;\n }\n }\n struct tls_key_creator {\n pthread_key_t TLS_KEY;\n tls_key_creator() : TLS_KEY(0) {\n pthread_key_create(&TLS_KEY, destroy_tls_data);\n }\n }; \n const tls_key_creator key; \n \n icache*& get_icache_ptr(const void* dht_ptr) {\n cache_map_type* cache_map_ptr = static_cast<cache_map_type*> \n (pthread_getspecific(key.TLS_KEY));\n if(cache_map_ptr == NULL) {\n cache_map_ptr = new cache_map_type();\n pthread_setspecific(key.TLS_KEY, cache_map_ptr);\n }\n ASSERT_NE(cache_map_ptr, NULL);\n ASSERT_NE(dht_ptr, NULL);\n return (*cache_map_ptr)[dht_ptr];\n }\n\n \n\n \n }; // end of delta dht impl\n}; // end of graphlab namespace\n\n\n\n\n\n\n\n\n\n" }, { "path": "src/graphlab/rpc/delta_dht.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n/**\n * Also contains code that is Copyright 2011 Yahoo! Inc. All rights\n * reserved. \n *\n * Contributed under the iCLA for:\n * Joseph Gonzalez (jegonzal@yahoo-inc.com) \n *\n */\n\n\n#ifndef GRAPHLAB_DELTA_DHT_HPP\n#define GRAPHLAB_DELTA_DHT_HPP\n\n\n#include <boost/unordered_map.hpp>\n#include <boost/functional/hash.hpp>\n\n\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/util/cache.hpp>\n\n\n\n\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n\n namespace delta_dht_impl {\n struct icache { virtual ~icache() { } };\n icache*& get_icache_ptr(const void* dht_ptr); \n }; // end of namespace delta_dht_impl\n\n\n namespace delta_predicate {\n template<typename ValueType, typename DeltaType>\n struct uses {\n size_t max_uses;\n uses(size_t max_uses = 100) : max_uses(max_uses) { }\n //! returns true if the predicate \n bool operator()(const ValueType& current,\n const DeltaType& delta,\n const size_t& uses) const {\n return uses < max_uses;\n }\n }; // end of uses\n\n }; // end of eviction predicates\n\n\n\n\n\n\n template<typename KeyType, typename ValueType,\n typename DeltaType = ValueType>\n class delta_dht {\n public:\n typedef KeyType key_type;\n typedef ValueType value_type;\n typedef DeltaType delta_type;\n\n typedef size_t size_type; \n \n typedef boost::unordered_map<key_type, value_type> data_map_type;\n \n struct cache_entry {\n value_type value;\n delta_type delta;\n size_t uses;\n cache_entry(const value_type& value = value_type()) : \n value(value), uses(0) { }\n };\n\n typedef cache::lru<key_type, cache_entry> cache_type;\n\n private:\n\n //! The remote procedure call manager \n mutable dc_dist_object<delta_dht> rpc;\n\n\n //! The data stored locally on this machine\n data_map_type data_map;\n\n //! The lock for the data map\n mutex data_lock;\n\n //! The master cache\n cache_type cache;\n\n //! The master cash rw lock\n mutex cache_lock;\n \n //! The maximum cache size\n size_t max_cache_size;\n\n size_t max_uses;\n\n //! the hash function\n boost::hash<key_type> hash_function;\n\n //! cache hits and misses\n mutable atomic<size_t> local; \n mutable atomic<size_t> hits;\n mutable atomic<size_t> misses;\n mutable atomic<size_t> background_updates;\n\n public:\n\n delta_dht(distributed_control& dc, \n size_t max_cache_size = 2056) : \n rpc(dc, this), \n max_cache_size(max_cache_size), max_uses(10) {\n rpc.barrier();\n }\n\n ~delta_dht() { rpc.full_barrier(); }\n \n void set_max_uses(size_t max) { max_uses = max; }\n\n size_t cache_local() const { return local.value; }\n size_t cache_hits() const { return hits.value; }\n size_t cache_misses() const { return misses.value; }\n size_t background_syncs() const { return background_updates.value; }\n\n size_t cache_size() const { \n cache_lock.lock();\n const size_t ret_val = cache.size(); \n cache_lock.unlock();\n return ret_val;\n }\n\n bool is_cached(const key_type& key) const { \n cache_lock.lock();\n const bool ret_value = cache.contains(key); \n cache_lock.unlock();\n return ret_value;\n }\n\n\n value_type operator[](const key_type& key) { \n if(is_local(key)) {\n ++local;\n data_lock.lock();\n const value_type value = data_map[key];\n data_lock.unlock();\n return value;\n } else { // on a remote machine check the cache \n // test for the key in the cache\n cache_lock.lock();\n if(cache.contains(key)) {\n ++hits;\n const value_type ret_value = cache[key].value;\n cache_lock.unlock();\n return ret_value;\n } else { // need to create a cache entry\n ++misses;\n // Free space in the cache if necessary\n while(cache.size() + 1 > max_cache_size) {\n ASSERT_GT(cache.size(), 0);\n const std::pair<key_type, cache_entry> pair = cache.evict();\n const key_type& key = pair.first;\n const cache_entry& entry = pair.second;\n send_delta(key, entry.delta);\n } \n // get the new entry from the server\n const value_type ret_value = (cache[key].value = get_master(key));\n cache_lock.unlock();\n return ret_value;\n }\n }\n } // end of operator []\n \n\n void apply_delta(const key_type& key, const delta_type& delta) {\n if(is_local(key)) {\n data_lock.lock();\n data_map[key] += delta;\n data_lock.unlock();\n } else {\n // update the cache entry if availablable\n cache_lock.lock();\n if(cache.contains(key)) {\n cache_entry& entry = cache[key];\n entry.value += delta;\n entry.delta += delta; \n if( entry.uses > max_uses ) {\n const delta_type accum_delta = entry.delta;\n entry.delta = delta_type();\n entry.uses = 0;\n cache_lock.unlock();\n send_delta(key, accum_delta);\n return;\n }\n }\n cache_lock.unlock(); \n }\n }\n\n\n\n //! empty the local cache\n void flush() {\n cache_lock.lock();\n while(cache.size() > 0) {\n const std::pair<key_type, cache_entry> pair = cache.evict();\n const key_type& key = pair.first;\n const cache_entry& entry = pair.second;\n send_delta(key, entry.delta);\n }\n cache_lock.unlock();\n }\n\n\n //! empty the local cache\n void barrier_flush() {\n flush();\n rpc.full_barrier();\n }\n \n \n void synchronize() {\n typedef typename cache_type::pair_type pair_type;\n cache_lock.lock();\n foreach(pair_type& pair, cache) {\n key_type& key = pair.first;\n cache_entry& entry = pair.second;\n if(entry.uses > 0) {\n const delta_type accum_delta = entry.delta;\n entry.delta = delta_type();\n entry.uses = 0;\n send_delta(key, accum_delta);\n }\n } // end of foreach\n cache_lock.unlock();\n }\n\n\n void synchronize(const key_type& key) {\n if(is_local(key)) return;\n cache_lock.lock();\n if(cache.contains(key)) {\n cache_entry& entry = cache[key];\n const delta_type accum_delta = entry.delta;\n entry.delta = delta_type();\n entry.uses = 0;\n cache_lock.unlock();\n send_delta(key, accum_delta);\n } else cache_lock.unlock();\n }\n\n\n size_t owning_cpu(const key_type& key) const {\n const size_t hash_value = hash_function(key);\n const size_t cpuid = hash_value % rpc.numprocs();\n return cpuid;\n }\n \n\n bool is_local(const key_type& key) const {\n return owning_cpu(key) == rpc.procid();\n } // end of is local \n\n\n delta_type delta(const key_type& key) const {\n if(!is_local(key)) {\n cache_lock.lock();\n if(cache.contains(key)) {\n const delta_type delta = cache[key].delta;\n cache_lock.unlock();\n return delta;\n }\n cache_lock.unlock();\n }\n return delta_type();\n }\n\n\n size_t local_size() const {\n data_lock.lock();\n const size_t result = data_map.size(); \n data_lock.unlock();\n return result;\n }\n\n\n size_t size() const {\n size_t sum = 0;\n for(size_t i = 0; i < rpc.numprocs(); ++i) {\n if(i == rpc.procid()) sum += local_size();\n else sum += rpc.remote_request(i, &delta_dht::local_size); \n }\n return sum;\n }\n\n size_t numprocs() const { return rpc.num_procs(); }\n size_t procid() const { return rpc.procid(); }\n\n\n value_type get_master(const key_type& key) {\n // If the data is stored locally just read and return\n if(is_local(key)) {\n data_lock.lock();\n const value_type ret_value = data_map[key];\n data_lock.unlock();\n return ret_value;\n } else {\n return rpc.remote_request(owning_cpu(key), \n &delta_dht::get_master, key);\n }\n } // end of direct get\n \n private:\n \n void send_delta(const key_type& key, const delta_type& delta) {\n // If the data is stored locally just read and return\n ASSERT_FALSE(is_local(key));\n const size_t calling_procid = procid();\n rpc.remote_call(owning_cpu(key), \n &delta_dht::send_delta_rpc, \n calling_procid, key, delta);\n \n } // end of send_delta\n \n void send_delta_rpc(const size_t& calling_procid, \n const key_type& key, const delta_type& delta) {\n // If the data is stored locally just read and return\n ASSERT_TRUE(is_local(key));\n data_lock.lock(); \n const value_type ret_value = (data_map[key] += delta);\n data_lock.unlock();\n rpc.remote_call(calling_procid, \n &delta_dht::send_delta_rpc_callback, key, ret_value); \n \n } // end of send_delta_rpc\n \n void send_delta_rpc_callback(const key_type& key, const value_type& new_value) {\n // If the data is stored locally just read and return\n ASSERT_FALSE(is_local(key));\n cache_lock.lock();\n if(cache.contains(key)) {\n cache_entry& entry = cache[key];\n entry.value = new_value;\n entry.value += entry.delta;\n }\n ++background_updates;\n cache_lock.unlock();\n } // end of send_delta_rpc_callback \n\n \n // void synchronize(const key_type& key, cache_entry& entry) {\n // const value_type delta = entry.current - entry.old;\n // entry.old = synchronize_rpc(key, delta);\n // entry.current = entry.old;\n // } // end of synchronize\n\n // value_type synchronize_rpc(const key_type& key, const value_type& delta) {\n // if(is_local(key)) {\n // data_lock.lock();\n // typename data_map_type::iterator iter = data_map.find(key);\n // ASSERT_TRUE(iter != data_map.end());\n // const value_type ret_value = (iter->second += delta);\n // data_lock.unlock();\n // return ret_value;\n // } else {\n // return rpc.remote_request(owning_cpu(key), \n // &delta_dht::synchronize_rpc, \n // key, delta);\n // }\n // } // end of synchronize_rpc\n\n }; // end of delta_dht\n\n\n}; // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n\n\n#endif\n\n\n" }, { "path": "src/graphlab/rpc/dht.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_DHT_HPP\n#define GRAPHLAB_DHT_HPP\n\n#include <boost/functional/hash.hpp>\n#include <boost/unordered_map.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/rpc/dc_dist_object.hpp>\n\nnamespace graphlab {\n\n /**\n * \\ingroup rpc\n * Implements a very rudimentary distributed key value store.\n */\n template <typename KeyType, typename ValueType>\n class dht { \n\n public:\n typedef boost::unordered_map<size_t, ValueType> storage_type;\n \n\n private:\n mutable dc_dist_object< dht > rpc;\n \n boost::hash<KeyType> hasher;\n mutex lock;\n storage_type storage;\n\n public:\n dht(distributed_control &dc) : rpc(dc, this) { }\n \n /**\n * Get the owner of the key\n */ \n procid_t owner(const KeyType& key) const {\n return hasher(key) % rpc.dc().numprocs();\n }\n \n /**\n * gets the value associated with a key.\n * Returns (true, Value) if the entry is available.\n * Returns (false, undefined) otherwise.\n */\n std::pair<bool, ValueType> get(const KeyType &key) const {\n // who owns the data?\n\n const size_t hashvalue = hasher(key);\n const size_t owningmachine = hashvalue % rpc.numprocs();\n std::pair<bool, ValueType> retval;\n // if it is me, we can return it\n if (owningmachine == rpc.dc().procid()) {\n\n lock.lock();\n typename storage_type::const_iterator iter = storage.find(hashvalue);\n retval.first = iter != storage.end();\n if (retval.first) retval.second = iter->second;\n lock.unlock();\n } else {\n retval = rpc.remote_request(owningmachine, \n &dht<KeyType,ValueType>::get, \n key);\n }\n return retval;\n }\n \n /**\n * gets the value associated with a key.\n * Returns (true, Value) if the entry is available.\n * Returns (false, undefined) otherwise.\n */\n request_future<std::pair<bool, ValueType> > get_future(const KeyType &key) const {\n // who owns the data?\n\n const size_t hashvalue = hasher(key);\n const size_t owningmachine = hashvalue % rpc.numprocs();\n std::pair<bool, ValueType> retval;\n // if it is me, we can return it\n if (owningmachine == rpc.dc().procid()) {\n\n lock.lock();\n typename storage_type::const_iterator iter = storage.find(hashvalue);\n retval.first = iter != storage.end();\n if (retval.first) retval.second = iter->second;\n lock.unlock();\n return retval;\n } else {\n return rpc.future_remote_request(owningmachine, \n &dht<KeyType,ValueType>::get, \n key);\n }\n }\n \n\n\n\n /**\n * Sets the newval to be the value associated with the key\n */\n void set(const KeyType &key, const ValueType &newval) { \n // who owns the data?\n const size_t hashvalue = hasher(key);\n const size_t owningmachine = hashvalue % rpc.numprocs();\n \n // if it is me, set it\n if (owningmachine == rpc.dc().procid()) {\n lock.lock();\n storage[hashvalue] = newval;\n lock.unlock();\n } else {\n rpc.remote_call(owningmachine, \n &dht<KeyType,ValueType>::set, \n key, newval);\n }\n }\n \n void print_stats() const {\n std::cerr << rpc.calls_sent() << \" calls sent\\n\";\n std::cerr << rpc.calls_received() << \" calls received\\n\";\n }\n \n /**\n Must be called by all machines simultaneously\n */\n void clear() {\n rpc.barrier();\n storage.clear();\n }\n\n };\n\n};\n#endif\n\n" }, { "path": "src/graphlab/rpc/distributed_event_log.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n#include <pthread.h>\n#include <string>\n#include <limits>\n#include <cfloat>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_dist_object.hpp>\n#include <graphlab/rpc/distributed_event_log.hpp>\n#include <graphlab/util/timer.hpp>\n#include <graphlab/logger/assertions.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n#include <graphlab/ui/metrics_server.hpp>\n#include <graphlab/macros_def.hpp>\n#define DISABLE_DISTRIBUTED_EVENT_LOG\nnamespace graphlab {\n\n\n// predeclaration the metric server handlers\nstatic std::pair<std::string, std::string> \nmetric_names_json(std::map<std::string, std::string>& vars);\n\nstatic std::pair<std::string, std::string> \nmetric_aggregate_json(std::map<std::string, std::string>& vars);\n\nstatic std::pair<std::string, std::string> \nmetric_by_machine_json(std::map<std::string, std::string>& vars);\n\n\nstatic size_t time_to_index(double t) {\n return std::floor(t / 5);\n}\n\nstatic double index_to_time(size_t t) {\n return 5 * t;\n}\n\n\nsize_t distributed_event_logger::allocate_log_entry(log_group* group) {\n log_entry_lock.lock();\n size_t id = 0;\n if (has_log_entry.first_zero_bit(id) == false) {\n logger(LOG_FATAL, \"More than 256 Log entries created. \"\n \"New log entries cannot be created\");\n // does not return\n }\n logs[id] = group;\n has_log_entry.set_bit(id);\n log_entry_lock.unlock();\n return id;\n}\n\nevent_log_thread_local_type* distributed_event_logger::get_thread_counter_ref() {\n void* v = pthread_getspecific(key);\n if (v == NULL) {\n // allocate a new thread local entry\n event_log_thread_local_type* entry = new event_log_thread_local_type;\n // set all values to 0\n for (size_t i = 0; i < MAX_LOG_SIZE; ++i) entry->values[i] = 0;\n // cast and write it to v. We need it later. \n // and set the thread local store\n v = (void*)(entry);\n pthread_setspecific(key, v);\n\n // register the key entry against the logger\n thread_local_count_lock.lock();\n // find an unused entry\n size_t b = 0;\n if (thread_local_count_slots.first_zero_bit(b) == false) {\n logger(LOG_FATAL, \"More than 1024 active threads. \"\n \"Log counters cannot be created\");\n // does not return\n }\n entry->thlocal_slot = b;\n thread_local_count[b] = entry;\n thread_local_count_slots.set_bit(b);\n thread_local_count_lock.unlock();\n }\n\n event_log_thread_local_type* entry = (event_log_thread_local_type*)(v);\n return entry;\n}\n\n/**\n * Receives the log information from each machine\n */ \nvoid distributed_event_logger::rpc_collect_log(size_t srcproc, size_t record_ctr,\n std::vector<double> srccounts) {\n foreach(size_t log, has_log_entry) {\n logs[log]->lock.lock();\n // insert the new counts\n size_t entryid = record_ctr;\n logs[log]->earliest_modified_log = \n std::min(entryid, logs[log]->earliest_modified_log);\n logs[log]->machine_log_modified = true;\n // resize all procs\n for (procid_t p = 0; p < logs[log]->machine.size(); ++p) {\n if (logs[log]->machine[p].size() < entryid + 1) {\n double prevvalue = 0;\n if (logs[log]->machine[p].size() > 0) {\n prevvalue = logs[log]->machine[p].back().value;\n }\n logs[log]->machine[p].resize(entryid + 1, log_entry(prevvalue));\n }\n }\n logs[log]->machine[srcproc][entryid].value = srccounts[log];\n logs[log]->lock.unlock();\n }\n}\n\nvoid distributed_event_logger::collect_instantaneous_log() {\n foreach(size_t log, has_log_entry) {\n if (logs[log]->logtype == log_type::INSTANTANEOUS) {\n logs[log]->lock.lock();\n // for each log entry which is a callback entry\n // call the callback to get the counts\n if (logs[log]->is_callback_entry) {\n logs[log]->sum_of_instantaneous_entries += logs[log]->callback();\n ++logs[log]->count_of_instantaneous_entries;\n }\n else {\n // sum it across all the threads\n foreach(size_t thr, thread_local_count_slots) {\n logs[log]->sum_of_instantaneous_entries += thread_local_count[thr]->values[log];\n }\n ++logs[log]->count_of_instantaneous_entries;\n }\n logs[log]->lock.unlock();\n }\n }\n}\n\n/** \n * Collects the machine level\n * log entry. and sends it to machine 0\n */\nvoid distributed_event_logger::local_collect_log(size_t record_ctr) {\n // put together an aggregate of all counters \n std::vector<double> combined_counts(MAX_LOG_SIZE, 0);\n\n // for each thread and for each log entry which is \n // not a callback entry. Accumulate the number of counts\n //\n foreach(size_t log, has_log_entry) {\n logs[log]->lock.lock();\n // cimulative entry. just add across all threads\n if (logs[log]->logtype == log_type::CUMULATIVE) {\n if (logs[log]->is_callback_entry) {\n combined_counts[log] = logs[log]->callback();\n } else {\n foreach(size_t thr, thread_local_count_slots) {\n size_t* current_thread_counts = thread_local_count[thr]->values;\n combined_counts[log] += current_thread_counts[log];\n }\n }\n }\n else {\n // take the average \n if (logs[log]->count_of_instantaneous_entries > 0) {\n combined_counts[log] = (double)logs[log]->sum_of_instantaneous_entries / \n logs[log]->count_of_instantaneous_entries;\n }\n else {\n combined_counts[log] = 0;\n }\n logs[log]->sum_of_instantaneous_entries = 0;\n logs[log]->count_of_instantaneous_entries = 0;\n }\n logs[log]->lock.unlock();\n }\n\n // send to machine 0\n if (rmi->procid() != 0) {\n rmi->control_call(0, &distributed_event_logger::rpc_collect_log,\n (size_t)rmi->procid(), record_ctr, combined_counts);\n }\n else {\n rpc_collect_log((size_t)0, record_ctr, combined_counts);\n }\n}\n\n// Called only by machine 0 to get the aggregate log\nvoid distributed_event_logger::build_aggregate_log() {\n ASSERT_EQ(rmi->procid(), 0);\n foreach(size_t log, has_log_entry) {\n logs[log]->lock.lock();\n if (logs[log]->machine_log_modified) {\n // what is the previous time the aggregate was computed?\n // The sum takes the open interval (prevtime, current_time]\n // thus the first time this is called, we may drop one entry\n // if we let prevtime initialize at 0\n size_t prevtime = logs[log]->earliest_modified_log;\n size_t lasttime = prevtime + 1;\n for (procid_t p = 0; p < logs[log]->machine.size(); ++p) {\n lasttime = std::max(lasttime, logs[log]->machine[p].size());\n }\n // if it is a CUMULATIVE log, take the latest entry from each machine\n // if it is an INSTANTANEOUS log, take the average of the last times.\n if (logs[log]->aggregate.size() < lasttime) {\n if (logs[log]->logtype == log_type::CUMULATIVE) {\n double lastval = 0;\n if (logs[log]->aggregate.size() > 0) {\n lastval = logs[log]->aggregate.rbegin()->value;\n }\n logs[log]->aggregate.resize(lasttime, log_entry(lastval));\n }\n else {\n logs[log]->aggregate.resize(lasttime);\n }\n }\n \n for (size_t t = prevtime; t < lasttime; ++t) {\n double sum = 0;\n for (procid_t p = 0; p < logs[log]->machine.size(); ++p) {\n if (t < logs[log]->machine[p].size()) {\n sum += logs[log]->machine[p][t].value;\n }\n }\n logs[log]->aggregate[t].value = sum;\n }\n logs[log]->earliest_modified_log = (size_t)(-1);\n logs[log]->machine_log_modified = false;\n }\n logs[log]->lock.unlock();\n }\n}\n\nvoid distributed_event_logger::periodic_timer() {\n periodic_timer_lock.lock();\n timer ti; ti.start();\n int tick_ctr = 0;\n int record_ctr = 0;\n\n int ticks_per_record = RECORD_FREQUENCY / TICK_FREQUENCY;\n\n while (!periodic_timer_stop){ \n collect_instantaneous_log();\n if (tick_ctr % ticks_per_record == 0) {\n local_collect_log(record_ctr);\n ++record_ctr;\n if (rmi->procid() == 0) build_aggregate_log();\n }\n // when is the next tick\n ++tick_ctr;\n int nexttick_time = tick_ctr * 1000 * TICK_FREQUENCY;\n int nexttick_interval = nexttick_time - ti.current_time_millis();\n // we lost a tick.\n if (nexttick_interval < 10) continue;\n periodic_timer_cond.timedwait_ms(periodic_timer_lock, nexttick_interval);\n }\n periodic_timer_lock.unlock();\n}\n\ndistributed_event_logger::distributed_event_logger():rmi(NULL) {\n pthread_key_create(&key, NULL);\n // clear the bit fields\n has_log_entry.clear();\n thread_local_count_slots.clear();\n periodic_timer_stop = false;\n}\n\nvoid distributed_event_logger::destroy_event_logger() {\n // kill the tick thread\n bool thread_was_started = false;\n periodic_timer_lock.lock();\n // if periodic_timer_stop is false, then\n // thread was started. signal it and wait for it later to \n // join\n if (periodic_timer_stop == false) {\n periodic_timer_stop = true;\n thread_was_started = true;\n periodic_timer_cond.signal();\n }\n periodic_timer_lock.unlock();\n if (thread_was_started) tick_thread.join();\n // make sure everyone has joined before I start freeing stuff\n rmi->full_barrier();\n delete rmi;\n pthread_key_delete(key);\n // here also free all the allocated memory!\n foreach(size_t thr, thread_local_count_slots) {\n if (thread_local_count[thr] != NULL) delete thread_local_count[thr];\n }\n foreach(size_t log, has_log_entry) {\n if (logs[log] != NULL) delete logs[log];\n }\n\n\n}\n\nvoid distributed_event_logger::set_dc(distributed_control& dc) {\n if (rmi == NULL) {\n rmi = new dc_dist_object<distributed_event_logger>(dc, this);\n // register a deletion callback since the distributed_event_logger\n // will be destroyed only after main\n\n dc.register_deletion_callback(boost::bind(\n &distributed_event_logger::destroy_event_logger, \n this));\n\n dc.barrier();\n // everyone starts the timer at the same time\n // at the one distributed synchronization point we have\n ti.start();\n // procid 0 waits 0.2s to skew the local timer a little\n // so everyone else's log has time to show up\n if (rmi->procid() == 0) {\n timer::sleep_ms(200);\n }\n periodic_timer_stop = false;\n // spawn a thread for the tick\n#ifndef DISABLE_DISTRIBUTED_EVENT_LOG\n tick_thread.launch(boost::bind(&distributed_event_logger::periodic_timer,\n this));\n#endif\n // register the metric server callbacks\n add_metric_server_callback(\"names.json\", metric_names_json);\n add_metric_server_callback(\"metrics_aggregate.json\", metric_aggregate_json);\n add_metric_server_callback(\"metrics_by_machine.json\", metric_by_machine_json);\n }\n}\n \nsize_t distributed_event_logger::create_log_entry(std::string name, \n std::string units,\n log_type::log_type_enum logtype) {\n // look for an entry with the same name\n bool has_existing = false;\n size_t existingid = 0;\n log_entry_lock.lock();\n foreach(size_t log, has_log_entry) {\n if (logs[log]->name == name) {\n ASSERT_MSG(logs[log]->is_callback_entry == false,\n \"Cannot convert callback log to counter log\");\n has_existing = true;\n existingid = log;\n break;\n }\n }\n log_entry_lock.unlock();\n if (has_existing) return existingid;\n\n log_group* group = new log_group;\n group->logtype = logtype;\n group->name = name;\n group->units = units;\n group->callback = NULL;\n group->is_callback_entry = false;\n group->earliest_modified_log = 1;\n group->machine_log_modified = false;\n group->sum_of_instantaneous_entries = 0.0;\n group->count_of_instantaneous_entries = 0;\n // only allocate the machine vector on the root machine.\n // no one else needs it \n if (rmi->procid() == 0) {\n group->machine.resize(rmi->numprocs());\n } \n // ok. get an ID\n size_t id = allocate_log_entry(group);\n // enforce that all machines are running this at the same time \n rmi->barrier();\n return id;\n}\n\nsize_t distributed_event_logger::create_callback_entry(std::string name, \n std::string units,\n boost::function<double(void)> callback,\n log_type::log_type_enum logtype) {\n bool has_existing = false;\n size_t existingid = 0;\n log_entry_lock.lock();\n foreach(size_t log, has_log_entry) {\n if (logs[log]->name == name) {\n has_existing = true;\n existingid = log;\n break;\n }\n }\n log_entry_lock.unlock();\n if (has_existing) {\n // ok... we have an existing entry. We may\n // overwrite the callback if the callback is NULL\n ASSERT_MSG(logs[existingid]->is_callback_entry == true,\n \"Cannot convert counter log to callback log\");\n\n logs[existingid]->lock.lock();\n ASSERT_MSG(logs[existingid]->callback == NULL, \n \"Cannot create another callback log entry with\"\n \"the same name %s\", name.c_str());\n logs[existingid]->callback = callback;\n logs[existingid]->lock.unlock();\n return existingid;\n }\n\n log_group* group = new log_group;\n group->logtype = logtype;\n group->name = name;\n group->units = units;\n group->earliest_modified_log = 0;\n group->machine_log_modified = false;\n group->callback = callback;\n group->is_callback_entry = true;\n group->sum_of_instantaneous_entries = 0.0;\n group->count_of_instantaneous_entries = 0;\n\n // only allocate the machine vector on the root machine.\n // no one else needs it \n if (rmi->procid() == 0) {\n group->machine.resize(rmi->numprocs());\n } \n // ok. get an ID\n size_t id = allocate_log_entry(group);\n // enforce that all machines are running this at the same time \n rmi->barrier();\n return id;\n}\n\nvoid distributed_event_logger::thr_inc_log_entry(size_t entry, size_t value) {\n event_log_thread_local_type* ev = get_thread_counter_ref();\n ASSERT_LT(entry, MAX_LOG_SIZE);\n ASSERT_EQ(logs[entry]->is_callback_entry, false);\n ev->values[entry] += value;\n}\n\nvoid distributed_event_logger::thr_dec_log_entry(size_t entry, size_t value) {\n event_log_thread_local_type* ev = get_thread_counter_ref();\n ASSERT_LT(entry, MAX_LOG_SIZE);\n // does not work for cumulative logs\n ASSERT_NE((int)logs[entry]->logtype, (int) log_type::CUMULATIVE);\n ASSERT_EQ(logs[entry]->is_callback_entry, false);\n ev->values[entry] -= value;\n}\n\n\nvoid distributed_event_logger::free_callback_entry(size_t entry) {\n ASSERT_LT(entry, MAX_LOG_SIZE);\n // does not work for cumulative logs\n logs[entry]->lock.lock();\n ASSERT_EQ(logs[entry]->is_callback_entry, true);\n logs[entry]->callback = NULL;\n logs[entry]->lock.unlock();\n}\n\ndistributed_event_logger& get_event_log() {\n static distributed_event_logger dist_event_log;\n return dist_event_log;\n}\n\n\n\n\n\n/*\n Used to process the names.json request\n*/\nstd::pair<std::string, std::string> \nstatic metric_names_json(std::map<std::string, std::string>& vars) {\n std::stringstream strm;\n char *pname = getenv(\"_\");\n std::string progname;\n if (pname != NULL) progname = pname;\n\n\n distributed_event_logger& evlog = get_event_log();\n log_group** logs = evlog.get_logs_ptr();\n fixed_dense_bitset<MAX_LOG_SIZE>& has_log_entry = evlog.get_logs_bitset();\n\n strm << \"{\\n\"\n << \" \\\"program_name\\\": \\\"\"<< progname << \"\\\",\\n\"\n << \" \\\"time\\\": \" << evlog.get_current_time() << \",\\n\"\n << \" \\\"metrics\\\": [\\n\";\n // output the metrics\n size_t nlogs = has_log_entry.popcount();\n\n size_t logcount = 0;\n foreach(size_t log, has_log_entry) {\n\n logs[log]->lock.lock();\n double rate_val = 0;\n size_t len = logs[log]->aggregate.size();\n if (len >= 1) { \n double logtime = index_to_time(logs[log]->aggregate.size() - 1);\n double logval = logs[log]->aggregate.rbegin()->value;\n double prevtime = 0;\n double prevval = 0;\n if (logs[log]->aggregate.size() >= 2) {\n prevtime = index_to_time(logs[log]->aggregate.size() - 2);\n prevval = logs[log]->aggregate[len - 2].value;\n }\n if (logs[log]->logtype == log_type::CUMULATIVE) {\n rate_val = (logval - prevval) / (logtime - prevtime);\n }\n else {\n rate_val = logval;\n }\n }\n\n strm << \" {\\n\"\n << \" \\\"id\\\":\" << log << \",\\n\"\n << \" \\\"name\\\": \\\"\" << logs[log]->name << \"\\\",\\n\"\n << \" \\\"units\\\": \\\"\" << logs[log]->units << \"\\\",\\n\"\n << \" \\\"cumulative\\\": \" << (int)(logs[log]->logtype) << \",\\n\"\n << \" \\\"rate_val\\\": \" << rate_val << \",\\n\"\n << \" \\\"value\\\": \" << ( logs[log]->aggregate.size() > 0 ?\n logs[log]->aggregate.rbegin()->value \n : 0 ) << \"\\n\"\n << \" }\\n\";\n\n logs[log]->lock.unlock();\n ++logcount;\n if (logcount < nlogs) strm << \",\";\n }\n strm << \" ]\\n\"\n << \"}\\n\";\n\n return std::make_pair(std::string(\"text/plain\"), strm.str());\n}\n\nstd::pair<std::string, std::string> \nstatic metric_aggregate_json(std::map<std::string, std::string>& vars) {\n double tstart = 0;\n double tend = DBL_MAX;\n bool rate = false;\n std::string name;\n // see what variables there are\n\n size_t idxstart = time_to_index(tstart);\n size_t idxend = (size_t)(-1);\n if (vars.count(\"name\")) name = vars[\"name\"];\n if (vars.count(\"tstart\")) {\n tstart = atof(vars[\"tstart\"].c_str());\n idxstart = time_to_index(tstart);\n }\n if (vars.count(\"tend\")) {\n tend = atof(vars[\"tend\"].c_str());\n idxend = time_to_index(tend) + 1;\n }\n if (vars.count(\"rate\")) rate = (atoi(vars[\"rate\"].c_str()) != 0);\n if (vars.count(\"tlast\")) {\n double tlast = atof(vars[\"tlast\"].c_str());\n tstart = get_event_log().get_current_time() - tlast;\n tstart = tstart < 0.0 ? 0.0 : tstart;\n tend = get_event_log().get_current_time();\n idxstart = time_to_index(tstart);\n idxend = time_to_index(tend) + 1;\n }\n\n // name is not optional\n name = trim(name);\n\n distributed_event_logger& evlog = get_event_log();\n log_group** logs = evlog.get_logs_ptr();\n fixed_dense_bitset<MAX_LOG_SIZE>& has_log_entry = evlog.get_logs_bitset();\n\n std::stringstream strm;\n\n size_t nlogs = has_log_entry.popcount();\n size_t logcount = 0;\n \n // if name is empty, I should extract all metrics\n bool extract_all = (name.length() == 0);\n\n // make a top level array\n\n strm << \"[\\n\";\n foreach(size_t log, has_log_entry) {\n\n if (logs[log]->name == name || extract_all) {\n\n logs[log]->lock.lock();\n strm << \" {\\n\"\n << \" \\\"id\\\":\" << log << \",\\n\"\n << \" \\\"name\\\": \\\"\" << logs[log]->name << \"\\\",\\n\"\n << \" \\\"units\\\": \\\"\" << logs[log]->units << \"\\\",\\n\"\n << \" \\\"name\\\": \\\"\" << logs[log]->name << \"\\\",\\n\"\n << \" \\\"cumulative\\\": \" << (int)(logs[log]->logtype) << \",\\n\"\n << \" \\\"record\\\": [\";\n\n std::vector<std::pair<double, double> > output_entries;\n // annoyingly, json does not let me put a trailing comma in the array.\n // thus I need to first write it to a vector, before dumping it to json\n size_t log_idxend = std::min(idxend, logs[log]->aggregate.size());\n for (size_t i = idxstart; i < log_idxend ; ++i) {\n double logtime = index_to_time(i);\n double logval = logs[log]->aggregate[i].value;\n // only cumulative logs can have rate\n if (rate == 0 || logs[log]->logtype == log_type::INSTANTANEOUS) {\n output_entries.push_back(std::make_pair(logtime, logval));\n }\n else {\n double prevval = 0;\n double prevtime = 0;\n if (i > 0) {\n prevtime = index_to_time(i - 1);\n prevval = logs[log]->aggregate[i - 1].value;\n }\n double currate = 0;\n // avoid divide by zero annoyances\n if (logtime > prevtime) {\n currate = (logval - prevval) / (logtime - prevtime);\n }\n output_entries.push_back(std::make_pair(logtime, currate));\n }\n }\n\n logs[log]->lock.unlock();\n for (size_t i = 0 ;i < output_entries.size(); ++i) {\n strm << \" [\" \n << output_entries[i].first << \", \" \n << output_entries[i].second \n << \"] \";\n // add a comma if this is not the last entry\n if (i < output_entries.size() - 1) strm << \", \";\n }\n strm << \"]\\n\"\n << \" }\\n\";\n\n // if I am not supposed to extract all, then I am done here.\n if (!extract_all) break;\n ++logcount;\n if (logcount < nlogs) strm << \",\\n\";\n }\n\n }\n\n strm << \"]\\n\";\n return std::make_pair(std::string(\"text/plain\"), strm.str());\n}\n\n\n\n\nstd::pair<std::string, std::string> \nstatic metric_by_machine_json(std::map<std::string, std::string>& vars) {\n double tstart = 0;\n double tend = DBL_MAX;\n bool rate = false;\n std::string name;\n size_t machine = 0;\n bool has_machine_filter = false;\n // see what variables there are\n size_t idxstart = 0;\n size_t idxend = (size_t)(-1);\n \n if (vars.count(\"name\")) name = vars[\"name\"];\n if (vars.count(\"machine\")) {\n has_machine_filter = true;\n machine = atoi(vars[\"machine\"].c_str());\n }\n if (vars.count(\"tstart\")) {\n tstart = atof(vars[\"tstart\"].c_str());\n idxstart = time_to_index(tstart);\n }\n if (vars.count(\"tend\")) {\n tend = atof(vars[\"tend\"].c_str());\n idxend = time_to_index(tend) + 1;\n }\n if (vars.count(\"rate\")) rate = (atoi(vars[\"rate\"].c_str()) != 0);\n if (vars.count(\"tlast\")) {\n double tlast = atof(vars[\"tlast\"].c_str());\n tstart = get_event_log().get_current_time() - tlast;\n tstart = tstart < 0.0 ? 0.0 : tstart;\n tend = get_event_log().get_current_time();\n idxstart = time_to_index(tstart);\n idxend = time_to_index(tend) + 1;\n }\n\n\n // name is not optional\n name = trim(name);\n\n distributed_event_logger& evlog = get_event_log();\n log_group** logs = evlog.get_logs_ptr();\n fixed_dense_bitset<MAX_LOG_SIZE>& has_log_entry = evlog.get_logs_bitset();\n\n std::stringstream strm;\n\n size_t nlogs = has_log_entry.popcount();\n size_t logcount = 0;\n \n // if name is empty, I should extract all metrics\n bool extract_all = (name.length() == 0);\n\n // make a top level array\n\n strm << \"[\\n\";\n foreach(size_t log, has_log_entry) {\n if (logs[log]->name == name || extract_all) {\n\n logs[log]->lock.lock();\n strm << \" {\\n\"\n << \" \\\"id\\\":\" << log << \",\\n\"\n << \" \\\"name\\\": \\\"\" << logs[log]->name << \"\\\",\\n\"\n << \" \\\"units\\\": \\\"\" << logs[log]->units << \"\\\",\\n\"\n << \" \\\"cumulative\\\": \" << (int)(logs[log]->logtype) << \",\\n\"\n << \" \\\"record\\\": \";\n \n std::vector<std::vector<std::pair<double, double> > > all_output_entries;\n // annoyingly, json does not let me put a trailing comma in the array.\n // thus I need to first write it to a vector, before dumping it to json\n // and annoying 2 dimensional output arrays...\n //\n size_t p_start = 0;\n size_t p_end = logs[log]->machine.size();\n if (has_machine_filter) {\n p_start = machine;\n p_end = machine + 1;\n }\n for (size_t p = p_start; p < p_end; ++p) {\n std::vector<log_entry>& current = logs[log]->machine[p];\n std::vector<std::pair<double, double> > output_entries;\n\n size_t log_idxend = std::min(idxend, current.size());\n for (size_t i = idxstart; i < log_idxend; ++i) {\n double logtime = index_to_time(i);\n double logval = current[i].value;\n \n if (logtime > tstart && logtime <= tend) {\n // only cumulative logs can have rate\n if (rate == 0 || logs[log]->logtype == log_type::INSTANTANEOUS) {\n output_entries.push_back(std::make_pair(logtime, logval));\n }\n else {\n double prevval = 0;\n double prevtime = 0;\n if (i > 0) {\n prevtime = index_to_time(i - 1);\n prevval = current[i - 1].value;\n }\n double currate = 0;\n // avoid divide by zero annoyances\n if (logtime > prevtime) {\n currate = (logval - prevval) / (logtime - prevtime);\n }\n output_entries.push_back(std::make_pair(logtime, currate));\n }\n }\n }\n all_output_entries.push_back(output_entries);\n }\n\n logs[log]->lock.unlock();\n strm << \"[ \";\n for (size_t p = 0; p < all_output_entries.size(); ++p) {\n std::vector<std::pair<double, double> >& output_entries = all_output_entries[p];\n strm << \"[ \";\n for (size_t i = 0 ;i < output_entries.size(); ++i) {\n strm << \" [\" \n << output_entries[i].first << \", \" \n << output_entries[i].second \n << \"] \";\n // add a comma if this is not the last entry\n if (i < output_entries.size() - 1) strm << \", \";\n }\n\n strm << \"] \";\n if (p < all_output_entries.size() - 1) strm << \", \";\n }\n strm << \"]\\n\"\n << \" }\\n\";\n\n\n // if I am not supposed to extract all, then I am done here.\n if (!extract_all) break;\n ++logcount;\n if (logcount < nlogs) strm << \",\\n\";\n }\n }\n\n strm << \"]\\n\";\n return std::make_pair(std::string(\"text/plain\"), strm.str());\n}\n\n\n\n\n} // namespace graphlab\n" }, { "path": "src/graphlab/rpc/distributed_event_log.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_DISTRIBUTED_EVENT_LOG_HPP\n#define GRAPHLAB_DISTRIBUTED_EVENT_LOG_HPP\n#include <iostream>\n#include <string>\n#include <vector>\n#include <boost/bind.hpp>\n#include <boost/function.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/parallel/atomic.hpp>\n#include <graphlab/util/timer.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n#include <graphlab/util/stl_util.hpp>\nnamespace graphlab {\n\n// forward declaration because we need this in the\n// class but we want dc_dist_object to be able\n// to use this class too.\ntemplate <typename T>\nclass dc_dist_object;\nclass distributed_control;\n\n\n\nconst size_t MAX_LOG_SIZE = 256;\nconst size_t MAX_LOG_THREADS = 1024;\nconst double TICK_FREQUENCY = 0.5;\nconst double RECORD_FREQUENCY = 5.0;\n\n\n\n/// A single entry in time\nstruct log_entry: public IS_POD_TYPE {\n // The value at the time. If this is a CUMULATIVE entry, this\n // will contain the total number of events since the start\n double value;\n\n explicit log_entry(double value = 0): value(value) { }\n};\n\n\nnamespace log_type {\nenum log_type_enum {\n INSTANTANEOUS = 0, ///< Sum of log values over time are not meaningful \n CUMULATIVE = 1 ///< Sum of log values over time are meaningful \n};\n}\n\n/// Logging information for a particular log entry (say \\#updates)\nstruct log_group{\n mutex lock;\n\n /// name of the group\n std::string name;\n\n /// unit of measurement\n std::string units;\n\n /// Set to true if this is a callback entry\n bool is_callback_entry;\n\n /// The type of log. Instantaneous or Cumulative \n log_type::log_type_enum logtype;\n\n boost::function<double(void)> callback;\n\n size_t sum_of_instantaneous_entries;\n size_t count_of_instantaneous_entries;\n\n bool machine_log_modified; \n size_t earliest_modified_log;\n\n /// machine[i] holds a vector of entries from machine i\n std::vector<std::vector<log_entry> > machine;\n /// aggregate holds vector of totals\n std::vector<log_entry> aggregate;\n};\n\n\n/**\n * This is the type that is held in the thread local store\n */\nstruct event_log_thread_local_type {\n /** The values written to by each thread. \n * An array with max length MAX_LOG_SIZE \n */\n size_t values[MAX_LOG_SIZE];\n size_t thlocal_slot;\n\n // These are used for time averaging instantaneous values\n};\n\n\nclass distributed_event_logger {\n private:\n // a key to allow multiple threads, each to have their\n // own counter. Decreases performance penalty of the\n // the event logger.\n pthread_key_t key;\n\n dc_dist_object<distributed_event_logger>* rmi;\n \n // The array of logs. We can only have a maximum of MAX_LOG_SIZE logs\n // This is only created on machine 0\n log_group* logs[MAX_LOG_SIZE];\n // this bit field is used to identify which log entries are active\n fixed_dense_bitset<MAX_LOG_SIZE> has_log_entry;\n mutex log_entry_lock;\n\n // A collection of slots, one for each thread, to hold \n // the current thread's active log counter.\n // Threads will write directly into here\n // and a master timer will sum it all up periodically\n event_log_thread_local_type* thread_local_count[MAX_LOG_THREADS];\n // a bitset which lets me identify which slots in thread_local_counts\n // are used.\n fixed_dense_bitset<MAX_LOG_THREADS> thread_local_count_slots; \n mutex thread_local_count_lock;\n\n // timer managing the frequency at which logs are transmitted to the root\n timer ti; \n thread tick_thread;\n\n size_t allocate_log_entry(log_group* group);\n /**\n * Returns a pointer to the current thread log counter\n * creating one if one does not already exist.\n */\n event_log_thread_local_type* get_thread_counter_ref();\n\n /**\n * Receives the log information from each machine\n */ \n void rpc_collect_log(size_t srcproc, size_t record_ctr,\n std::vector<double> srccounts);\n\n void collect_instantaneous_log(); \n /** \n * Collects the machine level\n * log entry. and sends it to machine 0\n */\n void local_collect_log(size_t record_ctr); \n \n // Called only by machine 0 to get the aggregate log\n void build_aggregate_log();\n\n mutex periodic_timer_lock;\n conditional periodic_timer_cond;\n bool periodic_timer_stop;\n\n /** a new thread spawns here and sleeps for 5 seconds at a time\n * when it wakes up it will insert log entries\n */\n void periodic_timer();\n public:\n distributed_event_logger();\n\n // called by the destruction of distributed_control\n void destroy_event_logger();\n\n\n /**\n * Associates the event log with a DC object.\n * Must be called by all machines simultaneously.\n * Can be called more than once, but only the first call will have\n * an effect.\n */\n void set_dc(distributed_control& dc);\n /**\n * Creates a new log entry with a given name and log type.\n * Returns the ID of the log. Must be called by \n * all machines simultaneously with the same settings.\n * units is the unit of measurement.\n */\n size_t create_log_entry(std::string name, std::string units, \n log_type::log_type_enum logtype);\n\n /**\n * Creates a new callback log entry with a given name and log type.\n * Returns the ID of the log. Must be called by \n * all machines simultaneously with the same settings.\n * units is the unit of measurement.\n * Callback will be triggered periodically.\n * Callback entries must be deleted once the callback goes\n * out of scope.\n */\n size_t create_callback_entry(std::string name, \n std::string units,\n boost::function<double(void)> callback,\n log_type::log_type_enum logtype);\n\n void free_callback_entry(size_t entry);\n\n /**\n * Increments the value of a log entry\n */\n void thr_inc_log_entry(size_t entry, size_t value);\n\n /**\n * Increments the value of a log entry\n */\n void thr_dec_log_entry(size_t entry, size_t value);\n\n\n /// \\cond GRAPHLAB_INTERNAL\n inline double get_current_time() const {\n return ti.current_time();\n }\n\n inline log_group** get_logs_ptr() {\n return logs;\n }\n\n inline fixed_dense_bitset<MAX_LOG_SIZE>& get_logs_bitset() {\n return has_log_entry;\n }\n\n /// \\endcond\n \n};\n\n\nextern distributed_event_logger& get_event_log();\n\n\n} // namespace graphlab\n#define DECLARE_EVENT(name) size_t name;\n\n#define INITIALIZE_EVENT_LOG(dc) graphlab::get_event_log().set_dc(dc);\n#define ADD_CUMULATIVE_EVENT(name, desc, units) \\\n name = graphlab::get_event_log().create_log_entry(desc, units, graphlab::log_type::CUMULATIVE);\n\n#define ADD_INSTANTANEOUS_EVENT(name, desc, units) \\\n name = graphlab::get_event_log().create_log_entry(desc, units, graphlab::log_type::INSTANTANEOUS);\n\n#define ADD_CUMULATIVE_CALLBACK_EVENT(name, desc, units, callback) \\\n name = graphlab::get_event_log().create_callback_entry(desc, units, callback, \\\n graphlab::log_type::CUMULATIVE);\n\n\n#define ADD_INSTANTANEOUS_CALLBACK_EVENT(name, desc, units, callback) \\\n name = graphlab::get_event_log().create_callback_entry(desc, units, callback, \\\n graphlab::log_type::INSTANTANEOUS);\n\n\n\n#define FREE_CALLBACK_EVENT(name) \\\n graphlab::get_event_log().free_callback_entry(name);\n\n#define INCREMENT_EVENT(name, count) graphlab::get_event_log().thr_inc_log_entry(name, count);\n#define DECREMENT_EVENT(name, count) graphlab::get_event_log().thr_dec_log_entry(name, count);\n\n#endif\n" }, { "path": "src/graphlab/rpc/evwrapdef.h", "content": "#define event_active gl_event_active\n#define event_active_nolock gl_event_active_nolock\n#define event_add gl_event_add\n#define event_assign gl_event_assign\n#define event_base_add_virtual gl_event_base_add_virtual\n#define event_base_assert_ok gl_event_base_assert_ok\n#define event_base_del_virtual gl_event_base_del_virtual\n#define event_base_dispatch gl_event_base_dispatch\n#define event_base_dump_events gl_event_base_dump_events\n#define event_base_free gl_event_base_free\n#define event_base_get_deferred_cb_queue gl_event_base_get_deferred_cb_queue\n#define event_base_get_features gl_event_base_get_features\n#define event_base_get_method gl_event_base_get_method\n#define event_base_gettimeofday_cached gl_event_base_gettimeofday_cached\n#define event_base_got_break gl_event_base_got_break\n#define event_base_got_exit gl_event_base_got_exit\n#define event_base_init_common_timeout gl_event_base_init_common_timeout\n#define event_base_loop gl_event_base_loop\n#define event_base_loopbreak gl_event_base_loopbreak\n#define event_base_loopexit gl_event_base_loopexit\n#define event_base_new gl_event_base_new\n#define event_base_new_with_config gl_event_base_new_with_config\n#define event_base_once gl_event_base_once\n#define event_base_priority_init gl_event_base_priority_init\n#define event_base_set gl_event_base_set\n#define event_base_start_iocp gl_event_base_start_iocp\n#define event_base_stop_iocp gl_event_base_stop_iocp\n#define event_config_avoid_method gl_event_config_avoid_method\n#define event_config_free gl_event_config_free\n#define event_config_new gl_event_config_new\n#define event_config_require_features gl_event_config_require_features\n#define event_config_set_flag gl_event_config_set_flag\n#define event_config_set_num_cpus_hint gl_event_config_set_num_cpus_hint\n#define event_debug_map_HT_CLEAR gl_event_debug_map_HT_CLEAR\n#define event_debug_map_HT_GROW gl_event_debug_map_HT_GROW\n#define _event_debug_map_HT_REP_IS_BAD gl__event_debug_map_HT_REP_IS_BAD\n#define _event_debug_mode_on gl__event_debug_mode_on\n#define event_debug_unassign gl_event_debug_unassign\n#define event_deferred_cb_cancel gl_event_deferred_cb_cancel\n#define event_deferred_cb_init gl_event_deferred_cb_init\n#define event_deferred_cb_queue_init gl_event_deferred_cb_queue_init\n#define event_deferred_cb_schedule gl_event_deferred_cb_schedule\n#define event_del gl_event_del\n#define event_dispatch gl_event_dispatch\n#define event_enable_debug_mode gl_event_enable_debug_mode\n#define event_free gl_event_free\n#define event_get_assignment gl_event_get_assignment\n#define event_get_base gl_event_get_base\n#define event_get_callback gl_event_get_callback\n#define event_get_callback_arg gl_event_get_callback_arg\n#define event_get_events gl_event_get_events\n#define event_get_fd gl_event_get_fd\n#define event_get_method gl_event_get_method\n#define event_get_struct_event_size gl_event_get_struct_event_size\n#define event_get_supported_methods gl_event_get_supported_methods\n#define event_get_version gl_event_get_version\n#define event_get_version_number gl_event_get_version_number\n#define event_global_current_base_ gl_event_global_current_base_\n#define event_global_setup_locks_ gl_event_global_setup_locks_\n#define event_init gl_event_init\n#define event_initialized gl_event_initialized\n#define event_loop gl_event_loop\n#define event_loopbreak gl_event_loopbreak\n#define event_loopexit gl_event_loopexit\n#define event_mm_calloc_ gl_event_mm_calloc_\n#define event_mm_free_ gl_event_mm_free_\n#define event_mm_malloc_ gl_event_mm_malloc_\n#define event_mm_realloc_ gl_event_mm_realloc_\n#define event_mm_strdup_ gl_event_mm_strdup_\n#define event_new gl_event_new\n#define event_once gl_event_once\n#define event_pending gl_event_pending\n#define event_priority_init gl_event_priority_init\n#define event_priority_set gl_event_priority_set\n#define event_reinit gl_event_reinit\n#define event_set gl_event_set\n#define event_set_mem_functions gl_event_set_mem_functions\n#define evthread_make_base_notifiable gl_evthread_make_base_notifiable\n#define _evthread_cond_fns gl__evthread_cond_fns\n#define _evthread_debug_get_real_lock gl__evthread_debug_get_real_lock\n#define evthread_enable_lock_debuging gl_evthread_enable_lock_debuging\n#define _evthread_id_fn gl__evthread_id_fn\n#define _evthread_is_debug_lock_held gl__evthread_is_debug_lock_held\n#define _evthread_lock_debugging_enabled gl__evthread_lock_debugging_enabled\n#define _evthread_lock_fns gl__evthread_lock_fns\n#define evthread_set_condition_callbacks gl_evthread_set_condition_callbacks\n#define evthread_set_id_callback gl_evthread_set_id_callback\n#define evthread_set_lock_callbacks gl_evthread_set_lock_callbacks\n#define evthread_setup_global_lock_ gl_evthread_setup_global_lock_\n#define evbuffer_add gl_evbuffer_add\n#define evbuffer_add_buffer gl_evbuffer_add_buffer\n#define evbuffer_add_cb gl_evbuffer_add_cb\n#define evbuffer_add_file gl_evbuffer_add_file\n#define evbuffer_add_printf gl_evbuffer_add_printf\n#define evbuffer_add_reference gl_evbuffer_add_reference\n#define evbuffer_add_vprintf gl_evbuffer_add_vprintf\n#define evbuffer_cb_clear_flags gl_evbuffer_cb_clear_flags\n#define evbuffer_cb_set_flags gl_evbuffer_cb_set_flags\n#define _evbuffer_chain_pin gl__evbuffer_chain_pin\n#define _evbuffer_chain_unpin gl__evbuffer_chain_unpin\n#define evbuffer_clear_flags gl_evbuffer_clear_flags\n#define evbuffer_commit_space gl_evbuffer_commit_space\n#define evbuffer_copyout gl_evbuffer_copyout\n#define _evbuffer_decref_and_unlock gl__evbuffer_decref_and_unlock\n#define evbuffer_defer_callbacks gl_evbuffer_defer_callbacks\n#define evbuffer_drain gl_evbuffer_drain\n#define evbuffer_enable_locking gl_evbuffer_enable_locking\n#define evbuffer_expand gl_evbuffer_expand\n#define _evbuffer_expand_fast gl__evbuffer_expand_fast\n#define evbuffer_find gl_evbuffer_find\n#define evbuffer_free gl_evbuffer_free\n#define evbuffer_freeze gl_evbuffer_freeze\n#define evbuffer_get_contiguous_space gl_evbuffer_get_contiguous_space\n#define evbuffer_get_length gl_evbuffer_get_length\n#define _evbuffer_incref gl__evbuffer_incref\n#define _evbuffer_incref_and_lock gl__evbuffer_incref_and_lock\n#define evbuffer_invoke_callbacks gl_evbuffer_invoke_callbacks\n#define evbuffer_lock gl_evbuffer_lock\n#define evbuffer_new gl_evbuffer_new\n#define evbuffer_peek gl_evbuffer_peek\n#define evbuffer_prepend gl_evbuffer_prepend\n#define evbuffer_prepend_buffer gl_evbuffer_prepend_buffer\n#define evbuffer_ptr_set gl_evbuffer_ptr_set\n#define evbuffer_pullup gl_evbuffer_pullup\n#define evbuffer_read gl_evbuffer_read\n#define evbuffer_readline gl_evbuffer_readline\n#define evbuffer_readln gl_evbuffer_readln\n#define _evbuffer_read_setup_vecs gl__evbuffer_read_setup_vecs\n#define evbuffer_remove gl_evbuffer_remove\n#define evbuffer_remove_buffer gl_evbuffer_remove_buffer\n#define evbuffer_remove_cb gl_evbuffer_remove_cb\n#define evbuffer_remove_cb_entry gl_evbuffer_remove_cb_entry\n#define evbuffer_reserve_space gl_evbuffer_reserve_space\n#define evbuffer_search gl_evbuffer_search\n#define evbuffer_search_eol gl_evbuffer_search_eol\n#define evbuffer_search_range gl_evbuffer_search_range\n#define evbuffer_setcb gl_evbuffer_setcb\n#define evbuffer_set_flags gl_evbuffer_set_flags\n#define evbuffer_set_parent gl_evbuffer_set_parent\n#define _evbuffer_testing_use_linear_file_access gl__evbuffer_testing_use_linear_file_access\n#define _evbuffer_testing_use_mmap gl__evbuffer_testing_use_mmap\n#define _evbuffer_testing_use_sendfile gl__evbuffer_testing_use_sendfile\n#define evbuffer_unfreeze gl_evbuffer_unfreeze\n#define evbuffer_unlock gl_evbuffer_unlock\n#define evbuffer_write gl_evbuffer_write\n#define evbuffer_write_atmost gl_evbuffer_write_atmost\n#define _bufferevent_add_event gl__bufferevent_add_event\n#define bufferevent_decref gl_bufferevent_decref\n#define _bufferevent_decref_and_unlock gl__bufferevent_decref_and_unlock\n#define _bufferevent_del_generic_timeout_cbs gl__bufferevent_del_generic_timeout_cbs\n#define bufferevent_disable gl_bufferevent_disable\n#define bufferevent_disable_hard gl_bufferevent_disable_hard\n#define bufferevent_enable gl_bufferevent_enable\n#define bufferevent_enable_locking gl_bufferevent_enable_locking\n#define bufferevent_flush gl_bufferevent_flush\n#define bufferevent_free gl_bufferevent_free\n#define _bufferevent_generic_adj_timeouts gl__bufferevent_generic_adj_timeouts\n#define bufferevent_get_base gl_bufferevent_get_base\n#define bufferevent_get_enabled gl_bufferevent_get_enabled\n#define bufferevent_getfd gl_bufferevent_getfd\n#define bufferevent_get_input gl_bufferevent_get_input\n#define bufferevent_get_output gl_bufferevent_get_output\n#define bufferevent_get_underlying gl_bufferevent_get_underlying\n#define bufferevent_incref gl_bufferevent_incref\n#define _bufferevent_incref_and_lock gl__bufferevent_incref_and_lock\n#define bufferevent_init_common gl_bufferevent_init_common\n#define _bufferevent_init_generic_timeout_cbs gl__bufferevent_init_generic_timeout_cbs\n#define bufferevent_lock gl_bufferevent_lock\n#define bufferevent_read gl_bufferevent_read\n#define bufferevent_read_buffer gl_bufferevent_read_buffer\n#define _bufferevent_run_eventcb gl__bufferevent_run_eventcb\n#define _bufferevent_run_readcb gl__bufferevent_run_readcb\n#define _bufferevent_run_writecb gl__bufferevent_run_writecb\n#define bufferevent_setcb gl_bufferevent_setcb\n#define bufferevent_setfd gl_bufferevent_setfd\n#define bufferevent_settimeout gl_bufferevent_settimeout\n#define bufferevent_set_timeouts gl_bufferevent_set_timeouts\n#define bufferevent_setwatermark gl_bufferevent_setwatermark\n#define bufferevent_suspend_read gl_bufferevent_suspend_read\n#define bufferevent_suspend_write gl_bufferevent_suspend_write\n#define bufferevent_unlock gl_bufferevent_unlock\n#define bufferevent_unsuspend_read gl_bufferevent_unsuspend_read\n#define bufferevent_unsuspend_write gl_bufferevent_unsuspend_write\n#define bufferevent_write gl_bufferevent_write\n#define bufferevent_write_buffer gl_bufferevent_write_buffer\n#define bufferevent_base_set gl_bufferevent_base_set\n#define bufferevent_new gl_bufferevent_new\n#define bufferevent_ops_socket gl_bufferevent_ops_socket\n#define bufferevent_priority_set gl_bufferevent_priority_set\n#define bufferevent_socket_connect gl_bufferevent_socket_connect\n#define bufferevent_socket_connect_hostname gl_bufferevent_socket_connect_hostname\n#define bufferevent_socket_get_dns_error gl_bufferevent_socket_get_dns_error\n#define bufferevent_socket_new gl_bufferevent_socket_new\n#define bufferevent_filter_new gl_bufferevent_filter_new\n#define bufferevent_ops_filter gl_bufferevent_ops_filter\n#define bufferevent_ops_pair gl_bufferevent_ops_pair\n#define bufferevent_pair_get_partner gl_bufferevent_pair_get_partner\n#define bufferevent_pair_new gl_bufferevent_pair_new\n#define evconnlistener_disable gl_evconnlistener_disable\n#define evconnlistener_enable gl_evconnlistener_enable\n#define evconnlistener_free gl_evconnlistener_free\n#define evconnlistener_get_base gl_evconnlistener_get_base\n#define evconnlistener_get_fd gl_evconnlistener_get_fd\n#define evconnlistener_new gl_evconnlistener_new\n#define evconnlistener_new_bind gl_evconnlistener_new_bind\n#define evconnlistener_set_cb gl_evconnlistener_set_cb\n#define evconnlistener_set_error_cb gl_evconnlistener_set_error_cb\n#define bufferevent_add_to_rate_limit_group gl_bufferevent_add_to_rate_limit_group\n#define _bufferevent_decrement_read_buckets gl__bufferevent_decrement_read_buckets\n#define bufferevent_decrement_read_limit gl_bufferevent_decrement_read_limit\n#define _bufferevent_decrement_write_buckets gl__bufferevent_decrement_write_buckets\n#define bufferevent_decrement_write_limit gl_bufferevent_decrement_write_limit\n#define bufferevent_get_max_to_read gl_bufferevent_get_max_to_read\n#define bufferevent_get_max_to_write gl_bufferevent_get_max_to_write\n#define bufferevent_get_read_limit gl_bufferevent_get_read_limit\n#define _bufferevent_get_read_max gl__bufferevent_get_read_max\n#define bufferevent_get_write_limit gl_bufferevent_get_write_limit\n#define _bufferevent_get_write_max gl__bufferevent_get_write_max\n#define bufferevent_rate_limit_group_decrement_read gl_bufferevent_rate_limit_group_decrement_read\n#define bufferevent_rate_limit_group_decrement_write gl_bufferevent_rate_limit_group_decrement_write\n#define bufferevent_rate_limit_group_free gl_bufferevent_rate_limit_group_free\n#define bufferevent_rate_limit_group_get_read_limit gl_bufferevent_rate_limit_group_get_read_limit\n#define bufferevent_rate_limit_group_get_totals gl_bufferevent_rate_limit_group_get_totals\n#define bufferevent_rate_limit_group_get_write_limit gl_bufferevent_rate_limit_group_get_write_limit\n#define bufferevent_rate_limit_group_new gl_bufferevent_rate_limit_group_new\n#define bufferevent_rate_limit_group_reset_totals gl_bufferevent_rate_limit_group_reset_totals\n#define bufferevent_rate_limit_group_set_cfg gl_bufferevent_rate_limit_group_set_cfg\n#define bufferevent_rate_limit_group_set_min_share gl_bufferevent_rate_limit_group_set_min_share\n#define bufferevent_remove_from_rate_limit_group gl_bufferevent_remove_from_rate_limit_group\n#define bufferevent_remove_from_rate_limit_group_internal gl_bufferevent_remove_from_rate_limit_group_internal\n#define bufferevent_set_rate_limit gl_bufferevent_set_rate_limit\n#define ev_token_bucket_cfg_free gl_ev_token_bucket_cfg_free\n#define ev_token_bucket_cfg_new gl_ev_token_bucket_cfg_new\n#define ev_token_bucket_get_tick gl_ev_token_bucket_get_tick\n#define ev_token_bucket_init gl_ev_token_bucket_init\n#define ev_token_bucket_update gl_ev_token_bucket_update\n#define event_changelist_add gl_event_changelist_add\n#define event_changelist_del gl_event_changelist_del\n#define event_changelist_freemem gl_event_changelist_freemem\n#define event_changelist_init gl_event_changelist_init\n#define event_changelist_remove_all gl_event_changelist_remove_all\n#define evmap_check_integrity gl_evmap_check_integrity\n#define evmap_io_active gl_evmap_io_active\n#define evmap_io_add gl_evmap_io_add\n#define evmap_io_clear gl_evmap_io_clear\n#define evmap_io_del gl_evmap_io_del\n#define evmap_io_get_fdinfo gl_evmap_io_get_fdinfo\n#define evmap_io_initmap gl_evmap_io_initmap\n#define evmap_signal_active gl_evmap_signal_active\n#define evmap_signal_add gl_evmap_signal_add\n#define evmap_signal_clear gl_evmap_signal_clear\n#define evmap_signal_del gl_evmap_signal_del\n#define evmap_signal_initmap gl_evmap_signal_initmap\n#define _event_debugx gl__event_debugx\n#define event_err gl_event_err\n#define event_errx gl_event_errx\n#define event_msgx gl_event_msgx\n#define event_set_fatal_callback gl_event_set_fatal_callback\n#define event_set_log_callback gl_event_set_log_callback\n#define event_sock_err gl_event_sock_err\n#define event_sock_warn gl_event_sock_warn\n#define event_warn gl_event_warn\n#define event_warnx gl_event_warnx\n#define evutil_addrinfo_append gl_evutil_addrinfo_append\n#define evutil_adjust_hints_for_addrconfig gl_evutil_adjust_hints_for_addrconfig\n#define evutil_ascii_strcasecmp gl_evutil_ascii_strcasecmp\n#define evutil_ascii_strncasecmp gl_evutil_ascii_strncasecmp\n#define evutil_closesocket gl_evutil_closesocket\n#define evutil_ersatz_socketpair gl_evutil_ersatz_socketpair\n#define evutil_format_sockaddr_port gl_evutil_format_sockaddr_port\n#define evutil_freeaddrinfo gl_evutil_freeaddrinfo\n#define evutil_gai_strerror gl_evutil_gai_strerror\n#define evutil_getaddrinfo gl_evutil_getaddrinfo\n#define evutil_getaddrinfo_async gl_evutil_getaddrinfo_async\n#define evutil_getaddrinfo_common gl_evutil_getaddrinfo_common\n#define evutil_getenv gl_evutil_getenv\n#define evutil_hex_char_to_int gl_evutil_hex_char_to_int\n#define evutil_inet_ntop gl_evutil_inet_ntop\n#define evutil_inet_pton gl_evutil_inet_pton\n#define EVUTIL_ISALNUM gl_EVUTIL_ISALNUM\n#define EVUTIL_ISALPHA gl_EVUTIL_ISALPHA\n#define EVUTIL_ISDIGIT gl_EVUTIL_ISDIGIT\n#define EVUTIL_ISLOWER gl_EVUTIL_ISLOWER\n#define EVUTIL_ISPRINT gl_EVUTIL_ISPRINT\n#define EVUTIL_ISSPACE gl_EVUTIL_ISSPACE\n#define EVUTIL_ISUPPER gl_EVUTIL_ISUPPER\n#define EVUTIL_ISXDIGIT gl_EVUTIL_ISXDIGIT\n#define evutil_make_listen_socket_reuseable gl_evutil_make_listen_socket_reuseable\n#define evutil_make_socket_closeonexec gl_evutil_make_socket_closeonexec\n#define evutil_make_socket_nonblocking gl_evutil_make_socket_nonblocking\n#define evutil_new_addrinfo gl_evutil_new_addrinfo\n#define evutil_open_closeonexec gl_evutil_open_closeonexec\n#define evutil_parse_sockaddr_port gl_evutil_parse_sockaddr_port\n#define evutil_read_file gl_evutil_read_file\n#define evutil_set_evdns_getaddrinfo_fn gl_evutil_set_evdns_getaddrinfo_fn\n#define evutil_snprintf gl_evutil_snprintf\n#define evutil_sockaddr_cmp gl_evutil_sockaddr_cmp\n#define evutil_sockaddr_is_loopback gl_evutil_sockaddr_is_loopback\n#define evutil_socket_connect gl_evutil_socket_connect\n#define evutil_socket_finished_connecting gl_evutil_socket_finished_connecting\n#define evutil_socketpair gl_evutil_socketpair\n#define evutil_strtoll gl_evutil_strtoll\n#define EVUTIL_TOLOWER gl_EVUTIL_TOLOWER\n#define EVUTIL_TOUPPER gl_EVUTIL_TOUPPER\n#define evutil_tv_to_msec gl_evutil_tv_to_msec\n#define evutil_vsnprintf gl_evutil_vsnprintf\n#define _evutil_weakrand gl__evutil_weakrand\n#define evutil_secure_rng_add_bytes gl_evutil_secure_rng_add_bytes\n#define evutil_secure_rng_get_bytes gl_evutil_secure_rng_get_bytes\n#define evutil_secure_rng_global_setup_locks_ gl_evutil_secure_rng_global_setup_locks_\n#define evutil_secure_rng_init gl_evutil_secure_rng_init\n#define _event_strlcpy gl__event_strlcpy\n#define selectops gl_selectops\n#define pollops gl_pollops\n#define epollops gl_epollops\n#define evsig_dealloc gl_evsig_dealloc\n#define evsig_global_setup_locks_ gl_evsig_global_setup_locks_\n#define evsig_init gl_evsig_init\n#define _evsig_restore_handler gl__evsig_restore_handler\n#define evsig_set_base gl_evsig_set_base\n#define _evsig_set_handler gl__evsig_set_handler\n#define evtag_consume gl_evtag_consume\n#define evtag_decode_int gl_evtag_decode_int\n#define evtag_decode_int64 gl_evtag_decode_int64\n#define evtag_decode_tag gl_evtag_decode_tag\n#define evtag_encode_int gl_evtag_encode_int\n#define evtag_encode_int64 gl_evtag_encode_int64\n#define evtag_encode_tag gl_evtag_encode_tag\n#define evtag_init gl_evtag_init\n#define evtag_marshal gl_evtag_marshal\n#define evtag_marshal_buffer gl_evtag_marshal_buffer\n#define evtag_marshal_int gl_evtag_marshal_int\n#define evtag_marshal_int64 gl_evtag_marshal_int64\n#define evtag_marshal_string gl_evtag_marshal_string\n#define evtag_marshal_timeval gl_evtag_marshal_timeval\n#define evtag_payload_length gl_evtag_payload_length\n#define evtag_peek gl_evtag_peek\n#define evtag_peek_length gl_evtag_peek_length\n#define evtag_unmarshal gl_evtag_unmarshal\n#define evtag_unmarshal_fixed gl_evtag_unmarshal_fixed\n#define evtag_unmarshal_header gl_evtag_unmarshal_header\n#define evtag_unmarshal_int gl_evtag_unmarshal_int\n#define evtag_unmarshal_int64 gl_evtag_unmarshal_int64\n#define evtag_unmarshal_string gl_evtag_unmarshal_string\n#define evtag_unmarshal_timeval gl_evtag_unmarshal_timeval\n#define evhttp_accept_socket gl_evhttp_accept_socket\n#define evhttp_accept_socket_with_handle gl_evhttp_accept_socket_with_handle\n#define evhttp_add_header gl_evhttp_add_header\n#define evhttp_add_server_alias gl_evhttp_add_server_alias\n#define evhttp_add_virtual_host gl_evhttp_add_virtual_host\n#define evhttp_bind_listener gl_evhttp_bind_listener\n#define evhttp_bind_socket gl_evhttp_bind_socket\n#define evhttp_bind_socket_with_handle gl_evhttp_bind_socket_with_handle\n#define evhttp_bound_socket_get_fd gl_evhttp_bound_socket_get_fd\n#define evhttp_bound_socket_get_listener gl_evhttp_bound_socket_get_listener\n#define evhttp_cancel_request gl_evhttp_cancel_request\n#define evhttp_clear_headers gl_evhttp_clear_headers\n#define evhttp_connection_base_new gl_evhttp_connection_base_new\n#define evhttp_connection_connect gl_evhttp_connection_connect\n#define evhttp_connection_fail gl_evhttp_connection_fail\n#define evhttp_connection_free gl_evhttp_connection_free\n#define evhttp_connection_get_base gl_evhttp_connection_get_base\n#define evhttp_connection_get_bufferevent gl_evhttp_connection_get_bufferevent\n#define evhttp_connection_get_peer gl_evhttp_connection_get_peer\n#define evhttp_connection_new gl_evhttp_connection_new\n#define evhttp_connection_reset gl_evhttp_connection_reset\n#define evhttp_connection_set_base gl_evhttp_connection_set_base\n#define evhttp_connection_set_closecb gl_evhttp_connection_set_closecb\n#define evhttp_connection_set_local_address gl_evhttp_connection_set_local_address\n#define evhttp_connection_set_local_port gl_evhttp_connection_set_local_port\n#define evhttp_connection_set_max_body_size gl_evhttp_connection_set_max_body_size\n#define evhttp_connection_set_max_headers_size gl_evhttp_connection_set_max_headers_size\n#define evhttp_connection_set_retries gl_evhttp_connection_set_retries\n#define evhttp_connection_set_timeout gl_evhttp_connection_set_timeout\n#define evhttp_decode_uri gl_evhttp_decode_uri\n#define evhttp_del_accept_socket gl_evhttp_del_accept_socket\n#define evhttp_del_cb gl_evhttp_del_cb\n#define evhttp_encode_uri gl_evhttp_encode_uri\n#define evhttp_find_header gl_evhttp_find_header\n#define evhttp_free gl_evhttp_free\n#define evhttp_htmlescape gl_evhttp_htmlescape\n#define evhttp_make_request gl_evhttp_make_request\n#define evhttp_new gl_evhttp_new\n#define evhttp_parse_firstline gl_evhttp_parse_firstline\n#define evhttp_parse_headers gl_evhttp_parse_headers\n#define evhttp_parse_query gl_evhttp_parse_query\n#define evhttp_parse_query_str gl_evhttp_parse_query_str\n#define evhttp_remove_header gl_evhttp_remove_header\n#define evhttp_remove_server_alias gl_evhttp_remove_server_alias\n#define evhttp_remove_virtual_host gl_evhttp_remove_virtual_host\n#define evhttp_request_free gl_evhttp_request_free\n#define evhttp_request_get_command gl_evhttp_request_get_command\n#define evhttp_request_get_connection gl_evhttp_request_get_connection\n#define evhttp_request_get_evhttp_uri gl_evhttp_request_get_evhttp_uri\n#define evhttp_request_get_host gl_evhttp_request_get_host\n#define evhttp_request_get_input_buffer gl_evhttp_request_get_input_buffer\n#define evhttp_request_get_input_headers gl_evhttp_request_get_input_headers\n#define evhttp_request_get_output_buffer gl_evhttp_request_get_output_buffer\n#define evhttp_request_get_output_headers gl_evhttp_request_get_output_headers\n#define evhttp_request_get_response_code gl_evhttp_request_get_response_code\n#define evhttp_request_get_uri gl_evhttp_request_get_uri\n#define evhttp_request_is_owned gl_evhttp_request_is_owned\n#define evhttp_request_new gl_evhttp_request_new\n#define evhttp_request_own gl_evhttp_request_own\n#define evhttp_request_set_chunked_cb gl_evhttp_request_set_chunked_cb\n#define evhttp_response_code gl_evhttp_response_code\n#define evhttp_send_error gl_evhttp_send_error\n#define evhttp_send_page gl_evhttp_send_page\n#define evhttp_send_reply gl_evhttp_send_reply\n#define evhttp_send_reply_chunk gl_evhttp_send_reply_chunk\n#define evhttp_send_reply_end gl_evhttp_send_reply_end\n#define evhttp_send_reply_start gl_evhttp_send_reply_start\n#define evhttp_set_allowed_methods gl_evhttp_set_allowed_methods\n#define evhttp_set_cb gl_evhttp_set_cb\n#define evhttp_set_gencb gl_evhttp_set_gencb\n#define evhttp_set_max_body_size gl_evhttp_set_max_body_size\n#define evhttp_set_max_headers_size gl_evhttp_set_max_headers_size\n#define evhttp_set_timeout gl_evhttp_set_timeout\n#define evhttp_start gl_evhttp_start\n#define evhttp_start_read gl_evhttp_start_read\n#define evhttp_uridecode gl_evhttp_uridecode\n#define evhttp_uriencode gl_evhttp_uriencode\n#define evhttp_uri_free gl_evhttp_uri_free\n#define evhttp_uri_get_fragment gl_evhttp_uri_get_fragment\n#define evhttp_uri_get_host gl_evhttp_uri_get_host\n#define evhttp_uri_get_path gl_evhttp_uri_get_path\n#define evhttp_uri_get_port gl_evhttp_uri_get_port\n#define evhttp_uri_get_query gl_evhttp_uri_get_query\n#define evhttp_uri_get_scheme gl_evhttp_uri_get_scheme\n#define evhttp_uri_get_userinfo gl_evhttp_uri_get_userinfo\n#define evhttp_uri_join gl_evhttp_uri_join\n#define evhttp_uri_new gl_evhttp_uri_new\n#define evhttp_uri_parse gl_evhttp_uri_parse\n#define evhttp_uri_parse_with_flags gl_evhttp_uri_parse_with_flags\n#define evhttp_uri_set_flags gl_evhttp_uri_set_flags\n#define evhttp_uri_set_fragment gl_evhttp_uri_set_fragment\n#define evhttp_uri_set_host gl_evhttp_uri_set_host\n#define evhttp_uri_set_path gl_evhttp_uri_set_path\n#define evhttp_uri_set_port gl_evhttp_uri_set_port\n#define evhttp_uri_set_query gl_evhttp_uri_set_query\n#define evhttp_uri_set_scheme gl_evhttp_uri_set_scheme\n#define evhttp_uri_set_userinfo gl_evhttp_uri_set_userinfo\n#define evdns_add_server_port gl_evdns_add_server_port\n#define evdns_add_server_port_with_base gl_evdns_add_server_port_with_base\n#define evdns_base_clear_nameservers_and_suspend gl_evdns_base_clear_nameservers_and_suspend\n#define evdns_base_count_nameservers gl_evdns_base_count_nameservers\n#define evdns_base_free gl_evdns_base_free\n#define evdns_base_load_hosts gl_evdns_base_load_hosts\n#define evdns_base_nameserver_add gl_evdns_base_nameserver_add\n#define evdns_base_nameserver_ip_add gl_evdns_base_nameserver_ip_add\n#define evdns_base_nameserver_sockaddr_add gl_evdns_base_nameserver_sockaddr_add\n#define evdns_base_new gl_evdns_base_new\n#define evdns_base_resolv_conf_parse gl_evdns_base_resolv_conf_parse\n#define evdns_base_resolve_ipv4 gl_evdns_base_resolve_ipv4\n#define evdns_base_resolve_ipv6 gl_evdns_base_resolve_ipv6\n#define evdns_base_resolve_reverse gl_evdns_base_resolve_reverse\n#define evdns_base_resolve_reverse_ipv6 gl_evdns_base_resolve_reverse_ipv6\n#define evdns_base_resume gl_evdns_base_resume\n#define evdns_base_search_add gl_evdns_base_search_add\n#define evdns_base_search_clear gl_evdns_base_search_clear\n#define evdns_base_search_ndots_set gl_evdns_base_search_ndots_set\n#define evdns_base_set_option gl_evdns_base_set_option\n#define evdns_cancel_request gl_evdns_cancel_request\n#define evdns_clear_nameservers_and_suspend gl_evdns_clear_nameservers_and_suspend\n#define evdns_close_server_port gl_evdns_close_server_port\n#define evdns_count_nameservers gl_evdns_count_nameservers\n#define evdns_err_to_string gl_evdns_err_to_string\n#define evdns_getaddrinfo gl_evdns_getaddrinfo\n#define evdns_getaddrinfo_cancel gl_evdns_getaddrinfo_cancel\n#define evdns_get_global_base gl_evdns_get_global_base\n#define evdns_init gl_evdns_init\n#define evdns_nameserver_add gl_evdns_nameserver_add\n#define evdns_nameserver_ip_add gl_evdns_nameserver_ip_add\n#define evdns_resolv_conf_parse gl_evdns_resolv_conf_parse\n#define evdns_resolve_ipv4 gl_evdns_resolve_ipv4\n#define evdns_resolve_ipv6 gl_evdns_resolve_ipv6\n#define evdns_resolve_reverse gl_evdns_resolve_reverse\n#define evdns_resolve_reverse_ipv6 gl_evdns_resolve_reverse_ipv6\n#define evdns_resume gl_evdns_resume\n#define evdns_search_add gl_evdns_search_add\n#define evdns_search_clear gl_evdns_search_clear\n#define evdns_search_ndots_set gl_evdns_search_ndots_set\n#define evdns_server_request_add_aaaa_reply gl_evdns_server_request_add_aaaa_reply\n#define evdns_server_request_add_a_reply gl_evdns_server_request_add_a_reply\n#define evdns_server_request_add_cname_reply gl_evdns_server_request_add_cname_reply\n#define evdns_server_request_add_ptr_reply gl_evdns_server_request_add_ptr_reply\n#define evdns_server_request_add_reply gl_evdns_server_request_add_reply\n#define evdns_server_request_drop gl_evdns_server_request_drop\n#define evdns_server_request_get_requesting_addr gl_evdns_server_request_get_requesting_addr\n#define evdns_server_request_respond gl_evdns_server_request_respond\n#define evdns_server_request_set_flags gl_evdns_server_request_set_flags\n#define evdns_set_log_fn gl_evdns_set_log_fn\n#define evdns_set_option gl_evdns_set_option\n#define evdns_set_random_bytes_fn gl_evdns_set_random_bytes_fn\n#define evdns_set_transaction_id_fn gl_evdns_set_transaction_id_fn\n#define evdns_shutdown gl_evdns_shutdown\n#define evrpc_add_hook gl_evrpc_add_hook\n#define evrpc_free gl_evrpc_free\n#define evrpc_get_reply gl_evrpc_get_reply\n#define evrpc_get_request gl_evrpc_get_request\n#define evrpc_hook_add_meta gl_evrpc_hook_add_meta\n#define evrpc_hook_find_meta gl_evrpc_hook_find_meta\n#define evrpc_hook_get_connection gl_evrpc_hook_get_connection\n#define evrpc_init gl_evrpc_init\n#define evrpc_make_request gl_evrpc_make_request\n#define evrpc_make_request_ctx gl_evrpc_make_request_ctx\n#define evrpc_pool_add_connection gl_evrpc_pool_add_connection\n#define evrpc_pool_free gl_evrpc_pool_free\n#define evrpc_pool_new gl_evrpc_pool_new\n#define evrpc_pool_remove_connection gl_evrpc_pool_remove_connection\n#define evrpc_pool_set_timeout gl_evrpc_pool_set_timeout\n#define evrpc_register_generic gl_evrpc_register_generic\n#define evrpc_register_rpc gl_evrpc_register_rpc\n#define evrpc_remove_hook gl_evrpc_remove_hook\n#define evrpc_reqstate_free gl_evrpc_reqstate_free\n#define evrpc_request_done gl_evrpc_request_done\n#define evrpc_request_get_pool gl_evrpc_request_get_pool\n#define evrpc_request_set_cb gl_evrpc_request_set_cb\n#define evrpc_request_set_pool gl_evrpc_request_set_pool\n#define evrpc_resume_request gl_evrpc_resume_request\n#define evrpc_send_request_generic gl_evrpc_send_request_generic\n#define evrpc_unregister_rpc gl_evrpc_unregister_rpc\n" }, { "path": "src/graphlab/rpc/evwrapundef.h", "content": "#undef event_active\n#undef event_active_nolock\n#undef event_add\n#undef event_assign\n#undef event_base_add_virtual\n#undef event_base_assert_ok\n#undef event_base_del_virtual\n#undef event_base_dispatch\n#undef event_base_dump_events\n#undef event_base_free\n#undef event_base_get_deferred_cb_queue\n#undef event_base_get_features\n#undef event_base_get_method\n#undef event_base_gettimeofday_cached\n#undef event_base_got_break\n#undef event_base_got_exit\n#undef event_base_init_common_timeout\n#undef event_base_loop\n#undef event_base_loopbreak\n#undef event_base_loopexit\n#undef event_base_new\n#undef event_base_new_with_config\n#undef event_base_once\n#undef event_base_priority_init\n#undef event_base_set\n#undef event_base_start_iocp\n#undef event_base_stop_iocp\n#undef event_config_avoid_method\n#undef event_config_free\n#undef event_config_new\n#undef event_config_require_features\n#undef event_config_set_flag\n#undef event_config_set_num_cpus_hint\n#undef event_debug_map_HT_CLEAR\n#undef event_debug_map_HT_GROW\n#undef _event_debug_map_HT_REP_IS_BAD\n#undef _event_debug_mode_on\n#undef event_debug_unassign\n#undef event_deferred_cb_cancel\n#undef event_deferred_cb_init\n#undef event_deferred_cb_queue_init\n#undef event_deferred_cb_schedule\n#undef event_del\n#undef event_dispatch\n#undef event_enable_debug_mode\n#undef event_free\n#undef event_get_assignment\n#undef event_get_base\n#undef event_get_callback\n#undef event_get_callback_arg\n#undef event_get_events\n#undef event_get_fd\n#undef event_get_method\n#undef event_get_struct_event_size\n#undef event_get_supported_methods\n#undef event_get_version\n#undef event_get_version_number\n#undef event_global_current_base_\n#undef event_global_setup_locks_\n#undef event_init\n#undef event_initialized\n#undef event_loop\n#undef event_loopbreak\n#undef event_loopexit\n#undef event_mm_calloc_\n#undef event_mm_free_\n#undef event_mm_malloc_\n#undef event_mm_realloc_\n#undef event_mm_strdup_\n#undef event_new\n#undef event_once\n#undef event_pending\n#undef event_priority_init\n#undef event_priority_set\n#undef event_reinit\n#undef event_set\n#undef event_set_mem_functions\n#undef evthread_make_base_notifiable\n#undef _evthread_cond_fns\n#undef _evthread_debug_get_real_lock\n#undef evthread_enable_lock_debuging\n#undef _evthread_id_fn\n#undef _evthread_is_debug_lock_held\n#undef _evthread_lock_debugging_enabled\n#undef _evthread_lock_fns\n#undef evthread_set_condition_callbacks\n#undef evthread_set_id_callback\n#undef evthread_set_lock_callbacks\n#undef evthread_setup_global_lock_\n#undef evbuffer_add\n#undef evbuffer_add_buffer\n#undef evbuffer_add_cb\n#undef evbuffer_add_file\n#undef evbuffer_add_printf\n#undef evbuffer_add_reference\n#undef evbuffer_add_vprintf\n#undef evbuffer_cb_clear_flags\n#undef evbuffer_cb_set_flags\n#undef _evbuffer_chain_pin\n#undef _evbuffer_chain_unpin\n#undef evbuffer_clear_flags\n#undef evbuffer_commit_space\n#undef evbuffer_copyout\n#undef _evbuffer_decref_and_unlock\n#undef evbuffer_defer_callbacks\n#undef evbuffer_drain\n#undef evbuffer_enable_locking\n#undef evbuffer_expand\n#undef _evbuffer_expand_fast\n#undef evbuffer_find\n#undef evbuffer_free\n#undef evbuffer_freeze\n#undef evbuffer_get_contiguous_space\n#undef evbuffer_get_length\n#undef _evbuffer_incref\n#undef _evbuffer_incref_and_lock\n#undef evbuffer_invoke_callbacks\n#undef evbuffer_lock\n#undef evbuffer_new\n#undef evbuffer_peek\n#undef evbuffer_prepend\n#undef evbuffer_prepend_buffer\n#undef evbuffer_ptr_set\n#undef evbuffer_pullup\n#undef evbuffer_read\n#undef evbuffer_readline\n#undef evbuffer_readln\n#undef _evbuffer_read_setup_vecs\n#undef evbuffer_remove\n#undef evbuffer_remove_buffer\n#undef evbuffer_remove_cb\n#undef evbuffer_remove_cb_entry\n#undef evbuffer_reserve_space\n#undef evbuffer_search\n#undef evbuffer_search_eol\n#undef evbuffer_search_range\n#undef evbuffer_setcb\n#undef evbuffer_set_flags\n#undef evbuffer_set_parent\n#undef _evbuffer_testing_use_linear_file_access\n#undef _evbuffer_testing_use_mmap\n#undef _evbuffer_testing_use_sendfile\n#undef evbuffer_unfreeze\n#undef evbuffer_unlock\n#undef evbuffer_write\n#undef evbuffer_write_atmost\n#undef _bufferevent_add_event\n#undef bufferevent_decref\n#undef _bufferevent_decref_and_unlock\n#undef _bufferevent_del_generic_timeout_cbs\n#undef bufferevent_disable\n#undef bufferevent_disable_hard\n#undef bufferevent_enable\n#undef bufferevent_enable_locking\n#undef bufferevent_flush\n#undef bufferevent_free\n#undef _bufferevent_generic_adj_timeouts\n#undef bufferevent_get_base\n#undef bufferevent_get_enabled\n#undef bufferevent_getfd\n#undef bufferevent_get_input\n#undef bufferevent_get_output\n#undef bufferevent_get_underlying\n#undef bufferevent_incref\n#undef _bufferevent_incref_and_lock\n#undef bufferevent_init_common\n#undef _bufferevent_init_generic_timeout_cbs\n#undef bufferevent_lock\n#undef bufferevent_read\n#undef bufferevent_read_buffer\n#undef _bufferevent_run_eventcb\n#undef _bufferevent_run_readcb\n#undef _bufferevent_run_writecb\n#undef bufferevent_setcb\n#undef bufferevent_setfd\n#undef bufferevent_settimeout\n#undef bufferevent_set_timeouts\n#undef bufferevent_setwatermark\n#undef bufferevent_suspend_read\n#undef bufferevent_suspend_write\n#undef bufferevent_unlock\n#undef bufferevent_unsuspend_read\n#undef bufferevent_unsuspend_write\n#undef bufferevent_write\n#undef bufferevent_write_buffer\n#undef bufferevent_base_set\n#undef bufferevent_new\n#undef bufferevent_ops_socket\n#undef bufferevent_priority_set\n#undef bufferevent_socket_connect\n#undef bufferevent_socket_connect_hostname\n#undef bufferevent_socket_get_dns_error\n#undef bufferevent_socket_new\n#undef bufferevent_filter_new\n#undef bufferevent_ops_filter\n#undef bufferevent_ops_pair\n#undef bufferevent_pair_get_partner\n#undef bufferevent_pair_new\n#undef evconnlistener_disable\n#undef evconnlistener_enable\n#undef evconnlistener_free\n#undef evconnlistener_get_base\n#undef evconnlistener_get_fd\n#undef evconnlistener_new\n#undef evconnlistener_new_bind\n#undef evconnlistener_set_cb\n#undef evconnlistener_set_error_cb\n#undef bufferevent_add_to_rate_limit_group\n#undef _bufferevent_decrement_read_buckets\n#undef bufferevent_decrement_read_limit\n#undef _bufferevent_decrement_write_buckets\n#undef bufferevent_decrement_write_limit\n#undef bufferevent_get_max_to_read\n#undef bufferevent_get_max_to_write\n#undef bufferevent_get_read_limit\n#undef _bufferevent_get_read_max\n#undef bufferevent_get_write_limit\n#undef _bufferevent_get_write_max\n#undef bufferevent_rate_limit_group_decrement_read\n#undef bufferevent_rate_limit_group_decrement_write\n#undef bufferevent_rate_limit_group_free\n#undef bufferevent_rate_limit_group_get_read_limit\n#undef bufferevent_rate_limit_group_get_totals\n#undef bufferevent_rate_limit_group_get_write_limit\n#undef bufferevent_rate_limit_group_new\n#undef bufferevent_rate_limit_group_reset_totals\n#undef bufferevent_rate_limit_group_set_cfg\n#undef bufferevent_rate_limit_group_set_min_share\n#undef bufferevent_remove_from_rate_limit_group\n#undef bufferevent_remove_from_rate_limit_group_internal\n#undef bufferevent_set_rate_limit\n#undef ev_token_bucket_cfg_free\n#undef ev_token_bucket_cfg_new\n#undef ev_token_bucket_get_tick\n#undef ev_token_bucket_init\n#undef ev_token_bucket_update\n#undef event_changelist_add\n#undef event_changelist_del\n#undef event_changelist_freemem\n#undef event_changelist_init\n#undef event_changelist_remove_all\n#undef evmap_check_integrity\n#undef evmap_io_active\n#undef evmap_io_add\n#undef evmap_io_clear\n#undef evmap_io_del\n#undef evmap_io_get_fdinfo\n#undef evmap_io_initmap\n#undef evmap_signal_active\n#undef evmap_signal_add\n#undef evmap_signal_clear\n#undef evmap_signal_del\n#undef evmap_signal_initmap\n#undef _event_debugx\n#undef event_err\n#undef event_errx\n#undef event_msgx\n#undef event_set_fatal_callback\n#undef event_set_log_callback\n#undef event_sock_err\n#undef event_sock_warn\n#undef event_warn\n#undef event_warnx\n#undef evutil_addrinfo_append\n#undef evutil_adjust_hints_for_addrconfig\n#undef evutil_ascii_strcasecmp\n#undef evutil_ascii_strncasecmp\n#undef evutil_closesocket\n#undef evutil_ersatz_socketpair\n#undef evutil_format_sockaddr_port\n#undef evutil_freeaddrinfo\n#undef evutil_gai_strerror\n#undef evutil_getaddrinfo\n#undef evutil_getaddrinfo_async\n#undef evutil_getaddrinfo_common\n#undef evutil_getenv\n#undef evutil_hex_char_to_int\n#undef evutil_inet_ntop\n#undef evutil_inet_pton\n#undef EVUTIL_ISALNUM\n#undef EVUTIL_ISALPHA\n#undef EVUTIL_ISDIGIT\n#undef EVUTIL_ISLOWER\n#undef EVUTIL_ISPRINT\n#undef EVUTIL_ISSPACE\n#undef EVUTIL_ISUPPER\n#undef EVUTIL_ISXDIGIT\n#undef evutil_make_listen_socket_reuseable\n#undef evutil_make_socket_closeonexec\n#undef evutil_make_socket_nonblocking\n#undef evutil_new_addrinfo\n#undef evutil_open_closeonexec\n#undef evutil_parse_sockaddr_port\n#undef evutil_read_file\n#undef evutil_set_evdns_getaddrinfo_fn\n#undef evutil_snprintf\n#undef evutil_sockaddr_cmp\n#undef evutil_sockaddr_is_loopback\n#undef evutil_socket_connect\n#undef evutil_socket_finished_connecting\n#undef evutil_socketpair\n#undef evutil_strtoll\n#undef EVUTIL_TOLOWER\n#undef EVUTIL_TOUPPER\n#undef evutil_tv_to_msec\n#undef evutil_vsnprintf\n#undef _evutil_weakrand\n#undef evutil_secure_rng_add_bytes\n#undef evutil_secure_rng_get_bytes\n#undef evutil_secure_rng_global_setup_locks_\n#undef evutil_secure_rng_init\n#undef _event_strlcpy\n#undef selectops\n#undef pollops\n#undef epollops\n#undef evsig_dealloc\n#undef evsig_global_setup_locks_\n#undef evsig_init\n#undef _evsig_restore_handler\n#undef evsig_set_base\n#undef _evsig_set_handler\n#undef evtag_consume\n#undef evtag_decode_int\n#undef evtag_decode_int64\n#undef evtag_decode_tag\n#undef evtag_encode_int\n#undef evtag_encode_int64\n#undef evtag_encode_tag\n#undef evtag_init\n#undef evtag_marshal\n#undef evtag_marshal_buffer\n#undef evtag_marshal_int\n#undef evtag_marshal_int64\n#undef evtag_marshal_string\n#undef evtag_marshal_timeval\n#undef evtag_payload_length\n#undef evtag_peek\n#undef evtag_peek_length\n#undef evtag_unmarshal\n#undef evtag_unmarshal_fixed\n#undef evtag_unmarshal_header\n#undef evtag_unmarshal_int\n#undef evtag_unmarshal_int64\n#undef evtag_unmarshal_string\n#undef evtag_unmarshal_timeval\n#undef evhttp_accept_socket\n#undef evhttp_accept_socket_with_handle\n#undef evhttp_add_header\n#undef evhttp_add_server_alias\n#undef evhttp_add_virtual_host\n#undef evhttp_bind_listener\n#undef evhttp_bind_socket\n#undef evhttp_bind_socket_with_handle\n#undef evhttp_bound_socket_get_fd\n#undef evhttp_bound_socket_get_listener\n#undef evhttp_cancel_request\n#undef evhttp_clear_headers\n#undef evhttp_connection_base_new\n#undef evhttp_connection_connect\n#undef evhttp_connection_fail\n#undef evhttp_connection_free\n#undef evhttp_connection_get_base\n#undef evhttp_connection_get_bufferevent\n#undef evhttp_connection_get_peer\n#undef evhttp_connection_new\n#undef evhttp_connection_reset\n#undef evhttp_connection_set_base\n#undef evhttp_connection_set_closecb\n#undef evhttp_connection_set_local_address\n#undef evhttp_connection_set_local_port\n#undef evhttp_connection_set_max_body_size\n#undef evhttp_connection_set_max_headers_size\n#undef evhttp_connection_set_retries\n#undef evhttp_connection_set_timeout\n#undef evhttp_decode_uri\n#undef evhttp_del_accept_socket\n#undef evhttp_del_cb\n#undef evhttp_encode_uri\n#undef evhttp_find_header\n#undef evhttp_free\n#undef evhttp_htmlescape\n#undef evhttp_make_request\n#undef evhttp_new\n#undef evhttp_parse_firstline\n#undef evhttp_parse_headers\n#undef evhttp_parse_query\n#undef evhttp_parse_query_str\n#undef evhttp_remove_header\n#undef evhttp_remove_server_alias\n#undef evhttp_remove_virtual_host\n#undef evhttp_request_free\n#undef evhttp_request_get_command\n#undef evhttp_request_get_connection\n#undef evhttp_request_get_evhttp_uri\n#undef evhttp_request_get_host\n#undef evhttp_request_get_input_buffer\n#undef evhttp_request_get_input_headers\n#undef evhttp_request_get_output_buffer\n#undef evhttp_request_get_output_headers\n#undef evhttp_request_get_response_code\n#undef evhttp_request_get_uri\n#undef evhttp_request_is_owned\n#undef evhttp_request_new\n#undef evhttp_request_own\n#undef evhttp_request_set_chunked_cb\n#undef evhttp_response_code\n#undef evhttp_send_error\n#undef evhttp_send_page\n#undef evhttp_send_reply\n#undef evhttp_send_reply_chunk\n#undef evhttp_send_reply_end\n#undef evhttp_send_reply_start\n#undef evhttp_set_allowed_methods\n#undef evhttp_set_cb\n#undef evhttp_set_gencb\n#undef evhttp_set_max_body_size\n#undef evhttp_set_max_headers_size\n#undef evhttp_set_timeout\n#undef evhttp_start\n#undef evhttp_start_read\n#undef evhttp_uridecode\n#undef evhttp_uriencode\n#undef evhttp_uri_free\n#undef evhttp_uri_get_fragment\n#undef evhttp_uri_get_host\n#undef evhttp_uri_get_path\n#undef evhttp_uri_get_port\n#undef evhttp_uri_get_query\n#undef evhttp_uri_get_scheme\n#undef evhttp_uri_get_userinfo\n#undef evhttp_uri_join\n#undef evhttp_uri_new\n#undef evhttp_uri_parse\n#undef evhttp_uri_parse_with_flags\n#undef evhttp_uri_set_flags\n#undef evhttp_uri_set_fragment\n#undef evhttp_uri_set_host\n#undef evhttp_uri_set_path\n#undef evhttp_uri_set_port\n#undef evhttp_uri_set_query\n#undef evhttp_uri_set_scheme\n#undef evhttp_uri_set_userinfo\n#undef evdns_add_server_port\n#undef evdns_add_server_port_with_base\n#undef evdns_base_clear_nameservers_and_suspend\n#undef evdns_base_count_nameservers\n#undef evdns_base_free\n#undef evdns_base_load_hosts\n#undef evdns_base_nameserver_add\n#undef evdns_base_nameserver_ip_add\n#undef evdns_base_nameserver_sockaddr_add\n#undef evdns_base_new\n#undef evdns_base_resolv_conf_parse\n#undef evdns_base_resolve_ipv4\n#undef evdns_base_resolve_ipv6\n#undef evdns_base_resolve_reverse\n#undef evdns_base_resolve_reverse_ipv6\n#undef evdns_base_resume\n#undef evdns_base_search_add\n#undef evdns_base_search_clear\n#undef evdns_base_search_ndots_set\n#undef evdns_base_set_option\n#undef evdns_cancel_request\n#undef evdns_clear_nameservers_and_suspend\n#undef evdns_close_server_port\n#undef evdns_count_nameservers\n#undef evdns_err_to_string\n#undef evdns_getaddrinfo\n#undef evdns_getaddrinfo_cancel\n#undef evdns_get_global_base\n#undef evdns_init\n#undef evdns_nameserver_add\n#undef evdns_nameserver_ip_add\n#undef evdns_resolv_conf_parse\n#undef evdns_resolve_ipv4\n#undef evdns_resolve_ipv6\n#undef evdns_resolve_reverse\n#undef evdns_resolve_reverse_ipv6\n#undef evdns_resume\n#undef evdns_search_add\n#undef evdns_search_clear\n#undef evdns_search_ndots_set\n#undef evdns_server_request_add_aaaa_reply\n#undef evdns_server_request_add_a_reply\n#undef evdns_server_request_add_cname_reply\n#undef evdns_server_request_add_ptr_reply\n#undef evdns_server_request_add_reply\n#undef evdns_server_request_drop\n#undef evdns_server_request_get_requesting_addr\n#undef evdns_server_request_respond\n#undef evdns_server_request_set_flags\n#undef evdns_set_log_fn\n#undef evdns_set_option\n#undef evdns_set_random_bytes_fn\n#undef evdns_set_transaction_id_fn\n#undef evdns_shutdown\n#undef evrpc_add_hook\n#undef evrpc_free\n#undef evrpc_get_reply\n#undef evrpc_get_request\n#undef evrpc_hook_add_meta\n#undef evrpc_hook_find_meta\n#undef evrpc_hook_get_connection\n#undef evrpc_init\n#undef evrpc_make_request\n#undef evrpc_make_request_ctx\n#undef evrpc_pool_add_connection\n#undef evrpc_pool_free\n#undef evrpc_pool_new\n#undef evrpc_pool_remove_connection\n#undef evrpc_pool_set_timeout\n#undef evrpc_register_generic\n#undef evrpc_register_rpc\n#undef evrpc_remove_hook\n#undef evrpc_reqstate_free\n#undef evrpc_request_done\n#undef evrpc_request_get_pool\n#undef evrpc_request_set_cb\n#undef evrpc_request_set_pool\n#undef evrpc_resume_request\n#undef evrpc_send_request_generic\n#undef evrpc_unregister_rpc\n" }, { "path": "src/graphlab/rpc/fiber_async_consensus.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <graphlab/rpc/fiber_async_consensus.hpp>\n#include <graphlab/parallel/fiber_control.hpp>\nnamespace graphlab {\n fiber_async_consensus::fiber_async_consensus(distributed_control &dc,\n size_t required_fibers_in_done,\n const dc_impl::dc_dist_object_base *attach)\n :rmi(dc, this), attachedobj(attach),\n last_calls_sent(0), last_calls_received(0),\n numactive(required_fibers_in_done),\n ncpus(required_fibers_in_done),\n done(false),\n trying_to_sleep(0),\n critical(ncpus, 0),\n sleeping(ncpus, 0),\n hastoken(dc.procid() == 0),\n cond(ncpus, 0){\n\n cur_token.total_calls_sent = 0;\n cur_token.total_calls_received = 0;\n cur_token.last_change = (procid_t)(rmi.numprocs() - 1);\n }\n\n void fiber_async_consensus::reset() {\n last_calls_sent = 0;\n last_calls_received = 0;\n numactive = ncpus;\n done = false;\n trying_to_sleep = false;\n critical = std::vector<char>(ncpus, 0);\n sleeping = std::vector<char>(ncpus, 0);\n hastoken = (rmi.procid() == 0);\n cur_token.total_calls_sent = 0;\n cur_token.total_calls_received = 0;\n cur_token.last_change = (procid_t)(rmi.numprocs() - 1);\n }\n\n void fiber_async_consensus::force_done() {\n m.lock();\n done = true;\n m.unlock();\n cancel();\n }\n\n void fiber_async_consensus::begin_done_critical_section(size_t cpuid) {\n trying_to_sleep.inc();\n critical[cpuid] = true;\n m.lock();\n }\n\n\n void fiber_async_consensus::cancel_critical_section(size_t cpuid) {\n m.unlock();\n critical[cpuid] = false;\n trying_to_sleep.dec();\n }\n \n bool fiber_async_consensus::end_done_critical_section(size_t cpuid) {\n // if done flag is set, quit immediately\n if (done) {\n m.unlock();\n critical[cpuid] = false;\n trying_to_sleep.dec();\n return true;\n }\n /*\n Assertion: Since numactive is decremented only within \n a critical section, and is incremented only within the same critical \n section. Therefore numactive is a valid counter of the number of fibers\n outside of this critical section. \n */\n --numactive;\n \n /*\n Assertion: If numactive is ever 0 at this point, the algorithm is done.\n WLOG, let the current fiber which just decremented numactive be fiber 0\n \n Since there is only 1 active fiber (0), there must be no fibers \n performing insertions, and are no other fibers which are waking up.\n All fibers must therefore be sleeping.\n */\n if (numactive == 0) {\n logstream(LOG_INFO) << rmi.procid() << \": Termination Possible\" << std::endl;\n if (hastoken) pass_the_token();\n }\n sleeping[cpuid] = true;\n while(1) {\n // here we are protected by the mutex again.\n \n // woken up by someone else. leave the \n // terminator\n if (sleeping[cpuid] == false || done) {\n break;\n }\n\n // put myself to sleep\n // this here is basically cond[cpuid].wait(m);\n cond[cpuid] = fiber_control::get_tid();\n ASSERT_NE(cond[cpuid], 0);\n fiber_control::deschedule_self(&m.m_mut);\n m.lock();\n cond[cpuid] = 0;\n }\n m.unlock();\n critical[cpuid] = false;\n trying_to_sleep.dec();\n return done;\n }\n \n void fiber_async_consensus::cancel() {\n /*\n Assertion: numactive > 0 if there is work to do.\n If there are fibers trying to sleep, lets wake them up\n */\n if (trying_to_sleep > 0 || numactive < ncpus) {\n m.lock();\n size_t oldnumactive = numactive;\n // once I acquire this lock, all fibers must be\n // in the following states\n // 1: still running and has not reached begin_critical_section()\n // 2: is sleeping in cond.wait()\n // 3: has called begin_critical_section() but has not acquired\n // the mutex\n // In the case of 1,3: These fibers will perform one more sweep\n // of their task queues. Therefore they will see any new job if available\n // in the case of 2: numactive must be < ncpus since numactive\n // is mutex protected. Then I can wake them up by\n // clearing their sleeping flags and broadcasting.\n if (numactive < ncpus) {\n // this is safe. Note that it is done from within \n // the critical section.\n for (size_t i = 0;i < ncpus; ++i) {\n numactive += sleeping[i];\n if (sleeping[i]) {\n sleeping[i] = 0;\n // this here was basically cond[i].signal();\n if (cond[i] != 0) fiber_control::schedule_tid(cond[i]);\n }\n }\n if (oldnumactive == 0 && !done) {\n logstream(LOG_INFO) << rmi.procid() << \": Waking\" << std::endl;\n }\n\n }\n m.unlock();\n }\n }\n\n void fiber_async_consensus::cancel_one(size_t cpuhint) {\n if (critical[cpuhint]) {\n m.lock();\n size_t oldnumactive = numactive;\n // see new_job() for detailed comments\n if (sleeping[cpuhint]) {\n numactive += sleeping[cpuhint];\n sleeping[cpuhint] = 0;\n if (oldnumactive == 0 && !done) {\n logstream(LOG_INFO) << rmi.procid() << \": Waking\" << std::endl;\n }\n\n // this here was basically cond[cpuhint].signal();\n if (cond[cpuhint] != 0) fiber_control::schedule_tid(cond[cpuhint]);\n \n }\n m.unlock();\n }\n }\n\n void fiber_async_consensus::receive_the_token(token &tok) {\n m.lock();\n // save the token\n hastoken = true;\n cur_token = tok;\n // if I am waiting on done, pass the token.\n logstream(LOG_INFO) << rmi.procid() << \": Token Received\" << std::endl;\n if (numactive == 0) {\n pass_the_token();\n }\n m.unlock();\n }\n\n void fiber_async_consensus::pass_the_token() {\n // note that this function does not acquire the token lock\n // the caller must acquire it \n assert(hastoken);\n // first check if we are done\n if (cur_token.last_change == rmi.procid() && \n cur_token.total_calls_received == cur_token.total_calls_sent) {\n logstream(LOG_INFO) << \"Completed Token: \" \n << cur_token.total_calls_received << \" \" \n << cur_token.total_calls_sent << std::endl;\n // we have completed a loop around!\n // broadcast a completion\n for (procid_t i = 0;i < rmi.numprocs(); ++i) {\n if (i != rmi.procid()) {\n rmi.control_call(i,\n &fiber_async_consensus::force_done);\n }\n }\n // set the complete flag\n // we can't call consensus() since it will deadlock\n done = true;\n // this is the same code as cancel(), but we can't call cancel \n // since we are holding on to a lock\n if (numactive < ncpus) {\n // this is safe. Note that it is done from within \n // the critical section.\n for (size_t i = 0;i < ncpus; ++i) {\n numactive += sleeping[i];\n if (sleeping[i]) {\n sleeping[i] = 0;\n // this here is basically cond[i].signal();\n size_t ch = cond[i];\n if (ch != 0) fiber_control::schedule_tid(ch);\n }\n }\n }\n\n }\n else {\n // update the token\n size_t callsrecv;\n size_t callssent;\n \n if (attachedobj) {\n callsrecv = attachedobj->calls_received();\n callssent = attachedobj->calls_sent();\n }\n else {\n callsrecv = rmi.dc().calls_received();\n callssent = rmi.dc().calls_sent();\n }\n\n if (callssent != last_calls_sent ||\n callsrecv != last_calls_received) {\n cur_token.total_calls_sent += callssent - last_calls_sent;\n cur_token.total_calls_received += callsrecv - last_calls_received;\n cur_token.last_change = rmi.procid();\n }\n //std::cout << \"Sending token: (\" << cur_token.total_calls_sent\n //<< \", \" << cur_token.total_calls_received << \")\" << std::endl;\n\n last_calls_sent = callssent;\n last_calls_received = callsrecv;\n // send it along.\n hastoken = false;\n logstream(LOG_INFO) << \"Passing Token \" << rmi.procid() << \"-->\" \n << (rmi.procid() + 1) % rmi.numprocs() << \": \"\n << cur_token.total_calls_received << \" \" \n << cur_token.total_calls_sent << std::endl;\n\n rmi.control_call((procid_t)((rmi.procid() + 1) % rmi.numprocs()),\n &fiber_async_consensus::receive_the_token,\n cur_token);\n }\n }\n}\n\n\n" }, { "path": "src/graphlab/rpc/fiber_async_consensus.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef FIBER_ASYNC_TERMINATOR_HPP\n#define FIBER_ASYNC_TERMINATOR_HPP\n\n#include <graphlab/parallel/pthread_tools.hpp>\n\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_dist_object_base.hpp>\n#include <graphlab/rpc/dc_dist_object.hpp>\n\n\nnamespace graphlab {\n /**\n * \\ingroup rpc\n * \\brief This implements a distributed consensus algorithm which waits\n * for global completion of all computation/RPC events on a given object.\n *\n * The use case is as follows:\n * \n * A collection of fibers on a collection of distributed machines, each\n * running the following\n * \n * \\code\n * while (work to be done) {\n * Do_stuff\n * }\n * \\endcode\n * \n * However, <tt>Do_stuff</tt> will include RPC calls which may introduce\n * work to other fibers/machines.\n * Figuring out when termination is possible is complex. For instance RPC calls \n * could be in-flight and not yet received. This fiber_async_consensus class \n * implements a solution built around the algorithm in\n * <i>Misra, J.: Detecting Termination of Distributed Computations Using Markers, SIGOPS, 1983</i>\n * extended to handle the mixed parallelism (distributed with threading) case.\n * \n * The main loop of the user has to be modified to:\n * \n * \\code\n * done = false;\n * while (!done) {\n * Do_stuff\n * // if locally, I see that there is no work to be done\n * // we begin the consensus checking\n * if (no work to be done) {\n * // begin the critical section and try again\n * consensus.begin_done_critical_section();\n * // if still no work to be done\n * if (no work to be done) {\n * done = consensus.end_done_critical_section();\n * }\n * else {\n * consensus.cancel_critical_section();\n * }\n * }\n * }\n * \n * \\endcode\n * \n * Additionally, incoming RPC calls which create work must ensure there are\n * active fiberswhich are capable of processing the work. An easy solution \n * will be to simply cancel_one(). Other more optimized solutions\n * include keeping a counter of the number of active fibers, and only calling\n * cancel() or cancel_one() if all fibers are asleep. (Note that the optimized\n * solution does require some care to ensure dead-lock free execution).\n *\n * This class works with fibers. For a version which works with regular \n * kernel threads see \\ref graphlab::async_consensus .\n *\n * \\see graphlab::async_consensus\n */\n class fiber_async_consensus {\n public:\n /** \\brief Constructs an asynchronous consensus object\n *\n * The consensus procedure waits till all fibers have no work to do and are \n * waiting in consensus, and there is no communication going on which\n * could wake up a thread. The consensus object is therefore associated\n * with a communication context, either a graphlab::dc_dist_object,\n * or the global context (the root distributed_control).\n * \n * \\param dc The distributed control object to use for communication\n * \\param required_fibers_in_done local consensus is achieved if this many\n * fibers are waiting for consensus locally.\n * \\param attach The context to associate with. If NULL, we associate with\n * the global context. \n */\n fiber_async_consensus(distributed_control &dc, size_t required_fibers_in_done = 1,\n const dc_impl::dc_dist_object_base* attach = NULL);\n\n\n /**\n * \\brief A thread enters the critical section by calling\n * this function. \n * \n * After which it should check its termination \n * condition locally. If termination condition\n * is still fulfilled, end_done_critical_section() should be called.\n * Otherwise cancel_critical_section() should be called.\n *\n * \\param cpuid Thread ID of the thread.\n */\n void begin_done_critical_section(size_t cpuid);\n\n /**\n * \\brief Leaves the critical section because termination condition\n * is not fullfilled.\n *\n * See begin_done_critical_section()\n * \\param cpuid Thread ID of the thread.\n */\n void cancel_critical_section(size_t cpuid);\n\n /**\n * \\brief Thread must call this function if termination condition\n * is fullfilled while in the critical section. \n *\n * See begin_done_critical_section()\n * \n * \\param cpuid Thread ID of the thread.\n * \\returns True if global consensus is achieved. False otherwise. \n */\n bool end_done_critical_section(size_t cpuid);\n\n /**\n * \\brief Forces the consensus to be set\n */\n void force_done();\n \n \n /// \\brief Wakes up all local fibers waiting in done()\n void cancel();\n\n /// \\brief Wakes up a specific thread waiting in done()\n void cancel_one(size_t cpuid);\n\n /// \\brief Returns true if consensus is achieved.\n bool is_done() const {\n return done;\n }\n /** \\brief Resets the consensus object. Must be called simultaneously by\n * exactly one thread on each machine.\n * This function is not safe to call while consensus is being achieved.\n */\n void reset();\n \n private:\n\n /**\n * The token object that is passed around the machines.\n * It counts the total number of RPC calls that has been sent\n * or received, as well as the machine which last changed the value.\n * When the token goes one full round with no change, consensus is\n * achieved.\n */\n struct token {\n size_t total_calls_sent;\n size_t total_calls_received;\n procid_t last_change;\n void save(oarchive &oarc) const {\n oarc << total_calls_sent << total_calls_received << last_change;\n }\n\n void load(iarchive &iarc) {\n iarc >> total_calls_sent >> total_calls_received >> last_change;\n }\n };\n\n \n dc_dist_object<fiber_async_consensus> rmi;\n const dc_impl::dc_dist_object_base* attachedobj;\n \n size_t last_calls_sent;\n size_t last_calls_received;\n\n \n \n /// counts the number of fibers which are not sleeping\n /// protected by the mutex\n size_t numactive; \n \n /// Total number of CPUs\n size_t ncpus;\n \n /// once flag is set, the terminator is invalid, and all fibers\n /// should leave\n bool done;\n \n /// set if abort() is called\n // bool forced_abort; \n \n /// Number of fibers which have called\n /// begin_critical_section(), and have not left end_critical_section()\n /// This is an atomic counter and is not protected.\n atomic<size_t> trying_to_sleep;\n \n /// critical[i] is set if thread i has called \n /// begin_critical_section(), but has not left end_critical_section()\n /// sum of critical should be the same as trying_to_sleep\n std::vector<char> critical;\n \n /// sleeping[i] is set if fibers[i] is in cond.wait()\n std::vector<char> sleeping;\n \n \n bool hastoken;\n /// If I have the token, the value of the token\n token cur_token;\n\n mutex m;\n\n /*\n * Now, to work with fibers, the basic trick here is that the \n * async_consensus implementation uses exactly one thread waiting on each\n * condition variable. As such, we can just replace this with a deschedule\n * operation.\n *\n * We set cond[cpuid] to the fiber ID if there is a thread\n * waiting on it, and 0 otherwise.\n */\n std::vector<size_t> cond;\n \n\n void receive_the_token(token &tok);\n void pass_the_token();\n };\n\n}\n#endif\n\n\n\n" }, { "path": "src/graphlab/rpc/fiber_buffered_exchange.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_FIBER_BUFFERED_EXCHANGE_HPP\n#define GRAPHLAB_FIBER_BUFFERED_EXCHANGE_HPP\n\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/parallel/fiber_control.hpp>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_dist_object.hpp>\n#include <graphlab/util/mpi_tools.hpp>\n\n\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n\n /**\n * \\ingroup rpc\n *\n * The buffered exchange provides high performance exchange of bulk data \n * between machines. This is like the \\ref graphlab::buffered_exchange, but is\n * much stricter. All send calls must occur within fibers, and all recv calls\n * must occur within fibers. Specifically, a collection of \"fiber-worker-local\"\n * send buffer and receive buffers are created. Sends by fibers write into the\n * buffer owned by the current worker handling the fiber. Receives similarly\n * read from the buffer owned by the current worker handling the fiber.\n * As such a bit more subtlety is needed to use this class correctly.\n *\n * For instance, if we are doing bulk exchanges of integers:\n * \\code\n * buffered_exchange<int> exchange(dc);\n * .. In parallel in fibers .. {\n * exchange.send([target machine], [value to send to target])\n * exchange.partial_flush();\n * }\n *\n * .. now in 1 thread ..\n * exchange.flush()\n *\n * .. In parallel in fibers .. {\n * procid_t proc;\n * std::vector<buffered_exchange<int>::buffer_record> buffer; // (an array of buffers)\n * while(exchange.recv(buffer)) {\n * for each buffer_record in buffer:\n * buffer_record.proc is the machine which sent the contents of this record\n * buffer_record.buffer is an array containing values sent by the machine buffer_record.proc\n * }\n * }\n * \n * .. now in 1 thread ..\n * exchange.recv(buffer, false); // get from all receive buffers\n * while(exchange.recv(buffer)) {\n * for each buffer_record in buffer:\n * buffer_record.proc is the machine which sent the contents of this record\n * buffer_record.buffer is an array containing values sent by the machine buffer_record.proc\n * }\n * \\endcode\n *\n * \\note The buffered exchange sends data in the background, so recv can be\n * called even before the flush calls.\n * \\note The last single threaded receive is not necessary if worker-affinity\n * is set correctly so that every worker is active in the parallel receiving\n * block.\n *\n * \\see graphlab::buffered_exchange\n */\n template<typename T>\n class fiber_buffered_exchange {\n public:\n typedef std::vector<T> buffer_type;\n\n struct buffer_record {\n procid_t proc;\n buffer_type buffer;\n buffer_record() : proc(-1) { }\n }; // end of buffer record\n typedef std::vector<buffer_record> recv_buffer_type;\n mutex lock;\n private:\n\n\n\n /** The rpc interface for this class */\n mutable dc_dist_object<fiber_buffered_exchange> rpc;\n\n std::vector<std::vector< buffer_record> > recv_buffers;\n\n\n struct send_record {\n oarchive* oarc;\n size_t numinserts;\n };\n\n std::vector<std::vector<send_record> > send_buffers;\n const size_t max_buffer_size;\n\n\n /**\n * Flushes the send buffer local to worker id \"wid\" and going to process proc\n */\n void flush_buffer(size_t wid, procid_t proc) {\n if(send_buffers[wid][proc].oarc) {\n // write the length at the end of the buffere are returning\n send_buffers[wid][proc].oarc->write(reinterpret_cast<char*>(&send_buffers[wid][proc].numinserts), sizeof(size_t));\n rpc.split_call_end(proc, send_buffers[wid][proc].oarc);\n// logstream(LOG_DEBUG) << rpc.procid() << \": Sending exchange of length \" \n// << send_buffers[wid][proc].oarc->off << \" to \" \n// << proc << std::endl;\n send_buffers[wid][proc].oarc = NULL;\n send_buffers[wid][proc].numinserts = 0;\n }\n }\n\n public:\n /**\n * Constructs a buffered exchange object.\n *\n * \\ref dc The master distributed_control object\n * \\ref max_buffer_size The size of the per thread and per target send buffer.\n */\n fiber_buffered_exchange(distributed_control& dc,\n const size_t max_buffer_size = DEFAULT_BUFFERED_EXCHANGE_SIZE) :\n rpc(dc, this),\n max_buffer_size(max_buffer_size) {\n send_buffers.resize(fiber_control::get_instance().num_workers());\n recv_buffers.resize(fiber_control::get_instance().num_workers());\n for (size_t i = 0;i < send_buffers.size(); ++i) {\n send_buffers[i].resize(dc.numprocs());\n for (size_t j = 0;j < send_buffers[i].size(); ++j) {\n send_buffers[i][j].oarc = NULL;\n send_buffers[i][j].numinserts = 0;\n }\n }\n rpc.barrier();\n }\n\n\n ~fiber_buffered_exchange() {\n // clear the send buffers\n for (size_t i = 0;i < send_buffers.size(); ++i) {\n for (size_t j = 0;j < send_buffers[i].size(); ++j) {\n if (send_buffers[i][j].oarc) rpc.split_call_cancel(send_buffers[i][j].oarc);\n }\n }\n }\n // fiber_buffered_exchange(distributed_control& dc, handler_type recv_handler,\n // size_t buffer_size = 1000) :\n // rpc(dc, this), send_buffers(dc.numprocs()), send_locks(dc.numprocs()),\n // max_buffer_size(buffer_size), recv_handler(recv_handler) { rpc.barrier(); }\n\n\n /**\n * Sends a value to a target machine.\n * Must be called from within a fiber\n */\n void send(const procid_t proc, const T& value) {\n size_t wid = fiber_control::get_worker_id();\n if (send_buffers[wid][proc].oarc == NULL) {\n send_buffers[wid][proc].oarc = rpc.split_call_begin(&fiber_buffered_exchange::rpc_recv);\n // write a header\n (*send_buffers[wid][proc].oarc) << rpc.procid();\n send_buffers[wid][proc].numinserts = 0;\n }\n\n (*(send_buffers[wid][proc].oarc)) << value;\n ++send_buffers[wid][proc].numinserts;\n\n\n if(send_buffers[wid][proc].oarc->off >= max_buffer_size) {\n flush_buffer(wid, proc);\n }\n } // end of send\n\n /**\n * Flushes the send buffers owned by the worker currently running the \n * current fiber.\n */\n void partial_flush() {\n for(procid_t proc = 0; proc < rpc.numprocs(); ++proc) {\n flush_buffer(fiber_control::get_worker_id(), proc);\n }\n }\n\n /**\n * Flushes all send buffers. Must be called only on one thread.\n * Will not return until all machines call flush.\n */\n void flush() {\n for(size_t i = 0; i < send_buffers.size(); ++i) {\n for (size_t j = 0;j < send_buffers[i].size(); ++j) {\n flush_buffer(i,j);\n }\n }\n rpc.dc().flush();\n rpc.full_barrier();\n } // end of flush\n\n /**\n * Receives a collection of buffers.\n * Must be called from within a fiber.\n * \\param ret_buffer If return value is true, this contains a collection of \n * buffers sent to this machine.\n * \\param self_buffer If true, only receives the worker local buffers.\n * \\returns true If ret_buffer contains values.\n */\n bool recv(std::vector<buffer_record>& ret_buffer,\n const bool self_buffer = true) {\n fiber_control::fast_yield();\n ret_buffer.clear();\n bool success = false;\n if (self_buffer) {\n // get from my own buffer\n size_t wid = fiber_control::get_worker_id();\n lock.lock();\n if(!recv_buffers[wid].empty()) {\n success = true;\n std::swap(ret_buffer, recv_buffers[wid]);\n } else {\n for (size_t i = 0;i < recv_buffers.size(); ++i) {\n if(!recv_buffers[i].empty()) {\n success = true;\n std::swap(ret_buffer, recv_buffers[i]);\n break;\n }\n }\n }\n lock.unlock();\n } else {\n for (size_t i = 0;i < recv_buffers.size(); ++i) {\n if(!recv_buffers[i].empty()) {\n success = true;\n std::swap(ret_buffer, recv_buffers[i]);\n break;\n }\n }\n }\n return success;\n } // end of recv\n\n\n\n /**\n * Returns the number of elements avalable for receiving. \n */\n size_t size() const {\n size_t count = 0;\n for (size_t i = 0;i < recv_buffers.size(); ++i) {\n count += recv_buffers[i].size();\n }\n return count;\n } // end of size\n\n /**\n * Returns true if there are no elements to receive.\n */\n bool empty() const { \n for (size_t i = 0;i < recv_buffers.size(); ++i) {\n if (recv_buffers[i].size() > 0) return false;\n }\n return true;\n }\n\n void clear() { }\n\n void barrier() { rpc.barrier(); }\n private:\n void rpc_recv(size_t len, wild_pointer w) {\n buffer_type tmp;\n iarchive iarc(reinterpret_cast<const char*>(w.ptr), len);\n // first desrialize the source process\n procid_t src_proc; iarc >> src_proc;\n// logstream(LOG_DEBUG) << rpc.procid() << \": Receiving exchange of length \"\n// << len << \" from \" << src_proc << std::endl;\n // create an iarchive which just points to the last size_t bytes\n // to get the number of elements\n iarchive numel_iarc(reinterpret_cast<const char*>(w.ptr) + len - sizeof(size_t),\n sizeof(size_t));\n size_t numel = 0; \n numel_iarc.read(reinterpret_cast<char*>(&numel), sizeof(size_t));\n //std::cout << \"Receiving: \" << numel << \"\\n\";\n tmp.resize(numel);\n for (size_t i = 0;i < numel; ++i) {\n iarc >> tmp[i];\n }\n\n size_t wid = fiber_control::get_worker_id();\n lock.lock();\n recv_buffers[wid].push_back(buffer_record());\n buffer_record& rec = recv_buffers[wid].back();\n rec.proc = src_proc;\n rec.buffer.swap(tmp);\n lock.unlock();\n } // end of rpc rcv\n\n\n\n }; // end of buffered exchange\n\n\n}; // end of graphlab namespace\n#include <graphlab/macros_undef.hpp>\n\n#endif\n\n\n" }, { "path": "src/graphlab/rpc/function_arg_types_def.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifdef __GLRPC_F0\n#error \"multiple includes of function arg types\" \n#endif\n\n#include <boost/type_traits/decay.hpp>\n#include <boost/type_traits/remove_pointer.hpp>\n#include <boost/type_traits/remove_const.hpp>\n#include <boost/type_traits/function_traits.hpp>\n\n#include <boost/function.hpp>\n/*\nA huge collection of useful typedefs.\nF0... F5: identifies the arguments for an RPC aware function F. Dropping const and dropping references \n (therefore allowing you to use F0....F5 to do casting.\n\n__GLRPC_NIF0... __GLRPC_NIF5: identifies the arguments for an RPC unaware function F\n\n__GLRPC_R0.... __GLRPC_R7: Identifies the actual arguments of the function F, without de-consting and de-reffing\n\n__GLRPC_FRESULT: de-const and de-refed type of F's return type\n\n__GLRPC_FARITY: the number of arguments F takes\n*/\n#define REMOVE_CONST_REF(REF) typename boost::remove_const<typename boost::remove_reference<REF>::type>::type\n// non-intrusive calls\n#define __GLRPC_NIF0 REMOVE_CONST_REF(typename boost::function<typename boost::remove_pointer<F>::type>::arg1_type)\n#define __GLRPC_NIF1 REMOVE_CONST_REF(typename boost::function<typename boost::remove_pointer<F>::type>::arg2_type)\n#define __GLRPC_NIF2 REMOVE_CONST_REF(typename boost::function<typename boost::remove_pointer<F>::type>::arg3_type)\n#define __GLRPC_NIF3 REMOVE_CONST_REF(typename boost::function<typename boost::remove_pointer<F>::type>::arg4_type)\n#define __GLRPC_NIF4 REMOVE_CONST_REF(typename boost::function<typename boost::remove_pointer<F>::type>::arg5_type)\n#define __GLRPC_NIF5 REMOVE_CONST_REF(typename boost::function<typename boost::remove_pointer<F>::type>::arg6_type)\n#define __GLRPC_NIF6 REMOVE_CONST_REF(typename boost::function<typename boost::remove_pointer<F>::type>::arg7_type)\n#define __GLRPC_NIF7 REMOVE_CONST_REF(typename boost::function<typename boost::remove_pointer<F>::type>::arg8_type)\n\n// for rpc aware functions\n#define __GLRPC_F0 __GLRPC_NIF2\n#define __GLRPC_F1 __GLRPC_NIF3\n#define __GLRPC_F2 __GLRPC_NIF4\n#define __GLRPC_F3 __GLRPC_NIF5\n#define __GLRPC_F4 __GLRPC_NIF6\n#define __GLRPC_F5 __GLRPC_NIF7\n\n\n\n#define __GLRPC_R0 typename boost::function<typename boost::remove_pointer<F>::type>::arg1_type\n#define __GLRPC_R1 typename boost::function<typename boost::remove_pointer<F>::type>::arg2_type\n#define __GLRPC_R2 typename boost::function<typename boost::remove_pointer<F>::type>::arg3_type\n#define __GLRPC_R3 typename boost::function<typename boost::remove_pointer<F>::type>::arg4_type\n#define __GLRPC_R4 typename boost::function<typename boost::remove_pointer<F>::type>::arg5_type\n#define __GLRPC_R5 typename boost::function<typename boost::remove_pointer<F>::type>::arg6_type\n#define __GLRPC_R6 typename boost::function<typename boost::remove_pointer<F>::type>::arg7_type\n#define __GLRPC_R7 typename boost::function<typename boost::remove_pointer<F>::type>::arg8_type\n\n#define __GLRPC_FRESULT REMOVE_CONST_REF(typename boost::function<typename boost::remove_pointer<F>::type>::result_type)\n\n#define __GLRPC_FARITY boost::function<typename boost::remove_pointer<F>::type>::arity\n\n" }, { "path": "src/graphlab/rpc/function_arg_types_undef.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#undef __GLRPC_F0\n#undef __GLRPC_F1\n#undef __GLRPC_F2\n#undef __GLRPC_F3\n#undef __GLRPC_F4\n#undef __GLRPC_F5\n#undef __GLRPC_NIF0\n#undef __GLRPC_NIF1\n#undef __GLRPC_NIF2\n#undef __GLRPC_NIF3\n#undef __GLRPC_NIF4\n#undef __GLRPC_NIF5\n#undef __GLRPC_NIF6\n#undef __GLRPC_NIF7\n#undef __GLRPC_R0\n#undef __GLRPC_R1\n#undef __GLRPC_R2\n#undef __GLRPC_R3\n#undef __GLRPC_R4\n#undef __GLRPC_R5\n#undef __GLRPC_R6\n#undef __GLRPC_R7\n\n#undef __GLRPC_FRESULT\n#undef REMOVE_CONST_REF\n#undef __GLRPC_FARITY\n\n" }, { "path": "src/graphlab/rpc/function_broadcast_issue.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef FUNCTION_BROADCAST_ISSUE_HPP\n#define FUNCTION_BROADCAST_ISSUE_HPP\n#include <iostream>\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/dc_send.hpp>\n#include <graphlab/rpc/dc_thread_get_send_buffer.hpp>\n#include <graphlab/rpc/function_call_dispatch.hpp>\n#include <graphlab/rpc/function_call_issue.hpp>\n#include <graphlab/rpc/is_rpc_call.hpp>\n#include <boost/preprocessor.hpp>\n#include <graphlab/rpc/function_arg_types_def.hpp>\n\nnamespace graphlab{\nnamespace dc_impl {\n\n/**\n\n\\ingroup rpc\n\\internal\n\\file function_broadcast_issue.hpp\n\nSee function_call_issue.hpp for details. This is equivalent to the macro\nexpansion in remote_call_issue with the difference that this takes an iterator \nsequence listing the machines to send to.\n\nThe code below generates the following for different number of arguments. Here, \nwe demonstrate the 1 argument version.\n\\code\ntemplate < typename Iterator, typename F, typename T0 > \nclass remote_broadcast_issue1 {\n public:\n static void exec (std::vector < dc_send * >&sender, unsigned char flags,\n Iterator target_begin, Iterator target_end,\n F remote_function, const T0 & i0) {\n oarchive arc;\n arc.buf = (char *) malloc (65536);\n arc.len = 65536;\n size_t len =\n dc_send::write_packet_header (arc, _get_procid (), flags,\n _get_sequentialization_key ());\n uint32_t beginoff = arc.off;\n dispatch_type d =\n function_call_issue_detail::dispatch_selector1 < typename is_rpc_call <\n F >::type, F, T0 >::dispatchfn ();\n arc << reinterpret_cast < size_t > (d);\n arc << reinterpret_cast < size_t > (remote_function);\n arc << i0;\n *(reinterpret_cast < uint32_t * >(arc.buf + len)) = arc.off - beginoff;\n Iterator iter = target_begin;\n while (iter != target_end) {\n oarchive *buf = get_thread_local_buffer (*iter);\n buf->write (arc.buf, arc.off);\n release_thread_local_buffer (*iter, flags & CONTROL_PACKET);\n ++iter;\n }\n free (arc.buf);\n }\n};\n\\endcode\n*/\n\n\n\n#define GENARGS(Z,N,_) BOOST_PP_CAT(const T, N) BOOST_PP_CAT(&i, N)\n#define GENI(Z,N,_) BOOST_PP_CAT(i, N)\n#define GENT(Z,N,_) BOOST_PP_CAT(T, N)\n#define GENARC(Z,N,_) arc << BOOST_PP_CAT(i, N);\n\n\n/**\nThe dispatch_selectorN structs are used to pick between the standard dispatcher and the nonintrusive dispatch\nby checking if the function is a RPC style call or not.\n*/\n#define REMOTE_BROADCAST_ISSUE_GENERATOR(Z,N,FNAME_AND_CALL) \\\ntemplate<typename Iterator, typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nclass BOOST_PP_CAT(FNAME_AND_CALL, N) { \\\n public: \\\n static void exec(std::vector<dc_send*>& sender, unsigned char flags, Iterator target_begin, Iterator target_end, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n oarchive arc; \\\n arc.buf = (char*)malloc(INITIAL_BUFFER_SIZE); \\\n arc.len = INITIAL_BUFFER_SIZE; \\\n size_t len = dc_send::write_packet_header(arc, _get_procid(), flags, _get_sequentialization_key()); \\\n uint32_t beginoff = arc.off; \\\n dispatch_type d = BOOST_PP_CAT(function_call_issue_detail::dispatch_selector,N)<typename is_rpc_call<F>::type, F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, T) >::dispatchfn(); \\\n arc << reinterpret_cast<size_t>(d); \\\n arc << reinterpret_cast<size_t>(remote_function); \\\n BOOST_PP_REPEAT(N, GENARC, _) \\\n *(reinterpret_cast<uint32_t*>(arc.buf + len)) = arc.off - beginoff; \\\n Iterator iter = target_begin; \\\n while(iter != target_end) { \\\n oarchive* buf = get_thread_local_buffer(*iter); \\\n buf->write(arc.buf, arc.off); \\\n release_thread_local_buffer(*iter, flags & CONTROL_PACKET); \\\n ++iter; \\\n } \\\n free(arc.buf); \\\n if (flags & FLUSH_PACKET) pull_flush_soon_thread_local_buffer(); \\\n }\\\n};\n\n\n\n/**\nGenerates a function call issue. 3rd argument is the issue name\n*/\nBOOST_PP_REPEAT(6, REMOTE_BROADCAST_ISSUE_GENERATOR, remote_broadcast_issue )\n\n\n\n#undef GENARC\n#undef GENT\n#undef GENI\n#undef GENARGS\n#undef REMOTE_BROADCAST_ISSUE_GENERATOR\n\n} // namespace dc_impl\n} // namespace graphlab\n\n#include <graphlab/rpc/function_arg_types_undef.hpp>\n\n#endif\n\n" }, { "path": "src/graphlab/rpc/function_call_dispatch.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef REPACK_DISPATCH_HPP\n#define REPACK_DISPATCH_HPP\n#include <iostream>\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/function_arg_types_def.hpp>\n#include <boost/preprocessor.hpp>\nnamespace graphlab {\nnamespace dc_impl {\n\n/**\n\\ingroup rpc\n\\internal\n\\file function_call_dispatch.hpp\n\nThis is an internal function and should not be used directly\n\nA \"call\" is an RPC which is performed asynchronously.\nThere are 2 types of calls. A \"basic\" call calls a standard C/C++ function\nand does not require the function to be modified.\nwhile the \"regular\" call requires the first 2 arguments of the function \nto be \"distributed_control &dc, procid_t source\".\n\nA \"dispatch\" is a wrapper function on the receiving side of an RPC\nwhich decodes the packet and performs the function call.\n\nThis scary looking piece of code is actually quite straightforward.\nGiven function F, as well as input types T1 ... Tn\nit will construct an input archive and deserialize the types T1.... Tn,\nand call the function f with it. This code dispatches to the \"intrusive\" \nform of a function call (that is the function call must take a distributed_control\nand a \"procid_t source\" as its first 2 arguments.\n\nFor instance, the 1 argument version of this is DISPATCH1:\n\n\\code\ntemplate<typename DcType, typename F , typename T0> \nvoid DISPATCH1 (DcType& dc, procid_t source, unsigned char packet_type_mask, \n const char* buf, size_t len) { \n iarchive iarc(buf, len);\n size_t s;\n iarc >> s;\n F f = reinterpret_cast<F>(s);\n T0 (f0) ;\n iarc >> (f0) ;\n f(dc, source , (f0) );\n charstring_free(f0);\n} \n\\endcode\n\ncharstring_free is a special template function which calls free(f1)\nonly if f1 is a character array (char*)\n\nAnd similarly, the non-intrusive dispatch a little below\n\nNote that the template around DcType is *deliberate*. This prevents this\nfunction from instantiating the distributed_control until as late as possible, \navoiding problems with circular references.\n*/\n\n#define GENFN(N) BOOST_PP_CAT(__GLRPC_F, N)\n#define GENFN2(N) BOOST_PP_CAT(f, N)\n#define GENARGS(Z,N,_) (BOOST_PP_CAT(f, N))\n#define GENPARAMS(Z,N,_) BOOST_PP_CAT(T, N) (BOOST_PP_CAT(f, N)) ; iarc >> (BOOST_PP_CAT(f, N)) ;\n#define CHARSTRINGFREE(Z,N,_) charstring_free(BOOST_PP_CAT(f, N));\n\n#define DISPATCH_GENERATOR(Z,N,_) \\\ntemplate<typename DcType, typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nvoid BOOST_PP_CAT(DISPATCH,N) (DcType& dc, procid_t source, unsigned char packet_type_mask, \\\n const char* buf, size_t len) { \\\n iarchive iarc(buf, len); \\\n size_t s; iarc >> s; F f = reinterpret_cast<F>(s); \\\n BOOST_PP_REPEAT(N, GENPARAMS, _) \\\n f(dc, source BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ); \\\n BOOST_PP_REPEAT(N, CHARSTRINGFREE, _) \\\n} \n\nBOOST_PP_REPEAT(6, DISPATCH_GENERATOR, _)\n\n#undef GENFN\n#undef GENFN2\n#undef GENARGS\n#undef GENPARAMS\n#undef DISPATCH_GENERATOR\n\n\n\n/**\nThis is similar, but generates the non-intrusive version of a \ndispatcher. That is, the target function does not need to take\n\"distributed_control &dc, procid_t source\" as its first 2 arguments.\n\ntemplate<typename DcType, typename F , typename T0> \nvoid NONINTRUSIVE_DISPATCH1(DcType& dc, procid_t source, \n unsigned char packet_type_mask, \n const char* buf, size_t len) { \n iarchive iarc(buf, len);\n size_t s;\n iarc >> s;\n F f = reinterpret_cast<F>(s);\n T0 (f0) ;\n iarc >> (f0) ;\n f( (f0) );\n charstring_free(f0);\n}\n*/\n\n\n\n#define GENFN(N) BOOST_PP_CAT(__GLRPC_NIF, N)\n#define GENFN2(N) BOOST_PP_CAT(f, N)\n#define GENARGS(Z,N,_) (BOOST_PP_CAT(f, N))\n#define GENPARAMS(Z,N,_) BOOST_PP_CAT(T, N) (BOOST_PP_CAT(f, N)) ; iarc >> (BOOST_PP_CAT(f, N)) ;\n#define CHARSTRINGFREE(Z,N,_) charstring_free(BOOST_PP_CAT(f, N));\n\n\n#define NONINTRUSIVE_DISPATCH_GENERATOR(Z,N,_) \\\ntemplate<typename DcType, typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nvoid BOOST_PP_CAT(NONINTRUSIVE_DISPATCH,N) (DcType& dc, procid_t source, unsigned char packet_type_mask, \\\n const char* buf, size_t len) { \\\n iarchive iarc(buf, len); \\\n size_t s; iarc >> s; F f = reinterpret_cast<F>(s); \\\n BOOST_PP_REPEAT(N, GENPARAMS, _) \\\n f(BOOST_PP_ENUM(N,GENARGS ,_) ); \\\n BOOST_PP_REPEAT(N, CHARSTRINGFREE, _) \\\n} \n\nBOOST_PP_REPEAT(6, NONINTRUSIVE_DISPATCH_GENERATOR, _)\n\n#undef GENFN\n#undef GENFN2\n#undef GENARGS\n#undef GENPARAMS\n#undef NONINTRUSIVE_DISPATCH_GENERATOR\n\n} // namespace dc_impl\n} // namespace graphlab\n\n\n#include <graphlab/rpc/function_arg_types_undef.hpp>\n#endif\n\n" }, { "path": "src/graphlab/rpc/function_call_issue.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef FUNCTION_CALL_ISSUE_HPP\n#define FUNCTION_CALL_ISSUE_HPP\n#include <iostream>\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/dc_send.hpp>\n#include <graphlab/rpc/function_call_dispatch.hpp>\n#include <graphlab/rpc/is_rpc_call.hpp>\n#include <graphlab/rpc/request_reply_handler.hpp>\n#include <boost/preprocessor.hpp>\n#include <graphlab/rpc/dc_thread_get_send_buffer.hpp>\n#include <graphlab/rpc/dc_compile_parameters.hpp>\n#include <graphlab/rpc/function_arg_types_def.hpp>\n\nnamespace graphlab{\nnamespace dc_impl {\n\n/**\n\\ingroup rpc\n\\internal\n * \\file function_call_issue.hpp\n\nThis is an internal function and should not be used directly\n\nA \"call\" is an RPC which is performed asynchronously.\nThere are 2 types of calls. A \"basic\" call calls a standard C/C++ function\nand does not require the function to be modified.\nwhile the \"regular\" call requires the first 2 arguments of the function\nto be \"distributed_control &dc, procid_t source\".\n\nAn \"issue\" is a wrapper function on the sending side of an RPC\nwhich encodes the packet and transmits it to the other side.\n(I realized later this is called a \"Marshaller\")\n\nNative Call Formats \\n\n=================== \\n\nThe format of a \"call\" packet is in the form of an archive and is as follows\n\n\\li (dispatch_type*) -- pointer to target machine's dispatcher function\n\\li (void*) -- pointer to target function\n\\li fn::arg1_type -- target function's 1st argument\n\\li fn::arg2_type -- target function's 2nd argument\n\\li ...\n\\li fn::argN_type -- target function's Nth argument\n\nArgument casting is deferred to as late as possible. So the data type of\narguments are the data types that the caller use to call the function.\nA dispatcher function will be instantiated with the input types, which will\nthen perform the type cast.\n\n\nThe code below generates the following for different number of arguments. Here, we\ndemonstrate the 1 argument version.\n\\code\nnamespace function_call_issue_detail {\n template < typename BoolType, typename F, typename T0 > struct dispatch_selector1 {\n static dispatch_type dispatchfn () {\n return dc_impl::NONINTRUSIVE_DISPATCH1 < distributed_control, F, T0 >;\n }\n };\n template < typename F, typename T0 > \n struct dispatch_selector1 <boost::mpl::bool_ < true >, F, T0 > {\n static dispatch_type dispatchfn () {\n return dc_impl::DISPATCH1 < distributed_control, F, T0 >;\n }\n };\n}\n\ntemplate < typename F, typename T0 > \nclass remote_call_issue1 {\n public:\n static void exec (dc_send * sender, unsigned char flags, procid_t target,\n F remote_function, const T0 & i0) {\n oarchive *ptr = get_thread_local_buffer (target);\n oarchive & arc = *ptr;\n if (reinterpret_cast < size_t > (remote_function) == reinterpret_cast <\n\tsize_t > (request_reply_handler)) {\n flags |= REPLY_PACKET;\n }\n size_t len =\n dc_send::write_packet_header (arc, _get_procid (), flags,\n\t\t\t\t _get_sequentialization_key ());\n uint32_t beginoff = arc.off;\n dispatch_type d =\n function_call_issue_detail::dispatch_selector1 < typename is_rpc_call <\n F >::type, F, T0 >::dispatchfn ();\n arc << reinterpret_cast < size_t > (d);\n arc << reinterpret_cast < size_t > (remote_function);\n arc << i0;\n *(reinterpret_cast < uint32_t * >(arc.buf + len)) = arc.off - beginoff;\n release_thread_local_buffer (target, flags & CONTROL_PACKET);\n }\n};\n\\endcode\n\nThe basic idea of the code is straightforward.\nThe receiving end cannot call the target function (remote_function) directly, since it has\nno means of understanding how to deserialize or to construct the stack for the remote_function.\nSo instead, we generate a \"dispatch\" function on the receiving side. The dispatch function\nis constructed according to the type information of the remote_function, and therefore knows\nhow to deserialize the data, and issue the function call. That is the \"dispatch_type\".\n\nHowever, since we defined two families of receiving functions:\n a non-intrusive version which does not take (dc, procid) as an argument\n and an intrusive version which does, the dispatch function must therefore be slightly different\n for each of them. That is what the dispatch_selector class performs.\n The first template argument of the dispatch_selector family of classes is a boolean flag which\n denotes whether the function is a non-intrusive call or not. This boolean flag itself\n is determined using the is_rpc_call<F>::type template.\n*/\n\n\n\n#define GENARGS(Z,N,_) BOOST_PP_CAT(const T, N) BOOST_PP_CAT(&i, N)\n#define GENI(Z,N,_) BOOST_PP_CAT(i, N)\n#define GENT(Z,N,_) BOOST_PP_CAT(T, N)\n#define GENARC(Z,N,_) arc << BOOST_PP_CAT(i, N);\n\n\n/**\nThe dispatch_selectorN structs are used to pick between the standard dispatcher and the nonintrusive dispatch\nby checking if the function is a RPC style call or not.\n*/\n#define REMOTE_CALL_ISSUE_GENERATOR(Z,N,FNAME_AND_CALL) \\\nnamespace function_call_issue_detail { \\\ntemplate <typename BoolType, typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nstruct BOOST_PP_CAT(dispatch_selector, N){ \\\n static dispatch_type dispatchfn() { return BOOST_PP_CAT(dc_impl::NONINTRUSIVE_DISPATCH,N)<distributed_control,F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N, GENT ,_) >; } \\\n};\\\ntemplate <typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nstruct BOOST_PP_CAT(dispatch_selector, N)<boost::mpl::bool_<true>, F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, T)>{ \\\n static dispatch_type dispatchfn() { return BOOST_PP_CAT(dc_impl::DISPATCH,N)<distributed_control,F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N, GENT ,_) >; } \\\n}; \\\n} \\\ntemplate<typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nclass BOOST_PP_CAT(FNAME_AND_CALL, N) { \\\n public: \\\n static void exec(dc_send* sender, unsigned char flags, procid_t target, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n oarchive* ptr = get_thread_local_buffer(target); \\\n oarchive& arc = *ptr; \\\n size_t len = dc_send::write_packet_header(arc, _get_procid(), flags, _get_sequentialization_key()); \\\n uint32_t beginoff = arc.off; \\\n dispatch_type d = BOOST_PP_CAT(function_call_issue_detail::dispatch_selector,N)<typename is_rpc_call<F>::type, F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, T) >::dispatchfn(); \\\n arc << reinterpret_cast<size_t>(d); \\\n arc << reinterpret_cast<size_t>(remote_function); \\\n BOOST_PP_REPEAT(N, GENARC, _) \\\n *(reinterpret_cast<uint32_t*>(arc.buf + len)) = arc.off - beginoff; \\\n release_thread_local_buffer(target, flags & CONTROL_PACKET); \\\n if (flags & FLUSH_PACKET) pull_flush_soon_thread_local_buffer(target); \\\n }\\\n};\n\n\n\n/**\nGenerates a function call issue. 3rd argument is the issue name\n*/\nBOOST_PP_REPEAT(6, REMOTE_CALL_ISSUE_GENERATOR, remote_call_issue )\n\n\n\n#undef GENARC\n#undef GENT\n#undef GENI\n#undef GENARGS\n#undef REMOTE_CALL_ISSUE_GENERATOR\n\n} // namespace dc_impl\n} // namespace graphlab\n\n#include <graphlab/rpc/function_arg_types_undef.hpp>\n\n#endif\n\n" }, { "path": "src/graphlab/rpc/function_ret_type.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef FUNCTION_RETURN_TYPE_HPP\n#define FUNCTION_RETURN_TYPE_HPP\n#include <boost/preprocessor.hpp>\n#include <graphlab/rpc/function_arg_types_def.hpp>\nnamespace graphlab {\nnamespace dc_impl {\n\n \n/**\n\\ingroup rpc\n\\internal\n\nThis struct performs two duties.\nFirstly, it provides a consistent interface through a function called ::fcallN<F>\nto complete a function call with a variable number of arguments.\nNext, it provides the type of the return value of the function in ::type.\nIf the return type is void, it is promoted to an int. This makes the output\ntype of the function call be always serializable, simplifying the implementation\nof \"requests\".\n*/\ntemplate <typename RetType>\nstruct function_ret_type {\n typedef RetType type;\n \n #define GENARGS(Z,N,_) BOOST_PP_CAT(__GLRPC_R, N) BOOST_PP_CAT(i, N)\n \n #define FCALL(Z, N, _) \\\n template <typename F> \\\n static RetType BOOST_PP_CAT(fcall, N)(F f BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N, GENARGS, _)){ \\\n return f(BOOST_PP_ENUM_PARAMS(N, i)); \\\n } \n \n BOOST_PP_REPEAT(8, FCALL , _ )\n\n #undef FCALL\n #undef GENARGS\n\n};\n\ntemplate <>\nstruct function_ret_type<void> {\n typedef size_t type;\n \n #define GENARGS(Z,N,_) BOOST_PP_CAT(__GLRPC_R, N) BOOST_PP_CAT(i, N)\n \n #define FCALL(Z, N, _) \\\n template <typename F> \\\n static size_t BOOST_PP_CAT(fcall, N)(F f BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N, GENARGS, _)){ \\\n f(BOOST_PP_ENUM_PARAMS(N, i)); \\\n return 0; \\\n } \n \n BOOST_PP_REPEAT(8, FCALL , _ )\n\n #undef FCALL\n #undef GENARGS\n\n};\n#include <graphlab/rpc/function_arg_types_undef.hpp>\n\n\n} // namespace dc_impl\n} // namespace graphlab\n\n#include <graphlab/rpc/mem_function_arg_types_def.hpp>\n\nnamespace graphlab {\nnamespace dc_impl {\n\n/**\nThis struct performs two duties.\nFirstly, it provides a consistent interface through a function called ::fcallN<F>\nto complete a \\b member function call with a variable number of arguments.\nNext, it provides the type of the return value of the function in ::type.\nIf the return type is void, it is promoted to an int. This makes the output\ntype of the function call be always serializable, simplifying the implementation\nof \"requests\".\n*/\ntemplate <typename RetType>\nstruct mem_function_ret_type {\n typedef RetType type;\n \n #define GENARGS(Z,N,_) BOOST_PP_CAT(__GLRPC_R, N) BOOST_PP_CAT(i, N)\n \n #define FCALL(Z, N, _) \\\n template <typename F, typename T> \\\n static RetType BOOST_PP_CAT(fcall, N)(F f , T t BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N, GENARGS, _)){ \\\n return (t->*f)(BOOST_PP_ENUM_PARAMS(N, i)); \\\n }\n\n BOOST_PP_REPEAT(8, FCALL , _ )\n\n #undef FCALL\n #undef GENARGS\n\n};\n\ntemplate <>\nstruct mem_function_ret_type<void> {\n typedef size_t type;\n \n #define GENARGS(Z,N,_) BOOST_PP_CAT(__GLRPC_R, N) BOOST_PP_CAT(i, N)\n \n #define FCALL(Z, N, _) \\\n template <typename F, typename T> \\\n static size_t BOOST_PP_CAT(fcall, N)(F f , T t BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N, GENARGS, _)){ \\\n (t->*f)(BOOST_PP_ENUM_PARAMS(N, i)); \\\n return 0; \\\n }\n\n BOOST_PP_REPEAT(8, FCALL , _ )\n\n #undef FCALL\n #undef GENARGS\n\n};\n\n\n\n\n} // namespace dc_impl\n} // namespace graphlab\n\n#include <graphlab/rpc/mem_function_arg_types_undef.hpp>\n\n#endif\n\n" }, { "path": "src/graphlab/rpc/get_current_process_hash.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <graphlab/rpc/get_current_process_hash.hpp>\n#include <graphlab/ui/mongoose/mongoose.h>\n\n#ifdef __APPLE__\n#include <stdio.h>\n#include <stdlib.h>\n#include <string.h>\n#include <errno.h>\n#include <libproc.h>\n#include <unistd.h>\n#endif\nnamespace graphlab {\nnamespace dc_impl {\n\n\n#ifdef __linux\nstd::string get_current_process_hash() {\n char buf[33];\n mg_md5_file(buf, \"/proc/self/exe\");\n buf[32] = '\\0';\n std::string ret = buf;\n if (ret.length() != 32) {\n ret = std::string(32, '0');\n }\n return ret;\n}\n#elif __APPLE__\nstd::string get_current_process_hash() {\n std::string ret;\n\n pid_t pid = getpid();\n char pathbuf[PROC_PIDPATHINFO_MAXSIZE];\n int pidsuccess = proc_pidpath (pid, pathbuf, sizeof(pathbuf));\n if (pidsuccess > 0) {\n char buf[33];\n mg_md5_file(buf, pathbuf);\n buf[32] = '\\0';\n ret = buf;\n }\n if (ret.length() != 32) {\n ret = std::string(32, '0');\n }\n return ret;\n}\n#endif\n\n} // dc_impl\n} // graphlab\n" }, { "path": "src/graphlab/rpc/get_current_process_hash.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_RPC_GET_CURRENT_PROCESS_HASH_HPP\n#define GRAPHLAB_RPC_GET_CURRENT_PROCESS_HASH_HPP\n#include <string>\nnamespace graphlab {\nnamespace dc_impl {\nstd::string get_current_process_hash();\n} // dc_impl \n} // graphlab\n#endif\n" }, { "path": "src/graphlab/rpc/is_rpc_call.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef IS_RPC_CALL_HPP\n#define IS_RPC_CALL_HPP\n#include <boost/type_traits/remove_pointer.hpp>\n#include <boost/type_traits/remove_const.hpp>\n#include <boost/type_traits/function_traits.hpp>\n#include <boost/type_traits/is_same.hpp>\n#include <boost/mpl/if.hpp>\n#include <boost/mpl/and.hpp>\n#include <boost/mpl/bool.hpp>\n#include <boost/mpl/less.hpp>\n#include <boost/mpl/comparison.hpp>\n#include <boost/mpl/int.hpp>\n\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/function_arg_types_def.hpp>\n\n\nnamespace graphlab {\nclass distributed_control;\nnamespace dc_impl {\n\nnamespace is_rpc_call_detail {\n/**\n\\ingroup rpc\n\\internal\nWhether the function has less than or equal to 2 arguments\n*/\ntemplate <typename F>\nstruct less_than_2_args {\n typedef typename boost::mpl::bool_<__GLRPC_FARITY < 2 >::type type; \n};\n\n\n/**\n\\ingroup rpc\n\\internal\nNow, arg1_type and arg_2 type may not exist in function_traits if the \nnumber of arguments is < 2. I will need to wrap it to make it safe\n*/\ntemplate <typename F, size_t nargs>\nstruct get_args{\n typedef __GLRPC_NIF0 arg1_type;\n typedef __GLRPC_NIF1 arg2_type;\n};\n\n// if 0 args. then make both void\ntemplate <typename F>\nstruct get_args<F, 0>{\n typedef void arg1_type;\n typedef void arg2_type;\n};\n\n// if 1 arg then make just make arg2 void\ntemplate <typename F>\nstruct get_args<F, 1>{\n typedef __GLRPC_NIF0 arg1_type;\n typedef void arg2_type;\n};\n\n\n\ntemplate <typename F>\nstruct check_first_arg {\n typedef typename boost::is_same<typename get_args<F,__GLRPC_FARITY>::arg1_type, distributed_control>::type type; \n};\n\ntemplate <typename F>\nstruct check_second_arg {\n typedef typename boost::is_integral<typename get_args<F,__GLRPC_FARITY>::arg2_type>::type type; \n};\n\n\n}\n\n/**\n * \\ingroup rpc\n * \\internal\n * ::type is true if F is an RPC call interface.\n * \\tparam F the function to test\n */\ntemplate <typename F>\nstruct is_rpc_call {\n typedef typename boost::mpl::if_< typename is_rpc_call_detail::less_than_2_args<F>::type,\n boost::false_type,\n typename boost::mpl::and_<\n typename is_rpc_call_detail::check_first_arg<F>::type, \n typename is_rpc_call_detail::check_second_arg<F>::type>::type >::type type;\n \n \n};\n\n// Varargs are all none RPC calls\n#define BLOCK_VAR_ARGS(Z,N,_) \\\ntemplate <typename RetType BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nstruct is_rpc_call<RetType (BOOST_PP_ENUM_PARAMS(N, T) BOOST_PP_COMMA_IF(N) ...)> { \\\n typedef boost::false_type type; \\\n}; \\\n\\\ntemplate <typename RetType BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nstruct is_rpc_call<RetType (*)(BOOST_PP_ENUM_PARAMS(N, T) BOOST_PP_COMMA_IF(N) ...)> { \\\n typedef boost::false_type type; \\\n};\nBOOST_PP_REPEAT(6, BLOCK_VAR_ARGS, _)\n#undef BLOCK_VAR_ARGS\n\n#define GEN_GET_USER_ARG(Z,N,_) \\\ntemplate <typename F, typename BoolType> \\\nstruct BOOST_PP_CAT(get_cleaned_rpc_or_basic_arg, N) { \\\n typedef BOOST_PP_CAT(__GLRPC_NIF, N) arg_type; \\\n}; \\\ntemplate <typename F> \\\nstruct BOOST_PP_CAT(get_cleaned_rpc_or_basic_arg, N) <F, boost::mpl::bool_<true> > { \\\n typedef BOOST_PP_CAT(__GLRPC_F, N) arg_type; \\\n}; \\\ntemplate <typename F> \\\nstruct BOOST_PP_CAT(get_cleaned_user_arg, N) { \\\n typedef typename BOOST_PP_CAT(get_cleaned_rpc_or_basic_arg, N)<F,typename is_rpc_call<F>::type>::arg_type arg_type; \\\n};\n\nBOOST_PP_REPEAT(6, GEN_GET_USER_ARG, _)\n#undef GEN_GET_USER_ARG\n\n} // namespace dc_impl\n} // namespace graphlab\n\n#include <graphlab/rpc/function_arg_types_undef.hpp>\n\n#endif\n\n" }, { "path": "src/graphlab/rpc/lazy_dht.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n/*\n \\author Yucheng Low (ylow)\n An implementation of a distributed integer -> integer map with caching\n capabilities. \n\n*/\n\n#ifndef GRAPHLAB_LAZY_DHT_HPP\n#define GRAPHLAB_LAZY_DHT_HPP\n\n#include <boost/unordered_map.hpp>\n#include <boost/intrusive/list.hpp>\n\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/util/synchronized_unordered_map.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n\n\n\nnamespace graphlab {\n\n /**\n \\internal\n \\ingroup rpc \n\n\n This implements a distributed key -> value map with caching\n capabilities. It is up to the user to determine cache\n invalidation policies. User explicitly calls the invalidate()\n function to clear local cache entries. This is an extremely lazy\n DHT in that it is up to the user to guarantee that the keys are\n unique. Any machine can call set on any key, and the result of\n the key will be stored locally. Reads on any unknown keys will be\n resolved using a broadcast operation.\n */\n\n template<typename KeyType, typename ValueType>\n class lazy_dht{\n public:\n\n typedef dc_impl::lru_list<KeyType, ValueType> lru_entry_type;\n /// datatype of the data map\n typedef boost::unordered_map<KeyType, ValueType> map_type;\n /// datatype of the local cache map\n typedef boost::unordered_map<KeyType, lru_entry_type* > cache_type;\n\n struct wait_struct {\n mutex mut;\n conditional cond;\n ValueType val;\n size_t numreplies;\n bool hasvalue;\n };\n\n typedef boost::intrusive::member_hook<lru_entry_type,\n typename lru_entry_type::lru_member_hook_type,\n &lru_entry_type::member_hook_> MemberOption;\n /// datatype of the intrusive LRU list embedded in the cache map\n typedef boost::intrusive::list<lru_entry_type, \n MemberOption, \n boost::intrusive::constant_time_size<false> > lru_list_type;\n\n /// Constructor. Creates the integer map.\n lazy_dht(distributed_control &dc, \n size_t max_cache_size = 65536):rmi(dc, this),data(11) {\n cache.rehash(max_cache_size);\n maxcache = max_cache_size;\n logger(LOG_INFO, \"%d Creating distributed_hash_table. Cache Limit = %d\", \n dc.procid(), maxcache);\n reqs = 0;\n misses = 0;\n dc.barrier();\n }\n\n\n ~lazy_dht() {\n data.clear();\n typename cache_type::iterator i = cache.begin();\n while (i != cache.end()) {\n delete i->second;\n ++i;\n }\n cache.clear();\n }\n \n \n /// Sets the key to the value\n void set(const KeyType& key, const ValueType &newval) {\n datalock.lock();\n data[key] = newval;\n datalock.unlock();\n }\n \n\n std::pair<bool, ValueType> get_owned(const KeyType &key) const {\n std::pair<bool, ValueType> ret;\n datalock.lock();\n typename map_type::const_iterator iter = data.find(key); \n if (iter == data.end()) {\n ret.first = false;\n }\n else {\n ret.first = true;\n ret.second = iter->second;\n }\n datalock.unlock();\n return ret;\n }\n \n void remote_get_owned(const KeyType &key, procid_t source, size_t ptr) const {\n std::pair<bool, ValueType> ret;\n datalock.lock();\n typename map_type::const_iterator iter = data.find(key); \n if (iter == data.end()) {\n ret.first = false;\n }\n else {\n ret.first = true;\n ret.second = iter->second;\n }\n datalock.unlock();\n rmi.remote_call(source, &lazy_dht<KeyType,ValueType>::get_reply, ptr, ret.second, ret.first);\n }\n\n void get_reply(size_t ptr, ValueType& val, bool hasvalue) {\n wait_struct* w = reinterpret_cast<wait_struct*>(ptr); \n w->mut.lock();\n if (hasvalue) {\n w->val = val;\n w->hasvalue = true;\n }\n w->numreplies--;\n if (w->numreplies == 0) w->cond.signal();\n w->mut.unlock();\n \n }\n\n /** Gets the value associated with the key. returns true on success.. */\n std::pair<bool, ValueType> get(const KeyType &key) const {\n std::pair<bool, ValueType> ret = get_owned(key);\n if (ret.first) return ret;\n \n wait_struct w;\n w.numreplies = rmi.numprocs() - 1;\n size_t ptr = reinterpret_cast<size_t>(&w);\n // otherwise I need to find someone with the key\n for (size_t i = 0;i < rmi.numprocs(); ++i) {\n if (i != rmi.procid()) {\n rmi.remote_call(i, &lazy_dht<KeyType,ValueType>::remote_get_owned, key, rmi.procid(), ptr);\n }\n }\n w.mut.lock();\n while (w.numreplies > 0) w.cond.wait(w.mut);\n w.mut.unlock();\n ret.first = w.hasvalue;\n ret.second = w.val;\n if (ret.first) update_cache(key, ret.second);\n return ret;\n }\n\n\n /** Gets the value associated with the key, reading from cache if available\n Note that the cache may be out of date. */\n std::pair<bool, ValueType> get_cached(const KeyType &key) const {\n std::pair<bool, ValueType> ret = get_owned(key);\n if (ret.first) return ret;\n \n reqs++;\n cachelock.lock();\n // check if it is in the cache\n typename cache_type::iterator i = cache.find(key);\n if (i == cache.end()) {\n // nope. not in cache. Call the regular get\n cachelock.unlock();\n misses++;\n return get(key);\n }\n else {\n // yup. in cache. return the value\n ret.first = true;\n ret.second = i->second->value;\n // shift the cache entry to the head of the LRU list\n lruage.erase(lru_list_type::s_iterator_to(*(i->second)));\n lruage.push_front(*(i->second));\n cachelock.unlock();\n return ret;\n }\n }\n\n /// Invalidates the cache entry associated with this key\n void invalidate(const KeyType &key) const{\n cachelock.lock();\n // is the key I am invalidating in the cache?\n typename cache_type::iterator i = cache.find(key);\n if (i != cache.end()) {\n // drop it from the lru list\n delete i->second;\n cache.erase(i);\n }\n cachelock.unlock();\n }\n\n\n double cache_miss_rate() {\n return double(misses) / double(reqs);\n }\n\n size_t num_gets() const {\n return reqs;\n }\n size_t num_misses() const {\n return misses;\n }\n\n size_t cache_size() const {\n return cache.size();\n }\n\n private:\n\n mutable dc_dist_object<lazy_dht<KeyType, ValueType> > rmi;\n \n mutex datalock;\n map_type data; /// The actual table data that is distributed\n\n \n mutex cachelock; /// lock for the cache datastructures\n mutable cache_type cache; /// The cache table\n mutable lru_list_type lruage; /// THe LRU linked list associated with the cache\n\n\n procid_t numprocs; /// NUmber of processors\n size_t maxcache; /// Maximum cache size allowed\n\n mutable size_t reqs;\n mutable size_t misses;\n \n\n\n \n\n /// Updates the cache with this new value\n void update_cache(const KeyType &key, const ValueType &val) const{\n cachelock.lock();\n typename cache_type::iterator i = cache.find(key);\n // create a new entry\n if (i == cache.end()) {\n cachelock.unlock();\n // if we are out of room, remove the lru entry\n if (cache.size() >= maxcache) remove_lru();\n cachelock.lock();\n // insert the element, remember the iterator so we can push it\n // straight to the LRU list\n std::pair<typename cache_type::iterator, bool> ret = cache.insert(std::make_pair(key, new lru_entry_type(key, val)));\n if (ret.second) lruage.push_front(*(ret.first->second));\n }\n else {\n // modify entry in place\n i->second->value = val;\n // swap to front of list\n //boost::swap_nodes(lru_list_type::s_iterator_to(i->second), lruage.begin());\n lruage.erase(lru_list_type::s_iterator_to(*(i->second)));\n lruage.push_front(*(i->second));\n }\n cachelock.unlock();\n }\n\n /// Removes the least recently used element from the cache\n void remove_lru() const{\n cachelock.lock();\n KeyType keytoerase = lruage.back().key;\n // is the key I am invalidating in the cache?\n typename cache_type::iterator i = cache.find(keytoerase);\n if (i != cache.end()) {\n // drop it from the lru list\n delete i->second;\n cache.erase(i);\n }\n cachelock.unlock();\n }\n\n };\n\n}\n#endif\n\n" }, { "path": "src/graphlab/rpc/mem_function_arg_types_def.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n// check for multiple inclusions\n#ifdef __GLRPC_F0\n#error \"multiple includes of function arg types\"\n#endif\n\n#include <boost/type_traits/decay.hpp>\n#include <graphlab/util/generics/remove_member_pointer.hpp>\n#include <boost/type_traits/remove_const.hpp>\n#include <boost/type_traits/function_traits.hpp>\n#include <boost/function.hpp>\n// This is the member function version of function_arg_types\n/*\nA huge collection of useful typedefs.\nF0... F5: identifies the arguments for an RPC aware function F. Dropping const and dropping references \n (therefore allowing you to use F0....F5 to do casting.\n\n__GLRPC_NIF0... __GLRPC_NIF5: identifies the arguments for an RPC unaware function F\n\n__GLRPC_R0.... __GLRPC_R7: Identifies the actual arguments of the function F, without de-consting and de-reffing\n\n__GLRPC_FRESULT: de-const and de-refed type of F's return type\n\n__GLRPC_FARITY: the number of arguments F takes\n*/\n#define REMOVE_CONST_REF(REF) typename boost::remove_const<typename boost::remove_reference<REF>::type>::type\n\n\n\n//#define F0 REMOVE_CONST_REF(typename boost::function<typename boost::remove_member_pointer<F>::type>::arg1_type)\n//#define __GLRPC_FRESULT REMOVE_CONST_REF(typename boost::function<typename boost::remove_member_pointer<F>::type>::result_type)\n\n#define __GLRPC_NIF0 REMOVE_CONST_REF(typename boost::function<typename boost::remove_member_pointer<F>::type>::arg1_type)\n#define __GLRPC_NIF1 REMOVE_CONST_REF(typename boost::function<typename boost::remove_member_pointer<F>::type>::arg2_type)\n#define __GLRPC_NIF2 REMOVE_CONST_REF(typename boost::function<typename boost::remove_member_pointer<F>::type>::arg3_type)\n#define __GLRPC_NIF3 REMOVE_CONST_REF(typename boost::function<typename boost::remove_member_pointer<F>::type>::arg4_type)\n#define __GLRPC_NIF4 REMOVE_CONST_REF(typename boost::function<typename boost::remove_member_pointer<F>::type>::arg5_type)\n#define __GLRPC_NIF5 REMOVE_CONST_REF(typename boost::function<typename boost::remove_member_pointer<F>::type>::arg6_type)\n#define __GLRPC_NIF6 REMOVE_CONST_REF(typename boost::function<typename boost::remove_member_pointer<F>::type>::arg7_type)\n#define __GLRPC_NIF7 REMOVE_CONST_REF(typename boost::function<typename boost::remove_member_pointer<F>::type>::arg8_type)\n\n\n\n#define __GLRPC_R0 typename boost::function<typename boost::remove_member_pointer<F>::type>::arg1_type\n#define __GLRPC_R1 typename boost::function<typename boost::remove_member_pointer<F>::type>::arg2_type\n#define __GLRPC_R2 typename boost::function<typename boost::remove_member_pointer<F>::type>::arg3_type\n#define __GLRPC_R3 typename boost::function<typename boost::remove_member_pointer<F>::type>::arg4_type\n#define __GLRPC_R4 typename boost::function<typename boost::remove_member_pointer<F>::type>::arg5_type\n#define __GLRPC_R5 typename boost::function<typename boost::remove_member_pointer<F>::type>::arg6_type\n#define __GLRPC_R6 typename boost::function<typename boost::remove_member_pointer<F>::type>::arg7_type\n#define __GLRPC_R7 typename boost::function<typename boost::remove_member_pointer<F>::type>::arg8_type\n\n#define __GLRPC_FRESULT REMOVE_CONST_REF(typename boost::function<typename boost::remove_member_pointer<F>::type>::result_type)\n\n#define __GLRPC_FARITY boost::function<typename boost::remove_member_pointer<F>::type>::arity\n\n" }, { "path": "src/graphlab/rpc/mem_function_arg_types_undef.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#undef __GLRPC_NIF0\n#undef __GLRPC_NIF1\n#undef __GLRPC_NIF2\n#undef __GLRPC_NIF3\n#undef __GLRPC_NIF4\n#undef __GLRPC_NIF5\n#undef __GLRPC_NIF6\n#undef __GLRPC_NIF7\n#undef __GLRPC_R0\n#undef __GLRPC_R1\n#undef __GLRPC_R2\n#undef __GLRPC_R3\n#undef __GLRPC_R4\n#undef __GLRPC_R5\n#undef __GLRPC_R6\n#undef __GLRPC_R7\n\n#undef __GLRPC_FRESULT\n#undef REMOVE_CONST_REF\n#undef __GLRPC_FARITY\n\n" }, { "path": "src/graphlab/rpc/object_broadcast_issue.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef OBJECT_BROADCAST_ISSUE_HPP\n#define OBJECT_BROADCAST_ISSUE_HPP\n#include <iostream>\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/dc_send.hpp>\n#include <graphlab/rpc/object_call_dispatch.hpp>\n#include <graphlab/rpc/object_call_issue.hpp>\n#include <graphlab/rpc/is_rpc_call.hpp>\n#include <graphlab/rpc/dc_thread_get_send_buffer.hpp>\n#include <boost/preprocessor.hpp>\n#include <graphlab/rpc/mem_function_arg_types_def.hpp>\n\nnamespace graphlab{\nnamespace dc_impl {\n\n/**\n\\ingroup rpc\n\\internal\n\\file object_broadcast_issue.hpp\n This is an internal function and should not be used directly\n\nSee object_call_issue.hpp for details. This is equivalent to the macro\nexpansion in object_call_issue with the difference that this takes an iterator \nsequence listing the machines to send to.\n\nThe code below generates the following for different number of arguments. Here, \nwe demonstrate the 1 argument version.\n\n\\code\ntemplate < typename Iterator, typename T, typename F, typename T0 > \nclass object_broadcast_issue1 {\n public:\n static void exec (dc_dist_object_base * rmi,\n std::vector < dc_send * >sender, unsigned char flags,\n Iterator target_begin, Iterator target_end, size_t objid,\n F remote_function, const T0 & i0) {\n oarchive arc;\n arc.buf = (char *) malloc (65536);\n arc.len = 65536;\n size_t len =\n dc_send::write_packet_header (arc, _get_procid (), flags,\n\t\t\t\t _get_sequentialization_key ());\n uint32_t beginoff = arc.off;\n dispatch_type d =\n dc_impl::OBJECT_NONINTRUSIVE_DISPATCH1 < distributed_control, T, F,\n T0 >;\n arc << reinterpret_cast < size_t > (d);\n serialize (arc, (char *) (&remote_function), sizeof (F));\n arc << objid;\n arc << i0;\n uint32_t curlen = arc.off - beginoff;\n *(reinterpret_cast < uint32_t * >(arc.buf + len)) = curlen;\n Iterator iter = target_begin;\n while (iter != target_end) {\n oarchive *buf = get_thread_local_buffer (*iter);\n buf->write (arc.buf, arc.off);\n release_thread_local_buffer (*iter, flags & CONTROL_PACKET);\n if ((flags & CONTROL_PACKET) == 0) {\n rmi->inc_bytes_sent ((*iter), curlen);\n }\n ++iter;\n }\n free (arc.buf);\n }\n};\n\n\\endcode\n*/\n\n#define GENARGS(Z,N,_) BOOST_PP_CAT(const T, N) BOOST_PP_CAT(&i, N)\n#define GENI(Z,N,_) BOOST_PP_CAT(i, N)\n#define GENT(Z,N,_) BOOST_PP_CAT(T, N)\n#define GENARC(Z,N,_) arc << BOOST_PP_CAT(i, N);\n\n\n#define REMOTE_BROADCAST_ISSUE_GENERATOR(Z,N,FNAME_AND_CALL) \\\ntemplate<typename Iterator, typename T, typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nclass BOOST_PP_CAT(BOOST_PP_TUPLE_ELEM(2,0,FNAME_AND_CALL), N) { \\\n public: \\\n static void exec(dc_dist_object_base* rmi, std::vector<dc_send*> sender, unsigned char flags, \\\n Iterator target_begin, Iterator target_end, size_t objid, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n oarchive arc; \\\n arc.buf = (char*)malloc(INITIAL_BUFFER_SIZE); \\\n arc.len = INITIAL_BUFFER_SIZE; \\\n size_t len = dc_send::write_packet_header(arc, _get_procid(), flags, _get_sequentialization_key()); \\\n uint32_t beginoff = arc.off; \\\n dispatch_type d = BOOST_PP_CAT(dc_impl::OBJECT_NONINTRUSIVE_DISPATCH,N)<distributed_control,T,F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N, GENT ,_) >; \\\n arc << reinterpret_cast<size_t>(d); \\\n serialize(arc, (char*)(&remote_function), sizeof(F)); \\\n arc << objid; \\\n BOOST_PP_REPEAT(N, GENARC, _) \\\n uint32_t curlen = arc.off - beginoff; \\\n *(reinterpret_cast<uint32_t*>(arc.buf + len)) = curlen; \\\n Iterator iter = target_begin; \\\n while(iter != target_end) { \\\n oarchive* buf = get_thread_local_buffer(*iter); \\\n buf->write(arc.buf, arc.off); \\\n release_thread_local_buffer(*iter, flags & CONTROL_PACKET); \\\n if ((flags & CONTROL_PACKET) == 0) { \\\n rmi->inc_bytes_sent((*iter), curlen); \\\n } \\\n ++iter; \\\n } \\\n free(arc.buf); \\\n if (flags & FLUSH_PACKET) pull_flush_soon_thread_local_buffer(); \\\n } \\\n};\n\n\n\n/**\nGenerates a function call issue. 3rd argument is a tuple (issue name, dispacther name)\n*/\nBOOST_PP_REPEAT(7, REMOTE_BROADCAST_ISSUE_GENERATOR, (object_broadcast_issue, _) )\n\n\n\n#undef GENARC\n#undef GENT\n#undef GENI\n#undef GENARGS\n#undef REMOTE_BROADCAST_ISSUE_GENERATOR\n\n} // namespace dc_impl\n} // namespace graphlab\n\n#include <graphlab/rpc/mem_function_arg_types_undef.hpp>\n\n#endif\n\n" }, { "path": "src/graphlab/rpc/object_call_dispatch.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_OBJECT_CALL_DISPATCH_HPP\n#define GRAPHLAB_OBJECT_CALL_DISPATCH_HPP\n#include <iostream>\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/mem_function_arg_types_def.hpp>\n#include <boost/preprocessor.hpp>\nnamespace graphlab {\nnamespace dc_impl {\n\n\n\n/**\n\\ingroup rpc\n\\internal\n\\file object_call_dispatch.hpp\nThis is an internal function and should not be used directly\n\nThis is similar to a regular function call in function_call_dispatch.hpp\nwith the only difference\nthat it needs to locate the object using dc.get_registered_object(...)\nAfter the function call, it also needs to increment the call count for\nthe object context.\n\\code\ntemplate<typename DcType, typename T, typename F , typename T0 > \nvoid OBJECT_NONINTRUSIVE_DISPATCH1(DcType& dc, procid_t source, \n unsigned char packet_type_mask, \n const char* buf, size_t len){ \n iarchive iarc(buf, len);\n F f;\n deserialize(iarc, (char*)(&f), sizeof(F));\n size_t objid;\n iarc >> objid;\n T* obj = reinterpret_cast<T*>(dc.get_registered_object(objid));\n T0 (f0) ;\n iarc >> (f0) ;\n (obj->*f)( (f0) );\n charstring_free(f0);\n if ((packet_type_mask & CONTROL_PACKET) == 0) dc.get_rmi_instance(objid)->inc_calls_received(source);\n}\n\\endcode\n*/\n\n\n\n#define GENFN(N) BOOST_PP_CAT(__GLRPC_NIF, N)\n#define GENFN2(N) BOOST_PP_CAT(f, N)\n#define GENARGS(Z,N,_) (BOOST_PP_CAT(f, N))\n\n/**\n * This macro defines and deserializes each of the parameters to the\n * function.\n */\n#define GENPARAMS(Z,N,_) \\\n BOOST_PP_CAT(T, N) (BOOST_PP_CAT(f, N)) ; \\\n iarc >> (BOOST_PP_CAT(f, N)) ;\n\n#define CHARSTRINGFREE(Z,N,_) charstring_free(BOOST_PP_CAT(f, N));\n\n\n#define OBJECT_NONINTRUSIVE_DISPATCH_GENERATOR(Z,N,_) \\\n template<typename DcType, typename T, \\\n typename F BOOST_PP_COMMA_IF(N) \\\n BOOST_PP_ENUM_PARAMS(N, typename T) > \\\n void BOOST_PP_CAT(OBJECT_NONINTRUSIVE_DISPATCH,N)(DcType& dc, \\\n procid_t source, \\\n unsigned char packet_type_mask, \\\n const char* buf, size_t len){ \\\n iarchive iarc(buf, len); \\\n F f; \\\n deserialize(iarc, (char*)(&f), sizeof(F)); \\\n size_t objid; \\\n iarc >> objid; \\\n T* obj = reinterpret_cast<T*>(dc.get_registered_object(objid)); \\\n /* Deserialize the arguments to f */ \\\n BOOST_PP_REPEAT(N, GENPARAMS, _); \\\n /* Invoke f */ \\\n (obj->*f)(BOOST_PP_ENUM(N,GENARGS ,_) ); \\\n /* Free the buffers for the args */ \\\n BOOST_PP_REPEAT(N, CHARSTRINGFREE, _) ; \\\n /* Count the call if not a control call */ \\\n if ((packet_type_mask & CONTROL_PACKET) == 0) \\\n dc.get_rmi_instance(objid)->inc_calls_received(source); \\\n } \n\n\n\n/**\n * This macro generates dispatch functions for functions for rpc calls\n * with up to 6 arguments.\n *\n * Remarks: If the compiler generates the following error \"Too\n * few/many arguments to function\" at this point is is due to the\n * caller not providing the correct number fo arguments in the RPC\n * call. Note that default arguments are NOT supported in rpc calls\n * and so all arguments must be provided.\n *\n */\nBOOST_PP_REPEAT(7, OBJECT_NONINTRUSIVE_DISPATCH_GENERATOR, _)\n\n\n\n#undef GENFN\n#undef GENFN2\n#undef GENARGS\n#undef GENPARAMS\n#undef NONINTRUSIVE_DISPATCH_GENERATOR\n\n} // namespace dc_impl\n} // namespace graphlab\n\n\n#include <graphlab/rpc/mem_function_arg_types_undef.hpp>\n#endif\n\n" }, { "path": "src/graphlab/rpc/object_call_issue.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef OBJECT_CALL_ISSUE_HPP\n#define OBJECT_CALL_ISSUE_HPP\n#include <iostream>\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/dc_send.hpp>\n#include <graphlab/rpc/object_call_dispatch.hpp>\n#include <graphlab/rpc/is_rpc_call.hpp>\n#include <graphlab/rpc/dc_thread_get_send_buffer.hpp>\n#include <boost/preprocessor.hpp>\n#include <graphlab/rpc/dc_compile_parameters.hpp>\n#include <graphlab/util/generics/blob.hpp>\n#include <graphlab/rpc/mem_function_arg_types_def.hpp>\n\nnamespace graphlab{\nnamespace dc_impl {\n\n/**\n\\ingroup rpc\n\\internal\n\\file object_call_issue.hpp\n This is an internal function and should not be used directly\n\nMarshalls a object function call to a remote machine.\nThis is similar to the regular function call in function_call_issue.hpp\nwith the only difference that the object id needs to be transmitted as well.\n\n\\code\ntemplate < typename T, typename F, typename T0 > \nclass object_call_issue1 {\n public:\n static void exec (dc_dist_object_base * rmi, dc_send * sender,\n unsigned char flags, procid_t target, size_t objid,\n F remote_function, const T0 & i0) {\n oarchive *ptr = get_thread_local_buffer (target);\n oarchive & arc = *ptr;\n size_t len =\n dc_send::write_packet_header (arc, _get_procid (), flags,\n\t\t\t\t _get_sequentialization_key ());\n uint32_t beginoff = arc.off;\n dispatch_type d =\n dc_impl::OBJECT_NONINTRUSIVE_DISPATCH1 < distributed_control, T, F,\n T0 >;\n arc << reinterpret_cast < size_t > (d);\n serialize (arc, (char *) (&remote_function), sizeof (F));\n arc << objid;\n arc << i0;\n uint32_t curlen = arc.off - beginoff;\n *(reinterpret_cast < uint32_t * >(arc.buf + len)) = curlen;\n release_thread_local_buffer (target, flags & CONTROL_PACKET);\n if ((flags & CONTROL_PACKET) == 0) {\n rmi->inc_bytes_sent (target, curlen);\n }\n }\n};\n\\endcode\n*/\n\n#define GENARGS(Z,N,_) BOOST_PP_CAT(const T, N) BOOST_PP_CAT(&i, N)\n#define GENI(Z,N,_) BOOST_PP_CAT(i, N)\n#define GENT(Z,N,_) BOOST_PP_CAT(T, N)\n#define GENARC(Z,N,_) arc << BOOST_PP_CAT(i, N);\n\n\n/**\nThe dispatch_selectorN structs are used to pick between the standard dispatcher and the nonintrusive dispatch\nby checking if the function is a RPC style call or not.\n*/\n#define REMOTE_CALL_ISSUE_GENERATOR(Z,N,FNAME_AND_CALL) \\\ntemplate<typename T, typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nclass BOOST_PP_CAT(BOOST_PP_TUPLE_ELEM(2,0,FNAME_AND_CALL), N) { \\\n public: \\\n static void exec(dc_dist_object_base* rmi, dc_send* sender, unsigned char flags, procid_t target, size_t objid, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n oarchive* ptr = get_thread_local_buffer(target); \\\n oarchive& arc = *ptr; \\\n size_t len = dc_send::write_packet_header(arc, _get_procid(), flags, _get_sequentialization_key()); \\\n uint32_t beginoff = arc.off; \\\n dispatch_type d = BOOST_PP_CAT(dc_impl::OBJECT_NONINTRUSIVE_DISPATCH,N)<distributed_control,T,F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N, GENT ,_) >; \\\n arc << reinterpret_cast<size_t>(d); \\\n serialize(arc, (char*)(&remote_function), sizeof(F)); \\\n arc << objid; \\\n BOOST_PP_REPEAT(N, GENARC, _) \\\n uint32_t curlen = arc.off - beginoff; \\\n *(reinterpret_cast<uint32_t*>(arc.buf + len)) = curlen; \\\n release_thread_local_buffer(target, flags & CONTROL_PACKET); \\\n if ((flags & CONTROL_PACKET) == 0) { \\\n rmi->inc_bytes_sent(target, curlen); \\\n } \\\n if (flags & FLUSH_PACKET) pull_flush_soon_thread_local_buffer(target); \\\n } \\\n \\\n};\n\n\n/**\n * \\ingroup rpc\n * \\internal\n *\n * This generates a \"split call\". Where the header of the call message\n * is written to with split_call_begin, and the message actually sent with\n * split_call_end(). It is then up to the user to serialize the message arguments\n * into the oarchive returned. The split call can provide performance gains \n * when the contents of the message are large, since this allows the user to\n * control the serialization process. For examples, see \n * \\ref dc_dist_object::split_call_begin\n */\ntemplate <typename T, typename F>\nclass object_split_call {\n public:\n static oarchive* split_call_begin(dc_dist_object_base* rmi, size_t objid, F remote_function) {\n oarchive* ptr = new oarchive;\n oarchive& arc = *ptr;\n arc.buf = (char*)malloc(INITIAL_BUFFER_SIZE); \n arc.len = INITIAL_BUFFER_SIZE; \n arc.advance(sizeof(packet_hdr));\n dispatch_type d = dc_impl::OBJECT_NONINTRUSIVE_DISPATCH2<distributed_control,T,F,size_t, wild_pointer>;\n arc << reinterpret_cast<size_t>(d);\n serialize(arc, (char*)(&remote_function), sizeof(F));\n arc << objid;\n // make a gap for the blob size argument\n // write the largest possible size_t. That will allow it to bypass\n // dynamic length encoding issues.\n // patch the header with the offset to this point. \n (*reinterpret_cast<size_t*>(arc.buf)) = arc.off + 1;\n arc << (size_t)(-1);\n return ptr;\n }\n static void split_call_cancel(oarchive* oarc) {\n free(oarc->buf);\n delete oarc;\n }\n\n/**\n * \\ingroup rpc\n * \\internal\n *\n * This sends a message first created with split_call_begin. The archive\n * pointer is consumed.\n */\n static void split_call_end(dc_dist_object_base* rmi,\n oarchive* oarc, dc_send* sender, procid_t target, unsigned char flags) {\n // header points to the location of the blob size argument\n size_t blobsize_offset = *reinterpret_cast<size_t*>(oarc->buf);\n (*reinterpret_cast<size_t*>(oarc->buf + blobsize_offset)) = oarc->off - blobsize_offset - sizeof(size_t);\n // write the packet header\n packet_hdr* hdr = reinterpret_cast<packet_hdr*>(oarc->buf);\n hdr->len = oarc->off - sizeof(packet_hdr);\n hdr->src = _get_procid();\n hdr->packet_type_mask = flags;\n hdr->sequentialization_key = _get_sequentialization_key();\n size_t len = hdr->len;\n write_thread_local_buffer(target, oarc->buf, oarc->off, flags & CONTROL_PACKET);\n if ((flags & CONTROL_PACKET) == 0) {\n rmi->inc_bytes_sent(target, len);\n }\n if (flags & FLUSH_PACKET) pull_flush_soon_thread_local_buffer(target); \n delete oarc;\n }\n};\n\n/**\nGenerates a function call issue. 3rd argument is a tuple (issue name, dispacther name)\n*/\n\nBOOST_PP_REPEAT(7, REMOTE_CALL_ISSUE_GENERATOR, (object_call_issue, _) )\n\n\n#undef GENARC\n#undef GENT\n#undef GENI\n#undef GENARGS\n#undef REMOTE_CALL_ISSUE_GENERATOR\n\n} // namespace dc_impl\n} // namespace graphlab\n\n#include <graphlab/rpc/mem_function_arg_types_undef.hpp>\n\n#endif\n\n" }, { "path": "src/graphlab/rpc/object_request_dispatch.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef OBJECT_REQUEST_DISPATCH_HPP\n#define OBJECT_REQUEST_DISPATCH_HPP\n#include <sstream>\n#include <iostream>\n#include <string>\n#include <functional>\n#include <algorithm>\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/request_reply_handler.hpp>\n#include <graphlab/rpc/function_ret_type.hpp>\n#include <boost/bind.hpp>\n#include <boost/mem_fn.hpp>\n#include <graphlab/rpc/mem_function_arg_types_def.hpp>\n#include <boost/preprocessor.hpp>\n\nnamespace graphlab {\nnamespace dc_impl{\n\n\n/**\n\\ingroup rpc\n\\internal\n\\file object_request_dispatch.hpp\n\nThis is an internal function and should not be used directly\n\nThis is the dispatch function for the an object request.\nThis is similar to the standard request dispatcher in request_dispatch.hpp\nexcept that the object needs to be located using the object id.\nAfter the function call, it also needs to increment the call count for\nthe object context.\n\n\\code\ntemplate < typename DcType, typename T, typename F, typename T0 > \nvoid OBJECT_NONINTRUSIVE_REQUESTDISPATCH1 (DcType & dc, procid_t source,\n unsigned char packet_type_mask,\n const char *buf, size_t len) {\n iarchive iarc (buf, len);\n F f;\n deserialize (iarc, (char *) (&f), sizeof (F));\n size_t objid;\n iarc >> objid;\n T *obj = reinterpret_cast < T * >(dc.get_registered_object (objid));\n size_t id;\n iarc >> id;\n T0 (f0);\n iarc >> (f0);\n typename function_ret_type < \n typename boost::remove_const < \n typename boost::remove_reference < \n typename boost::function < \n typename boost::remove_member_pointer < F >::type >::result_type >::type >::type >::type \n ret = mem_function_ret_type < \n typename boost::remove_const <\n typename boost::remove_reference < \n typename boost::function < \n typename boost::remove_member_pointer < F >::type >::result_type >::type >::type >::fcall1 (f, obj, (f0));\n\n charstring_free (f0);\n boost::iostreams::stream < resizing_array_sink > retstrm (128);\n oarchive oarc (retstrm);\n oarc << ret;\n retstrm.flush ();\n if ((packet_type_mask & CONTROL_PACKET) == 0) {\n dc.get_rmi_instance (objid)->inc_calls_received (source);\n dc.get_rmi_instance (objid)->inc_bytes_sent (source, retstrm->len);\n }\n if (packet_type_mask & CONTROL_PACKET) {\n dc.control_call (source, request_reply_handler, id,\n\t\t blob (retstrm->str, retstrm->len));\n }\n else {\n dc.reply_remote_call (source, request_reply_handler, id,\n\t\t\t blob (retstrm->str, retstrm->len));\n }\n free (retstrm->str);\n}\n\\endcode\n\n\n*/\n#define GENFN(N) BOOST_PP_CAT(__GLRPC_NIF, N)\n#define GENFN2(N) BOOST_PP_CAT(f, N)\n#define GENNIARGS(Z,N,_) (BOOST_PP_CAT(f, N))\n\n#define GENPARAMS(Z,N,_) \\\n BOOST_PP_CAT(T, N) (BOOST_PP_CAT(f, N)) ; \\\n iarc >> (BOOST_PP_CAT(f, N)) ;\n\n#define CHARSTRINGFREE(Z,N,_) charstring_free(BOOST_PP_CAT(f, N));\n\n#define NONINTRUSIVE_DISPATCH_GENERATOR(Z,N,_) \\\n template<typename DcType, typename T, \\\n typename F BOOST_PP_COMMA_IF(N) \\\n BOOST_PP_ENUM_PARAMS(N, typename T) > \\\n void BOOST_PP_CAT(OBJECT_NONINTRUSIVE_REQUESTDISPATCH,N) (DcType& dc, \\\n procid_t source, \\\n unsigned char packet_type_mask, \\\n const char* buf, size_t len) { \\\n iarchive iarc(buf, len); \\\n F f; \\\n deserialize(iarc, (char*)(&f), sizeof(F)); \\\n size_t objid; \\\n iarc >> objid; \\\n T* obj = reinterpret_cast<T*>(dc.get_registered_object(objid)); \\\n size_t id; iarc >> id; \\\n BOOST_PP_REPEAT(N, GENPARAMS, _); \\\n typename function_ret_type<__GLRPC_FRESULT>::type ret = \\\n mem_function_ret_type<__GLRPC_FRESULT>::BOOST_PP_CAT(fcall, N) \\\n (f, obj BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENNIARGS ,_)); \\\n BOOST_PP_REPEAT(N, CHARSTRINGFREE, _); \\\n boost::iostreams::stream<resizing_array_sink> retstrm(128); \\\n oarchive oarc(retstrm); \\\n oarc << ret; \\\n retstrm.flush(); \\\n if ((packet_type_mask & CONTROL_PACKET) == 0) { \\\n dc.get_rmi_instance(objid)->inc_calls_received(source); \\\n dc.get_rmi_instance(objid)->inc_bytes_sent(source, retstrm->len); \\\n } \\\n /*std::cerr << \"Request wait on \" << id << std::endl ; */ \\\n if (packet_type_mask & CONTROL_PACKET) { \\\n dc.control_call(source, \\\n request_reply_handler, \\\n id, \\\n blob(retstrm->str, retstrm->len)); \\\n } else if(packet_type_mask & FLUSH_PACKET) { \\\n dc.reply_remote_call(source, \\\n request_reply_handler, \\\n id, \\\n blob(retstrm->str, retstrm->len)); \\\n } else { \\\n dc.remote_call(source, \\\n request_reply_handler, \\\n id, \\\n blob(retstrm->str, retstrm->len)); \\\n } \\\n free(retstrm->str); \\\n /* std::cerr << \"Request received on \" << id << std::endl ; */ \\\n } \n\n\n\nBOOST_PP_REPEAT(6, NONINTRUSIVE_DISPATCH_GENERATOR, _)\n\n\n#undef GENFN\n#undef GENFN2\n#undef GENNIARGS\n#undef GENPARAMS\n#undef NONINTRUSIVE_DISPATCH_GENERATOR\n\n} // namespace dc_impl\n} // namespace graphlab\n#include <graphlab/rpc/mem_function_arg_types_undef.hpp>\n#endif\n\n" }, { "path": "src/graphlab/rpc/object_request_issue.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef OBJECT_REQUEST_ISSUE_HPP\n#define OBJECT_REQUEST_ISSUE_HPP\n#include <sstream>\n#include <iostream>\n#include <string>\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/request_reply_handler.hpp>\n#include <graphlab/rpc/object_request_dispatch.hpp>\n#include <graphlab/rpc/function_ret_type.hpp>\n#include <graphlab/rpc/mem_function_arg_types_def.hpp>\n#include <graphlab/rpc/request_future.hpp>\n#include <graphlab/rpc/dc_thread_get_send_buffer.hpp>\n#include <graphlab/rpc/dc_compile_parameters.hpp>\n#include <boost/preprocessor.hpp>\n\nnamespace graphlab {\nnamespace dc_impl {\n\n\n#define GENARGS(Z,N,_) BOOST_PP_CAT(const T, N) BOOST_PP_CAT(&i, N)\n#define GENT(Z,N,_) BOOST_PP_CAT(T, N)\n#define GENARC(Z,N,_) arc << BOOST_PP_CAT(i, N);\n\n\n/**\n\\internal\n\\ingroup rpc\n\\file object_request_issue.hpp\n\n\nThis is an internal function and should not be used directly\n\nThis is the marshall function for the an object member function call.\nThis is very similar to the standard function request issue in request_issue.hpp\n, with the only difference that an object id has to be transmitted\n\n\\code\ntemplate < typename T, typename F, typename T0 > \nclass object_request_issue1 {\n public:\n static void exec (dc_dist_object_base * rmi, dc_send * sender,\n size_t request_handle, unsigned char flags,\n procid_t target, size_t objid, F remote_function,\n const T0 & i0) {\n oarchive *ptr = get_thread_local_buffer (target);\n oarchive & arc = *ptr;\n size_t len =\n dc_send::write_packet_header (arc, _get_procid (), flags,\n\t\t\t\t _get_sequentialization_key ());\n uint32_t beginoff = arc.off;\n dispatch_type d =\n dc_impl::OBJECT_NONINTRUSIVE_REQUESTDISPATCH1 < distributed_control, T,\n F, T0 >;\n arc << reinterpret_cast < size_t > (d);\n serialize (arc, (char *) (&remote_function), sizeof (remote_function));\n arc << objid;\n arc << request_handle;\n arc << i0;\n uint32_t curlen = arc.off - beginoff;\n *(reinterpret_cast < uint32_t * >(arc.buf + len)) = curlen;\n release_thread_local_buffer (target, flags & CONTROL_PACKET);\n if ((flags & CONTROL_PACKET) == 0)\n rmi->inc_bytes_sent (target, curlen);\n pull_flush_thread_local_buffer (target);\n }\n};\n\\endcode\n\n\n*/\n#define REMOTE_REQUEST_ISSUE_GENERATOR(Z,N,FNAME_AND_CALL) \\\ntemplate<typename T,typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nclass BOOST_PP_CAT(FNAME_AND_CALL, N) { \\\n public: \\\n static void exec(dc_dist_object_base* rmi, dc_send* sender, size_t request_handle, unsigned char flags, procid_t target,size_t objid, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n oarchive* ptr = get_thread_local_buffer(target); \\\n oarchive& arc = *ptr; \\\n size_t len = dc_send::write_packet_header(arc, _get_procid(), flags, _get_sequentialization_key()); \\\n uint32_t beginoff = arc.off; \\\n dispatch_type d = BOOST_PP_CAT(dc_impl::OBJECT_NONINTRUSIVE_REQUESTDISPATCH,N)<distributed_control,T,F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N, GENT ,_) >; \\\n arc << reinterpret_cast<size_t>(d); \\\n serialize(arc, (char*)(&remote_function), sizeof(remote_function)); \\\n arc << objid; \\\n arc << request_handle; \\\n BOOST_PP_REPEAT(N, GENARC, _) \\\n uint32_t curlen = arc.off - beginoff; \\\n *(reinterpret_cast<uint32_t*>(arc.buf + len)) = curlen; \\\n release_thread_local_buffer(target, flags & CONTROL_PACKET); \\\n if ((flags & CONTROL_PACKET) == 0) \\\n rmi->inc_bytes_sent(target, curlen); \\\n if (flags & FLUSH_PACKET) pull_flush_soon_thread_local_buffer(target); \\\n }\\\n};\n\nBOOST_PP_REPEAT(6, REMOTE_REQUEST_ISSUE_GENERATOR, object_request_issue )\n\n\n\n#undef GENARC\n#undef GENT\n#undef GENARGS\n#undef REMOTE_REQUEST_ISSUE_GENERATOR\n\n\n} // namespace dc_impl\n} // namespace graphlab\n#include <graphlab/rpc/mem_function_arg_types_undef.hpp>\n\n#endif\n\n" }, { "path": "src/graphlab/rpc/pod_template_structs.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef POD_TEMPLATE_STRUCTS_HPP\n#define POD_TEMPLATE_STRUCTS_HPP\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/serialization/is_pod.hpp>\n\nnamespace graphlab {\nnamespace dc_impl {\nnamespace pod_template_detail {\n\ntemplate <typename F>\nstruct pod_call_struct0 : public IS_POD_TYPE{\n size_t dispatch_function;\n size_t objid;\n F remote_function;\n};\n\n\ntemplate <typename F, typename T0>\nstruct pod_call_struct1 : public IS_POD_TYPE{\n size_t dispatch_function;\n size_t objid;\n F remote_function;\n T0 t0;\n};\n\n\ntemplate <typename F, typename T0, typename T1>\nstruct pod_call_struct2 : public IS_POD_TYPE{\n size_t dispatch_function;\n size_t objid;\n F remote_function;\n T0 t0; T1 t1;\n};\n\ntemplate <typename F, typename T0, typename T1, typename T2>\nstruct pod_call_struct3 : public IS_POD_TYPE{\n size_t dispatch_function;\n size_t objid;\n F remote_function;\n T0 t0; T1 t1; T2 t2;\n};\n\n\ntemplate <typename F, typename T0, typename T1, typename T2,\n typename T3>\nstruct pod_call_struct4 : public IS_POD_TYPE{\n size_t dispatch_function;\n size_t objid;\n F remote_function;\n T0 t0; T1 t1; T2 t2; T3 t3;\n};\n\n\ntemplate <typename F, typename T0, typename T1, typename T2,\n typename T3, typename T4>\nstruct pod_call_struct5 : public IS_POD_TYPE{\n size_t dispatch_function;\n size_t objid;\n F remote_function;\n T0 t0; T1 t1; T2 t2; T3 t3; T4 t4;\n};\n\n\ntemplate <typename F, typename T0, typename T1, typename T2,\n typename T3, typename T4, typename T5>\nstruct pod_call_struct6 : public IS_POD_TYPE{\n size_t dispatch_function;\n size_t objid;\n F remote_function;\n T0 t0; T1 t1; T2 t2; T3 t3; T4 t4; T5 t5;\n};\n\n\ntemplate <typename F, typename T0, typename T1, typename T2,\n typename T3, typename T4, typename T5,\n typename T6>\nstruct pod_call_struct7 : public IS_POD_TYPE{\n size_t dispatch_function;\n size_t objid;\n F remote_function;\n T0 t0; T1 t1; T2 t2; T3 t3; T4 t4; T5 t5; T6 t6;\n};\n\n\ntemplate <typename F, typename T0, typename T1, typename T2,\n typename T3, typename T4, typename T5,\n typename T6, typename T7>\nstruct pod_call_struct8 : public IS_POD_TYPE{\n size_t dispatch_function;\n size_t objid;\n F remote_function;\n T0 t0; T1 t1; T2 t2; T3 t3; T4 t4; T5 t5; T6 t6; T7 t7;\n};\n\n}\n}\n}\n\n#endif" }, { "path": "src/graphlab/rpc/request_dispatch.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef REQUEST_DISPATCH_HPP\n#define REQUEST_DISPATCH_HPP\n#include <sstream>\n#include <iostream>\n#include <string>\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/request_reply_handler.hpp>\n#include <graphlab/rpc/function_ret_type.hpp>\n#include <graphlab/rpc/function_arg_types_def.hpp>\n#include <boost/preprocessor.hpp>\n\nnamespace graphlab {\nnamespace dc_impl{\n\n/**\n\\internal\n\\ingroup rpc\n\\file request_dispatch.hpp\n\nThis is an internal function and should not be used directly.\n\nGiven function F, as well as input types T1 ... Tn\nit will construct an input archive and deserialize the types T1.... Tn,\nand call the function f with it. The return value of the function\nis then returned to the caller through the reply call to the \nsource's request_reply_handler. This code dispatches to the \"intrusive\" \nform of a function call (that is the function call must take a distributed_control\nand a \"procid_t source\" as its first 2 arguments.\n\nFor instance, the 1 argument of this will be DISPATCH1:\n\\code\ntemplate < typename DcType, typename F, typename T0 > \nvoid REQUESTDISPATCH1 (DcType & dc, procid_t source,\n unsigned char packet_type_mask, \n const char *buf, size_t len) {\n iarchive iarc (buf, len);\n size_t s;\n iarc >> s;\n F f = reinterpret_cast < F > (s);\n size_t id;\n iarc >> id;\n T0 (f0);\n iarc >> (f0);\n typename function_ret_type < \n typename boost::remove_const <\n typename boost::remove_reference < \n typename boost::function <\n typename boost::remove_pointer <\n F >::type >::result_type >::type >::type >::type ret =\n function_ret_type < \n typename boost::remove_const <\n typename boost::remove_reference < \n typename boost::function <\n typename boost::remove_pointer <F>::type >::result_type >::type >::type >::fcall3 (f, dc, source, (f0));\n charstring_free (f0);\n boost::iostreams::stream < resizing_array_sink > retstrm (128);\n oarchive oarc (retstrm);\n oarc << ret;\n retstrm.flush ();\n if (packet_type_mask & CONTROL_PACKET) {\n dc.control_call (source, request_reply_handler, id,\n\t\t blob (retstrm->str, retstrm->len));\n }\n else {\n dc.reply_remote_call (source, request_reply_handler, id,\n\t\t\t blob (retstrm->str, retstrm->len));\n }\n free (retstrm->str);\n}\n\\endcode\n\ncharstring_free is a special template function which calls free(f1)\nonly if f1 is a character array (char*)\n\nNote that the template around DcType is *deliberate*. This prevents this\nfunction from instantiating the distributed_control until as late as possible, \navoiding problems with circular references.\n\n*/\n#define GENFN(N) BOOST_PP_CAT(__GLRPC_F, N)\n#define GENFN2(N) BOOST_PP_CAT(f, N)\n#define GENARGS(Z,N,_) (BOOST_PP_CAT(f, N))\n#define GENPARAMS(Z,N,_) BOOST_PP_CAT(T, N) (BOOST_PP_CAT(f, N)) ; iarc >> (BOOST_PP_CAT(f, N)) ;\n#define CHARSTRINGFREE(Z,N,_) charstring_free(BOOST_PP_CAT(f, N));\n\n\n#define DISPATCH_GENERATOR(Z,N,_) \\\ntemplate<typename DcType, typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nvoid BOOST_PP_CAT(REQUESTDISPATCH,N) (DcType& dc, procid_t source, unsigned char packet_type_mask, \\\n const char* buf, size_t len) { \\\n iarchive iarc(buf, len); \\\n size_t s; iarc >> s; F f = reinterpret_cast<F>(s); \\\n size_t id; iarc >> id; \\\n BOOST_PP_REPEAT(N, GENPARAMS, _) \\\n typename function_ret_type<__GLRPC_FRESULT>::type ret = function_ret_type<__GLRPC_FRESULT>::BOOST_PP_CAT(fcall, BOOST_PP_ADD(N, 2)) \\\n (f, dc, source BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_)); \\\n BOOST_PP_REPEAT(N, CHARSTRINGFREE, _) \\\n boost::iostreams::stream<resizing_array_sink> retstrm(128); \\\n oarchive oarc(retstrm); \\\n oarc << ret; \\\n retstrm.flush(); \\\n if (packet_type_mask & CONTROL_PACKET) { \\\n dc.control_call(source, request_reply_handler, id, blob(retstrm->str, retstrm->len));\\\n } \\\n else { \\\n dc.reply_remote_call(source, request_reply_handler, id, blob(retstrm->str, retstrm->len));\\\n } \\\n free(retstrm->str); \\\n} \n\nBOOST_PP_REPEAT(7, DISPATCH_GENERATOR, _)\n\n#undef GENFN\n#undef GENFN2\n#undef GENARGS\n#undef GENPARAMS\n#undef DISPATCH_GENERATOR\n\n/**\nSame as above, but is the non-intrusive version.\n*/\n#define GENFN(N) BOOST_PP_CAT(NIF, N)\n#define GENFN2(N) BOOST_PP_CAT(f, N)\n#define GENNIARGS(Z,N,_) (BOOST_PP_CAT(f, N))\n#define GENPARAMS(Z,N,_) BOOST_PP_CAT(T, N) (BOOST_PP_CAT(f, N)) ; iarc >> (BOOST_PP_CAT(f, N)) ;\n#define CHARSTRINGFREE(Z,N,_) charstring_free(BOOST_PP_CAT(f, N));\n\n#define NONINTRUSIVE_DISPATCH_GENERATOR(Z,N,_) \\\ntemplate<typename DcType, typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nvoid BOOST_PP_CAT(NONINTRUSIVE_REQUESTDISPATCH,N) (DcType& dc, procid_t source, unsigned char packet_type_mask, \\\n const char* buf, size_t len) { \\\n iarchive iarc(buf, len); \\\n size_t s; iarc >> s; F f = reinterpret_cast<F>(s); \\\n size_t id; iarc >> id; \\\n BOOST_PP_REPEAT(N, GENPARAMS, _) \\\n typename function_ret_type<__GLRPC_FRESULT>::type ret = function_ret_type<__GLRPC_FRESULT>::BOOST_PP_CAT(fcall, N) \\\n (f BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENNIARGS ,_)); \\\n BOOST_PP_REPEAT(N, CHARSTRINGFREE, _) \\\n boost::iostreams::stream<resizing_array_sink> retstrm(128); \\\n oarchive oarc(retstrm); \\\n oarc << ret; \\\n retstrm.flush(); \\\n if (packet_type_mask & CONTROL_PACKET) { \\\n dc.control_call(source, request_reply_handler, id, blob(retstrm->str, retstrm->len));\\\n } \\\n else if(packet_type_mask & FLUSH_PACKET) { \\\n dc.reply_remote_call(source, request_reply_handler, id, blob(retstrm->str, retstrm->len));\\\n } \\\n else { \\\n dc.remote_call(source, request_reply_handler, id, blob(retstrm->str, retstrm->len));\\\n } \\\n free(retstrm->str); \\\n} \n\nBOOST_PP_REPEAT(7, NONINTRUSIVE_DISPATCH_GENERATOR, _)\n\n\n#undef GENFN\n#undef GENFN2\n#undef GENNIARGS\n#undef GENPARAMS\n#undef NONINTRUSIVE_DISPATCH_GENERATOR\n\n} // namespace dc_impl\n} // namespace graphlab\n#include <graphlab/rpc/function_arg_types_undef.hpp>\n#endif\n\n" }, { "path": "src/graphlab/rpc/request_future.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef OBJECT_REQUEST_FUTURE_HPP\n#define OBJECT_REQUEST_FUTURE_HPP\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/request_reply_handler.hpp>\n#include <graphlab/rpc/function_ret_type.hpp>\n\nnamespace graphlab {\n\n\n /**\n * \\ingroup rpc\n * The result of a remote_request future call.\n * This class represents the outcome of a remote request sent to another\n * machine via the future-based remote_request_call. The future remote_request call\n * returns immediately with this object. Only when operator() is called on this\n * object, then it waits for a result from the remote machine. All remote_request\n * calls which return futures are linked below.\n *\n * example:\n * \\code\n * // this function returns immediately\n * graphlab::request_future<int> res = \n * rmi.future_remote_request(SOME_OTHER_MACHINE, \n * function_which_returns_an_integer, ...);\n *\n * ... we can do other stuff ... \n * // read the result, or wait for the result if it is not done yet.\n * int actual_result = res();\n * \\endcode\n *\n * \\see graphlab::distributed_control::future_remote_request\n * graphlab::dc_dist_object::future_remote_request\n * graphlab::fiber_remote_request\n * graphlab::object_fiber_remote_request\n *\n * The future object holds a copy of the result of the request, and the\n * operator() call returns a reference to this result (once it is available).\n */\ntemplate <typename T>\nstruct request_future {\n typedef typename dc_impl::function_ret_type<T>::type result_type;\n mutable std::auto_ptr<dc_impl::ireply_container> reply;\n result_type result;\n bool hasval;\n\n /// default constructor\n request_future(): \n reply(new dc_impl::basic_reply_container),\n hasval(false) { }\n\n\n /** constructor which allows you to specify a custom target container\n * This class takes ownership of the container and will free it when done.\n */\n request_future(dc_impl::ireply_container* container): \n reply(container),\n hasval(false) { }\n\n\n /** We can assign return values directly to the future in the\n * case where no remote calls are necessary. \n * Thus allowing the following to be written easily:\n * \\code\n * request_future<int> a_function(int arg) {\n * if (arg == 0) return rmi.future_remote_request(... somewhere else ...) ;\n * else return 10;\n * }\n * \\endcode\n */\n request_future(const T& val): \n reply(NULL),\n result(val), \n hasval(true) { }\n\n /// copy constructor \n request_future(const request_future<T>& val): \n reply(val.reply),\n result(val.result), \n hasval(val.hasval) { }\n\n /// operator=\n request_future& operator=(const request_future<T>& val) {\n reply = val.reply;\n result = val.result;\n hasval = val.hasval;\n return *this;\n }\n\n /**\n * \\internal\n * Returns a handle to the underlying container\n */\n size_t get_handle() {\n return reinterpret_cast<size_t>(reply.get());\n }\n\n /** \n * Waits for the request if it has not yet been received.\n */\n void wait() {\n if (!hasval) {\n reply->wait(); \n dc_impl::blob& receiveddata = reply->get_blob();\n iarchive iarc(receiveddata.c, receiveddata.len); \n iarc >> result; \n receiveddata.free(); \n hasval = true;\n }\n }\n\n /**\n * Returns true if the result is ready and \\ref operator()\n * can be called without blocking.\n */\n bool is_ready() {\n return (hasval || reply->ready());\n }\n\n /**\n * Waits for the request if it has not yet been received.\n * When the result is ready, it returns a reference to the received value.\n */\n result_type& operator()() {\n if (!hasval) wait();\n return result;\n }\n};\n\n\ntemplate <>\nstruct request_future<void> {\n typedef dc_impl::function_ret_type<void>::type result_type;\n mutable std::auto_ptr<dc_impl::ireply_container> reply;\n bool hasval;\n\n request_future(): \n reply(new dc_impl::basic_reply_container),\n hasval(false) { }\n\n request_future(dc_impl::ireply_container* container): \n reply(container),\n hasval(false) { }\n\n request_future(int val): \n reply(NULL),\n hasval(true) { }\n \n \n request_future(const request_future<void>& val): \n reply(val.reply),\n hasval(val.hasval) { }\n\n request_future& operator=(const request_future<void>& val) {\n reply = val.reply;\n hasval = val.hasval;\n return *this;\n }\n\n bool is_ready() {\n return (hasval || reply->ready());\n }\n\n\n size_t get_handle() {\n return reinterpret_cast<size_t>(reply.get());\n }\n\n void wait() {\n if (!hasval) {\n result_type result;\n reply->wait(); \n dc_impl::blob& receiveddata = reply->get_blob();\n iarchive iarc(receiveddata.c, receiveddata.len); \n iarc >> result; \n receiveddata.free(); \n hasval = true;\n }\n }\n\n result_type operator()() {\n if (!hasval) wait();\n return 0;\n }\n};\n\n\n}\n#endif\n" }, { "path": "src/graphlab/rpc/request_issue.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef REQUEST_ISSUE_HPP\n#define REQUEST_ISSUE_HPP\n#include <sstream>\n#include <iostream>\n#include <string>\n#include <graphlab/serialization/serialization_includes.hpp>\n#include <graphlab/rpc/dc_types.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/rpc/request_reply_handler.hpp>\n#include <graphlab/rpc/request_future.hpp>\n#include <graphlab/rpc/request_dispatch.hpp>\n#include <graphlab/rpc/function_ret_type.hpp>\n#include <graphlab/rpc/function_arg_types_def.hpp>\n#include <graphlab/rpc/dc_thread_get_send_buffer.hpp>\n#include <graphlab/rpc/dc_compile_parameters.hpp>\n#include <boost/preprocessor.hpp>\n\nnamespace graphlab {\nnamespace dc_impl {\n\n/**\n\n\\internal\n\\ingroup rpc\n\\file request_issue.hpp\n\nThis is an internal function and should not be used directly.\n\nThis is an internal function and should not be used directly.\nA request is an RPC which is performed \"synchronously\". The return value of the\nfunction is returned.\n\nThe format of the RPC request is in the form of an archive and is as follows\n\nThe format of a \"request\" packet is in the form of an archive and is as follows\n\n\\li (dispatch_type*) -- pointer to target machine's dispatcher function\n\\li (void*) -- pointer to target function\n\\li size_t -- return ID\n\\li fn::arg1_type -- target function's 1st argument\n\\li fn::arg2_type -- target function's 2nd argument\n\\li ...\n\\li fn::argN_type -- target function's Nth argument\n\n\nThe ID here is a pointer to a ireply_container datastructure. When the remote machine completes\nthe function call, it will issue an RPC to the function reply_increment_counter on the originating machine.\nThe reply_increment_counter function store the serialized return value in the ireply_container , as well\nas perform an atomic increment on the ireply_container .\n\nHere is an example of the marshall code for 1 argument\n\\code\nnamespace request_issue_detail {\n template < typename BoolType, typename F, typename T0 > \n struct dispatch_selector1 {\n static dispatch_type dispatchfn () {\n return dc_impl::NONINTRUSIVE_REQUESTDISPATCH1 < distributed_control, F,\n\tT0 >;\n }\n };\n template < typename F, typename T0 > \n struct dispatch_selector1 <boost::mpl::bool_ < true >, F, T0 > {\n static dispatch_type dispatchfn () {\n return dc_impl::REQUESTDISPATCH1 < distributed_control, F, T0 >;\n }\n };\n}\n\ntemplate < typename F, typename T0 > \nclass remote_request_issue1 {\n public:\n static void exec (dc_send * sender, size_t request_handle,\n unsigned char flags, procid_t target, F remote_function,\n const T0 & i0) {\n oarchive *ptr = get_thread_local_buffer (target);\n oarchive & arc = *ptr;\n size_t len =\n dc_send::write_packet_header (arc, _get_procid (), flags,\n\t\t\t\t _get_sequentialization_key ());\n uint32_t beginoff = arc.off;\n dispatch_type d =\n request_issue_detail::dispatch_selector1 < typename is_rpc_call <\n F >::type, F, T0 >::dispatchfn ();\n arc << reinterpret_cast < size_t > (d);\n arc << reinterpret_cast < size_t > (remote_function);\n arc << request_handle;\n arc << i0;\n *(reinterpret_cast < uint32_t * >(arc.buf + len)) = arc.off - beginoff;\n release_thread_local_buffer (target, flags & CONTROL_PACKET);\n pull_flush_thread_local_buffer (target);\n }\n};\n\\endcode\n\nIf the pointer to the dispatcher function is NULL, the next argument\nwill contain the name of the function. This is a \"portable\" call.\n\\see portable_issue.hpp\n*/\n\n#define GENARGS(Z,N,_) BOOST_PP_CAT(const T, N) BOOST_PP_CAT(&i, N)\n#define GENT(Z,N,_) BOOST_PP_CAT(T, N)\n#define GENARC(Z,N,_) arc << BOOST_PP_CAT(i, N);\n\n\n/**\nThe dispatch_selectorN structs are used to pick between the standard dispatcher and the nonintrusive dispatch\nby checking if the function is a RPC style call or not.\n*/\n#define REMOTE_REQUEST_ISSUE_GENERATOR(Z,N,FNAME_AND_CALL) \\\nnamespace request_issue_detail { \\\ntemplate <typename BoolType, typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nstruct BOOST_PP_CAT(dispatch_selector, N){ \\\n static dispatch_type dispatchfn() { return BOOST_PP_CAT(dc_impl::NONINTRUSIVE_REQUESTDISPATCH,N)<distributed_control,F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N, GENT ,_) >; } \\\n};\\\ntemplate <typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nstruct BOOST_PP_CAT(dispatch_selector, N)<boost::mpl::bool_<true>, F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, T)>{ \\\n static dispatch_type dispatchfn() { return BOOST_PP_CAT(dc_impl::REQUESTDISPATCH,N)<distributed_control,F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N, GENT ,_) >; } \\\n}; \\\n}\\\ntemplate<typename F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, typename T)> \\\nclass BOOST_PP_CAT(FNAME_AND_CALL, N) { \\\n public: \\\n static void exec(dc_send* sender, size_t request_handle, unsigned char flags, procid_t target, F remote_function BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM(N,GENARGS ,_) ) { \\\n oarchive* ptr = get_thread_local_buffer(target); \\\n oarchive& arc = *ptr; \\\n size_t len = dc_send::write_packet_header(arc, _get_procid(), flags, _get_sequentialization_key()); \\\n uint32_t beginoff = arc.off; \\\n dispatch_type d = BOOST_PP_CAT(request_issue_detail::dispatch_selector,N)<typename is_rpc_call<F>::type, F BOOST_PP_COMMA_IF(N) BOOST_PP_ENUM_PARAMS(N, T) >::dispatchfn(); \\\n arc << reinterpret_cast<size_t>(d); \\\n arc << reinterpret_cast<size_t>(remote_function); \\\n arc << request_handle; \\\n BOOST_PP_REPEAT(N, GENARC, _) \\\n *(reinterpret_cast<uint32_t*>(arc.buf + len)) = arc.off - beginoff; \\\n release_thread_local_buffer(target, flags & CONTROL_PACKET); \\\n if (flags & FLUSH_PACKET) pull_flush_soon_thread_local_buffer(target); \\\n }\\\n};\n\n\n/**\nGenerates a function call issue. 3rd argument is the issue name\n*/\nBOOST_PP_REPEAT(7, REMOTE_REQUEST_ISSUE_GENERATOR, remote_request_issue )\n\n\n\n#undef GENARC\n#undef GENT\n#undef GENARGS\n#undef REMOTE_REQUEST_ISSUE_GENERATOR\n\n\n} // namespace dc_impl\n} // namespace graphlab\n#include <graphlab/rpc/function_arg_types_undef.hpp>\n\n#endif\n\n" }, { "path": "src/graphlab/rpc/request_reply_handler.cpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <string>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/request_reply_handler.hpp>\n\nnamespace graphlab {\n\nvoid request_reply_handler(distributed_control &dc, procid_t src, \n size_t ptr, dc_impl::blob ret) {\n dc_impl::ireply_container* a = reinterpret_cast<dc_impl::ireply_container*>(ptr);\n a->receive(src, ret);\n}\n\n\n}\n\n" }, { "path": "src/graphlab/rpc/request_reply_handler.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef REPLY_INCREMENT_COUNTER_HPP\n#define REPLY_INCREMENT_COUNTER_HPP\n#include <string>\n#include <graphlab/parallel/atomic.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\nnamespace graphlab {\n\nclass distributed_control;\n\nnamespace dc_impl {\n/**\n\\ingroup rpc\n\\internal\nA wrapper around a char array. This structure \nis incapable of freeing itself and must be managed externally\n*/\nstruct blob {\n /// Constructs a blob containing a pointer to a character array with length len\n blob(char* c, size_t len):c(c),len(len) { };\n blob():c(NULL), len(0){ };\n \n char *c; ///< stored pointer \n size_t len; ///< stored length\n \n \n /// serialize the char array\n void save(oarchive& oarc) const {\n oarc << len;\n if (len > 0) serialize(oarc, c, len);\n }\n \n /// deserializes a char array. If there is already a char array here, it will be freed\n void load(iarchive& iarc) {\n if (c) ::free(c);\n c = NULL;\n iarc >> len;\n if (len > 0) {\n c = (char*) malloc(len);\n deserialize(iarc, c, len);\n }\n }\n \n /// Free the stored char array.\n void free() {\n if (c) {\n ::free(c);\n c = NULL;\n len = 0;\n }\n }\n};\n\n\n/**\n *\\internal\n * \\ingroup rpc\n * Abstract class for where the result of a request go into.\n */\nstruct ireply_container {\n ireply_container() { }\n virtual ~ireply_container() { }\n virtual void wait() = 0;\n virtual void receive(procid_t source, blob b) = 0;\n virtual bool ready() const = 0;\n virtual blob& get_blob() = 0;\n};\n\n\n/**\n\\internal\n\\ingroup rpc\nThe most basic container for replies. Only waits for one reply,\nand uses a mutex/condition variable pair to lock and wait on the reply value.\n\\see ireply_container \n*/\nstruct basic_reply_container: public ireply_container{\n blob val;\n mutex mut;\n conditional cond;\n bool valready;\n /**\n * Constructs a reply object which waits for 'retcount' replies.\n */\n basic_reply_container():valready(false) { }\n \n ~basic_reply_container() { \n val.free();\n }\n\n void receive(procid_t source, blob b) {\n mut.lock();\n val = b;\n valready = true;\n cond.signal();\n mut.unlock();\n }\n /**\n * Waits for all replies to complete. It is up to the \n * reply implementation to decrement the counter.\n */\n inline void wait() {\n mut.lock();\n while(!valready) cond.wait(mut);\n mut.unlock();\n }\n\n inline bool ready() const {\n return valready;\n }\n\n blob& get_blob() {\n return val;\n }\n};\n\n\n} // namespace dc_impl\n\n\n/**\n * \\internal\n * \\ingroup rpc\n * The RPC call to handle the result of a request.\n *\n * The basic protocol of a request is as such:\n * On the sender side, a request_future is created which contains within it\n * an instance of an ireply_container. A message is then sent to the target\n * machine containing the address of the ireply_container.\n * Once the target machine finishes evaluating the function, it issues a\n * call to the request_reply_handler function, passing the original address\n * into the ptr argument. The request_reply_handler then reinterprets the ptr\n * argument as an ireply_container object and calls the receive() function \n * on it.\n * \\see ireply_container\n */\nvoid request_reply_handler(distributed_control &dc, procid_t src, \n size_t ptr, dc_impl::blob ret);\n\n\n} // namespace graphlab\n\n#endif\n\n" }, { "path": "src/graphlab/rpc/rpc.dox", "content": "/**\n\\page RPC GraphLab RPC\n\nGraphLab RPC primary design goal was to provide a convenient and easy to use\nasynchronous communication system between \\b identical binaries running\non different machines over a distributed network. It therefore provides \nMPI-like capabilities together with RPC functionality. The GraphLab distributed \nimplementation is built on top of this RPC library.\n\nGraphLab RPC uses extensive template meta-programming techniques to provide\nan \\b IDL-free (http://en.wikipedia.org/wiki/Interface_description_language) \nRPC system, allowing arbitrary functions to be called on program running on \nremote machines (Note that all machines must be running the same binary).\n\nFor instance, this is a particularly interesting example:\n\\code\n#include <iostream>\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_init_from_mpi.hpp>\nusing namespace graphlab;\n\nint main(int argc, char ** argv) {\n mpi_tools::init(argc, argv);\n\n distributed_control dc;\n \n if (dc.procid() == 0 && dc.numprocs() >= 2) {\n dc.remote_call(1, printf, \"%d + %f = %s\\n\", 1, 2.0, \"three\");\n }\n dc.barrier();\n}\n\\endcode\n\nThe distributed_control constructor will first detect if MPI is initialized,\nand if it is, will use MPI to perform initialization (\\ref sec_spawning_mpi).\nIf MPI is not initialized, then the constructor will check if an alternate\nspawning process using environment variables is used (\\ref sec_spawning_rpcexec).\nThe environment variable based spawning process is less reliable, but useful\nin situations where MPI is not available.\n\n\nOnce the distributed_control object is created, \\ref graphlab::distributed_control::procid \"dc.procid()\"\nprovides the current machine number, while \\ref graphlab::distributed_control::numprocs \"dc.numprocs()\"\nprovide the total number of machines.\n\n\nThe if-condition is therefore entered by only the first machine, which\nperforms a remote call to the second machine (the first argument of remote_call\nis the target machine ID). The second machine will then\nexecute the equivalent of\n\\code\n printf(\"%d + %f = %s\\n\", 1, 2.0, \"three\");\n\\endcode\n\nWe will discuss the different aspects of the RPC library seperately:\n\\li \\ref Spawning \\n\n Initialization and Starting a distributed program using GraphLab RPC\n\\li \\ref Basic_RPC \\n\n Basic usage of the RPC library. Calling of simple functions.\n\\li \\ref OOP_RPC \\n\n Advanced usage of the RPC library. Creating and managing \n distributed object contexts.\n\\li \\ref Fiber_RPC \\n\n Fiber-compatible remote request calls.\n\n\\section sec_examples Examples\nThe tests/ directory include a collection of nine RPC examples demonstrating\nall the key features.\n\n\\li RPC Example 1: Basic Synchronous RPC \\ref rpc_example1.cpp\n\\li RPC Example 2: Asynchronous RPC with Built-in Serialization \\ref rpc_example2.cpp\n\\li RPC Example 3: Asynchronous RPC with Struct POD Serialization \\ref rpc_example3.cpp\n\\li RPC Example 4: Asynchronous RPC with Manual Serialization \\ref rpc_example4.cpp\n\\li RPC Example 5: Asynchronous RPC to printf \\ref rpc_example5.cpp\n\\li RPC Example 6: Asynchronous RPC with graphlab::any \\ref rpc_example6.cpp\n\\li RPC Example 7: Distributed Object \\ref rpc_example7.cpp\n\\li RPC Example 8: RPC using iterators over machines \\ref rpc_example8.cpp\n\\li RPC Example 9: Distributed Object RPC using iterators over machines \\ref rpc_example9.cpp\n\n\n\n\n\\section sec_spawning Spawning and Initialization\nSpawning is the process of starting an instance of GraphLab RPC on seperate \nmachines. GraphLab RPC supports two spawning methods: MPI or rpcexec.py \n(a script in the scripts/ directory). The MPI method is <b>strongly recommended</b>\nand is the most reliable.\n\n\n\\subsection sec_spawning_mpi Spawning with MPI\nGraphLab was tested with MPICH2, but should also with OpenMPI.\nRefer to the documentation for MPICH2 or OpenMPI to set up MPI and make sure\nthat you can run the basic test MPI programs (MPICH2 comes with an mpdringtest). \n\nNo additional configuration is necessary to spawn a GraphLab RPC program with MPI.\n\nThe GraphLab RPC program should begin with:\n\n\\code\n#include <graphlab/rpc/dc.hpp>\nusing namespace graphlab;\n\nint main(int argc, char ** argv) {\n mpi_tools::init(argc, argv);\n distributed_control dc;\n ...\n}\n\\endcode\n\nIn this case, distributed_control detects that MPI was initialized\nprior and will use MPI to perform initial negotiation of port numbers.\n\n\\subsection sec_spawning_rpcexec Spawning with rpcexec.py\nrpcexec.py provides an alternative, less reliable way \nto run a process on a collection of machines,\nusing ssh to communicate between them. <tt>rpcexec.py --help</tt> provides\nsome basic help.\n\nYou will first need to create a host file which is simply a list of host names\nand IP addresses:\n\\verbatim\nlocalhost\n192.168.1.5\nnode2\nnode3\nlocalhost\n192.168.1.5\nnode2\nnode3\n\\endverbatim\n\nRunning <tt>rpcexec.py -n [num to start] -f [hostsfile] `command`</tt> will read the first\nexecute the command on the first N hosts in the hostfile. For instance in this case, running\n\\verbatim\nrpcexec.py -n 5 -f hostsfile ls\n\\endverbatim\nwill run the <tt>ls</tt> bash command twice on the localhost, and once on \nthe three nodes : 192.168.1.5, node2, node3.\n\nrpcexec.py also supports a 'screen' (GNU Screen) mode. Running\n\\verbatim\nrpcexec.py -s lsscreen -n 3 -f hostsfile ls\n\\endverbatim\nwill create a `screen` session with 3 windows where one window ran `ls` on the\nlocalhost, while two other windows sshed into 192.168.1.5 and <tt>node2</tt>, \nrunning the `ls` on each of them. The screen session will be named \"lsscreen\" \n\nrpcexec.py will terminate immediately after creating the screen session.\n\\verbatim\nscreen -r lsscreen\n\\endverbatim\nwill display and resume the screen session.\n\nIf rpcexec.py is used to spawn the program, The GraphLab RPC program should \nbegin with:\n\\code\n#include <graphlab/rpc/dc.hpp>\nusing namespace graphlab;\n\nint main(int argc, char ** argv) {\n distributed_control dc;\n ...\n}\n\\endcode\n\nSince unlike MPI spawning, there is no existing channel for communicating\nport information between the machines. rpcexec.py therefore uses environment\nvariables to pass information to the GraphLab RPC process. The following \ntwo environment variables are used:\n\\li \\b SPAWNNODES A comma seperated list of hostnames participating in the distributed program\n\\li \\b SPAWNID: The index of the current machine into the SPAWNNODES list. First machine\nhas an index value of 0.\n\nA machine will listen on the port 10000 + SPAWNID.\n\nSee \\ref graphlab::dc_init_param \"dc_init_param\" for details about additional\nconfiguration options.\n\nThis spawning system is less flexibile due to the fixed port numbering. For instance,\na crashed process will keep the port in TIMED_WAIT for a few minutes, preventing\nthe next RPC process from running. This also prevents multiple different GraphLab RPC programs\nfrom running on the same set of the machines.\n\nThe MPI spawner is therefore the recommended method for starting the RPC system.\n\n\n\\section sec_rpc_usage RPC Usage Overview\n The graphlab::distributed_control object provides asynchronous, multi-threaded\n Remote Procedure Call (RPC) services to allow distributed GraphLab\n processes to communicate with each other. Currently, the only\n communication method implemented is TCP/IP. \n\n Each process is assigned a sequential process ID at starting at 0. \n i.e. The first process will have a process ID of 0, the second process\n will have an ID of 1, etc. graphlab::distributed_control::procid() can be used to\n obtain the current machine's process ID, and graphlab::distributed_control::numprocs()\n can be used to obtain the total number of processes. \n\n The primary functions used to communicate between processes are\n graphlab::distributed_control::remote_call() and\n graphlab::distributed_control::remote_request(). These functions are thread-safe and\n can be called very rapidly as they only write into a local buffer.\n Communication is handled by a background thread. On the remote side,\n RPC calls are handled in parallel by a thread pool, and thus may be \n parallelized arbitrarily. Operations such as\n graphlab::distributed_control::full_barrier(), or the sequentialization key\n can be used to get finer grained control over order of execution on the \n remote machine.\n\n A few other additional helper functions are also provided to support \n \"synchronous\" modes of communication. These functions are not thread-safe\n and can only be called on one thread per machine. These functions block \n until all machines call the same function. For instance, if gather() is \n called on one machine, it will not return until all machines call gather().\n\n \\li graphlab::distributed_control::barrier()\n \\li graphlab::distributed_control::full_barrier()\n \\li graphlab::distributed_control::broadcast()\n \\li graphlab::distributed_control::all_reduce()\n \\li graphlab::distributed_control::all_reduce2()\n \\li graphlab::distributed_control::gather()\n \\li graphlab::distributed_control::all_gather()\n\n\\subsection sec_basic_rpc_usage Basic RPC\n\nOnce the distributed_control is set up, it can be used to call functions on remote machines.\nFor instance in the earlier example:\n\\code\nif (dc.procid() == 0) {\n dc.remote_call(1, printf, \"%d + %f = %s\\n\", 1, 2.0, \"three\");\n}\n\\endcode\ncalls printf from machine 0 to machine 1 asynchronously.\n\nIn the GraphLab RPC terminology, a \\b call is a one-way remote function call, while a \n\\b request is a function call which has a return value. \\b calls are executed \nasynchronously and returns immediately, while \\b requests will wait for completion\nof the function on the remote machine.\n\nFor instance in the code below, machine 1 could print \neither \"hello world\", or \"world hello\".\n\\code\nif (dc.procid() == 0) {\n dc.remote_call(1, printf, \"hello \");\n dc.remote_call(1, printf, \"world \");\n}\n\\endcode\n\nRemote calls complete \\b immediately, regardless of how long the function\ntook on the other side. For instance, processor 0 will take almost no time\nrunning through this code.\n\\code\nif (dc.procid() == 0) {\n dc.remote_call(1, sleep, 1);\n}\n\\endcode\n\nHowever, since requests will wait for completion and send back the reply,\nthis could take about a second to run.\n\\code\nif (dc.procid() == 0) {\n dc.remote_request(1, sleep, 1);\n}\n\\endcode\n\nAll arguments and return values will be passed by value. Any argument type\nor return type can be used as long as it is \\ref Serialization \"serializable\".\n \n\n\\subsection sec_rpc_collective Collective Operations\nIn addition to regular RPC operations, A collection of MPI-like collective\noperations are also provided. A collective operation is a function which requires\nall machines to call the same function before execution can proceed.\n\n\\subsubsection sec_rpc_collective_barrier Barrier\nOne of the most useful operations is graphlab::distributed_control::barrier()\nThe barrier() is functionally equivalent to MPI_Barrier(). It requires all machines\nto hit the barrier, before execution is allowed to resume. \nFor instance in the code below, while processor 0 is busy working at compute Pi, \nall other machines will pause at the barrier and wait for the processor 0 to complete\ncomputation and hit the barrier, before execution can proceed.\n\\code\nif (dc.procid() == 0) {\n compute Pi to 1 million digits\n}\ndc.barrier();\n\\endcode\n\n\\subsubsection sec_rpc_collective_fullbarrier Full Barrier\nA \\ref graphlab::distributed_control::full_barrier() \"Full Barrier\" is also provided\nthrough graphlab::distributed_control::full_barrier(). A Full Barrier is like a barrier but \nguarantees that all RPC operations sent before the barrier must complete execution.\n\nFor instance in the example below,\nThe full barrier guarantees that the call to set_a_to_1() must complete on \nall remote machines before execution is allowed to proceed. \nAll machines will therefore print '1'.\n\\code\nint a = 0;\nvoid set_a_to_1() { a = 1; }\n\nint main(int argc, char** argv) {\n graphlab::mpi_tools::init(argc, argv);\n graphlab::distributed_control dc;\n\n dc.remote_call( [ another machine ], set_a_to_1);\n dc.full_barrier();\n std::cout << a;\n}\n\\endcode\n\nThe full_barrier is about 2-3x more costly than the regular barrier and should be\nused sparingly.\n\n\\subsubsection sec_rpc_collective_other_collectives Other Collectives\nIn addition to the barrier and the full barrier, operations such as broadcast,\ngather, all_gather are also provided.\nNote that the implementation of these operations are not particularly efficient\nas compared to native MPI implementations due to simplistic algorithm choices.\n\n\\subsection sec_rpc_sequentialization Sequentialization\nA slightly more unusual feature of the GraphLab RPC system is the ability to\nenforce sequentialization of a sequence of RPC calls. This is particularly\nuseful for asynchronous usages of this RPC library and can simplify code in many\ncases.\n\nFor instance, in the code below:\n\\code\nint a = 0;\nvoid set_a_to_1() { a = 1; }\nvoid print_a() { std::cout << a; }\n\nint main(int argc, char** argv) {\n graphlab::mpi_tools::init(argc, argv);\n graphlab::distributed_control dc;\n targetmachine = (dc.procid() + 1) % dc.numprocs();\n dc.remote_call(targetmachine, set_a_to_1);\n dc.remote_call(targetmachine, print_a);\n}\n\\endcode\nNote that due to the asynchronous nature of the remote_call, it is possible for\n<tt>print_a()</tt> to complete on the target machine, before the variable\n<tt>a</tt> is set to 1. Therefore, it is possible for the output to be '0'.\n\nA possible solution as suggested before is to change the remote_calls to\nremote_requests. However, requests incur a large performance penalty due to\nthe need to wait for replies.\n\nAlternatively, we can use the sequentialization key system:\n\\code\n// set the sequentialization key to a non-zero value\nchar oldkey = graphlab::distributed_control::set_sequentialization_key(123);\n\ndc.remote_call(targetmachine, set_a_to_1);\ndc.remote_call(targetmachine, print_a);\n\ngraphlab::distributed_control::set_sequentialization_key(oldkey);\n\\endcode\n\nEssentially all RPC calls made using the same key value (as long as the\nkey value is non-zero) will sequentialize. This enforces that calls/requests\nmade while a key is set will always be processed by the same thread in the\nthread pool on the target machine, ensuring sequentialization of the\n<tt>set_a_to_1</tt> and the <tt>print_a</tt> call. \n\nThe sequentialization key is unique to each \\b thread (thread-local) so\nsequentialization of RPC calls in one thread will not affect RPC calls made by\nother threads.\n\n\n\n\\section OOP_RPC Distributed Objects\n\nGraphLab provides a \"distributed object\" system which simplifies the process\nof designing data structures which provide distributed computation and storage.\n\nA GraphLab distributed object is an object which is instantiated at the same\ntime across all machines. The object internally contains a <tt>dc_dist_object</tt>\nwhich provides RPC communication between distributed instances.\n\nFor instance, say we run the following code using two machines:\n\\code\nint main(int argc, char** argv) {\n graphlab::mpi_tools::init(argc, argv);\n graphlab::distributed_control dc;\n\n graphlab::dht<std::string, std::string> str_map(dc);\n dc.barrier();\n\n if (dc.procid() == 0) {\n str_map.set(\"hello\", \"world\");\n }\n else if (dc.procid() == 1) {\n str_map.set(\"something\", \"other\");\n }\n dc.barrier();\n std::cout << str_map.get(\"hello\").second;\n std::cout << str_map.get(\"something\").second;\n}\n\\endcode\nThe DHT is a distributed object which provides a distributed key/value\nstore (a distributed \"Hash Table\"). Every entry is stored at a machine \ncorresponding to a hash of the key value. Note that it is created at the same time\non all the machines. The barrier() after creation ensures that the object is\ninstantiated properly on all machines before utilization.\n\nNow, after initialization, the <tt>set</tt> function of the dht will internally\nhash the key value and forward it to the right machine for processing. <tt>get</tt>\nis similar. However, since the distributed object system operates on \\b instances,\nit is possible to create multiple distributed objects easily. For instance,\nthe following code will create 50 different distributed key/value maps. \nstr_map[15] corresponds to the same DHT when accessed on any machine.\n\\code\ngraphlab::dht<std::string, std::string>* str_map[50];\nfor (size_t i = 0;i < 50; ++i) {\n str_map[i] = new graphlab::dht<std::string, std::string>(dc); \n}\n\\endcode\n\n\\subsection sec_oop_rpc_usage Usage\nWe will demonstrate usage of the distributed object system using a simple \ndistributed Hash Table example. Note that this is a \\b very \\b simple implementation,\nand is not entirely correct since we are going to ignore thread-safety. But it is\nsufficient to demonstrate the key concepts.\n\n\\code\nclass string_dht { \n private:\n std::map<int, std::string> local_storage;\n mutable dc_dist_object<string_dht> rmi;\n\\endcode\n\nFirst, each machine needs a local data storage. In this case we will simply \nuse a std::map. The key object that provides distributed access is the \n<tt>dc_dist_object\\<string_dht\\> rmi;</tt>. This object creates a \"context\" \nfor remote function calls, allowing the correct remote instance to be identified.\n\nWe will now look at the string_dht constructor. The rmi object constructor\nrequires a reference to the underlying distributed_control object, as well\nas a pointer to the current instance:\n\\code\n public:\n string_dht(distributed_control &dc): rmi(dc, this) { }\n\\endcode\n\nNow, to demonstrate how the RMI object is used, lets see the set() function\n\\code\nvoid set(int key, const std::string &newval) { \n procid_t owningmachine = key % rmi.numprocs();\n if (owningmachine == rmi.procid()) {\n local_storage[key] = newval;\n }\n\\endcode\nWe use a simple hash function to identify where the key-value pair should be \nstored. Observe that the RMI object provides pretty much the same functionality\nas the graphlab::distributed_control object, having both graphlab::dc_dist_object::numprocs()\nand graphlab::dc_dist_object::procid(). If the data is to be stored in the current machine, \nwe simply store it. Otherwise we will need to send it to a remote machine for\nstorage. This is the interesting case:\n\n\\code\n else {\n rmi.remote_call(owningmachine,\n &string_dht::set,\n key,\n newval);\n }\n}\n\\endcode\n\nThe RMI object supports the same family of call/request operations as \n\\ref sec_rpc_dc \"distributed_control\"\nHowever, it will only work with <b>member function pointers</b>. For instance in this case,\nwe will be calling the set() member function on the matching instance of the string_dht\nobject on a remote machine. (Note that the & is important and necessary)\n\nThe get() function is similar. However, we will have to use remote requests.\n\n\\code\nstd::string get(int key) { \n procid_t owningmachine = key % rmi.numprocs();\n if (owningmachine == rmi.procid()) {\n return local_storage[key];\n }\n else {\n return rmi.remote_request(owningmachine,\n &string_dht::get,\n key);\n }\n}\n\\endcode\n\nAs stated earlier, this code should not be used as it is due to several limitations\nsuch as the local_storage object is not thread-safe. Since incoming RPC calls are\ngenerally multithreaded, locks are necessary. See dht.hpp for an equivalent \n\"safe\" example of a simple DHT.\n\n\\subsection sec_oop_rpc_context Context\nEssentially, the dc_dist_object object supports the identical set of operations as the \ndistributed_control object, but restricted to the \\b context of a single object instance.\n\nIt includes all the regular call operations:\n\\li graphlab::dc_dist_object::remote_call()\n\\li graphlab::dc_dist_object::remote_request()\n\nAdditionally, this \\b context is entirely independent of the distributed_control\nobject, permitting its own set of collective operations such as\ngraphlab::dc_dist_object::broadcast, graphlab::dc_dist_object::barrier,\n graphlab::dc_dist_object::full_barrier, etc\n\nSince these collective operations also operate entirely within the context of the object\ninstance, this permits the use of parallel collectives. For instance, I could have\ntwo objects, and each object internally spawns threads to perform distributed computation;\nusing the RMI object to perform collective operations which are local to the object.\n\nIn particular, the graphlab::dc_dist_object::full_barrier() is worth taking note of. \nThe graphlab::distributed_control::full_barrier() ensures completion of ALL RPC calls\nincluding calls meant for distributed objects. Its barrier is therefore\n\\b global to the state of the program as a while.\nThe graphlab::dc_dist_object::full_barrier() however, only ensures completion of all RPC\ncalls within the object instance. Its barrier is therefore \\b local to the state\nof the distributed object. This allows each distributed object to run its own\nfull barriers without affecting other distributed objects.\n\n\\subsection sec_oop_rpc_notes Final Notes\nFinally, note that the RMI object can ONLY call member function pointers.\nIt cannot call other global functions (such as printf).\nThe global context can be accessed through graphlab::dc_dist_object::dc() which\nreturns the underlying distributed_control object, which can then be used\nto call global functions. For instance:\n\\code\nrmi.dc().remote_call(1, printf, \"hello \");\n\\endcode\n\n\n\n\\section Fiber_RPC Fiber Compatible Remote Requests\nTo support the fiber architecture required for the Warp Engine, we provide \nthe following functions:\n\n\\li graphlab::fiber_remote_request() \n\\li graphlab::object_fiber_remote_request()\n\nThese two functions are special in that unlike the remote_request functions,\nthey return immediately with a future object.\n\nFor instance:\n\n\\code\nint add_one(int a) {\n return a + 1;\n}\n\n... /* elsewhere */\ngraphlab::request_future<int> future = fiber_remote_request(1, /* call to machine 1 */\n add_one,\n 1);\n\\endcode\n\nWaiting on the future, using either:\n\\code\nint ret = future();\n/* Or, more explicitly ... */\nfuture.wait();\nint ret = future();\n\\endcode\n\nWill block until the result is available.\nThis wait, however, is optimized if the caller is in a fiber, in which case\nthe fiber is descheduled, allowing other fibers to execute while waiting for the \nresult.\n\nThe graphlab::object_fiber_remote_request() function is similar, but allows\nfor calling of member functions of a class.\n\n*/\n\n" }, { "path": "src/graphlab/rpc/rpc_includes.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n \n\n#ifndef GRAPHLAB_RPC_INCLUDES\n#define GRAPHLAB_RPC_INCLUDES\n\n#include <graphlab/rpc/dc.hpp>\n#include <graphlab/rpc/dc_init_from_mpi.hpp>\n\n#endif\n" }, { "path": "src/graphlab/rpc/sample_sort.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_RPC_SAMPLE_SORT_HPP\n#define GRAPHLAB_RPC_SAMPLE_SORT_HPP\n\n#include <vector>\n#include <algorithm>\n#include <utility>\n#include <graphlab/rpc/dc_dist_object.hpp>\n#include <graphlab/rpc/buffered_exchange.hpp>\n#include <graphlab/logger/assertions.hpp>\nnamespace graphlab {\n\nnamespace sample_sort_impl {\n template <typename Key, typename Value>\n struct pair_key_comparator {\n bool operator()(const std::pair<Key,Value>& k1,\n const std::pair<Key,Value>& k2) {\n return k1.first < k2.first;\n }\n };\n}\n\ntemplate <typename Key, typename Value>\nclass sample_sort {\n private:\n dc_dist_object<sample_sort<Key, Value> > rmi;\n\n typedef buffered_exchange<std::pair<Key, Value> > key_exchange_type;\n\n key_exchange_type key_exchange;\n std::vector<std::pair<Key, Value> > key_values;\n public:\n sample_sort(distributed_control& dc): rmi(dc, this), key_exchange(dc) { }\n\n template <typename KeyIterator, typename ValueIterator>\n void sort(KeyIterator kstart, KeyIterator kend,\n ValueIterator vstart, ValueIterator vend) {\n rmi.barrier();\n\n size_t num_entries = std::distance(kstart, kend);\n ASSERT_EQ(num_entries, std::distance(vstart, vend));\n\n // we will sample k * p entries\n std::vector<std::vector<Key> > sampled_keys(rmi.numprocs());\n for (size_t i = 0;i < 100 * rmi.numprocs(); ++i) {\n size_t idx = (rand() % num_entries); \n sampled_keys[rmi.procid()].push_back(*(kstart + idx));\n }\n\n rmi.all_gather(sampled_keys);\n // collapse into a single array and sort\n std::vector<Key> all_sampled_keys;\n for (size_t i = 0;i < sampled_keys.size(); ++i) {\n std::copy(sampled_keys[i].begin(), sampled_keys[i].end(),\n std::inserter(all_sampled_keys, all_sampled_keys.end()));\n }\n // sort the sampled keys and extract the ranges\n std::sort(all_sampled_keys.begin(), all_sampled_keys.end());\n std::vector<Key> ranges(rmi.numprocs());\n ranges[0] = Key();\n for(size_t i = 1; i < rmi.numprocs(); ++i) {\n ranges[i] = all_sampled_keys[sampled_keys[0].size() * i];\n }\n\n // begin shuffle \n KeyIterator kiter = kstart;\n ValueIterator viter = vstart;\n if (rmi.numprocs() < 8) {\n while(kiter != kend) {\n procid_t target_machine = 0;\n while (target_machine < rmi.numprocs() - 1 && \n ranges[target_machine + 1] < *kiter) ++target_machine;\n key_exchange.send(target_machine, std::make_pair(*kiter, *viter));\n ++kiter; ++viter;\n }\n } \n else {\n while(kiter != kend) {\n procid_t target_machine = \n std::upper_bound(ranges.begin(), ranges.end(), *kiter) \n - ranges.begin() - 1;\n key_exchange.send(target_machine, std::make_pair(*kiter, *viter));\n ++kiter; ++viter;\n }\n }\n key_exchange.flush();\n \n // read from key exchange \n procid_t recvid;\n typename key_exchange_type::buffer_type buffer;\n while(key_exchange.recv(recvid, buffer)) {\n std::copy(buffer.begin(), buffer.end(), \n std::inserter(key_values, key_values.end()));\n }\n std::sort(key_values.begin(), key_values.end(), \n sample_sort_impl::pair_key_comparator<Key,Value>());\n\n rmi.barrier();\n }\n\n std::vector<std::pair<Key, Value> >& result() {\n return key_values;\n }\n};\n\n\n} // namespace graphlab\n\n#endif\n" }, { "path": "src/graphlab/rpc/thread_local_send_buffer.cpp", "content": "#include <graphlab/rpc/thread_local_send_buffer.hpp>\n#include <graphlab/rpc/dc.hpp>\nnamespace graphlab {\nnamespace dc_impl {\n\nthread_local_buffer::thread_local_buffer() {\n // allocate the buffers\n dc = distributed_control::get_instance();\n size_t nprocs = dc->numprocs(); \n\n outbuf.resize(nprocs); \n for (size_t i = 0;i < outbuf.size(); ++i) {\n outbuf[i] = new inplace_lf_queue2<buffer_elem>;\n }\n current_archive.resize(nprocs); \n\n archive_locks.resize(nprocs);\n\n bytes_sent.resize(nprocs, 0);\n dc->register_send_buffer(this);\n procid = dc->procid();\n}\n\n\nthread_local_buffer::~thread_local_buffer() {\n dc->unregister_send_buffer(this);\n push_flush();\n // deallocate the buffers\n for (size_t i = 0; i < current_archive.size(); ++i) {\n if (current_archive[i].buf) {\n free(current_archive[i].buf);\n current_archive[i].buf = NULL;\n }\n }\n\n for (size_t i = 0;i < outbuf.size(); ++i) {\n delete outbuf[i];\n }\n outbuf.clear();\n}\n\nvoid thread_local_buffer::inc_calls_sent(procid_t target) {\n dc->inc_calls_sent(target);\n}\n\n\nvoid thread_local_buffer::push_flush() {\n for (size_t i = 0; i < outbuf.size(); ++i) {\n std::pair<buffer_elem*, buffer_elem*> bufs = extract(i);\n if (bufs.first != NULL) {\n while(bufs.first != bufs.second) {\n buffer_elem* prev = bufs.first;\n dc->write_to_buffer(i, bufs.first->buf, bufs.second->len);\n buffer_elem** next = &bufs.first->next;\n volatile buffer_elem** n = (volatile buffer_elem**)(next);\n while(__unlikely__((*n) == NULL)) {\n asm volatile(\"pause\\n\": : :\"memory\");\n }\n bufs.first = (buffer_elem*)(*n);\n delete prev;\n }\n dc->flush_soon(i);\n }\n }\n}\n\n\nvoid thread_local_buffer::pull_flush() {\n dc->flush();\n}\n\nvoid thread_local_buffer::pull_flush(procid_t p) {\n dc->flush(p);\n}\n\n\nvoid thread_local_buffer::pull_flush_soon() {\n dc->flush_soon();\n}\n\n\nvoid thread_local_buffer::pull_flush_soon(procid_t p) {\n dc->flush_soon(p);\n}\n\noarchive* thread_local_buffer::acquire(procid_t target) {\n archive_locks[target].lock();\n // need a new archive, or existing one at risk of being resized\n if (current_archive[target].buf == NULL) {\n current_archive[target].buf = (char*)malloc(INITIAL_BUFFER_SIZE);\n current_archive[target].off = 0;\n current_archive[target].len = INITIAL_BUFFER_SIZE;\n }\n prev_acquire_archive_size = current_archive[target].off;\n return ¤t_archive[target];\n}\n\n\nvoid thread_local_buffer::add_to_queue(procid_t target, char* ptr, size_t len) {\n buffer_elem* elem = new buffer_elem;\n ASSERT_NE(ptr, NULL);\n elem->buf = ptr;\n elem->len = len;\n elem->next = NULL;\n outbuf[target]->enqueue(elem);\n if (outbuf[target]->approx_size() > NUM_FULL_BUFFER_LIMIT) {\n pull_flush_soon(target);\n }\n}\n\nvoid thread_local_buffer::release(procid_t target, bool do_not_count_bytes_sent) {\n if (!do_not_count_bytes_sent) {\n bytes_sent[target] += current_archive[target].off - prev_acquire_archive_size - sizeof(packet_hdr);\n inc_calls_sent(target);\n }\n\n if (current_archive[target].off >= FULL_BUFFER_SIZE_LIMIT) {\n // shift the buffer into outbuf\n char* ptr = current_archive[target].buf;\n size_t len = current_archive[target].off;\n current_archive[target].buf = NULL; \n current_archive[target].off = 0;\n archive_locks[target].unlock();\n\n add_to_queue(target, ptr, len);\n\n } else {\n archive_locks[target].unlock();\n }\n}\n\n\nvoid thread_local_buffer::write(procid_t target, char* c, size_t len, \n bool do_not_count_bytes_sent) {\n if (!do_not_count_bytes_sent) {\n bytes_sent[target] += len;\n inc_calls_sent(target);\n }\n // make sure that messsages sent before this write are sent before this write\n if (current_archive[target].off) {\n archive_locks[target].lock();\n\n if (current_archive[target].off) {\n add_to_queue(target, current_archive[target].buf, current_archive[target].off);\n }\n current_archive[target].buf = NULL; \n current_archive[target].off = 0;\n archive_locks[target].unlock();\n }\n add_to_queue(target, c, len);\n}\n\n\nstd::pair<buffer_elem*, buffer_elem*> thread_local_buffer::extract(procid_t target) {\n if (current_archive[target].off > 0 ) {\n if (archive_locks[target].try_lock()) {\n char* ptr = current_archive[target].buf;\n size_t len = current_archive[target].off;\n if (len > 0) {\n current_archive[target].buf = NULL;\n current_archive[target].off = 0;\n }\n archive_locks[target].unlock();\n if (len > 0) {\n buffer_elem* elem = new buffer_elem;\n ASSERT_NE(ptr, NULL);\n elem->buf = ptr;\n elem->len = len;\n elem->next = NULL;\n outbuf[target]->enqueue(elem);\n }\n } \n } \n std::pair<buffer_elem*, buffer_elem*> ret;\n ret.first = outbuf[target]->dequeue_all();\n if (ret.first != NULL) {\n ASSERT_NE(ret.first->buf, NULL);\n ret.second = outbuf[target]->end_of_dequeue_list();\n return ret;\n } else {\n return std::pair<buffer_elem*, buffer_elem*>(NULL, NULL);\n }\n}\n\n\n\n} // dc_impl\n} // graphlab\n" }, { "path": "src/graphlab/rpc/thread_local_send_buffer.hpp", "content": "#ifndef GRAPHLAB_RPC_THREAD_LOCAL_SEND_BUFFER_HPP\n#define GRAPHLAB_RPC_THREAD_LOCAL_SEND_BUFFER_HPP\n#include <graphlab/serialization/oarchive.hpp>\n#include <graphlab/rpc/dc_compile_parameters.hpp>\n#include <graphlab/rpc/dc_internal_types.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n#include <graphlab/util/inplace_lf_queue2.hpp>\nnamespace graphlab {\nclass distributed_control;\n\n\nnamespace dc_impl {\n\nstruct thread_local_buffer {\n std::vector<inplace_lf_queue2<buffer_elem>* > outbuf;\n std::vector<size_t> bytes_sent;\n\n\n std::vector<mutex> archive_locks;\n std::vector<oarchive> current_archive;\n size_t prev_acquire_archive_size;\n\n procid_t procid;\n distributed_control* dc;\n\n thread_local_buffer();\n ~thread_local_buffer();\n\n /**\n * Must be called from within the thread owning this buffer.\n * Acquires a buffer to write to\n */\n oarchive* acquire(procid_t target);\n\n inline size_t get_bytes_sent(procid_t target) {\n return bytes_sent[target];\n }\n /**\n * Must be called from within the thread owning this buffer.\n * Releases a buffer previously acquired with acquire\n */\n void release(procid_t target, bool do_not_count_bytes_sent);\n\n void write(procid_t target, char* c, size_t len, bool do_not_count_bytes_sent);\n\n /**\n * Must be called from within the thread owning this buffer.\n * Flushes the buffer to the sender. This should really only be used\n * when the thread is dying since this incurs a large performance penalty by\n * locking up the sender.\n */\n void push_flush();\n\n\n /**\n * Can be called anywhere.\n * Flushes the buffer to the sender. This function blocks until all \n * buffers have been flushed. Equivalent to calling distributed_control::flush()\n */\n void pull_flush();\n\n /**\n * Can be called anywhere.\n * Flushes the buffer to the sender. This function blocks until all \n * buffers have been flushed. Equivalent to calling distributed_control::flush()\n */\n void pull_flush(procid_t p);\n\n /**\n * Can be called anywhere.\n * Flushes the buffer to the sender. This function requests a flush to happen\n * soon. Equivalent to calling distributed_control::flush()\n */\n void pull_flush_soon();\n\n\n /**\n * Can be called anywhere.\n * Flushes the buffer to the sender. This function requests a flush to happen\n * soon. Equivalent to calling distributed_control::flush()\n */\n void pull_flush_soon(procid_t p);\n\n /**\n * Extracts the buffer going to a given target.\n * The first element of the pair points to the head of the linked list\n * The linked list ends when the pointer becomes the second element of \n * the pair.\n */\n std::pair<buffer_elem*, buffer_elem*> extract(procid_t target);\n\n void inc_calls_sent(procid_t target);\n\n void add_to_queue(procid_t target, char* ptr, size_t len);\n};\n}\n}\n#endif\n" }, { "path": "src/graphlab/scheduler/CMakeLists.txt", "content": "project(GraphLab)\n" }, { "path": "src/graphlab/scheduler/fifo_scheduler.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <graphlab/scheduler/fifo_scheduler.hpp>\n\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n\nvoid fifo_scheduler::set_options(const graphlab_options& opts) {\n ncpus = opts.get_ncpus();\n std::vector<std::string> keys = opts.get_scheduler_args().get_option_keys();\n foreach(std::string opt, keys) {\n if (opt == \"multi\") {\n opts.get_scheduler_args().get_option(\"multi\", multi);\n } else {\n logstream(LOG_FATAL) << \"Unexpected Scheduler Option: \" << opt << std::endl;\n }\n }\n}\n\n// Initializes the internal datastructures\nvoid fifo_scheduler::initialize_data_structures() {\n current_queue.resize(ncpus, 0);\n size_t nqueues = std::max(multi * current_queue.size(), size_t(1));\n queues.resize(nqueues);\n locks.resize(nqueues);\n vertex_is_scheduled.resize(num_vertices);\n}\n\nfifo_scheduler::fifo_scheduler(size_t num_vertices,\n const graphlab_options& opts):\n multi(3), num_vertices(num_vertices) { \n ASSERT_GE(opts.get_ncpus(), 1);\n set_options(opts);\n initialize_data_structures();\n}\n\n\nvoid fifo_scheduler::set_num_vertices(const lvid_type numv) {\n num_vertices = numv;\n vertex_is_scheduled.resize(numv);\n}\n\nvoid fifo_scheduler::schedule(const lvid_type vid, double priority) {\n if (vid < num_vertices && !vertex_is_scheduled.set_bit(vid)) {\n /* \"Randomize\" the task queue task is put in. Note that we do\n not care if this counter is corrupted in race conditions\n Find first queue that is not locked and put task there (or\n after iteration limit) Choose two random queues and use the\n one which has smaller size */\n // M.D. Mitzenmacher The Power of Two Choices in Randomized\n // Load Balancing (1991)\n // http://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.\n size_t idx = 0;\n if(queues.size() > 1) {\n const uint32_t prod = \n random::fast_uniform(uint32_t(0), \n uint32_t(queues.size() * queues.size() - 1));\n const uint32_t r1 = prod / queues.size();\n const uint32_t r2 = prod % queues.size();\n idx = (queues[r1].size() < queues[r2].size()) ? r1 : r2; \n }\n locks[idx].lock(); queues[idx].push_back(vid); locks[idx].unlock();\n }\n}\n\n/** Get the next element in the queue */\nsched_status::status_enum fifo_scheduler::get_next(const size_t cpuid,\n lvid_type& ret_vid) {\n /* Check all of my queues for a task */\n // begin scanning from the machine's current queue\n size_t initial_idx = (current_queue[cpuid] % multi) + cpuid * multi;\n for(size_t i = 0; i < queues.size(); ++i) {\n const size_t idx = (initial_idx + i) % queues.size();\n // increment the current queue as long as I am scanning with in the \n // queues owned by this machine\n current_queue[cpuid] += (i < multi);\n\n // pick up the lock\n bool good = false;\n locks[idx].lock();\n while(!queues[idx].empty()) {\n // not empty, pop and verify\n ret_vid = queues[idx].front();\n queues[idx].pop_front();\n if (ret_vid < num_vertices) {\n good = vertex_is_scheduled.clear_bit(ret_vid);\n if (good) break;\n }\n }\n locks[idx].unlock();\n // managed to retrieve a task\n if(good) {\n return sched_status::NEW_TASK;\n }\n }\n return sched_status::EMPTY; \n} // end of get_next_task\n\n\nbool fifo_scheduler::empty() {\n for (size_t i = 0;i < queues.size(); ++i) {\n if (!queues[i].empty()) return false;\n }\n return true;\n}\n\n}\n" }, { "path": "src/graphlab/scheduler/fifo_scheduler.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_FIFO_SCHEDULER_HPP\n#define GRAPHLAB_FIFO_SCHEDULER_HPP\n\n#include <algorithm>\n#include <queue>\n\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/parallel/atomic.hpp>\n\n#include <graphlab/util/random.hpp>\n#include <graphlab/scheduler/ischeduler.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n\n#include <graphlab/options/graphlab_options.hpp>\n\nnamespace graphlab {\n\n /**\n * \\ingroup group_schedulers\n *\n * This class defines a multiple queue approximate fifo scheduler.\n * Each processor has its own in_queue which it puts new tasks in\n * and out_queue which it pulls tasks from. Once a processors\n * in_queue gets too large, the entire queue is placed at the end of\n * the shared master queue. Once a processors out queue is empty it\n * grabs the next out_queue from the master.\n */\n class fifo_scheduler : public ischeduler {\n \n public:\n\n typedef std::deque<lvid_type> queue_type;\n\n private:\n\n // a bitset denoting if a vertex is scheduled\n dense_bitset vertex_is_scheduled;\n // a collection of FIFO queues\n std::vector<queue_type> queues;\n // a parallel datastructure to queues containing all the locks\n std::vector<padded_simple_spinlock> locks;\n // the index of the queue currently accessed by a given CPU\n // when used, this is modded so that it ranges from 0 to multi - 1\n std::vector<size_t> current_queue; \n\n\n // the number of CPUs\n size_t ncpus;\n // The queue to CPU ratio\n size_t multi;\n // the number of vertices in the graph\n size_t num_vertices;\n \n \n \n void set_options(const graphlab_options& opts); \n\n // Initializes the internal datastructures\n void initialize_data_structures();\n public:\n\n fifo_scheduler(size_t num_vertices,\n const graphlab_options& opts);\n\n\n void set_num_vertices(const lvid_type numv);\n\n void schedule(const lvid_type vid, double priority = 1 /* ignored */ );\n\n /** Get the next element in the queue */\n sched_status::status_enum get_next(const size_t cpuid,\n lvid_type& ret_vid);\n\n\n bool empty();\n\n static void print_options_help(std::ostream& out) {\n out << \"\\t multi = [number of queues per thread. Default = 3].\\n\";\n }\n\n\n }; \n\n\n} // end of namespace graphlab\n\n#endif\n\n" }, { "path": "src/graphlab/scheduler/get_message_priority.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_SCHEDULER_GET_MESSAGE_PRIORITY_HPP\n#define GRAPHLAB_SCHEDULER_GET_MESSAGE_PRIORITY_HPP\n\n#include <boost/type_traits.hpp>\n#include <typeinfo>\n\nnamespace graphlab {\n \nnamespace scheduler_impl {\n\n template <typename T>\n struct implements_priority_member {\n template<typename U, double (U::*)() const> struct SFINAE {};\n template <typename U> static char test(SFINAE<U, &U::priority>*);\n template <typename U> static int test(...);\n static const bool value = (sizeof(test<T>(0)) == sizeof(char));\n };\n\n template <typename MessageType>\n typename boost::enable_if_c<implements_priority_member<MessageType>::value,\n double>::type\n get_message_priority(const MessageType &m) {\n return m.priority();\n }\n\n template <typename MessageType>\n typename boost::disable_if_c<implements_priority_member<MessageType>::value,\n double>::type\n get_message_priority(const MessageType &m) {\n return 1.0;\n }\n\n} //namespace scheduler_impl\n} //namespace graphlab\n\n\n#endif\n" }, { "path": "src/graphlab/scheduler/ischeduler.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n/**\n * Also contains code that is Copyright 2011 Yahoo! Inc. All rights\n * reserved.\n *\n * Contributed under the iCLA for:\n * Joseph Gonzalez (jegonzal@yahoo-inc.com)\n *\n */\n\n\n#ifndef GRAPHLAB_ISCHEDULER_HPP\n#define GRAPHLAB_ISCHEDULER_HPP\n\n#include <vector>\n#include <sstream>\n#include <ostream>\n\n#include <graphlab/graph/graph_basic_types.hpp>\n\n#include <graphlab/options/graphlab_options.hpp>\n\n\n\nnamespace graphlab {\n\n /**\n * This is an enumeration for the possible return values for\n * get_next_tasks\n */\n struct sched_status {\n /// \\brief the possible scheduler status.\n enum status_enum {\n NEW_TASK, /**< The get_next_tasks function returned a new task\n to be executed */\n EMPTY, /**< The schedule is empty. */\n };\n };\n\n /**\n * \\ingroup group_schedulers\n *\n * This describes the interface/concept for a scheduler. \n * The scheduler allows vertices to be scheduled, but deduplicates\n * repeated schedulings of the same vertex. The only guarantee is that\n * if a vertex is scheduled, the vertex will be popped at some point in\n * the future.\n * Note that all functions (with the exception of the\n * constructor and destructor and set_num_vertices()) must be thread-safe.\n */\n class ischeduler {\n public:\n\n /// destructor\n virtual ~ischeduler() {};\n\n /** Sets the number of vertices in the graph. Existing schedule\n * will not be cleared. Scheduler will not return a vertex ID\n * exceeding the number of vertices.\n */\n virtual void set_num_vertices(const lvid_type numv) = 0;\n\n /**\n * Adds vertex vid to the schedule. The new priority is the priority value\n */\n virtual void schedule(const lvid_type vid, double priority = 1) = 0;\n\n\n /**\n * This function is called by the engine to ask for the next\n * vertex to process. The vertex is \n * returned in ret_msg and ret_vid respectively.\n *\n * \\retval NEWTASK There is a new message to process\n * \\retval EMPTY There are no messages to process\n */\n virtual sched_status::status_enum\n get_next(const size_t cpuid, lvid_type& ret_vid) = 0;\n\n /// returns true if the scheduler is empty. Need not be consistent.\n virtual bool empty() = 0;\n\n /**\n * Print a help string describing the options that this scheduler\n * accepts.\n */\n static void print_options_help(std::ostream& out) { };\n\n };\n\n}\n#endif\n\n" }, { "path": "src/graphlab/scheduler/priority_scheduler.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <graphlab/scheduler/priority_scheduler.hpp>\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n\nvoid priority_scheduler::set_options(const graphlab_options& opts) {\n ncpus = opts.get_ncpus();\n std::vector<std::string> keys = opts.get_scheduler_args().get_option_keys();\n foreach(std::string opt, keys) {\n if (opt == \"multi\") {\n opts.get_scheduler_args().get_option(\"multi\", multi);\n } else if (opt == \"min_priority\") {\n opts.get_scheduler_args().get_option(\"min_priority\", min_priority);\n } else {\n logstream(LOG_FATAL) << \"Unexpected Scheduler Option: \" << opt << std::endl;\n }\n }\n}\n\n// Initializes the internal datastructures\nvoid priority_scheduler::initialize_data_structures() {\n current_queue.resize(ncpus, 0);\n size_t nqueues = std::max(multi * current_queue.size(), size_t(1));\n queues.resize(nqueues);\n locks.resize(nqueues);\n vertex_is_scheduled.resize(num_vertices);\n}\n\npriority_scheduler::priority_scheduler(size_t num_vertices,\n const graphlab_options& opts):\n multi(3), \n min_priority(-std::numeric_limits<double>::max()),\n num_vertices(num_vertices) { \n ASSERT_GE(opts.get_ncpus(), 1);\n set_options(opts);\n initialize_data_structures();\n}\n\n\nvoid priority_scheduler::set_num_vertices(const lvid_type numv) {\n num_vertices = numv;\n vertex_is_scheduled.resize(numv);\n}\n\nvoid priority_scheduler::schedule(const lvid_type vid, double priority) {\n if (vid < num_vertices && !vertex_is_scheduled.set_bit(vid)) {\n /* \"Randomize\" the task queue task is put in. Note that we do\n not care if this counter is corrupted in race conditions\n Find first queue that is not locked and put task there (or\n after iteration limit) Choose two random queues and use the\n one which has smaller size */\n // M.D. Mitzenmacher The Power of Two Choices in Randomized\n // Load Balancing (1991)\n // http://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.\n size_t idx = 0;\n if(queues.size() > 1) {\n const uint32_t prod = \n random::fast_uniform(uint32_t(0), \n uint32_t(queues.size() * queues.size() - 1));\n const uint32_t r1 = prod / queues.size();\n const uint32_t r2 = prod % queues.size();\n idx = (queues[r1].size() < queues[r2].size()) ? r1 : r2; \n }\n locks[idx].lock(); \n queues[idx].push_or_update(vid, priority); \n locks[idx].unlock();\n }\n}\n\n/** Get the next element in the queue */\nsched_status::status_enum priority_scheduler::get_next(const size_t cpuid,\n lvid_type& ret_vid) {\n /* Check all of my queues for a task */\n // begin scanning from the machine's current queue\n size_t initial_idx = (current_queue[cpuid] % multi) + cpuid * multi;\n for(size_t i = 0; i < queues.size(); ++i) {\n const size_t idx = (initial_idx + i) % queues.size();\n // increment the current queue as long as I am scanning with in the \n // queues owned by this machine\n current_queue[cpuid] += (i < multi);\n\n // pick up the lock\n bool good = false;\n locks[idx].lock();\n while(!queues[idx].empty() && queues[idx].top().second >= min_priority) {\n // not empty, pop and verify\n ret_vid = queues[idx].pop().first;\n if (ret_vid < num_vertices) {\n good = vertex_is_scheduled.clear_bit(ret_vid);\n if (good) break;\n }\n else continue;\n }\n locks[idx].unlock();\n // managed to retrieve a task\n if(good) {\n return sched_status::NEW_TASK;\n }\n }\n return sched_status::EMPTY; \n} // end of get_next_task\n\n\nbool priority_scheduler::empty() {\n for (size_t i = 0;i < queues.size(); ++i) {\n if (!queues[i].empty() && queues[i].top().second >= min_priority) {\n return false;\n }\n }\n return true;\n}\n\n}\n" }, { "path": "src/graphlab/scheduler/priority_scheduler.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_PRIORITY_SCHEDULER_HPP\n#define GRAPHLAB_PRIORITY_SCHEDULER_HPP\n\n#include <algorithm>\n#include <queue>\n\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/parallel/atomic.hpp>\n\n#include <graphlab/util/random.hpp>\n#include <graphlab/util/mutable_queue.hpp>\n#include <graphlab/scheduler/ischeduler.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n\n#include <graphlab/options/graphlab_options.hpp>\n\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n\n /**\n * \\ingroup group_schedulers \n *\n * This class defines a multiple queue approximate priority scheduler.\n * Each processor has its own in_queue which it puts new tasks in\n * and out_queue which it pulls tasks from. Once a processors\n * in_queue gets too large, the entire queue is placed at the end of\n * the shared master queue. Once a processors out queue is empty it\n * grabs the next out_queue from the master.\n */\n class priority_scheduler : public ischeduler {\n \n public:\n\n typedef mutable_queue<lvid_type, double> queue_type;\n\n private:\n\n // a bitset denoting if a vertex is scheduled\n dense_bitset vertex_is_scheduled;\n // a collection of priority queues\n std::vector<queue_type> queues;\n // a parallel datastructure to queues containing all the locks\n std::vector<padded_simple_spinlock> locks;\n // the index of the queue currently accessed by a given CPU\n // when used, this is modded so that it ranges from 0 to multi - 1\n std::vector<size_t> current_queue; \n\n\n // the number of CPUs\n size_t ncpus;\n // The queue to CPU ratio\n size_t multi;\n double min_priority; \n // the number of vertices in the graph\n size_t num_vertices;\n \n \n void set_options(const graphlab_options& opts);\n\n // Initializes the internal datastructures\n void initialize_data_structures();\n public:\n\n priority_scheduler(size_t num_vertices, const graphlab_options& opts);\n\n void set_num_vertices(const lvid_type numv);\n\n void schedule(const lvid_type vid, double priority = 1);\n\n /** Get the next element in the queue */\n sched_status::status_enum get_next(const size_t cpuid,\n lvid_type& ret_vid);\n\n bool empty();\n\n static void print_options_help(std::ostream& out) {\n out << \"\\t multi = [number of queues per thread. Default = 3].\\n\"\n << \"min_priority = [double, minimum priority required to receive \\n\"\n << \"\\t a message, default = -inf]\\n\";\n }\n\n\n }; \n\n\n} // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n#endif\n\n" }, { "path": "src/graphlab/scheduler/queued_fifo_scheduler.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <graphlab/scheduler/queued_fifo_scheduler.hpp>\n#include <graphlab/macros_def.hpp>\n\nnamespace graphlab {\n\nvoid queued_fifo_scheduler::set_options(const graphlab_options& opts) {\n // read the remaining options.\n std::vector<std::string> keys = opts.get_scheduler_args().get_option_keys();\n foreach(std::string opt, keys) {\n if (opt == \"queuesize\") {\n opts.get_scheduler_args().get_option(\"queuesize\", sub_queue_size);\n } else if (opt == \"multi\") {\n opts.get_scheduler_args().get_option(\"multi\", multi);\n } else {\n logstream(LOG_FATAL) << \"Unexpected Scheduler Option: \" << opt << std::endl;\n }\n }\n}\n\nvoid queued_fifo_scheduler::initialize_data_structures() {\n ASSERT_GT(ncpus * multi, 1);\n in_queues.resize(ncpus * multi);\n in_queue_locks.resize(ncpus * multi);\n out_queue_locks.resize(ncpus);\n out_queues.resize(ncpus);\n vertex_is_scheduled.resize(num_vertices);\n}\n\nqueued_fifo_scheduler::queued_fifo_scheduler(size_t num_vertices,\n const graphlab_options& opts) :\n ncpus(opts.get_ncpus()),\n num_vertices(num_vertices),\n multi(3),\n sub_queue_size(100) {\n ASSERT_GE(opts.get_ncpus(), 1);\n set_options(opts);\n initialize_data_structures();\n }\n\nvoid queued_fifo_scheduler::set_num_vertices(const lvid_type numv) {\n num_vertices = numv;\n vertex_is_scheduled.resize(numv);\n}\n\nvoid queued_fifo_scheduler::schedule(const lvid_type vid, double priority) {\n // If this is a new message, schedule it\n // the min priority will be taken care of by the get_next function\n if (vid < num_vertices && !vertex_is_scheduled.set_bit(vid)) {\n const size_t cpuid= \n random::fast_uniform(size_t(0), \n in_queues.size() - 1);\n in_queue_locks[cpuid].lock();\n queue_type& queue = in_queues[cpuid];\n queue.push_back(vid);\n if(queue.size() > sub_queue_size) {\n master_lock.lock();\n queue_type emptyq;\n master_queue.push_back(emptyq);\n master_queue.back().swap(queue);\n master_lock.unlock();\n }\n in_queue_locks[cpuid].unlock();\n } \n} // end of schedule\n\n/** Get the next element in the queue */\nsched_status::status_enum queued_fifo_scheduler::get_next(const size_t cpuid,\n lvid_type& ret_vid) {\n queue_type& myqueue = out_queues[cpuid];\n // if the local queue is empty try to get a queue from the master\n out_queue_locks[cpuid].lock();\n if(myqueue.empty()) {\n master_lock.lock();\n // if master queue is empty... \n if (!master_queue.empty()) {\n myqueue.swap(master_queue.front());\n master_queue.pop_front();\n master_lock.unlock();\n }\n else {\n master_lock.unlock();\n //try to steal from the inqueues\n for (size_t i = 0; i < in_queues.size(); ++i) {\n size_t idx = (i + multi * cpuid) % in_queues.size();\n if (!in_queues[idx].empty()) {\n in_queue_locks[idx].lock();\n // double check\n if(!in_queues[idx].empty()) {\n myqueue.swap(in_queues[idx]);\n }\n in_queue_locks[idx].unlock();\n if (!myqueue.empty()) break;\n } \n }\n }\n }\n // end of get next\n bool good = false;\n while(!myqueue.empty()) {\n // not empty, pop and verify\n ret_vid = myqueue.front();\n myqueue.pop_front();\n if (ret_vid < num_vertices) {\n good = vertex_is_scheduled.clear_bit(ret_vid);\n if (good) break;\n }\n }\n out_queue_locks[cpuid].unlock();\n\n if(good) {\n return sched_status::NEW_TASK;\n } else {\n return sched_status::EMPTY;\n }\n} // end of get_next_task\n\n\nbool queued_fifo_scheduler::empty() {\n for (size_t i = 0;i < out_queues.size(); ++i) {\n if (!out_queues[i].empty()) return false;\n }\n if (!master_queue.empty()) return false;\n for (size_t i = 0;i < in_queues.size(); ++i) {\n if (!in_queues[i].empty()) return false;\n }\n return true;\n}\n\n} // namespace graphlab\n" }, { "path": "src/graphlab/scheduler/queued_fifo_scheduler.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_QUEUED_FIFO_SCHEDULER_HPP\n#define GRAPHLAB_QUEUED_FIFO_SCHEDULER_HPP\n\n#include <algorithm>\n#include <queue>\n\n\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/parallel/pthread_tools.hpp>\n#include <graphlab/parallel/atomic.hpp>\n\n#include <graphlab/util/dense_bitset.hpp>\n\n#include <graphlab/util/random.hpp>\n#include <graphlab/scheduler/ischeduler.hpp>\n#include <graphlab/options/graphlab_options.hpp>\n\n\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n\n /**\n * \\ingroup group_schedulers\n *\n * This class defines a multiple queue approximate fifo scheduler.\n * Each processor has its own in_queue which it puts new tasks in\n * and out_queue which it pulls tasks from. Once a processors\n * in_queue gets too large, the entire queue is placed at the end of\n * the shared master queue. Once a processors out queue is empty it\n * grabs the next out_queue from the master.\n */\n class queued_fifo_scheduler: public ischeduler {\n \n public:\n\n typedef std::deque<lvid_type> queue_type;\n\n private:\n size_t ncpus;\n size_t num_vertices;\n size_t multi;\n dense_bitset vertex_is_scheduled;\n std::deque<queue_type> master_queue;\n mutex master_lock;\n size_t sub_queue_size;\n std::vector<queue_type> in_queues;\n std::vector<mutex> in_queue_locks;\n std::vector<queue_type> out_queues;\n std::vector<mutex> out_queue_locks;\n\n void set_options(const graphlab_options& opts);\n \n void initialize_data_structures();\n public:\n\n queued_fifo_scheduler(size_t num_vertices,\n const graphlab_options& opts); \n\n void set_num_vertices(const lvid_type numv);\n\n void schedule(const lvid_type vid, double priority = 1 /* ignored */);\n \n /** Get the next element in the queue */\n sched_status::status_enum get_next(const size_t cpuid,\n lvid_type& ret_vid);\n\n\n bool empty();\n\n /**\n * Print a help string describing the options that this scheduler\n * accepts.\n */\n static void print_options_help(std::ostream& out) {\n out << \"\\t queuesize: [the size at which a subqueue is \"\n << \"placed in the master queue. default = 100]\\n\";\n out << \"\\t multi = [number of queues per thread. Default = 3].\\n\";\n }\n\n\n };\n\n\n} // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n#endif\n\n" }, { "path": "src/graphlab/scheduler/scheduler_factory.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SCHEDULER_FACTORY_HPP\n#define GRAPHLAB_SCHEDULER_FACTORY_HPP\n\n#include <string>\n\n// Schedulers\n#include <graphlab/options/graphlab_options.hpp>\n#include <graphlab/scheduler/scheduler_list.hpp>\n\n#include <boost/preprocessor.hpp>\n\n\nnamespace graphlab {\n \n \n /**\n * helper for constructing graphlab engines.\n **/\n struct scheduler_factory {\n typedef ischeduler ischeduler_type;\n \n /**\n * Construct the a scheduler\n */\n template<typename Scheduler>\n static ischeduler_type* \n new_scheduler_impl(size_t num_vertices, const graphlab_options& opts) {\n ischeduler_type* scheduler_ptr = \n new Scheduler(num_vertices, opts);\n ASSERT_TRUE(scheduler_ptr != NULL);\n return scheduler_ptr;\n } // end of new_scheduler\n\n /**\n * This function returns a new scheduler for a particular engine\n */ \n static ischeduler_type* \n new_scheduler(size_t num_vertices, const graphlab_options& opts) {\n std::string scheduler_str = opts.get_scheduler_type();\n#define __GENERATE_NEW_SCHEDULER__(r_unused, data_unused, i, elem) \\\n BOOST_PP_EXPR_IF(i, else) \\\n if (scheduler_str == BOOST_PP_TUPLE_ELEM(3,0,elem)) { \\\n typedef BOOST_PP_TUPLE_ELEM(3,1,elem) \\\n scheduler_type; \\\n return new_scheduler_impl<scheduler_type> \\\n ( num_vertices, opts); \\\n } \n // generate the construction calls\n BOOST_PP_SEQ_FOR_EACH_I(__GENERATE_NEW_SCHEDULER__, _, __SCHEDULER_LIST__);\n#undef __GENERATE_NEW_SCHEDULER__ \n logstream(LOG_FATAL) \n << \"Invalid scheduler type: \" << scheduler_str << std::endl;\n return NULL;\n } // end of new_scheduler\n\n }; // end of class scheduler_factory\n\n}; // End of namespace graphlab\n\n\n\n\n#endif\n\n" }, { "path": "src/graphlab/scheduler/scheduler_includes.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n#ifndef GRAPHLAB_SCHEDULER_INCLUDES_HPP\n#define GRAPHLAB_SCHEDULER_INCLUDES_HPP\n\n#include <graphlab/scheduler/fifo_scheduler.hpp>\n#include <graphlab/scheduler/get_message_priority.hpp>\n#include <graphlab/scheduler/ischeduler.hpp>\n #include <graphlab/scheduler/priority_scheduler.hpp>\n#include <graphlab/scheduler/queued_fifo_scheduler.hpp>\n#include <graphlab/scheduler/scheduler_factory.hpp>\n#include <graphlab/scheduler/scheduler_list.hpp>\n#include <graphlab/scheduler/sweep_scheduler.hpp>\n#endif\n" }, { "path": "src/graphlab/scheduler/scheduler_list.cpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <algorithm>\n#include <graphlab/scheduler/scheduler_list.hpp>\n#include <graphlab/util/stl_util.hpp>\n\n\nnamespace graphlab {\n \n std::vector<std::string> get_scheduler_names() {\n std::vector<std::string> ret;\n#define __APPEND_TO_RET__(r_unused, data_unused, i, elem) \\\n ret.push_back(BOOST_PP_TUPLE_ELEM(3,0,elem));\n BOOST_PP_SEQ_FOR_EACH_I(__APPEND_TO_RET__, _, __SCHEDULER_LIST__)\n#undef __APPEND_TO_RET__\n return ret;\n }\n\n\n std::string get_scheduler_names_str() {\n std::string ret;\n std::vector<std::string> schednames;\n schednames = get_scheduler_names();\n for (size_t i = 0; i < schednames.size(); ++i) {\n if (i > 0) {\n ret = ret + \", \";\n }\n ret = ret + schednames[i];\n }\n return ret;\n }\n\n static std::string add_line_breaks(const std::string &s, size_t numcols) {\n size_t pos = 0;\n std::string ret;\n while(pos < s.length() - 1) {\n size_t oldpos = pos;\n pos = std::min(pos + numcols, s.length());\n \n size_t newpos = pos;\n // search backward for a space if we are not at the end of the\n // string\n if (pos < s.length()) {\n newpos = s.rfind(\" \", pos);\n }\n // if we get back to the old position, or we fail to find a\n // space, force the break\n if (newpos == std::string::npos || newpos == oldpos) {\n newpos = pos;\n }\n // break\n ret = ret + trim(s.substr(oldpos, newpos - oldpos)) + \"\\n\";\n pos = newpos;\n }\n return ret;\n }\n\n\n void print_scheduler_info(std::string s, std::ostream &out) {\n typedef char dummy_message_type; \n // this is annoying... I need to instantiate the graph<char, char> type to\n // even call the scheduler\n#define __GENERATE_SCHEDULER_HELP__(r_unused, data_unused, i, elem) \\\n BOOST_PP_EXPR_IF(i, else) if (s == BOOST_PP_TUPLE_ELEM(3,0,elem)) { \\\n out << \"\\n\"; \\\n out << BOOST_PP_TUPLE_ELEM(3,0,elem) << \" scheduler\\n\"; \\\n out << std::string(50, '-') << std::endl; \\\n out << add_line_breaks(BOOST_PP_TUPLE_ELEM(3,2,elem), 50) << \"\\n\" \\\n << \"Options: \\n\"; \\\n BOOST_PP_TUPLE_ELEM(3,1,elem) \\\n ::print_options_help(out); \\\n }\n /*\n * if (scheduler == \"sweep\") {\n * sweep_scheduler<graph<char,char> >::print_options_help(out);\n * }\n * ...\n */\n // generate the construction calls\n BOOST_PP_SEQ_FOR_EACH_I(__GENERATE_SCHEDULER_HELP__, _, __SCHEDULER_LIST__)\n else {\n out << \"Scheduler \" << s << \" not found\" << \"\\n\";\n }\n#undef __GENERATE_SCHEDULER_HELP__\n } // end of print scheduler info\n\n\n} // end of namespace graphlab\n\n\n\n" }, { "path": "src/graphlab/scheduler/scheduler_list.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SCHEDULER_LIST_HPP\n#define GRAPHLAB_SCHEDULER_LIST_HPP\n#include <string>\n#include <vector>\n#include <iostream>\n#include <boost/preprocessor.hpp>\n\n#define __SCHEDULER_LIST__ \\\n ((\"fifo\", fifo_scheduler, \\\n \"Standard FIFO task queue, poor parallelism, but task evaluation \" \\\n \"sequence is highly predictable. \" \\\n \"Useful for debugging and testing.\")) \\\n ((\"sweep\", sweep_scheduler, \\\n \"very fast dynamic scheduler. Scans all vertices in sequence, \" \\\n \"running all update tasks on each vertex evaluated.\")) \\\n ((\"priority\", priority_scheduler, \\\n \"Standard Priority queue, poor parallelism, but task evaluation \" \\\n \"sequence is highly predictable. Useful for debugging\")) \\\n ((\"queued_fifo\", queued_fifo_scheduler, \\\n \"This scheduler maintains a shared FIFO queue of FIFO queues. \" \\\n \"Each thread maintains its own smaller in and out queues. When a \" \\\n \"threads out queue is too large (greater than \\\"queuesize\\\") then \" \\\n \"the thread puts its out queue at the end of the master queue.\")) \n\n#include <graphlab/scheduler/fifo_scheduler.hpp>\n#include <graphlab/scheduler/sweep_scheduler.hpp>\n#include <graphlab/scheduler/priority_scheduler.hpp>\n#include <graphlab/scheduler/queued_fifo_scheduler.hpp>\n\n\nnamespace graphlab {\n /// get all the scheduler names\n std::vector<std::string> get_scheduler_names();\n\n /// get all the scheduler names concated into a string\n std::string get_scheduler_names_str();\n\n /// Display the scheduler options for a particular scheduler\n void print_scheduler_info(std::string s, std::ostream &out);\n}\n\n#endif\n\n" }, { "path": "src/graphlab/scheduler/sweep_scheduler.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include <graphlab/scheduler/sweep_scheduler.hpp>\n#include <graphlab/macros_def.hpp>\nnamespace graphlab {\n\nvoid sweep_scheduler::set_options(const graphlab_options& opts) {\n std::vector<std::string> keys = opts.get_scheduler_args().get_option_keys();\n bool max_iter_set = false;\n foreach(std::string opt, keys) {\n if (opt == \"order\") {\n opts.get_scheduler_args().get_option(\"order\", ordering);\n ASSERT_TRUE(ordering == \"random\" || ordering == \"ascending\");\n } else if (opt == \"strict\") {\n opts.get_scheduler_args().get_option(\"strict\", strict_round_robin);\n } else if (opt == \"max_iterations\") {\n opts.get_scheduler_args().get_option(\"max_iterations\", max_iterations);\n max_iter_set = true;\n } else {\n logstream(LOG_FATAL) << \"Unexpected Scheduler Option: \" << opt << std::endl;\n }\n }\n\n if (max_iter_set) {\n ASSERT_MSG(strict_round_robin, \n \"sweep_scheduler: \\\"strict\\\" must be set with \\\"max_iteration\\\"\");\n }\n}\n\nsweep_scheduler::sweep_scheduler(size_t num_vertices,\n const graphlab_options& opts) :\n ncpus(opts.get_ncpus()),\n num_vertices(num_vertices),\n strict_round_robin(true),\n max_iterations(std::numeric_limits<size_t>::max()),\n vertex_is_scheduled(num_vertices) {\n // initialize defaults\n ASSERT_GE(opts.get_ncpus(), 1);\n ordering = \"random\";\n set_options(opts);\n\n if (ordering == \"ascending\") {\n randomizer = 1;\n } else if(ordering == \"random\") {\n randomizer = 1500450271;\n }\n\n if(strict_round_robin) {\n logstream(LOG_INFO)\n << \"Using a strict round robin schedule.\" << std::endl;\n // Max iterations only applies to strict round robin\n if(max_iterations != std::numeric_limits<size_t>::max()) {\n logstream(LOG_INFO)\n << \"Using maximum iterations: \" << max_iterations << std::endl;\n }\n rr_index = 0;\n } else {\n // each cpu is responsible for its own subset of vertices\n // Initialize the cpu2index counters\n cpu2index.resize(ncpus);\n for(size_t i = 0; i < cpu2index.size(); ++i) cpu2index[i] = i;\n }\n vertex_is_scheduled.resize(num_vertices);\n} // end of constructor\n\n\nvoid sweep_scheduler::set_num_vertices(const lvid_type numv) {\n num_vertices = numv;\n vertex_is_scheduled.resize(numv);\n}\n\nvoid sweep_scheduler::schedule(const lvid_type vid, double priority) { \n if (vid < num_vertices) vertex_is_scheduled.set_bit(vid);\n} \n\n\nsched_status::status_enum sweep_scheduler::get_next(const size_t cpuid,\n lvid_type& ret_vid) { \n const size_t max_fails = (num_vertices/ncpus) + 1;\n // Check to see if max iterations have been achieved \n if(strict_round_robin && (rr_index / num_vertices) >= max_iterations) \n return sched_status::EMPTY;\n // Loop through all vertices that are associated with this\n // processor searching for a vertex with an active task\n for(size_t idx = get_and_inc_index(cpuid), fails = 0; \n fails <= max_fails; // \n idx = get_and_inc_index(cpuid), ++fails) {\n // It is possible that the get_and_inc_index could return an\n // invalid index if the number of cpus exceeds the number of\n // vertices. In This case we alwasy return empty\n if(__builtin_expect(idx >= num_vertices, false)) return sched_status::EMPTY;\n const lvid_type vid = (idx * randomizer) % num_vertices;\n bool success = vertex_is_scheduled.clear_bit(vid);\n while(success) { // Job found now decide whether to keep it\n ret_vid = vid; \n return sched_status::NEW_TASK;\n }\n } // end of for loop\n return sched_status::EMPTY;\n} // end of get_next\n\n\n}\n" }, { "path": "src/graphlab/scheduler/sweep_scheduler.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_SWEEP_SCHEDULER_HPP\n#define GRAPHLAB_SWEEP_SCHEDULER_HPP\n\n#include <queue>\n#include <cmath>\n#include <cassert>\n\n#include <graphlab/graph/graph_basic_types.hpp>\n#include <graphlab/scheduler/ischeduler.hpp>\n#include <graphlab/util/dense_bitset.hpp>\n#include <graphlab/parallel/atomic.hpp>\n#include <graphlab/options/graphlab_options.hpp>\n\n#include <graphlab/macros_def.hpp>\n\nnamespace graphlab {\n\n /** \\ingroup group_schedulers\n */\n class sweep_scheduler: public ischeduler {\n private:\n\n size_t ncpus;\n\n size_t num_vertices;\n bool strict_round_robin;\n atomic<size_t> rr_index;\n size_t max_iterations;\n size_t randomizer;\n\n std::vector<lvid_type> cpu2index;\n\n dense_bitset vertex_is_scheduled;\n std::string ordering;\n\n void set_options(const graphlab_options& opts);\n\n public:\n sweep_scheduler(size_t num_vertices,\n const graphlab_options& opts);\n\n void set_num_vertices(const lvid_type numv);\n\n void schedule(const lvid_type vid, double priority = 1 /* ignored */) ; \n\n\n \n sched_status::status_enum get_next(const size_t cpuid, lvid_type& ret_vid);\n \n \n static void print_options_help(std::ostream &out) {\n out << \"order = [string: {random, ascending} default=random]\\n\"\n << \"strict = [bool, use strict round robin schedule, default=true]\\n\"\n << \"max_iterations = [integer, maximum number of iterations \"\n << \" (requires strict=true) \\n\"\n << \"\\t default = inf]\\n\";\n } // end of print_options_help\n\n\n bool empty() {\n return (vertex_is_scheduled.popcount() == 0);\n }\n\n private:\n inline size_t get_and_inc_index(const size_t cpuid) {\n if (strict_round_robin) { \n return rr_index++ % num_vertices; \n } else {\n const size_t index = cpu2index[cpuid];\n cpu2index[cpuid] += ncpus;\n // Address loop around\n if (__builtin_expect(cpu2index[cpuid] >= num_vertices, false)) \n cpu2index[cpuid] = cpuid;\n return index;\n }\n }// end of next index\n\n };\n\n\n} // end of namespace graphlab\n#include <graphlab/macros_undef.hpp>\n\n#endif\n\n" }, { "path": "src/graphlab/serialization/CMakeLists.txt", "content": "project(GraphLab)\n\n" }, { "path": "src/graphlab/serialization/basic_types.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n/*\n This files defines the serializer/deserializer for all basic types\n (as well as string and pair) \n*/\n#ifndef ARCHIVE_BASIC_TYPES_HPP\n#define ARCHIVE_BASIC_TYPES_HPP\n\n#include <string>\n#include <graphlab/serialization/serializable_pod.hpp>\n#include <graphlab/logger/assertions.hpp>\n#include <stdint.h>\n\nnamespace graphlab {\n class oarchive;\n class iarchive;\n}\n\n\nnamespace graphlab {\n namespace archive_detail {\n\n /** Serialization of null terminated const char* strings.\n * This is necessary to serialize constant strings like\n * \\code \n * oarc << \"hello world\";\n * \\endcode\n */\n template <typename OutArcType>\n struct serialize_impl<OutArcType, const char*, false> {\n static void exec(OutArcType& oarc, const char* const& s) {\n // save the length\n // ++ for the \\0\n size_t length = strlen(s); length++;\n oarc << length;\n oarc.write(reinterpret_cast<const char*>(s), length);\n DASSERT_FALSE(oarc.fail());\n }\n };\n\n\n\n\n /// Serialization of fixed length char arrays\n template <typename OutArcType, size_t len>\n struct serialize_impl<OutArcType, char [len], false> {\n static void exec(OutArcType& oarc, const char s[len] ) { \n size_t length = len;\n oarc << length;\n oarc.write(reinterpret_cast<const char*>(s), length);\n DASSERT_FALSE(oarc.fail());\n }\n };\n\n\n /// Serialization of null terminated char* strings\n template <typename OutArcType>\n struct serialize_impl<OutArcType, char*, false> {\n static void exec(OutArcType& oarc, char* const& s) {\n // save the length\n // ++ for the \\0\n size_t length = strlen(s); length++;\n oarc << length;\n oarc.write(reinterpret_cast<const char*>(s), length);\n DASSERT_FALSE(oarc.fail());\n }\n };\n\n /// Deserialization of null terminated char* strings\n template <typename InArcType>\n struct deserialize_impl<InArcType, char*, false> {\n static void exec(InArcType& iarc, char*& s) {\n // Save the length and check if lengths match\n size_t length;\n iarc >> length;\n s = new char[length];\n //operator>> the rest\n iarc.read(reinterpret_cast<char*>(s), length);\n DASSERT_FALSE(iarc.fail());\n }\n };\n \n /// Deserialization of fixed length char arrays \n template <typename InArcType, size_t len>\n struct deserialize_impl<InArcType, char [len], false> {\n static void exec(InArcType& iarc, char s[len]) { \n size_t length;\n iarc >> length;\n ASSERT_LE(length, len);\n iarc.read(reinterpret_cast<char*>(s), length);\n DASSERT_FALSE(iarc.fail());\n }\n };\n\n\n\n /// Serialization of std::string\n template <typename OutArcType>\n struct serialize_impl<OutArcType, std::string, false> {\n static void exec(OutArcType& oarc, const std::string& s) {\n size_t length = s.length();\n oarc << length;\n oarc.write(reinterpret_cast<const char*>(s.c_str()), \n (std::streamsize)length);\n DASSERT_FALSE(oarc.fail());\n }\n };\n\n\n /// Deserialization of std::string\n template <typename InArcType>\n struct deserialize_impl<InArcType, std::string, false> {\n static void exec(InArcType& iarc, std::string& s) {\n //read the length\n size_t length;\n iarc >> length;\n //resize the string and read the characters\n s.resize(length);\n iarc.read(const_cast<char*>(s.c_str()), (std::streamsize)length);\n DASSERT_FALSE(iarc.fail());\n }\n };\n\n /// Serialization of std::pair\n template <typename OutArcType, typename T, typename U>\n struct serialize_impl<OutArcType, std::pair<T, U>, false > {\n static void exec(OutArcType& oarc, const std::pair<T, U>& s) {\n oarc << s.first << s.second;\n }\n };\n\n\n /// Deserialization of std::pair\n template <typename InArcType, typename T, typename U>\n struct deserialize_impl<InArcType, std::pair<T, U>, false > {\n static void exec(InArcType& iarc, std::pair<T, U>& s) {\n iarc >> s.first >> s.second;\n }\n };\n\n\n\n /** Serialization of 8 byte wide integers\n * \\code \n * oarc << vec.length();\n * \\endcode\n */\n template <typename OutArcType>\n struct serialize_impl<OutArcType, unsigned long , true> {\n static void exec(OutArcType& oarc, const unsigned long & s) {\n // only bottom 1 byte\n if ((s >> 8) == 0) {\n unsigned char c = 0;\n unsigned char trunc_s = s;\n oarc.direct_assign(c);\n oarc.direct_assign(trunc_s);\n }\n // only bottom 2 byte\n else if ((s >> 16) == 0) {\n unsigned char c = 1;\n unsigned short trunc_s = s;\n oarc.direct_assign(c);\n oarc.direct_assign(trunc_s);\n }\n // only bottom 4 byte\n else if ((s >> 32) == 0) {\n unsigned char c = 2;\n uint32_t trunc_s = s;\n oarc.direct_assign(c);\n oarc.direct_assign(trunc_s);\n } \n else {\n unsigned char c = 3;\n oarc.direct_assign(c);\n oarc.direct_assign(s);\n }\n }\n };\n\n\n /// Deserialization of 8 byte wide integer \n template <typename InArcType>\n struct deserialize_impl<InArcType, unsigned long , true> {\n static void exec(InArcType& iarc, unsigned long & s) {\n unsigned char c;\n iarc.read(reinterpret_cast<char*>(&c), 1);\n switch(c) {\n case 0: {\n unsigned char val;\n iarc.read(reinterpret_cast<char*>(&val), 1);\n s = val;\n break;\n }\n case 1: {\n unsigned short val;\n iarc.read(reinterpret_cast<char*>(&val), 2);\n s = val;\n break;\n }\n case 2: {\n uint32_t val;\n iarc.read(reinterpret_cast<char*>(&val), 4);\n s = val;\n break;\n }\n case 3: {\n iarc.read(reinterpret_cast<char*>(&s), 8);\n break;\n }\n default:\n ASSERT_LE(c, 3);\n };\n }\n };\n\n\n\n } // namespace archive_detail\n} // namespace graphlab\n \n#undef INT_SERIALIZE\n#endif\n\n" }, { "path": "src/graphlab/serialization/conditional_serialize.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SERIALIZATION_CONDITIONAL_SERIALIZE_HPP\n#define GRAPHLAB_SERIALIZATION_CONDITIONAL_SERIALIZE_HPP\n#include <graphlab/serialization/oarchive.hpp>\n#include <graphlab/serialization/iarchive.hpp>\nnamespace graphlab {\n\ntemplate <typename T>\nstruct conditional_serialize {\n bool hasval;\n T val;\n\n conditional_serialize(): hasval(false) { }\n conditional_serialize(T& val): hasval(true), val(val) { }\n\n conditional_serialize(const conditional_serialize& cs): hasval(cs.hasval), val(cs.val) { }\n conditional_serialize& operator=(const conditional_serialize& cs) {\n hasval = cs.hasval;\n val = cs.val;\n return (*this);\n }\n void save(oarchive& oarc) const {\n oarc << hasval;\n if (hasval) oarc << val;\n }\n\n void load(iarchive& iarc) {\n iarc >> hasval;\n if (hasval) iarc >> val;\n }\n};\n\n};\n\n#endif\n" }, { "path": "src/graphlab/serialization/has_load.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_HAS_LOAD_HPP\n#define GRAPHLAB_HAS_LOAD_HPP\n\n#include <typeinfo>\n\n\nnamespace graphlab {\n namespace archive_detail {\n\n /** SFINAE method to detect if a class T \n * implements a function void T::load(ArcType&)\n * \n * If T implements the method, has_load_method<ArcType,T>::value will be \n * true. Otherwise it will be false\n */\n template<typename ArcType, typename T>\n struct has_load_method\n {\n template<typename U, void (U::*)(ArcType&)> struct SFINAE {};\n template<typename U> static char Test(SFINAE<U, &U::load>*);\n template<typename U> static int Test(...);\n static const bool value = sizeof(Test<T>(0)) == sizeof(char);\n };\n\n /**\n * load_or_fail<ArcType, T>(arc, t)\n * will call this version of the function if\n * T implements void T::load(ArcType&).\n *\n * load_or_fail<ArcType, T>(arc, t) will therefore load the class successfully\n * if T implements the load function correctly. Otherwise, calling \n * load_or_fail will print an error message.\n */\n template <typename ArcType, typename ValueType>\n typename boost::enable_if_c<has_load_method<ArcType, ValueType>::value, void>::type \n load_or_fail(ArcType& o, ValueType &t) { \n t.load(o);\n }\n\n /**\n * load_or_fail<ArcType, T>(arc, t)\n * will call this version of the function if\n *\n * load_or_fail<ArcType, T>(arc, t) will therefore load the class successfully\n * if T implements the load function correctly. Otherwise, calling \n * load_or_fail will print an error message.\n * T does not implement void T::load(ArcType&).\n */\n template <typename ArcType, typename ValueType>\n typename boost::disable_if_c<has_load_method<ArcType, ValueType>::value, void>::type \n load_or_fail(ArcType& o, ValueType &t) { \n ASSERT_MSG(false, \"Trying to deserializable type %s without valid load method.\", typeid(ValueType).name()); \n }\n \n } // archive_detail\n} // graphlab\n\n#endif\n\n" }, { "path": "src/graphlab/serialization/has_save.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n\n#ifndef HAS_SAVE_HPP\n#define HAS_SAVE_HPP\n#include <typeinfo>\n\nnamespace graphlab {\nnamespace archive_detail {\n\n /** SFINAE method to detect if a class T \n * implements a function void T::save(ArcType&) const\n * \n * If T implements the method, has_save_method<ArcType,T>::value will be \n * true. Otherwise it will be false\n */\n template<typename ArcType, typename T>\n struct has_save_method\n {\n\t template<typename U, void (U::*)(ArcType&) const> struct SFINAE {};\n\t template<typename U> static char Test(SFINAE<U, &U::save>*);\n\t template<typename U> static int Test(...);\n\t static const bool value = sizeof(Test<T>(0)) == sizeof(char);\n };\n\n /**\n * save_or_fail<ArcType, T>(arc, t)\n * will call this version of the function if\n * T implements void T::save(ArcType&) const.\n * \n * save_or_fail<ArcType, T>(arc, t) will therefore save the class successfully\n * if T implements the save function correctly. Otherwise, calling \n * save_or_fail will print an error message.\n */\n template <typename ArcType, typename ValueType>\n typename boost::enable_if_c<has_save_method<ArcType, ValueType>::value, void>::type \n save_or_fail(ArcType& o, const ValueType &t) { \n t.save(o);\n }\n \n /**\n * save_or_fail<ArcType, T>(arc, t)\n * will call this version of the function if\n * \n * save_or_fail<ArcType, T>(arc, t) will therefore save the class successfully\n * if T implements the save function correctly. Otherwise, calling \n * save_or_fail will print an error message.\n * T does not implement void T::save(ArcType&) const.\n */\n template <typename ArcType, typename ValueType>\n typename boost::disable_if_c<has_save_method<ArcType, ValueType>::value, void>::type \n save_or_fail(ArcType& o, const ValueType &t) { \n ASSERT_MSG(false,\"Trying to serializable type %s without valid save method.\", typeid(ValueType).name()); \n }\n \n} // archive_detail\n} // graphlab\n\n#endif\n\n" }, { "path": "src/graphlab/serialization/iarchive.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SERIALIZE_HPP\n#include <graphlab/serialization/serialize.hpp>\n\n#else\n\n\n#ifndef GRAPHLAB_IARCHIVE_HPP\n#define GRAPHLAB_IARCHIVE_HPP\n\n#include <iostream>\n#include <graphlab/logger/assertions.hpp>\n#include <graphlab/serialization/is_pod.hpp>\n#include <graphlab/serialization/has_load.hpp>\nnamespace graphlab {\n\n /**\n * \\ingroup group_serialization\n * \\brief The serialization input archive object which, provided\n * with a reference to an istream, will read from the istream,\n * providing deserialization capabilities.\n *\n * Given a source of serialized bytes (written by an graphlab::oarchive),\n * in the form of a standard input stream, you can construct an iarchive\n * object by:\n * \\code\n * // where strm is an istream object\n * graphlab::iarchive iarc(strm);\n * \\endcode\n *\n * For instance, to deserialize from a file,\n * \\code\n * std::ifstream fin(\"inputfile.bin\");\n * graphlab::iarchive iarc(fin);\n * \\endcode\n *\n * Once the iarc object is constructed, \\ref sec_serializable\n * objects can be read from it using the >> stream operator.\n *\n * \\code\n * iarc >> a >> b >> c;\n * \\endcode\n *\n * Alternatively, data can be directly read from the stream using\n * the iarchive::read() and iarchive::read_char() functions.\n *\n * For more usage details, see \\ref serialization\n *\n * The iarchive object should not be used once the associated stream\n * object is closed or is destroyed.\n *\n * To use this class, include\n * graphlab/serialization/serialization_includes.hpp\n */\n class iarchive {\n public:\n std::istream* in;\n const char* buf;\n size_t off;\n size_t len;\n\n /// Directly reads a single character from the input stream\n inline char read_char() {\n char c;\n if (buf) {\n c = buf[off];\n ++off;\n } else {\n in->get(c);\n }\n return c;\n }\n\n /**\n * Directly reads a sequence of \"len\" bytes from the\n * input stream into the location pointed to by \"c\"\n */\n inline void read(char* c, size_t l) {\n if (buf) {\n memcpy(c, buf + off, l);\n off += l;\n } else {\n in->read(c, l);\n }\n }\n\n\n /// Returns true if the underlying stream is in a failure state\n inline bool fail() {\n return in == NULL ? off > len : in->fail();\n }\n\n /**\n * Constructs an iarchive object.\n * Takes a reference to a generic std::istream object and associates\n * the archive with it. Reads from the archive will read from the\n * assiciated input stream.\n */\n inline iarchive(std::istream& instream)\n : in(&instream), buf(NULL), off(0), len(0) { }\n\n inline iarchive(const char* buf, size_t len)\n : in(NULL), buf(buf), off(0), len(len) { }\n\n ~iarchive() {}\n };\n\n\n /**\n * \\ingroup group_serialization\n * \\brief\n * When this archive is used to deserialize an object,\n * and the object does not support serialization,\n * failure will only occur at runtime. Otherwise equivalent to\n * graphlab::iarchive.\n */\n class iarchive_soft_fail{\n public:\n\n iarchive *iarc;\n bool mine;\n\n /// Directly reads a single character from the input stream\n inline char read_char() {\n return iarc->read_char();\n }\n\n /**\n * Directly reads a sequence of \"len\" bytes from the\n * input stream into the location pointed to by \"c\"\n */\n inline void read(char* c, size_t len) {\n iarc->read(c, len);\n }\n\n /// Returns true if the underlying stream is in a failure state\n inline bool fail() {\n return iarc->fail();\n }\n\n /**\n * Constructs an iarchive_soft_fail object.\n * Takes a reference to a generic std::istream object and associates\n * the archive with it. Reads from the archive will read from the\n * assiciated input stream.\n */\n inline iarchive_soft_fail(std::istream &instream)\n : iarc(new iarchive(instream)), mine(true) {}\n\n /**\n * Constructs an iarchive_soft_fail object from an iarchive.\n * Both will share the same input stream\n */\n inline iarchive_soft_fail(iarchive &iarc)\n : iarc(&iarc), mine(false) {}\n\n inline ~iarchive_soft_fail() { if (mine) delete iarc; }\n };\n\n\n namespace archive_detail {\n\n /// called by the regular archive The regular archive will do a hard fail\n template <typename InArcType, typename T>\n struct deserialize_hard_or_soft_fail {\n inline static void exec(InArcType& iarc, T& t) {\n t.load(iarc);\n }\n };\n\n /// called by the soft fail archive\n template <typename T>\n struct deserialize_hard_or_soft_fail<iarchive_soft_fail, T> {\n inline static void exec(iarchive_soft_fail& iarc, T& t) {\n load_or_fail(*(iarc.iarc), t);\n }\n };\n\n\n /**\n Implementation of the deserializer for different types. This is the\n catch-all. If it gets here, it must be a non-POD and is a class. We\n therefore call the .save function. Here we pick between the archive\n types using serialize_hard_or_soft_fail\n */\n template <typename InArcType, typename T, bool IsPOD>\n struct deserialize_impl {\n inline static void exec(InArcType& iarc, T& t) {\n deserialize_hard_or_soft_fail<InArcType, T>::exec(iarc, t);\n }\n };\n\n // catch if type is a POD\n template <typename InArcType, typename T>\n struct deserialize_impl<InArcType, T, true>{\n inline static void exec(InArcType& iarc, T &t) {\n iarc.read(reinterpret_cast<char*>(&t),\n sizeof(T));\n }\n };\n\n } //namespace archive_detail\n\n /// \\cond GRAPHLAB_INTERNAL\n\n /**\n Allows Use of the \"stream\" syntax for serialization\n */\n template <typename T>\n inline iarchive& operator>>(iarchive& iarc, T &t) {\n archive_detail::deserialize_impl<iarchive,\n T,\n gl_is_pod<T>::value >::exec(iarc, t);\n return iarc;\n }\n\n\n\n /**\n Allows Use of the \"stream\" syntax for serialization\n */\n template <typename T>\n inline iarchive_soft_fail& operator>>(iarchive_soft_fail& iarc, T &t) {\n archive_detail::deserialize_impl<iarchive_soft_fail,\n T,\n gl_is_pod<T>::value >::exec(iarc, t);\n return iarc;\n }\n\n\n /**\n deserializes an arbitrary pointer + length from an archive\n */\n inline iarchive& deserialize(iarchive& iarc,\n void* str,\n const size_t length) {\n iarc.read(reinterpret_cast<char*>(str), (std::streamsize)length);\n assert(!iarc.fail());\n return iarc;\n }\n\n\n\n /**\n deserializes an arbitrary pointer + length from an archive\n */\n inline iarchive_soft_fail& deserialize(iarchive_soft_fail& iarc,\n void* str,\n const size_t length) {\n iarc.read(reinterpret_cast<char*>(str), (std::streamsize)length);\n assert(!iarc.fail());\n return iarc;\n }\n\n /// \\endcond GRAPHLAB_INTERNAL\n\n /**\n \\ingroup group_serialization\n\n \\brief Macro to make it easy to define out-of-place loads\n\n In the event that it is impractical to implement a save() and load()\n function in the class one wnats to serialize, it is necessary to define\n an \"out of save\" save and load.\n\n See \\ref sec_serializable_out_of_place for an example\n\n \\note important! this must be defined in the global namespace!\n */\n#define BEGIN_OUT_OF_PLACE_LOAD(arc, tname, tval) \\\n namespace graphlab{ namespace archive_detail { \\\n template <typename InArcType> \\\n struct deserialize_impl<InArcType, tname, false>{ \\\n static void exec(InArcType& arc, tname & tval) {\n\n#define END_OUT_OF_PLACE_LOAD() } }; } }\n\n\n\n\n} // namespace graphlab\n\n\n#endif\n\n#endif\n\n\n\n\n\n\n\n\n" }, { "path": "src/graphlab/serialization/is_pod.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_IS_POD_HPP\n#define GRAPHLAB_IS_POD_HPP\n#include <boost/type_traits.hpp>\n\nnamespace graphlab {\n\n /** \\ingroup group_serialization\n \\brief Inheriting from this type will force the serializer\n to treat the derived type as a POD type.\n */\n struct IS_POD_TYPE { };\n\n /**\n * \\ingroup group_serialization\n *\n * \\brief Tests if T is a POD type\n *\n * gl_is_pod<T>::value is true if T is a POD type (as determined by\n * boost::is_pod) or if T inherits from IS_POD_TYPE. gl_is_pod<T>::value\n * is false otherwise.\n */\n template <typename T>\n struct gl_is_pod{\n // it is a pod and is not an integer since we have special handlings for integers\n\n BOOST_STATIC_CONSTANT(bool, value = (boost::type_traits::ice_or<\n boost::is_scalar<T>::value,\n boost::is_base_of<IS_POD_TYPE, T>::value\n >::value));\n\n // standard POD detection is no good because things which contain pointers\n // are POD, but are not serializable\n // (T is POD and T is not an integer of size >= 2)\n /*BOOST_STATIC_CONSTANT(bool, value =\n (\n boost::type_traits::ice_and<\n boost::is_pod<T>::value,\n boost::type_traits::ice_not<\n boost::type_traits::ice_and<\n boost::is_integral<T>::value,\n sizeof(T) >= 2\n >::value\n >::value\n >::value\n ));*/\n\n };\n \n /// \\internal\n\n template <typename T>\n struct gl_is_pod_or_scaler{\n BOOST_STATIC_CONSTANT(bool, value =\n (\n boost::type_traits::ice_or<\n boost::is_scalar<T>::value,\n gl_is_pod<T>::value>::value\n ));\n };\n}\n\n#endif\n\n\n\n" }, { "path": "src/graphlab/serialization/iterator.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SERIALIZE_ITERATOR_HPP\n#define GRAPHLAB_SERIALIZE_ITERATOR_HPP\n\n#include <iterator>\n#include <graphlab/serialization/oarchive.hpp>\n#include <graphlab/serialization/iarchive.hpp>\n\nnamespace graphlab {\n\n /**\n * \\ingroup group_serialization\n \\brief Serializes the contents between the iterators begin and end.\n\n This function prefers random access iterators since it needs\n a distance between the begin and end iterator.\n This function as implemented will work for other input iterators\n but is extremely inefficient.\n \n \\tparam OutArcType The output archive type. This should not need to be\n specified. The compiler will typically infer this\n correctly.\n \\tparam RandomAccessIterator The iterator type. This should not need to be\n specified. The compiler will typically infer this\n correctly.\n \n \\param oarc A reference to the output archive to write to.\n \\param begin The start of the iterator range to write.\n \\param end The end of the iterator range to write.\n */\n template <typename OutArcType, typename RandomAccessIterator>\n void serialize_iterator(OutArcType& oarc, RandomAccessIterator begin,\n RandomAccessIterator end){\n const size_t vsize = std::distance(begin, end);\n oarc << vsize;\n // store each element\n for(; begin != end; ++begin) oarc << *begin;\n }\n\n\n /**\n \\ingroup group_serialization\n \\brief Serializes the contents between the iterators begin and end.\n\n This functions takes all iterator types, but takes a \"count\" for\n efficiency. This count is checked and will return failure if the number\n of elements serialized does not match the count\n \n \\tparam OutArcType The output archive type. This should not need to be\n specified. The compiler will typically infer this\n correctly.\n \\tparam InputIterator The iterator type. This should not need to be\n specified. The compiler will typically infer this\n correctly.\n\n \\param oarc A reference to the output archive to write to.\n \\param begin The start of the iterator range to write.\n \\param end The end of the iterator range to write.\n \\param vsize The distance between the iterators begin and end. Must match\n std::distance(begin, end);\n */\n template <typename OutArcType, typename InputIterator>\n void serialize_iterator(OutArcType& oarc, InputIterator begin,\n InputIterator end, size_t vsize){\n oarc << vsize;\n //store each element\n size_t count = 0;\n for(; begin != end; ++begin) { oarc << *begin; ++count; }\n // fail if count does not match\n ASSERT_EQ(count, vsize);\n }\n\n /**\n \\ingroup group_serialization\n \\brief The accompanying function to serialize_iterator()\n Reads elements from the stream and writes it to the output iterator.\n \n Note that this requires an additional template parameter T which is the\n \"type of object to deserialize\"\n This is necessary for instance for the map type. The \n <code>map<T,U>::value_type</code>\n is <code>pair<const T,U></code>which is not useful since I cannot assign to\n it. In this case, <code>T=pair<T,U></code>\n\n \\tparam OutArcType The output archive type. \n \\tparam T The type of values to deserialize\n \\tparam OutputIterator The type of the output iterator to be written to.\n This should not need to be specified. The compiler\n will typically infer this correctly.\n\n \\param iarc A reference to the input archive\n \\param result The output iterator to write to\n\n */\n template <typename InArcType, typename T, typename OutputIterator>\n void deserialize_iterator(InArcType& iarc, OutputIterator result) {\n // get the number of elements to deserialize\n size_t length = 0;\n iarc >> length;\n \n // iterate through and send to the output iterator\n for (size_t x = 0; x < length ; ++x){\n /**\n * A compiler error on this line means that one of the user\n * defined types currently trying to be serialized (e.g.,\n * vertex_data, edge_data, messages, gather_type, or\n * vertex_programs) does not have a default constructor.\n */\n T v;\n iarc >> v;\n (*result) = v;\n result++;\n }\n }\n \n \n} \n#endif \n\n" }, { "path": "src/graphlab/serialization/list.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SERIALIZE_LIST_HPP\n#define GRAPHLAB_SERIALIZE_LIST_HPP\n\n#include <list>\n\n#include <graphlab/serialization/iarchive.hpp>\n#include <graphlab/serialization/oarchive.hpp>\n#include <graphlab/serialization/iterator.hpp>\n\n\nnamespace graphlab {\nnamespace archive_detail {\n /** serializes a list */\n template <typename OutArcType, typename T>\n struct serialize_impl<OutArcType, std::list<T>, false > {\n static void exec(OutArcType& oarc, const std::list<T>& vec){\n serialize_iterator(oarc,vec.begin(),vec.end(), vec.size());\n }\n };\n\n /** deserializes a list */\n template <typename InArcType, typename T>\n struct deserialize_impl<InArcType, std::list<T>, false > {\n static void exec(InArcType& iarc, std::list<T>& vec){\n vec.clear();\n deserialize_iterator<T>(iarc, std::inserter(vec,vec.end()));\n }\n };\n} // archive_detail \n} // graphlab\n#endif \n\n" }, { "path": "src/graphlab/serialization/map.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SERIALIZE_MAP_HPP\n#define GRAPHLAB_SERIALIZE_MAP_HPP\n\n#include <map>\n#include <graphlab/serialization/iarchive.hpp>\n#include <graphlab/serialization/oarchive.hpp>\n#include <graphlab/serialization/iterator.hpp>\n\nnamespace graphlab {\n\nnamespace archive_detail {\n /** Serializes a map */\n template <typename OutArcType, typename T, typename U>\n struct serialize_impl<OutArcType, std::map<T,U>, false > {\n static void exec(OutArcType& oarc, const std::map<T,U>& vec){\n serialize_iterator(oarc,\n vec.begin(), vec.end(), vec.size());\n }\n };\n\n /** deserializes a map */\n \n template <typename InArcType, typename T, typename U>\n struct deserialize_impl<InArcType, std::map<T,U>, false > {\n static void exec(InArcType& iarc, std::map<T,U>& vec){\n vec.clear();\n deserialize_iterator<InArcType, \n std::pair<T,U> >(iarc, \n std::inserter(vec,vec.end()));\n }\n };\n\n} // archive_detail \n} // graphlab\n#endif \n\n" }, { "path": "src/graphlab/serialization/oarchive.hpp", "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n// This file should not be included directly. use serialize.hpp\n#ifndef GRAPHLAB_SERIALIZE_HPP\n#include <graphlab/serialization/serialize.hpp>\n\n#else\n\n#ifndef GRAPHLAB_OARCHIVE_HPP\n#define GRAPHLAB_OARCHIVE_HPP\n\n#include <iostream>\n#include <string>\n#include <graphlab/logger/assertions.hpp>\n#include <graphlab/serialization/is_pod.hpp>\n#include <graphlab/serialization/has_save.hpp>\n#include <graphlab/util/branch_hints.hpp>\nnamespace graphlab {\n\n /**\n * \\ingroup group_serialization\n * \\brief The serialization output archive object which, provided\n * with a reference to an ostream, will write to the ostream,\n * providing serialization capabilities.\n *\n * Given a standard output stream, you can construct an oarchive\n * object by:\n * \\code\n * // where strm is an ostream object\n * graphlab::oarchive oarc(strm);\n * \\endcode\n *\n * For instance, to serialize to a file,\n * \\code\n * std::ofstream fout(\"outputfile.bin\");\n * graphlab::oarchive oarc(fout);\n * \\endcode\n *\n * Once the oarc object is constructed, \\ref sec_serializable objects can be\n * written to it using the << stream operator.\n *\n * \\code\n * oarc << a << b << c;\n * \\endcode\n *\n * Alternatively, data can be directly written to the stream\n * using the oarchive::write() function.\n *\n * Written data can be deserialized using graphlab::iarchive.\n * For more usage details, see \\ref serialization\n *\n * The oarchive object should not be used once the associated stream\n * object is closed or is destroyed.\n *\n * The oarc object\n * does <b> not </b> flush the associated stream, and the user may need to\n * manually flush the associated stream to clear any stream buffers.\n * For instance, while the std::stringstream may be used for both output\n * and input, it is necessary to flush the stream before all bytes written to\n * the stringstream are available for input.\n *\n * To use this class, include\n * graphlab/serialization/serialization_includes.hpp\n */\n class oarchive{\n public:\n std::ostream* out;\n char* buf;\n size_t off;\n size_t len;\n /// constructor. Takes a generic std::ostream object\n inline oarchive(std::ostream& outstream)\n : out(&outstream),buf(NULL),off(0),len(0) {}\n\n inline oarchive(void)\n : out(NULL),buf(NULL),off(0),len(0) {}\n\n inline void expand_buf(size_t s) {\n if (__unlikely__(off + s > len)) {\n len = 2 * (s + len);\n buf = (char*)realloc(buf, len);\n }\n }\n /** Directly writes \"s\" bytes from the memory location\n * pointed to by \"c\" into the stream.\n */\n inline void write(const char* c, std::streamsize s) {\n if (out == NULL) {\n expand_buf(s);\n memcpy(buf + off, c, s);\n off += s;\n } else {\n out->write(c, s);\n }\n }\n template <typename T>\n inline void direct_assign(const T& t) {\n if (out == NULL) {\n expand_buf(sizeof(T));\n (*reinterpret_cast<T*>(buf + off)) = t;\n off += sizeof(T);\n }\n else {\n out->write(reinterpret_cast<const char*>(&t), sizeof(T));\n }\n }\n\n inline void advance(size_t s) {\n if (out == NULL) {\n expand_buf(s);\n off += s;\n } else {\n out->seekp(s, std::ios_base::cur);\n }\n }\n\n /// Returns true if the underlying stream is in a failure state\n inline bool fail() {\n return out == NULL ? false : out->fail();\n }\n\n inline ~oarchive() { }\n };\n\n /**\n * \\ingroup group_serialization\n * \\brief\n * When this archive is used to serialize an object,\n * and the object does not support serialization,\n * failure will only occur at runtime. Otherwise equivalent to\n * graphlab::oarchive\n */\n class oarchive_soft_fail {\n public:\n oarchive* oarc;\n bool mine;\n\n /// constructor. Takes a generic std::ostream object\n inline oarchive_soft_fail(std::ostream& outstream)\n : oarc(new oarchive(outstream)), mine(true) { }\n\n inline oarchive_soft_fail(oarchive& oarc):oarc(&oarc), mine(false) {\n }\n\n inline oarchive_soft_fail(void)\n : oarc(new oarchive) {}\n\n /** Directly writes \"s\" bytes from the memory location\n * pointed to by \"c\" into the stream.\n */\n inline void write(const char* c, std::streamsize s) {\n oarc->write(c, s);\n }\n template <typename T>\n inline void direct_assign(const T& t) {\n oarc->direct_assign(t);\n }\n\n inline bool fail() {\n return oarc->fail();\n }\n\n inline ~oarchive_soft_fail() {\n if (mine) delete oarc;\n }\n };\n\n namespace archive_detail {\n\n /// called by the regular archive The regular archive will do a hard fail\n template <typename OutArcType, typename T>\n struct serialize_hard_or_soft_fail {\n inline static void exec(OutArcType& oarc, const T& t) {\n t.save(oarc);\n }\n };\n\n /// called by the soft fail archive\n template <typename T>\n struct serialize_hard_or_soft_fail<oarchive_soft_fail, T> {\n inline static void exec(oarchive_soft_fail& oarc, const T& t) {\n // create a regular oarchive and\n // use the save_or_fail function which will\n // perform a soft fail\n save_or_fail(*(oarc.oarc), t);\n }\n };\n\n\n /**\n Implementation of the serializer for different types.\n This is the catch-all. If it gets here, it must be a non-POD and is a class.\n We therefore call the .save function.\n Here we pick between the archive types using serialize_hard_or_soft_fail\n */\n template <typename OutArcType, typename T, bool IsPOD>\n struct serialize_impl {\n static void exec(OutArcType& oarc, const T& t) {\n serialize_hard_or_soft_fail<OutArcType, T>::exec(oarc, t);\n }\n };\n\n /** Catch if type is a POD */\n template <typename OutArcType, typename T>\n struct serialize_impl<OutArcType, T, true> {\n inline static void exec(OutArcType& oarc, const T& t) {\n oarc.direct_assign(t);\n //oarc.write(reinterpret_cast<const char*>(&t), sizeof(T));\n }\n };\n\n /**\n Re-dispatch if for some reasons T already has a const\n */\n template <typename OutArcType, typename T>\n struct serialize_impl<OutArcType, const T, true> {\n inline static void exec(OutArcType& oarc, const T& t) {\n serialize_impl<OutArcType, T, true>::exec(oarc, t);\n }\n };\n\n /**\n Re-dispatch if for some reasons T already has a const\n */\n template <typename OutArcType, typename T>\n struct serialize_impl<OutArcType, const T, false> {\n inline static void exec(OutArcType& oarc, const T& t) {\n serialize_impl<OutArcType, T, false>::exec(oarc, t);\n }\n };\n }// archive_detail\n\n\n /// \\cond GRAPHLAB_INTERNAL\n\n /**\n Overloads the operator<< in the oarchive to\n allow the use of the stream syntax for serialization.\n It simply re-dispatches into the serialize_impl classes\n */\n template <typename T>\n inline oarchive& operator<<(oarchive& oarc, const T& t) {\n archive_detail::serialize_impl<oarchive,\n T,\n gl_is_pod<T>::value >::exec(oarc, t);\n return oarc;\n }\n\n /**\n Overloads the operator<< in the oarchive_soft_fail to\n allow the use of the stream syntax for serialization.\n It simply re-dispatches into the serialize_impl classes\n */\n template <typename T>\n inline oarchive_soft_fail& operator<<(oarchive_soft_fail& oarc,\n const T& t) {\n archive_detail::serialize_impl<oarchive_soft_fail,\n T,\n gl_is_pod<T>::value >::exec(oarc, t);\n return oarc;\n }\n\n\n /**\n Serializes an arbitrary pointer + length to an archive\n */\n inline oarchive& serialize(oarchive& oarc,\n const void* str,\n const size_t length) {\n // save the length\n oarc.write(reinterpret_cast<const char*>(str),\n (std::streamsize)length);\n assert(!oarc.fail());\n return oarc;\n }\n\n\n /**\n Serializes an arbitrary pointer + length to an archive\n */\n inline oarchive_soft_fail& serialize(oarchive_soft_fail& oarc,\n const void* str,\n const size_t length) {\n // save the length\n oarc.write(reinterpret_cast<const char*>(str),\n (std::streamsize)length);\n assert(!oarc.fail());\n return oarc;\n }\n\n /// \\endcond GRAPHLAB_INTERNAL\n\n}\n /**\n \\ingroup group_serialization\n \\brief Macro to make it easy to define out-of-place saves\n\n In the event that it is impractical to implement a save() and load()\n function in the class one wnats to serialize, it is necessary to define\n an \"out of save\" save and load.\n\n See \\ref sec_serializable_out_of_place for an example\n\n \\note important! this must be defined in the global namespace!\n */\n#define BEGIN_OUT_OF_PLACE_SAVE(arc, tname, tval) \\\n namespace graphlab{ namespace archive_detail { \\\n template <typename OutArcType> struct serialize_impl<OutArcType, tname, false> { \\\n static void exec(OutArcType& arc, const tname & tval) {\n\n#define END_OUT_OF_PLACE_SAVE() } }; } }\n\n\n#endif\n\n#endif\n\n\n\n\n\n\n\n\n\n" }, { "path": "src/graphlab/serialization/serializable_concept.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SERIALIZABLE\n#define GRAPHLAB_SERIALIZABLE\n#include <boost/concept/assert.hpp>\n#include <boost/concept/requires.hpp>\n#include <boost/concept_check.hpp>\n#include <sstream>\n#include <graphlab/serialization/serialize.hpp>\nnamespace graphlab {\n\n /**\n * \\brief Concept checks if a type T is serializable.\n *\n * This is a concept checking class for boost::concept and can be \n * used to enforce that a type T is \\ref sec_serializable, assignable and \n * default constructible. \n *\n * \\tparam T The type to test for serializability.\n */\n template <typename T>\n class Serializable : boost::Assignable<T>, boost::DefaultConstructible<T> {\n public:\n BOOST_CONCEPT_USAGE(Serializable) {\n std::stringstream strm;\n oarchive oarc(strm);\n iarchive iarc(strm);\n const T const_t = T();\n T t = T();\n // A compiler error on these lines implies that your type is not\n // serializable. See the documentaiton on how to make\n // serializable type.\n oarc << const_t;\n iarc >> t;\n }\n };\n\n} // namespace graphlab\n#endif\n\n" }, { "path": "src/graphlab/serialization/serializable_pod.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef SERIALIZABLE_POD_HPP\n#define SERIALIZABLE_POD_HPP\n\n#include <graphlab/serialization/is_pod.hpp>\n\n#define SERIALIZABLE_POD(tname) \\\nnamespace graphlab { \\\n template <> \\\n struct gl_is_pod<tname> { \\\n BOOST_STATIC_CONSTANT(bool, value = true); \\\n }; \\\n}\n\n#endif\n\n" }, { "path": "src/graphlab/serialization/serialization.dox", "content": "/**\n\n\\page serialization Serialization \n\nWe have a custom serialization scheme which is\ndesigned for performance rather than compatibility. It does not perform type\nchecking, It does not perform pointer tracking, and has only limited support\nacross platforms. It has been tested, and should be compatible across x86\nplatforms.\n\nFor a summary of all serialization functionality see \\ref group_serialization\n\nThere are two serialization classes \\ref graphlab::oarchive and \\ref\ngraphlab::iarchive. The former does output, while the latter does input. To\ninclude all serialization headers, \\#include\n<graphlab/serialization/serialization_includes.hpp>.\n\n\\section sec_basic_serialize Basic serialize/deserialize\n\nTo serialize data to disk, you just create an output archive, and associate it\nwiith an output stream.\n\nFor instance, to serialize to a file called \"file.bin\":\n\\code\n std::ofstream fout(\"file.bin\", std::fstream::binary); \n graphlab::oarchive oarc(fout); \n\\endcode\n\nThe << stream operators are then used to write data into the archive.\n\n\\code \n int i = 10; \n double j = 20; \n std::vector<float> v(10,1.0); // create a vector of 10 \"1.0\" values\n oarc << i << j << v; \n\\endcode\n\nTo read back, you use the iarchive with an input stream, and read back the\nvariables in the same order:\n\n\\code \n std::ifstream fin(\"file.bin\", std::fstream::binary); \n graphlab::iarchive iarc(fout);\n int i;\n double j; \n std::vector<float> v;\n iarc >> i >> j >> v; \n\\endcode\n\n\\section sec_serializable Serializable\nSo what type of data is serializable?\n\n\\subsection sec_serializable_integers Integer Types\nAll integer datatypes are serializable.\n\\li <code> bool </code>\n\\li <code> char </code> and <code> unsigned char </code>\n\\li <code> short </code>and <code> unsigned short</code>\n\\li <code> int </code>and <code> unsigned int</code>\n\\li <code> long </code>and <code> unsigned long </code>\n\\li <code> long long </code>and <code> unsigned long long</code>\n\nSince all fixed width integer types from stdint (int16_t, int32_t, etc)\nare derived from these basic types, all fixed width integer types\nare also serializable.\n\n\\li <code> int16_t </code> and <code> uint16_t </code>\n\\li <code> int32_t </code> and <code> uint32_t </code>\n\\li <code> int64_t </code> and <code> uint64_t </code>\n\nAll integer types are saved in their raw binary form without any additional\nre-encoding. It is therefore important to deserialize with the \nsame integer width as what was serialized.\n\nThe following code will fail in dramatic ways:\n\\code\n int i;\n oarc << i; // write some integer to a file\n\n ...\n // some time later we need to read back the integer.\n long j;\n iarc >> j; // this will fail \n\\endcode\n\n\\subsection sec_serializable_floats Floating Point Types\nAll floating point data types are serializable.\n\\li <code>float</code>\n\\li <code>double</code>\n\\li <code>long double</code> if your compiler supports quad precision.\n\nSimilar to integer types, all floating types are saved in raw binary form\nwithout re-encoding. You must deserialize with the same floating point width\nas what was serialized. (i.e. if you serialize a double, you <b>must</b>\ndeserialize a double.\n\n\\subsection sec_serializable_containers Containers\nThe following template containers are serializable as long as the contained\ntypes are all serializable. This can be recursively applied.\n\\li <code>std::vector</code>\n\\li <code>std::list</code>\n\\li <code>std::set</code>\n\\li <code>std::map</code>\n\\li <code>boost::unordered_set</code>\n\\li <code>boost::unordered_map</code>\n\nFor instance, a <code>std::vector<int></code> is serializable. \nA <code>std::list<std::vector<int> > </code> is therefore also serializable.\n\nThere is special handling for the std::vector<T> for performance in the event\nthat T is a simple POD (Plain Old Data) data type. POD types are data types\nwhich occupy a contiguous region in memory. For instance, basic types (double,\nint, etc), or structs which contains only basic types. Such types can be\ncopied or replicated using a simple mem-copy operation and can be greatly\nacceleration during serialization / deserialization. All basic data types\nare automatically POD types. We will discuss structs and other user types\nin the next section.\n\n\n\n\\section sec_serializable_user User Structs and Classes\n\nTo serialize a struct/class, all you need to do is to define a public load/save\nfunction. For instance:\n\n\\code \nclass TestClass{ \n public: \n int i, j; \n std::vector<int> k;\n void save(graphlab::oarchive& oarc) const { \n oarc << i << j << k; \n }\n void load(graphlab::iarchive& iarc) { \n iarc >> i >> j >> k; \n } \n}; \n\\endcode\nThe save() and load() function prototypes must match <b>exactly</b>.\nOther conditions are that the class must be Default Constructible:\n\\code\n // it must be possible to create a variable of TestClass type like this\n TestClass a;\n\\endcode\n\nAnd that the class must be Assignable:\n \\code\n TestClass a, b;\n // it must be possible to assign one variable of TestClass to another\n b = a;\n \\endcode \n\nAfter which, <code>TestClass</code> becomes serializable, and can be stored and\nread from an archive:\n\n\\code\nTestClass t; \n// set values to t\noarc << t; // write it to a file\n\n... some time afterwords...\n\nTestClass t2;\niarc >> t2; // read it to a file\n\\endcode\n\nSince <code>TestClass</code> is now serializable, containers of TestClass\nlisted in \\ref sec_serializable_containers are also serializable.\n\n\\section sec_serializable_pod POD Serialization \n\nAs mentioned in \\ref sec_serializable_containers, POD data types \noccupy a contiguous region in memory and hence can be serialized and \ndeserialized very quickly. Ideally, determination of whether a data type is \nPOD or not should be handled by the compiler. However, this capability is only\navailable in C++11 and not all compilers support it yet. We therefore\nimplemented a simple workaround which will allow you to identify to the \nserializer that a class is POD, and avoid writing a save/load function.\n\n\nWe consider the following Coordinate struct. \n\\code \nstruct Coordinate{ \n int x, y, z; \n}; \n\\endcode\n\nThis struct can be defined to be a POD type using an accelerated serializer by\nsimply inheriting from graphlab::IS_POD_TYPE \n\n\\code \nstruct Coordinate: public graphlab::IS_POD_TYPE{ \n int x, y, z; \n}; \n\\endcode\n\n\nNow, Coordinate variables, or even vector<Coordinate> variables will\nserialize/deserialize faster. Also, you avoid writing a save() and load()\nfunction.\n\n\\note\nCurrently POD detection is performed through the boost type traits library.\nWhen compilers implement std::is_pod (in C++11), POD detection will improve,\nincreasing the scope of types which can be serialized quickly and automatically.\nA minor concern is that the scope of POD types is still slightly too large,\nsince technically pointer types are POD, and those cannot not be \nserialized automatically.\n\n\\section sec_serializable_out_of_place Out of Place Serialization\nIn some situations, you may find that you need to make a data type serializable,\nbut the data type is implemented by someone else, in a different library,\nmaking it impossible to extend and write a member save() and load() function\nas described in \\ref sec_serializable_user.\n\nIn this situation, it is necessary to implement an \"Out of place\" \nserializer. This is unfortunately somewhat more complicated.\n\nFor instance, if there is an external type implemented by some other library\ncalled Matrix which I would like to make serializable. The following code will\nhave to be written in the <b>global namespace</b>\n\n\\code\nBEGIN_OUT_OF_PLACE_SAVE(oarc, Matrix, mat)\n // write the \"mat\" variable which is of the type Matrix\n // into the output archive oarc\nEND_OUT_OF_PLACE_SAVE()\n\nBEGIN_OUT_OF_PLACE_LOAD(iarc, Matrix, mat)\n // read the \"mat\" variable which is of the type Matrix\n // from the input archive iarc \nEND_OUT_OF_PLACE_LOAD()\n\\endcode\n\nTo facilitate reading and writing of data from the archives, the output oarchive \nobject provides an \\ref graphlab::oarchive::write() oarchive::write() \nfunction which directly writes a sequence of bytes to the stream. Similarly,\nthe input iarchive object provides a \n\\ref graphlab::iarchive::read() iarchive::read() function which directly reads\na sequence of bytes from the stream.\n\nFor instance, if the Matrix type example above is defined in the following way:\n\n\\code\nstruct Matrix {\n int width; // width of the matrix\n int height; // height of the matrix\n double* data; // an array containing all the values in the matrix\n int datalen; // the number of elements in the \"data\" array.\n}\n\\endcode\n\nAn \"out of place\" serializer could be implemented the following way:\n\\code\nBEGIN_OUT_OF_PLACE_SAVE(oarc, Matrix, mat)\n // store the dimensions of the matrix\n oarc << mat.width << mat.height;\n\n // store the length of the data array\n oarc << mat.datalen;\n\n // write the double array\n oarc.write((char*)(mat.data), sizeof(double) * mat.datalen);\nEND_OUT_OF_PLACE_SAVE()\n\nBEGIN_OUT_OF_PLACE_LOAD(iarc, Matrix, mat)\n // clear the matrix data if there is any\n if (mat.data != NULL) delete [] mat.data;\n\n // read the dimensions of the matrix\n iarc >> mat.width >> mat.height;\n\n // read the length of the data array\n iarc >> mat.datalen;\n\n // allocate sufficient storage for the array\n mat.data = new double[mat.datalen];\n\n // read the double array\n iarc.read((char*)(mat.data), sizeof(double) * mat.datalen);\n\nEND_OUT_OF_PLACE_LOAD()\n\\endcode\n\n*/\n" }, { "path": "src/graphlab/serialization/serialization_includes.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n#include <graphlab/serialization/serialize.hpp>\n#include <graphlab/serialization/serializable_concept.hpp>\n" }, { "path": "src/graphlab/serialization/serialize.hpp", "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SERIALIZE_HPP\n#define GRAPHLAB_SERIALIZE_HPP\n#include <graphlab/serialization/iarchive.hpp>\n#include <graphlab/serialization/oarchive.hpp>\n#include <graphlab/serialization/basic_types.hpp>\n\n#include <graphlab/serialization/list.hpp>\n#include <graphlab/serialization/set.hpp>\n#include <graphlab/serialization/vector.hpp>\n#include <graphlab/serialization/map.hpp>\n#include <graphlab/serialization/unordered_map.hpp>\n#include <graphlab/serialization/unordered_set.hpp>\n#include <graphlab/serialization/serializable_pod.hpp>\n#include <graphlab/serialization/unsupported_serialize.hpp>\n#include <graphlab/serialization/serialize_to_from_string.hpp>\n#include <graphlab/serialization/conditional_serialize.hpp>\n#endif\n\n" }, { "path": "src/graphlab/serialization/serialize_to_from_string.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef SERIALIZE_TO_FROM_STRING_HPP\n#define SERIALIZE_TO_FROM_STRING_HPP\n#include <sstream>\n#include <boost/iostreams/stream.hpp>\n\nnamespace graphlab {\n /**\n * \\ingroup group_serialization\n * \\brief Serializes a object to a string\n * \n * Converts a \\ref serializable object t to a string\n * using the serializer.\n * \n * \\tparam T the type of object to serialize. Typically\n * will be inferred by the compiler. \n *\n * \\param t The object to serializer\n * \\returns A string containing a serialized form of t \n *\n * \\see deserialize_from_string()\n */\n template <typename T>\n inline std::string serialize_to_string(const T &t) {\n std::stringstream strm;\n oarchive oarc(strm);\n oarc << t;\n strm.flush();\n return strm.str();\n }\n\n\n /**\n * \\ingroup group_serialization\n * \\brief Deserializes a object from a string\n * \n * Deserializes a \\ref serializable object t from a string\n * using the deserializer.\n * \n * \\tparam T the type of object to deserialize. Typically\n * will be inferred by the compiler. \n *\n * \\param s The string to deserialize \n * \\param t A reference to the object which will contain \n * the deserialized object when the function returns\n *\n * \\see serialize_from_string()\n */\n template <typename T>\n inline void deserialize_from_string(const std::string &s, T &t) {\n boost::iostreams::stream<boost::iostreams::array_source> \n istrm(s.c_str(), s.length()); \n iarchive iarc(istrm);\n iarc >> t;\n }\n}\n\n#endif\n" }, { "path": "src/graphlab/serialization/set.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SERIALIZE_SET_HPP\n#define GRAPHLAB_SERIALIZE_SET_HPP\n\n#include <set>\n#include <graphlab/serialization/iarchive.hpp>\n#include <graphlab/serialization/oarchive.hpp>\n#include <graphlab/serialization/iterator.hpp>\n\nnamespace graphlab {\nnamespace archive_detail {\n /** serializes a set */\n template <typename OutArcType, typename T>\n struct serialize_impl<OutArcType, std::set<T>, false > {\n static void exec(OutArcType& oarc, const std::set<T>& vec){\n serialize_iterator(oarc,\n vec.begin(), vec.end(), vec.size());\n }\n };\n\n /** deserializes a set */\n template <typename InArcType, typename T>\n struct deserialize_impl<InArcType, std::set<T>, false > {\n static void exec(InArcType& iarc, std::set<T>& vec){\n vec.clear();\n deserialize_iterator<InArcType, T>(iarc,\n std::inserter(vec,vec.end()));\n }\n };\n\n} // archive_detail \n} // graphlab\n\n#endif \n\n" }, { "path": "src/graphlab/serialization/unordered_map.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SERIALIZE_UNORDERED_MAP_HPP\n#define GRAPHLAB_SERIALIZE_UNORDERED_MAP_HPP\n\n#include <boost/unordered_map.hpp>\n#include <graphlab/serialization/iarchive.hpp>\n#include <graphlab/serialization/oarchive.hpp>\n#include <graphlab/serialization/iterator.hpp>\n\nnamespace graphlab {\n\nnamespace archive_detail {\n /** Serializes a map */\n template <typename OutArcType, typename T, typename U>\n struct serialize_impl<OutArcType, boost::unordered_map<T,U>, false > {\n static void exec(OutArcType& oarc, \n const boost::unordered_map<T,U>& vec){\n serialize_iterator(oarc, \n vec.begin(), vec.end(), vec.size());\n }\n };\n\n /** deserializes a map */\n \n template <typename InArcType, typename T, typename U>\n struct deserialize_impl<InArcType, boost::unordered_map<T,U>, false > {\n static void exec(InArcType& iarc, boost::unordered_map<T,U>& vec){\n vec.clear();\n // get the number of elements to deserialize\n size_t length = 0;\n iarc >> length; \n // iterate through and send to the output iterator\n for (size_t x = 0; x < length ; ++x){\n std::pair<T, U> v;\n iarc >> v;\n vec[v.first] = v.second;\n }\n }\n };\n\n} // archive_detail \n} // graphlab\n#endif \n\n" }, { "path": "src/graphlab/serialization/unordered_set.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SERIALIZE_UNORDERED_SET_HPP\n#define GRAPHLAB_SERIALIZE_UNORDERED_SET_HPP\n\n#include <boost/unordered_set.hpp>\n#include <graphlab/serialization/iarchive.hpp>\n#include <graphlab/serialization/oarchive.hpp>\n#include <graphlab/serialization/iterator.hpp>\n\nnamespace graphlab {\nnamespace archive_detail {\n /** serializes a set */\n template <typename OutArcType, typename T>\n struct serialize_impl<OutArcType, boost::unordered_set<T>, false > {\n static void exec(OutArcType& oarc, const boost::unordered_set<T>& vec){\n serialize_iterator(oarc,\n vec.begin(), vec.end(), vec.size());\n }\n };\n\n /** deserializes a set */\n template <typename InArcType, typename T>\n struct deserialize_impl<InArcType, boost::unordered_set<T>, false > {\n static void exec(InArcType& iarc, boost::unordered_set<T>& vec){\n vec.clear();\n // get the number of elements to deserialize\n size_t length = 0;\n iarc >> length; \n // iterate through and send to the output iterator\n for (size_t x = 0; x < length ; ++x){\n T v;\n iarc >> v;\n vec.insert(v);\n }\n }\n };\n\n} // archive_detail \n} // graphlab\n\n#endif \n\n" }, { "path": "src/graphlab/serialization/unsupported_serialize.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_UNSUPPORTED_SERIALIZE_HPP\n#define GRAPHLAB_UNSUPPORTED_SERIALIZE_HPP\n\n#include <graphlab/serialization/iarchive.hpp>\n#include <graphlab/serialization/oarchive.hpp>\n#include <graphlab/logger/logger.hpp>\n\nnamespace graphlab {\n\n /**\n * \\ingroup group_serialization\n * \\brief Inheritting from this class will prevent the serialization\n * of the derived class. Used for debugging purposes.\n * \n * Inheritting from this class will result in an assertion failure\n * if any attempt is made to serialize or deserialize the derived\n * class. This is largely used for debugging purposes to enforce\n * that certain types are never serialized \n */\n struct unsupported_serialize {\n void save(oarchive& archive) const { \n ASSERT_MSG(false, \"trying to serialize an unserializable object\");\n }\n void load(iarchive& archive) {\n ASSERT_MSG(false, \"trying to deserialize an unserializable object\");\n }\n }; // end of struct\n};\n\n\n/**\n\\ingroup group_serialization\n\\brief A macro which disables the serialization of type so that \nit will fault at runtime. \n\nWriting GRAPHLAB_UNSERIALIZABLE(T) for some typename T in the global namespace\nwill result in an assertion failure if any attempt is made to serialize or\ndeserialize the type T. This is largely used for debugging purposes to enforce\nthat certain types are never serialized. \n*/\n#define GRAPHLAB_UNSERIALIZABLE(tname) \\\n BEGIN_OUT_OF_PLACE_LOAD(arc, tname, tval) \\\n ASSERT_MSG(false,\"trying to deserialize an unserializable object\"); \\\n END_OUT_OF_PLACE_LOAD() \\\n \\\n BEGIN_OUT_OF_PLACE_SAVE(arc, tname, tval) \\\n ASSERT_MSG(false,\"trying to serialize an unserializable object\"); \\\n END_OUT_OF_PLACE_SAVE() \\\n\n\n#endif\n\n" }, { "path": "src/graphlab/serialization/vector.hpp", "content": "/** \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SERIALIZE_VECTOR_HPP\n#define GRAPHLAB_SERIALIZE_VECTOR_HPP\n#include <vector>\n#include <graphlab/serialization/iarchive.hpp>\n#include <graphlab/serialization/oarchive.hpp>\n#include <graphlab/serialization/iterator.hpp>\n\n\nnamespace graphlab {\n namespace archive_detail {\n /**\n * We re-dispatch vectors because based on the contained type,\n * it is actually possible to serialize them like a POD\n */\n template <typename OutArcType, typename ValueType, bool IsPOD>\n struct vector_serialize_impl {\n static void exec(OutArcType& oarc, const ValueType& vec) {\n // really this is an assert false. But the static assert\n // must depend on a template parameter \n BOOST_STATIC_ASSERT(sizeof(OutArcType) == 0);\n assert(false);\n };\n };\n /**\n * We re-dispatch vectors because based on the contained type,\n * it is actually possible to deserialize them like iarc POD\n */\n template <typename InArcType, typename ValueType, bool IsPOD>\n struct vector_deserialize_impl {\n static void exec(InArcType& iarc, ValueType& vec) {\n // really this is an assert false. But the static assert\n // must depend on a template parameter \n BOOST_STATIC_ASSERT(sizeof(InArcType) == 0);\n assert(false);\n };\n };\n \n /// If contained type is not a POD use the standard serializer\n template <typename OutArcType, typename ValueType>\n struct vector_serialize_impl<OutArcType, ValueType, false > {\n static void exec(OutArcType& oarc, const std::vector<ValueType>& vec) {\n oarc << size_t(vec.size());\n serialize_iterator(oarc,vec.begin(), vec.end());\n }\n };\n\n /// Fast vector serialization if contained type is a POD\n template <typename OutArcType, typename ValueType>\n struct vector_serialize_impl<OutArcType, ValueType, true > {\n static void exec(OutArcType& oarc, const std::vector<ValueType>& vec) {\n oarc << size_t(vec.size());\n serialize(oarc, &(vec[0]),sizeof(ValueType)*vec.size());\n }\n };\n\n /// If contained type is not a POD use the standard deserializer\n template <typename InArcType, typename ValueType>\n struct vector_deserialize_impl<InArcType, ValueType, false > {\n static void exec(InArcType& iarc, std::vector<ValueType>& vec){\n size_t len;\n iarc >> len;\n vec.clear(); vec.reserve(len);\n deserialize_iterator<InArcType, ValueType>(iarc, std::inserter(vec, vec.end()));\n }\n };\n\n /// Fast vector deserialization if contained type is a POD\n template <typename InArcType, typename ValueType>\n struct vector_deserialize_impl<InArcType, ValueType, true > {\n static void exec(InArcType& iarc, std::vector<ValueType>& vec){\n size_t len;\n iarc >> len;\n vec.clear(); vec.resize(len);\n deserialize(iarc, &(vec[0]), sizeof(ValueType)*vec.size());\n }\n };\n\n \n \n /**\n Serializes a vector */\n template <typename OutArcType, typename ValueType>\n struct serialize_impl<OutArcType, std::vector<ValueType>, false > {\n static void exec(OutArcType& oarc, const std::vector<ValueType>& vec) {\n vector_serialize_impl<OutArcType, ValueType, \n gl_is_pod_or_scaler<ValueType>::value >::exec(oarc, vec);\n }\n };\n /**\n deserializes a vector */\n template <typename InArcType, typename ValueType>\n struct deserialize_impl<InArcType, std::vector<ValueType>, false > {\n static void exec(InArcType& iarc, std::vector<ValueType>& vec){\n vector_deserialize_impl<InArcType, ValueType, \n gl_is_pod_or_scaler<ValueType>::value >::exec(iarc, vec);\n }\n };\n } // archive_detail\n} // namespace graphlab\n\n#endif \n\n" }, { "path": "src/graphlab/ui/CMakeLists.txt", "content": "project(GraphLab)\n\nsubdirs(mongoose)\n\n" }, { "path": "src/graphlab/ui/graphlab_visualization.js", "content": "google.load(\"jquery\", \"1.5\");\ngoogle.load(\"jqueryui\", \"1.7.2\");\ngoogle.load(\"visualization\", \"1\", \n {\"packages\":[\"corechart\", \"table\", \"gauge\"]});\n\n\nvar domain_str = \"http://localhost:8090\";\nvar update_interval = 5000;\nvar last_minutes=5;\n\nfunction update_domain(value) {\n console.log(\"Setting domain to \" + value);\n domain_str = value;\n initiate_job_info(); \n initiate_aggregate_info();\n initiate_node_info();\n}\n\n// Start the rendering of the UI\ngoogle.setOnLoadCallback(function() { \n initiate_job_info(); \n initiate_aggregate_info();\n initiate_node_info();\n});\n\nfunction initiate_job_info() {\n jQuery.getJSON(domain_str + \"/names.json\", process_job_info)\n .error(function() { \n console.log(\"Unable to access \" + domain_str + \" will try again.\");})\n .complete(function() {\n setTimeout(initiate_job_info, update_interval);\n });\n}\n\nfunction initiate_aggregate_info() {\n jQuery.getJSON(domain_str + \"/metrics_aggregate.json?rate=1&rounding=1&tlast=\" + (60*last_minutes), process_aggregate_info)\n .error(function() { \n console.log(\"Unable to access \" + domain_str + \" will try again.\");})\n .complete(function() {\n setTimeout(initiate_aggregate_info, update_interval);\n });\n}\n\n\n\nfunction initiate_node_info() {\n jQuery.getJSON(domain_str + \"/metrics_by_machine.json?rate=1&align=1&tlast=\" + (60*last_minutes) , \n process_node_info)\n .error(function() { \n console.log(\"Unable to access \" + domain_str + \" will try again.\");})\n .complete(function() {\n setTimeout(initiate_node_info, 5000);\n });\n}\n\n\n\nvar job_info_data = [];\nfunction process_job_info(data) {\n console.log(\"Processing job info.\");\n $(\"#program_name\").text(data.program_name);\n // $(\"#nprocs\").text(data.nprocs + \" processes\");\n $(\"#current_time\").text((data.time) + \" seconds\");\n\n/*\n // Render all the current values\n var container = $(\"#gauges\");\n var sorted_metrics = data.metrics.sort(function(a,b) { \n return a.id - b.id; \n });\n // Build an array of divs one for each metric with the name and value\n jQuery.each(sorted_metrics, function(i, metric) {\n var id = metric.id;\n var name = metric.name;\n var value = metric.rate_val;\n // if no job info has been created then create one as well as\n // the div to contain the display items\n if(job_info_data[id] == undefined) {\n // add a div the container\n var gauge_div_name = id + \"_info_gauge\";\n var str = \n \"<div class=\\\"metric_summary\\\" id=\\\"\" + gauge_div_name + \"\\\">\" +\n \"<div class=\\\"name\\\">\" + name + \"</div>\" +\n \"<div class=\\\"value\\\">\" + value + \"</div>\" +\n \"<div class=\\\"gauge\\\"></div>\" +\n \"</div>\";\n container.append(str);\n var div = container.children(\"#\" + gauge_div_name);\n var gauge = new google.visualization.Gauge($(div).children(\".gauge\")[0]);\n job_info_data[id] = {\n div: div,\n gauge: gauge,\n options: {\n animation: {duration: 100, easing: \"linear\"},\n width: 400, height: 120,\n min: 0, max: value + 1.0E-5},\n data: google.visualization.arrayToDataTable([\n ['Label', 'Value'], [name, value] ])\n };\n }\n // Get the job info\n var info = job_info_data[id];\n info.options.max = Math.max(info.options.max, value);\n // info.options.min = Math.min(info.options.min, value);\n info.data.setCell(0,1, value);\n info.gauge.draw(info.data, info.options);\n info.div.children(\".value\").text(value);\n\n });\n*/\n}\n\n\n\n\nvar aggregate_charts = []\n\nfunction process_aggregate_info(data) {\n console.log(\"Processing aggregate info.\");\n\n // Render all the current values\n var container = $(\"#aggregate\");\n var sorted_data = data.sort(function(a,b) { \n return a.id - b.id; \n });\n // Build an array of divs one for each metric with the name and value\n jQuery.each(sorted_data, function(i, metric) {\n var id = metric.id;\n var name = metric.name;\n var units = metric.units;\n if(aggregate_charts[id] == undefined) {\n // add a div the container\n var div_name = id + \"_aggregate_chart\";\n var str = \n \"<div class=\\\"aggregate\\\" id=\\\"\" + div_name + \"\\\">\" +\n \"<div class=\\\"name\\\">\" + name + \"</div>\" +\n \"<div class=\\\"chart\\\"></div>\" +\n \"</div>\";\n container.append(str);\n var div = $(container.children(\"#\" + div_name)).children(\".chart\")[0]; \n aggregate_charts[id] = {\n div: div,\n options: { title: name,\n vAxis: {title: (units + \" per Second\"),\n titleTextStyle: {color: 'red'},\n minValue: 0,\n maxValue: 10,\n viewWindow: {min: 0}\n },\n hAxis: {title: 'Time (seconds)'}},\n chart: new google.visualization.AreaChart(div),\n }\n }\n if(metric.record.length > 0) {\n // Update the chart\n var chart_info = aggregate_charts[id];\n chart_info.data = google.visualization.arrayToDataTable(\n [[\"Time\", \"Value\"]].concat(metric.record));\n chart_info.chart.draw(chart_info.data, chart_info.options);\n }\n });\n\n}\n\n\n\n\nfunction tensor_to_table(tensor) {\n var table = new google.visualization.DataTable();\n table.addColumn(\"number\", \"Time\");\n var numLines = tensor.length;\n var numRows = 0;\n for(var i = 0; i < numLines; ++i) {\n table.addColumn(\"number\", \"Node \" + i);\n }\n numRows = tensor[0].length;\n for(var i = 0; i < tensor.length; ++i) {\n numRows = Math.min(numRows, tensor[i].length);\n }\n table.addRows(numRows);\n var counter = 0;\n for(var i = 0; i < numLines; ++i) {\n for(var j = 0; j < numRows; ++j) {\n if (i == 0) table.setValue(j, 0, tensor[i][j][0]);\n if (tensor[i][j][1] >= 0) {\n table.setValue(j, i+1, tensor[i][j][1]);\n }\n }\n }\n return table;\n}\n\n\n\n\n\n\n\n\n\nvar node_charts = []\n\nfunction process_node_info(data) {\n console.log(\"Processing node info.\");\n\n // Render all the current values\n var container = $(\"#nodes\");\n var sorted_data = data.sort(function(a,b) { \n return a.id - b.id; \n });\n // Build an array of divs one for each metric with the name and value\n jQuery.each(sorted_data, function(i, metric) {\n var id = metric.id;\n var name = metric.name;\n var units = metric.units;\n if(node_charts[id] == undefined) {\n // add a div the container\n var div_name = id + \"_node_chart\";\n var str = \n \"<div class=\\\"node\\\" id=\\\"\" + div_name + \"\\\">\" +\n \"<div class=\\\"name\\\">\" + name + \"</div>\" +\n \"<div class=\\\"chart\\\"></div>\" +\n \"</div>\";\n container.append(str);\n var div = $(container.children(\"#\" + div_name)).children(\".chart\")[0]; \n node_charts[id] = {\n div: div,\n options: { title: name,\n //isStacked: true,\n enableInteractivity: 0,\n vAxis: {title: (units + \" per Second\"),\n titleTextStyle: {color: 'red'},\n minValue: 0,\n maxValue: 10,\n viewWindow: {min: 0}\n },\n hAxis: {title: 'Time (seconds)'}},\n chart: new google.visualization.LineChart(div),\n }\n }\n if(metric.record.length > 0) {\n // Update the chart\n var chart_info = node_charts[id];\n chart_info.data = tensor_to_table(metric.record);\n chart_info.chart.draw(chart_info.data, chart_info.options);\n }\n });\n\n}\n\n\n\n\n" }, { "path": "src/graphlab/ui/index.html", "content": "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\"\n\"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\n\n<html xmlns=\"http://www.w3.org/1999/xhtml\" xml:lang=\"en\" lang=\"en\">\n\n<head>\n<meta http-equiv=\"content-type\" content=\"text/html; charset=iso-8859-1\" />\n<meta name=\"author\" content=\"GraphLab Dev Team\" />\n<meta name=\"publisher\" content=\"Carnegie Mellon University\" />\n<meta name=\"copyright\" content=\"(c) 2012. GraphLab.org\" />\n<meta name=\"distribution\" content=\"global\" />\n<meta name=\"keywords\" content=\"GraphLabUI\" />\n<meta name=\"description\" content=\"GraphLab Dashboard\" />\n<meta name=\"robots\" content=\"all\" />\n<title>GraphLab Dashboard \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n

GraphLab Visualization Dashboard

\n\n\n\n\n
\n
\n \n
\n
\n\n
\n
\n
\n
\n
\n\n\n\n\n\n\n\n" }, { "path": "src/graphlab/ui/intel_demo/graph_builder.json", "content": "{ \"phase_name\" : \"parsing\",\r\n \"sys_metrics\" : [\r\n { \r\n \"label\": \"cpu\",\r\n \"units\": \"% Utilization\",\r\n \"id\": 1,\r\n \"values\": [\r\n [\"time\", \"phase1\", \"phas2\", \"phase3\"],\r\n [0, 23.4, 0, 17],\r\n [1, 2, 1, 9],\r\n [2, 0, 18, {} ],\r\n [3, 0, 2, {} ],\r\n [4, 0, 0, {}]]\r\n }, \r\n { \r\n \"label\": \"Network\",\r\n \"units\": \"MB / Second\",\r\n \"id\": 2,\r\n \"values\": [\r\n [\"time\", \"cpu1\", \"cpu2\", \"cpu3\"],\r\n [0, 23.4, 25, 17],\r\n [1, 24, 18, 9],\r\n [2, 27, 3, 5],\r\n [3, 40, 2, 8],\r\n [4, 75, 12, 2]]\r\n }, \r\n { \r\n \"label\": \"Disk\",\r\n \"units\": \"MB / Second\",\r\n \"id\": 3,\r\n \"values\": [\r\n [\"time\", \"cpu1\", \"cpu2\", \"cpu3\"],\r\n [0, 23.4, 25, 17],\r\n [1, 24, 18, 9],\r\n [2, 27, 3, 5],\r\n [3, 40, 2, 8],\r\n [4, 75, 12, 2]]\r\n\r\n }\r\n ]\r\n }" }, { "path": "src/graphlab/ui/intel_demo/graphlab_visualization.js", "content": "google.load(\"jquery\", \"1.5\");\ngoogle.load(\"jqueryui\", \"1.7.2\");\ngoogle.load(\"visualization\", \"1\", \n {\"packages\":[\"corechart\", \"table\", \"gauge\"]});\n\n\nvar domain_str = \"http://localhost:8090\";\nvar update_interval = 5000;\nvar last_minutes=5;\n\n// This function is called when the update domain button is clicked\nfunction update_domain(value) {\n console.log(\"Setting domain to \" + value);\n domain_str = value;\n start();\n}\n\n// Start the rendering of the UI\ngoogle.setOnLoadCallback(start);\n\n// Start rendering the UI\nfunction start() { \n initiate_graph_builder();\n initiate_job_info(); \n initiate_aggregate_info();\n initiate_node_info();\n}\n\nfunction initiate_job_info() {\n jQuery.getJSON(domain_str + \"/names.json\", process_job_info)\n .error(function() { \n console.log(\"Unable to access \" + domain_str + \" will try again.\");})\n .complete(function() {\n setTimeout(initiate_job_info, update_interval);\n });\n}\n\nfunction initiate_aggregate_info() {\n jQuery.getJSON(domain_str + \"/metrics_aggregate.json?rate=1&rounding=1&tlast=\" + (60*last_minutes), process_aggregate_info)\n .error(function() { \n console.log(\"Unable to access \" + domain_str + \" will try again.\");})\n .complete(function() {\n setTimeout(initiate_aggregate_info, update_interval);\n });\n}\n\n\n\nfunction initiate_node_info() {\n jQuery.getJSON(domain_str + \"/metrics_by_machine.json?rate=1&align=1&tlast=\" + (60*last_minutes) , \n process_node_info)\n .error(function() { \n console.log(\"Unable to access \" + domain_str + \" will try again.\");})\n .complete(function() {\n setTimeout(initiate_node_info, 5000);\n });\n}\n\n\n\nvar job_info_data = [];\nfunction process_job_info(data) {\n console.log(\"Processing job info.\");\n $(\"#program_name\").text(data.program_name);\n // $(\"#nprocs\").text(data.nprocs + \" processes\");\n $(\"#current_time\").text((data.time) + \" seconds\");\n\n/*\n // Render all the current values\n var container = $(\"#gauges\");\n var sorted_metrics = data.metrics.sort(function(a,b) { \n return a.id - b.id; \n });\n // Build an array of divs one for each metric with the name and value\n jQuery.each(sorted_metrics, function(i, metric) {\n var id = metric.id;\n var name = metric.name;\n var value = metric.rate_val;\n // if no job info has been created then create one as well as\n // the div to contain the display items\n if(job_info_data[id] == undefined) {\n // add a div the container\n var gauge_div_name = id + \"_info_gauge\";\n var str = \n \"
\" +\n \"
\" + name + \"
\" +\n \"
\" + value + \"
\" +\n \"
\" +\n \"
\";\n container.append(str);\n var div = container.children(\"#\" + gauge_div_name);\n var gauge = new google.visualization.Gauge($(div).children(\".gauge\")[0]);\n job_info_data[id] = {\n div: div,\n gauge: gauge,\n options: {\n animation: {duration: 100, easing: \"linear\"},\n width: 400, height: 120,\n min: 0, max: value + 1.0E-5},\n data: google.visualization.arrayToDataTable([\n ['Label', 'Value'], [name, value] ])\n };\n }\n // Get the job info\n var info = job_info_data[id];\n info.options.max = Math.max(info.options.max, value);\n // info.options.min = Math.min(info.options.min, value);\n info.data.setCell(0,1, value);\n info.gauge.draw(info.data, info.options);\n info.div.children(\".value\").text(value);\n\n });\n*/\n}\n\n\n\n\nvar aggregate_charts = []\n\nfunction process_aggregate_info(data) {\n console.log(\"Processing aggregate info.\");\n\n // Render all the current values\n var container = $(\"#aggregate\");\n var sorted_data = data.sort(function(a,b) { \n return a.id - b.id; \n });\n // Build an array of divs one for each metric with the name and value\n jQuery.each(sorted_data, function(i, metric) {\n var id = metric.id;\n var name = metric.name;\n var units = metric.units;\n if(aggregate_charts[id] == undefined) {\n // add a div the container\n var div_name = id + \"_aggregate_chart\";\n var str = \n \"
\" +\n \"
\" + name + \"
\" +\n \"
\" +\n \"
\";\n container.append(str);\n var div = $(container.children(\"#\" + div_name)).children(\".chart\")[0]; \n aggregate_charts[id] = {\n div: div,\n options: { title: name,\n vAxis: {title: (units + \" per Second\"),\n titleTextStyle: {color: 'red'},\n minValue: 0,\n maxValue: 10,\n viewWindow: {min: 0}\n },\n hAxis: {title: 'Time (seconds)'}},\n chart: new google.visualization.AreaChart(div),\n }\n }\n if(metric.record.length > 0) {\n // Update the chart\n var chart_info = aggregate_charts[id];\n chart_info.data = google.visualization.arrayToDataTable(\n [[\"Time\", \"Value\"]].concat(metric.record));\n chart_info.chart.draw(chart_info.data, chart_info.options);\n }\n });\n\n}\n\n\n\n\nfunction tensor_to_table(tensor) {\n var table = new google.visualization.DataTable();\n table.addColumn(\"number\", \"Time\");\n var numLines = tensor.length;\n var numRows = 0;\n for(var i = 0; i < numLines; ++i) {\n table.addColumn(\"number\", \"Node \" + i);\n }\n numRows = tensor[0].length;\n for(var i = 0; i < tensor.length; ++i) {\n numRows = Math.min(numRows, tensor[i].length);\n }\n table.addRows(numRows);\n var counter = 0;\n for(var i = 0; i < numLines; ++i) {\n for(var j = 0; j < numRows; ++j) {\n if (i == 0) table.setValue(j, 0, tensor[i][j][0]);\n if (tensor[i][j][1] >= 0) {\n table.setValue(j, i+1, tensor[i][j][1]);\n }\n }\n }\n return table;\n}\n\n\n\n\n\n\n\n\n\nvar node_charts = []\n\nfunction process_node_info(data) {\n console.log(\"Processing node info.\");\n\n // Render all the current values\n var container = $(\"#nodes\");\n var sorted_data = data.sort(function(a,b) { \n return a.id - b.id; \n });\n // Build an array of divs one for each metric with the name and value\n jQuery.each(sorted_data, function(i, metric) {\n var id = metric.id;\n var name = metric.name;\n var units = metric.units;\n if(node_charts[id] == undefined) {\n // add a div the container\n var div_name = id + \"_node_chart\";\n var str = \n \"
\" +\n \"
\" + name + \"
\" +\n \"
\" +\n \"
\";\n container.append(str);\n var div = $(container.children(\"#\" + div_name)).children(\".chart\")[0]; \n node_charts[id] = {\n div: div,\n options: { title: name,\n //isStacked: true,\n enableInteractivity: 0,\n vAxis: {title: (units + \" per Second\"),\n titleTextStyle: {color: 'red'},\n minValue: 0,\n maxValue: 10,\n viewWindow: {min: 0}\n },\n hAxis: {title: 'Time (seconds)'}},\n chart: new google.visualization.LineChart(div),\n }\n }\n if(metric.record.length > 0) {\n // Update the chart\n var chart_info = node_charts[id];\n chart_info.data = tensor_to_table(metric.record);\n chart_info.chart.draw(chart_info.data, chart_info.options);\n }\n });\n\n}\n\n\n\n\nfunction initiate_graph_builder() {\n jQuery.getJSON(\"graph_builder.json\", process_graph_builder)\n .error(function() { \n console.log(\"Unable to access graph_builder will try again.\");})\n .complete(function() {\n setTimeout(initiate_graph_builder, update_interval);\n });\n}\n\n\n\n\n\nvar graph_builder_charts = [];\n\nfunction process_graph_builder(data) {\n// console.log(data); \n console.log(\"Processing graph builder info.\");\n\n $(\"#graph_builder_phase_name\").text(data.phase_name);\n\n\n // Render all the current values\n var container = $(\"#graph_builder_info\");\n var sorted_data = data.sys_metrics.sort(function(a,b) { return a.id - b.id; });\n \n\n // Build an array of divs one for each metric with the name and value\n jQuery.each(sorted_data, function(i, metric) {\n var id = metric.id;\n var name = metric.label;\n var units = metric.units;\n if(graph_builder_charts[id] == undefined) {\n var div_name = id + \"_graph_builder_chart\";\n var str = \n \"
\" +\n \"
\" + name + \"
\" +\n \"
\" +\n \"
\";\n container.append(str);\n var div = $(container.children(\"#\" + div_name)).children(\".chart\")[0]; \n graph_builder_charts[id] = {\n div: div,\n options: { title: name,\n vAxis: {title: (units + \" per Second\"),\n titleTextStyle: {color: 'red'},\n minValue: 0,\n maxValue: 10,\n viewWindow: {min: 0}\n },\n hAxis: {title: 'Time (seconds)'}},\n chart: new google.visualization.AreaChart(div),\n }\n } \n if(metric.values.length > 0) {\n // Update the chart\n var chart_info = graph_builder_charts[id];\n chart_info.data = google.visualization.arrayToDataTable(metric.values);\n chart_info.chart.draw(chart_info.data, chart_info.options);\n }\n });\n\n \n\n}\n\n\n\n\n\n\n" }, { "path": "src/graphlab/ui/intel_demo/index.html", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\nGraphLab Dashboard \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n

GraphLab Visualization Dashboard

\n\n\n\n\n
\n
\n \n
\n
\n\n
\n
\n
\n
\n
\n
\n
\n
\n\n\n\n\n\n\n\n" }, { "path": "src/graphlab/ui/intel_demo/style.css", "content": "body {\n text-align: center;\n}\n#instrument_panel {\n text-align: center;\n}\n\n.metric_summary {\n display: inline-block;\n \n}\n\n.metric_summary .name {\n display: none;\n text-align: center;\n}\n\n.metric_summary .value {\n display: none;\n text-align: center;\n}\n\n\n.aggregate {\n display: inline-block;\n}\n\n.aggregate .chart {\n width: 400px;\n height: 200px;\n}\n\n.aggregate .name {\n display: none\n}\n\n\n.aggregate .chart {\n width: 400px;\n height: 200px;\n}\n\n.node {\n display: inline-block;\n}\n\n\n.node .name {\n display: none\n}\n\n\n.node .chart {\n width: 400px;\n height: 200px;\n}\n\n\n.graph_builder {\n display: inline-block;\n}\n\n.graph_builder .chart {\n width: 400px;\n height: 200px;\n}\n\n.graph_builder .name {\n display: none\n}\n\n\n.graph_builder .chart {\n width: 400px;\n height: 200px;\n}\n" }, { "path": "src/graphlab/ui/metrics_server.cpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n#include \n#include \n\n#include \n#include \n\n#include \n\n\nnamespace graphlab {\n\n\nstatic mg_context* metric_context = NULL;\n\nstatic rwlock& callback_lock() {\n static rwlock clock;\n return clock;\n}\n\n\nstatic std::map& callbacks() {\n static std::map cback;\n return cback;\n}\n\n\n\n\nstatic void* process_request(enum mg_event event,\n struct mg_connection* conn,\n const struct mg_request_info* info) {\n if (event == MG_NEW_REQUEST) {\n\n // get the URL being requested\n std::string url;\n if (info->uri != NULL) url = info->uri;\n // strip the starting /\n if (url.length() >= 1) url = url.substr(1, url.length() - 1);\n // get all the variables\n std::map variable_map;\n if (info->query_string != NULL) {\n std::string qs = info->query_string;\n std::vector terms = strsplit(qs, \"&\", true);\n // now for each term..\n foreach(std::string& term, terms) {\n // get the variable name\n std::vector key_val = strsplit(term, \"=\", true);\n if (key_val.size() > 0) {\n // use mg_get_var to read the actual variable.\n // since mg_get_var does http escape sequence decoding\n std::string key = key_val[0];\n char val_target[8192];\n int ret = mg_get_var(qs.c_str(), qs.length(), \n key.c_str(), val_target, 8192);\n if (ret >= 0) variable_map[key] = val_target;\n }\n }\n }\n callback_lock().readlock();\n // now redirect to the callback handlers. if we find one\n std::map::iterator iter = \n callbacks().find(url);\n\n if (iter != callbacks().end()) {\n std::pair returnval = iter->second(variable_map);\n\n callback_lock().rdunlock();\n\n std::string ctype = returnval.first;\n std::string body = returnval.second;\n mg_printf(conn,\n \"HTTP/1.1 200 OK\\r\\n\"\n \"Access-Control-Allow-Origin: *\\r\\n\"\n \"Access-Control-Allow-Methods: GET\\r\\n\"\n \"Content-Type: %s\\r\\n\"\n \"Content-Length: %d\\r\\n\" \n \"\\r\\n\",\n ctype.c_str(), (int) body.length());\n mg_write(conn, body.c_str(), body.length());\n }\n else {\n std::map::iterator iter404 =\n callbacks().find(\"404\");\n std::pair returnval;\n if (iter404 != callbacks().end()) returnval = iter404->second(variable_map);\n \n callback_lock().rdunlock();\n\n std::string ctype = returnval.first;\n std::string body = returnval.second;\n\n mg_printf(conn,\n \"HTTP/1.1 404 Not Found\\r\\n\"\n \"Access-Control-Allow-Origin: *\\r\\n\"\n \"Content-Type: %s\\r\\n\"\n \"Content-Length: %d\\r\\n\" \n \"\\r\\n\",\n ctype.c_str(), (int)body.length());\n mg_write(conn, body.c_str(), body.length());\n }\n\n return (void*)\"\";\n }\n else {\n return NULL;\n }\n}\n\n\n/*\n Simple 404 handler. Just reuturns a string \"Page Not Found\"\n */\nstd::pair \nfour_oh_four(std::map& varmap) {\n return std::make_pair(std::string(\"text/html\"), \n std::string(\"Page Not Found\"));\n}\n\n\n/*\n Echo handler. Returns a html with get keys and values\n */\nstd::pair \necho(std::map& varmap) {\n std::stringstream ret;\n std::map::iterator iter = varmap.begin();\n ret << \"\\n\";\n while (iter != varmap.end()) {\n ret << iter->first << \" = \" << iter->second << \"
\\n\"; \n ++iter;\n }\n ret << \"\\n\";\n ret.flush();\n return std::make_pair(std::string(\"text/html\"), ret.str());\n}\n\nstd::pair \nindex_page(std::map& varmap) {\n std::stringstream ret;\n ret << \"\\n\";\n ret << \"

Registered Handlers:

\\n\";\n callback_lock().readlock();\n std::map::const_iterator iter = \n callbacks().begin();\n while (iter != callbacks().end()) {\n // don't put in the index page callback\n if (iter->first != \"\") {\n ret << iter->first << \"
\\n\";\n }\n ++iter;\n }\n callback_lock().rdunlock();\n ret << \"\\n\";\n ret.flush();\n return std::make_pair(std::string(\"text/html\"), ret.str());\n}\n\n\nstatic void fill_builtin_callbacks() {\n callbacks()[\"404\"] = four_oh_four;\n callbacks()[\"echo\"] = echo;\n callbacks()[\"\"] = index_page;\n callbacks()[\"index.html\"] = index_page;\n}\n\n\nvoid add_metric_server_callback(std::string page, \n http_redirect_callback_type callback) {\n callback_lock().writelock();\n callbacks()[page] = callback;\n callback_lock().wrunlock();\n}\n\nvoid launch_metric_server() {\n if (distributed_control::get_instance_procid() == 0) {\n const char *options[] = {\"listening_ports\", \"8090\", NULL};\n metric_context = mg_start(process_request, (void*)(&(callbacks())), options);\n if(metric_context == NULL) {\n logstream(LOG_ERROR) << \"Unable to launch metrics server on port 8090. \"\n << \"Metrics server will not be available\" << std::endl;\n return;\n }\n fill_builtin_callbacks();\n\n char hostname[1024];\n std::string strhostname;\n if (gethostname(hostname, 1024) == 0) strhostname = hostname;\n logstream(LOG_EMPH) << \"Metrics server now listening on \" \n << \"http://\" << strhostname << \":8090\" << std::endl;\n }\n}\n\nvoid stop_metric_server() {\n if (distributed_control::get_instance_procid() == 0 && metric_context != NULL) {\n std::cout << \"Metrics server stopping.\" << std::endl;\n mg_stop(metric_context);\n }\n}\n\nvoid stop_metric_server_on_eof() {\n if (distributed_control::get_instance_procid() == 0 && metric_context != NULL) {\n char buff[128];\n // wait for ctrl-d\n logstream(LOG_EMPH) << \"Hit Ctrl-D to stop the metrics server\" << std::endl;\n while (fgets(buff, 128, stdin) != NULL );\n stop_metric_server(); \n }\n}\n\n} // namespace graphlab\n" }, { "path": "src/graphlab/ui/metrics_server.hpp", "content": "/* \n * Copyright (c) 2009 Carnegie Mellon University. \n * All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an \"AS\n * IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n * express or implied. See the License for the specific language\n * governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n * http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_METRICS_SERVER_HPP\n#define GRAPHLAB_METRICS_SERVER_HPP\n#include \n#include \n#include \n#include \n\n\nnamespace graphlab {\n\n\n\n/** \n \\ingroup httpserver\n The callback type used for add_metric_server_callback()\n See add_metric_server_callback() for details.\n */\ntypedef boost::function\n (std::map&)> \n http_redirect_callback_type;\n\n\n\n\n/**\n\n \\ingroup httpserver\n \\brief This is used to map a URL on the mtrics server \n to a processing function.\n \n The processing function must have the prototype\n \\code\n std::pair callback(std::map&)\n \\endcode\n\n The processing function takes a map of GET variables to their corresponding\n values, and returns a pair of strings. (content_type, content)\n \\li \\c content type is the http content_type header. For instance text/plain\n or text/html.\n \\li \\c content is the actual body\n\n For instance: The builtin 404 handler looks like this:\n\n \\code\n std::pair \n four_oh_four(std::map& varmap) {\n return std::make_pair(std::string(\"text/html\"), \n std::string(\"Page Not Found\"));\n }\n \\endcode\n\n \\note The callbacks are only processed on machine 0 since only machine 0\n launches the server.\n\n \\param page The page to map. For instance page = \"a.html\"\n will be shown on http://[server]/a.html\n \\param callback The callback function to use to process the page\n */\nvoid add_metric_server_callback(std::string page, \n http_redirect_callback_type callback);\n\n\n/**\n \\ingroup httpserver\n \\brief Starts the metrics reporting server.\n\n The function may be called by all machines simultaneously since it only\n does useful work on machine 0. Only machine 0 will launch the web server.\n */\nvoid launch_metric_server();\n\n\n\n/**\n \\ingroup httpserver\n \\brief Stops the metrics reporting server if one is started.\n\n The function may be called by all machines simultaneously since it only\n does useful work on machine 0.\n */\nvoid stop_metric_server();\n\n/**\n \\ingroup httpserver\n \\brief Waits for a ctrl-D on machine 0, and \n stops the metrics reporting server if one is started.\n\n The function may be called by all machines simultaneously since it only\n does useful work on machine 0. It waits for the stdin stream to close\n (when the user hit ctrl-d), then shuts down the server.\n*/\nvoid stop_metric_server_on_eof();\n\n} // graphlab \n#endif // GRAPHLAB_METRICS_SERVER_HPP\n" }, { "path": "src/graphlab/ui/mongoose/CMakeLists.txt", "content": "project(GraphLab)\n\n" }, { "path": "src/graphlab/ui/mongoose/LICENSE", "content": "Copyright (c) 2004-2010 Sergey Lyubka\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n" }, { "path": "src/graphlab/ui/mongoose/mongoose.cpp", "content": "// Copyright (c) 2004-2011 Sergey Lyubka\n//\n// Permission is hereby granted, free of charge, to any person obtaining a copy\n// of this software and associated documentation files (the \"Software\"), to deal\n// in the Software without restriction, including without limitation the rights\n// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n// copies of the Software, and to permit persons to whom the Software is\n// furnished to do so, subject to the following conditions:\n//\n// The above copyright notice and this permission notice shall be included in\n// all copies or substantial portions of the Software.\n//\n// THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\n// THE SOFTWARE.\n\n#define NO_SSL\n#define NO_CGI\n\n#if defined(_WIN32)\n#define _CRT_SECURE_NO_WARNINGS // Disable deprecation warning in VS2005\n#else\n#define _XOPEN_SOURCE 600 // For flockfile() on Linux\n#define _LARGEFILE_SOURCE // Enable 64-bit file offsets\n#define __STDC_FORMAT_MACROS // wants this for C++\n#define __STDC_LIMIT_MACROS // C++ wants that for INT64_MAX\n#endif\n\n#if defined(__SYMBIAN32__)\n#define NO_SSL // SSL is not supported\n#define NO_CGI // CGI is not supported\n#define PATH_MAX FILENAME_MAX\n#endif // __SYMBIAN32__\n\n#ifndef _WIN32_WCE // Some ANSI #includes are not available on Windows CE\n#include \n#include \n#include \n#include \n#include \n#endif // !_WIN32_WCE\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#if defined(_WIN32) && !defined(__SYMBIAN32__) // Windows specific\n#define _WIN32_WINNT 0x0400 // To make it link in VS2005\n#include \n\n#ifndef PATH_MAX\n#define PATH_MAX MAX_PATH\n#endif\n\n#ifndef _WIN32_WCE\n#include \n#include \n#include \n#else // _WIN32_WCE\n#include \n#define NO_CGI // WinCE has no pipes\n\ntypedef long off_t;\n#define BUFSIZ 4096\n\n#define errno GetLastError()\n#define strerror(x) _ultoa(x, (char *) _alloca(sizeof(x) *3 ), 10)\n#endif // _WIN32_WCE\n\n#define MAKEUQUAD(lo, hi) ((uint64_t)(((uint32_t)(lo)) | \\\n ((uint64_t)((uint32_t)(hi))) << 32))\n#define RATE_DIFF 10000000 // 100 nsecs\n#define EPOCH_DIFF MAKEUQUAD(0xd53e8000, 0x019db1de)\n#define SYS2UNIX_TIME(lo, hi) \\\n (time_t) ((MAKEUQUAD((lo), (hi)) - EPOCH_DIFF) / RATE_DIFF)\n\n// Visual Studio 6 does not know __func__ or __FUNCTION__\n// The rest of MS compilers use __FUNCTION__, not C99 __func__\n// Also use _strtoui64 on modern M$ compilers\n#if defined(_MSC_VER) && _MSC_VER < 1300\n#define STRX(x) #x\n#define STR(x) STRX(x)\n#define __func__ \"line \" STR(__LINE__)\n#define strtoull(x, y, z) strtoul(x, y, z)\n#define strtoll(x, y, z) strtol(x, y, z)\n#else\n#define __func__ __FUNCTION__\n#define strtoull(x, y, z) _strtoui64(x, y, z)\n#define strtoll(x, y, z) _strtoi64(x, y, z)\n#endif // _MSC_VER\n\n#define ERRNO GetLastError()\n#define NO_SOCKLEN_T\n#define SSL_LIB \"ssleay32.dll\"\n#define CRYPTO_LIB \"libeay32.dll\"\n#define DIRSEP '\\\\'\n#define IS_DIRSEP_CHAR(c) ((c) == '/' || (c) == '\\\\')\n#define O_NONBLOCK 0\n#if !defined(EWOULDBLOCK)\n#define EWOULDBLOCK WSAEWOULDBLOCK\n#endif // !EWOULDBLOCK\n#define _POSIX_\n#define INT64_FMT \"I64d\"\n\n#define WINCDECL __cdecl\n#define SHUT_WR 1\n#define snprintf _snprintf\n#define vsnprintf _vsnprintf\n#define sleep(x) Sleep((x) * 1000)\n\n#define pipe(x) _pipe(x, BUFSIZ, _O_BINARY)\n#define popen(x, y) _popen(x, y)\n#define pclose(x) _pclose(x)\n#define close(x) _close(x)\n#define dlsym(x,y) GetProcAddress((HINSTANCE) (x), (y))\n#define RTLD_LAZY 0\n#define fseeko(x, y, z) fseek((x), (y), (z))\n#define fdopen(x, y) _fdopen((x), (y))\n#define write(x, y, z) _write((x), (y), (unsigned) z)\n#define read(x, y, z) _read((x), (y), (unsigned) z)\n#define flockfile(x) EnterCriticalSection(&global_log_file_lock)\n#define funlockfile(x) LeaveCriticalSection(&global_log_file_lock)\n\n#if !defined(fileno)\n#define fileno(x) _fileno(x)\n#endif // !fileno MINGW #defines fileno\n\ntypedef HANDLE pthread_mutex_t;\ntypedef struct {HANDLE signal, broadcast;} pthread_cond_t;\ntypedef DWORD pthread_t;\n#define pid_t HANDLE // MINGW typedefs pid_t to int. Using #define here.\n\nstruct timespec {\n long tv_nsec;\n long tv_sec;\n};\n\nstatic int pthread_mutex_lock(pthread_mutex_t *);\nstatic int pthread_mutex_unlock(pthread_mutex_t *);\nstatic FILE *mg_fopen(const char *path, const char *mode);\n\n#if defined(HAVE_STDINT)\n#include \n#else\ntypedef unsigned int uint32_t;\ntypedef unsigned short uint16_t;\ntypedef unsigned __int64 uint64_t;\ntypedef __int64 int64_t;\n#define INT64_MAX 9223372036854775807\n#endif // HAVE_STDINT\n\n// POSIX dirent interface\nstruct dirent {\n char d_name[PATH_MAX];\n};\n\ntypedef struct DIR {\n HANDLE handle;\n WIN32_FIND_DATAW info;\n struct dirent result;\n} DIR;\n\n#else // UNIX specific\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n#include \n#include \n#if defined(__MACH__)\n#define SSL_LIB \"libssl.dylib\"\n#define CRYPTO_LIB \"libcrypto.dylib\"\n#else\n#if !defined(SSL_LIB)\n#define SSL_LIB \"libssl.so\"\n#endif\n#if !defined(CRYPTO_LIB)\n#define CRYPTO_LIB \"libcrypto.so\"\n#endif\n#endif\n#define DIRSEP '/'\n#define IS_DIRSEP_CHAR(c) ((c) == '/')\n#ifndef O_BINARY\n#define O_BINARY 0\n#endif // O_BINARY\n#define closesocket(a) close(a)\n#define mg_fopen(x, y) fopen(x, y)\n#define mg_mkdir(x, y) mkdir(x, y)\n#define mg_remove(x) remove(x)\n#define mg_rename(x, y) rename(x, y)\n#define ERRNO errno\n#define INVALID_SOCKET (-1)\n#define INT64_FMT PRId64\ntypedef int SOCKET;\n#define WINCDECL\n\n#endif // End of Windows and UNIX specific includes\n\n#include \"mongoose.h\"\n\n#define MONGOOSE_VERSION \"3.1\"\n#define PASSWORDS_FILE_NAME \".htpasswd\"\n#define CGI_ENVIRONMENT_SIZE 4096\n#define MAX_CGI_ENVIR_VARS 64\n#define ARRAY_SIZE(array) (sizeof(array) / sizeof(array[0]))\n\n#ifdef _WIN32\nstatic CRITICAL_SECTION global_log_file_lock;\nstatic pthread_t pthread_self(void) {\n return GetCurrentThreadId();\n}\n#endif // _WIN32\n\n#if defined(DEBUG)\n#define DEBUG_TRACE(x) do { \\\n flockfile(stdout); \\\n printf(\"*** %lu.%p.%s.%d: \", \\\n (unsigned long) time(NULL), (void *) pthread_self(), \\\n __func__, __LINE__); \\\n printf x; \\\n putchar('\\n'); \\\n fflush(stdout); \\\n funlockfile(stdout); \\\n} while (0)\n#else\n#define DEBUG_TRACE(x)\n#endif // DEBUG\n\n// Darwin prior to 7.0 and Win32 do not have socklen_t\n#ifdef NO_SOCKLEN_T\ntypedef int socklen_t;\n#endif // NO_SOCKLEN_T\n\n#if !defined(MSG_NOSIGNAL)\n#define MSG_NOSIGNAL 0\n#endif\n\ntypedef void * (*mg_thread_func_t)(void *);\n\nstatic const char *http_500_error = \"Internal Server Error\";\n\n// Snatched from OpenSSL includes. I put the prototypes here to be independent\n// from the OpenSSL source installation. Having this, mongoose + SSL can be\n// built on any system with binary SSL libraries installed.\ntypedef struct ssl_st SSL;\ntypedef struct ssl_method_st SSL_METHOD;\ntypedef struct ssl_ctx_st SSL_CTX;\n\n#define SSL_ERROR_WANT_READ 2\n#define SSL_ERROR_WANT_WRITE 3\n#define SSL_FILETYPE_PEM 1\n#define CRYPTO_LOCK 1\n\n#if defined(NO_SSL_DL)\nextern void SSL_free(SSL *);\nextern int SSL_accept(SSL *);\nextern int SSL_connect(SSL *);\nextern int SSL_read(SSL *, void *, int);\nextern int SSL_write(SSL *, const void *, int);\nextern int SSL_get_error(const SSL *, int);\nextern int SSL_set_fd(SSL *, int);\nextern SSL *SSL_new(SSL_CTX *);\nextern SSL_CTX *SSL_CTX_new(SSL_METHOD *);\nextern SSL_METHOD *SSLv23_server_method(void);\nextern int SSL_library_init(void);\nextern void SSL_load_error_strings(void);\nextern int SSL_CTX_use_PrivateKey_file(SSL_CTX *, const char *, int);\nextern int SSL_CTX_use_certificate_file(SSL_CTX *, const char *, int);\nextern int SSL_CTX_use_certificate_chain_file(SSL_CTX *, const char *);\nextern void SSL_CTX_set_default_passwd_cb(SSL_CTX *, mg_callback_t);\nextern void SSL_CTX_free(SSL_CTX *);\nextern unsigned long ERR_get_error(void);\nextern char *ERR_error_string(unsigned long, char *);\nextern int CRYPTO_num_locks(void);\nextern void CRYPTO_set_locking_callback(void (*)(int, int, const char *, int));\nextern void CRYPTO_set_id_callback(unsigned long (*)(void));\n#else\n// Dynamically loaded SSL functionality\nstruct ssl_func {\n const char *name; // SSL function name\n void (*ptr)(void); // Function pointer\n};\n\n#define SSL_free (* (void (*)(SSL *)) ssl_sw[0].ptr)\n#define SSL_accept (* (int (*)(SSL *)) ssl_sw[1].ptr)\n#define SSL_connect (* (int (*)(SSL *)) ssl_sw[2].ptr)\n#define SSL_read (* (int (*)(SSL *, void *, int)) ssl_sw[3].ptr)\n#define SSL_write (* (int (*)(SSL *, const void *,int)) ssl_sw[4].ptr)\n#define SSL_get_error (* (int (*)(SSL *, int)) ssl_sw[5].ptr)\n#define SSL_set_fd (* (int (*)(SSL *, SOCKET)) ssl_sw[6].ptr)\n#define SSL_new (* (SSL * (*)(SSL_CTX *)) ssl_sw[7].ptr)\n#define SSL_CTX_new (* (SSL_CTX * (*)(SSL_METHOD *)) ssl_sw[8].ptr)\n#define SSLv23_server_method (* (SSL_METHOD * (*)(void)) ssl_sw[9].ptr)\n#define SSL_library_init (* (int (*)(void)) ssl_sw[10].ptr)\n#define SSL_CTX_use_PrivateKey_file (* (int (*)(SSL_CTX *, \\\n const char *, int)) ssl_sw[11].ptr)\n#define SSL_CTX_use_certificate_file (* (int (*)(SSL_CTX *, \\\n const char *, int)) ssl_sw[12].ptr)\n#define SSL_CTX_set_default_passwd_cb \\\n (* (void (*)(SSL_CTX *, mg_callback_t)) ssl_sw[13].ptr)\n#define SSL_CTX_free (* (void (*)(SSL_CTX *)) ssl_sw[14].ptr)\n#define SSL_load_error_strings (* (void (*)(void)) ssl_sw[15].ptr)\n#define SSL_CTX_use_certificate_chain_file \\\n (* (int (*)(SSL_CTX *, const char *)) ssl_sw[16].ptr)\n\n#define CRYPTO_num_locks (* (int (*)(void)) crypto_sw[0].ptr)\n#define CRYPTO_set_locking_callback \\\n (* (void (*)(void (*)(int, int, const char *, int))) crypto_sw[1].ptr)\n#define CRYPTO_set_id_callback \\\n (* (void (*)(unsigned long (*)(void))) crypto_sw[2].ptr)\n#define ERR_get_error (* (unsigned long (*)(void)) crypto_sw[3].ptr)\n#define ERR_error_string (* (char * (*)(unsigned long,char *)) crypto_sw[4].ptr)\n\n// set_ssl_option() function updates this array.\n// It loads SSL library dynamically and changes NULLs to the actual addresses\n// of respective functions. The macros above (like SSL_connect()) are really\n// just calling these functions indirectly via the pointer.\nstatic struct ssl_func ssl_sw[] = {\n {\"SSL_free\", NULL},\n {\"SSL_accept\", NULL},\n {\"SSL_connect\", NULL},\n {\"SSL_read\", NULL},\n {\"SSL_write\", NULL},\n {\"SSL_get_error\", NULL},\n {\"SSL_set_fd\", NULL},\n {\"SSL_new\", NULL},\n {\"SSL_CTX_new\", NULL},\n {\"SSLv23_server_method\", NULL},\n {\"SSL_library_init\", NULL},\n {\"SSL_CTX_use_PrivateKey_file\", NULL},\n {\"SSL_CTX_use_certificate_file\",NULL},\n {\"SSL_CTX_set_default_passwd_cb\",NULL},\n {\"SSL_CTX_free\", NULL},\n {\"SSL_load_error_strings\", NULL},\n {\"SSL_CTX_use_certificate_chain_file\", NULL},\n {NULL, NULL}\n};\n\n#ifndef NO_SSL\n// Similar array as ssl_sw. These functions could be located in different lib.\nstatic struct ssl_func crypto_sw[] = {\n {\"CRYPTO_num_locks\", NULL},\n {\"CRYPTO_set_locking_callback\", NULL},\n {\"CRYPTO_set_id_callback\", NULL},\n {\"ERR_get_error\", NULL},\n {\"ERR_error_string\", NULL},\n {NULL, NULL}\n};\n#endif\n#endif // NO_SSL_DL\n\nstatic const char *month_names[] = {\n \"Jan\", \"Feb\", \"Mar\", \"Apr\", \"May\", \"Jun\",\n \"Jul\", \"Aug\", \"Sep\", \"Oct\", \"Nov\", \"Dec\"\n};\n\n// Unified socket address. For IPv6 support, add IPv6 address structure\n// in the union u.\nunion usa {\n struct sockaddr sa;\n struct sockaddr_in sin;\n#if defined(USE_IPV6)\n struct sockaddr_in6 sin6;\n#endif\n};\n\n// Describes a string (chunk of memory).\nstruct vec {\n const char *ptr;\n size_t len;\n};\n\n// Structure used by mg_stat() function. Uses 64 bit file length.\nstruct mgstat {\n int is_directory; // Directory marker\n int64_t size; // File size\n time_t mtime; // Modification time\n};\n\n// Describes listening socket, or socket which was accept()-ed by the master\n// thread and queued for future handling by the worker thread.\nstruct socket {\n struct socket *next; // Linkage\n SOCKET sock; // Listening socket\n union usa lsa; // Local socket address\n union usa rsa; // Remote socket address\n int is_ssl; // Is socket SSL-ed\n};\n\nenum {\n CGI_EXTENSIONS, CGI_ENVIRONMENT, PUT_DELETE_PASSWORDS_FILE, CGI_INTERPRETER,\n PROTECT_URI, AUTHENTICATION_DOMAIN, SSI_EXTENSIONS, ACCESS_LOG_FILE,\n SSL_CHAIN_FILE, ENABLE_DIRECTORY_LISTING, ERROR_LOG_FILE,\n GLOBAL_PASSWORDS_FILE, INDEX_FILES,\n ENABLE_KEEP_ALIVE, ACCESS_CONTROL_LIST, MAX_REQUEST_SIZE,\n EXTRA_MIME_TYPES, LISTENING_PORTS,\n DOCUMENT_ROOT, SSL_CERTIFICATE, NUM_THREADS, RUN_AS_USER, REWRITE,\n NUM_OPTIONS\n};\n\nstatic const char *config_options[] = {\n \"C\", \"cgi_pattern\", \"**.cgi$|**.pl$|**.php$\",\n \"E\", \"cgi_environment\", NULL,\n \"G\", \"put_delete_passwords_file\", NULL,\n \"I\", \"cgi_interpreter\", NULL,\n \"P\", \"protect_uri\", NULL,\n \"R\", \"authentication_domain\", \"mydomain.com\",\n \"S\", \"ssi_pattern\", \"**.shtml$|**.shtm$\",\n \"a\", \"access_log_file\", NULL,\n \"c\", \"ssl_chain_file\", NULL,\n \"d\", \"enable_directory_listing\", \"yes\",\n \"e\", \"error_log_file\", NULL,\n \"g\", \"global_passwords_file\", NULL,\n \"i\", \"index_files\", \"index.html,index.htm,index.cgi\",\n \"k\", \"enable_keep_alive\", \"yes\",\n \"l\", \"access_control_list\", NULL,\n \"M\", \"max_request_size\", \"16384\",\n \"m\", \"extra_mime_types\", NULL,\n \"p\", \"listening_ports\", \"8080\",\n \"r\", \"document_root\", \".\",\n \"s\", \"ssl_certificate\", NULL,\n \"t\", \"num_threads\", \"10\",\n \"u\", \"run_as_user\", NULL,\n \"w\", \"url_rewrite_patterns\", NULL,\n NULL\n};\n#define ENTRIES_PER_CONFIG_OPTION 3\n\nstruct mg_context {\n volatile int stop_flag; // Should we stop event loop\n SSL_CTX *ssl_ctx; // SSL context\n char *config[NUM_OPTIONS]; // Mongoose configuration parameters\n mg_callback_t user_callback; // User-defined callback function\n void *user_data; // User-defined data\n\n struct socket *listening_sockets;\n\n volatile int num_threads; // Number of threads\n pthread_mutex_t mutex; // Protects (max|num)_threads\n pthread_cond_t cond; // Condvar for tracking workers terminations\n\n struct socket queue[20]; // Accepted sockets\n volatile int sq_head; // Head of the socket queue\n volatile int sq_tail; // Tail of the socket queue\n pthread_cond_t sq_full; // Singaled when socket is produced\n pthread_cond_t sq_empty; // Signaled when socket is consumed\n};\n\nstruct mg_connection {\n struct mg_request_info request_info;\n struct mg_context *ctx;\n SSL *ssl; // SSL descriptor\n struct socket client; // Connected client\n time_t birth_time; // Time connection was accepted\n int64_t num_bytes_sent; // Total bytes sent to client\n int64_t content_len; // Content-Length header value\n int64_t consumed_content; // How many bytes of content is already read\n char *buf; // Buffer for received data\n char *path_info; // PATH_INFO part of the URL\n int must_close; // 1 if connection must be closed\n int buf_size; // Buffer size\n int request_len; // Size of the request + headers in a buffer\n int data_len; // Total size of data in a buffer\n};\n\nconst char **mg_get_valid_option_names(void) {\n return config_options;\n}\n\nstatic void *call_user(struct mg_connection *conn, enum mg_event event) {\n conn->request_info.user_data = conn->ctx->user_data;\n return conn->ctx->user_callback == NULL ? NULL :\n conn->ctx->user_callback(event, conn, &conn->request_info);\n}\n\nstatic int get_option_index(const char *name) {\n int i;\n\n for (i = 0; config_options[i] != NULL; i += ENTRIES_PER_CONFIG_OPTION) {\n if (strcmp(config_options[i], name) == 0 ||\n strcmp(config_options[i + 1], name) == 0) {\n return i / ENTRIES_PER_CONFIG_OPTION;\n }\n }\n return -1;\n}\n\nconst char *mg_get_option(const struct mg_context *ctx, const char *name) {\n int i;\n if ((i = get_option_index(name)) == -1) {\n return NULL;\n } else if (ctx->config[i] == NULL) {\n return \"\";\n } else {\n return ctx->config[i];\n }\n}\n\nstatic void sockaddr_to_string(char *buf, size_t len,\n const union usa *usa) {\n buf[0] = '\\0';\n#if defined(USE_IPV6)\n inet_ntop(usa->sa.sa_family, usa->sa.sa_family == AF_INET ?\n (void *) &usa->sin.sin_addr :\n (void *) &usa->sin6.sin6_addr, buf, len);\n#elif defined(_WIN32)\n // Only Windoze Vista (and newer) have inet_ntop()\n strncpy(buf, inet_ntoa(usa->sin.sin_addr), len);\n#else\n inet_ntop(usa->sa.sa_family, (void *) &usa->sin.sin_addr, buf, len);\n#endif\n}\n\n// Print error message to the opened error log stream.\nstatic void cry(struct mg_connection *conn, const char *fmt, ...) {\n char buf[BUFSIZ], src_addr[20];\n va_list ap;\n FILE *fp;\n time_t timestamp;\n\n va_start(ap, fmt);\n (void) vsnprintf(buf, sizeof(buf), fmt, ap);\n va_end(ap);\n\n // Do not lock when getting the callback value, here and below.\n // I suppose this is fine, since function cannot disappear in the\n // same way string option can.\n conn->request_info.log_message = buf;\n if (call_user(conn, MG_EVENT_LOG) == NULL) {\n fp = conn->ctx->config[ERROR_LOG_FILE] == NULL ? NULL :\n mg_fopen(conn->ctx->config[ERROR_LOG_FILE], \"a+\");\n\n if (fp != NULL) {\n flockfile(fp);\n timestamp = time(NULL);\n\n sockaddr_to_string(src_addr, sizeof(src_addr), &conn->client.rsa);\n fprintf(fp, \"[%010lu] [error] [client %s] \", (unsigned long) timestamp,\n src_addr);\n\n if (conn->request_info.request_method != NULL) {\n fprintf(fp, \"%s %s: \", conn->request_info.request_method,\n conn->request_info.uri);\n }\n\n (void) fprintf(fp, \"%s\", buf);\n fputc('\\n', fp);\n funlockfile(fp);\n if (fp != stderr) {\n fclose(fp);\n }\n }\n }\n conn->request_info.log_message = NULL;\n}\n\n#ifndef NO_SSL\n// Return OpenSSL error message\nstatic const char *ssl_error(void) {\n unsigned long err;\n err = ERR_get_error();\n return err == 0 ? \"\" : ERR_error_string(err, NULL);\n}\n#endif\n\n// Return fake connection structure. Used for logging, if connection\n// is not applicable at the moment of logging.\nstatic struct mg_connection *fc(struct mg_context *ctx) {\n static struct mg_connection fake_connection;\n fake_connection.ctx = ctx;\n return &fake_connection;\n}\n\nconst char *mg_version(void) {\n return MONGOOSE_VERSION;\n}\n\nstatic void mg_strlcpy(register char *dst, register const char *src, size_t n) {\n for (; *src != '\\0' && n > 1; n--) {\n *dst++ = *src++;\n }\n *dst = '\\0';\n}\n\nstatic int lowercase(const char *s) {\n return tolower(* (const unsigned char *) s);\n}\n\nstatic int mg_strncasecmp(const char *s1, const char *s2, size_t len) {\n int diff = 0;\n\n if (len > 0)\n do {\n diff = lowercase(s1++) - lowercase(s2++);\n } while (diff == 0 && s1[-1] != '\\0' && --len > 0);\n\n return diff;\n}\n\nstatic int mg_strcasecmp(const char *s1, const char *s2) {\n int diff;\n\n do {\n diff = lowercase(s1++) - lowercase(s2++);\n } while (diff == 0 && s1[-1] != '\\0');\n\n return diff;\n}\n\nstatic char * mg_strndup(const char *ptr, size_t len) {\n char *p;\n\n if ((p = (char *) malloc(len + 1)) != NULL) {\n mg_strlcpy(p, ptr, len + 1);\n }\n\n return p;\n}\n\nstatic char * mg_strdup(const char *str) {\n return mg_strndup(str, strlen(str));\n}\n\n// Like snprintf(), but never returns negative value, or the value\n// that is larger than a supplied buffer.\n// Thanks to Adam Zeldis to pointing snprintf()-caused vulnerability\n// in his audit report.\nstatic int mg_vsnprintf(struct mg_connection *conn, char *buf, size_t buflen,\n const char *fmt, va_list ap) {\n int n;\n\n if (buflen == 0)\n return 0;\n\n n = vsnprintf(buf, buflen, fmt, ap);\n\n if (n < 0) {\n cry(conn, \"vsnprintf error\");\n n = 0;\n } else if (n >= (int) buflen) {\n cry(conn, \"truncating vsnprintf buffer: [%.*s]\",\n n > 200 ? 200 : n, buf);\n n = (int) buflen - 1;\n }\n buf[n] = '\\0';\n\n return n;\n}\n\nstatic int mg_snprintf(struct mg_connection *conn, char *buf, size_t buflen,\n const char *fmt, ...) {\n va_list ap;\n int n;\n\n va_start(ap, fmt);\n n = mg_vsnprintf(conn, buf, buflen, fmt, ap);\n va_end(ap);\n\n return n;\n}\n\n// Skip the characters until one of the delimiters characters found.\n// 0-terminate resulting word. Skip the delimiter and following whitespaces if any.\n// Advance pointer to buffer to the next word. Return found 0-terminated word.\n// Delimiters can be quoted with quotechar.\nstatic char *skip_quoted(char **buf, const char *delimiters, const char *whitespace, char quotechar) {\n char *p, *begin_word, *end_word, *end_whitespace;\n\n begin_word = *buf;\n end_word = begin_word + strcspn(begin_word, delimiters);\n\n // Check for quotechar\n if (end_word > begin_word) {\n p = end_word - 1;\n while (*p == quotechar) {\n // If there is anything beyond end_word, copy it\n if (*end_word == '\\0') {\n *p = '\\0';\n break;\n } else {\n size_t end_off = strcspn(end_word + 1, delimiters);\n memmove (p, end_word, end_off + 1);\n p += end_off; // p must correspond to end_word - 1\n end_word += end_off + 1;\n }\n }\n for (p++; p < end_word; p++) {\n *p = '\\0';\n }\n }\n\n if (*end_word == '\\0') {\n *buf = end_word;\n } else {\n end_whitespace = end_word + 1 + strspn(end_word + 1, whitespace);\n\n for (p = end_word; p < end_whitespace; p++) {\n *p = '\\0';\n }\n\n *buf = end_whitespace;\n }\n\n return begin_word;\n}\n\n// Simplified version of skip_quoted without quote char\n// and whitespace == delimiters\nstatic char *skip(char **buf, const char *delimiters) {\n return skip_quoted(buf, delimiters, delimiters, 0);\n}\n\n\n// Return HTTP header value, or NULL if not found.\nstatic const char *get_header(const struct mg_request_info *ri,\n const char *name) {\n int i;\n\n for (i = 0; i < ri->num_headers; i++)\n if (!mg_strcasecmp(name, ri->http_headers[i].name))\n return ri->http_headers[i].value;\n\n return NULL;\n}\n\nconst char *mg_get_header(const struct mg_connection *conn, const char *name) {\n return get_header(&conn->request_info, name);\n}\n\n// A helper function for traversing comma separated list of values.\n// It returns a list pointer shifted to the next value, of NULL if the end\n// of the list found.\n// Value is stored in val vector. If value has form \"x=y\", then eq_val\n// vector is initialized to point to the \"y\" part, and val vector length\n// is adjusted to point only to \"x\".\nstatic const char *next_option(const char *list, struct vec *val,\n struct vec *eq_val) {\n if (list == NULL || *list == '\\0') {\n // End of the list\n list = NULL;\n } else {\n val->ptr = list;\n if ((list = strchr(val->ptr, ',')) != NULL) {\n // Comma found. Store length and shift the list ptr\n val->len = list - val->ptr;\n list++;\n } else {\n // This value is the last one\n list = val->ptr + strlen(val->ptr);\n val->len = list - val->ptr;\n }\n\n if (eq_val != NULL) {\n // Value has form \"x=y\", adjust pointers and lengths\n // so that val points to \"x\", and eq_val points to \"y\".\n eq_val->len = 0;\n eq_val->ptr = (const char *) memchr(val->ptr, '=', val->len);\n if (eq_val->ptr != NULL) {\n eq_val->ptr++; // Skip over '=' character\n eq_val->len = val->ptr + val->len - eq_val->ptr;\n val->len = (eq_val->ptr - val->ptr) - 1;\n }\n }\n }\n\n return list;\n}\n\nstatic int match_prefix(const char *pattern, int pattern_len, const char *str) {\n const char *or_str;\n int i, j, len, res;\n\n if ((or_str = (const char *) memchr(pattern, '|', pattern_len)) != NULL) {\n res = match_prefix(pattern, or_str - pattern, str);\n return res > 0 ? res :\n match_prefix(or_str + 1, (pattern + pattern_len) - (or_str + 1), str);\n }\n\n i = j = 0;\n res = -1;\n for (; i < pattern_len; i++, j++) {\n if (pattern[i] == '?' && str[j] != '\\0') {\n continue;\n } else if (pattern[i] == '$') {\n return str[j] == '\\0' ? j : -1;\n } else if (pattern[i] == '*') {\n i++;\n if (pattern[i] == '*') {\n i++;\n len = strlen(str + j);\n } else {\n len = strcspn(str + j, \"/\");\n }\n if (i == pattern_len) {\n return j + len;\n }\n do {\n res = match_prefix(pattern + i, pattern_len - i, str + j + len);\n } while (res == -1 && len-- > 0);\n return res == -1 ? -1 : j + res + len;\n } else if (pattern[i] != str[j]) {\n return -1;\n }\n }\n return j;\n}\n\n// HTTP 1.1 assumes keep alive if \"Connection:\" header is not set\n// This function must tolerate situations when connection info is not\n// set up, for example if request parsing failed.\nstatic int should_keep_alive(const struct mg_connection *conn) {\n const char *http_version = conn->request_info.http_version;\n const char *header = mg_get_header(conn, \"Connection\");\n return (!conn->must_close &&\n !conn->request_info.status_code != 401 &&\n !mg_strcasecmp(conn->ctx->config[ENABLE_KEEP_ALIVE], \"yes\") &&\n (header == NULL && http_version && !strcmp(http_version, \"1.1\"))) ||\n (header != NULL && !mg_strcasecmp(header, \"keep-alive\"));\n}\n\nstatic const char *suggest_connection_header(const struct mg_connection *conn) {\n return should_keep_alive(conn) ? \"keep-alive\" : \"close\";\n}\n\nstatic void send_http_error(struct mg_connection *conn, int status,\n const char *reason, const char *fmt, ...) {\n char buf[BUFSIZ];\n va_list ap;\n int len;\n\n conn->request_info.status_code = status;\n\n if (call_user(conn, MG_HTTP_ERROR) == NULL) {\n buf[0] = '\\0';\n len = 0;\n\n // Errors 1xx, 204 and 304 MUST NOT send a body\n if (status > 199 && status != 204 && status != 304) {\n len = mg_snprintf(conn, buf, sizeof(buf), \"Error %d: %s\", status, reason);\n cry(conn, \"%s\", buf);\n buf[len++] = '\\n';\n\n va_start(ap, fmt);\n len += mg_vsnprintf(conn, buf + len, sizeof(buf) - len, fmt, ap);\n va_end(ap);\n }\n DEBUG_TRACE((\"[%s]\", buf));\n\n mg_printf(conn, \"HTTP/1.1 %d %s\\r\\n\"\n \"Content-Type: text/plain\\r\\n\"\n \"Content-Length: %d\\r\\n\"\n \"Connection: %s\\r\\n\\r\\n\", status, reason, len,\n suggest_connection_header(conn));\n conn->num_bytes_sent += mg_printf(conn, \"%s\", buf);\n }\n}\n\n#if defined(_WIN32) && !defined(__SYMBIAN32__)\nstatic int pthread_mutex_init(pthread_mutex_t *mutex, void *unused) {\n unused = NULL;\n *mutex = CreateMutex(NULL, FALSE, NULL);\n return *mutex == NULL ? -1 : 0;\n}\n\nstatic int pthread_mutex_destroy(pthread_mutex_t *mutex) {\n return CloseHandle(*mutex) == 0 ? -1 : 0;\n}\n\nstatic int pthread_mutex_lock(pthread_mutex_t *mutex) {\n return WaitForSingleObject(*mutex, INFINITE) == WAIT_OBJECT_0? 0 : -1;\n}\n\nstatic int pthread_mutex_unlock(pthread_mutex_t *mutex) {\n return ReleaseMutex(*mutex) == 0 ? -1 : 0;\n}\n\nstatic int pthread_cond_init(pthread_cond_t *cv, const void *unused) {\n unused = NULL;\n cv->signal = CreateEvent(NULL, FALSE, FALSE, NULL);\n cv->broadcast = CreateEvent(NULL, TRUE, FALSE, NULL);\n return cv->signal != NULL && cv->broadcast != NULL ? 0 : -1;\n}\n\nstatic int pthread_cond_wait(pthread_cond_t *cv, pthread_mutex_t *mutex) {\n HANDLE handles[] = {cv->signal, cv->broadcast};\n ReleaseMutex(*mutex);\n WaitForMultipleObjects(2, handles, FALSE, INFINITE);\n return WaitForSingleObject(*mutex, INFINITE) == WAIT_OBJECT_0? 0 : -1;\n}\n\nstatic int pthread_cond_signal(pthread_cond_t *cv) {\n return SetEvent(cv->signal) == 0 ? -1 : 0;\n}\n\nstatic int pthread_cond_broadcast(pthread_cond_t *cv) {\n // Implementation with PulseEvent() has race condition, see\n // http://www.cs.wustl.edu/~schmidt/win32-cv-1.html\n return PulseEvent(cv->broadcast) == 0 ? -1 : 0;\n}\n\nstatic int pthread_cond_destroy(pthread_cond_t *cv) {\n return CloseHandle(cv->signal) && CloseHandle(cv->broadcast) ? 0 : -1;\n}\n\n// For Windows, change all slashes to backslashes in path names.\nstatic void change_slashes_to_backslashes(char *path) {\n int i;\n\n for (i = 0; path[i] != '\\0'; i++) {\n if (path[i] == '/')\n path[i] = '\\\\';\n // i > 0 check is to preserve UNC paths, like \\\\server\\file.txt\n if (path[i] == '\\\\' && i > 0)\n while (path[i + 1] == '\\\\' || path[i + 1] == '/')\n (void) memmove(path + i + 1,\n path + i + 2, strlen(path + i + 1));\n }\n}\n\n// Encode 'path' which is assumed UTF-8 string, into UNICODE string.\n// wbuf and wbuf_len is a target buffer and its length.\nstatic void to_unicode(const char *path, wchar_t *wbuf, size_t wbuf_len) {\n char buf[PATH_MAX], buf2[PATH_MAX], *p;\n\n mg_strlcpy(buf, path, sizeof(buf));\n change_slashes_to_backslashes(buf);\n\n // Point p to the end of the file name\n p = buf + strlen(buf) - 1;\n\n // Trim trailing backslash character\n while (p > buf && *p == '\\\\' && p[-1] != ':') {\n *p-- = '\\0';\n }\n\n // Protect from CGI code disclosure.\n // This is very nasty hole. Windows happily opens files with\n // some garbage in the end of file name. So fopen(\"a.cgi \", \"r\")\n // actually opens \"a.cgi\", and does not return an error!\n if (*p == 0x20 || // No space at the end\n (*p == 0x2e && p > buf) || // No '.' but allow '.' as full path\n *p == 0x2b || // No '+'\n (*p & ~0x7f)) { // And generally no non-ascii chars\n (void) fprintf(stderr, \"Rejecting suspicious path: [%s]\", buf);\n wbuf[0] = L'\\0';\n } else {\n // Convert to Unicode and back. If doubly-converted string does not\n // match the original, something is fishy, reject.\n memset(wbuf, 0, wbuf_len * sizeof(wchar_t));\n MultiByteToWideChar(CP_UTF8, 0, buf, -1, wbuf, (int) wbuf_len);\n WideCharToMultiByte(CP_UTF8, 0, wbuf, (int) wbuf_len, buf2, sizeof(buf2),\n NULL, NULL);\n if (strcmp(buf, buf2) != 0) {\n wbuf[0] = L'\\0';\n }\n }\n}\n\n#if defined(_WIN32_WCE)\nstatic time_t time(time_t *ptime) {\n time_t t;\n SYSTEMTIME st;\n FILETIME ft;\n\n GetSystemTime(&st);\n SystemTimeToFileTime(&st, &ft);\n t = SYS2UNIX_TIME(ft.dwLowDateTime, ft.dwHighDateTime);\n\n if (ptime != NULL) {\n *ptime = t;\n }\n\n return t;\n}\n\nstatic struct tm *localtime(const time_t *ptime, struct tm *ptm) {\n int64_t t = ((int64_t) *ptime) * RATE_DIFF + EPOCH_DIFF;\n FILETIME ft, lft;\n SYSTEMTIME st;\n TIME_ZONE_INFORMATION tzinfo;\n\n if (ptm == NULL) {\n return NULL;\n }\n\n * (int64_t *) &ft = t;\n FileTimeToLocalFileTime(&ft, &lft);\n FileTimeToSystemTime(&lft, &st);\n ptm->tm_year = st.wYear - 1900;\n ptm->tm_mon = st.wMonth - 1;\n ptm->tm_wday = st.wDayOfWeek;\n ptm->tm_mday = st.wDay;\n ptm->tm_hour = st.wHour;\n ptm->tm_min = st.wMinute;\n ptm->tm_sec = st.wSecond;\n ptm->tm_yday = 0; // hope nobody uses this\n ptm->tm_isdst =\n GetTimeZoneInformation(&tzinfo) == TIME_ZONE_ID_DAYLIGHT ? 1 : 0;\n\n return ptm;\n}\n\nstatic struct tm *gmtime(const time_t *ptime, struct tm *ptm) {\n // FIXME(lsm): fix this.\n return localtime(ptime, ptm);\n}\n\nstatic size_t strftime(char *dst, size_t dst_size, const char *fmt,\n const struct tm *tm) {\n (void) snprintf(dst, dst_size, \"implement strftime() for WinCE\");\n return 0;\n}\n#endif\n\nstatic int mg_rename(const char* oldname, const char* newname) {\n wchar_t woldbuf[PATH_MAX];\n wchar_t wnewbuf[PATH_MAX];\n\n to_unicode(oldname, woldbuf, ARRAY_SIZE(woldbuf));\n to_unicode(newname, wnewbuf, ARRAY_SIZE(wnewbuf));\n\n return MoveFileW(woldbuf, wnewbuf) ? 0 : -1;\n}\n\n\nstatic FILE *mg_fopen(const char *path, const char *mode) {\n wchar_t wbuf[PATH_MAX], wmode[20];\n\n to_unicode(path, wbuf, ARRAY_SIZE(wbuf));\n MultiByteToWideChar(CP_UTF8, 0, mode, -1, wmode, ARRAY_SIZE(wmode));\n\n return _wfopen(wbuf, wmode);\n}\n\nstatic int mg_stat(const char *path, struct mgstat *stp) {\n int ok = -1; // Error\n wchar_t wbuf[PATH_MAX];\n WIN32_FILE_ATTRIBUTE_DATA info;\n\n to_unicode(path, wbuf, ARRAY_SIZE(wbuf));\n\n if (GetFileAttributesExW(wbuf, GetFileExInfoStandard, &info) != 0) {\n stp->size = MAKEUQUAD(info.nFileSizeLow, info.nFileSizeHigh);\n stp->mtime = SYS2UNIX_TIME(info.ftLastWriteTime.dwLowDateTime,\n info.ftLastWriteTime.dwHighDateTime);\n stp->is_directory =\n info.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY;\n ok = 0; // Success\n }\n\n return ok;\n}\n\nstatic int mg_remove(const char *path) {\n wchar_t wbuf[PATH_MAX];\n to_unicode(path, wbuf, ARRAY_SIZE(wbuf));\n return DeleteFileW(wbuf) ? 0 : -1;\n}\n\nstatic int mg_mkdir(const char *path, int mode) {\n char buf[PATH_MAX];\n wchar_t wbuf[PATH_MAX];\n\n mode = 0; // Unused\n mg_strlcpy(buf, path, sizeof(buf));\n change_slashes_to_backslashes(buf);\n\n (void) MultiByteToWideChar(CP_UTF8, 0, buf, -1, wbuf, sizeof(wbuf));\n\n return CreateDirectoryW(wbuf, NULL) ? 0 : -1;\n}\n\n// Implementation of POSIX opendir/closedir/readdir for Windows.\nstatic DIR * opendir(const char *name) {\n DIR *dir = NULL;\n wchar_t wpath[PATH_MAX];\n DWORD attrs;\n\n if (name == NULL) {\n SetLastError(ERROR_BAD_ARGUMENTS);\n } else if ((dir = (DIR *) malloc(sizeof(*dir))) == NULL) {\n SetLastError(ERROR_NOT_ENOUGH_MEMORY);\n } else {\n to_unicode(name, wpath, ARRAY_SIZE(wpath));\n attrs = GetFileAttributesW(wpath);\n if (attrs != 0xFFFFFFFF &&\n ((attrs & FILE_ATTRIBUTE_DIRECTORY) == FILE_ATTRIBUTE_DIRECTORY)) {\n (void) wcscat(wpath, L\"\\\\*\");\n dir->handle = FindFirstFileW(wpath, &dir->info);\n dir->result.d_name[0] = '\\0';\n } else {\n free(dir);\n dir = NULL;\n }\n }\n\n return dir;\n}\n\nstatic int closedir(DIR *dir) {\n int result = 0;\n\n if (dir != NULL) {\n if (dir->handle != INVALID_HANDLE_VALUE)\n result = FindClose(dir->handle) ? 0 : -1;\n\n free(dir);\n } else {\n result = -1;\n SetLastError(ERROR_BAD_ARGUMENTS);\n }\n\n return result;\n}\n\nstruct dirent * readdir(DIR *dir) {\n struct dirent *result = 0;\n\n if (dir) {\n if (dir->handle != INVALID_HANDLE_VALUE) {\n result = &dir->result;\n (void) WideCharToMultiByte(CP_UTF8, 0,\n dir->info.cFileName, -1, result->d_name,\n sizeof(result->d_name), NULL, NULL);\n\n if (!FindNextFileW(dir->handle, &dir->info)) {\n (void) FindClose(dir->handle);\n dir->handle = INVALID_HANDLE_VALUE;\n }\n\n } else {\n SetLastError(ERROR_FILE_NOT_FOUND);\n }\n } else {\n SetLastError(ERROR_BAD_ARGUMENTS);\n }\n\n return result;\n}\n\n#define set_close_on_exec(fd) // No FD_CLOEXEC on Windows\n\nstatic int start_thread(struct mg_context *ctx, mg_thread_func_t f, void *p) {\n return _beginthread((void (__cdecl *)(void *)) f, 0, p) == -1L ? -1 : 0;\n}\n\nstatic HANDLE dlopen(const char *dll_name, int flags) {\n wchar_t wbuf[PATH_MAX];\n flags = 0; // Unused\n to_unicode(dll_name, wbuf, ARRAY_SIZE(wbuf));\n return LoadLibraryW(wbuf);\n}\n\n#if !defined(NO_CGI)\n#define SIGKILL 0\nstatic int kill(pid_t pid, int sig_num) {\n (void) TerminateProcess(pid, sig_num);\n (void) CloseHandle(pid);\n return 0;\n}\n\nstatic pid_t spawn_process(struct mg_connection *conn, const char *prog,\n char *envblk, char *envp[], int fd_stdin,\n int fd_stdout, const char *dir) {\n HANDLE me;\n char *p, *interp, cmdline[PATH_MAX], buf[PATH_MAX];\n FILE *fp;\n STARTUPINFOA si = { sizeof(si) };\n PROCESS_INFORMATION pi = { 0 };\n\n envp = NULL; // Unused\n\n // TODO(lsm): redirect CGI errors to the error log file\n si.dwFlags = STARTF_USESTDHANDLES | STARTF_USESHOWWINDOW;\n si.wShowWindow = SW_HIDE;\n\n me = GetCurrentProcess();\n (void) DuplicateHandle(me, (HANDLE) _get_osfhandle(fd_stdin), me,\n &si.hStdInput, 0, TRUE, DUPLICATE_SAME_ACCESS);\n (void) DuplicateHandle(me, (HANDLE) _get_osfhandle(fd_stdout), me,\n &si.hStdOutput, 0, TRUE, DUPLICATE_SAME_ACCESS);\n\n // If CGI file is a script, try to read the interpreter line\n interp = conn->ctx->config[CGI_INTERPRETER];\n if (interp == NULL) {\n buf[2] = '\\0';\n mg_snprintf(conn, cmdline, sizeof(cmdline), \"%s%c%s\", dir, DIRSEP, prog);\n if ((fp = fopen(cmdline, \"r\")) != NULL) {\n (void) fgets(buf, sizeof(buf), fp);\n if (buf[0] != '#' || buf[1] != '!') {\n // First line does not start with \"#!\". Do not set interpreter.\n buf[2] = '\\0';\n } else {\n // Trim whitespaces in interpreter name\n for (p = &buf[strlen(buf) - 1]; p > buf && isspace(*p); p--) {\n *p = '\\0';\n }\n }\n (void) fclose(fp);\n }\n interp = buf + 2;\n }\n\n (void) mg_snprintf(conn, cmdline, sizeof(cmdline), \"%s%s%s%c%s\",\n interp, interp[0] == '\\0' ? \"\" : \" \", dir, DIRSEP, prog);\n\n DEBUG_TRACE((\"Running [%s]\", cmdline));\n if (CreateProcessA(NULL, cmdline, NULL, NULL, TRUE,\n CREATE_NEW_PROCESS_GROUP, envblk, dir, &si, &pi) == 0) {\n cry(conn, \"%s: CreateProcess(%s): %d\",\n __func__, cmdline, ERRNO);\n pi.hProcess = (pid_t) -1;\n } else {\n (void) close(fd_stdin);\n (void) close(fd_stdout);\n }\n\n (void) CloseHandle(si.hStdOutput);\n (void) CloseHandle(si.hStdInput);\n (void) CloseHandle(pi.hThread);\n\n return (pid_t) pi.hProcess;\n}\n#endif // !NO_CGI\n\nstatic int set_non_blocking_mode(SOCKET sock) {\n unsigned long on = 1;\n return ioctlsocket(sock, FIONBIO, &on);\n}\n\n#else\nstatic int mg_stat(const char *path, struct mgstat *stp) {\n struct stat st;\n int ok;\n\n if (stat(path, &st) == 0) {\n ok = 0;\n stp->size = st.st_size;\n stp->mtime = st.st_mtime;\n stp->is_directory = S_ISDIR(st.st_mode);\n } else {\n ok = -1;\n }\n\n return ok;\n}\n\nstatic void set_close_on_exec(int fd) {\n (void) fcntl(fd, F_SETFD, FD_CLOEXEC);\n}\n\nstatic int start_thread(struct mg_context *ctx, mg_thread_func_t func,\n void *param) {\n pthread_t thread_id;\n pthread_attr_t attr;\n int retval;\n\n (void) pthread_attr_init(&attr);\n (void) pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);\n // TODO(lsm): figure out why mongoose dies on Linux if next line is enabled\n // (void) pthread_attr_setstacksize(&attr, sizeof(struct mg_connection) * 5);\n\n if ((retval = pthread_create(&thread_id, &attr, func, param)) != 0) {\n cry(fc(ctx), \"%s: %s\", __func__, strerror(retval));\n }\n\n return retval;\n}\n\n#ifndef NO_CGI\nstatic pid_t spawn_process(struct mg_connection *conn, const char *prog,\n char *envblk, char *envp[], int fd_stdin,\n int fd_stdout, const char *dir) {\n pid_t pid;\n const char *interp;\n\n envblk = NULL; // Unused\n\n if ((pid = fork()) == -1) {\n // Parent\n send_http_error(conn, 500, http_500_error, \"fork(): %s\", strerror(ERRNO));\n } else if (pid == 0) {\n // Child\n if (chdir(dir) != 0) {\n cry(conn, \"%s: chdir(%s): %s\", __func__, dir, strerror(ERRNO));\n } else if (dup2(fd_stdin, 0) == -1) {\n cry(conn, \"%s: dup2(%d, 0): %s\", __func__, fd_stdin, strerror(ERRNO));\n } else if (dup2(fd_stdout, 1) == -1) {\n cry(conn, \"%s: dup2(%d, 1): %s\", __func__, fd_stdout, strerror(ERRNO));\n } else {\n (void) dup2(fd_stdout, 2);\n (void) close(fd_stdin);\n (void) close(fd_stdout);\n\n // Execute CGI program. No need to lock: new process\n interp = conn->ctx->config[CGI_INTERPRETER];\n if (interp == NULL) {\n (void) execle(prog, prog, NULL, envp);\n cry(conn, \"%s: execle(%s): %s\", __func__, prog, strerror(ERRNO));\n } else {\n (void) execle(interp, interp, prog, NULL, envp);\n cry(conn, \"%s: execle(%s %s): %s\", __func__, interp, prog,\n strerror(ERRNO));\n }\n }\n exit(EXIT_FAILURE);\n } else {\n // Parent. Close stdio descriptors\n (void) close(fd_stdin);\n (void) close(fd_stdout);\n }\n\n return pid;\n}\n#endif // !NO_CGI\n\nstatic int set_non_blocking_mode(SOCKET sock) {\n int flags;\n\n flags = fcntl(sock, F_GETFL, 0);\n (void) fcntl(sock, F_SETFL, flags | O_NONBLOCK);\n\n return 0;\n}\n#endif // _WIN32\n\n// Write data to the IO channel - opened file descriptor, socket or SSL\n// descriptor. Return number of bytes written.\nstatic int64_t push(FILE *fp, SOCKET sock, SSL *ssl, const char *buf,\n int64_t len) {\n int64_t sent;\n int n, k;\n\n sent = 0;\n while (sent < len) {\n\n // How many bytes we send in this iteration\n k = len - sent > INT_MAX ? INT_MAX : (int) (len - sent);\n\n if (ssl != NULL) {\n n = SSL_write(ssl, buf + sent, k);\n } else if (fp != NULL) {\n n = fwrite(buf + sent, 1, (size_t) k, fp);\n if (ferror(fp))\n n = -1;\n } else {\n n = send(sock, buf + sent, (size_t) k, MSG_NOSIGNAL);\n }\n\n if (n < 0)\n break;\n\n sent += n;\n }\n\n return sent;\n}\n\n// Read from IO channel - opened file descriptor, socket, or SSL descriptor.\n// Return number of bytes read.\nstatic int pull(FILE *fp, SOCKET sock, SSL *ssl, char *buf, int len) {\n int nread;\n\n if (ssl != NULL) {\n nread = SSL_read(ssl, buf, len);\n } else if (fp != NULL) {\n // Use read() instead of fread(), because if we're reading from the CGI\n // pipe, fread() may block until IO buffer is filled up. We cannot afford\n // to block and must pass all read bytes immediately to the client.\n nread = read(fileno(fp), buf, (size_t) len);\n if (ferror(fp))\n nread = -1;\n } else {\n nread = recv(sock, buf, (size_t) len, 0);\n }\n\n return nread;\n}\n\nint mg_read(struct mg_connection *conn, void *buf, size_t len) {\n int n, buffered_len, nread;\n const char *buffered;\n\n assert((conn->content_len == -1 && conn->consumed_content == 0) ||\n conn->consumed_content <= conn->content_len);\n DEBUG_TRACE((\"%p %zu %lld %lld\", buf, len,\n conn->content_len, conn->consumed_content));\n nread = 0;\n if (conn->consumed_content < conn->content_len) {\n\n // Adjust number of bytes to read.\n int64_t to_read = conn->content_len - conn->consumed_content;\n if (to_read < (int64_t) len) {\n len = (int) to_read;\n }\n\n // How many bytes of data we have buffered in the request buffer?\n buffered = conn->buf + conn->request_len + conn->consumed_content;\n buffered_len = conn->data_len - conn->request_len;\n assert(buffered_len >= 0);\n\n // Return buffered data back if we haven't done that yet.\n if (conn->consumed_content < (int64_t) buffered_len) {\n buffered_len -= (int) conn->consumed_content;\n if (len < (size_t) buffered_len) {\n buffered_len = len;\n }\n memcpy(buf, buffered, (size_t)buffered_len);\n len -= buffered_len;\n buf = (char *) buf + buffered_len;\n conn->consumed_content += buffered_len;\n nread = buffered_len;\n }\n\n // We have returned all buffered data. Read new data from the remote socket.\n while (len > 0) {\n n = pull(NULL, conn->client.sock, conn->ssl, (char *) buf, (int) len);\n if (n <= 0) {\n break;\n }\n buf = (char *) buf + n;\n conn->consumed_content += n;\n nread += n;\n len -= n;\n }\n }\n return nread;\n}\n\nint mg_write(struct mg_connection *conn, const void *buf, size_t len) {\n return (int) push(NULL, conn->client.sock, conn->ssl, (const char *) buf,\n (int64_t) len);\n}\n\nint mg_printf(struct mg_connection *conn, const char *fmt, ...) {\n char buf[BUFSIZ];\n int len;\n va_list ap;\n\n va_start(ap, fmt);\n len = mg_vsnprintf(conn, buf, sizeof(buf), fmt, ap);\n va_end(ap);\n\n return mg_write(conn, buf, (size_t)len);\n}\n\n// URL-decode input buffer into destination buffer.\n// 0-terminate the destination buffer. Return the length of decoded data.\n// form-url-encoded data differs from URI encoding in a way that it\n// uses '+' as character for space, see RFC 1866 section 8.2.1\n// http://ftp.ics.uci.edu/pub/ietf/html/rfc1866.txt\nstatic size_t url_decode(const char *src, size_t src_len, char *dst,\n size_t dst_len, int is_form_url_encoded) {\n size_t i, j;\n int a, b;\n#define HEXTOI(x) (isdigit(x) ? x - '0' : x - 'W')\n\n for (i = j = 0; i < src_len && j < dst_len - 1; i++, j++) {\n if (src[i] == '%' &&\n isxdigit(* (const unsigned char *) (src + i + 1)) &&\n isxdigit(* (const unsigned char *) (src + i + 2))) {\n a = tolower(* (const unsigned char *) (src + i + 1));\n b = tolower(* (const unsigned char *) (src + i + 2));\n dst[j] = (char) ((HEXTOI(a) << 4) | HEXTOI(b));\n i += 2;\n } else if (is_form_url_encoded && src[i] == '+') {\n dst[j] = ' ';\n } else {\n dst[j] = src[i];\n }\n }\n\n dst[j] = '\\0'; // Null-terminate the destination\n\n return j;\n}\n\n// Scan given buffer and fetch the value of the given variable.\n// It can be specified in query string, or in the POST data.\n// Return NULL if the variable not found, or allocated 0-terminated value.\n// It is caller's responsibility to free the returned value.\nint mg_get_var(const char *buf, size_t buf_len, const char *name,\n char *dst, size_t dst_len) {\n const char *p, *e, *s;\n size_t name_len, len;\n\n name_len = strlen(name);\n e = buf + buf_len;\n len = -1;\n dst[0] = '\\0';\n\n // buf is \"var1=val1&var2=val2...\". Find variable first\n for (p = buf; p != NULL && p + name_len < e; p++) {\n if ((p == buf || p[-1] == '&') && p[name_len] == '=' &&\n !mg_strncasecmp(name, p, name_len)) {\n\n // Point p to variable value\n p += name_len + 1;\n\n // Point s to the end of the value\n s = (const char *) memchr(p, '&', (size_t)(e - p));\n if (s == NULL) {\n s = e;\n }\n assert(s >= p);\n\n // Decode variable into destination buffer\n if ((size_t) (s - p) < dst_len) {\n len = url_decode(p, (size_t)(s - p), dst, dst_len, 1);\n }\n break;\n }\n }\n\n return len;\n}\n\nint mg_get_cookie(const struct mg_connection *conn, const char *cookie_name,\n char *dst, size_t dst_size) {\n const char *s, *p, *end;\n int name_len, len = -1;\n\n dst[0] = '\\0';\n if ((s = mg_get_header(conn, \"Cookie\")) == NULL) {\n return 0;\n }\n\n name_len = strlen(cookie_name);\n end = s + strlen(s);\n\n for (; (s = strstr(s, cookie_name)) != NULL; s += name_len)\n if (s[name_len] == '=') {\n s += name_len + 1;\n if ((p = strchr(s, ' ')) == NULL)\n p = end;\n if (p[-1] == ';')\n p--;\n if (*s == '\"' && p[-1] == '\"' && p > s + 1) {\n s++;\n p--;\n }\n if ((size_t) (p - s) < dst_size) {\n len = (p - s) + 1;\n mg_strlcpy(dst, s, (size_t)len);\n }\n break;\n }\n\n return len;\n}\n\nstatic int convert_uri_to_file_name(struct mg_connection *conn, char *buf,\n size_t buf_len, struct mgstat *st) {\n struct vec a, b;\n const char *rewrite, *uri = conn->request_info.uri;\n char *p;\n int match_len, stat_result;\n\n buf_len--; // This is because memmove() for PATH_INFO may shift part\n // of the path one byte on the right.\n mg_snprintf(conn, buf, buf_len, \"%s%s\", conn->ctx->config[DOCUMENT_ROOT],\n uri);\n\n rewrite = conn->ctx->config[REWRITE];\n while ((rewrite = next_option(rewrite, &a, &b)) != NULL) {\n if ((match_len = match_prefix(a.ptr, a.len, uri)) > 0) {\n mg_snprintf(conn, buf, buf_len, \"%.*s%s\", b.len, b.ptr, uri + match_len);\n break;\n }\n }\n\n#if defined(_WIN32) && !defined(__SYMBIAN32__)\n //change_slashes_to_backslashes(buf);\n#endif // _WIN32\n\n if ((stat_result = mg_stat(buf, st)) != 0) {\n // Support PATH_INFO for CGI scripts.\n for (p = buf + strlen(buf); p > buf + 1; p--) {\n if (*p == '/') {\n *p = '\\0';\n if (match_prefix(conn->ctx->config[CGI_EXTENSIONS],\n strlen(conn->ctx->config[CGI_EXTENSIONS]), buf) > 0 &&\n (stat_result = mg_stat(buf, st)) == 0) {\n conn->path_info = p + 1;\n memmove(p + 2, p + 1, strlen(p + 1));\n p[1] = '/';\n break;\n } else {\n *p = '/';\n stat_result = -1;\n }\n }\n }\n }\n\n return stat_result;\n}\n\nstatic int sslize(struct mg_connection *conn, int (*func)(SSL *)) {\n return (conn->ssl = SSL_new(conn->ctx->ssl_ctx)) != NULL &&\n SSL_set_fd(conn->ssl, conn->client.sock) == 1 &&\n func(conn->ssl) == 1;\n}\n\n// Check whether full request is buffered. Return:\n// -1 if request is malformed\n// 0 if request is not yet fully buffered\n// >0 actual request length, including last \\r\\n\\r\\n\nstatic int get_request_len(const char *buf, int buflen) {\n const char *s, *e;\n int len = 0;\n\n DEBUG_TRACE((\"buf: %p, len: %d\", buf, buflen));\n for (s = buf, e = s + buflen - 1; len <= 0 && s < e; s++)\n // Control characters are not allowed but >=128 is.\n if (!isprint(* (const unsigned char *) s) && *s != '\\r' &&\n *s != '\\n' && * (const unsigned char *) s < 128) {\n len = -1;\n } else if (s[0] == '\\n' && s[1] == '\\n') {\n len = (int) (s - buf) + 2;\n } else if (s[0] == '\\n' && &s[1] < e &&\n s[1] == '\\r' && s[2] == '\\n') {\n len = (int) (s - buf) + 3;\n }\n\n return len;\n}\n\n// Convert month to the month number. Return -1 on error, or month number\nstatic int get_month_index(const char *s) {\n size_t i;\n\n for (i = 0; i < ARRAY_SIZE(month_names); i++)\n if (!strcmp(s, month_names[i]))\n return (int) i;\n\n return -1;\n}\n\n// Parse UTC date-time string, and return the corresponding time_t value.\nstatic time_t parse_date_string(const char *datetime) {\n static const unsigned short days_before_month[] = {\n 0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334\n };\n char month_str[32];\n int second, minute, hour, day, month, year, leap_days, days;\n time_t result = (time_t) 0;\n\n if (((sscanf(datetime, \"%d/%3s/%d %d:%d:%d\",\n &day, month_str, &year, &hour, &minute, &second) == 6) ||\n (sscanf(datetime, \"%d %3s %d %d:%d:%d\",\n &day, month_str, &year, &hour, &minute, &second) == 6) ||\n (sscanf(datetime, \"%*3s, %d %3s %d %d:%d:%d\",\n &day, month_str, &year, &hour, &minute, &second) == 6) ||\n (sscanf(datetime, \"%d-%3s-%d %d:%d:%d\",\n &day, month_str, &year, &hour, &minute, &second) == 6)) &&\n year > 1970 &&\n (month = get_month_index(month_str)) != -1) {\n year -= 1970;\n leap_days = year / 4 - year / 100 + year / 400;\n days = year * 365 + days_before_month[month] + (day - 1) + leap_days;\n result = days * 24 * 3600 + hour * 3600 + minute * 60 + second;\n }\n\n return result;\n}\n\n// Protect against directory disclosure attack by removing '..',\n// excessive '/' and '\\' characters\nstatic void remove_double_dots_and_double_slashes(char *s) {\n char *p = s;\n\n while (*s != '\\0') {\n *p++ = *s++;\n if (IS_DIRSEP_CHAR(s[-1])) {\n // Skip all following slashes and backslashes\n while (IS_DIRSEP_CHAR(s[0])) {\n s++;\n }\n\n // Skip all double-dots\n while (*s == '.' && s[1] == '.') {\n s += 2;\n }\n }\n }\n *p = '\\0';\n}\n\nstatic const struct {\n const char *extension;\n size_t ext_len;\n const char *mime_type;\n size_t mime_type_len;\n} builtin_mime_types[] = {\n {\".html\", 5, \"text/html\", 9},\n {\".htm\", 4, \"text/html\", 9},\n {\".shtm\", 5, \"text/html\", 9},\n {\".shtml\", 6, \"text/html\", 9},\n {\".css\", 4, \"text/css\", 8},\n {\".js\", 3, \"application/x-javascript\", 24},\n {\".ico\", 4, \"image/x-icon\", 12},\n {\".gif\", 4, \"image/gif\", 9},\n {\".jpg\", 4, \"image/jpeg\", 10},\n {\".jpeg\", 5, \"image/jpeg\", 10},\n {\".png\", 4, \"image/png\", 9},\n {\".svg\", 4, \"image/svg+xml\", 13},\n {\".torrent\", 8, \"application/x-bittorrent\", 24},\n {\".wav\", 4, \"audio/x-wav\", 11},\n {\".mp3\", 4, \"audio/x-mp3\", 11},\n {\".mid\", 4, \"audio/mid\", 9},\n {\".m3u\", 4, \"audio/x-mpegurl\", 15},\n {\".ram\", 4, \"audio/x-pn-realaudio\", 20},\n {\".xml\", 4, \"text/xml\", 8},\n {\".xslt\", 5, \"application/xml\", 15},\n {\".ra\", 3, \"audio/x-pn-realaudio\", 20},\n {\".doc\", 4, \"application/msword\", 19},\n {\".exe\", 4, \"application/octet-stream\", 24},\n {\".zip\", 4, \"application/x-zip-compressed\", 28},\n {\".xls\", 4, \"application/excel\", 17},\n {\".tgz\", 4, \"application/x-tar-gz\", 20},\n {\".tar\", 4, \"application/x-tar\", 17},\n {\".gz\", 3, \"application/x-gunzip\", 20},\n {\".arj\", 4, \"application/x-arj-compressed\", 28},\n {\".rar\", 4, \"application/x-arj-compressed\", 28},\n {\".rtf\", 4, \"application/rtf\", 15},\n {\".pdf\", 4, \"application/pdf\", 15},\n {\".swf\", 4, \"application/x-shockwave-flash\",29},\n {\".mpg\", 4, \"video/mpeg\", 10},\n {\".mpeg\", 5, \"video/mpeg\", 10},\n {\".mp4\", 4, \"video/mp4\", 9},\n {\".m4v\", 4, \"video/x-m4v\", 11},\n {\".asf\", 4, \"video/x-ms-asf\", 14},\n {\".avi\", 4, \"video/x-msvideo\", 15},\n {\".bmp\", 4, \"image/bmp\", 9},\n {NULL, 0, NULL, 0}\n};\n\n// Look at the \"path\" extension and figure what mime type it has.\n// Store mime type in the vector.\nstatic void get_mime_type(struct mg_context *ctx, const char *path,\n struct vec *vec) {\n struct vec ext_vec, mime_vec;\n const char *list, *ext;\n size_t i, path_len;\n\n path_len = strlen(path);\n\n // Scan user-defined mime types first, in case user wants to\n // override default mime types.\n list = ctx->config[EXTRA_MIME_TYPES];\n while ((list = next_option(list, &ext_vec, &mime_vec)) != NULL) {\n // ext now points to the path suffix\n ext = path + path_len - ext_vec.len;\n if (mg_strncasecmp(ext, ext_vec.ptr, ext_vec.len) == 0) {\n *vec = mime_vec;\n return;\n }\n }\n\n // Now scan built-in mime types\n for (i = 0; builtin_mime_types[i].extension != NULL; i++) {\n ext = path + (path_len - builtin_mime_types[i].ext_len);\n if (path_len > builtin_mime_types[i].ext_len &&\n mg_strcasecmp(ext, builtin_mime_types[i].extension) == 0) {\n vec->ptr = builtin_mime_types[i].mime_type;\n vec->len = builtin_mime_types[i].mime_type_len;\n return;\n }\n }\n\n // Nothing found. Fall back to \"text/plain\"\n vec->ptr = \"text/plain\";\n vec->len = 10;\n}\n\n#ifndef HAVE_MD5\ntypedef struct MD5Context {\n uint32_t buf[4];\n uint32_t bits[2];\n unsigned char in[64];\n} MD5_CTX;\n\n#if defined(__BYTE_ORDER) && (__BYTE_ORDER == 1234)\n#define byteReverse(buf, len) // Do nothing\n#else\nstatic void byteReverse(unsigned char *buf, unsigned longs) {\n uint32_t t;\n do {\n t = (uint32_t) ((unsigned) buf[3] << 8 | buf[2]) << 16 |\n ((unsigned) buf[1] << 8 | buf[0]);\n *(uint32_t *) buf = t;\n buf += 4;\n } while (--longs);\n}\n#endif\n\n#define F1(x, y, z) (z ^ (x & (y ^ z)))\n#define F2(x, y, z) F1(z, x, y)\n#define F3(x, y, z) (x ^ y ^ z)\n#define F4(x, y, z) (y ^ (x | ~z))\n\n#define MD5STEP(f, w, x, y, z, data, s) \\\n ( w += f(x, y, z) + data, w = w<>(32-s), w += x )\n\n// Start MD5 accumulation. Set bit count to 0 and buffer to mysterious\n// initialization constants.\nstatic void MD5Init(MD5_CTX *ctx) {\n ctx->buf[0] = 0x67452301;\n ctx->buf[1] = 0xefcdab89;\n ctx->buf[2] = 0x98badcfe;\n ctx->buf[3] = 0x10325476;\n\n ctx->bits[0] = 0;\n ctx->bits[1] = 0;\n}\n\nstatic void MD5Transform(uint32_t buf[4], uint32_t const in[16]) {\n register uint32_t a, b, c, d;\n\n a = buf[0];\n b = buf[1];\n c = buf[2];\n d = buf[3];\n\n MD5STEP(F1, a, b, c, d, in[0] + 0xd76aa478, 7);\n MD5STEP(F1, d, a, b, c, in[1] + 0xe8c7b756, 12);\n MD5STEP(F1, c, d, a, b, in[2] + 0x242070db, 17);\n MD5STEP(F1, b, c, d, a, in[3] + 0xc1bdceee, 22);\n MD5STEP(F1, a, b, c, d, in[4] + 0xf57c0faf, 7);\n MD5STEP(F1, d, a, b, c, in[5] + 0x4787c62a, 12);\n MD5STEP(F1, c, d, a, b, in[6] + 0xa8304613, 17);\n MD5STEP(F1, b, c, d, a, in[7] + 0xfd469501, 22);\n MD5STEP(F1, a, b, c, d, in[8] + 0x698098d8, 7);\n MD5STEP(F1, d, a, b, c, in[9] + 0x8b44f7af, 12);\n MD5STEP(F1, c, d, a, b, in[10] + 0xffff5bb1, 17);\n MD5STEP(F1, b, c, d, a, in[11] + 0x895cd7be, 22);\n MD5STEP(F1, a, b, c, d, in[12] + 0x6b901122, 7);\n MD5STEP(F1, d, a, b, c, in[13] + 0xfd987193, 12);\n MD5STEP(F1, c, d, a, b, in[14] + 0xa679438e, 17);\n MD5STEP(F1, b, c, d, a, in[15] + 0x49b40821, 22);\n\n MD5STEP(F2, a, b, c, d, in[1] + 0xf61e2562, 5);\n MD5STEP(F2, d, a, b, c, in[6] + 0xc040b340, 9);\n MD5STEP(F2, c, d, a, b, in[11] + 0x265e5a51, 14);\n MD5STEP(F2, b, c, d, a, in[0] + 0xe9b6c7aa, 20);\n MD5STEP(F2, a, b, c, d, in[5] + 0xd62f105d, 5);\n MD5STEP(F2, d, a, b, c, in[10] + 0x02441453, 9);\n MD5STEP(F2, c, d, a, b, in[15] + 0xd8a1e681, 14);\n MD5STEP(F2, b, c, d, a, in[4] + 0xe7d3fbc8, 20);\n MD5STEP(F2, a, b, c, d, in[9] + 0x21e1cde6, 5);\n MD5STEP(F2, d, a, b, c, in[14] + 0xc33707d6, 9);\n MD5STEP(F2, c, d, a, b, in[3] + 0xf4d50d87, 14);\n MD5STEP(F2, b, c, d, a, in[8] + 0x455a14ed, 20);\n MD5STEP(F2, a, b, c, d, in[13] + 0xa9e3e905, 5);\n MD5STEP(F2, d, a, b, c, in[2] + 0xfcefa3f8, 9);\n MD5STEP(F2, c, d, a, b, in[7] + 0x676f02d9, 14);\n MD5STEP(F2, b, c, d, a, in[12] + 0x8d2a4c8a, 20);\n\n MD5STEP(F3, a, b, c, d, in[5] + 0xfffa3942, 4);\n MD5STEP(F3, d, a, b, c, in[8] + 0x8771f681, 11);\n MD5STEP(F3, c, d, a, b, in[11] + 0x6d9d6122, 16);\n MD5STEP(F3, b, c, d, a, in[14] + 0xfde5380c, 23);\n MD5STEP(F3, a, b, c, d, in[1] + 0xa4beea44, 4);\n MD5STEP(F3, d, a, b, c, in[4] + 0x4bdecfa9, 11);\n MD5STEP(F3, c, d, a, b, in[7] + 0xf6bb4b60, 16);\n MD5STEP(F3, b, c, d, a, in[10] + 0xbebfbc70, 23);\n MD5STEP(F3, a, b, c, d, in[13] + 0x289b7ec6, 4);\n MD5STEP(F3, d, a, b, c, in[0] + 0xeaa127fa, 11);\n MD5STEP(F3, c, d, a, b, in[3] + 0xd4ef3085, 16);\n MD5STEP(F3, b, c, d, a, in[6] + 0x04881d05, 23);\n MD5STEP(F3, a, b, c, d, in[9] + 0xd9d4d039, 4);\n MD5STEP(F3, d, a, b, c, in[12] + 0xe6db99e5, 11);\n MD5STEP(F3, c, d, a, b, in[15] + 0x1fa27cf8, 16);\n MD5STEP(F3, b, c, d, a, in[2] + 0xc4ac5665, 23);\n\n MD5STEP(F4, a, b, c, d, in[0] + 0xf4292244, 6);\n MD5STEP(F4, d, a, b, c, in[7] + 0x432aff97, 10);\n MD5STEP(F4, c, d, a, b, in[14] + 0xab9423a7, 15);\n MD5STEP(F4, b, c, d, a, in[5] + 0xfc93a039, 21);\n MD5STEP(F4, a, b, c, d, in[12] + 0x655b59c3, 6);\n MD5STEP(F4, d, a, b, c, in[3] + 0x8f0ccc92, 10);\n MD5STEP(F4, c, d, a, b, in[10] + 0xffeff47d, 15);\n MD5STEP(F4, b, c, d, a, in[1] + 0x85845dd1, 21);\n MD5STEP(F4, a, b, c, d, in[8] + 0x6fa87e4f, 6);\n MD5STEP(F4, d, a, b, c, in[15] + 0xfe2ce6e0, 10);\n MD5STEP(F4, c, d, a, b, in[6] + 0xa3014314, 15);\n MD5STEP(F4, b, c, d, a, in[13] + 0x4e0811a1, 21);\n MD5STEP(F4, a, b, c, d, in[4] + 0xf7537e82, 6);\n MD5STEP(F4, d, a, b, c, in[11] + 0xbd3af235, 10);\n MD5STEP(F4, c, d, a, b, in[2] + 0x2ad7d2bb, 15);\n MD5STEP(F4, b, c, d, a, in[9] + 0xeb86d391, 21);\n\n buf[0] += a;\n buf[1] += b;\n buf[2] += c;\n buf[3] += d;\n}\n\nstatic void MD5Update(MD5_CTX *ctx, unsigned char const *buf, unsigned len) {\n uint32_t t;\n\n t = ctx->bits[0];\n if ((ctx->bits[0] = t + ((uint32_t) len << 3)) < t)\n ctx->bits[1]++;\n ctx->bits[1] += len >> 29;\n\n t = (t >> 3) & 0x3f;\n\n if (t) {\n unsigned char *p = (unsigned char *) ctx->in + t;\n\n t = 64 - t;\n if (len < t) {\n memcpy(p, buf, len);\n return;\n }\n memcpy(p, buf, t);\n byteReverse(ctx->in, 16);\n MD5Transform(ctx->buf, (uint32_t *) ctx->in);\n buf += t;\n len -= t;\n }\n\n while (len >= 64) {\n memcpy(ctx->in, buf, 64);\n byteReverse(ctx->in, 16);\n MD5Transform(ctx->buf, (uint32_t *) ctx->in);\n buf += 64;\n len -= 64;\n }\n\n memcpy(ctx->in, buf, len);\n}\n\nstatic void MD5Final(unsigned char digest[16], MD5_CTX *ctx) {\n unsigned count;\n unsigned char *p;\n\n count = (ctx->bits[0] >> 3) & 0x3F;\n\n p = ctx->in + count;\n *p++ = 0x80;\n count = 64 - 1 - count;\n if (count < 8) {\n memset(p, 0, count);\n byteReverse(ctx->in, 16);\n MD5Transform(ctx->buf, (uint32_t *) ctx->in);\n memset(ctx->in, 0, 56);\n } else {\n memset(p, 0, count - 8);\n }\n byteReverse(ctx->in, 14);\n\n ((uint32_t *) ctx->in)[14] = ctx->bits[0];\n ((uint32_t *) ctx->in)[15] = ctx->bits[1];\n\n MD5Transform(ctx->buf, (uint32_t *) ctx->in);\n byteReverse((unsigned char *) ctx->buf, 4);\n memcpy(digest, ctx->buf, 16);\n memset((char *) ctx, 0, sizeof(*ctx));\n}\n#endif // !HAVE_MD5\n\n// Stringify binary data. Output buffer must be twice as big as input,\n// because each byte takes 2 bytes in string representation\nstatic void bin2str(char *to, const unsigned char *p, size_t len) {\n static const char *hex = \"0123456789abcdef\";\n\n for (; len--; p++) {\n *to++ = hex[p[0] >> 4];\n *to++ = hex[p[0] & 0x0f];\n }\n *to = '\\0';\n}\n\n// Return stringified MD5 hash for list of vectors. Buffer must be 33 bytes.\nvoid mg_md5(char *buf, ...) {\n unsigned char hash[16];\n const char *p;\n va_list ap;\n MD5_CTX ctx;\n\n MD5Init(&ctx);\n\n va_start(ap, buf);\n while ((p = va_arg(ap, const char *)) != NULL) {\n MD5Update(&ctx, (const unsigned char *) p, (unsigned) strlen(p));\n }\n va_end(ap);\n\n MD5Final(hash, &ctx);\n bin2str(buf, hash, sizeof(hash));\n}\n\n\n\n\n// Return stringified MD5 hash for a file\nvoid mg_md5_file(char *buf, const char* file) {\n unsigned char hash[16];\n MD5_CTX ctx;\n\n MD5Init(&ctx);\n FILE* f = fopen(file, \"rb\");\n if (f == NULL) {\n buf[0] = '\\0';\n return;\n }\n char readbuf[4096];\n while (1) {\n unsigned readlen = fread(readbuf, 1, 4096, f);\n MD5Update(&ctx, (const unsigned char *) readbuf, readlen);\n if (readlen == 0) break;\n }\n fclose(f);\n MD5Final(hash, &ctx);\n bin2str(buf, hash, sizeof(hash));\n}\n\n\n// Check the user's password, return 1 if OK\nstatic int check_password(const char *method, const char *ha1, const char *uri,\n const char *nonce, const char *nc, const char *cnonce,\n const char *qop, const char *response) {\n char ha2[32 + 1], expected_response[32 + 1];\n\n // Some of the parameters may be NULL\n if (method == NULL || nonce == NULL || nc == NULL || cnonce == NULL ||\n qop == NULL || response == NULL) {\n return 0;\n }\n\n // NOTE(lsm): due to a bug in MSIE, we do not compare the URI\n // TODO(lsm): check for authentication timeout\n if (// strcmp(dig->uri, c->ouri) != 0 ||\n strlen(response) != 32\n // || now - strtoul(dig->nonce, NULL, 10) > 3600\n ) {\n return 0;\n }\n\n mg_md5(ha2, method, \":\", uri, NULL);\n mg_md5(expected_response, ha1, \":\", nonce, \":\", nc,\n \":\", cnonce, \":\", qop, \":\", ha2, NULL);\n\n return mg_strcasecmp(response, expected_response) == 0;\n}\n\n// Use the global passwords file, if specified by auth_gpass option,\n// or search for .htpasswd in the requested directory.\nstatic FILE *open_auth_file(struct mg_connection *conn, const char *path) {\n struct mg_context *ctx = conn->ctx;\n char name[PATH_MAX];\n const char *p, *e;\n struct mgstat st;\n FILE *fp;\n\n if (ctx->config[GLOBAL_PASSWORDS_FILE] != NULL) {\n // Use global passwords file\n fp = mg_fopen(ctx->config[GLOBAL_PASSWORDS_FILE], \"r\");\n if (fp == NULL)\n cry(fc(ctx), \"fopen(%s): %s\",\n ctx->config[GLOBAL_PASSWORDS_FILE], strerror(ERRNO));\n } else if (!mg_stat(path, &st) && st.is_directory) {\n (void) mg_snprintf(conn, name, sizeof(name), \"%s%c%s\",\n path, DIRSEP, PASSWORDS_FILE_NAME);\n fp = mg_fopen(name, \"r\");\n } else {\n // Try to find .htpasswd in requested directory.\n for (p = path, e = p + strlen(p) - 1; e > p; e--)\n if (IS_DIRSEP_CHAR(*e))\n break;\n (void) mg_snprintf(conn, name, sizeof(name), \"%.*s%c%s\",\n (int) (e - p), p, DIRSEP, PASSWORDS_FILE_NAME);\n fp = mg_fopen(name, \"r\");\n }\n\n return fp;\n}\n\n// Parsed Authorization header\nstruct ah {\n char *user, *uri, *cnonce, *response, *qop, *nc, *nonce;\n};\n\nstatic int parse_auth_header(struct mg_connection *conn, char *buf,\n size_t buf_size, struct ah *ah) {\n char *name, *value, *s;\n const char *auth_header;\n\n if ((auth_header = mg_get_header(conn, \"Authorization\")) == NULL ||\n mg_strncasecmp(auth_header, \"Digest \", 7) != 0) {\n return 0;\n }\n\n // Make modifiable copy of the auth header\n (void) mg_strlcpy(buf, auth_header + 7, buf_size);\n\n s = buf;\n (void) memset(ah, 0, sizeof(*ah));\n\n // Parse authorization header\n for (;;) {\n // Gobble initial spaces\n while (isspace(* (unsigned char *) s)) {\n s++;\n }\n name = skip_quoted(&s, \"=\", \" \", 0);\n // Value is either quote-delimited, or ends at first comma or space.\n if (s[0] == '\\\"') {\n s++;\n value = skip_quoted(&s, \"\\\"\", \" \", '\\\\');\n if (s[0] == ',') {\n s++;\n }\n } else {\n value = skip_quoted(&s, \", \", \" \", 0); // IE uses commas, FF uses spaces\n }\n if (*name == '\\0') {\n break;\n }\n\n if (!strcmp(name, \"username\")) {\n ah->user = value;\n } else if (!strcmp(name, \"cnonce\")) {\n ah->cnonce = value;\n } else if (!strcmp(name, \"response\")) {\n ah->response = value;\n } else if (!strcmp(name, \"uri\")) {\n ah->uri = value;\n } else if (!strcmp(name, \"qop\")) {\n ah->qop = value;\n } else if (!strcmp(name, \"nc\")) {\n ah->nc = value;\n } else if (!strcmp(name, \"nonce\")) {\n ah->nonce = value;\n }\n }\n\n // CGI needs it as REMOTE_USER\n if (ah->user != NULL) {\n conn->request_info.remote_user = mg_strdup(ah->user);\n } else {\n return 0;\n }\n\n return 1;\n}\n\n// Authorize against the opened passwords file. Return 1 if authorized.\nstatic int authorize(struct mg_connection *conn, FILE *fp) {\n struct ah ah;\n char line[256], f_user[256], ha1[256], f_domain[256], buf[BUFSIZ];\n\n if (!parse_auth_header(conn, buf, sizeof(buf), &ah)) {\n return 0;\n }\n\n // Loop over passwords file\n while (fgets(line, sizeof(line), fp) != NULL) {\n if (sscanf(line, \"%[^:]:%[^:]:%s\", f_user, f_domain, ha1) != 3) {\n continue;\n }\n\n if (!strcmp(ah.user, f_user) &&\n !strcmp(conn->ctx->config[AUTHENTICATION_DOMAIN], f_domain))\n return check_password(\n conn->request_info.request_method,\n ha1, ah.uri, ah.nonce, ah.nc, ah.cnonce, ah.qop,\n ah.response);\n }\n\n return 0;\n}\n\n// Return 1 if request is authorised, 0 otherwise.\nstatic int check_authorization(struct mg_connection *conn, const char *path) {\n FILE *fp;\n char fname[PATH_MAX];\n struct vec uri_vec, filename_vec;\n const char *list;\n int authorized;\n\n fp = NULL;\n authorized = 1;\n\n list = conn->ctx->config[PROTECT_URI];\n while ((list = next_option(list, &uri_vec, &filename_vec)) != NULL) {\n if (!memcmp(conn->request_info.uri, uri_vec.ptr, uri_vec.len)) {\n (void) mg_snprintf(conn, fname, sizeof(fname), \"%.*s\",\n filename_vec.len, filename_vec.ptr);\n if ((fp = mg_fopen(fname, \"r\")) == NULL) {\n cry(conn, \"%s: cannot open %s: %s\", __func__, fname, strerror(errno));\n }\n break;\n }\n }\n\n if (fp == NULL) {\n fp = open_auth_file(conn, path);\n }\n\n if (fp != NULL) {\n authorized = authorize(conn, fp);\n (void) fclose(fp);\n }\n\n return authorized;\n}\n\nstatic void send_authorization_request(struct mg_connection *conn) {\n conn->request_info.status_code = 401;\n (void) mg_printf(conn,\n \"HTTP/1.1 401 Unauthorized\\r\\n\"\n \"Content-Length: 0\\r\\n\"\n \"WWW-Authenticate: Digest qop=\\\"auth\\\", \"\n \"realm=\\\"%s\\\", nonce=\\\"%lu\\\"\\r\\n\\r\\n\",\n conn->ctx->config[AUTHENTICATION_DOMAIN],\n (unsigned long) time(NULL));\n}\n\nstatic int is_authorized_for_put(struct mg_connection *conn) {\n FILE *fp;\n int ret = 0;\n\n fp = conn->ctx->config[PUT_DELETE_PASSWORDS_FILE] == NULL ? NULL :\n mg_fopen(conn->ctx->config[PUT_DELETE_PASSWORDS_FILE], \"r\");\n\n if (fp != NULL) {\n ret = authorize(conn, fp);\n (void) fclose(fp);\n }\n\n return ret;\n}\n\nint mg_modify_passwords_file(const char *fname, const char *domain,\n const char *user, const char *pass) {\n int found;\n char line[512], u[512], d[512], ha1[33], tmp[PATH_MAX];\n FILE *fp, *fp2;\n\n found = 0;\n fp = fp2 = NULL;\n\n // Regard empty password as no password - remove user record.\n if (pass != NULL && pass[0] == '\\0') {\n pass = NULL;\n }\n\n (void) snprintf(tmp, sizeof(tmp), \"%s.tmp\", fname);\n\n // Create the file if does not exist\n if ((fp = mg_fopen(fname, \"a+\")) != NULL) {\n (void) fclose(fp);\n }\n\n // Open the given file and temporary file\n if ((fp = mg_fopen(fname, \"r\")) == NULL) {\n return 0;\n } else if ((fp2 = mg_fopen(tmp, \"w+\")) == NULL) {\n fclose(fp);\n return 0;\n }\n\n // Copy the stuff to temporary file\n while (fgets(line, sizeof(line), fp) != NULL) {\n if (sscanf(line, \"%[^:]:%[^:]:%*s\", u, d) != 2) {\n continue;\n }\n\n if (!strcmp(u, user) && !strcmp(d, domain)) {\n found++;\n if (pass != NULL) {\n mg_md5(ha1, user, \":\", domain, \":\", pass, NULL);\n fprintf(fp2, \"%s:%s:%s\\n\", user, domain, ha1);\n }\n } else {\n (void) fprintf(fp2, \"%s\", line);\n }\n }\n\n // If new user, just add it\n if (!found && pass != NULL) {\n mg_md5(ha1, user, \":\", domain, \":\", pass, NULL);\n (void) fprintf(fp2, \"%s:%s:%s\\n\", user, domain, ha1);\n }\n\n // Close files\n (void) fclose(fp);\n (void) fclose(fp2);\n\n // Put the temp file in place of real file\n (void) mg_remove(fname);\n (void) mg_rename(tmp, fname);\n\n return 1;\n}\n\nstruct de {\n struct mg_connection *conn;\n char *file_name;\n struct mgstat st;\n};\n\nstatic void url_encode(const char *src, char *dst, size_t dst_len) {\n static const char *dont_escape = \"._-$,;~()\";\n static const char *hex = \"0123456789abcdef\";\n const char *end = dst + dst_len - 1;\n\n for (; *src != '\\0' && dst < end; src++, dst++) {\n if (isalnum(*(const unsigned char *) src) ||\n strchr(dont_escape, * (const unsigned char *) src) != NULL) {\n *dst = *src;\n } else if (dst + 2 < end) {\n dst[0] = '%';\n dst[1] = hex[(* (const unsigned char *) src) >> 4];\n dst[2] = hex[(* (const unsigned char *) src) & 0xf];\n dst += 2;\n }\n }\n\n *dst = '\\0';\n}\n\nstatic void print_dir_entry(struct de *de) {\n char size[64], mod[64], href[PATH_MAX];\n\n if (de->st.is_directory) {\n (void) mg_snprintf(de->conn, size, sizeof(size), \"%s\", \"[DIRECTORY]\");\n } else {\n // We use (signed) cast below because MSVC 6 compiler cannot\n // convert unsigned __int64 to double. Sigh.\n if (de->st.size < 1024) {\n (void) mg_snprintf(de->conn, size, sizeof(size),\n \"%lu\", (unsigned long) de->st.size);\n } else if (de->st.size < 1024 * 1024) {\n (void) mg_snprintf(de->conn, size, sizeof(size),\n \"%.1fk\", (double) de->st.size / 1024.0);\n } else if (de->st.size < 1024 * 1024 * 1024) {\n (void) mg_snprintf(de->conn, size, sizeof(size),\n \"%.1fM\", (double) de->st.size / 1048576);\n } else {\n (void) mg_snprintf(de->conn, size, sizeof(size),\n \"%.1fG\", (double) de->st.size / 1073741824);\n }\n }\n (void) strftime(mod, sizeof(mod), \"%d-%b-%Y %H:%M\", localtime(&de->st.mtime));\n url_encode(de->file_name, href, sizeof(href));\n de->conn->num_bytes_sent += mg_printf(de->conn,\n \"%s%s\"\n \" %s  %s\\n\",\n de->conn->request_info.uri, href, de->st.is_directory ? \"/\" : \"\",\n de->file_name, de->st.is_directory ? \"/\" : \"\", mod, size);\n}\n\n// This function is called from send_directory() and used for\n// sorting directory entries by size, or name, or modification time.\n// On windows, __cdecl specification is needed in case if project is built\n// with __stdcall convention. qsort always requires __cdels callback.\nstatic int WINCDECL compare_dir_entries(const void *p1, const void *p2) {\n const struct de *a = (const struct de *) p1, *b = (const struct de *) p2;\n const char *query_string = a->conn->request_info.query_string;\n int cmp_result = 0;\n\n if (query_string == NULL) {\n query_string = \"na\";\n }\n\n if (a->st.is_directory && !b->st.is_directory) {\n return -1; // Always put directories on top\n } else if (!a->st.is_directory && b->st.is_directory) {\n return 1; // Always put directories on top\n } else if (*query_string == 'n') {\n cmp_result = strcmp(a->file_name, b->file_name);\n } else if (*query_string == 's') {\n cmp_result = a->st.size == b->st.size ? 0 :\n a->st.size > b->st.size ? 1 : -1;\n } else if (*query_string == 'd') {\n cmp_result = a->st.mtime == b->st.mtime ? 0 :\n a->st.mtime > b->st.mtime ? 1 : -1;\n }\n\n return query_string[1] == 'd' ? -cmp_result : cmp_result;\n}\n\nstatic int scan_directory(struct mg_connection *conn, const char *dir,\n void *data, void (*cb)(struct de *, void *)) {\n char path[PATH_MAX];\n struct dirent *dp;\n DIR *dirp;\n struct de de;\n\n if ((dirp = opendir(dir)) == NULL) {\n return 0;\n } else {\n de.conn = conn;\n\n while ((dp = readdir(dirp)) != NULL) {\n // Do not show current dir and passwords file\n if (!strcmp(dp->d_name, \".\") ||\n !strcmp(dp->d_name, \"..\") ||\n !strcmp(dp->d_name, PASSWORDS_FILE_NAME))\n continue;\n\n mg_snprintf(conn, path, sizeof(path), \"%s%c%s\", dir, DIRSEP, dp->d_name);\n\n // If we don't memset stat structure to zero, mtime will have\n // garbage and strftime() will segfault later on in\n // print_dir_entry(). memset is required only if mg_stat()\n // fails. For more details, see\n // http://code.google.com/p/mongoose/issues/detail?id=79\n if (mg_stat(path, &de.st) != 0) {\n memset(&de.st, 0, sizeof(de.st));\n }\n de.file_name = dp->d_name;\n\n cb(&de, data);\n }\n (void) closedir(dirp);\n }\n return 1;\n}\n\nstruct dir_scan_data {\n struct de *entries;\n int num_entries;\n int arr_size;\n};\n\nstatic void dir_scan_callback(struct de *de, void *data) {\n struct dir_scan_data *dsd = (struct dir_scan_data *) data;\n\n if (dsd->entries == NULL || dsd->num_entries >= dsd->arr_size) {\n dsd->arr_size *= 2;\n dsd->entries = (struct de *) realloc(dsd->entries, dsd->arr_size *\n sizeof(dsd->entries[0]));\n }\n if (dsd->entries == NULL) {\n // TODO(lsm): propagate an error to the caller\n dsd->num_entries = 0;\n } else {\n dsd->entries[dsd->num_entries].file_name = mg_strdup(de->file_name);\n dsd->entries[dsd->num_entries].st = de->st;\n dsd->entries[dsd->num_entries].conn = de->conn;\n dsd->num_entries++;\n }\n}\n\nstatic void handle_directory_request(struct mg_connection *conn,\n const char *dir) {\n int i, sort_direction;\n struct dir_scan_data data = { NULL, 0, 128 };\n\n if (!scan_directory(conn, dir, &data, dir_scan_callback)) {\n send_http_error(conn, 500, \"Cannot open directory\",\n \"Error: opendir(%s): %s\", dir, strerror(ERRNO));\n return;\n }\n\n sort_direction = conn->request_info.query_string != NULL &&\n conn->request_info.query_string[1] == 'd' ? 'a' : 'd';\n\n conn->must_close = 1;\n mg_printf(conn, \"%s\",\n \"HTTP/1.1 200 OK\\r\\n\"\n \"Connection: close\\r\\n\"\n \"Content-Type: text/html; charset=utf-8\\r\\n\\r\\n\");\n\n conn->num_bytes_sent += mg_printf(conn,\n \"Index of %s\"\n \"\"\n \"

Index of %s

\"\n      \"\"\n      \"\"\n      \"\"\n      \"\",\n      conn->request_info.uri, conn->request_info.uri,\n      sort_direction, sort_direction, sort_direction);\n\n  // Print first entry - link to a parent directory\n  conn->num_bytes_sent += mg_printf(conn,\n      \"\"\n      \"\\n\",\n      conn->request_info.uri, \"..\", \"Parent directory\", \"-\", \"-\");\n\n  // Sort and print directory entries\n  qsort(data.entries, (size_t) data.num_entries, sizeof(data.entries[0]),\n        compare_dir_entries);\n  for (i = 0; i < data.num_entries; i++) {\n    print_dir_entry(&data.entries[i]);\n    free(data.entries[i].file_name);\n  }\n  free(data.entries);\n\n  conn->num_bytes_sent += mg_printf(conn, \"%s\", \"
NameModifiedSize
%s %s  %s
\");\n  conn->request_info.status_code = 200;\n}\n\n// Send len bytes from the opened file to the client.\nstatic void send_file_data(struct mg_connection *conn, FILE *fp, int64_t len) {\n  char buf[BUFSIZ];\n  int to_read, num_read, num_written;\n\n  while (len > 0) {\n    // Calculate how much to read from the file in the buffer\n    to_read = sizeof(buf);\n    if ((int64_t) to_read > len)\n      to_read = (int) len;\n\n    // Read from file, exit the loop on error\n    if ((num_read = fread(buf, 1, (size_t)to_read, fp)) == 0)\n      break;\n\n    // Send read bytes to the client, exit the loop on error\n    if ((num_written = mg_write(conn, buf, (size_t)num_read)) != num_read)\n      break;\n\n    // Both read and were successful, adjust counters\n    conn->num_bytes_sent += num_written;\n    len -= num_written;\n  }\n}\n\nstatic int parse_range_header(const char *header, int64_t *a, int64_t *b) {\n  return sscanf(header, \"bytes=%\" INT64_FMT \"-%\" INT64_FMT, a, b);\n}\n\nstatic void gmt_time_string(char *buf, size_t buf_len, time_t *t) {\n  strftime(buf, buf_len, \"%a, %d %b %Y %H:%M:%S GMT\", gmtime(t));\n}\n\nstatic void handle_file_request(struct mg_connection *conn, const char *path,\n                                struct mgstat *stp) {\n  char date[64], lm[64], etag[64], range[64];\n  const char *msg = \"OK\", *hdr;\n  time_t curtime = time(NULL);\n  int64_t cl, r1, r2;\n  struct vec mime_vec;\n  FILE *fp;\n  int n;\n\n  get_mime_type(conn->ctx, path, &mime_vec);\n  cl = stp->size;\n  conn->request_info.status_code = 200;\n  range[0] = '\\0';\n\n  if ((fp = mg_fopen(path, \"rb\")) == NULL) {\n    send_http_error(conn, 500, http_500_error,\n        \"fopen(%s): %s\", path, strerror(ERRNO));\n    return;\n  }\n  set_close_on_exec(fileno(fp));\n\n  // If Range: header specified, act accordingly\n  r1 = r2 = 0;\n  hdr = mg_get_header(conn, \"Range\");\n  if (hdr != NULL && (n = parse_range_header(hdr, &r1, &r2)) > 0) {\n    conn->request_info.status_code = 206;\n    (void) fseeko(fp, (off_t) r1, SEEK_SET);\n    cl = n == 2 ? r2 - r1 + 1: cl - r1;\n    (void) mg_snprintf(conn, range, sizeof(range),\n        \"Content-Range: bytes \"\n        \"%\" INT64_FMT \"-%\"\n        INT64_FMT \"/%\" INT64_FMT \"\\r\\n\",\n        r1, r1 + cl - 1, stp->size);\n    msg = \"Partial Content\";\n  }\n\n  // Prepare Etag, Date, Last-Modified headers. Must be in UTC, according to\n  // http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.3\n  gmt_time_string(date, sizeof(date), &curtime);\n  gmt_time_string(lm, sizeof(lm), &stp->mtime);\n  (void) mg_snprintf(conn, etag, sizeof(etag), \"%lx.%lx\",\n      (unsigned long) stp->mtime, (unsigned long) stp->size);\n\n  (void) mg_printf(conn,\n      \"HTTP/1.1 %d %s\\r\\n\"\n      \"Date: %s\\r\\n\"\n      \"Last-Modified: %s\\r\\n\"\n      \"Etag: \\\"%s\\\"\\r\\n\"\n      \"Content-Type: %.*s\\r\\n\"\n      \"Content-Length: %\" INT64_FMT \"\\r\\n\"\n      \"Connection: %s\\r\\n\"\n      \"Accept-Ranges: bytes\\r\\n\"\n      \"%s\\r\\n\",\n      conn->request_info.status_code, msg, date, lm, etag, (int) mime_vec.len,\n      mime_vec.ptr, cl, suggest_connection_header(conn), range);\n\n  if (strcmp(conn->request_info.request_method, \"HEAD\") != 0) {\n    send_file_data(conn, fp, cl);\n  }\n  (void) fclose(fp);\n}\n\nvoid mg_send_file(struct mg_connection *conn, const char *path) {\n  struct mgstat st;\n  if (mg_stat(path, &st) == 0) {\n    handle_file_request(conn, path, &st);\n  } else {\n    send_http_error(conn, 404, \"Not Found\", \"%s\", \"File not found\");\n  }\n}\n\n\n// Parse HTTP headers from the given buffer, advance buffer to the point\n// where parsing stopped.\nstatic void parse_http_headers(char **buf, struct mg_request_info *ri) {\n  int i;\n\n  for (i = 0; i < (int) ARRAY_SIZE(ri->http_headers); i++) {\n    ri->http_headers[i].name = skip_quoted(buf, \":\", \" \", 0);\n    ri->http_headers[i].value = skip(buf, \"\\r\\n\");\n    if (ri->http_headers[i].name[0] == '\\0')\n      break;\n    ri->num_headers = i + 1;\n  }\n}\n\nstatic int is_valid_http_method(const char *method) {\n  return !strcmp(method, \"GET\") || !strcmp(method, \"POST\") ||\n    !strcmp(method, \"HEAD\") || !strcmp(method, \"CONNECT\") ||\n    !strcmp(method, \"PUT\") || !strcmp(method, \"DELETE\") ||\n    !strcmp(method, \"OPTIONS\") || !strcmp(method, \"PROPFIND\");\n}\n\n// Parse HTTP request, fill in mg_request_info structure.\nstatic int parse_http_request(char *buf, struct mg_request_info *ri) {\n  int status = 0;\n\n  // RFC says that all initial whitespaces should be ingored\n  while (*buf != '\\0' && isspace(* (unsigned char *) buf)) {\n    buf++;\n  }\n\n  ri->request_method = skip(&buf, \" \");\n  ri->uri = skip(&buf, \" \");\n  ri->http_version = skip(&buf, \"\\r\\n\");\n\n  if (is_valid_http_method(ri->request_method) &&\n      strncmp(ri->http_version, \"HTTP/\", 5) == 0) {\n    ri->http_version += 5;   // Skip \"HTTP/\"\n    parse_http_headers(&buf, ri);\n    status = 1;\n  }\n\n  return status;\n}\n\n// Keep reading the input (either opened file descriptor fd, or socket sock,\n// or SSL descriptor ssl) into buffer buf, until \\r\\n\\r\\n appears in the\n// buffer (which marks the end of HTTP request). Buffer buf may already\n// have some data. The length of the data is stored in nread.\n// Upon every read operation, increase nread by the number of bytes read.\nstatic int read_request(FILE *fp, SOCKET sock, SSL *ssl, char *buf, int bufsiz,\n                        int *nread) {\n  int n, request_len;\n\n  request_len = 0;\n  while (*nread < bufsiz && request_len == 0) {\n    n = pull(fp, sock, ssl, buf + *nread, bufsiz - *nread);\n    if (n <= 0) {\n      break;\n    } else {\n      *nread += n;\n      request_len = get_request_len(buf, *nread);\n    }\n  }\n\n  return request_len;\n}\n\n// For given directory path, substitute it to valid index file.\n// Return 0 if index file has been found, -1 if not found.\n// If the file is found, it's stats is returned in stp.\nstatic int substitute_index_file(struct mg_connection *conn, char *path,\n                                 size_t path_len, struct mgstat *stp) {\n  const char *list = conn->ctx->config[INDEX_FILES];\n  struct mgstat st;\n  struct vec filename_vec;\n  size_t n = strlen(path);\n  int found = 0;\n\n  // The 'path' given to us points to the directory. Remove all trailing\n  // directory separator characters from the end of the path, and\n  // then append single directory separator character.\n  while (n > 0 && IS_DIRSEP_CHAR(path[n - 1])) {\n    n--;\n  }\n  path[n] = DIRSEP;\n\n  // Traverse index files list. For each entry, append it to the given\n  // path and see if the file exists. If it exists, break the loop\n  while ((list = next_option(list, &filename_vec, NULL)) != NULL) {\n\n    // Ignore too long entries that may overflow path buffer\n    if (filename_vec.len > path_len - (n + 2))\n      continue;\n\n    // Prepare full path to the index file\n    (void) mg_strlcpy(path + n + 1, filename_vec.ptr, filename_vec.len + 1);\n\n    // Does it exist?\n    if (mg_stat(path, &st) == 0) {\n      // Yes it does, break the loop\n      *stp = st;\n      found = 1;\n      break;\n    }\n  }\n\n  // If no index file exists, restore directory path\n  if (!found) {\n    path[n] = '\\0';\n  }\n\n  return found;\n}\n\n// Return True if we should reply 304 Not Modified.\nstatic int is_not_modified(const struct mg_connection *conn,\n                           const struct mgstat *stp) {\n  const char *ims = mg_get_header(conn, \"If-Modified-Since\");\n  return ims != NULL && stp->mtime <= parse_date_string(ims);\n}\n\nstatic int forward_body_data(struct mg_connection *conn, FILE *fp,\n                             SOCKET sock, SSL *ssl) {\n  const char *expect, *buffered;\n  char buf[BUFSIZ];\n  int to_read, nread, buffered_len, success = 0;\n\n  expect = mg_get_header(conn, \"Expect\");\n  assert(fp != NULL);\n\n  if (conn->content_len == -1) {\n    send_http_error(conn, 411, \"Length Required\", \"\");\n  } else if (expect != NULL && mg_strcasecmp(expect, \"100-continue\")) {\n    send_http_error(conn, 417, \"Expectation Failed\", \"\");\n  } else {\n    if (expect != NULL) {\n      (void) mg_printf(conn, \"%s\", \"HTTP/1.1 100 Continue\\r\\n\\r\\n\");\n    }\n\n    buffered = conn->buf + conn->request_len;\n    buffered_len = conn->data_len - conn->request_len;\n    assert(buffered_len >= 0);\n    assert(conn->consumed_content == 0);\n\n    if (buffered_len > 0) {\n      if ((int64_t) buffered_len > conn->content_len) {\n        buffered_len = (int) conn->content_len;\n      }\n      push(fp, sock, ssl, buffered, (int64_t) buffered_len);\n      conn->consumed_content += buffered_len;\n    }\n\n    while (conn->consumed_content < conn->content_len) {\n      to_read = sizeof(buf);\n      if ((int64_t) to_read > conn->content_len - conn->consumed_content) {\n        to_read = (int) (conn->content_len - conn->consumed_content);\n      }\n      nread = pull(NULL, conn->client.sock, conn->ssl, buf, to_read);\n      if (nread <= 0 || push(fp, sock, ssl, buf, nread) != nread) {\n        break;\n      }\n      conn->consumed_content += nread;\n    }\n\n    if (conn->consumed_content == conn->content_len) {\n      success = 1;\n    }\n\n    // Each error code path in this function must send an error\n    if (!success) {\n      send_http_error(conn, 577, http_500_error, \"\");\n    }\n  }\n\n  return success;\n}\n\n#if !defined(NO_CGI)\n// This structure helps to create an environment for the spawned CGI program.\n// Environment is an array of \"VARIABLE=VALUE\\0\" ASCIIZ strings,\n// last element must be NULL.\n// However, on Windows there is a requirement that all these VARIABLE=VALUE\\0\n// strings must reside in a contiguous buffer. The end of the buffer is\n// marked by two '\\0' characters.\n// We satisfy both worlds: we create an envp array (which is vars), all\n// entries are actually pointers inside buf.\nstruct cgi_env_block {\n  struct mg_connection *conn;\n  char buf[CGI_ENVIRONMENT_SIZE]; // Environment buffer\n  int len; // Space taken\n  char *vars[MAX_CGI_ENVIR_VARS]; // char **envp\n  int nvars; // Number of variables\n};\n\n// Append VARIABLE=VALUE\\0 string to the buffer, and add a respective\n// pointer into the vars array.\nstatic char *addenv(struct cgi_env_block *block, const char *fmt, ...) {\n  int n, space;\n  char *added;\n  va_list ap;\n\n  // Calculate how much space is left in the buffer\n  space = sizeof(block->buf) - block->len - 2;\n  assert(space >= 0);\n\n  // Make a pointer to the free space int the buffer\n  added = block->buf + block->len;\n\n  // Copy VARIABLE=VALUE\\0 string into the free space\n  va_start(ap, fmt);\n  n = mg_vsnprintf(block->conn, added, (size_t) space, fmt, ap);\n  va_end(ap);\n\n  // Make sure we do not overflow buffer and the envp array\n  if (n > 0 && n < space &&\n      block->nvars < (int) ARRAY_SIZE(block->vars) - 2) {\n    // Append a pointer to the added string into the envp array\n    block->vars[block->nvars++] = block->buf + block->len;\n    // Bump up used length counter. Include \\0 terminator\n    block->len += n + 1;\n  }\n\n  return added;\n}\n\nstatic void prepare_cgi_environment(struct mg_connection *conn,\n                                    const char *prog,\n                                    struct cgi_env_block *blk) {\n  const char *s, *slash;\n  struct vec var_vec;\n  char *p, src_addr[20];\n  int  i;\n\n  blk->len = blk->nvars = 0;\n  blk->conn = conn;\n  sockaddr_to_string(src_addr, sizeof(src_addr), &conn->client.rsa);\n\n  addenv(blk, \"SERVER_NAME=%s\", conn->ctx->config[AUTHENTICATION_DOMAIN]);\n  addenv(blk, \"SERVER_ROOT=%s\", conn->ctx->config[DOCUMENT_ROOT]);\n  addenv(blk, \"DOCUMENT_ROOT=%s\", conn->ctx->config[DOCUMENT_ROOT]);\n\n  // Prepare the environment block\n  addenv(blk, \"%s\", \"GATEWAY_INTERFACE=CGI/1.1\");\n  addenv(blk, \"%s\", \"SERVER_PROTOCOL=HTTP/1.1\");\n  addenv(blk, \"%s\", \"REDIRECT_STATUS=200\"); // For PHP\n\n  // TODO(lsm): fix this for IPv6 case\n  addenv(blk, \"SERVER_PORT=%d\", ntohs(conn->client.lsa.sin.sin_port));\n\n  addenv(blk, \"REQUEST_METHOD=%s\", conn->request_info.request_method);\n  addenv(blk, \"REMOTE_ADDR=%s\", src_addr);\n  addenv(blk, \"REMOTE_PORT=%d\", conn->request_info.remote_port);\n  addenv(blk, \"REQUEST_URI=%s\", conn->request_info.uri);\n\n  // SCRIPT_NAME\n  assert(conn->request_info.uri[0] == '/');\n  slash = strrchr(conn->request_info.uri, '/');\n  if ((s = strrchr(prog, '/')) == NULL)\n    s = prog;\n  addenv(blk, \"SCRIPT_NAME=%.*s%s\", slash - conn->request_info.uri,\n         conn->request_info.uri, s);\n\n  addenv(blk, \"SCRIPT_FILENAME=%s\", prog);\n  addenv(blk, \"PATH_TRANSLATED=%s\", prog);\n  addenv(blk, \"HTTPS=%s\", conn->ssl == NULL ? \"off\" : \"on\");\n\n  if ((s = mg_get_header(conn, \"Content-Type\")) != NULL)\n    addenv(blk, \"CONTENT_TYPE=%s\", s);\n\n  if (conn->request_info.query_string != NULL)\n    addenv(blk, \"QUERY_STRING=%s\", conn->request_info.query_string);\n\n  if ((s = mg_get_header(conn, \"Content-Length\")) != NULL)\n    addenv(blk, \"CONTENT_LENGTH=%s\", s);\n\n  if ((s = getenv(\"PATH\")) != NULL)\n    addenv(blk, \"PATH=%s\", s);\n\n  if (conn->path_info != NULL) {\n    addenv(blk, \"PATH_INFO=%s\", conn->path_info);\n  }\n\n#if defined(_WIN32)\n  if ((s = getenv(\"COMSPEC\")) != NULL) {\n    addenv(blk, \"COMSPEC=%s\", s);\n  }\n  if ((s = getenv(\"SYSTEMROOT\")) != NULL) {\n    addenv(blk, \"SYSTEMROOT=%s\", s);\n  }\n  if ((s = getenv(\"SystemDrive\")) != NULL) {\n    addenv(blk, \"SystemDrive=%s\", s);\n  }\n#else\n  if ((s = getenv(\"LD_LIBRARY_PATH\")) != NULL)\n    addenv(blk, \"LD_LIBRARY_PATH=%s\", s);\n#endif // _WIN32\n\n  if ((s = getenv(\"PERLLIB\")) != NULL)\n    addenv(blk, \"PERLLIB=%s\", s);\n\n  if (conn->request_info.remote_user != NULL) {\n    addenv(blk, \"REMOTE_USER=%s\", conn->request_info.remote_user);\n    addenv(blk, \"%s\", \"AUTH_TYPE=Digest\");\n  }\n\n  // Add all headers as HTTP_* variables\n  for (i = 0; i < conn->request_info.num_headers; i++) {\n    p = addenv(blk, \"HTTP_%s=%s\",\n        conn->request_info.http_headers[i].name,\n        conn->request_info.http_headers[i].value);\n\n    // Convert variable name into uppercase, and change - to _\n    for (; *p != '=' && *p != '\\0'; p++) {\n      if (*p == '-')\n        *p = '_';\n      *p = (char) toupper(* (unsigned char *) p);\n    }\n  }\n\n  // Add user-specified variables\n  s = conn->ctx->config[CGI_ENVIRONMENT];\n  while ((s = next_option(s, &var_vec, NULL)) != NULL) {\n    addenv(blk, \"%.*s\", var_vec.len, var_vec.ptr);\n  }\n\n  blk->vars[blk->nvars++] = NULL;\n  blk->buf[blk->len++] = '\\0';\n\n  assert(blk->nvars < (int) ARRAY_SIZE(blk->vars));\n  assert(blk->len > 0);\n  assert(blk->len < (int) sizeof(blk->buf));\n}\n\nstatic void handle_cgi_request(struct mg_connection *conn, const char *prog) {\n  int headers_len, data_len, i, fd_stdin[2], fd_stdout[2];\n  const char *status, *status_text;\n  char buf[BUFSIZ], *pbuf, dir[PATH_MAX], *p;\n  struct mg_request_info ri;\n  struct cgi_env_block blk;\n  FILE *in, *out;\n  pid_t pid;\n\n  prepare_cgi_environment(conn, prog, &blk);\n\n  // CGI must be executed in its own directory. 'dir' must point to the\n  // directory containing executable program, 'p' must point to the\n  // executable program name relative to 'dir'.\n  (void) mg_snprintf(conn, dir, sizeof(dir), \"%s\", prog);\n  if ((p = strrchr(dir, DIRSEP)) != NULL) {\n    *p++ = '\\0';\n  } else {\n    dir[0] = '.', dir[1] = '\\0';\n    p = (char *) prog;\n  }\n\n  pid = (pid_t) -1;\n  fd_stdin[0] = fd_stdin[1] = fd_stdout[0] = fd_stdout[1] = -1;\n  in = out = NULL;\n\n  if (pipe(fd_stdin) != 0 || pipe(fd_stdout) != 0) {\n    send_http_error(conn, 500, http_500_error,\n        \"Cannot create CGI pipe: %s\", strerror(ERRNO));\n    goto done;\n  } else if ((pid = spawn_process(conn, p, blk.buf, blk.vars,\n          fd_stdin[0], fd_stdout[1], dir)) == (pid_t) -1) {\n    goto done;\n  } else if ((in = fdopen(fd_stdin[1], \"wb\")) == NULL ||\n      (out = fdopen(fd_stdout[0], \"rb\")) == NULL) {\n    send_http_error(conn, 500, http_500_error,\n        \"fopen: %s\", strerror(ERRNO));\n    goto done;\n  }\n\n  setbuf(in, NULL);\n  setbuf(out, NULL);\n\n  // spawn_process() must close those!\n  // If we don't mark them as closed, close() attempt before\n  // return from this function throws an exception on Windows.\n  // Windows does not like when closed descriptor is closed again.\n  fd_stdin[0] = fd_stdout[1] = -1;\n\n  // Send POST data to the CGI process if needed\n  if (!strcmp(conn->request_info.request_method, \"POST\") &&\n      !forward_body_data(conn, in, INVALID_SOCKET, NULL)) {\n    goto done;\n  }\n\n  // Now read CGI reply into a buffer. We need to set correct\n  // status code, thus we need to see all HTTP headers first.\n  // Do not send anything back to client, until we buffer in all\n  // HTTP headers.\n  data_len = 0;\n  headers_len = read_request(out, INVALID_SOCKET, NULL,\n      buf, sizeof(buf), &data_len);\n  if (headers_len <= 0) {\n    send_http_error(conn, 500, http_500_error,\n                    \"CGI program sent malformed HTTP headers: [%.*s]\",\n                    data_len, buf);\n    goto done;\n  }\n  pbuf = buf;\n  buf[headers_len - 1] = '\\0';\n  parse_http_headers(&pbuf, &ri);\n\n  // Make up and send the status line\n  status_text = \"OK\";\n  if ((status = get_header(&ri, \"Status\")) != NULL) {\n    conn->request_info.status_code = atoi(status);\n    status_text = status;\n    while (isdigit(* (unsigned char *) status_text) || *status_text == ' ') {\n      status_text++;\n    }\n  } else if (get_header(&ri, \"Location\") != NULL) {\n    conn->request_info.status_code = 302;\n  } else {\n    conn->request_info.status_code = 200;\n  }\n  if (get_header(&ri, \"Connection\") != NULL &&\n      !mg_strcasecmp(get_header(&ri, \"Connection\"), \"keep-alive\")) {\n    conn->must_close = 1;\n  }\n  (void) mg_printf(conn, \"HTTP/1.1 %d %s\\r\\n\", conn->request_info.status_code,\n                   status_text);\n\n  // Send headers\n  for (i = 0; i < ri.num_headers; i++) {\n    mg_printf(conn, \"%s: %s\\r\\n\",\n              ri.http_headers[i].name, ri.http_headers[i].value);\n  }\n  (void) mg_write(conn, \"\\r\\n\", 2);\n\n  // Send chunk of data that may be read after the headers\n  conn->num_bytes_sent += mg_write(conn, buf + headers_len,\n                                   (size_t)(data_len - headers_len));\n\n  // Read the rest of CGI output and send to the client\n  send_file_data(conn, out, INT64_MAX);\n\ndone:\n  if (pid != (pid_t) -1) {\n    kill(pid, SIGKILL);\n  }\n  if (fd_stdin[0] != -1) {\n    (void) close(fd_stdin[0]);\n  }\n  if (fd_stdout[1] != -1) {\n    (void) close(fd_stdout[1]);\n  }\n\n  if (in != NULL) {\n    (void) fclose(in);\n  } else if (fd_stdin[1] != -1) {\n    (void) close(fd_stdin[1]);\n  }\n\n  if (out != NULL) {\n    (void) fclose(out);\n  } else if (fd_stdout[0] != -1) {\n    (void) close(fd_stdout[0]);\n  }\n}\n#endif // !NO_CGI\n\n// For a given PUT path, create all intermediate subdirectories\n// for given path. Return 0 if the path itself is a directory,\n// or -1 on error, 1 if OK.\nstatic int put_dir(const char *path) {\n  char buf[PATH_MAX];\n  const char *s, *p;\n  struct mgstat st;\n  int len, res = 1;\n\n  for (s = p = path + 2; (p = strchr(s, DIRSEP)) != NULL; s = ++p) {\n    len = p - path;\n    if (len >= (int) sizeof(buf)) {\n      res = -1;\n      break;\n    }\n    memcpy(buf, path, len);\n    buf[len] = '\\0';\n\n    // Try to create intermediate directory\n    DEBUG_TRACE((\"mkdir(%s)\", buf));\n    if (mg_stat(buf, &st) == -1 && mg_mkdir(buf, 0755) != 0) {\n      res = -1;\n      break;\n    }\n\n    // Is path itself a directory?\n    if (p[1] == '\\0') {\n      res = 0;\n    }\n  }\n\n  return res;\n}\n\nstatic void put_file(struct mg_connection *conn, const char *path) {\n  struct mgstat st;\n  const char *range;\n  int64_t r1, r2;\n  FILE *fp;\n  int rc;\n\n  conn->request_info.status_code = mg_stat(path, &st) == 0 ? 200 : 201;\n\n  if ((rc = put_dir(path)) == 0) {\n    mg_printf(conn, \"HTTP/1.1 %d OK\\r\\n\\r\\n\", conn->request_info.status_code);\n  } else if (rc == -1) {\n    send_http_error(conn, 500, http_500_error,\n        \"put_dir(%s): %s\", path, strerror(ERRNO));\n  } else if ((fp = mg_fopen(path, \"wb+\")) == NULL) {\n    send_http_error(conn, 500, http_500_error,\n        \"fopen(%s): %s\", path, strerror(ERRNO));\n  } else {\n    set_close_on_exec(fileno(fp));\n    range = mg_get_header(conn, \"Content-Range\");\n    r1 = r2 = 0;\n    if (range != NULL && parse_range_header(range, &r1, &r2) > 0) {\n      conn->request_info.status_code = 206;\n      // TODO(lsm): handle seek error\n      (void) fseeko(fp, (off_t) r1, SEEK_SET);\n    }\n    if (forward_body_data(conn, fp, INVALID_SOCKET, NULL))\n      (void) mg_printf(conn, \"HTTP/1.1 %d OK\\r\\n\\r\\n\",\n          conn->request_info.status_code);\n    (void) fclose(fp);\n  }\n}\n\nstatic void send_ssi_file(struct mg_connection *, const char *, FILE *, int);\n\nstatic void do_ssi_include(struct mg_connection *conn, const char *ssi,\n                           char *tag, int include_level) {\n  char file_name[BUFSIZ], path[PATH_MAX], *p;\n  FILE *fp;\n\n  // sscanf() is safe here, since send_ssi_file() also uses buffer\n  // of size BUFSIZ to get the tag. So strlen(tag) is always < BUFSIZ.\n  if (sscanf(tag, \" virtual=\\\"%[^\\\"]\\\"\", file_name) == 1) {\n    // File name is relative to the webserver root\n    (void) mg_snprintf(conn, path, sizeof(path), \"%s%c%s\",\n        conn->ctx->config[DOCUMENT_ROOT], DIRSEP, file_name);\n  } else if (sscanf(tag, \" file=\\\"%[^\\\"]\\\"\", file_name) == 1) {\n    // File name is relative to the webserver working directory\n    // or it is absolute system path\n    (void) mg_snprintf(conn, path, sizeof(path), \"%s\", file_name);\n  } else if (sscanf(tag, \" \\\"%[^\\\"]\\\"\", file_name) == 1) {\n    // File name is relative to the currect document\n    (void) mg_snprintf(conn, path, sizeof(path), \"%s\", ssi);\n    if ((p = strrchr(path, DIRSEP)) != NULL) {\n      p[1] = '\\0';\n    }\n    (void) mg_snprintf(conn, path + strlen(path),\n        sizeof(path) - strlen(path), \"%s\", file_name);\n  } else {\n    cry(conn, \"Bad SSI #include: [%s]\", tag);\n    return;\n  }\n\n  if ((fp = mg_fopen(path, \"rb\")) == NULL) {\n    cry(conn, \"Cannot open SSI #include: [%s]: fopen(%s): %s\",\n        tag, path, strerror(ERRNO));\n  } else {\n    set_close_on_exec(fileno(fp));\n    if (match_prefix(conn->ctx->config[SSI_EXTENSIONS],\n                     strlen(conn->ctx->config[SSI_EXTENSIONS]), path) > 0) {\n      send_ssi_file(conn, path, fp, include_level + 1);\n    } else {\n      send_file_data(conn, fp, INT64_MAX);\n    }\n    (void) fclose(fp);\n  }\n}\n\n#if !defined(NO_POPEN)\nstatic void do_ssi_exec(struct mg_connection *conn, char *tag) {\n  char cmd[BUFSIZ];\n  FILE *fp;\n\n  if (sscanf(tag, \" \\\"%[^\\\"]\\\"\", cmd) != 1) {\n    cry(conn, \"Bad SSI #exec: [%s]\", tag);\n  } else if ((fp = popen(cmd, \"r\")) == NULL) {\n    cry(conn, \"Cannot SSI #exec: [%s]: %s\", cmd, strerror(ERRNO));\n  } else {\n    send_file_data(conn, fp, INT64_MAX);\n    (void) pclose(fp);\n  }\n}\n#endif // !NO_POPEN\n\nstatic void send_ssi_file(struct mg_connection *conn, const char *path,\n                          FILE *fp, int include_level) {\n  char buf[BUFSIZ];\n  int ch, len, in_ssi_tag;\n\n  if (include_level > 10) {\n    cry(conn, \"SSI #include level is too deep (%s)\", path);\n    return;\n  }\n\n  in_ssi_tag = 0;\n  len = 0;\n\n  while ((ch = fgetc(fp)) != EOF) {\n    if (in_ssi_tag && ch == '>') {\n      in_ssi_tag = 0;\n      buf[len++] = (char) ch;\n      buf[len] = '\\0';\n      assert(len <= (int) sizeof(buf));\n      if (len < 6 || memcmp(buf, \"Bufsize\n    /// 0--------------H...body...T--(Part A)--->Bufsize\n    /// T--(Part B)----H...body....ccccccccccccc>Bufsize\n    /// 0cccccccccccT--H...body....ccccccccccccc>Bufsize\n\n    // First we copy the contents into Part A\n    std::streamsize firstcopy = std::min(clen, bufsize - tail);\n    memcpy(buffer + tail, c, (size_t)firstcopy);\n    // Move the tail to the end\n    tail += firstcopy;\n    // If tail moved to the end wrap around\n    if (tail == bufsize) tail = 0;\n    // If the copy is not complete\n    if (firstcopy < clen) {\n      // Assert: This only happens on wrape around\n      ASSERT_EQ(tail, 0);\n      // Determine what is left to be coppied\n      std::streamsize secondcopy = clen - firstcopy;\n      ASSERT_GT(secondcopy, 0);\n      // Do the copy and advance the pointer\n      memcpy(buffer, c + firstcopy, (size_t)secondcopy);    \n      tail += secondcopy;\n      \n    }\n    return clen;\n  }\n\n  std::streamsize safe_circular_char_buffer::\n  introspective_read(char* &s, std::streamsize clen) {\n    ASSERT_GT(clen,0);\n    // early termination check\n    if(empty() || clen == 0) {\n      s = NULL;\n      return 0;\n    }\n    const std::streamsize curtail = tail;\n    \n    s = buffer + head;\n    // how much we do read?  we can go up to the end of the requested\n    // size or until a looparound\n    // case 1: no looparound  |------H......T----->\n    // case 2: looparound     |...T--H............>\n    std::streamsize available_readlen = 0;\n    const bool loop_around(curtail < head);\n    if (loop_around) available_readlen = bufsize - head; \n    else available_readlen = curtail - head; \n    ASSERT_GE(available_readlen, 0);\n    const std::streamsize actual_readlen =\n      std::min(available_readlen, clen);\n    ASSERT_GT(actual_readlen, 0);\n    return actual_readlen;\n  }\n\n\n  std::streamsize safe_circular_char_buffer::\n  blocking_introspective_read(char* &s, std::streamsize clen) {\n    // try to read\n    std::streamsize ret = introspective_read(s, clen);\n    if (ret != 0) return ret;  \n    // if read failed. acquire the lock and try again\n    while(1) {\n      iswaiting = true;\n      mut.lock();\n      while (empty() && !done) cond.wait(mut);\n      iswaiting = false;\n      mut.unlock();    \n      std::streamsize ret = introspective_read(s, clen);\n      if (ret != 0) return ret;\n      if (done) return 0;\n    }\n  }\n\n  \n  void safe_circular_char_buffer::\n  advance_head(const std::streamsize advance_len) {\n    ASSERT_GE(advance_len, 0);\n    ASSERT_LE(advance_len, size());\n    // advance the head forward as far as possible\n    head += advance_len;\n    // If head wraps around move head to begginning and then offset\n    if (head >= bufsize) head -= bufsize;    \n  } // end of advance head\n\n\n\n} // end of namespace\n\n\n"
  },
  {
    "path": "src/graphlab/util/safe_circular_char_buffer.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef SAFE_CIRCULAR_CHAR_BUFFER_HPP\n#define SAFE_CIRCULAR_CHAR_BUFFER_HPP\n#include \n#include \n#include \n\nnamespace graphlab {\n\n/**\n\\ingroup util\nA non-resizing circular char buffer\nwith thread-safe write operations and a single reader \n*/\nclass safe_circular_char_buffer {\n public:\n  safe_circular_char_buffer(std::streamsize bufsize = 10485760 /*10 MB */);\n\n  ~safe_circular_char_buffer();\n  \n  /**\n   * Stops the buffer and signals any blocking calls.\n   */\n  void stop_reader();\n\n\n  /**\n   * Determine if the buffer is empty\n   */\n  bool empty() const;\n\n  inline bool is_done() const { \n    return done;\n  }\n  \n  inline bool reader_is_blocked() const {\n    return iswaiting;\n  }\n  /**\n   * Get the total contents currently stored in the buffer.\n   */\n  std::streamsize size() const;\n\n  /**\n   * Get the amount of free space reamining in the buffer\n   */\n  std::streamsize free_space() const;\n\n  /** Gets the size of the buffer. \n     \\note: The useable space is reserved_size() - 1 */\n  inline std::streamsize reserved_size() const {\n    return bufsize - 1;\n  }\n\n  \n  /**\n   * Returns 0 if the write doesn't fit\n   *\n   * This function acquires the critical section\n   * to perform the write\n   */\n  std::streamsize write(const char* c, std::streamsize clen);\n\n  /**\n   * Returns 0 if the write doesn't fit\n   *\n   * This does the same as write(), but does not acquire the critical\n   * section. The caller should ensure safety\n   */\n  std::streamsize write_unsafe(const char* c, std::streamsize clen);\n\n\n  /**\n   * Returns a pointer (through s) and a length of the read.  This\n   * pointer is a direct pointer into the internal buffer of this\n   * datastructure. The pointer is valid as long as no other\n   * operations are performed on this structure.  The length of the\n   * introspective_read may be less than the number of bytes\n   * requested. Multiple calls to introspective_read may be necessary\n   * to read all data in the buffer. If the function returns 0, the\n   * buffer is empty.\n   * \n   * No locks are acquired on this call.\n   */  \n  std::streamsize introspective_read(char* &s, std::streamsize clen);\n  \n  \n  /**\n   * Same as introspective read. But blocks until there is something to read\n   * This function does not acquire a critical section.\n   */\n  std::streamsize blocking_introspective_read(char* &s, \n                                              std::streamsize clen);\n\n\n  void advance_head(const std::streamsize advance_len);\n  \n  \n  /** When begin critical section returns, it is \n      guaranteed that no other writer will be touching\n      the tail of the queue */\n  inline void begin_critical_section() {\n    mut.lock();\n  }\n  \n  /** Releases a critical section acquired by begin_critical_section */\n  inline void end_critical_section() {\n    mut.unlock();\n  }\n\n  /** Releases a critical section acquired by begin_critical_section,\n   and signals the reader to begin reading if the reader is blocked */  \n  inline void end_critical_section_with_signal() {\n    cond.signal();\n    mut.unlock();\n  }\n  \n  \n private:\n  char* buffer;\n  std::streamsize bufsize; // current size of the buffer\n\n  /** \n   * points to the head of the queue.  Reader reads from here\n   */\n  std::streamsize head;  \n  \n  /** \n   * points to one past the end of the queue.  writer writes to\n   * here. if tail == head, buffer must be empty\n   */\n  std::streamsize tail;  \n\n  mutex mut;\n  conditional cond;\n  \n  volatile bool done; // Once \n  volatile bool iswaiting;\n};\n\n}\n\n#endif\n\n"
  },
  {
    "path": "src/graphlab/util/small_map.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SMALL_MAP_HPP\n#define GRAPHLAB_SMALL_MAP_HPP\n\n\n#include \n#include \n#include \n#include \n\n#include \n#include \n\n#include \n\n#include \nnamespace graphlab {\n\n  template\n  class small_map {\n    template< size_t T1, size_t T2 >\n    struct max_type { enum max_value { value = T1 < T2? T2 : T1 }; };\n    struct less {\n      bool operator()(const std::pair& a, \n                      const std::pair& b) const {\n        return a.first < b.first;\n      } \n    };\n  \n\n\n  public: \n    \n    typedef small_set< MAX_DIM, std::pair, less > \n    small_set_type;\n    typedef typename small_set_type::value_type value_type;\n    typedef typename small_set_type::iterator   iterator;\n    typedef typename small_set_type::const_iterator const_iterator;\n    typedef KeyT   key_type;\n    typedef ValueT mapped_type;\n\n\n  public:\n    //! construct an empty map\n    small_map() { }\n    \n    //! Construct a map with a single element\n    small_map(const key_type& key, const mapped_type& value) :\n      set(std::make_pair(key, value)) {  }\n    \n\n    //! Get the begin iterator\n    inline iterator begin() { return set.begin(); }\n\n    //! get the end iterator\n    inline iterator end() { return set.end(); }\n\n\n    //! Get the begin iterator\n    inline const_iterator begin() const { return set.begin(); }\n\n    //! Get the end iterator\n    inline const_iterator end() const { return set.end(); }\n\n    //! get the size of the set\n    inline size_t size() const { return set.size(); }\n\n    //! determine if there are any elements in the set\n    inline bool empty() const { return set.empty(); }\n\n    \n    //! test whether the set contains the given element\n    bool contains(const value_type& pair) const {\n      return set.contains(pair);\n    }\n\n    //! test whether the set contains the given element\n    bool contains(const key_type& key) const {\n      const_iterator iter =\n        std::lower_bound(set.begin(), \n                         set.end(), \n                         std::make_pair(key, mapped_type()));\n      return (iter != set.end()) && (iter->first == key);\n    }\n\n    //! test whether the set has the given key\n    inline bool has_key(const key_type& key) {\n      return contains(key);\n    }\n\n\n    //! test whether the set contains the given set of element\n    template\n    bool contains(const small_map& other) const {\n      return set.contains(other.set);\n    }\n\n    template\n    bool operator==(const small_map& other) const {\n      return set == other.set;\n    }\n\n    //! Lookup an element in the map\n    inline const mapped_type& operator[](const key_type& key) const {\n      value_type* const ptr = \n        std::lower_bound(set.begin(), set.end(), \n                         std::make_pair(key, mapped_type()),\n                         less());\n      ASSERT_NE(ptr, set.end());\n      ASSERT_TURE(ptr->first == key);\n      return ptr->second;\n    }\n\n    //! Lookup an element in the map\n    inline mapped_type& operator[](const key_type& key) {\n      value_type* ptr = \n        std::lower_bound(set.begin(), set.end(), \n                         std::make_pair(key, mapped_type()),\n                         less());\n      if(ptr != end() && (key == ptr->first) ) { return ptr->second; }\n      // Add the entry to the map\n      set += std::make_pair(key, ValueT());\n      ptr = \n        std::lower_bound(set.begin(), set.end(),\n                         std::make_pair(key, mapped_type()),\n                         less());\n      ASSERT_NE(ptr, set.end());\n      ASSERT_TRUE(ptr->first == key);\n      return ptr->second;\n    }\n\n    inline mapped_type& safe_find(const key_type& key) {\n      value_type* const ptr = \n        std::lower_bound(set.begin(), set.end(), \n                         std::make_pair(key, mapped_type()),\n                         less());\n      ASSERT_NE(ptr, set.end());\n      ASSERT_TRUE(ptr->first == key);\n      return ptr->second;\n    }\n\n    //! Take the union of two maps\n    template\n    inline small_map::value, KeyT, ValueT> \n    operator+(const small_map& other) const {      \n      typedef small_map::value, KeyT, ValueT >\n        result_type;\n      result_type result;\n      result.set = set + other.set;\n      return result;\n    }\n\n\n\n  private:\n    small_set_type set;\n    \n\n\n  }; // end of small map\n\n\n  template\n  std::ostream&\n  operator<<(std::ostream& out,\n            const graphlab::small_map& map) {\n    typedef std::pair pair_type;\n    size_t index = 0;\n    out << '{';\n    foreach(const pair_type& pair, map) {\n      out << pair.first << \"->\" << pair.second;\n      if(++index < map.size()) out << \", \";\n    }\n    return out << '}';\n  }\n\n}; // end graphlab\n\n#include \n#endif\n\n"
  },
  {
    "path": "src/graphlab/util/small_set.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SMALL_SET_HPP\n#define GRAPHLAB_SMALL_SET_HPP\n\n\n#include \n#include \n#include \n#include \n\n#include \n#include \n\n\n\n#include \nnamespace graphlab {\n\n\n  /**\n   * This class implements a dense set of fixed maximum size which\n   * support quick operations with stack allocation.\n   */\n  template >\n  class small_set {\n  public: // typedefs\n    typedef T value_type;\n    typedef T& reference;\n    typedef const T& const_reference;\n    typedef ptrdiff_t difference_type;\n    typedef size_t size_type;\n    typedef T* iterator;\n    typedef const T* const_iterator;\n    enum sizes {max_dim_type = MAX_DIM };\n\n\n    template< size_t T1, size_t T2 >\n    struct max_type { enum max_value { value = T1 < T2? T2 : T1 }; };\n\n    struct Equals { \n      inline bool operator()(const T& a, const T& b) const {\n        return !Less()(a,b) && !Less()(b,a);\n      }\n    }; // end of equals\n\n  public:\n    //! Construct an empty set\n    small_set() : nelems(0) { }\n\n    //! Create a stack set with just one element\n    small_set(const T& elem) : nelems(1) { values[0] = elem; }\n    \n    /**\n     * Create a stack from an std set by adding each element one at a\n     * time\n     */\n    template\n    small_set(const std::set& other) : nelems(other.size()) { \n      ASSERT_LE(nelems, MAX_DIM);\n      size_t index = 0;\n      foreach(const OtherT& elem, other) values[index++] = elem;\n    }\n\n\n    /**\n     * Create a stack from an std set by adding each element one at a\n     * time\n     */\n    template\n    small_set(const small_set& other) : \n      nelems(other.size()) { \n      ASSERT_LE(nelems, MAX_DIM);\n      size_t index = 0;\n      for(const T* elem = other.begin(); elem != other.end(); ++elem) \n        values[index++] = *elem;\n    }\n\n\n    //! Get the begin iterator\n    inline T* begin() { return values; }\n\n    //! get the end iterator\n    inline T* end() { return values + nelems; }\n\n\n    //! Get the begin iterator\n    inline const T* begin() const { return values; }\n\n    //! Get the end iterator\n    inline const T* end() const { return values + nelems; }\n\n    //! get the size of the set\n    inline size_t size() const { return nelems; }\n\n    //! determine if there are any elements in the set\n    inline bool empty() const { return size() == 0; }\n\n    \n    //! test whether the set contains the given element\n    bool contains(const T& elem) const {\n      return std::binary_search(begin(), end(), elem, Less());\n    }\n\n    //! test whether the set contains the given set of element\n    template\n    bool contains(const small_set& other) const {\n      return std::includes(begin(), end(), \n                           other.begin(), other.end(), Less());\n    }\n\n\n    /**\n     * Test if this set is contained in other.  If so this returns\n     * true. \n     */\n    template\n    bool operator<=(const small_set& other) const {\n      return other.contains(*this);\n    }\n\n\n    /**\n     * Test if this set is contained in other.  If so this returns\n     * true. \n     */\n    template\n    bool operator<(const small_set& other) const {\n      return other.contains(*this) && size() < other.size();\n    }\n\n    template\n    bool operator==(const small_set& other) const {\n      if(size() != other.size()) return false;\n      return std::equal(begin(), end(), other.begin(), Equals());\n    }\n\n\n\n    //! insert an element into this set\n    inline void insert(const T& elem) {\n      *this += elem;\n    }\n\n\n    //! insert a range of elements into this set\n    inline void insert(const T* begin, const T* end) {\n      // Ensure that other size is not negative\n      ASSERT_LE(begin, end);\n      // Ensure that the other set has an appropriate size\n      const size_t other_size = end - begin;  \n      ASSERT_LE(other_size, MAX_DIM);\n      // Construct a temporary small set representing the range\n      small_set other;\n      for(size_t i = 0; i < other_size; ++i) {\n        other[i] = begin[i];\n        // Ensure that the other set is sorted\n        if(i+1 < other_size) ASSERT_LT(begin[i], begin[i+1]);\n      }\n      // Insert it into this small set using the + operation\n      *this += other;\n    }\n\n\n    //! remove an element from the set\n    void erase(const T& elem) { *this -= elem; }\n    \n\n    //! get the element at a particular location\n    virtual const T& operator[](size_t index) const {\n      ASSERT_LT(index, nelems);\n      return values[index];\n    }\n\n\n    \n\n    // //! Take the union of two sets\n    // inline small_set operator+(const small_set& other) const {\n    //   small_set result;\n    //   size_t i = 0, j = 0;\n    //   while(i < size() && j < other.size()) {\n    //     assert(result.nelems < MAX_DIM);\n    //     if(values[i] < other.values[j])  // This comes first\n    //       result.values[result.nelems++] = values[i++];\n    //     else if (values[i] > other.values[j])  // other comes first\n    //       result.values[result.nelems++] = other.values[j++];\n    //     else {  // both are same\n    //       result.values[result.nelems++] = values[i++]; j++;\n    //     }\n    //   }\n    //   // finish writing this\n    //   while(i < size()) {\n    //     assert(result.nelems < MAX_DIM);\n    //     result.values[result.nelems++] = values[i++];\n    //   }\n    //   // finish writing other\n    //   while(j < other.size()) {\n    //     assert(result.nelems < MAX_DIM);\n    //     result.values[result.nelems++] = other.values[j++];\n    //   }\n    //   return result;\n    // }\n\n\n    //! Take the union of two sets\n    inline small_set operator+(const T& elem) const {\n      small_set result(*this);\n      return result += elem;\n    }\n\n\n    //! Take the union of two sets\n    template\n    inline small_set< max_type::value, T, Less> \n    operator+(const small_set& other) const {\n      typedef small_set< max_type::value, T, Less>\n        result_type;\n      result_type result;\n      const T* new_end = \n        std::set_union(begin(), end(),\n                       other.begin(), other.end(),\n                       safe_iterator(result.begin(), \n                                     result.absolute_end()),\n                       Less()).begin;\n      result.nelems = new_end - result.begin();\n      ASSERT_LE(result.nelems, result_type::max_dim_type);\n      return result;\n    }\n\n\n    //! Add the other set to this set\n    template\n    inline small_set& operator+=(const small_set& other) {\n      *this = *this + other;\n      return *this;\n    }\n\n\n    //! Add an element to this set. This is \"optimized\" since it is\n    //! used frequently\n    inline small_set& operator+=(const T& elem) {\n      // // Find where elem should be inserted\n      // size_t index = 0;\n      // for(; index < nelems && values[index] < elem; ++index);      \n      T* ptr(std::lower_bound(begin(), end(), elem, Less()));     \n      // if the element already exists return\n      if(ptr != end() && !(elem < *ptr) ) return *this;\n      // otherwise the element does not exist so add it at the current\n      // location and increment the number of elements\n      T tmp(elem); nelems++; // advances end\n      ASSERT_LE(nelems, MAX_DIM);\n      // Insert the element at index swapping out the rest of the\n      // array\n      for(; ptr < end(); ++ptr) std::swap(*ptr, tmp);      \n      // Finished return\n      return *this;\n    }\n\n\n\n    //! Remove the other set from this set\n    template\n    small_set& operator-=(const small_set& other) {\n      // if(other.size() == 0) return *this;    \n      // // Backup the old nelems and reset nelems\n      // size_t old_nelems = size(); nelems = 0;\n      // for(size_t i = 0, j = 0; i < old_nelems; ++i ) {\n      //   // advance the other index\n      //   for( ; j < other.size() && values[i] > other.values[j]; ++j);\n      //   // otherwise check equality or move forward\n      //   if(j >= other.size() || (values[i] != other.values[j])) \n      //     values[nelems++] = values[i];\n      // }\n      // ASSERT_LE(nelems, MAX_DIM);\n      *this = *this - other;\n      return *this;\n    }\n\n    //! subtract the right set form the left set\n    template\n    small_set operator-(const small_set& other) const {\n      // small_set result = *this;\n      // result -= other;\n      small_set result;\n      T* const new_end =\n        std::set_difference(begin(), end(), \n                            other.begin(), other.end(),\n                            safe_iterator(result.begin(),\n                                          result.absolute_end()),\n                            Less()).begin;\n      result.nelems = new_end - result.begin();\n      ASSERT_LE(result.nelems, MAX_DIM);\n      return result;\n    }\n\n    //! Take the intersection of two sets\n    template\n    small_set operator*(const small_set& other) const {\n      small_set result;\n      const T* new_end = \n        std::set_intersection(begin(), end(), \n                              other.begin(), other.end(),\n                              safe_iterator(result.begin(), \n                                            result.absolute_end()),\n                              Less()).begin;\n      result.nelems = new_end - result.end();\n      ASSERT_LE(result.nelems, MAX_DIM);\n      return result;\n    }\n\n    //! Take the intersection of two sets\n    template\n    small_set operator*=(const small_set& other)  {\n      *this = *this * other;\n      return *this;\n    }\n\n    //! Load the set form file\n    void load(iarchive& arc) {\n      arc >> nelems;\n      assert(nelems <= MAX_DIM);\n      for(size_t i = 0; i < nelems; ++i) {\n        arc >> values[i];\n        if( i > 0 ) assert(values[i] > values[i-1]);\n      }\n    }\n    \n    //! Save the set to file\n    void save(oarchive& arc) const {\n      arc << nelems;\n      for(size_t i = 0; i < nelems; ++i) arc << values[i];\n    }\n  private:\n    size_t nelems;\n    T values[MAX_DIM];\n\n                     \n    //! get the end iterator to the complete range\n    inline T* absolute_end() { return values + MAX_DIM; }\n\n\n    struct safe_iterator : \n      public std::iterator  {\n      T* begin;\n      const T* end;\n      safe_iterator(const safe_iterator& other) :\n        begin(other.begin), end(other.end) { }\n      safe_iterator(T* begin, const T* end) : \n        begin(begin), end(end) {\n        ASSERT_NE(begin, NULL);\n        ASSERT_NE(end, NULL);\n        ASSERT_LE(begin, end);\n      }\n      inline safe_iterator& operator++() { ++begin; return *this; }\n      inline safe_iterator& operator++(int) {\n        safe_iterator tmp(*this); operator++(); return tmp;\n      }\n      inline bool operator==(const safe_iterator& other) {\n        ASSERT_EQ(end, other.end);\n        return begin == other.begin;\n      }\n      inline bool operator!=(const safe_iterator& other) {\n        return !operator==(other);\n      }\n      T& operator*() { ASSERT_LT(begin, end); return *begin; }\n    };   \n  }; // end of small_set\n\n  template\n  std::ostream&\n  operator<<(std::ostream& out, const graphlab::small_set& set) {\n    out << \"{\";\n    for(size_t i = 0; i < set.size(); ++i) {\n      out << set[i];\n      if(i + 1 < set.size()) out << \", \";\n    }\n    out << \"}\";\n    return out;\n  }\n}; // end of graphlab namespace\n\n\n\n\n#include \n#endif\n\n"
  },
  {
    "path": "src/graphlab/util/stl_util.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n// Probabilistic Reasoning Library (PRL)\n// Copyright 2009 (see AUTHORS.txt for a list of contributors)\n//\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n//\n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU\n// Lesser General Public License for more details.\n//\n// You should have received a copy of the GNU Lesser General Public\n// License along with this library; if not, write to the Free Software\n// Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA\n\n#ifndef GRAPHLAB_STL_UTIL_HPP\n#define GRAPHLAB_STL_UTIL_HPP\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n// #include \n// #include \n// #include \n\n#include \n\nnamespace graphlab {\n\n  // Functions on sets\n  //============================================================================\n\n  /**\n   * computes the union of two sets.\n   */\n  template \n  std::set set_union(const std::set& a, const std::set& b) {\n    std::set output;\n    std::set_union(a.begin(), a.end(), \n                   b.begin(), b.end(),\n                   std::inserter(output, output.begin()));\n    return output;\n  }\n  \n  template \n  std::set set_union(const std::set& a, const T& b) {\n    std::set output = a;\n    output.insert(b);\n    return output;\n  }\n\n  template \n  std::set set_intersect(const std::set& a, const std::set& b) {\n    std::set output;\n    std::set_intersection(a.begin(), a.end(), \n                          b.begin(), b.end(),\n                          std::inserter(output, output.begin()));\n    return output;\n  }\n\n  template \n  std::set set_difference(const std::set& a, const std::set& b) {\n    std::set output;\n    std::set_difference(a.begin(), a.end(), \n                        b.begin(), b.end(),\n                        std::inserter(output, output.begin()));\n    return output;\n  }\n\n\n  template \n  std::set set_difference(const std::set& a, const T& b) {\n    std::set output = a;\n    output.erase(b);\n    return output;\n  }\n\n  //! @return 2 sets: \n  template \n  std::pair,std::set > \n  set_partition(const std::set& s, const std::set& partition) {\n    std::set a, b;\n    a = set_intersect(s, partition);\n    b = set_difference(s, partition);\n    return std::make_pair(a, b);\n  }\n\n  template \n  bool set_disjoint(const std::set& a, const std::set& b) {\n    return (intersection_size(a,b) == 0);\n  }\n  \n  template \n  bool set_equal(const std::set& a, const std::set& b) {\n    if (a.size() != b.size()) return false;\n    return a == b; // defined in \n  }\n  \n  template \n  bool includes(const std::set& a, const std::set& b) {\n    return std::includes(a.begin(), a.end(), b.begin(), b.end());\n  }\n\n  /** \n   * Returns true if $a \\subseteq b$\n   */\n  template \n  bool is_subset(const std::set& a, const std::set& b) {\n    return includes(b, a);\n  }\n\n  template \n  bool is_superset(const std::set& a,const std::set& b) {\n    return includes(a, b);\n  }\n  \n  //! Writes a human representation of the set to the supplied stream.\n  template \n  std::ostream& operator<<(std::ostream& out, const std::set& s) {\n    return print_range(out, s, '{', ' ', '}');\n  }\n\n  // Functions on maps\n  //============================================================================\n\n  /**\n   * constant lookup in a map. assertion failure of key not found in map\n   */\n  template \n  const T& safe_get(const std::map& map,\n                    const Key& key) {\n    typedef typename std::map::const_iterator iterator;\n    iterator iter = map.find(key);\n    ASSERT_TRUE(iter != map.end());\n    return iter->second;\n  } // end of safe_get\n\n  /**\n   * constant lookup in a map. If key is not found in map, \n   * 'default_value' is returned. Note that this can't return a reference\n   * and must return a copy\n   */\n  template \n  const T safe_get(const std::map& map,\n                    const Key& key, const T default_value) {\n    typedef typename std::map::const_iterator iterator;\n    iterator iter = map.find(key);\n    if (iter == map.end())   return default_value;\n    else return iter->second;\n  } // end of safe_get\n\n  /**\n   * Transform each key in the map using the key_map\n   * transformation. The resulting map will have the form\n   * output[key_map[i]] = map[i]\n   */\n  template \n  std::map\n  rekey(const std::map& map,\n        const std::map& key_map) {\n    std::map output;\n    typedef std::pair pair_type;\n    foreach(const pair_type& pair, map) {\n      output[safe_get(key_map, pair.first)] = pair.second;\n    }\n    return output;\n  }\n\n  /**\n   * Transform each key in the map using the key_map\n   * transformation. The resulting map will have the form\n   output[i] = remap[map[i]]\n  */\n  template \n  std::map\n  remap(const std::map& map,\n        const std::map& val_map) {\n    std::map output;\n    typedef std::pair pair_type;\n    foreach(const pair_type& pair, map) {\n      output[pair.first] = safe_get(val_map, pair.second);\n    }\n    return output;\n  }\n\n  /**\n   * Inplace version of remap\n   */\n  template \n  void remap(std::map& map,\n             const std::map& val_map) {\n    typedef std::pair pair_type;\n    foreach(pair_type& pair, map) {\n      pair.second = safe_get(val_map, pair.second);\n    }\n  }\n\n  /**\n   * Computes the union of two maps\n   */\n  template \n  std::map \n  map_union(const std::map& a,\n            const std::map& b) {\n    // Initialize the output map\n    std::map output;\n    std::set_union(a.begin(), a.end(),\n                   b.begin(), b.end(),\n                   std::inserter(output, output.begin()),\n                   output.value_comp());\n    return output;\n  }\n  \n  /**\n   * Computes the intersection of two maps\n   */\n  template \n  std::map \n  map_intersect(const std::map& a,\n                const std::map& b) {\n    // Initialize the output map\n    std::map output;\n    // compute the intersection\n    std::set_intersection(a.begin(), a.end(),\n                          b.begin(), b.end(),\n                          std::inserter(output, output.begin()),\n                          output.value_comp());\n    return output;\n  }\n  \n  /**\n   * Returns the entries of a map whose keys show up in the set keys\n   */\n  template \n  std::map \n  map_intersect(const std::map& m,\n                const std::set& keys) {\n    std::map output;\n    foreach(const Key& key, keys) {\n      typename std::map::const_iterator it = m.find(key);\n      if (it != m.end())\n        output[key] = it->second;\n    }\n    return output;\n  }\n\n  /**\n   * Computes the difference between two maps\n   */\n  template \n  std::map \n  map_difference(const std::map& a,\n                 const std::map& b) {\n    // Initialize the output map\n    std::map output;\n    // compute the intersection\n    std::set_difference(a.begin(), a.end(),\n                        b.begin(), b.end(),\n                        std::inserter(output, output.begin()),\n                        output.value_comp());\n    return output;\n  }\n\n\n  /**\n   * Returns the set of keys in a map\n   */\n  template \n  std::set keys(const std::map& map) {\n    std::set output;\n    typedef std::pair pair_type;\n    foreach(const pair_type& pair, map) {\n      output.insert(pair.first);\n    }\n    return output;\n  }\n\n  /**\n   * Get teh set of keys in a map as a vector\n   */\n  template \n  std::vector keys_as_vector(const std::map& map) {\n    std::vector output(map.size());   \n    typedef std::pair pair_type;\n    size_t i = 0;\n    foreach(const pair_type& pair, map) {\n      output[i++] = pair.first;\n    }\n    return output;\n  }\n\n\n  /**\n   * Gets the values from a map\n   */\n  template \n  std::set values(const std::map& map) {\n    std::set output;\n    typedef std::pair pair_type;\n    foreach(const pair_type& pair, map) {\n      output.insert(pair.second);\n    }\n    return output;\n  }\n  \n  template \n  std::vector values(const std::map& m, \n                        const std::set& keys) {\n    std::vector output;\n\n    foreach(const Key &i, keys) {\n      output.push_back(safe_get(m, i));\n    }\n    return output;\n  }\n  \n  template \n  std::vector values(const std::map& m, \n                        const std::vector& keys) {\n    std::vector output;\n    foreach(const Key &i, keys) {\n      output.push_back(safe_get(m, i));\n    }\n    return output;\n  }\n  \n  //! Creates an identity map (a map from elements to themselves)\n  template \n  std::map make_identity_map(const std::set& keys) {\n    std::map m;\n    foreach(const Key& key, keys) \n      m[key] = key;\n    return m;\n  }\n\n  //! Writes a map to the supplied stream.\n  template \n  std::ostream& operator<<(std::ostream& out, const std::map& m) {\n    out << \"{\";\n    for (typename std::map::const_iterator it = m.begin(); \n         it != m.end();) {\n      out << it->first << \"-->\" << it->second;\n      if (++it != m.end()) out << \" \";\n    }\n    out << \"}\";\n    return out;\n  }\n\n  /** Removes white space (space and tabs) from the beginning and end of str,\n      returning the resultant string\n  */\n  inline std::string trim(const std::string& str) {\n    std::string::size_type pos1 = str.find_first_not_of(\" \\t\");\n    std::string::size_type pos2 = str.find_last_not_of(\" \\t\");\n    return str.substr(pos1 == std::string::npos ? 0 : pos1,\n                      pos2 == std::string::npos ? str.size()-1 : pos2-pos1+1);\n  }\n\n  /**\n  * Convenience function for using std streams to convert anything to a string\n  */\n  template\n  std::string tostr(const T& t) {\n    std::stringstream strm;\n    strm << t;\n    return strm.str();\n  }\n\n  /**\n  * Convenience function for using std streams to convert a string to anything\n  */\n  template\n  T fromstr(const std::string& str) {\n    std::stringstream strm(str);\n    T elem; \n    strm >> elem;\n    ASSERT_FALSE(strm.fail());\n    return elem;\n  }\n\n  /**\n  Returns a string representation of the number,\n  padded to 'npad' characters using the pad_value character\n  */\n  inline std::string pad_number(const size_t number,\n                                const size_t npad,\n                                const char pad_value = '0') {\n    std::stringstream strm;\n    strm << std::setw((int)npad) << std::setfill(pad_value)\n         << number;\n    return strm.str();\n  }\n\n\n  // inline std::string change_suffix(const std::string& fname,\n  //                                  const std::string& new_suffix) {             \n  //   size_t pos = fname.rfind('.');\n  //   assert(pos != std::string::npos); \n  //   const std::string new_base(fname.substr(0, pos));\n  //   return new_base + new_suffix;\n  // } // end of change_suffix\n\n\n  /**\n  Using splitchars as delimiters, splits the string into a vector of strings.\n  if auto_trim is true, trim() is called on all the extracted strings\n  before returning.\n  */\n  inline std::vector strsplit(const std::string& str, \n                                           const std::string& splitchars,\n                                           const bool auto_trim = false) {\n    std::vector tokens;\n    for(size_t beg = 0, end = 0; end != std::string::npos; beg = end+1) {\n      end = str.find_first_of(splitchars, beg);\n      if(auto_trim) {\n        if(end - beg > 0) {\n          std::string tmp = trim(str.substr(beg, end - beg));\n          if(!tmp.empty()) tokens.push_back(tmp);\n        }\n      } else tokens.push_back(str.substr(beg, end - beg));\n    }\n    return tokens;\n    // size_t pos = 0;\n    // while(1) {\n    //   size_t nextpos = s.find_first_of(splitchars, pos);\n    //   if (nextpos != std::string::npos) {\n    //     ret.push_back(s.substr(pos, nextpos - pos));\n    //     pos = nextpos + 1;\n    //   } else {\n    //     ret.push_back(s.substr(pos));\n    //     break;\n    //   }\n    // }\n    // return ret;\n  }\n}; // end of namespace graphlab\n\n#include \n\n#endif\n\n"
  },
  {
    "path": "src/graphlab/util/synchronized_unordered_map.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SYNCHRONIZED_UNORDERED_MAP\n#define GRAPHLAB_SYNCHRONIZED_UNORDERED_MAP\n#include \n#include \n#include \n\nnamespace graphlab {\n\n/// \\ingroup util_internal\ntemplate \nclass synchronized_unordered_map {\n public:\n  typedef boost::unordered_map container;\n  typedef typename container::iterator iterator;\n  typedef typename container::const_iterator const_iterator;\n\n  typedef std::pair datapointer;\n  typedef std::pair const_datapointer;\n  typedef Data value_type;\n  typedef size_t key_type;\n  \n private:\n   std::vector data;\n   std::vector lock; \n   size_t nblocks;\n public:\n   synchronized_unordered_map(size_t numblocks):data(numblocks), \n                                                lock(numblocks),\n                                                nblocks(numblocks) { \n    for (size_t i = 0;i < numblocks; ++i) {\n      data[i].max_load_factor(1.0);\n    }\n  }\n\n   std::pair find(size_t key) {\n     size_t b = key % nblocks;\n     lock[b].readlock();\n     iterator iter = data[b].find(key);\n     std::pair ret = std::make_pair(iter != data[b].end(), &(iter->second));\n     lock[b].rdunlock();\n     return ret;\n   }\n   \n   /**\n   return std::pair\n   if not found, iterator is invalid\n   */\n   std::pair find(size_t key) const {\n     size_t b = key % nblocks;\n     lock[b].readlock();\n     const_iterator iter = data[b].find(key);\n     std::pair ret = std::make_pair(iter != data[b].end(), &(iter->second));\n     lock[b].rdunlock();\n     return ret;\n   }\n   \n   // care must be taken that  you do not access an erased iterator\n   void erase(size_t key) {\n     size_t b = key % nblocks;\n     lock[b].writelock();\n     data[b].erase(key);\n     lock[b].wrunlock();\n   }\n\n   template\n   void erase_if(size_t key, Predicate pred) {\n     size_t b = key % nblocks;\n     lock[b].writelock();\n     iterator iter = data[b].find(key); \n      \n     if (iter != data[b].end() && pred(iter->second))  data[b].erase(key);\n     lock[b].wrunlock();\n   }\n\n   value_type& insert(size_t key, const value_type &val) {\n     size_t b = key % nblocks;\n     lock[b].writelock();\n     data[b][key] = val;\n     value_type& ret = data[b][key];\n     lock[b].wrunlock();\n     return ret;\n   }\n\n    /**\n    returns std::pair\n    on success, iterator will point to the entry\n    on failure, iterator will point to an existing entry\n    */\n   std::pair insert_with_failure_detect(size_t key, const value_type &val) {\n     std::pair ret ;\n     \n     size_t b = key % nblocks;\n     lock[b].writelock();\n     //search for it\n     iterator iter = data[b].find(key);\n     // if it not in the table, write and return\n     if (iter == data[b].end()) {\n      data[b][key] = val;\n      ret = std::make_pair(true,  &(data[b].find(key)->second));\n     }\n     else {\n      ret = std::make_pair(false,  &(iter->second));\n     }\n     lock[b].wrunlock();\n     return ret;\n   }\n\n   void clear() {\n     for (size_t i = 0;i < data.size(); ++i) {\n       data[i].clear();\n     }\n   }\n};\n}\n#endif\n\n"
  },
  {
    "path": "src/graphlab/util/synchronized_unordered_map2.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef SYNCHRONIZED_UNORDERED_MAP2\n#define SYNCHRONIZED_UNORDERED_MAP2\n#include \n#include \n#include \n\n\nnamespace graphlab {\n/*\n \\ingroup util_internal\nAn alternate form of the synchronized unordered map, built around the\nuse of critical sections\n*/\ntemplate \nclass synchronized_unordered_map2 {\n public:\n  typedef boost::unordered_map container;\n  typedef typename container::iterator iterator;\n  typedef typename container::const_iterator const_iterator;\n\n  typedef std::pair datapointer;\n  typedef std::pair const_datapointer;\n  typedef Data value_type;\n  typedef size_t key_type;\n\n private:\n   std::vector data;\n   std::vector lock;\n   size_t nblocks;\n public:\n   synchronized_unordered_map2(size_t numblocks):data(numblocks),\n                                                lock(numblocks),\n                                                nblocks(numblocks) {\n    for (size_t i = 0;i < numblocks; ++i) {\n      data[i].max_load_factor(1.0);\n    }\n  }\n\n   std::pair find(size_t key) {\n    size_t b = key % nblocks;\n     iterator iter = data[b].find(key);\n     std::pair ret = std::make_pair(iter != data[b].end(), &(iter->second));\n     return ret;\n   }\n\n   /**\n   return std::pair\n   if not found, iterator is invalid\n   */\n   std::pair find(size_t key) const {\n     size_t b = key % nblocks;\n     const_iterator iter = data[b].find(key);\n     std::pair ret = std::make_pair(iter != data[b].end(), &(iter->second));\n     return ret;\n   }\n\n   // care must be taken that  you do not access an erased iterator\n   void erase(size_t key) {\n     size_t b = key % nblocks;\n     data[b].erase(key);\n   }\n\n   template\n   void erase_if(size_t key, Predicate pred) {\n     size_t b = key % nblocks;\n     iterator iter = data[b].find(key);\n\n     if (iter != data[b].end() && pred(iter->second))  data[b].erase(key);\n   }\n\n   value_type& insert(size_t key, const value_type &val) {\n     size_t b = key % nblocks;\n     data[b][key] = val;\n     value_type& ret = data[b][key];\n     return ret;\n   }\n\n   void read_critical_section(size_t key) {\n    size_t b = key % nblocks;\n    lock[b].readlock();\n   }\n   void write_critical_section(size_t key) {\n    size_t b = key % nblocks;\n    lock[b].writelock();\n   }\n   void release_critical_section(size_t key) {\n    size_t b = key % nblocks;\n    lock[b].unlock();\n   }\n    /**\n    returns std::pair\n    on success, iterator will point to the entry\n    on failure, iterator will point to an existing entry\n    */\n   std::pair insert_with_failure_detect(size_t key, const value_type &val) {\n     std::pair ret ;\n\n     size_t b = key % nblocks;\n     //search for it\n     iterator iter = data[b].find(key);\n     // if it not in the table, write and return\n     if (iter == data[b].end()) {\n      data[b][key] = val;\n      ret = std::make_pair(true,  &(data[b].find(key)->second));\n     }\n     else {\n      ret = std::make_pair(false,  &(iter->second));\n     }\n     return ret;\n   }\n\n   void clear() {\n     for (size_t i = 0;i < data.size(); ++i) {\n       data[i].clear();\n     }\n   }\n};\n}\n#endif\n\n"
  },
  {
    "path": "src/graphlab/util/system_usage.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_SYSTEM_USAGE_HPP\n#define GRAPHLAB_SYSTEM_USAGE_HPP\n\n\n\n\n#include \n\n\nnamespace graphlab {\n\n\n\n\n\n};\n\n\n#endif\n\n"
  },
  {
    "path": "src/graphlab/util/timer.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n\n#include \n#include \n\nstd::ostream&  operator<<(std::ostream& out, const graphlab::timer& t) {\n  return out << t.current_time();\n} \n\nnamespace graphlab {\n  void alarm_wakeup(int i);\n  \n  \n  class hundredms_timer {\n  public:\n    hundredms_timer() {    \n      stop = false;\n      tout_val.it_interval.tv_sec = 0;\n      tout_val.it_interval.tv_usec = 0;\n      tout_val.it_value.tv_sec = 0;\n      tout_val.it_value.tv_usec = 50000;\n      signal(SIGALRM,alarm_wakeup); /* set the Alarm signal capture */    \n      setitimer(ITIMER_REAL, &tout_val,0);\n      ti.start();\n    }\n    size_t ctr; \n    timer ti;\n    struct itimerval tout_val;\n    bool stop;\n    \n    ~hundredms_timer() {  \n      stop = true;\n      signal(SIGALRM, SIG_IGN);\n    }\n  };\n  \n  hundredms_timer hmstimer;\n  \n  \n  void alarm_wakeup(int i) {\n    if (hmstimer.stop) return;\n    signal(SIGALRM,alarm_wakeup);\n    // compute the desired time till the next 100ms tick by using a real timer call\n    double realtime = hmstimer.ti.current_time() ;\n    // round down\n    hmstimer.ctr = (size_t)(realtime * 10);\n    setitimer(ITIMER_REAL, &(hmstimer.tout_val), 0);   \n  }\n\n  /**\n   * Precision of deciseconds \n   */\n  float timer::approx_time_seconds() {\n    return float(hmstimer.ctr) / 10;\n  }\n\n  /**\n   * Precision of deciseconds \n   */\n  size_t timer::approx_time_millis() {\n    return hmstimer.ctr * 100;\n  }\n\n\n  void timer::sleep(size_t sleeplen) {\n    struct timespec timeout;\n    timeout.tv_sec = sleeplen;\n    timeout.tv_nsec = 0;\n    while (nanosleep(&timeout, &timeout) == -1);\n  }\n\n\n  /**\n  Sleeps for sleeplen milliseconds.\n  */\n  void timer::sleep_ms(size_t sleeplen) {\n    struct timespec timeout;\n    timeout.tv_sec = sleeplen / 1000;\n    timeout.tv_nsec = (sleeplen % 1000) * 1000000;\n    while (nanosleep(&timeout, &timeout) == -1);\n  }\n  \n\n  \n  \nstatic unsigned long long rtdsc_ticks_per_sec = 0; \nstatic mutex rtdsc_ticks_per_sec_mutex;\n\nunsigned long long estimate_ticks_per_second() {\n  if (rtdsc_ticks_per_sec == 0) {\n    rtdsc_ticks_per_sec_mutex.lock();\n      if (rtdsc_ticks_per_sec == 0) {\n      unsigned long long tstart = rdtsc();\n      graphlab::timer::sleep(1);\n      unsigned long long tend = rdtsc();\n      rtdsc_ticks_per_sec = tend - tstart;\n      }\n    rtdsc_ticks_per_sec_mutex.unlock();\n  }\n  return rtdsc_ticks_per_sec;\n}\n\n}\n\n"
  },
  {
    "path": "src/graphlab/util/timer.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_TIMER_HPP\n#define GRAPHLAB_TIMER_HPP\n\n#include \n#include \n\n#include \n\nnamespace graphlab {\n  /**\n   * \\ingroup util \n   *\n   * \\brief A simple class that can be used for benchmarking/timing up\n   * to microsecond resolution.\n   *\n   * Standard Usage\n   * =================\n   *\n   * The timer is used by calling \\ref graphlab::timer::start and then\n   * by getting the current time since start by calling \n   * \\ref graphlab::timer::current_time.\n   * \n   * For example:\n   * \n   * \\code\n   * #include \n   *\n   *\n   * graphlab::timer timer;\n   * timer.start();\n   * // do something\n   * std::cout << \"Elapsed time: \" << timer.current_time() << std::endl; \n   * \\endcode\n   *\n   * Fast approximate time\n   * ====================\n   *\n   * Calling current item in a tight loop can be costly and so we\n   * provide a faster less accurate timing primitive which reads a\n   * local time variable that is updated roughly every 100 millisecond.\n   * These are the \\ref graphlab::timer::approx_time_seconds and\n   * \\ref graphlab::timer::approx_time_millis.\n   */\n  class timer {\n  private:\n    /**\n     * \\brief The internal start time for this timer object\n     */\n    timeval start_time_;   \n  public:\n    /**\n     * \\brief The timer starts on construction but can be restarted by\n     * calling \\ref graphlab::timer::start.\n     */\n    inline timer() { start(); }\n    \n    /**\n     * \\brief Reset the timer.\n     */\n    inline void start() { gettimeofday(&start_time_, NULL); }\n    \n    /** \n     * \\brief Returns the elapsed time in seconds since \n     * \\ref graphlab::timer::start was last called.\n     *\n     * @return time in seconds since \\ref graphlab::timer::start was called.\n     */\n    inline double current_time() const {\n      timeval current_time;\n      gettimeofday(¤t_time, NULL);\n      double answer = \n       // (current_time.tv_sec + ((double)current_time.tv_usec)/1.0E6) -\n       // (start_time_.tv_sec + ((double)start_time_.tv_usec)/1.0E6);\n        (double)(current_time.tv_sec - start_time_.tv_sec) + \n        ((double)(current_time.tv_usec - start_time_.tv_usec))/1.0E6;\n       return answer;\n    } // end of current_time\n\n    /** \n     * \\brief Returns the elapsed time in milliseconds since \n     * \\ref graphlab::timer::start was last called.\n     *\n     * @return time in milliseconds since \\ref graphlab::timer::start was called.\n     */\n    inline double current_time_millis() const { return current_time() * 1000; }\n\n    /**\n     * \\brief Get the number of seconds (as a floating point value)\n     * since the Unix Epoch.\n     */\n    static double sec_of_day() {\n      timeval current_time;\n      gettimeofday(¤t_time, NULL);\n      double answer = \n        (double)current_time.tv_sec + ((double)current_time.tv_usec)/1.0E6;\n      return answer;\n    } // end of sec_of_day\n\n    /**\n     * \\brief Returns only the micro-second component of the \n     * time since the Unix Epoch.\n     */\n    static size_t usec_of_day() {\n      timeval current_time;\n      gettimeofday(¤t_time, NULL);\n      size_t answer = \n        (size_t)current_time.tv_sec * 1000000 + (size_t)current_time.tv_usec;\n      return answer;\n    } // end of usec_of_day\n\n    /**\n     * \\brief Returns the time since program start.\n     * \n     * This value is only updated once every 100ms and is therefore\n     * approximate (but fast).\n     */\n    static float approx_time_seconds();\n    \n    /**\n     * \\brief Returns the time since program start.\n     * \n     * This value is only updated once every 100ms and is therefore\n     * approximate (but fast).\n     */\n    static size_t approx_time_millis();\n\n    /**\n     * Sleeps for sleeplen seconds\n     */\n    static void sleep(size_t sleeplen);\n\n    /**\n     * Sleeps for sleeplen milliseconds.\n     */\n    static void sleep_ms(size_t sleeplen);\n  }; // end of Timer\n  \n\n  unsigned long long estimate_ticks_per_second();\n\n  #if defined(__i386__)\n  static inline unsigned long long rdtsc(void)\n  {\n    unsigned long long int x;\n      __asm__ volatile (\".byte 0x0f, 0x31\" : \"=A\" (x));\n      return x;\n  }\n  #elif defined(__x86_64__)\n  static inline unsigned long long rdtsc(void)\n  {\n    unsigned hi, lo;\n    __asm__ __volatile__ (\"rdtsc\" : \"=a\"(lo), \"=d\"(hi));\n    return ( (unsigned long long)lo) | ( ((unsigned long long)hi)<<32 );\n  }\n  #else\n  static inline unsigned long long rdtsc(void) {\n    return 0;\n  }\n  #endif\n\n\n} // end of graphlab namespace\n\n/** \n * Convenience function. Allows you to call \"cout << ti\" where ti is\n * a timer object and it will print the number of seconds elapsed\n * since ti.start() was called.\n */\nstd::ostream&  operator<<(std::ostream& out, const graphlab::timer& t);\n\n\n#endif\n\n"
  },
  {
    "path": "src/graphlab/util/tracepoint.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n\n\nnamespace graphlab {\n\nvoid trace_count::print(std::ostream& out, unsigned long long tpersec) const {\n  if (tpersec == 0) {\n    out << name << \": \" << description << \"\\n\";\n    out << \"Events:\\t\" << count.value << \"\\n\";\n    out << \"Total:\\t\" << total.value << \"ticks \\n\";\n    if (count.value > 0) {\n      out << \"Mean:\\t\" << (double)total.value / count.value << \"ticks \\n\";\n      out << \"Min:\\t\" << minimum << \"ticks \\n\";\n      out << \"Max:\\t\" << maximum << \"ticks \\n\";\n    }\n  }\n  else {\n    double tperms = (double)tpersec / 1000;\n    out << name << \": \" << description << \"\\n\";\n    out << \"Events:\\t\" << count.value << \"\\n\";\n    out << \"Total:\\t\" << (double)total.value / tperms << \" ms \\n\";\n    if (count.value > 0) {\n      out << \"Mean:\\t\" << (double)total.value / count.value / tperms << \" ms \\n\";\n      out << \"Min:\\t\" << (double)minimum / tperms << \" ms \\n\";\n      out << \"Max:\\t\" << (double)maximum / tperms << \" ms \\n\";\n    }\n  }\n}\n\n\nstatic mutex printlock;\n\ntrace_count::~trace_count() {\n#ifdef USE_TRACEPOINT\n  printlock.lock();\n  print(std::cout, estimate_ticks_per_second());\n  std::cout.flush();\n  printlock.unlock();\n#endif\n}\n\n} // namespace graphlab\n\n"
  },
  {
    "path": "src/graphlab/util/tracepoint.hpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_UTIL_TRACEPOINT_HPP\n#define GRAPHLAB_UTIL_TRACEPOINT_HPP\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\nnamespace graphlab{\n\nstruct trace_count{\n  std::string name;\n  std::string description;\n  bool print_on_destruct;\n  graphlab::atomic count;\n  graphlab::atomic total;\n  unsigned long long minimum;\n  unsigned long long maximum;\n  inline trace_count(std::string name = \"\",\n                    std::string description = \"\",\n                    bool print_on_destruct = true):\n                      name(name),\n                      description(description),\n                      print_on_destruct(print_on_destruct),\n                      count(0),\n                      total(0),\n                      minimum(std::numeric_limits::max()),\n                      maximum(0) { }\n\n  /**\n   * Initializes the tracer with a name, a description\n   * and whether to print on destruction\n   */\n  inline void initialize(std::string n,\n                  std::string desc,\n                  bool print_out = true) {\n    name = n;\n    description = desc;\n    print_on_destruct = print_out;\n  }\n\n  /**\n   * Adds an event time to the trace\n   */\n  inline void incorporate(unsigned long long val)  __attribute__((always_inline)) {\n    count.inc();\n    total.inc(val);\n    while(1) {\n      unsigned long long m = minimum;\n      if (__likely__(val > m || graphlab::atomic_compare_and_swap(minimum, m, val))) break;\n    }\n    while(1) {\n      unsigned long long m = maximum;\n      if (__likely__(val < m || graphlab::atomic_compare_and_swap(maximum, m, val))) break;\n    }\n  }\n  \n  /**\n   * Adds the counts in a second tracer to the current tracer.\n   */  \n  inline void incorporate(const trace_count &val)  __attribute__((always_inline)) {\n    count.inc(val.count.value);\n    total.inc(val.total.value);\n    while(1) {\n      unsigned long long m = minimum;\n      if (__likely__(val.minimum > m || graphlab::atomic_compare_and_swap(minimum, m, val.minimum))) break;\n    }\n    while(1) {\n      unsigned long long m = maximum;\n      if (__likely__(val.maximum < m || graphlab::atomic_compare_and_swap(maximum, m, val.maximum))) break;\n    }\n  }\n\n  /**\n   * Adds the counts in a second tracer to the current tracer.\n   */\n  inline trace_count& operator+=(trace_count &val) {\n    incorporate(val);\n    return *this;\n  }\n\n  /**\n   * Destructor. Will print to cout if initialize() is called\n   * with \"true\" as the 3rd argument\n   */\n  ~trace_count();\n\n  /**\n   * Prints the tracer counts\n   */\n  void print(std::ostream& out, unsigned long long tpersec = 0) const;\n};\n\n} // namespace\n\n/**\n * DECLARE_TRACER(name)\n * creates a tracing object with a given name. This creates a variable\n * called \"name\" which is of type trace_count. and is equivalent to:\n *\n * graphlab::trace_count name;\n * \n * The primary reason to use this macro instead of just writing\n * the code above directly, is that the macro is ignored and compiles\n * to nothing when tracepoints are disabled.\n *\n *\n * INITIALIZE_TRACER(name, desc)\n * The object with name \"name\" created by DECLARE_TRACER must be in scope.\n * This initializes the tracer \"name\" with a description, and\n * configures the tracer to print when the tracer \"name\" is destroyed.\n *\n *\n * INITIALIZE_TRACER_NO_PRINT(name, desc)\n * The object with name \"name\" created by DECLARE_TRACER must be in scope.\n * This initializes the tracer \"name\" with a description, and\n * configures the tracer to NOT print when the tracer \"name\" is destroyed.\n *\n * BEGIN_TRACEPOINT(name)\n * END_TRACEPOINT(name)\n * The object with name \"name\" created by DECLARE_TRACER must be in scope.\n * Times a block of code. Every END_TRACEPOINT must be matched with a\n * BEGIN_TRACEPOINT within the same scope. Tracepoints are parallel.\n * \n\n  \nExample Usage:\n  DECLARE_TRACER(classname_someevent)\n  INITIALIZE_TRACER(classname_someevent, \"hello world\");\n  Then later on...\n  BEGIN_TRACEPOINT(classname_someevent)\n  ...\n  END_TRACEPOINT(classname_someevent)\n *\n*/\n\n\n#ifdef USE_TRACEPOINT\n#define DECLARE_TRACER(name) graphlab::trace_count name;\n\n#define INITIALIZE_TRACER(name, description) name.initialize(#name, description);\n#define INITIALIZE_TRACER_NO_PRINT(name, description) name.initialize(#name, description, false);\n\n#define BEGIN_TRACEPOINT(name) unsigned long long __ ## name ## _trace_ = rdtsc();\n#define END_TRACEPOINT(name) name.incorporate(rdtsc() - __ ## name ## _trace_);\n#define END_AND_BEGIN_TRACEPOINT(endname, beginname) unsigned long long __ ## beginname ## _trace_ = rdtsc(); \\\n                                                     endname.incorporate(__ ## beginname ## _trace_ - __ ## endname ## _trace_);\n\n#define CREATE_ACCUMULATING_TRACEPOINT(name) graphlab::trace_count __ ## name ## _acc_trace_; \\\n                                             unsigned long long __ ## name ## _acc_trace_elem_;\n#define BEGIN_ACCUMULATING_TRACEPOINT(name) __ ## name ## _acc_trace_elem_ = rdtsc();\n#define END_ACCUMULATING_TRACEPOINT(name) __ ## name ## _acc_trace_.incorporate(rdtsc() - __ ## name ## _acc_trace_elem_);\n\n#define END_AND_BEGIN_ACCUMULATING_TRACEPOINT(endname, beginname) __ ## beginname ## _acc_trace_elem_ = rdtsc(); \\\n                                                                  __ ## endname ## _acc_trace_.incorporate(__ ## beginname ## _acc_trace_elem_ - __ ## endname ## _acc_trace_elem_)\n\n#define STORE_ACCUMULATING_TRACEPOINT(name) name.incorporate(__ ## name ## _acc_trace_);\n#else\n#define DECLARE_TRACER(name)\n#define INITIALIZE_TRACER(name, description)\n#define INITIALIZE_TRACER_NO_PRINT(name, description) \n\n#define BEGIN_TRACEPOINT(name) \n#define END_TRACEPOINT(name) \n\n#define CREATE_ACCUMULATING_TRACEPOINT(name) \n#define BEGIN_ACCUMULATING_TRACEPOINT(name) \n#define END_ACCUMULATING_TRACEPOINT(name)\n#define STORE_ACCUMULATING_TRACEPOINT(name)\n\n#define END_AND_BEGIN_ACCUMULATING_TRACEPOINT(endname, beginname)\n#define END_AND_BEGIN_TRACEPOINT(endname, beginname) \n#endif           \n\n#endif\n"
  },
  {
    "path": "src/graphlab/util/uint128.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_UINT128_HPP\n#define GRAPHLAB_UINT128_HPP\n#include \n#include \n#include \n#include \n\nnamespace graphlab {\n  \n/**\n * A 128 bit numeric type. This type is a union of a 16-byte character array (bytes),\n * and struct of two 64-bit integers (ints.high and ints.low).\n */\nunion gl_uint128_t {\n  struct {\n    uint64_t high;\n    uint64_t low;  \n  } ints;\n  char bytes[16];\n  \n  gl_uint128_t() { }\n  \n  /**\n   * Constructs a 128-bit type from a 64-bit value.\n   * It simply clears the \"high\" 64 bits of the 128-bit integer, and sets\n   * the low 64-bits to the input\n   */\n  explicit gl_uint128_t(uint64_t val) {\n    ints.high = 0;\n    ints.low = val;\n  }\n};\n\n/**\n * Sets all 128bits of the the gl_uint128_t to 'true'.\n * Or the 128-bit integer representation of \"-1\"\n */\ninline gl_uint128_t fill_128b() {\n  gl_uint128_t i;\n  i.ints.high = (uint64_t)(-1);\n  i.ints.low = (uint64_t)(-1);\n  return i;\n}\n\n/**\n * Prints the 128-bit integer as hexadecimal\n */\ninline std::ostream& operator<<(std::ostream& out, const gl_uint128_t &val) {\n  static char hexchar[17] = \"0123456789abcdef\";\n  \n  for (size_t i = 0;i < 16; ++i) {\n    out << hexchar[(val.bytes[i] >> 4) & 15];\n    out << hexchar[val.bytes[i] & 15];\n  }\n  return out;\n}\n\n}\n\nSERIALIZABLE_POD(graphlab::gl_uint128_t);\n\n#endif\n\n"
  },
  {
    "path": "src/graphlab/util/union_find.hpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_UTIL_UNION_FIND_HPP\n#define GRAPHLAB_UTIL_UNION_FIND_HPP\n#include \n#include \n#include \n\nnamespace graphlab {\n// IDType must be an integer type and its maximum \n// value must be larger than the length of the sequence\ntemplate \nclass union_find {\n  private:\n    std::vector > setid;\n\n    bool is_root(IDType i) {\n      return setid[i].first == (IDType)i;\n    }\n    \n  public:\n    union_find() { }\n    void init(IDType s) {\n      setid.resize((size_t)s);\n      for (size_t i = 0; i < setid.size() ;++i) {\n        setid[i].first = (IDType)(i);\n        setid[i].second = 0;\n      }\n    }\n    \n    void merge(IDType i, IDType j) {\n      IDType iroot = find(i);\n      IDType jroot = find(j);\n      if (iroot == jroot) return;\n      else if (setid[iroot].second < setid[jroot].second) {\n        setid[iroot].first = jroot;\n      }\n      else if (setid[iroot].second > setid[jroot].second) {\n        setid[jroot].first = iroot;\n      }\n      else {\n        setid[jroot].first = iroot;\n        // make sure we don't overflow\n        if (setid[iroot].second + 1 > setid[iroot].second) {\n          setid[iroot].second = setid[iroot].second + 1;\n        }\n      }\n    }\n\n    IDType find(IDType i) {\n      IDType root = i;\n      if (is_root(root)) return root;\n      \n      // get the id of the root element\n      while (!is_root(root)) { root = setid[root].first; }\n      \n      // update the parents and ranks all the way up\n      IDType cur = i;\n      while (!is_root(cur)) {\n        IDType parent = setid[cur].first;\n        setid[cur].first = root;\n        cur = parent;\n      }\n      \n      return setid[i].first;\n    }\n};\n\n\nclass concurrent_union_find {\n  private:\n    union elem{\n      struct {\n        uint32_t next;\n        uint32_t rank;\n      } d;\n      uint64_t val;\n    };\n    \n    std::vector setid;\n\n    bool is_root(uint32_t i) {\n      return setid[i].d.next == i;\n    }\n\n    bool updateroot(uint32_t x, uint32_t oldrank,\n                    uint32_t y, uint32_t newrank) {\n      elem old; old.d.next = x; old.d.rank = oldrank;\n      elem newval; newval.d.next = y; newval.d.rank = newrank;\n      return atomic_compare_and_swap(setid[x].val, old.val, newval.val);\n    }\n    \n  public:\n    concurrent_union_find() { }\n    void init(uint32_t s) {\n      setid.resize((size_t)s);\n      for (size_t i = 0; i < setid.size() ;++i) {\n        setid[i].d.next = (uint32_t)(i);\n        setid[i].d.rank = 0;\n      }\n    }\n\n    void merge(uint32_t x, uint32_t y) {\n\n      uint32_t xr, yr;\n      while(1) {\n        x = find(x);\n        y = find(y);\n        if (x == y) return;\n        xr = setid[x].d.rank;\n        yr = setid[y].d.rank;\n\n        if (xr > yr || (xr == yr && x > y)) {\n          std::swap(x,y); std::swap(xr, yr);\n        }\n\n        if (updateroot(x, xr, y, xr)) break;\n      }\n      if (xr == yr) {\n        __sync_add_and_fetch(&(setid[y].d.rank), 1);\n      }\n    }\n\n    uint32_t find(uint32_t x) {\n      if (is_root(x)) return x;\n      \n      uint32_t y = x;\n      // get the id of the root element\n      while (!is_root(x)) { x = setid[x].d.next; }\n\n      // update the parents and ranks all the way up\n      while (setid[y].d.rank < setid[x].d.rank) {\n        uint32_t t = setid[y].d.next;\n        atomic_compare_and_swap(setid[y].d.next, t, x);\n        y = setid[t].d.next;\n      }\n      return x;\n    }\n};\n}\n#endif"
  },
  {
    "path": "src/graphlab/util/util_includes.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n#include \n#include \n#include \n#include \n// #include \n// #include \n#include \n#include \n#include \n#include \n#include \n"
  },
  {
    "path": "src/graphlab/util/web_util.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \n#include \n\n\n\nnamespace graphlab {\n  namespace web_util {\n\n    std::string url_decode(const std::string& url) {\n#define HEXTOI(x) (isdigit(x) ? x - '0' : x - 'W')\n      std::string ret_str;\n      for (size_t i = 0; i < url.size(); ++i) {\n        if (url[i] == '%' && \n            (i+1 < url.size() && isxdigit(url[i+1])) &&\n            (i+1 < url.size() && isxdigit(url[i+2]))) {\n          const char a = tolower(url[i+1]);\n          const char b = tolower(url[i+2]);\n          const char new_char = ((HEXTOI(a) << 4) | HEXTOI(b));\n          i += 2;\n          ret_str.push_back(new_char);\n        } else if (url[i] == '+') {\n          ret_str.push_back(' ');\n        } else {\n          ret_str.push_back(url[i]);\n        }\n      }\n#undef HEXTOI\n      return ret_str;\n    } // end of url decode\n\n\n\n    std::map parse_query(const std::string& query) {\n      std::vector pairs = graphlab::strsplit(query, \",=\", true);\n      std::map map;\n      for(size_t i = 0; i+1 < pairs.size(); i+=2) \n        map[url_decode(pairs[i])] = url_decode(pairs[i+1]);\n      return map;\n    } // end of parse url query\n\n  } // end of namespace web_util\n \n}; // end of namespace GraphLab\n\n"
  },
  {
    "path": "src/graphlab/util/web_util.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_WEB_UTIL_HPP\n#define GRAPHLAB_WEB_UTIL_HPP\n\n#include \n#include \n\n\nnamespace graphlab {\n  namespace web_util {\n\n    /**\n     * \\brief decode a url by converting escape characters\n     */\n    std::string url_decode(const std::string& url); \n\n    /**\n     * \\brief convert a query string into a map\n     */\n    std::map parse_query(const std::string& query);\n\n  } // end of namespace web_util\n \n}; // end of namespace GraphLab\n#endif\n\n\n\n\n\n"
  },
  {
    "path": "src/graphlab/version.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_VERSION_HPP\n#define GRAPHLAB_VERSION_HPP\n\n#define GRAPHLAB_VERSION \"2_1_0\"\n\n#define GRAPHLAB_VERSION_MAJOR 2\n#define GRAPHLAB_VERSION_MINOR 1\n#define GRAPHLAB_VERSION_REVISION 0\n\n#endif\n\n"
  },
  {
    "path": "src/graphlab/vertex_program/CMakeLists.txt",
    "content": "project(GraphLab)\n\n\n\n\n"
  },
  {
    "path": "src/graphlab/vertex_program/context.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_CONTEXT_HPP\n#define GRAPHLAB_CONTEXT_HPP\n\n#include \n#include \n#include \n\n#include \nnamespace graphlab {\n\n\n\n  /**\n   * \\brief The context object mediates the interaction between the\n   * vertex program and the graphlab execution environment and\n   * implements the \\ref icontext interface.\n   *\n   * \\tparam Engine the engine that is using this context.\n   */\n  template\n  class context : \n    public icontext {\n  public:\n    // Type members ===========================================================\n    /** The engine that created this context object */\n    typedef Engine engine_type;\n\n    /** The parent type */\n    typedef icontext icontext_type;\n    typedef typename icontext_type::graph_type graph_type;\n    typedef typename icontext_type::vertex_id_type vertex_id_type;\n    typedef typename icontext_type::vertex_type vertex_type;   \n    typedef typename icontext_type::message_type message_type;\n    typedef typename icontext_type::gather_type gather_type;\n\n\n\n  private:\n    /** A reference to the engine that created this context */\n    engine_type& engine;\n    /** A reference to the graph that is being operated on by the engine */\n    graph_type& graph;\n       \n  public:        \n    /** \n     * \\brief Construct a context for a particular engine and graph pair.\n     */\n    context(engine_type& engine, graph_type& graph) : \n      engine(engine), graph(graph) { }\n\n    size_t num_vertices() const { return graph.num_vertices(); }\n\n    /**\n     * Get the number of edges in the graph\n     */\n    size_t num_edges() const { return graph.num_edges(); }\n\n    // /**\n    //  * Get an estimate of the number of update functions executed up\n    //  * to this point.\n    //  */\n    // size_t num_updates() const { return engine.num_updates(); }\n\n    size_t procid() const { return graph.procid(); }\n      \n    size_t num_procs() const { return graph.numprocs(); }\n\n    std::ostream& cout() const {\n      return graph.dc().cout();\n    }\n\n    std::ostream& cerr() const {\n      return graph.dc().cerr();\n    }\n\n    /**\n     * Get the elapsed time in seconds\n     */\n    float elapsed_seconds() const { return engine.elapsed_seconds(); }\n\n    /**\n     * Return the current interation number (if supported).\n     */\n    int iteration() const { return engine.iteration(); }\n\n    /**\n     * Force the engine to stop executing additional update functions.\n     */\n    void stop() { engine.internal_stop(); }\n\n    /**\n     * Send a message to a vertex.\n     */\n    void signal(const vertex_type& vertex, \n                const message_type& message = message_type()) {\n      engine.internal_signal(vertex, message);\n    }\n\n    /**\n     * Send a message to an arbitrary vertex ID.\n     * \\warning If sending to neighboring vertices, the \\ref signal()\n     * function is more efficientas it permits sender side message combining.\n     */\n    void signal_vid(vertex_id_type vid, \n                    const message_type& message = message_type()) {\n      engine.internal_signal_gvid(vid, message);\n    }\n\n\n    /**\n     * Post a change to the cached sum for the vertex\n     */\n    void post_delta(const vertex_type& vertex, \n                    const gather_type& delta) {\n      engine.internal_post_delta(vertex, delta);\n    }\n\n    /**\n     * Invalidate the cached gather on the vertex.\n     */\n    virtual void clear_gather_cache(const vertex_type& vertex) { \n      engine.internal_clear_gather_cache(vertex);      \n    }\n\n\n                                                \n\n  }; // end of context\n  \n} // end of namespace\n#include \n\n#endif\n\n"
  },
  {
    "path": "src/graphlab/vertex_program/icontext.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_ICONTEXT_HPP\n#define GRAPHLAB_ICONTEXT_HPP\n\n#include \n#include \n#include \n#include \n\n#include \nnamespace graphlab {\n\n\n\n  /**\n   * \\brief The context object mediates the interaction between the\n   * vertex program and the graphlab execution environment.\n   *\n   * Each of the vertex program (see \\ref ivertex_program) methods is\n   * passed a reference to the engine's context.  The context allows\n   * vertex programs to access information about the current execution\n   * and send information (through icontext::signal,\n   * icontext::post_delta, and icontext::clear_gather_cache) to the\n   * graphlab engines (see \\ref iengine).\n   *\n   * \\tparam GraphType the type of graph (typically \\ref distributed_graph)\n   * \\tparam GatherType the user defined gather type (see \n   * \\ref ivertex_program::gather_type).\n   * \\tparam MessageType the user defined message type (see \n   * \\ref ivertex_program::message_type).\n   */\n  template\n  class icontext {\n  public:\n    // Type members ===========================================================\n    \n    /**\n     * \\brief the user graph type (typically \\ref distributed_graph)\n     */\n    typedef GraphType graph_type;   \n\n    /**\n     * \\brief the opaque vertex_type defined in the ivertex_program::graph_type\n     * (typically distributed_graph::vertex_type)\n     */\n    typedef typename graph_type::vertex_type vertex_type;\n\n    /**\n     * \\brief the global vertex identifier (see\n     * graphlab::vertex_id_type).\n     */\n    typedef typename graph_type::vertex_id_type vertex_id_type;\n\n    /**\n     * The message type specified by the user-defined vertex-program.\n     * (see ivertex_program::message_type)\n     */\n    typedef MessageType message_type;\n\n    /**\n     * The type returned by the gather operation.  (see\n     * ivertex_program::gather_type)\n     */\n    typedef GatherType gather_type;\n\n   \n  public:        \n    /** \\brief icontext destructor */\n    virtual ~icontext() { }\n    \n    /**\n     * \\brief Get the total number of vertices in the graph.\n     *\n     * \\return the total number of vertices in the entire graph.\n     */\n    virtual size_t num_vertices() const { return 0; }\n\n    /**\n     * \\brief Get the number of edges in the graph.\n     *\n     * Each direction counts as a separate edge.\n     *\n     * \\return the total number of edges in the entire graph.\n     */\n    virtual size_t num_edges() const { return 0; }\n\n    /**\n     * \\brief Get the id of this process.\n     *\n     * The procid is a number between 0 and \n     * \\ref graphlab::icontext::num_procs\n     * \n     * \\warning Each process may have many threads\n     *\n     * @return the process of this machine.\n     */\n    virtual size_t procid() const { return 0; }\n\n    /**\n     * \\brief Returns a standard output object (like cout)\n     *        which only prints once even when running distributed.\n     * \n     * This returns a C++ standard output stream object\n     * which maps directly to std::cout on machine with \n     * process ID 0, and to empty output streamss\n     * on all other processes. Calling,\n     * \\code\n     *   context.cout() << \"Hello World!\";\n     * \\endcode\n     * will therefore only print if the code is run on machine 0.\n     * This is useful in the finalize operation in aggregators.\n     */\n    virtual std::ostream& cout() const { return std::cout; }\n\n    /**\n     * \\brief Returns a standard error object (like cerr)\n     *        which only prints once even when running distributed.\n     * \n     * This returns a C++ standard output stream object\n     * which maps directly to std::cerr on machine with \n     * process ID 0, and to empty output streamss\n     * on all other processes. Calling,\n     * \\code\n     *   context.cerr() << \"Hello World!\";\n     * \\endcode\n     * will therefore only print if the code is run on machine 0.\n     * This is useful in the finalize operation in aggregators.\n     */\n    virtual std::ostream& cerr() const { return std::cerr; } \n\n    /**\n     * \\brief Get the number of processes in the current execution.\n     *\n     * This is typically the number of mpi jobs created:\n     * \\code\n     * %> mpiexec -n 16 ./pagerank\n     * \\endcode\n     * would imply that num_procs() returns 16.\n     *\n     * @return the number of processes in the current execution\n     */\n    virtual size_t num_procs() const { return 0; }\n\n    /**\n     * \\brief Get the elapsed time in seconds since start was called.\n     * \n     * \\return runtine in seconds\n     */\n    virtual float elapsed_seconds() const { return 0.0; }\n\n    /**\n     * \\brief Return the current interation number (if supported).\n     *\n     * \\return the current interation number if support or -1\n     * otherwise.\n     */\n    virtual int iteration() const { return -1; } \n\n    /**\n     * \\brief Signal the engine to stop executing additional update\n     * functions.\n     *\n     * \\warning The execution engine will stop *eventually* and\n     * additional update functions may be executed prior to when the\n     * engine stops. For-example the synchronous engine (see \\ref\n     * synchronous_engine) will complete the current super-step before\n     * terminating.\n     */\n    virtual void stop() { } \n\n    /**\n     * \\brief Signal a vertex with a particular message.\n     *\n     * This function is an essential part of the GraphLab abstraction\n     * and is used to encode iterative computation. Typically a vertex\n     * program will signal neighboring vertices during the scatter\n     * phase.  A vertex program may choose to signal neighbors on when\n     * changes made during the previos phases break invariants or warrant\n     * future computation on neighboring vertices.\n     * \n     * The signal function takes two arguments. The first is mandatory\n     * and specifies which vertex to signal.  The second argument is\n     * optional and is used to send a message.  If no message is\n     * provided then the default message is used.\n     *\n     * \\param vertex [in] The vertex to send the message to\n     * \\param message [in] The message to send, defaults to message_type(). \n     */\n    virtual void signal(const vertex_type& vertex, \n                        const message_type& message = message_type()) { }\n\n    /**\n     * \\brief Send a message to a vertex ID.\n     *\n     * \\warning This function will be slow since the current machine\n     * do not know the location of the vertex ID.  If possible use the\n     * the icontext::signal call instead.\n     *\n     * \\param gvid [in] the vertex id of the vertex to signal\n     * \\param message [in] the message to send to that vertex, \n     * defaults to message_type().\n     */\n    virtual void signal_vid(vertex_id_type gvid, \n                            const message_type& message = message_type()) { }\n\n    /**\n     * \\brief Post a change to the cached sum for the vertex\n     * \n     * Often a vertex program will be signaled due to a change in one\n     * or a few of its neighbors.  However the gather operation will\n     * be rerun on all neighbors potentially producing the same value\n     * as previous invocations and wasting computation time.  To\n     * address this some engines support caching (see \\ref\n     * gather_caching for details) of the gather phase.\n     *\n     * When caching is enabled the engines save a copy of the previous\n     * gather for each vertex.  On subsequent calls to gather if their\n     * is a cached gather then the gather phase is skipped and the\n     * cached value is passed to the ivertex_program::apply function.\n     * Therefore it is the responsibility of the vertex program to\n     * update the cache values for neighboring vertices. This is\n     * accomplished by using the icontext::post_delta function.\n     * Posted deltas are atomically added to the cache.\n     *\n     * \\param vertex [in] the vertex whose cache we want to update\n     * \\param delta [in] the change that we want to *add* to the\n     * current cache.\n     *\n     */\n    virtual void post_delta(const vertex_type& vertex, \n                            const gather_type& delta) { } \n\n    /**\n     * \\brief Invalidate the cached gather on the vertex.\n     *\n     * When caching is enabled clear_gather_cache clears the cache\n     * entry forcing a complete invocation of the subsequent gather.\n     *\n     * \\param vertex [in] the vertex whose cache to clear.\n     */\n    virtual void clear_gather_cache(const vertex_type& vertex) { } \n\n  }; // end of icontext\n  \n} // end of namespace\n#include \n\n#endif\n\n"
  },
  {
    "path": "src/graphlab/vertex_program/ivertex_program.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n * Also contains code that is Copyright 2011 Yahoo! Inc.  All rights\n * reserved. \n *\n */\n\n#ifndef GRAPHLAB_IVERTEX_PROGRAM_HPP\n#define GRAPHLAB_IVERTEX_PROGRAM_HPP\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\n#if defined(__cplusplus) && __cplusplus >= 201103L\n// for whatever reason boost concept is broken under C++11. \n// Temporary workaround. TOFIX\n#undef BOOST_CONCEPT_ASSERT\n#define BOOST_CONCEPT_ASSERT(unused)\n#endif\n\n\nnamespace graphlab {\n  \n  /**\n   * \\brief The ivertex_program class defines the vertex program\n   * interface that all vertex programs should extend and implement.\n   * The vertex-program is used to encode the user-define computation\n   * in a GraphLab program.\n   *\n   * Overview\n   * ==================\n   *\n   * A vertex program represents the primary user defined computation\n   * in GraphLab.  A unique instance of the vertex program is run on\n   * each vertex in the graph and can interact with neighboring vertex\n   * programs through the gather and scatter functions as well as by\n   * signaling neighboring vertex-programs.  Conceptually the\n   * vertex-program is a class which represents the parts of an\n   * update-function in the original formulation of the GraphLab\n   * abstraction.  Moreover many graph-structured programs can be\n   * written in the following pattern:\n   *\n   * \\code\n   * graphlab::update_function(Vertex center, Neighborhood nbrs) {\n   *   // nbrs represents the state of neighboring vertices and edges\n   *\n   *   // Gather Phase: \n   *   sum = EMPTY;\n   *   for(edge in nbrs.in_edges()) {\n   *      // The sum is a general commutative associative operation\n   *      if(sum == EMPTY) sum = gather_function(center, edge, edge.neighbor());\n   *      else sum += gather_function(center, edge, edge.neighbor());\n   *   }\n   *\n   *   // Apply Phase:\n   *   center = apply_function(center, sum);\n   *\n   *   // Scatter Phase:\n   *   for(edge in nbrs.out_edges()) {\n   *     edge = scatter_function(center, edge, edge.neighbor());\n   *     if(condition is met) trigger_neighbor();\n   *   }\n   *\n   * }\n   * \\endcode\n   *\n   * Vertex programs express computation by implementing what we call\n   * the *Gather-Apply-Scatter (GAS)* model which decomposes the\n   * vertex program into a parallel gather phase, followed by an\n   * atomic apply phase, and finally a parallel scatter phase.  This\n   * decomposition allows us to execute a single vertex program on\n   * several machines simultaneously and move computation to the data.  \n   *\n   * We therefore decompose the update function logic into member\n   * functions of the vertex-program class that are invoked in the\n   * following manner:\n   *\n   * \\code\n   * For the center vertex vtx:\n   *   vprog.init(ctx, vtx, msg);\n   *   // Gather Phase: \n   *   vprog::gather_type sum = EMPTY;\n   *   ParallelFor(adjacent edges in direction vprog.gather_edges(ctx, vtx) )\n   *     if(sum == EMPTY) sum = vprog.gather(ctx, vtx, edge);\n   *     else sum += vprog.gather(ctx, vtx, edge);\n   *   // Apply Phase\n   *   vprog.apply(ctx, vtx, sum);\n   *   // Scatter Phase\n   *   ParallelFor(adjacent edges in direction vprog.scatter_edges(ctx, vtx) )\n   *     vprog.scatter(ctx, vtx, edge);\n   *   // Vertex program is destroyed\n   *   vprog = vertex_program();\n   * \\endcode\n   *\n   * All user define vertex programs must extend the ivertex_program\n   * interface and implement the ivertex_program::apply function.\n   * Most vertex programs will also implement the\n   * ivertex_program::gather and ivertex_program::scatter functions.\n   *\n   * The state of a vertex program *does not* persist between\n   * invocations of \\ref ivertex_program::init.  Moreover prior to\n   * each call to init the vertex program's previous state is\n   * cleared. Therefore any persistent state must be saved into the\n   * vertex data.\n   *\n   * The vertex program depends on several key types which are\n   * template arguments to ivertex_program interface.\n   * \n   * \\li graph_type: the type of graph used to store the data for this\n   * vertex program.  This currently always the distributed_graph.\n   *\n   * \\li gather_type: the type used in the gather phase and must\n   * implement the operator+= function.\n   *\n   * \\li message_type: The type used for signaling and is typically\n   * empty.  However if a message type is desired it must implement\n   * the operator+= to allow message merging across the network.  In\n   * addition the message type may also implement the priority()\n   * function which returns a double assigning a priority to the\n   * reception of the message (used by the asynchronous engines). We\n   * provide a basic set of simple prioritized messages in \n   * \\ref graphlab::signals.\n   *\n   * All user-defined types including the vertex data, edge data,\n   * vertex-program, gather type, and message type must be\n   * serializable (see \\ref sec_serializable) and default\n   * constructible to enable movement between machines.\n   *\n   * Advanced Features\n   * ======================\n   *\n   * While the basic Gather-Apply-Scatter approach to graph structure\n   * computation can express a wide range of algorithms there are some\n   * situation where additional features could either simplify the\n   * design or provide additional efficiency. \n   *\n   *\n   * Messaging \n   * ----------------------\n   * \n   * Vertex-programs can trigger adjacent vertex programs by sending a\n   * signal which can contain a message to neighbor vertices.  By\n   * default the message type is empty however it is possible for the\n   * user to define a message type.  For example the following\n   * message_type could be used to implement pagerank:\n   *\n   * \\code\n   * struct pagerank_message : public graphlab::IS_POD_TYPE {\n   *   double value;\n   *   double priority() const { return std::fabs(value); }\n   *   message_type& operator+=(const message_type& other) {\n   *     value += other.value; \n   *     return *this;\n   *   }\n   * }; \n   * \\endcode\n   *\n   * Unlike other messaging abstractions, GraphLab always _merges_\n   * messages destined to the same vertex.  This allows the GraphLab\n   * engines to minimize network communication and more evenly balance\n   * computation.  Messages are combined using the operator+=\n   * function.\n   *\n   * As mentioned earlier some engines may prioritize the _reception_\n   * of messages.  Messages can optionally (it is not required)\n   * provide a priority function which is used to prioritize message\n   * reception.  The engine then attempts to prioritize the reception\n   * of higher priority messages first.\n   *\n   * The message is received in the \\ref ivertex_program::init\n   * function.  The single message passed into \n   * \\ref ivertex_program::init represents the sum of all messages \n   * destined to that vertex since the vertex-program was last invoked.\n   * \n   * The GraphLab messaging framework allows us to write\n   * Pregel-like vertex-programs of the form:\n   *\n   * \\code\n   * typedef graphlab::empty gather_type;\n   * class pregel_pagerank : \n   *   public ivertex_program,\n   *   public graphlab::IS_POD_TYPE {\n   *\n   *   // Store a local copy of the message data\n   *   double message_value;\n   *\n   *   // Receive the inbound message (sum of messages)\n   *   void init(icontext_type& context, const vertex_type& vertex, \n   *             const message_type& msg) { \n   *     message_value = message.value;\n   *   }\n   *\n   *   // Skip the gather phase\n   *   edge_dir_type gather_edges(icontext_type& context,\n   *                              const vertex_type& vertex) const { \n   *     return graphlab::NO_EDGES; \n   *   }\n   *\n   *   // Update the pagerank using the message\n   *   void apply(icontext_type& context, vertex_type& vertex, \n   *              const gather_type& total) {\n   *     vertex.data() += message_value;      \n   *   }\n   *\n   *   // Scatter along out edges\n   *   edge_dir_type scatter_edges(icontext_type& context,\n   *                               const vertex_type& vertex) const { \n   *     return OUT_EDGES; \n   *   }\n   *\n   *   // Compute new messages encoding the change in the pagerank of\n   *   // adjacent vertices.\n   *   void scatter(icontext_type& context, const vertex_type& vertex, \n   *                edge_type& edge) const { \n   *     pagerank_message msg;\n   *     msg.value = message_value * (1 - RESET_PROBABILITY);\n   *     context.signal(edge.target(), msg);\n   *   }\n   * }; \n   * \\endcode\n   *\n   * Notice that the gather phase is skipped and instead the gather\n   * computation is accomplished using the messages.  However unlike\n   * Pregel the scatter function which computs and sends the new\n   * message is actually run on the machine that is receiving the\n   * message.  \n   *\n   * The message abstraction is surprisingly powerful and can often\n   * often express computation that can be written using the Gather\n   * operation.  However, the message combination is done outside of\n   * the consistency model and so can lead to more confusing code.\n   *\n   * Gather Caching\n   * ---------------------\n   *\n   * In many applications the gather computation can be costly and\n   * high-degree vertices will be signaled often even only a small\n   * fraction of its neighbors values have changed.  In this case\n   * running the gather function on all neighbors can be wasteful.  To\n   * address this important issue the GraphLab engines expose a gather\n   * caching mechanism.  However to take advantage of the gather\n   * caching the vertex-program must notify the engine when a cache is\n   * no longer valid and can even correct the cache to ensure that it\n   * remains valid.\n   *\n   * \\todo finish documenting gather caching\n   *\n   * \n   * \n   */\n  template \n  class ivertex_program {    \n  public:\n\n    // User defined type members ==============================================\n    /**\n     * \\brief The user defined vertex data associated with each vertex\n     * in the graph (see \\ref distributed_graph::vertex_data_type).\n     *\n     * The vertex data is the data associated with each vertex in the\n     * graph.  Unlike the vertex-program the vertex data of adjacent\n     * vertices is visible to other vertex programs during the gather\n     * and scatter phases and persists between executions of the\n     * vertex-program.\n     *\n     * The vertex data type must be serializable \n     * (see \\ref sec_serializable)\n     */\n    typedef typename Graph::vertex_data_type vertex_data_type;\n\n    /**\n     * \\cond GRAPHLAB_INTERNAL\n     *\n     * \\brief GraphLab Requires that the vertex data type be Serializable.  See\n     * \\ref sec_serializable for details. \n     */\n    BOOST_CONCEPT_ASSERT((graphlab::Serializable));\n    /// \\endcond\n\n\n\n    /**\n     * \\brief The user defined edge data associated with each edge in\n     * the graph.\n     *\n     * The edge data type must be serializable \n     * (see \\ref sec_serializable)\n     *\n     */\n    typedef typename Graph::edge_data_type edge_data_type;\n\n    /**\n     * \\cond GRAPHLAB_INTERNAL\n     *\n     * \\brief GraphLab Requires that the edge data type be Serializable.  See\n     * \\ref sec_serializable for details. \n     */\n    BOOST_CONCEPT_ASSERT((graphlab::Serializable));\n    /// \\endcond\n\n\n    /**\n     * \\brief The user defined gather type is used to accumulate the\n     * results of the gather function during the gather phase and must\n     * implement the operator += operation.\n     *\n     * The gather type plays the following role in the vertex program:\n     * \n     * \\code\n     * gather_type sum = EMPTY;\n     * for(edges in vprog.gather_edges()) {\n     *   if(sum == EMPTY) sum = vprog.gather(...);\n     *   else sum += vprog.gather( ... );\n     * }\n     * vprog.apply(..., sum);\n     * \\endcode\n     *\n     * In addition to implementing the operator+= operation the gather\n     * type must also be serializable (see \\ref sec_serializable).\n     */\n    typedef GatherType gather_type;\n    \n    /**\n     * \\cond GRAPHLAB_INTERNAL\n     * \\brief GraphLab Requires that the gather type be default\n     * constructible.\n     *\n     * \\code\n     * class gather_type {\n     * public:\n     *   gather_type() { }\n     * };  \n     * \\endcode\n     */\n    BOOST_CONCEPT_ASSERT((boost::DefaultConstructible));\n    /// \\endcond\n\n    /**\n     * \\cond GRAPHLAB_INTERNAL\n     *\n     * \\brief GraphLab Requires that gather type be Serializable.  See\n     * \\ref sec_serializable for detials\n     */\n    BOOST_CONCEPT_ASSERT((graphlab::Serializable));\n    /// \\endcond\n\n    /**\n     * \\cond GRAPHLAB_INTERNAL\n     *\n     * \\brief GraphLab Requires that gather type support operator+=.\n     */\n    BOOST_CONCEPT_ASSERT((graphlab::OpPlusEq));\n    /// \\endcond\n\n\n\n    /**\n     * \\cond GRAPHLAB_INTERNAL\n     *\n     *  \\brief GraphLab Requires that the gather type be serializable\n     *\n     * \\code\n     * class gather_type {\n     * public:\n     *   gather_type() { }\n     * };  \n     * \\endcode\n     */\n    BOOST_CONCEPT_ASSERT((boost::DefaultConstructible));\n    /// \\endcond\n\n\n\n    /**\n     * The message type which must be provided by the vertex_program\n     */\n    typedef MessageType message_type;\n\n\n    /**\n     * \\cond GRAPHLAB_INTERNAL\n     *\n     * \\brief GraphLab requires that the message type be default\n     * constructible.\n     *\n     * \\code\n     * class message_type {\n     * public:\n     *   message_type() { }\n     * };  \n     * \\endcode\n     * \n     */\n    BOOST_CONCEPT_ASSERT((boost::DefaultConstructible));\n    /// \\endcond \n\n    /**\n     * \\cond GRAPHLAB_INTERNAL\n     *\n     * \\brief GraphLab requires that the message type be Serializable.\n     * See \\ref sec_serializable for detials\n     */\n    BOOST_CONCEPT_ASSERT((graphlab::Serializable));\n    /// \\endcond\n\n    /**\n     * \\cond GRAPHLAB_INTERNAL\n     *\n     * \\brief GraphLab requires that message type support operator+=.\n     */\n    BOOST_CONCEPT_ASSERT((graphlab::OpPlusEq));\n    /// \\endcond\n\n\n    // Graph specific type members ============================================\n    /**\n     * \\brief The graph type associative with this vertex program. \n     *\n     * The graph type is specified as a template argument and will\n     * usually be \\ref distributed_graph.\n     */\n    typedef Graph graph_type;\n\n    /**\n     * \\brief The unique integer id used to reference vertices in the graph.\n     * \n     * See \\ref graphlab::vertex_id_type for details.\n     */\n    typedef typename graph_type::vertex_id_type vertex_id_type;\n    \n    /**\n     * \\brief The opaque vertex object type used to get vertex\n     * information.\n     *\n     * The vertex type is defined by the graph.  \n     * See \\ref distributed_graph::vertex_type for details.\n     */\n    typedef typename graph_type::vertex_type vertex_type;\n    \n    /**\n     * \\brief The opaque edge_object type used to access edge\n     * information.\n     * \n     * The edge type is defined by the graph.  \n     * See \\ref distributed_graph::edge_type for details.\n     */\n    typedef typename graph_type::edge_type edge_type;\n\n    /**\n     * \\brief The type used to define the direction of edges used in\n     * gather and scatter.\n     * \n     * Possible values include:\n     *\n     * \\li graphlab::NO_EDGES : Do not process any edges\n     * \n     * \\li graphlab::IN_EDGES : Process only inbound edges to this\n     * vertex\n     * \n     * \\li graphlab::OUT_EDGES : Process only outbound edges to this\n     * vertex\n     *\n     * \\li graphlab::ALL_EDGES : Process both inbound and outbound\n     * edges on this vertes.\n     * \n     * See \\ref graphlab::edge_dir_type for details.\n     */\n    typedef graphlab::edge_dir_type edge_dir_type;\n\n    // Additional Types =======================================================\n    \n    /**\n     * \\brief The context type is used by the vertex program to\n     * communicate with the engine.\n     *\n     * The context and provides facilities for signaling adjacent\n     * vertices (sending messages), interacting with the GraphLab\n     * gather cache (posting deltas), and accessing engine state.\n     *\n     */\n    typedef icontext icontext_type;\n   \n    // Functions ==============================================================\n    /**\n     * \\brief Standard virtual destructor for an abstract class.\n     */\n    virtual ~ivertex_program() { }\n\n    /**\n     * \\brief This called by the engine to receive a message to this\n     * vertex program.  The vertex program can use this to initialize\n     * any state before entering the gather phase.  The init function\n     * is invoked _once_ per execution of the vertex program.\n     *\n     * If the vertex program does not implement this function then the\n     * default implementation (NOP) is used.\n     *\n     * \\param [in,out] context The context is used to interact with the engine\n     *\n     * \\param [in] vertex The vertex on which this vertex-program is\n     * running. Note that the vertex is constant and its value should\n     * not be modified within the init function.  If there is some\n     * message state that is needed then it must be saved to the\n     * vertex-program and not the vertex data.\n     *\n     * \\param [in] message The sum of all the signal calls to this\n     * vertex since it was last run.\n     */\n    virtual void init(icontext_type& context,\n                      const vertex_type& vertex, \n                      const message_type& msg) { /** NOP */ }\n    \n\n    /**\n     * \\brief Returns the set of edges on which to run the gather\n     * function.  The default edge direction is in edges.\n     *\n     * The gather_edges function is invoked after the init function\n     * has completed.\n     *\n     * \\warning The gather_edges function may be invoked multiple\n     * times for the same execution of the vertex-program and should\n     * return the same value.  In addition it cannot modify the\n     * vertex-programs state or the vertex data.\n     *\n     * Possible return values include:\n     *\n     * \\li graphlab::NO_EDGES : The gather phase is completely skipped\n     * potentially reducing network communication.\n     * \n     * \\li graphlab::IN_EDGES : The gather function is only run on\n     * inbound edges to this vertex.\n     * \n     * \\li graphlab::OUT_EDGES : The gather function is only run on\n     * outbound edges to this vertex.\n     *\n     * \\li graphlab::ALL_EDGES : The gather function is run on both\n     * inbound and outbound edges to this vertes.\n     * \n     * \\param [in,out] context The context is used to interact with\n     * the engine\n     *\n     * \\param [in] vertex The vertex on which this vertex-program is\n     * running. Note that the vertex is constant and its value should\n     * not be modified.\n     * \n     * \\return One of graphlab::NO_EDGES, graphlab::IN_EDGES,\n     * graphlab::OUT_EDGES, or graphlab::ALL_EDGES.\n     * \n     */\n    virtual edge_dir_type gather_edges(icontext_type& context,\n                                       const vertex_type& vertex) const { \n      return IN_EDGES; \n    }\n\n\n    /**\n     * \\brief The gather function is called on all the \n     * \\ref ivertex_program::gather_edges in parallel and returns the \n     * \\ref gather_type which are added to compute the final output of \n     * the gather phase.  \n     *\n     * The gather function is the core computational element of the\n     * Gather phase and is responsible for collecting the information\n     * about the state of adjacent vertices and edges.  \n     *\n     * \\warning The gather function is executed in parallel on\n     * multiple machines and therefore cannot modify the\n     * vertex-program's state or the vertex data.\n     *\n     * A default implementation of the gather function is provided\n     * which will fail if invoked. \n     *\n     * \\param [in,out] context The context is used to interact with\n     * the engine\n     *\n     * \\param [in] vertex The vertex on which this vertex-program is\n     * running. Note that the vertex is constant and its value should\n     * not be modified.\n     *\n     * \\param [in,out] edge The adjacent edge to be processed.  The\n     * edge is not constant and therefore the edge data can be\n     * modified. \n     *\n     * \\return the result of the gather computation which will be\n     * \"summed\" to produce the input to the apply operation.  The\n     * behavior of the \"sum\" is defined by the \\ref gather_type.\n     * \n     */\n    virtual gather_type gather(icontext_type& context, \n                               const vertex_type& vertex, \n                               edge_type& edge) const {\n      logstream(LOG_FATAL) << \"Gather not implemented!\" << std::endl;\n      return gather_type();\n    };\n\n\n    /**\n     * \\brief The apply function is called once the gather phase has\n     * completed and must be implemented by all vertex programs.\n     *\n     * The apply function is responsible for modifying the vertex data\n     * and is run only once per vertex per execution of a vertex\n     * program.  In addition the apply function can modify the state\n     * of the vertex program.\n     *\n     * If a vertex has no neighbors than the apply function is called\n     * passing the default value for the gather_type.\n     *\n     * \\param [in,out] context The context is used to interact with\n     * the engine\n     *\n     * \\param [in,out] vertex The vertex on which this vertex-program is\n     * running. \n     *\n     * \\param [in] total The result of the gather phase.  If a vertex\n     * has no neighbors then the total is the default value (i.e.,\n     * gather_type()) of the gather type.\n     * \n     */\n    virtual void apply(icontext_type& context, \n                       vertex_type& vertex, \n                       const gather_type& total) = 0;\n\n    /**\n     * \\brief Returns the set of edges on which to run the scatter\n     * function.  The default edge direction is out edges.\n     *\n     * The scatter_edges function is invoked after the apply function\n     * has completed.\n     *\n     * \\warning The scatter_edges function may be invoked multiple\n     * times for the same execution of the vertex-program and should\n     * return the same value.  In addition it cannot modify the\n     * vertex-programs state or the vertex data.\n     *\n     * Possible return values include:\n     *\n     * \\li graphlab::NO_EDGES : The scatter phase is completely\n     * skipped potentially reducing network communication.\n     * \n     * \\li graphlab::IN_EDGE : The scatter function is only run on\n     * inbound edges to this vertex.\n     * \n     * \\li graphlab::OUT_EDGES : The scatter function is only run on\n     * outbound edges to this vertex.\n     *\n     * \\li graphlab::ALL_EDGES : The scatter function is run on both\n     * inbound and outbound edges to this vertes.\n     * \n     * \\param [in,out] context The context is used to interact with\n     * the engine\n     *\n     * \\param [in] vertex The vertex on which this vertex-program is\n     * running. Note that the vertex is constant and its value should\n     * not be modified.\n     * \n     * \\return One of graphlab::NO_EDGES, graphlab::IN_EDGES,\n     * graphlab::OUT_EDGES, or graphlab::ALL_EDGES.\n     * \n     */\n    virtual edge_dir_type scatter_edges(icontext_type& context,\n                                        const vertex_type& vertex) const { \n      return OUT_EDGES; \n    }\n\n    /**\n     * \\brief Scatter is called on all scatter_edges() in parallel\n     * after the apply function has completed and is typically\n     * responsible for updating edge data, signaling (messaging)\n     * adjacent vertices, and updating the gather cache state when\n     * caching is enabled.\n     *\n     * The scatter function is almost identical to the gather function\n     * except that nothing is returned. \n     *\n     * \\warning The scatter function is executed in parallel on\n     * multiple machines and therefore cannot modify the\n     * vertex-program's state or the vertex data.\n     *\n     * A default implementation of the gather function is provided\n     * which will fail if invoked. \n     *\n     * \\param [in,out] context The context is used to interact with\n     * the engine\n     *\n     * \\param [in] vertex The vertex on which this vertex-program is\n     * running. Note that the vertex is constant and its value should\n     * not be modified.\n     *\n     * \\param [in,out] edge The adjacent edge to be processed.  The\n     * edge is not constant and therefore the edge data can be\n     * modified. \n     *\n     */\n    virtual void scatter(icontext_type& context, const vertex_type& vertex, \n                         edge_type& edge) const { \n      logstream(LOG_FATAL) << \"Scatter not implemented!\" << std::endl;\n    };\n\n\n    /** \n     * \\internal\n     * Used to signal the start of a local gather.\n     * Called on each machine which is doing a gather operation.\n     * Semantics are that, a complete gather involves\n     * \n     * \\code\n     * On each machine with edges adjacent to vertex being updated:\n     *   vprogram.pre_local_gather(g) // passed by reference\n     *   foreach edge adjacent to vertex:\n     *     if ( ... first gather ... ) g = vprogram.gather(edge)\n     *     else g += vprogram.gather(edge)\n     *   end\n     *   vprogram.post_local_gather(g) // passed by reference\n     * \\endcode\n     */\n    virtual void pre_local_gather(gather_type&) const {\n    }\n\n    /** \n     * \\internal\n     * Used to signal the end of a local gather.\n     * Called on each machine which is doing a gather operation.\n     * Semantics are that, a complete gather involves\n     * \n     * \\code\n     * On each machine with edges adjacent to vertex being updated:\n     *   vprogram.pre_local_gather(g) // passed by reference\n     *   foreach edge adjacent to vertex:\n     *     if ( ... first gather ... ) g = vprogram.gather(edge)\n     *     else g += vprogram.gather(edge)\n     *   end\n     *   vprogram.post_local_gather(g) // passed by reference\n     * \\endcode\n     */\n    virtual void post_local_gather(gather_type&) const {\n    }\n\n  };  // end of ivertex_program\n \n}; //end of namespace graphlab\n#include \n\n#endif\n"
  },
  {
    "path": "src/graphlab/vertex_program/messages.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#ifndef GRAPHLAB_MESSAGES_HPP\n#define GRAPHLAB_MESSAGES_HPP\n\n#include \n\nnamespace graphlab {\n\n  namespace messages {\n\n    /**\n     * The priority of two messages is the sum\n     */\n    struct sum_priority : public graphlab::IS_POD_TYPE {\n      double value;\n      sum_priority(const double value = 0) : value(value) { }\n      double priority() const { return value; }\n      sum_priority& operator+=(const sum_priority& other) {\n        value += other.value;\n        return *this;\n      }\n    }; // end of sum_priority message\n\n    /**\n     * The priority of two messages is the max\n     */\n    struct max_priority : public graphlab::IS_POD_TYPE {\n      double value;\n      max_priority(const double value = 0) : value(value) { }\n      double priority() const { return value; }\n      max_priority& operator+=(const max_priority& other) {\n        value = std::max(value, other.value);\n        return *this;\n      }\n    }; // end of max_priority message\n\n\n  }; // end of messages namespace\n\n\n}; // end of graphlab namespace\n#endif\n"
  },
  {
    "path": "src/graphlab/vertex_program/op_plus_eq_concept.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_OP_PLUS_EQ_CONCEPT\n#define GRAPHLAB_OP_PLUS_EQ_CONCEPT\n\n#include \n#include \n#include \n#include \n#include \n\n\nnamespace graphlab {\n\n  /**\n   * \\brief Concept checks if a type T supports operator+=\n   *\n   * This is a concept checking class for boost::concept and can be\n   * used to enforce that a type T is \"additive.\"  In particular many\n   * types in GraphLab (e.g., messages, gather_type, as well as\n   * aggregation types) must support operator+=.  To achieve this the\n   * class should implement:\n   *\n   * \\code\n   * class gather_type {\n   *   int member1;\n   * public:\n   *   gather_type& operator+=(const gather_type& other) {\n   *     member1 += other.member1;\n   *     return *this;\n   *   } // end of operator+=\n   * };\n   * \\endcode\n   *\n   * \\tparam T The type to test for additivity\n   */\n  template \n  class OpPlusEq :  boost::Assignable, public boost::DefaultConstructible {\n   public:\n    BOOST_CONCEPT_USAGE(OpPlusEq) {\n      T t1 = T();\n      const T t2 = T();\n      // A compiler error on these lines implies that your type does\n      // not support operator+= when this is required (e.g.,\n      // gather_type or aggregator types)\n      t1 += t2;\n    }\n  };\n} // namespace graphlab\n#endif\n\n"
  },
  {
    "path": "src/graphlab/vertex_program/vertex_program_includes.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n#include \n#include \n#include \n\n\n"
  },
  {
    "path": "src/graphlab/warp.hpp",
    "content": "#include \n#include \n#include \n#include \n#include \n"
  },
  {
    "path": "src/graphlab/zookeeper/key_value.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \nextern \"C\" {\n#include \n}\n\nnamespace graphlab{\nnamespace zookeeper {\n\n\nkey_value::key_value(std::vector zkhosts,\n                     std::string _prefix,\n                     std::string serveridentifier):\n    prefix(_prefix), next_callback_id(0), closing(false) {\n  serveridentifier = serveridentifier;\n  // construct hosts list\n  std::string hosts = boost::algorithm::join(zkhosts, \",\");\n  prefix = normalize_path(prefix);\n  if (prefix[0] != '/') prefix = \"/\" + prefix;\n  // we need to block the watcher from running until everything is ready\n  handle = zookeeper_init(hosts.c_str(), watcher, 10000, NULL, (void*)this, 0);\n  assert(handle != NULL);\n  // create the prefix if it does not already exist\n  if (prefix != \"/\") create_dir(handle,\n                                prefix.substr(0, prefix.length() - 1),\n                                \"zk_key_value\");\n  if (prefix != \"/\") {\n    masters_path = prefix + \"masters/\";\n    values_path = prefix + \"values/\";\n  }\n  else {\n    masters_path = \"/masters/\";\n    values_path = \"/values/\";\n  }\n\n  create_dir(handle,\n             masters_path.substr(0, masters_path.length() - 1),\n             \"zk_key_value\");\n\n  create_dir(handle,\n             values_path.substr(0, values_path.length() - 1),\n             \"zk_key_value\");\n\n  datalock.lock();\n  std::vector unused1, unused2, unused3;\n  get_all_keys_locked(unused1, unused2, unused3);\n  datalock.unlock();\n}\n\nkey_value::~key_value() {\n  if (handle == NULL) return;\n  datalock.lock();\n  closing = true;\n  datalock.unlock();\n  // cleanup\n  std::set::const_iterator iter = my_values.begin();\n  while (iter != my_values.end()) {\n    int version = data[*iter].remote_version;\n    if (version >= 0) {\n      std::string value_node = get_sequence_node_path(values_path + (*iter) + \"-\",\n                                                     version);\n      delete_node(handle, value_node, \"zk_key_value cleanup\");\n    }\n    std::string master_node = masters_path + (*iter);\n    delete_node(handle, master_node, \"zk_key_value cleanup\");\n    ++iter;\n  }\n  /* should not try to delete it. It will mess up any watches.\n\n  delete_dir(handle,\n             masters_path.substr(0, masters_path.length() - 1),\n             \"zk_key_value cleanup\");\n  delete_dir(handle,\n             values_path.substr(0, values_path.length() - 1),\n             \"zk_key_value cleanup\");\n  if (prefix != \"/\") delete_dir(handle,\n                                prefix.substr(0, prefix.length() - 1),\n                                \"zk_key_value cleanup\");\n  */\n  zookeeper_close(handle);\n}\n\n\n/** Inserts a value to the key value store. Returns true on success.\n * False on failure (indicating the key already exists)\n */\nbool key_value::insert(const std::string& key, const std::string& value) {\n  if (key.length() == 0) return false;\n  if (my_values.count(key)) return modify(key, value);\n  // ok try to create the master node\n  int ret = create_ephemeral_node(handle,\n                                  masters_path + key,\n                                  serveridentifier,\n                                  \"zk_key_value insert\");\n  if (ret == ZNODEEXISTS) return false;\n  else {\n    // ok we own this key\n    my_values.insert(key);\n    return modify(key, value);\n  }\n}\n\n\nbool key_value::modify(const std::string& key, const std::string& value) {\n  if (key.length() == 0) return false;\n  if (my_values.count(key) == 0) return false;\n  datalock.lock();\n  // add a - to the end\n  std::pair ret = create_ephemeral_sequence_node(handle,\n                                                           values_path + key + \"-\",\n                                                           value,\n                                                           \"zk_key_value modify\");\n  assert(ret.first == ZOK);\n  // update the cache\n  lazy_value& val = data[key];\n  val.has_value = true;\n  int prev_remote_version = val.remote_version;\n  val.stored_version = ret.second;\n  val.value = value;\n  // try to delete the previous remote version node\n  if (prev_remote_version >= 0) {\n    std::string old_node_path = get_sequence_node_path(values_path + key + \"-\",\n                                                       prev_remote_version);\n    delete_node(handle, old_node_path, \"zk_key_value modify-cleanup\");\n  }\n  datalock.unlock();\n  return true;\n}\n\n\nbool key_value::erase(const std::string& key) {\n  if (key.length() == 0) return false;\n  if (my_values.count(key) == 0) return false;\n  datalock.lock();\n  lazy_value& val = data[key];\n  // find the current version\n  int cur_remote_version = std::max(val.stored_version, val.remote_version);\n  // try to delete it\n  if (cur_remote_version >= 0) {\n    std::string old_node_path = get_sequence_node_path(values_path + key + \"-\",\n                                                       cur_remote_version);\n    delete_node(handle, old_node_path, \"zk_key_value erase-value\");\n  }\n  std::string master_node = masters_path + key;\n  delete_node(handle, master_node, \"zk_key_value erase-master\");\n  // don't fully remove it from the data map yet\n  // let the trigger take care of it\n  my_values.erase(my_values.find(key));\n  datalock.unlock();\n  return true;\n}\n\n\n\nbool key_value::get_all_keys_locked(\n    std::vector& out_newkeys,\n    std::vector& out_deletedkeys,\n    std::vector& out_modifiedkeys) {\n\n\n  struct String_vector children;\n  children.count = 0;\n  children.data = NULL;\n\n  // get a list of all the keys and set the watch\n  std::string values_node = values_path.substr(0, values_path.length() - 1);\n  std::string master_node = masters_path.substr(0, masters_path.length() - 1);\n  int stat = zoo_get_children(handle, values_node.c_str(), 1, &children);\n  if (stat == ZCLOSING) return false;\n  if (stat != 0) {\n    print_stat(stat, \"zk_key_value get_all_keys values\", values_path);\n    return false;\n  }\n\n  struct String_vector masters;\n  stat = zoo_get_children(handle, master_node.c_str(), 1, &masters);\n  if (stat == ZCLOSING) return false;\n  if (stat != 0) {\n    print_stat(stat, \"zk_key_value get_all_keys masters\", masters_path);\n    free_String_vector(&children);\n    return false;\n  }\n\n  std::vector masterkeys = String_vector_to_vector(&masters);\n  std::vector allkeys = String_vector_to_vector(&children);\n  /*\n  for (size_t i = 0;i < masterkeys.size(); ++i) {\n    std::cout << \"\\t\" << masterkeys[i] << \"\\n\";\n  }\n  for (size_t i = 0;i < allkeys.size(); ++i) {\n    std::cout << \"\\t\" << allkeys[i] << \"\\n\";\n  }*/\n  free_String_vector(&children);\n  free_String_vector(&masters);\n\n  fill_data_locked(allkeys, masterkeys, out_newkeys, out_deletedkeys, out_modifiedkeys);\n  return true;\n}\n\nstd::pair key_value::get(const std::string& key) {\n  datalock.lock();\n  // search for the key in the map\n  std::map::const_iterator iter = data.find(key);\n  if (iter == data.end()) {\n    datalock.unlock();\n    return std::pair(false, \"\");\n  }\n  // see if we have a cached version\n  if (iter->second.has_value) {\n    // yup. we have a cached copy. return that\n    std::pair value(true, iter->second.value);\n    datalock.unlock();\n    return value;\n  }\n  // otherwise, we need to get the data.\n  // figure out the node we need to query\n  int remote_version = iter->second.remote_version;\n  std::string node = get_sequence_node_path(values_path + key + \"-\",\n                                            remote_version);\n  datalock.unlock();\n  // ok. try to query the node\n  //std::cout << \"Getting value for \" << node << \"\\n\";\n  std::pair value = get_node_value(handle, node, \"zk_key_value get\");\n  // if successful, return that\n  if (value.first) {\n    // cache the value\n    datalock.lock();\n    std::map::iterator iter = data.begin();\n    if (iter != data.end() && iter->second.remote_version == remote_version) {\n      iter->second.has_value = true;\n      iter->second.stored_version = remote_version;\n      iter->second.value = value.second;\n    }\n    datalock.unlock();\n  }\n  // otherwise... the node is missing. The watch should delete it eventually\n  return value;\n}\n\n\nvoid key_value::fill_data_locked(const std::vector& keys,\n                                 const std::vector& masterkeys,\n                                 std::vector& out_newkeys,\n                                 std::vector& out_deletedkeys,\n                                 std::vector& out_modifiedkeys) {\n  std::set masterkeyset;\n  for (size_t i = 0;i < masterkeys.size(); ++i) {\n    masterkeyset.insert(masterkeys[i]);\n  }\n  std::map key_and_version;\n  for (size_t i = 0;i < keys.size(); ++i) {\n    // this must be a sequence node!\n    assert(keys[i].length() > 10);\n    // where the sequence number is expected to start\n    size_t num_start = keys[i].length() - 10;\n    size_t key_length = num_start - 1;\n\n    // Ex: abc-1234567890\n    // length = 14\n    // num_start = 4\n    // key_length = 3\n    //\n    // some sanity checks. The format must be [key]-%10d\n    // check for the dash ('-')\n    assert(num_start > 0);\n    assert(keys[i][num_start - 1] == '-');\n    int version = atoi(keys[i].c_str() + num_start);\n    std::string keyname = keys[i].substr(0, key_length);\n    if (masterkeyset.count(keyname)) {\n      key_and_version[keyname] = std::max(version, key_and_version[keyname]);\n    }\n  }\n\n  // now scan against the actual data and compute a diff\n  // first search for deleted keys\n  // scan the data map against the key_and_version map\n  {\n    std::map::const_iterator iter = data.begin();\n    while (iter != data.end()) {\n      if (key_and_version.count(iter->first) == 0) {\n        out_deletedkeys.push_back(iter->first);\n      }\n      ++iter;\n    }\n  }\n  // now actually delete it from the data map\n  for (size_t i = 0;i < out_deletedkeys.size(); ++i) {\n    data.erase(data.find(out_deletedkeys[i]));\n  }\n\n  // ok. now loop through the key_and_version map and handle new and\n  // modified keys\n  {\n    std::map::const_iterator iter = key_and_version.begin();\n    while (iter != key_and_version.end()) {\n      std::map::iterator data_iter = data.find(iter->first);\n      if (data_iter == data.end()) {\n        // key not found. this is a new key\n        out_newkeys.push_back(iter->first);\n        data[iter->first].remote_version = iter->second;\n        data[iter->first].has_value = (data[iter->first].stored_version == data[iter->first].remote_version);\n      } else {\n        // key found. this is an existing key\n        // if the remote version changed, it was modified\n        if (data_iter->second.remote_version == -1) {\n          out_newkeys.push_back(iter->first);\n        }\n        else if (data_iter->second.remote_version < iter->second) {\n          out_modifiedkeys.push_back(iter->first);\n        }\n        // invalidate the local value\n        data_iter->second.remote_version = std::max(data_iter->second.remote_version, iter->second);\n        data[iter->first].has_value = (data[iter->first].stored_version == data[iter->first].remote_version);\n      }\n      ++iter;\n    }\n  }\n}\n\n// ------------- watch implementation ---------------\n\nint key_value::add_callback(callback_type fn) {\n  datalock.lock();\n  size_t cur_callback_id = next_callback_id;\n  callbacks[cur_callback_id] = fn;\n  ++next_callback_id;\n  datalock.unlock();\n  return cur_callback_id;\n}\n\nbool key_value::remove_callback(int fnid) {\n  bool ret = false;\n  datalock.lock();\n  std::map::iterator iter = callbacks.find(fnid);\n  if (iter != callbacks.end()) {\n    ret = true;\n    callbacks.erase(iter);\n  }\n  datalock.unlock();\n  return ret;\n}\n\nvoid key_value::watcher(zhandle_t *zh,\n                        int type,\n                        int state,\n                        const char *path,\n                        void *watcherCtx) {\n  key_value* slist = reinterpret_cast(watcherCtx);\n  if (type == ZOO_CHILD_EVENT) {\n    slist->datalock.lock();\n    if (!slist->closing) {\n      std::vector newkeys, deletedkeys, modifiedkeys;\n      bool ret = slist->get_all_keys_locked(newkeys, deletedkeys, modifiedkeys);\n      slist->datalock.unlock();\n      if (ret && !slist->callbacks.empty()) {\n        std::map::iterator iter = slist->callbacks.begin();\n        while (iter != slist->callbacks.end()) {\n          iter->second(slist, newkeys, deletedkeys, modifiedkeys);\n          ++iter;\n        }\n      }\n    } else {\n      slist->datalock.unlock();\n    }\n  }\n}\n\n\n\n\n} // namespace zookeeper\n} // namespace graphlab\n"
  },
  {
    "path": "src/graphlab/zookeeper/key_value.hpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef ZOOKEEPER_KEY_VALUE_HPP\n#define ZOOKEEPER_KEY_VALUE_HPP\n#include \n#include \n#include \n#include \n#include \n#include \nextern \"C\" {\n#include \n}\n\n\nnamespace graphlab{\nnamespace zookeeper {\n\n\n/**\n *  A simple zookeeper service to maintain a key value store\n *  The service provides the ability to watch for changes\n *  through the use of callbacks.\n *  Keys are \"owned\" by their creators, and duplicate keys\n *  are not permitted. Owners can change the values of their owned keys.\n *  Keys are destroyed when their owners die.\n *\n *  The natural implementation will be to create a node for each key\n *  and have the node contain the actual value of the key. However, it is very\n *  difficult to watch for changes here because if there are a large number of\n *  keys, we have to set a data watch on each key, and zookeeper does not like\n *  it if you make too many watches.\n *\n *  The solution:\n *  For each key\n *   - An ephemeral masters/[key]\" node is created.\n *     This is used to identify the machine currently owning the key, and make\n *     sure that there can only be one owner for each key.\n *   - A SEQUENCE EPHEMERAL node with the name \"values/[key]-%10d\" is created\n *     whenever the value of the key changes. The contents of the node\n *     are the contents of the key.\n *   - Now a single watch on the entire values directory is sufficient to\n *     identify any data changes.\n */\nclass key_value {\n public:\n\n  ///  Joins a zookeeper cluster.\n  ///  Zookeeper nodes will be created in the prefix \"prefix\".\n  key_value(std::vector zkhosts,\n            std::string prefix,\n            std::string serveridentifier);\n  /// destructor\n  ~key_value();\n\n  /** Inserts a value to the key value store. Returns true on success.\n   * False on failure (indicating the key already exists)\n   */\n  bool insert(const std::string& key, const std::string& value);\n\n  /** Modifies the value in the key value store. Returns true on success.\n   * False on failure. This instance must own the key (created the key)\n   * to modify its value.\n   */\n  bool modify(const std::string& key, const std::string& value);\n\n  /** Removed a key in the key value store. Returns true on success.\n   * False on failure. This instance must own the key (created the key)\n   * to delete the key.\n   */\n  bool erase(const std::string& key);\n\n\n  /// Gets a value of a key. First element of the pair is if the key was found\n  std::pair get(const std::string& key);\n\n\n  typedef boost::function& out_newkeys,\n                               const std::vector& out_deletedkeys,\n                               const std::vector& out_modifiedkeys)\n                          >  callback_type;\n\n  /** Adds a callback which will be triggered when any key/value\n   * changes. The callback arguments will be the key_value object,\n   * and the new complete key-value mapping.\n   * Calling this function will a NULL argument deletes\n   * the callback. Note that the callback may be triggered in a different thread.\n   *\n   * Returns the id of the callback. Calling remove_callback with the id\n   * disables the callback.\n   */\n  int add_callback(callback_type fn);\n\n\n  /** Removes a callback identified by an ID. Returns true on success,\n   * false on failure */\n  bool remove_callback(int fnid);\n private:\n  std::string serveridentifier;\n  std::string prefix;\n  std::string masters_path;\n  std::string values_path;\n  zhandle_t* handle;\n\n  recursive_mutex datalock;\n\n  std::map callbacks;\n  int next_callback_id;\n  bool closing;\n\n  // a list of all the values I created\n  std::set my_values;\n\n  struct lazy_value {\n    bool has_value;\n    int stored_version;\n    int remote_version;\n    std::string value;\n    lazy_value():has_value(false), stored_version(-1), remote_version(-1) {}\n    lazy_value(const lazy_value& lv):\n        has_value(lv.has_value),\n        stored_version(lv.stored_version),\n        remote_version(lv.remote_version),\n        value(lv.value) {}\n  };\n\n  std::map data;\n\n  bool get_all_keys_locked(std::vector& out_newkeys,\n                           std::vector& out_deletedkeys,\n                           std::vector& out_modifiedkeys);\n\n  void fill_data_locked(const std::vector& keys,\n                        const std::vector& masterkeys,\n                        std::vector& out_newkeys,\n                        std::vector& out_deletedkeys,\n                        std::vector& out_modifiedkeys);\n\n  static void watcher(zhandle_t *zh,\n                    int type,\n                    int state,\n                    const char *path,\n                    void *watcherCtx);\n\n\n};\n\n\n} // namespace zookeeper\n} // namespace graphlab\n#endif\n\n"
  },
  {
    "path": "src/graphlab/zookeeper/server_list.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n#include \n#include \nextern \"C\" {\n#include \n}\n\nnamespace graphlab{\nnamespace zookeeper {\n\nserver_list::server_list(std::vector zkhosts,\n                         std::string _prefix,\n                         std::string _serveridentifier) :\n    prefix(_prefix), serveridentifier(_serveridentifier), callback(NULL) {\n  // construct hosts list\n  std::string hosts = boost::algorithm::join(zkhosts, \",\");\n  prefix = normalize_path(prefix);\n  if (prefix[0] != '/') prefix = \"/\" + prefix;\n  handle = zookeeper_init(hosts.c_str(), watcher, 10000, NULL, (void*)this, 0);\n  // create the prefix if it does not already exist\n  if (prefix != \"/\") create_dir(handle,\n                                prefix.substr(0, prefix.length() - 1),\n                                \"zk_server_list\");\n\n  assert(handle != NULL);\n\n}\n\nserver_list::~server_list() {\n  if (handle != NULL) zookeeper_close(handle);\n}\n\n\n\n\nstd::vector server_list::get_all_servers(std::string name_space) {\n  boost::algorithm::trim(name_space); assert(name_space.length() > 0);\n  struct String_vector children;\n  children.count = 0;\n  children.data = NULL;\n\n  std::vector ret;\n\n  // effective path is prefix + name_space\n  std::string path = prefix + name_space;\n\n  int stat = zoo_get_children(handle, path.c_str(), 0, &children);\n  // if there are no children quit\n  if (stat == ZNONODE) return ret;\n  ret = String_vector_to_vector(&children);\n  free_String_vector(&children);\n  return ret;\n}\n\n/// Joins a namespace\nvoid server_list::join(std::string name_space) {\n  boost::algorithm::trim(name_space); assert(name_space.length() > 0);\n  create_dir(handle,\n             prefix + name_space,\n             \"zk_server_list\");\n  std::string path = normalize_path(prefix + name_space) + serveridentifier;\n  int stat = create_ephemeral_node(handle, path, \"\");\n  if (stat == ZNODEEXISTS) {\n    std::cerr << \"Server \" << serveridentifier << \" already exists!\" << std::endl;\n  }\n  if (stat != ZOK) assert(false);\n}\n\nvoid server_list::leave(std::string name_space) {\n  boost::algorithm::trim(name_space); assert(name_space.length() > 0);\n  std::string path = normalize_path(prefix + name_space) + serveridentifier;\n  delete_node(handle, path, \"zk_server_list leave\");\n  // also try to delete its parents if they become empty\n  delete_dir(handle, prefix + name_space, \"zk_server_list leave cleanup\");\n  if (prefix != \"/\") delete_dir(handle,\n                                prefix.substr(0, prefix.length() - 1),\n                                \"zk_server_list leave cleanup\");\n}\n\n\n// ------------- watch implementation ---------------\n\n\nstd::vector server_list::watch_changes(std::string name_space) {\n  boost::algorithm::trim(name_space); assert(name_space.length() > 0);\n  struct String_vector children;\n  children.count = 0;\n  children.data = NULL;\n  std::vector ret;\n\n  std::string path = prefix + name_space;\n  watchlock.lock();\n  if (watches.count(path)) {\n    watchlock.unlock();\n    return get_all_servers(name_space);\n  }\n  watches.insert(path);\n\n  int stat = zoo_get_children(handle, path.c_str(), 1, &children);\n  watchlock.unlock();\n  // if there are no children quit\n  if (stat == ZNONODE) return ret;\n  print_stat(stat, \"zk_server_list watch_changes\", path);\n  ret = String_vector_to_vector(&children);\n  free_String_vector(&children);\n  return ret;\n\n}\n\nvoid server_list::stop_watching(std::string name_space) {\n  boost::algorithm::trim(name_space); assert(name_space.length() > 0);\n  std::string path = prefix + name_space;\n  watchlock.lock();\n  watches.erase(path);\n  watchlock.unlock();\n}\n\nvoid server_list::set_callback(boost::function server)\n                                              > fn) {\n  watchlock.lock();\n  callback = fn;\n  watchlock.unlock();\n}\n\nvoid server_list::issue_callback(std::string path) {\n  watchlock.lock();\n  // search for the path in the watch set\n  bool found = watches.count(path);\n  if (found) {\n    struct String_vector children;\n    children.count = 0;\n    children.data = NULL;\n    std::vector ret;\n    // reissue the watch\n    int stat = zoo_get_children(handle, path.c_str(), 1, &children);\n    print_stat(stat, \"zk serverlist issue_callback\", path);\n    ret = String_vector_to_vector(&children);\n    free_String_vector(&children);\n\n    // if a callback is registered\n    if (callback != NULL) {\n      callback(this, path, ret);\n    }\n  }\n  watchlock.unlock();\n}\n\nvoid server_list::watcher(zhandle_t *zh,\n                          int type,\n                          int state,\n                          const char *path,\n                          void *watcherCtx) {\n  server_list* slist = reinterpret_cast(watcherCtx);\n  if (type == ZOO_CHILD_EVENT) {\n    std::string strpath = path;\n    slist->issue_callback(path);\n  }\n}\n\n\n\n\n} // namespace zookeeper\n} // namespace graphlab\n"
  },
  {
    "path": "src/graphlab/zookeeper/server_list.hpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_ZOOKEEPER_SERVER_LIST_HPP\n#define GRAPHLAB_ZOOKEEPER_SERVER_LIST_HPP\n\n#include \n#include \n#include \n#include \n#include \nextern \"C\" {\n#include \n}\nnamespace graphlab{\nnamespace zookeeper {\n\n/**\n *  A simple zookeeper service to maintain a list of servers.\n *  The service provides the ability to watch for new servers leaving\n *  and joining the service through the use of callbacks.\n */\nclass server_list {\n public:\n\n  ///  Joins a zookeeper cluster.\n  ///  Zookeeper nodes will be created in the prefix \"prefix\".\n  ///  The current machine will be identified as \"serveridentifier\"\n  server_list(std::vector zkhosts,\n              std::string prefix,\n              std::string serveridentifier);\n\n  /// destructor\n  ~server_list();\n\n  /// Joins a namespace\n  void join(std::string name_space);\n\n  /// Leaves a namespace. Note that if this machine crashes, or if the\n  /// server list is destroyed, it will automatically leave the namespace.\n  void leave(std::string name_space);\n\n  /// gets a list of all servers in a namespace\n  std::vector get_all_servers(std::string name_space);\n\n  /// Watches for changes to a namespace while returning the current contents\n  /// When changes occur, the callback is called.\n  std::vector watch_changes(std::string name_space);\n\n  /// Removes the watch callback.\n  void stop_watching(std::string name_space);\n\n\n  /** Adds a callback which will be triggered when any namespace in the prefix\n   * changes. The callback arguments will be the server_list object, the\n   * namespace which changed, and the new list of servers in the name space.\n   * Calling this function will a NULL argument deletes\n   * the callback. Note that the callback may be triggered in a different thread.\n   */\n  void set_callback(boost::function server)\n                                         > fn);\n\n private:\n  std::string prefix, serveridentifier;\n  zhandle_t* handle;\n\n  recursive_mutex watchlock;\n  std::set watches;\n\n  boost::function)> callback;\n\n  void issue_callback(std::string path);\n\n  static void watcher(zhandle_t *zh,\n                    int type,\n                    int state,\n                    const char *path,\n                    void *watcherCtx);\n\n\n};\n\n} // namespace zookeeper\n} // namespace graphlab\n#endif\n"
  },
  {
    "path": "src/graphlab/zookeeper/zookeeper_common.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \nextern \"C\" {\n#include \n}\n\n\n\nnamespace graphlab{\nnamespace zookeeper {\n\n\n// frees a zookeeper String_vector\nvoid free_String_vector(struct String_vector* strings) {\n  if (strings->data) {\n    for (size_t i = 0;i < (size_t)(strings->count); ++i) {\n      free(strings->data[i]);\n    }\n    free(strings->data);\n    strings->data = NULL;\n    strings->count = 0;\n  }\n}\n\n// convert a zookeeper String_vector to a c++ vector\nstd::vector String_vector_to_vector(\n    const struct String_vector* strings) {\n  std::vector ret;\n  for (size_t i = 0;i < (size_t)(strings->count); ++i) {\n    ret.push_back(strings->data[i]);\n  }\n  return ret;\n}\n\n// print a few zookeeper error status\nvoid print_stat(int stat,\n                const std::string& prefix,\n                const std::string& path) {\n  if (stat == ZNONODE) {\n    std::cerr << prefix << \": Node missing\" << path << std::endl;\n  }\n  else if (stat == ZNOAUTH) {\n    std::cerr << prefix << \": No permission to list children of node \"\n              << path << std::endl;\n  }\n  else if (stat == ZNODEEXISTS) {\n    std::cerr << prefix << \": Node \" << path << \" already exists.\" << std::endl;\n  }\n  else if (stat == ZNOTEMPTY) {\n    std::cerr << prefix << \": Node \" << path << \" not empty.\" << std::endl;\n  }\n  else if (stat != ZOK) {\n    std::cerr << prefix << \": Unexpected error \" << stat\n              << \" on path \" << path << std::endl;\n  }\n}\n\n// adds a trailing / to the path name if there is not one already\nstd::string normalize_path(std::string prefix) {\n  boost::algorithm::trim(prefix);\n  if (prefix.length() == 0) return \"/\";\n  else if (prefix[prefix.length() - 1] != '/') return prefix + \"/\";\n  else return prefix;\n}\n\nint create_dir(zhandle_t* handle, const std::string& name,\n               const std::string& stat_message) {\n  int stat = zoo_create(handle, name.c_str(), NULL, -1,\n                       &ZOO_OPEN_ACL_UNSAFE, 0, NULL, 0);\n  // we are ok with ZNODEEXISTS\n  // if (stat == ZOK || stat == ZNODEEXISTS) return stat;\n  if (stat != ZOK) print_stat(stat, stat_message + \" create_dir\", name);\n  return stat;\n}\n\nint delete_dir(zhandle_t* handle, const std::string& name,\n               const std::string& stat_message) {\n  int stat = zoo_delete(handle, name.c_str(), -1);\n  // we are ok if the node is not empty in which case\n  // there are still machines in the name space\n  // if (stat == ZOK || stat == ZNOTEMPTY) return stat;\n  if (stat != ZOK) print_stat(stat, stat_message + \" delete_dir\", name);\n  return stat;\n}\n\nint create_ephemeral_node(zhandle_t* handle,\n                 const std::string& path,\n                 const std::string& value,\n                 const std::string& stat_message) {\n  int stat = zoo_create(handle, path.c_str(), value.c_str(), value.length(),\n                        &ZOO_OPEN_ACL_UNSAFE, ZOO_EPHEMERAL, NULL, 0);\n  // if (stat == ZOK) return stat;\n  if (stat != ZOK) print_stat(stat, stat_message + \" create_ephemeral_node\", path);\n  return stat;\n}\n\nint delete_node(zhandle_t* handle,\n                           const std::string& path,\n                           const std::string& stat_message) {\n  int stat = zoo_delete(handle, path.c_str(), -1);\n  //  if (stat == ZOK) return stat;\n  if (stat != ZOK) print_stat(stat, stat_message + \" delete_node\", path);\n  return stat;\n}\n\n\nstd::string get_sequence_node_path(const std::string& path,\n                                   const int version) {\n  char versionstring[16];\n  sprintf(versionstring, \"%010d\", version);\n  std::string actualpath = path + versionstring;\n  return actualpath;\n}\n\nint delete_sequence_node(zhandle_t* handle,\n                         const std::string& path,\n                         const int version,\n                         const std::string& stat_message) {\n  std::string actualpath = get_sequence_node_path(path, version);\n  int stat = zoo_delete(handle, actualpath.c_str(), -1);\n  //  if (stat == ZOK) return stat;\n  if (stat != ZOK) print_stat(stat, stat_message + \" delete_sequence_node\", actualpath);\n  return stat;\n}\n\nstd::pair create_ephemeral_sequence_node(zhandle_t* handle,\n                                                  const std::string& path,\n                                                  const std::string& value,\n                                                  const std::string& stat_message) {\n  // make sure we always have enough room for the version number\n  assert(path.length() + 10 < 1024);\n  char retpathbuffer[1024];\n  int stat = zoo_create(handle, path.c_str(), value.c_str(), value.length(),\n                        &ZOO_OPEN_ACL_UNSAFE, ZOO_EPHEMERAL | ZOO_SEQUENCE,\n                        retpathbuffer, 1024);\n  // if (stat == ZOK) return stat;\n  if (stat != ZOK) print_stat(stat, stat_message + \" create_ephemeral_sequence_node\", path);\n  int retlen = strlen(retpathbuffer);\n  assert(retlen > 10);\n  int version = atoi(retpathbuffer + (retlen - 10));\n  return std::pair(stat, version);\n}\n\n\nstd::pair get_node_value(zhandle_t* handle,\n                                            const std::string& node,\n                                            const std::string& stat_message) {\n  char buffer[1024];\n  int length = 1024;\n  int stat = zoo_get(handle, node.c_str(), 0, buffer, &length, NULL);\n  if (stat != ZOK) print_stat(stat, stat_message + \" get_node_value\", node);\n  if (stat != ZOK) return std::pair(false, \"\");\n\n  // we are good here\n  if (length <= 1024) {\n    // ok. it fit inside the buffer\n    // we can return\n    if (length < 0) return std::pair(true, \"\");\n    else return std::pair(true, std::string(buffer, length));\n  }\n  else {\n    while(1) {\n      // buffer not long enough. The length parameter constains the actual length\n      // try again. keep looping until we succeed\n      char* newbuffer = new char[length];\n      int stat = zoo_get(handle, node.c_str(), 0, newbuffer, &length, NULL);\n      if (stat != ZOK) print_stat(stat, stat_message + \" get_node_value\", node);\n      std::string retval(newbuffer, length);\n      delete newbuffer;\n\n      if (stat != ZOK) print_stat(stat, stat_message + \" get_node_value\", node);\n      if (stat != ZOK) return std::pair(false, \"\");\n      if (length < 0) return std::pair(true, \"\");\n      else return std::pair(true, retval);\n    }\n  }\n}\n\n\n} // graphlab\n} // zookeeper\n\n"
  },
  {
    "path": "src/graphlab/zookeeper/zookeeper_common.hpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#ifndef GRAPHLAB_ZOOKEEPER_COMMON_HPP\n#define GRAPHLAB_ZOOKEEPER_COMMON_HPP\n#include \n#include \n\nextern \"C\" {\n#include \n}\n\n\nnamespace graphlab{\nnamespace zookeeper {\n\n/// frees a zookeeper String_vector\nvoid free_String_vector(struct String_vector* strings);\n\n/// convert a zookeeper String_vector to a c++ vector\nstd::vector String_vector_to_vector(\n    const struct String_vector* strings);\n\n/// print a few zookeeper error status\nvoid print_stat(int stat,\n                const std::string& prefix,\n                const std::string& path);\n\n/// adds a trailing / to the path name if there is not one already\nstd::string normalize_path(std::string prefix);\n\n/// Creates a zookeeper directory\nint create_dir(zhandle_t* handle, const std::string& path,\n               const std::string& stat_message = \"\");\n\n/// Deletes a zookeeper directory\nint delete_dir(zhandle_t* handle, const std::string& path,\n               const std::string& stat_message = \"\");\n\n/// Creates a zookeeper ephemeral node\nint create_ephemeral_node(zhandle_t* handle,\n                 const std::string& path,\n                 const std::string& value,\n                 const std::string& stat_message = \"\");\n\n/// Deletes a zookeeper ephemeral node\nint delete_node(zhandle_t* handle,\n                const std::string& path,\n                const std::string& stat_message = \"\");\n\n/// Deletes a zookeeper sequence node\nint delete_sequence_node(zhandle_t* handle,\n                         const std::string& path,\n                         const int version,\n                         const std::string& stat_message = \"\");\n\n/// Gets the effective node name for a sequence node of a particular sequence number\nstd::string get_sequence_node_path(const std::string& path,\n                                   const int version);\n\n\n/// Creates a zookeeper ephemeral sequence nodea\n/// Returns a pair of (status, version)\nstd::pair create_ephemeral_sequence_node(zhandle_t* handle,\n                                                   const std::string& path,\n                                                   const std::string& value,\n                                                   const std::string& stat_message = \"\");\n\n/// Gets the value in a node. output is a pair of (success, value)\nstd::pair get_node_value(zhandle_t* handle,\n                                            const std::string& node,\n                                            const std::string& stat_message = \"\");\n\n\n} // graphlab\n} // zookeeper\n\n#endif\n"
  },
  {
    "path": "src/graphlab.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n \n\n#ifndef GRAPHLAB_MASTER_INCLUDES\n#define GRAPHLAB_MASTER_INCLUDES\n\n\n\n\n// #include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n\n#endif\n\n"
  },
  {
    "path": "tests/CMakeLists.txt",
    "content": "project(tests)\nsubdirs(data)\n\nADD_CXXTEST(random_test.cxx)\n\n# move into toolkit\n#ADD_CXXTEST(factor_test.cxx)\nADD_CXXTEST(small_map_test.cxx)\nADD_CXXTEST(small_set_test.cxx)\n\nADD_CXXTEST(dense_bitset_test.cxx)\nADD_CXXTEST(serializetests.cxx)\nADD_CXXTEST(thread_tools.cxx)\n\nADD_CXXTEST(test_lock_free_pool.cxx)\nADD_CXXTEST(lock_free_pushback.cxx)\nADD_CXXTEST(union_find_test.cxx)\n\nADD_CXXTEST(empty_test.cxx)\n# ADD_CXXTEST(scheduler_test.cxx)\n\nADD_CXXTEST(csr_storage_test.cxx)\nADD_CXXTEST(local_graph_test.cxx)\nadd_graphlab_executable(distributed_graph_test distributed_graph_test.cpp)\nadd_graphlab_executable(distributed_ingress_test distributed_ingress_test.cpp)\n\nadd_graphlab_executable(cuckootest cuckootest.cpp)\nadd_graphlab_executable(dc_consensus_test dc_consensus_test.cpp)\nadd_graphlab_executable(distributed_chandy_misra_test distributed_chandy_misra_test.cpp)\nadd_graphlab_executable(dc_fiber_consensus_test dc_fiber_consensus_test.cpp)\nadd_graphlab_executable(dc_test_sequentialization dc_test_sequentialization.cpp)\nadd_graphlab_executable(hdfs_test hdfs_test.cpp)\nadd_graphlab_executable(test_parsers test_parsers.cpp)\n\nadd_graphlab_executable(synchronous_engine_test synchronous_engine_test.cpp)\nadd_graphlab_executable(async_consistent_test async_consistent_test.cpp)\n\nadd_graphlab_executable(sfinae_function_test sfinae_function_test.cpp)\n\nadd_test(synchronous_engine_test synchronous_engine_test)\nadd_test(async_consistent_test async_consistent_test)\n\n# copyfile(runtests.sh)\n\nadd_graphlab_executable(mini_web_server mini_web_server.cpp)\n\nadd_graphlab_executable(test_vertex_set test_vertex_set.cpp)\n\nadd_test(test_vertex_set test_vertex_set)\nadd_graphlab_executable(arbitrary_signal_test arbitrary_signal_test.cpp)\n\n\nadd_graphlab_executable(sort_test sort_test.cpp)\n\nadd_graphlab_executable(hopscotch_test hopscotch_test.cpp)\n\nadd_graphlab_executable(fiber_test fiber_test.cpp)\nadd_graphlab_executable(fibo_fiber_test fibo_fiber_test.cpp)\n"
  },
  {
    "path": "tests/arbitrary_signal_test.cpp",
    "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \n#include \n#include \n\n\n// #include \n\n#include \n\ntypedef graphlab::distributed_graph graph_type;\n\n\nclass test_uf:\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\npublic:\n  edge_dir_type\n  gather_edges(icontext_type& context, \n               const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& total) {\n    if (vertex.id() < 99) context.signal_vid(vertex.id() + 1);\n  }\n  edge_dir_type scatter_edges(icontext_type& context, \n                              const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n}; // end of count neighbors\n\n\ntypedef graphlab::async_consistent_engine agg_engine_type;\n//typedef graphlab::synchronous_engine agg_engine_type;\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_WARNING);\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::dc_init_param rpc_parameters;\n  graphlab::init_param_from_mpi(rpc_parameters);\n  graphlab::distributed_control dc(rpc_parameters);\n\n  graphlab::command_line_options clopts(\"Test code.\");\n  clopts.set_scheduler_type(\"queued_fifo\");\n  std::cout << \"Creating a powerlaw graph\" << std::endl;\n  graph_type graph(dc, clopts);\n  graph.load_synthetic_powerlaw(100);\n\n\n  typedef agg_engine_type engine_type;\n  engine_type engine(dc, graph, clopts);\n  engine.signal(0);\n  engine.start();\n\n  ASSERT_EQ(engine.num_updates(), 100);\n  graphlab::mpi_tools::finalize();\n} // end of main\n\n\n\n\n\n"
  },
  {
    "path": "tests/async_consistent_test.cpp",
    "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \n#include \n#include \n\n\n// #include \n\n#include \n\ntypedef graphlab::distributed_graph graph_type;\n\n\nclass count_in_neighbors :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\npublic:\n  edge_dir_type\n  gather_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::IN_EDGES;\n  }\n  gather_type\n  gather(icontext_type& context, const vertex_type& vertex,\n         edge_type& edge) const {\n    return 1;\n  }\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& total) {\n    ASSERT_EQ( total, int(vertex.num_in_edges()) );\n  }\n  edge_dir_type\n  scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n}; // end of count neighbors\n\n\nvoid test_in_neighbors(graphlab::distributed_control& dc,\n                       graphlab::command_line_options& clopts,\n                       graph_type& graph) {\n  std::cout << \"Constructing an engine for in neighbors\" << std::endl;\n  typedef graphlab::async_consistent_engine engine_type;\n  engine_type engine(dc, graph, clopts);\n  std::cout << \"Scheduling all vertices to count their neighbors\" << std::endl;\n  engine.signal_all();\n  std::cout << \"Running!\" << std::endl;\n  engine.start();\n  std::cout << \"Finished\" << std::endl;\n}\n\n\nclass count_out_neighbors :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\npublic:\n  edge_dir_type\n  gather_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::OUT_EDGES;\n  }\n  gather_type\n  gather(icontext_type& context, const vertex_type& vertex,\n         edge_type& edge) const {\n    return 1;\n  }\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& total) {\n    ASSERT_EQ( total, int(vertex.num_out_edges()) );\n  }\n  edge_dir_type\n  scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n}; // end of count neighbors\n\nvoid test_out_neighbors(graphlab::distributed_control& dc,\n                        graphlab::command_line_options& clopts,\n                        graph_type& graph) {\n  std::cout << \"Constructing an engine for out neighbors\" << std::endl;\n  typedef graphlab::async_consistent_engine engine_type;\n  engine_type engine(dc, graph, clopts);\n  std::cout << \"Scheduling all vertices to count their neighbors\" << std::endl;\n  engine.signal_all();\n  std::cout << \"Running!\" << std::endl;\n  engine.start();\n  std::cout << \"Finished\" << std::endl;\n}\n\n\nclass count_all_neighbors :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\npublic:\n  void init(icontext_type& context, const vertex_type& vertex,\n                    const message_type& msg) {\n    ASSERT_EQ(msg, 100);\n  }\n  \n  edge_dir_type\n  gather_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  }\n  gather_type\n  gather(icontext_type& context, const vertex_type& vertex,\n         edge_type& edge) const {\n    return 1;\n  }\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& total) {\n    ASSERT_EQ( total, int(vertex.num_in_edges() + vertex.num_out_edges() ) );\n  }\n  edge_dir_type\n  scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n}; // end of count neighbors\n\nvoid test_all_neighbors(graphlab::distributed_control& dc,\n                        graphlab::command_line_options& clopts,\n                        graph_type& graph) {\n  std::cout << \"Constructing an engine for all neighbors\" << std::endl;\n  typedef graphlab::async_consistent_engine engine_type;\n  engine_type engine(dc, graph, clopts);\n  std::cout << \"Scheduling all vertices to count their neighbors\" << std::endl;\n  engine.signal_all(100);\n  std::cout << \"Running!\" << std::endl;\n  engine.start();\n  std::cout << \"Finished\" << std::endl;\n}\n\n\n\n\n\n\n\n\n\n\n\n// Make a slow version so that the asynchronous aggregators get a change\n// to run. Basically, sleep a bit on apply.\nclass count_all_neighbors_slow :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\npublic:\n  void init(icontext_type& context, const vertex_type& vertex,\n                    const message_type& msg) {\n    ASSERT_EQ(msg, 100);\n  }\n\n  edge_dir_type\n  gather_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  }\n  gather_type\n  gather(icontext_type& context, const vertex_type& vertex,\n         edge_type& edge) const {\n    return 1;\n  }\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& total) {\n    graphlab::timer::sleep_ms(100);\n    ASSERT_EQ( total, int(vertex.num_in_edges() + vertex.num_out_edges() ) );\n  }\n  edge_dir_type\n  scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n}; // end of count neighbors\n\n\n\n\n\n\ntypedef graphlab::async_consistent_engine agg_engine_type;\n\nsize_t agg_map(agg_engine_type::icontext_type& context,\n              const agg_engine_type::vertex_type& vtx) {\n  return 1;\n}\n\nvoid agg_finalize(agg_engine_type::icontext_type& context,\n                  size_t result) {\n  std::cout << \"Aggregator: #vertices = \" << result << std::endl;\n}\n\n\nsize_t agg_edge_map(agg_engine_type::icontext_type& context,\n              const agg_engine_type::edge_type& vtx) {\n  return 1;\n}\n\nvoid agg_edge_finalize(agg_engine_type::icontext_type& context,\n                  size_t result) {\n  std::cout << \"Aggregator: #edges= \" << result << std::endl;\n}\n\n\n\nsize_t identity_vertex_map(agg_engine_type::vertex_type vtx) {\n  return vtx.data();\n}\nsize_t identity_edge_map(agg_engine_type::edge_type e) {\n  return e.data();\n}\n\n\nsize_t identity_vertex_map_context(agg_engine_type::icontext_type& context,\n                                   agg_engine_type::vertex_type vtx) {\n  return vtx.data();\n}\nsize_t identity_edge_map_context(agg_engine_type::icontext_type& context,\n                                 agg_engine_type::edge_type e) {\n  return e.data();\n}\n\n\n\nvoid  set_vertex_to_one(agg_engine_type::vertex_type vtx) {\n  vtx.data() = 1;\n}\nvoid  set_edge_to_one(agg_engine_type::edge_type e) {\n  e.data() = 1;\n}\n\n\nvoid  vertex_plus_one(agg_engine_type::vertex_type vtx) {\n  ++vtx.data();\n}\n\n\nvoid vertex_minus_one_context(agg_engine_type::icontext_type& context,\n                                agg_engine_type::vertex_type vtx) {\n  --vtx.data();\n}\n\nvoid edge_plus_one(agg_engine_type::edge_type e) {\n  ++e.data();\n}\n\n\nvoid edge_minus_one_context(agg_engine_type::icontext_type& context,\n                              agg_engine_type::edge_type e) {\n  --e.data();\n}\n\nvoid test_aggregator(graphlab::distributed_control& dc,\n                     graphlab::command_line_options& clopts,\n                     graph_type& graph) {\n  std::cout << \"Constructing an engine for all neighbors\" << std::endl;\n  agg_engine_type engine(dc, graph, clopts);\n  engine.add_vertex_aggregator(\"num_vertices_counter\", agg_map, agg_finalize);\n  engine.add_edge_aggregator(\"num_edges_counter\", agg_edge_map, agg_edge_finalize);\n  // reset all\n  graph.transform_vertices(set_vertex_to_one);\n  graph.transform_edges(set_edge_to_one);\n  \n  ASSERT_EQ(graph.map_reduce_vertices(identity_vertex_map), graph.num_vertices());\n  graph.transform_vertices(vertex_plus_one);\n  ASSERT_EQ(graph.map_reduce_vertices(identity_vertex_map), 2 * graph.num_vertices());\n  engine.transform_vertices(vertex_minus_one_context);\n  ASSERT_EQ(graph.map_reduce_vertices(identity_vertex_map), graph.num_vertices());\n  ASSERT_EQ(engine.map_reduce_vertices(identity_vertex_map_context), graph.num_vertices());\n  \n  ASSERT_EQ(graph.map_reduce_edges(identity_edge_map), graph.num_edges());\n  graph.transform_edges(edge_plus_one);\n  ASSERT_EQ(graph.map_reduce_edges(identity_edge_map), 2 * graph.num_edges());\n  engine.transform_edges(edge_minus_one_context);\n  ASSERT_EQ(graph.map_reduce_edges(identity_edge_map), graph.num_edges());\n  ASSERT_EQ(engine.map_reduce_edges(identity_edge_map_context), graph.num_edges());\n  \n  ASSERT_TRUE(engine.aggregate_now(\"num_vertices_counter\"));\n  ASSERT_TRUE(engine.aggregate_now(\"num_edges_counter\"));\n  ASSERT_TRUE(engine.aggregate_periodic(\"num_vertices_counter\", 0.2));\n  ASSERT_TRUE(engine.aggregate_periodic(\"num_edges_counter\", 0.2));\n  std::cout << \"Scheduling all vertices to count their neighbors\" << std::endl;\n  engine.signal_all(100);\n  std::cout << \"Running!\" << std::endl;\n  engine.start();\n  std::cout << \"Finished\" << std::endl;\n}\n\n\n\n\n\n\n\n\n\nint main(int argc, char** argv) {\n\n  global_logger().set_log_level(LOG_INFO);\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::dc_init_param rpc_parameters;\n  graphlab::init_param_from_mpi(rpc_parameters);\n  graphlab::distributed_control dc(rpc_parameters);\n\n  graphlab::command_line_options clopts(\"Test code.\");\n  clopts.set_scheduler_type(\"queued_fifo\");\n  std::cout << \"Creating a powerlaw graph\" << std::endl;\n  graph_type graph(dc, clopts);\n  graph.load_synthetic_powerlaw(100);\n\n  test_in_neighbors(dc, clopts, graph);\n  test_out_neighbors(dc, clopts, graph);\n  test_all_neighbors(dc, clopts, graph);\n  test_aggregator(dc, clopts, graph);\n  graphlab::mpi_tools::finalize();\n} // end of main\n\n\n\n\n\n"
  },
  {
    "path": "tests/chandy_misra.cxx",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n#include \nusing namespace graphlab;\n\ntypedef graph graph_type;\n\n\n\nclass ChandyMisraTest: public CxxTest::TestSuite {\n public:\n  void test_cm() {\n    graph_type g;\n    for (size_t i = 0;i < 25; ++i) g.add_vertex(0);\n    for (size_t i = 0;i < 25; ++i) {\n      for (size_t j = 0;j < 25; ++j) {\n        if ((i != j) && (rand() % 1000 <= 100)) {\n          ASSERT_NE(i, j);\n          g.add_edge(i, j, 0);\n        }\n      }\n    }\n    g.finalize();\n    chandy_misra cm(g);\n    for (size_t i = 0;i < 100; ++i) {\n      TS_ASSERT_EQUALS(cm.make_philosopher_hungry(i % 25), i % 25);\n      std::vector r = cm.philosopher_stops_eating(i % 25);\n      TS_ASSERT_EQUALS(r.size(), size_t(0));\n    }\n    // test more aggressive\n    for (size_t k = 0;k < 10; ++k) {\n      dense_bitset locked, ready, complete;\n      locked.resize(25); ready.resize(25);\n      locked.clear(); ready.clear();\n      complete.resize(25); complete.clear();\n\n      for (size_t i = 0;i < 25; ++i) {\n        locked.set_bit(i);\n        vertex_id_type ret = cm.make_philosopher_hungry(i);\n        if (ret != (vertex_id_type)(-1)) {\n          complete.set_bit(ret);\n          ready.set_bit(ret);\n        }\n      }\n      cm.complete_consistency_check();\n      \n      while(1) {\n        if (ready.popcount() == 0 && complete.popcount() == g.num_vertices()) break;\n        foreach(size_t i, ready) {\n          ready.clear_bit(i);\n          std::vector r = cm.philosopher_stops_eating(i);\n          cm.complete_consistency_check();\n          \n          foreach(vertex_id_type j, r) {\n            TS_ASSERT(locked.get(j));\n            complete.set_bit(j);\n            ready.set_bit(j);\n          }\n        }\n      }\n      cm.no_locks_consistency_check();\n      cm.complete_consistency_check();\n    }\n    cm.no_locks_consistency_check();\n    cm.complete_consistency_check();\n    {\n      // test very aggressive\n      std::vector ctr(25, 10);\n      size_t n = 25 * 10;\n      std::queue ready;\n      for (size_t i = 0;i < 25; ++i) {\n        vertex_id_type ret = cm.make_philosopher_hungry(i);\n        if (ret != vertex_id_type(-1)) ready.push(ret);\n      }\n      while(!ready.empty()) {\n        size_t i = ready.front(); ready.pop();\n        TS_ASSERT(ctr[i] > 0);\n        ctr[i]--;\n        n--;\n        std::vector r = cm.philosopher_stops_eating(i);\n        foreach(vertex_id_type v, r) ready.push(v);\n        if (ctr[i] > 0) {\n          vertex_id_type ret = cm.make_philosopher_hungry(i);\n          if (ret != vertex_id_type(-1)) ready.push(ret);\n        }\n        cm.complete_consistency_check();\n      }\n      TS_ASSERT_EQUALS(n, size_t(0));\n    }\n  }\n\n  void test_parallel() {\n    \n  }\n};\n"
  },
  {
    "path": "tests/csr_storage_test.cxx",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n#include \n#include \n\n#include \n#include \n#include \n#include \n\nclass csr_storage_test : public CxxTest::TestSuite {  \n public:\n  typedef int valuetype;\n  typedef size_t keytype;\n  typedef size_t sizetype;\n\n  typedef graphlab::csr_storage csr_storage;\n  typedef graphlab::dynamic_csr_storage dcsr2_t;\n  typedef graphlab::dynamic_csr_storage dcsr4_t;\n  typedef graphlab::dynamic_csr_storage dcsr8_t;\n  typedef graphlab::dynamic_csr_storage dcsr16_t;\n  typedef graphlab::dynamic_csr_storage dcsr64_t;\n  \n public:\n  csr_storage_test() {\n    keytype keyin_arr[] = {1, 3, 6, 9, 5, 2};\n    valuetype valin_arr[] = {3, 2, 1, 4, 4, 4};\n\n    _keyin.assign(keyin_arr, keyin_arr + sizeof(keyin_arr) / sizeof(keytype));\n    _valin.assign(valin_arr, valin_arr + sizeof(valin_arr) / sizeof(valuetype));\n\n    keytype keyout_arr[] = {1, 2, 3, 5, 6, 9};\n    valuetype valout_arr[] = {3, 4, 2, 4, 1, 4};\n\n    _keyout.assign(keyout_arr, keyout_arr + sizeof(keyout_arr) / sizeof(keytype));\n    _valout.assign(valout_arr, valout_arr + sizeof(valout_arr) / sizeof(valuetype));\n  }\n\n  void test_csr_storage() {\n    std::cout << \"Test csr_storage constructor\" << std::endl;\n    csr_storage csr(get_keyin(), get_valin());\n    check(csr, get_keyout(), get_valout());\n    csr.print(std::cout);\n    printf(\"+ Pass test: csr_storage constructor :)\\n\\n\");\n  }\n\n  void test_csr_storage2() {\n    std::cout << \"Test csr_storage wrap \" << std::endl;\n    std::vector keys(get_keyin());\n    std::vector values(get_valin());\n\n    std::vector permute_index;\n    std::vector prefix;\n\n    graphlab::counting_sort(keys, permute_index, &prefix);\n    graphlab::outofplace_shuffle(values, permute_index);\n\n    csr_storage csr;\n    csr.wrap(prefix, values);\n    check(csr, get_keyout(), get_valout());\n    printf(\"+ Pass test: csr_storage wrap :)\\n\\n\");\n  }\n\n  template\n  void dynamic_csr_storage_constructor_test() {\n    std::cout << \"Test dynamic csr_storage constructor\" << std::endl;\n    csr_type csr(get_keyin(), get_valin());\n    check(csr, get_keyout(), get_valout());\n    printf(\"+ Pass test: dynamic_csr_storage constructor :)\\n\\n\");\n\n    std::cout << \"Test dynamic csr_storage wrap\" << std::endl;\n\n    csr.clear();\n    std::vector keys(get_keyin());\n    std::vector values(get_valin());\n    std::vector permute_index;\n    std::vector prefix;\n    \n    graphlab::counting_sort(keys, permute_index, &prefix);\n    graphlab::outofplace_shuffle(values, permute_index);\n\n    csr.wrap(prefix, values);\n    check(csr, get_keyout(), get_valout());\n    printf(\"+ Pass test: dynamic_csr_storage wrap:)\\n\\n\");\n  }\n\n  template\n  void dynamic_csr_storage_insertion_test() {\n    std::cout << \"Test dynamic csr_storage insertion\" << std::endl;\n    std::vector keys(get_keyin());\n    std::vector values(get_valin());\n\n    csr_type csr;\n    for (size_t i = 0; i < keys.size(); ++i) {\n      csr.insert(keys[i], values[i]);\n    }\n    csr.get_values().print(std::cerr);\n    check(csr, get_keyout(), get_valout());\n    csr.repack();\n    check(csr, get_keyout(), get_valout());\n    printf(\"+ Pass test: dynamic_csr_storage insertion:)\\n\\n\");\n  }\n\n  void test_dynamic_csr_storage_constructor() {\n      dynamic_csr_storage_constructor_test();\n      dynamic_csr_storage_constructor_test();\n      dynamic_csr_storage_constructor_test();\n      dynamic_csr_storage_constructor_test();\n  }\n \n  void test_dynamic_csr_storage_insertion() {\n      dynamic_csr_storage_insertion_test();\n      dynamic_csr_storage_insertion_test();\n      dynamic_csr_storage_insertion_test();\n      dynamic_csr_storage_insertion_test();\n  }\n \n  template\n  void dynamic_csr_storage_range_insertion_test(size_t nkey, size_t nval) {\n    std::cout << \"Test dynamic csr_storage range insertion\" << std::endl;\n    csr_type csr;\n    for (size_t i = 0; i < nkey; ++i) {\n      std::vector vals(nval, i);\n      csr.insert(i, vals.begin(), vals.end());\n    }\n    // csr.print(std::cout);\n    check_dcsr(csr, nkey, nval);\n\n    csr.clear();\n    ASSERT_EQ(csr.num_keys(), 0);\n    ASSERT_EQ(csr.num_values(), 0);\n    ASSERT_EQ(csr.get_values().num_blocks(), 0);\n\n    for (int i = nkey-1; i >= 0; --i) {\n      std::vector vals(nval, i);\n      csr.insert((keytype)i, vals.begin(), vals.end());\n    }\n    csr.get_values().print(std::cout);\n    check_dcsr(csr, nkey, nval);\n    std::cout << \"test repack...\" << std::endl;\n    csr.repack();\n    check_dcsr(csr, nkey, nval);\n    printf(\"+ Pass test: dynamic_csr_storage range insertion:)\\n\\n\");\n  }\n\n  void test_dynamic_csr_storage_range_insertion() {\n      dynamic_csr_storage_range_insertion_test(4, 4);\n      dynamic_csr_storage_range_insertion_test(6, 9);\n      dynamic_csr_storage_range_insertion_test(8, 3);\n      dynamic_csr_storage_range_insertion_test(20, 64);\n  }\n\n  void test_dynamic_csr_storage_stress_insertion() {\n      stress_insertion_test(4, 4);\n      stress_insertion_test(6, 9);\n      stress_insertion_test(8, 3);\n      stress_insertion_test(982, 294);\n  }\n\n\n\n  template\n  void stress_insertion_test(size_t nkey, size_t nval) {\n    std::cout << \"Test dynamic csr_storage stess insertion\" << std::endl;\n    // stress test single insertion\n    csr_type csr;\n\n    for (size_t j = 0; j < nval; ++j) {\n      for (size_t i = 0; i < nkey; i+=2) {\n        csr.insert(i, i);\n      }\n    }\n    for (size_t j = 0; j < nval; ++j) {\n      for (int i = nkey-1; i >= 0; i-=2) {\n        csr.insert((keytype)i, i);\n      }\n    }\n    check_dcsr(csr, nkey, nval);\n\n    csr.clear();\n    // stress test range insertion\n    for (size_t i = 0; i < nkey; i+=2) {\n      std::vector values(nval, i);\n      csr.insert(i, values.begin(), values.end());\n    }\n    for (int i = nkey-1; i>=0; i-=2) {\n      std::vector values(nval, i);\n      csr.insert((keytype)i, values.begin(), values.end());\n    }\n    check_dcsr(csr, nkey, nval);\n    printf(\"+ Pass test: dynamic_csr_storage stress insertion:)\\n\\n\");\n  }\n\n private:\n  template\n      void check(csr_type& csr,\n                 std::vector keyout,\n                 std::vector valout) {\n        typedef typename csr_type::iterator iterator;\n        size_t id = 0;\n        for (size_t i = 0; i < csr.num_keys(); ++i) {\n          iterator iter = csr.begin(i);\n          while (iter != csr.end(i)) {\n            ASSERT_EQ(i, keyout[id]);\n            ASSERT_EQ(*iter, valout[id]); \n            ++iter;\n            ++id;\n          }\n        }\n      }\n  template\n      void check_dcsr(csr_type& csr,\n                      size_t nkey,\n                      size_t nval) {\n    ASSERT_EQ(csr.num_keys(), nkey);\n    ASSERT_EQ(csr.num_values(), nkey*nval);\n    for (size_t i = 0; i < csr.num_keys(); ++i) {\n      typename csr_type::iterator iter = csr.begin(i);\n      size_t size = 0;\n      while(iter != csr.end(i)) {\n        ASSERT_EQ(*iter, (valuetype)i);\n        ++iter;\n        ++size;\n      }\n      ASSERT_EQ(size, nval);\n    }\n    csr.meminfo(std::cout);\n  }\n\n\n  std::vector get_keyin() { return std::vector(_keyin); }\n\n  std::vector get_valin() { return std::vector(_valin); }\n  \n  std::vector get_keyout() { return std::vector(_keyout); }\n\n  std::vector get_valout() { return std::vector(_valout); }\n\n  std::vector _keyin;\n  std::vector _keyout;\n  std::vector _valin;\n  std::vector _valout;\n}; // end of test\n"
  },
  {
    "path": "tests/cuckootest.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\nvoid sanity_checks() {\n  boost::unordered_map um;\n  graphlab::cuckoo_map_pow2 cm(-1);\n  ASSERT_TRUE(cm.begin() == cm.end());\n  for (size_t i = 0;i < 10000; ++i) {\n    cm[17 * i] = i;\n    um[17 * i] = i;\n  }\n\n  for (size_t i = 0;i < 10000; ++i) {\n    assert(cm[17 * i] == i);\n    assert(um[17 * i] == i);\n  }\n  assert(cm.size() == 10000);\n  assert(um.size() == 10000);\n\n  for (size_t i = 0;i < 10000; i+=2) {\n    cm.erase(17*i);\n    um.erase(17*i);\n  }\n  for (size_t i = 0;i < 10000; i+=2) {\n    assert(cm.count(17*i) == i % 2);\n    assert(um.count(17*i) == i % 2);\n    if (cm.count(17*i)) {\n      assert(cm.find(17*i)->second == i);\n    }\n  }\n\n  assert(cm.size() == 5000);\n  assert(um.size() == 5000);\n\n  typedef graphlab::cuckoo_map_pow2::value_type vpair;\n  {\n    size_t cnt = 0;\n    foreach(vpair &v, cm) {\n      ASSERT_EQ(v.second, um[v.first]);\n      ++cnt;\n    }\n    ASSERT_EQ(cnt, 5000);\n  }\n  {\n    size_t cnt = 0;\n    foreach(const vpair &v, cm) {\n      ASSERT_EQ(v.second, um[v.first]);\n      ++cnt;\n    }\n    ASSERT_EQ(cnt, 5000);\n  }\n}\n\n\n\n\nvoid sanity_checks2() {\n  boost::unordered_map um;\n  graphlab::cuckoo_map cm(-1);\n  ASSERT_TRUE(cm.begin() == cm.end());\n\n  for (size_t i = 0;i < 10000; ++i) {\n    cm[17 * i] = i;\n    um[17 * i] = i;\n  }\n\n  for (size_t i = 0;i < 10000; ++i) {\n    assert(cm[17 * i] == i);\n    assert(um[17 * i] == i);\n  }\n  assert(cm.size() == 10000);\n  assert(um.size() == 10000);\n\n  for (size_t i = 0;i < 10000; i+=2) {\n    cm.erase(17*i);\n    um.erase(17*i);\n  }\n  for (size_t i = 0;i < 10000; i+=2) {\n    assert(cm.count(17*i) == i % 2);\n    assert(um.count(17*i) == i % 2);\n    if (cm.count(17*i)) {\n      assert(cm.find(17*i)->second == i);\n    }\n  }\n\n  assert(cm.size() == 5000);\n  assert(um.size() == 5000);\n\n  typedef graphlab::cuckoo_map::value_type vpair;\n  {\n    size_t cnt = 0;\n    foreach(vpair &v, cm) {\n      ASSERT_EQ(v.second, um[v.first]);\n      ++cnt;\n    }\n    ASSERT_EQ(cnt, 5000);\n  }\n  {\n    size_t cnt = 0;\n    foreach(const vpair &v, cm) {\n      ASSERT_EQ(v.second, um[v.first]);\n      ++cnt;\n    }\n    ASSERT_EQ(cnt, 5000);\n  }\n}\n\nstd::string randstring(size_t len) {\n  std::string ret; ret.reserve(len);\n  for (size_t i = 0;i < len; ++i) {\n    ret = ret + graphlab::random::fast_uniform('A','Z');\n  }\n  return ret;\n}\n\nvoid more_interesting_data_types_check() {\n  boost::unordered_map um;\n  graphlab::cuckoo_map_pow2 cm(\"\");\n  for (size_t i = 0;i < 10000; ++i) {\n    std::string s = randstring(16);\n    cm[s] = s;\n    um[s] = s;\n  }\n\n  assert(cm.size() == 10000);\n  assert(um.size() == 10000);\n\n  \n  typedef boost::unordered_map::value_type vpair;\n  foreach(vpair& v, um) {\n    ASSERT_EQ(v.second, cm[v.first]);\n  }\n\n\n  foreach(vpair& v, cm) {\n    ASSERT_EQ(v.second, um[v.first]);\n  }\n\n\n  foreach(const vpair& v, cm) {\n    ASSERT_EQ(v.second, um[v.first]);\n  }\n\n  // test assignment\n  graphlab::cuckoo_map_pow2 cm2(\"\");\n  cm2 = cm;\n\n  foreach(vpair& v, um) {\n    ASSERT_EQ(v.second, cm2[v.first]);\n  }\n\n\n  foreach(vpair& v, cm2) {\n    ASSERT_EQ(v.second, um[v.first]);\n  }\n\n\n  foreach(const vpair& v, cm2) {\n    ASSERT_EQ(v.second, um[v.first]);\n  }\n\n  std::stringstream strm;\n  graphlab::oarchive oarc(strm);\n  oarc << cm;\n  strm.flush();\n\n  \n  cm2.clear();\n  ASSERT_EQ(cm2.size(), 0);\n  graphlab::iarchive iarc(strm);\n  iarc >> cm2;\n  ASSERT_EQ(cm2.size(), 10000);\n\n  foreach(vpair& v, um) {\n    ASSERT_EQ(v.second, cm2[v.first]);\n  }\n\n\n  foreach(vpair& v, cm2) {\n    ASSERT_EQ(v.second, um[v.first]);\n  }\n\n\n  foreach(const vpair& v, cm2) {\n    ASSERT_EQ(v.second, um[v.first]);\n  }\n}\n\n\nvoid more_interesting_data_types_check2() {\n  boost::unordered_map um;\n  graphlab::cuckoo_map cm(\"\");\n  for (size_t i = 0;i < 10000; ++i) {\n    std::string s = randstring(16);\n    cm[s] = s;\n    um[s] = s;\n  }\n\n  assert(cm.size() == 10000);\n  assert(um.size() == 10000);\n\n\n  typedef boost::unordered_map::value_type vpair;\n  foreach(vpair& v, um) {\n    ASSERT_EQ(v.second, cm[v.first]);\n  }\n\n\n  foreach(vpair& v, cm) {\n    ASSERT_EQ(v.second, um[v.first]);\n  }\n\n\n  foreach(const vpair& v, cm) {\n    ASSERT_EQ(v.second, um[v.first]);\n  }\n\n\n  // test assignment\n  graphlab::cuckoo_map cm2(\"\");\n  cm2 = cm;\n\n  foreach(vpair& v, um) {\n    ASSERT_EQ(v.second, cm2[v.first]);\n  }\n\n\n  foreach(vpair& v, cm2) {\n    ASSERT_EQ(v.second, um[v.first]);\n  }\n\n\n  foreach(const vpair& v, cm2) {\n    ASSERT_EQ(v.second, um[v.first]);\n  }\n\n\n  std::stringstream strm;\n  graphlab::oarchive oarc(strm);\n  oarc << cm;\n  strm.flush();\n\n\n  cm2.clear();\n  ASSERT_EQ(cm2.size(), 0);\n  graphlab::iarchive iarc(strm);\n  iarc >> cm2;\n  ASSERT_EQ(cm2.size(), 10000);\n\n  foreach(vpair& v, um) {\n    ASSERT_EQ(v.second, cm2[v.first]);\n  }\n\n\n  foreach(vpair& v, cm2) {\n    ASSERT_EQ(v.second, um[v.first]);\n  }\n\n\n  foreach(const vpair& v, cm2) {\n    ASSERT_EQ(v.second, um[v.first]);\n  }\n}\n\n\nvoid benchmark() {\n  graphlab::timer ti;\n\n  size_t NUM_ELS = 10000000;\n  \n  std::vector v;\n  uint32_t u = 0;\n  for (size_t i = 0;i < NUM_ELS; ++i) {\n    v.push_back(u);\n    u += 1 + rand() % 8;\n  }\n  std::random_shuffle(v.begin(), v.end());\n  graphlab::memory_info::print_usage();\n\n  {\n    boost::unordered_map um;\n    ti.start();\n    for (size_t i = 0;i < NUM_ELS; ++i) {\n      um[v[i]] = i;\n    }\n    std::cout <<  NUM_ELS / 1000000 << \"M unordered map inserts in \" << ti.current_time() << \" (Load factor = \" << um.load_factor() << \")\" << std::endl;\n\n    graphlab::memory_info::print_usage();\n    \n    ti.start();\n    for (size_t i = 0;i < 10000000; ++i) {\n      size_t t = um[v[i]];\n      assert(t == i);\n    }\n    std::cout << \"10M unordered map successful probes in \" << ti.current_time() << std::endl;\n    um.clear();\n  }\n\n  {\n    graphlab::cuckoo_map cm(-1, 128);\n\n    //cm.reserve(102400);\n    ti.start();\n    for (size_t i = 0;i < NUM_ELS; ++i) {\n      cm[v[i]] = i;\n      if (i % 1000000 == 0) std::cout << cm.load_factor() << std::endl;\n\n    }\n    std::cout <<  NUM_ELS / 1000000 << \"M cuckoo map inserts in \" << ti.current_time() << \" (Load factor = \" << cm.load_factor() << \")\" << std::endl;\n\n    graphlab::memory_info::print_usage();\n\n    ti.start();\n    for (size_t i = 0;i < 10000000; ++i) {\n      size_t t = cm[v[i]];\n      assert(t == i);\n    }\n    std::cout << \"10M cuckoo map successful probes in \" << ti.current_time() << std::endl;\n\n  }\n  \n  {\n    graphlab::cuckoo_map_pow2 cm(-1, 128);\n    \n    //cm.reserve(102400);\n    ti.start();\n    for (size_t i = 0;i < NUM_ELS; ++i) {\n      cm[v[i]] = i;\n      if (i % 1000000 == 0) std::cout << cm.load_factor() << std::endl;\n\n    }\n    std::cout << NUM_ELS / 1000000 << \"M cuckoo map pow2 inserts in \" << ti.current_time() << \" (Load factor = \" << cm.load_factor() << \")\" << std::endl;\n\n    graphlab::memory_info::print_usage();\n\n    ti.start();\n    for (size_t i = 0;i < 10000000; ++i) {\n      size_t t = cm[v[i]];\n      assert(t == i);\n    }\n    std::cout << \"10M cuckoo map pow2 successful probes in \" << ti.current_time() << std::endl;\n\n  }\n}\n\n\n\n\nvoid benchmark_strings() {\n  graphlab::timer ti;\n\n  size_t NUM_ELS = 1000000;\n\n  std::vector v;\n  for (size_t i = 0;i < NUM_ELS; ++i) {\n    v.push_back(randstring(16));\n  }\n  graphlab::memory_info::print_usage();\n\n  {\n    boost::unordered_map um;\n    ti.start();\n    for (size_t i = 0;i < NUM_ELS; ++i) {\n      um[v[i]] = v[i];\n    }\n    std::cout <<  NUM_ELS / 1000000 << \"M unordered map inserts in \" << ti.current_time() << \" (Load factor = \" << um.load_factor() << \")\" << std::endl;\n\n    graphlab::memory_info::print_usage();\n\n    ti.start();\n    for (size_t i = 0;i < 1000000; ++i) {\n      std::string t = um[v[i]];\n      assert(t == v[i]);\n    }\n    std::cout << \"1M unordered map successful probes in \" << ti.current_time() << std::endl;\n    um.clear();\n  }\n\n  {\n    graphlab::cuckoo_map cm(\"\", 128);\n\n    //cm.reserve(102400);\n    ti.start();\n    for (size_t i = 0;i < NUM_ELS; ++i) {\n      cm[v[i]] = v[i];\n      if (i % 1000000 == 0) std::cout << cm.load_factor() << std::endl;\n\n    }\n    std::cout <<  NUM_ELS / 1000000 << \"M cuckoo map inserts in \" << ti.current_time() << \" (Load factor = \" << cm.load_factor() << \")\" << std::endl;\n\n    graphlab::memory_info::print_usage();\n\n    ti.start();\n    for (size_t i = 0;i < 1000000; ++i) {\n      std::string t = cm[v[i]];\n      assert(t == v[i]);\n    }\n    std::cout << \"1M cuckoo map successful probes in \" << ti.current_time() << std::endl;\n\n  }\n\n  {\n    graphlab::cuckoo_map_pow2 cm(\"\", 128);\n\n    //cm.reserve(102400);\n    ti.start();\n    for (size_t i = 0;i < NUM_ELS; ++i) {\n      cm[v[i]] = v[i];\n      if (i % 1000000 == 0) std::cout << cm.load_factor() << std::endl;\n\n    }\n    std::cout << NUM_ELS / 1000000 << \"M cuckoo map pow2 inserts in \" << ti.current_time() << \" (Load factor = \" << cm.load_factor() << \")\" << std::endl;\n\n    graphlab::memory_info::print_usage();\n\n    ti.start();\n    for (size_t i = 0;i < 1000000; ++i) {\n      std::string t = cm[v[i]];\n      assert(t == v[i]);\n    }\n    std::cout << \"1M cuckoo map pow2 successful probes in \" << ti.current_time() << std::endl;\n\n  }\n}\n\n\nvoid save_load_test() {\n  typedef graphlab::cuckoo_map_pow2 cuckoo_map_type;\n  cuckoo_map_type map(-1);\n  for(uint32_t i = 0; i < 10000; ++i) map[i] = i;\n  std::ofstream fout(\"tmp.txt\");\n  graphlab::oarchive oarc(fout);\n  std::string t = \"The end.\";\n  oarc << map << t;\n  fout.close();\n  std::ifstream fin(\"tmp.txt\");\n  graphlab::iarchive iarc(fin);\n  cuckoo_map_type map2(-1);\n  std::string txt;\n  iarc >> map2;\n  iarc >> txt;\n  ASSERT_EQ(txt, std::string(\"The end.\"));\n  for(uint32_t i = 0; i < 10000; ++i) \n    ASSERT_EQ(map[i], i);\n} // end of save load test\n\n\n\nvoid cuckoo_set_sanity_checks() {\n  boost::unordered_set um;\n  graphlab::cuckoo_set_pow2 cm(-1, 2, 2);\n  ASSERT_TRUE(cm.begin() == cm.end());\n  for (size_t i = 0;i < 10000; ++i) {\n    cm.insert(17 * i);\n    um.insert(17 * i);\n  }\n\n  for (size_t i = 0;i < 10000; ++i) {\n    assert(cm.count(17 * i) == 1);\n    assert(um.count(17 * i) == 1);\n  }\n  assert(cm.size() == 10000);\n  assert(um.size() == 10000);\n\n  for (size_t i = 0;i < 10000; i+=2) {\n    cm.erase(17*i);\n    um.erase(17*i);\n  }\n  for (size_t i = 0;i < 10000; i+=2) {\n    assert(cm.count(17*i) == i % 2);\n    assert(um.count(17*i) == i % 2);\n  }\n\n  assert(cm.size() == 5000);\n  assert(um.size() == 5000);\n\n  std::ofstream fout(\"tmp.txt\");\n  graphlab::oarchive oarc(fout);\n  oarc << cm;\n  fout.close();\n  std::ifstream fin(\"tmp.txt\");\n  graphlab::iarchive iarc(fin);\n  graphlab::cuckoo_set_pow2 set2(-1);\n  iarc >> set2;\n  assert(set2.size() == 5000);\n}\n\n\n\n\n\nint main(int argc, char** argv) {\n  std::cout << \"Basic Sanity Checks... \";\n  std::cout.flush();\n  sanity_checks();\n  sanity_checks2();\n  more_interesting_data_types_check();\n  more_interesting_data_types_check2();\n  save_load_test();\n\n\n  cuckoo_set_sanity_checks();\n\n  std::cout << \"Done\" << std::endl;\n\n\n  // std::cout << \"\\n\\n\\nRunning Benchmarks. uint32-->uint32\" << std::endl;\n  // benchmark();\n\n\n  // std::cout << \"\\n\\n\\nRunning Benchmarks. string-->string\" << std::endl;\n  // benchmark_strings();\n\n}\n"
  },
  {
    "path": "tests/data/CMakeLists.txt",
    "content": "project(tests)\n\ncopy_files(*)\n\n"
  },
  {
    "path": "tests/data/test_adj/test.adj",
    "content": "0\t1\t5\r\n1\t2\t0\t5\r\n2\t2\t0\t5\r\n3\t2\t0\t5\r\n"
  },
  {
    "path": "tests/data/test_snap/test.snap",
    "content": "# Snap Comments \r\n# Blah Blah Blah\r\n0\t5\r\n1\t0\r\n1\t5\r\n2\t0\r\n2\t5\r\n3\t0\r\n3\t5\r\n"
  },
  {
    "path": "tests/data/test_tsv/test.tsv",
    "content": "0\t5\r\n1\t0\r\n1\t5\r\n2\t0\r\n2\t5\r\n3\t0\r\n3\t5\r\n"
  },
  {
    "path": "tests/dc_consensus_test.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \nusing namespace graphlab;\n\n\n\n\nclass simple_engine_test {\n public:\n  dc_dist_object rmi;\n  blocking_queue queue;\n  async_consensus cons;\n  atomic numactive;;\n\n  simple_engine_test(distributed_control &dc):rmi(dc, this), cons(dc, 4) {\n    numactive.value = 4; \n    dc.barrier();\n  }\n\n  void add_task_local(size_t i) {\n    queue.enqueue(i);\n    if (numactive.value < 4) cons.cancel();\n  }  \n  \n  void task(size_t i) {\n    if (i < 5) std::cout << \"Task \" << i << std::endl;\n    if (i > 0) {\n      if (rmi.numprocs() == 1) {\n        add_task_local(i - 1);\n      }\n      else {\n        rmi.remote_call((procid_t)((rmi.procid() + 1) % rmi.numprocs()),\n                    &simple_engine_test::add_task_local,\n                    i - 1);\n      }\n    }\n  }\n  \n  bool try_terminate(size_t cpuid, std::pair &job) {\n    job.second = false;\n    \n    numactive.dec();\n    cons.begin_done_critical_section(cpuid);\n    job = queue.try_dequeue();\n    if (job.second == false) {\n      bool ret = cons.end_done_critical_section(cpuid);\n      numactive.inc();\n      return ret;\n    }\n    else {\n      cons.cancel_critical_section(cpuid);\n      numactive.inc();\n      return false;\n    }\n  }\n  \n  void thread(size_t cpuid) {\n    while(1) {\n       std::pair job = queue.try_dequeue();\n       if (job.second == false) {\n          bool ret = try_terminate(cpuid, job);\n          if (ret == true) break;\n          if (ret == false && job.second == false) continue;\n       }\n       task(job.first);\n    }\n  }\n  \n  void start_thread() {\n    thread_group thrgrp; \n    for (size_t i = 0;i < 4; ++i) {\n      thrgrp.launch(boost::bind(\n                            &simple_engine_test::thread,\n                            this, i));\n    }\n    \n    thrgrp.join();\n    ASSERT_EQ(queue.size(), 0);\n  }\n};\n\n\nint main(int argc, char ** argv) {\n  /** Initialization */\n  mpi_tools::init(argc, argv);\n  global_logger().set_log_level(LOG_DEBUG);\n\n  dc_init_param param;\n  if (init_param_from_mpi(param) == false) {\n    return 0;\n  }\n  distributed_control dc(param);\n  simple_engine_test test(dc);\n  test.add_task_local(1000);\n  test.start_thread();\n  mpi_tools::finalize();\n}\n"
  },
  {
    "path": "tests/dc_fiber_consensus_test.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \nusing namespace graphlab;\n\n#define NTHREADS 1000\n\n\nclass simple_engine_test {\n public:\n  dc_dist_object rmi;\n  blocking_queue queue;\n  fiber_async_consensus cons;\n  atomic numactive;;\n\n  simple_engine_test(distributed_control &dc):rmi(dc, this), cons(dc, NTHREADS) {\n    numactive.value = NTHREADS; \n    dc.barrier();\n  }\n\n  void add_task_local(size_t i) {\n    queue.enqueue(i);\n    if (numactive.value < NTHREADS) cons.cancel();\n  }  \n  \n  void task(size_t i) {\n    if (i < 5) std::cout << \"Task \" << i << std::endl;\n    if (i > 0) {\n      if (rmi.numprocs() == 1) {\n        add_task_local(i - 1);\n      }\n      else {\n        rmi.remote_call((procid_t)((rmi.procid() + 1) % rmi.numprocs()),\n                    &simple_engine_test::add_task_local,\n                    i - 1);\n      }\n    }\n  }\n  \n  bool try_terminate(size_t cpuid, std::pair &job) {\n    job.second = false;\n    \n    numactive.dec();\n    cons.begin_done_critical_section(cpuid);\n    job = queue.try_dequeue();\n    if (job.second == false) {\n      bool ret = cons.end_done_critical_section(cpuid);\n      numactive.inc();\n      return ret;\n    }\n    else {\n      cons.cancel_critical_section(cpuid);\n      numactive.inc();\n      return false;\n    }\n  }\n  \n  void thread(size_t cpuid) {\n    while(1) {\n       std::pair job = queue.try_dequeue();\n       if (job.second == false) {\n          bool ret = try_terminate(cpuid, job);\n          if (ret == true) break;\n          if (ret == false && job.second == false) continue;\n       }\n       task(job.first);\n    }\n  }\n  \n  void start_thread() {\n    fiber_group thrgrp; \n    for (size_t i = 0;i < NTHREADS; ++i) {\n      thrgrp.launch(boost::bind(\n                            &simple_engine_test::thread,\n                            this, i));\n    }\n    \n    thrgrp.join();\n    ASSERT_EQ(queue.size(), 0);\n  }\n};\n\n\nint main(int argc, char ** argv) {\n  /** Initialization */\n  mpi_tools::init(argc, argv);\n  global_logger().set_log_level(LOG_DEBUG);\n\n  dc_init_param param;\n  if (init_param_from_mpi(param) == false) {\n    return 0;\n  }\n  distributed_control dc(param);\n  simple_engine_test test(dc);\n  test.add_task_local(300);\n  test.start_thread();\n  dc.barrier();\n  mpi_tools::finalize();\n}\n"
  },
  {
    "path": "tests/dc_test_sequentialization.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \nusing namespace graphlab;\n\n\n\n\nclass seq_test {\n public:\n  dc_dist_object rmi;\n  std::vector ctr;\n  seq_test(distributed_control &dc):rmi(dc, this), ctr(100,0) {\n    rmi.barrier();\n  }\n  \n  void recv(size_t idx, size_t val) {\n    ASSERT_EQ(thread::thread_id(), idx);\n    ASSERT_EQ(ctr[idx], val);\n    ++ctr[idx];\n  }\n\n  void run() {\n    for (size_t i = 1; i < 2; ++i) {\n      rmi.dc().set_sequentialization_key(i);\n      for (size_t j = 0;j < 1000000; ++j) {\n        rmi.remote_call(1, &seq_test::recv, i, j);\n      }\n    }\n  }\n};\n\n\nint main(int argc, char ** argv) {\n  /** Initialization */\n  mpi_tools::init(argc, argv);\n  global_logger().set_log_level(LOG_DEBUG);\n\n  dc_init_param param;\n  if (init_param_from_mpi(param) == false) {\n    return 0;\n  }\n  distributed_control dc(param);\n  seq_test test(dc);\n  if (dc.procid() == 0) {\n    test.run();\n  }\n  dc.full_barrier();\n  mpi_tools::finalize();\n}\n"
  },
  {
    "path": "tests/dcsc_test.cpp",
    "content": "#include \n#include \n#include \n#include \n#include \nusing namespace graphlab;\n\nint main(int argc, char** argv) {\n  dcsc_store store;\n  std::cout << store;\n  // basic tests\n  store.insert(1, 2, 1);\n  store.insert(2, 5, 2);\n  store.insert(4, 4, 3);\n  store.insert(4, 5, 4);\n  store.insert(0, 1, 5);\n  store.insert(0, 5, 6);\n  store.insert(0, 3, 7);\n  store.insert(3, 3, 8);\n  store.insert(4, 3, 9);\n\n  std::cout << store;\n\n  std::cout << \"\\n\\nPrinting column 0\\n\";\n  typedef dcsc_store::entry_type entry_type;\n  foreach(const entry_type e, store.get_column(0)) {\n    std::cout << \"(\" << e.row() << \", \" << e.column() << \") = \" << e.value() << \"\\n\";\n  }\n\n  std::cout << \"\\n\\nPrinting column 5\\n\";\n  foreach(entry_type e, store.get_column(5)) {\n    std::cout << \"(\" << e.row() << \", \" << e.column() << \") = \" << e.value() << \"\\n\";\n  }\n\n\n  std::cout << \"\\n\\nChanging column 3 to all 1s\\n\";\n  foreach(entry_type e, store.get_column(3)) {\n    e.value() = 1;\n  }\n\n  std::cout << store;\n\n  srand(10);\n  store.clear();\n  std::vector row, col, val;\n  for (size_t i = 0;i < 10000; ++i) {\n    row.push_back(rand());\n    col.push_back(rand());\n    val.push_back(rand());\n  }\n\n  store.construct(row.begin(), row.end(),\n                  col.begin(), col.end(),\n                  val.begin(), val.end());\n\n  for (size_t i = 0;i < 10000; ++i) {\n    assert(store.find(row[i], col[i]) == val[i]);\n  }\n\n\n}\n"
  },
  {
    "path": "tests/dense_bitset_test.cxx",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \nusing namespace graphlab;\n\nclass DenseBitsetTestSuite : public CxxTest::TestSuite {\npublic:\n  void test_densebitset(void) {\n    dense_bitset d;\n    d.resize(100);\n    d.clear();\n    size_t probelocations[7] = {0, 10, 12, 50, 66, 81, 99};\n    // test setting\n    for (size_t i= 0;i < 7; ++i) {\n      d.set_bit(probelocations[i]);\n    }\n    \n    for (size_t i = 0;i< 100; ++i) {\n      bool inprobe=false;\n      for (size_t j = 0;j <7; ++j) inprobe |= (probelocations[j] == i);\n      TS_ASSERT_EQUALS(d.get(i), inprobe);\n    }\n\n    // test iteration\n    size_t iter = (size_t)(-1);\n    TS_ASSERT_EQUALS(d.first_bit(iter), true)\n    for (size_t i= 0;i < 7; ++i) {\n      TS_ASSERT_EQUALS(iter, probelocations[i]);\n      bool ret = d.next_bit(iter);\n      TS_ASSERT_EQUALS(ret, i < 6);\n    }\n    size_t ctr = 0;\n    foreach(iter, d) {\n      TS_ASSERT(ctr < 7);\n      TS_ASSERT_EQUALS(iter, probelocations[ctr]);\n      ++ctr;\n    }\n    \n    std::stringstream strm;\n    graphlab::oarchive oarc(strm);\n    oarc << d;\n    strm.flush();\n    graphlab::iarchive iarc(strm);\n    dense_bitset d2;\n    iarc >> d2;\n\n\n    for (size_t i = 0;i< 100; ++i) {\n      bool inprobe=false;\n      for (size_t j = 0;j <7; ++j) inprobe |= (probelocations[j] == i);\n      TS_ASSERT_EQUALS(d2.get(i), inprobe);\n    }\n    // testclearing\n    for (size_t i= 0;i < 7; ++i) {\n      d.clear_bit(probelocations[i]);\n    }\n    for (size_t i = 0;i< 100; ++i) {\n      TS_ASSERT_EQUALS(d.get(i), false);\n    }\n\n    d.fill();\n    ASSERT_EQ(d.popcount(), d.size());\n    d.invert();\n    ASSERT_EQ(d.popcount(), 0);\n    d.invert();\n    ASSERT_EQ(d.popcount(), d.size());\n\n    d2.fill();\n    ASSERT_EQ(d2.popcount(), d2.size());\n\n\n  }\n\n\n  void test_fixeddensebitset(void) {\n    fixed_dense_bitset<100> d;\n    size_t probelocations[7] = {0, 10, 12, 50, 66, 81, 99};\n    // test setting\n    for (size_t i= 0;i < 7; ++i) {\n      d.set_bit(probelocations[i]);\n    }\n    \n    for (size_t i = 0;i< 100; ++i) {\n      bool inprobe=false;\n      for (size_t j = 0;j <7; ++j) inprobe |= (probelocations[j] == i);\n      TS_ASSERT_EQUALS(d.get(i), inprobe);\n    }\n\n    // test iteration\n    size_t iter = (size_t)(-1);\n    TS_ASSERT_EQUALS(d.first_bit(iter), true)\n    for (size_t i= 0;i < 7; ++i) {\n      TS_ASSERT_EQUALS(iter, probelocations[i]);\n      bool ret = d.next_bit(iter);\n      TS_ASSERT_EQUALS(ret, i < 6);\n    }\n    \n    size_t ctr = 0;\n    foreach(iter, d) {\n      TS_ASSERT(ctr < 7);\n      TS_ASSERT_EQUALS(iter, probelocations[ctr]);\n      ++ctr;\n    }\n\n    std::stringstream strm;\n    graphlab::oarchive oarc(strm);\n    oarc << d;\n    strm.flush();\n    graphlab::iarchive iarc(strm);\n    fixed_dense_bitset<100> d2;\n    iarc >> d2;\n\n\n    for (size_t i = 0;i< 100; ++i) {\n      bool inprobe=false;\n      for (size_t j = 0;j <7; ++j) inprobe |= (probelocations[j] == i);\n      TS_ASSERT_EQUALS(d2.get(i), inprobe);\n    }\n    \n    \n    // testclearing\n    for (size_t i= 0;i < 7; ++i) {\n      d.clear_bit(probelocations[i]);\n    }\n    for (size_t i = 0;i< 100; ++i) {\n      TS_ASSERT_EQUALS(d.get(i), false);\n    }\n\n    d.fill();\n    ASSERT_EQ(d.popcount(), d.size());\n\n    d2.fill();\n    ASSERT_EQ(d2.popcount(), d2.size());\n\n  }\n\n\n};\n\n"
  },
  {
    "path": "tests/dht_performance_test.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n#include \n#include     \n#include \n#include \nusing namespace graphlab;\n\nstd::string randstring(size_t len) {\n  std::string str;\n  str.resize(len);\n  const char *charset=\"ab\";\n  size_t charsetlen = 64;\n  for (size_t i = 0;i < len; ++i) {\n    str[i] = charset[rand()  % charsetlen];\n  }\n  return str;\n}\n\nint main(int argc, char ** argv) {\n  //mpi_tools::init(argc, argv);\n  global_logger().set_log_level(LOG_INFO);\n\n  dc_init_param param;\n  mpi_tools::init(argc, argv);\n  if (!init_param_from_mpi(param)) {\n    return 0;\n  }\n  \n  global_logger().set_log_level(LOG_DEBUG);\n  distributed_control dc(param);\n  std::cout << \"I am machine id \" << dc.procid() \n            << \" in \" << dc.numprocs() << \" machines\"< testdht(dc);\n  \n  std::vector > data;\n  const size_t NUMSTRINGS = 10000;\n  const size_t strlen[4] = {16, 128, 1024, 10240};\n  // fill rate\n  for (size_t l = 0; l < 4; ++l) {\n    timer ti;\n    ti.start();\n    if (dc.procid() == 0) {\n      std::cout << \"String Length = \" << strlen[l] << std::endl;\n      data.clear();\n      for (size_t i = 0;i < NUMSTRINGS; ++i) {\n        data.push_back(std::make_pair(randstring(8), randstring(strlen[l])));\n      }\n      std::cout << \"10k random strings generated\" << std::endl;\n      std::cout << \"Starting set\" << std::endl;\n      for (size_t i = 0;i < NUMSTRINGS; ++i) {\n        testdht.set(data[i].first, data[i].second);\n        if (i % 100 == 0) {\n          std::cout << \".\";\n          std::cout.flush();\n        }\n      }\n      std::cout << \"10k insertions in \" << ti.current_time() << std::endl;\n    }\n      dc.full_barrier();\n    if (dc.procid() == 0) {\n      std::cout << \"--> Time to Insertion Barrier \" << ti.current_time() << std::endl;\n    }\n    // get rate\n    if (dc.procid() == 0) {\n      std::cout << \"Starting get\" << std::endl;\n\n      timer ti;\n      ti.start();\n      for (size_t i = 0;i < NUMSTRINGS; ++i) {\n        std::pair ret = testdht.get(data[i].first);\n        assert(ret.first);\n        if (i % 100 == 0) {\n          std::cout << \".\";\n          std::cout.flush();\n        }\n      }\n      std::cout << \"10k reads in \" << ti.current_time() << std::endl;\n    }\n    testdht.clear();\n  }\n  dc.barrier();\n  testdht.print_stats();\n  mpi_tools::finalize();\n}\n"
  },
  {
    "path": "tests/distributed_chandy_misra_test.cpp",
    "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n\n#include \n\n#define INITIAL_NLOCKS_TO_ACQUIRE 1000\ngraphlab::mutex mt;\ngraphlab::conditional cond;\nstd::vector lockable_vertices;\nboost::unordered_map demand_set;\nboost::unordered_map current_demand_set;\nboost::unordered_map locked_set;\nsize_t nlocksacquired ;\n\nsize_t nlocks_to_acquire;\n\nstruct vertex_data {\n  uint32_t nupdates;\n  double value, old_value;\n  vertex_data(double value = 1) :\n    nupdates(0), value(value), old_value(0) { }\n}; // End of vertex data\nSERIALIZABLE_POD(vertex_data);\nstd::ostream& operator<<(std::ostream& out, const vertex_data& vdata) {\n  return out << \"Rank=\" << vdata.value;\n}\n\nstruct edge_data { }; // End of edge data\nSERIALIZABLE_POD(edge_data);\n\ntypedef graphlab::distributed_graph graph_type;\n\n\ngraphlab::distributed_chandy_misra *locks;\ngraph_type *ggraph;\ngraphlab::blocking_queue locked_elements;\n\n\nvoid callback(graphlab::vertex_id_type v) {\n  //logstream(LOG_INFO) << \"Locked \" << ggraph->global_vid(v) << std::endl;\n  mt.lock();\n  ASSERT_EQ(current_demand_set[v], 1);\n  locked_set[v]++;\n  nlocksacquired++;\n  mt.unlock();\n//  graphlab::my_sleep(1);\n  locked_elements.enqueue(v);\n}\n\n\nvoid thread_stuff() {\n  std::pair deq;\n  while(1) {\n    deq = locked_elements.dequeue();\n    if (deq.second == false) break;\n    else {\n      locks->philosopher_stops_eating(deq.first);\n      mt.lock();\n      current_demand_set[deq.first] = 0;\n      bool getnextlock = nlocks_to_acquire > 0;\n      if (nlocks_to_acquire > 0) {\n        nlocks_to_acquire--;\n        if (nlocks_to_acquire % 100 == 0) {\n          std::cout << \"Remaining: \" << nlocks_to_acquire << std::endl;\n        }\n      }\n      if (nlocks_to_acquire == 0 &&\n        nlocksacquired == INITIAL_NLOCKS_TO_ACQUIRE + lockable_vertices.size()) cond.signal();\n      mt.unlock();\n\n      if (getnextlock > 0) {\n        graphlab::vertex_id_type toacquire = 0;\n        while(1) {\n          mt.lock();\n           toacquire = lockable_vertices[graphlab::random::rand() %\n                                         lockable_vertices.size()];\n          if (current_demand_set[toacquire] == 0) {\n            current_demand_set[toacquire] = 1;\n            demand_set[toacquire]++;\n            mt.unlock();\n            break;\n          }\n          mt.unlock();\n        }\n        locks->make_philosopher_hungry(toacquire);\n      }\n    }\n  }\n}\n\n\nint main(int argc, char** argv) {\n//   global_logger().set_log_level(LOG_INFO);\n//   global_logger().set_log_to_console(true);\n\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::dc_init_param rpc_parameters;\n  graphlab::init_param_from_mpi(rpc_parameters);\n  graphlab::distributed_control dc(rpc_parameters);\n\n\n  // Parse command line options -----------------------------------------------\n  graphlab::command_line_options clopts(\"distributed chandy misra test.\");\n  std::string format = \"adj\";\n  std::string graph_dir = \"\";\n  clopts.attach_option(\"graph\", graph_dir,\n                       \"The graph file.  If none is provided \"\n                       \"then a toy graph will be created\");\n  clopts.add_positional(\"graph\");\n  clopts.attach_option(\"format\",format,\n                       \"The graph file format: {metis, snap, tsv, adj, bin}\");\n  size_t ring = 0;\n  clopts.attach_option(\"ring\", ring,\n                       \"The size of the ring. \"\n                       \"If ring=0 then the graph file is used.\");\n  size_t randomconnect = 0;\n  clopts.attach_option(\"randomconnect\", randomconnect,\n                       \"The size of a randomly connected network. \"\n                       \"If randomconnect=0 then the graph file is used.\");\n\n  if(!clopts.parse(argc, argv)) {\n    std::cout << \"Error in parsing command line arguments.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  std::cout << dc.procid() << \": Starting.\" << std::endl;\n  graphlab::timer timer; timer.start();\n  graph_type graph(dc, clopts);\n  ggraph = &graph;\n  if(ring > 0) {\n    if(dc.procid() == 0) {\n      for(size_t i = 0; i < ring; ++i) graph.add_edge(i, i + 1);\n      graph.add_edge(ring, 0);\n    }\n  } else if(randomconnect > 0) {\n    if(dc.procid() == 0) {\n      for(size_t i = 0; i < randomconnect; ++i) {\n        std::vector v(randomconnect, false);\n        v[i] = true;\n        for (size_t r = 0; r < randomconnect /2 ; ++r) {\n          size_t t = graphlab::random::rand() % randomconnect;\n          if (v[t] == false && t > i) {\n            graph.add_edge(i, t);\n            //            std::cout << i << \"->\" << t << \"\\n\";\n            v[t] = true;\n          }\n        }\n      }\n    }\n  } else {\n    std::vector graph_files;\n    graphlab::fs_util::list_files_with_prefix(graph_dir, \"\", graph_files);\n    for(size_t i = 0; i < graph_files.size(); ++i) {\n      if (i % dc.numprocs() == dc.procid()) {\n        const std::string graph_fname = graph_dir + graph_files[i];\n        std::cout << \"Loading graph from structure file: \" << graph_fname\n                  << std::endl;\n        graph.load_format(graph_fname, format);\n      }\n    }\n  }\n  std::cout << dc.procid() << \": Enter Finalize\" << std::endl;\n  graph.finalize();\n  \n  boost::unordered_set eidset1;\n  boost::unordered_set eidset2;\n  typedef graph_type::local_edge_type  local_edge_type;\n  typedef graph_type::local_edge_list_type local_edge_list_type;\n \n  for (size_t v = 0; v < graph.num_local_vertices(); ++v) {\n    const local_edge_list_type& in_edges = graph.l_in_edges(v);\n    foreach(const local_edge_type& edge, in_edges) {\n      size_t edgeid = edge.id();\n      ASSERT_TRUE(eidset1.find(edgeid) == eidset1.end());\n      eidset1.insert(edgeid);\n    }\n    const local_edge_list_type& out_edges = graph.l_out_edges(v);\n    foreach(const local_edge_type& edge, out_edges) {\n      size_t edgeid = edge.id();\n      ASSERT_TRUE(eidset1.find(edgeid) == eidset1.end());\n      ASSERT_TRUE(eidset2.find(edgeid) == eidset2.end());\n      eidset2.insert(edgeid);\n    }\n  }\n  ASSERT_EQ(eidset1.size(), eidset2.size());\n  eidset1.clear(); eidset2.clear();\n  \n  std::cout << \" ===============================================================\"\n            << std::endl;\n  std::cout << dc.procid() << \": Finished in \" << timer.current_time() << std::endl;\n\n  std::cout\n    << \"========== Graph statistics on proc \" << dc.procid()\n    << \" ===============\"\n    << \"\\n Num vertices: \" << graph.num_vertices()\n    << \"\\n Num edges: \" << graph.num_edges()\n    << \"\\n Num replica: \" << graph.num_replicas()\n    << \"\\n Replica to vertex ratio: \"\n    << (float)graph.num_replicas()/graph.num_vertices()\n    << \"\\n --------------------------------------------\"\n    << \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n    << \"\\n Num local vertices: \" << graph.num_local_vertices()\n    << \"\\n Replica to own ratio: \"\n    << (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n    << \"\\n Num local edges: \" << graph.num_local_edges()\n    << \"\\n Edge balance ratio: \" << (float)graph.num_local_edges()/graph.num_edges()\n    << std::endl;\n\n  // for (graphlab::vertex_id_type v = 0; v < graph.num_local_vertices(); ++v) {\n  //   std::cout << graph.l_get_vertex_record(v).gvid << \": \" << graph.l_get_vertex_record(v).owner << \":\";\n  //   foreach(graphlab::procid_t pid,  graph.l_get_vertex_record(v).get_replicas()) {\n  //     std::cout << pid << \" \";\n  //   }\n  //   std::cout << \"\\n\";\n  // }\n  dc.barrier();\n  locks = new graphlab::distributed_chandy_misra(dc, graph, callback);\n  nlocksacquired = 0;\n  nlocks_to_acquire = INITIAL_NLOCKS_TO_ACQUIRE;\n  dc.full_barrier();\n  for (graphlab::vertex_id_type v = 0; v < graph.num_local_vertices(); ++v) {\n    if (graph.l_get_vertex_record(v).owner == dc.procid()) {\n      demand_set[v] = 1;\n      current_demand_set[v] = 1;\n      lockable_vertices.push_back(v);\n    }\n  }\n  dc.full_barrier();\n  graphlab::thread_group thrs;\n  for (size_t i = 0;i < 10; ++i) {\n    thrs.launch(thread_stuff);\n  }\n  for (graphlab::vertex_id_type v = 0; v < graph.num_local_vertices(); ++v) {\n    if (graph.l_get_vertex_record(v).owner == dc.procid()) {\n      //std::cout << dc.procid() << \": Lock Req for \" << graph.l_get_vertex_record(v).gvid << std::endl;\n      locks->make_philosopher_hungry(v);\n    }\n  }\n  mt.lock();\n  while (nlocksacquired != INITIAL_NLOCKS_TO_ACQUIRE + lockable_vertices.size()) cond.wait(mt);\n  mt.unlock();\n  dc.barrier();\n  locked_elements.stop_blocking();\n  thrs.join();\n  std::cout << INITIAL_NLOCKS_TO_ACQUIRE + lockable_vertices.size() << \" Locks to acquire\\n\";\n  std::cout << nlocksacquired << \" Locks Acquired in total\\n\";\n  boost::unordered_map::const_iterator iter = demand_set.begin();\n  bool bad = (nlocksacquired != INITIAL_NLOCKS_TO_ACQUIRE + lockable_vertices.size());\n  while (iter != demand_set.end()) {\n    if(locked_set[iter->first] != iter->second) {\n      std::cout << graph.l_get_vertex_record(iter->first).gvid << \" mismatch: \"\n                << locked_set[iter->first] << \", \" << iter->second << \"\\n\";\n      bad = true;\n    }\n    ++iter;\n  }\n  if (bad) {\n    locks->print_out();\n  }\n  dc.barrier();\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // End of main\n\n\n"
  },
  {
    "path": "tests/distributed_graph_test.cpp",
    "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n// standard C++ headers\n#include \n#include \n#include \n\n\ntemplate\nstd::vector operator+=(std::vector& v1, const std::vector& v2) {\n  for (size_t i = 0; i < v2.size(); ++i)\n    v1.push_back(v2[i]);\n  return v1;\n}\n\n\n#include \n#include \n#include \n#include \n#include \n\n\ngraphlab::distributed_control* dc;\n\ntemplate\nclass map_reduce;\n\nnamespace tests{\nclass distributed_graph_test  {\n public:\n   struct vertex_data: public graphlab::IS_POD_TYPE  {\n     size_t value;\n     vertex_data() : value(0) { }\n     vertex_data(size_t n) : value(n) { }\n     bool operator==(const vertex_data& other)  const {\n       return value == other.value;\n     }\n   };\n\n   struct edge_data: public graphlab::IS_POD_TYPE  {\n     int from;\n     int to;\n     edge_data (int f = 0, int t = 0) : from(f), to(t) {}\n     bool operator==(const edge_data& other)  const {\n       return ((from == other.from) && (to == other.to));\n     }\n   };\n\n   /**\n    * Test adding vertex.\n    */\n   void test_add_vertex() {\n     graphlab::distributed_graph g(*dc);\n     test_add_vertex_impl(g, 100);\n     test_add_vertex_impl(g, 1000);\n     test_add_vertex_impl(g, 10000);\n     dc->cout() << \"\\n+ Pass test: graph add vertex. :) \\n\";\n   }\n\n   /**\n    * Test adding edges\n    */\n   void test_add_edge() {\n     graphlab::distributed_graph g(*dc);\n     test_add_edge_impl(g, 10);\n     test_add_edge_impl(g, 1000);\n     test_add_edge_impl(g, 10000);\n     dc->cout() << \"\\n+ Pass test: graph add edge. :) \\n\";\n   }\n\n   /**\n    * Test adding edges\n    */\n   void test_dynamic_add_edge() {\n     graphlab::distributed_graph g(*dc);\n     if (g.is_dynamic()) {\n       test_add_edge_impl(g, 10, true);\n       test_add_edge_impl(g, 1000, true);\n       test_add_edge_impl(g, 10000, true);\n       dc->cout() << \"\\n+ Pass test: graph dynamically add edge. :) \\n\";\n     } else {\n       dc->cout() << \"\\n- Graph does not support dynamic. Please compile with -DUSE_DYNAMIC_GRAPH \\n\";\n     }\n   }\n\n   /**\n    * Test save load\n    */\n   void test_save_load() {\n     graphlab::distributed_graph g(*dc);\n     for (size_t i = 0; i < 10; ++i) {\n       g.add_edge(i, (i+1), edge_data(i, i+1));\n     }\n     g.finalize();\n     test_save_load_impl(g);\n     if (g.is_dynamic()) {\n       for (size_t i = 0; i < 10; ++i) {\n         g.add_edge(i+1, (i), edge_data(i+1, i));\n       }\n       g.finalize();\n       test_save_load_impl(g);\n     }\n     dc->cout() << \"\\n+ Pass test: graph save load binary. :) \\n\";\n   }\n\n private: \n   template\n       void test_add_vertex_impl(Graph& g, size_t nverts) {\n         g.clear();\n         ASSERT_EQ(g.num_vertices(), 0);\n         for (size_t i = 0; i < nverts; ++i) {\n           g.add_vertex(i, vertex_data(i));\n         }\n         ASSERT_EQ(g.num_vertices(), 0); \n         g.finalize();\n         for (size_t i = 0; i < g.num_local_vertices(); ++i) {\n           ASSERT_EQ(g.l_vertex(i).data().value, g.global_vid(i));\n         }\n         ASSERT_EQ(g.num_vertices(), nverts);\n\n         // Test dynamic graph capability\n         if (g.is_dynamic()) {\n           // dynamic graph should support adding vertices after finalization\n           // add more vertices and override existing vertex values\n           for (size_t i = 0; i < 2*nverts; ++i) {\n             g.add_vertex(i, vertex_data(i*2));\n           }\n           g.finalize();\n           ASSERT_EQ(g.num_vertices(), 2*nverts);\n           for (size_t i = 0; i < g.num_local_vertices(); ++i) {\n             ASSERT_EQ(g.l_vertex(i).data().value, g.global_vid(i) * 2);\n           }\n         }\n       }\n\n   template\n       void test_add_edge_impl(Graph& g, size_t nedges, bool use_dynamic = false) {\n         typedef typename Graph::vertex_id_type vertex_id_type;\n         srand(0);\n         g.clear();\n         ASSERT_EQ(g.num_edges(), 0);\n         boost::unordered_map > out_edges;\n         boost::unordered_map > in_edges;\n         boost::unordered_set< std::pair > all_edges;\n         while (all_edges.size() < nedges) {\n           vertex_id_type src = rand() % (int)(3*sqrt(nedges));\n           vertex_id_type dst = rand() % (int)(3*sqrt(nedges));\n           if (src == dst)\n             continue;\n           std::pair pair(src, dst);\n           if (!all_edges.count(pair))  {\n             all_edges.insert(pair);\n             if (!out_edges.count(src)) {\n               out_edges[src] = std::vector();\n             } \n             if (!in_edges.count(dst)) {\n               in_edges[dst] = std::vector();\n             }\n             in_edges[dst].push_back(src);\n             out_edges[src].push_back(dst);\n           }\n         }\n         typedef typename boost::unordered_set< std::pair >::value_type pair_type; \n         int count = 0;\n         foreach (const pair_type& p, all_edges) {\n           if (count++ % dc->numprocs() == dc->procid()) {\n             g.add_edge(p.first, p.second, edge_data(p.first, p.second));\n           }\n           if (use_dynamic && count % (nedges/5) == 0) {\n             g.finalize();\n           } \n         }\n         if (!use_dynamic)\n           ASSERT_EQ(g.num_edges(), 0); \n\n         g.finalize();\n         check_adjacency(g, in_edges, out_edges, all_edges.size());\n         check_edge_data(g);\n         check_vertex_info(g);\n       }\n\n   template\n       void test_save_load_impl(Graph& g) {\n         typedef typename Graph::local_edge_type local_edge_type;\n\n         using namespace boost::filesystem;\n         path ph = unique_path();\n         if (create_directory(ph)) {\n           path prefix = ph;\n           prefix /= \"test\"; \n           dc->cout() << \"Save to path: \" << prefix.string() << std::endl;\n           g.save_binary(prefix.string());\n\n           Graph g2(*dc);\n           g2.load_binary(prefix.string());\n           ASSERT_EQ(g.num_vertices(), g2.num_vertices());\n           ASSERT_EQ(g.num_edges(), g2.num_edges());\n\n           for (size_t i = 0; i < g.num_local_vertices(); ++i) {\n             // check vertex records\n             ASSERT_TRUE(g.l_get_vertex_record(i) == g2.l_get_vertex_record(i));\n             // check vertex data \n             ASSERT_TRUE(g.l_vertex(i).data() == g2.l_vertex(i).data());\n\n             // check local in edges\n             ASSERT_EQ(g.l_in_edges(i).size(), g2.l_in_edges(i).size());\n             size_t in_edge_size = g.l_in_edges(i).size();\n             for (size_t j = 0; j < in_edge_size; ++j) {\n               ASSERT_EQ(g.l_in_edges(i)[j].source().lvid,\n                         g2.l_in_edges(i)[j].source().lvid);\n               ASSERT_EQ(g.l_in_edges(i)[j].target().lvid,\n                         g2.l_in_edges(i)[j].target().lvid);\n               ASSERT_TRUE(g.l_in_edges(i)[j].data() == g2.l_in_edges(i)[j].data());\n             }\n\n             // check local out edges\n             ASSERT_EQ(g.l_out_edges(i).size(), g2.l_out_edges(i).size());\n             size_t out_edge_size = g.l_out_edges(i).size();\n             for (size_t j = 0; j < out_edge_size; ++j) {\n               ASSERT_EQ(g.l_out_edges(i)[j].source().lvid,\n                         g2.l_out_edges(i)[j].source().lvid);\n               ASSERT_EQ(g.l_out_edges(i)[j].target().lvid,\n                         g2.l_out_edges(i)[j].target().lvid);\n               ASSERT_TRUE(g.l_out_edges(i)[j].data() == g2.l_out_edges(i)[j].data());\n             }\n           }\n           dc->cout() << \"Remove path: \" << ph.string()<< std::endl;\n           remove_all(ph);\n         } else {\n           dc->cout() << \"Unable to create tmp directory:\" << ph.string() << std::endl;\n         }\n       }\n\n   template\n       void check_edge_data(Graph& g) {\n         typedef typename Graph::local_edge_list_type local_edge_list_type;\n         typedef typename Graph::local_edge_type local_edge_type;\n         typedef typename Graph::vertex_type vertex_type;\n         typedef typename Graph::vertex_id_type vertex_id_type;\n         for (size_t i = 0; i < g.num_local_vertices(); ++i) {\n           const local_edge_list_type& in_edges = g.l_in_edges(i);\n           foreach (const local_edge_type& e, in_edges) {\n             ASSERT_EQ(e.data().from, g.global_vid(e.source().id()));\n             ASSERT_EQ(e.data().to, g.global_vid(e.target().id()));\n           }\n           const local_edge_list_type& out_edges = g.l_out_edges(i);\n           foreach (const local_edge_type& e, out_edges) {\n             ASSERT_EQ(e.data().from, g.global_vid(e.source().id()));\n             ASSERT_EQ(e.data().to, g.global_vid(e.target().id()));\n           }\n         }\n       }\n\n   /**\n    * Helper function to check the in/out edges of the graph.\n    */\n   template\n       void check_adjacency(Graph& g, \n                            boost::unordered_map >& in_edges,\n                            boost::unordered_map >& out_edges,\n                            size_t nedges) {\n         typedef typename Graph::local_edge_list_type local_edge_list_type;\n         typedef typename Graph::local_edge_type local_edge_type;\n         typedef typename Graph::vertex_type vertex_type;\n         typedef typename Graph::vertex_id_type vertex_id_type;\n\n         // check total edge size \n         ASSERT_EQ(g.num_edges(), nedges);\n         size_t sum_local_edges = g.num_local_edges();\n         dc->all_reduce(sum_local_edges);\n         ASSERT_EQ(g.num_edges(), sum_local_edges);\n\n         // check local edge size\n         size_t local_in_edge_size = 0;\n         size_t local_out_edge_size = 0;\n         for (size_t i = 0; i < g.num_local_vertices(); ++i) {\n           local_in_edge_size += g.l_in_edges(i).size();\n           local_out_edge_size += g.l_out_edges(i).size();\n         }\n         ASSERT_EQ(local_in_edge_size, g.num_local_edges());\n         ASSERT_EQ(local_out_edge_size, g.num_local_edges());\n\n         // check adjacency list\n         typedef map_reduce< vertex_id_type, std::vector > dist_adj_type;\n         dist_adj_type local_out_adj, local_in_adj;\n\n         for (size_t i = 0; i < g.num_local_vertices(); ++i) {\n           std::vector outids, inids;\n           vertex_id_type gvid = g.global_vid(i);\n           const local_edge_list_type& ls_out = g.l_out_edges(i);\n           const local_edge_list_type& ls_in = g.l_in_edges(i);\n           foreach (const local_edge_type& e, ls_out) {\n             ASSERT_EQ(e.source().id(), i);\n             outids.push_back(g.global_vid(e.target().id()));\n           }\n           foreach (const local_edge_type& e, ls_in) {\n             ASSERT_EQ(e.target().id(), i);\n             inids.push_back(g.global_vid(e.source().id()));\n           }\n           local_out_adj.data[gvid] = outids;\n           local_in_adj.data[gvid] = inids;\n         }\n         dc->all_reduce(local_out_adj);\n         dc->all_reduce(local_in_adj);\n\n         typedef typename boost::unordered_map >::const_iterator iter_type;\n\n         // check out adjacency \n         for (iter_type it = out_edges.begin(); it != out_edges.end(); ++it) {\n           vertex_id_type id = it->first;\n           std::vector expected = it->second;\n           std::vector actual = local_out_adj.data[id];\n           std::sort(actual.begin(), actual.end()); std::sort(expected.begin(), expected.end());\n           ASSERT_EQ(actual.size(), expected.size());\n           if (g.vid2lvid.count(id))\n             ASSERT_EQ(g.num_out_edges(id), expected.size());\n           for (size_t i = 0; i < actual.size(); ++i) {\n             ASSERT_EQ(actual[i], expected[i]);\n           }\n         }\n\n         // check in adjacency\n         for (iter_type it = in_edges.begin(); it != in_edges.end(); ++it) {\n           vertex_id_type id = it->first;\n           std::vector expected = it->second;\n           std::vector actual = local_in_adj.data[id];\n           std::sort(actual.begin(), actual.end()); std::sort(expected.begin(), expected.end());\n           ASSERT_EQ(actual.size(), expected.size());\n           if (g.vid2lvid.count(id))\n             ASSERT_EQ(g.num_in_edges(id), expected.size());\n           for (size_t i = 0; i < actual.size(); ++i) {\n             ASSERT_EQ(actual[i], expected[i]);\n           }\n         }\n       }\n\n   template\n       struct vertex_info {\n         typename Graph::vertex_id_type vid;\n         typename Graph::vertex_data_type data;\n         typename Graph::mirror_type mirrors;\n         graphlab::procid_t master;\n         size_t num_in_edges, num_out_edges;\n\n         bool operator==(const vertex_info& other) {\n           return ((master == other.master) && \n                   (vid == other.vid) && \n                   (data == other.data) &&\n                   (mirrors == other.mirrors) &&\n                   (num_in_edges == other.num_in_edges) &&\n                   (num_out_edges == other.num_out_edges));\n         }\n\n         void load(graphlab::iarchive& arc) {\n           arc >> vid\n               >> master \n               >> mirrors\n               >> num_in_edges\n               >> num_out_edges\n               >> data;\n         }\n\n         void save(graphlab::oarchive& arc) const {\n           arc << vid\n               << master\n               << mirrors\n               << num_in_edges\n               << num_out_edges\n               << data;\n         } // end of save\n       };\n\n   template\n       void check_vertex_info(Graph& g) {\n         typedef typename Graph::vertex_id_type vertex_id_type;\n         typedef typename Graph::vertex_data_type vertex_data_type;\n         typedef typename Graph::vertex_type vertex_type;\n         typedef typename Graph::local_vertex_type local_vertex_type;\n         typedef vertex_info vinfo_type;\n         typedef typename boost::unordered_map vinfo_map_type; \n\n         vinfo_map_type vid2info;\n         std::vector vids;\n\n         for (size_t i = 0; i < g.num_local_vertices(); ++i) {\n           vertex_type v = g.vertex(g.global_vid(i));\n           local_vertex_type lv = g.l_vertex(i);\n           ASSERT_EQ(v.local_id(), lv.id());\n           ASSERT_EQ(v.id(), lv.global_id());\n\n           vinfo_type info;\n           info.vid = v.id();\n           info.num_in_edges = v.num_in_edges();\n           info.num_out_edges = v.num_out_edges();\n           info.data = v.data();\n           info.mirrors = lv.mirrors();\n           info.master = lv.owner();\n           // master should not be in the mirror set\n           ASSERT_TRUE(info.mirrors.get(info.master) == 0);\n\n           vid2info[v.id()] = info;\n           if (lv.owned()) \n             vids.push_back(v.id());\n         }\n\n         // gather the vid->record map on each machine\n         std::vector vinfo_map_gather(dc->numprocs());\n         vinfo_map_gather[dc->procid()] = vid2info;\n         dc->all_gather(vinfo_map_gather);\n         dc->all_reduce(vids);\n\n         ASSERT_EQ(vids.size(), g.num_vertices());\n\n         // check the consistency of vertex_record on each machine. \n         foreach(vertex_id_type vid, vids) {\n           std::vector records;\n           std::vector mirror_expected;\n\n           for (size_t i = 0; i < vinfo_map_gather.size(); ++i) {\n             if (vinfo_map_gather[i].count(vid)) {\n               records.push_back(vinfo_map_gather[i][vid]);\n               mirror_expected.push_back(i);\n             }\n           }\n\n           // check vertex records are consistent  across machines.\n           for (size_t i = 1; i < records.size(); ++i) {\n             ASSERT_TRUE(records[i] == records[0]);\n           }\n\n           // recevied record size == mirror size + 1\n           ASSERT_EQ(records.size(), records[0].mirrors.popcount()+1);\n\n           for (size_t i = 0; i < mirror_expected.size(); ++i) {\n             size_t procid =  mirror_expected[i];\n             ASSERT_TRUE(records[0].mirrors.get(procid) || (records[0].master == procid));\n           }\n         } // end for loop over all vertices\n       }\n\n\n}; // end of distributed_graph_test\n\n} // namespace\n\nusing namespace tests;\n\ntemplate\nclass map_reduce {\n public:\n   boost::unordered_map data; \n   void save(graphlab::oarchive& oarc) const {\n     oarc << data;\n   }\n   void load(graphlab::iarchive& iarc) {\n     iarc >> data;\n   }\n   map_reduce& operator+=(const map_reduce& other) {\n     for (typename boost::unordered_map::const_iterator it = other.data.begin();\n          it != other.data.end(); ++it) {\n       K key = it->first;\n       V val = it->second;\n       if (data.count(key)) {\n         data[key] += val;\n       } else {\n         data[key] = val;\n       }\n     }\n     return *this;\n   }\n};\n\n\n\n\nint main(int argc, char** argv) {\n  graphlab::mpi_tools::init(argc, argv);\n  dc = new graphlab::distributed_control();\n\n  // run tests\n  distributed_graph_test testsuit; \n  testsuit.test_add_vertex();\n  testsuit.test_add_edge();\n  testsuit.test_dynamic_add_edge();\n  testsuit.test_save_load();\n\n  delete(dc);\n  graphlab::mpi_tools::finalize();\n}\n\n#include \n"
  },
  {
    "path": "tests/distributed_ingress_test.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n// standard C++ headers\n#include \n#include \n#include \n#include \n#include \n#include \n// #include \n#include \n\ntypedef char vertex_data;\ntypedef std::string edge_data;\n\ntypedef graphlab::distributed_graph graph_type;\ntypedef graph_type::vertex_record vertex_record; \n\nint main(int argc, char** argv) {\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  global_logger().set_log_level(LOG_INFO); \n  graphlab::command_line_options clopts(\"Distributed graph load test.\");\n  std::string graphpath; \n  bool gzip = false;\n  std::string prefix = \"\"; \n  std::string format = \"adj\";\n\n  clopts.attach_option(\"graph\", graphpath,\n                       \"The graph path \\n\");\n\n  clopts.attach_option(\"prefix\", prefix,\n                       \"The prefix for load/save binary file\\n\");\n\n  clopts.attach_option(\"gzip\", gzip,\n                       \"The input is in gzip format\\n\");\n\n  clopts.attach_option(\"format\", format,\n                       \"format of the graph: {adj, snap}\\n\");\n\n  \n  if(!clopts.parse(argc, argv)) {\n    logstream(LOG_FATAL) << \"Error in parsing command line arguments.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  graphlab::timer mytimer; mytimer.start();\n  // global_logger().set_log_to_console(true);\n\n  graph_type graph(dc, clopts);\n  \n  graph.load_format(graphpath, format);\n\n  // size_t heap_size_load;\n  // size_t allocate_size_load;\n  // MallocExtension::instance()->GetNumericProperty(\"generic.heap_size\", &heap_size_load);\n  // MallocExtension::instance()->GetNumericProperty(\"generic.current_allocated_bytes\", &allocate_size_load);\n\n  // if (dc.procid() == 0) {\n  //   std::cout << \"==========================================\" << std::endl;\n  //   std::cout << \"Heap Size (before finalize): \" << (double)heap_size_load/(1024*1024) << \"MB\" << \"\\n\";\n  //   std::cout << \"Allocated Size (before finalize): \" << (double)allocate_size_load/(1024*1024) << \"MB\" << \"\\n\";\n  //   std::cout << \"==========================================\" << std::endl;\n  // }\n\n  double time_to_load = mytimer.current_time();\n  graph.finalize();\n  double time_all = mytimer.current_time();\n\n  std::cout << dc.procid() << \": Finished in \" << mytimer.current_time() << std::endl;\n    std::cout \n      << \"========== Graph statistics on proc \" << dc.procid() << \" ===============\"\n      << \"\\n Num vertices: \" << graph.num_vertices()\n      << \"\\n Num edges: \" << graph.num_edges()\n      << \"\\n Num replica: \" << graph.num_replicas()\n      << \"\\n Replica to vertex ratio: \" \n      << (float)graph.num_replicas()/graph.num_vertices()\n      << \"\\n --------------------------------------------\" \n      << \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n      << \"\\n Num local vertices: \" << graph.num_local_vertices()\n      << \"\\n Replica to own ratio: \" \n      << (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n      << \"\\n Num local edges: \" << graph.num_local_edges()\n      << \"\\n Edge balance ratio: \" << (float)graph.num_local_edges()/graph.num_edges()\n      << \"\\n --------------------------------------------\" \n      << std::endl;\n      std::cout << \"==========================================\" << std::endl;\n\n   // size_t heap_size;\n   // size_t allocate_size;\n   // MallocExtension::instance()->GetNumericProperty(\"generic.heap_size\", &heap_size);\n   // MallocExtension::instance()->GetNumericProperty(\"generic.current_allocated_bytes\", &allocate_size);\n   // if (dc.procid() == 0) {\n   //   std::cout << \"Heap Size: \" << (double)heap_size/(1024*1024) << \"MB\" << \"\\n\";\n   //   std::cout << \"Allocated Size: \" << (double)allocate_size/(1024*1024) << \"MB\" << \"\\n\";\n   // }\n\n  if (dc.procid() == 0) {\n   std::ofstream fout;\n   std::vector keys = clopts.get_graph_args().get_option_keys();\n   std::string ingress_method = \"random\";\n   std::string constraint_graph = \"na\";\n   std::string bufsize = \"50000\";\n   bool usehash = false; \n   bool userecent = false; \n\n   foreach (std::string opt, keys) {\n     if (opt == \"ingress\") {\n       clopts.get_graph_args().get_option(\"ingress\", ingress_method);\n     } else if (opt == \"bufsize\") {\n       clopts.get_graph_args().get_option(\"bufsize\", bufsize);\n     } else if (opt == \"usehash\") {\n       clopts.get_graph_args().get_option(\"usehash\", usehash);\n     } else if (opt == \"userecent\") {\n       clopts.get_graph_args().get_option(\"userecent\", userecent);\n     } else if (opt == \"constrained_graph\") {\n       clopts.get_graph_args().get_option(\"constrained_graph\", constraint_graph);\n     }\n   }\n\n   fout.open(\"result.txt\");\n   fout << \"#ingress: \" << ingress_method  << std::endl\n     << \"#constraint: \" << constraint_graph << std::endl\n     << \"#bufsize: \" << bufsize << std::endl\n     << \"#usehash: \" << usehash << std::endl\n     << \"#userecent: \" << userecent\n     << std::endl;\n\n   fout << \"Num procs: \" << dc.numprocs() << std::endl;\n   fout << \"Replication factor: \" << (float)graph.num_replicas()/graph.num_vertices() << std::endl;\n   fout << \"Balance factor: \" << (float)graph.num_local_edges()/graph.num_edges() << std::endl;\n   // fout << \"Heap size (load): \" << (double)heap_size_load/(1024*1024) << std::endl;\n   // fout << \"Allocated size (load): \" << (double)allocate_size_load/(1024*1024) << std::endl;\n   // fout << \"Heap size (finalize): \" << (double)heap_size/(1024*1024) << std::endl;\n   // fout << \"Allocated size (finalize): \" << (double)allocate_size/(1024*1024) << std::endl;\n   fout << \"Runtime (load): \" << time_to_load << std::endl;\n   fout << \"Runtime (total): \" << time_all << std::endl;\n   fout.close();\n  }\n\n  // graph.get_local_graph().save_adjacency(\"partition_\"+boost::lexical_cast(dc.procid())+\".txt\");\n  // graph.save_format(\"partition\", \"snap\", false, 1);\n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // end of main\n\n#include \n"
  },
  {
    "path": "tests/empty_test.cxx",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n\n#include \n\n#include \n\n#include \n\nusing namespace graphlab;\n\nclass empty_test : public CxxTest::TestSuite {\npublic:\n\n  \n  void test_empty() {\n    std::vector v;\n    v.resize(100);\n    TS_ASSERT_EQUALS(v.size(), (size_t)100);\n    size_t c = 0;\n    foreach(empty e, v) {\n      e = empty();\n      ++c;\n    }\n    TS_ASSERT_EQUALS(c, (size_t)100);\n\n    TS_ASSERT_EQUALS(v.end() - v.begin(), (int)100);\n    TS_ASSERT_EQUALS(v.rend() - v.rbegin(), (int)100);\n    \n    std::vector v2;\n    v2.assign(v.begin(), v.end());\n    TS_ASSERT_EQUALS(v2.size(), (size_t)100);\n    v2.assign(v.rbegin(), v.rend());\n    TS_ASSERT_EQUALS(v2.size(), (size_t)100);\n\n    v.insert(v.begin(), empty());\n    TS_ASSERT_EQUALS(v.size(), (size_t)101);\n     v.insert(v.end(), empty());\n    TS_ASSERT_EQUALS(v.size(), (size_t)102);\n\n    std::vector::const_iterator iter = v.begin();\n    (*iter);\n    ++iter; TS_ASSERT_EQUALS(v.end() - iter, (int)101);\n    --iter; TS_ASSERT_EQUALS(v.end() - iter, (int)102);\n    iter+=10; TS_ASSERT_EQUALS(v.end() - iter, (int)92);\n    iter-=10; TS_ASSERT_EQUALS(v.end() - iter, (int)102);\n    std::vector::const_iterator iter2 = iter;\n    iter2 += 10;\n    TS_ASSERT_EQUALS(iter2 - iter, (int)10);\n  }\n\n};\n\n"
  },
  {
    "path": "tests/engine_terminator_bench.cxx",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n\n#include \n\n#define RING_SIZE 200\n#define NUM_ITERATIONS 1000\n\ntypedef graphlab::graph graph_type;\n\n\nclass increment_update : public\n                  graphlab::iupdate_functor {\n public:\n  void operator()(icontext_type& context) {\n    ++context.vertex_data();\n    foreach(edge_type edge, context.out_edges()) {\n      const vertex_id_type nbr_id = edge.target();\n      if (context.const_vertex_data(nbr_id) < NUM_ITERATIONS) {\n        context.schedule(nbr_id, *this);    \n      }\n    }\n  }\n}; // end of shortest path update functor\n\nvoid make_graph(graph_type &graph) {\n  for (size_t i = 0;i < RING_SIZE; ++i) {\n    graph.add_vertex(0);\n  }\n  \n  for (size_t i = 0;i < RING_SIZE; ++i) {\n    graph.add_edge(i, (i+1) % RING_SIZE, 0);\n  }\n}\n\n\nclass EngineTerminatorTestSuite: public CxxTest::TestSuite {\n public:  \n  void test_engine_terminator() {\n    // Create a graphlab core\n    graphlab::core core;\n    make_graph(core.graph());\n    core.graph().finalize();\n    for (size_t ncpus = 1; ncpus <= 8; ++ncpus) {\n      core.set_ncpus(ncpus);\n      core.set_scope_type(\"edge\");\n      core.schedule(0, increment_update());\n      const double runtime = core.start();\n      std::cout << ncpus << \" Procs: \" << runtime << std::endl;\n      // check the graph and reset it\n      for (size_t i = 0;i < RING_SIZE; ++i) {\n        if (core.graph().vertex_data(i) != (size_t)NUM_ITERATIONS) {\n          std::cout << \"vertex \" << i << \" \";\n          TS_ASSERT_EQUALS(core.graph().vertex_data(i), (size_t)NUM_ITERATIONS);\n        }\n        core.graph().vertex_data(i) = 0;\n      }\n    }\n  }\n};\n\n\n\n"
  },
  {
    "path": "tests/fiber_test.cpp",
    "content": "#include \n#include \n#include \nusing namespace graphlab;\nint numticks = 0;\nvoid threadfn() {\n\n  timer ti; ti.start();\n  while(1) {\n    if (ti.current_time() >= 1) break;\n    fiber_control::yield();\n    __sync_fetch_and_add(&numticks, 1);\n  }\n}\n\n\nvoid threadfn2() {\n\n  timer ti; ti.start();\n  while(1) {\n    if (ti.current_time() >= 2) break;\n    fiber_control::yield();\n    __sync_fetch_and_add(&numticks, 2);\n  }\n}\n\nint main(int argc, char** argv) {\n  timer ti; ti.start();\n  fiber_group group;\n  fiber_group group2;\n  for (int i = 0;i < 100000; ++i) {\n    group.launch(threadfn);\n    group2.launch(threadfn2);\n  }\n  group.join();\n  std::cout << \"Completion in \" << ti.current_time() << \"s\\n\";\n  std::cout << \"Context Switches: \" << numticks << \"\\n\";\n  group2.join();\n  std::cout << \"Completion in \" << ti.current_time() << \"s\\n\";\n  std::cout << \"Context Switches: \" << numticks << \"\\n\";\n}\n"
  },
  {
    "path": "tests/fibo_fiber_test.cpp",
    "content": "#include \n#include \n#include \nusing namespace graphlab;\n\nstruct fibonacci_compute_promise {\n  mutex* lock;\n  size_t argument;\n  size_t result;\n  size_t parent_tid;\n  bool result_set;\n};\n\nvoid fibonacci(fibonacci_compute_promise* promise) {\n  //std::cout << promise->argument << \"\\n\";\n  if (promise->argument == 1 ||  promise->argument == 2) {\n    promise->result = 1;\n  } else {\n    // recursive case\n    mutex lock;\n    fibonacci_compute_promise left, right;\n    left.lock = &lock;\n    left.argument = promise->argument - 1;\n    left.result_set = false;\n    left.parent_tid = fiber_control::get_tid();\n\n    right.lock = &lock;\n    right.argument = promise->argument - 2;\n    right.result_set = false;\n    right.parent_tid = fiber_control::get_tid();\n\n    fiber_control::get_instance().launch(boost::bind(fibonacci, &left));\n    fiber_control::get_instance().launch(boost::bind(fibonacci, &right));\n\n    // wait on the left and right promise\n    lock.lock();\n    while (left.result_set == false || right.result_set == false) {\n      fiber_control::deschedule_self(&lock.m_mut);\n      lock.lock();\n    }\n    lock.unlock();\n\n    assert(left.result_set);\n    assert(right.result_set);\n    promise->result = left.result + right.result;\n  }\n  promise->lock->lock();\n  promise->result_set = true;\n  if (promise->parent_tid) fiber_control::schedule_tid(promise->parent_tid);\n  promise->lock->unlock();\n}\n\n\nint main(int argc, char** argv) {\n\n  timer ti; ti.start();\n\n  fibonacci_compute_promise promise;\n  mutex lock;\n  promise.lock = &lock;\n  promise.result_set = false;\n  promise.argument = 24;\n  promise.parent_tid = 0;\n  fiber_control::get_instance().launch(boost::bind(fibonacci, &promise));\n  fiber_control::get_instance().join();\n  assert(promise.result_set);\n  std::cout << \"Fib(\" << promise.argument << \") = \" << promise.result << \"\\n\";\n\n  std::cout << \"Completion in \" << ti.current_time() << \"s\\n\";\n  std::cout << fiber_control::get_instance().total_threads_created() << \" threads created\\n\";\n}\n"
  },
  {
    "path": "tests/hdfs_test.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n\n\n#include \n\nint main(int argc, char **argv) {\n  {\n    graphlab::hdfs hdfs;\n    const bool write = true;\n    graphlab::hdfs::fstream file(hdfs, \"/tmp/joeytest.txt\", write);\n    file.good();\n    file << \"Hello World\\n\";\n    file.close();\n    std::vector files = hdfs.list_files(\"/tmp/\");\n    for(size_t i = 0; i < files.size(); ++i) \n      std::cout << files[i] << std::endl;\n  }\n\n  {\n    graphlab::hdfs hdfs;\n    graphlab::hdfs::fstream file(hdfs, \"/tmp/joeytest.txt\");\n    file.good();\n    std::string answer;\n    std::getline(file, answer);\n    std::cout << \"contents: \" << std::endl;\n    std::cout << answer << std::endl;\n    file.close();\n  }\n    std::cout << \"Done!\" << std::endl;\n}\n"
  },
  {
    "path": "tests/hopscotch_test.cpp",
    "content": "#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n\n\n\nboost::unordered_map um2;\ngraphlab::hopscotch_map cm2;\n\nvoid hopscotch_map_sanity_checks() {\n  const size_t NINS = 1500000;\n  ASSERT_TRUE(cm2.begin() == cm2.end());\n  for (size_t i = 0;i < NINS; ++i) {\n    cm2[17 * i] = i;\n    um2[17 * i] = i;\n  }\n\n  for (size_t i = 0;i < NINS; ++i) {\n    assert(cm2[17 * i] == i);\n    assert(um2[17 * i] == i);\n  }\n  assert(cm2.size() == NINS);\n  assert(um2.size() == NINS);\n\n  for (size_t i = 0;i < NINS; i+=2) {\n    cm2.erase(17*i);\n    um2.erase(17*i);\n  }\n  for (size_t i = 0;i < NINS; i+=2) {\n    assert(cm2.count(17*i) == i % 2);\n    assert(um2.count(17*i) == i % 2);\n    if (cm2.count(17*i)) {\n      assert(cm2.find(17*i)->second == i);\n    }\n  }\n\n  assert(cm2.size() == NINS / 2);\n  assert(um2.size() == NINS / 2);\n\n  typedef graphlab::hopscotch_map::value_type vpair;\n  {\n    size_t cnt = 0;\n    foreach(vpair &v, cm2) {\n      ASSERT_EQ(v.second, um2[v.first]);\n      ++cnt;\n    }\n    ASSERT_EQ(cnt, NINS / 2);\n  }\n  {\n    size_t cnt = 0;\n    foreach(const vpair &v, cm2) {\n      ASSERT_EQ(v.second, um2[v.first]);\n      ++cnt;\n    }\n    ASSERT_EQ(cnt, NINS / 2);\n  }\n\n  std::stringstream strm;\n  graphlab::oarchive oarc(strm);\n  oarc << cm2;\n  strm.flush();\n\n  cm2.clear();\n  ASSERT_EQ(cm2.size(), 0);\n  graphlab::iarchive iarc(strm);\n  iarc >> cm2;\n  ASSERT_EQ(cm2.size(), NINS / 2);\n\n}\n\n\n\n\n\nstruct bad_hasher {\n  size_t operator()(uint32_t a) const {\n    return 1;\n  }\n};\n\n\n\nvoid hopscotch_high_collision_sanity_checks() {\n  const size_t NINS = 15000;\n  boost::unordered_map um2;\n  graphlab::hopscotch_map cm2;\n  ASSERT_TRUE(cm2.begin() == cm2.end());\n  for (size_t i = 0;i < NINS; ++i) {\n    cm2[17 * i] = i;\n    um2[17 * i] = i;\n  }\n\n  for (size_t i = 0;i < NINS; ++i) {\n    assert(cm2[17 * i] == i);\n    assert(um2[17 * i] == i);\n  }\n  assert(cm2.size() == NINS);\n  assert(um2.size() == NINS);\n\n  for (size_t i = 0;i < NINS; i+=2) {\n    cm2.erase(17*i);\n    um2.erase(17*i);\n  }\n  for (size_t i = 0;i < NINS; i+=2) {\n    assert(cm2.count(17*i) == i % 2);\n    assert(um2.count(17*i) == i % 2);\n    if (cm2.count(17*i)) {\n      assert(cm2.find(17*i)->second == i);\n    }\n  }\n\n  assert(cm2.size() == NINS / 2);\n  assert(um2.size() == NINS / 2);\n\n  typedef graphlab::hopscotch_map::value_type vpair;\n  {\n    size_t cnt = 0;\n    foreach(vpair &v, cm2) {\n      ASSERT_EQ(v.second, um2[v.first]);\n      ++cnt;\n    }\n    ASSERT_EQ(cnt, NINS / 2);\n  }\n  {\n    size_t cnt = 0;\n    foreach(const vpair &v, cm2) {\n      ASSERT_EQ(v.second, um2[v.first]);\n      ++cnt;\n    }\n    ASSERT_EQ(cnt, NINS / 2);\n  }\n\n  std::stringstream strm;\n  graphlab::oarchive oarc(strm);\n  oarc << cm2;\n  strm.flush();\n\n  cm2.clear();\n  ASSERT_EQ(cm2.size(), 0);\n  graphlab::iarchive iarc(strm);\n  iarc >> cm2;\n  ASSERT_EQ(cm2.size(), NINS / 2);\n\n}\n\n\n\n\n\n\n\n\n\n\nvoid benchmark() {\n  graphlab::timer ti;\n\n  size_t NUM_ELS = 10000000;\n\n  std::vector v;\n  uint32_t u = 0;\n  for (size_t i = 0;i < NUM_ELS; ++i) {\n    v.push_back(u);\n    u += 1 + rand() % 8;\n  }\n  std::random_shuffle(v.begin(), v.end());\n  graphlab::memory_info::print_usage();\n\n  {\n    boost::unordered_map um;\n    ti.start();\n    for (size_t i = 0;i < NUM_ELS; ++i) {\n      um[v[i]] = i;\n    }\n    std::cout <<  NUM_ELS / 1000000 << \"M unordered map inserts in \" << ti.current_time() << \" (Load factor = \" << um.load_factor() << \")\" << std::endl;\n\n    graphlab::memory_info::print_usage();\n\n    ti.start();\n    for (size_t i = 0;i < 10000000; ++i) {\n      size_t t = um[v[i]];\n      assert(t == i);\n    }\n    std::cout << \"10M unordered map successful probes in \" << ti.current_time() << std::endl;\n    um.clear();\n  }\n\n  {\n    graphlab::cuckoo_map_pow2 cm(-1, 128);\n\n    //cm.reserve(102400);\n    ti.start();\n    for (size_t i = 0;i < NUM_ELS; ++i) {\n      cm[v[i]] = i;\n  //    if (i % 1000000 == 0) std::cout << cm.load_factor() << std::endl;\n\n    }\n    std::cout << NUM_ELS / 1000000 << \"M cuckoo map pow2 inserts in \" << ti.current_time() << \" (Load factor = \" << cm.load_factor() << \")\" << std::endl;\n\n    graphlab::memory_info::print_usage();\n\n    ti.start();\n    for (size_t i = 0;i < 10000000; ++i) {\n      size_t t = cm[v[i]];\n      assert(t == i);\n    }\n    std::cout << \"10M cuckoo map pow2 successful probes in \" << ti.current_time() << std::endl;\n  }\n\n{\n    graphlab::hopscotch_map cm;\n    ti.start();\n    for (size_t i = 0;i < NUM_ELS; ++i) {\n      cm[v[i]] = i;\n//      if (i % 1000000 == 0) std::cout << cm.load_factor() << std::endl;\n\n    }\n    std::cout << NUM_ELS / 1000000 << \"M hopscotch inserts in \" << ti.current_time() << \" (Load factor = \" << cm.load_factor() << \")\" << std::endl;\n\n    graphlab::memory_info::print_usage();\n\n    ti.start();\n    for (size_t i = 0;i < 10000000; ++i) {\n      size_t t = cm[v[i]];\n      assert(t == i);\n    }\n    std::cout << \"10M hopscotch successful probes in \" << ti.current_time() << std::endl;\n\n  }\n}\n\n\n\nint main(int argc, char** argv) {\n  std::cout << \"Hopscotch Map Sanity Checks... \\n\";\n  hopscotch_map_sanity_checks();\n\n  std::cout << \"Hopscotch High Collision Sanity Checks... \\n\";\n  hopscotch_high_collision_sanity_checks();\n\n  std::cout << \"Map Benchmarks... \\n\";\n  benchmark();\n  std::cout << \"Done\" << std::endl;\n}\n"
  },
  {
    "path": "tests/local_graph_test.cxx",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n// standard C++ headers\n#include \n#include \n\n// includes the entire graphlab framework\n#include \n#include \n#include \n#include \n\n/**\n * Unit test for graphlab::local_graph.hpp\n */\nclass local_graph_test : public CxxTest::TestSuite {\npublic:\n  struct vertex_data {\n    size_t value;\n    vertex_data() : value(0) { }\n    vertex_data(size_t n) : value(n) { }\n  };\n\n  struct edge_data { \n    int from; \n    int to;\n    edge_data (int f = 0, int t = 0) : from(f), to(t) {}\n  };\n\n  /**\n   * Test add vertex and add edges\n   */\n  void test_add_vertex() {\n    graphlab::local_graph g;\n    test_add_vertex_impl(g, 100);\n    test_add_vertex_impl(g, 10000);\n    test_add_vertex_impl(g, 100000);\n    std::cout << \"\\n+ Pass test: graph add vertex. :) \\n\";\n\n    graphlab::dynamic_local_graph g2;\n    test_add_vertex_impl(g2, 100);\n    test_add_vertex_impl(g2, 10000);\n    test_add_vertex_impl(g2, 100000);\n    std::cout << \"\\n+ Pass test: dynamic graph add vertex. :) \\n\";\n  }\n\n  void test_add_edge() {\n    graphlab::local_graph g;\n    test_add_edge_impl(g, 100);\n    test_add_edge_impl(g, 10000);\n    test_add_edge_impl(g, 100000);\n    std::cout << \"\\n+ Pass test: graph add edge. :) \\n\";\n    graphlab::dynamic_local_graph g2;\n    test_add_edge_impl(g2, 100);\n    test_add_edge_impl(g2, 10000);\n    test_add_edge_impl(g2, 100000);\n      std::cout << \"\\n+ Pass test: dynamic graph add edge. :) \\n\";\n  }\n\n  void test_dynamic_add_edge() {\n    graphlab::dynamic_local_graph g2;\n    test_add_edge_impl(g2, 100, true); // add edge dynamically\n    test_add_edge_impl(g2, 10000, true);\n    test_add_edge_impl(g2, 100000, true);\n    std::cout << \"\\n+ Pass test: graph dynamicly add edge. :) \\n\";\n  }\n\n  void test_powerlaw_graph() {\n    graphlab::local_graph g;\n    graphlab::dynamic_local_graph g2;\n    test_powerlaw_graph_impl(g, 100); // add edge (powerlaw) \n    test_powerlaw_graph_impl(g, 10000);\n\n    test_powerlaw_graph_impl(g2, 100); // add edge (powerlaw) \n    test_powerlaw_graph_impl(g2, 10000);\n\n    test_powerlaw_graph_impl(g2, 100, true); // add edge (powerlaw) dynamically\n    test_powerlaw_graph_impl(g2, 10000, true);\n    std::cout << \"\\n+ Pass test: powerlaw graph add edge. :) \\n\";\n  }\n\n  void test_edge_case() {\n    graphlab::local_graph g;\n    test_edge_case_impl(g);\n    std::cout << \"\\n+ Pass test: edge case test. :) \\n\";\n\n    graphlab::dynamic_local_graph g2;\n    test_edge_case_impl(g2);\n    std::cout << \"\\n+ Pass test: dynamic graph edge case test. :) \\n\";\n  }\n\n\n  void test_sparse_graph() {\n    graphlab::local_graph g;\n    test_sparse_graph_impl(g);\n    std::cout << \"\\n+ Pass test: sparse graph test. :) \\n\";\n\n    graphlab::dynamic_local_graph g2;\n    test_sparse_graph_impl(g2);\n    std::cout << \"\\n+ Pass test: sparse dyanmic graph test. :) \\n\";\n  }\n\n  void test_grid_graph() {\n    graphlab::local_graph g;\n    test_grid_graph_impl(g);\n    std::cout << \"\\n+ Pass test: grid graph test. :) \\n\";\n\n    graphlab::dynamic_local_graph g2;\n    test_grid_graph_impl(g2);\n    std::cout << \"\\n+ Pass test: grid dynamic graph test. :) \\n\";\n  }\n\nprivate: \n  template\n  void test_add_vertex_impl(Graph& g, size_t nverts) {\n    g.clear();\n    ASSERT_EQ(g.num_vertices(), 0);\n    for (size_t i = 0; i < nverts; ++i) {\n      g.add_vertex(i, vertex_data(i));\n    }\n    ASSERT_EQ(g.num_vertices(), nverts); \n    for (size_t i = 0; i < g.num_vertices(); ++i) {\n      ASSERT_EQ(g.vertex(i).data().value, i);\n    }\n    g.finalize();\n    ASSERT_EQ(g.num_vertices(), nverts);\n\n    // graph should still support adding vertices after finalization\n    // add more vertices and override existing vertex values\n    for (size_t i = 0; i < 2*nverts; ++i) {\n      g.add_vertex(i, vertex_data(i*2));\n    }\n    ASSERT_EQ(g.num_vertices(), 2*nverts);\n    for (size_t i = 0; i < g.num_vertices(); ++i) {\n      ASSERT_EQ(g.vertex(i).data().value, 2*i);\n    }\n  }\n\n  /**\n   * Helper function to check the in/out edges of the graph.\n   */\n  template\n  void check_adjacency(Graph& g, \n                       boost::unordered_map >& in_edges,\n                       boost::unordered_map >& out_edges,\n                       size_t nedges) {\n    typedef typename Graph::edge_list_type edge_list_type;\n    typedef typename Graph::edge_type edge_type;\n    typedef typename Graph::vertex_type vertex_type;\n    typedef typename Graph::vertex_id_type vertex_id_type;\n\n    // check size \n    ASSERT_EQ(g.num_edges(), nedges);\n\n    size_t nedges_actual = 0;\n    // check out edges\n    typedef typename boost::unordered_map >::iterator iter_type;\n    for (iter_type it = out_edges.begin(); it != out_edges.end(); ++it) {\n      vertex_id_type src = it->first;\n      std::set dst_expected = std::set(it->second.begin(), it->second.end());\n      const edge_list_type& ls = g.out_edges(src);\n      foreach (const edge_type& e, ls) {\n        ASSERT_EQ(e.source().id(), src);\n        ASSERT_TRUE(dst_expected.count(e.target().id()) == 1);\n        dst_expected.erase(e.target().id());\n      }\n      nedges_actual += ls.size();\n    }\n    ASSERT_EQ(nedges_actual, g.num_edges());\n    ASSERT_EQ(nedges_actual, nedges);\n\n    nedges_actual = 0;\n    // check in edges\n    for (iter_type it = in_edges.begin(); it != in_edges.end(); ++it) {\n      vertex_id_type dst = it->first;\n      std::set src_expected = std::set(it->second.begin(), it->second.end());\n      const edge_list_type& ls = g.in_edges(dst);\n      foreach (const edge_type& e, ls) {\n        ASSERT_EQ(e.target().id(), dst);\n        ASSERT_TRUE(src_expected.count(e.source().id()) == 1);\n        src_expected.erase(e.source().id());\n      }\n      nedges_actual += ls.size();\n    }\n    ASSERT_EQ(nedges_actual, g.num_edges());\n    ASSERT_EQ(nedges_actual, nedges);\n  }\n\ntemplate\n  void check_edge_data(Graph& g) {\n      typedef typename Graph::edge_list_type edge_list_type;\n      typedef typename Graph::edge_type edge_type;\n      typedef typename Graph::vertex_type vertex_type;\n      typedef typename Graph::vertex_id_type vertex_id_type;\n    for (size_t i = 0; i < g.num_vertices(); ++i) {\n      const edge_list_type& in_edges = g.in_edges(i);\n      foreach (const edge_type& e, in_edges) {\n        ASSERT_EQ(e.data().from, e.source().id());\n        ASSERT_EQ(e.data().to, e.target().id());\n      }\n      const edge_list_type& out_edges = g.out_edges(i);\n      foreach (const edge_type& e, out_edges) {\n        ASSERT_EQ(e.data().from, e.source().id());\n        ASSERT_EQ(e.data().to, e.target().id());\n      }\n    }\n  } \n\n  template\n  void test_add_edge_impl(Graph& g, size_t nedges, bool use_dynamic=false) {\n    typedef typename Graph::vertex_id_type vertex_id_type;\n    srand(0);\n    g.clear();\n    ASSERT_EQ(g.num_edges(), 0);\n    boost::unordered_map > out_edges;\n    boost::unordered_map > in_edges;\n    boost::unordered_set< std::pair > all_edges;\n    while (all_edges.size() < nedges) {\n      vertex_id_type src = rand() % (int)(3*sqrt(nedges));\n      vertex_id_type dst = rand() % (int)(3*sqrt(nedges));\n      if (src == dst)\n        continue;\n      std::pair pair(src, dst);\n      if (!all_edges.count(pair))  {\n        all_edges.insert(pair);\n        if (!out_edges.count(src)) {\n          out_edges[src] = std::vector();\n        } \n        if (!in_edges.count(dst)) {\n          in_edges[dst] = std::vector();\n        }\n        in_edges[dst].push_back(src);\n        out_edges[src].push_back(dst);\n      }\n    }\n    typedef typename boost::unordered_set< std::pair >::value_type pair_type; \n    size_t count = 0;\n    foreach (const pair_type& p, all_edges) {\n      g.add_edge(p.first, p.second, edge_data(p.first, p.second));\n      ++count;\n      if (use_dynamic && (all_edges.size()/5) == 0) {\n        g.finalize();\n      }\n    }\n    if (!use_dynamic)\n      ASSERT_EQ(g.num_edges(), 0); \n    g.finalize();\n    check_adjacency(g, in_edges, out_edges, all_edges.size());\n    check_edge_data(g);\n  }\n\n  \n  template\n  void test_edge_case_impl(Graph& g) {\n    // TODO: \n    // self edges\n    // duplicate edges\n    std::cout << \"Warning: test not implemented\" << std::endl;\n  }\n\n  /**\n   * Construct a star like sparse graph and test the in/out neighbors.\n   */\n  template\n  void test_sparse_graph_impl (Graph& g) {\n    typedef typename Graph::edge_list_type edge_list_type;\n    typedef typename Graph::edge_type edge_type;\n    typedef typename Graph::vertex_type vertex_type;\n    typedef typename Graph::vertex_id_type vertex_id_type;\n\n    size_t num_v = 10;\n    size_t num_e = 6;\n\n    for (size_t i = 0; i < num_v; ++i) {\n      vertex_data vdata;\n      g.add_vertex(vertex_id_type(i), vdata);\n    }\n\n    /**\n     * Create a star graph.\n     */\n    g.add_edge(1,3,edge_data(1,3));\n    g.add_edge(2,3,edge_data(2,3));\n    g.add_edge(4,3,edge_data(4,3));\n    g.add_edge(5,3,edge_data(5,3));\n    g.add_edge(3,2, edge_data(3,2));\n    g.add_edge(3,5, edge_data(3,5));\n    g.finalize();\n\n    ASSERT_EQ(g.num_vertices(), num_v);\n    ASSERT_EQ(g.num_edges(), num_e);\n\n    /**\n     * Test number of in/out edges.\n     */\n    for (vertex_id_type i = 0; i < 6; ++i) {\n      edge_list_type inedges = g.in_edges(i);\n      edge_list_type outedges = g.out_edges(i);\n      size_t arr_insize[] = {0,0,1,4,0,1};\n      size_t arr_outsize[] = {0,1,1,2,1,1};\n      if (i != 3) {\n        ASSERT_EQ(inedges.size(), arr_insize[i]);\n        ASSERT_EQ(outedges.size(), arr_outsize[i]);\n        if (outedges.size() > 0)\n          {\n            ASSERT_EQ(outedges[0].source().id(), i);\n            ASSERT_EQ(outedges[0].target().id(), 3);\n            edge_data data = (outedges[0]).data();\n            ASSERT_EQ(data.from, i);\n            ASSERT_EQ(data.to, 3);\n          }\n      } else {\n        std::set out_neighbors;\n        out_neighbors.insert(5);\n        out_neighbors.insert(2);\n        ASSERT_EQ(outedges.size(), out_neighbors.size());\n        for (size_t j = 0; j < 2; ++j) {\n          edge_data data = (outedges[j]).data();\n          ASSERT_EQ(data.from, 3);\n          ASSERT_TRUE(out_neighbors.count(data.to) == 1);\n          out_neighbors.erase(data.to);\n        }\n\n        std::set in_neighbors;\n        in_neighbors.insert(5);\n        in_neighbors.insert(4);\n        in_neighbors.insert(2);\n        in_neighbors.insert(1);\n        ASSERT_EQ(inedges.size(), in_neighbors.size());\n        for (size_t j = 0; j < 4; ++j) {\n          edge_data data = (inedges[j]).data();\n          ASSERT_EQ(data.to, 3);\n          ASSERT_TRUE(in_neighbors.count(data.from) == 1);\n          in_neighbors.erase(data.from);\n        }\n      }\n    }\n\n    for (vertex_id_type i = 6; i < num_v; ++i) {\n      edge_list_type inedges = g.in_edges(i);\n      edge_list_type outedges = g.out_edges(i);\n      ASSERT_EQ(0, inedges.size());\n      ASSERT_EQ(0, outedges.size());\n    }\n  }\n\n  /**\n     In this function, we construct the 3 by 3 grid graph.\n  */\n  template\n  void test_grid_graph_impl(Graph& g, bool verbose = false) {\n    typedef typename Graph::edge_list_type edge_list_type;\n    typedef typename Graph::edge_type edge_type;\n    typedef typename Graph::vertex_type vertex_type;\n    typedef typename Graph::vertex_id_type vertex_id_type;\n\n    g.clear();\n    if (verbose) \n      std::cout << \"-----------Begin Grid Test: ID Accessors--------------------\" << std::endl;\n    size_t dim = 3;\n    size_t num_vertices = 0;\n    size_t num_edge = 0;\n\n\n    // here we create dim * dim vertices.\n    for (size_t i = 0; i < dim * dim; ++i) {\n      // create the vertex data, randomizing the color\n      vertex_data vdata;\n      vdata.value = 0;\n      // create the vertex\n      g.add_vertex(vertex_id_type(i), vdata);\n      ++num_vertices;\n    }\n\n    // create the edges. The add_edge(i,j,edgedata) function creates\n    // an edge from i->j. with the edgedata attached.   edge_data edata;\n\n    for (size_t i = 0;i < dim; ++i) {\n      for (size_t j = 0;j < dim - 1; ++j) {\n        // add the horizontal edges in both directions\n        //\n        g.add_edge(dim * i + j, dim * i + j + 1, edge_data(dim*i+j, dim*i+j+1));\n        g.add_edge(dim * i + j + 1, dim * i + j, edge_data(dim*i+j+1, dim*i+j));\n\n        // add the vertical edges in both directions\n        g.add_edge(dim * j + i, dim * (j + 1) + i, edge_data(dim*j+i, dim*(j+1)+i));\n        g.add_edge(dim * (j + 1) + i, dim * j + i, edge_data(dim*(j+1)+i, dim*j+i));\n        num_edge += 4;\n      }\n    }\n\n    // the graph is now constructed\n    // we need to call finalize. \n    g.finalize();\n\n    if (verbose) printf(\"Test num_vertices()...\\n\");\n    ASSERT_EQ(g.num_vertices(), num_vertices);\n    if (verbose) printf(\"+ Pass test: num_vertices :)\\n\\n\");\n\n    if (verbose) printf(\"Test num_edges()...\\n\");\n    ASSERT_EQ(g.num_edges(), num_edge);\n    if (verbose) printf(\"+ Pass test: num_edges :)\\n\\n\");\n\n    // Symmetric graph: #inneighbor == outneighbor\n    if (verbose) printf(\"Test num_in_neighbors() == num_out_neighbors() ...\\n\");\n    for (size_t i = 0; i < num_vertices; ++i) {\n      ASSERT_EQ(g.in_edges(i).size(), g.vertex(i).num_in_edges());\n      ASSERT_EQ(g.out_edges(i).size(), g.vertex(i).num_out_edges());\n      ASSERT_EQ(g.in_edges(i).size(), g.out_edges(i).size());\n    }\n    ASSERT_EQ(g.in_edges(4).size(), 4);\n    ASSERT_EQ(g.in_edges(0).size(), 2);\n    if (verbose) printf(\"+ Pass test: #in = #out...\\n\\n\");\n\n\n    if (verbose) \n      printf(\"Test iterate over in/out_edges and get edge data: \\n\");\n    for (vertex_id_type i = 0; i < num_vertices; ++i) {\n      const edge_list_type& out_edges = g.out_edges(i);\n      const edge_list_type& in_edges = g.in_edges(i);\n\n      if (verbose) {\n        std::cout << \"Test v: \" << i << \"\\n\"\n                  << \"In edge ids: \";\n        foreach(edge_type edge, in_edges) \n            std::cout << \"(\" << edge.data().from << \",\"\n                      << edge.data().to << \") \";\n        std::cout <\n  void test_powerlaw_graph_impl(Graph& g, size_t nverts, bool use_dynamic = false, double alpha = 2.1) {\n    graphlab::random::seed(0);\n    g.clear();\n    typedef typename Graph::edge_list_type edge_list_type;\n    typedef typename Graph::edge_type edge_type;\n    typedef typename Graph::vertex_type vertex_type;\n    typedef typename Graph::vertex_id_type vertex_id_type;\n\n    boost::unordered_map > out_edges;\n    boost::unordered_map > in_edges;\n    boost::unordered_set< std::pair > all_edges;\n\n      // construct powerlaw out degree distribution \n      std::vector prob(nverts, 0);\n      for(size_t i = 0; i < prob.size(); ++i)\n        prob[i] = std::pow(double(i+1), -alpha);\n      graphlab::random::pdf2cdf(prob);\n\n      vertex_id_type dst = 0;\n\n      // A large prime number\n      const size_t HASH_OFFSET = 2654435761;\n      // construct powerlaw graph with no dup edges\n      for(vertex_id_type src  = 0; src < nverts; ++src) {\n        const size_t out_degree = graphlab::random::multinomial_cdf(prob) + 1;\n        for(size_t i = 0; i < out_degree; ++i) {\n          dst = (dst + HASH_OFFSET)  % nverts;\n          while (src == dst) {\n            dst = (dst + HASH_OFFSET)  % nverts;\n          }\n          std::pair pair(src, dst);\n          if (!all_edges.count(pair))  {\n            all_edges.insert(pair);\n            if (!out_edges.count(src)) {\n              out_edges[src] = std::vector();\n            } \n            if (!in_edges.count(dst)) {\n              in_edges[dst] = std::vector();\n            }\n            in_edges[dst].push_back(src);\n            out_edges[src].push_back(dst);\n          }\n        }\n      }\n\n    typedef typename boost::unordered_set< std::pair >::value_type pair_type; \n    size_t count = 0;\n    foreach (const pair_type& p, all_edges) {\n      g.add_edge(p.first, p.second, edge_data(p.first, p.second));\n      ++count;\n      if (use_dynamic && count % (all_edges.size()/5) == 0) {\n        g.finalize();\n      }\n    }\n    if (!use_dynamic)\n      ASSERT_EQ(g.num_edges(), 0); \n    g.finalize();\n    check_adjacency(g, in_edges, out_edges, all_edges.size());\n    check_edge_data(g);\n  }\n};\n\n#include \n"
  },
  {
    "path": "tests/lock_free_pushback.cxx",
    "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \nusing namespace graphlab;\n\nstd::vector vec;\nlockfree_push_back > pusher(vec, 0);\n\nvoid testthread(size_t range0, size_t range1) {\n  for (size_t i = range0;i < range1; ++i) {\n    pusher.push_back(i);\n  }\n}\n\nclass LockFreePushBack : public CxxTest::TestSuite {\npublic:\n\n  void test_lockfree_push_back(void) {\n    thread_group thr;\n    for (size_t i = 0;i < 16; ++i) {\n      thr.launch(boost::bind(testthread, i * 100000, (i+1) * 100000));\n    }\n\n    thr.join();\n\n    TS_ASSERT_EQUALS(pusher.size(), (size_t)16 * 100000);\n    vec.resize(pusher.size());\n    std::sort(vec.begin(), vec.end());\n    for (size_t i = 0;i < vec.size(); ++i) {\n      TS_ASSERT_EQUALS(vec[i], i);\n    }\n  }\n};\n\n"
  },
  {
    "path": "tests/mini_web_server.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\n#include \n\nstatic void *callback(enum mg_event event,\n                      struct mg_connection *conn,\n                      const struct mg_request_info *request_info) {\n  if (event == MG_NEW_REQUEST) {\n    assert(request_info != NULL);\n    \n    const std::string url = (request_info->uri == NULL)?\n      std::string(\"/\") : std::string(request_info->uri) ;\n    const std::string query = (request_info->query_string == NULL)?\n      std::string(\"\") : std::string(request_info->query_string) ;\n\n    std::string response = \"
URL: (\" + url + \")
 
    \";\n\n \n\n    std::map map = graphlab::web_util::parse_query(query);\n    typedef std::map::value_type pair_type;\n    foreach(pair_type pair, map) \n      response += \"
  •  \" + pair.first + \" -- \" + pair.second + \"
    • \";\n    response += \"
\";\n    \n\n\n    mg_printf(conn,\n              \"HTTP/1.1 200 OK\\r\\n\"\n              \"Content-Type: text/html\\r\\n\"\n              \"Content-Length: %d\\r\\n\"        // Always set Content-Length\n              \"\\r\\n\"\n              \"%s\",\n              int(response.size()), response.c_str());\n \n    // Mark as processed\n    return (void*)(1);\n  } else {\n    return NULL;\n  }\n}\n\nint main(void) {\n  struct mg_context *ctx;\n  const char *options[] = {\"listening_ports\", \"8080\", NULL};\n\n  ctx = mg_start(&callback, NULL, options);\n  getchar();  // Wait until user hits \"enter\"\n  mg_stop(ctx);\n\n  return 0;\n}\n\n\n\n\n\n\n\n\n\n\n//// Using lib event\n\n// #include \n// #include \n// #include \n// #include \n\n// #include \n// #include \n// #include \n\n// void generic_handler(struct evhttp_request *req, void *arg)\n// {\n//   struct evbuffer *buf;\n//   buf = evbuffer_new();\n  \n//   if (buf == NULL)\n//     err(1, \"failed to create response buffer\");\n//   for(size_t i = 0; i < 1000; ++i) {\n//     evbuffer_add_printf(buf, \"Requested: %s\\n\", evhttp_request_uri(req));\n//   }\n//   evhttp_send_reply(req, HTTP_OK, \"OK\", buf);\n// }\n\n// int main(int argc, char **argv)\n// {\n//     struct evhttp *httpd;\n\n//     event_init();\n//     httpd = evhttp_start(\"0.0.0.0\", 8080);\n\n//     /* Set a callback for requests to \"/specific\". */\n//     /* evhttp_set_cb(httpd, \"/specific\", another_handler, NULL); */\n\n//     /* Set a callback for all other requests. */\n//     evhttp_set_gencb(httpd, generic_handler, NULL);\n\n//     event_dispatch();\n\n//     /* Not reached in this code as it is now. */\n//     evhttp_free(httpd);\n\n//     return 0;\n// }\n"
  },
  {
    "path": "tests/random_test.cxx",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n\n\n#include \n\n#include \n#include \n#include \n\n#include \n\n\ntypedef double vertex_data_type;\ntypedef double edge_data_type;\n\n\n\ntemplate\nvoid uniform_speed(const size_t max_iter) {\n  NumType sum(0);\n  graphlab::timer ti;\n  ti.start();\n  for(size_t i = 0; i < max_iter; ++i) {\n    sum += (NumType)(graphlab::random::uniform(0, 10));\n  }\n  double slow_time = ti.current_time();\n  ti.start();\n  for(size_t i = 0; i < max_iter; ++i) {\n    sum += (NumType)(graphlab::random::fast_uniform(0, 10));\n  }\n  double fast_time = ti.current_time();\n  std::cout << slow_time << \", \" << fast_time << std::endl; \n}\n\n\nclass thread_worker {\npublic:\n  std::vector values;\n  void run() {\n    namespace random = graphlab::random;\n    for(size_t i = 0; i < values.size(); ++i) {\n      values[i] = random::uniform(0,3);\n    }\n  }\n};\n\ntemplate\nstd::ostream& operator<<(std::ostream& out, const std::vector& values) {\n  out << \"{\";\n  for(size_t i = 0; i < values.size(); ++i) {\n    out << values[i];\n    if(i + 1 < values.size()) out << \", \";\n  }\n  return out << \"}\";\n}\n\n\nstd::vector operator+(const std::vector& v1, \n                           const std::vector& v2) {\n  assert(v1.size() == v2.size());\n  std::vector result(v1.size());\n  for(size_t i = 0; i < result.size(); ++i) {\n    result[i] = v1[i] + v2[i];\n  }\n  return result;\n}\n\n\n\n\nclass RandomTestSuite: public CxxTest::TestSuite {\n  size_t iterations;\n  \n  public:\n\n  RandomTestSuite() : iterations(1E8) { }\n  \n \n  void test_nondet_generator() {\n    graphlab::random::nondet_seed();\n    graphlab::random::nondet_seed();\n    graphlab::random::nondet_seed();\n  }\n\n\n  void test_random_number_generators() {\n    std::cout << std::endl;\n    std::cout << \"beginning seed\" << std::endl;\n    namespace random = graphlab::random;\n    graphlab::random::seed();\n    graphlab::random::time_seed();\n    graphlab::random::nondet_seed();\n    graphlab::random::seed(12345);\n    std::cout << \"finished\" << std::endl;\n\n    const size_t num_iterations(20);\n    std::vector workers(10);\n    for(size_t i = 0; i < workers.size(); ++i) \n      workers[i].values.resize(num_iterations);\n    graphlab::thread_group threads;\n    for(size_t i = 0; i < workers.size(); ++i) {\n      threads.launch(boost::bind(&thread_worker::run, &(workers[i])));\n    }\n    threads.join();\n    for(size_t i = 0; i < workers.size(); ++i) {\n      std::cout << workers[i].values << std::endl;\n    }\n    std::vector sum(workers[0].values.size());\n    for(size_t i = 0; i < workers.size(); ++i) {\n      sum = sum + workers[i].values;\n    }\n    std::cout << \"Result: \" << sum << std::endl;\n  }\n\n\n\n\n\n  void test_shuffle() {\n    namespace random = graphlab::random;\n    random::nondet_seed();\n    std::vector numbers(100);\n    for(size_t i = 0; i < numbers.size(); ++i) numbers[i] = (int)i + 1;\n    for(size_t j = 0; j < 10; ++j) {\n      // shuffle the numbers\n      random::shuffle(numbers);\n      std::cout << numbers << std::endl;\n    }\n  }\n\n\n\n  // void test_speed() {\n  //   namespace random = graphlab::random;\n  //   std::cout << \"speed test run: \" << std::endl;\n  //   const size_t MAX_ITER(10000);\n  //   std::cout << \"size_t:   \"; \n  //   uniform_speed(MAX_ITER);\n  //   std::cout << \"int:      \"; \n  //   uniform_speed(MAX_ITER);\n  //   std::cout << \"uint32_t: \"; \n  //   uniform_speed(MAX_ITER);\n  //   std::cout << \"uint16_t: \"; \n  //   uniform_speed(MAX_ITER);\n  //   std::cout << \"char:     \"; \n  //   uniform_speed(MAX_ITER);\n  //   std::cout << \"float:    \"; \n  //   uniform_speed(MAX_ITER);\n  //   std::cout << \"double:   \"; \n  //   uniform_speed(MAX_ITER);\n    \n  //   std::cout << \"gaussian: \";\n  //   double sum = 0;\n  //   graphlab::timer time;\n  //   time.start();\n  //   for(size_t i = 0; i < MAX_ITER; ++i) \n  //     sum += random::gaussian();\n  //   std::cout << time.current_time() << std::endl;\n    \n  //   std::cout << \"shuffle:  \"; \n  //   std::vector numbers(6);\n  //   for(size_t i = 0; i < numbers.size(); ++i) numbers[i] = (int)i + 1;\n  //   time.start();\n  //   for(size_t j = 0; j < MAX_ITER/numbers.size(); ++j) {\n  //     // shuffle the numbers\n  //     random::shuffle(numbers);\n  //   }\n  //   std::cout << time.current_time() << \", \";\n  //   time.start();\n  //   for(size_t j = 0; j < MAX_ITER/numbers.size(); ++j) {\n  //     // shuffle the numbers\n  //     std::random_shuffle(numbers.begin(), numbers.end());\n  //   }\n  //   std::cout << time.current_time() << std::endl;    \n  // }\n\n\n  \n  \n};\n"
  },
  {
    "path": "tests/runtests.sh",
    "content": "#!/bin/bash\n\nfunction quit_if_bad_retvalue {\n  if [ $? -eq 0 ]; then\n    echo \"PASS\"\n  else\n    echo \"FAIL. Program returned with failure\"\n    exit 1\n  fi\n}\n\nfunction test_rpc_prog {\n  echo \"Testing $1 ...\"\n  echo \"---------$1-------------\" >> $stdoutfname\n  echo \"---------$1-------------\" >> $stderrfname \n  mpiexec -n 2 -host $localhostname ./$1  >> $stdoutfname 2>> $stderrfname\n  if [ $? -ne 0 ]; then\n    echo \"FAIL. Program returned with failure\"\n    exit 1\n  fi\n  str=\"mpiexec -n 2 -host $localhostname ./$1 2> /dev/null | grep \\\"$2\\\"\"\n  #echo $str\n  e=`eval $str`\n  if [ -z \"$e\" ] ; then\n    echo \"Expected program output not obtained\"\n    exit 1\n  fi\n}\n\nstdoutfname=$PWD/stdout.log\nstderrfname=$PWD/stderr.log\necho $PWD | grep debug > /dev/null\ndbgpath=$?\necho $PWD | grep release > /dev/null\nrelpath=$?\necho $PWD | grep profile > /dev/null\npropath=$?\n\nif [ $dbgpath -eq 1 ]; then\n  if [ $relpath -eq 1 ]; then\n    if [ $propath -eq 1 ]; then\n\techo \"This test must be run from either ./release/tests/, ./debug/tests/, or ./profile/tests/ in Graphlab root folder\"\n        echo \"Please compile GraphLab first, using the instructions on http://graphlab.org/download.html and try again from the approprite folder\"\n        exit 1\n    fi \n  fi\nfi\n\n\n\nrm -f $stdoutfname $stderrfname\n\nif [ $# -eq 0 ]; then\n\necho \"Running Standard unit tests\"\necho \"===========================\"\nctest -O testlog.txt\n./anytests\n./anytests_loader\n# delete extra generated files\nrm -f dg*\n\nfi\n\necho | tee -a $stdoutfname\necho \"Running application tests\"| tee -a $stdoutfname\necho \"=========================\"| tee -a $stdoutfname\necho \"GraphLab collaborative filtering library\"| tee -a $stdoutfname\nsomefailed=0\nif [ -f ../demoapps/pmf/pmf ]; then\n  pushd . > /dev/null\n  cd ../demoapps/pmf\n  echo \"---------PMF-------------\" >> $stdoutfname\n  OUTFILE=smalltest.out\n  ./pmf --show_version=true\n  if [ $? -eq 2 ]; then\n    echo \"detected Eigen based pmf\"| tee -a $stdoutfname\n    OUTFILE=smalltest_eigen.out\n  else\n    echo \"detected it++ based pmf\"| tee -a $stdoutfname\n  fi\n  echo \"********************TEST1************************\" >> $stdoutfname\n  ./pmf --unittest 1 --ncpus=1 --debug=true >> $stdoutfname 2>& 1\n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 1 (Alternating least squares)\"| tee -a $stdoutfname\n  else\n     somefailed=1\n     echo \"FAIL --unittest=1\"| tee -a $stdoutfname\n  fi\n  echo \"********************TEST2************************\" >> $stdoutfname\n  ./pmf --unittest 71 --ncpus=1 --debug=true >> $stdoutfname 2>& 1\n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 2 (Lanczos)\"| tee -a $stdoutfname\n  else\n     somefailed=1\n     echo \"FAIL --unittest=71 (Lanczos)\"| tee -a $stdoutfname\n  fi\n  echo \"********************TEST4************************\" >> $stdoutfname\n  ./pmf --unittest 91 --ncpus=1 --debug=true >> $stdoutfname 2>& 1\n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 3 (Weighted ALS)\"| tee -a $stdoutfname\n  else\n     somefailed=1\n     echo \"FAIL --unittest=91 (weighted alternating least squares)\"| tee -a $stdoutfname\n  fi\n  echo \"********************TEST5************************\" >> $stdoutfname\n ./pmf --unittest 101 --ncpus=1 >> $stdoutfname 2>& 1 \n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 4 (CoSaMP)\"| tee -a $stdoutfname\n  else\n     echo \"FAIL --unittest=101 (CoSaMP)\"| tee -a $stdoutfname\n     somefailed=1\n  fi\n  echo \"********************TEST6************************\" >> $stdoutfname\n ./pmf --unittest 131  >> $stdoutfname 2>& 1 \n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 5 (SVD)\"| tee -a $stdoutfname\n  else\n     echo \"FAIL --unittest=131 (SVD)\"| tee -a $stdoutfname\n     somefailed=1\n  fi\n  popd > /dev/null\n\nelse\n  echo \"PMF not found. \"| tee -a $stdoutfname\nfi\necho\n\n\n\necho \"GraphLab clustring library\"| tee -a $stdoutfname\nif [ -f ../demoapps/clustering/glcluster ]; then\n  pushd . > /dev/null\n  cd ../demoapps/clustering\n  echo \"---------CLUSTERING-------------\" >> $stdoutfname\n  echo \"---------CLUSTERING-------------\" >> $stderrfname\n  echo \"********************TEST1************************\" >> $stdoutfname\n  ./glcluster --unittest 1  $stdoutfname 2>& 1\n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 1 (Math functions)\"| tee -a $stdoutfname\n  else\n     somefailed=1\n     echo \"FAIL --unittest=1 (Math functions)\"| tee -a $stdoutfname\n  fi\n  echo \"********************TEST2************************\" >> $stdoutfname\n  ./glcluster --unittest 2 >> $stdoutfname 2>& 1\n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 2 (Distance functions)\"| tee -a $stdoutfname\n  else\n     somefailed=1\n     echo \"FAIL --unittest=2 (Distance functions)\"| tee -a $stdoutfname\n  fi\n  echo \"********************TEST3************************\" >> $stdoutfname\n  ./glcluster --unittest 4 >> $stdoutfname 2>& 1\n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 3 (Floating point math functions)\"| tee -a $stdoutfname\n  else\n     somefailed=1\n     echo \"FAIL --unittest=3 (Floating point math functions)\"| tee -a $stdoutfname\n  fi\n  popd  > /dev/null\nelse\n  echo \"Clustering library not found. \"| tee -a $stdoutfname\nfi\n \necho | tee -a $stdoutfname\necho \"GraphLab Linear Solvers Library\"| tee -a $stdoutfname\nif [ -f ../demoapps/gabp/gabp ]; then\n  pushd . > /dev/null\n  cd ../demoapps/gabp\n  echo \"---------GABP-------------\" >> $stdoutfname\n  echo \"********************TEST1************************\" >> $stdoutfname\n  ./gabp --unittest=1 --ncpus=1 --debug=true >> $stdoutfname 2>& 1\n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 1 (GaBP non-square)\"| tee -a $stdoutfname\n  else\n     somefailed=1\n     echo \"FAIL --unittest=1 (GabP non-square)\"| tee -a $stdoutfname\n  fi\n  echo \"********************TEST2************************\" >> $stdoutfname\n  ./gabp --unittest=2 --ncpus=1 --debug=true >> $stdoutfname 2>& 1\n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 2 (GaBP square)\"| tee -a $stdoutfname\n  else\n     somefailed=1\n     echo \"FAIL --unittest 2 (GaBP square)\" | tee -a $stdoutfname\n  fi\n  echo \"********************TEST3************************\" >> $stdoutfname\n  ./gabp --unittest=3 --ncpus=1 --debug=true >> $stdoutfname 2>& 1\n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 3 (Jacobi)\"| tee -a $stdoutfname\n  else\n     somefailed=1\n     echo \"FAIL --unittest=3 (Jacobi)\"| tee -a $stdoutfname\n  fi\n  echo \"********************TEST4************************\" >> $stdoutfname\n ./gabp --unittest=4 --ncpus=1 >> $stdoutfname 2>& 1 \n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 4 (Conjugate Gradient - square)\"| tee -a $stdoutfname\n  else\n     echo \"FAIL --unittest=4 (Conjugate Gradient - square)\"| tee -a $stdoutfname\n     somefailed=1\n  fi\n  echo \"********************TEST5************************\" >> $stdoutfname\n ./gabp --unittest=5  >> $stdoutfname 2>& 1 \n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 5 (Conjugate Gradient - non square)\"| tee -a $stdoutfname\n  else\n     echo \"FAIL --unittest=5 (Conjugate Gradient- non square)\"| tee -a $stdoutfname\n     somefailed=1\n  fi\n  echo \"********************TEST6************************\" >> $stdoutfname\n ./gabp --unittest=51  >> $stdoutfname 2>& 1 \n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 6 (Conjugate Gradient, matrix market format)\"| tee -a $stdoutfname\n  else\n     echo \"FAIL --unittest=6 (Conjugate Gradient- matrix market)\"| tee -a $stdoutfname\n     somefailed=1\n  fi\n  echo \"********************TEST7************************\" >> $stdoutfname\n ./gabp --unittest=21  >> $stdoutfname 2>& 1 \n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 7 (GaBP, matrix market format, square, regularization)\"| tee -a $stdoutfname\n  else\n     echo \"FAIL --unittest=7 (gaBP, matrix market format, square, regularization)\"| tee -a $stdoutfname\n     somefailed=1\n  fi\n   echo \"********************TEST8************************\" >> $stdoutfname\n ./gabp --unittest=22  >> $stdoutfname 2>& 1 \n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 8 (Jacobi, matrix market format, symmetric)\"| tee -a $stdoutfname\n  else\n     echo \"FAIL --unittest=8 (Jacobi, matrix market format, symmetric)\"| tee -a $stdoutfname\n     somefailed=1\n  fi\n   echo \"********************TEST9************************\" >> $stdoutfname\n ./gabp --unittest=23  >> $stdoutfname 2>& 1 \n  if [ $? -eq 0 ]; then\n     echo \"PASS TEST 9 (Conjugate Gradient, matrix market format, symmetric)\"| tee -a $stdoutfname\n  else\n     echo \"FAIL --unittest=9 (Conjugate Gradient, matrix market, symmetric)\"| tee -a $stdoutfname\n     somefailed=1\n  fi\n    popd > /dev/null\nelse\n  echo \"Linear solver library not found. \"| tee -a $stdoutfname\nfi\necho\n\n\n  if [ $somefailed == 1 ]; then\n     echo \"**** FAILURE LOG **************\" >> $stdoutfname\n     cat $stderrfname >> $stdoutfname\n     echo \"**** CONFIGURE.DEPS **************\" >> $stdoutfname\n     cat ../../configure.deps >> $stdoutfname\n     echo \"**** CONFIG.LOG **************\" >> $stdoutfname\n     cat ../../config.log >> $stdoutfname\n     echo \"**** SYSTEM STATS **************\" >> $stdoutfname\n     echo `date` >> $stdoutfname\n     echo `uname -a` >> $stdoutfname\n     echo `echo $USER` >> $stdoutfname\n     echo \"Some of the tests failed\".\n     echo \"Please email stdout.log to danny.bickson@gmail.com\"\n     echo \"Thanks for helping improve GraphLab!\"\n  fi\n\n\n\n\nif [ -f ../demoapps/demo/demo ]; then\n  pushd . > /dev/null\n  cd ../demoapps/demo\n  echo \"Demo...\"\n  echo \"---------demo-------------\" >> $stdoutfname\n  echo \"---------demo-------------\" >> $stderrfname\n  \n  ./demo  >> $stdoutfname 2>> $stderrfname\n  quit_if_bad_retvalue\n  popd > /dev/null\nelse\n  echo \"demo not found. \"\nfi\n\necho\necho \"RPC Tests\"\necho \"=========\"\necho \"Testing for availability of an MPI daemon\"\nlocalhostname=`hostname`\nmpdtrace\nif [ $? -eq 0 ]; then\n  echo \"MPI available\"\nelse\n  echo \"MPI not available. Distributed/RPC tests not running.\"\n  exit 1\nfi\n\n\n\ntest_rpc_prog rpc_example1 \"5 plus 1 is : 6\\\\|11 plus 1 is : 12\"\ntest_rpc_prog rpc_example2 \"hello world!\\\\|1, 2, 1,\"\ntest_rpc_prog rpc_example3 \"1.a = 10\\\\|10.b = 0\\\\|string = hello world!\"\ntest_rpc_prog rpc_example4 \"1.a = 10\\\\|10.b = 0\\\\|string = hello world!\"\ntest_rpc_prog rpc_example5 \"1 + 2.000000 = three\"\ntest_rpc_prog rpc_example6 \"10\\\\|15\\\\|hello world\\\\|10.5\\\\|10\"\ntest_rpc_prog rpc_example7 \"set from 1\\\\|set from 1\\\\|set from 0\\\\|set from 0\\\\|set from 1\\\\|set from 1\\\\|set from 0\\\\|set from 0\"\n\necho\necho \"Distributed GraphLab Tests\"\necho \"==========================\"\n\necho \"Testing Distributed disk graph construction...\"\necho \"---------distributed_dg_construction_test-------------\" >> $stdoutfname\necho \"---------distributed_dg_construction_test-------------\" >> $stderrfname \nmpiexec -n 2 -host $localhostname ./distributed_dg_construction_test >> $stdoutfname 2>> $stderrfname\nquit_if_bad_retvalue\nrm -f dg*\n\necho \"Testing Distributed Graph ...\"\necho \"---------distributed_graph_test-------------\" >> $stdoutfname\necho \"---------distributed_graph_test-------------\" >> $stderrfname \n./distributed_graph_test -g\nmpiexec -n 2 -host $localhostname ./distributed_graph_test -b >> $stdoutfname 2>> $stderrfname\nquit_if_bad_retvalue\nrm -f dg*\n"
  },
  {
    "path": "tests/scheduler_test.cxx",
    "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n\n\nusing namespace graphlab;\n\ndistributed_control dc;\n\nstruct message_type {\n  int value;\n  double pr;\n\n  explicit message_type(int value = 0, double pr = 0):value(value),pr(pr) { }\n  \n  double priority() const {\n    return pr;\n  }\n  message_type& operator+=(const message_type& other) {\n    value += other.value;\n    pr = other.value;\n    return *this;\n  }\n};\n\n\nconst size_t NCPUS = 4;\nconst size_t NUM_VERTICES = 101;\nstd::vector > correctness_counter;\n\n\ntemplate \nvoid test_scheduler_basic_functionality_single_threaded() {\n  graphlab_options opts;\n  opts.set_ncpus(NCPUS);\n  SchedulerType sched(NUM_VERTICES, opts);\n  const size_t target_value = 100;\n  \n  // inject a sequence of messages which will sum to 100 per vertex\n  for (size_t c = 0;c < target_value; ++c) {\n    for (size_t i = 0; i < NUM_VERTICES; ++i) {\n      sched.schedule(i, message_type(1, 1.0));\n    }\n  }\n  correctness_counter.clear();\n  correctness_counter.resize(NUM_VERTICES, atomic(0));\n  sched.start();\n  \n  // pull stuff out\n  bool allcpus_done = false; \n  while(!allcpus_done) {\n    allcpus_done = true;\n    for (size_t i = 0; i < NCPUS; ++i) {\n      vertex_id_type v; message_type m;\n      sched_status::status_enum ret = sched.get_next(i, v, m);\n      if (ret == sched_status::NEW_TASK) {\n        allcpus_done = false;\n        correctness_counter[v].inc(m.value);\n      }\n    }\n  }\n\n  // check the counters\n  for(size_t i = 0; i < NUM_VERTICES; ++i) {\n    TS_ASSERT_EQUALS(correctness_counter[i].value, (int)target_value);\n  }\n}\n\n\n\n\n\ntemplate \nvoid test_basic_functionality_thread(SchedulerType& sched, \n                                     async_consensus& consensus, \n                                     size_t schedule_count,\n                                     size_t threadid) {\n  size_t c = 0;\n  vertex_id_type v; message_type m;\n  while(1) {\n    // process as many tasks as I can\n    while(1) {\n      sched_status::status_enum ret = sched.get_next(threadid, v, m);\n      if (ret == sched_status::NEW_TASK) {\n        correctness_counter[v].inc(m.value);\n      }\n      else {\n        break;\n      }\n    }\n    \n    // schedule 1 cycle. If I schedule stuff I go back to processing tasks\n    if (c < schedule_count) {\n      for (size_t i = 0; i < NUM_VERTICES; ++i) {\n        sched.schedule(i, message_type(1, 1.0));\n        consensus.cancel();\n      }\n      ++c;\n      continue;\n    }\n    \n    // nothing to schedule, nothing to run. try to quit\n    consensus.begin_done_critical_section(threadid);\n    sched_status::status_enum ret = sched.get_next(threadid, v, m);\n    if (ret == sched_status::NEW_TASK) {\n      // there is task. cancel, process it, and look back\n      consensus.cancel_critical_section(threadid);\n      correctness_counter[v].inc(m.value);\n    }\n    else {\n      // no more tasks try to finish up\n      bool ret = consensus.end_done_critical_section(threadid);\n      if (ret) break;\n    }\n  }\n}\n\n\ntemplate \nvoid test_scheduler_basic_functionality_parallel() {\n  graphlab_options opts;\n  opts.set_ncpus(NCPUS);\n  SchedulerType sched(NUM_VERTICES, opts);\n  async_consensus consensus(dc, NCPUS);\n  \n  const size_t schedule_count = 10000;\n  const size_t target_value = schedule_count * NCPUS + 1;\n\n  correctness_counter.clear();\n  correctness_counter.resize(NUM_VERTICES, atomic(0));\n\n  // inject a sequence of messages which will sum to 100 per vertex\n  for (size_t i = 0; i < NUM_VERTICES; ++i) {\n    sched.schedule(i, message_type(1, 1.0));\n  }\n  sched.start();\n  \n  thread_group group;\n  for (size_t i = 0;i < NCPUS;++i) {\n    group.launch(boost::bind(test_basic_functionality_thread,\n                             boost::ref(sched), boost::ref(consensus), schedule_count, i));\n  }\n\n  group.join();\n  // check the counters\n  for(size_t i = 0; i < NUM_VERTICES; ++i) {\n    TS_ASSERT_EQUALS(correctness_counter[i].value, (int)target_value);\n  }\n}\n\n\n\n\n\n\n\n/*\n * Like test_basic_functionality_thread, but only increments the \n * correctness_counter by 1, and checks to make sure the priority is at least\n * 100.0\n */\ntemplate \nvoid test_scheduler_min_priority_thread(SchedulerType& sched, \n                                        async_consensus& consensus, \n                                        size_t schedule_count,\n                                        size_t threadid) {\n  size_t c = 0;\n  vertex_id_type v; message_type m;\n  while(1) {\n    // process as many tasks as I can\n    while(1) {\n      sched_status::status_enum ret = sched.get_next(threadid, v, m);\n      if (ret == sched_status::NEW_TASK) {\n        TS_ASSERT_LESS_THAN_EQUALS(100.0, m.priority());\n        correctness_counter[v].inc(1);\n      }\n      else {\n        break;\n      }\n    }\n    \n    // schedule 1 cycle. If I schedule stuff I go back to processing tasks\n    if (c < schedule_count) {\n      for (size_t i = 0; i < NUM_VERTICES; ++i) {\n        sched.schedule(i, message_type(1, 1.0));\n        consensus.cancel();\n      }\n      ++c;\n      continue;\n    }\n    \n    // nothing to schedule, nothing to run. try to quit\n    consensus.begin_done_critical_section(threadid);\n    sched_status::status_enum ret = sched.get_next(threadid, v, m);\n    if (ret == sched_status::NEW_TASK) {\n      TS_ASSERT_LESS_THAN_EQUALS(100.0, m.priority());\n      // there is task. cancel, process it, and look back\n      consensus.cancel_critical_section(threadid);\n      correctness_counter[v].inc(1);\n    }\n    else {\n      // no more tasks try to finish up\n      bool ret = consensus.end_done_critical_section(threadid);\n      if (ret) break;\n    }\n  }\n}\n\n\ntemplate \nvoid test_scheduler_min_priority_parallel() {\n  graphlab_options opts;\n  opts.set_ncpus(NCPUS);\n  opts.get_scheduler_args().set_option(\"min_priority\", 100.0);\n  \n  SchedulerType sched(NUM_VERTICES, opts);\n  async_consensus consensus(dc, NCPUS);\n  \n  const size_t schedule_count = 10000;\n  const size_t maximum_value = (size_t)((schedule_count * NCPUS + 101.0) / 100.0);\n\n  correctness_counter.clear();\n  correctness_counter.resize(NUM_VERTICES, atomic(0));\n\n  // inject a sequence of messages which will sum to 100 per vertex\n  for (size_t i = 0; i < NUM_VERTICES; ++i) {\n    sched.schedule(i, message_type(1, 101.0));\n  }\n  sched.start();\n  \n  thread_group group;\n  for (size_t i = 0;i < NCPUS;++i) {\n    group.launch(boost::bind(test_scheduler_min_priority_thread,\n                             boost::ref(sched), boost::ref(consensus), schedule_count, i));\n  }\n\n  group.join();\n  // check the counters\n  for(size_t i = 0; i < NUM_VERTICES; ++i) {\n    TS_ASSERT_LESS_THAN_EQUALS(correctness_counter[i].value, (int)maximum_value);\n  }\n}\n\nclass SerializeTestSuite : public CxxTest::TestSuite {\npublic:\n  void test_scheduler_basic_single_threaded() {\n    test_scheduler_basic_functionality_single_threaded >();\n    test_scheduler_basic_functionality_single_threaded >();\n    test_scheduler_basic_functionality_single_threaded >();\n    test_scheduler_basic_functionality_single_threaded >();\n  }\n  \n  void test_scheduler_basic_parallel() {\n    test_scheduler_basic_functionality_parallel >();\n    test_scheduler_basic_functionality_parallel >();\n    test_scheduler_basic_functionality_parallel >();\n    test_scheduler_basic_functionality_parallel >();\n  }\n  \n    \n  void test_scheduler_min_priority() {\n    test_scheduler_min_priority_parallel >();\n    test_scheduler_min_priority_parallel >();\n    test_scheduler_min_priority_parallel >();\n    test_scheduler_min_priority_parallel >();\n  }\n\n};\n\n"
  },
  {
    "path": "tests/serializetests.cxx",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n#include \n\n#include \n\n#include \n#include \n#include \n#include \n\n#include \n#include \n\n\nusing namespace graphlab;\n\n\n\nstruct A{\n  int z;\n  void save(oarchive &a) const {\n    a << z;\n  }\n  void load(iarchive &a) {\n    a >> z;\n  }\n};\n\nclass TestClass{\npublic:\n  int i;\n  int j;\n  std::vector k;\n  A l;\n  void save(oarchive &a) const {\n    a << i << j << k << l;\n  }\n  void load(iarchive &a) {\n    a >> i >> j >> k >> l;\n  }\n};\n\n\nstruct pod_class_1: public graphlab::IS_POD_TYPE {\n  size_t x;\n};\n\nstruct pod_class_2 {\n  size_t x;\n}; \nSERIALIZABLE_POD(pod_class_2);\n\n\nclass SerializeTestSuite : public CxxTest::TestSuite {\npublic:\n\n  // Look for the class TestClass() to see the most interesting tutorial on how to\n  // use the serializer\n  void test_basic_datatype(void) {\n    char t1 = 'z';\n    bool t2 = true;\n    int t3 = 10;\n    int t4 = 18345;\n    long t5 = 30921233;\n    long long t6 = (long long)(t5)*100;\n    float t7 = 10.35;\n    double t8 = 3.14156;\n    const char *t9 = \"hello world\";\n    const char * t10 = \"blue\";\n    graphlab::any t11;\n    t11 = size_t(10);\n    char r1;\n    bool r2;\n    int r3;\n    int r4;\n    long r5;\n    long long r6;\n    float r7;\n    double r8;\n    char r9[100];\n    char r10[10];\n    graphlab::any r11;\n\n    // serialize t1-10\n    std::ofstream f;\n    f.open(\"test.bin\",std::fstream::binary);\n    oarchive a(f);\n    a << t1 << t2 << t3 << t4 << t5 << t6 << t7 << t8;\n    serialize(a, t9, strlen(t9) + 1);\n    serialize(a, t10, strlen(t10) + 1);\n    a << t11;\n    f.close();\n\n    // deserialize into r1-10\n    std::ifstream g;\n    g.open(\"test.bin\",std::fstream::binary);\n    iarchive b(g);\n    b >> r1 >> r2 >> r3 >> r4 >> r5 >> r6 >> r7 >> r8;\n    deserialize(b, &r9, strlen(t9) + 1);\n    deserialize(b, r10, strlen(t10) + 1);\n    b >> r11;\n    g.close();\n\n    TS_ASSERT_EQUALS(t1, r1);\n    TS_ASSERT_EQUALS(t2, r2);\n    TS_ASSERT_EQUALS(t3, r3);\n    TS_ASSERT_EQUALS(t4, r4);\n    TS_ASSERT_EQUALS(t5, r5);\n    TS_ASSERT_EQUALS(t6, r6);\n    TS_ASSERT_EQUALS(t7, r7);\n    TS_ASSERT_EQUALS(t8, r8);\n    TS_ASSERT_SAME_DATA(t9, r9, strlen(t9) + 1);\n    TS_ASSERT_SAME_DATA(t10, r10, strlen(t10) + 1);\n    TS_ASSERT_EQUALS(r11.as(), t11.as());\n  }\n\n  void test_vector_serialization(void) {\n    std::vector v;\n    for (int i = 0;i< 10; ++i) {\n      v.push_back(i);\n    }\n    std::ofstream f;\n    f.open(\"test.bin\",std::fstream::binary);\n    oarchive a(f);\n    a << v;\n    f.close();\n\n    std::vector w;\n    std::ifstream g;\n    iarchive b(g);\n    g.open(\"test.bin\",std::fstream::binary);\n    b >> w;\n    g.close();\n\n    for (int i = 0;i< 10; ++i) {\n      TS_ASSERT_EQUALS(v[i], w[i]);\n    }\n  }\n\n\n  void test_class_serialization(void) {\n    // create a test class\n    TestClass t;\n    t.i=10;\n    t.j=20;\n    t.k.push_back(30);\n\n    //serialize\n    std::ofstream f;\n    f.open(\"test.bin\",std::fstream::binary);\n    oarchive a(f);\n    a << t;\n    f.close();\n    //deserialize into t2\n    TestClass t2;\n    std::ifstream g;\n    g.open(\"test.bin\",std::fstream::binary);\n    iarchive b(g);\n    b >> t2;\n    g.close();\n    // check\n    TS_ASSERT_EQUALS(t.i, t2.i);\n    TS_ASSERT_EQUALS(t.j, t2.j);\n    TS_ASSERT_EQUALS(t.k.size(), t2.k.size());\n    TS_ASSERT_EQUALS(t.k[0], t2.k[0]);\n  }\n\n  void test_vector_of_classes(void) {\n    // create a vector of test classes\n    std::vector vt;\n    vt.resize(10);\n    for (int i=0;i<10;i++) {\n      vt[i].i=i;\n      vt[i].j=i*21;\n      vt[i].k.resize(10);\n      vt[i].k[i]=i*51;\n    }\n\n    //serialize\n    std::ofstream f;\n    f.open(\"test.bin\",std::fstream::binary);\n    oarchive a(f);\n    a << vt;\n    f.close();\n\n    //deserialize into vt2\n    std::vector vt2;\n    std::ifstream g;\n    g.open(\"test.bin\",std::fstream::binary);\n    iarchive b(g);\n    b >> vt2;\n    g.close();\n    // check\n    TS_ASSERT_EQUALS(vt.size(), vt2.size());\n    for (size_t i=0;i<10;i++) {\n      TS_ASSERT_EQUALS(vt[i].i, vt2[i].i);\n      TS_ASSERT_EQUALS(vt[i].j, vt2[i].j);\n      TS_ASSERT_EQUALS(vt[i].k.size(), vt2[i].k.size());\n      for (size_t j = 0; j < vt[i].k.size(); ++j) {\n        TS_ASSERT_EQUALS(vt[i].k[j], vt2[i].k[j]);\n      }\n    }\n  }\n\n  void test_vector_of_strings(void) {\n    std::string x = \"Hello world\";\n    std::string y = \"This is a test\";\n    std::vector v;\n    v.push_back(x); v.push_back(y);\n\n    std::ofstream f;\n    f.open(\"test.bin\",std::fstream::binary);\n    oarchive a(f);\n    a << v;\n    f.close();\n\n    //deserialize into vt2\n    std::vector v2;\n    std::ifstream g;\n    g.open(\"test.bin\",std::fstream::binary);\n    iarchive b(g);\n    b >> v2;\n    g.close();\n    TS_ASSERT_EQUALS(v[0], v2[0]);\n    TS_ASSERT_EQUALS(v[1], v2[1]);\n  }\n\n  void test_map_serialization(void) {\n    std::map v;\n    v[\"one\"] = 1;\n    v[\"two\"] = 2;\n    v[\"three\"] = 3;\n\n    std::ofstream f;\n    f.open(\"test.bin\",std::fstream::binary);\n    oarchive a(f);\n    a << v;\n    f.close();\n\n    //deserialize into vt2\n    std::map v2;\n    std::ifstream g;\n    g.open(\"test.bin\",std::fstream::binary);\n    iarchive b(g);\n    b >> v2;\n    g.close();\n    TS_ASSERT_EQUALS(v[\"one\"], v2[\"one\"]);\n    TS_ASSERT_EQUALS(v[\"two\"], v2[\"two\"]);\n    TS_ASSERT_EQUALS(v[\"three\"], v2[\"three\"]);\n  }\n\n  void test_repeated_array_serialization(void) {\n    typedef std::map intmap;\n    std::vector buffer;\n    std::vector sizes(5);\n    std::cout << \"Making maps =====================================\" << std::endl;\n    for(size_t i = 0; i < sizes.size(); ++i) {\n      std::stringstream strm;\n      oarchive arc(strm);\n      intmap im;\n      im[i] = i;\n      im[10*i] = 10*i;\n      if(i % 2 == 0) im[i+sizes.size()] = 3;\n      for(intmap::const_iterator iter = im.begin(); iter != im.end(); ++iter) {\n        std::cout << \"[\" << iter->first << \", \" << iter->second << \"]\\t\";\n      }\n      std::cout << std::endl;\n      arc << im;\n      std::string str(strm.str());\n      sizes[i] = str.size();\n      int index = buffer.size();\n      buffer.resize(index + str.size());\n      memcpy(&buffer[index], str.c_str(), str.size());\n    }\n    std::cout << \"reading maps =====================================\" << std::endl;\n    namespace bio = boost::iostreams;\n    typedef bio::stream icharstream;\n  \n\n    \n    for(size_t i = 0, offset=0; i < sizes.size(); ++i) {\n      icharstream strm(&buffer[offset], sizes[i]);\n      offset += sizes[i];\n      intmap im;\n      iarchive arc(strm);\n      arc >> im;\n      for(intmap::const_iterator iter = im.begin(); iter != im.end(); ++iter) {\n        std::cout << \"[\" << iter->first << \", \" << iter->second << \"]\\t\";\n      }\n      std::cout << std::endl;\n    }\n\n  }\n  \n  void test_boost_unordered_map(void) {\n    boost::unordered_map m;\n    m[\"hello\"] = 1;\n    m[\"world\"] = 2;\n    std::ofstream f;\n    f.open(\"test.bin\",std::fstream::binary);\n    oarchive a(f);\n    a << m;\n    f.close();\n\n    boost::unordered_map m2;\n    std::ifstream g;\n    iarchive b(g);\n    g.open(\"test.bin\",std::fstream::binary);\n    b >> m2;\n    g.close();\n\n    TS_ASSERT_EQUALS(m[\"hello\"], m2[\"hello\"]);\n    TS_ASSERT_EQUALS(m[\"world\"], m2[\"world\"]);\n  }\n\n\n  void test_boost_unordered_set(void) {\n    boost::unordered_set m;\n    m.insert(\"hello\");\n    m.insert(\"world\");\n    std::ofstream f;\n    f.open(\"test.bin\",std::fstream::binary);\n    oarchive a(f);\n    a << m;\n    f.close();\n\n    boost::unordered_set m2;\n    std::ifstream g;\n    iarchive b(g);\n    g.open(\"test.bin\",std::fstream::binary);\n    b >> m2;\n    g.close();\n\n    TS_ASSERT(m2.find(\"hello\") != m2.end());\n    TS_ASSERT(m2.find(\"world\") != m2.end());\n  }\n  \n  void test_pod_method_1() {\n    std::vector p1;\n    for (size_t i = 0;i < 1000; ++i) {\n        pod_class_1 p;\n        p.x = i;\n        p1.push_back(p);\n    }\n    \n    std::ofstream f;\n    f.open(\"test.bin\",std::fstream::binary);\n    oarchive a(f);\n    a << p1;\n    f.close();\n\n    std::vector p2;\n    \n    std::ifstream g;\n    iarchive b(g);\n    g.open(\"test.bin\",std::fstream::binary);\n    b >> p2;\n    g.close();\n\n    for (size_t i = 0;i < 1000; ++i) {\n        TS_ASSERT_EQUALS(p1[i].x, p2[i].x);\n    }\n  }\n  \n    void test_pod_method_2() {\n    std::vector p1;\n    for (size_t i = 0;i < 1000; ++i) {\n        pod_class_2 p;\n        p.x = i;\n        p1.push_back(p);\n    }\n    \n    std::ofstream f;\n    f.open(\"test.bin\",std::fstream::binary);\n    oarchive a(f);\n    a << p1;\n    f.close();\n\n    std::vector p2;\n    \n    std::ifstream g;\n    iarchive b(g);\n    g.open(\"test.bin\",std::fstream::binary);\n    b >> p2;\n    g.close();\n\n    for (size_t i = 0;i < 1000; ++i) {\n        TS_ASSERT_EQUALS(p1[i].x, p2[i].x);\n    }\n  }\n};\n\n"
  },
  {
    "path": "tests/sfinae_function_test.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n\nstruct ts{\n  int i;\n};\n\n\nvoid by_value(ts) {\n}\n\nvoid by_const_value(const ts) {\n}\n\nvoid by_reference(ts&) {\n}\n\nvoid by_const_reference(const ts&) {\n}\n\n\nstruct functor_by_value{\n  void operator()(ts) { }\n};\n\nstruct functor_by_const_value{\n  void operator()(const ts) { }\n};\n\nstruct functor_by_reference{\n  void operator()(ts&) { }\n};\n\nstruct functor_by_const_reference{\n  void operator()(const ts&) { }\n};\n\n\nstruct const_functor_by_value{\n  void operator()(ts) const { }\n};\n\nstruct const_functor_by_const_value{\n  void operator()(const ts) const { }\n};\n\nstruct const_functor_by_reference{\n  void operator()(ts&) const { }\n};\n\nstruct const_functor_by_const_reference{\n  void operator()(const ts&) const { }\n};\n\n\n\nstruct overload_functor_by_value{\n  void operator()(ts) { }\n  void operator()(ts) const { }\n};\n\nstruct overload_functor_by_const_value{\n  void operator()(const ts) { }\n  void operator()(const ts) const { }\n};\n\nstruct overload_functor_by_reference{\n  void operator()(ts&) { }\n  void operator()(ts&) const { }\n};\n\nstruct overload_functor_by_const_reference{\n  void operator()(const ts&) { }\n  void operator()(const ts&) const { }\n};\n\n\n/*\n * Returns true if T is a function which matches void(const ts&)\n * or if T is a functor with a void operator()(const ts&) const\n */\ntemplate \nint test_function_is_const_ref(T t) {\n  return graphlab::test_function_or_const_functor_1::value;\n}\n\nint main(int argc, char** argv) {\n  std::cout << test_function_is_const_ref(by_value) << std::endl;\n  std::cout << test_function_is_const_ref(by_const_value) << std::endl;\n  std::cout << test_function_is_const_ref(by_reference) << std::endl;\n  std::cout << test_function_is_const_ref(by_const_reference) << std::endl;\n\n  std::cout << test_function_is_const_ref(functor_by_value()) << std::endl;\n  std::cout << test_function_is_const_ref(functor_by_const_value()) << std::endl;\n  std::cout << test_function_is_const_ref(functor_by_reference()) << std::endl;\n  std::cout << test_function_is_const_ref(functor_by_const_reference()) << std::endl;\n\n  std::cout << test_function_is_const_ref(const_functor_by_value()) << std::endl;\n  std::cout << test_function_is_const_ref(const_functor_by_const_value()) << std::endl;\n  std::cout << test_function_is_const_ref(const_functor_by_reference()) << std::endl;\n  std::cout << test_function_is_const_ref(const_functor_by_const_reference()) << std::endl;\n\n  std::cout << test_function_is_const_ref(overload_functor_by_value()) << std::endl;\n  std::cout << test_function_is_const_ref(overload_functor_by_const_value()) << std::endl;\n  std::cout << test_function_is_const_ref(overload_functor_by_reference()) << std::endl;\n  std::cout << test_function_is_const_ref(overload_functor_by_const_reference()) << std::endl;\n}"
  },
  {
    "path": "tests/small_map_test.cxx",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n\n#include \n\n#include \n\n#include \n#include \n\nusing namespace graphlab;\n\n#include \nclass test_small_map : public CxxTest::TestSuite {\npublic:\n\n  void test_lookup() {\n    typedef small_map<32, size_t, double> map_type;\n    map_type map;\n    std::map stdmap;\n    map[5] = stdmap[5] = 5.1;\n    map[1] = stdmap[1] = 1.1;\n    map[2] = stdmap[2] = 2.1;    \n    ASSERT_EQ(map.size(), stdmap.size());\n    std::cout << std::endl;\n    std::cout << map << std::endl;\n    typedef std::pair pair_type;\n    foreach(pair_type pair, stdmap) {\n      ASSERT_EQ(map[pair.first], pair.second);\n      ASSERT_EQ(map.safe_find(pair.first), pair.second);\n      ASSERT_TRUE(map.has_key(pair.first));\n    }\n    foreach(pair_type pair, map) {\n      ASSERT_EQ(stdmap[pair.first], pair.second);\n    }\n\n    map_type map2;\n    std::map stdmap2;\n    map2[0] = stdmap2[0] = 0.2;\n    map2[5] = stdmap2[5] = 5.2;\n    map2[2] = stdmap2[2] = 2.2;    \n    map2[1] = stdmap2[1] = 1.2;\n    map2[8] = stdmap2[8] = 8.2;\n    ASSERT_EQ(map2.size(), stdmap2.size());\n    map_type map3 = map + map2;\n    std::map stdmap3 = \n      graphlab::map_union(stdmap, stdmap2);\n\n    std::cout << map3 << std::endl;\n    \n    foreach(pair_type pair, stdmap3) {\n      ASSERT_EQ(map3[pair.first], pair.second);\n      ASSERT_EQ(map3.safe_find(pair.first), pair.second);\n      ASSERT_TRUE(map3.has_key(pair.first));\n    }\n    foreach(pair_type pair, map3) {\n      ASSERT_EQ(stdmap3[pair.first], pair.second);\n    }\n\n    \n  }\n\n\n  // void test_lookup() {\n  //   typedef small_map<32, size_t, std::string> map_type;\n  //   map_type map;\n  //   std::map stdmap;\n  //   map[5] = stdmap[5] = \"five\";\n  //   map[1] = stdmap[1] = \"one\";\n  //   map[2] = stdmap[2] = \"two\";    \n  //   ASSERT_EQ(map.size(), stdmap.size());\n  //   std::cout << std::endl;\n  //   std::cout << map << std::endl;\n  //   typedef std::pair pair_type;\n  //   foreach(pair_type pair, stdmap) {\n  //     ASSERT_EQ(map[pair.first], pair.second);\n  //     ASSERT_EQ(map.safe_find(pair.first), pair.second);\n  //     ASSERT_TRUE(map.has_key(pair.first));\n  //   }\n  //   foreach(pair_type pair, map) {\n  //     ASSERT_EQ(stdmap[pair.first], pair.second);\n  //   }\n\n  //   map_type map2;\n  //   std::map stdmap2;\n  //   map2[0] = stdmap2[0] = \"ZERO\";\n  //   map2[5] = stdmap2[5] = \"FIVE\";\n  //   map2[2] = stdmap2[2] = \"TWO\";    \n  //   map2[1] = stdmap2[1] = \"ONE\";\n  //   map2[8] = stdmap2[8] = \"EIGHT\";\n  //   ASSERT_EQ(map2.size(), stdmap2.size());\n  //   map_type map3 = map + map2;\n  //   std::map stdmap3 = \n  //     graphlab::map_union(stdmap, stdmap2);\n\n  //   std::cout << map3 << std::endl;\n    \n  //   foreach(pair_type pair, stdmap3) {\n  //     ASSERT_EQ(map3[pair.first], pair.second);\n  //     ASSERT_EQ(map3.safe_find(pair.first), pair.second);\n  //     ASSERT_TRUE(map3.has_key(pair.first));\n  //   }\n  //   foreach(pair_type pair, map3) {\n  //     ASSERT_EQ(stdmap3[pair.first], pair.second);\n  //   }\n\n    \n  // }\n\n\n\n};\n#include \n"
  },
  {
    "path": "tests/small_set_test.cxx",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n\n#include \n\n#include \n\n#include \n#include \n\nusing namespace graphlab;\n\n#include \nclass test_small_set : public CxxTest::TestSuite {\npublic:\n\n  \n  void test_union() {\n    std::cout << std::endl;\n    std::cout << \"Testing set union\" << std::endl;\n\n    typedef small_set<10, int> set_type;\n    typedef small_set<5, int> small_set_type;\n    small_set<0, int> empty_set;\n    small_set<10, int> set1;\n    small_set<10, int> set2;\n    set1 += set_type(1) + small_set_type(3) + set_type(2) + empty_set;\n    set1 += 1;\n    set1 += 3;\n    set1 += 2;\n    set1 += empty_set;\n    set1 += 1;\n    std::set true_set1;\n    true_set1.insert(1);\n    true_set1.insert(2);\n    true_set1.insert(3);\n    ASSERT_EQ(set_type(true_set1), set1);\n    std::cout << \"set1: \" << set1 << std::endl;\n    set2 += set_type(2) + small_set_type(5) + small_set_type(3) + set_type(7);\n    set2.insert(0);\n    set2 += 7;\n    set2 += 0;\n\n    std::set true_set2;\n    true_set2.insert(0);\n    true_set2.insert(2);\n    true_set2.insert(5);\n    true_set2.insert(3);\n    true_set2.insert(7);\n    ASSERT_EQ(set_type(true_set2), set2);    \n    std::cout << \"set2: \" << set2 << std::endl;\n\n    small_set<7, int> set3 = set1 + set2;\n    std::set true_set3 = set_union(true_set1, true_set2);\n    ASSERT_EQ(set_type(true_set3), set3);\n    std::cout << \"set3 = set1 + set2: \" << set3 << std::endl;\n    std::cout << \"set3 + set3: \" << (set3  + set3) << std::endl;\n    ASSERT_EQ(set_type(true_set3), (set3 + set3));    \n  }\n\n\n\n  void test_intersection() {\n    std::cout << std::endl;\n    std::cout << \"Testing set union\" << std::endl;\n\n    typedef small_set<10, int> set_type;\n    typedef small_set<5, int> small_set_type;\n    small_set<0, int> empty_set;\n    small_set<10, int> set1;\n    small_set<10, int> set2;\n    set1 += set_type(1) + small_set_type(3) + set_type(2) + empty_set;\n    set1.insert(8);\n    // do some intersections\n    set1 *= set1;\n    set1 = set1 * set1;\n    std::set true_set1;\n    true_set1.insert(1);\n    true_set1.insert(2);\n    true_set1.insert(3);\n    true_set1.insert(8);\n    ASSERT_EQ(set_type(true_set1), set1);\n    std::cout << \"set1: \" << set1 << std::endl;\n    \n    set2 += set_type(2) + small_set_type(5) + small_set_type(3) + set_type(7);\n    set2.insert(0);\n    set2 += 4;\n\n    std::set true_set2;\n    true_set2.insert(0);\n    true_set2.insert(2);\n    true_set2.insert(5);\n    true_set2.insert(3);\n    true_set2.insert(7);\n    true_set2.insert(4);\n    ASSERT_EQ(set_type(true_set2), set2);    \n    std::cout << \"set2: \" << set2 << std::endl;\n\n    small_set<7, int> set3 = set1 * set2;\n    std::set true_set3 = set_intersect(true_set1, true_set2);\n    ASSERT_EQ(set_type(true_set3), set3);\n    std::cout << \"set3 = set1 * set2: \" << set3 << std::endl;\n    std::cout << \"set3 * set3: \" << (set3  + set3) << std::endl;\n    ASSERT_EQ(set_type(true_set3), (set3 + set3));    \n  }\n\n\n  void test_difference() {\n    std::cout << std::endl;\n    std::cout << \"Testing set diff\" << std::endl;\n\n    typedef small_set<10, int> set_type;\n    typedef small_set<5, int> small_set_type;\n    small_set<0, int> empty_set;\n    small_set<10, int> set1;\n    small_set<10, int> set2;\n    set1 += set_type(1) + small_set_type(3) + set_type(2) + empty_set;\n    set1.insert(8);\n    // do some intersections\n    ASSERT_EQ(empty_set, set1 - set1);\n    ASSERT_EQ(empty_set, empty_set - empty_set);\n    empty_set = (empty_set - set1);\n    ASSERT_EQ(empty_set, empty_set - set1);\n    ASSERT_EQ(set1, set1 - empty_set);\n    std::set true_set1;\n    true_set1.insert(1);\n    true_set1.insert(2);\n    true_set1.insert(3);\n    true_set1.insert(8);\n    ASSERT_EQ(set_type(true_set1), set1);\n    std::cout << \"set1: \" << set1 << std::endl;\n    set2 += set_type(2) + small_set_type(5) + small_set_type(3) + set_type(7);\n    set2.insert(0);\n    set2 += 4;\n    std::set true_set2;\n    true_set2.insert(0);\n    true_set2.insert(2);\n    true_set2.insert(5);\n    true_set2.insert(3);\n    true_set2.insert(7);\n    true_set2.insert(4);\n    ASSERT_EQ(set_type(set_difference(true_set1, true_set2)),\n              set1 - set2);\n    ASSERT_EQ(set_type(set_difference(true_set2, true_set1)),\n              set2 - set1);\n  }\n\n  void test_range_iteration() {\n    typedef std::pair pair_type;\n    typedef small_set<20, pair_type > set_type;\n    set_type set = \n      set_type(std::make_pair(1, \"hello\")) + \n      set_type(std::make_pair(2, \"world\"));\n    foreach(const pair_type& value, set) {\n      std::cout << value.first << value.second << \", \";\n    }\n    std::cout << std::endl;\n  }\n\n  \n\n};\n\n#include \n"
  },
  {
    "path": "tests/sort_test.cpp",
    "content": "#include \n#include \n#include \n\nusing namespace graphlab;\n\nint main(int argc, char** argv) {\n  mpi_tools::init(argc, argv);\n  distributed_control dc;\n  std::vector keys;\n  std::vector values;\n  for (size_t i = 0;i < 1000000; ++i) {\n    size_t s = rand();\n    keys.push_back(s); values.push_back(s);\n  }\n\n  sample_sort sorter(dc);\n  sorter.sort(keys.begin(), keys.end(),\n              values.begin(), values.end());\n\n  std::vector > > result(dc.numprocs());\n  \n  std::swap(result[dc.procid()], sorter.result());\n  dc.gather(result, 0);\n  if (dc.procid() == 0) {\n    // test that it is sorted and the values are correct\n    size_t last = 0;\n    for (size_t i = 0;i < result.size(); ++i) {\n      dc.cout() << result[i].size() << \",\";\n      for (size_t j = 0; j < result[i].size(); ++j) {\n        ASSERT_EQ(result[i][j].first, result[i][j].second);\n        ASSERT_GE(result[i][j].first, last);\n        last = result[i][j].first;\n      }\n    }\n    dc.cout() << std::endl;\n  }\n  mpi_tools::finalize();\n}\n"
  },
  {
    "path": "tests/synchronous_engine_test.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \n#include \n#include \n\n\n// #include \n\n#include \n\ntypedef graphlab::distributed_graph graph_type;\n\n\nclass count_in_neighbors : \n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\npublic:\n  edge_dir_type \n  gather_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::IN_EDGES;\n  }\n  gather_type \n  gather(icontext_type& context, const vertex_type& vertex, \n         edge_type& edge) const {\n    // if (edge.target().id() == 7)\n    //   std::cout << edge.source().id() << \"\\t\" << edge.target().id() << std::endl;\n    return 1;    \n  }\n  void apply(icontext_type& context, vertex_type& vertex, \n             const gather_type& total) {\n    // if (total != int(vertex.num_in_edges())) {\n    //   std::cout << \"test fail vid : \" << vertex.id() << std::endl;\n    // }\n    ASSERT_EQ( total, int(vertex.num_in_edges()) );\n    context.signal(vertex);\n  }\n  edge_dir_type\n  scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n}; // end of count neighbors\n\n\nvoid test_in_neighbors(graphlab::distributed_control& dc,\n                       graphlab::command_line_options& clopts,\n                       graph_type& graph) {\n  std::cout << \"Constructing a syncrhonous engine for in neighbors\" << std::endl;\n  typedef graphlab::synchronous_engine engine_type;\n  engine_type engine(dc, graph, clopts);\n  std::cout << \"Scheduling all vertices to count their neighbors\" << std::endl;\n  engine.signal_all();\n\n  std::cout << \"Running!\" << std::endl;\n  engine.start();\n  std::cout << \"Finished\" << std::endl;\n\n  if (graph.is_dynamic()) {\n    std::cout << \"Test engine on dynamic graph !\" << std::endl;\n    graph.load_synthetic_powerlaw(10000);\n    graph.finalize();\n    engine.signal_all();\n    std::cout << \"Running!\" << std::endl;\n    engine.start();\n    std::cout << \"Finished\" << std::endl;\n  }\n}\n\n\nclass count_out_neighbors : \n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\npublic:\n  edge_dir_type \n  gather_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::OUT_EDGES;\n  }\n  gather_type \n  gather(icontext_type& context, const vertex_type& vertex, \n         edge_type& edge) const {\n    return 1;\n  }\n  void apply(icontext_type& context, vertex_type& vertex, \n             const gather_type& total) {\n    ASSERT_EQ( total, int(vertex.num_out_edges()) );\n    context.signal(vertex);\n  }\n  edge_dir_type \n  scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n}; // end of count neighbors\n\nvoid test_out_neighbors(graphlab::distributed_control& dc,\n                        graphlab::command_line_options& clopts,\n                        graph_type& graph) {\n  std::cout << \"Constructing a syncrhonous engine for out neighbors\" << std::endl;\n  typedef graphlab::synchronous_engine engine_type;\n  engine_type engine(dc, graph, clopts);\n  std::cout << \"Scheduling all vertices to count their neighbors\" << std::endl;\n  engine.signal_all();\n  std::cout << \"Running!\" << std::endl;\n  engine.start();\n  std::cout << \"Finished\" << std::endl;\n\n  if (graph.is_dynamic()) {\n    std::cout << \"Test engine on dynamic graph !\" << std::endl;\n    graph.load_synthetic_powerlaw(10000);\n    graph.finalize();\n    engine.signal_all();\n    std::cout << \"Running!\" << std::endl;\n    engine.start();\n    std::cout << \"Finished\" << std::endl;\n  }\n}\n\n\nclass count_all_neighbors : \n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\npublic:\n  edge_dir_type \n  gather_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  }\n  gather_type \n  gather(icontext_type& context, const vertex_type& vertex, \n         edge_type& edge) const {\n    return 1;\n  }\n  void apply(icontext_type& context, vertex_type& vertex, \n             const gather_type& total) {\n    ASSERT_EQ( total, int(vertex.num_in_edges() + vertex.num_out_edges() ) );\n    context.signal(vertex);\n  }\n  edge_dir_type \n  scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n}; // end of count neighbors\n\nvoid test_all_neighbors(graphlab::distributed_control& dc,\n                        graphlab::command_line_options& clopts,\n                        graph_type& graph) {\n  std::cout << \"Constructing a syncrhonous engine for all neighbors\" << std::endl;\n  typedef graphlab::synchronous_engine engine_type;\n  engine_type engine(dc, graph, clopts);\n  std::cout << \"Scheduling all vertices to count their neighbors\" << std::endl;\n  engine.signal_all();\n  std::cout << \"Running!\" << std::endl;\n  engine.start();\n  std::cout << \"Finished\" << std::endl;\n  if (graph.is_dynamic()) {\n    std::cout << \"Test engine on dynamic graph !\" << std::endl;\n    graph.load_synthetic_powerlaw(10000);\n    graph.finalize();\n    engine.signal_all();\n    std::cout << \"Running!\" << std::endl;\n    engine.start();\n    std::cout << \"Finished\" << std::endl;\n  }\n}\n\n\n\n\nclass basic_messages : \n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\n  int message_value;\npublic:\n\n  void init(icontext_type& context, const vertex_type& vertex,\n                    const message_type& msg) {\n    message_value = msg;\n  } \n\n  edge_dir_type \n  gather_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::IN_EDGES;\n  }\n \n  gather_type gather(icontext_type& context, const vertex_type& vertex, \n         edge_type& edge) const {\n    return 1;\n  }\n\n  void apply(icontext_type& context, vertex_type& vertex, \n             const gather_type& total) {\n    context.signal(vertex, 0);\n    if(message_value < 0) {\n      // first iteration has wrong messages\n      return;\n    }\n    ASSERT_EQ(total, message_value);\n\n  }\n\n  edge_dir_type \n  scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::OUT_EDGES;\n  }\n\n  void scatter(icontext_type& context, const vertex_type& vertex, \n               edge_type& edge) const {\n    context.signal(edge.target(), 1);\n  }\n\n}; // end of test_messages\n\nvoid test_messages(graphlab::distributed_control& dc,\n                   graphlab::command_line_options& clopts,\n                   graph_type& graph) {\n  std::cout << \"Testing messages\" << std::endl;\n  typedef graphlab::synchronous_engine engine_type;\n  engine_type engine(dc, graph, clopts);\n  std::cout << \"Scheduling all vertices to test messages\" << std::endl;\n  engine.signal_all(-1);\n  std::cout << \"Running!\" << std::endl;\n  engine.start();\n  std::cout << \"Finished\" << std::endl;\n  if (graph.is_dynamic()) {\n    engine.init();\n    std::cout << \"Test engine on dynamic graph !\" << std::endl;\n    graph.load_synthetic_powerlaw(10000);\n    graph.finalize();\n    engine.signal_all(-1);\n    std::cout << \"Running!\" << std::endl;\n    engine.start();\n    std::cout << \"Finished\" << std::endl;\n  }\n}\n\n\n\n\n\n\n\nclass count_aggregators : \n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\npublic:\n  edge_dir_type \n  gather_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::IN_EDGES;\n  }\n  gather_type \n  gather(icontext_type& context, const vertex_type& vertex, \n         edge_type& edge) const {\n    ASSERT_LT(vertex.data(), 100);\n    ASSERT_GE(vertex.data(), 0);\n    return vertex.data();\n  }\n  void apply(icontext_type& context, vertex_type& vertex, \n             const gather_type& total) {\n    ASSERT_EQ( total, context.iteration() * vertex.num_in_edges() );\n    vertex.data() = context.iteration() + 1; \n    if(context.iteration() < 10) context.signal(vertex);\n  }\n  edge_dir_type \n  scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n}; // end of count aggregators\n\nint iteration_counter(count_aggregators::icontext_type& context,\n                      const graph_type::vertex_type& vertex) {\n  ASSERT_LT(vertex.data(), 100);\n  return vertex.data();\n}\nint finalize_iter = 0;\nvoid iteration_finalize(count_aggregators::icontext_type& context,\n                        const int& total) {\n  std::cout << \"Finalized\" << std::endl;\n  ASSERT_EQ(total, context.num_vertices() * (context.iteration()+1));\n  ASSERT_EQ(finalize_iter++, context.iteration());\n}\n\n\nvoid test_count_aggregators(graphlab::distributed_control& dc,\n                            graphlab::command_line_options& clopts,\n                            graph_type& graph) {\n  std::cout << \"Constructing a syncrhonous engine for aggregators\" << std::endl;\n  typedef graphlab::synchronous_engine engine_type;\n  engine_type engine(dc, graph, clopts);\n  engine.add_vertex_aggregator(\"iteration_counter\", \n                                    iteration_counter, iteration_finalize);\n  engine.aggregate_periodic(\"iteration_counter\", 0);\n  std::cout << \"Scheduling all vertices to count their neighbors\" << std::endl;\n  engine.signal_all();\n  std::cout << \"Running!\" << std::endl;\n  engine.start();\n  std::cout << \"Finished\" << std::endl;\n  ASSERT_EQ(finalize_iter, engine.iteration());\n}\n\n\n\n\nint main(int argc, char** argv) {\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::dc_init_param rpc_parameters;\n  graphlab::init_param_from_mpi(rpc_parameters);\n  graphlab::distributed_control dc(rpc_parameters);\n\n  graphlab::command_line_options clopts(\"Test code.\");\n  clopts.engine_args.set_option(\"max_iterations\", 10);\n  std::cout << \"Creating a powerlaw graph\" << std::endl;\n  graph_type graph(dc, clopts);\n  graph.load_synthetic_powerlaw(10000);\n  graph.finalize();\n  test_in_neighbors(dc, clopts, graph);\n  test_out_neighbors(dc, clopts, graph);\n  test_all_neighbors(dc, clopts, graph);\n  test_messages(dc, clopts, graph);\n  test_count_aggregators(dc, clopts, graph);\n\n  graphlab::mpi_tools::finalize();\n} // end of main\n\n\n\n\n\n"
  },
  {
    "path": "tests/test_lock_free_pool.cxx",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n\nusing namespace graphlab;\nlock_free_pool pool;\n\nvoid exec() {\n  size_t *s = NULL;\n  for (size_t i = 0;i < 1000000; ++i) {\n    while(1) {\n      s = pool.alloc();\n      if (s == NULL) continue;\n      else {\n       for (size_t j = 0;j < 10; ++j) (*s)++;\n       pool.free(s);\n       break;\n      }\n    } \n  }\n}\n\n\nclass LockFreePoolTestSuite: public CxxTest::TestSuite {\n public:  \n  void test_lock_free_pool() {\n    size_t nthreads = 8;\n    \n    pool.reset_pool(32);\n    thread_group g;\n    for (size_t i = 0; i < nthreads; ++i) {\n      g.launch(exec);\n    }\n    while(1) {\n      try {\n        g.join();\n        break;\n      }\n      catch(const char* c ) {\n        std::cout << c << \"\\n\";\n      }\n    }\n    \n    std::vector alldata = pool.unsafe_get_pool_ref();\n    size_t total = 0;\n    for (size_t i = 0;i < alldata.size(); ++i) {\n      total += alldata[i];\n    }\n    TS_ASSERT_EQUALS(total, 10000000 * nthreads);\n  }\n};\n"
  },
  {
    "path": "tests/test_parsers.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n\ntypedef graphlab::distributed_graph graph_type;\n\nvoid check_structure(graph_type &graph) {\n  ASSERT_EQ(graph.num_vertices(), 5);\n  ASSERT_EQ(graph.num_edges(), 7);\n  // check vertex 0 \n  {\n    graph_type::vertex_type vtype = graph.vertex(0);\n    graph_type::local_edge_list_type v0_out = graph_type::local_vertex_type(vtype).out_edges();\n    ASSERT_EQ(v0_out.size(), 1);\n    ASSERT_EQ(v0_out[0].target().global_id(), 5);\n  }\n  // vertex 1\n  {\n    graph_type::vertex_type vtype = graph.vertex(1);\n    graph_type::local_edge_list_type v0_out = graph_type::local_vertex_type(vtype).out_edges();\n    ASSERT_EQ(v0_out.size(), 2);\n    ASSERT_EQ(v0_out[0].target().global_id(), 0);\n    ASSERT_EQ(v0_out[1].target().global_id(), 5);\n  }\n  \n  // vertex 2\n  {\n    graph_type::vertex_type vtype = graph.vertex(2);\n    graph_type::local_edge_list_type v0_out = graph_type::local_vertex_type(vtype).out_edges();\n    ASSERT_EQ(v0_out.size(), 2);\n    ASSERT_EQ(v0_out[0].target().global_id(), 0);\n    ASSERT_EQ(v0_out[1].target().global_id(), 5);\n  }\n  // vertex 3\n  {\n    graph_type::vertex_type vtype = graph.vertex(3);\n    graph_type::local_edge_list_type v0_out = graph_type::local_vertex_type(vtype).out_edges();\n    ASSERT_EQ(v0_out.size(), 2);\n    ASSERT_EQ(v0_out[0].target().global_id(), 0);\n    ASSERT_EQ(v0_out[1].target().global_id(), 5);\n  }\n}\n\n\n\nvoid test_adj(graphlab::distributed_control& dc) {\n  graphlab::distributed_graph graph(dc);\n  graph.load_format(\"data/test_adj\", \"adj\");\n  graph.finalize();\n  check_structure(graph);  \n}\n\nvoid test_snap(graphlab::distributed_control& dc) {\n  graphlab::distributed_graph graph(dc);\n  graph.load_format(\"data/test_snap\", \"snap\");\n  graph.finalize();\n  check_structure(graph);  \n}\n\nvoid test_tsv(graphlab::distributed_control& dc) {\n  graphlab::distributed_graph graph(dc);\n  graph.load_format(\"data/test_tsv\", \"tsv\");\n  graph.finalize();\n  check_structure(graph);  \n}\n\nvoid test_powerlaw(graphlab::distributed_control& dc) {\n  graphlab::distributed_graph graph(dc);\n  graph.load_synthetic_powerlaw(1000);\n  graph.finalize();\n  ASSERT_EQ(graph.num_vertices(), 1000);\n  std::cout << graph.num_edges() << \" Edges\\n\";\n}\n\n\nvoid test_save_load(graphlab::distributed_control& dc) {\n  graphlab::distributed_graph graph(dc);\n  graph.load_synthetic_powerlaw(1000);\n  graph.finalize();\n  ASSERT_EQ(graph.num_vertices(), 1000);\n  graph.save_format(\"data/plawtest_tsv\", \"tsv\");\n  graph.save_format(\"data/plawtest_jrl\", \"graphjrl\");\n  // load it back\n  graphlab::distributed_graph graph2(dc);\n  graph2.load_format(\"data/plawtest_tsv\", \"tsv\");\n  graph2.finalize();\n  ASSERT_EQ(graph.num_vertices(), graph2.num_vertices());\n  ASSERT_EQ(graph.num_edges(), graph2.num_edges());\n\n  graphlab::distributed_graph graph3(dc);\n  graph3.load_format(\"data/plawtest_jrl\", \"graphjrl\");\n  graph3.finalize();\n  ASSERT_EQ(graph.num_vertices(), graph3.num_vertices());\n  ASSERT_EQ(graph.num_edges(), graph3.num_edges());\n\n}\n\n\nint main(int argc, char** argv) {\n  graphlab::distributed_control dc;\n  test_adj(dc);\n  test_snap(dc);\n  test_tsv(dc);\n  test_powerlaw(dc);\n  test_save_load(dc);\n};\n\n"
  },
  {
    "path": "tests/test_vertex_set.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n#include \n#include \n#include \n\n\n// #include \n\n#include \n\ntypedef graphlab::distributed_graph graph_type;\n\n\nbool select_out_degree_le(graph_type::vertex_type vtx, size_t ndeg) {\n  return vtx.num_out_edges() <= ndeg;\n}\n\n\nbool select_out_degree_eq(graph_type::vertex_type vtx, size_t ndeg) {\n  return vtx.num_out_edges() == ndeg;\n}\n\n\nbool select_vid_modulo(graph_type::vertex_type vtx, size_t divisor) {\n  return (vtx.id() % divisor) == 0;\n}\n\n\nsize_t is_divisible(graph_type::vertex_type vtx, size_t divisor) {\n  return (vtx.id() % divisor) == 0;\n}\n\nsize_t count_edges(graph_type::edge_type e) {\n  return 1;\n}\n\nvoid set_to_one(graph_type::vertex_type vtx) {\n  vtx.data() = 1;\n}\n\nint vertex_data_identity(graph_type::vertex_type vtx) {\n  return vtx.data();\n}\n\n\n\nint main(int argc, char** argv) {\n\n  global_logger().set_log_level(LOG_INFO);\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n  graph_type graph(dc);\n  graph.load_synthetic_powerlaw(100000);\n  graph.finalize();\n\n  dc.cout() << graph.vertex_set_size(graph.complete_set()) << \" Vertices\\n\";\n\n  ASSERT_EQ(graph.vertex_set_size(graph.complete_set()), graph.num_vertices());\n  // select all vertices which have <= 1 neighbors\n  graphlab::vertex_set small = graph.select(boost::bind(select_out_degree_le, _1, 1));\n  dc.cout() << graph.vertex_set_size(small) << \" vertices with out degree <= 1\\n\";\n\n  // all vertices which have  > 1 neighbors\n  graphlab::vertex_set connected = graph.complete_set() - small; \n  dc.cout() << graph.vertex_set_size(connected) << \" vertices with out degree > 1\\n\";\n\n  // union of the two of them should give me all vertices\n  graphlab::vertex_set all = small | connected;\n  ASSERT_EQ(graph.vertex_set_size(all), graph.num_vertices());\n\n  // select all vertices with an even ID\n  graphlab::vertex_set even_id = graph.select(boost::bind(select_vid_modulo, _1, 2));\n  // select all vertices with ID divisible by 3\n  graphlab::vertex_set div_3_id = graph.select(boost::bind(select_vid_modulo, _1, 3));\n\n  // intersect\n  graphlab::vertex_set div_6_id = even_id & div_3_id;\n\n  // count the number of IDs which are divisible by 6 \n  size_t num_div_6 = graph.map_reduce_vertices(boost::bind(is_divisible, _1, 6)); \n\n  ASSERT_EQ(num_div_6, 1 + (graph.num_vertices() - 1) / 6);\n\n  // do a restricted map_reduce\n  size_t num_div_6_restricted = graph.map_reduce_vertices\n                                (boost::bind(is_divisible, _1, 6), div_6_id); \n\n  ASSERT_EQ(num_div_6, num_div_6_restricted);\n\n  ASSERT_EQ(graph.vertex_set_size(div_6_id), num_div_6);\n\n\n\n  graphlab::vertex_set out_deg_one = graph.select(boost::bind(select_out_degree_eq, _1, 1));\n  // test edge mapreduce\n  size_t num_small_edges = graph.map_reduce_edges(\n                            count_edges, out_deg_one, graphlab::OUT_EDGES);\n  // since the set only has stuff with out degree == 1... the number\n  // of edges must match the size of out_deg_one\n\n  ASSERT_EQ(num_small_edges, graph.vertex_set_size(out_deg_one));\n\n  // test transform \n  // set vdata to 1 for the vertices with out degree 1 \n  graph.transform_vertices(set_to_one, out_deg_one);\n  size_t total = graph.map_reduce_vertices(vertex_data_identity, out_deg_one);\n  ASSERT_EQ(total, graph.vertex_set_size(out_deg_one)); \n\n  // test neighborhood selection \n  // extract the set of out neighbors of out_deg_one \n  graphlab::vertex_set out_nbrs = graph.neighbors(out_deg_one, graphlab::OUT_EDGES);\n\n  dc.cout() << graph.vertex_set_size(out_nbrs) << \" nbr size\\n\";\n  // extract the set of in neighbors of these out neighbors\n  graphlab::vertex_set out_nbrs_in_nbrs = graph.neighbors(out_nbrs, graphlab::IN_EDGES);\n\n  dc.cout() << graph.vertex_set_size(out_nbrs_in_nbrs) << \" nbr nbr size\\n\";\n  // this set must contain the original out_deg_one set\n  ASSERT_TRUE(graph.vertex_set_empty((out_deg_one & out_nbrs_in_nbrs) - out_deg_one));\n  graphlab::mpi_tools::finalize();\n}\n\n"
  },
  {
    "path": "tests/thread_tools.cxx",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n\nusing namespace graphlab;\n\natomic testval;\n\nvoid test_inc() {\n  usleep(100000);\n  testval.inc();\n}\n\nvoid test_dec() {\n  usleep(100000);\n  testval.dec();\n}\n\n\n\nvoid thread_assert_false() {\n  ASSERT_TRUE(false);\n}\n\n\nvoid test_group_exception_forwarding(){\n  std::cout << \"\\n\";\n  std::cout << \"----------------------------------------------------------------\\n\";\n  std::cout << \"This test will print a  large number of assertional failures\\n\";\n  std::cout << \"and back traces. This is intentional as we are testing the\\n\" ;\n  std::cout << \"exception forwarding scheme\\n\";\n  std::cout << \"----------------------------------------------------------------\\n\";\n  std::cout << std::endl;\n\n  thread_group group;\n\n  \n  thread thr3;\n  thr3.launch(thread_assert_false);\n  try {\n    thr3.join();\n  }\n  catch(const char* c) {\n    std::cout << \"Exception \" << c << \" forwarded successfully!\" << std::endl;\n  }\n  \n  \n  for (size_t i = 0;i < 10; ++i) {\n    group.launch(thread_assert_false);\n  }\n  \n  size_t numcaught = 0;\n  while (group.running_threads() > 0) {\n    try {\n      group.join();\n    }\n    catch (const char* c){\n      std::cout << \"Exception \" << c << \" forwarded successfully!\" << std::endl;\n      numcaught++;\n    }\n  }\n  std::cout << \"Caught \" << numcaught << \" exceptions!\" << std::endl;\n  TS_ASSERT_EQUALS(numcaught, (size_t)10);\n}\n\nvoid test_pool(){\n  testval.value = 0;\n  thread_pool pool(4);\n  for (size_t j = 0;j < 10; ++j) {\n    for (size_t i = 0;i < 10; ++i) {\n      pool.launch(test_inc);\n    }\n    for (size_t i = 0;i < 10; ++i) {\n      pool.launch(test_dec);\n    }\n    pool.set_cpu_affinity(j % 2);\n  }\n  \n  pool.join();\n  TS_ASSERT_EQUALS(testval.value, 0);\n}\n\nvoid test_pool_exception_forwarding(){\n  std::cout << \"\\n\";\n  std::cout << \"----------------------------------------------------------------\\n\";\n  std::cout << \"This test will print a  large number of assertional failures\\n\";\n  std::cout << \"and back traces. This is intentional as we are testing the\\n\" ;\n  std::cout << \"exception forwarding scheme\\n\";\n  std::cout << \"----------------------------------------------------------------\\n\";\n  std::cout << std::endl;\n  thread_pool pool(10);\n\n  \n  thread thr3;\n  thr3.launch(thread_assert_false);\n  try {\n    thr3.join();\n  }\n  catch(const char* c) {\n    std::cout << \"Exception \" << c << \" forwarded successfully!\" << std::endl;\n  }\n  \n  \n  for (size_t i = 0;i < 10; ++i) {\n    pool.launch(thread_assert_false);\n    if (i == 50) {\n      pool.set_cpu_affinity(true);\n    }\n  }\n  \n  size_t numcaught = 0;\n  while (1) {\n    try {\n      pool.join();\n      break;\n    }\n    catch (const char* c){\n      std::cout << \"Exception \" << c << \" forwarded successfully!\" << std::endl;\n      numcaught++;\n    }\n  }\n  std::cout << \"Caught \" << numcaught << \" exceptions!\" << std::endl;\n  TS_ASSERT_EQUALS(numcaught, (size_t)10);\n}\n\n\n\n\n\n\nclass ThreadToolsTestSuite : public CxxTest::TestSuite {\npublic:\n  void test_thread_group_exception(void) {\n   test_group_exception_forwarding();\n  }\n\n  void test_thread_pool(void) {\n   test_pool();\n  }\n   \n  void test_thread_pool_exception(void) {\n    test_pool_exception_forwarding();\n  }\n\n};\n"
  },
  {
    "path": "tests/union_find_test.cxx",
    "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n\ngraphlab::concurrent_union_find uf2;\n\nvoid add_even() {\n  for (size_t i = 2; i < 1000000; i+=2) {\n    size_t unionwith = 0;\n    while(1){\n      unionwith = graphlab::random::fast_uniform((size_t)0, i - 1);\n      if (unionwith % 2 == 0) break;\n    }\n    uf2.merge(i, unionwith);\n  }\n}\n\n\nvoid add_odd() {\n  for (size_t i = 3; i < 1000000; i+=2) {\n    size_t unionwith = 0;\n    while(1){\n      unionwith = graphlab::random::fast_uniform((size_t)0, i - 1);\n      if (unionwith % 2 == 1) break;\n    }\n    uf2.merge(i, unionwith);\n  }\n}\n\n\nclass UnionFindTest: public CxxTest::TestSuite {\n public:\n  void test_union_find() {\n    graphlab::union_find uf;\n    uf.init(1000);\n    // union all the odd together and all the even together\n    for (size_t i = 2; i < 1000; i+=2) {\n      size_t unionwith = 0;\n      while(1){\n        unionwith = graphlab::random::fast_uniform((size_t)0, i - 1);\n        if (unionwith % 2 == 0) break;\n      }\n      uf.merge(i, unionwith);\n    }\n    \n    // union all the odd together and all the even together\n    for (size_t i = 3; i < 1000; i+=2) {\n      size_t unionwith = 0;\n      while(1){\n        unionwith = graphlab::random::fast_uniform((size_t)0, i - 1);\n        if (unionwith % 2 == 1) break;\n      }\n      uf.merge(i, unionwith);\n    }\n\n    // assert that all evens are together and all odds are together\n    size_t evenid = uf.find(0);\n    for (size_t i = 0; i < 1000; i+=2) {\n      TS_ASSERT_EQUALS(uf.find(i), evenid);\n    }\n\n    size_t oddid = uf.find(1);\n    for (size_t i = 1; i < 1000; i+=2) {\n      TS_ASSERT_EQUALS(uf.find(i), oddid);\n    }\n  }\n\n  void test_union_find2() {\n    uf2.init(1000000);\n\n    graphlab::thread_group tg;\n    tg.launch(add_even);\n    tg.launch(add_even);\n    tg.launch(add_even);\n    tg.launch(add_odd);\n    tg.launch(add_odd);\n    tg.launch(add_odd);\n\n    tg.join();\n\n\n    // assert that all evens are together and all odds are together\n    size_t evenid = uf2.find(0);\n    for (size_t i = 0; i < 1000000; i+=2) {\n      TS_ASSERT_EQUALS(uf2.find(i), evenid);\n    }\n\n    size_t oddid = uf2.find(1);\n    for (size_t i = 1; i < 1000000; i+=2) {\n      TS_ASSERT_EQUALS(uf2.find(i), oddid);\n    }\n  }\n};"
  },
  {
    "path": "toolkits/CMakeLists.txt",
    "content": "project(GraphLab)\n\n# link_libraries(${Boost_LIBRARIES})\n# link_libraries(${GraphLab_LIBRARIES})\n\n\n\nmacro(add_all_subdirectories retval curdir)\n  file(GLOB sub-dir RELATIVE ${curdir} *)\n  set(list_of_dirs \"\")\n  foreach(dir ${sub-dir})\n    if(IS_DIRECTORY ${curdir}/${dir})\n    STRING(SUBSTRING ${dir} 0 1 firstchar)\n        if(${firstchar} STREQUAL \".\" OR ${firstchar} STREQUAL \"_\" )\n        else(${firstchar} STREQUAL \".\" OR ${firstchar} STREQUAL \"_\")\n          set(list_of_dirs ${list_of_dirs} ${dir})\n          message(STATUS \"Detected Toolkit: \" ${dir})\n          add_subdirectory(${dir})\n        endif()\n    endif()\n  endforeach()\n  set(${retval} ${list_of_dirs})\nendmacro()\n\nadd_all_subdirectories(retval, ${CMAKE_CURRENT_SOURCE_DIR})\n\n"
  },
  {
    "path": "toolkits/README.md",
    "content": "# Toolkits Overview\n\n## Collaborative Filtering\n\n![Collaborative Filtering](../images/collaborative_filtering.jpg?raw=true \"collaborative_filtering\")\n\nThe collaborative filtering toolkit provides tools to identify patterns of user interests and make targeted recommendations. Learn more about collaborative filtering [here.](http://en.wikipedia.org/wiki/Collaborative_Filter)\n\n![ALS equation](../images/als_eqn.gif \"als_eqn\")\n\nMost of the algorithms take the rating matrix R, which is a sparse matrix holding the rating given by users to movies, and builds a linear model, finding two low dimensional matrices\n\nU and V s.t. their product approximates R: R ~ UV.\n\nWe implement multiple collaborative filtering algorithms: ALS (Alternating least squares), SGD (Stochastic Gradient Descent) , Bias SGD, Weighted-ALS, Sparse-ALS, SVD++.\n\n## Clustering\n\n![](../images/kmeans1.gif \"kmeans\")\n\n## KMeans++\n\nGraphLab PowerGraph has a fast scalable implementation of the [Kmeans++](http://en.wikipedia.org/wiki/K-means%2B%2B) algorithm: a robust method of grouping datapoints into clusters.\n\n## Computer Vision\n\n![Image Stitching](../images/panorama-1024x251.png \"Image Stitching\")\n\nThe Computer Vision Toolkit aims to provide fully distributed wrappers to algorithms in [OpenCV](http://opencv.org/ \"OpenCV\"), an open-source library aimed at real-time computer vision. Currently, the only implemented algorithm is Image-Stitching, where the goal is to create a composite panoramic image from a collection of images. Learn more about computer vision [here.](http://en.wikipedia.org/wiki/Computer_vision)\n\nGraphLab PowerGraph Computer Vision Toolkit has become it’s own spin-off project called [CloudCV](http://cloudcv.org \"CloudCV\"), a comprehensive system that will aims to provide access to state-of-the-art computer vision algorithms on the cloud.\n\n![CloudCV](../images/cloudcv-1024x489.png \"CloudCV\")\n\nCloudCV: Large-Scale Parallel Computer Vision on the Cloud\n\n\n## Graphical Models\n![](../doc/images/noisy_img.jpeg)\n![](../images/Slide1.jpg \"Slide1\")\n![](../doc/images/pred_img.jpeg)\n\n[Graphical models](http://en.wikipedia.org/wiki/Graphical_model) provide a compact interpretable representation of complex statistical phenomena by encoding random variables as vertices in a graph and relationships between those variables as edges. The Graphical Models toolkit provides a collection of methods to make predictions under uncertainty, and for reasoning about structured noisy data.\n\nThe main components of Graphical Models toolkit are:\n\n1.  Distributed Dual Decomposition: performs maximum _a posteriori_ (MAP) inference in general [Markov Random Fields](http://en.wikipedia.org/wiki/Markov_random_field) via the Dual Decomposition algorithm. The MRF is assumed to be provided in the standard [UAI file format](http://www.cs.huji.ac.il/project/PASCAL/fileFormat.php). Maintained by [Dhruv Batra](http://filebox.ece.vt.edu/~dbatra/).\n2.  [Structured Prediction](http://docs.graphlab.org/graphical_models.html): that applies the [Loopy Belief propagation (LBP)](http://en.wikipedia.org/wiki/Belief_propagation) algorithm to a pair-wise [Markov Random Field](http://en.wikipedia.org/wiki/Markov_random_field) encoding the classic [Potts Model](http://en.wikipedia.org/wiki/Potts_model).\n\n## Graph Analytics\n\n![](../images/r-300x278.jpg \"r\")\n\nThe Graph Analytics Toolkit aims to provide high performance, distributed tools for graph mining, for use in community detection, social network discovery, etc.\n\nThe toolkit currently implements the following tools:\n\n### Triangle Counting\n![](../images/triangle_weak_community-150x150.jpg \"triangle_weak_community\")\n![](../images/triangle_strong_community-150x150.jpg \"triangle_strong_community\")\n\n\nTwo triangle counting program:\n\n1.  **Undirected Triangle Counting**: counts the total number of triangles in a graph, or the the number of triangles each vertex is in\n2.  **Directed Triangle Counting:** Counts the number of types of triangles each vertex is in\n\n### PageRank\n\n![](../images/800px-PageRank-hi-res-300x215.png \"800px-PageRank-hi-res\")\n\nA classical graph algorithm which assigns each vertex a numerical importance value based on random walk properties. Learn more about page rank [here.](http://en.wikipedia.org/wiki/PageRank)\n\n### KCore Decomposition\n\n![](../images/kcore2-300x300.jpg \"kcore\")\n\nIdentifies a hierarchical ordering of the vertices in the graph, allowing discovery of the central components of the network.\n\n## Topic Modeling\n\n![](../images/topic-300x179.gif \"topic\")\n\n## Latent Dirichlet Allocation\n\nThe topic modelling toolbox currently implemented the [Latent Dirichlet Allocation](http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) algorithm for deriving semantic topic information from a corpus of plain text.\n\n## Linear Solvers\n\nGraphLab PowerGraph iterative solvers, for solving a linear system of the type Ax = b.\n\nCurrently [Jacobi method](http://en.wikipedia.org/wiki/Jacobi_method) is implemented."
  },
  {
    "path": "toolkits/clustering/CMakeLists.txt",
    "content": "project(GraphProcessing)\n\nadd_graphlab_executable(kmeans kmeans.cpp)\nadd_graphlab_executable(generate_synthetic generate_synthetic.cpp)\nadd_graphlab_executable(spectral_clustering spectral_clustering.cpp)\nadd_graphlab_executable(graph_laplacian_for_sc graph_laplacian_for_sc.cpp)\n"
  },
  {
    "path": "toolkits/clustering/clustering.dox",
    "content": "\t/**\n\n\\page clustering Clustering\n\nThis toolkit will contain implementations of clustering algorithm.\n\nCurrently the algorithms implemented are \n\n\\li \\ref clustering_kmeans \"KMeans++\"\n\\li \\ref clustering_spectral_clustering \"Spectral Clustering\"\n\n\n\\section clustering_kmeans KMeans++\n\nThe \\c kmeans program implements the \nKMeans++ algorithm described \nby \n\nArthur, D. and Vassilvitskii, S. (2007). \"k-means++: the advantages of careful seeding\". \nProceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms. pp. 1027–1035.\n\nIt takes as input a collection of files where each line in each file represents\na data point.  Each line must contains a list of numbers, white-space or comma\nseparated. Each line must be the same length. \n\nFor instance in this example input file, there are 6 datapoints, one per line,\nand each datapoint is a point in a 5-dimensional space.  \n\n\\verbatim\n-10.7551  6.82178 5.33455 -2.08247  2.86694 \n-1.36687  10.8464 -5.28851  -4.26768  -5.50659  \n-8.79834  8.01002 5.33418 0.102824  3.23318 \n-8.64345  6.81946 1.2309  -4.46784  2.26341 \n-8.29782  7.1154  3.32559 -2.59422  2.33936 \n-8.12504  8.98924 4.15027 0.253153  1.75911 \n\\endverbatim\n\n\\subsection clustering_kmeans_synthetic Synthetic Data\nExample synthetic data can be generated by running\n\n\\verbatim\n> ./generate_synthetic [Number of Clusters] [Number of Dimensions] [Number of datapoints]\n\\endverbatim\n\nThis will generate a file called \\c synthetic.txt , and will also output to screen\nthe cluster centers.\n\nFor instance:\n\\verbatim\n> ./generate_synthetic 2 3 10\nUsage: generate_synthetic [NClusters] [Dimensions] [Ndata]\nCenter 0 at: -6.69675 0.355189  -4.88601  \nCenter 1 at: 5.85919  0.0388327 5.50007  \n\n> cat synthetic.txt\n-4.31568  -0.396959 -6.29507  \n-4.56112  -1.74917  -4.57874  \n4.54508 0.102845  6.35385 \n4.87746 -0.832591 7.06942 \n-5.91254  -0.278006 -4.25934  \n6.95139 0.120139  4.89531 \n-6.28538  -0.88527  -4.74988  \n-6.84791  0.887664  -4.91919  \n7.47117 1.67911 6.02221 \n-4.78011  1.2099  -4.55519  \n\\endverbatim\n\n\\note Mathematically, the synthetic generator draws centers from the [-10,10]^D\nuniform hypercube and draws data points by sampling uniformly from the centers,\nand around a unit gaussian around each center.\n\n\n\\subsection clustering_kmeans_running Running KMeans\nTo run Kmeans clustering:\n\\verbatim\n> ./kmeans --data=[data prefix] --cluster=[N cluster] --output-clusters=[cluster output file]\n\\endverbatim\n\nAll files beginning with the data prefix will be loaded. The data may be on\nHDFS.  The \\c --cluster option is the number of clusters to solve for.\n\nThe cluster output file must be a target accessible on the local file system.\n(In the distributed case, it must be accessible from the 0th machine).\nThis file will contain a list of all the solved cluster centers.\n\nFor instance, running on the synthetic data above:\n\\verbatim\n>./kmeans --data=synthetic.txt --clusters=2 --output-clusters=cluster.txt\nNumber of datapoints: 10\nValidating data...Initializing using Kmeans++\nRunning Kmeans...\nKmeans iteration 1: # points with changed assignments = 0\nWriting Cluster Centers...\n\n>cat cluster.txt\n-5.45046  -0.201973 -4.8929 \n5.96127 0.267376  6.0852  \n\\endverbatim\n\n\nTo also output the cluster center assignments for each datapoint,\nadd the option:\n\\verbatim\n--output-data=[output prefix]\n\\endverbatim\n\nTne output prefix is where the output data will be written. This\nmay be located on HDFS. For instance, if the output_prefix is \"v_out\",\nthe output files will be written to:\n\n\\verbatim\nv_out_1_of_16\nv_out_2_of_16\n...\nv_out_16_of_16\n\\endverbatim\n\nEach line in the output files contain firstly, the original data point, followed\n$by the cluster number it was assigned to. These need not be in the same order\nas the original input.\n\nFor instance, running kmeans on the example synthetic data above may produce\nthe following output:\n\n\\verbatim\n>./kmeans --data=synthetic.txt --clusters=2 --output-clusters=cluster.txt --output-data=data.txt\nNumber of datapoints: 10\nValidating data...Initializing using Kmeans++\nRunning Kmeans...\nKmeans iteration 1: # points with changed assignments = 0\nWriting Cluster Centers...\nWriting Data with cluster assignments...\n\n>cat data.txt\n-4.78011  1.2099  -4.55519  0\n7.47117 1.67911 6.02221 1\n-6.84791  0.887664  -4.91919  0\n-6.28538  -0.88527  -4.74988  0\n6.95139 0.120139  4.89531 1\n-5.91254  -0.278006 -4.25934  0\n4.87746 -0.832591 7.06942 1\n4.54508 0.102845  6.35385 1\n-4.56112  -1.74917  -4.57874  0\n-4.31568  -0.396959 -6.29507  0\n\\endverbatim\n\nThis program can also run distributed by using\n\\verbatim\n> mpiexec -n [N machines] --hostfile [host file] ./kmeans ....\n\\endverbatim\nSee your MPI documentation for details on how to launch this job.\nAll machines must have access to the input graph location and the output graph \nlocation. Graphs may be on HDFS. \nIf you have problems loading HDFS files, see the \\ref FAQ.\n\n\n\\subsection clustering_kmeans_sparse --sparse Option\n\nIf --sparse=1 is set (default is 0), the program will use \na sparse vector representation. \nThe file format is [feature id]:[value] [feature id]:[value] ...\nEach line corresponds to a datapoint.\n[feature id] must be positive integer or zero.\n\nFor instance:\n\\verbatim\n> cat synthetic_sparse.txt\n0:-4.31568  3:-0.396959 5:-6.29507  \n5:-4.56112  9:-1.74917  \n4:4.54508 5:0.102845  10:6.35385 \n0:4.87746 7:-0.832591 \n\\endverbatim\n\n\n\\subsection clustering_kmeans_id --id Option\n\nIf --id=1 is set (default is 0), the program will use \nid for each data point. The id of a data point must be written at the head \nof each line of the input data.\n\nFor instance:\n\\verbatim\n> cat synthetic_with_id.txt\n1 -4.31568  -0.396959 -6.29507  \n2 -4.56112  -1.74917  -4.57874  \n3 4.54508 0.102845  6.35385 \n4 4.87746 -0.832591 7.06942 \n\\endverbatim\n\nThe output data will consist of two columns: one for the ids and the other for \nthe assigned clusters.\n\nFor instance:\n\\verbatim\n> cat data_with_id.txt\n1 0\n2 0  \n3 1\n4 1 \n\\endverbatim\n\n--id can be used with --sparse.\n\n\\subsection clustering_kmeans_edge_data Adding Pairwise Reward\n\nThis program can consider pairwise rewards (and penalty) by using \n--pairwise-reward=[file prefix]. All files beginning with \n[file prefix] will be loaded for pairwise rewards. Each \nline of the pairwise reward file holds [id1] [id2] [weight].\nThis option must be used with --id=1.\n \nFor instance:\n\\verbatim\n> cat pairwise_data.txt\n1 2 10\n2 4 -10\n\\endverbatim\n\nIn this example the evaluation function will gain 10 when data 1 \nand data 2 are in the same cluster; it will gain nothing otherwise.\n\n\n\\subsection clustering_kmeans_options Options\n\\li \\b --data (Required). The prefix from which to load the input data\n\\li \\b --clusters (Required). The number of cluster centers\n\\li \\b --ncpus (Optional. Default 2) The number of processors that will be used\nfor computation. Due to some implementation limitations within GraphLab,\n   this parameter is not respected. It will use all processors on your machine\n   if ran in Linux, and will use only 1 processor if ran on Mac \n\\li \\b --output-data  (Optional) A target prefix to write the output data \n   with cluster assignments. May be on HDFS.\n\\li \\b --output-clusters (Optional) A target location to write the cluster centers.\n   Must be on the local file system.\n\\li \\b --sparse (Optional. Default 0) If set at 1, will use sparse vector representation\n\\li \\b --id (Optional. Default 0) If set at 1, will use ids for data points\n\\li \\b --pairwise-reward (Optional) If set, will consider pairwise rewards written in the \n   files beginning with the given argument\n\\li \\b --max-iteration (Optional) The max number of iterations\n\n\n\n\n\n\n\n\\section clustering_spectral_clustering Spectral Clustering\n\nThis program takes as input a collection of files where each line in each file represents \na data point. Each line must contains an id and a list of numbers, white-space or comma \nseparated. Each line must be the same length. \nFor instance in this example input file, there are 6 datapoints, one per line,\nand each datapoint is a point in a 5-dimensional space.  \n\n\\verbatim\n1 -10.7551  6.82178 5.33455 -2.08247  2.86694\n2 -1.36687  10.8464 -5.28851  -4.26768  -5.50659\n3 -8.79834  8.01002 5.33418 0.102824  3.23318 \n4 -8.64345  6.81946 1.2309  -4.46784  2.26341 \n5 -8.29782  7.1154  3.32559 -2.59422  2.33936\n6 -8.12504  8.98924 4.15027 0.253153  1.75911 \n\\endverbatim\n\nThe ids of data points must start from 1 and must not skip any numbers.\n\nTo run spectral clustering, the minimal set of options required are:\n\n\\verbatim\n> ./spectral_clustering --data=[data prefix] --clusters=[N cluster]\n\\endverbatim\n\nThis program uses svd in Graphlab Collaborative Filtering Toolkit, \nkmeans in Graphlab Clustering Toolkit and eigen_vector_normalization in \nGraphlab Graph Analytics Toolkit.  The paths to the directories are\nspecified by --svd-dir, --kmeans-dir and \n--graph-analytics-dir, respectively.\n\nThe program will create some intermediate files. The final clustering\nresult is written in files named [data prefix].result with a suffix,\nfor example [data prefix].result_1_of_4. The clustering result\ndata will consist of two columns: one for the ids and the other for the\nassigned clusters. For instance:\n\n\\verbatim\n1 1\n2 0\n3 1\n4 1\n5 1\n\\endverbatim\n\nNOTE: To run the spectral clustering in a distributed setting, you must use the \n\"mpi-args\" option, not like other graphlab toolkits. \nThe spectral clustering calls other graphlab programs.\nWhen \"--mpi-args\" is set, these graphlab programs are called with \"mpiexec\" and the \nstring written after the \"mpi-args\" option.\nFor example, if you set --mpi-args=\"-n 4 --hostfile host\", the program calls the \nother graphlab programs with \"mpiexec -n 4 --hostfile host\".\n\n\\subsection Options\nRelevant options are:\n\\li \\b --data (Required). The prefix from which to load the input data\n\\li \\b --clusters (Required). The number of clusters\n\\li \\b --sigma (Optional. Default 0.1). Scale parameter for Gaussian kernel.\nThis value is often critical to the clustering result.\n\\li \\b --t-nearest (Optional. Default 20). Number of nearest neighbors (=t). \nWill use only the t-nearest similarities for each datapoint. If set at 0,\nwill use all similarities. \n\\li \\b --similarity-thres (Optional). Threshold to discard small similarities.\nIf a value is set, similarities less than this value will be discarded.\n\\li \\b --svd-dir (Optional Default ../collaborative_filtering/).\nPath to the directory where Graphlab svd is located\n\\li \\b --kmeans-dir (Optional. Default ../clustering/). \nPath to the directory where Graphlab kmeans is located\n\\li \\b --graph-analytics-dir (Optional. Default ../graph_analytics/).\nPath to the directory where Graphlab eigen_vector_normalization is located\n\\li \\b --ncpus (Optional. Default 2). The number of processors that will be used for computation. \nDue to some implementation limitations within GraphLab, \nthis parameter is not respected. It will use all processors on your machine \nif ran in Linux, and will use only 1 processor if ran on Mac \n\\li \\b --graph_opts (Optional, Default empty). Any additional graph options. See\n  graphlab::distributed_graph a list of options.\n\\li \\b --mpi-args (Optional, Default empty). If set, will execute mipexec with the given string.\n\n\n*/\n\n\n"
  },
  {
    "path": "toolkits/clustering/generate_synthetic.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \n#include \n\nint main(int argc, char** argv) {\n  std::ofstream fout(\"synthetic.txt\");\n  \n  size_t num_clusters = 2;\n  size_t dim = 2;\n  size_t ndata = 10000;\n  if (argc >= 2) num_clusters = atoi(argv[1]);\n  if (argc >= 3) dim = atoi(argv[2]);\n  if (argc >= 4) ndata = atoi(argv[3]);\n\n  std::cout << \"Usage: generate_synthetic [NClusters] [Dimensions] [Ndata]\\n\";\n\n  std::vector< std::vector > centers(num_clusters);\n  for (size_t i = 0;i < centers.size(); ++i) {\n    std::cout << \"Center \" << i << \" at: \" ; \n    for (size_t j = 0; j < dim; ++j) {\n      double r = graphlab::random::fast_uniform(-10,10);\n      std::cout << r << \"\\t\";\n      centers[i].push_back(r);\n    }\n    std::cout << \"\\n\";\n  }\n\n  // now generate data points\n  // \n  for (size_t i = 0;i < ndata; ++i) {\n    size_t c = graphlab::random::fast_uniform(0, centers.size() - 1);\n    for (size_t j = 0; j < dim; ++j) {\n      double d = graphlab::random::gaussian() + centers[c][j];\n      fout << d << \"\\t\";\n    }\n    fout << \"\\n\";\n  }\n}\n"
  },
  {
    "path": "toolkits/clustering/graph_laplacian_for_sc.cpp",
    "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n\n//shared parameters\nfloat gaussian_kernel_scale_parameter = 0.1;\nfloat threshold_to_discard_small_similarities = 0.0;\nsize_t number_of_nearest_neighbors= 20;\n\n\n//data point\nstruct vertex_data {\n  std::vector x;\n  float D_ii;\n  vertex_data():x(), D_ii(0.0) {};\n  explicit vertex_data(const std::vector& x_in) :\n      x(x_in), D_ii(0.0) {}\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << x.size();\n    for(size_t i=0;i> size;\n    for(size_t i=0;i> temp;\n      x.push_back(temp);\n    }\n    iarc >> D_ii;\n  }\n};\n\n//similarity\nstruct edge_data{\n  float A_ij;\n  bool nearest;\n  edge_data() : A_ij(0.0), nearest(false){}\n  void save(graphlab::oarchive& oarc) const {\n    oarc << A_ij << nearest;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> A_ij >> nearest;\n  }\n};\n\ntypedef graphlab::distributed_graph graph_type;\n\n\n//[vertex_id] [element1] [element2] [element3] ...\nbool line_parser(graph_type& graph, const std::string& filename,\n    const std::string& line) {\n  if (line.empty()) return true;\n  size_t id = 0;\n  namespace qi = boost::spirit::qi;\n  namespace ascii = boost::spirit::ascii;\n  namespace phoenix = boost::phoenix;\n  vertex_data vtx;\n  const bool success = qi::phrase_parse\n    (line.begin(), line.end(),\n     //  Begin grammar\n     (\n      qi::ulong_[phoenix::ref(id) = qi::_1] >> -qi::char_(\",\") >>\n      (qi::double_[phoenix::push_back(phoenix::ref(vtx.x), qi::_1)] % -qi::char_(\",\") )\n      )\n     ,\n     //  End grammar\n     ascii::space);\n  if (!success) return false;\n  graph.add_vertex(id, vtx);\n\n  for(size_t i=1;i& v1, const std::vector& v2) {\n  float ret = 0.0;\n  for (size_t i = 0; i < v1.size(); ++i) {\n    float tmp = v1[i] - v2[i];\n    ret += tmp * tmp;\n  }\n  return exp(-ret / gaussian_kernel_scale_parameter);\n}\n\n//calculate similarities between data points\nvoid calc_similarities(graph_type::edge_type& edata) {\n  edata.data().A_ij = similarity(edata.source().data().x, edata.target().data().x);\n}\n\n\n//discard small similarities (Optional)\nvoid discard_small_similarity(graph_type::edge_type& edata) {\n  if(edata.data().A_ij < threshold_to_discard_small_similarities)\n    edata.data().A_ij = 0.0;\n}\n\n\n//gather T-nearest neighbor (Optional)\nstruct top_t_similarity{\n  std::vector ids;\n  std::vector sims;\n  top_t_similarity(): ids(number_of_nearest_neighbors, std::numeric_limits::max()),\n      sims(number_of_nearest_neighbors, -1.0){}\n  top_t_similarity(size_t id, float sim): ids(number_of_nearest_neighbors, std::numeric_limits::max()),\n      sims(number_of_nearest_neighbors, -1.0){\n    ids[0] = id;\n    sims[0] = sim;\n  }\n\n  top_t_similarity& operator+=(const top_t_similarity& other){\n    std::vector new_ids;\n    std::vector new_sims;\n    size_t pos1=0;\n    size_t pos2=0;\n    while(pos1+pos2 < number_of_nearest_neighbors){\n      if(sims[pos1] >= other.sims[pos2]){\n        new_ids.push_back(ids[pos1]);\n        new_sims.push_back(sims[pos1]);\n        pos1++;\n      }else{\n        new_ids.push_back(other.ids[pos2]);\n        new_sims.push_back(other.sims[pos2]);\n        pos2++;\n      }\n    }\n    ids = new_ids;\n    sims = new_sims;\n    return *this;\n  }\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << ids.size();\n    for(size_t i=0;i> size;\n    for(size_t i=0;i> id;\n      ids.push_back(id);\n    }\n    for(size_t i=0;i> sim;\n      sims.push_back(sim);\n    }\n  }\n};\n\n//get T-nearest neighbor and discard others (Optional)\nclass t_nearest: public graphlab::ivertex_program, public graphlab::IS_POD_TYPE {\nprivate:\n  float threshold;\n\npublic:\n  t_nearest():threshold(0.0){}\n\n  edge_dir_type gather_edges(icontext_type& context,\n      const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  }\n  top_t_similarity gather(icontext_type& context, const vertex_type& vertex,\n      edge_type& edge) const {\n    if(edge.target().id() == vertex.id()){//in edge\n      return top_t_similarity(edge.source().id(), edge.data().A_ij);\n    }else{//out edge\n      return top_t_similarity(edge.target().id(), edge.data().A_ij);\n    }\n  }\n\n  //assign a cluster, considering the clusters of neighbors\n  void apply(icontext_type& context, vertex_type& vertex,\n      const gather_type& total) {\n    threshold = total.sims[number_of_nearest_neighbors-1];\n//    std::cout << vertex.id() << \"\\t\" << total.ids[0] << \"-\" << total.sims[0] << \", \"\n//        << total.ids[1] << \"-\" << total.sims[1] << std::endl;\n  }\n\n  edge_dir_type scatter_edges(icontext_type& context,\n      const vertex_type& vertex) const {\n      return graphlab::ALL_EDGES;\n  }\n  void scatter(icontext_type& context, const vertex_type& vertex,\n      edge_type& edge) const {\n    if(edge.data().A_ij >= threshold)\n      edge.data().nearest = true;\n  }\n};\n\n//discard small similarities (Optional)\nvoid make_other_similarities_zero(graph_type::edge_type& edata) {\n  if(edata.data().nearest == false)\n    edata.data().A_ij = 0.0;\n}\n\n\n//compute sums over rows and then take inverse square root\nclass calc_degrees: public graphlab::ivertex_program, public graphlab::IS_POD_TYPE {\npublic:\n  //gather A_ij\n  edge_dir_type gather_edges(icontext_type& context,\n      const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  }\n  float gather(icontext_type& context, const vertex_type& vertex,\n      edge_type& edge) const {\n    return edge.data().A_ij;\n  }\n\n  //assign a cluster, considering the clusters of neighbors\n  void apply(icontext_type& context, vertex_type& vertex,\n      const gather_type& total) {\n    vertex.data().D_ii = 1.0 / sqrt(total);\n  }\n\n  edge_dir_type scatter_edges(icontext_type& context,\n      const vertex_type& vertex) const {\n      return graphlab::NO_EDGES;\n  }\n  void scatter(icontext_type& context, const vertex_type& vertex,\n      edge_type& edge) const {\n  }\n};\n\n//multiply D^-1/2\nvoid mult_D(graph_type::edge_type& edata) {\n  edata.data().A_ij = edata.data().A_ij * edata.source().data().D_ii * edata.target().data().D_ii;\n}\n\nstruct max_min_similarity{\n  float max_sim;\n  float min_sim;\n\n  max_min_similarity(): max_sim(0.0), min_sim(0.0){}\n  explicit max_min_similarity(float similarity): max_sim(similarity),\n      min_sim(similarity){}\n\n  max_min_similarity& operator+=(const max_min_similarity& other){\n    if(max_sim < 1.0 && other.max_sim < 1.0){\n      max_sim = std::max(max_sim, other.max_sim);\n    }else if(other.max_sim < 1.0){\n      max_sim = other.max_sim;\n    }\n    if(min_sim > 0.0 && other.min_sim > 0.0){\n      min_sim = std::min(min_sim, other.min_sim);\n    }else if(other.min_sim > 0.0){\n      min_sim = other.min_sim;\n    }\n    return *this;\n  }\n  void save(graphlab::oarchive& oarc) const {\n    oarc << max_sim << min_sim;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> max_sim >> min_sim;\n  }\n};\nmax_min_similarity absolute_edge_data(const graph_type::edge_type& edge) {\n  return max_min_similarity(edge.data().A_ij);\n}\n\nstruct max_vid{\n  size_t vid;\n  max_vid(): vid(0){}\n  explicit max_vid(size_t in_vid): vid(in_vid){}\n\n  max_vid& operator+=(const max_vid& other){\n    vid = std::max(vid, other.vid);\n    return *this;\n  }\n  void save(graphlab::oarchive& oarc) const {\n    oarc << vid;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> vid;\n  }\n};\nmax_vid absolute_vertex_data(const graph_type::vertex_type& vertex) {\n  return max_vid(vertex.id());\n}\n\nclass graph_writer {\npublic:\n  std::string save_vertex(graph_type::vertex_type v) {\n    std::stringstream strm;\n    size_t vid = v.id();\n    if(vid == 0)\n      return \"\";\n    strm << vid << \" \" << vid << \" 1.0\\n\";\n    return strm.str();\n  }\n\n  std::string save_edge(graph_type::edge_type e) {\n    const float& A_ij = e.data().A_ij;\n    std::stringstream strm;\n    if(A_ij > 0.0){\n      strm << e.source().id() << \" \" << e.target().id() << \" \" <<\n          A_ij << \"\\n\";\n      strm << e.target().id() << \" \" << e.source().id() << \" \" <<\n          A_ij << \"\\n\";\n    }\n    return strm.str();\n  }\n};\n\nint main(int argc, char** argv) {\n  std::cout << \"construct graph Laplacian for spectral clustering.\\n\\n\";\n\n  //parse command line\n  std::string datafile;\n  graphlab::command_line_options clopts\n    (\"Constructing graph Laplacian for spectral clustering\");\n  clopts.attach_option(\"data\", datafile,\n                       \"Input file. Each line hold a sample id followed by a white-space or \"\n                       \"comma separated numeric vector. Id should start from 1\");\n  clopts.attach_option(\"sigma\",  gaussian_kernel_scale_parameter,\n                       \"Scale parameter for Gaussian kernel.\");\n  clopts.attach_option(\"similarity-thres\", threshold_to_discard_small_similarities,\n                       \"Threshold to discard small similarities. \");\n  clopts.attach_option(\"t-nearest\", number_of_nearest_neighbors,\n                      \"Number of nearest neighbors (=t). Will use only the t-nearest similarities \"\n                      \"for each datapoint. If set at 0, will use all similarities.\");\n  if(!clopts.parse(argc, argv)) return EXIT_FAILURE;\n  if (datafile == \"\") {\n    std::cout << \"--data is not optional\\n\";\n    return EXIT_FAILURE;\n  }\n  gaussian_kernel_scale_parameter *= 2.0*gaussian_kernel_scale_parameter;\n\n  //construct graph\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  graph_type graph(dc, clopts);\n  graph.load(\n      datafile,\n      line_parser);\n  graph.finalize();\n\n  time_t start, end;\n  time(&start);\n  size_t data_num = graph.map_reduce_vertices(absolute_vertex_data).vid;\n\n  //calculate similarities\n  graph.transform_edges(calc_similarities);\n\n  //show the max similarity less than 1 and the min similarity grater than 0\n  max_min_similarity stat = graph.map_reduce_edges(absolute_edge_data);\n  dc.cout() << \"max squared distance(min similarity): \"\n      << -log(stat.min_sim)*gaussian_kernel_scale_parameter\n      << \"(\" << stat.min_sim << \")\\n\"\n      << \"min squared distance(max similarity):\"\n      << -log(stat.max_sim)*gaussian_kernel_scale_parameter\n      << \"(\" << stat.max_sim << \")\\n\";\n\n\n  //if t is set, use only t-nearest similarities\n  if(number_of_nearest_neighbors > 0){\n    if(number_of_nearest_neighbors > data_num-1)\n      number_of_nearest_neighbors = data_num-1;\n    dc.cout() << \"use only the \" << number_of_nearest_neighbors\n        << \"-nearest similarities for each datapoint\\n\";\n    graphlab::omni_engine engine_nearest(dc, graph, \"sync\", clopts);\n    engine_nearest.signal_all();\n    engine_nearest.start();\n    graph.transform_edges(make_other_similarities_zero);\n  }\n  //if threshold is set, discard similarities less then the threshold\n  if(threshold_to_discard_small_similarities > 0.0){\n    dc.cout() << \"discard small similarities less than \"\n        << threshold_to_discard_small_similarities << \"\\n\";\n    graph.transform_edges(discard_small_similarity);\n  }\n\n  //sum elements over rows (calculate the degree matrix D)\n  graphlab::omni_engine engine(dc, graph, \"sync\", clopts);\n  engine.signal_all();\n  engine.start();\n  //multiply D\n  graph.transform_edges(mult_D);\n  time(&end);\n\n  dc.cout() << \"graph calculation time is \" << (end - start) << \" sec\\n\";\n  dc.cout() << \"writing...\\n\";\n\n  //write results\n  const std::string outputname = datafile + \".glap\";\n  graph.save(\n      outputname + \"_diag\",\n      graph_writer(), false, //set to true if each output file is to be gzipped\n      true, //whether vertices are saved\n      false,1); //whether edges are saved\n  graph.save(\n      outputname + \"_other\",\n      graph_writer(), false, //set to true if each output file is to be gzipped\n      false, //whether vertices are saved\n      true,1); //whether edges are saved\n\n  //write the number of data\n  const std::string datanum_filename = datafile + \".datanum\";\n  std::ofstream ofs(datanum_filename.c_str());\n  if(!ofs) {\n    std::cout << \"can't create file for number of data\" << std::endl;\n    return EXIT_FAILURE;\n  }\n  ofs << data_num;\n\n  graphlab::mpi_tools::finalize();\n\n  return EXIT_SUCCESS;\n}\n\n"
  },
  {
    "path": "toolkits/clustering/kmeans.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * This implements the classical \"k-means\" clustering algorithm.\n *\n * It takes as input file a series of lines where each line is a comma separated\n * or space separated list of values representing a vector. For instance:\n *\n * \\verbatim\n * 1.1, 1.5, 0.9\n * 0.3, 0.4, -1.1\n * ...\n * \\endverbatim\n *\n * It constructs a graph with a single vertex for each data point and simply\n * uses the \"Map-Reduce\" scheme to perform a k-means clustering of all\n * the datapoints.\n */\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n\n#include \n\n\nsize_t NUM_CLUSTERS = 0;\nbool IS_SPARSE = false;\n\nstruct cluster {\n  cluster(): count(0), changed(false) { }\n  std::vector center;\n  std::map center_sparse;\n  size_t count;\n  bool changed;\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << center << count << changed << center_sparse;\n  }\n\n  void load(graphlab::iarchive& iarc) {\n    iarc >> center >> count >> changed >> center_sparse;\n  }\n};\n\nstd::vector CLUSTERS;\n\n// the current cluster to initialize\nsize_t KMEANS_INITIALIZATION;\n\nstruct vertex_data{\n  std::vector point;\n  std::map point_sparse;\n  size_t best_cluster;\n  double best_distance;\n  bool changed;\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << point << best_cluster << best_distance << changed << point_sparse;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> point >> best_cluster >> best_distance >> changed >> point_sparse;\n  }\n};\n\n//use edges when edge weight file is given\nstruct edge_data {\n  double weight;\n\n  edge_data() :\n      weight(0.0) {\n  }\n  explicit edge_data(double w) :\n      weight(w) {\n  }\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << weight;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> weight;\n  }\n};\n\n// helper function to compute distance between points\ndouble sqr_distance(const std::vector& a,\n                    const std::vector& b) {\n  ASSERT_EQ(a.size(), b.size());\n  double total = 0;\n  for (size_t i = 0;i < a.size(); ++i) {\n    double d = a[i] - b[i];\n    total += d * d;\n  }\n  return total;\n}\n\ndouble sqr_distance(const std::map& a,\n                    const std::map& b) {\n  double total = 0.0;\n  for(std::map::const_iterator iter = a.begin();\n      iter != a.end(); ++iter){\n    size_t id = (*iter).first;\n    double val = (*iter).second;\n    if(b.find(id) != b.end()){\n      double d = val - b.at(id);\n      total += d*d;\n    }else{\n      total += val * val;\n    }\n  }\n  for(std::map::const_iterator iter = b.begin();\n      iter != b.end(); ++iter){\n    double val = (*iter).second;\n    if(a.find((*iter).first) == a.end()){\n      total += val * val;\n    }\n  }\n\n  return total;\n\n////   cosine distance is better for sparse datapoints?\n//    double ip = 0.0;\n//    double lenA = 0.0;\n//    double lenB = 0.0;\n//    for(std::map::const_iterator iter = a.begin();\n//        iter != a.end(); ++iter){\n//      size_t id = (*iter).first;\n//      double val = (*iter).second;\n//      if(b.find(id) != b.end()){\n//        ip += val * b.at(id);\n//      }\n//      lenA += val*val;\n//    }\n//\n//    if(ip == 0.0 || lenA == 0.0)\n//      return 1.0;\n//\n//    for(std::map::const_iterator iter = b.begin();\n//        iter != b.end(); ++iter){\n//      double val = (*iter).second;\n//      lenB += val * val;\n//    }\n//\n//    if(lenB == 1.0)\n//      return 1.0;\n//\n//    return 1.0 - ip/(sqrt(lenA)*sqrt(lenB));\n\n}\n\n\n// helper function to add two vectors\nstd::vector& plus_equal_vector(std::vector& a,\n                                       const std::vector& b) {\n  ASSERT_EQ(a.size(), b.size());\n  for (size_t i = 0;i < a.size(); ++i) {\n    a[i] += b[i];\n  }\n  return a;\n}\n\n// helper function to add two vectors\nstd::map& plus_equal_vector(std::map& a,\n                                       const std::map& b) {\n  for(std::map::const_iterator iter = b.begin();\n    iter != b.end(); ++iter){\n    size_t id = (*iter).first;\n    double val = (*iter).second;\n    if(a.find(id) != a.end()){\n      a[id] += b.at(id);\n    }else{\n      a.insert(std::make_pair(id, val));\n    }\n  }\n  return a;\n}\n\n// helper function to scale a vector vectors\nstd::vector& scale_vector(std::vector& a, double d) {\n  for (size_t i = 0;i < a.size(); ++i) {\n    a[i] *= d;\n  }\n  return a;\n}\n\n// helper function to scale a vector vectors\nstd::map& scale_vector(std::map& a, double d) {\n  for(std::map::iterator iter = a.begin();\n    iter != a.end(); ++iter){\n  size_t id = (*iter).first;\n  double val = (*iter).second;\n  a[id] = val*d;\n//    (*iter).second *= d;\n  }\n  return a;\n}\n\n\ntypedef graphlab::distributed_graph graph_type;\n\ngraphlab::atomic NEXT_VID;\n\n// Read a line from a file and creates a vertex\nbool vertex_loader(graph_type& graph, const std::string& fname,\n                   const std::string& line) {\n  if (line.empty()) return true;\n  namespace qi = boost::spirit::qi;\n  namespace ascii = boost::spirit::ascii;\n  namespace phoenix = boost::phoenix;\n  vertex_data vtx;\n  const bool success = qi::phrase_parse\n    (line.begin(), line.end(),\n     //  Begin grammar\n     (\n      (qi::double_[phoenix::push_back(phoenix::ref(vtx.point), qi::_1)] % -qi::char_(\",\") )\n      )\n     ,\n     //  End grammar\n     ascii::space);\n\n  if (!success) return false;\n  vtx.best_cluster = (size_t)(-1);\n  vtx.best_distance = std::numeric_limits::infinity();\n  vtx.changed = false;\n  graph.add_vertex(NEXT_VID.inc_ret_last(1), vtx);\n  return true;\n}\n\n// Read a line from a file and creates a vertex\nbool vertex_loader_sparse(graph_type& graph, const std::string& fname,\n                   const std::string& line) {\n  if (line.empty()) return true;\n\n  vertex_data vtx;\n  boost::char_separator sep(\" \");\n  boost::tokenizer< boost::char_separator > tokens(line, sep);\n  BOOST_FOREACH (const std::string& t, tokens) {\n    std::string::size_type pos = t.find(\":\");\n    if(pos > 0){\n      size_t id = (size_t)std::atoi(t.substr(0, pos).c_str());\n      double val = std::atof(t.substr(pos+1, t.length() - pos -1).c_str());\n      vtx.point_sparse.insert(std::make_pair(id, val));\n    }\n  }\n  vtx.best_cluster = (size_t)(-1);\n  vtx.best_distance = std::numeric_limits::infinity();\n  vtx.changed = false;\n  graph.add_vertex(NEXT_VID.inc_ret_last(1), vtx);\n  return true;\n}\n\n// Read a line from a file and creates a vertex\nbool vertex_loader_with_id(graph_type& graph, const std::string& fname,\n                   const std::string& line) {\n  if (line.empty()) return true;\n  size_t id = 0;\n  namespace qi = boost::spirit::qi;\n  namespace ascii = boost::spirit::ascii;\n  namespace phoenix = boost::phoenix;\n  vertex_data vtx;\n  const bool success = qi::phrase_parse\n    (line.begin(), line.end(),\n     //  Begin grammar\n     (\n      qi::ulong_[phoenix::ref(id) = qi::_1] >> -qi::char_(\",\") >>\n      (qi::double_[phoenix::push_back(phoenix::ref(vtx.point), qi::_1)] % -qi::char_(\",\") )\n      )\n     ,\n     //  End grammar\n     ascii::space);\n\n  if (!success) return false;\n  vtx.best_cluster = (size_t)(-1);\n  vtx.best_distance = std::numeric_limits::infinity();\n  vtx.changed = false;\n  graph.add_vertex(id, vtx);\n  return true;\n}\n\n// Read a line from a file and creates a vertex\nbool vertex_loader_with_id_sparse(graph_type& graph, const std::string& fname,\n                   const std::string& line) {\n  if (line.empty()) return true;\n\n  vertex_data vtx;\n  size_t id = 0;\n  boost::char_separator sep(\" \");\n  boost::tokenizer > tokens(line, sep);\n  bool first = true;\n  BOOST_FOREACH (const std::string& t, tokens) {\n    if(first){\n      id = (size_t)std::atoi(t.c_str());\n      first = false;\n    }else{\n      std::string::size_type pos = t.find(\":\");\n      if(pos > 0){\n        size_t id = (size_t)std::atoi(t.substr(0, pos).c_str());\n        double val = std::atof(t.substr(pos+1, t.length() - pos -1).c_str());\n        vtx.point_sparse.insert(std::make_pair(id, val));\n      }\n    }\n  }\n  vtx.best_cluster = (size_t)(-1);\n  vtx.best_distance = std::numeric_limits::infinity();\n  vtx.changed = false;\n  graph.add_vertex(id, vtx);\n  return true;\n}\n\n\n\n//call this when edge weight file is given.\n//each line should be [source id] [target id] [weight].\n//directions of edges are ignored.\nbool edge_loader(graph_type& graph, const std::string& filename,\n    const std::string& textline) {\n  if (textline.empty())\n    return true;\n  std::stringstream strm(textline);\n  size_t source_vid = 0;\n  size_t target_vid = 0;\n  double weight = 0.0;\n  strm >> source_vid;\n  strm.ignore(1);\n  strm >> target_vid;\n  strm.ignore(1);\n  strm >> weight;\n  if(source_vid != target_vid)\n    graph.add_edge(source_vid, target_vid, edge_data(weight));\n  return true;\n}\n\n\n// A set of Map Reduces to compute the maximum and minimum vector sizes\n// to ensure that all vectors have the same length\nstruct max_point_size_reducer: public graphlab::IS_POD_TYPE {\n  size_t max_point_size;\n\n  static max_point_size_reducer get_max_point_size(const graph_type::vertex_type& v) {\n    max_point_size_reducer r;\n    r.max_point_size = v.data().point.size();\n    return r;\n  }\n\n  max_point_size_reducer& operator+=(const max_point_size_reducer& other) {\n    max_point_size = std::max(max_point_size, other.max_point_size);\n    return *this;\n  }\n};\n\nstruct min_point_size_reducer: public graphlab::IS_POD_TYPE {\n  size_t min_point_size;\n\n  static min_point_size_reducer get_min_point_size(const graph_type::vertex_type& v) {\n    min_point_size_reducer r;\n    r.min_point_size = v.data().point.size();\n    return r;\n  }\n\n  min_point_size_reducer& operator+=(const min_point_size_reducer& other) {\n    min_point_size = std::min(min_point_size, other.min_point_size);\n    return *this;\n  }\n};\n\n\n/*\n * This transform vertices call is only used during\n * the initialization phase. It computes distance to\n * cluster[KMEANS_INITIALIZATION] and assigns itself\n * to the new cluster KMEANS_INITIALIZATION if the new distance\n * is smaller that its previous cluster assignment\n */\nvoid kmeans_pp_initialization(graph_type::vertex_type& v) {\n  double d = sqr_distance(v.data().point,\n                          CLUSTERS[KMEANS_INITIALIZATION].center);\n  if (v.data().best_distance > d) {\n    v.data().best_distance = d;\n    v.data().best_cluster = KMEANS_INITIALIZATION;\n  }\n}\n\nvoid kmeans_pp_initialization_sparse(graph_type::vertex_type& v) {\n  double d = sqr_distance(v.data().point_sparse,\n                          CLUSTERS[KMEANS_INITIALIZATION].center_sparse);\n  if (v.data().best_distance > d) {\n    v.data().best_distance = d;\n    v.data().best_cluster = KMEANS_INITIALIZATION;\n  }\n}\n\n\n/*\n * Draws a random sample from the data points that is \n * proportionate to the \"best distance\" stored in the vertex.\n */\nstruct random_sample_reducer {\n  std::vector vtx;\n  double weight;\n\n  random_sample_reducer():weight(0) { }\n  random_sample_reducer(const std::vector& vtx,\n                        double weight):vtx(vtx),weight(weight) { }\n\n  static random_sample_reducer get_weight(const graph_type::vertex_type& v) {\n    if (v.data().best_cluster == (size_t)(-1)) {\n      return random_sample_reducer(v.data().point, 1);\n    }\n    else {\n      return random_sample_reducer(v.data().point,\n                                   v.data().best_distance);\n    }\n  }\n\n  random_sample_reducer& operator+=(const random_sample_reducer& other) {\n    double totalweight = weight + other.weight;\n    // if any weight is too small, just quit\n    if (totalweight <= 0) return *this;\n\n    double myp = weight / (weight + other.weight);\n    if (graphlab::random::bernoulli(myp)) {\n      weight += other.weight;\n      return *this;\n    }\n    else {\n      vtx = other.vtx;\n      weight += other.weight;\n      return *this;\n    }\n  }\n\n  void save(graphlab::oarchive &oarc) const {\n    oarc << vtx << weight;\n  }\n\n  void load(graphlab::iarchive& iarc) {\n    iarc >> vtx >> weight;\n  }\n};\n\nstruct random_sample_reducer_sparse{\n  std::map vtx;\n  double weight;\n\n  random_sample_reducer_sparse():weight(0) { }\n  random_sample_reducer_sparse(const std::map& vtx,\n                        double weight):vtx(vtx),weight(weight) { }\n\n  static random_sample_reducer_sparse get_weight(const graph_type::vertex_type& v) {\n    if (v.data().best_cluster == (size_t)(-1)) {\n      return random_sample_reducer_sparse(v.data().point_sparse, 1);\n    }\n    else {\n      return random_sample_reducer_sparse(v.data().point_sparse,\n                                   v.data().best_distance);\n    }\n  }\n\n  random_sample_reducer_sparse& operator+=(const random_sample_reducer_sparse& other) {\n    double totalweight = weight + other.weight;\n    // if any weight is too small, just quit\n    if (totalweight <= 0) return *this;\n\n    double myp = weight / (weight + other.weight);\n    if (graphlab::random::bernoulli(myp)) {\n      weight += other.weight;\n      return *this;\n    }\n    else {\n      vtx = other.vtx;\n      weight += other.weight;\n      return *this;\n    }\n  }\n\n  void save(graphlab::oarchive &oarc) const {\n    oarc << vtx << weight;\n  }\n\n  void load(graphlab::iarchive& iarc) {\n    iarc >> vtx >> weight;\n  }\n};\n\n\n/*\n * This transform vertices call is used during the \n * actual k-means iteration. It computes distance to \n * all \"changed\" clusters and reassigns itself if necessary\n */\nvoid kmeans_iteration(graph_type::vertex_type& v) {\n  // if current vertex's cluster was modified, we invalidate the distance.\n  // and we need to recompute to all existing clusters\n  // otherwise, we just need to recompute to changed cluster centers.\n  size_t prev_asg = v.data().best_cluster;\n  if (CLUSTERS[v.data().best_cluster].changed) {\n    // invalidate. recompute to all\n    v.data().best_cluster = (size_t)(-1);\n    v.data().best_distance = std::numeric_limits::infinity();\n    for (size_t i = 0;i < NUM_CLUSTERS; ++i) {\n      if (CLUSTERS[i].center.size() > 0 || CLUSTERS[i].center_sparse.size() > 0) {\n        double d = 0.0;\n        if(IS_SPARSE == true)\n          d = sqr_distance(v.data().point_sparse, CLUSTERS[i].center_sparse);\n        else\n          d = sqr_distance(v.data().point, CLUSTERS[i].center);\n        if (d < v.data().best_distance) {\n          v.data().best_distance = d;\n          v.data().best_cluster = i;\n        }\n      }\n    }\n  }\n  else {\n    // just compute distance to what has changed\n    for (size_t i = 0;i < NUM_CLUSTERS; ++i) {\n      if (CLUSTERS[i].changed &&\n          (CLUSTERS[i].center.size() > 0 || CLUSTERS[i].center_sparse.size() > 0)) {\n        double d = 0.0;\n        if(IS_SPARSE == true)\n          d = sqr_distance(v.data().point_sparse, CLUSTERS[i].center_sparse);\n        else\n          d= sqr_distance(v.data().point, CLUSTERS[i].center);\n        if (d < v.data().best_distance) {\n          v.data().best_distance = d;\n          v.data().best_cluster = i;\n        }\n      }\n    }\n  }\n  v.data().changed = (prev_asg != v.data().best_cluster);\n}\n\n//gathered information\n//used when edge weight file is given\nstruct neighbor_info {\n  std::map cw_map;\n\n  neighbor_info() :\n      cw_map() {\n  }\n  neighbor_info(size_t clst, double weight) :\n      cw_map() {\n    cw_map.insert(std::make_pair(clst, weight));\n  }\n\n  neighbor_info& operator+=(const neighbor_info& other) {\n    for (std::map::const_iterator iter = other.cw_map.begin();\n        iter != other.cw_map.end(); iter++) {\n      size_t clst = iter->first;\n      if (cw_map.find(clst) == cw_map.end()) {\n        cw_map.insert(std::make_pair(clst, iter->second));\n      } else {\n        cw_map[clst] += iter->second;\n      }\n    }\n    return *this;\n  }\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << cw_map;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> cw_map;\n  }\n};\n\n//used when edge weight file is given\nclass cluster_assignment: public graphlab::ivertex_program, public graphlab::IS_POD_TYPE {\npublic:\n  //gather on all the edges\n  edge_dir_type gather_edges(icontext_type& context,\n      const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  }\n\n  //for each edge gather the weights and the assigned clusters of the neighbors\n  neighbor_info gather(icontext_type& context, const vertex_type& vertex,\n      edge_type& edge) const {\n    if (edge.source().id() == vertex.id()) { //out edge\n      return neighbor_info(edge.target().data().best_cluster,\n          edge.data().weight);\n    } else { //in edge\n      return neighbor_info(edge.source().data().best_cluster,\n          edge.data().weight);\n    }\n  }\n\n  //assign a cluster, considering the clusters of neighbors\n  void apply(icontext_type& context, vertex_type& vertex,\n      const gather_type& total) {\n    size_t past_clst = vertex.data().best_cluster;\n    vertex.data().best_cluster = (size_t) (-1);\n    vertex.data().best_distance = std::numeric_limits::infinity();\n    for (size_t i = 0; i < NUM_CLUSTERS; ++i) {\n      if (CLUSTERS[i].center.size() > 0 || CLUSTERS[i].center_sparse.size() > 0) {\n        double d = 0.0;\n        if(IS_SPARSE == true)\n          d = sqr_distance(vertex.data().point_sparse, CLUSTERS[i].center_sparse);\n        else\n          d = sqr_distance(vertex.data().point, CLUSTERS[i].center);\n        //consider neighbors\n        const std::map& cw_map = total.cw_map;\n        for (std::map::const_iterator iter = cw_map.begin();\n            iter != cw_map.end(); iter++) {\n          size_t neighbor_cluster = iter->first;\n          double total_wieght = iter->second;\n          if (i == neighbor_cluster)\n            d -= total_wieght;\n        }\n        if (d < vertex.data().best_distance) {\n          vertex.data().best_distance = d;\n          vertex.data().best_cluster = i;\n        }\n      }\n    }\n    vertex.data().changed = (past_clst != vertex.data().best_cluster);\n  }\n\n  //send signals to the neighbors when the cluster assignment has changed\n  edge_dir_type scatter_edges(icontext_type& context,\n      const vertex_type& vertex) const {\n    if (vertex.data().changed)\n      return graphlab::ALL_EDGES;\n    else\n      return graphlab::NO_EDGES;\n  }\n\n  void scatter(icontext_type& context, const vertex_type& vertex,\n      edge_type& edge) const {\n  }\n};\n\n\n\n/*\n * computes new cluster centers\n * Also accumulates a counter counting the number of vertices which\n * assignments changed.\n */\nstruct cluster_center_reducer {\n  std::vector new_clusters;\n  size_t num_changed;\n  double cost;\n\n  cluster_center_reducer():new_clusters(NUM_CLUSTERS), num_changed(0), cost(0) { }\n\n  static cluster_center_reducer get_center(const graph_type::vertex_type& v) {\n    cluster_center_reducer cc;\n    ASSERT_NE(v.data().best_cluster, (size_t)(-1));\n\n    if(IS_SPARSE == true)\n      cc.new_clusters[v.data().best_cluster].center_sparse = v.data().point_sparse;\n    else\n      cc.new_clusters[v.data().best_cluster].center = v.data().point;\n    cc.new_clusters[v.data().best_cluster].count = 1;\n    cc.num_changed = v.data().changed;\n    cc.cost = v.data().best_distance;\n    return cc;\n  }\n\n  cluster_center_reducer& operator+=(const cluster_center_reducer& other) {\n    for (size_t i = 0;i < NUM_CLUSTERS; ++i) {\n      if (new_clusters[i].count == 0) new_clusters[i] = other.new_clusters[i];\n      else if (other.new_clusters[i].count > 0) {\n        if(IS_SPARSE == true)\n          plus_equal_vector(new_clusters[i].center_sparse, other.new_clusters[i].center_sparse);\n        else\n          plus_equal_vector(new_clusters[i].center, other.new_clusters[i].center);\n        new_clusters[i].count += other.new_clusters[i].count;\n      }\n    }\n    num_changed += other.num_changed;\n    cost += other.cost;\n    return *this;\n  }\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << new_clusters << num_changed <> new_clusters >> num_changed >> cost;\n  }\n};\n\nstruct vertex_writer {\n  std::string save_vertex(graph_type::vertex_type v) {\n    std::stringstream strm;\n    for (size_t i = 0;i < v.data().point.size(); ++i) {\n      strm << v.data().point[i] << \"\\t\";\n    }\n    strm << v.data().best_distance << \"\\t\";\n    strm << v.data().best_cluster << \"\\n\";\n    strm.flush();\n    return strm.str();\n  }\n\n  std::string save_edge(graph_type::edge_type e) { return \"\"; }\n};\n\nstruct vertex_writer_sparse {\n  std::string save_vertex(graph_type::vertex_type v) {\n    std::stringstream strm;\n    for(std::map::iterator iter = v.data().point_sparse.begin();\n        iter != v.data().point_sparse.end();++iter){\n      strm << (*iter).first << \":\" << (*iter).second << \" \";\n    }\n    strm << v.data().best_cluster << \"\\n\";\n    strm.flush();\n    return strm.str();\n  }\n\n  std::string save_edge(graph_type::edge_type e) { return \"\"; }\n};\n\nstruct vertex_writer_with_id {\n  std::string save_vertex(graph_type::vertex_type v) {\n    std::stringstream strm;\n    strm << v.id() << \"\\t\";\n    strm << v.data().best_cluster+1 << \"\\n\";\n    strm.flush();\n    return strm.str();\n  }\n\n  std::string save_edge(graph_type::edge_type e) { return \"\"; }\n};\n\n\nint main(int argc, char** argv) {\n  std::cout << \"Computes a K-means clustering of data.\\n\\n\";\n\n  graphlab::command_line_options clopts\n    (\"K-means clustering. The input data file is provided by the \"\n     \"--data argument which is non-optional. The format of the data file is a \"\n     \"collection of lines, where each line contains a comma or white-space \"\n     \"separated lost of numeric values representing a vector. Every line \"\n     \"must have the same number of values. The required --clusters=N \"\n     \"argument denotes the number of clusters to generate. To store the output \"\n     \"see the --output-cluster and --output-data arguments\");\n\n  std::string datafile;\n  std::string outcluster_file;\n  std::string outdata_file;\n  std::string edgedata_file;\n  size_t MAX_ITERATION = 0;\n  bool use_id = false;\n  clopts.attach_option(\"data\", datafile,\n                       \"Input file. Each line holds a white-space or comma separated numeric vector\");\n  clopts.attach_option(\"clusters\", NUM_CLUSTERS,\n                       \"The number of clusters to create.\");\n  clopts.attach_option(\"output-clusters\", outcluster_file,\n                       \"If set, will write a file containing cluster centers \"\n                       \"to this filename. This must be on the local filesystem \"\n                       \"and must be accessible to the root node.\");\n  clopts.attach_option(\"output-data\", outdata_file,\n                       \"If set, will output a copy of the input data with an additional \"\n                       \"two columns. The first added column is the distance to assigned \"\n\t\t       \"center and the last is the assigned cluster centers. The output \"\n                       \"will be written to a sequence of filenames where each file is \"\n                       \"prefixed by this value. This may be on HDFS.\");\n  clopts.attach_option(\"sparse\", IS_SPARSE,\n                       \"If set to true, will use a sparse vector representation.\"\n                       \"The file format is [feature id]:[value] [feature id]:[value] ...\"\n                       \", where [feature id] must be positive integer or zero.\");\n  clopts.attach_option(\"id\", use_id,\n                       \"If set to true, will use ids for data points. The id of a data point \"\n                       \"must be written at the head of each line of the input data. \"\n                       \"The output data will consist of two columns: the first one \"\n                       \"denotes the ids; the second one denotes the assigned clusters.\");\n  clopts.attach_option(\"pairwise-reward\", edgedata_file,\n                       \"If set, will consider pairwise rewards when clustering. \"\n                       \"Each line of the file beginning with the argument holds [id1] [id2] \"\n                       \"[reward]. This mode must be used with --id option.\");\n  clopts.attach_option(\"max-iteration\", MAX_ITERATION,\n                       \"The max number of iterations\");\n\n  if(!clopts.parse(argc, argv)) return EXIT_FAILURE;\n  if (datafile == \"\") {\n    std::cout << \"--data is not optional\\n\";\n    return EXIT_FAILURE;\n  }\n  if (NUM_CLUSTERS == 0) {\n    std::cout << \"--clusters is not optional\\n\";\n    return EXIT_FAILURE;\n  }\n  if(edgedata_file.size() > 0){\n    if(use_id == false){\n      std::cout << \"--id is not optional when you use edge data\\n\";\n      return EXIT_FAILURE;\n    }\n  }\n\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  // load graph\n  graph_type graph(dc, clopts);\n  NEXT_VID = (((graphlab::vertex_id_type)1 << 31) / dc.numprocs()) * dc.procid();\n  if(IS_SPARSE == true){\n    if(use_id){\n      graph.load(datafile, vertex_loader_with_id_sparse);\n    }else{\n      graph.load(datafile, vertex_loader_sparse);\n    }\n  }else{\n    if(use_id){\n      graph.load(datafile, vertex_loader_with_id);\n    }else{\n      graph.load(datafile, vertex_loader);\n    }\n  }\n  if(edgedata_file.size() > 0){\n    graph.load(edgedata_file, edge_loader);\n  }\n  graph.finalize();\n  dc.cout() << \"Number of datapoints: \" << graph.num_vertices() << std::endl;\n\n  if (graph.num_vertices() < NUM_CLUSTERS) {\n    dc.cout() << \"More clusters than datapoints! Cannot proceed\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  dc.cout() << \"Validating data...\";\n\n\n  CLUSTERS.resize(NUM_CLUSTERS);\n  // make sure all have the same array length\n  if(IS_SPARSE == false){\n    size_t max_p_size = graph.map_reduce_vertices\n                                  (max_point_size_reducer::get_max_point_size).max_point_size;\n    size_t min_p_size = graph.map_reduce_vertices\n                                  (min_point_size_reducer::get_min_point_size).min_point_size;\n    if (max_p_size != min_p_size) {\n      dc.cout() << \"Data has dimensionality ranging from \" << min_p_size << \" to \" << max_p_size\n                << \"! K-means cannot proceed!\" << std::endl;\n      return EXIT_FAILURE;\n    }\n    // allocate clusters\n    for (size_t i = 0;i < NUM_CLUSTERS; ++i) {\n      CLUSTERS[i].center.resize(max_p_size);\n    }\n  }\n\n  dc.cout() << \"Initializing using Kmeans++\\n\";\n  // ok. perform kmeans++ initialization\n  for (KMEANS_INITIALIZATION = 0;\n       KMEANS_INITIALIZATION < NUM_CLUSTERS;\n       ++KMEANS_INITIALIZATION) {\n\n    if(IS_SPARSE == true){\n      random_sample_reducer_sparse rs = graph.map_reduce_vertices\n                                        (random_sample_reducer_sparse::get_weight);\n      CLUSTERS[KMEANS_INITIALIZATION].center_sparse = rs.vtx;\n      graph.transform_vertices(kmeans_pp_initialization_sparse);\n    }else{\n      random_sample_reducer rs = graph.map_reduce_vertices\n                                        (random_sample_reducer::get_weight);\n      CLUSTERS[KMEANS_INITIALIZATION].center = rs.vtx;\n      graph.transform_vertices(kmeans_pp_initialization);\n    }\n  }\n\n  // \"reset\" all clusters\n  for (size_t i = 0; i < NUM_CLUSTERS; ++i) CLUSTERS[i].changed = true;\n  // perform Kmeans iteration\n\n  dc.cout() << \"Running Kmeans...\\n\";\n  bool clusters_changed = true;\n  size_t iteration_count = 0;\n  while(clusters_changed) {\n\t\tif(MAX_ITERATION > 0 && iteration_count >= MAX_ITERATION)\n\t\t\tbreak;\n\n    cluster_center_reducer cc = graph.map_reduce_vertices\n                                    (cluster_center_reducer::get_center);\n    // the first round (iteration_count == 0) is not so meaningful\n    // since I am just recomputing the centers from the output of the KMeans++\n    // initialization\n    if (iteration_count > 0) {\n      dc.cout() << \"Kmeans iteration \" << iteration_count << \": \" <<\n                 \"# points with changed assignments = \" << cc.num_changed << \n\t\t \" total cost: \" << cc.cost << std::endl;\n    }\n    for (size_t i = 0;i < NUM_CLUSTERS; ++i) {\n      double d = cc.new_clusters[i].count;\n      if(IS_SPARSE){\n        if (d > 0) scale_vector(cc.new_clusters[i].center_sparse, 1.0 / d);\n        if (cc.new_clusters[i].count == 0 && CLUSTERS[i].count > 0) {\n          dc.cout() << \"Cluster \" << i << \" lost\" << std::endl;\n          CLUSTERS[i].center_sparse.clear();\n          CLUSTERS[i].count = 0;\n          CLUSTERS[i].changed = false;\n        }\n        else {\n          CLUSTERS[i] = cc.new_clusters[i];\n          CLUSTERS[i].changed = true;\n        }\n      }else{\n        if (d > 0) scale_vector(cc.new_clusters[i].center, 1.0 / d);\n        if (cc.new_clusters[i].count == 0 && CLUSTERS[i].count > 0) {\n          dc.cout() << \"Cluster \" << i << \" lost\" << std::endl;\n          CLUSTERS[i].center.clear();\n          CLUSTERS[i].count = 0;\n          CLUSTERS[i].changed = false;\n        }\n        else {\n          CLUSTERS[i] = cc.new_clusters[i];\n          CLUSTERS[i].changed = true;\n        }\n      }\n    }\n    clusters_changed = iteration_count == 0 || cc.num_changed > 0;\n\n    if(edgedata_file.size() > 0){\n      clopts.engine_args.set_option(\"factorized\", true);\n      graphlab::omni_engine engine(dc, graph, \"async\", clopts);\n      engine.signal_all();\n      engine.start();\n    }else{\n      graph.transform_vertices(kmeans_iteration);\n    }\n\n    ++iteration_count;\n  }\n\n\n  if (!outcluster_file.empty() && dc.procid() == 0) {\n    dc.cout() << \"Writing Cluster Centers...\" << std::endl;\n    std::ofstream fout(outcluster_file.c_str());\n    if(IS_SPARSE){\n      for (size_t i = 0;i < NUM_CLUSTERS; ++i) {\n        if(use_id)\n          fout << i+1 << \"\\t\";\n        for (std::map::iterator iter = CLUSTERS[i].center_sparse.begin();\n             iter != CLUSTERS[i].center_sparse.end();++iter) {\n          fout << (*iter).first << \":\" << (*iter).second << \" \";\n        }\n        fout << \"\\n\";\n      }\n    }else{\n      for (size_t i = 0;i < NUM_CLUSTERS; ++i) {\n        if(use_id)\n          fout << i+1 << \"\\t\";\n        for (size_t j = 0; j < CLUSTERS[i].center.size(); ++j) {\n          fout << CLUSTERS[i].center[j] << \" \";\n        }\n        fout << \"\\n\";\n      }\n    }\n  }\n\n  if (!outdata_file.empty()) {\n    dc.cout() << \"Writing Data with cluster assignments...\\n\" << std::endl;\n    if(use_id){\n      graph.save(outdata_file, vertex_writer_with_id(), false, true, false, 1);\n    }else{\n      if(IS_SPARSE == true)\n        graph.save(outdata_file, vertex_writer_sparse(), false, true, false, 1);\n      else\n        graph.save(outdata_file, vertex_writer(), false, true, false, 1);\n    }\n  }\n\n  graphlab::mpi_tools::finalize();\n}\n\n\n"
  },
  {
    "path": "toolkits/clustering/spectral_clustering.cpp",
    "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\n#include \n#include \n\n//remove assigned options from arguments\nstd::string get_arg_str_without(int argc, char** argv,\n    std::vector remove_opts) {\n  std::stringstream strm;\n  bool skip_next = false;\n  for (int i = 1; i < argc; ++i) {\n    bool skip = false;\n    for (size_t j = 0; j < remove_opts.size(); ++j) {\n      std::string with_equal = remove_opts[j] + \"=\";\n      if (strncmp(with_equal.c_str(), argv[i], with_equal.size()) == 0) {\n        skip = true;\n      } else if (strncmp(remove_opts[j].c_str(), argv[i], remove_opts[j].size())\n          == 0) {\n        skip = true;\n        skip_next = true;\n      }\n    }\n    if (skip == false && skip_next == false) {\n      strm << argv[i] << \" \";\n    } else if (skip == false && skip_next == true) {\n      skip_next = false;\n    }\n  }\n  return strm.str();\n}\n\nbool call_graph_laplacian_construction(const std::string& mpi_args,\n    const std::string& filename, const float sigma, const float epsilon,\n    const size_t num_nearests, const std::string& args) {\n  std::stringstream strm;\n  if (mpi_args.length() > 0)\n    strm << \"mpiexec \" << mpi_args << \" \";\n  strm << \"./graph_laplacian_for_sc \";\n  strm << \" --data=\" << filename;\n  strm << \" --sigma=\" << sigma;\n  strm << \" --similarity-thres=\" << epsilon;\n  strm << \" --t-nearest=\" << num_nearests;\n  strm << \" \" << args;\n  std::cout << \"CALLING >\" << strm.str() << std::endl;\n  int sys_ret = system(strm.str().c_str());\n  if (sys_ret != 0) {\n    std::cout << \"system call fails\" << std::endl;\n    return false;\n  }\n\n  return true;\n}\n\nvoid make_initial_vector_file(const std::string& filename, const size_t num_data){\n  std::ofstream ofs((filename + \".init\").c_str());\n  for(size_t i=0;i 0)\n    strm << \"mpiexec \" << mpi_args << \" \";\n  strm << svd_dir << \"svd \" + filename + \".glap\";\n  strm << \" --rows=\" << num_data+1;\n  strm << \" --cols=\" << num_data;\n  strm << \" --nsv=\" << num_clusters;\n  strm << \" --nv=\" << rank;\n//  strm << \" --tol=1e-10\";\n//  strm << \" --max_iter=20\";\n  strm << \" --quiet=1\";\n  strm << \" --input_file_offset=1\";\n  strm << \" --save_vectors=1\";\n  strm << \" --ortho_repeats=3\";\n//  strm << \" --id=1\";\n  strm << \" --prediction=\" << filename;\n  strm << \" --initial_vector=\" << filename + \".init\";\n  strm << \" \" << args;\n  std::cout << \"CALLING >\" << strm.str() << std::endl;\n  int sys_ret = system(strm.str().c_str());\n  if (sys_ret != 0) {\n    std::cout << \"system call fails\" << std::endl;\n    return false;\n  }\n\n  return true;\n}\n\nbool call_eigen_vector_normalization(const std::string& mpi_args,\n    const std::string& filename, const std::string& graph_analytics_dir,\n    const size_t num_clusters, const size_t rank, const size_t num_data,\n    const std::string& args) {\n  std::stringstream strm;\n  if (mpi_args.length() > 0)\n    strm << \"mpiexec \" << mpi_args << \" \";\n  strm << graph_analytics_dir << \"eigen_vector_normalization\";\n  strm << \" --data=\" << filename;\n  strm << \" --clusters=\" << num_clusters;\n  strm << \" --rank=\" << rank;\n  strm << \" --data-num=\" << num_data;\n  strm << \" \" << args;\n  std::cout << \"CALLING >\" << strm.str() << std::endl;\n  int sys_ret = system(strm.str().c_str());\n  if (sys_ret != 0) {\n    std::cout << \"system call fails\" << std::endl;\n    return false;\n  }\n\n  return true;\n}\n\nbool call_kmeans(const std::string& mpi_args, const std::string& filename,\n    const std::string& kmeans_dir, const size_t num_clusters,\n    const std::string& args) {\n  //call svd\n  std::stringstream strm;\n  if (mpi_args.length() > 0)\n    strm << \"mpiexec \" << mpi_args << \" \";\n  strm << kmeans_dir << \"kmeans \";\n  strm << \" --data \" << filename << \".compressed\";\n  strm << \" --clusters \" << num_clusters;\n  strm << \" --output-data \" << filename << \".result\";\n  strm << \" --id=1\";\n  strm << \" \" << args;\n  std::cout << \"CALLING >\" << strm.str() << std::endl;\n  int sys_ret = system(strm.str().c_str());\n  if (sys_ret != 0) {\n    std::cout << \"system call fails\" << std::endl;\n    return false;\n  }\n  return true;\n}\n\n\nbool call_kmeans_as_preprocess(const std::string& mpi_args, const std::string& filename,\n    const std::string& kmeans_dir, const size_t num_clusters,\n    const std::string& args) {\n  //call svd\n  std::stringstream strm;\n  if (mpi_args.length() > 0)\n    strm << \"mpiexec \" << mpi_args << \" \";\n  strm << kmeans_dir << \"kmeans \";\n  strm << \" --data \" << filename;\n  strm << \" --clusters \" << num_clusters;\n  strm << \" --output-data \" << filename << \".pre.labels\";\n  strm << \" --output-clusters \" << filename << \".pre.centers\";\n  strm << \" --id=1\";\n  strm << \" \" << args;\n  std::cout << \"CALLING >\" << strm.str() << std::endl;\n  int sys_ret = system(strm.str().c_str());\n  if (sys_ret != 0) {\n    std::cout << \"system call fails\" << std::endl;\n    return false;\n  }\n  return true;\n}\n\n\n//select good value of rank (TODO)\nint get_lanczos_rank(const size_t num_clusters, const size_t num_data) {\n  size_t rank = 1;\n  if (num_data < 1000) {\n    if (num_clusters + 10 <= num_data)\n      rank = num_clusters + 10;\n    else\n      rank = num_data;\n  } else if (num_data < 10000) {\n    rank = num_clusters + 100;\n  } else if (num_data < 100000) {\n    rank = num_clusters + 150;\n  } else if (num_data < 1000000) {\n    rank = num_clusters + 200;\n  } else {\n    rank = num_clusters + 300;\n  }\n  return rank;\n//  return num_clusters+2;\n}\n\nvoid read_pairs_with_prefix(std::vector >& ret, const std::string& prefix){\n  std::string directory_name;\n  std::string original_path(prefix);\n  boost::filesystem::path path(prefix);\n  std::string search_prefix;\n  if (boost::filesystem::is_directory(path)) {\n    // if this is a directory\n    // force a \"/\" at the end of the path\n    // make sure to check that the path is non-empty. (you do not\n    // want to make the empty path \"\" the root path \"/\" )\n    directory_name = path.native();\n  }\n  else {\n    directory_name = path.parent_path().native();\n    search_prefix = path.filename().native();\n    directory_name = (directory_name.empty() ? \".\" : directory_name);\n  }\n  std::vector files;\n  graphlab::fs_util::list_files_with_prefix(directory_name, search_prefix, files);\n  if (files.size() == 0) {\n    logstream(LOG_WARNING) << \"No files found matching \" << original_path << std::endl;\n  }\n  for(size_t i = 0; i < files.size(); ++i) {\n    std::ifstream ifs(files[i].c_str());\n    if (!ifs) {\n      std::cout << \"can't read \" << files[i] << std::endl;\n      return;\n    }\n    while( !ifs.eof() ) {\n      std::vector pair;\n      size_t id = 0;\n      size_t label = 0;\n      ifs >> id;\n//      ifs.ignore(1);\n      ifs >> label;\n      if(id > 0 && label > 0){\n        pair.push_back(id);\n        pair.push_back(label);\n        ret.push_back(pair);\n      }\n    }\n  }\n}\n\nint recover_labels(const std::string& prefix){\n  const std::string kmeans_result_prefix = prefix + \".pre.labels\";\n  const std::string spectral_result_prefix = prefix + \".pre.centers.result\";\n  const std::string outfile = prefix + \".result_1_of_1\";\n\n  std::vector > kmeans_result;\n  read_pairs_with_prefix(kmeans_result, kmeans_result_prefix);\n  std::vector > spectral_result;\n  read_pairs_with_prefix(spectral_result, spectral_result_prefix);\n\n  std::map label_map;\n  for(size_t i=0;i > times;\n  time(&start);\n\n  std::string datafile;\n  std::string graph_analytics_dir = \"../graph_analytics/\";\n  std::string svd_dir = \"../collaborative_filtering/\";\n  std::string kmeans_dir = \"./\";\n  std::string mpi_args;\n  size_t num_clusters = 0;\n  size_t num_nearests = 30;\n  float sigma = 1.0;\n  float epsilon = 0.0;\n  size_t pre_kmeans_clusters = 0;\n  size_t sv = 0;\n  //parse command line\n  graphlab::command_line_options clopts(\n          \"Spectral clustering. The input data file is provided by the \"\n          \"--data argument which is non-optional. The format of the data file is a \"\n          \"collection of lines, where each line contains a data id followed by a \"\n          \"comma or white-space separated list of numeric values representing a vector. \"\n          \"Every line must have the same number of values. The required --clusters=N \"\n          \"argument denotes the number of clusters to generate.\");\n  clopts.attach_option(\"data\", datafile,\n          \"Input file. Each line holds a data id followed by a white-space \"\n          \"or comma separated numeric vector\");\n  clopts.attach_option(\"clusters\", num_clusters,\n          \"The number of clusters to create\");\n  clopts.attach_option(\"sigma\", sigma,\n          \"Scale parameter for Gaussian kernel\");\n  clopts.attach_option(\"t-nearest\", num_nearests,\n          \"Number of nearest neighbors (=t). Will use only the t-nearest similarities \"\n          \"for each datapoint. If set at 0, will use all similarities.\");\n  clopts.attach_option(\"similarity-thres\", epsilon,\n          \"Threshold to discard small similarities\");\n  clopts.attach_option(\"svd-dir\", svd_dir,\n          \"Path to the directory where Graphlab svd is located\");\n  clopts.attach_option(\"kmeans-dir\", kmeans_dir,\n          \"Path to the directory where Graphlab kmeans is located\");\n  clopts.attach_option(\"graph-analytics-dir\", graph_analytics_dir,\n          \"Path to the directory where Graphlab eigen_vector_normalization is located\");\n  clopts.attach_option(\"pre-kmeans-clusters\", pre_kmeans_clusters,\n          \"If set, will perform kmeans as a preprocess with the given cluster number.\");\n  clopts.attach_option(\"mpi-args\", mpi_args,\n          \"If set, will execute mipexec with the given arguments. \"\n          \"For example, --mpi-args=\\\"-n [N machines] --hostfile [host file]\\\"\");\n  clopts.attach_option(\"sv\", sv,\n          \"Number of vectors in each iteration in the Lanczos svd.\");\n  if (!clopts.parse(argc, argv))\n    return EXIT_FAILURE;\n  if (datafile == \"\") {\n    std::cout << \"--data is not optional\\n\";\n    return EXIT_FAILURE;\n  }\n  if (num_clusters == 0) {\n    std::cout << \"--cluster is not optional\\n\";\n    return EXIT_FAILURE;\n  }\n\n  std::vector remove_opts;\n  remove_opts.push_back(\"--data\");\n  remove_opts.push_back(\"--svd-dir\");\n  remove_opts.push_back(\"--graph-analytics-dir\");\n  remove_opts.push_back(\"--kmeans-dir\");\n  remove_opts.push_back(\"--clusters\");\n  remove_opts.push_back(\"--sigma\");\n  remove_opts.push_back(\"--similarity-thres\");\n  remove_opts.push_back(\"--mpi-args\");\n  remove_opts.push_back(\"--t-nearest\");\n  remove_opts.push_back(\"--pre-kmeans-clusters\");\n  remove_opts.push_back(\"--sv\");\n  std::string other_args = get_arg_str_without(argc, argv, remove_opts);\n\n  //preprocess by kmeans for fast clustering\n  if(pre_kmeans_clusters > 0){\n    if(pre_kmeans_clusters < num_clusters){\n      std::cout << \"the number of --pre-kmeans-clusters must be bigger than the number of clusters\\n\";\n      return EXIT_FAILURE;\n    }\n    time(&mid);\n    if(call_kmeans_as_preprocess(mpi_args, datafile, kmeans_dir, pre_kmeans_clusters, other_args) == false)\n      return EXIT_FAILURE;\n    //modify settings\n    datafile = datafile + \".pre.centers\";\n    num_nearests = 0;\n    time(&end);\n    times.push_back(std::pair(\"kmeans preprocess\",(end - mid)));\n  }\n\n  //construct graph laplacian\n  time(&mid);\n  if (call_graph_laplacian_construction(mpi_args, datafile, sigma, epsilon,\n      num_nearests, other_args) == false) {\n    return EXIT_FAILURE;\n  }\n  time(&end);\n  times.push_back(std::pair(\"graph laplacian\",(end - mid)));\n\n  //eigen value decomposition\n  //read number of data\n  size_t num_data = 0;\n  const std::string datanum_filename = datafile + \".datanum\";\n  std::ifstream ifs(datanum_filename.c_str());\n  if (!ifs) {\n    std::cout << \"can't read number of data.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n  ifs >> num_data;\n  //determine the sv of Lanczos method\n  if(sv == 0){\n    sv = get_lanczos_rank(num_clusters, num_data);\n  }else{\n    if(sv < num_clusters)\n      sv = num_clusters;\n  }\n  time(&mid);\n  if (call_svd(mpi_args, datafile, svd_dir, num_clusters, sv, num_data,\n      other_args) == false) {\n    return EXIT_FAILURE;\n  }\n  if (call_eigen_vector_normalization(mpi_args, datafile, graph_analytics_dir,\n      num_clusters, sv, num_data, other_args) == false) {\n    return EXIT_FAILURE;\n  }\n  time(&end);\n  times.push_back(std::pair(\"eigen decomposition\",(end - mid)));\n\n  //run kmeans\n  time(&mid);\n  if (call_kmeans(mpi_args, datafile, kmeans_dir, num_clusters, other_args)\n      == false) {\n    return EXIT_FAILURE;\n  }\n  time(&end);\n  times.push_back(std::pair(\"kmeans\",(end - mid)));\n\n  //recover cluster membership if preprocess with kmeans was done\n  if(pre_kmeans_clusters > 0){\n    //remove \".pre.centers\"\n    datafile = datafile.substr(0, datafile.size() - 12);\n    recover_labels(datafile);\n  }\n\n  time(&end);\n\n  std::cout << \"computation times:\\n\";\n  for(size_t i=0;i\n#include \n#include \n#include \"stats.hpp\"\n#include \"cdf.hpp\"\n\n//when using negative node id range, we are not allowed to use\n//0 and 1 so we add 2.\nconst static int SAFE_NEG_OFFSET=2;\nconst double pi = 3.14159265;\nconst double gaussian_normalization = 1/sqrt(2 * pi);\ndouble beta = 1;\nbool debug = false;\n\n\nenum data_role_type { TRAIN = 0, VALIDATE = 1, PREDICT =2 };\n\ndata_role_type mode;\n/** \n */\nstruct vertex_data : graphlab::IS_POD_TYPE{\n\tint y;\n\tfloat xT_mu; \n\tfloat sigma;\n\tfloat predict;\n\tfloat err;\n\tfloat likelihood;\n\tfloat weights;\n\tdata_role_type type;\n\n\tvertex_data() {\n\t\txT_mu = 0;\n\t\ty = 0;\n\t\tsigma  = 1;\n\t\tpredict = 0;\n\t\terr = 0;\n\t\tlikelihood = 0;\n\t\tweights = 0;\n\t\ttype = TRAIN;\n\t}\n\n\n}; // end of vertex data\n\n\n/**\n * \\brief The edge data stores the entry in the matrix.\n *\n * In addition the edge data adpredictoro stores the most recent error estimate.\n */\nstruct edge_data : public graphlab::IS_POD_TYPE {\n\t/**\n\t * \\brief The type of data on the edge;\n\t *\n\t * \\li *Train:* the observed value is correct and used in training\n\t * \\li *Validate:* the observed value is correct but not used in training\n\t * \\li *Predict:* The observed value is not correct and should not be\n\t *        used in training.\n\t */\n\n\t/** \\brief the observed value for the edge */\n\tfloat x_ij;\n\n\t/** \\brief The train/validation/test designation of the edge */\n\tdata_role_type role;\n\n\t/** \\brief basic initialization */\n\tedge_data(float x_ij = 1, data_role_type role = TRAIN) :\n\t\tx_ij(x_ij), role(role) { }\n\n}; // end of edge data\n\n\n\n/**\n * \\brief The graph type is defined in terms of the vertex and edge\n * data.\n */ \ntypedef graphlab::distributed_graph graph_type;\n\n\n\n\n\n\n/* compute v(t) according to equation (9) left */\ndouble v(double t){\n        double phit = phi(t);\n        if (phit == 0)\n            phit = 1e-5;\n\tdouble ret = gaussian_normalization * exp(-t*t/2) / phit;\n        if (std::isinf(ret)){\n           std::cout<<\"BUG: \" << ret << \" \" << t << \" \" << exp(-t*t/2) << \" phi(t)\" << phi(t) << std::endl;\n           assert(false);\n        }\n        return ret;\n}\n\n/* compute w(t) according to equation (9) right */\ndouble w(double t){\n\tdouble vt = v(t);\n\treturn vt * (vt+t);\n}\n\n\nstruct gather_type: public graphlab::IS_POD_TYPE{\n\tfloat sigma;\n\tfloat mu;\n\tfloat mult_sigma;\n\n\tgather_type(){ sigma = 0; mu = 0; mult_sigma = 1; }\n\tgather_type& operator+=(const gather_type& other) {\n\t\tsigma += other.sigma;\n\t\tmu += other.mu;\n\t\tmult_sigma *= other.mult_sigma;\n\t\treturn *this;\n\t}\n};\n\n/** compute probability for click as given in equation (2) */\nfloat ctr_predict( const vertex_data& data, \n\t\tconst float rating, \n\t\tdouble & prediction, \n\t\tvoid * extra = NULL){\n\n\tassert(beta > 0);\n\tprediction = data.xT_mu;\n\tdouble prob = phi(data.xT_mu * data.y / beta);\n\tif (debug)\n\t\t//std::cout<<\"prediction: \" << prediction << \" y: \" << data.y << std::endl;\n\t\tprintf(\"prediction %12.8lf y: %d \\n\", prediction, data.y);\n\treturn prob; \n}\n\n\n\ngather_type adpredictor_map(graph_type::edge_type edge, graph_type::vertex_type other) {\n\tgather_type ret;\n\tassert(edge.data().x_ij  == 1);\n\t/* compute equation (6) */\n\tret.sigma = edge.data().x_ij * other.data().sigma;\n\tret.mu = edge.data().x_ij * other.data().xT_mu;\t\t\t\n\treturn ret;\n}\n\n// the function arguments of the combiner must match the return type of the\n// map function.\nvoid adpredictor_combine2(gather_type &a, const gather_type & b, const vertex_data unused) {\n\ta.mu += b.mu;\n\ta.mult_sigma *= b.mult_sigma;\n}\n\n\n\ngather_type adpredictor_map2(graph_type::edge_type edge, graph_type::vertex_type other, vertex_data  vertex){\n\tgather_type ret;\n\tassert(vertex.sigma > 0);\n\tassert(other.data().y == -1 || other.data().y == 1);\n        assert(edge.data().x_ij == 1);\n\tdouble product = other.data().y * other.data().xT_mu / sqrt(other.data().sigma);\n\t//assert(product > 0);\n\tret.mu = (other.data().y * edge.data().x_ij * vertex.sigma / sqrt(other.data().sigma))  * v(product);\n        if (std::isinf(ret.mu)){\n          std::cout<<\"BUG: \" << ret.mu << \" vertex.sigma \" << vertex.sigma << \" other.data().sigma \" << other.data().sigma << \" v(prod) \" << v(product);\n          assert(false);\n        }\n\tdouble factor = 1.0 - (edge.data().x_ij * vertex.sigma / other.data().sigma)*w(product);\n        if (factor <= 0){\n          std::cout<<\"BUG: \" << product << \" \" << ret.mu << \" \" << factor < 0);\n        }\n\tret.sigma = factor;\n\treturn ret;\n}\n\nvoid adpredictor_update(graph_type::vertex_type vertex) {\n\t//go over all row nodes\n\tif ( vertex.num_out_edges() > 0){\n                if (debug) printf(\"Entered vertex %lu role %d \\n\", vertex.id(), vertex.data().type);\n\t\tif (vertex.data().type == TRAIN){\n\t\t\tvertex_data & row = vertex.data(); \n\t\t\trow.likelihood = 0;\n\t\t\trow.err = 0;\n\t\t\tassert(row.y == -1 || row.y == 1);\n\t\t\tassert(beta > 0);\n\n\t\t\tif (debug)\n\t\t\t\tstd::cout<<\"Entered item \" << vertex.id() << \" y: \" << row.y << std::endl;\n\t\t\trow.sigma = beta*beta;\n\t\t\trow.xT_mu = 0;\n\n\t\t\tgather_type sum = graphlab::warp::map_reduce_neighborhood(vertex, graphlab::OUT_EDGES, adpredictor_map);\n\t\t\trow.sigma = sum.sigma;\n\t\t\trow.xT_mu = sum.mu;\n\n\t\t\tdouble prediction;\n\t\t\tdouble ret = ctr_predict(row, row.y, prediction);\n\t\t\tdouble predicted_target = prediction < 0 ? -1: 1;\n\t\t\tif ((predicted_target == -1  && row.y == 1) || (predicted_target == 1 && row.y == -1))\n\t\t\t\trow.err += 1.0;  \n\t\t\tif (debug)\n\t\t\t\tstd::cout<<\"Prediction was: \" << prediction << \" real value: \" << row.y << std::endl;\n\t\t\trow.likelihood += ret;\n\n\t\t\tassert(row.sigma > 0);\n\t\t}\n\t\telse if (vertex.data().type == VALIDATE || vertex.data().type == PREDICT){\n\t\t\tvertex_data & row = vertex.data(); \n\t\t\trow.likelihood = 0;\n\t\t\trow.err = 0;\n\t\t\tif (vertex.data().type == VALIDATE) \n     \t\t\t \tassert(row.y == -1 || row.y == 1);\n\t\t\tgather_type sum = graphlab::warp::map_reduce_neighborhood(vertex, graphlab::OUT_EDGES, adpredictor_map);\n\t\t\trow.predict = sum.mu;\n\t\t\tdouble predict = sum.mu > 0 ? 1 : -1;                       \n\t\t\tif (predict != row.y && vertex.data().type == VALIDATE)\n\t\t\t\trow.err++;\n\t\t}\n                else assert(false);\n\n\t}\n}\n\nvoid adpredictor_update2(graph_type::vertex_type vertex) {\n\tif (vertex.num_in_edges() > 0){\n\t\tgather_type sum = graphlab::warp::map_reduce_neighborhood(vertex, graphlab::IN_EDGES, vertex.data(), adpredictor_map2,adpredictor_combine2);\n\t\tvertex.data().sigma *= sum.mult_sigma;\n\t\tvertex.data().xT_mu += sum.mu;\n\t}\n}\ngather_type count_vertices(const graph_type::vertex_type& vertex) {\n\tgather_type ret;\n\tif (vertex.data().type == TRAIN){\n\t\tret.mu = 1;\n\t}\n\telse if (vertex.data().type == VALIDATE){\n\t\tret.sigma = 1;\n\t}\n\treturn ret;\n}\n\nstruct model_saver {\n\ttypedef graph_type::vertex_type vertex_type;\n\ttypedef graph_type::edge_type   edge_type;\n\t/* save the linear model, using the format:\n\t */\n\tstd::string save_vertex(const vertex_type& vertex) const {\n                if (vertex.num_in_edges() == 0 || vertex.data().type != TRAIN)\n\t\t  return \"\";\n\n\t\tstd::stringstream strm;\n\t\tstrm << vertex.id() << \" \" << vertex.data().xT_mu << \" \" << std::endl;\n\t\treturn strm.str();\n\t}\n\tstd::string save_edge(const edge_type& edge) const {\n\t\treturn \"\";\n\t}\n}; // end of prediction_saver\n\nstruct prediction_saver {\n\ttypedef graph_type::vertex_type vertex_type;\n\ttypedef graph_type::edge_type   edge_type;\n\t/* save the linear model, using the format:\n\t */\n\tstd::string save_vertex(const vertex_type& vertex) const {\n                if (vertex.num_out_edges() == 0 || vertex.data().type != PREDICT)\n\t\t  return \"\";\n\n\t\tstd::stringstream strm;\n\t\tstrm << vertex.id() << \" \" << vertex.data().predict << \" \" << std::endl;\n\t\treturn strm.str();\n\t}\n\tstd::string save_edge(const edge_type& edge) const {\n\t\treturn \"\";\n\t}\n}; // end of prediction_saver\n\n\n\n/**\n * \\brief The graph loader function is a line parser used for\n * distributed graph construction.\n */\ninline bool graph_loader(graph_type& graph, \n\t\tconst std::string& filename,\n\t\tconst std::string& line) {\n\tASSERT_FALSE(line.empty()); \n\n\t// Parse the line\n\tstd::stringstream strm(line);\n\tfloat weight = 0;\n\tfloat label = 0;\n\n\tstrm >> label;\n\tif (label != -1 && label != 1)\n\t\tlogstream(LOG_FATAL)<<\"Each line must have label -1 or 1 as the first item in the row. Row was : \" << line << \" label: \" << label << std::endl;\n\n\t// Determine the role of the data\n\tdata_role_type role = TRAIN;\n\tif(boost::ends_with(filename,\".validate\")) role = VALIDATE;\n\telse if(boost::ends_with(filename, \".predict\")) role = PREDICT;\n\n\tint myid = rand();\n\tint num_vals = 0;\n\twhile (strm.good()) {\n\t\tgraphlab::vertex_id_type target;\n\t\tstrm >> target;\n\t\tif (strm.fail()) break;\n\t\tchar col;\n\t\tstrm >> col;\n\t\tif (strm.fail()) break;\n\t\tstrm >> weight;\n\t\tif (strm.fail()) break;\n\t\tif (weight != 1)\n\t\t\tlogstream(LOG_FATAL)<<\"Currently we support only binary edges. Line was: \" << line << \" in file: \" << filename << std::endl;\n\t\tnum_vals++;\n\t\ttarget = -(graphlab::vertex_id_type(target + SAFE_NEG_OFFSET));\n\t\tgraph.add_edge(myid, target, edge_data(weight, role));\n\t}  \n\n\tif (num_vals == 0)\n\t\tlogstream(LOG_FATAL)<<\"Failed to load line: \" << line << \" in file: \" << filename << std::endl;\n\n\tvertex_data data;\n\tdata.y = label;\n\tdata.type = role;\n        if (debug) printf(\"Adding vertex %u with role %d\\n\", myid, role);\n\tgraph.add_vertex(myid, data);\n\treturn true; // successful load\n} // end of graph_loader\n\n\n\n\n\n\nint MAX_ITER = 5;\n\ngather_type calc_error(const graph_type::vertex_type& vertex) {\n\tgather_type ret;\n\tif (mode == vertex.data().type){\n\t\tret.mu = vertex.data().err;\n\t\tret.sigma = vertex.data().likelihood;\n\t}\n\treturn ret;\n}\n\n\nint main(int argc, char** argv) {\n\tglobal_logger().set_log_level(LOG_INFO);\n\tglobal_logger().set_log_to_console(true);\n\n\t// Parse command line options -----------------------------------------------\n\tconst std::string description = \n\t\t\"adPredictor algorithm\";\n\tgraphlab::command_line_options clopts(description);\n\tstd::string input_dir;\n\tstd::string save_model;\n\tstd::string exec_type = \"synchronous\";\n\tclopts.attach_option(\"matrix\", input_dir,\n\t\t\t\"The directory containing the matrix file\");\n\tclopts.add_positional(\"matrix\");\n\tclopts.attach_option(\"max_iter\", MAX_ITER,\n\t\t\t\"The maxumum number of udpates allowed for a vertex\");\n\tclopts.attach_option(\"debug\", debug, \n\t\t\t\"debug - additional verbose info\"); \n\tclopts.attach_option(\"save_model\", save_model,\n\t\t\t\"The prefix (folder and filename) to save predictions.\");\n\tclopts.attach_option(\"beta\", beta, \"gaussian bandwidth\");\n\n\tif(!clopts.parse(argc, argv) || input_dir == \"\") {\n\t\tstd::cout << \"Error in parsing command line arguments.\" << std::endl;\n\t\tclopts.print_description();\n\t\treturn EXIT_FAILURE;\n\t}\n\n\tgraphlab::mpi_tools::init(argc, argv);\n\tgraphlab::distributed_control dc;\n\n\tdc.cout() << \"Loading graph.\" << std::endl;\n\tgraphlab::timer timer; \n\tgraph_type graph(dc, clopts);  \n\tgraph.load(input_dir, graph_loader); \n\tdc.cout() << \"Loading graph. Finished in \" \n\t\t<< timer.current_time() << std::endl;\n\tdc.cout() << \"Finalizing graph.\" << std::endl;\n\ttimer.start();\n\tgraph.finalize();\n\tdc.cout() << \"Finalizing graph. Finished in \" \n\t\t<< timer.current_time() << std::endl;\n\n\n\tdc.cout() \n\t\t<< \"========== Graph statistics on proc \" << dc.procid() \n\t\t<< \" ===============\"\n\t\t<< \"\\n Num vertices: \" << graph.num_vertices()\n\t\t<< \"\\n Num edges: \" << graph.num_edges()\n\t\t<< \"\\n Num replica: \" << graph.num_replicas()\n\t\t<< \"\\n Replica to vertex ratio: \" \n\t\t<< float(graph.num_replicas())/graph.num_vertices()\n\t\t<< \"\\n --------------------------------------------\" \n\t\t<< \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n\t\t<< \"\\n Num local vertices: \" << graph.num_local_vertices()\n\t\t<< \"\\n Replica to own ratio: \" \n\t\t<< (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n\t\t<< \"\\n Num local edges: \" << graph.num_local_edges()\n\t\t//<< \"\\n Begin edge id: \" << graph.global_eid(0)\n\t\t<< \"\\n Edge balance ratio: \" \n\t\t<< float(graph.num_local_edges())/graph.num_edges()\n\t\t<< std::endl;\n\n\tdc.cout() << \"Running adPredictor\" << std::endl;\n\tdc.cout() << \"(C) Code by Danny Bickson, GraphLab Inc. \" << std::endl;\n\tdc.cout() << \"Please send bug reports to danny.bickson@gmail.com\" << std::endl;\n\ttimer.start();\n\n\tgather_type edge_count = graph.map_reduce_vertices(count_vertices);\n\tdc.cout()<<\"Training rows: \" << edge_count.mu << \" validation rows: \" << edge_count.sigma << std::endl;\n        if (edge_count.mu <= 0)\n          logstream(LOG_FATAL)<< \"Failed to read training data. Aborting\" << std::endl;\n\n\tgraphlab::timer mytimer; mytimer.start();\n\n\tfor (int i = 0; i < MAX_ITER; ++i) {\n\t\tgraphlab::warp::parfor_all_vertices(graph, adpredictor_update); \n\t\tgraphlab::warp::parfor_all_vertices(graph, adpredictor_update2); \n\t\tmode = TRAIN;\n\t\tgather_type ret = graph.map_reduce_vertices(calc_error);\n\t\tdc.cout() << i << \") Log likelihood: \" << std::setw(10) << ret.sigma << \" Avg error: \" << std::setw(10) << ret.mu/edge_count.mu << std::endl; \n\t\tmode = VALIDATE;\n\t\tret = graph.map_reduce_vertices(calc_error);\n\t\tdc.cout() << i << \" Avg validation error: \" << std::setw(10) << ret.mu/edge_count.sigma << std::endl; \n\n\t}\n\n\tconst double runtime = timer.current_time();\n\tdc.cout() << \"----------------------------------------------------------\"\n\t\t<< std::endl\n\t\t<< \"Final Runtime (seconds):   \" << runtime << std::endl;\n\n\n\t// Make predictions ---------------------------------------------------------\n\tif(!save_model.empty()) {\n\t\tstd::cout << \"Saving predictions\" << std::endl;\n\t\tconst bool gzip_output = false;\n\t\tconst bool save_vertices = true;\n\t\tconst bool save_edges = false;\n\t\tconst size_t threads_per_machine = 1;\n\t\t//save the predictions\n\t\tgraph.save(save_model, model_saver(), gzip_output, save_vertices, save_edges, threads_per_machine);\n\t\tgraph.save(save_model + \".predict\", prediction_saver(), gzip_output, save_vertices, save_edges, threads_per_machine);\n\t}\n\n\tgraphlab::mpi_tools::finalize();\n\treturn EXIT_SUCCESS;\n} // end of main\n\n\n\n"
  },
  {
    "path": "toolkits/collaborative_filtering/als.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * \\file\n * \n * \\brief The main file for the ALS matrix factorization algorithm.\n *\n * This file contains the main body of the ALS matrix factorization\n * algorithm. \n */\n\n#include \n\n#include \n#include \n#include \n#include \n#include \n\n\n\n\n// This file defines the serialization code for the eigen types.\n#include \"eigen_serialization.hpp\"\n\n#include \n#include \n#include \"stats.hpp\"\n\n#include \n\nconst int SAFE_NEG_OFFSET = 2; //add 2 to negative node id\n//to prevent -0 and -1 which arenot allowed\n\n\n\n/**\n * \\brief We use the eigen library's vector type to represent\n * mathematical vectors.\n */\ntypedef Eigen::VectorXd vec_type;\n\n/**\n * \\brief We use the eigen library's matrix type to represent\n * matrices.\n */\ntypedef Eigen::MatrixXd mat_type;\n\n\n\n\n/** \n * \\ingroup toolkit_matrix_factorization\n *\n * \\brief the vertex data type which contains the latent factor.\n *\n * Each row and each column in the matrix corresponds to a different\n * vertex in the ALS graph.  Associated with each vertex is a factor\n * (vector) of latent parameters that represent that vertex.  The goal\n * of the ALS algorithm is to find the values for these latent\n * parameters such that the non-zero entries in the matrix can be\n * predicted by taking the dot product of the row and column factors.\n */\nstruct vertex_data {\n  /**\n   * \\brief A shared \"constant\" that specifies the number of latent\n   * values to use.\n   */\n  static size_t NLATENT;\n  /** \\brief The number of times this vertex has been updated. */\n  uint32_t nupdates;\n  /** \\brief The most recent L1 change in the factor value */\n  float residual; //! how much the latent value has changed\n  /** \\brief The latent factor for this vertex */\n  vec_type factor;\n  /** \n   * \\brief Simple default constructor which randomizes the vertex\n   *  data \n   */\n  vertex_data() : nupdates(0), residual(1) { randomize(); } \n  /** \\brief Randomizes the latent factor */\n  void randomize() { factor.resize(NLATENT); factor.setRandom(); }\n  /** \\brief Save the vertex data to a binary archive */\n  void save(graphlab::oarchive& arc) const { \n    arc << nupdates << residual << factor;        \n  }\n  /** \\brief Load the vertex data from a binary archive */\n  void load(graphlab::iarchive& arc) { \n    arc >> nupdates >> residual >> factor;\n  }\n}; // end of vertex data\n\n\nsize_t vertex_data::NLATENT = 20;\n\n/**\n * \\brief The edge data stores the entry in the matrix.\n *\n * In addition the edge data also stores the most recent error estimate.\n */\nstruct edge_data : public graphlab::IS_POD_TYPE {\n  /**\n   * \\brief The type of data on the edge;\n   *\n   * \\li *Train:* the observed value is correct and used in training\n   * \\li *Validate:* the observed value is correct but not used in training\n   * \\li *Predict:* The observed value is not correct and should not be\n   *        used in training.\n   */\n  enum data_role_type { TRAIN, VALIDATE, PREDICT  };\n\n  /** \\brief the observed value for the edge */\n  float obs;\n\n  /** \\brief The train/validation/test designation of the edge */\n  data_role_type role;\n\n  /** \\brief basic initialization */\n  edge_data(float obs = 0, data_role_type role = TRAIN) :\n    obs(obs), role(role) { }\n\n}; // end of edge data\n\n\n/**\n * \\brief The graph type is defined in terms of the vertex and edge\n * data.\n */ \ntypedef graphlab::distributed_graph graph_type;\n\n#include \"implicit.hpp\"\n\nstats_info count_edges(const graph_type::edge_type & edge){\n  stats_info ret;\n\n  if (edge.data().role == edge_data::TRAIN)\n     ret.training_edges = 1;\n  else if (edge.data().role == edge_data::VALIDATE)\n     ret.validation_edges = 1;\n  ret.max_user = (size_t)edge.source().id();\n  ret.max_item = (-edge.target().id()-SAFE_NEG_OFFSET);\n  return ret;\n}\n\n\n\n/**\n * \\brief Given a vertex and an edge return the other vertex in the\n * edge.\n */\ninline graph_type::vertex_type\nget_other_vertex(graph_type::edge_type& edge, \n                 const graph_type::vertex_type& vertex) {\n  return vertex.id() == edge.source().id()? edge.target() : edge.source();\n}; // end of get_other_vertex\n\n\n\n\n/**\n * \\brief The gather type used to construct XtX and Xty needed for the ALS\n * update\n *\n * To compute the ALS update we need to compute the sum of \n * \\code\n *  sum: XtX = nbr.factor.transpose() * nbr.factor \n *  sum: Xy  = nbr.factor * edge.obs\n * \\endcode\n * For each of the neighbors of a vertex. \n *\n * To do this in the Gather-Apply-Scatter model the gather function\n * computes and returns a pair consisting of XtX and Xy which are then\n * added. The gather type represents that tuple and provides the\n * necessary gather_type::operator+= operation.\n *\n */\nclass gather_type {\npublic:\n  /**\n   * \\brief Stores the current sum of nbr.factor.transpose() *\n   * nbr.factor\n   */\n  mat_type XtX;\n\n  /**\n   * \\brief Stores the current sum of nbr.factor * edge.obs\n   */\n  vec_type Xy;\n\n  /** \\brief basic default constructor */\n  gather_type() { }\n\n  /**\n   * \\brief This constructor computes XtX and Xy and stores the result\n   * in XtX and Xy\n   */\n  gather_type(const vec_type& X, const double y) :\n    XtX(X.size(), X.size()), Xy(X.size()) {\n    XtX.triangularView() = X * X.transpose();\n    Xy = X * y;\n  } // end of constructor for gather type\n\n  /** \\brief Save the values to a binary archive */\n  void save(graphlab::oarchive& arc) const { arc << XtX << Xy; }\n\n  /** \\brief Read the values from a binary archive */\n  void load(graphlab::iarchive& arc) { arc >> XtX >> Xy; }  \n\n  /** \n   * \\brief Computes XtX += other.XtX and Xy += other.Xy updating this\n   * tuples value\n   */\n  gather_type& operator+=(const gather_type& other) {\n    if(other.Xy.size() == 0) {\n      ASSERT_EQ(other.XtX.rows(), 0);\n      ASSERT_EQ(other.XtX.cols(), 0);\n    } else {\n      if(Xy.size() == 0) {\n        ASSERT_EQ(XtX.rows(), 0); \n        ASSERT_EQ(XtX.cols(), 0);\n        XtX = other.XtX; Xy = other.Xy;\n      } else {\n        XtX.triangularView() += other.XtX;  \n        Xy += other.Xy;\n      }\n    }\n    return *this;\n  } // end of operator+=\n\n}; // end of gather type\n\n\n\n/**\n * \\brief ALS vertex program implements the alternating least squares\n * algorithm in the Gather-Apply-Scatter abstraction.\n *\n * The ALS update treats adjacent vertices (rows or columns) as \"X\"\n * (independent) values and the edges (matrix entries) as observed \"y\"\n * (dependent) values and then updates the current vertex value as a\n * weight \"w\" such that:\n *\n *    y = X * w + noise\n *\n * This is accomplished using the following equation:\n *\n *    w = inv(X' * X) * (X' * y)\n *\n * We implement this in the Gather-Apply-Scatter model by:\n *\n *  1) Gather: returns the tuple (X' * X, X' * y)\n *     Sum:   (aX' * aX, aX * ay) + (bX' * bX, bX * by) = \n *                 (aX' * aX + bX' * bX, aX * ay + bX * by)\n *\n *  2) Apply: Solves  inv(X' * X) * (X' * y)\n *\n *  3) Scatter: schedules the update of adjacent vertices if this\n *      vertex has changed sufficiently and the edge is not well\n *      predicted.\n *\n * \n */ \nclass als_vertex_program : \n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\npublic:\n  /** The convergence tolerance */\n  static double TOLERANCE;\n  static double LAMBDA;\n  static size_t MAX_UPDATES;\n  static double MAXVAL;\n  static double MINVAL;\n  static int    REGNORMAL; //regularization type\n\n  /** The set of edges to gather along */\n  edge_dir_type gather_edges(icontext_type& context, \n                             const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of gather_edges \n\n  /** The gather function computes XtX and Xy */\n  gather_type gather(icontext_type& context, const vertex_type& vertex, \n                     edge_type& edge) const {\n    if(edge.data().role == edge_data::TRAIN) {\n      const vertex_type other_vertex = get_other_vertex(edge, vertex);\n      return gather_type(other_vertex.data().factor, edge.data().obs);\n    } else return gather_type();\n  } // end of gather function\n\n  /** apply collects the sum of XtX and Xy */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& sum) {\n    // Get and reset the vertex data\n    vertex_data& vdata = vertex.data(); \n    // Determine the number of neighbors.  Each vertex has only in or\n    // out edges depending on which side of the graph it is located\n    if(sum.Xy.size() == 0) { vdata.residual = 0; ++vdata.nupdates; return; }\n    mat_type XtX = sum.XtX;\n    vec_type Xy = sum.Xy;\n    // Add regularization\n    double regularization = LAMBDA;\n    if (REGNORMAL)\n      regularization = LAMBDA*vertex.num_out_edges();\n    for(int i = 0; i < XtX.rows(); ++i) \n      XtX(i,i) += regularization; \n    // Solve the least squares problem using eigen ----------------------------\n    const vec_type old_factor = vdata.factor;\n    vdata.factor = XtX.selfadjointView().ldlt().solve(Xy);\n    // Compute the residual change in the factor factor -----------------------\n    vdata.residual = (vdata.factor - old_factor).cwiseAbs().sum() / XtX.rows();\n    ++vdata.nupdates;\n  } // end of apply\n  \n  /** The edges to scatter along */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of scatter edges\n\n  /** Scatter reschedules neighbors */  \n  void scatter(icontext_type& context, const vertex_type& vertex, \n               edge_type& edge) const {\n    edge_data& edata = edge.data();\n    if(edata.role == edge_data::TRAIN) {\n      const vertex_type other_vertex = get_other_vertex(edge, vertex);\n      const vertex_data& vdata = vertex.data();\n      const vertex_data& other_vdata = other_vertex.data();\n      //TODO:\n      //    Do we need to cap the prediction value into [min, max] here?\n      const double pred = vdata.factor.dot(other_vdata.factor);\n      const float error = std::fabs(edata.obs - pred);\n      const double priority = (error * vdata.residual); \n      // Reschedule neighbors ------------------------------------------------\n      if( priority > TOLERANCE && other_vdata.nupdates < MAX_UPDATES) \n        context.signal(other_vertex, priority);\n    }\n  } // end of scatter function\n\n\n  /**\n   * \\brief Signal all vertices on one side of the bipartite graph\n   */\n  static graphlab::empty signal_left(icontext_type& context,\n                                     const vertex_type& vertex) {\n    if(vertex.num_out_edges() > 0) context.signal(vertex);\n    return graphlab::empty();\n  } // end of signal_left \n\n\n\n}; // end of als vertex program\n\n\n\n/**\n * \\brief The graph loader function is a line parser used for\n * distributed graph construction.\n */\ninline bool graph_loader(graph_type& graph, \n                         const std::string& filename,\n                         const std::string& line) {\n  ASSERT_FALSE(line.empty()); \n  namespace qi = boost::spirit::qi;\n  namespace ascii = boost::spirit::ascii;\n  namespace phoenix = boost::phoenix;\n  // Determine the role of the data\n  edge_data::data_role_type role = edge_data::TRAIN;\n  if(boost::ends_with(filename,\".validate\")) role = edge_data::VALIDATE;\n  else if(boost::ends_with(filename, \".predict\")) role = edge_data::PREDICT;\n  // Parse the line\n  graph_type::vertex_id_type source_id(-1), target_id(-1);\n  float obs(0); \n  const bool success = qi::phrase_parse\n    (line.begin(), line.end(),       \n     //  Begin grammar\n     (\n      qi::ulong_[phoenix::ref(source_id) = qi::_1] >> -qi::char_(',') >>\n      qi::ulong_[phoenix::ref(target_id) = qi::_1] >> \n      -(-qi::char_(',') >> qi::float_[phoenix::ref(obs) = qi::_1])\n      )\n     ,\n     //  End grammar\n     ascii::space); \n\n  if(!success) return false;\n\n  if(role == edge_data::TRAIN || role == edge_data::VALIDATE){\n    if (obs < als_vertex_program::MINVAL || obs > als_vertex_program::MAXVAL)\n      logstream(LOG_FATAL)<<\"Rating values should be between \" << als_vertex_program::MINVAL << \" and \" << als_vertex_program::MAXVAL << \". Got value: \" << obs << \" [ user: \" << source_id << \" to item: \" < 0) {\n      const double validation_error = \n        std::sqrt(agg.validation_error / info.validation_edges);\n      context.cout() << \"\\tValidation RMSE: \" << validation_error; \n    }\n    context.cout() << std::endl;\n  }\n}; // end of error aggregator\n\n\n\n\n/**\n * \\brief The prediction saver is used by the graph.save routine to\n * output the final predictions back to the filesystem.\n */\nstruct prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    return \"\"; //nop\n  }\n  std::string save_edge(const edge_type& edge) const {\n    if(edge.data().role == edge_data::PREDICT) {\n      std::stringstream strm;\n      double prediction =\n        edge.source().data().factor.dot(edge.target().data().factor);\n\t  prediction = std::min(als_vertex_program::MAXVAL, prediction);\n\t  prediction = std::max(als_vertex_program::MINVAL, prediction);\n      strm << edge.source().id() << '\\t';\n      strm << (-edge.target().id() - SAFE_NEG_OFFSET) << '\\t';\n      strm << prediction << '\\n';\n      return strm.str();\n    } else return \"\";\n  }\n}; // end of prediction_saver\n\n\nstruct linear_model_saver_U {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid factor1 factor2 ... factorNLATENT \\n\n  */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() > 0){\n      std::string ret = boost::lexical_cast(vertex.id()) + \" \";\n      for (uint i=0; i< vertex_data::NLATENT; i++)\n        ret += boost::lexical_cast(vertex.data().factor[i]) + \" \";\n        ret += \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\nstruct linear_model_saver_V {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid factor1 factor2 ... factorNLATENT \\n\n  */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() == 0){\n      std::string ret = boost::lexical_cast(-vertex.id()-SAFE_NEG_OFFSET) + \" \";\n      for (uint i=0; i< vertex_data::NLATENT; i++)\n        ret += boost::lexical_cast(vertex.data().factor[i]) + \" \";\n        ret += \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\n\n\n/**\n * \\brief The engine type used by the ALS matrix factorization\n * algorithm.\n *\n * The ALS matrix factorization algorithm currently uses the\n * synchronous engine.  However we plan to add support for alternative\n * engines in the future.\n */\ntypedef graphlab::omni_engine engine_type;\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n\n  // Parse command line options -----------------------------------------------\n  const std::string description = \n    \"Compute the ALS factorization of a matrix.\";\n  graphlab::command_line_options clopts(description);\n  std::string input_dir;\n  std::string predictions;\n  size_t interval = 10;\n  std::string exec_type = \"synchronous\";\n  clopts.attach_option(\"matrix\", input_dir,\n                       \"The directory containing the matrix file\");\n  clopts.add_positional(\"matrix\");\n  clopts.attach_option(\"D\",  vertex_data::NLATENT,\n                       \"Number of latent parameters to use.\");\n  clopts.attach_option(\"max_iter\", als_vertex_program::MAX_UPDATES,\n                       \"The maxumum number of udpates allowed for a vertex\");\n  clopts.attach_option(\"lambda\", als_vertex_program::LAMBDA, \n                       \"ALS regularization weight\"); \n  clopts.attach_option(\"tol\", als_vertex_program::TOLERANCE,\n                       \"residual termination threshold\");\n  clopts.attach_option(\"maxval\", als_vertex_program::MAXVAL, \"max allowed value\");\n  clopts.attach_option(\"minval\", als_vertex_program::MINVAL, \"min allowed value\");\n  clopts.attach_option(\"interval\", interval, \n                       \"The time in seconds between error reports\");\n  clopts.attach_option(\"predictions\", predictions,\n                       \"The prefix (folder and filename) to save predictions.\");\n  clopts.attach_option(\"engine\", exec_type, \n                       \"The engine type synchronous or asynchronous\");\n  clopts.attach_option(\"regnormal\", als_vertex_program::REGNORMAL, \n                       \"regularization type. 1 = weighted according to neighbors num. 0 = no weighting - just lambda\");\n  \n  parse_implicit_command_line(clopts);\n  \n  if(!clopts.parse(argc, argv) || input_dir == \"\") {\n    std::cout << \"Error in parsing command line arguments.\" << std::endl;\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n\n\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n  dc.cout() << \"minval: \" << als_vertex_program::MINVAL << std::endl;\n  dc.cout() << \"maxval: \" << als_vertex_program::MAXVAL << std::endl;\n  \n  dc.cout() << \"Loading graph.\" << std::endl;\n  graphlab::timer timer; \n  graph_type graph(dc, clopts);  \n  graph.load(input_dir, graph_loader); \n  dc.cout() << \"Loading graph. Finished in \" \n            << timer.current_time() << std::endl;\n\n  if (dc.procid() == 0) \n    add_implicit_edges(implicitratingtype, graph, dc);\n  \n  dc.cout() << \"Finalizing graph.\" << std::endl;\n  timer.start();\n   graph.finalize();\n  dc.cout() << \"Finalizing graph. Finished in \" \n            << timer.current_time() << std::endl;\n\n  if (!graph.num_edges() || !graph.num_vertices())\n     logstream(LOG_FATAL)<< \"Failed to load graph. Check your input path: \" << input_dir << std::endl;     \n\n\n  dc.cout() \n      << \"========== Graph statistics on proc \" << dc.procid() \n      << \" ===============\"\n      << \"\\n Num vertices: \" << graph.num_vertices()\n      << \"\\n Num edges: \" << graph.num_edges()\n      << \"\\n Num replica: \" << graph.num_replicas()\n      << \"\\n Replica to vertex ratio: \" \n      << float(graph.num_replicas())/graph.num_vertices()\n      << \"\\n --------------------------------------------\" \n      << \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n      << \"\\n Num local vertices: \" << graph.num_local_vertices()\n      << \"\\n Replica to own ratio: \" \n      << (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n      << \"\\n Num local edges: \" << graph.num_local_edges()\n      //<< \"\\n Begin edge id: \" << graph.global_eid(0)\n      << \"\\n Edge balance ratio: \" \n      << float(graph.num_local_edges())/graph.num_edges()\n      << std::endl;\n \n  dc.cout() << \"Creating engine\" << std::endl;\n  engine_type engine(dc, graph, exec_type, clopts);\n\n  // Add error reporting to the engine\n  const bool success = engine.add_edge_aggregator\n    (\"error\", error_aggregator::map, error_aggregator::finalize) &&\n    engine.aggregate_periodic(\"error\", interval);\n  ASSERT_TRUE(success);\n  \n\n  // Signal all vertices on the vertices on the left (liberals) \n  engine.map_reduce_vertices(als_vertex_program::signal_left);\n  info = graph.map_reduce_edges(count_edges);\n  dc.cout()<<\"Training edges: \" << info.training_edges << \" validation edges: \" << info.validation_edges << std::endl;\n\n  // Run ALS ---------------------------------------------------------\n  dc.cout() << \"Running ALS\" << std::endl;\n  timer.start();\n  engine.start();  \n\n  const double runtime = timer.current_time();\n  dc.cout() << \"----------------------------------------------------------\"\n            << std::endl\n            << \"Final Runtime (seconds):   \" << runtime \n            << std::endl\n            << \"Updates executed: \" << engine.num_updates() << std::endl\n            << \"Update Rate (updates/second): \" \n            << engine.num_updates() / runtime << std::endl;\n\n  // Compute the final training error -----------------------------------------\n  dc.cout() << \"Final error: \" << std::endl;\n  engine.aggregate_now(\"error\");\n\n  // Make predictions ---------------------------------------------------------\n  if(!predictions.empty()) {\n    std::cout << \"Saving predictions\" << std::endl;\n    const bool gzip_output = false;\n    const size_t threads_per_machine = 2;\n\n    //save the predictions\n    graph.save(predictions, prediction_saver(),\n               gzip_output, false, \n               true, threads_per_machine);\n    //save the linear model\n    graph.save(predictions + \".U\", linear_model_saver_U(),\n\t\tgzip_output, true, false, threads_per_machine);\n    graph.save(predictions + \".V\", linear_model_saver_V(),\n\t\tgzip_output, true, false, threads_per_machine);\n  \n  }\n             \n\n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // end of main\n\n\n\n"
  },
  {
    "path": "toolkits/collaborative_filtering/biassgd.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * \\file\n * \n * \\brief The main file for the BIAS-SGD matrix factorization algorithm.\n *\n * This file contains the main body of the BIAS-SGD matrix factorization\n * algorithm. \n */\n\n#include \n#include \n#include \"eigen_serialization.hpp\"\n#include \n#include \n\n\n\ntypedef Eigen::VectorXd vec_type;\ntypedef Eigen::MatrixXd mat_type;\n\n//when using negative node id range, we are not allowed to use\n//0 and 1 so we add 2.\nconst static int SAFE_NEG_OFFSET=2;\nstatic bool debug;\nint iter = 0;\n/** \n * \\ingroup toolkit_matrix_pvecization\n *\n * \\brief the vertex data type which contains the latent pvec.\n *\n * Each row and each column in the matrix corresponds to a different\n * vertex in the BIASSGD graph.  Associated with each vertex is a pvec\n * (vector) of latent parameters that represent that vertex.  The goal\n * of the BIASSGD algorithm is to find the values for these latent\n * parameters such that the non-zero entries in the matrix can be\n * predicted by taking the dot product of the row and column pvecs.\n */\nstruct vertex_data {\n  /**\n   * \\brief A shared \"constant\" that specifies the number of latent\n   * values to use.\n   */\n  static size_t NLATENT;\n  /** \\brief The number of times this vertex has been updated. */\n  uint32_t nupdates;\n  /** \\brief The latent pvec for this vertex */\n  vec_type pvec;\n  double bias;\n  /** \n   * \\brief Simple default constructor which randomizes the vertex\n   *  data \n   */\n  vertex_data() : nupdates(0), bias(0) { if (debug) pvec = vec_type::Ones(NLATENT); else randomize(); } \n  /** \\brief Randomizes the latent pvec */\n  void randomize() { pvec.resize(NLATENT); pvec.setRandom(); }\n  /** \\brief Save the vertex data to a binary archive */\n  void save(graphlab::oarchive& arc) const { \n    arc << nupdates << pvec << bias;\n  }\n  /** \\brief Load the vertex data from a binary archive */\n  void load(graphlab::iarchive& arc) { \n    arc >> nupdates >> pvec >> bias;\n  }\n}; // end of vertex data\n\n\n/**\n * \\brief The edge data stores the entry in the matrix.\n *\n * In addition the edge data biassgdo stores the most recent error estimate.\n */\nstruct edge_data : public graphlab::IS_POD_TYPE {\n  /**\n   * \\brief The type of data on the edge;\n   *\n   * \\li *Train:* the observed value is correct and used in training\n   * \\li *Validate:* the observed value is correct but not used in training\n   * \\li *Predict:* The observed value is not correct and should not be\n   *        used in training.\n   */\n  enum data_role_type { TRAIN, VALIDATE, PREDICT  };\n\n  /** \\brief the observed value for the edge */\n  float obs;\n\n  /** \\brief The train/validation/test designation of the edge */\n  data_role_type role;\n\n  /** \\brief basic initialization */\n  edge_data(float obs = 0, data_role_type role = PREDICT) :\n    obs(obs), role(role) { }\n\n}; // end of edge data\n\n\n/**\n * \\brief The graph type is defined in terms of the vertex and edge\n * data.\n */ \ntypedef graphlab::distributed_graph graph_type;\n\n#include \"implicit.hpp\"\n\ndouble extract_l2_error(const graph_type::edge_type & edge);\n\n\n/**\n * \\brief Given a vertex and an edge return the other vertex in the\n * edge.\n */\ninline graph_type::vertex_type\nget_other_vertex(graph_type::edge_type& edge, \n                 const graph_type::vertex_type& vertex) {\n  return vertex.id() == edge.source().id()? edge.target() : edge.source();\n}; // end of get_other_vertex\n\n\n\n\n\n/**\n *\n */\nclass gather_type {\npublic:\n  vec_type pvec; //vector of gradient updates\n  double bias; //bias change\n\n  /** \\brief basic default constructor */\n  gather_type() { }\n\n  /**\n   */\n  gather_type(const vec_type& X, double _bias) {\n    pvec = X;\n    bias = _bias;\n  } // end of constructor for gather type\n\n  /** \\brief Save the values to a binary archive */\n  void save(graphlab::oarchive& arc) const { arc << pvec << bias; }\n\n  /** \\brief Read the values from a binary archive */\n  void load(graphlab::iarchive& arc) { arc >> pvec >> bias; }  \n\n  /** \n   */\n  gather_type& operator+=(const gather_type& other) {\n    if (pvec.size() == 0){\n      pvec = other.pvec;\n      bias = other.bias;\n      return *this;\n    }\n    else if (other.pvec.size() == 0)\n      return *this;\n    //sum up gradient updates\n    pvec += other.pvec;\n    bias += other.bias;\n    return *this;\n  } // end of operator+=\n\n}; // end of gather type\n\n//typedef gather_type message_type;\n\n/**\n * BIASSGD vertex program type\n */ \nclass biassgd_vertex_program : \n  public graphlab::ivertex_program {\npublic:\n  /** The convergence tolerance */\n  static double TOLERANCE;\n  static double LAMBDA;\n  static double GAMMA;\n  static double MAXVAL;\n  static double MINVAL;\n  static double STEP_DEC;\n  static bool debug;\n  static size_t MAX_UPDATES;\n  static double GLOBAL_MEAN;\n  static size_t NUM_TRAINING_EDGES;\n  static uint   USERS;\n\n  gather_type pmsg;\n  void save(graphlab::oarchive& arc) const { \n    arc << pmsg;\n  }\n  /** \\brief Load the vertex data from a binary archive */\n  void load(graphlab::iarchive& arc) { \n    arc >> pmsg;\n  }\n\n  /** The set of edges to gather along */\n  edge_dir_type gather_edges(icontext_type& context, \n                             const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of gather_edges \n\n  /** The gather function computes XtX and Xy */\n  gather_type gather(icontext_type& context, const vertex_type& vertex, \n                     edge_type& edge) const {\n    //if(edge.data().role == edge_data::TRAIN) {\n   vec_type delta, other_delta;\n   double bias =0, other_bias = 0;\n\n   //this is a user node\n   if (vertex.num_in_edges() == 0){\n      vertex_type other_vertex(get_other_vertex(edge, vertex));\n      vertex_type my_vertex(vertex);\n      //predict rating\n      double pred = biassgd_vertex_program::GLOBAL_MEAN + \n        edge.source().data().bias + edge.target().data().bias + \n        vertex.data().pvec.dot(other_vertex.data().pvec);\n      pred = std::min(pred, biassgd_vertex_program::MAXVAL);\n      pred = std::max(pred, biassgd_vertex_program::MINVAL); \n      //compute the error\n      const float err = (pred - edge.data().obs);\n      if (debug)\n        std::cout<<\"entering edge \" << (int)edge.source().id() << \":\" << (int)edge.target().id() << \" err: \" << err << \" rmse: \" << err*err < TOLERANCE && other_vertex.data().nupdates < MAX_UPDATES) \n          context.signal(other_vertex, gather_type(other_delta, other_bias));\n       }\n      return gather_type(delta, bias);\n    } \n    else return gather_type(delta, bias);\n  } // end of gather function\n\n//typedef vec_type message_type;\n void init(icontext_type& context,\n                              const vertex_type& vertex,\n                              const message_type& msg) {\n     if (vertex.num_in_edges() > 0){\n        pmsg = msg;\n     }\n  }\n  /** apply graident updates to feature vector */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& sum) {\n    // Get and reset the vertex data\n    vertex_data& vdata = vertex.data(); \n    if (sum.pvec.size() > 0){\n      vdata.pvec += sum.pvec; \n      vdata.bias += sum.bias;\n      assert(vertex.num_in_edges() == 0);\n    }\n    else if (pmsg.pvec.size() > 0){\n      vdata.pvec += pmsg.pvec;\n      vdata.bias += pmsg.bias;\n      assert(vertex.num_out_edges() == 0); \n    }\n    ++vdata.nupdates;\n  } // end of apply\n  \n  /** The edges to scatter along */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of scatter edges\n\n  /** Scatter reschedules neighbors */  \n  void scatter(icontext_type& context, const vertex_type& vertex, \n               edge_type& edge) const {\n    edge_data& edata = edge.data();\n    if(edata.role == edge_data::TRAIN) {\n      const vertex_type other_vertex = get_other_vertex(edge, vertex);\n      // Reschedule neighbors ------------------------------------------------\n      if(other_vertex.data().nupdates < MAX_UPDATES) \n        context.signal(other_vertex, gather_type(vec_type::Zero(vertex_data::NLATENT),0));\n    }\n  } // end of scatter function\n\n\n  /**\n   * \\brief Signal all vertices on one side of the bipartite graph\n   */\n  static graphlab::empty signal_left(icontext_type& context,\n                                     vertex_type& vertex) {\n    if(vertex.num_out_edges() > 0) context.signal(vertex, gather_type(vec_type::Zero(vertex_data::NLATENT),0));\n    return graphlab::empty();\n  } // end of signal_left \n\n}; // end of biassgd vertex program\n\n\nstruct error_aggregator : public graphlab::IS_POD_TYPE {\n  typedef biassgd_vertex_program::icontext_type icontext_type;\n  typedef graph_type::edge_type edge_type;\n  double train_error, validation_error;\n  size_t ntrain, nvalidation;\n  error_aggregator() : \n    train_error(0), validation_error(0), ntrain(0), nvalidation(0) { }\n  error_aggregator& operator+=(const error_aggregator& other) {\n    train_error += other.train_error;\n    assert(!std::isnan(train_error));\n    validation_error += other.validation_error;\n    ntrain += other.ntrain;\n    nvalidation += other.nvalidation;\n    return *this;\n  }\n  static error_aggregator map(icontext_type& context, const graph_type::edge_type& edge) {\n    error_aggregator agg;\n    if (edge.data().role == edge_data::TRAIN){\n      agg.train_error = extract_l2_error(edge); agg.ntrain = 1;\n      assert(!std::isnan(agg.train_error));\n    }\n    else if (edge.data().role == edge_data::VALIDATE){\n      agg.validation_error = extract_l2_error(edge); agg.nvalidation = 1;\n    }\n    return agg;\n  }\n\n\n  static void finalize(icontext_type& context, const error_aggregator& agg) {\n    iter++;\n    if (iter%2 == 0)\n      return; \n    ASSERT_GT(agg.ntrain, 0);\n    const double train_error = std::sqrt(agg.train_error / agg.ntrain);\n    assert(!std::isnan(train_error));\n    context.cout() << std::setw(8) << context.elapsed_seconds() << \"  \" << std::setw(8) << train_error;\n    if(agg.nvalidation > 0) {\n      const double validation_error = \n        std::sqrt(agg.validation_error / agg.nvalidation);\n      context.cout() << \"   \" << std::setw(8) << validation_error; \n    }\n    context.cout() << std::endl;\n    biassgd_vertex_program::GAMMA *= biassgd_vertex_program::STEP_DEC;\n  }\n}; // end of error aggregator\n\n/**\n * \\brief Given an edge compute the error associated with that edge\n */\ndouble extract_l2_error(const graph_type::edge_type & edge) {\n  double pred = biassgd_vertex_program::GLOBAL_MEAN + \n      edge.source().data().bias +\n      edge.target().data().bias + \n      edge.source().data().pvec.dot(edge.target().data().pvec);\n  pred = std::min(biassgd_vertex_program::MAXVAL, pred);\n  pred = std::max(biassgd_vertex_program::MINVAL, pred);\n  double rmse = (edge.data().obs - pred) * (edge.data().obs - pred);\n  assert(rmse <= pow(biassgd_vertex_program::MAXVAL-biassgd_vertex_program::MINVAL,2));\n  return rmse;\n} // end of extract_l2_error\n\n\nstruct prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    return \"\"; //nop\n  }\n  std::string save_edge(const edge_type& edge) const {\n   if (edge.data().role != edge_data::PREDICT)\n      return \"\";\n\n std::stringstream strm;\n  double prediction =  biassgd_vertex_program::GLOBAL_MEAN + \n      edge.source().data().bias +\n      edge.target().data().bias + \n      edge.source().data().pvec.dot(edge.target().data().pvec);\n    prediction = std::min(biassgd_vertex_program::MAXVAL, prediction);\n    prediction = std::max(biassgd_vertex_program::MINVAL, prediction);\n    strm << edge.source().id() << '\\t' \n         << -edge.target().id()-SAFE_NEG_OFFSET << '\\t'\n         << prediction << '\\n';\n    return strm.str();\n  }\n}; // end of prediction_saver\n\nstruct linear_model_saver_U {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n  */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() > 0){\n      std::string ret = boost::lexical_cast(vertex.id()) + \" \";\n      for (uint i=0; i< vertex_data::NLATENT; i++)\n        ret += boost::lexical_cast(vertex.data().pvec[i]) + \" \";\n        ret += \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\nstruct linear_model_saver_V {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n  */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() == 0){\n      std::string ret = boost::lexical_cast(-vertex.id()-SAFE_NEG_OFFSET) + \" \";\n      for (uint i=0; i< vertex_data::NLATENT; i++)\n        ret += boost::lexical_cast(vertex.data().pvec[i]) + \" \";\n        ret += \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\nstruct linear_model_saver_bias_U {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n  */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() > 0){\n      std::string ret = boost::lexical_cast(vertex.id()) + \" \";\n      ret += boost::lexical_cast(vertex.data().bias) + \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \nstruct linear_model_saver_bias_V {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n  */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() == 0){\n      std::string ret = boost::lexical_cast(-vertex.id()-SAFE_NEG_OFFSET) + \" \";\n      ret += boost::lexical_cast(vertex.data().bias) + \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\n\n/**\n * \\brief The graph loader function is a line parser used for\n * distributed graph construction.\n */\ninline bool graph_loader(graph_type& graph, \n    const std::string& filename,\n    const std::string& line) {\n\n // Parse the line\n  std::stringstream strm(line);\n  graph_type::vertex_id_type source_id(-1), target_id(-1);\n  float obs(0);\n  strm >> source_id >> target_id;\n\n  if (source_id == graph_type::vertex_id_type(-1) || target_id == graph_type::vertex_id_type(-1)){\n    logstream(LOG_WARNING)<<\"Failed to read input line: \"<< line << \" in file: \"  << filename << \" (or node id is -1). \" << std::endl;\n    return true;\n  }\n\n  // Determine the role of the data\n  edge_data::data_role_type role = edge_data::TRAIN;\n  if(boost::ends_with(filename,\".validate\")) role = edge_data::VALIDATE;\n  else if(boost::ends_with(filename, \".predict\")) role = edge_data::PREDICT;\n \n  if(role == edge_data::TRAIN || role == edge_data::VALIDATE){\n    strm >> obs;\n    if (obs < biassgd_vertex_program::MINVAL || obs > biassgd_vertex_program::MAXVAL){\n      logstream(LOG_WARNING)<<\"Rating values should be between \" << biassgd_vertex_program::MINVAL << \" and \" << biassgd_vertex_program::MAXVAL << \". Got value: \" << obs << \" [ user: \" << source_id << \" to item: \" < engine_type;\n\ndouble calc_global_mean(const graph_type::edge_type & edge){\n  if (edge.data().role == edge_data::TRAIN)\n     return edge.data().obs;\n  else return 0;\n}\n\nsize_t count_edges(const graph_type::edge_type & edge){\n  if (edge.data().role == edge_data::TRAIN)\n     return 1;\n  else return 0;\n}\n\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n\n  // Parse command line options -----------------------------------------------\n  const std::string description = \n    \"Compute the bias-SGD factorization of a matrix.\";\n  graphlab::command_line_options clopts(description);\n  std::string input_dir;\n  std::string predictions;\n  size_t interval = 0;\n  std::string exec_type = \"synchronous\";\n  clopts.attach_option(\"matrix\", input_dir,\n                       \"The directory containing the matrix file\");\n  clopts.add_positional(\"matrix\");\n  clopts.attach_option(\"D\", vertex_data::NLATENT,\n                       \"Number of latent parameters to use.\");\n  clopts.attach_option(\"engine\", exec_type, \n                       \"The engine type synchronous or asynchronous\");\n  clopts.attach_option(\"max_iter\", biassgd_vertex_program::MAX_UPDATES,\n                       \"The maxumum number of udpates allowed for a vertex\");\n  clopts.attach_option(\"lambda\", biassgd_vertex_program::LAMBDA, \n                       \"SGD regularization weight\"); \n  clopts.attach_option(\"gamma\", biassgd_vertex_program::GAMMA, \n                       \"SGD step size\"); \n  clopts.attach_option(\"debug\", biassgd_vertex_program::debug, \n                       \"debug - additional verbose info\"); \n  clopts.attach_option(\"tol\", biassgd_vertex_program::TOLERANCE,\n                       \"residual termination threshold\");\n  clopts.attach_option(\"maxval\", biassgd_vertex_program::MAXVAL, \"max allowed value\");\n  clopts.attach_option(\"minval\", biassgd_vertex_program::MINVAL, \"min allowed value\");\n  clopts.attach_option(\"step_dec\", biassgd_vertex_program::STEP_DEC, \"multiplicative step decrement\");\n  clopts.attach_option(\"interval\", interval, \n                       \"The time in seconds between error reports\");\n  clopts.attach_option(\"predictions\", predictions,\n                       \"The prefix (folder and filename) to save predictions.\");\n\n  parse_implicit_command_line(clopts);\n\n  if(!clopts.parse(argc, argv) || input_dir == \"\") {\n    std::cout << \"Error in parsing command line arguments.\" << std::endl;\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n debug = biassgd_vertex_program::debug;\n  //  omp_set_num_threads(clopts.get_ncpus());\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  \n  dc.cout() << \"Loading graph.\" << std::endl;\n  graphlab::timer timer; \n  graph_type graph(dc, clopts);  \n  graph.load(input_dir, graph_loader); \n  dc.cout() << \"Loading graph. Finished in \" \n            << timer.current_time() << std::endl;\n  if (dc.procid() == 0) \n    add_implicit_edges(implicitratingtype, graph, dc);\n\ndc.cout() << \"Finalizing graph.\" << std::endl;\n  timer.start();\n  graph.finalize();\n  dc.cout() << \"Finalizing graph. Finished in \" \n            << timer.current_time() << std::endl;\n\n  if (!graph.num_edges() || !graph.num_vertices())\n     logstream(LOG_FATAL)<< \"Failed to load graph. Check your input path: \" << input_dir << std::endl;     \n\n\n  dc.cout() \n      << \"========== Graph statistics on proc \" << dc.procid() \n      << \" ===============\"\n      << \"\\n Num vertices: \" << graph.num_vertices()\n      << \"\\n Num edges: \" << graph.num_edges()\n      << \"\\n Num replica: \" << graph.num_replicas()\n      << \"\\n Replica to vertex ratio: \" \n      << float(graph.num_replicas())/graph.num_vertices()\n      << \"\\n --------------------------------------------\" \n      << \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n      << \"\\n Num local vertices: \" << graph.num_local_vertices()\n      << \"\\n Replica to own ratio: \" \n      << (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n      << \"\\n Num local edges: \" << graph.num_local_edges()\n      //<< \"\\n Begin edge id: \" << graph.global_eid(0)\n      << \"\\n Edge balance ratio: \" \n      << float(graph.num_local_edges())/graph.num_edges()\n      << std::endl;\n \n  dc.cout() << \"Creating engine\" << std::endl;\n  engine_type engine(dc, graph, exec_type, clopts);\n\n  // Add error reporting to the engine\n  const bool success = engine.add_edge_aggregator\n    (\"error\", error_aggregator::map, error_aggregator::finalize) &&\n    engine.aggregate_periodic(\"error\", interval);\n  ASSERT_TRUE(success);\n  \n\n  biassgd_vertex_program::GLOBAL_MEAN = graph.map_reduce_edges(calc_global_mean);\n  biassgd_vertex_program::NUM_TRAINING_EDGES = graph.map_reduce_edges(count_edges);\n  biassgd_vertex_program::GLOBAL_MEAN /= biassgd_vertex_program::NUM_TRAINING_EDGES;\n  dc.cout() << \"Global mean is: \" <(biassgd_vertex_program::signal_left);\n \n\n  dc.cout() << \"Running Bias-SGD\" << std::endl;\n  dc.cout() << \"(C) Code by Danny Bickson, CMU \" << std::endl;\n  dc.cout() << \"Please send bug reports to danny.bickson@gmail.com\" << std::endl;\n  dc.cout() << \"Time   Training    Validation\" <\n\n// IMPLEMENTATION OF GAUSSIAN CFD\n// TAKEN FROM : http://www.johndcook.com/cpp_phi.html\n\n// constants\nconst double phi_a1 =  0.254829592;\nconst double phi_a2 = -0.284496736;\nconst double phi_a3 =  1.421413741;\nconst double phi_a4 = -1.453152027;\nconst double phi_a5 =  1.061405429;\nconst double phi_p  =  0.3275911;\n\n\ndouble phi(double x)\n{\n    // Save the sign of x\n    int sign = 1;\n    if (x < 0)\n        sign = -1;\n    x = fabs(x)/sqrt(2.0);\n\n    // A&S formula 7.1.26\n    double t = 1.0/(1.0 + phi_p*x);\n    double y = 1.0 - (((((phi_a5*t + phi_a4)*t) + phi_a3)*t + phi_a2)*t + phi_a1)*t*exp(-x*x);\n\n    return 0.5*(1.0 + sign*y);\n}\n\n\n#endif\n"
  },
  {
    "path": "toolkits/collaborative_filtering/collaborative_filtering.dox",
    "content": "/**\n\n\\page collaborative_filtering Collaborative Filtering\n\n\n\\brief The collaborative filtering toolkit contains tools for computing a linear model of the data, and predicting missing values based on this linear model. This is useful when computing recommendations for users. \n\nThe collaborative filtering toolkit is written by Danny Bickson, CMU. Please send any code related questions to our Google group. Any other inquiries can be directed to Danny.Bickson@gmail.com. You are more then welcome to visit my applied machine learning blog.\n\n\\section History\nIn GraphLab v1, the collaborative filtering package was implemented and optimized for a multicore machine.  \nCurrently this version is deprecated and no longer supported.\n\nIf you intend to utilize a single multicore machines it is recommended to take a look at GraphChi collaborative filtering toolkit, which can scale to datasets with billions of recommendations. \n\nIn GraphLAb v2, the collaborative filtering toolkit is distributed, targeted for a cluster with a few machines, \nthat way we can scale to much larger models.  \n\nThe GraphLab collaborative filtering toolkit 2.1 is under active\ndevelopment. Please contact us if you encounter any issues or would\nlike an additional features.\n\n\\section Algorithms\nThe collaborative filtering toolkit in GraphLab v.2 currently contains:\n\n - \\ref ALS \"Alternating Least Squares (ALS)\"\n\\verbatim\nYunhong Zhou, Dennis Wilkinson, Robert Schreiber and Rong Pan. Large-Scale Parallel Collaborative Filtering for the Netflix Prize. Proceedings of the 4th international conference on Algorithmic Aspects in Information and Management. Shanghai, China pp. 337-348, 2008.\n\\endverbatim\n - \\ref CCD++ \"ALS with parallel coordinate descent (CCD++)\"\n\\verbatim\nH.-F. Yu, C.-J. Hsieh, S. Si, I. S. Dhillon, Scalable Coordinate Descent Approaches to Parallel Matrix Factorization for Recommender Systems. IEEE International Conference on Data Mining(ICDM), December 2012.\nSteffen Rendle, Zeno Gantner, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2011. Fast context-aware recommendations with factorization machines. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval (SIGIR '11). ACM, New York, NY, USA, 635-644. \n\\endverbatim\n - \\ref SGD \"Stochastic gradient descent (SGD)\"\n\\verbatim\n Matrix Factorization Techniques for Recommender Systems Yehuda Koren, Robert Bell, Chris Volinsky In IEEE Computer, Vol. 42, No. 8. (07 August 2009), pp. 30-37. \nTakács, G, Pilászy, I., Németh, B. and Tikk, D. (2009). Scalable Collaborative Filtering Approaches for Large Recommender Systems. Journal of Machine Learning Research, 10, 623-656.\n\\endverbatim\n - \\ref BIAS_SGD \"Bias stochastic gradient descnet (Bias-SGD)\"\n\\verbatim\nY. Koren. Factorization Meets the Neighborhood: a Multifaceted Collaborative Filtering Model. In ACM KDD 2008. Equation (5).\n\\endverbatim\n - \\ref SVD_PLUS_PLUS \"SVD++\"\n\\verbatim\nY. Koren. Factorization Meets the Neighborhood: a Multifaceted Collaborative Filtering Model. In ACM KDD 2008. \n\\endverbatim\n - \\ref WALS \"Weighted-ALS\"\n\\verbatim\nCollaborative Filtering for Implicit Feedback Datasets Hu, Y.; Koren, Y.; Volinsky, C. IEEE International Conference on Data Mining (ICDM 2008), IEEE (2008). \nD. Needell, J. A. Tropp CoSaMP: Iterative signal recovery from incomplete and inaccurate samples Applied and Computational Harmonic Analysis, Vol. 26, No. 3. (17 Apr 2008), pp. 301-321. \n\\endverbatim\n - \\ref SALS \"Sparse-ALS\"\n\\verbatim\nXi Chen, Yanjun Qi, Bing Bai, Qihang Lin and Jaime Carbonell. Sparse Latent Semantic Analysis. In SIAM International Conference on Data Mining (SDM), 2011. \n\\endverbatim  \nIn the future we hope to implement to rest of V1 algorithms, like NMF, NMF, BPTF, etc.\n - \\ref NMF \"Non-negative matrix factorization\"\n\\verbatim\nNMF Lee, D..D., and Seung, H.S., (2001), 'Algorithms for Non-negative Matrix\nFactorization', Adv. Neural Info. Proc. Syst. 13, 556-562.\n\\endverbatim\n - \\ref SVD \"Restarted lanczos algorithm\"\n\\verbatim\nV. Hern´andez, J. E. Rom´an and A. Tom´as. STR-8: Restarted Lanczos Bidiagonalization for the SVD in SLEPc. \n\\endverbatim\n\n- \\ref ADPREDICTOR \"Adpredictor\"\n\\verbatim\nGraepel, Thore, et al. \"Web-scale bayesian click-through rate prediction for sponsored search advertising in microsoft's bing search engine.\" Proceedings of the 27th International Conference on Machine Learning (ICML-10). 2010.\n\\endverbatim\n\n\\section Input Input\n\nThe input to GraphLab v2.1 collaborative filtering toolkit should be prepared inside a directory. All files in the directory will be read in parallel by GraphLab. Each file has the following text format:\n\\verbatim\n[ user ] [ item ] [ rating] \\n\n\\endverbatim\nNamely, each row holds one rating. user and item are unsigned integers, and the rating is a double value.  user and item does not have to be consecutive integers. Here are some allowed inputs:\n\\verbatim\n1000 2 5.0\n3 7 12.0\n6 2 2.1\n\\endverbatim\n\nThere are three types of input files read from your input directory path:\n*.predict - test file\n*.validate - validation file\nall other files - are training files.\n\nTraining files, are the historic recommendations that the linear model is build from. Validation files, are historic recommendations put aside, not used for training, but for the validation of the model. Test files are user/item pairs to compute recommendations as learned by the trained model.\n\nNote: for weighted-ALS, the input has the follwoing format:\n\\verbatim\n[user] [item] [weight] [rating] \\n\n\\endverbatim\nSince each rating has its associated weight.\n\n\\section RATINGS Computing ratings (test predictions)\n\nOptionally, you can compute ratings for user/item pairs using the computed linear model. The prediction output Filename is specified by: --predictions=filename . If the --prediction command line is not used, the output is not saved. Additionally you need to prepare a file named somefilename.predict inside of your training folder, with user item pairs in the following format: \n\\verbatim\n[user] [item]\\n\n\\endverbatim\nThe program computes the prediction based on the computed linear models for every user item pair.\nThe output format for the prediction is:\n\\verbatim\n[user] [item] [rating]\\n\n\\endverbatim\n\n\\section Output Output\nThe linear model is saved to the files: filename.U_X_of_Y and filename.V_X_of_Y\nWhre X is the part number and Y is the total number of parts. On default there are two parts.\nU and V' are the matrices which their product U*V' approximates the matrix A.\nOuptut format for the linear model matrix U is:\n\\verbatim\nuser factor1 factor2 .. factorN \\n\n\\endverbatim\n\nOuptut format for the linear model matrix V is:\n\\verbatim\nitem factor1 factor2 .. factorN \\n\n\\endverbatim\n\nIt is possible to merge the files together using the cat command. For example:\n\\verbatim\n>  cat filename.U_2_of_2 >> filename.U_1_of_2\n\\endverbatim\nwill append the contents of part two of the matrix U into part one.\n\nIt is further possible to sort the output using the user or item id using the sort command.\n\\verbatim\n>  sort -g -k 1,1 filename.U_1_of_2 > filename.U.sorted\n\\endverbatim\nThat way each row will contain one user (for matrix U) or one item (for matrix V) feature vectors in sorted order.\n\n\nFor bias-SGD and SVD++, two additional files are created: filename.bias.U and filename.bias.V\nwith the biasses.\n\nNOTE: Output files are NOT sorted. Use the sort Linux command to sort them by user / item id. \nFor example:\n\\verbatim\nsort -g -k 1,1 filename.U > filename.U.sorted # sorts user features, by the first column (user id)\n\\endverbatim\n\n\\section ALS ALS\nALS (Alternating least squares)\n\nPros: Simple to use, not many command line arguments\n\nCons: intermediate accuracy, higher computational overhead\n\nALS is a simple yet powerful algorithm. In this model the prediction is computed as:\n   r_ui = p_u * q_i\nWhere r_ui is a scalar rating of user u to item i, and p_u is the user feature vector of size D, q_i is the item feature vector of size D and the product is a vector product. \nThe output of ALS is two matrices: filename.U and filename.V. The matrix U holds the user feature vectors in each row. (Each vector has exactly D columns). The matrix V holds the feature vectors for each time (Each vector has again exactly D columns). In linear algebra notation the rating matrix R ~ UV\n\n\nBelow are ALS related command line options:\n\\verbatim\n--D=XX\tSet D the feature vector width. High width results in higher accuracy but slower execution time. Typical values are 20 -  100.\n--lambda=XX\tSet regularization. Regularization helps to prevent overfitting. \n--max_iter=XX The number of iterations.\n--maxval=XX\tMaximum allowed rating\n--minval=XX\tMin allowed rating\n--predictions=XX\tFile name to write prediction to. Note that you will need a user/item pair input file named something.predict to enable predictions (see section: ratings).\n\\endverbatim\n\nAnd here is an exmaple ALS run:\n\\li Download the files: smallnetflix_mm.train and smallnetflix_mm.validate and save them inside a directory called smallnetflix/.\n\\li Run:\n\\verbatim\nbickson@thrust:~/graphlab2.1/graphlabapi/debug/toolkits/collaborative_filtering$ ./als smallnetflix/ --max_iter=5 --lambda=0.065 --ncpus=8 --minval=1 --maxval=5\nTCP Communication layer constructed.\nLoading graph.\nINFO:     distributed_graph.hpp(load_from_posixfs:1743): Loading graph from file: smallnetflix/smallnetflix_mm.train\nINFO:     distributed_graph.hpp(load_from_posixfs:1743): Loading graph from file: smallnetflix/smallnetflix_mm.validate\nLoading graph. Finished in 20.4732\nFinalizing graph.\nINFO:     distributed_ingress_base.hpp(finalize:165): Finalizing Graph...\nINFO:     distributed_ingress_base.hpp(exchange_global_info:489): Graph info: \n\t nverts: 97266\n\t nedges: 3843340\n\t nreplicas: 97266\n\t replication factor: 1\nFinalizing graph. Finished in 4.72823\n========== Graph statistics on proc 0 ===============\n Num vertices: 97266\n Num edges: 3843340\n Num replica: 97266\n Replica to vertex ratio: 1\n --------------------------------------------\n Num local own vertices: 97266\n Num local vertices: 97266\n Replica to own ratio: 1\n Num local edges: 3843340\n Edge balance ratio: 1\nCreating engine\nRunning ALS\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 93705\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n17.8\t2.99666\t5.76023\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 1\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n33\t2.07403\t3.99939\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 2\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 93702\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n50.9\t0.896588\t1.76014\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 3\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n66.1\t0.755783\t1.45845\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 4\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 93547\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n83.9\t0.697824\t1.35614\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 5\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n99.1\t0.677978\t1.33864\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 6\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 91661\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n116.9\t0.666121\t1.3114\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 7\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3560\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n132\t0.657644\t1.31769\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 8\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 90443\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n149.7\t0.651672\t1.3017\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 9\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\n\n...\n\\endverbatim\n\\section CCD_PLUS_PLUS \"CCD++\"\nALS (Alternating least squares) with parallel coordinate descent\n\nPros: Simple to use, not many command line arguments, faster than ALS\n\nCons: less accurate than ALS\n\nCCD++ is a recent improvement of ALS which uses parallel coordinate descent using a single dimension at a time\nand thus avoids the need of costly linear algebra operations (especially when the dimension of the feature vector is high).\nIn this model the prediction is computed as:\n   r_ui = p_u * q_i\nWhere r_ui is a scalar rating of user u to item i, and p_u is the user feature vector of size D, q_i is the item feature vector of size D and the product is a vector product. \nThe output of ALS is two matrices: filename.U and filename.V. The matrix U holds the user feature vectors in each row. (Each vector has exactly D columns). The matrix V holds the feature vectors for each time (Each vector has again exactly D columns). In linear algebra notation the rating matrix R ~ UV\n\n\nBelow are CCD++ related command line options:\n\\verbatim\n--D=XX\tSet D the feature vector width. High width results in higher accuracy but slower execution time. Typical values are 20 -  100.\n--lambda=XX\tSet regularization. Regularization helps to prevent overfitting. \n--max_iter=XX The number of iterations.\n--maxval=XX\tMaximum allowed rating\n--minval=XX\tMin allowed rating\n--predictions=XX\tFile name to write prediction to. Note that you will need a user/item pair input file named something.predict to enable predictions (see section: ratings).\n\\endverbatim\n\n\n\\section SGD \"Stochastic gradient descent (SGD)\"\n\nPros: fast method\nCons: need to tune step size, more iterations are needed relative to ALS.\n\nSGD is a simple gradient descent algorithm. Prediction in SGD is done as in ALS:\n   r_ui = p_u * q_i\nWhere r_ui is a scalar rating of user u to item i, and p_u is the user feature vector of size D, q_i is the item feature vector of size D and the product is a vector product. \nThe output of ALS is two matrices: filename.U and filename.V. The matrix U holds the user feature vectors in each row. (Each vector has exactly D columns). The matrix V holds the feature vectors for each time (Each vector has again exactly D columns). In linear algebra notation the rating matrix R ~ UV\n\n\\verbatim\n--gamma=XX\tGradient descent step size\n--lambda=XX\tGradient descent regularization\n--step_dec=XX\tMultiplicative step decrease. Should be between 0.1 to 1. Default is 0.9.\n--D=X\t\tFeature vector width. Common values are 20 - 150.\n--max_iter=XX\tMax number of iterations\n--maxval=XX\tMaximum allowed rating\n--minval=XX\tMin allowed rating\n--predictions=XX\tFile name to write prediction to. Note that you will need a user/item pair input file named something.predict to enable predictions (see section: ratings).\n--tol=XX\tStop computation when absolute error of prediction is less than tolerance. Default is 1e-3.\n\\endverbatim\n\nHere is an example SGD run on small Netflix data:\n\\li Download the files: smallnetflix_mm.train and smallnetflix_mm.validate and save them inside a directory called smallnetflix/.\n\\li Run:\n\\verbatim\nbickson@thrust:~/graphlab2.1/graphlabapi/debug/toolkits/collaborative_filtering$ ./sgd smallnetflix  --ncpus=8 --prediction=out --max_iter=10  --gamma=1e-3 --lambda=5e-4 --step_dec=0.9999 --minval=1 --maxval=5\nTCP Communication layer constructed.\nLoading graph.\nINFO:     distributed_graph.hpp(load_from_posixfs:1743): Loading graph from file: smallnetflix/smallnetflix_mm.train\nINFO:     distributed_graph.hpp(load_from_posixfs:1743): Loading graph from file: smallnetflix/smallnetflix_mm.validate\nLoading graph. Finished in 8.13307\nFinalizing graph.\nINFO:     distributed_ingress_base.hpp(finalize:165): Finalizing Graph...\nINFO:     distributed_ingress_base.hpp(exchange_global_info:489): Graph info: \n\t nverts: 97266\n\t nedges: 3843340\n\t nreplicas: 97266\n\t replication factor: 1\nFinalizing graph. Finished in 4.71821\n========== Graph statistics on proc 0 ===============\n Num vertices: 97266\n Num edges: 3843340\n Num replica: 97266\n Replica to vertex ratio: 1\n --------------------------------------------\n Num local own vertices: 97266\n Num local vertices: 97266\n Replica to own ratio: 1\n Num local edges: 3843340\n Edge balance ratio: 1\nCreating engine\nWARNING:  distributed_aggregator.hpp(test_vertex_mapper_type:344): \nVertex Map Function does not pass strict runtime type checks. \nFunction prototype should be \n\t ReductionType f(icontext_type&, const vertex_type&)\nIf you are not intentionally violating the abstraction, we recommend fixing your function for safety reasons\nRunning SGD\n(C) Code by Danny Bickson, CMU \nPlease send bug reports to danny.bickson@gmail.com\nTime   Training    Validation\n       RMSE        RMSE \nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 93705\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n6.2\t3.36002\t3.49087\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 1\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n15.2\t2.08215\t2.49183\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 3\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n24.6\t1.91162\t2.05136\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 5\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n33.7\t1.77294\t1.80171\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 7\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n42.6\t1.74585\t1.68424\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 9\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n51.7\t1.63199\t1.56293\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 11\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n60.7\t1.58655\t1.50337\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 13\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n69.8\t1.48326\t1.4251\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 15\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n78.8\t1.43588\t1.38834\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 17\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n87.8\t1.34333\t1.33439\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 19\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n----------------------------------------------------------\nFinal Runtime (seconds):   90.064\nUpdates executed: 972660\nUpdate Rate (updates/second): 10799.7\nFinal error: \n91.7\t2.04374\t1.87504\nSaving predictions\n\\endverbatim\n\n\n\\section BIAS_SGD BIAS-SGD\n\nPros: fast method\nCons: need to tune step size\n\nBias-SGD is a simple gradient descent algorithm, where besides of the feature vector we also compute item and user biases (how much their average rating differs from the global average).\nPrediction in bias-SGD is done as follows:\n\nr_ui = global_mean_rating + b_u + b_i + p_u * q_i\n\nWhere global_mean_rating is the global mean rating, b_u is the bias of user u, b_i is the bias of item i and p_u and q_i are feature vectors as in ALS. You can read more about bias-SGD in reference [N]. \n\nThe output of bias-SGD consists of two matrices: filename.U and filename.V. The matrix U holds the user feature vectors in each row. (Each vector has exactly D columns). The matrix V holds the feature vectors for each time (Each vector has again exactly D columns). Additionally, the output consists of two vectors: bias for each user, bias for each item. Last, the global mean rating is also given as output.\n\n\\verbatim\n--gamma=XX\tGradient descent step size\n--lambda=XX\tGradient descent regularization\n--step_dec=XX\tMultiplicative step decrease. Should be between 0.1 to 1. Default is 0.9\n--D=X\t\tFeature vector width. Common values are 20 - 150.\n--max_iter=XX\tMax number of iterations\n--maxval=XX\tMaximum allowed rating\n--minval=XX\tMin allowed rating\n--predictions=XX\tFile name to write prediction to. Note that you will need a user/item pair input file named something.predict to enable predictions (see section: ratings).\n\\endverbatim\n\nExample for running bias-SGD\n\\li Download the files: smallnetflix_mm.train and smallnetflix_mm.validate and save them inside a directory called smallnetflix/.\n\\li Run:\n\n\\verbatim\nibickson@thrust:~/graphlab2.1/graphlabapi/debug/toolkits/collaborative_filtering$ ./biassgd smallnetflix  --ncpus=8 --prediction=out --max_iter=10\nTCP Communication layer constructed.\nLoading graph.\nINFO:     distributed_graph.hpp(load_from_posixfs:1743): Loading graph from file: smallnetflix/smallnetflix_mm.train\nINFO:     distributed_graph.hpp(load_from_posixfs:1743): Loading graph from file: smallnetflix/smallnetflix_mm.validate\nLoading graph. Finished in 7.59514\nFinalizing graph.\nINFO:     distributed_ingress_base.hpp(finalize:165): Finalizing Graph...\nINFO:     distributed_ingress_base.hpp(exchange_global_info:489): Graph info: \n\t nverts: 97266\n\t nedges: 3843340\n\t nreplicas: 97266\n\t replication factor: 1\nFinalizing graph. Finished in 4.93781\n========== Graph statistics on proc 0 ===============\n Num vertices: 97266\n Num edges: 3843340\n Num replica: 97266\n Replica to vertex ratio: 1\n --------------------------------------------\n Num local own vertices: 97266\n Num local vertices: 97266\n Replica to own ratio: 1\n Num local edges: 3843340\n Edge balance ratio: 1\nCreating engine\nGlobal mean is: 3.5992\nWARNING:  distributed_aggregator.hpp(test_vertex_mapper_type:344): \nVertex Map Function does not pass strict runtime type checks. \nFunction prototype should be \n\t ReductionType f(icontext_type&, const vertex_type&)\nIf you are not intentionally violating the abstraction, we recommend fixing your function for safety reasons\nRunning Bias-SGD\n(C) Code by Danny Bickson, CMU \nPlease send bug reports to danny.bickson@gmail.com\nTime   Training    Validation\n       RMSE        RMSE \nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 93705\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n7.1     1.13985    1.15723\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 1\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n17.5    1.03638    1.07782\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 3\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n27.9    1.00508    1.05466\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 5\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n38.3    0.987878    1.04218\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 7\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n48.8    0.976675    1.03377\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 9\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n59.1    0.968729    1.02827\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 11\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n69.5    0.962782    1.0236\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 13\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n80      0.958178    1.02056\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 15\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n90.3    0.95442    1.01745\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 17\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n100.6    0.95139    1.01548\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 19\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 3561\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n----------------------------------------------------------\nFinal Runtime (seconds):   102.971\nUpdates executed: 972660\nUpdate Rate (updates/second): 9445.96\nFinal error: \n104.7    1.04346    1.13552\nSaving predictions\n\\endverbatim\n\n\n\\section SVD_PLUS_PLUS SVD++\nPros: more accurate method than SGD once tuned, relatively fast method\nCons: a lot of parameters for tuning, immune to numerical errors when parameters are out of scope.\n\nKoren SVD++ is an algorithm which is slightly more fancy than bias-SGD and give somewhat better prediction results.\n\nBasic configuration --svdpp_step_dec=XX Multiplicative step decrement (between 0.1 to 1). Default is 0.9\n\\verbatim\n--item_bias_step=XX Item bias step size\n--item_bias_reg=XX  Item bias regularization\n--user_bias_step=XX  User bias step size\n--user_bias_reg=XX User bias regularization\n--user_fctr_step=XX  User factor step size\n--user_fctr_reg=XX User factor regularization\n--item_fctr_step=XX Item factor step size\n--item_fctr_reg=XX  Item factor regularization\n--item_fctr2_step=XX  Item factor2 step size\n--item_fctr2_reg=XX Item factor2 regularization\n--D=X Feature vector width. Common values are 20 - 150.\n--step_dec=XX\tMultiplicative step decrease. Should be between 0.1 to 1. Default is 0.9\n--max_iter=XX\tMax number of iterations\n--maxval=XX\tMaximum allowed rating\n--minval=XX\tMin allowed rating\n--predictions=XX\tFile name to write prediction to. Note that you will need a user/item pair input file named something.predict to enable predictions (see section: ratings).\n\\endverbatim\n\n Prediction in Koren’s SVD++ algorithm is computed as follows:\n\\verbatim\nr_ui = global_mean_rating + b_u + b_i + q_u * ( p_i + w_i )\n\\endverbatim\nWhere r_ui is the scalar rating for user u to item i, global_mean_rating is the global mean rating, b_u is a scalar bias for user u, b_i is a scalar bias for item i, q_u is a feature vectors of length D for user u, p_i is a feature vector of length D for item i, and w_i is an additional feature vector of length D (the weight). The product is a vector product. \n\nThe output of Koren’s SVD++ is 5 output files:\n\\verbatim\nGlobal mean ratings - include the scalar global mean rating.\nuser_bias  - includes a vector with bias for each user\nmovie_bias - includes a vector with bias for each movie\nmatrix U - includes in each row the feature vector q_u of size D.\nmatrix V - includes in each row the sum of feature vectors p_i + w_i of size D.\n\\endverbatim\n\n\\section WALS Weighted-ALS\n\nPros: Simple to use, allows iteration of weights which can be thought of confidence in the recommendation\nCons: intermediate accuracy, higher computational overhead\n\nWeighted-ALS is a simple yet powerful algorithm. In this model the prediction is computed as:\n   r_ui = p_u * q_i\nWhere r_ui is a scalar rating of user u to item i, and p_u is the user feature vector of size D, q_i is the item feature vector of size D and the product is a vector product. \nThe output of ALS is two matrices: filename.U and filename.V. The matrix U holds the user feature vectors in each row. (Each vector has exactly D columns). The matrix V holds the feature vectors for each time (Each vector has again exactly D columns). In linear algebra notation the rating matrix R ~ UV\n\n\nBelow are WALS related command line options:\n\\verbatim\n--D=XX\tSet D the feature vector width. High width results in higher accuracy but slower execution time. Typical values are 20 -  100.\n--lambda=XX\tSet regularization. Regularization helps to prevent overfitting. \n--max_iter=XX The number of iterations.\n--maxval=XX\tMaximum allowed rating\n--minval=XX\tMin allowed rating\n--predictions=XX\tFile name to write prediction to. Note that you will need a user/item pair input file named something.predict to enable predictions (see section: ratings).\n\n\\endverbatim\n\nAnd here is an exmaple WALS run:\n\\li Download the files: time_smallnetflix.train and time_smallnetflix.validate and save them inside a directory called timenetflix/.\n\\li Run:\n\\verbatim\nbickson@thrust:~/graphlab2.1/graphlabapi/debug/toolkits/collaborative_filtering$ ./wals smallnetflix/ --max_iter=5 --lambda=0.065 --ncpus=8\nbickson@thrust:~/graphlab2.1/graphlabapi/debug/toolkits/collaborative_filtering$ ./wals timenetflix/\nTCP Communication layer constructed.\nLoading graph.\nINFO:     distributed_graph.hpp(load_from_posixfs:1823): Loading graph from file: timenetflix/time_smallnetflix.train\nINFO:     distributed_graph.hpp(load_from_posixfs:1823): Loading graph from file: timenetflix/time_smallnetflix.validate\nLoading graph. Finished in 8.69352\nFinalizing graph.\nINFO:     distributed_ingress_base.hpp(finalize:166): Finalizing Graph...\nINFO:     distributed_ingress_base.hpp(exchange_global_info:493): Graph info: \n   nverts: 97266\n   nedges: 3843340\n   nreplicas: 97266\n   replication factor: 1\nFinalizing graph. Finished in 5.2593\n========== Graph statistics on proc 0 ===============\n Num vertices: 97266\n Num edges: 3843340\n Num replica: 97266\n Replica to vertex ratio: 1\n --------------------------------------------\n Num local own vertices: 97266\n Num local vertices: 97266\n Replica to own ratio: 1\n Num local edges: 3843340\n Edge balance ratio: 1\nCreating engine\nRunning Weighted-ALS\nINFO:     synchronous_engine.hpp(start:1260): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1309):   Active vertices: 93705\nINFO:     synchronous_engine.hpp(start:1358):    Running Aggregators\n64.8  24.3996 67.9795\nINFO:     synchronous_engine.hpp(start:1260): 0: Starting iteration: 1\nINFO:     synchronous_engine.hpp(start:1309):   Active vertices: 3561\nINFO:     synchronous_engine.hpp(start:1358):    Running Aggregators\n119.2 18.866  75.5397\nINFO:     synchronous_engine.hpp(start:1260): 0: Starting iteration: 2\nINFO:     synchronous_engine.hpp(start:1309):   Active vertices: 93704\nINFO:     synchronous_engine.hpp(start:1358):    Running Aggregators\n184 10.6131 43.162\nINFO:     synchronous_engine.hpp(start:1260): 0: Starting iteration: 3\nINFO:     synchronous_engine.hpp(start:1309):   Active vertices: 3561\nINFO:     synchronous_engine.hpp(start:1358):    Running Aggregators\n238.5 8.49288 28.1034\nINFO:     synchronous_engine.hpp(start:1260): 0: Starting iteration: 4\nINFO:     synchronous_engine.hpp(start:1309):   Active vertices: 93702\nINFO:     synchronous_engine.hpp(start:1358):    Running Aggregators\n303 7.24041 22.2866\nINFO:     synchronous_engine.hpp(start:1260): 0: Starting iteration: 5\nINFO:     synchronous_engine.hpp(start:1309):   Active vertices: 3561\nINFO:     synchronous_engine.hpp(start:1358):    Running Aggregators\n357.6 6.74309 19.951\nINFO:     synchronous_engine.hpp(start:1260): 0: Starting iteration: 6\nINFO:     synchronous_engine.hpp(start:1309):   Active vertices: 93585\nINFO:     synchronous_engine.hpp(start:1358):    Running Aggregators\n422.1 6.42464 19.061\nINFO:     synchronous_engine.hpp(start:1260): 0: Starting iteration: 7\nINFO:     synchronous_engine.hpp(start:1309):   Active vertices: 3561\nINFO:     synchronous_engine.hpp(start:1358):    Running Aggregators\n476.5 6.25972 17.2991\nINFO:     synchronous_engine.hpp(start:1260): 0: Starting iteration: 8\nINFO:     synchronous_engine.hpp(start:1309):   Active vertices: 93164\nINFO:     synchronous_engine.hpp(start:1358):    Running Aggregators\n542.3 6.13234 16.8022\nINFO:     synchronous_engine.hpp(start:1260): 0: Starting iteration: 9\nINFO:     synchronous_engine.hpp(start:1309):   Active vertices: 3561\n\n...\n\\endverbatim\n\n\\section SALS Sparse-ALS \nPros: Generate sparse factor matrices, that can be clustered into similar user/item groups\nCons: less accurate linear model because of the sparsification step\n\nThis algorithm is based on ALS, but an additional sparsifying step is performed on either the user feature vectors, the item feature vectors or both. This algorithm is useful for spectral clustering: first the rating matrix is factorized into a product of one or two sparse matrices, and then clustering can be computed on the feature matrices to detect similar users or items. \n\nThe underlying algorithm which is used for sparsifying is CoSaMP. See reference  on the top of this page. \n\nBelow are sparse-ALS related command line options:\n\\verbatim\n--user_sparsity=XX  A number between 0.5 to 1 which defines how sparse is the resulting user feature factor matrix\n--movie_sparsity=XX A number between 0.5 to 1 which defines how sparse is the resulting movie feature factor matrix\n--algorithm=XX An integer between 1 to 3 which defines the run mode.\n1 = SPARSE_USR_FACTOR\n2 = SPARSE_ITM_FACTOR\n3 = SPARSE_BOTH_FACTORS\n\\endverbatim\nPrediction in sparse-ALS is computed like in ALS.\n\n\\section NMF \"Non-negative matrix factorization\"\n\nNon-negative matrix factorization (NMF) is based on Lee and Seung [reference H]. \nPrediction is computed like in ALS:\n    r_ui = p_u * q_i\n\nNamely the scalar prediction r of user u is composed of the vector product of the user feature vector p_u (of size D), with the item feature vector q_i (of size D). The only difference is that both p_u and q_i have all nonnegative values.\nThe output of NMF is two matrices: filename.U and filename.V. The matrix U holds the user feature vectors in each row. (Each vector has exactly D columns). The matrix V holds the feature vectors for each time (Each vector has again exactly D columns). In linear algebra notation the rating matrix R ~ UV, U>=0, V>=0.\n\n\\subsection \"NMF cost function and properties\"\n\nUnlike many of the other methods who is Euclidean distance, NMF cost function is:\n      KL( UV’ || A) \nNamely the KL divergence between the approximating product UV’  and the original matrix A.\nThe objective is not computed in GraphLab, but you can easily compute it in Matlab if needed. \n\nNMF is a gradient descent type algorithm which is supposed to always converge. However it may converge to a local minima. The algorithm starts from a random solution and that is why different runs may converge to different solution. For debugging, if you are interested in verifying that multiple runs converge to the same point, use the flag --debug=true when running.\n\n\\section SVD Restarted Lanczos Iteration (SVD)\nSVD is implemented using the restarted lanczos algorithm.\nThe input is a sparse matrix market format input file.\nThe output are 3 files: one file containing the singular values, and two dense matrix market format files containing the matrices U and V.\n\nNote: for larger models, it is advised to use svd_onesided since it significantly saved memory.\n\nHere is an example Matrix Market input file for the matrix A2:\n\n<235|0>bickson@bigbro6:~/ygraphlab/graphlabapi/debug/toolkits/parsers$ \ncat A2\n\\verbatim\n0 0  0.8147236863931789\n0 1 0.9133758561390194\n0 2  0.2784982188670484\n0 3  0.9648885351992765\n1 0  0.9057919370756192\n1 1  0.6323592462254095\n1 2  0.5468815192049838\n1 3  0.1576130816775483\n2 0  0.1269868162935061\n2 1  0.09754040499940952\n2 2  0.9575068354342976\n2 3  0.9705927817606157\n\\endverbatim\n\nCeate a directory named A2, and inside it put the file A2.\n\n\nHere is an for running SVD (using one mpi node, one core)\n\\verbatim\nbickson@thrust:~/graphlab2.1/graphlabapi/debug/toolkits/collaborative_filtering$ ./svd A2 --rows=3 --cols=4 --nsv=3 --nv=4 --max_iter=3 --quiet=1 --save_vectors=1 --predictions=out\nTCP Communication layer constructed.\nLoading graph.\nLoading graph. Finished in 0.004996\nFinalizing graph.\nFinalizing graph. Finished in 0.374135\n========== Graph statistics on proc 0 ===============\n Num vertices: 7\n Num edges: 12\n Num replica: 7\n Replica to vertex ratio: 1\n --------------------------------------------\n Num local own vertices: 7\n Num local vertices: 7\n Replica to own ratio: 1\n Num local edges: 12\n Edge balance ratio: 1\nCreating engine\nRunning SVD (gklanczos)\n(C) Code by Danny Bickson, CMU \nPlease send bug reports to danny.bickson@gmail.com\nset status to tol\n Number of computed signular values 4\nSingular value 0        2.16097 Error estimate:   1.05039e-15\nSingular value 1        0.97902 Error estimate:   1.32491e-15\nSingular value 2       0.554159 Error estimate:   9.92283e-16\nSingular value 3    1.05388e-64 Error estimate:   3.42194e-16\n----------------------------------------------------------\nFinal Runtime (seconds):   0.54851\nUpdates executed: 59\nUpdate Rate (updates/second): 107.564\n\\endverbatim\n\nFor running with multiple mpi nodes run:\n\\verbatim \nmpiexec -n XX ./svd [ rest of the command line aguments ]\n\\endverbatim\n\n\\subsection Command line arguments\n\n\\verbatim\n--training  Input file directory.\n--nv  Buffer size of vectors. Typically the buffer size should be greater than the number of singular values you look for. When nv is higher, accuracy will be higher, but running time slower.\n--nsv Number of singular values requested. Should be typically less than --nv\n--ortho_repeats Number of repeats on the orthogonalization step. Default is 1 (no repeats). Increase this number for higher accuracy but slower execution. Maximal allowed values is 3.\n--max_iter  Number of allowed restarts. The minimum is 2= no restart. When max_iter is higher, the result will be more accurate, but the running slower. \n--save_vectors=true Save the factorized matrices U and V to file. \n--predictions=XX\tFile name to write prediction to. Note that you will need a user/item pair input file named something.predict to enable predictions (see section: test ratings).\n--tol Convergence threshold. For large matrices set this number set this number higher (for example 1e-1, while for small matrices you can set it to 1e-16). As smaller the convergence threshold execution is slower.\n\\endverbatim\n--input_file_offset - for 1 based array index, use 1, for 0 based array index use 0. (Namely, array first index starts from 1 or 0).\n\nNote: for improving accuracy tol should be reduced. max_iter and nv should be increased.\n\n\\subsection SVD0 \"SVD Output\"\nOn default, the singular values will be written to an output file. When using --save_vectors=1 the singular vectors of the matrices U and V will be written into file as well.\nHere is an example of the output files created by the A2 example:\n\\verbatim\n-rw-r--r--   1 bickson  staff       136 Nov 17 14:19 outsingular_values\n-rw-r--r--   1 bickson  staff       353 Nov 17 14:19 out.V_1_of_1\n-rw-r--r--   1 bickson  staff       244 Nov 17 14:19 out.U_1_of_1\n\\endverbatim\n
\nThe singular_values file has a straightforward format:\n\\verbatim\n%%GraphLab SVD Solver library. This file contains the singular values.\n2.160971174556\n0.9790200922132\n0.5541592674291\n1.69699593375e-64\n\\endverbatim\n\nNow let's compate GraphLba's output to Matlab execution:\n\\verbatim\n>> A2\nans =\n\n    0.8147    0.9134    0.2785    0.9649\n    0.9058    0.6324    0.5469    0.1576\n    0.1270    0.0975    0.9575    0.9706\n\n>>[u,d,v] = svd(A2)\n\nu =\n\n   -0.7019    0.2772    0.6561\n   -0.5018    0.4613   -0.7317\n   -0.5055   -0.8428   -0.1847\n\n\nd =\n\n    2.1610         0         0\n         0    0.9790         0\n         0         0    0.5542\n\n\nv =\n\n   -0.5047    0.5481   -0.2737\n   -0.4663    0.4726    0.2139\n   -0.4414   -0.4878   -0.7115\n   -0.5770   -0.4882    0.6108\n\\endverbatim\n\nAnd here is GraphLab output:\n\\verbatim\n#> cat out.U_1_of_1\n1 -0.70192004675202879 -0.27716662376092144 -0.6561150132717597 -5.4738221262688167e-48 \n2 -0.50180137502007927 -0.46130533561664677 0.73170538289640219 0 \n3 -0.50547366696553819 0.84283809240244056 0.18471686983009383 -1.3684555315672042e-48 \n\n#> cat out.V_1_of_1\n1 -0.50467448230910661 -0.54813128066725625 0.27370653410216472 -1.0321593257419978e-64 \n2 -0.46633647481132878 -0.47257174529014068 -0.21394855212543565 1.2159686437710111e-64 \n3 -0.44142369037529217 0.48778719783126245 0.71152275587696578 4.1893513895261429e-65 \n4 -0.57704339935747928 0.4881513027120063 -0.61077501430760717 -4.0044466804328635e-65 \n\n#> cat out.singular_values\n%%GraphLab SVD Solver library. This file contains the singular values.\n2.160971174556\n0.9790200922132\n0.5541592674291\n1.69699593375e-64\n\\endverbatim\n\n\\subsection SVD1 \"Understanding the error measure\"\n Following Slepc, the error measure is computed by a combination of:\nsqrt( ||Av_i - sigma(i) u_i ||_2^2 + ||A^Tu_i - sigma(i) V_i ||_2^2 ) / sigma(i)\n\n\nNamely, the deviation of the approximation sigma(i) u_i  from Av_i , and vice versa.\n\n\\subsection SVD2 \"Scalability\"\n Currently the code was tested with up to 3.5 billion non-zeros on a 24 core machine. Each Lanczos iteration takes about 30 seconds. \n\n\\subsection SVD3 \"Difference to Mahout\"\n Mahout SVD solver is implemented using the same Lanczos algorithm. However, there are several differences\n1) In Mahout there are no restarts, so quality of the solution deteriorates very rapidly, after 5-10 iterations the solution is no longer accurate. Running without restarts can be done using our solution with the --max_iter=2 flag.\n2) In Mahout there is a single orthonornalization step in each iteration while in our implementation there are two (after computation of u_i and after v_i ).\n3) In Mahout there is no error estimation while we provide for each singular value the approximated error.\n4) Our solution is typically x100 times faster than Mahout.\n\n\\section ADPREDICTOR Adpredictor\nIn a nutshell, AdPredictor computes a linear regression model with probit link function.\nThe input to the algorithm are observations of the type\n\\verbatim\n-1 3:1 4:1 6:1 9:1\n1 4:1 5:1 18:1 19:1\n...\n\\endverbatim\n\nwhere the first field -1 is the action (did not click) or 1 (clicked). Next there are pairs of binary features.\nThe output of the algorithm are weights for each feature. When a new ad comes in, we should simply sum up the weights for the matching features. If the weights are smaller than zero then the prediction is -1 and vice versa.\n\nAdpredictor takes file in libsvm format. You should prepare a sub folder with the training file and validation (file needs to end with .validate).\n\nYou can run adpredictor using the command:\n\\verbatim\n./adpredictor --matrix=folder/ --max_iter=10 --beta=1\n\\endverbatim\n\n\\section Implicit \"Implicit Ratings\"\nImplicit rating handles the case where we have only positive examples (for example when a user bought a certain product) but we never have indication when a user DID NOT buy another product. The following paper \n\\verbatim\nPan, Yunhong Zhou, Bin Cao, Nathan N. Liu, Rajan Lukose, Martin Scholz, and Qiang Yang. 2008. One-Class Collaborative Filtering. In Proceedings of the 2008 Eighth IEEE International Conference on Data Mining (ICDM '08). IEEE Computer Society, Washington, DC, USA, 502-511. \n\\endverbatim\nproposes to add negative examples at random for unobserved user/item pairs. Implicit rating is implemented in the collaborative filtering library and can be used with any of the algorithms explained above. \n\n\\verbatim\n--implicitratingtype=1  Adds implicit ratings at random\n--implicitratingpercentage  A number between 1e-8 to 0.8  which determines what is the percentage of edges to add to the sparse model. 0 means none while 1 means fully dense model. \n--implicitratingvalue   The value of the rating added. On default it is zero, but you can change it. \n--implicitratingweight  Weight of the implicit rating (for WALS) OR\nTime of the explicit rating (for tensor algorithms)\n--users - the number of users. Note that users have to have consecutive ids between 1 and users.\n--items - the number of items. Note that items have to have consecutive ids between 1 and items.\n\\endverbatim\n\nExample for adding implicit ratings:\n\\verbatim\n./als --matrix=smallnetflix/ --users=95526 --items=3561 --implicitratingtype=1 --implicitratingpercentage=0.01\nTCP Communication layer constructed.\nLoading graph.\nINFO:     distributed_graph.hpp(load_from_posixfs:1823): Loading graph from file: smallnetflix/smallnetflix_mm.train\nINFO:     distributed_graph.hpp(load_from_posixfs:1823): Loading graph from file: smallnetflix/smallnetflix_mm.validate\nLoading graph. Finished in 1.17598\nGoing to add: 3401680 implicit edges. users: 95526 items: 3561\nFinished adding 3401680 implicit edges. \nFinalizing graph.\nINFO:     distributed_ingress_base.hpp(finalize:166): Finalizing Graph...\n^C\n\\endverbatim\n\n\\section Acknowledgements Acknowledgements\n\\li Liang Xiong, CMU for providing the Matlab code of BPTF, numerous discussions and infinite support!! Thanks!!\n\\li Timmy Wilson, Smarttypes.org for providing twitter network snapshot example, and Python scripts for reading the output.\n\\li Sanmi Koyejo, from the University of Austin, Texas, for providing Python scripts for preparing the inputs.\n\\li Dan Brickely, from VU University Amsertdam, for helping debugging installation and prepare the input in Octave.\n\\li Nicholas Ampazis, University of the Aegean, for providing his SVD++ source ode.\n\\li Yehuda Koren, Yahoo! Research, for providing his SVD++ source code implementation.\n\\li Marinka Zitnik, University of Ljubljana, Slovenia, for helping debugging ALS and suggesting NMF algos to implement.\n\\li Joel Welling from Pittsburgh Supercomputing Center, for optimizing GraphLab on BlackLight supercomputer and simplifying installation procedure.\n\\li Sagar Soni from Gujarat Technological University and Hasmukh Goswami College of Engineering for helping testing the code.\n\\li Young Cha, UCLA for testing the code.\n\\li Mohit Singh for helping improve documentation.\n\\li Nicholas Kolegraff for testing our examples.\n\\li Theo Throuillon, Ecole Nationale Superieure d'Informatique et de Mathematiques Appliquees de Grenoble for debugging NMF.\n\\li Qiang Yan, Chinese Academy of Science for providing time-svd++, bias-SVD, RBM and LIBFM code that the Graphlab version is based on.\n\\li Ramakrishnan Kannan, Georgia Tech, for helping debugging and simplifying usage.\n\\li Charles Martin, GLG, for debugging NMF.\n\\li Alex Hasha, bundle.com for improving SGD and bias-SGD documentation and usability.\n\\li Zhao Yu (Jason Chao), douban.com, for identifying SGD/bias-SGD bugs.\n*/\n"
  },
  {
    "path": "toolkits/collaborative_filtering/cosamp.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *  \n *  Code written by Danny Bickson, CMU\n *  Any changes to the code must include this original license notice in full.\n *  This file implements the shooting algorithm for solving Lasso problem\n */\n\n\n#ifndef _COSAMP_HPP\n#define _COSAMP_HPP\n\n\nivec sort_union(ivec a, ivec b){\n   ivec ab = concat(a,b);\n   sort(ab);\n  for (int i=1; i< ab.size(); i++){\n      if (ab[i] == ab[i-1])\n        del(ab,i);\n   }\n   return ab;\n}\n\n\nvec CoSaMP(const mat & Phi, const vec & u, int K, int max_iter, double tol1, int D){\n\n  assert(K<= 2*D);\n  assert(K>=1);\n\n  assert(Phi.rows() == Phi.cols());\n  assert(Phi.rows() == D);\n  assert(u.size() == D);\n  \n\n  vec Sest = zeros(D);\n  vec utrue = Sest;\n  vec v = u;\n  int t=1;\n  ivec T2;\n\n  while (t\n\n#include \n\n\n\n/**\n * \\brief Use the GraphLab serialization macros to add a save routine\n * to the Eigen::VectorXd object.\n */\nBEGIN_OUT_OF_PLACE_SAVE(arc, Eigen::VectorXd, vec) {\n  typedef Eigen::VectorXd::Index index_type;\n  typedef Eigen::VectorXd::Scalar scalar_type;\n  const index_type size = vec.size();\n  arc << size;\n  graphlab::serialize(arc, vec.data(), size * sizeof(scalar_type));\n} END_OUT_OF_PLACE_SAVE()\n\n\n/**\n * \\brief Use the GraphLab serialization macros to add a load routine\n * to the Eigen::VectorXd object.\n */\nBEGIN_OUT_OF_PLACE_LOAD(arc, Eigen::VectorXd, vec) {\n  typedef Eigen::VectorXd::Index index_type;\n  typedef Eigen::VectorXd::Scalar scalar_type;\n  index_type size = 0;\n  arc >> size;\n  vec.resize(size);\n  graphlab::deserialize(arc, vec.data(), size * sizeof(scalar_type));\n} END_OUT_OF_PLACE_LOAD()\n\n\n/**\n * \\brief Use the GraphLab serialization macros to add a save routine\n * to the Eigen::MatrixXd object.\n */\nBEGIN_OUT_OF_PLACE_SAVE(arc, Eigen::MatrixXd, mat) {\n  typedef Eigen::MatrixXd::Index index_type;\n  typedef Eigen::MatrixXd::Scalar scalar_type;\n  const index_type rows = mat.rows();\n  const index_type cols = mat.cols();\n  arc << rows << cols;\n  graphlab::serialize(arc, mat.data(), rows*cols*sizeof(scalar_type));\n} END_OUT_OF_PLACE_SAVE()\n\n\n\n/**\n * \\brief Use the GraphLab serialization macros to add a load routine\n * to the Eigen::MatrixXd object.\n */\nBEGIN_OUT_OF_PLACE_LOAD(arc, Eigen::MatrixXd, mat) {\n  typedef Eigen::MatrixXd::Index index_type; \n  typedef Eigen::MatrixXd::Scalar scalar_type;\n  index_type rows=0, cols=0;\n  arc >> rows >> cols;\n  mat.resize(rows,cols);\n  graphlab::deserialize(arc, mat.data(), rows*cols*sizeof(scalar_type));\n} END_OUT_OF_PLACE_LOAD()\n\n\n\n\n\n\n#endif\n"
  },
  {
    "path": "toolkits/collaborative_filtering/eigen_wrapper.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n/**\n * Code by Danny Bickson, CMU\n */\n#ifndef EIGEN_WRAPPER\n#define EIGEN_WRAPPER\n\n\n/**\n * SET OF WRAPPER FUNCTIONS FOR EIGEN\n *\n *\n */\n\n#include \n#include \n#include \n\n#include \"Eigen/Dense\"\n#define EIGEN_YES_I_KNOW_SPARSE_MODULE_IS_NOT_STABLE_YET\n#include \"Eigen/Sparse\"\n#include \"Eigen/Cholesky\"\n#include \"Eigen/Eigenvalues\"\n#include \"Eigen/SVD\"\n#define EIGEN_DONT_PARALLELIZE //eigen parallel for loop interfers with ours.\nusing namespace Eigen;\n\ntypedef MatrixXd mat;\ntypedef VectorXd vec;\ntypedef VectorXi ivec;\ntypedef MatrixXi imat;\ntypedef Matrix matst;\ntypedef SparseVector sparse_vec;\n\ninline void debug_print_vec(const char * name,const vec& _vec, int len){\n  printf(\"%s ) \", name);\n  for (int i=0; i< len; i++)\n    if (_vec[i] == 0)\n      printf(\"      0    \");\n    else printf(\"%12.4g    \", _vec[i]);\n  printf(\"\\n\");\n}\ninline void debug_print_vec(const char * name,const double* _vec, int len){\n  printf(\"%s ) \", name);\n  for (int i=0; i< len; i++)\n    if (_vec[i] == 0)\n      printf(\"      0    \");\n    else printf(\"%12.4g    \", _vec[i]);\n  printf(\"\\n\");\n}\nmat randn1(int dx, int dy, int col);\n\ntemplate\ninline void set_val(mat &A, int row, int col, data val){\n  A(row, col) = val;\n}\ninline double get_val(const mat &A, int row, int col){\n  return A(row, col);\n}\ninline int get_val(const imat &A, int row, int col){\n  return A(row, col);\n}\ninline vec get_col(const mat& A, int col){\n  return A.col(col);\n}\ninline vec get_row(const mat& A, int row){\n  return A.row(row);\n}\ninline void set_col(mat& A, int col, const vec & val){\n  A.col(col) = val;\n}\ninline void set_row(mat& A, int row, const vec & val){\n  A.row(row) = val;\n}\n\ninline mat eye(int size){\n  return mat::Identity(size, size);\n}\ninline vec ones(int size){\n  return vec::Ones(size);\n}\ninline vec init_vec(const double * array, int size){\n  vec ret(size);\n  memcpy(ret.data(), array, size*sizeof(double));\n  return ret;\n}\ninline mat init_mat(const char * string, int row, int col){\n  mat out(row, col);\n  char buf[2056];\n  strcpy(buf, string);\n  char *pch = strtok(buf,\" \\r\\n\\t;\");\n  for (int i=0; i< row; i++){\n    for (int j=0; j< col; j++){\n      out(i,j) = atof(pch);\n      pch = strtok (NULL, \" \\r\\n\\t;\");\n    }\n  }\n  return out;\n}\ninline imat init_imat(const char * string, int row, int col){\n  imat out(row, col);\n  char buf[2056];\n  strcpy(buf, string);\n  char *pch = strtok(buf,\" \\r\\n\\t;\");\n  for (int i=0; i< row; i++){\n    for (int j=0; j< col; j++){\n      out(i,j) = atol(pch);\n      pch = strtok (NULL, \" \\r\\n\\t;\");\n    }\n  }\n  return out;\n}\ninline vec init_vec(const char * string, int size){\n  vec out(size);\n  char buf[2056];\n  strcpy(buf, string);\n  char *pch = strtok (buf,\" \\r\\n\\t;\");\n  int i=0;\n  while (pch != NULL)\n    {\n      out(i) =atof(pch);\n      pch = strtok (NULL, \" \\r\\n\\t;\");\n      i++;\n    }\n  assert(i == size);\n  return out;\n}\ninline vec init_dbl_vec(const char * string, int size){\n  return init_vec(string, size);\n}\n\ninline vec zeros(int size){\n  return vec::Zero(size);\n}\ninline mat zeros(int rows, int cols){\n  return mat::Zero(rows, cols);\n}\ninline vec head(const vec& v, int num){\n  return v.head(num);\n}\ninline vec mid(const vec&v, int start, int num){\n  return v.segment(start, std::min(num, (int)(v.size()-start)));\n}\ninline vec tail(const vec&v,  int num){\n  return v.segment(v.size() - num, num);\n}\ninline ivec head(const ivec& v, int num){\n  return v.head(num);\n}\ninline void sort(ivec &a){\n  std::sort(a.data(), a.data()+a.size());\n}\ninline void sort(vec & a){\n  std::sort(a.data(), a.data()+a.size());\n}\ninline ivec sort_index(const vec&a){\n  ivec ret(a.size()); \n  std::vector > D;\n  // \t\n  D.reserve(a.size());\n  for (int i=0;i(a.coeff(i),i));\n  std::sort(D.begin(),D.end());\n  for (int i=0;i\ninline double sumsum(const mat & A){\n  return A.sum();\n}\ninline double norm(const mat &A, int pow=2){\n  return A.squaredNorm();\n}\ninline mat inv(const mat&A){\n  return A.inverse();\n}\ninline bool inv(const mat&A, mat &out){\n  out = A.inverse();\n  return true;\n}\ninline mat outer_product(const vec&a, const vec&b){\n  return a*b.transpose();\n}\n//Eigen does not sort eigenvalues, as done in matlab\ninline bool eig_sym(const mat & T, vec & eigenvalues, mat & eigenvectors){\n  //\n  //Column  of the returned matrix is an eigenvector corresponding to eigenvalue number  as returned by eigenvalues(). The eigenvectors are normalized to have (Euclidean) norm equal to one.\n  SelfAdjointEigenSolver solver(T);\n  eigenvectors = solver.eigenvectors();\n  eigenvalues = solver.eigenvalues(); \n  ivec index = sort_index(eigenvalues);\n  sort(eigenvalues);\n  vec eigenvalues2 = eigenvalues.reverse();\n  mat T2 = zeros(eigenvectors.rows(), eigenvectors.cols());\n  for (int i=0; i< eigenvectors.cols(); i++){\n    set_col(T2, index[i], get_col(eigenvectors, i));\n  }   \n  eigenvectors = T2;\n  eigenvalues = eigenvalues2;\n  return true;\n}\n\ninline vec elem_mult(const vec&a, const vec&b){\n  vec ret = a;\n  for (int i=0; i(from,to);\n  return ret;\n}\ninline int randi(int from, int to){\n  return internal::random(from,to);\n}\ninline ivec concat(const ivec&a, const ivec&b){ \n  ivec ret(a.size()+b.size());\n  ret << a,b;\n  return ret;\n}\ninline void del(ivec&a, int i){\n  memcpy(a.data()+i, a.data() + i+1, (a.size() - i - 1)*sizeof(int)); \n  a.conservativeResize(a.size() - 1); //resize without deleting values!\n}\ninline mat get_cols(const mat&A, ivec & cols){\n  mat a(A.rows(), cols.size());\n  for (int i=0; i< cols.size(); i++)\n    set_col(a, i, get_col(A, cols[i]));\n  return a;\n}\ninline mat get_cols(const mat&A, int start_col, int end_col){\n  assert(end_col > start_col);\n  assert(end_col <= A.cols());\n  assert(start_col >= 0);\n  mat a(A.rows(), end_col-start_col);\n  for (int i=0; i< end_col-start_col; i++)\n    set_col(a, i, get_col(A, i));\n  return a;\n}\ninline void set_val(vec & v, int pos, double val){\n  v(pos) = val;\n}\ninline double dot(const vec&a, const vec& b){\n  return a.dot(b);\n}\ninline vec reverse(vec& a){\n  return a.reverse();\n}\ninline ivec reverse(ivec& a){\n  return a.reverse();\n}\ninline const double * data(const mat &A){\n  return A.data();\n}\ninline const int * data(const imat &A){\n  return A.data();\n}\ninline const double * data(const vec &v){\n  return v.data();\n}\n\nclass it_file{\n  std::fstream fb;\n\npublic:\n  it_file(const char * name){\n    fb.open(name, std::fstream::in);\n    fb.close();\n\n    if (fb.fail()){\n      fb.clear(std::fstream::failbit);\n      fb.open(name, std::fstream::out | std::fstream::trunc );\n    }\n    else {\n      fb.open(name, std::fstream::in);\n    }\n   \n    if (!fb.is_open()){\n      perror(\"Failed opening file \");\n      printf(\"filename is: %s\\n\", name);\n      assert(false);\n    }\n  \n  };\n  std::fstream & operator<<(const std::string str){\n    int size = str.size();\n    fb.write((char*)&size, sizeof(int));\n    assert(!fb.fail());\n    fb.write(str.c_str(), size);\n    return fb;\n  }\n  std::fstream &operator<<(mat & A){\n    int rows = A.rows(), cols = A.cols();\n    fb.write( (const char*)&rows, sizeof(int));\n    fb.write( (const char *)&cols, sizeof(int));\n    for (int i=0; i< A.rows(); i++)\n      for (int j=0; j< A. cols(); j++){\n        double val = A(i,j);\n        fb.write( (const char *)&val, sizeof(double));\n        assert(!fb.fail());\n      }\n    return fb;\n  }\n  std::fstream &operator<<(const vec & v){\n    int size = v.size();\n    fb.write( (const char*)&size, sizeof(int));\n    assert(!fb.fail());\n    for (int i=0; i< v.size(); i++){\n      double val = v(i);\n      fb.write( (const char *)&val, sizeof(double));\n      assert(!fb.fail());\n    }\n    return fb;\n  }\n  std::fstream & operator<<(const double &v){\n    fb.write((const char*)&v, sizeof(double));\n    return fb;\n  }\n  std::fstream & operator>>(std::string  str){\n    int size = -1;\n    fb.read((char*)&size, sizeof(int));\n    if (fb.fail() || fb.eof()){\n      perror(\"Failed reading file\");\n      assert(false);\n    }\n     \n    char buf[256];\n    fb.read(buf, std::min(256,size));\n    assert(!fb.fail());\n    assert(!strncmp(str.c_str(), buf, std::min(256,size)));\n    return fb;\n  }\n\n  std::fstream &operator>>(mat & A){\n    int rows, cols;\n    fb.read( (char *)&rows, sizeof(int));\n    assert(!fb.fail());\n    fb.read( (char *)&cols, sizeof(int));\n    assert(!fb.fail());\n    A = mat(rows, cols);\n    double val;\n    for (int i=0; i< A.rows(); i++)\n      for (int j=0; j< A. cols(); j++){\n        fb.read((char*)&val, sizeof(double));\n        assert(!fb.fail());\n        A(i,j) = val;\n      }\n    return fb;\n  }\n  std::fstream &operator>>(vec & v){\n    int size;\n    fb.read((char*)&size, sizeof(int));\n    assert(!fb.fail());\n    assert(size >0);\n    v = vec(size);\n    double val;\n    for (int i=0; i< v.size(); i++){\n      fb.read((char*)& val, sizeof(double));\n      assert(!fb.fail());\n      v(i) = val;\n    }\n    return fb;\n  }\n\n  std::fstream &operator>>(double &v){\n    fb.read((char*)&v, sizeof(double));\n    assert(!fb.fail());\n    return fb;\n  }\n\n  void close(){\n    fb.close();\n  }\n};\n\n#define Name(a) std::string(a)\ninline void set_size(sparse_vec &v, int size){\n  //did not find a way to declare vector dimension, yet\n}\ninline void set_new(sparse_vec&v, int ind, double val){\n  v.insert(ind) = val;\n} \ninline int nnz(sparse_vec& v){\n  return v.nonZeros();\n}\ninline int get_nz_index(sparse_vec &v, sparse_vec::InnerIterator& i){\n  return i.index();\n}\ninline double get_nz_data(sparse_vec &v, sparse_vec::InnerIterator& i){\n  return i.value();\n}\n#define FOR_ITERATOR(i,v)                       \\\n  for (sparse_vec::InnerIterator i(v); i; ++i)\n\ntemplate\ninline double sum_sqr(const T& a);\n\ntemplate<>\ninline double sum_sqr(const vec & a){\n  vec ret = a.array().pow(2);\n  return ret.sum();\n}\ntemplate<>\ninline double sum_sqr(const sparse_vec & a){\n  double sum=0;\n  FOR_ITERATOR(i,a){\n    sum+= powf(i.value(),2);\n  }\n  return sum;\n}\n\ninline double trace(const mat & a){\n  return a.trace();\n}\ninline double get_nz_data(sparse_vec &v, int i){\n  assert(nnz(v) > i);\n  int cnt=0;\n  FOR_ITERATOR(j, v){\n    if (cnt == i){\n      return j.value();\n    }\n    cnt++;\n  }\n  return 0.0;\n}\ninline void print(sparse_vec & vec){\n  int cnt = 0;\n  FOR_ITERATOR(i, vec){\n    std::cout<= 20)\n      break;\n  }\n  std::cout< svdEigen(A, Eigen::ComputeFullU | Eigen::ComputeFullV);\n  U= svdEigen.matrixU();\n  V= svdEigen.matrixV();\n  singular_values =svdEigen.singularValues(); \n}\n#endif\n"
  },
  {
    "path": "toolkits/collaborative_filtering/implicit.hpp",
    "content": "#ifndef _IMPLICIT_HPP__\n#define _IMPLICIT_HPP__\n/**\n * @file\n * @author  Danny Bickson\n * @version 1.0\n *\n * @section LICENSE\n *\n * Copyright [2012] [Carnegie Mellon University]\n * \n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n * \n * http://www.apache.org/licenses/LICENSE-2.0\n * \n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n * \n * header file for handling the addition of implicit edges\n */\n\n#include \"eigen_wrapper.hpp\"\n#include \"stats.hpp\"\n\nenum{\n  IMPLICIT_RATING_DISABLED = 0,\n  IMPLICIT_RATING_RANDOM = 1\n};\n\ndouble implicitratingweight;\ndouble implicitratingvalue;\ndouble implicitratingpercentage;\nint    implicitratingtype;\n\ntemplate\nuint add_implicit_edges4(int type, graph_type & graph, graphlab::distributed_control & dc){\n\n  switch(type){\n    case IMPLICIT_RATING_DISABLED: return 0;\n    case IMPLICIT_RATING_RANDOM: break;\n    default: assert(false);\n  };\n\n  uint added = 0;\n  size_t M = info.max_user;\n  size_t N = info.max_item;\n  uint toadd  = implicitratingpercentage*N*M;\n  dc.cout()<<\"Going to add: \" << toadd << \" implicit edges. users: \" << M << \" items: \" << N << std::endl;\n  assert(toadd >= 1);\n  for (uint j=0; j< toadd; j++){\n    ivec item = ::randi(1,0,N-1);\n    ivec user = ::randi(1,0,M-1);\n    graph.add_edge(user[0], -(graphlab::vertex_id_type(item[0] + SAFE_NEG_OFFSET)), als_edge_type(implicitratingvalue, edge_data::TRAIN, implicitratingweight));\n    added++;\n  } \n  dc.cout()<<\"Finished adding \" << toadd << \" implicit edges. \" << std::endl;\n  return added;\n};\n\ntemplate\nuint add_implicit_edges(int type, graph_type & graph, graphlab::distributed_control & dc){\n\n  switch(type){\n    case IMPLICIT_RATING_DISABLED: return 0;\n    case IMPLICIT_RATING_RANDOM: break;\n    default: assert(false);\n  };\n\n  uint added = 0;\n  size_t M = info.max_user;\n  size_t N = info.max_item;\n  uint toadd  = implicitratingpercentage*N*M;\n  dc.cout()<<\"Going to add: \" << toadd << \" implicit edges. users: \" << M << \" items: \" << N <= 1);\n  for (uint j=0; j< toadd; j++){\n    ivec item = ::randi(1,0,N-1);\n    ivec user = ::randi(1,0,M-1);\n    graph.add_edge(user[0], -(graphlab::vertex_id_type(item[0] + SAFE_NEG_OFFSET)), als_edge_type(implicitratingvalue));\n    added++;\n  } \n  dc.cout()<<\"Finished adding \" << toadd << \" implicit edges. \" << std::endl;\n  return added;\n};\n\nvoid parse_implicit_command_line(graphlab::command_line_options & clopts){\n   clopts.attach_option(\"implicitratingweight\", implicitratingweight,\"implicit rating weight\");\n   clopts.attach_option(\"implicitratingvalue\", implicitratingvalue, \"implicit rating value\");\n   clopts.attach_option(\"implicitratingtype\", implicitratingtype, \"implicit rating type (-=disabled, 1=random)\");\n   if (implicitratingtype != IMPLICIT_RATING_RANDOM && implicitratingtype != IMPLICIT_RATING_DISABLED)\n     logstream(LOG_FATAL)<<\"Implicit rating type should be either 0 (IMPLICIT_RATING_DISABLED) or 1 (IMPLICIT_RATING_RANDOM)\" << std::endl;\n   clopts.attach_option(\"implicitratingpercentage\", implicitratingpercentage, \"implicit rating percentage (1e-8,0.8)\");\n   if (implicitratingpercentage < 1e-8 && implicitratingpercentage > 0.8)\n     logstream(LOG_FATAL)<<\"Implicit rating percentage should be (1e-8, 0.8)\" << std::endl;\n  clopts.attach_option(\"users\", info.max_user, \"max user id (for implicit ratings)\");\n  clopts.attach_option(\"items\", info.max_item, \"max item id (for implicit ratings)\");\n\n}\n#endif //_IMPLICIT_HPP__\n"
  },
  {
    "path": "toolkits/collaborative_filtering/make_synthetic_als_data.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n#include \n#include \n\ntypedef Eigen::VectorXd vec_type;\ntypedef Eigen::MatrixXd mat_type;\n\n\n#include \n#include \n\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n\n  // Parse command line options -----------------------------------------------\n  const std::string description = \n    \"Creates a folder with synthetic training data\";\n  graphlab::command_line_options clopts(description, false);\n  std::string output_folder = \"synthetic_data\";\n  size_t nfiles            = 5;\n  size_t D                 = 20;\n  size_t nusers            = 1000;\n  size_t nmovies           = 10000;\n  size_t nvalidate       = 2;\n  size_t npredict          = 1;\n  double noise             = 0.1;\n  double stdev             = 2;\n  double alpha             = 1.8;\n\n\n  clopts.attach_option(\"dir\", output_folder,\n                       \"Location to create the data files\");\n  clopts.attach_option(\"nfiles\", nfiles,\n                       \"The number of files to generate.\");\n  clopts.attach_option(\"D\", D, \"Number of latent dimensions.\");\n  clopts.attach_option(\"nusers\", nusers,\n                       \"The number of users.\");\n  clopts.attach_option(\"nmovies\", nmovies,\n                       \"The number of movies.\");\n  clopts.attach_option(\"alpha\", alpha,\n                       \"The power-law constant.\");\n  clopts.attach_option(\"nvalidate\", nvalidate,\n                       \"The validate ratings pers user\");\n  clopts.attach_option(\"npredict\", npredict,\n                       \"The predict ratings pers user\");\n\n  clopts.attach_option(\"noise\", noise,\n                       \"The standard deviation noise parameter\");\n  clopts.attach_option(\"stdev\", stdev,\n                       \"The standard deviation in latent factor values\");\n\n  if(!clopts.parse(argc, argv)) {\n    std::cout << \"Error in parsing command line arguments.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  std::cout << \"Creating data directory: \" << output_folder << std::endl;\n  boost::filesystem::path directory(output_folder);\n  if(!boost::filesystem::create_directory(output_folder)) {\n    logstream(LOG_ERROR) \n      << \"Error creating directory: \" << directory << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  std::cout << \"Opening files:\" << std::endl;\n  std::vector< std::ofstream* > train_files(nfiles);\n  std::vector< std::ofstream* > validate_files(nfiles);\n  std::vector< std::ofstream* > predict_files(nfiles);\n  for(size_t i = 0; i < nfiles; ++i) {\n    const std::string train_fname = \n      output_folder + \"/graph_\" + graphlab::tostr(i) + \".tsv\";\n    train_files[i] = new std::ofstream(train_fname.c_str());\n    if(!train_files[i]->good()) {\n      logstream(LOG_ERROR) \n        << \"Error creating file: \" << train_fname;\n    }\n\n    const std::string validate_fname = \n      output_folder + \"/graph_\" + graphlab::tostr(i) + \".tsv.validate\";\n    validate_files[i] = new std::ofstream(validate_fname.c_str());\n    if(!validate_files[i]->good()){\n      logstream(LOG_ERROR) \n        << \"Error creating file: \" << train_fname;\n    }       \n\n    const std::string predict_fname = \n      output_folder + \"/graph_\" + graphlab::tostr(i) + \".tsv.predict\";\n    predict_files[i] = new std::ofstream(predict_fname.c_str());\n    if(!predict_files[i]->good()){\n      logstream(LOG_ERROR) \n        << \"Error creating file: \" << train_fname;\n    }       \n  }\n  \n\n\n\n  // Make synthetic latent factors\n  std::vector< vec_type > user_factors(nusers);\n  std::vector< vec_type > movie_factors(nmovies);\n  // Create a shared random number generator\n  graphlab::random::generator gen; gen.seed(31413);\n  \n  std::cout << \"Constructing latent user factors\" << std::endl;\n  foreach(vec_type& factor, user_factors) {\n    factor.resize(D);\n    // Randomize the factor\n    for(size_t d = 0; d < D; ++d) \n      factor(d) = gen.gaussian(0, stdev);\n  }\n\n  std::cout << \"Constructing latent movie factors\" << std::endl;\n  foreach(vec_type& factor, movie_factors) {\n    factor.resize(D);\n    // Randomize the factor\n    for(size_t d = 0; d < D; ++d) \n      factor(d) = gen.gaussian(0, stdev);\n  }\n\n  size_t nedges_train = 0;\n  ASSERT_GT(nusers, nvalidate + npredict);\n  // Make power-law probability vector\n  std::vector prob(nusers - nvalidate - npredict);\n  for(size_t i = 0; i < prob.size(); ++i)\n    prob[i] = std::pow(double(i+1), -alpha);\n  graphlab::random::pdf2cdf(prob);\n  for(size_t movie_id = 0, user_id = 0; movie_id < nmovies; ++movie_id) {\n    // Add power-law out degree ratings\n    const size_t out_degree = gen.multinomial_cdf(prob) + 1;\n    for(size_t i = 0; i < out_degree; ++i) {\n      user_id = (user_id + 2654435761)  % nusers;\n      const size_t file_id = user_id % nfiles;\n      const double rating = \n        user_factors[user_id].dot(movie_factors[movie_id]);\n      *(train_files[file_id])\n        << user_id << '\\t' << (movie_id + nusers) << '\\t' << rating << '\\n';\n      nedges_train++;\n    }\n    // Add a few extra validate ratings\n    for(size_t i = 0; i < nvalidate; ++i) {\n      user_id = (user_id + 2654435761)  % nusers;\n      const size_t file_id = user_id % nfiles;\n      const double rating = \n        user_factors[user_id].dot(movie_factors[movie_id]);\n      *(validate_files[file_id])\n        << user_id << '\\t' << (movie_id + nusers) << '\\t' << rating << '\\n';\n    }\n    // Add a few extra predict ratings\n    for(size_t i = 0; i < npredict; ++i) {\n      user_id = (user_id + 2654435761)  % nusers;\n      const size_t file_id = user_id % nfiles;\n      *(predict_files[file_id])\n        << user_id << '\\t' << (movie_id + nusers) << '\\n';\n    }\n  } // end of loop over movies\n  for(size_t i = 0; i < nfiles; ++i) {\n    train_files[i]->close(); \n    delete train_files[i]; train_files[i] = NULL;\n    validate_files[i]->close(); \n    delete validate_files[i]; validate_files[i] = NULL;\n    predict_files[i]->close(); \n    delete predict_files[i]; predict_files[i] = NULL;\n  }\n  std::cout << \"Created \" << nedges_train \n            << \" training edges.\" << std::endl;\n\n\n\n} // end of main\n"
  },
  {
    "path": "toolkits/collaborative_filtering/math.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef _MATH_HPP\n#define _MATH_HPP\n\n#include \"types.hpp\"\n#include \"graphlab.hpp\"\n#include \"graphlab/util/tracepoint.hpp\"\n\n\nDECLARE_TRACER(Axbtrace);\nDECLARE_TRACER(Axbtrace2);\nDECLARE_TRACER(vecequals);\nDECLARE_TRACER(orthogonalize_vs_alltrace);\nDECLARE_TRACER(als_lapack_trace);\nDECLARE_TRACER(orth1);\nDECLARE_TRACER(orth2);\nDECLARE_TRACER(orth3);\n\ndouble regularization = 0;\nbool debug;\nbool regnormal;\n\nvoid print_vec(const char * name, const vec & pvec, bool high);\n\nstruct math_info{\n  //for Axb operation\n  int increment;\n  double  c;\n  double  d;\n  int x_offset, b_offset , y_offset, r_offset, div_offset, prev_offset, mat_offset, vec_offset;\n  int orthogonalization;\n  bool A_offset, A_transpose;\n  std::vector names;\n  bool use_diag;\n  int ortho_repeats;\n  int start, end;\n  bool update_function;\n\n  //for backslash operation\n  bool dist_sliced_mat_backslash;\n  mat eDT;\n  double maxval, minval;\n\n  math_info(){\n    reset_offsets();\n  }\n\n  void reset_offsets(){\n    increment = 2;\n    c=1.0; d=0.0;\n    x_offset = b_offset = y_offset = r_offset = div_offset = prev_offset = mat_offset = vec_offset = -1;\n    A_offset = false;\n    A_transpose = false;\n    use_diag = true;\n    start = end = -1;\n    update_function = false;\n    dist_sliced_mat_backslash = false;\n    orthogonalization = 0;\n  }\n  int increment_offset(){\n    return increment++;\n  }\n\n\n};\n\n\nbipartite_graph_descriptor info;\nmath_info mi;\nclass DistMat; \nclass DistDouble;\nclass DistSlicedMat;\nDistSlicedMat * curMat = NULL;\ngather_type alphas;\ngather_type sum_alpha;\n\n\n\n#define MAX_PRINT_ITEMS 25\ndouble runtime = 0;\n\nusing namespace graphlab;\n\nvec curvec;\n/***\n * UPDATE FUNCTION (ROWS)\n */\nclass Axb :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\n    float last_change;\n    public:\n    /* Gather the weighted rank of the adjacent page   */\n    double gather(icontext_type& context, const vertex_type& vertex,\n        edge_type& edge) const {\n\n      if (edge.data().role == edge_data::PREDICT)\n         return 0;\n\n      bool brows = vertex.id() < (uint)info.get_start_node(false);\n      if (info.is_square()) \n        brows = !mi.A_transpose;\n      if (mi.A_offset  && mi.x_offset >= 0){\n        double val = edge.data().obs * (brows ? edge.target().data().pvec[mi.x_offset] :\n            edge.source().data().pvec[mi.x_offset]);\n        //printf(\"gather edge on vertex %d val %lg obs %lg\\n\", vertex.id(), val, edge.data().obs);\n        return val;\n      }\n      //printf(\"edge on vertex %d val %lg\\n\", vertex.id(), 0.0);\n      return 0;\n    }\n\n    /* Use the total rank of adjacent pages to update this page */\n    void apply(icontext_type& context, vertex_type& vertex,\n        const double& total) {\n\n      //printf(\"Entered apply on node %d value %lg\\n\", vertex.id(), total);\n      vertex_data & user = vertex.data();\n      assert(mi.x_offset >=0 || mi.y_offset >= 0);\n      assert(mi.r_offset >=0);\n\n      /* perform orthogonalization of current vector */\n      if (mi.orthogonalization){\n         for (int i=mi.mat_offset; i< mi.vec_offset; i++){\n            vertex.data().pvec[mi.vec_offset] -= alphas.pvec[i-mi.mat_offset] * vertex.data().pvec[i]; \n         }\n         return;\n      }\n\n      double val = total;\n      //assert(total != 0 || mi.y_offset >= 0);\n\n      //store previous value for convergence detection\n      if (mi.prev_offset >= 0)\n        user.pvec[mi.prev_offset ] = user.pvec[mi.r_offset];\n\n      assert(mi.x_offset >=0 || mi.y_offset>=0);\n      if (mi.A_offset  && mi.x_offset >= 0){\n        if  (info.is_square() && mi.use_diag)// add the diagonal term\n          val += (/*mi.c**/ (user.A_ii+ regularization) * user.pvec[mi.x_offset]);\n        //printf(\"node %d added diag term: %lg\\n\", vertex.id(), user.A_ii);\n        val *= mi.c;\n      }\n      /***** COMPUTE r = c*I*x  *****/\n      else if (!mi.A_offset && mi.x_offset >= 0){\n        val = mi.c*user.pvec[mi.x_offset];\n      }\n\n      /**** COMPUTE r+= d*y (optional) ***/\n      if (mi.y_offset>= 0){\n        val += mi.d*user.pvec[mi.y_offset]; \n      }\n\n      /***** compute r = (... ) / div */\n      if (mi.div_offset >= 0){\n        val /= user.pvec[mi.div_offset];\n      }\n\n      user.pvec[mi.r_offset] = val;\n      //printf(\"Exit apply on node %d value %lg\\n\", vertex.id(), val);\n    }\n\n    edge_dir_type gather_edges(icontext_type& context,\n        const vertex_type& vertex) const {\n      if (vertex.id() < rows)\n        return OUT_EDGES;\n      else return IN_EDGES;\n    }\n\n\n    edge_dir_type scatter_edges(icontext_type& context,\n        const vertex_type& vertex) const {\n      return NO_EDGES;\n    }\n\n    /* The scatter function just signal adjacent pages */\n    //void scatter(icontext_type& context, const vertex_type& vertex,\n    //    edge_type& edge) const {\n    //}\n\n  }; \n\nvoid init_lanczos_mapr( graph_type::vertex_type& vertex) {\n  assert(actual_vector_len > 0);\n  vertex.data().pvec = zeros(actual_vector_len);\n} \n\n\n\n\nvoid init_math(graph_type * _pgraph, bipartite_graph_descriptor & _info, double ortho_repeats = 3, \n    bool update_function = false){\n  pgraph = _pgraph;\n  info = _info;\n  mi.reset_offsets();\n  mi.update_function = update_function;\n  mi.ortho_repeats = ortho_repeats;\n}\n\n\nclass DistVec{\n  public:\n    int offset; //real location in memory\n    int display_offset; //offset to print out\n    int prev_offset;\n    std::string name; //optional\n    bool transpose;\n    bipartite_graph_descriptor info;\n    int start; \n    int end;\n\n    void init(){\n      start = info.get_start_node(!transpose);\n      end = info.get_end_node(!transpose);\n      assert(start < end && start >= 0 && end >= 1);\n      //debug_print(name);\n    };\n\n    int size(){ return end-start; }\n\n    DistVec(const bipartite_graph_descriptor &_info, int _offset, bool _transpose, const std::string & _name){\n      offset = _offset;\n      display_offset = _offset;\n      name = _name;\n      info = _info;\n      transpose = _transpose;\n      prev_offset = -1;\n      init();\n    }\n    DistVec(const bipartite_graph_descriptor &_info, int _offset, bool _transpose, const std::string & _name, int _prev_offset){\n      offset = _offset;\n      display_offset = _offset;\n      name = _name;\n      info = _info;\n      transpose = _transpose;\n      assert(_prev_offset < data_size);\n      prev_offset = _prev_offset;\n      init();\n    }\n\n\n    DistVec& operator-(){\n      mi.d=-1.0;\n      return *this; \n    }\n    DistVec& operator-(const DistVec & other){\n      mi.x_offset = offset;\n      mi.y_offset = other.offset;\n      transpose = other.transpose;\n      if (mi.d == 0)\n        mi.d = -1.0;\n      else \n        mi.d*=-1.0;\n      return *this;\n    }\n    DistVec& operator+(){\n      if (mi.d == 0)\n        mi.d=1.0;\n      return *this;\n    }\n    DistVec& orthogonalize(){\n      mi.orthogonalization = 1;\n      return *this;\n    }\n\n    DistVec& operator+(const DistVec &other){\n      mi.x_offset =offset;\n      mi.y_offset = other.offset;\n      transpose = other.transpose;\n      return *this; \n    }\n    DistVec& operator+(const DistMat &other);\n\n    DistVec& operator-(const DistMat &other);\n\n    DistVec& operator/(const DistVec &other){\n      mi.div_offset = other.offset;\n      return *this;\n    }\n    DistVec& operator/(const DistDouble & other);\n\n    DistVec& operator/(double val){\n      assert(val != 0);\n      assert(mi.d == 0);\n      mi.d = 1/val;\n      return *this;\n    }\n\n    DistVec& operator=(const DistVec & vec);\n\n    DistVec& operator=(const vec & pvec);\n\n    vec to_vec(int dmax = -1, int doffset = -1);\n\n\n    void debug_print(const char * name){\n      if (debug){\n        std::cout<to_vec(MAX_PRINT_ITEMS, mi.r_offset == -1? offset:mi.r_offset);\n        for (int i=0; i< pvec.size(); i++){  \n          //TODO printf(\"%.5lg \", fabs(pgraph->vertex_data(i).pvec[(mi.r_offset==-1)?offset:mi.r_offset]));\n          printf(\"%.5lg \", fabs(pvec[i]));\n        }\n        printf(\"\\n\");\n      }\n    }\n    void debug_print(std::string name){ return debug_print(name.c_str());}\n\n    double operator[](int i){\n      assert(i < end - start);\n      assert(false);\n      // TODO   return pgraph->vertex_data(i+start).pvec[offset];\n    }\n\n    DistDouble operator*(const DistVec & other);\n\n    DistVec& operator*(const double val){\n      assert(val!= 0);\n      mi.d=val;\n      return *this;\n    }\n    DistVec& operator*(const DistDouble &dval);\n\n    DistMat &operator*(DistMat & v);\n\n    DistVec& _transpose() { \n      /*if (!config.square){\n        start = n; end = m+n;\n        }*/\n      return *this;\n    }\n\n    DistVec& operator=(DistMat &mat);\n\n};\n\nDistVec * pcurrent = NULL;\n\n\nclass DistSlicedMat{\n  public:\n    bipartite_graph_descriptor info;\n    int start_offset;\n    int end_offset; \n    std::string name; //optional\n    int start;\n    int end;\n    bool transpose;\n\n    DistSlicedMat(int _start_offset, int _end_offset, bool _transpose, const bipartite_graph_descriptor &_info, std::string _name){\n      assert(_start_offset < _end_offset);\n      assert(_start_offset >= 0);\n      assert(_info.total() > 0);\n      transpose = _transpose;\n      info = _info;\n      init();\n      start_offset = _start_offset;\n      end_offset = _end_offset;\n      name = _name;\n    }\n\n    DistSlicedMat& operator=(DistMat & other);\n\n    void init(){\n      start = info.get_start_node(!transpose);\n      end = info.get_end_node(!transpose);\n      assert(start < end && start >= 0 && end >= 1);\n      //debug_print(name);\n    };\n\n    int size(int dim){ return (dim == 1) ? (end-start) : (end_offset - start_offset) ; }\n\n    void set_cols(int start_col, int end_col, const mat& pmat){\n      assert(start_col >= 0);\n      assert(end_col <= end_offset - start_offset);\n      assert(pmat.rows() == end-start);\n      assert(pmat.cols() >= end_col - start_col);\n      for (int i=start_col; i< end_col; i++)\n        this->operator[](i) = get_col(pmat, i-start_col);\n    }\n    mat get_cols(int start_col, int end_col){\n      assert(start_col < end_offset - start_offset);\n      assert(start_offset + end_col <= end_offset);\n      mat retmat = zeros(end-start, end_col - start_col);\n      for (int i=start_col; i< end_col; i++)\n        set_col(retmat, i-start_col, this->operator[](i-start_col).to_vec());\n      return retmat;\n    }\n\n    void operator=(mat & pmat){\n      assert(end_offset-start_offset <= pmat.cols());\n      assert(end-start == pmat.rows());\n      set_cols(0, pmat.cols(), pmat);\n    }\n\n    std::string get_name(int pos){\n      assert(pos < end_offset - start_offset);\n      assert(pos >= 0);\n      return name;\n    }\n\n    DistVec operator[](int pos){\n      assert(pos < end_offset-start_offset);\n      assert(pos >= 0);\n      DistVec ret(info, start_offset + pos, transpose, get_name(pos));\n      ret.display_offset = pos;\n      return ret;\n    }\n\n};\n\n\n\nvoid assign_vec(graph_type::vertex_type & vertex){\n  if (!info.is_square())\n    assert(vertex.id() - pcurrent->start >= 0 && vertex.id() - pcurrent->start < curvec.size());\n  vertex.data().pvec[pcurrent->offset] = curvec[vertex.id() - pcurrent->start];\n}  \n\ngather_type output_vector(const graph_type::vertex_type & vertex){\n   assert(pcurrent && pcurrent->offset >= 0 && pcurrent->offset < vertex.data().pvec.size());\n   gather_type ret;\n   assert(pcurrent->end - pcurrent->start > 0);\n   assert(vertex.id() - pcurrent->start >= 0);\n   ret.pvec = vec::Zero(pcurrent->end - pcurrent->start);\n   ret.pvec[vertex.id() - pcurrent->start] = vertex.data().pvec[pcurrent->offset];\n   return ret;\n}\nbool select_in_range(const graph_type::vertex_type & vertex){\n   return vertex.id() >= (uint)pcurrent->start && vertex.id() < (uint)pcurrent->end;\n}\nDistVec& DistVec::operator=(const DistVec & vec){\n      assert(offset < (info.is_square() ? 2*data_size: data_size));\n      if (mi.x_offset == -1 && mi.y_offset == -1){\n        mi.y_offset = vec.offset;\n      }  \n      mi.r_offset = offset;\n      assert(prev_offset < data_size);\n      mi.prev_offset = prev_offset;\n      if (mi.d == 0.0)\n        mi.d=1.0;\n      transpose = vec.transpose;\n      end = vec.end; \n      start = vec.start;\n      mi.start = start;\n      mi.end = end;\n      INITIALIZE_TRACER(Axbtrace2, \"Update function Axb\");\n      BEGIN_TRACEPOINT(Axbtrace2);\n      pcurrent = (DistVec*)&vec;\n      start_engine();\n      debug_print(name);\n      mi.reset_offsets();\n      return *this;\n    }\n\nDistVec& DistVec::operator=(const vec & pvec){\n  assert(offset >= 0);\n  assert(pvec.size() == info.num_nodes(true) || pvec.size() == info.num_nodes(false));\n  assert(start < end);\n  if (!info.is_square() && pvec.size() == info.num_nodes(false)){\n    transpose = true;\n  }\n  else {\n    transpose = false;\n  }\n  //#pragma omp parallel for    \n  INITIALIZE_TRACER(vecequals, \"vector assignment\");\n  BEGIN_TRACEPOINT(vecequals);\n  //for (int i=start; i< end; i++){  \n  //  pgraph->vertex_data(i).pvec[offset] = pvec[i-start];\n  //}\n  pcurrent = this;\n  curvec = pvec;\n  graphlab::vertex_set nodes = pgraph->select(select_in_range);\n  pgraph->transform_vertices(assign_vec, nodes);\n  END_TRACEPOINT(vecequals);\n  debug_print(name);\n  return *this;       \n}\n\nvec DistVec::to_vec(int dmax, int doffset){\n  pcurrent = this;\n  if (doffset >= 0)\n    pcurrent->offset = doffset;\n  if (dmax >= 0)\n    pcurrent->end = std::min(pcurrent->start + dmax, pcurrent->end);\n  graphlab::vertex_set nodes = pgraph->select(select_in_range);\n  //    for (int i=start; i< end; i++){\n  //      //TODO ret[i-start] = pgraph->vertex_data(i).pvec[offset];\n  //    }\n  gather_type curvec = pgraph->map_reduce_vertices(output_vector, nodes);\n  return curvec.pvec;\n}\n\n\n\n/*\n * wrapper for computing r = c*A*x+d*b*y\n */\nclass DistMat{\n  public:\n    bool transpose;\n    bipartite_graph_descriptor info;\n\n    DistMat(const bipartite_graph_descriptor& _info) { \n      info = _info;\n      transpose = false;\n    };\n\n\n    DistMat &operator*(const DistVec & v){\n      mi.x_offset = v.offset;\n      mi.A_offset = true;\n      //v.transpose = transpose;\n      //r_offset = A_offset;\n      return *this;\n    }\n    DistMat &operator*(const DistDouble &d);\n\n    DistMat &operator-(){\n      mi.c=-1.0;\n      return *this;\n    }\n\n    DistMat &operator/(const DistVec & v){\n      mi.div_offset = v.offset;\n      return *this;\n    }\n\n    DistMat &operator+(){\n      mi.c=1.0;\n      return *this;\n    }\n    DistMat &operator+(const DistVec &v){\n      mi.y_offset = v.offset;\n      if (mi.d == 0.0)\n        mi.d=1.0;\n      return *this;\n    }\n    DistMat &operator-(const DistVec &v){\n      mi.y_offset = v.offset;\n      if (mi.d == 0.0)\n        mi.d=-1.0;\n      else \n        mi.d*=-1.0;\n      return *this;\n    }\n    DistMat & _transpose(){\n      transpose = true;\n      mi.A_transpose = true;\n      return *this;\n    }\n    DistMat & operator~(){\n      return _transpose();\n    }\n    DistMat & backslash(DistSlicedMat & U){\n      mi.dist_sliced_mat_backslash = true;\n      transpose = U.transpose;\n      return *this;\n    }\n    void set_use_diag(bool use){\n      mi.use_diag = use;\n    }   \n};\n\n\nDistVec& DistVec::operator=(DistMat &mat){\n  mi.r_offset = offset;\n  assert(prev_offset < data_size);\n  mi.prev_offset = prev_offset;\n  transpose = mat.transpose;\n  mi.start = info.get_start_node(!transpose);\n  mi.end = info.get_end_node(!transpose);\n  INITIALIZE_TRACER(Axbtrace, \"Axb update function\");\n  BEGIN_TRACEPOINT(Axbtrace);\n  pcurrent = this;\n  int old_start = start; int old_end = end;\n  start = mi.start; end = mi.end;\n  start_engine();\n  start = old_start; end = old_end;\n  END_TRACEPOINT(Axbtrace);\n  debug_print(name);\n  mi.reset_offsets();\n  mat.transpose = false;\n  return *this;\n}\nDistVec& DistVec::operator+(const DistMat &other){\n  mi.y_offset = offset;\n  transpose = other.transpose;\n  return *this; \n}\nDistVec& DistVec::operator-(const DistMat & other){\n  mi.y_offset = offset;\n  transpose = other.transpose;\n  if (mi.c == 0)\n    mi.c = -1;\n  else mi.c *= -1;\n  return *this;\n}\n\nDistMat& DistVec::operator*(DistMat & v){\n  mi.x_offset = offset;\n  mi.A_offset = true;\n  return v;\n}\n\n\nclass DistDouble{\n  public:\n    double val;\n    std::string name;\n\n    DistDouble() {};\n    DistDouble(double _val) : val(_val) {};\n\n\n    DistVec& operator*(DistVec & dval){\n      mi.d=val;\n      return dval;\n    }\n    DistMat& operator*(DistMat & mat){\n      mi.c = val;\n      return mat;\n    }\n\n    DistDouble  operator/(const DistDouble dval){\n      DistDouble mval;\n      mval.val = val / dval.val;\n      return mval;\n    }\n    bool operator<(const double other){\n      return val < other;\n    }\n    DistDouble & operator=(const DistDouble & other){\n      val = other.val;\n      debug_print(name);\n      return *this;\n    }\n    bool operator==(const double _val){\n      return val == _val;\n    }\n    void debug_print(const char * name){\n      std::cout<=0 && mi.b_offset >= 0);\n\n  double val = 0;\n  for (int i=start; i< end; i++){  \n    assert(false);//not yet\n    //TODO const vertex_data * data = &pgraph->vertex_data(i);\n    //TODO double * pv = (double*)&data->pvec[0];\n    //TODO val += mi.d* pv[mi.y_offset] * pv[mi.b_offset];\n  }\n  mi.reset_offsets();\n  DistDouble mval;\n  mval.val = val;\n  return mval;\n}\nDistVec& DistVec::operator*(const DistDouble &dval){\n  mi.d = dval.val;\n  return *this;\n}\n\n\nint size(DistMat & A, int pos){\n  assert(pos == 1 || pos == 2);\n  return A.info.num_nodes(!A.transpose);\n}\n\nDistMat &DistMat::operator*(const DistDouble &d){\n  mi.c = d.val;\n  return *this;\n}\n\nDistDouble sqrt(DistDouble & dval){\n  DistDouble mval;\n  mval.val=sqrt(dval.val);\n  return mval;\n}\n\ngather_type calc_norm(const graph_type::vertex_type & vertex){\n  gather_type ret;\n  assert(pcurrent && pcurrent->offset < vertex.data().pvec.size());\n  ret.training_rmse = pow(vertex.data().pvec[pcurrent->offset], 2);\n  return ret;\n}\n\nDistDouble norm(const DistVec &vec){\n  assert(vec.offset>=0);\n  assert(vec.start < vec.end);\n\n  DistDouble mval;\n  mval.val = 0;\n  pcurrent = (DistVec*)&vec;\n  vertex_set nodes = pgraph->select(select_in_range);\n  //for (int i=vec.start; i < vec.end; i++){\n    // TODO const vertex_data * data = &pgraph->vertex_data(i);\n    //double * px = (double*)&data->pvec[0];\n    // mval.val += px[vec.offset]*px[vec.offset];\n  gather_type ret = pgraph->map_reduce_vertices(calc_norm);\n  //}\n  mval.val = sqrt(ret.training_rmse);\n  return mval;\n}\n\n\nDistDouble norm(DistMat & mat){\n  DistVec vec(info, 0, mat.transpose, \"norm\");\n  vec = mat;\n  return norm((const DistVec&)vec);\n}\n\nvec diag(DistMat & mat){\n  assert(info.is_square());\n  vec ret = zeros(info.total());\n  for (int i=0; i< info.total(); i++){\n    //TODO ret[i] = pgraph->vertex_data(i).A_ii;\n    assert(false);\n  }\n  return ret;\n}\n\nint curoffset = -1;\ngather_type map_reduce_ortho(const graph_type::vertex_type & vertex){\n  gather_type ret;\n  assert(curoffset >= 0);\n  assert(curMat && curMat->start_offset - pcurrent->offset);\n  ret.pvec = vec::Zero(curoffset);\n  assert(curMat != NULL && curMat->start_offset < pcurrent->offset);\n  //for (int i=mat.start_offset; i< current.offset; i++){\n  for (int i=curMat->start_offset; i< pcurrent->offset; i++){\n    ret.pvec[i - curMat->start_offset] = vertex.data().pvec[i] * vertex.data().pvec[pcurrent->offset];\n  }\n  //printf(\"map_Reduce_ortho: node %d\\n\", vertex.id());\n  //std::cout<offset >= 0 && pcurrent->offset < vertex.data().pvec.size());\n    ret.training_rmse = pow(vertex.data().pvec[pcurrent->offset], 2);\n    return ret;\n  }\n  void divide_by_sum(graph_type::vertex_type& vertex){\n    assert(pcurrent->offset >= 0 && pcurrent->offset < vertex.data().pvec.size());\n    vertex.data().pvec[pcurrent->offset] /= sum_alpha.training_rmse;\n  }\n\n\n\n  void transform_ortho(graph_type::vertex_type & vertex){\n    assert(curMat != NULL && curMat->start_offset < pcurrent->offset);\n    for (int i=curMat->start_offset; i< pcurrent->offset; i++){\n      //assert(alphas.pvec[i-curMat->start_offset] != 0);\n      vertex.data().pvec[pcurrent->offset] -= alphas.pvec[i-curMat->start_offset] * vertex.data().pvec[i]; \n    }\n  }\n\n  bool selected_node(const graph_type::vertex_type& vertex){\n    if (info.is_square())\n      return true;\n    else return ((vertex.id() >= (uint)info.get_start_node(!pcurrent->transpose)) &&\n        (vertex.id() < (uint)info.get_end_node(!pcurrent->transpose)));\n  }\n\n\n  double orthogonalize_vs_all(DistSlicedMat & mat, int _curoffset, double &alpha){\n    assert(mi.ortho_repeats >=1 && mi.ortho_repeats <= 3);\n    curoffset = _curoffset;\n    curMat = &mat;\n    mi.mat_offset = mat.start_offset;\n    INITIALIZE_TRACER(orthogonalize_vs_alltrace, \"orthogonalization step - optimized\");\n    BEGIN_TRACEPOINT(orthogonalize_vs_alltrace);\n    bool old_debug = debug;\n    debug = false;\n    DistVec current = mat[curoffset];\n    pcurrent =¤t;\n    mi.vec_offset = pcurrent->offset;\n    assert(mat.start_offset <= current.offset); \n    vertex_set nodes = pgraph->select(selected_node);\n    if (curoffset > 0){\n      for (int j=0; j < mi.ortho_repeats; j++){\n        INITIALIZE_TRACER(orth1, \"map reduce in ortho\");\n        BEGIN_TRACEPOINT(orth1);\n        alphas = pgraph->map_reduce_vertices(map_reduce_ortho, nodes);\n        END_TRACEPOINT(orth1);\n        //pgraph->transform_vertices(transform_ortho, nodes);\n        mat[_curoffset] = mat[_curoffset].orthogonalize(); \n      } //for ortho_repeast \n    }\n\n    debug = old_debug;\n    current.debug_print(current.name);\n    INITIALIZE_TRACER(orth2, \"map reduce in ortho2\");\n    BEGIN_TRACEPOINT(orth2);\n    sum_alpha = pgraph->map_reduce_vertices(map_reduce_sum_power, nodes);\n    END_TRACEPOINT(orth2);\n    sum_alpha.training_rmse = sqrt(sum_alpha.training_rmse);\n    alpha = sum_alpha.training_rmse;\n    if (alpha >= 1e-10 ){\n       INITIALIZE_TRACER(orth3, \"transform_vertices in ortho3\");\n       BEGIN_TRACEPOINT(orth3);\n       //pgraph->transform_vertices(divide_by_sum, nodes);    \n       mat[_curoffset] = mat[_curoffset] / alpha;\n       END_TRACEPOINT(orth3);\n    }\n    END_TRACEPOINT(orthogonalize_vs_alltrace);\n    return alpha;\n  }\n\n\n  DistVec& DistVec::operator/(const DistDouble & other){\n    assert(other.val != 0);\n    assert(mi.d == 0);\n    mi.d = 1/other.val;\n    return *this;\n  }\n\n#endif //_MATH_HPP\n"
  },
  {
    "path": "toolkits/collaborative_filtering/nmf.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * \\file\n * This code iplements the NMF algorithm described in the paper:\n * Lee, D..D., and Seung, H.S., (2001), 'Algorithms for Non-negative Matrix\n * Factorization', Adv. Neural Info. Proc. Syst. 13, 556-562.\n */\n\n#include \n#include \n\n#include \n#include \"eigen_serialization.hpp\"\n#include \n#include \n#include \"stats.hpp\"\n\ntypedef Eigen::VectorXd vec;\ntypedef Eigen::MatrixXd mat_type;\n\n//when using negative node id range, we are not allowed to use\n//0 and 1 so we add 2.\nconst static int SAFE_NEG_OFFSET=2;\nconst double epsilon = 1e-16;\nstatic bool debug;\nint iter = 0;\n\nbool isuser(uint node){\n  return ((int)node) >= 0;\n}\n/** \n * \\ingroup toolkit_matrix_pvecization\n *\n * \\brief the vertex data type which contains the latent pvec.\n *\n * Each row and each column in the matrix corresponds to a different\n * vertex in the SGD graph.  Associated with each vertex is a pvec\n * (vector) of latent parameters that represent that vertex.  The goal\n * of the SGD algorithm is to find the values for these latent\n * parameters such that the non-zero entries in the matrix can be\n * predicted by taking the dot product of the row and column pvecs.\n */\nstruct vertex_data {\n  /**\n   * \\brief A shared \"constant\" that specifies the number of latent\n   * values to use.\n   */\n  static size_t NLATENT;\n  /** \\brief The latent pvec for this vertex */\n  vec pvec;\n\n  double train_rmse;\n  double validation_rmse;\n  /** \n   * \\brief Simple default constructor which randomizes the vertex\n   *  data \n   */\n  vertex_data() { if (debug) pvec = vec::Ones(NLATENT); else randomize(); train_rmse = validation_rmse = 0; } \n  /** \\brief Randomizes the latent pvec */\n  void randomize() { pvec.resize(NLATENT); pvec.setRandom(); }\n  /** \\brief Save the vertex data to a binary archive */\n  void save(graphlab::oarchive& arc) const { \n    arc << pvec << train_rmse << validation_rmse;\n  }\n  /** \\brief Load the vertex data from a binary archive */\n  void load(graphlab::iarchive& arc) { \n    arc >> pvec >>train_rmse >> validation_rmse;\n  }\n}; // end of vertex data\n\n\n/**\n * \\brief The edge data stores the entry in the matrix.\n *\n * In addition the edge data nmfo stores the most recent error estimate.\n */\nstruct edge_data : public graphlab::IS_POD_TYPE {\n  /**\n   * \\brief The type of data on the edge;\n   *\n   * \\li *Train:* the observed value is correct and used in training\n   * \\li *Validate:* the observed value is correct but not used in training\n   * \\li *Predict:* The observed value is not correct and should not be\n   *        used in training.\n   */\n  enum data_role_type { TRAIN, VALIDATE, PREDICT  };\n\n  /** \\brief the observed value for the edge */\n  float weight;\n\n  /** \\brief The train/validation/test designation of the edge */\n  data_role_type role;\n\n  /** \\brief basic initialization */\n  edge_data(float weight = 0, data_role_type role = PREDICT) :\n    weight(weight), role(role) { }\n\n}; // end of edge data\n\n\n\n/**\n * \\brief The graph type is defined in terms of the vertex and edge\n * data.\n */ \ntypedef graphlab::distributed_graph graph_type;\n\n\n\ndouble extract_l2_error(const graph_type::edge_type & edge);\n\n\n/**\n * \\brief Given a vertex and an edge return the other vertex in the\n * edge.\n */\ninline graph_type::vertex_type\nget_other_vertex(graph_type::edge_type& edge, \n    const graph_type::vertex_type& vertex) {\n  return vertex.id() == edge.source().id()? edge.target() : edge.source();\n}; // end of get_other_vertex\n\n\nclass gather_type {\n  public:\n    vec pvec;\n    double training_rmse;\n    double validation_rmse;\n    gather_type() { training_rmse = validation_rmse = 0; }\n    gather_type(const vec & _pvec, double _train_rmse, double _validation_rmse){ pvec = _pvec; training_rmse = _train_rmse; validation_rmse = _validation_rmse; }\n    void reset(){ pvec = vec::Zero(vertex_data::NLATENT); training_rmse = 0; validation_rmse = 0; }\n    void save(graphlab::oarchive& arc) const { arc << pvec << training_rmse << validation_rmse; }\n    void load(graphlab::iarchive& arc) { arc >> pvec >> training_rmse >> validation_rmse; }  \n    gather_type& operator+=(const gather_type& other) {\n      pvec += other.pvec;\n      training_rmse += other.training_rmse;\n      validation_rmse += other.validation_rmse;\n      return *this;\n    } \n\n};\n\ngather_type x1;\ngather_type x2;\ngather_type * px;\n\n\nbool isuser_node(const graph_type::vertex_type& vertex){\n  return isuser(vertex.id());\n}\n/**\n * SGD vertex program type\n */ \nclass nmf_vertex_program :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE{\n    public:\n      /** The convergence tolerance */\n      static double TOLERANCE;\n      static double MAXVAL;\n      static double MINVAL;\n      static bool debug;\n      static size_t MAX_UPDATES;\n\n      /** compute a missing value based on NMF algorithm */\n      static float nmf_predict(const vertex_data& user, \n          const vertex_data& movie, \n          const float rating, \n          double & prediction){\n\n        prediction = user.pvec.dot(movie.pvec);\n        //truncate prediction to allowed values\n        prediction = std::min((double)prediction, nmf_vertex_program::MAXVAL);\n        prediction = std::max((double)prediction, nmf_vertex_program::MINVAL);\n        //return the squared error\n        float err = rating - prediction;\n        assert(!std::isnan(err));\n        return err*err; \n\n      }\n\n\n      /** The set of edges to gather along */\n      edge_dir_type gather_edges(icontext_type& context, \n          const vertex_type& vertex) const { \n        //UNUSED \n        return graphlab::ALL_EDGES; \n      }; // end of gather_edges \n\n      /** The gather function computes XtX and Xy */\n      gather_type gather(icontext_type& context, const vertex_type& vertex, \n          edge_type& edge) const {\n\n        if (edge.data().role == edge_data::TRAIN || edge.data().role == edge_data::VALIDATE){\n          const vertex_type other_vertex = get_other_vertex(edge, vertex);\n          double prediction = 0;\n          double rmse = nmf_predict(vertex.data(), other_vertex.data(), edge.data().weight, prediction);\n          if (prediction == 0)\n            logstream(LOG_FATAL)<<\"Got into numerical error!\" << std::endl;\n          if (edge.data().role == edge_data::TRAIN)\n            return gather_type(other_vertex.data().pvec * (edge.data().weight / prediction), rmse, 0);\n          else //validation\n            return gather_type(vec::Zero(vertex_data::NLATENT), 0, rmse);\n\n        }\n        return gather_type(vec::Zero(vertex_data::NLATENT), 0, 0);\n      } // end of gather function\n\n      void apply(icontext_type& context, vertex_type& vertex,\n          const gather_type& sum) {\n        vertex_data& vdata = vertex.data();  \n        if (vdata.pvec.sum() != 0){\n          for (uint i=0; i< vertex_data::NLATENT; i++){\n            vdata.pvec[i] *= sum.pvec[i] / px->pvec[i];\n            ASSERT_NE(px->pvec[i] , 0);\n            if (vdata.pvec[i] < epsilon)\n              vdata.pvec[i] = epsilon;\n          }\n        }\n        vdata.train_rmse = sum.training_rmse;\n        vdata.validation_rmse = sum.validation_rmse;\n      }\n\n      edge_dir_type scatter_edges(icontext_type& context,\n          const vertex_type& vertex) const { \n        //UNUSED \n        return graphlab::ALL_EDGES; \n      }; // end of scatter edges\n\n      void scatter(icontext_type& context, const vertex_type& vertex, \n          edge_type& edge) const {\n        //we do not schedule any more neighbors to run\n      } \n      static void verify_rows(graph_type::vertex_type& vertex){\n        if (isuser(vertex.id()) && vertex.num_out_edges() == 0)\n          logstream(LOG_FATAL)<<\"NMF algorithm can not work when the row \" << vertex.id() << \" of the matrix contains all zeros\" << std::endl;\n      }\n\n      static gather_type pre_iter(const graph_type::vertex_type & vertex){\n        gather_type ret;\n        ret.pvec = vertex.data().pvec;\n        ret.training_rmse = vertex.data().train_rmse;\n        ret.validation_rmse = vertex.data().validation_rmse;\n        return ret;\n      }\n\n\n      static graphlab::empty signal_left(icontext_type& context,\n          const vertex_type& vertex) {\n        if(vertex.num_out_edges() > 0) context.signal(vertex);\n        return graphlab::empty();\n      } // end of signal_left \n\n      static graphlab::empty signal_right(icontext_type& context,\n          const vertex_type& vertex) {\n        if(vertex.num_in_edges() > 0) context.signal(vertex);\n        return graphlab::empty();\n      } // end of signal_left \n\n  }; // end of nmf vertex program\n\ngather_type count_edges(nmf_vertex_program::icontext_type & context, const graph_type::edge_type& edge) {\n  gather_type ret;\n  if (edge.data().role == edge_data::TRAIN){\n    ret.training_rmse = 1;\n  }\n  else if (edge.data().role == edge_data::VALIDATE){\n    ret.validation_rmse = 1;\n  }\n  if (edge.data().weight < 0)\n    logstream(LOG_FATAL)<<\"Found a negative entry in matirx row \" << edge.source().id() << \" with value: \" << edge.data().weight << std::endl;\n  return ret;\n}\n\n\nstruct prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n     */\n  std::string save_vertex(const vertex_type& vertex) const {\n    return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    if (edge.data().role != edge_data::PREDICT)\n      return \"\";\n\n    std::stringstream strm;\n    const double prediction = \n      edge.source().data().pvec.dot(edge.target().data().pvec);\n    strm << edge.source().id() << '\\t' \n      << -edge.target().id()-SAFE_NEG_OFFSET << '\\t'\n      << prediction << '\\n';\n    return strm.str();\n  }\n}; // end of prediction_saver\n\nstruct linear_model_saver_U {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n     */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() > 0){\n      std::string ret = boost::lexical_cast(vertex.id()) + \") \";\n      for (uint i=0; i< vertex_data::NLATENT; i++)\n        ret += boost::lexical_cast(vertex.data().pvec[i]) + \" \";\n      ret += \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\nstruct linear_model_saver_V {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n     */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() == 0){\n      std::string ret = boost::lexical_cast(-vertex.id()-SAFE_NEG_OFFSET) + \") \";\n      for (uint i=0; i< vertex_data::NLATENT; i++)\n        ret += boost::lexical_cast(vertex.data().pvec[i]) + \" \";\n      ret += \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\n\n\n/**\n * \\brief The graph loader function is a line parser used for\n * distributed graph construction.\n */\ninline bool graph_loader(graph_type& graph, \n    const std::string& filename,\n    const std::string& line) {\n  ASSERT_FALSE(line.empty()); \n\n // Parse the line\n  std::stringstream strm(line);\n  graph_type::vertex_id_type source_id(-1), target_id(-1);\n  float weight(0);\n  strm >> source_id >> target_id;\n\n  if (source_id == graph_type::vertex_id_type(-1) || target_id == graph_type::vertex_id_type(-1)){\n    logstream(LOG_WARNING)<<\"Failed to read input line: \"<< line << \" in file: \"  << filename << \" (or node id is -1). \" << std::endl;\n    return true;\n  }\n\n  // Determine the role of the data\n  edge_data::data_role_type role = edge_data::TRAIN;\n  if(boost::ends_with(filename,\".validate\")) role = edge_data::VALIDATE;\n  else if(boost::ends_with(filename, \".predict\")) role = edge_data::PREDICT;\n \n  // for test files (.predict) no need to read the actual rating value.\n  if(role == edge_data::TRAIN || role == edge_data::VALIDATE){\n    strm >> weight;\n    if (weight < nmf_vertex_program::MINVAL || weight > nmf_vertex_program::MAXVAL)\n      logstream(LOG_FATAL)<<\"Rating values should be between \" << nmf_vertex_program::MINVAL << \" and \" << nmf_vertex_program::MAXVAL << \". Got value: \" << weight << \" [ user: \" << source_id << \" to item: \" < engine_type;\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n\n  // Parse command line options -----------------------------------------------\n  const std::string description = \n    \"Compute the ALS factorization of a matrix.\";\n  graphlab::command_line_options clopts(description);\n  std::string input_dir;\n  std::string predictions;\n  std::string exec_type = \"synchronous\";\n  clopts.attach_option(\"matrix\", input_dir,\n      \"The directory containing the matrix file\");\n  clopts.add_positional(\"matrix\");\n  clopts.attach_option(\"D\", vertex_data::NLATENT,\n      \"Number of latent parameters to use.\");\n  clopts.attach_option(\"engine\", exec_type, \n      \"The engine type synchronous or asynchronous\");\n  clopts.attach_option(\"max_iter\", nmf_vertex_program::MAX_UPDATES,\n      \"The maxumum number of udpates allowed for a vertex\");\n  clopts.attach_option(\"debug\", nmf_vertex_program::debug, \n      \"debug - additional verbose info\"); \n  clopts.attach_option(\"maxval\", nmf_vertex_program::MAXVAL, \"max allowed value\");\n  clopts.attach_option(\"minval\", nmf_vertex_program::MINVAL, \"min allowed value\");\n  clopts.attach_option(\"predictions\", predictions,\n      \"The prefix (folder and filename) to save predictions.\");\n\n  if(!clopts.parse(argc, argv) || input_dir == \"\") {\n    std::cout << \"Error in parsing command line arguments.\" << std::endl;\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  debug = nmf_vertex_program::debug;\n  \n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n  dc.cout() << \"Loading graph.\" << std::endl;\n  graphlab::timer timer; \n  graph_type graph(dc, clopts);  \n  graph.load(input_dir, graph_loader); \n  dc.cout() << \"Loading graph. Finished in \" \n    << timer.current_time() << std::endl;\n  dc.cout() << \"Finalizing graph.\" << std::endl;\n  timer.start();\n  graph.finalize();\n  dc.cout() << \"Finalizing graph. Finished in \" \n    << timer.current_time() << std::endl;\n\n  if (!graph.num_edges() || !graph.num_vertices())\n     logstream(LOG_FATAL)<< \"Failed to load graph. Check your input path: \" << input_dir << std::endl;     \n\n\n\n  dc.cout() \n    << \"========== Graph statistics on proc \" << dc.procid() \n    << \" ===============\"\n    << \"\\n Num vertices: \" << graph.num_vertices()\n    << \"\\n Num edges: \" << graph.num_edges()\n    << \"\\n Num replica: \" << graph.num_replicas()\n    << \"\\n Replica to vertex ratio: \" \n    << float(graph.num_replicas())/graph.num_vertices()\n    << \"\\n --------------------------------------------\" \n    << \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n    << \"\\n Num local vertices: \" << graph.num_local_vertices()\n    << \"\\n Replica to own ratio: \" \n    << (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n    << \"\\n Num local edges: \" << graph.num_local_edges()\n    //<< \"\\n Begin edge id: \" << graph.global_eid(0)\n    << \"\\n Edge balance ratio: \" \n    << float(graph.num_local_edges())/graph.num_edges()\n    << std::endl;\n\n  dc.cout() << \"Creating engine\" << std::endl;\n  engine_type engine(dc, graph, exec_type, clopts);\n\n\n  // Run the NMF ---------------------------------------------------------\n  dc.cout() << \"Running NMF\" << std::endl;\n  dc.cout() << \"(C) Code by Danny Bickson, CMU \" << std::endl;\n  dc.cout() << \"Please send bug reports to danny.bickson@gmail.com\" << std::endl;\n  dc.cout() << \"Time   Training    Validation\" <(count_edges);\n  dc.cout()<<\"Training edges: \" << edge_count.training_rmse << \" validation edges: \" << edge_count.validation_rmse << std::endl;\n\n  graphlab::vertex_set left = graph.select(isuser_node);\n  graphlab::vertex_set right = ~left;\n  graph.transform_vertices(nmf_vertex_program::verify_rows, left);\n\n  graphlab::timer mytimer; mytimer.start();\n\n  for (uint j=0; j< nmf_vertex_program::MAX_UPDATES; j++){\n    x1 = graph.map_reduce_vertices(nmf_vertex_program::pre_iter,right);\n    px = &x1;\n    for (int i=0; i< (int)vertex_data::NLATENT; i++)\n      ASSERT_NE(px->pvec[i], 0);\n    \n    dc.cout()<< std::setw(8) << mytimer.current_time() << \" \" << sqrt(x1.training_rmse/edge_count.training_rmse);\n    if (edge_count.validation_rmse > 0)\n      dc.cout() << \" \" << std::setw(8) << sqrt(x1.validation_rmse/edge_count.validation_rmse) << std::endl;\n    else dc.cout() << std::endl;\n    engine.map_reduce_vertices(nmf_vertex_program::signal_left);\n    engine.start();\n    x1.reset();\n\n    x2 = graph.map_reduce_vertices(nmf_vertex_program::pre_iter,left);\n    px = &x2;\n\n    engine.map_reduce_vertices(nmf_vertex_program::signal_right);\n    engine.start();\n    x2.reset();\n  }\n\n  const double runtime = timer.current_time();\n  dc.cout() << \"----------------------------------------------------------\"\n    << std::endl\n    << \"Final Runtime (seconds):   \" << runtime \n                                        << std::endl\n                                        << \"Updates executed: \" << engine.num_updates() << std::endl\n                                        << \"Update Rate (updates/second): \" \n                                          << engine.num_updates() / runtime << std::endl;\n\n\n  // Make predictions ---------------------------------------------------------\n  if(!predictions.empty()) {\n    std::cout << \"Saving predictions\" << std::endl;\n    const bool gzip_output = false;\n    const bool save_vertices = false;\n    const bool save_edges = true;\n    const size_t threads_per_machine = 1;\n    //save the predictions\n    graph.save(predictions, prediction_saver(),\n        gzip_output, save_vertices, \n        save_edges, threads_per_machine);\n    //save the linear model\n    graph.save(predictions + \".U\", linear_model_saver_U(),\n        gzip_output, save_edges, save_vertices, threads_per_machine);\n    graph.save(predictions + \".V\", linear_model_saver_V(),\n        gzip_output, save_edges, save_vertices, threads_per_machine);\n\n  }\n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // end of main\n\n\n\n"
  },
  {
    "path": "toolkits/collaborative_filtering/printouts.hpp",
    "content": "#ifndef PRINTOUTS\n#define PRINTOUTS\n#define MAX_PRINTOUT_LEN 25\n\nbool absolute_value = true;\n\ninline double fabs2(double val){\n  if (absolute_value)\n    return fabs(val);\n  else return val;\n}\n\n\ngather_type collect_vec(const graph_type::vertex_type& vertex) {\n  assert(pcurrent);\n  assert(pcurrent->offset >= 0 && pcurrent->offset < vertex.data().pvec.size());\n  int len=std::min(pcurrent->end-pcurrent->start,MAX_PRINTOUT_LEN);\n  assert(len > 0);\n  gather_type ret;\n  ret.pvec = vec::Zero(len);\n  if (vertex.id() >= (uint)pcurrent->start && vertex.id() < (uint)pcurrent->start +len)\n    ret.pvec[vertex.id() - pcurrent->start] = vertex.data().pvec[pcurrent->offset];\n  return ret;\n}\n\n\nvoid print_vec(const char * name, const DistVec & vec, bool high = false){\n if (!debug)\n   return;\n  int i;\n printf(\"%s[%d]\\n\", name, vec.offset);\n pcurrent = (DistVec*)&vec;\n int len=std::min(vec.end-vec.start,MAX_PRINTOUT_LEN);\n gather_type ret = pgraph->map_reduce_vertices(collect_vec);\n for (i= 0; i< len; i++){\n  if (high)\n    printf(\"%15.15lg \", fabs2(ret.pvec[i]));\n  else\n    printf(\"%.5lg \", fabs2(ret.pvec[i]));\n  }\n printf(\"\\n\");\n}\n\nvoid print_vec(const char * name, const vec & pvec, bool high = false){\n if (!debug)\n   return;\n  printf(\"%s\\n\", name);\n for (int i= 0; i< std::min((int)pvec.size(), MAX_PRINTOUT_LEN); i++){\n  if (high)\n   printf(\"%15.15lg \", fabs2(pvec[i]));\n  else\n   printf(\"%.5lg \", fabs2(pvec[i]));\n  }\n printf(\"\\n\");\n}\nvoid print_mat(const char * name, const mat & pmat, bool high = false){\n if (!debug)\n   return;\n  printf(\"%s\\n\", name);\n mat pmat2 = transpose((mat&)pmat);\n if (pmat2.cols() == 1)\n    pmat2 = pmat2.transpose();\n for (int i= 0; i< std::min((int)pmat2.rows(), MAX_PRINTOUT_LEN); i++){\n  for (int j=0; j< std::min((int)pmat2.cols(), MAX_PRINTOUT_LEN); j++){\n    if (high)\n      printf(\"%15.15lg \", fabs2(get_val(pmat2, i, j)));\n    else\n     printf(\"%.5lg \", fabs2(get_val(pmat2, i, j)));\n  }\n  printf(\"\\n\");\n  \n  }\n}\n\nvoid print_vec_pos(std::string name, vec & v, int i){\n if (!debug)\n   return;\n   if (i == -1)\n    printf(\"%s\\n\", name.c_str());\n  else {\n    printf(\"%s[%d]: %.5lg\\n\", name.c_str(), i, fabs(v[i]));\n    return;\n  }\n  for (int j=0; j< std::min((int)v.size(),MAX_PRINTOUT_LEN); j++){\n   printf(\"%.5lg\", fabs2(v(j)));\n   if (v.size() > 1)\n    printf(\" \");\n  }\n  printf(\"\\n\");\n}\n\n\n#define PRINT_VEC(a) print_vec(#a,a,0)\n#define PRINT_VEC2(a,b) print_vec(a,b,0)\n#define PRINT_VEC3(a,b,c) print_vec_pos(a,b,c)\n#define PRINT_VEC2_HIGH(a,i) print_vec(#a,a[i],1)\n#define PRINT_INT(a) if (debug) printf(\"%s: %d\\n\", #a, a);\n#define PRINT_NAMED_INT(a,b) if (debug) printf(\"%s: %d\\n\",a, b);\n#define PRINT_DBL(a) if (debug) printf(\"%s: %.5lg\\n\", #a, a);\n#define PRINT_NAMED_DBL(a,b) if (debug) printf(\"%s: %.5lg\\n\", a, b);\n#define PRINT_MAT(a) print_mat(#a, a, 0);\n#define PRINT_MAT2(a,b) print_mat(a,b,0);\n#endif\n"
  },
  {
    "path": "toolkits/collaborative_filtering/sgd.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * \\file\n * \n * \\brief The main file for the SGD matrix factorization algorithm.\n *\n * This file contains the main body of the SGD matrix factorization\n * algorithm. \n */\n\n#include \n#include \n\n#include \n#include \"eigen_serialization.hpp\"\n#include \n\n\ntypedef Eigen::VectorXd vec_type;\ntypedef Eigen::MatrixXd mat_type;\n\n//when using negative node id range, we are not allowed to use\n//0 and 1 so we add 2.\nconst static int SAFE_NEG_OFFSET=2;\nstatic bool debug;\nint iter = 0;\n\nbool isuser(uint node){\n\treturn ((int)node) >= 0;\n}\n\n/** \n * \\ingroup toolkit_matrix_pvecization\n *\n * \\brief the vertex data type which contains the latent pvec.\n *\n * Each row and each column in the matrix corresponds to a different\n * vertex in the SGD graph.  Associated with each vertex is a pvec\n * (vector) of latent parameters that represent that vertex.  The goal\n * of the SGD algorithm is to find the values for these latent\n * parameters such that the non-zero entries in the matrix can be\n * predicted by taking the dot product of the row and column pvecs.\n */\nstruct vertex_data {\n\t/**\n\t * \\brief A shared \"constant\" that specifies the number of latent\n\t * values to use.\n\t */\n\tstatic size_t NLATENT;\n\t/** \\brief The latent pvec for this vertex */\n\tvec_type pvec;\n\n\tint nupdates;\n\t/** \n\t * \\brief Simple default constructor which randomizes the vertex\n\t *  data \n\t */\n\tvertex_data() : nupdates(0) { if (debug) pvec = vec_type::Ones(NLATENT); else randomize(); } \n\t/** \\brief Randomizes the latent pvec */\n\tvoid randomize() { pvec.resize(NLATENT); pvec.setRandom(); }\n\t/** \\brief Save the vertex data to a binary archive */\n\tvoid save(graphlab::oarchive& arc) const { \n\t\tarc << nupdates << pvec;\n\t}\n\t/** \\brief Load the vertex data from a binary archive */\n\tvoid load(graphlab::iarchive& arc) { \n\t\tarc >> nupdates >> pvec;\n\t}\n}; // end of vertex data\n\n\n/**\n * \\brief The edge data stores the entry in the matrix.\n *\n * In addition the edge data sgdo stores the most recent error estimate.\n */\nstruct edge_data : public graphlab::IS_POD_TYPE {\n\t/**\n\t * \\brief The type of data on the edge;\n\t *\n\t * \\li *Train:* the observed value is correct and used in training\n\t * \\li *Validate:* the observed value is correct but not used in training\n\t * \\li *Predict:* The observed value is not correct and should not be\n\t *        used in training.\n\t */\n\tenum data_role_type { TRAIN, VALIDATE, PREDICT  };\n\n\t/** \\brief the observed value for the edge */\n\tfloat obs;\n\n\t/** \\brief The train/validation/test designation of the edge */\n\tdata_role_type role;\n\n\t/** \\brief basic initialization */\n\tedge_data(float obs = 0, data_role_type role = PREDICT) :\n\t\tobs(obs), role(role) { }\n\n}; // end of edge data\n\n\n/**\n * \\brief The graph type is defined in terms of the vertex and edge\n * data.\n */ \ntypedef graphlab::distributed_graph graph_type;\n\n#include \"implicit.hpp\"\n\nstats_info count_edges(const graph_type::edge_type & edge){\n\tstats_info ret;\n\n\tif (edge.data().role == edge_data::TRAIN)\n\t\tret.training_edges = 1;\n\telse if (edge.data().role == edge_data::VALIDATE)\n\t\tret.validation_edges = 1;\n\tret.max_user = (size_t)edge.source().id();\n\tret.max_item = (-edge.target().id()-SAFE_NEG_OFFSET);\n\treturn ret;\n}\n\n\ndouble extract_l2_error(const graph_type::edge_type & edge);\n\n\n/**\n * \\brief Given a vertex and an edge return the other vertex in the\n * edge.\n */\ninline graph_type::vertex_type\nget_other_vertex(graph_type::edge_type& edge, \n\t\tconst graph_type::vertex_type& vertex) {\n\treturn vertex.id() == edge.source().id()? edge.target() : edge.source();\n}; // end of get_other_vertex\n\n\n\n\n\n/**\n *\n */\nclass gather_type {\n\tpublic:\n\t\t/**\n\t\t * \\brief Stores the current sum of nbr.pvec.transpose() *\n\t\t * nbr.pvec\n\t\t */\n\n\t\t/**\n\t\t * \\brief Stores the current sum of nbr.pvec * edge.obs\n\t\t */\n\t\tvec_type pvec;\n\t\t/** \\brief basic default constructor */\n\t\tgather_type() { }\n\n\t\t/**\n\t\t * \\brief This constructor computes XtX and Xy and stores the result\n\t\t * in XtX and Xy\n\t\t */\n\t\tgather_type(const vec_type& X) {\n\t\t\tpvec = X;\n\t\t} // end of constructor for gather type\n\n\t\t/** \\brief Save the values to a binary archive */\n\t\tvoid save(graphlab::oarchive& arc) const { arc << pvec; }\n\n\t\t/** \\brief Read the values from a binary archive */\n\t\tvoid load(graphlab::iarchive& arc) { arc >> pvec; }  \n\n\t\t/** \n\t\t */\n\t\tgather_type& operator+=(const gather_type& other) {\n\t\t\tif (pvec.size() == 0){\n\t\t\t\tpvec = other.pvec;\n\t\t\t\treturn *this;\n\t\t\t}\n\t\t\telse if (other.pvec.size() == 0)\n\t\t\t\treturn *this;\n\t\t\tpvec += other.pvec;\n\t\t\treturn *this;\n\t\t} // end of operator+=\n\n}; // end of gather type\n\ntypedef vec_type message_type;\n\nbool isuser_node(const graph_type::vertex_type& vertex){\n\treturn isuser(vertex.id());\n}\n\n/**\n * SGD vertex program type\n */ \nclass sgd_vertex_program : \n\tpublic graphlab::ivertex_program {\n\t\tpublic:\n\t\t\t/** The convergence tolerance */\n\t\t\tstatic double TOLERANCE;\n\t\t\tstatic double LAMBDA;\n\t\t\tstatic double GAMMA;\n\t\t\tstatic double MAXVAL;\n\t\t\tstatic double MINVAL;\n\t\t\tstatic double STEP_DEC;\n\t\t\tstatic bool debug;\n\t\t\tstatic size_t MAX_UPDATES;\n\t\t\tvec_type pmsg;\n\n\t\t\tvoid save(graphlab::oarchive& arc) const { \n\t\t\t\tarc << pmsg;\n\t\t\t}\n\t\t\t/** \\brief Load the vertex data from a binary archive */\n\t\t\tvoid load(graphlab::iarchive& arc) { \n\t\t\t\tarc >> pmsg;\n\t\t\t}\n\n\t\t\t/** The set of edges to gather along */\n\t\t\tedge_dir_type gather_edges(icontext_type& context, \n\t\t\t\t\tconst vertex_type& vertex) const { \n\t\t\t\treturn graphlab::ALL_EDGES; \n\t\t\t}; // end of gather_edges \n\n\n\t\t\tgather_type gather(icontext_type& context, const vertex_type& vertex, \n\t\t\t\t\tedge_type& edge) const {\n\n\t\t\t\tvec_type delta, other_delta;\n\t\t\t\t//this is user node\n\t\t\t\tif (vertex.num_in_edges() == 0){\n\t\t\t\t\tvertex_type my_vertex(vertex);\n\t\t\t\t\t//get a copy of the item node\n\t\t\t\t\tvertex_type other_vertex(get_other_vertex(edge, vertex));\n\t\t\t\t\t//compute the current prediction by computing a dot production of user and item nodes\n\t\t\t\t\tdouble pred = vertex.data().pvec.dot(other_vertex.data().pvec);\n\t\t\t\t\t//truncte predictions into allowed range\n\t\t\t\t\tpred = std::min(pred, sgd_vertex_program::MAXVAL);\n\t\t\t\t\tpred = std::max(pred, sgd_vertex_program::MINVAL); \n\t\t\t\t\t//compute the prediction error \n\t\t\t\t\tconst float err = edge.data().obs - pred;\n\t\t\t\t\tif (debug)\n\t\t\t\t\t\tstd::cout<<\"entering edge \" << (int)edge.source().id() << \":\" << (int)edge.target().id() << \" err: \" << err << \" rmse: \" << err*err < TOLERANCE && other_vertex.data().nupdates < MAX_UPDATES) \n\t\t\t\t\t\t\tcontext.signal(other_vertex, other_delta);\n\t\t\t\t\t}\n\t\t\t\t} \n\t\t\t\treturn gather_type(delta);\n\t\t\t} // end of gather function\n\n\t\t\tvoid init(icontext_type& context,\n\t\t\t\t\tconst vertex_type& vertex,\n\t\t\t\t\tconst message_type& msg) {\n\t\t\t\t//if this is an item node, store the change in the gradient (sum of changes) to be\n\t\t\t\t//applied in the apply() function\n\t\t\t\tif (vertex.num_in_edges() > 0){\n\t\t\t\t\tpmsg = msg;\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tvoid apply(icontext_type& context, vertex_type& vertex,\n\t\t\t\t\tconst gather_type& sum) {\n\n\t\t\t\tvertex_data& vdata = vertex.data(); \n\t\t\t\t//this is a user node, update the gradient using the comulative sum of gradient updates computed in gather\n\t\t\t\tif (sum.pvec.size() > 0){\n\t\t\t\t\tvdata.pvec += sum.pvec; \n\t\t\t\t\tassert(vertex.num_in_edges() == 0);\n\t\t\t\t}\n\t\t\t\t//if this is an item node, update the gradient using the received sum from the init() function\n\t\t\t\telse if (pmsg.size() > 0){\n\t\t\t\t\tvdata.pvec += pmsg;\n\t\t\t\t\tassert(vertex.num_out_edges() == 0); \n\t\t\t\t}\n\t\t\t\t++vdata.nupdates;\n\t\t\t} // end of apply\n\n\t\t\t/** The edges to scatter along */\n\t\t\tedge_dir_type scatter_edges(icontext_type& context,\n\t\t\t\t\tconst vertex_type& vertex) const { \n\t\t\t\treturn graphlab::ALL_EDGES; \n\t\t\t}; // end of scatter edges\n\n\t\t\t/** Scatter reschedules neighbors */  \n\t\t\tvoid scatter(icontext_type& context, const vertex_type& vertex, \n\t\t\t\t\tedge_type& edge) const {\n\t\t\t\tedge_data& edata = edge.data();\n\t\t\t\tif(edata.role == edge_data::TRAIN) {\n\t\t\t\t\tconst vertex_type other_vertex = get_other_vertex(edge, vertex);\n\t\t\t\t\t// Reschedule neighbors ------------------------------------------------\n\t\t\t\t\tif(other_vertex.data().nupdates < MAX_UPDATES) \n\t\t\t\t\t\tcontext.signal(other_vertex, vec_type::Zero(vertex_data::NLATENT));\n\t\t\t\t}\n\t\t\t} // end of scatter function\n\n\n\t\t\t/**\n\t\t\t * \\brief Signal all vertices on one side of the bipartite graph\n\t\t\t */\n\t\t\tstatic graphlab::empty signal_left(icontext_type& context,\n\t\t\t\t\tvertex_type& vertex) {\n\t\t\t\tif(vertex.num_out_edges() > 0) context.signal(vertex, vec_type::Zero(vertex_data::NLATENT));\n\t\t\t\treturn graphlab::empty();\n\t\t\t} // end of signal_left \n\n\t}; // end of sgd vertex program\n\n\nstruct error_aggregator : public graphlab::IS_POD_TYPE {\n\ttypedef sgd_vertex_program::icontext_type icontext_type;\n\ttypedef graph_type::edge_type edge_type;\n\tdouble train_error, validation_error;\n\n\terror_aggregator() : \n\t\ttrain_error(0), validation_error(0){ }\n\terror_aggregator& operator+=(const error_aggregator& other) {\n\t\ttrain_error += other.train_error;\n\t\tassert(!std::isnan(train_error));\n\t\tvalidation_error += other.validation_error;\n\t\treturn *this;\n\t}\n\tstatic error_aggregator map(icontext_type& context, const graph_type::edge_type& edge) {\n\t\terror_aggregator agg;\n\t\tif (edge.data().role == edge_data::TRAIN){\n\t\t\tif (isuser_node(edge.source())) \n\t\t\t\tagg.train_error = extract_l2_error(edge); \n\t\t\tassert(!std::isnan(agg.train_error));\n\t\t}\n\t\telse if (edge.data().role == edge_data::VALIDATE){\n\t\t\tif (isuser_node(edge.source())) \n\t\t\t\tagg.validation_error = extract_l2_error(edge); \n\t\t}\n\t\treturn agg;\n\t}\n\n\n\tstatic void finalize(icontext_type& context, const error_aggregator& agg) {\n\t\titer++;\n\t\tif (iter%2 == 0)\n\t\t\treturn; \n\t\tconst double train_error = std::sqrt(agg.train_error / info.training_edges);\n\t\tassert(!std::isnan(train_error));\n\t\tcontext.cout() << std::setw(8) << context.elapsed_seconds()  << \"  \" << std::setw(8) << train_error;\n\t\tif(info.validation_edges > 0) {\n\t\t\tconst double validation_error = \n\t\t\t\tstd::sqrt(agg.validation_error / info.validation_edges);\n\t\t\tcontext.cout() << \"   \" << std::setw(8) << validation_error; \n\t\t}\n\t\tcontext.cout() << std::endl;\n\t\tsgd_vertex_program::GAMMA *= sgd_vertex_program::STEP_DEC;\n\t}\n}; // end of error aggregator\n\n/**\n * \\brief Given an edge compute the error associated with that edge\n */\ndouble extract_l2_error(const graph_type::edge_type & edge) {\n\tdouble pred = \n\t\tedge.source().data().pvec.dot(edge.target().data().pvec);\n\tpred = std::min(sgd_vertex_program::MAXVAL, pred);\n\tpred = std::max(sgd_vertex_program::MINVAL, pred);\n\tdouble rmse = (edge.data().obs - pred) * (edge.data().obs - pred);\n\tassert(rmse <= pow(sgd_vertex_program::MAXVAL-sgd_vertex_program::MINVAL,2));\n\treturn rmse;\n} // end of extract_l2_error\n\n\nstruct prediction_saver {\n\ttypedef graph_type::vertex_type vertex_type;\n\ttypedef graph_type::edge_type   edge_type;\n\t/* save the linear model, using the format:\n\t   nodeid) factor1 factor2 ... factorNLATENT \\n\n\t */\n\tstd::string save_vertex(const vertex_type& vertex) const {\n\t\treturn \"\";\n\t}\n\tstd::string save_edge(const edge_type& edge) const {\n\t\tif (edge.data().role != edge_data::PREDICT)\n\t\t\treturn \"\";\n\n\t\tstd::stringstream strm;\n\t\tconst double prediction = \n\t\t\tedge.source().data().pvec.dot(edge.target().data().pvec);\n\t\tstrm << edge.source().id() << '\\t' \n\t\t\t<< -edge.target().id()-SAFE_NEG_OFFSET << '\\t'\n\t\t\t<< prediction << '\\n';\n\t\treturn strm.str();\n\t}\n}; // end of prediction_saver\n\nstruct linear_model_saver_U {\n\ttypedef graph_type::vertex_type vertex_type;\n\ttypedef graph_type::edge_type   edge_type;\n\t/* save the linear model, using the format:\n\t   nodeid) factor1 factor2 ... factorNLATENT \\n\n\t */\n\tstd::string save_vertex(const vertex_type& vertex) const {\n\t\tif (vertex.num_out_edges() > 0){\n\t\t\tstd::string ret = boost::lexical_cast(vertex.id()) + \") \";\n\t\t\tfor (uint i=0; i< vertex_data::NLATENT; i++)\n\t\t\t\tret += boost::lexical_cast(vertex.data().pvec[i]) + \" \";\n\t\t\tret += \"\\n\";\n\t\t\treturn ret;\n\t\t}\n\t\telse return \"\";\n\t}\n\tstd::string save_edge(const edge_type& edge) const {\n\t\treturn \"\";\n\t}\n}; \n\nstruct linear_model_saver_V {\n\ttypedef graph_type::vertex_type vertex_type;\n\ttypedef graph_type::edge_type   edge_type;\n\t/* save the linear model, using the format:\n\t   nodeid) factor1 factor2 ... factorNLATENT \\n\n\t */\n\tstd::string save_vertex(const vertex_type& vertex) const {\n\t\tif (vertex.num_out_edges() == 0){\n\t\t\tstd::string ret = boost::lexical_cast(-vertex.id()-SAFE_NEG_OFFSET) + \") \";\n\t\t\tfor (uint i=0; i< vertex_data::NLATENT; i++)\n\t\t\t\tret += boost::lexical_cast(vertex.data().pvec[i]) + \" \";\n\t\t\tret += \"\\n\";\n\t\t\treturn ret;\n\t\t}\n\t\telse return \"\";\n\t}\n\tstd::string save_edge(const edge_type& edge) const {\n\t\treturn \"\";\n\t}\n}; \n\n\n\n/**\n * \\brief The graph loader function is a line parser used for\n * distributed graph construction.\n */\ninline bool graph_loader(graph_type& graph, \n\t\tconst std::string& filename,\n\t\tconst std::string& line) {\n\n\t// Parse the line\n\tstd::stringstream strm(line);\n\tgraph_type::vertex_id_type source_id(-1), target_id(-1);\n\tfloat obs(0);\n\tstrm >> source_id >> target_id;\n\n\tif (source_id == graph_type::vertex_id_type(-1) || target_id == graph_type::vertex_id_type(-1)){\n\t\tlogstream(LOG_WARNING)<<\"Failed to read input line: \"<< line << \" in file: \"  << filename << \" (or node id is -1). \" << std::endl;\n\t\treturn true;\n\t}\n\n\t// Determine the role of the data\n\tedge_data::data_role_type role = edge_data::TRAIN;\n\tif(boost::ends_with(filename,\".validate\")) role = edge_data::VALIDATE;\n\telse if(boost::ends_with(filename, \".predict\")) role = edge_data::PREDICT;\n\n\tif(role == edge_data::TRAIN || role == edge_data::VALIDATE){\n\t\tstrm >> obs;\n\t\tif (obs < sgd_vertex_program::MINVAL || obs > sgd_vertex_program::MAXVAL){\n\t\t\tlogstream(LOG_WARNING)<<\"Rating values should be between \" << sgd_vertex_program::MINVAL << \" and \" << sgd_vertex_program::MAXVAL << \". Got value: \" << obs << \" [ user: \" << source_id << \" to item: \" < engine_type;\n\nint main(int argc, char** argv) {\n\tglobal_logger().set_log_level(LOG_INFO);\n\tglobal_logger().set_log_to_console(true);\n\n\t// Parse command line options -----------------------------------------------\n\tconst std::string description = \n\t\t\"Compute the SGD factorization of a matrix.\";\n\tgraphlab::command_line_options clopts(description);\n\tstd::string input_dir;\n\tstd::string predictions;\n\tsize_t interval = 0;\n\tstd::string exec_type = \"synchronous\";\n\tclopts.attach_option(\"matrix\", input_dir,\n\t\t\t\"The directory containing the matrix file\");\n\tclopts.add_positional(\"matrix\");\n\tclopts.attach_option(\"D\", vertex_data::NLATENT,\n\t\t\t\"Number of latent parameters to use.\");\n\tclopts.attach_option(\"engine\", exec_type, \n\t\t\t\"The engine type synchronous or asynchronous\");\n\tclopts.attach_option(\"max_iter\", sgd_vertex_program::MAX_UPDATES,\n\t\t\t\"The maxumum number of udpates allowed for a vertex\");\n\tclopts.attach_option(\"lambda\", sgd_vertex_program::LAMBDA, \n\t\t\t\"SGD regularization weight\"); \n\tclopts.attach_option(\"gamma\", sgd_vertex_program::GAMMA, \n\t\t\t\"SGD step size\"); \n\tclopts.attach_option(\"debug\", sgd_vertex_program::debug, \n\t\t\t\"debug - additional verbose info\"); \n\tclopts.attach_option(\"tol\", sgd_vertex_program::TOLERANCE,\n\t\t\t\"residual termination threshold\");\n\tclopts.attach_option(\"maxval\", sgd_vertex_program::MAXVAL, \"max allowed value\");\n\tclopts.attach_option(\"minval\", sgd_vertex_program::MINVAL, \"min allowed value\");\n\tclopts.attach_option(\"step_dec\", sgd_vertex_program::STEP_DEC, \"multiplicative step decrement\");\n\tclopts.attach_option(\"interval\", interval, \n\t\t\t\"The time in seconds between error reports\");\n\tclopts.attach_option(\"predictions\", predictions,\n\t\t\t\"The prefix (folder and filename) to save predictions.\");\n\n\tparse_implicit_command_line(clopts);\n\n\tif(!clopts.parse(argc, argv) || input_dir == \"\") {\n\t\tstd::cout << \"Error in parsing command line arguments.\" << std::endl;\n\t\tclopts.print_description();\n\t\treturn EXIT_FAILURE;\n\t}\n\tdebug = sgd_vertex_program::debug;\n\t//  omp_set_num_threads(clopts.get_ncpus());\n\t///! Initialize control plain using mpi\n\tgraphlab::mpi_tools::init(argc, argv);\n\tgraphlab::distributed_control dc;\n\n\tdc.cout() << \"Loading graph.\" << std::endl;\n\tgraphlab::timer timer; \n\tgraph_type graph(dc, clopts);  \n\tgraph.load(input_dir, graph_loader); \n\tdc.cout() << \"Loading graph. Finished in \" \n\t\t<< timer.current_time() << std::endl;\n\n\tif (dc.procid() == 0) \n\t\tadd_implicit_edges(implicitratingtype, graph, dc);\n\n\n\tdc.cout() << \"Finalizing graph.\" << std::endl;\n\ttimer.start();\n\tgraph.finalize();\n\tdc.cout() << \"Finalizing graph. Finished in \" \n\t\t<< timer.current_time() << std::endl;\n\n        if (!graph.num_edges() || !graph.num_vertices())\n          logstream(LOG_FATAL)<< \"Failed to load graph. Check your input path: \" << input_dir << std::endl;     \n\n\n\n\tdc.cout() \n\t\t<< \"========== Graph statistics on proc \" << dc.procid() \n\t\t<< \" ===============\"\n\t\t<< \"\\n Num vertices: \" << graph.num_vertices()\n\t\t<< \"\\n Num edges: \" << graph.num_edges()\n\t\t<< \"\\n Num replica: \" << graph.num_replicas()\n\t\t<< \"\\n Replica to vertex ratio: \" \n\t\t<< float(graph.num_replicas())/graph.num_vertices()\n\t\t<< \"\\n --------------------------------------------\" \n\t\t<< \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n\t\t<< \"\\n Num local vertices: \" << graph.num_local_vertices()\n\t\t<< \"\\n Replica to own ratio: \" \n\t\t<< (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n\t\t<< \"\\n Num local edges: \" << graph.num_local_edges()\n\t\t//<< \"\\n Begin edge id: \" << graph.global_eid(0)\n\t\t<< \"\\n Edge balance ratio: \" \n\t\t<< float(graph.num_local_edges())/graph.num_edges()\n\t\t<< std::endl;\n\n\tdc.cout() << \"Creating engine\" << std::endl;\n\tengine_type engine(dc, graph, exec_type, clopts);\n\n\t// Add error reporting to the engine\n\tconst bool success = engine.add_edge_aggregator\n\t\t(\"error\", error_aggregator::map, error_aggregator::finalize) &&\n\t\tengine.aggregate_periodic(\"error\", interval);\n\tASSERT_TRUE(success);\n\n\n\t// Signal all vertices on the vertices on the left (libersgd) \n\tengine.map_reduce_vertices(sgd_vertex_program::signal_left);\n\tinfo = graph.map_reduce_edges(count_edges);\n\tdc.cout()<<\"Training edges: \" << info.training_edges << \" validation edges: \" << info.validation_edges << std::endl;\n\n\n\t// Run the PageRank ---------------------------------------------------------\n\tdc.cout() << \"Running SGD\" << std::endl;\n\tdc.cout() << \"(C) Code by Danny Bickson, CMU \" << std::endl;\n\tdc.cout() << \"Please send bug reports to danny.bickson@gmail.com\" << std::endl;\n\tdc.cout() << \"Time   Training    Validation\" <\n#define EIGEN_DONT_PARALLELIZE //eigen parallel for loop interfers with ours.\n\n#include \n#include \n#include \n#include \n#include \n\n\n\n\n// This file defines the serialization code for the eigen types.\n#include \"eigen_serialization.hpp\"\n#include \"eigen_wrapper.hpp\"\n#include \"stats.hpp\"\n#include \n#include \n\n\n#include \n\nconst int SAFE_NEG_OFFSET = 2; //add 2 to negative node id\n//to prevent -0 and -1 which arenot allowed\n\n/**\n * \\brief We use the eigen library's vector type to represent\n * mathematical vectors.\n */\ntypedef Eigen::VectorXd vec;\ntypedef Eigen::VectorXi ivec;\n\n/**\n * \\brief We use the eigen library's matrix type to represent\n * matrices.\n */\ntypedef Eigen::MatrixXd mat;\n#include \"cosamp.hpp\"\n\n//algorithm run modes\nenum {\n  SPARSE_USR_FACTOR = 1, SPARSE_ITM_FACTOR = 2, SPARSE_BOTH_FACTORS = 3\n};\n\nint algorithm = SPARSE_USR_FACTOR;\ndouble user_sparsity = 0.8;\ndouble movie_sparsity = 0.8;\n\n\n\n/** \n * \\ingroup toolkit_matrix_factorization\n *\n * \\brief the vertex data type which contains the latent factor.\n *\n * Each row and each column in the matrix corresponds to a different\n * vertex in the ALS graph.  Associated with each vertex is a factor\n * (vector) of latent parameters that represent that vertex.  The goal\n * of the ALS algorithm is to find the values for these latent\n * parameters such that the non-zero entries in the matrix can be\n * predicted by taking the dot product of the row and column factors.\n */\nstruct vertex_data {\n  /**\n   * \\brief A shared \"constant\" that specifies the number of latent\n   * values to use.\n   */\n  static size_t NLATENT;\n  /** \\brief The number of times this vertex has been updated. */\n  uint32_t nupdates;\n  /** \\brief The most recent L1 change in the factor value */\n  float residual; //! how much the latent value has changed\n  /** \\brief The latent factor for this vertex */\n  vec factor;\n  /** \n   * \\brief Simple default constructor which randomizes the vertex\n   *  data \n   */\n  vertex_data() : nupdates(0), residual(1) { randomize(); } \n  /** \\brief Randomizes the latent factor */\n  void randomize() { factor.resize(NLATENT); factor.setRandom(); }\n  /** \\brief Save the vertex data to a binary archive */\n  void save(graphlab::oarchive& arc) const { \n    arc << nupdates << residual << factor;        \n  }\n  /** \\brief Load the vertex data from a binary archive */\n  void load(graphlab::iarchive& arc) { \n    arc >> nupdates >> residual >> factor;\n  }\n}; // end of vertex data\n\n\nsize_t vertex_data::NLATENT = 20;\n\n/**\n * \\brief The edge data stores the entry in the matrix.\n *\n * In addition the edge data also stores the most recent error estimate.\n */\nstruct edge_data : public graphlab::IS_POD_TYPE {\n  /**\n   * \\brief The type of data on the edge;\n   *\n   * \\li *Train:* the observed value is correct and used in training\n   * \\li *Validate:* the observed value is correct but not used in training\n   * \\li *Predict:* The observed value is not correct and should not be\n   *        used in training.\n   */\n  enum data_role_type { TRAIN, VALIDATE, PREDICT  };\n\n  /** \\brief the observed value for the edge */\n  float obs;\n\n  /** \\brief The train/validation/test designation of the edge */\n  data_role_type role;\n\n  /** \\brief basic initialization */\n  edge_data(float obs = 0, data_role_type role = PREDICT) :\n    obs(obs), role(role) { }\n\n}; // end of edge data\n\n\n/**\n * \\brief The graph type is defined in terms of the vertex and edge\n * data.\n */ \ntypedef graphlab::distributed_graph graph_type;\n\n#include \"implicit.hpp\"\n\nstats_info count_edges(const graph_type::edge_type & edge){\n  stats_info ret;\n\n  if (edge.data().role == edge_data::TRAIN)\n     ret.training_edges = 1;\n  else if (edge.data().role == edge_data::VALIDATE)\n     ret.validation_edges = 1;\n  ret.max_user = (size_t)edge.source().id();\n  ret.max_item = (size_t)edge.target().id();\n  return ret;\n}\n\n\n/**\n * \\brief Given a vertex and an edge return the other vertex in the\n * edge.\n */\ninline graph_type::vertex_type\nget_other_vertex(graph_type::edge_type& edge, \n                 const graph_type::vertex_type& vertex) {\n  return vertex.id() == edge.source().id()? edge.target() : edge.source();\n}; // end of get_other_vertex\n\n\n\n\n/**\n * \\brief The gather type used to construct XtX and Xty needed for the ALS\n * update\n *\n * To compute the ALS update we need to compute the sum of \n * \\code\n *  sum: XtX = nbr.factor.transpose() * nbr.factor \n *  sum: Xy  = nbr.factor * edge.obs\n * \\endcode\n * For each of the neighbors of a vertex. \n *\n * To do this in the Gather-Apply-Scatter model the gather function\n * computes and returns a pair consisting of XtX and Xy which are then\n * added. The gather type represents that tuple and provides the\n * necessary gather_type::operator+= operation.\n *\n */\nclass gather_type {\npublic:\n  /**\n   * \\brief Stores the current sum of nbr.factor.transpose() *\n   * nbr.factor\n   */\n  mat XtX;\n\n  /**\n   * \\brief Stores the current sum of nbr.factor * edge.obs\n   */\n  vec Xy;\n\n  /** \\brief basic default constructor */\n  gather_type() { }\n\n  /**\n   * \\brief This constructor computes XtX and Xy and stores the result\n   * in XtX and Xy\n   */\n  gather_type(const vec& X, const double y) :\n    XtX(X.size(), X.size()), Xy(X.size()) {\n    XtX = X * X.transpose();\n    Xy = X * y;\n  } // end of constructor for gather type\n\n  /** \\brief Save the values to a binary archive */\n  void save(graphlab::oarchive& arc) const { arc << XtX << Xy; }\n\n  /** \\brief Read the values from a binary archive */\n  void load(graphlab::iarchive& arc) { arc >> XtX >> Xy; }  \n\n  /** \n   * \\brief Computes XtX += other.XtX and Xy += other.Xy updating this\n   * tuples value\n   */\n  gather_type& operator+=(const gather_type& other) {\n    if(other.Xy.size() == 0) {\n      ASSERT_EQ(other.XtX.rows(), 0);\n      ASSERT_EQ(other.XtX.cols(), 0);\n    } else {\n      if(Xy.size() == 0) {\n        ASSERT_EQ(XtX.rows(), 0); \n        ASSERT_EQ(XtX.cols(), 0);\n        XtX = other.XtX; Xy = other.Xy;\n      } else {\n        XtX.triangularView() += other.XtX;  \n        Xy += other.Xy;\n      }\n    }\n    return *this;\n  } // end of operator+=\n\n}; // end of gather type\n\n\n\n/**\n * \\brief ALS vertex program implements the alternating least squares\n * algorithm in the Gather-Apply-Scatter abstraction.\n *\n * The ALS update treats adjacent vertices (rows or columns) as \"X\"\n * (independent) values and the edges (matrix entries) as observed \"y\"\n * (dependent) values and then updates the current vertex value as a\n * weight \"w\" such that:\n *\n *    y = X * w + noise\n *\n * This is accomplished using the following equation:\n *\n *    w = inv(X' * X) * (X * y)\n *\n * We implement this in the Gather-Apply-Scatter model by:\n *\n *  1) Gather: returns the tuple (X' * X, X * y)\n *     Sum:   (aX' * aX, aX * ay) + (bX' * bX, bX * by) = \n *                 (aX' * aX + bX' * bX, aX * ay + bX * by)\n *\n *  2) Apply: Solves  inv(X' * X) * (X * y)\n *\n *  3) Scatter: schedules the update of adjacent vertices if this\n *      vertex has changed sufficiently and the edge is not well\n *      predicted.\n *\n * \n */ \nclass als_vertex_program : \n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\npublic:\n  /** The convergence tolerance */\n  static double TOLERANCE;\n  static double LAMBDA;\n  static size_t MAX_UPDATES;\n  static double MAXVAL;\n  static double MINVAL;\n \n  /** The set of edges to gather along */\n  edge_dir_type gather_edges(icontext_type& context, \n                             const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of gather_edges \n\n  /** The gather function computes XtX and Xy */\n  gather_type gather(icontext_type& context, const vertex_type& vertex, \n                     edge_type& edge) const {\n    if(edge.data().role == edge_data::TRAIN) {\n      const vertex_type other_vertex = get_other_vertex(edge, vertex);\n      return gather_type(other_vertex.data().factor, edge.data().obs);\n    } else return gather_type();\n  } // end of gather function\n\n  /** apply collects the sum of XtX and Xy */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& sum) {\n    // Get and reset the vertex data\n    vertex_data& vdata = vertex.data(); \n    // Determine the number of neighbors.  Each vertex has only in or\n    // out edges depending on which side of the graph it is located\n    if(sum.Xy.size() == 0) { vdata.residual = 0; ++vdata.nupdates; return; }\n    mat XtX = sum.XtX;\n    vec Xy = sum.Xy;\n    // Add regularization\n    for(int i = 0; i < XtX.rows(); ++i) XtX(i,i) += LAMBDA; // /nneighbors;\n    // Solve the least squares problem using eigen ----------------------------\n    const vec old_factor = vdata.factor;\n   \n    long nodeid = (long)vertex.id(); \n    bool isuser = nodeid >= 0;\n    if (algorithm == SPARSE_BOTH_FACTORS || (algorithm == SPARSE_USR_FACTOR && isuser) || \n        (algorithm == SPARSE_ITM_FACTOR && !isuser)){ \n      double sparsity_level = 1.0;\n      if (isuser)\n        sparsity_level -= user_sparsity;\n      else sparsity_level -= movie_sparsity;\n      vdata.factor = CoSaMP(XtX, Xy, ceil(sparsity_level*(double)vertex_data::NLATENT), 10, 1e-4, vertex_data::NLATENT);\n    }\n    else vdata.factor = XtX.selfadjointView().ldlt().solve(Xy);\n\n    // Compute the residual change in the factor factor -----------------------\n    vdata.residual = (vdata.factor - old_factor).cwiseAbs().sum() / XtX.rows();\n    ++vdata.nupdates;\n  } // end of apply\n  \n  /** The edges to scatter along */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of scatter edges\n\n  /** Scatter reschedules neighbors */  \n  void scatter(icontext_type& context, const vertex_type& vertex, \n               edge_type& edge) const {\n    edge_data& edata = edge.data();\n    if(edata.role == edge_data::TRAIN) {\n      const vertex_type other_vertex = get_other_vertex(edge, vertex);\n      const vertex_data& vdata = vertex.data();\n      const vertex_data& other_vdata = other_vertex.data();\n      const double pred = vdata.factor.dot(other_vdata.factor);\n      const float error = std::fabs(edata.obs - pred);\n      const double priority = (error * vdata.residual); \n      // Reschedule neighbors ------------------------------------------------\n      if( priority > TOLERANCE && other_vdata.nupdates < MAX_UPDATES) \n        context.signal(other_vertex, priority);\n    }\n  } // end of scatter function\n\n\n  /**\n   * \\brief Signal all vertices on one side of the bipartite graph\n   */\n  static graphlab::empty signal_left(icontext_type& context,\n                                     const vertex_type& vertex) {\n    if(vertex.num_out_edges() > 0) context.signal(vertex);\n    return graphlab::empty();\n  } // end of signal_left \n\n}; // end of als vertex program\n\n\n\n/**\n * \\brief The graph loader function is a line parser used for\n * distributed graph construction.\n */\ninline bool graph_loader(graph_type& graph, \n                         const std::string& filename,\n                         const std::string& line) {\n  ASSERT_FALSE(line.empty()); \n  namespace qi = boost::spirit::qi;\n  namespace ascii = boost::spirit::ascii;\n  namespace phoenix = boost::phoenix;\n  // Determine the role of the data\n  edge_data::data_role_type role = edge_data::TRAIN;\n  if(boost::ends_with(filename,\".validate\")) role = edge_data::VALIDATE;\n  else if(boost::ends_with(filename, \".predict\")) role = edge_data::PREDICT;\n  // Parse the line\n  graph_type::vertex_id_type source_id(-1), target_id(-1);\n  float obs(0); \n  const bool success = qi::phrase_parse\n    (line.begin(), line.end(),       \n     //  Begin grammar\n     (\n      qi::ulong_[phoenix::ref(source_id) = qi::_1] >> -qi::char_(',') >>\n      qi::ulong_[phoenix::ref(target_id) = qi::_1] >> \n      -(-qi::char_(',') >> qi::float_[phoenix::ref(obs) = qi::_1])\n      )\n     ,\n     //  End grammar\n     ascii::space); \n\n  if(!success) return false;\n\n  if(role == edge_data::TRAIN || role == edge_data::VALIDATE){\n    if (obs < als_vertex_program::MINVAL || obs > als_vertex_program::MAXVAL)\n      logstream(LOG_FATAL)<<\"Rating values should be between \" << als_vertex_program::MINVAL << \" and \" << als_vertex_program::MAXVAL << \". Got value: \" << obs << \" [ user: \" << source_id << \" to item: \" < 0) {\n      const double validation_error = \n        std::sqrt(agg.validation_error / info.validation_edges);\n      context.cout() << \"\\t\" << validation_error; \n    }\n    context.cout() << std::endl;\n  }\n}; // end of error aggregator\n\n\n\n\n/**\n * \\brief The prediction saver is used by the graph.save routine to\n * output the final predictions back to the filesystem.\n */\nstruct prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    return \"\"; //nop\n  }\n  std::string save_edge(const edge_type& edge) const {\n    if(edge.data().role == edge_data::PREDICT) {\n      std::stringstream strm;\n      const double prediction = \n        edge.source().data().factor.dot(edge.target().data().factor);\n      strm << edge.source().id() << '\\t';\n      strm << (-edge.target().id() - SAFE_NEG_OFFSET) << '\\t';\n      strm << prediction << '\\n';\n      return strm.str();\n    } else return \"\";\n  }\n}; // end of prediction_saver\n\n\nstruct linear_model_saver_U {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n  */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() > 0){\n      std::string ret = boost::lexical_cast(vertex.id()) + \" \";\n      for (uint i=0; i< vertex_data::NLATENT; i++)\n        ret += boost::lexical_cast(vertex.data().factor[i]) + \" \";\n        ret += \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\nstruct linear_model_saver_V {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n  */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() == 0){\n      std::string ret = boost::lexical_cast(-vertex.id()-SAFE_NEG_OFFSET) + \") \";\n      for (uint i=0; i< vertex_data::NLATENT; i++)\n        ret += boost::lexical_cast(vertex.data().factor[i]) + \" \";\n        ret += \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\n\n\n/**\n * \\brief The engine type used by the ALS matrix factorization\n * algorithm.\n *\n * The ALS matrix factorization algorithm currently uses the\n * synchronous engine.  However we plan to add support for alternative\n * engines in the future.\n */\ntypedef graphlab::omni_engine engine_type;\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n\n  // Parse command line options -----------------------------------------------\n  const std::string description = \n    \"Compute the ALS factorization of a matrix.\";\n  graphlab::command_line_options clopts(description);\n  std::string input_dir, output_dir;\n  std::string predictions;\n  size_t interval = 1;\n  std::string exec_type = \"synchronous\";\n  clopts.attach_option(\"matrix\", input_dir,\n                       \"The directory containing the matrix file\");\n  clopts.add_positional(\"matrix\");\n  clopts.attach_option(\"D\",  vertex_data::NLATENT,\n                       \"Number of latent parameters to use.\");\n  clopts.attach_option(\"max_iter\", als_vertex_program::MAX_UPDATES,\n                       \"The maxumum number of udpates allowed for a vertex\");\n  clopts.attach_option(\"lambda\", als_vertex_program::LAMBDA, \n                       \"ALS regularization weight\"); \n  clopts.attach_option(\"tol\", als_vertex_program::TOLERANCE,\n                       \"residual termination threshold\");\n  clopts.attach_option(\"maxval\", als_vertex_program::MAXVAL, \"max allowed value\");\n  clopts.attach_option(\"minval\", als_vertex_program::MINVAL, \"min allowed value\");\n  clopts.attach_option(\"interval\", interval, \n                       \"The time in seconds between error reports\");\n  clopts.attach_option(\"predictions\", predictions,\n                       \"The prefix (folder and filename) to save predictions.\");\n  clopts.attach_option(\"user_sparsity\", user_sparsity, \"sparsity of user factors\");\n  clopts.attach_option(\"movie_sparsity\", movie_sparsity, \"sparsity of item factors\");\n  clopts.attach_option(\"algorithm\", algorithm, \"run mode. 1 = SPARSE_USR_FACTOR, 2 = SPARSE_ITM_FACTOR, 3 = SPARSE_BOTH_FACTORS\");\n\n  clopts.attach_option(\"engine\", exec_type, \n                       \"The engine type synchronous or asynchronous\");\n  clopts.attach_option(\"output\", output_dir,\n                       \"Output results\");\n  \n  parse_implicit_command_line(clopts);\n  \n  if(!clopts.parse(argc, argv) || input_dir == \"\") {\n    std::cout << \"Error in parsing command line arguments.\" << std::endl;\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  if (user_sparsity < 0.5 || user_sparsity >= 1)\n    logstream(LOG_FATAL)<<\"Sparsity level should be [0.5,1). Please run again using --user_sparsity=XX in this range\" << std::endl;\n\n  if (movie_sparsity < 0.5 || movie_sparsity >= 1)\n    logstream(LOG_FATAL)<<\"Sparsity level should be [0.5,1). Please run again using --movie_sparsity=XX in this range\" << std::endl;\n\nif (algorithm != SPARSE_USR_FACTOR && algorithm != SPARSE_BOTH_FACTORS && algorithm != SPARSE_ITM_FACTOR)\n    logstream(LOG_FATAL)<<\"Algorithm should be 1 for SPARSE_USR_FACTOR, 2 for SPARSE_ITM_FACTOR and 3 for SPARSE_BOTH_FACTORS\" << std::endl;\n\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  \n  dc.cout() << \"Loading graph.\" << std::endl;\n  graphlab::timer timer; \n  graph_type graph(dc, clopts);  \n  graph.load(input_dir, graph_loader); \n  dc.cout() << \"Loading graph. Finished in \" \n            << timer.current_time() << std::endl;\n\n  if (dc.procid() == 0) \n    add_implicit_edges(implicitratingtype, graph, dc);\n\n  dc.cout() << \"Finalizing graph.\" << std::endl;\n  timer.start();\n  graph.finalize();\n  dc.cout() << \"Finalizing graph. Finished in \" \n            << timer.current_time() << std::endl;\n\n  if (!graph.num_edges() || !graph.num_vertices())\n     logstream(LOG_FATAL)<< \"Failed to load graph. Check your input path: \" << input_dir << std::endl;     \n\n\n\n  dc.cout() \n      << \"========== Graph statistics on proc \" << dc.procid() \n      << \" ===============\"\n      << \"\\n Num vertices: \" << graph.num_vertices()\n      << \"\\n Num edges: \" << graph.num_edges()\n      << \"\\n Num replica: \" << graph.num_replicas()\n      << \"\\n Replica to vertex ratio: \" \n      << float(graph.num_replicas())/graph.num_vertices()\n      << \"\\n --------------------------------------------\" \n      << \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n      << \"\\n Num local vertices: \" << graph.num_local_vertices()\n      << \"\\n Replica to own ratio: \" \n      << (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n      << \"\\n Num local edges: \" << graph.num_local_edges()\n      //<< \"\\n Begin edge id: \" << graph.global_eid(0)\n      << \"\\n Edge balance ratio: \" \n      << float(graph.num_local_edges())/graph.num_edges()\n      << std::endl;\n \n  dc.cout() << \"Creating engine\" << std::endl;\n  engine_type engine(dc, graph, exec_type, clopts);\n\n  // Add error reporting to the engine\n  const bool success = engine.add_edge_aggregator\n    (\"error\", error_aggregator::map, error_aggregator::finalize) &&\n    engine.aggregate_periodic(\"error\", interval);\n  ASSERT_TRUE(success);\n  \n\n  // Signal all vertices on the vertices on the left (liberals) \n  engine.map_reduce_vertices(als_vertex_program::signal_left);\n  info = graph.map_reduce_edges(count_edges);\n  dc.cout()<<\"Training edges: \" << info.training_edges << \" validation edges: \" << info.validation_edges << std::endl;\n\n \n\n  dc.cout() << \"Running Sparse-ALS\" << std::endl;\n  dc.cout() << \"(C) Code by Danny Bickson, CMU \" << std::endl;\n  dc.cout() << \"Please send bug reports to danny.bickson@gmail.com\" << std::endl;\n  dc.cout() << \"Time   Training    Validation\" <> validation_edges >> training_edges >> max_user >> max_item; }  \n\n};\n\n  \nstats_info info;\n\n\n#endif //TK_STATS\n"
  },
  {
    "path": "toolkits/collaborative_filtering/svd.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n *\n *  Implementation of the Lanczos algorithm, as given in:\n *  http://www.grycap.upv.es/slepc/documentation/reports/str8.pdf\n *  (Restarted Lanczos Bidiagonalization for the SVD in SLEPc)\n * \n *  Code written by Danny Bickson, CMU, June 2011\n * */\n\n\n#include \"eigen_wrapper.hpp\"\n#include \"types.hpp\"\n#include \"eigen_serialization.hpp\"\n#include \n#include \n\n\n\n//when using negative node id range, we are not allowed to use\n//0 and 1 so we add 2.\nint iter = 0;\n//LANCZOS VARIABLES\nint max_iter = 10;\nint actual_vector_len = 0;\nint nv = 0;\nint nsv = 0;\ndouble tol = 1e-8;\nbool finished = false;\ndouble ortho_repeats = 3;\nbool update_function = false;\nbool save_vectors = false;\nbool use_ids = true;\nstd::string datafile; \nstd::string vecfile;\nint unittest;\nint nodes = 0;\nint data_size = 0;\nstd::string predictions;\nint rows = -1, cols = -1;\nbool quiet = false;\nint nconv = 0;\nint n = 0; \nint kk = 0;\nbool binary = false; //if true, all edges = 1\nmat a,PT;\nbool v_vector = false;\nint input_file_offset = 0; //if set to non zero, each row/col id will be reduced the input_file_offset\nvec singular_values;\n\nDECLARE_TRACER(svd_bidiagonal);\nDECLARE_TRACER(svd_error_estimate);\nDECLARE_TRACER(svd_error2);\nDECLARE_TRACER(matproduct);\nDECLARE_TRACER(svd_swork);\nDECLARE_TRACER(svd_vectors);\n\nvoid start_engine();\n\nstruct vertex_data {\n  /** \\brief The number of times this vertex has been updated. */\n  vec pvec;\n  double A_ii;\n\n  /** \n   * \\brief Simple default constructor which randomizes the vertex\n   *  data \n   */\n  vertex_data() : A_ii(0) { randomize(); } \n  /** \\brief Randomizes the latent pvec */\n  void randomize() { pvec.resize(data_size); pvec.setRandom(); }\n  /** \\brief Save the vertex data to a binary archive */\n  void save(graphlab::oarchive& arc) const { \n    arc << pvec << A_ii;\n  }\n  /** \\brief Load the vertex data from a binary archive */\n  void load(graphlab::iarchive& arc) { \n    arc >> pvec >> A_ii;\n  }\n}; // end of vertex data\n\n\nclass gather_type {\n  public:\n    vec pvec;\n    double training_rmse;\n    double validation_rmse;\n    gather_type() { training_rmse = validation_rmse = 0; }\n    void save(graphlab::oarchive& arc) const { arc << pvec << training_rmse << validation_rmse; }\n    void load(graphlab::iarchive& arc) { arc >> pvec >> training_rmse >> validation_rmse; }  \n    gather_type& operator+=(const gather_type& other) {\n      pvec += other.pvec;\n      training_rmse += other.training_rmse;\n      validation_rmse += other.validation_rmse;\n      return *this;\n    } \n\n};\n\ngather_type ret;\n\n\n/**\n * \\brief The edge data stores the entry in the matrix.\n *\n * In addition the edge data svdo stores the most recent error estimate.\n */\nstruct edge_data : public graphlab::IS_POD_TYPE {\n  /**\n   * \\brief The type of data on the edge;\n   *\n   * \\li *Train:* the observed value is correct and used in training\n   * \\li *Validate:* the observed value is correct but not used in training\n   * \\li *Predict:* The observed value is not correct and should not be\n   *        used in training.\n   */\n  enum data_role_type { TRAIN, VALIDATE, PREDICT  };\n\n  /** \\brief the observed value for the edge */\n  double obs;\n\n  /** \\brief The train/validation/test designation of the edge */\n  data_role_type role;\n\n  /** \\brief basic initialization */\n  edge_data(double obs = 0, data_role_type role = PREDICT) :\n    obs(obs), role(role) { }\n\n}; // end of edge data\n\n\n/**\n * \\brief The graph type is defined in terms of the vertex and edge\n * data.\n */ \ntypedef graphlab::distributed_graph graph_type;\ngraph_type * pgraph;\n\n/**\n * \\brief Given a vertex and an edge return the other vertex in the\n * edge.\n */\ninline graph_type::vertex_type get_other_vertex(graph_type::edge_type& edge, \n    const graph_type::vertex_type& vertex) {\n  return vertex.id() == edge.source().id()? edge.target() : edge.source();\n}; // end of get_other_vertex\n\n//typedef double gather_type;\ntypedef double message_type;\n\n\n#include \"math.hpp\" //uses vertex_data and edge_data so has to be included here\n#include \"printouts.hpp\" // the same\ntypedef graphlab::omni_engine engine_type;\nengine_type * pengine = NULL;\n\n\n\n\n/**\n * \\brief The prediction saver is used by the graph.save routine to\n * output the final predictions back to the filesystem.\n */\nstruct prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    return \"\"; //nop\n  }\n  std::string save_edge(const edge_type& edge) const {\n    if(edge.data().role == edge_data::PREDICT) {\n      std::stringstream strm;\n      Eigen::DiagonalMatrix diagonal_matrix(nconv);      \n      diagonal_matrix.diagonal() = singular_values;\n     const double prediction = \n        edge.source().data().pvec.head(nconv).transpose() * diagonal_matrix * edge.target().data().pvec.head(nconv);\n      strm << (edge.source().id()+input_file_offset) << '\\t';\n      strm << (edge.target().id()-rows+input_file_offset) << '\\t';\n      strm << std::setprecision(8) < where k is the number of converged singular values\n  */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.id() < rows){\n      std::string ret = boost::lexical_cast(vertex.id()+input_file_offset) + \" \";\n      for (uint i=0; i< nconv; i++)\n        ret += boost::lexical_cast(vertex.data().pvec[i]) + \" \";\n      ret += \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n \n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\nstruct linear_model_saver_V {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid factor1 factor2 ... factorNLATENT \\n\n  */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.id() >= rows){\n      std::string ret = boost::lexical_cast(vertex.id()-rows+input_file_offset) + \" \";\n      for (uint i=0; i< nconv; i++)\n        ret += boost::lexical_cast(vertex.data().pvec[i]) + \" \";\n      ret += \"\\n\";\n      return ret;\n  }\n  else return \"\";\n}\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\n\n/**\n * \\brief The graph loader function is a line parser used for\n * distributed graph construction.\n */\ninline bool graph_loader(graph_type& graph, \n    const std::string& filename,\n    const std::string& line) {\n\n  //no need to parse\n  if (filename == vecfile)\n    return true;\n  if (boost::ends_with(filename,\"singular_values\") || boost::ends_with(filename, \"_v0\"))\n    return true;\n  if (line.find(\"#\") != std::string::npos)\n    return true;\n\n\n  // Determine the role of the data\n  edge_data::data_role_type role = edge_data::TRAIN;\n  if (boost::ends_with(filename,\".predict\")) \n    role = edge_data::PREDICT;\n \n  // Parse the line\n  std::stringstream strm(line);\n  graph_type::vertex_id_type source_id(-1), target_id(-1);\n  float obs = 1;\n  strm >> source_id >> target_id;\n  if (source_id == graph_type::vertex_id_type(-1) || target_id == graph_type::vertex_id_type(-1)){\n    logstream(LOG_WARNING)<<\"Failed to read input line: \"<< line << \" in file: \"  << filename << \" (or node id is -1). \" << std::endl;\n    return true;\n  }\n\n  if (input_file_offset != 0){\n     source_id-=input_file_offset; \n     target_id-=input_file_offset;\n  }\n  //if (source_id >= (uint)rows)\n  if (source_id > (uint)rows)\n    logstream(LOG_FATAL)<<\"Problem at input line: [ \" << line << \" ] row id ( = \" << source_id+input_file_offset << \" ) should be <= than matrix rows (= \" << rows << \" ) \" << std::endl;\n  //if (target_id >= (uint)cols)\n  if (target_id > (uint)cols)\n    logstream(LOG_FATAL)<<\"Problem at input line: [ \" << line << \" ] col id ( = \" << target_id+input_file_offset << \" ) should be <= than matrix cols (= \" << cols << \" ) \" << std::endl;\n\n  if (!binary)\n     strm >> obs;\n  if (!info.is_square())\n    target_id = rows + target_id;\n\n  if (source_id == target_id){\n      vertex_data data;\n      data.A_ii = obs;\n      graph.add_vertex(source_id, data);\n  }\n  // Create an edge and add it to the graph\n  else graph.add_edge(source_id, target_id, edge_data(obs, role)); \n  return true; // successful load\n} // end of graph_loader\n\n\nvoid init_lanczos(graph_type * g, bipartite_graph_descriptor & info){\n\n  if (g->num_vertices() == 0)\n    logstream(LOG_FATAL)<<\"Failed to load graph. Aborting\" << std::endl;\n\n  data_size = nsv + nv+1 + max_iter;\n  actual_vector_len = data_size;\n  if (info.is_square())\n    actual_vector_len = 2*data_size;\n\n  //assert(pengine);\n  assert(actual_vector_len > 0);\n  pgraph->transform_vertices(init_lanczos_mapr);\n\n  logstream(LOG_INFO)<<\"Allocated a total of: \" << ((double)actual_vector_len * g->num_vertices() * sizeof(double)/ 1e6) << \" MB for storing vectors.\" << std::endl;\n}\n\nvoid swork_vec(graph_type::vertex_type & vertex){\n  vertex.data().pvec[nconv+kk] = 0;\n  for (int ttt=nconv; ttt < nconv+n; ttt++){\n    vertex.data().pvec[nconv+kk] += curvec(ttt-nconv)*vertex.data().pvec[ttt];\n  }\n}  \n\nvoid compute_ritz(graph_type::vertex_type & vertex){\n  if (!info.is_square())\n    assert(vertex.id() - pcurrent->start >= 0);\n \n  assert(nconv + n < vertex.data().pvec.size());\n  assert(pcurrent->offset >= 0 && pcurrent->offset < vertex.data().pvec.size());\n\n  int offset = pcurrent->offset + nconv;\n  assert(offset + n < actual_vector_len);\n  if (info.is_square() && !v_vector)\n    offset += data_size;\n  vec tmp = init_vec(&vertex.data().pvec[offset], n);\n  tmp = tmp.transpose() * (v_vector ? PT : a);\n  memcpy(&vertex.data().pvec[offset] ,&tmp[0], kk*sizeof(double)); \n  if (debug)\n     std::cout<<\"Entered ritz with \" << offset << \" , v_vector: \" << v_vector << \"data_size: \" << data_size << \" kk: \" << kk << std::endl;\n}  \n\n\n\nvoid lanczos(bipartite_graph_descriptor & info, timer & mytimer, vec & errest, \n    const std::string & vecfile){\n\n  int its = 1;\n  DistMat A(info);\n  DistSlicedMat U(info.is_square() ? data_size : 0, info.is_square() ? 2*data_size : data_size, true, info, \"U\");\n  DistSlicedMat V(0, data_size, false, info, \"V\");\n  vec alpha, beta, b;\n  vec sigma = zeros(data_size);\n  errest = zeros(nv);\n  DistVec v_0(info, 0, false, \"v_0\");\n  if (vecfile.size() == 0)\n    v_0 = randu(size(A,2));\n  PRINT_VEC2(\"svd->V\", v_0);\n  PRINT_VEC(V[0]);\n   \n  DistDouble vnorm = norm(v_0);\n  v_0=v_0/vnorm;\n  PRINT_INT(nv);\n\n  while(nconv < nsv && its < max_iter){\n    logstream(LOG_EMPH)<<\"Starting iteration: \" << its << \" at time: \" << mytimer.current_time() << std::endl;\n    int k = nconv;\n    n  = nv;\n    PRINT_INT(k);\n    PRINT_INT(n);\n\n    alpha = zeros(n);\n    beta = zeros(n);\n\n    U[k] = V[k]*A._transpose();\n    PRINT_VEC(U[k]);\n    orthogonalize_vs_all(U, k, alpha(0));\n    PRINT_VEC(U[k]);\n    PRINT_VEC3(\"alpha\", alpha, 0);\n\n    for (int i=k+1; inconv\", nconv);\n    n = nv - nconv;\n    PRINT_INT(n);\n    alpha.conservativeResize(n);\n    beta.conservativeResize(n);\n    \n    PRINT_MAT2(\"Q\",eye(n));\n    PRINT_MAT2(\"PT\",eye(n));\n    PRINT_VEC2(\"alpha\",alpha);\n    PRINT_VEC2(\"beta\",beta);\n\n    mat T=diag(alpha);\n    for (int i=0; isigma[i]\", sigma(i));\n      PRINT_NAMED_DBL(\"Q[j*n+n-1]\",a(n-1,j));\n      PRINT_NAMED_DBL(\"beta[n-1]\",beta(n-1));\n      errest(i) = abs(a(n-1,j)*beta(n-1));\n      PRINT_NAMED_DBL(\"svd->errest[i]\", errest(i));\n      if (alpha(j) >  tol){\n        errest(i) = errest(i) / alpha(j);\n        PRINT_NAMED_DBL(\"svd->errest[i]\", errest(i));\n      }\n      if (errest(i) < tol){\n        kk = kk+1;\n        PRINT_NAMED_INT(\"k\",kk);\n      }\n\n\n      if (nconv +kk >= nsv){\n        printf(\"set status to tol\\n\");\n        finished = true;\n      }\n    }//end for\n    PRINT_NAMED_INT(\"k\",kk);\n    END_TRACEPOINT(svd_error_estimate)\n\n    //vec v;\n    if (!finished){\n      BEGIN_TRACEPOINT(svd_swork);\n      curvec=get_col(PT,kk); \n      DistVec v = V[nconv];\n      pcurrent = &v;\n      graphlab::vertex_set nodes = pgraph->select(select_in_range);\n      pgraph->transform_vertices(swork_vec, nodes);\n      \n      PRINT_MAT2(\"swork\", curvec);\n      PRINT_VEC2(\"svd->V\",V[nconv]);\n      //PRINT_VEC2(\"v[0]\",v); \n      END_TRACEPOINT(svd_swork);\n    }\n\n    //compute the ritz eigenvectors of the converged singular triplets\n    DistVec v = V[nconv];\n    if (kk > 0){\n      PRINT_VEC2(\"svd->V\", V[nconv]);\n      BEGIN_TRACEPOINT(matproduct);\n      v = V[nconv];\n      pcurrent = &v;\n      v_vector = true;\n      graphlab::vertex_set nodes = pgraph->select(select_in_range);\n      pgraph->transform_vertices(compute_ritz, nodes);\n      PRINT_VEC2(\"svd->V\", V[nconv]);\n      v = U[nconv];\n      pcurrent = &v;\n      v_vector = false;\n      PRINT_VEC2(\"svd->U\", U[nconv]);\n      nodes = pgraph->select(select_in_range);\n      pgraph->transform_vertices(compute_ritz, nodes);\n      END_TRACEPOINT(matproduct);\n      PRINT_VEC2(\"svd->U\", U[nconv]);\n    }\n\n    nconv=nconv+kk;\n    if (finished)\n      break;\n\n    V[nconv]=v;\n    PRINT_VEC2(\"svd->V\", V[nconv]);\n    PRINT_NAMED_INT(\"svd->nconv\", nconv);\n\n    its++;\n    PRINT_NAMED_INT(\"svd->its\", its);\n    PRINT_NAMED_INT(\"svd->nconv\", nconv);\n    PRINT_NAMED_INT(\"nv\",nv);\n\n  } // end(while)\n\n  printf(\" Number of computed signular values %d\",nconv);\n  printf(\"\\n\");\n  DistVec normret(info, nconv, false, \"normret\");\n  DistVec normret_tranpose(info, nconv, true, \"normret_tranpose\");\n  INITIALIZE_TRACER(svd_error2, \"svd error2\");\n  BEGIN_TRACEPOINT(svd_error2);\n  for (int i=0; i < std::min(nsv,nconv); i++){\n    normret = V[i]*A._transpose() -U[i]*sigma(i);\n    double n1 = norm(normret).toDouble();\n    PRINT_DBL(n1);\n    normret_tranpose = U[i]*A -V[i]*sigma(i);\n    double n2 = norm(normret_tranpose).toDouble();\n    PRINT_DBL(n2);\n    double err=sqrt(n1*n1+n2*n2);\n    PRINT_DBL(err);\n    PRINT_DBL(tol);\n    if (sigma(i)>tol){\n      err = err/sigma(i);\n    }\n    PRINT_DBL(err);\n    PRINT_DBL(sigma(i));\n    printf(\"Singular value %d \\t%13.6g\\tError estimate: %13.6g\\n\", i, sigma(i),err);\n  }\n  END_TRACEPOINT(svd_error2);\n\n  if (save_vectors){\n    if (nconv == 0)\n      logstream(LOG_FATAL)<<\"No converged vectors. Aborting the save operation\" << std::endl;\n    if (predictions == \"\")\n      logstream(LOG_FATAL)<<\"Please specify prediction output fie name using the --predictions=filename command\"<save(predictions + \".U\", linear_model_saver_U(),\n          gzip_output, save_edges, save_vertices, threads_per_machine);\n      pgraph->save(predictions + \".V\", linear_model_saver_V(),\n          gzip_output, save_edges, save_vertices, threads_per_machine);\n    END_TRACEPOINT(svd_vectors);\n  }\n\n  sigma.conservativeResize(nconv);\n  singular_values = sigma;\n  if(!predictions.empty()) {\n    std::cout << \"Saving predictions\" << std::endl;\n    const bool gzip_output = false;\n    const bool save_vertices = false;\n    const bool save_edges = true;\n    const size_t threads_per_machine = 1;\n\n    //save the predictions\n    pgraph->save(predictions, prediction_saver(),\n               gzip_output, save_vertices, \n               save_edges, threads_per_machine);\n  }\n \n}\n\nvoid start_engine(){\n  vertex_set nodes = pgraph->select(selected_node);\n  pengine->signal_vset(nodes);\n  pengine->start();\n}\n\nvoid write_output_vector(const std::string datafile, const vec & output, bool issparse, std::string comment)\n{\n  FILE * f = fopen(datafile.c_str(),\"w\");\n  if (f == NULL)\n    logstream(LOG_FATAL)<<\"Failed to open file: \" << datafile << \" for writing. \" << std::endl;\n\n  if (comment.size() > 0) // add a comment to the matrix market header\n    fprintf(f, \"%c%s\\n\", '%', comment.c_str());\n    for (int j=0; j<(int)output.size(); j++){\n    fprintf(f, \"%10.13g\\n\", output[j]);\n  }\n\n  fclose(f);\n}\n\n\nint main(int argc, char** argv) {\n  global_logger().set_log_to_console(true);\n\n  INITIALIZE_TRACER(svd_bidiagonal, \"svd bidiagonal\");\n  INITIALIZE_TRACER(svd_error_estimate, \"svd error estimate\");\n  INITIALIZE_TRACER(svd_swork, \"Svd swork\");\n  INITIALIZE_TRACER(matproduct, \"computing ritz eigenvectors\");\n  INITIALIZE_TRACER(svd_bidiagonal, \"svd bidiagonal\");\n  INITIALIZE_TRACER(svd_vectors, \"svd vectors\");\n\n  // Parse command line options -----------------------------------------------\n  const std::string description = \n    \"Compute the gklanczos factorization of a matrix.\";\n  graphlab::command_line_options clopts(description);\n  std::string input_dir, output_dir;\n  std::string exec_type = \"synchronous\";\n  clopts.attach_option(\"matrix\", input_dir,\n      \"The directory containing the matrix file\");\n  clopts.add_positional(\"matrix\");\n  clopts.attach_option(\"initial_vector\", vecfile,\"optional initial vector\");\n  clopts.attach_option(\"debug\", debug, \"Display debug output.\");\n  clopts.attach_option(\"unittest\", unittest,  \n      \"unit testing 0=None, 1=3x3 matrix, 2=10x10 matrix, 3 = 100x100 matrix\");\n  clopts.attach_option(\"max_iter\", max_iter, \"max iterations\");\n  clopts.attach_option(\"ortho_repeats\", ortho_repeats, \"orthogonalization iterations. 1 = low accuracy but fast, 2 = medium accuracy, 3 = high accuracy but slow.\");\n  clopts.attach_option(\"nv\", nv, \"Number of vectors in each iteration\");\n  clopts.attach_option(\"nsv\", nsv, \"Number of requested singular values to comptue\"); \n  clopts.attach_option(\"tol\", tol, \"convergence threshold\");\n  clopts.attach_option(\"save_vectors\", save_vectors, \"save output matrices U and V.\");\n  clopts.attach_option(\"rows\", rows, \"number of rows\");\n  clopts.attach_option(\"cols\", cols, \"number of cols\");\n  clopts.attach_option(\"quiet\", quiet, \"quiet mode (less verbose)\");\n  clopts.attach_option(\"predictions\", predictions, \"predictions file prefix\");\n  clopts.attach_option(\"binary\", binary, \"If true, all edges are weighted as one\");\n  clopts.attach_option(\"input_file_offset\", input_file_offset, \"input file node id offset (default 0)\");\n  if(!clopts.parse(argc, argv) || input_dir == \"\") {\n    std::cout << \"Error in parsing command line arguments.\" << std::endl;\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  if (quiet){\n    global_logger().set_log_level(LOG_ERROR);\n    debug = false;\n  }\n  if (unittest == 1){\n    datafile = \"gklanczos_testA/\"; \n    vecfile = \"gklanczos_testA_v0\";\n    nsv = 3; nv = 3;\n    rows = 3; cols = 4;\n    debug = true;\n    input_file_offset = 1;\n  }\n  else if (unittest == 2){\n    datafile = \"gklanczos_testB/\";\n    vecfile = \"gklanczos_testB_v0\";\n    nsv = 10; nv = 10;\n    rows = 10; cols = 10;\n    debug = true;  max_iter = 100;\n    input_file_offset = 1;\n  }\n  else if (unittest == 3){\n    datafile = \"gklanczos_testC/\";\n    vecfile = \"gklanczos_testC_v0\";\n    nsv = 4; nv = 10;\n    rows = 25; cols = 25;\n    debug = true;  max_iter = 100;\n    input_file_offset = 1;\n  }\n  else if (unittest == 4){\n    datafile= \"A2/\";\n    vecfile = \"A2/A2_v0\";\n    nsv=3; nv = 4; \n    rows=3; cols = 4;\n    debug=true; max_iter=3;\n    input_file_offset = 1;\n  }\n\n\n\n  if (rows <= 0 || cols <= 0)\n    logstream(LOG_FATAL)<<\"Please specify number of rows/cols of the input matrix\" << std::endl;\n\n  if (rows == cols){\n    logstream(LOG_WARNING)<<\"GraphLab SVD does not support square matrices. Increasing row size by one.\" << std::endl;\n    rows++; \n  }\n  info.rows = rows;\n  info.cols = cols;\n\n  if (nv < nsv){\n    logstream(LOG_FATAL)<<\"Please set the number of vectors --nv=XX, to be at least the number of support vectors --nsv=XX or larger\" << std::endl;\n  }\n\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n  dc.cout() << \"Loading graph.\" << std::endl;\n  graphlab::timer timer; \n  graph_type graph(dc, clopts);  \n  graph.load(input_dir, graph_loader); \n  pgraph = &graph;\n  dc.cout() << \"Loading graph. Finished in \" \n    << timer.current_time() << std::endl;\n  dc.cout() << \"Finalizing graph.\" << std::endl;\n  timer.start();\n  graph.finalize();\n  dc.cout() << \"Finalizing graph. Finished in \" \n    << timer.current_time() << std::endl;\n\n  if (!graph.num_edges() || !graph.num_vertices())\n     logstream(LOG_FATAL)<< \"Failed to load graph. Check your input path: \" << input_dir << std::endl;     \n\n  dc.cout() \n    << \"========== Graph statistics on proc \" << dc.procid() \n    << \" ===============\"\n    << \"\\n Num vertices: \" << graph.num_vertices()\n    << \"\\n Num edges: \" << graph.num_edges()\n    << \"\\n Num replica: \" << graph.num_replicas()\n    << \"\\n Replica to vertex ratio: \" \n    << float(graph.num_replicas())/graph.num_vertices()\n    << \"\\n --------------------------------------------\" \n    << \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n    << \"\\n Num local vertices: \" << graph.num_local_vertices()\n    << \"\\n Replica to own ratio: \" \n    << (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n    << \"\\n Num local edges: \" << graph.num_local_edges()\n    //<< \"\\n Begin edge id: \" << graph.global_eid(0)\n    << \"\\n Edge balance ratio: \" \n    << float(graph.num_local_edges())/graph.num_edges()\n    << std::endl;\n\n  dc.cout() << \"Creating engine\" << std::endl;\n  engine_type engine(dc, graph, exec_type, clopts);\n  pengine = &engine;\n\n  dc.cout() << \"Running SVD (gklanczos)\" << std::endl;\n  dc.cout() << \"(C) Code by Danny Bickson, CMU \" << std::endl;\n  dc.cout() << \"Please send bug reports to danny.bickson@gmail.com\" << std::endl;\n  timer.start();\n\n  init_lanczos(&graph, info);\n  init_math(&graph, info, ortho_repeats, update_function);\n  if (vecfile.size() > 0){\n    std::cout << \"Load inital vector from file\" << vecfile << std::endl;\n    FILE * file = fopen((vecfile).c_str(), \"r\");\n    if (file == NULL)\n      logstream(LOG_FATAL)<<\"Failed to open initial vector\"<< std::endl;\n    vec input = vec::Zero(rows);\n    double val = 0;\n    for (int i=0; i< rows; i++){\n      int rc = fscanf(file, \"%lg\\n\", &val);\n      if (rc != 1)\n        logstream(LOG_FATAL)<<\"Failed to read initial vector (on line: \"<< i << \" ) \" << std::endl;\n      input[i] = val;\n    }\n    fclose(file);\n    DistVec v0(info, 0, false, \"v0\");\n    v0 = input;\n  }  \n\n  vec errest;\n  lanczos( info, timer, errest, vecfile);\n\n  if (graphlab::mpi_tools::rank()==0)\n    write_output_vector(predictions + \".singular_values\", singular_values, false, \"%GraphLab SVD Solver library. This file contains the singular values.\");\n\n  const double runtime = timer.current_time();\n  dc.cout() << \"----------------------------------------------------------\"\n    << std::endl\n    << \"Final Runtime (seconds):   \" << runtime \n                                        << std::endl\n                                        << \"Updates executed: \" << engine.num_updates() << std::endl\n                                        << \"Update Rate (updates/second): \" \n                                          << engine.num_updates() / runtime << std::endl;\n\n  // Compute the final training error -----------------------------------------\n  if (unittest == 1){\n    assert(errest.size() == 3);\n    for (int i=0; i< errest.size(); i++)\n      assert(errest[i] < 1e-30);\n  }\n  else if (unittest == 2){\n    assert(errest.size() == 10);\n    for (int i=0; i< errest.size(); i++)\n      assert(errest[i] < 1e-15);\n  }\n  else if (unittest == 4){\n    assert(pow(singular_values[0]-  2.16097, 2) < 1e-8);\n    assert(pow(singular_values[2]-  0.554159, 2) < 1e-8);\n   }\n \n\n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // end of main\n\n\n\n"
  },
  {
    "path": "toolkits/collaborative_filtering/svdpp.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * \\file\n * \n * \\brief The main file for the BIAS-SGD matrix factorization algorithm.\n *\n * This file contains the main body of the BIAS-SGD matrix factorization\n * algorithm. \n */\n\n#include \n#include \n#include \"eigen_serialization.hpp\"\n#include \n#include \n\n\n\ntypedef Eigen::VectorXd vec_type;\ntypedef Eigen::MatrixXd mat_type;\n\n//when using negative node id range, we are not allowed to use\n//0 and 1 so we add 2.\nconst static int SAFE_NEG_OFFSET=2;\nstatic bool debug;\nint iter = 0;\nfloat itmBiasStep = 1e-4;\nfloat itmBiasReg = 1e-4;\nfloat usrBiasStep = 1e-4;\nfloat usrBiasReg = 1e-4;\nfloat usrFctrStep = 1e-4;\nfloat usrFctrReg = 1e-4;\nfloat itmFctrStep = 1e-4;\nfloat itmFctrReg = 1e-4; //gamma7\nfloat itmFctr2Step = 1e-4;\nfloat itmFctr2Reg = 1e-4;\n/** \n * \\ingroup toolkit_matrix_pvecization\n *\n * \\brief the vertex data type which contains the latent pvec.\n *\n * Each row and each column in the matrix corresponds to a different\n * vertex in the BIASSGD graph.  Associated with each vertex is a pvec\n * (vector) of latent parameters that represent that vertex.  The goal\n * of the BIASSGD algorithm is to find the values for these latent\n * parameters such that the non-zero entries in the matrix can be\n * predicted by taking the dot product of the row and column pvecs.\n */\nstruct vertex_data {\n  /**\n   * \\brief A shared \"constant\" that specifies the number of latent\n   * values to use.\n   */\n  static size_t NLATENT;\n  /** \\brief The number of times this vertex has been updated. */\n  uint32_t nupdates;\n  /** \\brief The latent pvec for this vertex */\n  vec_type pvec;\n  vec_type weight;\n  double bias;\n  /** \n   * \\brief Simple default constructor which randomizes the vertex\n   *  data \n   */\n  vertex_data() : nupdates(0) { randomize(); } \n  /** \\brief Randomizes the latent pvec */\n  void randomize() { pvec.resize(NLATENT); pvec.setRandom(); weight.resize(NLATENT); weight.setRandom(); }\n  /** \\brief Save the vertex data to a binary archive */\n  void save(graphlab::oarchive& arc) const { \n    arc << nupdates << pvec << weight << bias;\n  }\n  /** \\brief Load the vertex data from a binary archive */\n  void load(graphlab::iarchive& arc) { \n    arc >> nupdates >> pvec >> weight >> bias;\n  }\n}; // end of vertex data\n\n\n/**\n * \\brief The edge data stores the entry in the matrix.\n *\n * In addition the edge data svdppo stores the most recent error estimate.\n */\nstruct edge_data : public graphlab::IS_POD_TYPE {\n  /**\n   * \\brief The type of data on the edge;\n   *\n   * \\li *Train:* the observed value is correct and used in training\n   * \\li *Validate:* the observed value is correct but not used in training\n   * \\li *Predict:* The observed value is not correct and should not be\n   *        used in training.\n   */\n  enum data_role_type { TRAIN, VALIDATE, PREDICT  };\n\n  /** \\brief the observed value for the edge */\n  float obs;\n\n  /** \\brief The train/validation/test designation of the edge */\n  data_role_type role;\n\n  /** \\brief basic initialization */\n  edge_data(float obs = 0, data_role_type role = PREDICT) :\n    obs(obs), role(role) { }\n\n}; // end of edge data\n\n\n/**\n * \\brief The graph type is defined in terms of the vertex and edge\n * data.\n */ \ntypedef graphlab::distributed_graph graph_type;\n\n#include \"implicit.hpp\"\n\ndouble extract_l2_error(const graph_type::edge_type & edge);\n\n\n/**\n * \\brief Given a vertex and an edge return the other vertex in the\n * edge.\n */\ninline graph_type::vertex_type\nget_other_vertex(graph_type::edge_type& edge, \n    const graph_type::vertex_type& vertex) {\n  return vertex.id() == edge.source().id()? edge.target() : edge.source();\n}; // end of get_other_vertex\n\n\n\n\n\n/**\n * \\brief The gather type used to construct XtX and Xty needed for the BIASSGD\n * update\n *\n * To compute the ALS update we need to compute the sum of \n * \\code\n *  sum: XtX = nbr.pvec.transpose() * nbr.pvec \n *  sum: Xy  = nbr.pvec * edge.obs\n * \\endcode\n * For each of the neighbors of a vertex. \n *\n * To do this in the Gather-Apply-Scatter model the gather function\n * computes and returns a pair consisting of XtX and Xy which are then\n * added. The gather type represents that tuple and provides the\n * necessary gather_type::operator+= operation.\n *\n */\nclass gather_type {\n  public:\n    /**\n     * \\brief Stores the current sum of nbr.pvec.transpose() *\n     * nbr.pvec\n     */\n\n    /**\n     * \\brief Stores the current sum of nbr.pvec * edge.obs\n     */\n    vec_type pvec;\n    vec_type weight;\n    double bias;\n\n    /** \\brief basic default constructor */\n    gather_type() { }\n\n    /**\n     * \\brief This constructor computes XtX and Xy and stores the result\n     * in XtX and Xy\n     */\n    gather_type(const vec_type& X, const vec_type & _weight, double _bias) {\n      pvec = X;\n      bias = _bias;\n      weight = _weight;\n    } // end of constructor for gather type\n\n    /** \\brief Save the values to a binary archive */\n    void save(graphlab::oarchive& arc) const { arc << pvec << bias << weight; }\n\n    /** \\brief Read the values from a binary archive */\n    void load(graphlab::iarchive& arc) { arc >> pvec >> bias >> weight; }  \n\n    /** \n     * \\brief Computes XtX += other.XtX and Xy += other.Xy updating this\n     * tuples value\n     */\n    gather_type& operator+=(const gather_type& other) {\n      if (pvec.size() == 0){\n        pvec = other.pvec;\n        bias = other.bias;\n        weight = other.weight;\n        return *this;\n      }\n      else if (other.pvec.size() == 0)\n        return *this;\n      pvec += other.pvec;\n      bias += other.bias;\n      weight += other.weight;\n      return *this;\n    } // end of operator+=\n\n}; // end of gather type\n\n//typedef gather_type message_type;\n\n\nenum{\n  PHASE1 = 0, PHASE2 = 1\n};\n\n/**\n * BIASSGD vertex program type\n */ \nclass svdpp_vertex_program : \n  public graphlab::ivertex_program {\n    public:\n      /** The convergence tolerance */\n      static double TOLERANCE;\n      static double LAMBDA;\n      static double GAMMA;\n      static double MAXVAL;\n      static double MINVAL;\n      static double STEP_DEC;\n      static bool debug;\n      static size_t MAX_UPDATES;\n      static double GLOBAL_MEAN;\n      static size_t NUM_TRAINING_EDGES;\n      static uint   USERS;\n\n      gather_type pmsg;\n      void save(graphlab::oarchive& arc) const { \n        arc << pmsg;\n      }\n      /** \\brief Load the vertex data from a binary archive */\n      void load(graphlab::iarchive& arc) { \n        arc >> pmsg;\n      }\n\n      /** The set of edges to gather along */\n      edge_dir_type gather_edges(icontext_type& context, \n          const vertex_type& vertex) const { \n        return graphlab::ALL_EDGES; \n      }; // end of gather_edges \n\n      gather_type gather(icontext_type& context, const vertex_type& vertex, \n          edge_type& edge) const {\n        vec_type step = vec_type::Zero(vertex_data::NLATENT);\n        double bias =0, other_bias = 0;\n        vec_type delta, other_delta;\n\n        //user node\n        if (vertex.num_in_edges() == 0){\n          vertex_type other_vertex(get_other_vertex(edge, vertex));\n          vertex_type my_vertex(vertex);\n\n          int phase = my_vertex.data().nupdates % 2; \n          if (phase == PHASE1){\n            //my_vertex.data().weight += movie.weight;\n            context.signal(other_vertex, gather_type(vec_type::Zero(vertex_data::NLATENT), vec_type::Zero(vertex_data::NLATENT), 0));\n            return gather_type(vec_type::Zero(vertex_data::NLATENT), other_vertex.data().weight, 0);\n          }\n          else if (phase == PHASE2){\n            //vertex_data & my_data = my_vertex.data();\n            double pred = svdpp_vertex_program::GLOBAL_MEAN + \n              my_vertex.data().bias + other_vertex.data().bias + my_vertex.data().pvec.dot(other_vertex.data().pvec+other_vertex.data().weight);\n            pred = std::min(pred, svdpp_vertex_program::MAXVAL);\n            pred = std::max(pred, svdpp_vertex_program::MINVAL); \n            const float err = edge.data().obs - pred;\n            if (debug)\n              std::cout<<\"entering edge \" << (int)edge.source().id() << \":\" << (int)edge.target().id() << \" err: \" << err << \" rmse: \" << err*err < 0){\n          if (phase == PHASE1){\n            pmsg = msg;\n          }\n          else if (phase == PHASE2){\n            pmsg = msg;\n          }\n        }\n      }\n\n      /** apply collects the sum of XtX and Xy */\n      void apply(icontext_type& context, vertex_type& vertex,\n          const gather_type& sum) {\n        vertex_data& vdata = vertex.data(); \n        int phase = vdata.nupdates %2;\n\n        if (phase == PHASE1){\n          //user node receives the sum of movie weights\n          if (vertex.num_out_edges() > 0){\n            vertex.data().weight = sum.weight;\n            float usrNorm = double(1.0/sqrt(vertex.num_out_edges()));\n            vertex.data().weight *= usrNorm;\n          }\n          //movie node doe nothing\n          else {}\n        }\n        else if (phase == PHASE2){\n          //user node update gradients and bias\n          if (vertex.num_in_edges() == 0){\n            vdata.pvec += sum.pvec;\n            vdata.bias += sum.bias;\n            //does not update weight here (since was done in phase1)\n          }\n          //movie node\n          else {\n            vdata.weight += pmsg.weight; //step\n            vdata.pvec += pmsg.pvec;\n            vdata.bias += pmsg.bias;\n          }\n        }\n        ++vdata.nupdates;\n      } // end of apply\n\n      /** The edges to scatter along */\n      edge_dir_type scatter_edges(icontext_type& context,\n          const vertex_type& vertex) const { \n        return graphlab::ALL_EDGES; \n      }; // end of scatter edges\n\n      /** Scatter reschedules neighbors */  \n      void scatter(icontext_type& context, const vertex_type& vertex, \n          edge_type& edge) const {\n        edge_data& edata = edge.data();\n        if(edata.role == edge_data::TRAIN) {\n          const vertex_type other_vertex = get_other_vertex(edge, vertex);\n          // Reschedule neighbors ------------------------------------------------\n          if(other_vertex.data().nupdates < MAX_UPDATES) \n            context.signal(other_vertex, gather_type(vec_type::Zero(vertex_data::NLATENT),vec_type::Zero(vertex_data::NLATENT),0));\n        }\n      } // end of scatter function\n\n\n      /**\n       * \\brief Signal all vertices on one side of the bipartite graph\n       */\n      static graphlab::empty signal_left(icontext_type& context,\n          vertex_type& vertex) {\n        if(vertex.num_out_edges() > 0) context.signal(vertex, gather_type(vec_type::Zero(vertex_data::NLATENT),vec_type::Zero(vertex_data::NLATENT),0));\n        return graphlab::empty();\n      } // end of signal_left \n\n  }; // end of svdpp vertex program\n\n\nstruct error_aggregator : public graphlab::IS_POD_TYPE {\n  typedef svdpp_vertex_program::icontext_type icontext_type;\n  typedef graph_type::edge_type edge_type;\n  double train_error, validation_error;\n  size_t ntrain, nvalidation;\n  error_aggregator() : \n    train_error(0), validation_error(0), ntrain(0), nvalidation(0) { }\n  error_aggregator& operator+=(const error_aggregator& other) {\n    train_error += other.train_error;\n    assert(!std::isnan(train_error));\n    validation_error += other.validation_error;\n    ntrain += other.ntrain;\n    nvalidation += other.nvalidation;\n    return *this;\n  }\n  static error_aggregator map(icontext_type& context, const graph_type::edge_type& edge) {\n    error_aggregator agg;\n    if (edge.data().role == edge_data::TRAIN){\n      agg.train_error = extract_l2_error(edge); agg.ntrain = 1;\n      assert(!std::isnan(agg.train_error));\n    }\n    else if (edge.data().role == edge_data::VALIDATE){\n      agg.validation_error = extract_l2_error(edge); agg.nvalidation = 1;\n    }\n    return agg;\n  }\n\n\n  static void finalize(icontext_type& context, const error_aggregator& agg) {\n    iter++;\n    if (iter%2 == 0)\n      return; \n    ASSERT_GT(agg.ntrain, 0);\n    const double train_error = std::sqrt(agg.train_error / agg.ntrain);\n    assert(!std::isnan(train_error));\n    context.cout() << std::setw(8) << context.elapsed_seconds() << std::setw(8) << train_error;\n    if(agg.nvalidation > 0) {\n      const double validation_error = \n        std::sqrt(agg.validation_error / agg.nvalidation);\n      context.cout() << std::setw(8) << validation_error; \n    }\n    context.cout() << std::endl;\n    usrBiasStep *= svdpp_vertex_program::STEP_DEC;\n    itmBiasStep *= svdpp_vertex_program::STEP_DEC;\n    usrFctrStep  *= svdpp_vertex_program::STEP_DEC;\n    itmFctrStep  *= svdpp_vertex_program::STEP_DEC;\n    itmFctr2Step *= svdpp_vertex_program::STEP_DEC;\n\n  }\n}; // end of error aggregator\n\n/**\n * \\brief Given an edge compute the error associated with that edge\n */\ndouble extract_l2_error(const graph_type::edge_type & edge) {\n  double pred = svdpp_vertex_program::GLOBAL_MEAN + \n    edge.source().data().bias +\n    edge.target().data().bias + \n    edge.source().data().pvec.dot(edge.target().data().pvec);\n  pred = std::min(svdpp_vertex_program::MAXVAL, pred);\n  pred = std::max(svdpp_vertex_program::MINVAL, pred);\n  double rmse = (edge.data().obs - pred) * (edge.data().obs - pred);\n  assert(rmse <= pow(svdpp_vertex_program::MAXVAL-svdpp_vertex_program::MINVAL,2));\n  return rmse;\n} // end of extract_l2_error\n\n\nstruct prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    return \"\"; //nop\n  }\n  std::string save_edge(const edge_type& edge) const {\n    if (edge.data().role != edge_data::PREDICT)\n      return \"\";\n\n    std::stringstream strm;\n    double pred = svdpp_vertex_program::GLOBAL_MEAN +\n      edge.target().data().bias + edge.source().data().bias + edge.source().data().pvec.dot(edge.target().data().pvec+edge.target().data().weight);\n      pred = std::min(pred, svdpp_vertex_program::MAXVAL);\n      pred = std::max(pred, svdpp_vertex_program::MINVAL);\n    strm << edge.source().id() << '\\t' \n      << -edge.target().id()-SAFE_NEG_OFFSET << '\\t'\n      << pred << '\\n';\n    return strm.str();\n  }\n}; // end of prediction_saver\n\nstruct linear_model_saver_U {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n     */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() > 0){\n      std::string ret = boost::lexical_cast(vertex.id()) + \" \";\n      for (uint i=0; i< vertex_data::NLATENT; i++)\n        ret += boost::lexical_cast(vertex.data().pvec[i]) + \" \";\n      ret += \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\nstruct linear_model_saver_V {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n     */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() == 0){\n      std::string ret = boost::lexical_cast(-vertex.id()-SAFE_NEG_OFFSET) + \" \";\n      for (uint i=0; i< vertex_data::NLATENT; i++)\n        ret += boost::lexical_cast(vertex.data().pvec[i]) + \" \";\n      ret += \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\nstruct linear_model_saver_bias_U {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n     */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() > 0){\n      std::string ret = boost::lexical_cast(vertex.id()) + \" \";\n      ret += boost::lexical_cast(vertex.data().bias) + \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \nstruct linear_model_saver_bias_V {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n     */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() == 0){\n      std::string ret = boost::lexical_cast(-vertex.id()-SAFE_NEG_OFFSET) + \" \";\n      ret += boost::lexical_cast(vertex.data().bias) + \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\n\n\n\n/**\n * \\brief The graph loader function is a line parser used for\n * distributed graph construction.\n */\ninline bool graph_loader(graph_type& graph, \n    const std::string& filename,\n    const std::string& line) {\n\n // Parse the line\n  std::stringstream strm(line);\n  graph_type::vertex_id_type source_id(-1), target_id(-1);\n  float obs(0);\n  strm >> source_id >> target_id;\n\n  if (source_id == graph_type::vertex_id_type(-1) || target_id == graph_type::vertex_id_type(-1)){\n    logstream(LOG_WARNING)<<\"Failed to read input line: \"<< line << \" in file: \"  << filename << \" (or node id is -1). \" << std::endl;\n    return true;\n  }\n\n  // Determine the role of the data\n  edge_data::data_role_type role = edge_data::TRAIN;\n  if(boost::ends_with(filename,\".validate\")) role = edge_data::VALIDATE;\n  else if(boost::ends_with(filename, \".predict\")) role = edge_data::PREDICT;\n \n  if(role == edge_data::TRAIN || role == edge_data::VALIDATE){\n    strm >> obs;\n    if (obs < svdpp_vertex_program::MINVAL || obs > svdpp_vertex_program::MAXVAL){\n      logstream(LOG_WARNING)<<\"Rating values should be between \" << svdpp_vertex_program::MINVAL << \" and \" << svdpp_vertex_program::MAXVAL << \". Got value: \" << obs << \" [ user: \" << source_id << \" to item: \" < engine_type;\n\n  double calc_global_mean(const graph_type::edge_type & edge){\n    if (edge.data().role == edge_data::TRAIN)\n      return edge.data().obs;\n    else return 0;\n  }\n\n  size_t count_edges(const graph_type::edge_type & edge){\n    if (edge.data().role == edge_data::TRAIN)\n      return 1;\n    else return 0;\n  }\n\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n\n  // Parse command line options -----------------------------------------------\n  const std::string description = \n    \"Compute the ALS factorization of a matrix.\";\n  graphlab::command_line_options clopts(description);\n  std::string input_dir, output_dir;\n  std::string predictions;\n  size_t interval = 0;\n  std::string exec_type = \"synchronous\";\n  clopts.attach_option(\"matrix\", input_dir,\n      \"The directory containing the matrix file\");\n  clopts.add_positional(\"matrix\");\n  clopts.attach_option(\"D\", vertex_data::NLATENT,\n      \"Number of latent parameters to use.\");\n  clopts.attach_option(\"engine\", exec_type, \n      \"The engine type synchronous or asynchronous\");\n  clopts.attach_option(\"max_iter\", svdpp_vertex_program::MAX_UPDATES,\n      \"The maxumum number of udpates allowed for a vertex\");\n  clopts.attach_option(\"lambda\", svdpp_vertex_program::LAMBDA, \n      \"SGD regularization weight\"); \n  clopts.attach_option(\"gamma\", svdpp_vertex_program::GAMMA, \n      \"SGD step size\"); \n  clopts.attach_option(\"debug\", svdpp_vertex_program::debug, \n      \"debug - additional verbose info\"); \n  clopts.attach_option(\"tol\", svdpp_vertex_program::TOLERANCE,\n      \"residual termination threshold\");\n  clopts.attach_option(\"maxval\", svdpp_vertex_program::MAXVAL, \"max allowed value\");\n  clopts.attach_option(\"minval\", svdpp_vertex_program::MINVAL, \"min allowed value\");\n  clopts.attach_option(\"step_dec\", svdpp_vertex_program::STEP_DEC, \"multiplicative step decrement\");\n  clopts.attach_option(\"user_bias_step\", usrBiasStep, \"user_bias_step\");\n  clopts.attach_option(\"user_bias_reg\", usrBiasReg, \"user_bias_reg\");\n  clopts.attach_option(\"item_bias_step\",itmBiasStep, \"item_bias_step\");\n  clopts.attach_option(\"item_bias_reg\", itmBiasReg, \"item_bias_reg\");\n  clopts.attach_option(\"user_factor_step\", usrFctrStep, \"user_factor_step\");\n  clopts.attach_option(\"user_factor_reg\", usrFctrReg, \"user_factor_reg\");\n  clopts.attach_option(\"item_factor_step\", itmFctrStep, \"item_factor_step\");\n  clopts.attach_option(\"item_factor_reg\", itmFctrReg, \"item_factor_reg\");\n  clopts.attach_option(\"item_factor2_step\", itmFctr2Step, \"item_factor2_step\");\n  clopts.attach_option(\"item_factor2_reg\", itmFctr2Reg, \"item_factor2_reg\");\n  clopts.attach_option(\"interval\", interval, \"The time in seconds between error reports\");\n  clopts.attach_option(\"predictions\", predictions,\n      \"The prefix (folder and filename) to save predictions.\");\n  clopts.attach_option(\"output\", output_dir, \"Output results\");\n\n  parse_implicit_command_line(clopts);\n\n  if(!clopts.parse(argc, argv) || input_dir == \"\") {\n    std::cout << \"Error in parsing command line arguments.\" << std::endl;\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  debug = svdpp_vertex_program::debug;\n  //  omp_set_num_threads(clopts.get_ncpus());\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n  dc.cout() << \"Loading graph.\" << std::endl;\n  graphlab::timer timer; \n  graph_type graph(dc, clopts);  \n  graph.load(input_dir, graph_loader); \n  dc.cout() << \"Loading graph. Finished in \" \n    << timer.current_time() << std::endl;\n \n  if (dc.procid() == 0) \n    add_implicit_edges(implicitratingtype, graph, dc);\n\n  dc.cout() << \"Finalizing graph.\" << std::endl;\n  timer.start();\n  graph.finalize();\n  dc.cout() << \"Finalizing graph. Finished in \" \n    << timer.current_time() << std::endl;\n\n  if (!graph.num_edges() || !graph.num_vertices())\n     logstream(LOG_FATAL)<< \"Failed to load graph. Check your input path: \" << input_dir << std::endl;     \n\n\n  dc.cout() \n    << \"========== Graph statistics on proc \" << dc.procid() \n    << \" ===============\"\n    << \"\\n Num vertices: \" << graph.num_vertices()\n    << \"\\n Num edges: \" << graph.num_edges()\n    << \"\\n Num replica: \" << graph.num_replicas()\n    << \"\\n Replica to vertex ratio: \" \n    << float(graph.num_replicas())/graph.num_vertices()\n    << \"\\n --------------------------------------------\" \n    << \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n    << \"\\n Num local vertices: \" << graph.num_local_vertices()\n    << \"\\n Replica to own ratio: \" \n    << (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n    << \"\\n Num local edges: \" << graph.num_local_edges()\n    //<< \"\\n Begin edge id: \" << graph.global_eid(0)\n    << \"\\n Edge balance ratio: \" \n    << float(graph.num_local_edges())/graph.num_edges()\n    << std::endl;\n\n  dc.cout() << \"Creating engine\" << std::endl;\n  engine_type engine(dc, graph, exec_type, clopts);\n\n  // Add error reporting to the engine\n  const bool success = engine.add_edge_aggregator\n    (\"error\", error_aggregator::map, error_aggregator::finalize) &&\n    engine.aggregate_periodic(\"error\", interval);\n  ASSERT_TRUE(success);\n\n\n  svdpp_vertex_program::GLOBAL_MEAN = graph.map_reduce_edges(calc_global_mean);\n  svdpp_vertex_program::NUM_TRAINING_EDGES = graph.map_reduce_edges(count_edges);\n  svdpp_vertex_program::GLOBAL_MEAN /= svdpp_vertex_program::NUM_TRAINING_EDGES;\n  dc.cout() << \"Global mean is: \" <(svdpp_vertex_program::signal_left);\n\n\n  // Run the PageRank ---------------------------------------------------------\n  dc.cout() << \"Running SVD++\" << std::endl;\n  dc.cout() << \"(C) Code by Danny Bickson, CMU \" << std::endl;\n  dc.cout() << \"Please send bug reports to danny.bickson@gmail.com\" << std::endl;\n  dc.cout() << \"Time   Training    Validation\" <\n\n#include \n#include \n#include \n#include \n#include \n\n\n\n\n// This file defines the serialization code for the eigen types.\n#include \"eigen_serialization.hpp\"\n\n#include \n#include \n#include \"stats.hpp\"\n\n#include \n\nconst int SAFE_NEG_OFFSET = 2; //add 2 to negative node id\n//to prevent -0 and -1 which arenot allowed\n\n/**\n * \\brief We use the eigen library's vector type to represent\n * mathematical vectors.\n */\ntypedef Eigen::VectorXd vec_type;\n\n/**\n * \\brief We use the eigen library's matrix type to represent\n * matrices.\n */\ntypedef Eigen::MatrixXd mat_type;\n\n\n/**\n * \\brief Remap the target id of each edge into a different id space\n * than the source id.\n */\nbool REMAP_TARGET = true;\n\n\n\n/** \n * \\ingroup toolkit_matrix_factorization\n *\n * \\brief the vertex data type which contains the latent factor.\n *\n * Each row and each column in the matrix corresponds to a different\n * vertex in the ALS graph.  Associated with each vertex is a factor\n * (vector) of latent parameters that represent that vertex.  The goal\n * of the ALS algorithm is to find the values for these latent\n * parameters such that the non-zero entries in the matrix can be\n * predicted by taking the dot product of the row and column factors.\n */\nstruct vertex_data {\n  /**\n   * \\brief A shared \"constant\" that specifies the number of latent\n   * values to use.\n   */\n  static size_t NLATENT;\n  /** \\brief The number of times this vertex has been updated. */\n  uint32_t nupdates;\n  /** \\brief The most recent L1 change in the factor value */\n  float residual; //! how much the latent value has changed\n  /** \\brief The latent factor for this vertex */\n  vec_type factor;\n  /** \n   * \\brief Simple default constructor which randomizes the vertex\n   *  data \n   */\n  vertex_data() : nupdates(0), residual(1) { randomize(); } \n  /** \\brief Randomizes the latent factor */\n  void randomize() { factor.resize(NLATENT); factor.setRandom(); }\n  /** \\brief Save the vertex data to a binary archive */\n  void save(graphlab::oarchive& arc) const { \n    arc << nupdates << residual << factor;        \n  }\n  /** \\brief Load the vertex data from a binary archive */\n  void load(graphlab::iarchive& arc) { \n    arc >> nupdates >> residual >> factor;\n  }\n}; // end of vertex data\n\n\nsize_t vertex_data::NLATENT = 20;\n\n/**\n * \\brief The edge data stores the entry in the matrix.\n *\n * In addition the edge data also stores the most recent error estimate.\n */\nstruct edge_data : public graphlab::IS_POD_TYPE {\n  /**\n   * \\brief The type of data on the edge;\n   *\n   * \\li *Train:* the observed value is correct and used in training\n   * \\li *Validate:* the observed value is correct but not used in training\n   * \\li *Predict:* The observed value is not correct and should not be\n   *        used in training.\n   */\n  enum data_role_type { TRAIN, VALIDATE, PREDICT  };\n\n  /** \\brief the observed value for the edge */\n  float obs;\n\n  /** \\brief the weight or time of the observation */\n  float weight; \n  \n  /** \\brief The train/validation/test designation of the edge */\n  data_role_type role;\n\n  /** \\brief basic initialization */\n  edge_data(float obs = 0, data_role_type role = TRAIN, float weight = 1) :\n    obs(obs), weight(weight), role(role) { }\n\n}; // end of edge data\n\n\n/**\n * \\brief The graph type is defined in terms of the vertex and edge\n * data.\n */ \ntypedef graphlab::distributed_graph graph_type;\n\n#include \"implicit.hpp\"\n\nstats_info count_edges(const graph_type::edge_type & edge){\n  stats_info ret;\n\n  if (edge.data().role == edge_data::TRAIN)\n     ret.training_edges = 1;\n  else if (edge.data().role == edge_data::VALIDATE)\n     ret.validation_edges = 1;\n  ret.max_user = (size_t)edge.source().id();\n  ret.max_item = (size_t)edge.target().id();\n  return ret;\n}\n\n\n\n/**\n * \\brief Given a vertex and an edge return the other vertex in the\n * edge.\n */\ninline graph_type::vertex_type\nget_other_vertex(graph_type::edge_type& edge, \n                 const graph_type::vertex_type& vertex) {\n  return vertex.id() == edge.source().id()? edge.target() : edge.source();\n}; // end of get_other_vertex\n\n\n\n\n/**\n * \\brief The gather type used to construct XtX and Xty needed for the ALS\n * update\n *\n * To compute the ALS update we need to compute the sum of \n * \\code\n *  sum: XtX = nbr.factor.transpose() * nbr.factor \n *  sum: Xy  = nbr.factor * edge.obs\n * \\endcode\n * For each of the neighbors of a vertex. \n *\n * To do this in the Gather-Apply-Scatter model the gather function\n * computes and returns a pair consisting of XtX and Xy which are then\n * added. The gather type represents that tuple and provides the\n * necessary gather_type::operator+= operation.\n *\n */\nclass gather_type {\npublic:\n  /**\n   * \\brief Stores the current sum of nbr.factor.transpose() *\n   * nbr.factor\n   */\n  mat_type XtX;\n\n  /**\n   * \\brief Stores the current sum of nbr.factor * edge.obs\n   */\n  vec_type Xy;\n\n  /**\n   * \\brief Stores the weight of this edge\n   */\n  float weight;\n\n  /** \\brief basic default constructor */\n  gather_type() { }\n\n  /**\n   * \\brief This constructor computes XtX and Xy and stores the result\n   * in XtX and Xy\n   */\n  gather_type(const vec_type& X, const double y, const float weight) :\n    XtX(X.size(), X.size()), Xy(X.size()) {\n    XtX.triangularView() = X * X.transpose() * weight;\n    Xy = X * y * weight;\n  } // end of constructor for gather type\n\n  /** \\brief Save the values to a binary archive */\n  void save(graphlab::oarchive& arc) const { arc << XtX << Xy << weight; }\n\n  /** \\brief Read the values from a binary archive */\n  void load(graphlab::iarchive& arc) { arc >> XtX >> Xy >> weight; }  \n\n  /** \n   * \\brief Computes XtX += other.XtX and Xy += other.Xy updating this\n   * tuples value\n   */\n  gather_type& operator+=(const gather_type& other) {\n    if(other.Xy.size() == 0) {\n      ASSERT_EQ(other.XtX.rows(), 0);\n      ASSERT_EQ(other.XtX.cols(), 0);\n    } else {\n      if(Xy.size() == 0) {\n        ASSERT_EQ(XtX.rows(), 0); \n        ASSERT_EQ(XtX.cols(), 0);\n        XtX = other.XtX; Xy = other.Xy;\n      } else {\n        XtX.triangularView() += other.XtX;  \n        Xy += other.Xy;\n      }\n    }\n    return *this;\n  } // end of operator+=\n\n}; // end of gather type\n\n\n\n/**\n * \\brief WALS vertex program implements the alternating least squares\n * algorithm in the Gather-Apply-Scatter abstraction.\n *\n * The ALS update treats adjacent vertices (rows or columns) as \"X\"\n * (independent) values and the edges (matrix entries) as observed \"y\"\n * (dependent) values and then updates the current vertex value as a\n * weight \"w\" such that:\n *\n *    y = X * w + noise\n *\n * This is accomplished using the following equation:\n *\n *    w = inv(X' * X) * (X * y)\n *\n * We implement this in the Gather-Apply-Scatter model by:\n *\n *  1) Gather: returns the tuple (X' * X, X * y)\n *     Sum:   (aX' * aX, aX * ay) + (bX' * bX, bX * by) = \n *                 (aX' * aX + bX' * bX, aX * ay + bX * by)\n *\n *  2) Apply: Solves  inv(X' * X) * (X * y)\n *\n *  3) Scatter: schedules the update of adjacent vertices if this\n *      vertex has changed sufficiently and the edge is not well\n *      predicted.\n *\n * \n */ \nclass als_vertex_program : \n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\npublic:\n  /** The convergence tolerance */\n  static double TOLERANCE;\n  static double LAMBDA;\n  static size_t MAX_UPDATES;\n  static double MAXVAL;\n  static double MINVAL;\n \n  /** The set of edges to gather along */\n  edge_dir_type gather_edges(icontext_type& context, \n                             const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of gather_edges \n\n  /** The gather function computes XtX and Xy */\n  gather_type gather(icontext_type& context, const vertex_type& vertex, \n                     edge_type& edge) const {\n    if(edge.data().role == edge_data::TRAIN) {\n      const vertex_type other_vertex = get_other_vertex(edge, vertex);\n      return gather_type(other_vertex.data().factor, edge.data().obs, edge.data().weight);\n    } else return gather_type();\n  } // end of gather function\n\n  /** apply collects the sum of XtX and Xy */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& sum) {\n    // Get and reset the vertex data\n    vertex_data& vdata = vertex.data(); \n    // Determine the number of neighbors.  Each vertex has only in or\n    // out edges depending on which side of the graph it is located\n    if(sum.Xy.size() == 0) { vdata.residual = 0; ++vdata.nupdates; return; }\n    mat_type XtX = sum.XtX;\n    vec_type Xy = sum.Xy;\n    // Add regularization\n    for(int i = 0; i < XtX.rows(); ++i) XtX(i,i) += LAMBDA; // /nneighbors;\n    // Solve the least squares problem using eigen ----------------------------\n    const vec_type old_factor = vdata.factor;\n    vdata.factor = XtX.selfadjointView().ldlt().solve(Xy);\n    // Compute the residual change in the factor factor -----------------------\n    vdata.residual = (vdata.factor - old_factor).cwiseAbs().sum() / XtX.rows();\n    ++vdata.nupdates;\n  } // end of apply\n  \n  /** The edges to scatter along */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of scatter edges\n\n  /** Scatter reschedules neighbors */  \n  void scatter(icontext_type& context, const vertex_type& vertex, \n               edge_type& edge) const {\n    edge_data& edata = edge.data();\n    if(edata.role == edge_data::TRAIN) {\n      const vertex_type other_vertex = get_other_vertex(edge, vertex);\n      const vertex_data& vdata = vertex.data();\n      const vertex_data& other_vdata = other_vertex.data();\n      const double pred = vdata.factor.dot(other_vdata.factor);\n      const float error = std::fabs(edata.obs - pred);\n      const double priority = (error * vdata.residual); \n      // Reschedule neighbors ------------------------------------------------\n      if( priority > TOLERANCE && other_vdata.nupdates < MAX_UPDATES) \n        context.signal(other_vertex, priority);\n    }\n  } // end of scatter function\n\n\n  /**\n   * \\brief Signal all vertices on one side of the bipartite graph\n   */\n  static graphlab::empty signal_left(icontext_type& context,\n                                     const vertex_type& vertex) {\n    if(vertex.num_out_edges() > 0) context.signal(vertex);\n    return graphlab::empty();\n  } // end of signal_left \n\n}; // end of als vertex program\n\n\n\n/**\n * \\brief The graph loader function is a line parser used for\n * distributed graph construction.\n */\ninline bool graph_loader(graph_type& graph, \n                         const std::string& filename,\n                         const std::string& line) {\n  \n // Parse the line\n  std::stringstream strm(line);\n  graph_type::vertex_id_type source_id(-1), target_id(-1);\n  float obs(0), weight(1);\n  strm >> source_id >> target_id;\n\n  if (source_id == graph_type::vertex_id_type(-1) || target_id == graph_type::vertex_id_type(-1)){\n    logstream(LOG_WARNING)<<\"Failed to read input line: \"<< line << \" in file: \"  << filename << \" (or node id is -1). \" << std::endl;\n    return true;\n  }\n\n  // Determine the role of the data\n  edge_data::data_role_type role = edge_data::TRAIN;\n  if(boost::ends_with(filename,\".validate\")) role = edge_data::VALIDATE;\n  else if(boost::ends_with(filename, \".predict\")) role = edge_data::PREDICT;\n \n  // for test files (.predict) no need to read the actual rating value.\n  if(role == edge_data::TRAIN || role == edge_data::VALIDATE){\n    strm >> obs >> weight;\n    if (obs < als_vertex_program::MINVAL || obs > als_vertex_program::MAXVAL)\n      logstream(LOG_FATAL)<<\"Rating values should be between \" << als_vertex_program::MINVAL << \" and \" << als_vertex_program::MAXVAL << \". Got value: \" << obs << \" [ user: \" << source_id << \" to item: \" < 0) {\n      const double validation_error = \n        std::sqrt(agg.validation_error / info.validation_edges);\n      context.cout() << \"\\t\" << validation_error; \n    }\n    context.cout() << std::endl;\n  }\n}; // end of error aggregator\n\n\n\n\n/**\n * \\brief The prediction saver is used by the graph.save routine to\n * output the final predictions back to the filesystem.\n */\nstruct prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    return \"\"; //nop\n  }\n  std::string save_edge(const edge_type& edge) const {\n    if(edge.data().role == edge_data::PREDICT) {\n      std::stringstream strm;\n      const double prediction = \n        edge.source().data().factor.dot(edge.target().data().factor);\n      strm << edge.source().id() << '\\t';\n      if(REMAP_TARGET) strm << (-edge.target().id() - SAFE_NEG_OFFSET) << '\\t';\n      else strm << edge.target().id() << '\\t';\n      strm << prediction << '\\n';\n      return strm.str();\n    } else return \"\";\n  }\n}; // end of prediction_saver\n\n\nstruct linear_model_saver_U {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n  */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() > 0){\n      std::string ret = boost::lexical_cast(vertex.id()) + \" \";\n      for (uint i=0; i< vertex_data::NLATENT; i++)\n        ret += boost::lexical_cast(vertex.data().factor[i]) + \" \";\n        ret += \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\nstruct linear_model_saver_V {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n  */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() == 0){\n      std::string ret = boost::lexical_cast(-vertex.id()-SAFE_NEG_OFFSET) + \") \";\n      for (uint i=0; i< vertex_data::NLATENT; i++)\n        ret += boost::lexical_cast(vertex.data().factor[i]) + \" \";\n        ret += \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\n\n\n/**\n * \\brief The engine type used by the ALS matrix factorization\n * algorithm.\n *\n * The ALS matrix factorization algorithm currently uses the\n * synchronous engine.  However we plan to add support for alternative\n * engines in the future.\n */\ntypedef graphlab::omni_engine engine_type;\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n\n  // Parse command line options -----------------------------------------------\n  const std::string description = \n    \"Compute the Weighted-ALS factorization of a matrix.\";\n  graphlab::command_line_options clopts(description);\n  std::string input_dir, output_dir;\n  std::string predictions;\n  size_t interval = 10;\n  std::string exec_type = \"synchronous\";\n  clopts.attach_option(\"matrix\", input_dir,\n                       \"The directory containing the matrix file\");\n  clopts.add_positional(\"matrix\");\n  clopts.attach_option(\"D\",  vertex_data::NLATENT,\n                       \"Number of latent parameters to use.\");\n  clopts.attach_option(\"max_iter\", als_vertex_program::MAX_UPDATES,\n                       \"The maxumum number of udpates allowed for a vertex\");\n  clopts.attach_option(\"lambda\", als_vertex_program::LAMBDA, \n                       \"wALS regularization weight\"); \n  clopts.attach_option(\"tol\", als_vertex_program::TOLERANCE,\n                       \"residual termination threshold\");\n  clopts.attach_option(\"maxval\", als_vertex_program::MAXVAL, \"max allowed value\");\n  clopts.attach_option(\"minval\", als_vertex_program::MINVAL, \"min allowed value\");\n  clopts.attach_option(\"interval\", interval, \n                       \"The time in seconds between error reports\");\n  clopts.attach_option(\"predictions\", predictions,\n                       \"The prefix (folder and filename) to save predictions.\");\n  clopts.attach_option(\"engine\", exec_type, \n                       \"The engine type synchronous or asynchronous\");\n  // clopts.attach_option(\"remap\", REMAP_TARGET,\n  //                      \"Renumber target vertex ids (internally) so that they\\n\" \n  //                      \"are in a different range allowing user 0 to connect to movie 0\");\n  clopts.attach_option(\"output\", output_dir,\n                       \"Output results\");\n\n  parse_implicit_command_line(clopts);\n\n  if(!clopts.parse(argc, argv) || input_dir == \"\") {\n    std::cout << \"Error in parsing command line arguments.\" << std::endl;\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  \n  dc.cout() << \"Loading graph.\" << std::endl;\n  graphlab::timer timer; \n  graph_type graph(dc, clopts);  \n  graph.load(input_dir, graph_loader); \n  dc.cout() << \"Loading graph. Finished in \" \n            << timer.current_time() << std::endl;\n\n  if (dc.procid() == 0) \n    add_implicit_edges4(implicitratingtype, graph, dc);\n  \n  dc.cout() << \"Finalizing graph.\" << std::endl;\n  timer.start();\n  graph.finalize();\n  dc.cout() << \"Finalizing graph. Finished in \" \n            << timer.current_time() << std::endl;\n  if (!graph.num_edges() || !graph.num_vertices())\n     logstream(LOG_FATAL)<< \"Failed to load graph. Check your input path: \" << input_dir << std::endl;     \n\n\n\n  dc.cout() \n      << \"========== Graph statistics on proc \" << dc.procid() \n      << \" ===============\"\n      << \"\\n Num vertices: \" << graph.num_vertices()\n      << \"\\n Num edges: \" << graph.num_edges()\n      << \"\\n Num replica: \" << graph.num_replicas()\n      << \"\\n Replica to vertex ratio: \" \n      << float(graph.num_replicas())/graph.num_vertices()\n      << \"\\n --------------------------------------------\" \n      << \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n      << \"\\n Num local vertices: \" << graph.num_local_vertices()\n      << \"\\n Replica to own ratio: \" \n      << (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n      << \"\\n Num local edges: \" << graph.num_local_edges()\n      //<< \"\\n Begin edge id: \" << graph.global_eid(0)\n      << \"\\n Edge balance ratio: \" \n      << float(graph.num_local_edges())/graph.num_edges()\n      << std::endl;\n \n  dc.cout() << \"Creating engine\" << std::endl;\n  engine_type engine(dc, graph, exec_type, clopts);\n\n  // Add error reporting to the engine\n  const bool success = engine.add_edge_aggregator\n    (\"error\", error_aggregator::map, error_aggregator::finalize) &&\n    engine.aggregate_periodic(\"error\", interval);\n  ASSERT_TRUE(success);\n  \n\n  // Signal all vertices on the vertices on the left (liberals) \n  engine.map_reduce_vertices(als_vertex_program::signal_left);\n  info = graph.map_reduce_edges(count_edges);\n  dc.cout()<<\"Training edges: \" << info.training_edges << \" validation edges: \" << info.validation_edges << std::endl;\n\n \n\n  // Run the WALS ---------------------------------------------------------\n  dc.cout() << \"Running Weighted-ALS\" << std::endl;\n  timer.start();\n  engine.start();  \n\n  const double runtime = timer.current_time();\n  dc.cout() << \"----------------------------------------------------------\"\n            << std::endl\n            << \"Final Runtime (seconds):   \" << runtime \n            << std::endl\n            << \"Updates executed: \" << engine.num_updates() << std::endl\n            << \"Update Rate (updates/second): \" \n            << engine.num_updates() / runtime << std::endl;\n\n  // Compute the final training error -----------------------------------------\n  dc.cout() << \"Final error: \" << std::endl;\n  engine.aggregate_now(\"error\");\n\n  // Make predictions ---------------------------------------------------------\n  if(!predictions.empty()) {\n    std::cout << \"Saving predictions\" << std::endl;\n    const bool gzip_output = false;\n    const bool save_vertices = false;\n    const bool save_edges = true;\n    const size_t threads_per_machine = 2;\n\n    //save the predictions\n    graph.save(predictions, prediction_saver(),\n               gzip_output, save_vertices, \n               save_edges, threads_per_machine);\n    //save the linear model\n    graph.save(predictions + \".U\", linear_model_saver_U(),\n\t\tgzip_output, save_edges, save_vertices, threads_per_machine);\n    graph.save(predictions + \".V\", linear_model_saver_V(),\n\t\tgzip_output, save_edges, save_vertices, threads_per_machine);\n  \n  }\n             \n\n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // end of main\n\n\n\n"
  },
  {
    "path": "toolkits/collaborative_filtering/warp_als_coord.cpp",
    "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * \\file\n *\n * Matrix factorization with the Alternative Least Squares (ALS)  - parallel coordinate descent algorithm.\n * See the papers:\n * H.-F. Yu, C.-J. Hsieh, S. Si, I. S. Dhillon, Scalable Coordinate Descent Approaches to Parallel Matrix Factorization for Recommender Systems. IEEE International Conference on Data Mining(ICDM), December 2012.\n * Steffen Rendle, Zeno Gantner, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2011. Fast context-aware recommendations with factorization machines. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval (SIGIR '11). ACM, New York, NY, USA, 635-644.\n * Written by Danny Bickson, CMU\n */\n\n#include \n#include \n#include \n#include \n#include \n#include \"eigen_serialization.hpp\"\n#include \n\ntypedef Eigen::VectorXd vec_type;\n\n#define ALS_COORD_MAP_REDUCE 0\n#define ALS_COORD_TRANSFORM 1\n//when using negative node id range, we are not allowed to use\n//0 and 1 so we add 2.\nconst static int SAFE_NEG_OFFSET=2;\nconst static int regnormal = 0;\nstatic bool debug;\nint max_iter = 10;\ndouble maxval = 1e100;\ndouble minval = -1e100;\nstd::string predictions;\nbool isuser(uint node){\n  return ((int)node) >= 0;\n}\n\n/**\n * \\ingroup toolkit_matrix_factorization\n *\n * \\brief the vertex data type which contains the latent pvec.\n *\n * Each row and each column in the matrix corresponds to a different\n * vertex in the SGD graph.  Associated with each vertex is a pvec\n * (vector) of latent parameters that represent that vertex.  The goal\n * of the SGD algorithm is to find the values for these latent\n * parameters such that the non-zero entries in the matrix can be\n * predicted by taking the dot product of the row and column pvecs.\n */\nstruct vertex_data {\n  /**\n   * \\brief A shared \"constant\" that specifies the number of latent\n   * values to use.\n   */\n  static size_t NLATENT;\n  /** \\brief The latent pvec for this vertex */\n  vec_type pvec;\n  vec_type prev;\n  float z;\n  int t; //index inside the latent feature vector\n\n  /**\n   * \\brief Simple default constructor which randomizes the vertex\n   *  data\n   */\n  vertex_data() : t(0),z(0) { if (debug) pvec = vec_type::Ones(NLATENT); else randomize(); prev = vec_type::Zero(NLATENT); }\n  /** \\brief Randomizes the latent pvec */\n  void randomize() { pvec.resize(NLATENT); pvec.setRandom(); }\n  /** \\brief Save the vertex data to a binary archive */\n  void save(graphlab::oarchive& arc) const {\n    arc << pvec << t << prev << z;\n  }\n  /** \\brief Load the vertex data from a binary archive */\n  void load(graphlab::iarchive& arc) {\n    arc >> pvec >> t >> prev >> z;\n  }\n}; // end of vertex data\n\nstd::size_t hash_value(vertex_data const& b) {\n  return (size_t)b.pvec[0]*1000;\n}\n\n\n/**\n * \\brief The edge data stores the entry in the matrix.\n *\n * In addition the edge data sgdo stores the most recent error estimate.\n */\nstruct edge_data : public graphlab::IS_POD_TYPE {\n  /**\n   * \\brief The type of data on the edge;\n   *\n   * \\li *Train:* the observed value is correct and used in training\n   * \\li *Validate:* the observed value is correct but not used in training\n   * \\li *Predict:* The observed value is not correct and should not be\n   *        used in training.\n   */\n  enum data_role_type { TRAIN, VALIDATE, PREDICT  };\n\n  /** \\brief the observed value for the edge */\n  float obs;\n  \n  /** \\brief cached value for A_ij - prediction */\n  float R_ij;\n\n  /** \\brief The train/validation/test designation of the edge */\n  data_role_type role;\n\n  /** \\brief basic initialization */\n  edge_data(float obs = 0, data_role_type role = PREDICT) :\n    obs(obs), role(role), R_ij(0) { }\n\n}; // end of edge data\n\nstd::size_t hash_value(edge_data const& b) {\n  return boost::hash_value(b.obs);\n}\n\n\n/**\n * \\brief The graph type is defined in terms of the vertex and edge\n * data.\n */\ntypedef graphlab::distributed_graph graph_type;\ntypedef graphlab::gl3engine engine_type;\n\nbool isuser_node(const graph_type::vertex_type& vertex){\n  return isuser(vertex.id());\n}\n\n\n\n/**\n * \\brief The graph loader function is a line parser used for\n * distributed graph construction.\n */\ninline bool graph_loader(graph_type& graph,\n                         const std::string& filename,\n                         const std::string& line) {\n  ASSERT_FALSE(line.empty());\n  // Determine the role of the data\n  edge_data::data_role_type role = edge_data::TRAIN;\n  if(boost::ends_with(filename,\".validate\")) role = edge_data::VALIDATE;\n  else if(boost::ends_with(filename, \".predict\")) role = edge_data::PREDICT;\n  // Parse the line\n  std::stringstream strm(line);\n  graph_type::vertex_id_type source_id(-1), target_id(-1);\n  float obs(0);\n  strm >> source_id >> target_id;\n\n  // for test files (.predict) no need to read the actual rating value.\n  if(role == edge_data::TRAIN || role == edge_data::VALIDATE) {\n    strm >> obs;\n  }\n  target_id = -(graphlab::vertex_id_type(target_id + SAFE_NEG_OFFSET));\n\n  // Create an edge and add it to the graph\n  graph.add_edge(source_id, target_id, edge_data(obs, role));\n  return true; // successful load\n} // end of graph_loader\n\ndouble LAMBDA = 0.001;\n\nclass gather_type {\npublic:\n  double numerator;\n  double denominator;\n\n  gather_type() { \n    numerator = 0;\n    denominator = 0;\n  }\n\n  gather_type(double numerator, double denominator) : numerator(numerator),\n     denominator(denominator){\n  }\n\n  /** \\brief Save the values to a binary archive */\n  void save(graphlab::oarchive& arc) const { arc << numerator << denominator; }\n\n  /** \\brief Read the values from a binary archive */\n  void load(graphlab::iarchive& arc) { arc >> numerator >> denominator; }  \n\n  /** \n   * sums up values\n   */\n  gather_type& operator+=(const gather_type& other) {\n    numerator += other.numerator;\n    denominator += other.denominator;\n    return *this;\n  } // end of operator+=\n\n}; // end of gather type\n\n\n\ngather_type als_coord_map(const graph_type::vertex_type& center,\n                         graph_type::edge_type& edge,\n                         const graph_type::vertex_type& other) {\n\n   if (center.data().t == 0){\n     double prediction = center.data().pvec.dot(other.data().pvec);\n     prediction = std::min(prediction, maxval);\n     prediction = std::max(prediction, minval);\n     edge.data().R_ij = edge.data().obs - prediction;\n   }\n   //compute numerator of equation (6) in ICDM paper above\n   //             (A_ij        - w_i^T*h_j  + wit          * h_jt        )*h_jt \n   gather_type ret((edge.data().R_ij\n                               + center.data().pvec[center.data().t] * other.data().pvec[center.data().t])*other.data().pvec[center.data().t],\n   //compute denominator of equation (6) in ICDM paper above\n   //h_jt^2\n     pow(other.data().pvec[center.data().t], 2));\n   return ret;\n\n}\n\nvoid als_coord_transform(const graph_type::vertex_type& center,\n                         graph_type::edge_type& edge,\n                         const graph_type::vertex_type& other) {\n   //update using equation (7) in ICDM paper\n   //R_ij     -= (z             - w_it         )*h_jt\n   edge.data().R_ij -= (center.data().z - center.data().prev[center.data().t])*other.data().pvec[center.data().t];\n}\n\n\n//sum up two numerators and denomenators\nvoid als_coord_combine(gather_type& v1, const gather_type& v2) {\n    v1 += v2;\n}\n\n//the main update function\nvoid als_coord_function(engine_type::context_type& context,\n                  graph_type::vertex_type& vertex) {\n       \n   double regularization = LAMBDA;\n   for (vertex.data().t=0; vertex.data().t< (int)vertex_data::NLATENT; vertex.data().t++){\n     gather_type frac =  context.map_reduce(ALS_COORD_MAP_REDUCE, graphlab::ALL_EDGES);\n     assert(frac.denominator > 0);\n     vertex.data().z = (frac.numerator/(frac.denominator+regularization));  \n     vertex.data().prev = vertex.data().pvec;\n     //update using equation (8) in ICDM paper\n     //w_it                              = z;\n     vertex.data().pvec[vertex.data().t] = vertex.data().z;\n  \n     //update the cached R_ij using equation (7) in ICDM paper \n     context.edge_transform(ALS_COORD_TRANSFORM, graphlab::ALL_EDGES);\n   }\n\n}\n\n\n/**\n * \\brief Given an edge compute the error associated with that edge\n */\ndouble extract_l2_error(const graph_type::edge_type & edge) {\n  double pred =\n      edge.source().data().pvec.dot(edge.target().data().pvec);\n  double rmse = (edge.data().obs - pred) * (edge.data().obs - pred);\n  return rmse;\n} // end of extract_l2_error\n\n\n\nsize_t vertex_data::NLATENT = 20;\n/**\n * \\brief The prediction saver is used by the graph.save routine to\n * output the final predictions back to the filesystem.\n */\nstruct prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    return \"\"; //nop\n  }\n  std::string save_edge(const edge_type& edge) const {\n    if(edge.data().role == edge_data::PREDICT) {\n      std::stringstream strm;\n      double prediction = \n        edge.source().data().pvec.dot(edge.target().data().pvec);\n      prediction = std::min(prediction, maxval);\n      prediction = std::max(prediction, minval);\n      strm << edge.source().id() << '\\t';\n      strm << (-edge.target().id() - SAFE_NEG_OFFSET) << '\\t';\n      strm << prediction << '\\n';\n      return strm.str();\n    } else return \"\";\n  }\n}; // end of prediction_saver\n\n\nstruct linear_model_saver_U {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n  */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() > 0){\n      std::string ret = boost::lexical_cast(vertex.id()) + \" \";\n      for (uint i=0; i< vertex_data::NLATENT; i++)\n        ret += boost::lexical_cast(vertex.data().pvec[i]) + \" \";\n        ret += \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\nstruct linear_model_saver_V {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  /* save the linear model, using the format:\n     nodeid) factor1 factor2 ... factorNLATENT \\n\n  */\n  std::string save_vertex(const vertex_type& vertex) const {\n    if (vertex.num_out_edges() == 0){\n      std::string ret = boost::lexical_cast(-vertex.id()-SAFE_NEG_OFFSET) + \") \";\n      for (uint i=0; i< vertex_data::NLATENT; i++)\n        ret += boost::lexical_cast(vertex.data().pvec[i]) + \" \";\n        ret += \"\\n\";\n      return ret;\n    }\n    else return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\n\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n\n  // Parse command line options -----------------------------------------------\n  const std::string description =\n      \"Compute the ALS factorization of a matrix.\";\n  graphlab::command_line_options clopts(description);\n  std::string input_dir;\n  std::string exec_type = \"synchronous\";\n  clopts.attach_option(\"matrix\", input_dir,\n                       \"The directory containing the matrix file\");\n  clopts.add_positional(\"matrix\");\n  clopts.attach_option(\"D\", vertex_data::NLATENT,\n                       \"Number of latent parameters to use.\");\n  clopts.attach_option(\"maxval\", maxval, \"max allowed value\");\n  clopts.attach_option(\"minval\", minval, \"min allowed value\");\n  clopts.attach_option(\"predictions\", predictions,\n                       \"The prefix (folder and filename) to save predictions.\");\n  clopts.attach_option(\"lambda\", LAMBDA,\n                       \"regularization weight\");\n  clopts.attach_option(\"max_iter\", max_iter,\n                       \"number of iterations\");\n  if(!clopts.parse(argc, argv) || input_dir == \"\") {\n    std::cout << \"Error in parsing command line arguments.\" << std::endl;\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n  dc.cout() << \"Loading graph.\" << std::endl;\n  graphlab::timer timer;\n  graph_type graph(dc, clopts);\n  graph.load(input_dir, graph_loader);\n  dc.cout() << \"Loading graph. Finished in \"\n            << timer.current_time() << std::endl;\n\n  dc.cout() << \"Finalizing graph.\" << std::endl;\n  timer.start();\n  graph.finalize();\n  dc.cout() << \"Finalizing graph. Finished in \"\n            << timer.current_time() << std::endl;\n\n  if (!graph.num_edges() || !graph.num_vertices())\n     logstream(LOG_FATAL)<< \"Failed to load graph. Check your input path: \" << input_dir << std::endl;     \n\n\n  dc.cout()\n      << \"========== Graph statistics on proc \" << dc.procid()\n      << \" ===============\"\n      << \"\\n Num vertices: \" << graph.num_vertices()\n      << \"\\n Num edges: \" << graph.num_edges()\n      << \"\\n Num replica: \" << graph.num_replicas()\n      << \"\\n Replica to vertex ratio: \"\n      << float(graph.num_replicas())/graph.num_vertices()\n      << \"\\n --------------------------------------------\"\n      << \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n      << \"\\n Num local vertices: \" << graph.num_local_vertices()\n      << \"\\n Replica to own ratio: \"\n      << (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n      << \"\\n Num local edges: \" << graph.num_local_edges()\n      //<< \"\\n Begin edge id: \" << graph.global_eid(0)\n        << \"\\n Edge balance ratio: \"\n        << float(graph.num_local_edges())/graph.num_edges()\n        << std::endl;\n\n  dc.cout() << \"Creating engine\" << std::endl;\n\n  engine_type engine(dc, graph, clopts);\n  engine.register_map_reduce(ALS_COORD_MAP_REDUCE, als_coord_map, als_coord_combine);\n  engine.register_edge_transform(ALS_COORD_TRANSFORM, als_coord_transform);\n  for (int i=0; i< max_iter; i++){\n     engine.parfor_all_local_vertices(als_coord_function);\n     engine.wait();\n     double rmse = graph.map_reduce_edges(extract_l2_error);\n     dc.cout() << \"RMSE = \" << sqrt(rmse / graph.num_edges()) << std::endl;\n  }\n\n  const double runtime = timer.current_time();\n  dc.cout() << \"----------------------------------------------------------\"\n            << std::endl\n            << \"Final Runtime (seconds):   \" << runtime;\n\n  // Compute the final training error -----------------------------------------\n  dc.cout() << \"Final error: \" << std::endl;\n  // Make predictions ---------------------------------------------------------\n  if(!predictions.empty()) {\n    std::cout << \"Saving predictions\" << std::endl;\n    const bool gzip_output = false;\n    const bool save_vertices = false;\n    const bool save_edges = true;\n    const size_t threads_per_machine = 2;\n\n    //save the predictions\n    graph.save(predictions, prediction_saver(),\n               gzip_output, save_vertices, \n               save_edges, threads_per_machine);\n    //save the linear model\n    graph.save(predictions + \".U\", linear_model_saver_U(),\n\t\tgzip_output, save_edges, save_vertices, threads_per_machine);\n    graph.save(predictions + \".V\", linear_model_saver_V(),\n\t\tgzip_output, save_edges, save_vertices, threads_per_machine);\n  \n  }\n \n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // end of main\n\n\n\n"
  },
  {
    "path": "toolkits/collaborative_filtering/warp_nmf.cpp",
    "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * \\file\n *\n * \\brief The main file for the NMF matrix factorization algorithm.\n *\n */\n\n#include \n#include \n#include \n#include \n#include \"eigen_serialization.hpp\"\n#include \n\n#define VERTEX_DELTA_TASK_ID 0\n\ntypedef Eigen::VectorXd vec_type;\ntypedef Eigen::MatrixXd mat_type;\n\n//when using negative node id range, we are not allowed to use\n//0 and 1 so we add 2.\nconst static int SAFE_NEG_OFFSET=2;\nstatic bool debug;\nint iter = 0;\nenum { PHASE1, PHASE2};\nint phase = PHASE1;\ndouble epsilon = 1e-16;\n\nbool isuser(uint node){\n  return ((int)node) >= 0;\n}\n\n/**\n * \\ingroup toolkit_matrix_pvecization\n *\n * \\brief the vertex data type which contains the latent pvec.\n *\n * Each row and each column in the matrix corresponds to a different\n * vertex in the SGD graph.  Associated with each vertex is a pvec\n * (vector) of latent parameters that represent that vertex.  The goal\n * of the SGD algorithm is to find the values for these latent\n * parameters such that the non-zero entries in the matrix can be\n * predicted by taking the dot product of the row and column pvecs.\n */\nstruct vertex_data {\n  /**\n   * \\brief A shared \"constant\" that specifies the number of latent\n   * values to use.\n   */\n  static size_t NLATENT;\n  /** \\brief The latent pvec for this vertex */\n  vec_type pvec;\n\n  int nupdates;\n\n  /**\n   * \\brief Simple default constructor which randomizes the vertex\n   *  data\n   */\n  vertex_data() { if (debug) pvec = vec_type::Ones(NLATENT); else randomize(); }\n  /** \\brief Randomizes the latent pvec */\n  void randomize() { pvec.resize(NLATENT); pvec.setRandom(); }\n  /** \\brief Save the vertex data to a binary archive */\n  void save(graphlab::oarchive& arc) const {\n    arc << pvec;\n  }\n  /** \\brief Load the vertex data from a binary archive */\n  void load(graphlab::iarchive& arc) {\n    arc >> pvec;\n  }\n}; // end of vertex data\n\nstd::size_t hash_value(vertex_data const& b) {\n  return b.nupdates;\n}\n\n\n/**\n * \\brief The edge data stores the entry in the matrix.\n *\n * In addition the edge data sgdo stores the most recent error estimate.\n */\nstruct edge_data : public graphlab::IS_POD_TYPE {\n  /**\n   * \\brief The type of data on the edge;\n   *\n   * \\li *Train:* the observed value is correct and used in training\n   * \\li *Validate:* the observed value is correct but not used in training\n   * \\li *Predict:* The observed value is not correct and should not be\n   *        used in training.\n   */\n  enum data_role_type { TRAIN, VALIDATE, PREDICT  };\n\n  /** \\brief the observed value for the edge */\n  float obs;\n\n  /** \\brief The train/validation/test designation of the edge */\n  data_role_type role;\n\n  /** \\brief basic initialization */\n  edge_data(float obs = 0, data_role_type role = PREDICT) :\n    obs(obs), role(role) { }\n\n}; // end of edge data\n\nstd::size_t hash_value(edge_data const& b) {\n  return boost::hash_value(b.obs);\n}\n\n\n/**\n * \\brief The graph type is defined in terms of the vertex and edge\n * data.\n */\ntypedef graphlab::distributed_graph graph_type;\ntypedef graphlab::gl3engine engine_type;\n\n\nvec_type x1;\nvec_type x2;\nvec_type * px;\n\nbool isuser_node(const graph_type::vertex_type& vertex){\n  return isuser(vertex.id());\n}\n\n\n/**\n * \\brief The graph loader function is a line parser used for\n * distributed graph construction.\n */\ninline bool graph_loader(graph_type& graph,\n                         const std::string& filename,\n                         const std::string& line) {\n  ASSERT_FALSE(line.empty());\n  // Determine the role of the data\n  edge_data::data_role_type role = edge_data::TRAIN;\n  if(boost::ends_with(filename,\".validate\")) role = edge_data::VALIDATE;\n  else if(boost::ends_with(filename, \".predict\")) role = edge_data::PREDICT;\n  // Parse the line\n  std::stringstream strm(line);\n  graph_type::vertex_id_type source_id(-1), target_id(-1);\n  float obs(0);\n  strm >> source_id >> target_id;\n\n  // for test files (.predict) no need to read the actual rating value.\n  if(role == edge_data::TRAIN || role == edge_data::VALIDATE) {\n    strm >> obs;\n  }\n  target_id = -(graphlab::vertex_id_type(target_id + SAFE_NEG_OFFSET));\n\n  // Create an edge and add it to the graph\n  graph.add_edge(source_id, target_id, edge_data(obs, role));\n  return true; // successful load\n} // end of graph_loader\n\n\n\nvoid vertex_delta(graph_type::vertex_type& vtx, const vec_type& delta) {\n  if (delta.sum() != 0)\n    vtx.data().pvec.array() *= delta.array() / px->array(); \n  for (uint i=0; i< vertex_data::NLATENT; i++)\n    if (vtx.data().pvec[i] < epsilon)\n      vtx.data().pvec[i] = epsilon;\n}\n\nvoid nmf_function(engine_type::context_type& context,\n                  graph_type::edge_type& edge) {\n  double pred = edge.source().data().pvec.dot(edge.target().data().pvec);\n  if (pred == 0)\n     logstream(LOG_FATAL)<<\"Got into numerical error!\" << std::endl;    \n  vec_type delta;\n  delta = (phase == PHASE1 ? edge.target().data().pvec : edge.source().data().pvec) * edge.data().obs / pred;\n  context.send_delta(VERTEX_DELTA_TASK_ID, phase == PHASE1 ? edge.source() : edge.target(), delta);\n}\n\n\nvec_type count_edges(const graph_type::edge_type& edge) {\n  vec_type ret = vec_type::Zero(2);\n  if (edge.data().role == edge_data::TRAIN){\n    ret[0] = 1;\n  }\n  else if (edge.data().role == edge_data::VALIDATE){\n    ret[1] = 1;\n  }\n  if (edge.data().obs < 0)\n    logstream(LOG_FATAL)<<\"Found a negative entry in matirx row \" << edge.source().id() << \" with value: \" << edge.data().obs << std::endl;\n  return ret;\n}\n\nvoid verify_rows(\n    graph_type::vertex_type& vertex){\n        if (isuser(vertex.id()) && vertex.num_out_edges() == 0)\n          logstream(LOG_FATAL)<<\"NMF algorithm can not work when the row \" << vertex.id() << \" of the matrix contains all zeros\" << std::endl;\n}\n\nvec_type pre_iter( const graph_type::vertex_type & vertex){\n  return vertex.data().pvec;\n}\n\n\nvoid sync_function(engine_type::context_type& context,\n                   graph_type::vertex_type& vertex) {\n  context.synchronize(vertex);\n}\n\n/**\n * \\brief Given an edge compute the error associated with that edge\n */\ndouble extract_l2_error(const graph_type::edge_type & edge) {\n  double pred =\n      edge.source().data().pvec.dot(edge.target().data().pvec);\n  double rmse = (edge.data().obs - pred) * (edge.data().obs - pred);\n  return rmse;\n} // end of extract_l2_error\n\n\n\nsize_t vertex_data::NLATENT = 20;\n\n\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n\n  // Parse command line options -----------------------------------------------\n  const std::string description =\n      \"Compute the ALS factorization of a matrix.\";\n  graphlab::command_line_options clopts(description);\n  std::string input_dir;\n  size_t interval = 0;\n  size_t ITERATIONS = 10;\n  std::string exec_type = \"synchronous\";\n  clopts.attach_option(\"matrix\", input_dir,\n                       \"The directory containing the matrix file\");\n  clopts.add_positional(\"matrix\");\n  clopts.attach_option(\"D\", vertex_data::NLATENT,\n                       \"Number of latent parameters to use.\");\n  clopts.attach_option(\"interval\", interval,\n                       \"The time in seconds between error reports\");\n  clopts.attach_option(\"iterations\", ITERATIONS,\n                       \"number of NMF iterations\");\n  if(!clopts.parse(argc, argv) || input_dir == \"\") {\n    std::cout << \"Error in parsing command line arguments.\" << std::endl;\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n  dc.cout() << \"Loading graph.\" << std::endl;\n  graphlab::timer timer;\n  graph_type graph(dc, clopts);\n  graph.load(input_dir, graph_loader);\n  dc.cout() << \"Loading graph. Finished in \"\n            << timer.current_time() << std::endl;\n\n  dc.cout() << \"Finalizing graph.\" << std::endl;\n  timer.start();\n  graph.finalize();\n  dc.cout() << \"Finalizing graph. Finished in \"\n            << timer.current_time() << std::endl;\n\n  if (!graph.num_edges() || !graph.num_vertices())\n     logstream(LOG_FATAL)<< \"Failed to load graph. Check your input path: \" << input_dir << std::endl;     \n\n\n\n  dc.cout()\n      << \"========== Graph statistics on proc \" << dc.procid()\n      << \" ===============\"\n      << \"\\n Num vertices: \" << graph.num_vertices()\n      << \"\\n Num edges: \" << graph.num_edges()\n      << \"\\n Num replica: \" << graph.num_replicas()\n      << \"\\n Replica to vertex ratio: \"\n      << float(graph.num_replicas())/graph.num_vertices()\n      << \"\\n --------------------------------------------\"\n      << \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n      << \"\\n Num local vertices: \" << graph.num_local_vertices()\n      << \"\\n Replica to own ratio: \"\n      << (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n      << \"\\n Num local edges: \" << graph.num_local_edges()\n      //<< \"\\n Begin edge id: \" << graph.global_eid(0)\n        << \"\\n Edge balance ratio: \"\n        << float(graph.num_local_edges())/graph.num_edges()\n        << std::endl;\n\n  dc.cout() << \"Creating engine\" << std::endl;\n\n\n  engine_type engine(dc, graph, clopts);\n  vec_type edge_count = graph.map_reduce_edges(count_edges);\n  dc.cout()<<\"Training edges: \" << edge_count[0] << \" validation edges: \" << edge_count[1] << std::endl;\n\n  graphlab::vertex_set left = graph.select(isuser_node);\n  graphlab::vertex_set right = ~left;\n  graph.transform_vertices(verify_rows, left);\n\n  engine.register_vertex_delta(VERTEX_DELTA_TASK_ID, vertex_delta);\n\n  dc.cout() << \"Running NMF\" << std::endl;\n\n  timer.start();\n  for (size_t i = 0;i < ITERATIONS; ++i) {\n    phase = PHASE1;\n    x1 = graph.map_reduce_vertices(pre_iter,right);\n    px = &x1;\n    dc.cout() <<\"x1 is: \" << x1 << std::endl;\n\n    engine.parfor_all_local_edges(nmf_function); //todo - only left\n    engine.parfor_all_local_vertices(sync_function); //todo - only left\n    engine.wait();\n\n    phase = PHASE2;\n\n    x2 = graph.map_reduce_vertices(pre_iter,left);\n    px = &x2;\n    dc.cout() <<\"x2 is: \" << x2 << std::endl;\n\n    engine.parfor_all_local_edges(nmf_function); //todo only right\n    engine.parfor_all_local_vertices(sync_function); //todo only right\n    engine.wait();\n   \n    double rmse = graph.map_reduce_edges(extract_l2_error);\n    dc.cout() << \"RMSE = \" << sqrt(rmse / graph.num_edges()) << std::endl;\n \n  }\n\n\n\n\n  const double runtime = timer.current_time();\n  dc.cout() << \"----------------------------------------------------------\"\n            << std::endl\n            << \"Final Runtime (seconds):   \" << runtime;\n\n  // Compute the final training error -----------------------------------------\n  dc.cout() << \"Final error: \" << std::endl;\n\n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // end of main\n\n\n\n"
  },
  {
    "path": "toolkits/computer_vision/CMakeLists.txt",
    "content": "project(ComputerVision)\n\nadd_graphlab_executable(grabcut grabcut.cpp)\nrequires_opencv(grabcut)\n\nadd_graphlab_executable(stitching stitching.cpp)\nrequires_opencv(stitching)\n\nadd_graphlab_executable(stitching_detailed stitching_detailed.cpp)\nrequires_opencv(stitching_detailed)\n\nadd_graphlab_executable(stitch stitch_main.cpp)\nrequires_opencv(stitch)\nrequires_eigen(stitch)\n\nadd_graphlab_executable(stitch_full stitch_full_main.cpp)\nrequires_opencv(stitch_full)\nrequires_eigen(stitch_full)\ntarget_link_libraries( stitch_full ${OpenCV_LIBS} )\nset (CMAKE_C_FLAGS \"-g -Wall\")\nset (CMAKE_CXX_FLAGS \"-g -Wall\")\n\n"
  },
  {
    "path": "toolkits/computer_vision/computer_vision.dox",
    "content": "/**\n\n\\page computer_vision Computer Vision\n\n\n\\brief GraphLab Computer Vision Toolkit aims to provide fully distributed wrappers to algorithms in OpenCV, an open-source library aimed at real-time computer vision. \nEventually, GraphLab Computer Vision Toolkit will become it’s own spin-off project called CloudCV, a system that will provide access to state-of-the-art computer vision algorithms on the cloud.\n\nCurrently, the only implemented algorithm is Image-Stitching, where the goal is to create a composite panoramic image from a collection of images.\n\n\\section image_stitching Panoramic Image Stitching\n\n\\image html panorama_small.png\n\nThe goal in image stiching is to create a composite panoramic image from a collection of images. The standard pipeline consists of four main steps:\n\n\\li Feature Extraction: where distinctive points (or keypoints) are identified in each image and a feature descriptor (SIFT, SURF, etc) is computed for each keypoint.\n\\li Image/Feature Matching: where features are matched between pairs of images to estimate relative camera transformations.\n\\li Global Refinement: of camera transformation parameters across all images.\n\\li Seam Blending: where seams are estimated between pairs of images and blending is performed.\n\nSee the following for details about the pipeline:\n\\verbatim\nM. Brown and D. Lowe. \nAutomatic Panoramic Image Stitching using Invariant Features. \nInternational Journal of Computer Vision, 74(1), pages 59-73, 2007.\n\\endverbatim\n\nThe stiching code in this toolkit is based on OpenCV Stitching Module.\n\nImplemented by Dhruv Batra and Prakriti Banik.\n\n\\section running_stitch Running Stitch\n\nThe program requires a directory that contains all images from the panorama. Currently (and temporarily for now), \nthe program also requires an adjacency list indicating the overlap between images. We are working on incorporating code that will estimate this adjacency list directly from the images. \n\n\\verbatim\n> ./stitch --img /path/to/image/dir --graph /path/to/adjacency/list.txt \n\\endverbatim\n\nThe adjacency list file format stores on each line, \na vertex (image id), followed by a list of all vertices (image ids) that contain overlapping visual content. Each line has the following format:\n\n[image ID]  [number of neighbouring vertices/images] [neighbour-image ID 1] [neighbour-image ID 2] [neighbour-image ID 3] ...\n\nHere's an example adjacency list file with 3 images (numbered 0,1,2) in a chain graph (0-1-2):\n\\verbatim\n0 1 1\n1 2 0 2\n2 1 1\n\\endverbatim\n\n\n\\section computer_vision_options Options\n\nOther arguments are:\n\n\\li --help Display the help message describing the list of\noptions.\n\n\\li --output (Optional, default \"./\") The output directory in which to save\nthe final mosiac.\n\n\\li --verbose (Optional, default 0) How much information to print out.\n\n\\li --work_megapix (Optional, default 0.6 Mpx) Resolution for image matching step. See other details in stitch_opts.hpp.\n\n\\li --engine (Optional, Default: asynchronous) The engine type to\nuse when executing the vertex-programs\n       - synchronous: All LoopyBP updates are run at the same\n         time (Synchronous BP). This engine exposes greater parallelism but is less\n         computationally efficient.\n       - asynchronous: LoopyBP updates are run asynchronous\n         with priorities (Residual BP).  This engine is has greater\n         overhead and exposes less parallelism but can substantially\n         improve the rate over convergence.\n\n\\li --ncpus (Optional, Default 2) The number of local computation \nthreads to use on each machine.  This should typically match the number \nof physical cores. \n\n\\li --scheduler (Optional, Default sweep) The scheduler to use when \nrunning with the asynchronous engine.  The default is typically sufficient. \n\n\\li --engine_opts (Optional, Default empty) Any additional engine\noptions. See --engine_help for a list of options.\n\n\n\\li --graph_opts (Optional, Default empty) Any additional graph\noptions. See --graph_help for a list of options.\n\n\\li --scheduler_opts (Optional, Default empty) Any additional scheduler\noptions. See --scheduler_help for a list of options.\n\n*/\n\n\n"
  },
  {
    "path": "toolkits/computer_vision/eigen_serialization.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n#include \"eigen_serialization.hpp\"\n\n\n\ngraphlab::oarchive& operator<<(graphlab::oarchive& arc, const Eigen::VectorXd& vec) {\n  typedef Eigen::VectorXd::Index index_type;\n  typedef Eigen::VectorXd::Scalar scalar_type;\n  const index_type size = vec.size();\n  arc << size;\n  graphlab::serialize(arc, vec.data(), size * sizeof(scalar_type));\n  return arc;\n} // end of save vector\n\ngraphlab::iarchive& operator>>(graphlab::iarchive& arc, Eigen::VectorXd& vec) {\n  typedef Eigen::VectorXd::Index index_type;\n  typedef Eigen::VectorXd::Scalar scalar_type;\n  index_type size = 0;\n  arc >> size;\n  vec.resize(size);\n  graphlab::deserialize(arc, vec.data(), size * sizeof(scalar_type));\n  return arc;\n} // end of save vector\n\n\ngraphlab::oarchive& operator<<(graphlab::oarchive& arc, const Eigen::MatrixXd& mat) {\n  typedef Eigen::MatrixXd::Index index_type;\n  typedef Eigen::MatrixXd::Scalar scalar_type;\n  const index_type rows = mat.rows();\n  const index_type cols = mat.cols();\n  arc << rows << cols;\n  graphlab::serialize(arc, mat.data(), rows*cols*sizeof(scalar_type));\n  return arc;\n} // end of save matrix\n\ngraphlab::iarchive& operator>>(graphlab::iarchive& arc,  Eigen::MatrixXd& mat) {\n  typedef Eigen::MatrixXd::Index index_type; \n  typedef Eigen::MatrixXd::Scalar scalar_type;\n  index_type rows=0, cols=0;\n  arc >> rows >> cols;\n  mat.resize(rows,cols);\n  graphlab::deserialize(arc, mat.data(), rows*cols*sizeof(scalar_type));\n  return arc;\n} // end of load matrix\n"
  },
  {
    "path": "toolkits/computer_vision/eigen_serialization.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n#ifndef EIGEN_SERIALIZATION_HPP\n#define EIGEN_SERIALIZATION_HPP\n\n\n\n#include \n\n#include \n\n\n\nBEGIN_OUT_OF_PLACE_SAVE(arc, Eigen::VectorXd, vec) {\n  typedef Eigen::VectorXd::Index index_type;\n  typedef Eigen::VectorXd::Scalar scalar_type;\n  const index_type size = vec.size();\n  arc << size;\n  graphlab::serialize(arc, vec.data(), size * sizeof(scalar_type));\n} END_OUT_OF_PLACE_SAVE()\n\n\nBEGIN_OUT_OF_PLACE_LOAD(arc, Eigen::VectorXd, vec) {\n  typedef Eigen::VectorXd::Index index_type;\n  typedef Eigen::VectorXd::Scalar scalar_type;\n  index_type size = 0;\n  arc >> size;\n  vec.resize(size);\n  graphlab::deserialize(arc, vec.data(), size * sizeof(scalar_type));\n} END_OUT_OF_PLACE_LOAD()\n\n\nBEGIN_OUT_OF_PLACE_SAVE(arc, Eigen::MatrixXd, mat) {\n  typedef Eigen::MatrixXd::Index index_type;\n  typedef Eigen::MatrixXd::Scalar scalar_type;\n  const index_type rows = mat.rows();\n  const index_type cols = mat.cols();\n  arc << rows << cols;\n  graphlab::serialize(arc, mat.data(), rows*cols*sizeof(scalar_type));\n} END_OUT_OF_PLACE_SAVE()\n\n\nBEGIN_OUT_OF_PLACE_LOAD(arc, Eigen::MatrixXd, mat) {\n  typedef Eigen::MatrixXd::Index index_type; \n  typedef Eigen::MatrixXd::Scalar scalar_type;\n  index_type rows=0, cols=0;\n  arc >> rows >> cols;\n  mat.resize(rows,cols);\n  graphlab::deserialize(arc, mat.data(), rows*cols*sizeof(scalar_type));\n} END_OUT_OF_PLACE_LOAD()\n\n\n\n\n\n// inline graphlab::oarchive& operator<<(graphlab::oarchive& arc, const Eigen::VectorXd& vec) {\n//   typedef Eigen::VectorXd::Index index_type;\n//   typedef Eigen::VectorXd::Scalar scalar_type;\n//   const index_type size = vec.size();\n//   arc << size;\n//   graphlab::serialize(arc, vec.data(), size * sizeof(scalar_type));\n//   return arc;\n// } // end of save vector\n\n// inline graphlab::iarchive& operator>>(graphlab::iarchive& arc, Eigen::VectorXd& vec) {\n//   typedef Eigen::VectorXd::Index index_type;\n//   typedef Eigen::VectorXd::Scalar scalar_type;\n//   index_type size = 0;\n//   arc >> size;\n//   vec.resize(size);\n//   graphlab::deserialize(arc, vec.data(), size * sizeof(scalar_type));\n//   return arc;\n// } // end of save vector\n\n\n// inline graphlab::oarchive& operator<<(graphlab::oarchive& arc, const Eigen::MatrixXd& mat) {\n//   typedef Eigen::MatrixXd::Index index_type;\n//   typedef Eigen::MatrixXd::Scalar scalar_type;\n//   const index_type rows = mat.rows();\n//   const index_type cols = mat.cols();\n//   arc << rows << cols;\n//   graphlab::serialize(arc, mat.data(), rows*cols*sizeof(scalar_type));\n//   return arc;\n// } // end of save matrix\n\n// inline graphlab::iarchive& operator>>(graphlab::iarchive& arc,  Eigen::MatrixXd& mat) {\n//   typedef Eigen::MatrixXd::Index index_type; \n//   typedef Eigen::MatrixXd::Scalar scalar_type;\n//   index_type rows=0, cols=0;\n//   arc >> rows >> cols;\n//   mat.resize(rows,cols);\n//   graphlab::deserialize(arc, mat.data(), rows*cols*sizeof(scalar_type));\n//   return arc;\n// } // end of load matrix\n\n\n#endif\n"
  },
  {
    "path": "toolkits/computer_vision/gcgraph.hpp",
    "content": "/*M///////////////////////////////////////////////////////////////////////////////////////\n//\n//  IMPORTANT: READ BEFORE DOWNLOADING, COPYING, INSTALLING OR USING.\n//\n//  By downloading, copying, installing or using the software you agree to this license.\n//  If you do not agree to this license, do not download, install,\n//  copy or use the software.\n//\n//\n//                        Intel License Agreement\n//                For Open Source Computer Vision Library\n//\n// Copyright (C) 2000, Intel Corporation, all rights reserved.\n// Third party copyrights are property of their respective owners.\n//\n// Redistribution and use in source and binary forms, with or without modification,\n// are permitted provided that the following conditions are met:\n//\n//   * Redistribution's of source code must retain the above copyright notice,\n//     this list of conditions and the following disclaimer.\n//\n//   * Redistribution's in binary form must reproduce the above copyright notice,\n//     this list of conditions and the following disclaimer in the documentation\n//     and/or other materials provided with the distribution.\n//\n//   * The name of Intel Corporation may not be used to endorse or promote products\n//     derived from this software without specific prior written permission.\n//\n// This software is provided by the copyright holders and contributors \"as is\" and\n// any express or implied warranties, including, but not limited to, the implied\n// warranties of merchantability and fitness for a particular purpose are disclaimed.\n// In no event shall the Intel Corporation or contributors be liable for any direct,\n// indirect, incidental, special, exemplary, or consequential damages\n// (including, but not limited to, procurement of substitute goods or services;\n// loss of use, data, or profits; or business interruption) however caused\n// and on any theory of liability, whether in contract, strict liability,\n// or tort (including negligence or otherwise) arising in any way out of\n// the use of this software, even if advised of the possibility of such damage.\n//\n//M*/\n\n#ifndef _CV_GCGRAPH_H_\n#define _CV_GCGRAPH_H_\n\ntemplate  class GCGraph\n{\npublic:\n    GCGraph();\n    GCGraph( unsigned int vtxCount, unsigned int edgeCount );\n    ~GCGraph();\n    void create( unsigned int vtxCount, unsigned int edgeCount );\n    int addVtx();\n    void addEdges( int i, int j, TWeight w, TWeight revw );\n    void addTermWeights( int i, TWeight sourceW, TWeight sinkW );\n    TWeight maxFlow();\n    bool inSourceSegment( int i );\nprivate:\n    class Vtx\n    {\n    public:\n        Vtx *next; // initialized and used in maxFlow() only\n        int parent;\n        int first;\n        int ts;\n        int dist;\n        TWeight weight;\n        uchar t; \n    };\n    class Edge\n    {\n    public:\n        int dst;\n        int next;\n        TWeight weight;\n    };\n\n    std::vector vtcs;\n    std::vector edges;\n    TWeight flow;\n};\n\ntemplate \nGCGraph::GCGraph()\n{\n    flow = 0;\n}\ntemplate \nGCGraph::GCGraph( unsigned int vtxCount, unsigned int edgeCount )\n{\n    create( vtxCount, edgeCount );\n}\ntemplate \nGCGraph::~GCGraph()\n{\n}\ntemplate \nvoid GCGraph::create( unsigned int vtxCount, unsigned int edgeCount )\n{\n    vtcs.reserve( vtxCount );\n    edges.reserve( edgeCount + 2 );\n    flow = 0;\n}\n\ntemplate \nint GCGraph::addVtx()\n{\n    Vtx v;\n    memset( &v, 0, sizeof(Vtx));\n    vtcs.push_back(v);\n    return (int)vtcs.size() - 1;\n}\n\ntemplate \nvoid GCGraph::addEdges( int i, int j, TWeight w, TWeight revw )\n{\n    CV_Assert( i>=0 && i<(int)vtcs.size() );\n    CV_Assert( j>=0 && j<(int)vtcs.size() );\n    CV_Assert( w>=0 && revw>=0 );\n    CV_Assert( i != j );\n\n    if( !edges.size() )\n        edges.resize( 2 );\n\n    Edge fromI, toI;\n    fromI.dst = j;\n    fromI.next = vtcs[i].first;\n    fromI.weight = w;\n    vtcs[i].first = (int)edges.size();\n    edges.push_back( fromI );\n\n    toI.dst = i;\n    toI.next = vtcs[j].first;\n    toI.weight = revw;\n    vtcs[j].first = (int)edges.size();\n    edges.push_back( toI );\n}\n\ntemplate \nvoid GCGraph::addTermWeights( int i, TWeight sourceW, TWeight sinkW )\n{\n    CV_Assert( i>=0 && i<(int)vtcs.size() );\n\n    TWeight dw = vtcs[i].weight;\n    if( dw > 0 )\n        sourceW += dw;\n    else\n        sinkW -= dw;\n    flow += (sourceW < sinkW) ? sourceW : sinkW;\n    vtcs[i].weight = sourceW - sinkW;\n}\n\ntemplate \nTWeight GCGraph::maxFlow()\n{\n    const int TERMINAL = -1, ORPHAN = -2;\n    Vtx stub, *nilNode = &stub, *first = nilNode, *last = nilNode;\n    int curr_ts = 0;\n    stub.next = nilNode;\n    Vtx *vtxPtr = &vtcs[0];\n    Edge *edgePtr = &edges[0];\n\n    std::vector orphans;\n\n    // initialize the active queue and the graph vertices\n    for( int i = 0; i < (int)vtcs.size(); i++ )\n    {\n        Vtx* v = vtxPtr + i;\n        v->ts = 0;\n        if( v->weight != 0 )\n        {\n            last = last->next = v;\n            v->dist = 1;\n            v->parent = TERMINAL;\n            v->t = v->weight < 0;\n        }\n        else\n            v->parent = 0;        \n    }\n    first = first->next;\n    last->next = nilNode;\n    nilNode->next = 0;\n\n    // run the search-path -> augment-graph -> restore-trees loop\n    for(;;)\n    {\n        Vtx* v, *u;\n        int e0 = -1, ei = 0, ej = 0;\n        TWeight minWeight, weight;\n        uchar vt;\n\n        // grow S & T search trees, find an edge connecting them\n        while( first != nilNode )\n        {\n            v = first;\n            if( v->parent )\n            {\n                vt = v->t;\n                for( ei = v->first; ei != 0; ei = edgePtr[ei].next )\n                {\n                    if( edgePtr[ei^vt].weight == 0 )\n                        continue;\n                    u = vtxPtr+edgePtr[ei].dst;\n                    if( !u->parent )\n                    {\n                        u->t = vt;\n                        u->parent = ei ^ 1;\n                        u->ts = v->ts;\n                        u->dist = v->dist + 1;\n                        if( !u->next )\n                        {\n                            u->next = nilNode;\n                            last = last->next = u;\n                        }\n                        continue;\n                    }\n\n                    if( u->t != vt )\n                    {\n                        e0 = ei ^ vt;\n                        break;\n                    }\n\n                    if( u->dist > v->dist+1 && u->ts <= v->ts )\n                    {\n                        // reassign the parent\n                        u->parent = ei ^ 1;\n                        u->ts = v->ts;\n                        u->dist = v->dist + 1;\n                    }\n                }\n                if( e0 > 0 )\n                    break;\n            }\n            // exclude the vertex from the active list\n            first = first->next;\n            v->next = 0;\n        }\n\n        if( e0 <= 0 )\n            break;\n\n        // find the minimum edge weight along the path\n        minWeight = edgePtr[e0].weight;\n        assert( minWeight > 0 );\n        // k = 1: source tree, k = 0: destination tree\n        for( int k = 1; k >= 0; k-- )\n        {\n            for( v = vtxPtr+edgePtr[e0^k].dst;; v = vtxPtr+edgePtr[ei].dst )\n            {\n                if( (ei = v->parent) < 0 )\n                    break;\n                weight = edgePtr[ei^k].weight;\n                minWeight = MIN(minWeight, weight);\n                assert( minWeight > 0 );\n            }\n            weight = fabs(v->weight);\n            minWeight = MIN(minWeight, weight);\n            assert( minWeight > 0 );\n        }\n\n        // modify weights of the edges along the path and collect orphans\n        edgePtr[e0].weight -= minWeight;\n        edgePtr[e0^1].weight += minWeight;\n        flow += minWeight;\n\n        // k = 1: source tree, k = 0: destination tree\n        for( int k = 1; k >= 0; k-- )\n        {\n            for( v = vtxPtr+edgePtr[e0^k].dst;; v = vtxPtr+edgePtr[ei].dst )\n            {\n                if( (ei = v->parent) < 0 )\n                    break;\n                edgePtr[ei^(k^1)].weight += minWeight;\n                if( (edgePtr[ei^k].weight -= minWeight) == 0 )\n                {\n                    orphans.push_back(v);\n                    v->parent = ORPHAN;\n                }\n            }\n\n            v->weight = v->weight + minWeight*(1-k*2);\n            if( v->weight == 0 )\n            {\n               orphans.push_back(v);\n               v->parent = ORPHAN;\n            }\n        }\n\n        // restore the search trees by finding new parents for the orphans\n        curr_ts++;\n        while( !orphans.empty() )\n        {\n            Vtx* v = orphans.back();\n            orphans.pop_back();\n\n            int d, minDist = INT_MAX;\n            e0 = 0;\n            vt = v->t;\n\n            for( ei = v->first; ei != 0; ei = edgePtr[ei].next )\n            {\n                if( edgePtr[ei^(vt^1)].weight == 0 )\n                    continue;\n                u = vtxPtr+edgePtr[ei].dst;\n                if( u->t != vt || u->parent == 0 )\n                    continue;\n                // compute the distance to the tree root\n                for( d = 0;; )\n                {\n                    if( u->ts == curr_ts )\n                    {\n                        d += u->dist;\n                        break;\n                    }\n                    ej = u->parent;\n                    d++;\n                    if( ej < 0 )\n                    {\n                        if( ej == ORPHAN )\n                            d = INT_MAX-1;\n                        else\n                        {\n                            u->ts = curr_ts;\n                            u->dist = 1;\n                        }\n                        break;\n                    }\n                    u = vtxPtr+edgePtr[ej].dst;\n                }\n\n                // update the distance\n                if( ++d < INT_MAX )\n                {\n                    if( d < minDist )\n                    {\n                        minDist = d;\n                        e0 = ei;\n                    }\n                    for( u = vtxPtr+edgePtr[ei].dst; u->ts != curr_ts; u = vtxPtr+edgePtr[u->parent].dst )\n                    {\n                        u->ts = curr_ts;\n                        u->dist = --d;\n                    }\n                }\n            }\n\n            if( (v->parent = e0) > 0 )\n            {\n                v->ts = curr_ts;\n                v->dist = minDist;\n                continue;\n            }\n\n            /* no parent is found */\n            v->ts = 0;\n            for( ei = v->first; ei != 0; ei = edgePtr[ei].next )\n            {\n                u = vtxPtr+edgePtr[ei].dst;\n                ej = u->parent;\n                if( u->t != vt || !ej )\n                    continue;\n                if( edgePtr[ei^(vt^1)].weight && !u->next )\n                {\n                    u->next = nilNode;\n                    last = last->next = u;\n                }\n                if( ej > 0 && vtxPtr+edgePtr[ej].dst == v )\n                {\n                    orphans.push_back(u);\n                    u->parent = ORPHAN;\n                }\n            }\n        }\n    }\n    return flow;\n}\n\ntemplate \nbool GCGraph::inSourceSegment( int i )\n{\n    CV_Assert( i>=0 && i<(int)vtcs.size() );\n    return vtcs[i].t == 0;\n};\n\n#endif\n"
  },
  {
    "path": "toolkits/computer_vision/grabcut.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \"opencv2/highgui/highgui.hpp\"\n#include \"opencv2/imgproc/imgproc.hpp\"\n\n#include \n\nusing namespace std;\nusing namespace cv;\n\nvoid help()\n{\n    cout << \"\\nThis program demonstrates GrabCut segmentation -- select an object in a region\\n\"\n    \t\t\"and then grabcut will attempt to segment it out.\\n\"\n    \t\t\"Call:\\n\"\n    \t\t\"./grabcut \\n\"\n    \t\"\\nSelect a rectangular area around the object you want to segment\\n\" <<\n        \"\\nHot keys: \\n\"\n        \"\\tESC - quit the program\\n\"\n        \"\\tr - restore the original image\\n\"\n        \"\\tn - next iteration\\n\"\n        \"\\n\"\n        \"\\tleft mouse button - set rectangle\\n\"\n        \"\\n\"\n        \"\\tCTRL+left mouse button - set GC_BGD pixels\\n\"\n        \"\\tSHIFT+left mouse button - set CG_FGD pixels\\n\"\n        \"\\n\"\n        \"\\tCTRL+right mouse button - set GC_PR_BGD pixels\\n\"\n        \"\\tSHIFT+right mouse button - set CG_PR_FGD pixels\\n\" << endl;\n}\n\nconst Scalar RED = Scalar(0,0,255);\nconst Scalar PINK = Scalar(230,130,255);\nconst Scalar BLUE = Scalar(255,0,0);\nconst Scalar LIGHTBLUE = Scalar(255,255,160);\nconst Scalar GREEN = Scalar(0,255,0);\n\nconst int BGD_KEY = CV_EVENT_FLAG_CTRLKEY;\nconst int FGD_KEY = CV_EVENT_FLAG_SHIFTKEY;\n\nvoid getBinMask( const Mat& comMask, Mat& binMask )\n{\n    if( comMask.empty() || comMask.type()!=CV_8UC1 )\n        CV_Error( CV_StsBadArg, \"comMask is empty or has incorrect type (not CV_8UC1)\" );\n    if( binMask.empty() || binMask.rows!=comMask.rows || binMask.cols!=comMask.cols )\n        binMask.create( comMask.size(), CV_8UC1 );\n    binMask = comMask & 1;\n}\n\nclass GCApplication\n{\npublic:\n    enum{ NOT_SET = 0, IN_PROCESS = 1, SET = 2 };\n    static const int radius = 2;\n    static const int thickness = -1;\n\n    void reset();\n    void setImageAndWinName( const Mat& _image, const string& _winName );\n    void showImage() const;\n    void mouseClick( int event, int x, int y, int flags, void* param );\n    int nextIter();\n    int getIterCount() const { return iterCount; }\nprivate:\n    void setRectInMask();\n    void setLblsInMask( int flags, Point p, bool isPr );\n\n    const string* winName;\n    const Mat* image;\n    Mat mask;\n    Mat bgdModel, fgdModel;\n\n    uchar rectState, lblsState, prLblsState;\n    bool isInitialized;\n\n    Rect rect;\n    vector fgdPxls, bgdPxls, prFgdPxls, prBgdPxls;\n    int iterCount;\n};\n\nvoid GCApplication::reset()\n{\n    if( !mask.empty() )\n        mask.setTo(Scalar::all(GC_BGD));\n    bgdPxls.clear(); fgdPxls.clear();\n    prBgdPxls.clear();  prFgdPxls.clear();\n\n    isInitialized = false;\n    rectState = NOT_SET;\n    lblsState = NOT_SET;\n    prLblsState = NOT_SET;\n    iterCount = 0;\n}\n\nvoid GCApplication::setImageAndWinName( const Mat& _image, const string& _winName  )\n{\n    if( _image.empty() || _winName.empty() )\n        return;\n    image = &_image;\n    winName = &_winName;\n    mask.create( image->size(), CV_8UC1);\n    reset();\n}\n\nvoid GCApplication::showImage() const\n{\n    if( image->empty() || winName->empty() )\n        return;\n\n    Mat res;\n    Mat binMask;\n    if( !isInitialized )\n        image->copyTo( res );\n    else\n    {\n        getBinMask( mask, binMask );\n        image->copyTo( res, binMask );\n    }\n\n    vector::const_iterator it;\n    for( it = bgdPxls.begin(); it != bgdPxls.end(); ++it )\n        circle( res, *it, radius, BLUE, thickness );\n    for( it = fgdPxls.begin(); it != fgdPxls.end(); ++it )\n        circle( res, *it, radius, RED, thickness );\n    for( it = prBgdPxls.begin(); it != prBgdPxls.end(); ++it )\n        circle( res, *it, radius, LIGHTBLUE, thickness );\n    for( it = prFgdPxls.begin(); it != prFgdPxls.end(); ++it )\n        circle( res, *it, radius, PINK, thickness );\n\n    if( rectState == IN_PROCESS || rectState == SET )\n        rectangle( res, Point( rect.x, rect.y ), Point(rect.x + rect.width, rect.y + rect.height ), GREEN, 2);\n\n    imshow( *winName, res );\n}\n\nvoid GCApplication::setRectInMask()\n{\n    assert( !mask.empty() );\n    mask.setTo( GC_BGD );\n    rect.x = max(0, rect.x);\n    rect.y = max(0, rect.y);\n    rect.width = min(rect.width, image->cols-rect.x);\n    rect.height = min(rect.height, image->rows-rect.y);\n    (mask(rect)).setTo( Scalar(GC_PR_FGD) );\n}\n\nvoid GCApplication::setLblsInMask( int flags, Point p, bool isPr )\n{\n    vector *bpxls, *fpxls;\n    uchar bvalue, fvalue;\n    if( !isPr )\n    {\n        bpxls = &bgdPxls;\n        fpxls = &fgdPxls;\n        bvalue = GC_BGD;\n        fvalue = GC_FGD;\n    }\n    else\n    {\n        bpxls = &prBgdPxls;\n        fpxls = &prFgdPxls;\n        bvalue = GC_PR_BGD;\n        fvalue = GC_PR_FGD;\n    }\n    if( flags & BGD_KEY )\n    {\n        bpxls->push_back(p);\n        circle( mask, p, radius, bvalue, thickness );\n    }\n    if( flags & FGD_KEY )\n    {\n        fpxls->push_back(p);\n        circle( mask, p, radius, fvalue, thickness );\n    }\n}\n\nvoid GCApplication::mouseClick( int event, int x, int y, int flags, void* )\n{\n    // TODO add bad args check\n    switch( event )\n    {\n    case CV_EVENT_LBUTTONDOWN: // set rect or GC_BGD(GC_FGD) labels\n        {\n            bool isb = (flags & BGD_KEY) != 0,\n                 isf = (flags & FGD_KEY) != 0;\n            if( rectState == NOT_SET && !isb && !isf )\n            {\n                rectState = IN_PROCESS;\n                rect = Rect( x, y, 1, 1 );\n            }\n            if ( (isb || isf) && rectState == SET )\n                lblsState = IN_PROCESS;\n        }\n        break;\n    case CV_EVENT_RBUTTONDOWN: // set GC_PR_BGD(GC_PR_FGD) labels\n        {\n            bool isb = (flags & BGD_KEY) != 0,\n                 isf = (flags & FGD_KEY) != 0;\n            if ( (isb || isf) && rectState == SET )\n                prLblsState = IN_PROCESS;\n        }\n        break;\n    case CV_EVENT_LBUTTONUP:\n        if( rectState == IN_PROCESS )\n        {\n            rect = Rect( Point(rect.x, rect.y), Point(x,y) );\n            rectState = SET;\n            setRectInMask();\n            assert( bgdPxls.empty() && fgdPxls.empty() && prBgdPxls.empty() && prFgdPxls.empty() );\n            showImage();\n        }\n        if( lblsState == IN_PROCESS )\n        {\n            setLblsInMask(flags, Point(x,y), false);\n            lblsState = SET;\n            showImage();\n        }\n        break;\n    case CV_EVENT_RBUTTONUP:\n        if( prLblsState == IN_PROCESS )\n        {\n            setLblsInMask(flags, Point(x,y), true);\n            prLblsState = SET;\n            showImage();\n        }\n        break;\n    case CV_EVENT_MOUSEMOVE:\n        if( rectState == IN_PROCESS )\n        {\n            rect = Rect( Point(rect.x, rect.y), Point(x,y) );\n            assert( bgdPxls.empty() && fgdPxls.empty() && prBgdPxls.empty() && prFgdPxls.empty() );\n            showImage();\n        }\n        else if( lblsState == IN_PROCESS )\n        {\n            setLblsInMask(flags, Point(x,y), false);\n            showImage();\n        }\n        else if( prLblsState == IN_PROCESS )\n        {\n            setLblsInMask(flags, Point(x,y), true);\n            showImage();\n        }\n        break;\n    }\n}\n\nint GCApplication::nextIter()\n{\n    if( isInitialized )\n        grabCut( *image, mask, rect, bgdModel, fgdModel, 1 );\n    else\n    {\n        if( rectState != SET )\n            return iterCount;\n\n        if( lblsState == SET || prLblsState == SET )\n            grabCut( *image, mask, rect, bgdModel, fgdModel, 1, GC_INIT_WITH_MASK );\n        else\n            grabCut( *image, mask, rect, bgdModel, fgdModel, 1, GC_INIT_WITH_RECT );\n\n        isInitialized = true;\n    }\n    iterCount++;\n\n    bgdPxls.clear(); fgdPxls.clear();\n    prBgdPxls.clear(); prFgdPxls.clear();\n\n    return iterCount;\n}\n\nGCApplication gcapp;\n\nvoid on_mouse( int event, int x, int y, int flags, void* param )\n{\n    gcapp.mouseClick( event, x, y, flags, param );\n}\n\nint main( int argc, char** argv )\n{\n    if( argc!=2 )\n    {\n    \thelp();\n        return 1;\n    }\n    string filename = argv[1];\n    if( filename.empty() )\n    {\n    \tcout << \"\\nDurn, couldn't read in \" << argv[1] << endl;\n        return 1;\n    }\n    Mat image = imread( filename, 1 );\n    if( image.empty() )\n    {\n        cout << \"\\n Durn, couldn't read image filename \" << filename << endl;\n    \treturn 1;\n    }\n\n    help();\n\n    const string winName = \"image\";\n    cvNamedWindow( winName.c_str(), CV_WINDOW_AUTOSIZE );\n    cvSetMouseCallback( winName.c_str(), on_mouse, 0 );\n\n    gcapp.setImageAndWinName( image, winName );\n    gcapp.showImage();\n\n    for(;;)\n    {\n        int c = cvWaitKey(0);\n        switch( (char) c )\n        {\n        case '\\x1b':\n            cout << \"Exiting ...\" << endl;\n            goto exit_main;\n        case 'r':\n            cout << endl;\n            gcapp.reset();\n            gcapp.showImage();\n            break;\n        case 'n':\n            int iterCount = gcapp.getIterCount();\n            cout << \"<\" << iterCount << \"... \";\n            int newIterCount = gcapp.nextIter();\n            if( newIterCount > iterCount )\n            {\n                gcapp.showImage();\n                cout << iterCount << \">\" << endl;\n            }\n            else\n                cout << \"rect must be determined>\" << endl;\n            break;\n        }\n    }\n\nexit_main:\n    cvDestroyWindow( winName.c_str() );\n    return 0;\n}\n"
  },
  {
    "path": "toolkits/computer_vision/opencv_serialization.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n#include \"opencv_serialization.hpp\"\n\n\n\n//////////////////////////////////////////////////\n// For Size\ngraphlab::oarchive& operator<<(graphlab::oarchive& arc, const cv::Size& img_size) \n{\n    arc << img_size.width << img_size.height;  \n    return arc;\n} \n\ngraphlab::iarchive& operator>>(graphlab::iarchive& arc, cv::Size& img_size) \n{\n    arc >> img_size.width >> img_size.height;  \n    return arc;\n} \n\n\n//////////////////////////////////////////////////\n// For Point2f\ngraphlab::oarchive& operator<<(graphlab::oarchive& arc, const cv::Point2f& pt) \n{\n    arc << pt.x << pt.y;\n    return arc;\n} \n\ngraphlab::iarchive& operator>>(graphlab::iarchive& arc, cv::Point2f& pt) \n{\n    arc >> pt.x >> pt.y;\n    return arc;\n} \n\n\n//////////////////////////////////////////////////\n// For KeyPoint\ngraphlab::oarchive& operator<<(graphlab::oarchive& arc, const cv::KeyPoint& keypoint) \n{\n    arc << keypoint.pt \n    << keypoint.size << keypoint.angle << keypoint.response\n    << keypoint.octave << keypoint.class_id;\n    return arc;\n} \n\ngraphlab::iarchive& operator>>(graphlab::iarchive& arc, cv::KeyPoint& keypoint) \n{\n    arc >> keypoint.pt \n    >> keypoint.size >> keypoint.angle >> keypoint.response\n    >> keypoint.octave >> keypoint.class_id;\n    return arc;\n} \n\n\n//////////////////////////////////////////////////\n// For Mat\ngraphlab::oarchive& operator<<(graphlab::oarchive& arc, const cv::Mat& mat) \n{\n    size_t elem_size = mat.elemSize();\n    size_t elem_type = mat.type();\n    \n    arc << mat.cols << mat.rows \n    << elem_size << elem_type;\n    \n    const size_t data_size = mat.cols * mat.rows * elem_size;\n    graphlab::serialize(arc, mat.ptr(), data_size);\n    return arc;\n} \n\ngraphlab::iarchive& operator>>(graphlab::oarchive& arc, const cv::Mat& mat) \n{\n    int cols, rows; size_t elem_size, elem_type;\n    \n    arc >> cols >> rows \n    >> elem_size >> elem_type;\n    \n    mat.create(rows, cols, elem_type);\n    \n    size_t data_size = mat.cols * mat.rows * elem_size;\n    graphlab::deserialize(arc, mat.ptr(), data_size);\n    return arc;\n} \n\n\n//////////////////////////////////////////////////\n// For ImageFeatures\ngraphlab::oarchive& operator<<(graphlab::oarchive& arc, const cv::detail::ImageFeatures& features) \n{\n    arc << features.img_idx << features.img_size\n    << features.keypoints\n    << features.descriptors;    \n    return arc;\n} \n\n\ngraphlab::iarchive& operator>>(graphlab::oarchive& arc, const cv::detail::ImageFeatures& features) \n{\n    arc >> features.img_idx >> features.img_size\n    >> features.keypoints\n    >> features.descriptors;\n    return arc;\n} \n\n\n//////////////////////////////////////////////////\n// For DMatch\ngraphlab::oarchive& operator<<(graphlab::oarchive& arc, const cv::DMatch& match) \n{\n    arc << match.queryIdx << match.trainIdx << match.imgIdx\n    << match.distance;     \n    return arc;\n}\n\n\ngraphlab::iarchive& operator>>(graphlab::oarchive& arc, const cv::DMatch& match) \n{\n    arc >> match.queryIdx >> match.trainIdx >> match.imgIdx\n    >> match.distance;     \n    return arc;\n}\n\n\n//////////////////////////////////////////////////\n// For MatchesInfo \ngraphlab::oarchive& operator<<(graphlab::oarchive& arc, const cv::detail::MatchesInfo& matchesinfo) \n{\n    arc << matchesinfo.src_img_idx << matchesinfo.dst_img_idx\n    << matchesinfo.matches \n    << matchesinfo.inliers_mask << matchesinfo.num_inliers\n    << matchesinfo.H \n    << matchesinfo.confidence;\n    return arc;\n}\n\n\ngraphlab::iarchive& operator>>(graphlab::oarchive& arc, const cv::detail::MatchesInfo& matchesinfo) \n{\n    arc >> matchesinfo.src_img_idx >> matchesinfo.dst_img_idx\n    >> matchesinfo.matches \n    >> matchesinfo.inliers_mask >> matchesinfo.num_inliers\n    >> matchesinfo.H \n    >> matchesinfo.confidence;\n    return arc;\n}\n\n\n//////////////////////////////////////////////////\n// For CameraParams\ngraphlab::oarchive& operator<<(graphlab::oarchive& arc, const cv::detail::CameraParams& camera) \n{\n    arc << camera.focal << camera.aspect \n    << camera.ppx << camera.ppy\n    << camera.R << camera.t;\n    return arc;\n} \n\n\ngraphlab::iarchive& operator>>(graphlab::oarchive& arc, const cv::detail::CameraParams& camera) \n{\n    arc >> camera.focal >> camera.aspect \n    >> camera.ppx >> camera.ppy\n    >> camera.R >> camera.t;\n    return arc;\n} \n"
  },
  {
    "path": "toolkits/computer_vision/opencv_serialization.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n#ifndef OPENCV_SERIALIZATION_HPP\n#define OPENCV_SERIALIZATION_HPP\n\n#include \n\n#include \"opencv2/opencv_modules.hpp\"\n#include \"opencv2/opencv.hpp\"\n#include \"opencv2/stitching/stitcher.hpp\"\n\n\n//////////////////////////////////////////////////\n// For Size\nBEGIN_OUT_OF_PLACE_SAVE(arc, cv::Size, img_size) \n{\n    arc << img_size.width << img_size.height;\n} END_OUT_OF_PLACE_SAVE()\n\n\nBEGIN_OUT_OF_PLACE_LOAD(arc, cv::Size, img_size) \n{\n    arc >> img_size.width >> img_size.height;\n} END_OUT_OF_PLACE_LOAD()\n\n\n//////////////////////////////////////////////////\n// For Point2f\nBEGIN_OUT_OF_PLACE_SAVE(arc, cv::Point2f, pt) \n{\n    arc << pt.x << pt.y;\n} END_OUT_OF_PLACE_SAVE()\n\n\nBEGIN_OUT_OF_PLACE_LOAD(arc, cv::Point2f, pt) \n{\n    arc >> pt.x >> pt.y;\n} END_OUT_OF_PLACE_LOAD()\n\n\n//////////////////////////////////////////////////\n// For KeyPoint\nBEGIN_OUT_OF_PLACE_SAVE(arc, cv::KeyPoint, keypoint) \n{\n    arc << keypoint.pt \n    << keypoint.size << keypoint.angle << keypoint.response\n    << keypoint.octave << keypoint.class_id;\n} END_OUT_OF_PLACE_SAVE()\n\n\nBEGIN_OUT_OF_PLACE_LOAD(arc, cv::KeyPoint, keypoint) \n{\n    arc >> keypoint.pt \n    >> keypoint.size >> keypoint.angle >> keypoint.response\n    >> keypoint.octave >> keypoint.class_id;\n} END_OUT_OF_PLACE_LOAD()\n\n\n//////////////////////////////////////////////////\n// For Mat\nBEGIN_OUT_OF_PLACE_SAVE(arc, cv::Mat, mat) \n{\n    size_t elem_size = mat.elemSize();\n    size_t elem_type = mat.type();\n    \n    arc << mat.cols << mat.rows \n    << elem_size << elem_type;\n    \n    const size_t data_size = mat.cols * mat.rows * elem_size;\n    graphlab::serialize(arc, mat.ptr(), data_size);\n} END_OUT_OF_PLACE_SAVE()\n\n\nBEGIN_OUT_OF_PLACE_LOAD(arc, cv::Mat, mat) \n{\n    int cols, rows; size_t elem_size, elem_type;\n\n    arc >> cols >> rows \n    >> elem_size >> elem_type;\n    \n    mat.create(rows, cols, elem_type);\n    \n    size_t data_size = mat.cols * mat.rows * elem_size;\n    graphlab::deserialize(arc, mat.ptr(), data_size);    \n} END_OUT_OF_PLACE_LOAD()\n\n\n//////////////////////////////////////////////////\n// For ImageFeatures\nBEGIN_OUT_OF_PLACE_SAVE(arc, cv::detail::ImageFeatures, features) \n{\n    arc << features.img_idx << features.img_size\n    << features.keypoints\n    << features.descriptors;\n} END_OUT_OF_PLACE_SAVE()\n\n\nBEGIN_OUT_OF_PLACE_LOAD(arc, cv::detail::ImageFeatures, features) \n{\n    arc >> features.img_idx >> features.img_size\n    >> features.keypoints\n    >> features.descriptors;\n} END_OUT_OF_PLACE_LOAD()\n\n\n//////////////////////////////////////////////////\n// For DMatch\nBEGIN_OUT_OF_PLACE_SAVE(arc, cv::DMatch, match) \n{\n    arc << match.queryIdx << match.trainIdx << match.imgIdx\n    << match.distance;     \n} END_OUT_OF_PLACE_SAVE()\n\n\nBEGIN_OUT_OF_PLACE_LOAD(arc, cv::DMatch, match) \n{\n    arc >> match.queryIdx >> match.trainIdx >> match.imgIdx\n    >> match.distance;     \n} END_OUT_OF_PLACE_LOAD()\n\n\n//////////////////////////////////////////////////\n// For MatchesInfo \nBEGIN_OUT_OF_PLACE_SAVE(arc, cv::detail::MatchesInfo, matchesinfo) \n{\n    arc << matchesinfo.src_img_idx << matchesinfo.dst_img_idx\n    << matchesinfo.matches \n    << matchesinfo.inliers_mask << matchesinfo.num_inliers\n    << matchesinfo.H \n    << matchesinfo.confidence;    \n} END_OUT_OF_PLACE_SAVE()\n\n\nBEGIN_OUT_OF_PLACE_LOAD(arc, cv::detail::MatchesInfo, matchesinfo) \n{\n    arc >> matchesinfo.src_img_idx >> matchesinfo.dst_img_idx\n    >> matchesinfo.matches \n    >> matchesinfo.inliers_mask >> matchesinfo.num_inliers\n    >> matchesinfo.H \n    >> matchesinfo.confidence;\n} END_OUT_OF_PLACE_LOAD()\n\n\n//////////////////////////////////////////////////\n// For CameraParams\nBEGIN_OUT_OF_PLACE_SAVE(arc, cv::detail::CameraParams, camera) \n{\n    arc << camera.focal << camera.aspect \n    << camera.ppx << camera.ppy\n    << camera.R << camera.t;\n} END_OUT_OF_PLACE_SAVE()\n\n\nBEGIN_OUT_OF_PLACE_LOAD(arc, cv::detail::CameraParams, camera) \n{\n    arc >> camera.focal >> camera.aspect \n    >> camera.ppx >> camera.ppy\n    >> camera.R >> camera.t;\n} END_OUT_OF_PLACE_LOAD()\n\n#endif\n"
  },
  {
    "path": "toolkits/computer_vision/precomp.hpp",
    "content": "/*M///////////////////////////////////////////////////////////////////////////////////////\r\n//\r\n//  IMPORTANT: READ BEFORE DOWNLOADING, COPYING, INSTALLING OR USING.\r\n//\r\n//  By downloading, copying, installing or using the software you agree to this license.\r\n//  If you do not agree to this license, do not download, install,\r\n//  copy or use the software.\r\n//\r\n//\r\n//                          License Agreement\r\n//                For Open Source Computer Vision Library\r\n//\r\n// Copyright (C) 2000-2008, Intel Corporation, all rights reserved.\r\n// Copyright (C) 2009, Willow Garage Inc., all rights reserved.\r\n// Third party copyrights are property of their respective owners.\r\n//\r\n// Redistribution and use in source and binary forms, with or without modification,\r\n// are permitted provided that the following conditions are met:\r\n//\r\n//   * Redistribution's of source code must retain the above copyright notice,\r\n//     this list of conditions and the following disclaimer.\r\n//\r\n//   * Redistribution's in binary form must reproduce the above copyright notice,\r\n//     this list of conditions and the following disclaimer in the documentation\r\n//     and/or other materials provided with the distribution.\r\n//\r\n//   * The name of the copyright holders may not be used to endorse or promote products\r\n//     derived from this software without specific prior written permission.\r\n//\r\n// This software is provided by the copyright holders and contributors \"as is\" and\r\n// any express or implied warranties, including, but not limited to, the implied\r\n// warranties of merchantability and fitness for a particular purpose are disclaimed.\r\n// In no event shall the Intel Corporation or contributors be liable for any direct,\r\n// indirect, incidental, special, exemplary, or consequential damages\r\n// (including, but not limited to, procurement of substitute goods or services;\r\n// loss of use, data, or profits; or business interruption) however caused\r\n// and on any theory of liability, whether in contract, strict liability,\r\n// or tort (including negligence or otherwise) arising in any way out of\r\n// the use of this software, even if advised of the possibility of such damage.\r\n//\r\n//M*/\r\n\r\n#ifndef __OPENCV_STITCHING_PRECOMP_H__\r\n#define __OPENCV_STITCHING_PRECOMP_H__\r\n\r\n#ifdef HAVE_CVCONFIG_H \r\n#include \"cvconfig.h\"\r\n#endif\r\n#include \"opencv2/opencv_modules.hpp\"\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \"opencv2/core/core.hpp\"\r\n#include \"opencv2/core/internal.hpp\"\r\n#include \"opencv2/stitching/stitcher.hpp\"\r\n#include \"opencv2/stitching/detail/autocalib.hpp\"\r\n#include \"opencv2/stitching/detail/blenders.hpp\"\r\n#include \"opencv2/stitching/detail/camera.hpp\"\r\n#include \"opencv2/stitching/detail/exposure_compensate.hpp\"\r\n#include \"opencv2/stitching/detail/matchers.hpp\"\r\n#include \"opencv2/stitching/detail/motion_estimators.hpp\"\r\n#include \"opencv2/stitching/detail/seam_finders.hpp\"\r\n#include \"opencv2/stitching/detail/util.hpp\"\r\n#include \"opencv2/stitching/detail/warpers.hpp\"\r\n#include \"opencv2/imgproc/imgproc.hpp\"\r\n#include \"opencv2/features2d/features2d.hpp\"\r\n#include \"opencv2/calib3d/calib3d.hpp\"\r\n#ifdef HAVE_OPENCV_GPU\r\n# include \"opencv2/gpu/gpu.hpp\"\r\n#endif\r\n\r\n//#include \"../../imgproc/src/gcgraph.hpp\"\r\n#include \"gcgraph.hpp\"\r\n\r\n#ifdef HAVE_TEGRA_OPTIMIZATION\r\n# include \"opencv2/stitching/stitching_tegra.hpp\"\r\n#endif\r\n\r\n#endif\r\n"
  },
  {
    "path": "toolkits/computer_vision/seam_finders_gr.hpp",
    "content": "/*M///////////////////////////////////////////////////////////////////////////////////////\r\n//\r\n//  IMPORTANT: READ BEFORE DOWNLOADING, COPYING, INSTALLING OR USING.\r\n//\r\n//  By downloading, copying, installing or using the software you agree to this license.\r\n//  If you do not agree to this license, do not download, install,\r\n//  copy or use the software.\r\n//\r\n//\r\n//                          License Agreement\r\n//                For Open Source Computer Vision Library\r\n//\r\n// Copyright (C) 2000-2008, Intel Corporation, all rights reserved.\r\n// Copyright (C) 2009, Willow Garage Inc., all rights reserved.\r\n// Third party copyrights are property of their respective owners.\r\n//\r\n// Redistribution and use in source and binary forms, with or without modification,\r\n// are permitted provided that the following conditions are met:\r\n//\r\n//   * Redistribution's of source code must retain the above copyright notice,\r\n//     this list of conditions and the following disclaimer.\r\n//\r\n//   * Redistribution's in binary form must reproduce the above copyright notice,\r\n//     this list of conditions and the following disclaimer in the documentation\r\n//     and/or other materials provided with the distribution.\r\n//\r\n//   * The name of the copyright holders may not be used to endorse or promote products\r\n//     derived from this software without specific prior written permission.\r\n//\r\n// This software is provided by the copyright holders and contributors \"as is\" and\r\n// any express or implied warranties, including, but not limited to, the implied\r\n// warranties of merchantability and fitness for a particular purpose are disclaimed.\r\n// In no event shall the Intel Corporation or contributors be liable for any direct,\r\n// indirect, incidental, special, exemplary, or consequential damages\r\n// (including, but not limited to, procurement of substitute goods or services;\r\n// loss of use, data, or profits; or business interruption) however caused\r\n// and on any theory of liability, whether in contract, strict liability,\r\n// or tort (including negligence or otherwise) arising in any way out of\r\n// the use of this software, even if advised of the possibility of such damage.\r\n//\r\n//M*/\r\n\r\n#ifndef __SEAM_FINDERS_GR_HPP__\r\n#define __SEAM_FINDERS_GR_HPP__\r\n\r\n#include \"precomp.hpp\"\r\n#include \"opencv2/stitching/detail/seam_finders.hpp\"\r\n\r\nnamespace cv {\r\nnamespace detail {\r\n\r\nvoid PairwiseSeamFinder::find(const vector &src, const vector &corners,\r\n                              vector &masks)\r\n{\r\n    LOGLN(\"Finding seams...\");\r\n    if (src.size() == 0) \r\n        return;\r\n\r\n    int64 t = getTickCount();\r\n\r\n    images_ = src;\r\n    sizes_.resize(src.size());\r\n    for (size_t i = 0; i < src.size(); ++i)\r\n        sizes_[i] = src[i].size();\r\n    corners_ = corners;\r\n    masks_ = masks;\r\n    run();\r\n\r\n    LOGLN(\"Finding seams, time: \" << ((getTickCount() - t) / getTickFrequency()) << \" sec\");\r\n}\r\n\r\n\r\nvoid PairwiseSeamFinder::run()\r\n{\r\n    for (size_t i = 0; i < sizes_.size() - 1; ++i)\r\n    {\r\n        for (size_t j = i + 1; j < sizes_.size(); ++j)\r\n        {\r\n            Rect roi;\r\n            if (overlapRoi(corners_[i], corners_[j], sizes_[i], sizes_[j], roi))\r\n                findInPair(i, j, roi);\r\n        }\r\n    }\r\n}\r\n\r\n\r\nvoid VoronoiSeamFinder::find(const vector &sizes, const vector &corners,\r\n                             vector &masks)\r\n{\r\n    LOGLN(\"Finding seams...\");\r\n    if (sizes.size() == 0) \r\n        return;\r\n\r\n    int64 t = getTickCount();\r\n\r\n    sizes_ = sizes;\r\n    corners_ = corners;\r\n    masks_ = masks;\r\n    run();\r\n\r\n    LOGLN(\"Finding seams, time: \" << ((getTickCount() - t) / getTickFrequency()) << \" sec\");\r\n}\r\n\r\n\r\nvoid VoronoiSeamFinder::findInPair(size_t first, size_t second, Rect roi)\r\n{\r\n    const int gap = 10;\r\n    Mat submask1(roi.height + 2 * gap, roi.width + 2 * gap, CV_8U);\r\n    Mat submask2(roi.height + 2 * gap, roi.width + 2 * gap, CV_8U);\r\n\r\n    Size img1 = sizes_[first], img2 = sizes_[second];\r\n    Mat mask1 = masks_[first], mask2 = masks_[second];\r\n    Point tl1 = corners_[first], tl2 = corners_[second];\r\n\r\n    // Cut submasks with some gap\r\n    for (int y = -gap; y < roi.height + gap; ++y)\r\n    {\r\n        for (int x = -gap; x < roi.width + gap; ++x)\r\n        {\r\n            int y1 = roi.y - tl1.y + y;\r\n            int x1 = roi.x - tl1.x + x;\r\n            if (y1 >= 0 && x1 >= 0 && y1 < img1.height && x1 < img1.width)\r\n                submask1.at(y + gap, x + gap) = mask1.at(y1, x1);\r\n            else\r\n                submask1.at(y + gap, x + gap) = 0;\r\n\r\n            int y2 = roi.y - tl2.y + y;\r\n            int x2 = roi.x - tl2.x + x;\r\n            if (y2 >= 0 && x2 >= 0 && y2 < img2.height && x2 < img2.width)\r\n                submask2.at(y + gap, x + gap) = mask2.at(y2, x2);\r\n            else\r\n                submask2.at(y + gap, x + gap) = 0;\r\n        }\r\n    }\r\n\r\n    Mat collision = (submask1 != 0) & (submask2 != 0);\r\n    Mat unique1 = submask1.clone(); unique1.setTo(0, collision);\r\n    Mat unique2 = submask2.clone(); unique2.setTo(0, collision);\r\n\r\n    Mat dist1, dist2;\r\n    distanceTransform(unique1 == 0, dist1, CV_DIST_L1, 3);\r\n    distanceTransform(unique2 == 0, dist2, CV_DIST_L1, 3);\r\n\r\n    Mat seam = dist1 < dist2;\r\n\r\n    for (int y = 0; y < roi.height; ++y)\r\n    {\r\n        for (int x = 0; x < roi.width; ++x)\r\n        {\r\n            if (seam.at(y + gap, x + gap))\r\n                mask2.at(roi.y - tl2.y + y, roi.x - tl2.x + x) = 0;\r\n            else\r\n                mask1.at(roi.y - tl1.y + y, roi.x - tl1.x + x) = 0;\r\n        }\r\n    }\r\n}\r\n\r\n\r\nclass GraphCutSeamFinder::Impl : public PairwiseSeamFinder\r\n{\r\npublic:\r\n    Impl(int cost_type, float terminal_cost, float bad_region_penalty)\r\n        : cost_type_(cost_type), terminal_cost_(terminal_cost), bad_region_penalty_(bad_region_penalty) {}\r\n\r\n\t~Impl() {}\r\n\r\n    void find(const vector &src, const vector &corners, vector &masks);\r\n    void findInPair(size_t first, size_t second, Rect roi);\r\n\r\nprivate:\r\n    void setGraphWeightsColor(const Mat &img1, const Mat &img2, \r\n                              const Mat &mask1, const Mat &mask2, GCGraph &graph);\r\n    void setGraphWeightsColorGrad(const Mat &img1, const Mat &img2, const Mat &dx1, const Mat &dx2, \r\n                                  const Mat &dy1, const Mat &dy2, const Mat &mask1, const Mat &mask2, \r\n                                  GCGraph &graph);\r\n\r\n    vector dx_, dy_;\r\n    int cost_type_;\r\n    float terminal_cost_;\r\n    float bad_region_penalty_;\r\n};\r\n\r\n\r\nvoid GraphCutSeamFinder::Impl::find(const vector &src, const vector &corners,\r\n                                    vector &masks)\r\n{\r\n    // Compute gradients\r\n    dx_.resize(src.size());\r\n    dy_.resize(src.size());\r\n    Mat dx, dy;\r\n    for (size_t i = 0; i < src.size(); ++i)\r\n    {\r\n        CV_Assert(src[i].channels() == 3);\r\n        Sobel(src[i], dx, CV_32F, 1, 0);\r\n        Sobel(src[i], dy, CV_32F, 0, 1);\r\n        dx_[i].create(src[i].size(), CV_32F);\r\n        dy_[i].create(src[i].size(), CV_32F);\r\n        for (int y = 0; y < src[i].rows; ++y)\r\n        {\r\n            const Point3f* dx_row = dx.ptr(y);\r\n            const Point3f* dy_row = dy.ptr(y);\r\n            float* dx_row_ = dx_[i].ptr(y);\r\n            float* dy_row_ = dy_[i].ptr(y);\r\n            for (int x = 0; x < src[i].cols; ++x)\r\n            {\r\n                dx_row_[x] = normL2(dx_row[x]);\r\n                dy_row_[x] = normL2(dy_row[x]);\r\n            }\r\n        }\r\n    }\r\n    PairwiseSeamFinder::find(src, corners, masks);\r\n}\r\n\r\n\r\nvoid GraphCutSeamFinder::Impl::setGraphWeightsColor(const Mat &img1, const Mat &img2,\r\n                                                    const Mat &mask1, const Mat &mask2, GCGraph &graph)\r\n{\r\n    const Size img_size = img1.size();\r\n\r\n    // Set terminal weights\r\n    for (int y = 0; y < img_size.height; ++y)\r\n    {\r\n        for (int x = 0; x < img_size.width; ++x)\r\n        {\r\n            int v = graph.addVtx();\r\n            graph.addTermWeights(v, mask1.at(y, x) ? terminal_cost_ : 0.f,\r\n                                    mask2.at(y, x) ? terminal_cost_ : 0.f);\r\n        }\r\n    }\r\n\r\n    // Set regular edge weights\r\n    const float weight_eps = 1.f;\r\n    for (int y = 0; y < img_size.height; ++y)\r\n    {\r\n        for (int x = 0; x < img_size.width; ++x)\r\n        {\r\n            int v = y * img_size.width + x;\r\n            if (x < img_size.width - 1)\r\n            {\r\n                float weight = normL2(img1.at(y, x), img2.at(y, x)) +\r\n                               normL2(img1.at(y, x + 1), img2.at(y, x + 1)) +\r\n                               weight_eps;\r\n                if (!mask1.at(y, x) || !mask1.at(y, x + 1) ||\r\n                    !mask2.at(y, x) || !mask2.at(y, x + 1))\r\n                    weight += bad_region_penalty_;\r\n                graph.addEdges(v, v + 1, weight, weight);\r\n            }\r\n            if (y < img_size.height - 1)\r\n            {\r\n                float weight = normL2(img1.at(y, x), img2.at(y, x)) +\r\n                               normL2(img1.at(y + 1, x), img2.at(y + 1, x)) +\r\n                               weight_eps;\r\n                if (!mask1.at(y, x) || !mask1.at(y + 1, x) ||\r\n                    !mask2.at(y, x) || !mask2.at(y + 1, x))\r\n                    weight += bad_region_penalty_;\r\n                graph.addEdges(v, v + img_size.width, weight, weight);\r\n            }\r\n        }\r\n    }\r\n}\r\n\r\n\r\nvoid GraphCutSeamFinder::Impl::setGraphWeightsColorGrad(\r\n        const Mat &img1, const Mat &img2, const Mat &dx1, const Mat &dx2, \r\n        const Mat &dy1, const Mat &dy2, const Mat &mask1, const Mat &mask2, \r\n        GCGraph &graph)\r\n{\r\n    const Size img_size = img1.size();\r\n\r\n    // Set terminal weights\r\n    for (int y = 0; y < img_size.height; ++y)\r\n    {\r\n        for (int x = 0; x < img_size.width; ++x)\r\n        {\r\n            int v = graph.addVtx();\r\n            graph.addTermWeights(v, mask1.at(y, x) ? terminal_cost_ : 0.f,\r\n                                    mask2.at(y, x) ? terminal_cost_ : 0.f);\r\n        }\r\n    }\r\n\r\n    // Set regular edge weights\r\n    const float weight_eps = 1.f;\r\n    for (int y = 0; y < img_size.height; ++y)\r\n    {\r\n        for (int x = 0; x < img_size.width; ++x)\r\n        {\r\n            int v = y * img_size.width + x;\r\n            if (x < img_size.width - 1)\r\n            {\r\n                float grad = dx1.at(y, x) + dx1.at(y, x + 1) +\r\n                             dx2.at(y, x) + dx2.at(y, x + 1) + weight_eps;\r\n                float weight = (normL2(img1.at(y, x), img2.at(y, x)) +\r\n                                normL2(img1.at(y, x + 1), img2.at(y, x + 1))) / grad + \r\n                               weight_eps;\r\n                if (!mask1.at(y, x) || !mask1.at(y, x + 1) ||\r\n                    !mask2.at(y, x) || !mask2.at(y, x + 1))\r\n                    weight += bad_region_penalty_;\r\n                graph.addEdges(v, v + 1, weight, weight);\r\n            }\r\n            if (y < img_size.height - 1)\r\n            {\r\n                float grad = dy1.at(y, x) + dy1.at(y + 1, x) + \r\n                             dy2.at(y, x) + dy2.at(y + 1, x) + weight_eps;\r\n                float weight = (normL2(img1.at(y, x), img2.at(y, x)) + \r\n                                normL2(img1.at(y + 1, x), img2.at(y + 1, x))) / grad + \r\n                               weight_eps;\r\n                if (!mask1.at(y, x) || !mask1.at(y + 1, x) ||\r\n                    !mask2.at(y, x) || !mask2.at(y + 1, x))\r\n                    weight += bad_region_penalty_;\r\n                graph.addEdges(v, v + img_size.width, weight, weight);\r\n            }\r\n        }\r\n    }\r\n}\r\n\r\n\r\nvoid GraphCutSeamFinder::Impl::findInPair(size_t first, size_t second, Rect roi)\r\n{\r\n    Mat img1 = images_[first], img2 = images_[second];\r\n    Mat dx1 = dx_[first], dx2 = dx_[second];\r\n    Mat dy1 = dy_[first], dy2 = dy_[second];\r\n    Mat mask1 = masks_[first], mask2 = masks_[second];\r\n    Point tl1 = corners_[first], tl2 = corners_[second];\r\n\r\n    const int gap = 10;\r\n    Mat subimg1(roi.height + 2 * gap, roi.width + 2 * gap, CV_32FC3);\r\n    Mat subimg2(roi.height + 2 * gap, roi.width + 2 * gap, CV_32FC3);\r\n    Mat submask1(roi.height + 2 * gap, roi.width + 2 * gap, CV_8U);\r\n    Mat submask2(roi.height + 2 * gap, roi.width + 2 * gap, CV_8U);\r\n    Mat subdx1(roi.height + 2 * gap, roi.width + 2 * gap, CV_32F);\r\n    Mat subdy1(roi.height + 2 * gap, roi.width + 2 * gap, CV_32F);\r\n    Mat subdx2(roi.height + 2 * gap, roi.width + 2 * gap, CV_32F);\r\n    Mat subdy2(roi.height + 2 * gap, roi.width + 2 * gap, CV_32F);\r\n\r\n    // Cut subimages and submasks with some gap\r\n    for (int y = -gap; y < roi.height + gap; ++y)\r\n    {\r\n        for (int x = -gap; x < roi.width + gap; ++x)\r\n        {\r\n            int y1 = roi.y - tl1.y + y;\r\n            int x1 = roi.x - tl1.x + x;\r\n            if (y1 >= 0 && x1 >= 0 && y1 < img1.rows && x1 < img1.cols)\r\n            {\r\n                subimg1.at(y + gap, x + gap) = img1.at(y1, x1);\r\n                submask1.at(y + gap, x + gap) = mask1.at(y1, x1);\r\n                subdx1.at(y + gap, x + gap) = dx1.at(y1, x1);\r\n                subdy1.at(y + gap, x + gap) = dy1.at(y1, x1);\r\n            }\r\n            else\r\n            {\r\n                subimg1.at(y + gap, x + gap) = Point3f(0, 0, 0);\r\n                submask1.at(y + gap, x + gap) = 0;\r\n                subdx1.at(y + gap, x + gap) = 0.f;\r\n                subdy1.at(y + gap, x + gap) = 0.f;\r\n            }\r\n\r\n            int y2 = roi.y - tl2.y + y;\r\n            int x2 = roi.x - tl2.x + x;\r\n            if (y2 >= 0 && x2 >= 0 && y2 < img2.rows && x2 < img2.cols)\r\n            {\r\n                subimg2.at(y + gap, x + gap) = img2.at(y2, x2);\r\n                submask2.at(y + gap, x + gap) = mask2.at(y2, x2);\r\n                subdx2.at(y + gap, x + gap) = dx2.at(y2, x2);\r\n                subdy2.at(y + gap, x + gap) = dy2.at(y2, x2);\r\n            }\r\n            else\r\n            {\r\n                subimg2.at(y + gap, x + gap) = Point3f(0, 0, 0);\r\n                submask2.at(y + gap, x + gap) = 0;\r\n                subdx2.at(y + gap, x + gap) = 0.f;\r\n                subdy2.at(y + gap, x + gap) = 0.f;\r\n            }\r\n        }\r\n    }\r\n\r\n    const int vertex_count = (roi.height + 2 * gap) * (roi.width + 2 * gap);\r\n    const int edge_count = (roi.height - 1 + 2 * gap) * (roi.width + 2 * gap) +\r\n                           (roi.width - 1 + 2 * gap) * (roi.height + 2 * gap);\r\n    GCGraph graph(vertex_count, edge_count);\r\n\r\n    switch (cost_type_)\r\n    {\r\n    case GraphCutSeamFinder::COST_COLOR:\r\n        setGraphWeightsColor(subimg1, subimg2, submask1, submask2, graph);\r\n        break;\r\n    case GraphCutSeamFinder::COST_COLOR_GRAD:\r\n        setGraphWeightsColorGrad(subimg1, subimg2, subdx1, subdx2, subdy1, subdy2, \r\n                                 submask1, submask2, graph);\r\n        break;\r\n    default:\r\n        CV_Error(CV_StsBadArg, \"unsupported pixel similarity measure\");\r\n    }\r\n\r\n    graph.maxFlow();\r\n\r\n    for (int y = 0; y < roi.height; ++y)\r\n    {\r\n        for (int x = 0; x < roi.width; ++x)\r\n        {\r\n            if (graph.inSourceSegment((y + gap) * (roi.width + 2 * gap) + x + gap))\r\n            {\r\n                if (mask1.at(roi.y - tl1.y + y, roi.x - tl1.x + x))\r\n                    mask2.at(roi.y - tl2.y + y, roi.x - tl2.x + x) = 0;\r\n            }\r\n            else\r\n            {\r\n                if (mask2.at(roi.y - tl2.y + y, roi.x - tl2.x + x))\r\n                    mask1.at(roi.y - tl1.y + y, roi.x - tl1.x + x) = 0;\r\n            }\r\n        }\r\n    }\r\n}\r\n\r\n\r\nGraphCutSeamFinder::GraphCutSeamFinder(int cost_type, float terminal_cost, float bad_region_penalty)\r\n    : impl_(new Impl(cost_type, terminal_cost, bad_region_penalty)) {}\r\n\r\nGraphCutSeamFinder::~GraphCutSeamFinder() {}\r\n\r\n\r\nvoid GraphCutSeamFinder::find(const vector &src, const vector &corners,\r\n                              vector &masks)\r\n{\r\n    impl_->find(src, corners, masks);\r\n}\r\n\r\n\r\n#ifdef HAVE_OPENCV_GPU\r\nvoid GraphCutSeamFinderGpu::find(const vector &src, const vector &corners,\r\n                                 vector &masks)\r\n{\r\n    // Compute gradients\r\n    dx_.resize(src.size());\r\n    dy_.resize(src.size());\r\n    Mat dx, dy;\r\n    for (size_t i = 0; i < src.size(); ++i)\r\n    {\r\n        CV_Assert(src[i].channels() == 3);\r\n        Sobel(src[i], dx, CV_32F, 1, 0);\r\n        Sobel(src[i], dy, CV_32F, 0, 1);\r\n        dx_[i].create(src[i].size(), CV_32F);\r\n        dy_[i].create(src[i].size(), CV_32F);\r\n        for (int y = 0; y < src[i].rows; ++y)\r\n        {\r\n            const Point3f* dx_row = dx.ptr(y);\r\n            const Point3f* dy_row = dy.ptr(y);\r\n            float* dx_row_ = dx_[i].ptr(y);\r\n            float* dy_row_ = dy_[i].ptr(y);\r\n            for (int x = 0; x < src[i].cols; ++x)\r\n            {\r\n                dx_row_[x] = normL2(dx_row[x]);\r\n                dy_row_[x] = normL2(dy_row[x]);\r\n            }\r\n        }\r\n    }\r\n    PairwiseSeamFinder::find(src, corners, masks);\r\n}\r\n\r\n\r\nvoid GraphCutSeamFinderGpu::findInPair(size_t first, size_t second, Rect roi)\r\n{\r\n    Mat img1 = images_[first], img2 = images_[second];\r\n    Mat dx1 = dx_[first], dx2 = dx_[second];\r\n    Mat dy1 = dy_[first], dy2 = dy_[second];\r\n    Mat mask1 = masks_[first], mask2 = masks_[second];\r\n    Point tl1 = corners_[first], tl2 = corners_[second];\r\n\r\n    const int gap = 10;\r\n    Mat subimg1(roi.height + 2 * gap, roi.width + 2 * gap, CV_32FC3);\r\n    Mat subimg2(roi.height + 2 * gap, roi.width + 2 * gap, CV_32FC3);\r\n    Mat submask1(roi.height + 2 * gap, roi.width + 2 * gap, CV_8U);\r\n    Mat submask2(roi.height + 2 * gap, roi.width + 2 * gap, CV_8U);\r\n    Mat subdx1(roi.height + 2 * gap, roi.width + 2 * gap, CV_32F);\r\n    Mat subdy1(roi.height + 2 * gap, roi.width + 2 * gap, CV_32F);\r\n    Mat subdx2(roi.height + 2 * gap, roi.width + 2 * gap, CV_32F);\r\n    Mat subdy2(roi.height + 2 * gap, roi.width + 2 * gap, CV_32F);\r\n\r\n    // Cut subimages and submasks with some gap\r\n    for (int y = -gap; y < roi.height + gap; ++y)\r\n    {\r\n        for (int x = -gap; x < roi.width + gap; ++x)\r\n        {\r\n            int y1 = roi.y - tl1.y + y;\r\n            int x1 = roi.x - tl1.x + x;\r\n            if (y1 >= 0 && x1 >= 0 && y1 < img1.rows && x1 < img1.cols)\r\n            {\r\n                subimg1.at(y + gap, x + gap) = img1.at(y1, x1);\r\n                submask1.at(y + gap, x + gap) = mask1.at(y1, x1);\r\n                subdx1.at(y + gap, x + gap) = dx1.at(y1, x1);\r\n                subdy1.at(y + gap, x + gap) = dy1.at(y1, x1);\r\n            }\r\n            else\r\n            {\r\n                subimg1.at(y + gap, x + gap) = Point3f(0, 0, 0);\r\n                submask1.at(y + gap, x + gap) = 0;\r\n                subdx1.at(y + gap, x + gap) = 0.f;\r\n                subdy1.at(y + gap, x + gap) = 0.f;\r\n            }\r\n\r\n            int y2 = roi.y - tl2.y + y;\r\n            int x2 = roi.x - tl2.x + x;\r\n            if (y2 >= 0 && x2 >= 0 && y2 < img2.rows && x2 < img2.cols)\r\n            {\r\n                subimg2.at(y + gap, x + gap) = img2.at(y2, x2);\r\n                submask2.at(y + gap, x + gap) = mask2.at(y2, x2);\r\n                subdx2.at(y + gap, x + gap) = dx2.at(y2, x2);\r\n                subdy2.at(y + gap, x + gap) = dy2.at(y2, x2);\r\n            }\r\n            else\r\n            {\r\n                subimg2.at(y + gap, x + gap) = Point3f(0, 0, 0);\r\n                submask2.at(y + gap, x + gap) = 0;\r\n                subdx2.at(y + gap, x + gap) = 0.f;\r\n                subdy2.at(y + gap, x + gap) = 0.f;\r\n            }\r\n        }\r\n    }\r\n    \r\n    Mat terminals, leftT, rightT, top, bottom;\r\n\r\n    switch (cost_type_)\r\n    {\r\n    case GraphCutSeamFinder::COST_COLOR:\r\n        setGraphWeightsColor(subimg1, subimg2, submask1, submask2, \r\n                             terminals, leftT, rightT, top, bottom);\r\n        break;\r\n    case GraphCutSeamFinder::COST_COLOR_GRAD:\r\n        setGraphWeightsColorGrad(subimg1, subimg2, subdx1, subdx2, subdy1, subdy2, \r\n                                 submask1, submask2, terminals, leftT, rightT, top, bottom);\r\n        break;\r\n    default:\r\n        CV_Error(CV_StsBadArg, \"unsupported pixel similarity measure\");\r\n    }\r\n\r\n    gpu::GpuMat terminals_d(terminals);\r\n    gpu::GpuMat leftT_d(leftT);\r\n    gpu::GpuMat rightT_d(rightT);\r\n    gpu::GpuMat top_d(top);\r\n    gpu::GpuMat bottom_d(bottom);\r\n    gpu::GpuMat labels_d, buf_d;\r\n\r\n    gpu::graphcut(terminals_d, leftT_d, rightT_d, top_d, bottom_d, labels_d, buf_d);\r\n\r\n    Mat_ labels = (Mat)labels_d;\r\n    for (int y = 0; y < roi.height; ++y)\r\n    {\r\n        for (int x = 0; x < roi.width; ++x)\r\n        {\r\n            if (labels(y + gap, x + gap))\r\n            {\r\n                if (mask1.at(roi.y - tl1.y + y, roi.x - tl1.x + x))\r\n                    mask2.at(roi.y - tl2.y + y, roi.x - tl2.x + x) = 0;\r\n            }\r\n            else\r\n            {\r\n                if (mask2.at(roi.y - tl2.y + y, roi.x - tl2.x + x))\r\n                    mask1.at(roi.y - tl1.y + y, roi.x - tl1.x + x) = 0;\r\n            }\r\n        }\r\n    }\r\n}\r\n\r\n\r\nvoid GraphCutSeamFinderGpu::setGraphWeightsColor(const Mat &img1, const Mat &img2, const Mat &mask1, const Mat &mask2, \r\n                                                 Mat &terminals, Mat &leftT, Mat &rightT, Mat &top, Mat &bottom)\r\n{\r\n    const Size img_size = img1.size();\r\n\r\n    terminals.create(img_size, CV_32S);\r\n    leftT.create(Size(img_size.height, img_size.width), CV_32S);\r\n    rightT.create(Size(img_size.height, img_size.width), CV_32S);\r\n    top.create(img_size, CV_32S);\r\n    bottom.create(img_size, CV_32S);\r\n\r\n    Mat_ terminals_(terminals);\r\n    Mat_ leftT_(leftT);\r\n    Mat_ rightT_(rightT);\r\n    Mat_ top_(top);\r\n    Mat_ bottom_(bottom);\r\n\r\n    // Set terminal weights\r\n    for (int y = 0; y < img_size.height; ++y)\r\n    {\r\n        for (int x = 0; x < img_size.width; ++x)\r\n        {\r\n            float source = mask1.at(y, x) ? terminal_cost_ : 0.f;\r\n            float sink = mask2.at(y, x) ? terminal_cost_ : 0.f;\r\n            terminals_(y, x) = saturate_cast((source - sink) * 255.f);\r\n        }\r\n    }\r\n\r\n    // Set regular edge weights\r\n    const float weight_eps = 1.f;\r\n    for (int y = 0; y < img_size.height; ++y)\r\n    {\r\n        for (int x = 0; x < img_size.width; ++x)\r\n        {\r\n            if (x > 0)\r\n            {\r\n                float weight = normL2(img1.at(y, x - 1), img2.at(y, x - 1)) +\r\n                               normL2(img1.at(y, x), img2.at(y, x)) +\r\n                               weight_eps;\r\n                if (!mask1.at(y, x - 1) || !mask1.at(y, x) ||\r\n                    !mask2.at(y, x - 1) || !mask2.at(y, x))\r\n                    weight += bad_region_penalty_;\r\n                leftT_(x, y) = saturate_cast(weight * 255.f);\r\n            }\r\n            else\r\n                leftT_(x, y) = 0;\r\n\r\n            if (x < img_size.width - 1)\r\n            {\r\n                float weight = normL2(img1.at(y, x), img2.at(y, x)) +\r\n                               normL2(img1.at(y, x + 1), img2.at(y, x + 1)) +\r\n                               weight_eps;\r\n                if (!mask1.at(y, x) || !mask1.at(y, x + 1) ||\r\n                    !mask2.at(y, x) || !mask2.at(y, x + 1))\r\n                    weight += bad_region_penalty_;\r\n                rightT_(x, y) = saturate_cast(weight * 255.f);\r\n            }\r\n            else\r\n                rightT_(x, y) = 0;\r\n\r\n            if (y > 0)\r\n            {\r\n                float weight = normL2(img1.at(y - 1, x), img2.at(y - 1, x)) +\r\n                               normL2(img1.at(y, x), img2.at(y, x)) +\r\n                               weight_eps;\r\n                if (!mask1.at(y - 1, x) || !mask1.at(y, x) ||\r\n                    !mask2.at(y - 1, x) || !mask2.at(y, x))\r\n                    weight += bad_region_penalty_;\r\n                top_(y, x) = saturate_cast(weight * 255.f);\r\n            }\r\n            else\r\n                top_(y, x) = 0;\r\n\r\n            if (y < img_size.height - 1)\r\n            {\r\n                float weight = normL2(img1.at(y, x), img2.at(y, x)) +\r\n                               normL2(img1.at(y + 1, x), img2.at(y + 1, x)) +\r\n                               weight_eps;\r\n                if (!mask1.at(y, x) || !mask1.at(y + 1, x) ||\r\n                    !mask2.at(y, x) || !mask2.at(y + 1, x))\r\n                    weight += bad_region_penalty_;\r\n                bottom_(y, x) = saturate_cast(weight * 255.f);\r\n            }\r\n            else\r\n                bottom_(y, x) = 0;\r\n        }\r\n    }\r\n}\r\n\r\n\r\nvoid GraphCutSeamFinderGpu::setGraphWeightsColorGrad(\r\n        const Mat &img1, const Mat &img2, const Mat &dx1, const Mat &dx2,\r\n        const Mat &dy1, const Mat &dy2, const Mat &mask1, const Mat &mask2, \r\n        Mat &terminals, Mat &leftT, Mat &rightT, Mat &top, Mat &bottom)\r\n{\r\n    const Size img_size = img1.size();\r\n\r\n    terminals.create(img_size, CV_32S);\r\n    leftT.create(Size(img_size.height, img_size.width), CV_32S);\r\n    rightT.create(Size(img_size.height, img_size.width), CV_32S);\r\n    top.create(img_size, CV_32S);\r\n    bottom.create(img_size, CV_32S);\r\n\r\n    Mat_ terminals_(terminals);\r\n    Mat_ leftT_(leftT);\r\n    Mat_ rightT_(rightT);\r\n    Mat_ top_(top);\r\n    Mat_ bottom_(bottom);\r\n\r\n    // Set terminal weights\r\n    for (int y = 0; y < img_size.height; ++y)\r\n    {\r\n        for (int x = 0; x < img_size.width; ++x)\r\n        {\r\n            float source = mask1.at(y, x) ? terminal_cost_ : 0.f;\r\n            float sink = mask2.at(y, x) ? terminal_cost_ : 0.f;\r\n            terminals_(y, x) = saturate_cast((source - sink) * 255.f);\r\n        }\r\n    }\r\n\r\n    // Set regular edge weights\r\n    const float weight_eps = 1.f;\r\n    for (int y = 0; y < img_size.height; ++y)\r\n    {\r\n        for (int x = 0; x < img_size.width; ++x)\r\n        {\r\n            if (x > 0)\r\n            {\r\n                float grad = dx1.at(y, x - 1) + dx1.at(y, x) +\r\n                             dx2.at(y, x - 1) + dx2.at(y, x) + weight_eps;\r\n                float weight = (normL2(img1.at(y, x - 1), img2.at(y, x - 1)) +\r\n                                normL2(img1.at(y, x), img2.at(y, x))) / grad +\r\n                               weight_eps;\r\n                if (!mask1.at(y, x - 1) || !mask1.at(y, x) ||\r\n                    !mask2.at(y, x - 1) || !mask2.at(y, x))\r\n                    weight += bad_region_penalty_;\r\n                leftT_(x, y) = saturate_cast(weight * 255.f);\r\n            }\r\n            else\r\n                leftT_(x, y) = 0;\r\n\r\n            if (x < img_size.width - 1)\r\n            {\r\n                float grad = dx1.at(y, x) + dx1.at(y, x + 1) +\r\n                             dx2.at(y, x) + dx2.at(y, x + 1) + weight_eps;\r\n                float weight = (normL2(img1.at(y, x), img2.at(y, x)) +\r\n                                normL2(img1.at(y, x + 1), img2.at(y, x + 1))) / grad +\r\n                               weight_eps;\r\n                if (!mask1.at(y, x) || !mask1.at(y, x + 1) ||\r\n                    !mask2.at(y, x) || !mask2.at(y, x + 1))\r\n                    weight += bad_region_penalty_;\r\n                rightT_(x, y) = saturate_cast(weight * 255.f);\r\n            }\r\n            else\r\n                rightT_(x, y) = 0;\r\n\r\n            if (y > 0)\r\n            {\r\n                float grad = dy1.at(y - 1, x) + dy1.at(y, x) +\r\n                             dy2.at(y - 1, x) + dy2.at(y, x) + weight_eps;\r\n                float weight = (normL2(img1.at(y - 1, x), img2.at(y - 1, x)) +\r\n                                normL2(img1.at(y, x), img2.at(y, x))) / grad +\r\n                               weight_eps;\r\n                if (!mask1.at(y - 1, x) || !mask1.at(y, x) ||\r\n                    !mask2.at(y - 1, x) || !mask2.at(y, x))\r\n                    weight += bad_region_penalty_;\r\n                top_(y, x) = saturate_cast(weight * 255.f);\r\n            }\r\n            else\r\n                top_(y, x) = 0;\r\n\r\n            if (y < img_size.height - 1)\r\n            {\r\n                float grad = dy1.at(y, x) + dy1.at(y + 1, x) +\r\n                             dy2.at(y, x) + dy2.at(y + 1, x) + weight_eps;\r\n                float weight = (normL2(img1.at(y, x), img2.at(y, x)) +\r\n                                normL2(img1.at(y + 1, x), img2.at(y + 1, x))) / grad +\r\n                               weight_eps;\r\n                if (!mask1.at(y, x) || !mask1.at(y + 1, x) ||\r\n                    !mask2.at(y, x) || !mask2.at(y + 1, x))\r\n                    weight += bad_region_penalty_;\r\n                bottom_(y, x) = saturate_cast(weight * 255.f);\r\n            }\r\n            else\r\n                bottom_(y, x) = 0;\r\n        }\r\n    }\r\n}\r\n\r\n#endif\r\n\r\n} // namespace detail\r\n} // namespace cv\r\n\r\n#endif\r\n\r\n"
  },
  {
    "path": "toolkits/computer_vision/stitch_full_main.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n *\n * \\brief This file contains an example of graphlab used for stitching\n * multiple images into a panorama. The code is based on a example\n * stiching application in OpenCV.\n *\n *  \\author Dhruv Batra\n */\n\n/* No adjacency list is externally given here for constructing the graph, \n * rather an adjacency list is created here for a fully connected graph*/\n\n#include \"stitch_main.hpp\"\n\nOptions opts;\n\n/////////////////////////////////////////////////////////////////////////\nint main(int argc, char** argv)\n{\n   \n    int64 app_start_time = getTickCount();\n    ///////////////////////////////////////////////////////\n    // Set up Graphlab\n    global_logger().set_log_level(LOG_INFO);\n    global_logger().set_log_to_console(true);\n\n    ///! Initialize control plain using mpi\n    graphlab::mpi_tools::init(argc, argv);\n    graphlab::distributed_control dc;\n\n    ///////////////////////////////////////////////////////\n    // Set up OpenCV\n    cv::setBreakOnError(true);\n\n    ///////////////////////////////////////////////////////\n    // Graphlab parse input\n    const std::string description = \"Image Stitching\";\n    graphlab::command_line_options clopts(description);\n\n    string img_dir;\n    string graph_path;\n    \n    clopts.attach_option(\"img\", img_dir,\n                         \"The directory containing the images\");\n    clopts.add_positional(\"img\");\n    \n    clopts.add_positional(\"graph\");\n    clopts.attach_option(\"output\", opts.output_dir,\n                         \"The directory in which to save the output\");\n    clopts.attach_option(\"verbose\", opts.verbose,\n                         \"Verbosity of Printing: 0 (default, no printing) or 1 (lots).\");\n    clopts.attach_option(\"work_megapix\", opts.work_megapix,\n                         \"Resolution for image registration step. The default is 0.6 Mpx.\");\n    clopts.attach_option(\"engine\", opts.exec_type,\n                         \"The type of engine to use {async, sync}.\");\n\n    if(!clopts.parse(argc, argv))\n    {\n        graphlab::mpi_tools::finalize();\n        return clopts.is_set(\"help\")? EXIT_SUCCESS : EXIT_FAILURE;\n    }\n   \n    if(img_dir.empty())\n    {\n        logstream(LOG_ERROR) << \"No image directory was provided.\" << std::endl;\n        return EXIT_FAILURE;\n    }\n       \n    if (opts.work_megapix > 10)\n    {\n        logstream(LOG_ERROR) << \"Inappropriate value for work_megapix.\" << std::endl;\n        return EXIT_FAILURE;\n    }\n   \n   \n    // display settings  \n    dc.cout()\n    << \"ncpus:          \" << clopts.get_ncpus() << std::endl\n    << \"engine:         \" << opts.exec_type << std::endl\n    << \"scheduler:      \" << clopts.get_scheduler_type() << std::endl\n    << \"img_dir:        \" << img_dir << std::endl\n    << \"work_megapix:   \" << opts.work_megapix << std::endl\n    << \"verbose:        \" << opts.verbose << std::endl;\n   \n   \n    ///////////////////////////////////////////////////////\n    // Feature Graph\n    graph_type graph_feat(dc, clopts);\n       \n    // load the graph\n    graph_loader(dc, graph_feat, img_dir);\n    graph_feat.finalize();\n   \n    ///////////////////////////////////////////////////////\n    // Graphlab Engine\n    engine_type engine_feat(dc, graph_feat, opts.exec_type, clopts);\n   \n    ///////////////////////////////////////////////////////\n    // Run Aggregator to find size of largest image\n    engine_feat.add_vertex_aggregator(\"find_largest_img\", find_largest_img, set_scales);\n    engine_feat.aggregate_now(\"find_largest_img\");\n\n   \n    ///////////////////////////////////////////////////////\n    // Computer features in parallel on vertices\n    graph_feat.transform_vertices(compute_features);\n\n    ///////////////////////////////////////////////////////\n    // Match features in parallel on edges\n    //graph_feat.transform_edges(match_features);\n    // For now, features matching is done in serial manner.\n\n    //if (dc.procid()==0) {\n    ///////////////////////////////////////////////////////\n    // Compile features\n    typedef vector VecVD;\n    VecVD vdlist = engine_feat.map_reduce_vertices(compile_vertices);\n   \n    vector features(vdlist.size());\n    for (size_t i=0; i!=vdlist.size(); ++i)\n    {\n        features[i] = vdlist[i].features;\n    }\n       \n    int num_images = features.size();\n    ////////////////////////////////////////////////////////////////////////////////////////////////////\n    \n    LOG(\"Pairwise matching\");\n    int64 t1 = getTickCount();\n    vector pairwise_matches;\n    BestOf2NearestMatcher matcher(opts.try_gpu, opts.match_conf);\n    matcher(features, pairwise_matches);\n    matcher.collectGarbage();\n    LOGLN(\"\\nPairwise matching, time: \" << ((getTickCount() - t1) / getTickFrequency()) << \" sec\");\n    LOGLN(\"pairwise_matches.size() = \" << pairwise_matches.size() << \"\\n\");\n    /*for (size_t i=0; i!=pairwise_matches.size(); ++i) \n    {\n        LOGLN(\"src_img_idx = \" << pairwise_matches[i].src_img_idx  << \"\\t\");\n        LOGLN(\"dst_img_idx = \" << pairwise_matches[i].dst_img_idx  << \"\\t\");\n        LOGLN(\"matches.size() = \" << pairwise_matches[i].matches.size()  << \"\\t\");\n        LOGLN(\"num_inliers = \" << pairwise_matches[i].num_inliers  << \"\\t\");\n        LOGLN(\"confidence = \" << pairwise_matches[i].confidence  << \"\\n\");\n    }*/ ////we can uncomment this part to match output with stitching_detailed similar way\n        \n    ///////////////////////////////////////////////////////////////////\n    /* Compile matches\n    typedef vector VecED;\n    VecED edlist = engine_feat.map_reduce_edges(compile_edges);\n   \n    if ((opts.verbose > 0) & (dc.procid()==0))\n        logstream(LOG_EMPH) << \"edlist.size() =  \" << edlist.size()\n        << \"\\n\"; */\n    ///////////////////////////////////////////////////////////////////\n    \n    //vector pairwise_matches(edlist.size()); //not needed for serial implementation\n\n    int r,c; int pair_idx;\n    for (size_t i=0; i!=pairwise_matches.size(); ++i)\n    {\n        IND2SUB_RM(i,r,c,num_images)\n       \n        if (r==c)\n            continue;\n       \n        if (rc) // Swap & invert a few things in the match\n        {\n            if (!pairwise_matches[i].H.empty())\n                pairwise_matches[i].H = pairwise_matches[i].H.inv();\n           \n            for (size_t j = 0; j < pairwise_matches[i].matches.size(); ++j)\n                std::swap(pairwise_matches[i].matches[j].queryIdx,\n                          pairwise_matches[i].matches[j].trainIdx);\n        }\n       \n        if ((opts.verbose > 0) & (dc.procid()==0))\n            logstream(LOG_EMPH) << \"#Matches in Pair \"\n            \"(\" << pairwise_matches[i].src_img_idx\n            << \",\" << pairwise_matches[i].dst_img_idx << \")\"\n            << \": (\" << pairwise_matches[i].matches.size()\n            << \",\" << pairwise_matches[i].num_inliers\n            << \",\" << pairwise_matches[i].confidence << \")\"\n            << \"\\n\";\n\n    }\n    //edlist.clear();*/ not needed for serial implementation\n   \n    ///////////////////////////////////////////////////////\n    // Leave only images we are sure are from the same panorama\n    \n    vector indices = leaveBiggestComponent(features, pairwise_matches, opts.conf_thresh);\n    vector img_path(indices.size());\n    for (size_t i=0; i!=indices.size(); ++i)\n    {\n        img_path[i] = vdlist[indices[i]].img_path;\n    }\n\n      \n\n    ///////////////////////////////////////////////////////\n    // Homography-Based Initialization\n    int64 t;\n    t = getTickCount();\n    HomographyBasedEstimator estimator;\n    vector cameras;\n    estimator(features, pairwise_matches, cameras);\n    logstream(LOG_EMPH) << \"Homography-based init, time: \" << ((getTickCount() - t) / getTickFrequency()) << \" sec\\n\";\n   \n    LOGLN(\"Camera's size: \" << cameras.size() << \"\\n\");    //added for testing\n\n    for (size_t i = 0; i < cameras.size(); ++i)\n    {\n        Mat R;\n\tcameras[i].R.convertTo(R, CV_32F);\n        cameras[i].R = R;\n        if (dc.procid() == 0)\n            logstream(LOG_EMPH) << \"Initial intrinsics #\" << i << \":\\n\" << cameras[i].K() << \"\\n\\n\";\n    }\n\n    LOGLN(\"Homography-Based Initialization ended...\\n\");\n   \n    ///////////////////////////////////////////////////////\n    // Bundle Adjustment\n    t = getTickCount();\n    Ptr adjuster;\n    if (opts.ba_cost_func == \"reproj\") adjuster = new detail::BundleAdjusterReproj();\n    else if (opts.ba_cost_func == \"ray\") adjuster = new detail::BundleAdjusterRay();\n    else\n    {\n        cout << \"Unknown bundle adjustment cost function: '\" << opts.ba_cost_func << \"'.\\n\";\n        return -1;\n    }\n    adjuster->setConfThresh(opts.conf_thresh);\n    Mat_ refine_mask = Mat::zeros(3, 3, CV_8U);\n    if (opts.ba_refine_mask[0] == 'x') refine_mask(0,0) = 1;\n    if (opts.ba_refine_mask[1] == 'x') refine_mask(0,1) = 1;\n    if (opts.ba_refine_mask[2] == 'x') refine_mask(0,2) = 1;\n    if (opts.ba_refine_mask[3] == 'x') refine_mask(1,1) = 1;\n    if (opts.ba_refine_mask[4] == 'x') refine_mask(1,2) = 1;\n    adjuster->setRefinementMask(refine_mask);\n    //LOGLN(\"I will enter adjuster calculation now...\\n\");\n    (*adjuster)(features, pairwise_matches, cameras);\n    if (dc.procid() == 0)\n        logstream(LOG_EMPH) << \"Bundle Adjustment, time: \" << ((getTickCount() - t) / getTickFrequency()) << \" sec\\n\";\n\n    ///////////////////////////////////////////////////////\n    // Find median focal length    \n    vector focals;\n    for (size_t i = 0; i < cameras.size(); ++i)\n    {\n        if (dc.procid() == 0)\n            logstream(LOG_EMPH) << \"Camera #\" << i << \":\\n\" << cameras[i].K() << \"\\n\\n\";\n        focals.push_back(cameras[i].focal);\n    }\n   \n    sort(focals.begin(), focals.end());\n\n    if (opts.verbose > 2)\n    \tLOGLN(\"Focals size: \" << focals.size() << \"\tfocals: \" << focals << \"\\n\");    \n\n    if (focals.size() % 2 == 1)\n        opts.warped_image_scale = static_cast(focals[focals.size() / 2]);\n    else\n        opts.warped_image_scale = static_cast(focals[focals.size() / 2 - 1] + focals[focals.size() / 2]) * 0.5f;\n   \n    ///////////////////////////////////////////////////////\n    // Wave-Correction\n    vector rmats;\n    WaveCorrectKind wave_correct; // in future it needs to be added in clopts to change it in runtime\n    if (opts.wave_correct_type == \"horiz\")\n\twave_correct = detail::WAVE_CORRECT_HORIZ;\n    if (opts.wave_correct_type == \"vert\")\n\twave_correct = detail::WAVE_CORRECT_VERT;\n    for (size_t i = 0; i < cameras.size(); ++i)\n        rmats.push_back(cameras[i].R);\n    waveCorrect(rmats, wave_correct);\n    for (size_t i = 0; i < cameras.size(); ++i)\n        cameras[i].R = rmats[i];\n   \n    //} // End of if procid=0\n\n   \n    ///////////////////////////////////////////////////////\n    // Create a second graph with cameras\n    graph_type graph_cam(dc, clopts);\n\n    // load the graph\n    if (dc.procid()==0)\n\tgraph_loader(graph_cam, img_dir, cameras, img_path, indices, pairwise_matches);\n\n    graph_cam.finalize();\n    vdlist.clear();\n\n    ///////////////////////////////////////////////////////\n    // Warp Images in parallel on vertices\n    graph_cam.transform_vertices(warp_images);\n\n    ///////////////////////////////////////////////////////\n    // Gain Normalize\n\n    ///////////////////////////////////////////////////////\n    // Find seams in parallel on edges\n    graph_cam.transform_edges(find_seams);\n\n    ///////////////////////////////////////////////////////\n    // Composite Images in parallel on vertices\n    graph_cam.transform_vertices(composite_images);\n   \n\n    ///////////////////////////////////////////////////////\n    // Second Graphlab Engine\n    engine_type engine_cam(dc, graph_cam, opts.exec_type, clopts);\n\n    ///////////////////////////////////////////////////////\n    // blend images, gather vertices\n    VecVD veclist = engine_cam.map_reduce_vertices(compile_vertices);\n    vector corner(veclist.size());\n    vector img_warped(veclist.size());\n    vector mask_warped(veclist.size());\n    vector size(veclist.size());\n    Mat img_warped_s;\n\n    for (size_t i=0; i!=veclist.size(); ++i)\n    {\n        corner[i] = veclist[i].corner;\n\n\tif (opts.verbose > 2)\n            LOGLN(\"Blending corners x : \" << veclist[i].corner.x << \"   y : \" << veclist[i].corner.y << \"\\n\");\n\n        img_warped[i] = veclist[i].img_warped;\n        mask_warped[i] = veclist[i].mask_warped;\n        size[i] = veclist[i].warp_size;\n\n        if (opts.verbose > 2)\n            LOGLN(\"Blending sizes height : \" << size[i].height << \"   width : \" << size[i].width << \"\\n\");\n    }\n    veclist.clear();\n   \n    num_images = corner.size();\n    \n    Ptr blender;\n    \n    int blend_type;\n    if (opts.blending_type == \"no\")\n\tblend_type = Blender::NO;\n    if (opts.blending_type == \"feather\")\n\tblend_type = Blender::FEATHER;\n    if (opts.blending_type == \"multiband\")\n        blend_type = Blender::MULTI_BAND;\n\n    bool try_gpu = false;\n            \n    if (blender.empty())\n    {\n        blender = Blender::createDefault(blend_type, try_gpu);\n        Size dst_sz = resultRoi(corner, size).size();\n        float blend_width = sqrt(static_cast(dst_sz.area())) * opts.blend_strength / 100.f;\n\n        if (blend_width < 1.f)\n            blender = Blender::createDefault(Blender::NO, try_gpu);\n        else if (blend_type == Blender::MULTI_BAND)\n        {\n            MultiBandBlender* mb = dynamic_cast(static_cast(blender));\n            mb->setNumBands(static_cast(ceil(log(blend_width)/log(2.)) - 1.));\n            LOGLN(\"Multi-band blender, number of bands: \" << mb->numBands());\n        }\n        else if (blend_type == Blender::FEATHER)\n        {\n            FeatherBlender* fb = dynamic_cast(static_cast(blender));\n            fb->setSharpness(1.f/blend_width);\n            LOGLN(\"Feather blender, sharpness: \" << fb->sharpness());\n        }\n        blender->prepare(corner, size);\n    }\n\n    // Blend all images \n    for (int j=0; j!=num_images; ++j)\n    {\n\timg_warped[j].convertTo(img_warped_s, CV_16S);\n        blender->feed(img_warped_s, mask_warped[j], corner[j]);\n        img_warped_s.release();\n    }\n    \n    Mat result, result_mask, resized_result;\n    blender->blend(result, result_mask);\n\n    resize(result, resized_result, Size(), opts.output_scale, opts.output_scale);\n    imwrite(opts.result_name, resized_result);\n           \n   // imwrite(opts.result_name, result);\n    LOGLN(\"Finished, total time: \" << ((getTickCount() - app_start_time) / getTickFrequency()) << \" sec\");\n\n    ///////////////////////////////////////////////////////\n    // Run everything\n//    engine.signal_all();\n//    graphlab::timer timer;\n//    engine.start();  \n//    const double runtime = timer.current_time();\n//    dc.cout()\n//    << \"----------------------------------------------------------\" << std::endl\n//    << \"Final Runtime (seconds):   \" << runtime\n//    << std::endl\n//    << \"Updates executed: \" << engine.num_updates() << std::endl\n//    << \"Update Rate (updates/second): \"\n//    << engine.num_updates() / runtime << std::endl;\n       \n}\n\n\n"
  },
  {
    "path": "toolkits/computer_vision/stitch_grlab.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n *\n * \\brief This file contains an example of graphlab used for stitching\n * multiple images into a panorama. The code is based on a example\n * stiching application in OpenCV.\n *\n *  \\author Dhruv Batra\n */\n\n\n#ifndef __STITCH_GRLAB_HPP__\n#define __STITCH_GRLAB_HPP__\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n\n\n#include \n\n#include \n#include \n\n\n#include \"eigen_serialization.hpp\"\n#include \"opencv_serialization.hpp\"\n#include \"stitch_opts.hpp\"\n#include \"utils.hpp\"\n//#include \"seam_finders_gr.hpp\"\n#include \"precomp.hpp\"\n\n#include \n\n#include \"opencv2/opencv_modules.hpp\"\n#include \"opencv2/highgui/highgui.hpp\"\n#include \"opencv2/stitching/detail/autocalib.hpp\"\n#include \"opencv2/stitching/detail/blenders.hpp\"\n#include \"opencv2/stitching/detail/camera.hpp\"\n#include \"opencv2/stitching/detail/exposure_compensate.hpp\"\n#include \"opencv2/stitching/detail/matchers.hpp\"\n#include \"opencv2/stitching/detail/motion_estimators.hpp\"\n#include \"opencv2/stitching/detail/seam_finders.hpp\"\n#include \"opencv2/stitching/detail/util.hpp\"\n#include \"opencv2/stitching/detail/warpers.hpp\"\n#include \"opencv2/stitching/warpers.hpp\"\n\nusing namespace std;\nusing namespace cv;\nusing namespace cv::detail;\n\ntypedef Eigen::VectorXd vec;\ntypedef Eigen::MatrixXd mat;\n\n\n/////////////////////////////////////////////////////////////////////////\n// Edge and Vertex data and Graph Type\nstruct vertex_data\n{\n    bool empty; // used to quickly check if this is a dummy vertex.\n   \n    // path to image\n    std::string img_path;\n   \n    cv::Mat full_img;       // Original image\n    cv::Mat img;            // Used for feature computation\n    cv::Mat img_warped;     // Used by gain compensator\n    cv::Mat img_warped_f;   // Used by seam_finder\n   \n    cv::Size full_img_size;\n\n    cv:: Size warp_size;\n        \n    cv::detail::ImageFeatures features;\n   \n    cv::detail::CameraParams camera;\n   \n    cv::Point2f corner;\n    //cv::Mat mask;\n    cv::Mat mask_warped;\n   \n    // constructor\n    vertex_data() : empty(true)\n    { }\n   \n    void save(graphlab::oarchive& arc) const\n    {\n        arc << empty << img_path\n        << full_img << img << img_warped << img_warped_f\n        << full_img_size << warp_size << features << camera\n        << corner\n        << mask_warped;\n    }\n    void load(graphlab::iarchive& arc)\n    {\n        arc >> empty >> img_path\n        >> full_img >> img >> img_warped >> img_warped_f\n        >> full_img_size >> warp_size >> features >> camera\n        >> corner \n        >> mask_warped;\n    }\n   //>> mask\n    vertex_data operator+ (vertex_data& othervertex)\n    {\n        vertex_data sum;\n       \n        if (!empty && !othervertex.empty)\n        {\n            logstream(LOG_ERROR) << \"Don't know about to merge two non-empty vertex-data structures\" << std::endl;\n            //return EXIT_FAILURE;\n        }\n        else if (!empty && othervertex.empty)\n            sum = *this;\n        else if (empty && !othervertex.empty)\n            sum = othervertex;        \n        // Nothing to do if both empty.\n       \n        return sum;\n    }\n\n    vertex_data& operator+= (const vertex_data& othervertex)\n    {\n        if (!empty && !othervertex.empty)\n        {\n            logstream(LOG_ERROR) << \"Don't know about to merge two non-empty vertex-data structures\" << std::endl;\n            //return EXIT_FAILURE;\n        }\n        else if (empty && !othervertex.empty)\n            *this = othervertex;        \n        // Nothing to do if both empty or othervertex empty.\n       \n        return *this;\n    }\n}; // End of vertex data\n\n\n//typedef graphlab::empty edge_data;\nstruct edge_data\n{\n    bool empty; // used to quickly check if this is a dummy edge.\n\n    cv::detail::MatchesInfo matchinfo;\n   \n    // constructor\n    edge_data() : empty(true)\n    { }\n   \n    void save(graphlab::oarchive& arc) const\n    {\n        arc << empty << matchinfo;\n    }\n    void load(graphlab::iarchive& arc)\n    {\n        arc >> empty >> matchinfo;\n    }\n\n    edge_data operator+ (edge_data& otheredge)\n    {\n        edge_data sum;\n       \n        if (!empty && !otheredge.empty)\n        {\n            logstream(LOG_ERROR) << \"Don't know about to merge two non-empty edge-data structures\" << std::endl;\n            //return EXIT_FAILURE;\n        }\n        else if (!empty && otheredge.empty)\n            sum = *this;\n        else if (empty && !otheredge.empty)\n            sum = otheredge;        \n        // Nothing to do if both empty.\n       \n        return sum;\n    }\n\n    edge_data& operator+= (const edge_data& otheredge)\n    {\n        if (!empty && !otheredge.empty)\n        {\n            logstream(LOG_ERROR) << \"Don't know about to merge two non-empty edge-data structures\" << std::endl;\n            //return EXIT_FAILURE;\n        }\n        else if (empty && !otheredge.empty)\n            *this = otheredge;        \n        // Nothing to do if both empty or othervertex empty.\n       \n        return *this;\n    }\n}; // End of edge data\n\n/**\n * The graph type\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n/////////////////////////////////////////////////////////////////////////\n// GraphLab Vertex Program (and Gather Type)\n/**\n * The type passed around during the gather phase\n */\n//struct gather_type\n//{\n//    \n//    gather_type& operator+=(const gather_type& other)\n//    {\n//    } // end of operator +=\n//    void save(graphlab::oarchive& arc) const\n//    {\n//        arc << delf_i << delf_j;\n//    }\n//    void load(graphlab::iarchive& arc)\n//    {\n//        arc >> delf_i >> delf_j;\n//    }\n//}; // end of gather type\ntypedef graphlab::empty gather_type;\n\n/**\n * The core stitching update function.  \n */\nclass stitch_vertex_program :\npublic graphlab::ivertex_program,\npublic graphlab::IS_POD_TYPE\n{\nprivate:\n   \npublic:\n   \n    stitch_vertex_program()  { }\n   \n    edge_dir_type gather_edges(icontext_type& context,\n                               const vertex_type& vertex) const\n    {\n        return graphlab::ALL_EDGES;\n    }; // end of gather_edges\n\n    // Run the gather operation over all in edges\n    gather_type gather(icontext_type& context, const vertex_type& target_vertex,\n                       edge_type& edge) const\n    {\n        return gather_type();\n    } // end of gather\n\n    void apply(icontext_type& context, vertex_type& vertex,\n               const gather_type& sum)\n    {\n       \n        // Get vertex data\n        vertex_data &vdata = vertex.data();\n\n        logstream(LOG_EMPH) << \"Features in image #\" << vertex.id() << \": \" << vdata.features.keypoints.size() << \"\\n\";\n    } // end of apply\n\n    edge_dir_type scatter_edges(icontext_type& context,\n                                const vertex_type& vertex) const\n    {\n        //return graphlab::ALL_EDGES;\n        return graphlab::NO_EDGES;\n    }; // end of gather_edges\n   \n};\n\n\n/**\n * Define the engine type\n */\n//typedef graphlab::synchronous_engine engine_type;\n//typedef graphlab::async_consistent_engine engine_type;\ntypedef graphlab::omni_engine engine_type;\n\n#endif\n\n"
  },
  {
    "path": "toolkits/computer_vision/stitch_main.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n */\n\n\n/**\n *\n * \\brief This file contains an example of graphlab used for stitching\n * multiple images into a panorama. The code is based on a example\n * stiching application in OpenCV.\n *\n *  \\author Dhruv Batra\n */\n\n\n#include \"stitch_main.hpp\"\n\nOptions opts;\n\n/////////////////////////////////////////////////////////////////////////\nint main(int argc, char** argv)\n{\n   \n    int64 app_start_time = getTickCount();\n    ///////////////////////////////////////////////////////\n    // Set up Graphlab\n    global_logger().set_log_level(LOG_INFO);\n    global_logger().set_log_to_console(true);\n\n    ///! Initialize control plain using mpi\n    graphlab::mpi_tools::init(argc, argv);\n    graphlab::distributed_control dc;\n\n    ///////////////////////////////////////////////////////\n    // Set up OpenCV\n    cv::setBreakOnError(true);\n\n    ///////////////////////////////////////////////////////\n    // Graphlab parse input\n    const std::string description = \"Image Stitching\";\n    graphlab::command_line_options clopts(description);\n\n    string img_dir;\n    string graph_path;\n    \n    clopts.attach_option(\"img\", img_dir,\n                         \"The directory containing the images\");\n    clopts.add_positional(\"img\");\n    clopts.attach_option(\"graph\", graph_path,\n                         \"The path to the adjacency list file (could be the prefix in case of multiple files)\");\n    clopts.add_positional(\"graph\");\n    clopts.attach_option(\"output\", opts.output_dir,\n                         \"The directory in which to save the output\");\n    clopts.attach_option(\"verbose\", opts.verbose,\n                         \"Verbosity of Printing: 0 (default, no printing) or 1 (lots).\");\n    clopts.attach_option(\"work_megapix\", opts.work_megapix,\n                         \"Resolution for image registration step. The default is 0.6 Mpx.\");\n    clopts.attach_option(\"engine\", opts.exec_type,\n                         \"The type of engine to use {async, sync}.\");\n\n    if(!clopts.parse(argc, argv))\n    {\n        graphlab::mpi_tools::finalize();\n        return clopts.is_set(\"help\")? EXIT_SUCCESS : EXIT_FAILURE;\n    }\n   \n    if(img_dir.empty())\n    {\n        logstream(LOG_ERROR) << \"No image directory was provided.\" << std::endl;\n        return EXIT_FAILURE;\n    }\n   \n    if(graph_path.empty())\n    {\n        logstream(LOG_ERROR) << \"No adjacency file provided.\" << std::endl;\n        return EXIT_FAILURE;\n    }\n   \n    if (opts.work_megapix > 10)\n    {\n        logstream(LOG_ERROR) << \"Inappropriate value for work_megapix.\" << std::endl;\n        return EXIT_FAILURE;\n    }\n   \n   \n    // display settings  \n    dc.cout()\n    << \"ncpus:          \" << clopts.get_ncpus() << std::endl\n    << \"engine:         \" << opts.exec_type << std::endl\n    << \"scheduler:      \" << clopts.get_scheduler_type() << std::endl\n    << \"img_dir:        \" << img_dir << std::endl\n    << \"graph_path:     \" << graph_path << std::endl\n    << \"work_megapix:   \" << opts.work_megapix << std::endl\n    << \"verbose:        \" << opts.verbose << std::endl;\n   \n   \n    ///////////////////////////////////////////////////////\n    // Feature Graph\n    graph_type graph_feat(dc, clopts);\n       \n    // load the graph\n    //graph.load(img_dir, vertex_loader);\n    vertex_loader(dc, graph_feat, img_dir);\n    graph_feat.load(graph_path, edge_loader);\n    graph_feat.finalize();\n   \n    ///////////////////////////////////////////////////////\n    // Graphlab Engine\n    engine_type engine_feat(dc, graph_feat, opts.exec_type, clopts);\n   \n    ///////////////////////////////////////////////////////\n    // Run Aggregator to find size of largest image\n    engine_feat.add_vertex_aggregator(\"find_largest_img\", find_largest_img, set_scales);\n    engine_feat.aggregate_now(\"find_largest_img\");\n\n   \n    ///////////////////////////////////////////////////////\n    // Computer features in parallel on vertices\n    graph_feat.transform_vertices(compute_features);\n\n    ///////////////////////////////////////////////////////\n    // Match features in parallel on edges\n    graph_feat.transform_edges(match_features);\n\n   \n    //if (dc.procid()==0) {\n    ///////////////////////////////////////////////////////\n    // Compile features\n    typedef vector VecVD;\n    VecVD vdlist = engine_feat.map_reduce_vertices(compile_vertices);\n   \n    vector features(vdlist.size());\n    for (size_t i=0; i!=vdlist.size(); ++i)\n    {\n        features[i] = vdlist[i].features;\n    }\n    vdlist.clear();\n   \n    int num_images = features.size();\n   \n    ///////////////////////////////////////////////////////\n    // Compile matches\n    typedef vector VecED;\n    VecED edlist = engine_feat.map_reduce_edges(compile_edges);\n   \n    if ((opts.verbose > 0) & (dc.procid()==0))\n        logstream(LOG_EMPH) << \"edlist.size() =  \" << edlist.size()\n        << \"\\n\";\n\n\n    vector pairwise_matches(edlist.size());\n    int r,c; int pair_idx;\n    for (size_t i=0; i!=edlist.size(); ++i)\n    {\n        IND2SUB_RM(i,r,c,num_images)\n       \n        if (r==c)\n            continue;\n       \n        if (rc) // Swap & invert a few things in the match\n        {\n            if (!pairwise_matches[i].H.empty())\n                pairwise_matches[i].H = pairwise_matches[i].H.inv();\n           \n            for (size_t j = 0; j < pairwise_matches[i].matches.size(); ++j)\n                std::swap(pairwise_matches[i].matches[j].queryIdx,\n                          pairwise_matches[i].matches[j].trainIdx);\n        }\n       \n        if ((opts.verbose > 0) & (dc.procid()==0))\n            logstream(LOG_EMPH) << \"#Matches in Pair \"\n            \"(\" << pairwise_matches[i].src_img_idx\n            << \",\" << pairwise_matches[i].dst_img_idx << \")\"\n            << \": (\" << pairwise_matches[i].matches.size()\n            << \",\" << pairwise_matches[i].num_inliers\n            << \",\" << pairwise_matches[i].confidence << \")\"\n            << \"\\n\";\n\n    }\n    edlist.clear();\n\n    ///////////////////////////////////////////////////////\n    // Homography-Based Initialization\n    int64 t;\n    t = getTickCount();\n    HomographyBasedEstimator estimator;\n    vector cameras;\n    estimator(features, pairwise_matches, cameras);\n    logstream(LOG_EMPH) << \"Homography-based init, time: \" << ((getTickCount() - t) / getTickFrequency()) << \" sec\\n\";\n   \n    for (size_t i = 0; i < cameras.size(); ++i)\n    {\n        Mat R;\n        cameras[i].R.convertTo(R, CV_32F);\n        cameras[i].R = R;\n        if (dc.procid() == 0)\n            logstream(LOG_EMPH) << \"Initial intrinsics #\" << i << \":\\n\" << cameras[i].K() << \"\\n\\n\";\n    }\n\n   \n    ///////////////////////////////////////////////////////\n    // Bundle Adjustment\n    t = getTickCount();\n    Ptr adjuster;\n    //adjuster = new detail::BundleAdjusterRay();\n    if (opts.ba_cost_func == \"reproj\") adjuster = new detail::BundleAdjusterReproj();\n    else if (opts.ba_cost_func == \"ray\") adjuster = new detail::BundleAdjusterRay();\n    else\n    {\n        cout << \"Unknown bundle adjustment cost function: '\" << opts.ba_cost_func << \"'.\\n\";\n        return -1;\n    }\n    adjuster->setConfThresh(opts.conf_thresh);\n    Mat_ refine_mask = Mat::zeros(3, 3, CV_8U);\n    if (opts.ba_refine_mask[0] == 'x') refine_mask(0,0) = 1;\n    if (opts.ba_refine_mask[1] == 'x') refine_mask(0,1) = 1;\n    if (opts.ba_refine_mask[2] == 'x') refine_mask(0,2) = 1;\n    if (opts.ba_refine_mask[3] == 'x') refine_mask(1,1) = 1;\n    if (opts.ba_refine_mask[4] == 'x') refine_mask(1,2) = 1;\n    adjuster->setRefinementMask(refine_mask);\n    (*adjuster)(features, pairwise_matches, cameras);\n    if (dc.procid() == 0)\n        logstream(LOG_EMPH) << \"Bundle Adjustment, time: \" << ((getTickCount() - t) / getTickFrequency()) << \" sec\\n\";\n\n    ///////////////////////////////////////////////////////\n    // Find median focal length    \n    vector focals;\n    for (size_t i = 0; i < cameras.size(); ++i)\n    {\n        if (dc.procid() == 0)\n            logstream(LOG_EMPH) << \"Camera #\" << i << \":\\n\" << cameras[i].K() << \"\\n\\n\";\n        focals.push_back(cameras[i].focal);\n    }\n   \n    sort(focals.begin(), focals.end());\n\n    if (opts.verbose > 2)\n    \tLOGLN(\"Focals size: \" << focals.size() << \"\tfocals: \" << focals << \"\\n\");    \n\n    if (focals.size() % 2 == 1)\n        opts.warped_image_scale = static_cast(focals[focals.size() / 2]);\n    else\n        opts.warped_image_scale = static_cast(focals[focals.size() / 2 - 1] + focals[focals.size() / 2]) * 0.5f;\n   \n    ///////////////////////////////////////////////////////\n    // Wave-Correction\n    vector rmats;\n    WaveCorrectKind wave_correct; // in future it needs to be added in clopts to change it in runtime\n    if (opts.wave_correct_type == \"horiz\")\n\twave_correct = detail::WAVE_CORRECT_HORIZ;\n    if (opts.wave_correct_type == \"vert\")\n\twave_correct = detail::WAVE_CORRECT_VERT;\n    for (size_t i = 0; i < cameras.size(); ++i)\n        rmats.push_back(cameras[i].R);\n    waveCorrect(rmats, wave_correct);\n    for (size_t i = 0; i < cameras.size(); ++i)\n        cameras[i].R = rmats[i];\n   \n    //} // End of if procid=0\n\n   \n    ///////////////////////////////////////////////////////\n    // Create a second graph with cameras\n    graph_type graph_cam(dc, clopts);\n\n    // load the graph\n    if (dc.procid()==0) {\n    vertex_loader(graph_cam, img_dir, cameras);\n    graph_cam.load(graph_path, edge_loader);\n    }\n    graph_cam.finalize();\n\n    ///////////////////////////////////////////////////////\n    // Warp Images in parallel on vertices\n    graph_cam.transform_vertices(warp_images);\n\n    ///////////////////////////////////////////////////////\n    // Gain Normalize\n\n    ///////////////////////////////////////////////////////\n    // Find seams in parallel on edges\n    graph_cam.transform_edges(find_seams);\n\n    ///////////////////////////////////////////////////////\n    // Composite Images in parallel on vertices\n    graph_cam.transform_vertices(composite_images);\n   \n\n    ///////////////////////////////////////////////////////\n    // Second Graphlab Engine\n    engine_type engine_cam(dc, graph_cam, opts.exec_type, clopts);\n\n    ///////////////////////////////////////////////////////\n    // blend images, gather vertices\n    VecVD veclist = engine_cam.map_reduce_vertices(compile_vertices);\n    vector corner(veclist.size());\n    vector img_warped(veclist.size());\n    vector mask_warped(veclist.size());\n    vector size(veclist.size());\n    Mat img_warped_s;\n\n    for (size_t i=0; i!=veclist.size(); ++i)\n    {\n        corner[i] = veclist[i].corner;\n\n\tif (opts.verbose > 2)\n            LOGLN(\"Blending corners x : \" << veclist[i].corner.x << \"   y : \" << veclist[i].corner.y << \"\\n\");\n\n        img_warped[i] = veclist[i].img_warped;\n        mask_warped[i] = veclist[i].mask_warped;\n        size[i] = veclist[i].warp_size;\n\n        if (opts.verbose > 2)\n            LOGLN(\"Blending sizes height : \" << size[i].height << \"   width : \" << size[i].width << \"\\n\");\n    }\n    veclist.clear();\n   \n    num_images = corner.size();\n    \n    Ptr blender;\n    \n    int blend_type;\n    if (opts.blending_type == \"no\")\n\tblend_type = Blender::NO;\n    if (opts.blending_type == \"feather\")\n\tblend_type = Blender::FEATHER;\n    if (opts.blending_type == \"multiband\")\n        blend_type = Blender::MULTI_BAND;\n\n    bool try_gpu = false;\n            \n    if (blender.empty())\n    {\n        blender = Blender::createDefault(blend_type, try_gpu);\n        Size dst_sz = resultRoi(corner, size).size();\n        float blend_width = sqrt(static_cast(dst_sz.area())) * opts.blend_strength / 100.f;\n        if (blend_width < 1.f)\n            blender = Blender::createDefault(Blender::NO, try_gpu);\n        else if (blend_type == Blender::MULTI_BAND)\n        {\n            MultiBandBlender* mb = dynamic_cast(static_cast(blender));\n            mb->setNumBands(static_cast(ceil(log(blend_width)/log(2.)) - 1.));\n            LOGLN(\"Multi-band blender, number of bands: \" << mb->numBands());\n        }\n        else if (blend_type == Blender::FEATHER)\n        {\n            FeatherBlender* fb = dynamic_cast(static_cast(blender));\n            fb->setSharpness(1.f/blend_width);\n            LOGLN(\"Feather blender, sharpness: \" << fb->sharpness());\n        }\n        blender->prepare(corner, size);\n    }\n\n    // Blend all images \n    for (int j=0; j!=num_images; ++j)\n    {\n\timg_warped[j].convertTo(img_warped_s, CV_16S);\n        blender->feed(img_warped_s, mask_warped[j], corner[j]);\n        img_warped_s.release();\n    }\n    \n    Mat result, result_mask;\n    blender->blend(result, result_mask);\n\n    imwrite(opts.result_name, result);\n    \n    LOGLN(\"Finished, total time: \" << ((getTickCount() - app_start_time) / getTickFrequency()) << \" sec\");\n\n    ///////////////////////////////////////////////////////\n    // Run everything\n//    engine.signal_all();\n//    graphlab::timer timer;\n//    engine.start();  \n//    const double runtime = timer.current_time();\n//    dc.cout()\n//    << \"----------------------------------------------------------\" << std::endl\n//    << \"Final Runtime (seconds):   \" << runtime\n//    << std::endl\n//    << \"Updates executed: \" << engine.num_updates() << std::endl\n//    << \"Update Rate (updates/second): \"\n//    << engine.num_updates() / runtime << std::endl;\n       \n}\n\n\n"
  },
  {
    "path": "toolkits/computer_vision/stitch_main.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n *\n * \\brief This file contains an example of graphlab used for stitching\n * multiple images into a panorama. The code is based on a example\n * stiching application in OpenCV.\n *\n *  \\author Dhruv Batra\n */\n\n\n#ifndef __STITCH_MAIN_HPP__\n#define __STITCH_MAIN_HPP__\n\n#include \"utils.hpp\"\n#include \"stitch_grlab.hpp\"\n#include \"iostream\"\n\n/////////////////////////////////////////////////////////////////////////\n// Graph Loader (used to read images and load the vertex and edge data of\n// the graph. There is an edge between every pair of vertices in the graph,  \n// i.e. the graph is fully connected. (Used in stitch_full_main)\n\nbool graph_loader(graphlab::distributed_control& dc, graph_type& graph, string img_dir)\n{\n    // force a \"/\" at the end of the path\n    // make sure to check that the path is non-empty. (you do not\n    // want to make the empty path \"\" the root path \"/\" )\n    string path = img_dir;\n    if (path.length() > 0 && path[path.length() - 1] != '/') path = path + \"/\";\n    \n    vector graph_files;\n    string search_prefix;\n    graphlab::fs_util::list_files_with_prefix(path, search_prefix, graph_files);\n    \n    if (graph_files.size() == 0)\n        logstream(LOG_WARNING) << \"No files found in \" << path << std::endl;\n    \n    \n    if (opts.verbose > 2)\n        logstream(LOG_EMPH)\n        << \"Total number of images: \" << graph_files.size() << \"\\n\";\n    \n    // vertex data & id\n    graphlab::vertex_id_type vid(-1);\n    graphlab::vertex_id_type other_vid;\n    \n    ///////////////////////////////////////////////////////\n    // Loop over files\n    for(size_t i = 0; i < graph_files.size(); ++i)\n    {\n        // Each machine loads corresponding file\n        if (i % dc.numprocs() == dc.procid())\n        {\n            if (opts.verbose > 0)\n                logstream(LOG_EMPH)\n                << \"Process: \" << dc.procid() << \"/\" << dc.numprocs() << \" \"\n                << \"picked image: \" << graph_files[i] << \"\\n\";\n            \n            \n            vid = i;\n            vertex_data vdata;\n            vdata.empty = false;\n            vdata.img_path = graph_files[i];\n            vdata.features.img_idx = i;\n            \n            graph.add_vertex(vid, vdata);\n            \n            if (opts.verbose > 2)\n                logstream(LOG_EMPH)\n                << \"Vertex \" << i << \" Image: \" << vdata.img_path << \"\\n\"; \n            \n        }\n    }\n    \n    // Adding edges between every pair of vertices to create a fully connected graph\n    // no duplicate edges are added\n    for(size_t i = 0; i < graph_files.size()-1; ++i)\n    {\n        vid = i;\n        for(size_t j = i+1; j < graph_files.size(); ++j)\n        {\n            other_vid = j;\n            if (opts.verbose > 0)\n                logstream(LOG_EMPH) << \"Adding edge: (\" << vid << \",\" << other_vid << \")\\n\";\n            \n            edge_data edata; edata.empty = false;\n            graph.add_edge(vid,other_vid,edata);\n        }\n    }\n    return true;\n}\n\n// Second Graph loader that only a single machine calls and pre-loads cameras.\n// It adds only the selected edges to the subgraph of the fully connected graph.\n// (Used in stitch_full_main)\nbool graph_loader(graph_type& graph, string img_dir, vector& cameras, \n                  vector& img_path, vector indices, vector& pairwise_matches)\n{\n    // force a \"/\" at the end of the path\n    // make sure to check that the path is non-empty. (you do not\n    // want to make the empty path \"\" the root path \"/\" )\n    string path = img_dir;\n    if (path.length() > 0 && path[path.length() - 1] != '/') path = path + \"/\";\n    \n    // vertex data & id   \n    graphlab::vertex_id_type vid(-1);\n    graphlab::vertex_id_type other_vid;\n    \n    if (opts.verbose > 2)\n        logstream(LOG_EMPH)\n        << \"Total number of vertices in the second graph: \" << indices.size() << \"\\n\";\n    \n    ///////////////////////////////////////////////////////\n    // Loop over files\n    for(size_t i = 0; i < indices.size(); ++i)\n    {\n        vid = i;\n        vertex_data vdata;\n        vdata.empty = false;\n        vdata.img_path = img_path[i];\n        vdata.features.img_idx = i;\n        \n        vdata.camera = cameras[i]; // addition to above function.\n        \n        graph.add_vertex(vid, vdata);\n    }\n    \n    // Adding edges between selected pair of vertices to create a subgraph of the fully connected graph\n    // no duplicate edges are added\n    \n    if (opts.verbose > 2)\n        logstream(LOG_EMPH)\n        << \"Pairwise_matches size : \" << pairwise_matches.size() << \"\\n\";\n    \n    for(size_t i = 0; i < pairwise_matches.size(); ++i)\n    {\n        if (opts.verbose > 2)\n            logstream(LOG_EMPH)\n            << \"Pairwise_matches : (\" << pairwise_matches[i].src_img_idx << \",\" << pairwise_matches[i].dst_img_idx << \")\\n\";\n        \n        if (pairwise_matches[i].src_img_idx >= 0 && pairwise_matches[i].dst_img_idx >= 0)\n        {\n            if (pairwise_matches[i].src_img_idx < pairwise_matches[i].dst_img_idx)\t// no duplicate edges are allowed\n            {\n                vid = pairwise_matches[i].src_img_idx;\n                other_vid = pairwise_matches[i].dst_img_idx;\n                if (opts.verbose > 0)\n            \t    logstream(LOG_EMPH) << \"Adding edge: (\" << vid << \",\" << other_vid << \")\\n\";\n                \n                edge_data edata; edata.empty = false;\n                graph.add_edge(vid,other_vid,edata);\n            }\n        }\n    }\n    \n    return true;\n}\n\n/////////////////////////////////////////////////////////////////////////\n// Vertex Loader (used to read images and load the vertex data of the graph)\n//bool vertex_loader(graph_type& graph, const std::string& fname,\n//                   const std::string& line)\nbool vertex_loader(graphlab::distributed_control& dc, graph_type& graph, string img_path)\n{\n    // force a \"/\" at the end of the path\n    // make sure to check that the path is non-empty. (you do not\n    // want to make the empty path \"\" the root path \"/\" )\n    string path = img_path;\n    if (path.length() > 0 && path[path.length() - 1] != '/') path = path + \"/\";\n    \n    vector graph_files;\n    string search_prefix;\n    graphlab::fs_util::list_files_with_prefix(path, search_prefix, graph_files);\n    \n    if (graph_files.size() == 0)\n        logstream(LOG_WARNING) << \"No files found in \" << path << std::endl;\n    \n    // vertex data & id\n    graphlab::vertex_id_type vid(-1);\n    \n    ///////////////////////////////////////////////////////\n    // Loop over files\n    for(size_t i = 0; i < graph_files.size(); ++i)\n    {\n        // Each machine loads corresponding file\n        if (i % dc.numprocs() == dc.procid())\n        {\n            if (opts.verbose > 0)\n                logstream(LOG_EMPH)\n                << \"Process: \" << dc.procid() << \"/\" << dc.numprocs() << \" \"\n                << \"picked image: \" << graph_files[i] << \"\\n\";\n            \n            \n            vid = i;\n            vertex_data vdata;\n            vdata.empty = false;\n            vdata.img_path = graph_files[i];\n            vdata.features.img_idx = i;\n            \n            graph.add_vertex(vid, vdata);\n            \n        }\n    }\n    \n    return true;\n}\n\n// Second loader that only a single machine calls and pre-loads cameras.\nbool vertex_loader(graph_type& graph, string img_path, vector& cameras)\n{\n    // force a \"/\" at the end of the path\n    // make sure to check that the path is non-empty. (you do not\n    // want to make the empty path \"\" the root path \"/\" )\n    string path = img_path;\n    if (path.length() > 0 && path[path.length() - 1] != '/') path = path + \"/\";\n    \n    vector graph_files;\n    string search_prefix;\n    graphlab::fs_util::list_files_with_prefix(path, search_prefix, graph_files);\n    \n    if (graph_files.size() == 0)\n        logstream(LOG_WARNING) << \"No files found in \" << path << std::endl;\n    \n    // vertex data & id\n    graphlab::vertex_id_type vid(-1);\n    \n    ///////////////////////////////////////////////////////\n    // Loop over files\n    for(size_t i = 0; i < graph_files.size(); ++i)\n    {\n        vid = i;\n        vertex_data vdata;\n        vdata.empty = false;\n        vdata.img_path = graph_files[i];\n        vdata.features.img_idx = i;\n        \n        vdata.camera = cameras[i]; // addition to above function.\n        \n        graph.add_vertex(vid, vdata);\n    }\n    \n    return true;\n}\n\n/////////////////////////////////////////////////////////////////////////\n// Edge Loader (used to read the adjacency list and add edges to the graph)\nbool edge_loader(graph_type& graph, const std::string& fname,\n                 const std::string& textline)\n{\n    if ( textline.length() == 0 || textline[0] == '#' )\n        return true; // empty or comment line, return\n    \n    std::stringstream strm(textline);\n    graphlab::vertex_id_type vid;\n    \n    // first entry in the line is a vertex ID\n    strm >> vid;\n    \n    if (opts.verbose > 0)\n        logstream(LOG_EMPH) << \"Here's the input: \"\n        << textline << \"\\n\"\n        << vid << \"\\n\";\n    \n    // Line should contain at least 1 more number (degree of node)\n    if (!strm.good())\n    {\n        logstream(LOG_ERROR) << \"The following ajacency list line is incomplete(check adj_list standard):\\n\"\n        << textline << std::endl;\n        return EXIT_FAILURE;\n    }\n    \n    // second entry is the out-degree\n    int outdeg;\n    strm >> outdeg;\n    \n    graphlab::vertex_id_type other_vid;\n    for (int i=0; i!=outdeg; ++i)\n    {\n        // Line should contain more numbers (id of neighbours)\n        if (!strm.good())\n        {\n            logstream(LOG_ERROR) << \"The following ajacency list line is incomplete(check adj_list standard):\\n\"\n            << textline << std::endl;\n            return EXIT_FAILURE;\n        }\n        \n        strm >> other_vid;\n        \n        // only add edges in one direction\n        if (other_vid < vid)\n            continue;\n        \n        if (opts.verbose > 0)\n            logstream(LOG_EMPH) << \"Adding edge: (\" << vid << \",\" << other_vid << \")\\n\";\n        \n        edge_data edata; edata.empty = false;\n        graph.add_edge(vid,other_vid,edata);\n    }\n    \n    return true;\n}\n/////////////////////////////////////////////////////////////////////////\n// Map-Aggregator function to find the largest image size\nstruct ImgArea\n{\n    double full_img_area;\n    \n    void save(graphlab::oarchive& arc) const\n    {\n        arc << full_img_area;\n    }\n    void load(graphlab::iarchive& arc)\n    {\n        arc >> full_img_area;\n    }\n    \n    ImgArea operator+= (ImgArea other) // computes max\n    {\n        ImgArea max;\n        max.full_img_area =\n        (full_img_area > other.full_img_area) ? full_img_area : other.full_img_area;\n        return max;\n    }\n};\n\nImgArea find_largest_img(engine_type::icontext_type& context,\n                          graph_type::vertex_type& vertex)\n{\n    // Get vertex data\n    vertex_data &vdata = vertex.data();\n    \n    // Ignore hdfs-setup for now. Just read from file directly.\n    Mat full_img = imread(vdata.img_path);\n    \n    ImgArea imgarea;\n    imgarea.full_img_area = full_img.size().area();\n    \n    if (opts.verbose > 2)\n        LOGLN(\"largest image area: \" << imgarea.full_img_area << \"\\n\");\n    \n    return imgarea;\n}\n\nvoid set_scales(engine_type::icontext_type& context, ImgArea& largestimg)\n{\n    \n    if (opts.work_megapix > 0)\n        opts.work_scale = min(1.0, sqrt(opts.work_megapix * 1e6 / largestimg.full_img_area));\n    \n    if (opts.seam_megapix > 0)\n\topts.seam_scale = min(1.0, sqrt(opts.seam_megapix * 1e6 / largestimg.full_img_area));\n    \n    if (opts.compose_megapix > 0)\n        opts.compose_scale = min(1.0, sqrt(opts.compose_megapix * 1e6 / largestimg.full_img_area));\n\n    if (opts.output_megapix > 0)\n        opts.output_scale = min(1.0, sqrt(opts.output_megapix * 1e6 / largestimg.full_img_area));\n    \n    opts.seam_work_aspect = opts.seam_scale / opts.work_scale;\n    opts.compose_seam_aspect = opts.compose_scale / opts.seam_scale;\n    opts.compose_work_aspect = opts.compose_scale / opts.work_scale;\n  //  opts.output_work_aspect = opts.output_scale / opts.work_scale;\n}\n\n\n/////////////////////////////////////////////////////////////////////////\n// Function to extract features in vertex parallel\nvoid compute_features(graph_type::vertex_type& vertex)\n{\n    // Get vertex data\n    vertex_data &vdata = vertex.data();\n    \n    // Load image\n    //            // open the stream\n    //            std::ifstream in_file(graph_files[i].c_str(),\n    //                                  std::ios_base::in | std::ios_base::binary);\n    //\n    //            boost::iostreams::filtering_stream fin;  \n    //            fin.push(in_file);\n    //            \n    //            // Get data from stream into a buffer\n    //            fin.pop();\n    \n    // Ignore the above hdfs-setup for now. Just read from file directly.\n    Mat &full_img = vdata.full_img;\n    Mat &img = vdata.img;\n    full_img = imread(vdata.img_path);\n    \n    if ( abs(opts.work_scale-1) > 1e-3 )\n        resize(full_img, img, Size(), opts.work_scale, opts.work_scale);\n    else\n        img = full_img;\n    \n    if (img.empty())\n        logstream(LOG_ERROR) << \"Could not imread image: \" << vdata.img_path << \"\\n\";\n    \n    // compute features\n    SurfFeaturesFinder finder;\n    finder(img, vdata.features);\n    finder.collectGarbage();\n    \n    if (opts.verbose > 0)\n    {\n        logstream(LOG_EMPH) << \"Features in image #\" << vertex.id() << \": \" << vdata.features.keypoints.size() << \"\\n\";\n        LOGLN(\"Size of feature image #\" << img.cols << \"  \" << img.rows); //\n    }\n    \n}\n\n/////////////////////////////////////////////////////////////////////////\n// Function to compute feature-matches in parallel on edges\nvoid match_features(graph_type::edge_type& edge)\n{\n    // Get edge data\n    edge_data &edata = edge.data();\n    \n    // Get vertex ids of two vertices involved\n    vertex_data &vdata1 = edge.source().data();\n    vertex_data &vdata2 = edge.target().data();\n    \n    // Match features\n    BestOf2NearestMatcher matcher;\n    matcher(vdata1.features, vdata2.features, edata.matchinfo);\n    matcher.collectGarbage();\n    \n    if (opts.verbose > 0)\n        logstream(LOG_EMPH) << \"#Matches in Image Pair \"\n        \"(\" << edge.source().id() << \",\" << edge.target().id() << \")\"\n        << \": (\" << edata.matchinfo.matches.size()\n        << \",\" << edata.matchinfo.num_inliers << \")\"\n        << \"\\n\";\n}\n\n/////////////////////////////////////////////////////////////////////////\n// Function to warp images in parallel\nvoid warp_images(graph_type::vertex_type& vertex)\n{\n    // Get vertex data\n    vertex_data &vdata = vertex.data();\n    \n    Mat full_img = imread(vdata.img_path);\n    \n    Mat &img = vdata.img;\n    Mat &img_warped = vdata.img_warped;\n    Mat &img_warped_f = vdata.img_warped_f;\n    //Mat &mask = vdata.mask;\n    Mat mask;\n    Mat &mask_warped = vdata.mask_warped;\n    CameraParams &camera = vdata.camera;\n    Point2f &corner = vdata.corner;\n    Size &size = vdata.warp_size;\n    \n    if (full_img.empty())\n        logstream(LOG_ERROR) << \"Could not imread image: \" << vdata.img_path << \"\\n\";\n    \n    vdata.full_img_size = full_img.size();\n    \n    // Scale image if necessary\n    double seam_scale = min(1.0, sqrt(opts.seam_megapix * 1e6 / full_img.size().area()));\n    \n    if ( abs(seam_scale-1) > 1e-3 )\n        resize(full_img, img, Size(), seam_scale, seam_scale);\n    else\n        img = full_img.clone();\n    \n    // Prepare images mask\n    mask.create(img.size(), CV_8U);\n    mask.setTo(Scalar::all(255));\n    \n    if (opts.verbose > 2)\n        LOGLN(\"Size of mask image #\" << img.cols << \"  \" << img.rows); //\n    \n    // Warp images and their masks\n    Ptr warper_creator;\n    \n    if (opts.warp_type == \"plane\") warper_creator = new cv::PlaneWarper();\n    else if (opts.warp_type == \"cylindrical\") warper_creator = new cv::CylindricalWarper();\n    else if (opts.warp_type == \"spherical\") warper_creator = new cv::SphericalWarper();\n    else if (opts.warp_type == \"fisheye\") warper_creator = new cv::FisheyeWarper();\n    else if (opts.warp_type == \"stereographic\") warper_creator = new cv::StereographicWarper();\n    else if (opts.warp_type == \"compressedPlaneA2B1\") warper_creator = new cv::CompressedRectilinearWarper(2, 1);\n    else if (opts.warp_type == \"compressedPlaneA1.5B1\") warper_creator = new cv::CompressedRectilinearWarper(1.5, 1);\n    else if (opts.warp_type == \"compressedPlanePortraitA2B1\") warper_creator = new cv::CompressedRectilinearPortraitWarper(2, 1);\n    else if (opts.warp_type == \"compressedPlanePortraitA1.5B1\") warper_creator = new cv::CompressedRectilinearPortraitWarper(1.5, 1);\n    else if (opts.warp_type == \"paniniA2B1\") warper_creator = new cv::PaniniWarper(2, 1);\n    else if (opts.warp_type == \"paniniA1.5B1\") warper_creator = new cv::PaniniWarper(1.5, 1);\n    else if (opts.warp_type == \"paniniPortraitA2B1\") warper_creator = new cv::PaniniPortraitWarper(2, 1);\n    else if (opts.warp_type == \"paniniPortraitA1.5B1\") warper_creator = new cv::PaniniPortraitWarper(1.5, 1);\n    else if (opts.warp_type == \"mercator\") warper_creator = new cv::MercatorWarper();\n    else if (opts.warp_type == \"transverseMercator\") warper_creator = new cv::TransverseMercatorWarper();\n    \n    \n    if (warper_creator.empty())\n        logstream(LOG_ERROR) << \"Can't create the following warper '\" << opts.warp_type << \"'\\n\";\n    \n    Ptr warper = warper_creator->create(static_cast(opts.warped_image_scale * opts.seam_work_aspect));\n    Mat_ K;\n    camera.K().convertTo(K, CV_32F);\n    float swa = (float)opts.seam_work_aspect;\n    K(0,0) *= swa; K(0,2) *= swa;\n    K(1,1) *= swa; K(1,2) *= swa;\n    \n    corner = warper->warp(img, K, camera.R, INTER_LINEAR, BORDER_REFLECT, img_warped);\n    if (opts.verbose > 2)\n        LOGLN(\"Warp corners x : \" << corner.x << \"   y : \" << corner.y << \"\\n\");\n    \n    size = img_warped.size();\n    if (opts.verbose > 2)\n        LOGLN(\"Warp sizes height : \" << size.height << \"   width : \" << size.width << \"\\n\");\n    \n    warper->warp(mask, K, camera.R, INTER_NEAREST, BORDER_CONSTANT, mask_warped);\n    \n    img_warped.convertTo(img_warped_f, CV_32F);\n    \n    // If no gain compensator, then clear.\n    img_warped.release();\n    \n}\n\n/////////////////////////////////////////////////////////////////////////\n// Function to composite images in parallel\nvoid composite_images(graph_type::vertex_type& vertex)\n{\n    // Get vertex data\n    vertex_data &vdata = vertex.data();\n    CameraParams &camera = vdata.camera;\n    Point2f &corner = vdata.corner;\n    Mat full_img = imread(vdata.img_path);\t//we have to check it later for speed\n    \n    Mat &img_warped = vdata.img_warped;\t\t//added by me\n    Mat &mask_warped = vdata.mask_warped;\t//added by me\n    Size &size = vdata.warp_size;\t\t//added by me\n    Mat mask, dilated_mask, seam_mask, masks_warped;\t\t//added by me\n    \n    \n    if (full_img.empty())\n        logstream(LOG_ERROR) << \"Could not imread image: \" << vdata.img_path << \"\\n\";\n    \n    Mat &img = vdata.img;\n    \n    // Update warped image scale\n    Ptr warper_creator;\n    \n    if (opts.warp_type == \"plane\") warper_creator = new cv::PlaneWarper();\n    else if (opts.warp_type == \"cylindrical\") warper_creator = new cv::CylindricalWarper();\n    else if (opts.warp_type == \"spherical\") warper_creator = new cv::SphericalWarper();\n    else if (opts.warp_type == \"fisheye\") warper_creator = new cv::FisheyeWarper();\n    else if (opts.warp_type == \"stereographic\") warper_creator = new cv::StereographicWarper();\n    else if (opts.warp_type == \"compressedPlaneA2B1\") warper_creator = new cv::CompressedRectilinearWarper(2, 1);\n    else if (opts.warp_type == \"compressedPlaneA1.5B1\") warper_creator = new cv::CompressedRectilinearWarper(1.5, 1);\n    else if (opts.warp_type == \"compressedPlanePortraitA2B1\") warper_creator = new cv::CompressedRectilinearPortraitWarper(2, 1);\n    else if (opts.warp_type == \"compressedPlanePortraitA1.5B1\") warper_creator = new cv::CompressedRectilinearPortraitWarper(1.5, 1);\n    else if (opts.warp_type == \"paniniA2B1\") warper_creator = new cv::PaniniWarper(2, 1);\n    else if (opts.warp_type == \"paniniA1.5B1\") warper_creator = new cv::PaniniWarper(1.5, 1);\n    else if (opts.warp_type == \"paniniPortraitA2B1\") warper_creator = new cv::PaniniPortraitWarper(2, 1);\n    else if (opts.warp_type == \"paniniPortraitA1.5B1\") warper_creator = new cv::PaniniPortraitWarper(1.5, 1);\n    else if (opts.warp_type == \"mercator\") warper_creator = new cv::MercatorWarper();\n    else if (opts.warp_type == \"transverseMercator\") warper_creator = new cv::TransverseMercatorWarper();\n    \n    if (warper_creator.empty())\n        logstream(LOG_ERROR) << \"Can't create the following warper '\" << opts.warp_type << \"'\\n\";\n    \n    Ptr warper = warper_creator->create(static_cast(opts.warped_image_scale * opts.compose_work_aspect));\n    \n    // Update intrinsics\n    camera.focal *= opts.compose_work_aspect;\n    camera.ppx *= opts.compose_work_aspect;\n    camera.ppy *= opts.compose_work_aspect;\n    \n    // Update corner and size\n    //vdata.full_img_size = full_img.size();\n    Size sz = vdata.full_img_size;\n    if (std::abs(opts.compose_scale - 1) > 1e-1)\n    {\n        sz.width = cvRound(vdata.full_img_size.width * opts.compose_scale);\n        sz.height = cvRound(vdata.full_img_size.height * opts.compose_scale);\n    }\n    \n    Mat K;\n    camera.K().convertTo(K, CV_32F);\n    Rect roi = warper->warpRoi(sz, K, camera.R);\n    corner = roi.tl();\n    if (opts.verbose > 2)\n        LOGLN(\"Compose corner x : \" << corner.x << \"   y : \" << corner.y << \"\\n\");\n    \n    size = roi.size();\n    if (opts.verbose > 2)\n        LOGLN(\"Compose size height : \" << size.height << \"   width : \" << size.width << \"\\n\");\n    \n    if (abs(opts.compose_scale - 1) > 1e-1)\n        resize(full_img, img, Size(), opts.compose_scale, opts.compose_scale);\n    else\n        img = full_img;\n    Size img_size = img.size();\n    \n    // Warp the current image\n    warper->warp(img, K, camera.R, INTER_LINEAR, BORDER_REFLECT, img_warped);\n    \n    // Warp the current image mask\n    mask.create(img_size, CV_8U);\n    mask.setTo(Scalar::all(255));\n    warper->warp(mask, K, camera.R, INTER_NEAREST, BORDER_CONSTANT, masks_warped);\n    \n    // Compensate exposure\n    //compensator->apply(img_idx, corner[img_idx], img_warped, mask_warped);\n    \n    img.release();\n    mask.release();\n    \n    dilate(mask_warped, dilated_mask, Mat());\n    resize(dilated_mask, seam_mask, masks_warped.size());\n    mask_warped = seam_mask & masks_warped;\n    \n}\n\n/////////////////////////////////////////////////////////////////////////\n// Function to compute feature-matches in parallel on edges\nvoid find_seams(graph_type::edge_type& edge)\n{\n    // Get edge data\n    //edge_data &edata = edge.data(); //commented by me as it was unused\n\t\n    // Get vertex ids of two vertices involved\n    vertex_data &vdata1 = edge.source().data();\n    vertex_data &vdata2 = edge.target().data();\n    \n    // Not sure why this is needed anymore?\n    //Ptr seam_finder;\n    //seam_finder = new detail::GraphCutSeamFinder(GraphCutSeamFinderBase::COST_COLOR);\n    \n    \n    // Code from PairwiseSeamFinder::Impl::findInPair()\n    //Mat img1 = images_[first], img2 = images_[second];\n    Mat &img1 = vdata1.img_warped_f; Mat &img2 = vdata2.img_warped_f;\n    vector src; src.push_back(img1); src.push_back(img2);\n    \n    vector dx_(2), dy_(2);  \n    Mat dx, dy;\n    for (size_t i = 0; i < src.size(); ++i)\n    {\n        CV_Assert(src[i].channels() == 3);\n        Sobel(src[i], dx, CV_32F, 1, 0);\n        Sobel(src[i], dy, CV_32F, 0, 1);\n        dx_[i].create(src[i].size(), CV_32F);\n        dy_[i].create(src[i].size(), CV_32F);\n        for (int y = 0; y < src[i].rows; ++y)\n        {\n            const Point3f* dx_row = dx.ptr(y);\n            const Point3f* dy_row = dy.ptr(y);\n            float* dx_row_ = dx_[i].ptr(y);\n            float* dy_row_ = dy_[i].ptr(y);\n            for (int x = 0; x < src[i].cols; ++x)\n            {\n                dx_row_[x] = normL2(dx_row[x]);\n                dy_row_[x] = normL2(dy_row[x]);\n            }\n        }\n    }\n    \n    //Mat dx1 = dx_[first], dx2 = dx_[second];\n    //Mat dy1 = dy_[first], dy2 = dy_[second];\n    Mat &dx1 = dx_[0]; Mat &dx2 = dx_[1];\n    Mat &dy1 = dy_[0]; Mat &dy2 = dy_[1];\n    \n    //Mat mask1 = masks_[first], mask2 = masks_[second];\n    Mat &mask1 = vdata1.mask_warped; Mat &mask2 = vdata2.mask_warped;\n    //Point tl1 = corners_[first], tl2 = corners_[second];\n    Point2f &tl1 = vdata1.corner; Point2f &tl2 = vdata2.corner;\n    \n    Rect roi;\n    overlapRoi(tl1, tl2, img1.size(), img2.size(), roi);\n    \n    const int gap = 10;\n    Mat subimg1(roi.height + 2 * gap, roi.width + 2 * gap, CV_32FC3);\n    Mat subimg2(roi.height + 2 * gap, roi.width + 2 * gap, CV_32FC3);\n    Mat submask1(roi.height + 2 * gap, roi.width + 2 * gap, CV_8U);\n    Mat submask2(roi.height + 2 * gap, roi.width + 2 * gap, CV_8U);\n    Mat subdx1(roi.height + 2 * gap, roi.width + 2 * gap, CV_32F);\n    Mat subdy1(roi.height + 2 * gap, roi.width + 2 * gap, CV_32F);\n    Mat subdx2(roi.height + 2 * gap, roi.width + 2 * gap, CV_32F);\n    Mat subdy2(roi.height + 2 * gap, roi.width + 2 * gap, CV_32F);\n    \n    // Cut subimages and submasks with some gap\n    for (int y = -gap; y < roi.height + gap; ++y)\n    {\n        for (int x = -gap; x < roi.width + gap; ++x)\n        {\n            int y1 = roi.y - tl1.y + y;\n            int x1 = roi.x - tl1.x + x;\n            if (y1 >= 0 && x1 >= 0 && y1 < img1.rows && x1 < img1.cols)\n            {\n                subimg1.at(y + gap, x + gap) = img1.at(y1, x1);\n                submask1.at(y + gap, x + gap) = mask1.at(y1, x1);\n                subdx1.at(y + gap, x + gap) = dx1.at(y1, x1);\n                subdy1.at(y + gap, x + gap) = dy1.at(y1, x1);\n            }\n            else\n            {\n                subimg1.at(y + gap, x + gap) = Point3f(0, 0, 0);\n                submask1.at(y + gap, x + gap) = 0;\n                subdx1.at(y + gap, x + gap) = 0.f;\n                subdy1.at(y + gap, x + gap) = 0.f;\n            }\n            \n            int y2 = roi.y - tl2.y + y;\n            int x2 = roi.x - tl2.x + x;\n            if (y2 >= 0 && x2 >= 0 && y2 < img2.rows && x2 < img2.cols)\n            {\n                subimg2.at(y + gap, x + gap) = img2.at(y2, x2);\n                submask2.at(y + gap, x + gap) = mask2.at(y2, x2);\n                subdx2.at(y + gap, x + gap) = dx2.at(y2, x2);\n                subdy2.at(y + gap, x + gap) = dy2.at(y2, x2);\n            }\n            else\n            {\n                subimg2.at(y + gap, x + gap) = Point3f(0, 0, 0);\n                submask2.at(y + gap, x + gap) = 0;\n                subdx2.at(y + gap, x + gap) = 0.f;\n                subdy2.at(y + gap, x + gap) = 0.f;\n            }\n        }\n    }\n    \n    const int vertex_count = (roi.height + 2 * gap) * (roi.width + 2 * gap);\n    const int edge_count = (roi.height - 1 + 2 * gap) * (roi.width + 2 * gap) + (roi.width - 1 + 2 * gap) * (roi.height + 2 * gap);\n    GCGraph graph(vertex_count, edge_count);\t\n    \n    \n    const Size img_size = subimg1.size();\n    \n    if (opts.seam_find_type.compare(\"gc_color\") ==0)\n    {\n    \t// Set terminal weights\n    \tfor (int y = 0; y < img_size.height; ++y)\n    \t{\n            for (int x = 0; x < img_size.width; ++x)\n            {\n                int v = graph.addVtx();\n                graph.addTermWeights(v, submask1.at(y, x) ? opts.terminal_cost : 0.f, \n                                     submask2.at(y, x) ? opts.terminal_cost : 0.f);\n            }\n    \t}\n        \n    \t// Set regular edge weights\n    \tconst float weight_eps = 1.f;\n        \n        for (int y = 0; y < img_size.height; ++y)\n    \t{\n            for (int x = 0; x < img_size.width; ++x)\n            {\n                int v = y * img_size.width + x;\n                \n                if (x < img_size.width - 1)\n                {\n                    float weight = normL2(subimg1.at(y, x), subimg2.at(y, x)) +\n                    normL2(subimg1.at(y, x + 1), subimg2.at(y, x + 1)) + weight_eps;\n                    \n                    if (!submask1.at(y, x) || !submask1.at(y, x + 1) || \n                        !submask2.at(y, x) || !submask2.at(y, x + 1))\n                        weight += opts.bad_region_penalty;\n                    \n                    graph.addEdges(v, v + 1, weight, weight);\n            \t}\n            \tif (y < img_size.height - 1)\n            \t{\n                    float weight = normL2(subimg1.at(y, x), subimg2.at(y, x)) +\n                    normL2(subimg1.at(y + 1, x), subimg2.at(y + 1, x)) + weight_eps;\n                    \n                    if (!submask1.at(y, x) || !submask1.at(y + 1, x) ||\n                    \t!submask2.at(y, x) || !submask2.at(y + 1, x))\n                        weight += opts.bad_region_penalty;\n                    \n                    graph.addEdges(v, v + img_size.width, weight, weight);\n            \t}\n       \t    }\n        }\n    }\n    \n    \n    else if (opts.seam_find_type.compare(\"gc_colorgrad\") ==0)\n    {\n        // Set terminal weights\n    \tfor (int y = 0; y < img_size.height; ++y)\n    \t{\n            for (int x = 0; x < img_size.width; ++x)\n            {\n                int v = graph.addVtx();\n                graph.addTermWeights(v, submask1.at(y, x) ? opts.terminal_cost : 0.f, \n                                     submask2.at(y, x) ? opts.terminal_cost : 0.f);\n            }\n        }\n        \n        // Set regular edge weights\n        const float weight_eps = 1.f;\n        for (int y = 0; y < img_size.height; ++y)\n        {\n            for (int x = 0; x < img_size.width; ++x)\n            {\n                int v = y * img_size.width + x;\n                if (x < img_size.width - 1)\n                {\n                    float grad = subdx1.at(y, x) + subdx1.at(y, x + 1) +\n                    subdx2.at(y, x) + subdx2.at(y, x + 1) + weight_eps;\n                    float weight = (normL2(subimg1.at(y, x), subimg2.at(y, x)) +\n                                    normL2(subimg1.at(y, x + 1), subimg2.at(y, x + 1))) / grad + weight_eps;\n                    if (!submask1.at(y, x) || !submask1.at(y, x + 1) ||\n                        !submask2.at(y, x) || !submask2.at(y, x + 1))\n                        weight += opts.bad_region_penalty;\n                    \n                    graph.addEdges(v, v + 1, weight, weight);\n                }\n                if (y < img_size.height - 1)\n                {\n                    float grad = subdy1.at(y, x) + subdy1.at(y + 1, x) + \n                    subdy2.at(y, x) + subdy2.at(y + 1, x) + weight_eps;\n                    float weight = (normL2(subimg1.at(y, x), subimg2.at(y, x)) + \n                                    normL2(subimg1.at(y + 1, x), subimg2.at(y + 1, x))) / grad + weight_eps;\n                    \n                    if (!submask1.at(y, x) || !submask1.at(y + 1, x) ||\n                        !submask2.at(y, x) || !submask2.at(y + 1, x))\n                        weight += opts.bad_region_penalty;\n                    \n                    graph.addEdges(v, v + img_size.width, weight, weight);\n                }\n            }\n        }\n    }\n    \n    else\n        CV_Error(CV_StsBadArg, \"unsupported pixel similarity measure\");\n    \n    graph.maxFlow();\n    \n    for (int y = 0; y < roi.height; ++y)\n    {\n        for (int x = 0; x < roi.width; ++x)\n        {\n            if (graph.inSourceSegment((y + gap) * (roi.width + 2 * gap) + x + gap))\n            {\n                if (mask1.at(roi.y - tl1.y + y, roi.x - tl1.x + x))\n                    mask2.at(roi.y - tl2.y + y, roi.x - tl2.x + x) = 0;\n            }\n            else\n            {\n                if (mask2.at(roi.y - tl2.y + y, roi.x - tl2.x + x))\n                    mask1.at(roi.y - tl1.y + y, roi.x - tl1.x + x) = 0;\n            }\n        }\n    }\n}\n\n/////////////////////////////////////////////////////////////////////////\n// Map Function to compile a list of features\n//vector compile_features(const graph_type::vertex_type& vertex)\nvector compile_vertices(engine_type::icontext_type& context,\n                                     const graph_type::vertex_type& vertex)\n{\n    vector temp(context.num_vertices());\n    \n    temp[vertex.id()] = vertex.data();\n    return temp;\n}\n\n/////////////////////////////////////////////////////////////////////////\n// Map Function to compile a list of matches\n//vector compile_features(const graph_type::vertex_type& vertex)\nvector compile_edges(engine_type::icontext_type& context,\n                                const graph_type::edge_type& edge)\n{\n    \n    int edlist_len = context.num_vertices() * context.num_vertices();\n    vector temp(edlist_len);\n    \n    int pair_idx = edge.source().id() * context.num_vertices() + edge.target().id();\n    temp[pair_idx] = edge.data();\n    return temp;\n}\n\n#endif\n\n"
  },
  {
    "path": "toolkits/computer_vision/stitch_opts.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n *\n * \\brief This file contains an example of graphlab used for stitching\n * multiple images into a panorama. The code is based on a example\n * stiching application in OpenCV.\n *\n *  \\author Dhruv Batra\n */\n\n\n#ifndef __STITCH_OPTS_HPP__\n#define __STITCH_OPTS_HPP__\n\n#include \n\n#include \"opencv2/opencv_modules.hpp\"\n#include \"opencv2/highgui/highgui.hpp\"\n#include \"opencv2/stitching/detail/autocalib.hpp\"\n#include \"opencv2/stitching/detail/blenders.hpp\"\n#include \"opencv2/stitching/detail/camera.hpp\"\n#include \"opencv2/stitching/detail/exposure_compensate.hpp\"\n#include \"opencv2/stitching/detail/matchers.hpp\"\n#include \"opencv2/stitching/detail/motion_estimators.hpp\"\n#include \"opencv2/stitching/detail/seam_finders.hpp\"\n#include \"opencv2/stitching/detail/util.hpp\"\n#include \"opencv2/stitching/detail/warpers.hpp\"\n#include \"opencv2/stitching/warpers.hpp\"\n\n\n/////////////////////////////////////////////////////////////////////////\n// Option Struct\nstruct Options\n{\n    // graphlab options\n    std::string exec_type;\n   \n    // input output dirs\n    std::string output_dir;\n\n    int verbose;\n   \n    bool try_gpu;\n    // size of images\n    double work_megapix;\n    double seam_megapix;\n    double compose_megapix;\n    double output_megapix;\n\n    double work_scale;\n    double seam_scale;\n    double compose_scale;\n    double output_scale;\n   \n    double seam_work_aspect;\n    double compose_seam_aspect;\n    double compose_work_aspect;\n //   double output_work_aspect;\n   \n    double warped_image_scale;\n    std::string warp_type;\n\n    // match options\n    double conf_thresh;\n    float match_conf;\n\n    // seam options\n    std::string seam_find_type;\n    float terminal_cost;\n    float bad_region_penalty; \n\n    //wave correction options\n    std::string wave_correct_type;\n\n    //bundle adjustment options\n    std::string ba_cost_func;\n    std::string ba_refine_mask;\n\n    //gain compensation options\n    std::string expose_comp_type;\n\n    //blending options\n    std::string blending_type;\n    float blend_strength;\n\n    //saving output\n    std::string result_name;\n        \n   //saving the adjacency list for creating the graph\n   //std::string graph_name;\n   \n    // Default values\n    Options():\n    exec_type(\"async\"),\n    output_dir(\"./\"),\n    verbose(0),\n    try_gpu(false),\n    work_megapix(0.6), seam_megapix(0.1), compose_megapix(-1), output_megapix(0.6),\n    work_scale(1), seam_scale(1), compose_scale(1), output_scale(1),\n    seam_work_aspect(1/6), compose_seam_aspect(1), compose_work_aspect(1),\n    warped_image_scale(-1), warp_type(\"spherical\"),\n    conf_thresh(1.f), match_conf(0.3f),\n    seam_find_type(\"gc_color\"), terminal_cost(10000.f), bad_region_penalty(1000.f),\n    wave_correct_type(\"horiz\"),\n    ba_cost_func(\"ray\"),\n    ba_refine_mask(\"xxxxx\"),\n    expose_comp_type(\"gain_blocks\"),\n    blending_type(\"multiband\"), blend_strength(5),\n    result_name(\"result_stitch.jpg\")\n    {}\n};\n\n// output_megapix(1), output_scale(1), \nextern Options opts;\n\n#endif\n\n"
  },
  {
    "path": "toolkits/computer_vision/stitching.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n\n#include \n#include \n#include \"opencv2/highgui/highgui.hpp\"\n#include \"opencv2/stitching/stitcher.hpp\"\n\nusing namespace std;\nusing namespace cv;\n\nbool try_use_gpu = false;\nvector imgs;\nstring result_name = \"result.png\";\n\nvoid printUsage();\nint parseCmdArgs(int argc, char** argv);\n\nint main(int argc, char* argv[])\n{\n    int retval = parseCmdArgs(argc, argv);\n    if (retval) return -1;\n\n    Mat pano;\n    Stitcher stitcher = Stitcher::createDefault(try_use_gpu);\n    Stitcher::Status status = stitcher.stitch(imgs, pano);\n\n    if (status != Stitcher::OK)\n    {\n        cout << \"Can't stitch images, error code = \" << status << endl;\n        return -1;\n    }\n\n    imwrite(result_name, pano);\n    return 0;\n}\n\n\nvoid printUsage()\n{\n    cout <<\n        \"Rotation model images stitcher.\\n\\n\"\n        \"stitching img1 img2 [...imgN]\\n\\n\"\n        \"Flags:\\n\"\n        \"  --try_use_gpu (yes|no)\\n\"\n        \"      Try to use GPU. The default value is 'no'. All default values\\n\"\n        \"      are for CPU mode.\\n\"\n        \"  --output \\n\"\n        \"      The default is 'result.jpg'.\\n\";\n}\n\n\nint parseCmdArgs(int argc, char** argv)\n{\n    if (argc == 1)\n    {\n        printUsage();\n        return -1;\n    }\n    for (int i = 1; i < argc; ++i)\n    {\n        if (string(argv[i]) == \"--help\" || string(argv[i]) == \"/?\")\n        {\n            printUsage();\n            return -1;\n        }\n        else if (string(argv[i]) == \"--try_use_gpu\")\n        {\n            if (string(argv[i + 1]) == \"no\")\n                try_use_gpu = false;\n            else if (string(argv[i + 1]) == \"yes\")\n                try_use_gpu = true;\n            else\n            {\n                cout << \"Bad --try_use_gpu flag value\\n\";\n                return -1;\n            }\n            i++;\n        }\n        else if (string(argv[i]) == \"--output\")\n        {\n            result_name = argv[i + 1];\n            i++;\n        }\n        else\n        {\n            Mat img = imread(argv[i]);\n            if (img.empty())\n            {\n                cout << \"Can't read image '\" << argv[i] << \"'\\n\";\n                return -1;\n            }\n            imgs.push_back(img);\n        }\n    }\n    return 0;\n}\n\n\n"
  },
  {
    "path": "toolkits/computer_vision/stitching_detailed.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \n#include \n#include \n#include \"opencv2/opencv_modules.hpp\"\n#include \"opencv2/highgui/highgui.hpp\"\n#include \"opencv2/stitching/detail/autocalib.hpp\"\n#include \"opencv2/stitching/detail/blenders.hpp\"\n#include \"opencv2/stitching/detail/camera.hpp\"\n#include \"opencv2/stitching/detail/exposure_compensate.hpp\"\n#include \"opencv2/stitching/detail/matchers.hpp\"\n#include \"opencv2/stitching/detail/motion_estimators.hpp\"\n#include \"opencv2/stitching/detail/seam_finders.hpp\"\n#include \"opencv2/stitching/detail/util.hpp\"\n#include \"opencv2/stitching/detail/warpers.hpp\"\n#include \"opencv2/stitching/warpers.hpp\"\n\nusing namespace std;\nusing namespace cv;\nusing namespace cv::detail;\n\nvoid printUsage()\n{\n    cout <<\n        \"Rotation model images stitcher.\\n\\n\"\n        \"stitching_detailed img1 img2 [...imgN] [flags]\\n\\n\"\n        \"Flags:\\n\"\n        \"  --preview\\n\"\n        \"      Run stitching in the preview mode. Works faster than usual mode,\\n\"\n        \"      but output image will have lower resolution.\\n\"\n        \"  --try_gpu (yes|no)\\n\"\n        \"      Try to use GPU. The default value is 'no'. All default values\\n\"\n        \"      are for CPU mode.\\n\"\n        \"\\nMotion Estimation Flags:\\n\"\n        \"  --work_megapix \\n\"\n        \"      Resolution for image registration step. The default is 0.6 Mpx.\\n\"\n        \"  --features (surf|orb)\\n\"\n        \"      Type of features used for images matching. The default is surf.\\n\"\n        \"  --match_conf \\n\"\n        \"      Confidence for feature matching step. The default is 0.65 for surf and 0.3 for orb.\\n\"\n        \"  --conf_thresh \\n\"\n        \"      Threshold for two images are from the same panorama confidence.\\n\"\n        \"      The default is 1.0.\\n\"\n        \"  --ba (reproj|ray)\\n\"\n        \"      Bundle adjustment cost function. The default is ray.\\n\"\n        \"  --ba_refine_mask (mask)\\n\"\n        \"      Set refinement mask for bundle adjustment. It looks like 'x_xxx',\\n\"\n        \"      where 'x' means refine respective parameter and '_' means don't\\n\"\n        \"      refine one, and has the following format:\\n\"\n        \"      . The default mask is 'xxxxx'. If bundle\\n\"\n        \"      adjustment doesn't support estimation of selected parameter then\\n\"\n        \"      the respective flag is ignored.\\n\"\n        \"  --wave_correct (no|horiz|vert)\\n\"\n        \"      Perform wave effect correction. The default is 'horiz'.\\n\"\n        \"  --save_graph \\n\"\n        \"      Save matches graph represented in DOT language to  file.\\n\"\n        \"      Labels description: Nm is number of matches, Ni is number of inliers,\\n\"\n        \"      C is confidence.\\n\"\n        \"\\nCompositing Flags:\\n\"\n        \"  --warp (plane|cylindrical|spherical|fisheye|stereographic|compressedPlaneA2B1|compressedPlaneA1.5B1|compressedPlanePortraitA2B1|compressedPlanePortraitA1.5B1|paniniA2B1|paniniA1.5B1|paniniPortraitA2B1|paniniPortraitA1.5B1|mercator|transverseMercator)\\n\"\n        \"      Warp surface type. The default is 'spherical'.\\n\"\n        \"  --seam_megapix \\n\"\n        \"      Resolution for seam estimation step. The default is 0.1 Mpx.\\n\"\n        \"  --seam (no|voronoi|gc_color|gc_colorgrad)\\n\"\n        \"      Seam estimation method. The default is 'gc_color'.\\n\"\n        \"  --compose_megapix \\n\"\n        \"      Resolution for compositing step. Use -1 for original resolution.\\n\"\n        \"      The default is -1.\\n\"\n        \"  --expos_comp (no|gain|gain_blocks)\\n\"\n        \"      Exposure compensation method. The default is 'gain_blocks'.\\n\"\n        \"  --blend (no|feather|multiband)\\n\"\n        \"      Blending method. The default is 'multiband'.\\n\"\n        \"  --blend_strength \\n\"\n        \"      Blending strength from [0,100] range. The default is 5.\\n\"\n        \"  --output \\n\"\n        \"      The default is 'result.jpg'.\\n\";\n}\n\n\n// Default command line args\nvector img_names;\nbool preview = false;\nbool try_gpu = false;\ndouble work_megapix = 0.6;\ndouble seam_megapix = 0.1;\ndouble compose_megapix = -1;\nfloat conf_thresh = 1.f;\nstring features = \"surf\";\nstring ba_cost_func = \"ray\";\nstring ba_refine_mask = \"xxxxx\";\nbool do_wave_correct = true;\nWaveCorrectKind wave_correct = detail::WAVE_CORRECT_HORIZ;\nbool save_graph = false;\nstd::string save_graph_to;\nstring warp_type = \"spherical\";\nint expos_comp_type = ExposureCompensator::GAIN_BLOCKS;\nfloat match_conf = 0.3f;\nstring seam_find_type = \"gc_color\";\nint blend_type = Blender::MULTI_BAND;\nfloat blend_strength = 5;\nstring result_name = \"result.jpg\";\n\nint parseCmdArgs(int argc, char** argv)\n{\n    if (argc == 1)\n    {\n        printUsage();\n        return -1;\n    }\n    for (int i = 1; i < argc; ++i)\n    {\n        if (string(argv[i]) == \"--help\" || string(argv[i]) == \"/?\")\n        {\n            printUsage();\n            return -1;\n        }\n        else if (string(argv[i]) == \"--preview\")\n        {\n            preview = true;\n        }\n        else if (string(argv[i]) == \"--try_gpu\")\n        {\n            if (string(argv[i + 1]) == \"no\")\n                try_gpu = false;\n            else if (string(argv[i + 1]) == \"yes\")\n                try_gpu = true;\n            else\n            {\n                cout << \"Bad --try_gpu flag value\\n\";\n                return -1;\n            }\n            i++;\n        }\n        else if (string(argv[i]) == \"--work_megapix\")\n        {\n            work_megapix = atof(argv[i + 1]);\n            i++;\n        }\n        else if (string(argv[i]) == \"--seam_megapix\")\n        {\n            seam_megapix = atof(argv[i + 1]);\n            i++;\n        }\n        else if (string(argv[i]) == \"--compose_megapix\")\n        {\n            compose_megapix = atof(argv[i + 1]);\n            i++;\n        }\n        else if (string(argv[i]) == \"--result\")\n        {\n            result_name = argv[i + 1];\n            i++;\n        }\n        else if (string(argv[i]) == \"--features\")\n        {\n            features = argv[i + 1];\n            if (features == \"orb\")\n                match_conf = 0.3f;\n            i++;\n        }\n        else if (string(argv[i]) == \"--match_conf\")\n        {\n            match_conf = static_cast(atof(argv[i + 1]));\n            i++;\n        }\n        else if (string(argv[i]) == \"--conf_thresh\")\n        {\n            conf_thresh = static_cast(atof(argv[i + 1]));\n            i++;\n        }\n        else if (string(argv[i]) == \"--ba\")\n        {\n            ba_cost_func = argv[i + 1];\n            i++;\n        }\n        else if (string(argv[i]) == \"--ba_refine_mask\")\n        {\n            ba_refine_mask = argv[i + 1];\n            if (ba_refine_mask.size() != 5)\n            {\n                cout << \"Incorrect refinement mask length.\\n\";\n                return -1;\n            }\n            i++;\n        }\n        else if (string(argv[i]) == \"--wave_correct\")\n        {\n            if (string(argv[i + 1]) == \"no\")\n                do_wave_correct = false;\n            else if (string(argv[i + 1]) == \"horiz\")\n            {\n                do_wave_correct = true;\n                wave_correct = detail::WAVE_CORRECT_HORIZ;\n            }\n            else if (string(argv[i + 1]) == \"vert\")\n            {\n                do_wave_correct = true;\n                wave_correct = detail::WAVE_CORRECT_VERT;\n            }\n            else\n            {\n                cout << \"Bad --wave_correct flag value\\n\";\n                return -1;\n            }\n            i++;\n        }\n        else if (string(argv[i]) == \"--save_graph\")\n        {\n            save_graph = true;\n            save_graph_to = argv[i + 1];\n            i++;\n        }\n        else if (string(argv[i]) == \"--warp\")\n        {\n            warp_type = string(argv[i + 1]);\n            i++;\n        }\n        else if (string(argv[i]) == \"--expos_comp\")\n        {\n            if (string(argv[i + 1]) == \"no\")\n                expos_comp_type = ExposureCompensator::NO;\n            else if (string(argv[i + 1]) == \"gain\")\n                expos_comp_type = ExposureCompensator::GAIN;\n            else if (string(argv[i + 1]) == \"gain_blocks\")\n                expos_comp_type = ExposureCompensator::GAIN_BLOCKS;\n            else\n            {\n                cout << \"Bad exposure compensation method\\n\";\n                return -1;\n            }\n            i++;\n        }\n        else if (string(argv[i]) == \"--seam\")\n        {\n            if (string(argv[i + 1]) == \"no\" ||\n                string(argv[i + 1]) == \"voronoi\" ||\n                string(argv[i + 1]) == \"gc_color\" ||\n                string(argv[i + 1]) == \"gc_colorgrad\")\n                seam_find_type = argv[i + 1];\n            else\n            {\n                cout << \"Bad seam finding method\\n\";\n                return -1;\n            }\n            i++;\n        }\n        else if (string(argv[i]) == \"--blend\")\n        {\n            if (string(argv[i + 1]) == \"no\")\n                blend_type = Blender::NO;\n            else if (string(argv[i + 1]) == \"feather\")\n                blend_type = Blender::FEATHER;\n            else if (string(argv[i + 1]) == \"multiband\")\n                blend_type = Blender::MULTI_BAND;\n            else\n            {\n                cout << \"Bad blending method\\n\";\n                return -1;\n            }\n            i++;\n        }\n        else if (string(argv[i]) == \"--blend_strength\")\n        {\n            blend_strength = static_cast(atof(argv[i + 1]));\n            i++;\n        }\n        else if (string(argv[i]) == \"--output\")\n        {\n            result_name = argv[i + 1];\n            i++;\n        }\n        else\n            img_names.push_back(argv[i]);\n    }\n    if (preview)\n    {\n        compose_megapix = 0.6;\n    }\n    return 0;\n}\n\n\nint main(int argc, char* argv[])\n{\n    int64 app_start_time = getTickCount();\n    cv::setBreakOnError(true);\n\n    int retval = parseCmdArgs(argc, argv);\n    if (retval)\n        return retval;\n\n    // Check if have enough images\n    int num_images = static_cast(img_names.size());\n    if (num_images < 2)\n    {\n        LOGLN(\"Need more images\");\n        return -1;\n    }\n\n    double work_scale = 1, seam_scale = 1, compose_scale = 1;\n    bool is_work_scale_set = false, is_seam_scale_set = false, is_compose_scale_set = false;\n\n    LOGLN(\"Finding features...\");\n    int64 t = getTickCount();\n\n    Ptr finder;\n    if (features == \"surf\")\n    {\n#ifdef HAVE_OPENCV_GPU\n        if (try_gpu && gpu::getCudaEnabledDeviceCount() > 0)\n            finder = new SurfFeaturesFinderGpu();\n        else\n#endif\n            finder = new SurfFeaturesFinder();\n    }\n    else if (features == \"orb\")\n    {\n        finder = new OrbFeaturesFinder();\n    }\n    else\n    {\n        cout << \"Unknown 2D features type: '\" << features << \"'.\\n\";\n        return -1;\n    }\n\n    Mat full_img, img;\n    vector features(num_images);\n    vector images(num_images);\n    vector full_img_sizes(num_images);\n    double seam_work_aspect = 1;\n\n    for (int i = 0; i < num_images; ++i)\n    {\n        full_img = imread(img_names[i]);\n        full_img_sizes[i] = full_img.size();\n\n        if (full_img.empty())\n        {\n            LOGLN(\"Can't open image \" << img_names[i]);\n            return -1;\n        }\n        if (work_megapix < 0)\n        {\n            img = full_img;\n            work_scale = 1;\n            is_work_scale_set = true;\n        }\n        else\n        {\n            if (!is_work_scale_set)\n            {\n                work_scale = min(1.0, sqrt(work_megapix * 1e6 / full_img.size().area()));\n                is_work_scale_set = true;\n            }\n            resize(full_img, img, Size(), work_scale, work_scale);\n        }\n        if (!is_seam_scale_set)\n        {\n            seam_scale = min(1.0, sqrt(seam_megapix * 1e6 / full_img.size().area()));\n            seam_work_aspect = seam_scale / work_scale;\n            is_seam_scale_set = true;\n        }\n\n        (*finder)(img, features[i]);\n        features[i].img_idx = i;\n        LOGLN(\"Features in image #\" << i << \": \" << features[i].keypoints.size());\n\n        resize(full_img, img, Size(), seam_scale, seam_scale);\n        images[i] = img.clone();\n    }\n\n    finder->collectGarbage();\n    full_img.release();\n    img.release();\n\n    LOGLN(\"Finding features, time: \" << ((getTickCount() - t) / getTickFrequency()) << \" sec\");\n\n    LOG(\"Pairwise matching\");\n    t = getTickCount();\n    vector pairwise_matches;\n    BestOf2NearestMatcher matcher(try_gpu, match_conf);\n    matcher(features, pairwise_matches);\n    matcher.collectGarbage();\n    LOGLN(\"Pairwise matching, time: \" << ((getTickCount() - t) / getTickFrequency()) << \" sec\");\n    LOGLN(\"pairwise_matches.size() = \" << pairwise_matches.size() << \"\\n\");\n    for (size_t i=0; i!=pairwise_matches.size(); ++i) \n    {\n        LOGLN(\"src_img_idx = \" << pairwise_matches[i].src_img_idx  << \"\\t\");\n        LOGLN(\"dst_img_idx = \" << pairwise_matches[i].dst_img_idx  << \"\\t\");\n        LOGLN(\"matches.size() = \" << pairwise_matches[i].matches.size()  << \"\\t\");\n        LOGLN(\"num_inliers = \" << pairwise_matches[i].num_inliers  << \"\\t\");\n        LOGLN(\"confidence = \" << pairwise_matches[i].confidence  << \"\\n\");\n    }\n    \n    // Check if we should save matches graph\n    if (save_graph)\n    {\n        LOGLN(\"Saving matches graph...\");\n        ofstream f(save_graph_to.c_str());\n        f << matchesGraphAsString(img_names, pairwise_matches, conf_thresh);\n//        for (int i=0; i!=pairwise_matches.size(); ++i)\n//            f << pairwise_matches[i].src_img_idx << \" \" << pairwise_matches[i].dst_img_idx << \"\\n\";\n    }\n\n    // Leave only images we are sure are from the same panorama\n    vector indices = leaveBiggestComponent(features, pairwise_matches, conf_thresh);\n    vector img_subset;\n    vector img_names_subset;\n    vector full_img_sizes_subset;\n    for (size_t i = 0; i < indices.size(); ++i)\n    {\n        img_names_subset.push_back(img_names[indices[i]]);\n        img_subset.push_back(images[indices[i]]);\n        full_img_sizes_subset.push_back(full_img_sizes[indices[i]]);\n    }\n\n    images = img_subset;\n    img_names = img_names_subset;\n    full_img_sizes = full_img_sizes_subset;\n\n    // Check if we still have enough images\n    num_images = static_cast(img_names.size());\n    if (num_images < 2)\n    {\n        LOGLN(\"Need more images\");\n        return -1;\n    }\n\n    LOG(\"Homography-based init\\n\");\n    t = getTickCount();\n    HomographyBasedEstimator estimator;\n    vector cameras;\n    estimator(features, pairwise_matches, cameras);\n    LOGLN(\"Homography-based init, time: \" << ((getTickCount() - t) / getTickFrequency()) << \" sec\");\n\n    for (size_t i = 0; i < cameras.size(); ++i)\n    {\n        Mat R;\n        cameras[i].R.convertTo(R, CV_32F);\n        cameras[i].R = R;\n        LOGLN(\"Initial intrinsics #\" << indices[i]+1 << \":\\n\" << cameras[i].K());\n    }\n\n    LOG(\"Bundle Adjustment\\n\");\n    t = getTickCount();\n    Ptr adjuster;\n    if (ba_cost_func == \"reproj\") adjuster = new detail::BundleAdjusterReproj();\n    else if (ba_cost_func == \"ray\") adjuster = new detail::BundleAdjusterRay();\n    else \n    { \n        cout << \"Unknown bundle adjustment cost function: '\" << ba_cost_func << \"'.\\n\"; \n        return -1; \n    }\n    adjuster->setConfThresh(conf_thresh);\n    Mat_ refine_mask = Mat::zeros(3, 3, CV_8U);\n    if (ba_refine_mask[0] == 'x') refine_mask(0,0) = 1;\n    if (ba_refine_mask[1] == 'x') refine_mask(0,1) = 1;\n    if (ba_refine_mask[2] == 'x') refine_mask(0,2) = 1;\n    if (ba_refine_mask[3] == 'x') refine_mask(1,1) = 1;\n    if (ba_refine_mask[4] == 'x') refine_mask(1,2) = 1;\n    adjuster->setRefinementMask(refine_mask);\n    (*adjuster)(features, pairwise_matches, cameras);\n    LOGLN(\"Bundle Adjustment, time: \" << ((getTickCount() - t) / getTickFrequency()) << \" sec\");\n\n    // Find median focal length\n\n    vector focals;\n    for (size_t i = 0; i < cameras.size(); ++i)\n    {\n        LOGLN(\"Camera #\" << indices[i]+1 << \":\\n\" << cameras[i].K());\n        focals.push_back(cameras[i].focal);\n    }\n\n    sort(focals.begin(), focals.end());\n    //LOGLN(\"Focals size: \" << focals.size() << \"\\n\");\n    //LOGLN(\" focals: \" << focals[0] << \"\\t\" << focals[1] << \"\\t\" << focals[2] << \"\\n\");  \n    //LOGLN(\"Focals size: \" << focals.size() << \"\\n\");    \n\n \n    float warped_image_scale;\n    if (focals.size() % 2 == 1)\n        warped_image_scale = static_cast(focals[focals.size() / 2]);\n    else\n        warped_image_scale = static_cast(focals[focals.size() / 2 - 1] + focals[focals.size() / 2]) * 0.5f;\n\n    if (do_wave_correct)\n    {\n        vector rmats;\n        for (size_t i = 0; i < cameras.size(); ++i)\n            rmats.push_back(cameras[i].R);\n        waveCorrect(rmats, wave_correct);\n        for (size_t i = 0; i < cameras.size(); ++i)\n            cameras[i].R = rmats[i];\n    }\n\n    LOGLN(\"Warping images (auxiliary)... \");\n    t = getTickCount();\n\n    vector corners(num_images);\n    vector masks_warped(num_images);\n    vector images_warped(num_images);\n    vector sizes(num_images);\n    vector masks(num_images);\n\n    // Preapre images masks\n    for (int i = 0; i < num_images; ++i)\n    {\n        masks[i].create(images[i].size(), CV_8U);\n        masks[i].setTo(Scalar::all(255));\n    }\n\n    // Warp images and their masks\n\n    Ptr warper_creator;\n#ifdef HAVE_OPENCV_GPU\n    if (try_gpu && gpu::getCudaEnabledDeviceCount() > 0)\n    {\n        if (warp_type == \"plane\") warper_creator = new cv::PlaneWarperGpu();\n        else if (warp_type == \"cylindrical\") warper_creator = new cv::CylindricalWarperGpu();\n        else if (warp_type == \"spherical\") warper_creator = new cv::SphericalWarperGpu();\n    }\n    else\n#endif\n    {\n        if (warp_type == \"plane\") warper_creator = new cv::PlaneWarper();\n        else if (warp_type == \"cylindrical\") warper_creator = new cv::CylindricalWarper();\n        else if (warp_type == \"spherical\") warper_creator = new cv::SphericalWarper();\n\t\telse if (warp_type == \"fisheye\") warper_creator = new cv::FisheyeWarper();\n\t\telse if (warp_type == \"stereographic\") warper_creator = new cv::StereographicWarper();\n\t\telse if (warp_type == \"compressedPlaneA2B1\") warper_creator = new cv::CompressedRectilinearWarper(2, 1);\n\t\telse if (warp_type == \"compressedPlaneA1.5B1\") warper_creator = new cv::CompressedRectilinearWarper(1.5, 1);\n\t\telse if (warp_type == \"compressedPlanePortraitA2B1\") warper_creator = new cv::CompressedRectilinearPortraitWarper(2, 1);\n\t\telse if (warp_type == \"compressedPlanePortraitA1.5B1\") warper_creator = new cv::CompressedRectilinearPortraitWarper(1.5, 1);\n\t\telse if (warp_type == \"paniniA2B1\") warper_creator = new cv::PaniniWarper(2, 1);\n\t\telse if (warp_type == \"paniniA1.5B1\") warper_creator = new cv::PaniniWarper(1.5, 1);\n\t\telse if (warp_type == \"paniniPortraitA2B1\") warper_creator = new cv::PaniniPortraitWarper(2, 1);\n\t\telse if (warp_type == \"paniniPortraitA1.5B1\") warper_creator = new cv::PaniniPortraitWarper(1.5, 1);\n\t\telse if (warp_type == \"mercator\") warper_creator = new cv::MercatorWarper();\n\t\telse if (warp_type == \"transverseMercator\") warper_creator = new cv::TransverseMercatorWarper();\n    }\n\n    if (warper_creator.empty())\n    {\n        cout << \"Can't create the following warper '\" << warp_type << \"'\\n\";\n        return 1;\n    }\n    \n    Ptr warper = warper_creator->create(static_cast(warped_image_scale * seam_work_aspect));\n\n    for (int i = 0; i < num_images; ++i)\n    {\n        Mat_ K;\n        cameras[i].K().convertTo(K, CV_32F);\n        float swa = (float)seam_work_aspect;\n        K(0,0) *= swa; K(0,2) *= swa;\n        K(1,1) *= swa; K(1,2) *= swa;\n\n        corners[i] = warper->warp(images[i], K, cameras[i].R, INTER_LINEAR, BORDER_REFLECT, images_warped[i]);\n        //cout << \"corners x : \" << corners[i].x << \"   y : \" << corners[i].y << endl;\n        sizes[i] = images_warped[i].size();\n        //cout << \"sizes height : \" << sizes[i].height << \"   width : \" << sizes[i].width << endl;\n\n        warper->warp(masks[i], K, cameras[i].R, INTER_NEAREST, BORDER_CONSTANT, masks_warped[i]);\n    }\n\n    vector images_warped_f(num_images);\n    for (int i = 0; i < num_images; ++i)\n        images_warped[i].convertTo(images_warped_f[i], CV_32F);\n\n    LOGLN(\"Warping images, time: \" << ((getTickCount() - t) / getTickFrequency()) << \" sec\");\n\n    Ptr compensator = ExposureCompensator::createDefault(expos_comp_type);\n    compensator->feed(corners, images_warped, masks_warped);\n\n    Ptr seam_finder;\n    if (seam_find_type == \"no\")\n        seam_finder = new detail::NoSeamFinder();\n    else if (seam_find_type == \"voronoi\")\n        seam_finder = new detail::VoronoiSeamFinder();\n    else if (seam_find_type == \"gc_color\")\n    {\n#ifdef HAVE_OPENCV_GPU\n        if (try_gpu && gpu::getCudaEnabledDeviceCount() > 0)\n            seam_finder = new detail::GraphCutSeamFinderGpu(GraphCutSeamFinderBase::COST_COLOR);\n        else\n#endif\n            seam_finder = new detail::GraphCutSeamFinder(GraphCutSeamFinderBase::COST_COLOR);\n    }\n    else if (seam_find_type == \"gc_colorgrad\")\n    {\n#ifdef HAVE_OPENCV_GPU\n        if (try_gpu && gpu::getCudaEnabledDeviceCount() > 0)\n            seam_finder = new detail::GraphCutSeamFinderGpu(GraphCutSeamFinderBase::COST_COLOR_GRAD);\n        else\n#endif\n            seam_finder = new detail::GraphCutSeamFinder(GraphCutSeamFinderBase::COST_COLOR_GRAD);\n    }\n    if (seam_finder.empty())\n    {\n        cout << \"Can't create the following seam finder '\" << seam_find_type << \"'\\n\";\n        return 1;\n    }\n\n    seam_finder->find(images_warped_f, corners, masks_warped);\n\n    // Release unused memory\n    images.clear();\n    images_warped.clear();\n    images_warped_f.clear();\n    masks.clear();\n\n    LOGLN(\"Compositing...\");\n    t = getTickCount();\n\n    Mat img_warped, img_warped_s;\n    Mat dilated_mask, seam_mask, mask, mask_warped;\n    Ptr blender;\n    double compose_seam_aspect = 1;\n    double compose_work_aspect = 1;\n\n    for (int img_idx = 0; img_idx < num_images; ++img_idx)\n    {\n        LOGLN(\"Compositing image #\" << indices[img_idx]+1);\n\n        // Read image and resize it if necessary\n        full_img = imread(img_names[img_idx]);\n        if (!is_compose_scale_set)\n        {\n            if (compose_megapix > 0)\n                compose_scale = min(1.0, sqrt(compose_megapix * 1e6 / full_img.size().area()));\n            is_compose_scale_set = true;\n\n            // Compute relative scales\n            compose_seam_aspect = compose_scale / seam_scale;\n            compose_work_aspect = compose_scale / work_scale;\n\n            // Update warped image scale\n            warped_image_scale *= static_cast(compose_work_aspect);\n            warper = warper_creator->create(warped_image_scale);\n\n            // Update corners and sizes\n            for (int i = 0; i < num_images; ++i)\n            {\n                // Update intrinsics\n                cameras[i].focal *= compose_work_aspect;\n                cameras[i].ppx *= compose_work_aspect;\n                cameras[i].ppy *= compose_work_aspect;\n\n                // Update corner and size\n                Size sz = full_img_sizes[i];\n                if (std::abs(compose_scale - 1) > 1e-1)\n                {\n                    sz.width = cvRound(full_img_sizes[i].width * compose_scale);\n                    sz.height = cvRound(full_img_sizes[i].height * compose_scale);\n                }\n\n                Mat K;\n                cameras[i].K().convertTo(K, CV_32F);\n                Rect roi = warper->warpRoi(sz, K, cameras[i].R);\n                corners[i] = roi.tl();\n                //cout << \"corner x : \" << corners[i].x << \"   y : \" << corners[i].y << endl;\n                sizes[i] = roi.size();\n                //cout << \"size height : \" << sizes[i].height << \"   width : \" << sizes[i].width << endl;\n            }\n        }\n        if (abs(compose_scale - 1) > 1e-1)\n            resize(full_img, img, Size(), compose_scale, compose_scale);\n        else\n            img = full_img;\n        full_img.release();\n        Size img_size = img.size();\n\n        Mat K;\n        cameras[img_idx].K().convertTo(K, CV_32F);\n\n        // Warp the current image\n        warper->warp(img, K, cameras[img_idx].R, INTER_LINEAR, BORDER_REFLECT, img_warped);\n\n        // Warp the current image mask\n        mask.create(img_size, CV_8U);\n        mask.setTo(Scalar::all(255));\n        warper->warp(mask, K, cameras[img_idx].R, INTER_NEAREST, BORDER_CONSTANT, mask_warped);\n        \n        // Compensate exposure\n        compensator->apply(img_idx, corners[img_idx], img_warped, mask_warped);\n\n        img_warped.convertTo(img_warped_s, CV_16S);\n        img_warped.release();\n        img.release();\n        mask.release();\n\n        dilate(masks_warped[img_idx], dilated_mask, Mat());\n        resize(dilated_mask, seam_mask, mask_warped.size());\n        mask_warped = seam_mask & mask_warped;\n\n        if (blender.empty())\n        {\n            blender = Blender::createDefault(blend_type, try_gpu);\n            Size dst_sz = resultRoi(corners, sizes).size();\n            float blend_width = sqrt(static_cast(dst_sz.area())) * blend_strength / 100.f;\n            if (blend_width < 1.f)\n                blender = Blender::createDefault(Blender::NO, try_gpu);\n            else if (blend_type == Blender::MULTI_BAND)\n            {\n                MultiBandBlender* mb = dynamic_cast(static_cast(blender));\n                mb->setNumBands(static_cast(ceil(log(blend_width)/log(2.)) - 1.));\n                LOGLN(\"Multi-band blender, number of bands: \" << mb->numBands());\n            }\n            else if (blend_type == Blender::FEATHER)\n            {\n                FeatherBlender* fb = dynamic_cast(static_cast(blender));\n                fb->setSharpness(1.f/blend_width);\n                LOGLN(\"Feather blender, sharpness: \" << fb->sharpness());\n            }\n            blender->prepare(corners, sizes);\n        }\n\n        // Blend the current image\n        blender->feed(img_warped_s, mask_warped, corners[img_idx]);\n    }\n\n    Mat result, result_mask;\n    blender->blend(result, result_mask);\n\n    LOGLN(\"Compositing, time: \" << ((getTickCount() - t) / getTickFrequency()) << \" sec\");\n\n    imwrite(result_name, result);\n\n    LOGLN(\"Finished, total time: \" << ((getTickCount() - app_start_time) / getTickFrequency()) << \" sec\");\n    return 0;\n}\n\n\n"
  },
  {
    "path": "toolkits/computer_vision/utils.hpp",
    "content": "\n// utils.hpp - miscellaneous utilities \n// Originally from Nicol N. Schraudolph's isinf package\n// Later expanded by Dhruv Batra\n\n#ifndef UTILS_HPP\n#define UTILS_HPP\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n// row-major array access\n#define ARR_RM(arr, r_ind, c_ind, ncols) (*(arr + r_ind*ncols + c_ind))\n// col-major array access\n#define ARR_CM(arr, r_ind, c_ind, nrows) (*(arr + c_ind*nrows + r_ind))\n\n// row-major ind2sub\n#define IND2SUB_RM(ind,r,c,ncols) \\\n        r = floor(ind/ncols);     \\\n        c = ind % ncols;\n// column-major ind2sub\n#define IND2SUB_CM(ind,r,c,nrows) \\\n        c = floor(ind/nrows);     \\\n        r = ind % nrows;     \n\n// row-major sub2ind\n#define SUB2IND_RM(r,c,ncols) r*ncols + c\n\n// col-major sub2ind\n#define SUB2IND_CM(r,c,nrows) c*nrows + r\n\n// operators & formatted I/O for vectors\n// inner product\ntemplate \ninline T operator*(const std::vector& a, const std::vector& b)\n{\n    assert(a.size() == b.size());\n    T sum(0);\n    for (size_t i = 0; i < a.size(); ++i)\n        sum += a[i]*b[i];\n    return sum;\n}\n\n// element-wise sum\ntemplate \ninline std::vector& operator+(const std::vector& a, const std::vector& b)\n{\n    assert(a.size() == b.size());\n    std::vector sum(a.size());\n    for (size_t i = 0; i < a.size(); ++i)\n        sum[i] = a[i]+b[i];\n    return sum;\n}\n\ntemplate \ninline std::vector& operator+=(std::vector& a, const T& b)\n{\n    typename std::vector::iterator i(a.begin());\n    while (i != a.end()) *i++ += b;\n    return a;\n}\n\ntemplate \ninline std::vector& operator-=(std::vector& a, const T& b)\n{\n    typename std::vector::iterator i(a.begin());\n    while (i != a.end()) *i++ -= b;\n    return a;\n}\n\ntemplate \ninline std::vector& operator*=(std::vector& a, const T& b)\n{\n    typename std::vector::iterator i(a.begin());\n    while (i != a.end()) *i++ *= b;\n    return a;\n}\n\ntemplate \ninline std::vector& operator/=(std::vector& a, const T& b)\n{\n    typename std::vector::iterator i(a.begin());\n    while (i != a.end()) *i++ /= b;\n    return a;\n}\n\ntemplate \ninline std::vector& operator+=(std::vector& a, const std::vector& b)\n{\n    assert(a.size() == b.size());\n    typename std::vector::iterator i(a.begin());\n    typename std::vector::const_iterator j(b.begin());\n    while (i != a.end()) *i++ += *j++;\n    return a;\n}\n\ntemplate \ninline std::vector& operator-=(std::vector& a, const std::vector& b)\n{\n    assert(a.size() == b.size());\n    typename std::vector::iterator i(a.begin());\n    typename std::vector::const_iterator j(b.begin());\n    while (i != a.end()) *i++ -= *j++;\n    return a;\n}\n\ntemplate \ninline std::vector& operator*=(std::vector& a, const std::vector& b)\n{\n    assert(a.size() == b.size());\n    typename std::vector::iterator i(a.begin());\n    typename std::vector::const_iterator j(b.begin());\n    while (i != a.end()) *i++ *= *j++;\n    return a;\n}\n\ntemplate \ninline std::vector& operator/=(std::vector& a, const std::vector& b)\n{\n    assert(a.size() == b.size());\n    typename std::vector::iterator i(a.begin());\n    typename std::vector::const_iterator j(b.begin());\n    while (i != a.end()) *i++ /= *j++;\n    return a;\n}\n\ntemplate \ninline std::ostream& operator<<(std::ostream& os, const std::vector& x)\n{\n    typename std::vector::const_iterator i(x.begin());\n    while(i != x.end()) os << *i++ << ' ';\n    return os;\n}\n\ntemplate \ninline std::istream& operator>>(std::istream& is, std::vector& x)\n{\n    std::string s;\n    const size_t n = x.size();\n\n    while (x.size() == n)\n    {\n        getline(is, s);\n        if (is.fail()) break;\n\n        std::istringstream iss(s);\n        T item;\n\n        iss >> item;\n        while (iss.good())\n        {\n            x.push_back(item);\n            iss >> item;\n        }\n        if (!iss.fail())\n            x.push_back(item);\n    }\n\n    return is;\n}\n\n// Function to write a vector to file (Assumes << is defined for type T)\n// CHECK_NULL is provided by Danny Tarlow's Nymph Utils\ntemplate  void WriteToFile(std::string fname, std::vector vecx)\n{\n    std::ofstream fout; \n    fout.open(fname.c_str());\n    \n    //CHECK_NULL(fout.fail(),\"Could not open file for writing results\\n\");\n    \n    fout << vecx;\n    \n    fout.close();\n}\n\n\n#endif\n\n"
  },
  {
    "path": "toolkits/extensions/CMakeLists.txt",
    "content": "project(Extensions)\n\nif(CPP11)\n\nadd_library(graphlab_extension STATIC \n            MurmurHash3.cpp \n            extension.cpp \n            extension_graph.cpp\n            extension_pagerank.cpp)\nadd_dependencies(graphlab_extension graphlab)\ntarget_link_libraries(graphlab_extension graphlab)\n\nadd_extension_executable(pagerank_extension_driver\n                         pagerank_extension_driver.cpp) \n\nendif()\n"
  },
  {
    "path": "toolkits/extensions/MurmurHash3.cpp",
    "content": "//-----------------------------------------------------------------------------\n// MurmurHash3 was written by Austin Appleby, and is placed in the public\n// domain. The author hereby disclaims copyright to this source code.\n\n// Note - The x86 and x64 versions do _not_ produce the same results, as the\n// algorithms are optimized for their respective platforms. You can still\n// compile and run any of them on any platform, but your performance with the\n// non-native version will be less than optimal.\n\n#include \"MurmurHash3.h\"\n\n//-----------------------------------------------------------------------------\n// Platform-specific functions and macros\n\n// Microsoft Visual Studio\n\n#if defined(_MSC_VER)\n\n#define FORCE_INLINE\t__forceinline\n\n#include \n\n#define ROTL32(x,y)\t_rotl(x,y)\n#define ROTL64(x,y)\t_rotl64(x,y)\n\n#define BIG_CONSTANT(x) (x)\n\n// Other compilers\n\n#else\t// defined(_MSC_VER)\n\n#define\tFORCE_INLINE __attribute__((always_inline))\n\ninline uint32_t rotl32 ( uint32_t x, int8_t r )\n{\n  return (x << r) | (x >> (32 - r));\n}\n\ninline uint64_t rotl64 ( uint64_t x, int8_t r )\n{\n  return (x << r) | (x >> (64 - r));\n}\n\n#define\tROTL32(x,y)\trotl32(x,y)\n#define ROTL64(x,y)\trotl64(x,y)\n\n#define BIG_CONSTANT(x) (x##LLU)\n\n#endif // !defined(_MSC_VER)\n\n//-----------------------------------------------------------------------------\n// Block read - if your platform needs to do endian-swapping or can only\n// handle aligned reads, do the conversion here\n\nFORCE_INLINE uint32_t getblock ( const uint32_t * p, int i )\n{\n  return p[i];\n}\n\nFORCE_INLINE uint64_t getblock ( const uint64_t * p, int i )\n{\n  return p[i];\n}\n\n//-----------------------------------------------------------------------------\n// Finalization mix - force all bits of a hash block to avalanche\n\nFORCE_INLINE uint32_t fmix ( uint32_t h )\n{\n  h ^= h >> 16;\n  h *= 0x85ebca6b;\n  h ^= h >> 13;\n  h *= 0xc2b2ae35;\n  h ^= h >> 16;\n\n  return h;\n}\n\n//----------\n\nFORCE_INLINE uint64_t fmix ( uint64_t k )\n{\n  k ^= k >> 33;\n  k *= BIG_CONSTANT(0xff51afd7ed558ccd);\n  k ^= k >> 33;\n  k *= BIG_CONSTANT(0xc4ceb9fe1a85ec53);\n  k ^= k >> 33;\n\n  return k;\n}\n\n//-----------------------------------------------------------------------------\n\nvoid MurmurHash3_x86_32 ( const void * key, int len,\n                          uint32_t seed, void * out )\n{\n  const uint8_t * data = (const uint8_t*)key;\n  const int nblocks = len / 4;\n\n  uint32_t h1 = seed;\n\n  const uint32_t c1 = 0xcc9e2d51;\n  const uint32_t c2 = 0x1b873593;\n\n  //----------\n  // body\n\n  const uint32_t * blocks = (const uint32_t *)(data + nblocks*4);\n\n  for(int i = -nblocks; i; i++)\n  {\n    uint32_t k1 = getblock(blocks,i);\n\n    k1 *= c1;\n    k1 = ROTL32(k1,15);\n    k1 *= c2;\n    \n    h1 ^= k1;\n    h1 = ROTL32(h1,13); \n    h1 = h1*5+0xe6546b64;\n  }\n\n  //----------\n  // tail\n\n  const uint8_t * tail = (const uint8_t*)(data + nblocks*4);\n\n  uint32_t k1 = 0;\n\n  switch(len & 3)\n  {\n  case 3: k1 ^= tail[2] << 16;\n  case 2: k1 ^= tail[1] << 8;\n  case 1: k1 ^= tail[0];\n          k1 *= c1; k1 = ROTL32(k1,15); k1 *= c2; h1 ^= k1;\n  };\n\n  //----------\n  // finalization\n\n  h1 ^= len;\n\n  h1 = fmix(h1);\n\n  *(uint32_t*)out = h1;\n} \n\n//-----------------------------------------------------------------------------\n\nvoid MurmurHash3_x86_128 ( const void * key, const int len,\n                           uint32_t seed, void * out )\n{\n  const uint8_t * data = (const uint8_t*)key;\n  const int nblocks = len / 16;\n\n  uint32_t h1 = seed;\n  uint32_t h2 = seed;\n  uint32_t h3 = seed;\n  uint32_t h4 = seed;\n\n  const uint32_t c1 = 0x239b961b; \n  const uint32_t c2 = 0xab0e9789;\n  const uint32_t c3 = 0x38b34ae5; \n  const uint32_t c4 = 0xa1e38b93;\n\n  //----------\n  // body\n\n  const uint32_t * blocks = (const uint32_t *)(data + nblocks*16);\n\n  for(int i = -nblocks; i; i++)\n  {\n    uint32_t k1 = getblock(blocks,i*4+0);\n    uint32_t k2 = getblock(blocks,i*4+1);\n    uint32_t k3 = getblock(blocks,i*4+2);\n    uint32_t k4 = getblock(blocks,i*4+3);\n\n    k1 *= c1; k1  = ROTL32(k1,15); k1 *= c2; h1 ^= k1;\n\n    h1 = ROTL32(h1,19); h1 += h2; h1 = h1*5+0x561ccd1b;\n\n    k2 *= c2; k2  = ROTL32(k2,16); k2 *= c3; h2 ^= k2;\n\n    h2 = ROTL32(h2,17); h2 += h3; h2 = h2*5+0x0bcaa747;\n\n    k3 *= c3; k3  = ROTL32(k3,17); k3 *= c4; h3 ^= k3;\n\n    h3 = ROTL32(h3,15); h3 += h4; h3 = h3*5+0x96cd1c35;\n\n    k4 *= c4; k4  = ROTL32(k4,18); k4 *= c1; h4 ^= k4;\n\n    h4 = ROTL32(h4,13); h4 += h1; h4 = h4*5+0x32ac3b17;\n  }\n\n  //----------\n  // tail\n\n  const uint8_t * tail = (const uint8_t*)(data + nblocks*16);\n\n  uint32_t k1 = 0;\n  uint32_t k2 = 0;\n  uint32_t k3 = 0;\n  uint32_t k4 = 0;\n\n  switch(len & 15)\n  {\n  case 15: k4 ^= tail[14] << 16;\n  case 14: k4 ^= tail[13] << 8;\n  case 13: k4 ^= tail[12] << 0;\n           k4 *= c4; k4  = ROTL32(k4,18); k4 *= c1; h4 ^= k4;\n\n  case 12: k3 ^= tail[11] << 24;\n  case 11: k3 ^= tail[10] << 16;\n  case 10: k3 ^= tail[ 9] << 8;\n  case  9: k3 ^= tail[ 8] << 0;\n           k3 *= c3; k3  = ROTL32(k3,17); k3 *= c4; h3 ^= k3;\n\n  case  8: k2 ^= tail[ 7] << 24;\n  case  7: k2 ^= tail[ 6] << 16;\n  case  6: k2 ^= tail[ 5] << 8;\n  case  5: k2 ^= tail[ 4] << 0;\n           k2 *= c2; k2  = ROTL32(k2,16); k2 *= c3; h2 ^= k2;\n\n  case  4: k1 ^= tail[ 3] << 24;\n  case  3: k1 ^= tail[ 2] << 16;\n  case  2: k1 ^= tail[ 1] << 8;\n  case  1: k1 ^= tail[ 0] << 0;\n           k1 *= c1; k1  = ROTL32(k1,15); k1 *= c2; h1 ^= k1;\n  };\n\n  //----------\n  // finalization\n\n  h1 ^= len; h2 ^= len; h3 ^= len; h4 ^= len;\n\n  h1 += h2; h1 += h3; h1 += h4;\n  h2 += h1; h3 += h1; h4 += h1;\n\n  h1 = fmix(h1);\n  h2 = fmix(h2);\n  h3 = fmix(h3);\n  h4 = fmix(h4);\n\n  h1 += h2; h1 += h3; h1 += h4;\n  h2 += h1; h3 += h1; h4 += h1;\n\n  ((uint32_t*)out)[0] = h1;\n  ((uint32_t*)out)[1] = h2;\n  ((uint32_t*)out)[2] = h3;\n  ((uint32_t*)out)[3] = h4;\n}\n\n//-----------------------------------------------------------------------------\n\nvoid MurmurHash3_x64_128 ( const void * key, const int len,\n                           const uint32_t seed, void * out )\n{\n  const uint8_t * data = (const uint8_t*)key;\n  const int nblocks = len / 16;\n\n  uint64_t h1 = seed;\n  uint64_t h2 = seed;\n\n  const uint64_t c1 = BIG_CONSTANT(0x87c37b91114253d5);\n  const uint64_t c2 = BIG_CONSTANT(0x4cf5ad432745937f);\n\n  //----------\n  // body\n\n  const uint64_t * blocks = (const uint64_t *)(data);\n\n  for(int i = 0; i < nblocks; i++)\n  {\n    uint64_t k1 = getblock(blocks,i*2+0);\n    uint64_t k2 = getblock(blocks,i*2+1);\n\n    k1 *= c1; k1  = ROTL64(k1,31); k1 *= c2; h1 ^= k1;\n\n    h1 = ROTL64(h1,27); h1 += h2; h1 = h1*5+0x52dce729;\n\n    k2 *= c2; k2  = ROTL64(k2,33); k2 *= c1; h2 ^= k2;\n\n    h2 = ROTL64(h2,31); h2 += h1; h2 = h2*5+0x38495ab5;\n  }\n\n  //----------\n  // tail\n\n  const uint8_t * tail = (const uint8_t*)(data + nblocks*16);\n\n  uint64_t k1 = 0;\n  uint64_t k2 = 0;\n\n  switch(len & 15)\n  {\n  case 15: k2 ^= uint64_t(tail[14]) << 48;\n  case 14: k2 ^= uint64_t(tail[13]) << 40;\n  case 13: k2 ^= uint64_t(tail[12]) << 32;\n  case 12: k2 ^= uint64_t(tail[11]) << 24;\n  case 11: k2 ^= uint64_t(tail[10]) << 16;\n  case 10: k2 ^= uint64_t(tail[ 9]) << 8;\n  case  9: k2 ^= uint64_t(tail[ 8]) << 0;\n           k2 *= c2; k2  = ROTL64(k2,33); k2 *= c1; h2 ^= k2;\n\n  case  8: k1 ^= uint64_t(tail[ 7]) << 56;\n  case  7: k1 ^= uint64_t(tail[ 6]) << 48;\n  case  6: k1 ^= uint64_t(tail[ 5]) << 40;\n  case  5: k1 ^= uint64_t(tail[ 4]) << 32;\n  case  4: k1 ^= uint64_t(tail[ 3]) << 24;\n  case  3: k1 ^= uint64_t(tail[ 2]) << 16;\n  case  2: k1 ^= uint64_t(tail[ 1]) << 8;\n  case  1: k1 ^= uint64_t(tail[ 0]) << 0;\n           k1 *= c1; k1  = ROTL64(k1,31); k1 *= c2; h1 ^= k1;\n  };\n\n  //----------\n  // finalization\n\n  h1 ^= len; h2 ^= len;\n\n  h1 += h2;\n  h2 += h1;\n\n  h1 = fmix(h1);\n  h2 = fmix(h2);\n\n  h1 += h2;\n  h2 += h1;\n\n  ((uint64_t*)out)[0] = h1;\n  ((uint64_t*)out)[1] = h2;\n}\n\n//-----------------------------------------------------------------------------\n\n"
  },
  {
    "path": "toolkits/extensions/MurmurHash3.h",
    "content": "//-----------------------------------------------------------------------------\n// MurmurHash3 was written by Austin Appleby, and is placed in the public\n// domain. The author hereby disclaims copyright to this source code.\n\n#ifndef _MURMURHASH3_H_\n#define _MURMURHASH3_H_\n\n//-----------------------------------------------------------------------------\n// Platform-specific functions and macros\n\n// Microsoft Visual Studio\n\n#if defined(_MSC_VER)\n\ntypedef unsigned char uint8_t;\ntypedef unsigned long uint32_t;\ntypedef unsigned __int64 uint64_t;\n\n// Other compilers\n\n#else\t// defined(_MSC_VER)\n\n#include \n\n#endif // !defined(_MSC_VER)\n\n//-----------------------------------------------------------------------------\n\nvoid MurmurHash3_x86_32  ( const void * key, int len, uint32_t seed, void * out );\n\nvoid MurmurHash3_x86_128 ( const void * key, int len, uint32_t seed, void * out );\n\nvoid MurmurHash3_x64_128 ( const void * key, int len, uint32_t seed, void * out );\n\n//-----------------------------------------------------------------------------\n\n#endif // _MURMURHASH3_H_\n"
  },
  {
    "path": "toolkits/extensions/example.txt",
    "content": "void pagerank(extension_graph& graph) {\n  graph.transform_field(\"pr\", [](var v){ return 0.15; });    \n  graph.GAS(\n      [](const vars&) { return graphlab::IN_EDGES; },             // gather_edges\n      [](const vars&, vars&, const vars& other, edge_direction) { // gather\n          return vget(other.field(\"pr\")) / \n                vget(other.field(\"out_degree\")) ;\n      }, \n      [](var& a, const var& b) {                                  // combine\n          vget(a) += vget(b); \n      }, \n      [](vars& v, const var& result) -> bool {                    // apply\n          double pr = 0.15 + 0.85 * vget(result); \n          v.field(\"change\") = \n              std::fabs(pr - vget(v.field(\"pr\"))) / \n              vget(v.field(\"out_degree\"));\n          v.field(\"pr\") = pr;         \n          return false; \n      }, \n      [](const vars& v) {                                        // scatter_edges\n          return vget(v.field(\"change\")) > 0.01 ? \n                                graphlab::OUT_EDGES : graphlab::NO_EDGES; \n      },\n      [](const vars&, const vars&, const vars&, edge_direction) {// scatter \n          return true; \n      }\n  ); // scatter\n}\n\n\n"
  },
  {
    "path": "toolkits/extensions/extension.cpp",
    "content": "#include \n#include \n#include \n#include \n#include \"extension_gas.hpp\"\n\nnamespace graphlab {\nnamespace extension {\n\nvar& operator+=(var& value, const var& other) {\n  const double* other_double = boost::get(&other);\n  const std::string* other_string = boost::get(&other);\n  const Eigen::VectorXd* other_vector = boost::get(&other);\n  const Eigen::MatrixXd* other_matrix = boost::get(&other);\n\n  if ( double* val = boost::get( &value) ) {\n    if (other_double!= NULL) value = (double)(*val) + (*other_double);\n    else ASSERT_MSG(false, \"Type mismatch in operator+=\");\n  } else if ( std::string* val = boost::get( &value) ) {\n    if (other_string != NULL) (*val) += (*other_string);\n    else ASSERT_MSG(false, \"Type mismatch in operator+=\");\n  } else if ( Eigen::VectorXd* val = boost::get( &value) ) {\n    if (other_vector!= NULL) (*val) += (*other_vector);\n    else ASSERT_MSG(false, \"Type mismatch in operator+=\");\n  } else if ( Eigen::MatrixXd* val = boost::get( &value) ) {\n    if (other_matrix!= NULL) (*val) += (*other_matrix);\n    else ASSERT_MSG(false, \"Type mismatch in operator+=\");\n  }\n  return value;\n}\n\n\n\n\n\n\n\n\n\n// lets get more than we will ever need so it will never need to resize\nstd::vector descriptor_set(65536);\nlockfree_push_back > \n          descriptor_access(descriptor_set, 0);\n \n\n\nvar vars::empty_var;\n}\n}\n"
  },
  {
    "path": "toolkits/extensions/extension_data.hpp",
    "content": "#ifndef GRAPHLAB_EXTENSION_DATA_HPP\n#define GRAPHLAB_EXTENSION_DATA_HPP\n\n#include \n#include \n#include \n#include \n#include \n#include \"../collaborative_filtering/eigen_serialization.hpp\"\n#include \"MurmurHash3.h\"\n\n// Here I define the basic \"var\" variant type.\n// which is basically a boost::variant around a double, string, vector and matrix\n\nnamespace graphlab {\nnamespace extension {\n\n// the key here is that we are going to lock down the \n// type system for GraphLab to a wrapped boost::variant\ntypedef boost::variant var;\n\nextern var& operator+=(var& value, const var& other);\n   \n\n/**\n * Gets a typechecked value from the variant var\n */\ntemplate \ninline const T& get(const var& v) {\n  const T* val = boost::get(&v);\n  if (val == NULL) {\n    logger_once(LOG_ERROR, \"Reading invalid type from var\");\n    static T t;\n    return t;\n  }\n  return *val;\n}\n\n/**\n * Gets a typechecked value from the variant var\n */\ntemplate \ninline T& get(var& v) {\n  T* val = boost::get(&v);\n  if (val == NULL) {\n    logger_once(LOG_ERROR, \"Reading invalid type from var\");\n    static T t;\n    return t;\n  }\n  return *val;\n}\n\n\n\n//////////////////////////////////////////////////////////////\n// Here we briefly escape out tp the global namespace       //\n// to define serializers and deserializers for the var      //\n//////////////////////////////////////////////////////////////\n}}\n\nBEGIN_OUT_OF_PLACE_SAVE(oarc, graphlab::extension::var, value) {\n  if ( const double* val = boost::get( &value) ) {\n    oarc << char(1) << (*val);\n  } else if ( const std::string* val = boost::get( &value) ) {\n    oarc << char(2) << (*val);\n  } else if ( const Eigen::VectorXd* val = boost::get( &value) ) {\n    oarc << char(3) << (*val);\n  } else if ( const Eigen::MatrixXd* val = boost::get( &value) ) {\n    oarc << char(4) << (*val);\n  }\n} END_OUT_OF_PLACE_SAVE()\n\n\nBEGIN_OUT_OF_PLACE_LOAD(iarc, graphlab::extension::var, value) {\n  char content_type;\n  iarc >> content_type;\n  if (content_type == 1) {\n    double val; iarc >> val; value = val;\n  } else if (content_type == 2) {\n    std::string val; iarc >> val; value = val;\n  } else if (content_type == 3) {\n    Eigen::VectorXd val; iarc >> val; value = val;\n  } else if (content_type == 4) {\n    Eigen::MatrixXd val; iarc >> val; value = val;\n  }\n} END_OUT_OF_PLACE_LOAD()\nnamespace graphlab { namespace extension { \n\n//////////////////////////////////////////////////////////////\n// Returning to your regular progamming                     //\n//////////////////////////////////////////////////////////////\n\n\n\ntypedef uint32_t key_id_type;\n\n/** vars is a dynamic struct with a mapping from string->var.\n  * Internally, it is stored as key_id_type->var where the key is\n  * a hash value of the string.\n  * we assume that the murmurhash will never collide for the small\n  * namespaces considered.\n */\ninline key_id_type get_id_from_name(const char* key) {\n  uint32_t ret = 0;\n  MurmurHash3_x86_32((void*)key, strlen(key), 12345, (void*)(&ret));\n  return ret;\n}\n// overload for string\ninline key_id_type get_id_from_name(const std::string& key) {\n  uint32_t ret = 0;\n  MurmurHash3_x86_32((void*)key.c_str(), key.length(), 12345, (void*)(&ret));\n  return ret;\n}\n// overload for int\ninline key_id_type get_id_from_name(key_id_type key) {\n  return key;\n}\n\n\n\n/**\n * A dynamic struct storing mappings from string->var where var\n * is a variant.\n * fields can be accessed with operator() or \".field()\"\n */\nstruct vars {\n  std::vector > table;\n  static var empty_var;\n  simple_spinlock lock;\n  vars() { }\n  ~vars() {\n    }\n\n  void clear() {\n    lock.lock();\n    for(const std::pair& p: table) {\n      delete p.second;\n    }\n    table.clear();\n    lock.unlock();\n  }\n\n  vars& operator=(const vars& v) {\n    clear();\n    for (size_t i = 0;i < v.table.size(); ++i) {\n      field(v.table[i].first) = *(v.table[i].second);\n    }\n    return *this;\n  }\n\n  void save(oarchive& oarc) const {\n    lock.lock();\n    oarc << (size_t)table.size();\n    for(const std::pair& p: table) {\n      oarc << p.first << (*p.second);\n    }\n    lock.unlock();\n  }\n  \n  void load(iarchive& iarc) {\n    size_t tsize;\n    iarc >> tsize;\n    for (size_t i = 0;i < tsize; ++i) {\n      key_id_type key; iarc >> key;\n      iarc >> field(key);\n    }\n  }\n\n  var& operator()(const std::string& key) {\n    return field(key);\n  }\n  const var& operator()(const std::string& key) const {\n    return field(key);\n  }\n  var& operator()(const char* key) {\n    return field(key);\n  }\n  const var& operator()(const char* key) const {\n    return field(key);\n  }\n  var& operator()(key_id_type key) {\n    return field(key);\n  }\n  const var& operator()(key_id_type key) const {\n    return field(key);\n  }\n\n  var& field(const std::string& _key) {\n    key_id_type key = get_id_from_name(_key);\n    return field(key);\n  }\n\n  const var& field(const std::string& _key) const {\n    key_id_type key = get_id_from_name(_key);\n    return field(key);\n  }\n  \n  var& field(const char* _key) {\n    key_id_type key = get_id_from_name(_key);\n    return field(key);\n  }\n  const var& field(const char* _key) const {\n    key_id_type key = get_id_from_name(_key);\n    return field(key);\n  }\n   \n  var& field(key_id_type key) {\n    lock.lock();\n\n    for(std::pair& p: table) {\n      if (p.first == key) {\n        lock.unlock();\n        return *(p.second);\n      }\n    }\n    var* ret = new var;\n    // force slow resize to limit memory usage\n    // assume that field creation is not a common operation.\n    table.reserve(table.size() + 1);\n    table.push_back(std::make_pair(key, ret));\n    lock.unlock();\n    return *ret;\n  }\n\n  const var& field(key_id_type key) const {\n    for(const std::pair& p: table) {\n      if (p.first == key) {\n        return *(p.second);\n      }\n    }\n    return empty_var;\n  }\n\n}; \n\n\n\ntypedef distributed_graph internal_graph_type;\n\n} // namespace extension\n} // namespace graphlab\n\n#endif\n"
  },
  {
    "path": "toolkits/extensions/extension_gas.hpp",
    "content": "#ifndef GRAPHLAB_EXTENSION_GAS_HPP\n#define GRAPHLAB_EXTENSION_GAS_HPP\n\n#include \n#include \n#include \"extension_data.hpp\"\n#include \"extension_gas_base_types.hpp\"\n\nnamespace graphlab {\nnamespace extension{\n\n  /// A collection of all the user ops together\nstruct gas_op_descriptor{\n  gather_functor* gather_op;\n  gather_select_functor* gather_select_op;\n  combiner_functor* combiner_op;\n  apply_functor* apply_op;\n  scatter_functor* scatter_op;\n  scatter_select_functor* scatter_select_op;\n};\n// the active set of GAS sets to run\nextern std::vector descriptor_set;\nextern lockfree_push_back > descriptor_access;\ntypedef uint32_t descriptor_id_type;\n\n\n\n// A wrapper around the gather operation\nstruct gather_var {\n  var v;\n  descriptor_id_type descriptor_id;\n  combiner_functor* combiner_op;\n\n  gather_var():descriptor_id(-1), combiner_op(NULL) { }\n  var& operator+=(const var& other) {\n    (*combiner_op)(v, other);\n    return v;\n  }\n\n  var& operator+=(const gather_var& other) {\n    (*combiner_op)(v, other.v);\n    return v;\n  }\n\n\n  inline void save(graphlab::oarchive& oarc) const {\n    oarc << v << descriptor_id;\n  }\n\n  inline void load(graphlab::iarchive& iarc) {\n    iarc >> v >> descriptor_id;\n    gas_op_descriptor gas;\n    bool ret = descriptor_access.query_unsafe(descriptor_id, gas);\n    if (ret) combiner_op = gas.combiner_op;\n  }\n};\n\n\n\n// a wrapper around the message\nstruct message_type: public graphlab::IS_POD_TYPE {\n  descriptor_id_type descriptor;\n  message_type(): descriptor(-1){ }\n  message_type(descriptor_id_type d): descriptor(d) { };\n  message_type& operator+=(const message_type& other) {\n    return *this;\n  }\n};\n\nstruct extension_update_functor: \n    public graphlab::ivertex_program {\n                                                                                  \npublic:\n    typedef graphlab::ivertex_program parent_type;\n\n    descriptor_id_type descriptor_id;\n//    gas_op_descriptor gas;\n\n    extension_update_functor():descriptor_id(-1) {\n    }\n    \n    extension_update_functor(size_t id):descriptor_id(id) { }\n\n    inline void init(icontext_type& context,\n                     const vertex_type& vertex, \n                     const message_type& msg) { \n      descriptor_id = msg.descriptor;\n    }\n\n\n    edge_dir_type gather_edges(icontext_type& context,\n                               const vertex_type& vertex) const {\n      gas_op_descriptor* gas = \n          descriptor_access.query_unsafe(descriptor_id);\n      ASSERT_TRUE(gas != NULL);\n      if (gas->gather_select_op) {\n        return (*gas->gather_select_op)(vertex.data());\n      } else {\n        return ALL_EDGES;\n      }\n    }\n\n    inline gather_var gather(icontext_type& context, \n                      const vertex_type& vertex,\n                      edge_type& edge) const {\n      gas_op_descriptor* gas = \n          descriptor_access.query_unsafe(descriptor_id);\n      ASSERT_TRUE(gas != NULL);\n      vertex_type other_vertex = edge.source().id() == vertex.id() ? \n                                    edge.target() : edge.source();\n      gather_var ret;\n      ret.v  = (*gas->gather_op)(vertex.data(), \n                                edge.data(), \n                                other_vertex.data(),\n                                edge.source().id() == vertex.id() ? \n                                      OUT_EDGE : IN_EDGE);\n      ret.descriptor_id = descriptor_id;\n      ret.combiner_op = gas->combiner_op;\n      return ret;\n    }\n\n    inline void apply(icontext_type& context, vertex_type& vertex,\n                      const gather_type& total) {\n      gas_op_descriptor* gas = \n          descriptor_access.query_unsafe(descriptor_id);\n      ASSERT_TRUE(gas != NULL);\n      bool sched = (*gas->apply_op)(vertex.data(), total.v);\n      if (sched) context.signal(vertex, descriptor_id);\n    }\n\n    edge_dir_type scatter_edges(icontext_type& context,\n                               const vertex_type& vertex) const {\n      gas_op_descriptor* gas = \n          descriptor_access.query_unsafe(descriptor_id);\n      ASSERT_TRUE(gas != NULL);\n      if (gas->scatter_select_op) {\n        return (*gas->scatter_select_op)(vertex.data());\n      } else {\n        return ALL_EDGES;\n      }\n    }\n\n    inline void scatter(icontext_type& context, const vertex_type& vertex,\n                 edge_type& edge) const {\n      gas_op_descriptor* gas = \n          descriptor_access.query_unsafe(descriptor_id);\n      ASSERT_TRUE(gas != NULL);\n      vertex_type other_vertex = edge.source().id() == vertex.id() ? \n          edge.target() : edge.source();\n\n      bool ret = (*gas->scatter_op)(vertex.data(), \n                                   edge.data(), \n                                   other_vertex.data(),\n                                   edge.source().id() == vertex.id() ? \n                                   OUT_EDGE : IN_EDGE);\n      if (ret) {\n        context.signal(other_vertex, descriptor_id);\n      }\n    }\n\n    inline void save(graphlab::oarchive& oarc) const {\n      oarc << descriptor_id;\n    }\n\n    inline void load(graphlab::iarchive& iarc) {\n      iarc >> descriptor_id;\n    }\n};\n\n\n\n\n\n} // namespace extension\n} // namespace graphlab\n\n#endif\n"
  },
  {
    "path": "toolkits/extensions/extension_gas_base_types.hpp",
    "content": "#ifndef GRAPHLAB_EXTENSION_GAS_BASE_TYPES_HPP\n#define GRAPHLAB_EXTENSION_GAS_BASE_TYPES_HPP\n#include \"extension_data.hpp\"\n\nnamespace graphlab {\nnamespace extension {\n// edge direction enum\nenum edge_direction {\n  IN_EDGE, OUT_EDGE\n};\n\n\n//////////////////////////////////////////////////////////////\n// Here we will produce base types of all the functors      //\n//////////////////////////////////////////////////////////////\n\n// the combiner base type\nstruct combiner_functor {\n  virtual void operator()(var& , const var&) = 0;\n};\n\n\n// the gather base type\nstruct gather_functor {\n  virtual var operator()(const vars& center, \n                         vars& edge, \n                         const vars& other,\n                         edge_direction direction) = 0;\n};\n\n//gather select base type\nstruct gather_select_functor {\n  virtual edge_dir_type operator()(const vars& center) {\n    return ALL_EDGES;\n  }\n};\n\n// the apply base type\nstruct apply_functor {\n  // return true to schedule self\n  virtual bool operator()(vars& center, \n                          const var& gather_result) = 0;\n};\n\n// scatter select base type\nstruct scatter_select_functor {\n  virtual edge_dir_type operator()(const vars& center) {\n    return ALL_EDGES;\n  }\n};\n\n\n// the scatter base type\nstruct scatter_functor {\n  // return true to schedule other\n  virtual bool operator()(const vars& center, \n                          vars& edge, \n                          const vars& other,\n                          edge_direction direction) = 0;\n};\n\n\n\n} // extension\n} // graphlab \n#endif \n"
  },
  {
    "path": "toolkits/extensions/extension_gas_lambda_wrapper.hpp",
    "content": "#ifndef GRAPHLAB_EXTENSION_GAS_LAMBDA_WRAPPER\n#define GRAPHLAB_EXTENSION_GAS_LAMBDA_WRAPPER\n\n\n#include \"extension_gas.hpp\"\n/*\n  Implements a collection of lightweight generic wrappers around the GAS, \n  transform and map operations.\n  These wrappers do nothing but store a functor which is compatible with the \n  base types in extension_gas_base_types.hpp, and call them.\n*/\n\nnamespace graphlab {\nnamespace extension {\n\n\ntemplate \nstruct transform_field_wrapper {\n  Functor* f;\n  key_id_type field;\n  typedef void result_type;\n  \n  transform_field_wrapper(Functor* f, FieldType field):\n        f(f),field(get_id_from_name(field)) { }\n\n  void operator()(internal_graph_type::vertex_type& vtx) {\n    var& v = vtx.data()(field);\n    v = (*f)(v);\n  }\n};\n\n\ntemplate \nstruct map_field_wrapper {\n  Functor* f;\n  key_id_type field;\n  typedef var result_type;\n\n  map_field_wrapper(Functor* f, FieldType field):\n        f(f),field(get_id_from_name(field)) { }\n\n  var operator()(internal_graph_type::vertex_type& vtx) {\n    var& v = vtx.data()(field);\n    return (*f)(v);\n  }\n};\n\n\n// the combiner base type\ntemplate \nstruct generic_combiner: public combiner_functor {\n  CombinerType* ct;\n  void operator()(var& a, const var& b) {\n    (*ct)(a, b);\n  }\n};\n\ntemplate \nstruct generic_gather_select : public gather_select_functor {\n  GatherSelectType* gt;\n  edge_dir_type operator()(const vars& center) {\n    return (*gt)(center);\n  }\n};\n\n\ntemplate \nstruct generic_gather : public gather_functor {\n  GatherType* gt;\n  var operator()(const vars& center, \n                 vars& edge, \n                 const vars& other,\n                 edge_direction direction) {\n    return (*gt)(center, edge, other, direction);\n  }\n};\n\ntemplate \nstruct generic_apply : public apply_functor {\n  ApplyType* at;\n  bool operator()(vars& center, \n                  const var& gather_result) {\n    return (*at)(center, gather_result);\n  }\n};\n\ntemplate \nstruct generic_scatter_select : public scatter_select_functor {\n  ScatterSelectType* st;\n  edge_dir_type operator()(const vars& center) {\n    return (*st)(center);\n  }\n};\n\n\ntemplate \nstruct generic_scatter : public scatter_functor {\n  ScatterType* st;\n  bool operator()(const vars& center, \n                 vars& edge, \n                 const vars& other,\n                 edge_direction direction) {\n    return (*st)(center, edge, other, direction);\n  }\n};\n\n\n} // extension\n} // graphlab\n#endif\n"
  },
  {
    "path": "toolkits/extensions/extension_graph.cpp",
    "content": "#include \n#include \"extension_graph.hpp\"\n\nnamespace graphlab {\nnamespace extension {\n\n\n\nvoid extension_graph::synchronous_dispatch_new_engine(size_t desc_id) {\n  synchronous_engine sync_engine(rmi.dc(), \n                                                           internal_graph,\n                                                           __glopts);\n  sync_engine.signal_all(desc_id);\n  sync_engine.start();\n}\n\n\n} // extension\n} // graphlab\n"
  },
  {
    "path": "toolkits/extensions/extension_graph.hpp",
    "content": "#ifndef GRAPHLAB_EXTENSION_GRAPH_HPP\n#define GRAPHLAB_EXTENSION_GRAPH_HPP\n\n#include \"extension_data.hpp\"\n#include \"extension_gas.hpp\"\n#include \"extension_gas_lambda_wrapper.hpp\"\n#include \"extension_main.hpp\"\nnamespace graphlab {\nnamespace dc_impl {\nextern distributed_control* get_last_dc();\n}\nnamespace extension {\n\nstruct extension_graph_writer{\n  std::string field;\n  extension_graph_writer(std::string field):field(field) { }\n  std::string save_vertex(internal_graph_type::vertex_type v) {\n    std::stringstream strm;\n    strm << v.id() << \"\\t\" << v.data()(field) << \"\\n\";\n    return strm.str();\n  }\n\n  std::string save_edge(internal_graph_type::edge_type e) {\n    std::stringstream strm;\n    strm << e.source().id() << \"\\t\" << e.target().id() << e.data()(field) << \"\\n\";\n    return strm.str();\n  }\n}; \n\n\nclass extension_graph {\n public:\n  dc_dist_object rmi;\n  internal_graph_type internal_graph;\n  mutex lock;\n  bool finalized;\n\n  extension_graph()\n     :rmi(*dc_impl::get_last_dc(), this), \n     internal_graph(*dc_impl::get_last_dc(), __glopts),finalized(false) { }\n\n\n  extension_graph(distributed_control& dc, \n                  const graphlab_options& opts = graphlab_options() ) \n     :rmi(dc, this), internal_graph(dc, opts),finalized(false) { }\n\n  template \n  void transform_field(FieldType field,\n                          TransformType transform_functor) {\n    finalize();\n    lock.lock();\n    transform_field_wrapper fw(&transform_functor, field);\n    internal_graph.transform_vertices(fw);\n    lock.unlock();\n  }\n\n/*\n  template \n  var map_reduce_field(FieldType field,\n                       MapFunctor map_functor) {\n    finalize();\n    lock.lock();\n    map_field_wrapper mw(&map_functor, field);\n    var ret = internal_graph.map_reduce_vertices(mw);\n    lock.unlock();\n    return ret;\n  }\n*/\n\n  void load_structure(std::string prefix, std::string format) {\n    lock.lock();\n    internal_graph.load_format(prefix, format);\n    lock.unlock();\n  }\n\n  void save_vertices(std::string prefix, std::string field) {\n    internal_graph.save(prefix, extension_graph_writer(field),\n                        false,    // do not gzip\n                        true,     // save vertices\n                        false);   // do not save edges\n  }\n\n  internal_graph_type& graph() {\n    return internal_graph;\n  }\n\n  void finalize() {\n    lock.lock();\n    if (!finalized) {\n      internal_graph.finalize();\n      internal_graph.transform_vertices([] (internal_graph_type::vertex_type& v) {\n                                        v.data().field(\"in_degree\") = (double)v.num_in_edges();\n                                        v.data().field(\"out_degree\") = (double)v.num_out_edges();\n                                        });\n    }\n    lock.unlock();\n  }\n\n  void synchronous_dispatch_new_engine(size_t desc_id);\n\n  /// GAS which defaults to all out and all in edges\n  template \n    void GAS(GatherType gather,\n             CombinerType combiner,\n             ApplyType apply,\n             ScatterType scatter,\n             size_t iterations = 0) {\n      finalize();\n      generic_gather g;\n      g.gt = &gather;\n\n      generic_combiner c;\n      c.ct = &combiner;\n\n      generic_apply a;\n      a.at = &apply;\n\n      generic_scatter s;\n      s.st = &scatter;\n\n      gas_op_descriptor gd;\n      gd.gather_select_op = NULL;\n      gd.gather_op = &g;\n      gd.combiner_op = &c;\n      gd.apply_op = &a;\n      gd.scatter_select_op = NULL;\n      gd.scatter_op = &s;\n\n      lock.lock();\n      descriptor_id_type descid = descriptor_access.push_back(gd);\n      lock.unlock();\n\n      synchronous_dispatch_new_engine(descid);\n    }\n\n  /// Regular GAS\n  template \n    void GAS(GatherSelectType gatherselect,\n             GatherType gather,\n             CombinerType combiner,\n             ApplyType apply,\n             ScatterSelectType scatterselect,\n             ScatterType scatter,\n             size_t iterations = 0) {\n      finalize();\n      generic_gather_select gs;\n      gs.gt = &gatherselect;\n\n      generic_gather g;\n      g.gt = &gather;\n\n      generic_combiner c;\n      c.ct = &combiner;\n\n      generic_apply a;\n      a.at = &apply;\n\n      generic_scatter_select ss;\n      ss.st = &scatterselect;\n\n      generic_scatter s;\n      s.st = &scatter;\n\n      gas_op_descriptor gd;\n      gd.gather_select_op = &gs;\n      gd.gather_op = &g;\n      gd.combiner_op = &c;\n      gd.apply_op = &a;\n      gd.scatter_select_op = &ss;\n      gd.scatter_op = &s;\n\n      lock.lock();\n      descriptor_id_type descid = descriptor_access.push_back(gd);\n      lock.unlock();\n\n      synchronous_dispatch_new_engine(descid);\n    }\n};\n\n\n\n\n} // namespace extension\n} // namespace graphlab\n\n#endif\n"
  },
  {
    "path": "toolkits/extensions/extension_main.cpp",
    "content": "#include \n#include \n#include \nint __real_main(int argc, char** argv);\ngraphlab::command_line_options __glopts(\"\");\n\n\nint actual_main(int argc, char** argv) {\n  graphlab::mpi_tools::init(argc, argv);\n  if (!__glopts.parse(argc, argv, true)) return false;\n  // rebuild argc argv with unrecognized options\n  std::vector vs = __glopts.unrecognized();\n  char** newargv = new char*[vs.size() + 1];\n  newargv[0] = argv[0];\n  for (size_t i = 0;i < vs.size(); ++ i) {\n    newargv[i + 1] = (char*)(vs[i].c_str());\n  }\n  int ret = __real_main(vs.size() + 1, newargv);\n  if (graphlab::dc_impl::get_last_dc()) delete graphlab::dc_impl::get_last_dc();\n  graphlab::mpi_tools::finalize();\n  return ret;\n}\n\n#if 1\n// don't seem to be able to get -wrap main working correctly\nint main(int argc, char** argv) {\n  return actual_main(argc, argv);\n}\n#else\nint __wrap_main(int argc, char** argv) {\n  return actual_main(argc, argv);\n}\n\n#endif\n \n"
  },
  {
    "path": "toolkits/extensions/extension_main.hpp",
    "content": "#ifndef GRAPHLAB_EXTENSION_MAIN_HPP\n#define GRAPHLAB_EXTENSION_MAIN_HPP\n#include \nextern graphlab::command_line_options __glopts;\n#endif\n"
  },
  {
    "path": "toolkits/extensions/extension_pagerank.cpp",
    "content": "#include \"extensions.hpp\"\n\nnamespace graphlab {\nnamespace extension {\n\nvoid pagerank(extension_graph& graph, \n              const std::string PR_FIELD_NAME,\n              double tolerance) {\n  const std::string PR_CHANGE_NAME = PR_FIELD_NAME + \"_change\";\n\n  key_id_type PR_FIELD = get_id_from_name(PR_FIELD_NAME);\n  key_id_type PR_CHANGE = get_id_from_name(PR_CHANGE_NAME);\n  key_id_type OUT_DEG = get_id_from_name(\"out_degree\");\n\n  graph.transform_field(PR_FIELD, [](var v){ return 0.15; });    \n  timer ti;\n  graph.GAS(\n      [](const vars&) { return graphlab::IN_EDGES; },             // gather_edges\n      [=](const vars&, vars&, const vars& other, edge_direction) { // gather\n          return get(other(PR_FIELD)) / \n                get(other(OUT_DEG)) ;\n      }, \n      [](var& a, const var& b) {                                  // combine\n          get(a) += get(b); \n      }, \n      [=](vars& v, const var& result) -> bool {                    // apply\n          double pr = 0.15 + 0.85 * get(result); \n          v(PR_CHANGE) = \n              std::fabs(pr - get(v(PR_FIELD))) / \n              get(v(OUT_DEG));\n          v(PR_FIELD) = pr;         \n          return false; \n      }, \n      [=](const vars& v) {                                        // scatter_edges\n          return get(v.field(PR_CHANGE)) > tolerance ? \n                                graphlab::OUT_EDGES : graphlab::NO_EDGES; \n      },\n      [](const vars&, const vars&, const vars&, edge_direction) {// scatter \n          return true; \n      }\n  ); // scatter\n  std::cout << \"PageRank complete in \" << ti.current_time() << \"s\" << std::endl;\n}\n\n\n} // namespace extension\n} // namespace graphlab\n\n"
  },
  {
    "path": "toolkits/extensions/extensions.hpp",
    "content": "#include \"extension_graph.hpp\"\n\n\n#ifndef GRAPHLAB_EXTENSIONS_HPP\n#define GRAPHLAB_EXTENSIONS_HPP\n#if 1\n// we have to use this unreliable hack\n// don't seem to be able to get -wrap main\n// working. TOFIX\n#define main __real_main\n#endif\n\n\nnamespace graphlab {\nnamespace extension {\n\n// prototype for all implemented extensions\nvoid pagerank(extension_graph& graph, \n              const std::string PR_FIELD,\n              double tolerance);\n\n} // namespace extension\n} // namespace graphlab\n\n\n\n\n#endif \n"
  },
  {
    "path": "toolkits/extensions/pagerank_extension_driver.cpp",
    "content": "#include \"extensions.hpp\"\n#include \nusing namespace graphlab::extension;\n\nint main(int argc, char** argv) {\n  extension_graph graph;\n  if (argc < 2) {\n    std::cout << argv[0] << \" [input prefix] optional:[output prefix]\\n\"; \n    return 0;\n  }\n  graph.load_structure(argv[1], \"snap\");\n  pagerank(graph, \"pr\", 0.01);\n  if (argc > 2) {\n    graph.save_vertices(argv[2], \"pr\");\n  }\n}\n"
  },
  {
    "path": "toolkits/graph_algorithms/CMakeLists.txt",
    "content": "project(Djikstra)\nadd_graphlab_executable(djikstra djikstra.cpp)\nadd_graphlab_executable(prestige prestige.cpp)\nadd_graphlab_executable(betweeness betweeness.cpp)\nadd_graphlab_executable(closeness closeness.cpp)\n"
  },
  {
    "path": "toolkits/graph_algorithms/betweeness.cpp",
    "content": "/*\n * Copyright (c) 2014 Daniel McEnnis.\n * portions of main Copyright (c) 2009 Carnegie Mellon\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n#include \n#include \n#include \n\n/*\n * Djikstra Graph Node Class\n *\n * This class contains the information about a single graphlab node.\n * id - current best path's previous node id - next node on path to root\n * cost - current cost of the path to route by the current route: Note - this\n *       can become inaccurate in the course of calculations and must be recalculated\n *       by traversing the shortest path tree to get an accurate result.\n * launched - has execution of this node been sheduled\n * done - has execution of this node been completed\n */\nclass DjikstraNode {\npublic:\n    long id;\n    double cost;\n    bool launched;\n    bool done;\n\n    DjikstraNode(){\n        id = 0;\n        cost = 1e100;\n        launched = false;\n        done=false;\n    }\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << id << cost << launched << done;\n  }\n\n  void load(graphlab::iarchive& iarc) {\n    iarc >> id >> cost >> launched >> done;\n  }\n};\n\n/*\n * PrestigeAnalysisNode\n * Graph Node class for running multiple djikstra tree algorithms simultaneously\n * Contains a map of node id's to DjikstraNode instances\n * bookkeeping components\n *\n */\nclass PrestigeAnalysisNode {\npublic:\n    std::map djikstra_pieces;\n    double local_value;\n    double total;\n    long count;\n    int edge_count;\n\n    PrestigeAnalysisNode(){\n        local_value=0.0;\n        total=0.0;\n        count=0;\n        edge_count=-1;\n    }\n\n    void save(graphlab::oarchive& oarc) const {\n        oarc << djikstra_pieces << local_value << total << count << edge_count;\n    }\n\n    void load(graphlab::iarchive& iarc) {\n        iarc >> djikstra_pieces >> local_value >> total >> count >> edge_count;\n    }\n};\n\n/*\n * Gather class for the Djikstra algorithm.\n * id: node id of the incoming edge's other end\n * cost: shortest path cost at the time this node gathers its edges\n * edge_count: a count of gathered edges\n *\n */\nclass Gather {\npublic:\n    unsigned long id;\n    double cost;\n    int edge_count;\n\n    Gather(){\n        id=0;\n        cost=0.0;\n        edge_count=1;\n    }\n\n    Gather& operator+=(const Gather& other){\n        if(other.id < 0){\n            return *this;\n        }\n        if(this->id < 0){\n            return *this;\n        }\n        if (cost <= other.cost){\n            this->edge_count++;\n            return *this;\n        }\n        this->edge_count += other.edge_count;\n        return *this;\n    }\n\n\n    void save(graphlab::oarchive& oarc) const {\n       oarc << id << cost << edge_count;\n     }\n\n     void load(graphlab::iarchive& iarc) {\n       iarc >> id >> cost >> edge_count;\n     }\n\n};\n\n/*\n * GatherMultiTree\n * map of djisktra root id's to their asociated content for that tree\n *\n */\nclass GatherMultiTree {\npublic:\n    std::map content;\n    int edge_count;\n\n    GatherMultiTree(){\n        edge_count=0;\n    }\n\n    GatherMultiTree& operator+=(const GatherMultiTree& other){\n        return *this;\n    }\n\n    void save(graphlab::oarchive& oarc) const {\n        oarc << content << edge_count;\n    }\n\n    void load(graphlab::iarchive& iarc) {\n        iarc >> content >> edge_count;\n    }\n};\n\n\ntypedef PrestigeAnalysisNode vertex_data_type;\n\ntypedef GatherMultiTree gather_type;\n\n// The graph type is determined by the vertex and edge data types\ntypedef graphlab::distributed_graph graph_type;\n\n/*\n * Loads graphs in the form 'id (id edge_strength)*'\n *\n */\nbool line_parser(graph_type& graph, const std::string& filename, const std::string& textline) {\n  std::stringstream strm(textline);\n  graphlab::vertex_id_type vid;\n  // first entry in the line is a vertex ID\n  strm >> vid;\n  PrestigeAnalysisNode node;\n  // insert this vertex with its label\n  graph.add_vertex(vid, node);\n  // while there are elements in the line, continue to read until we fail\n  double edge_val=1.0;\n  while(1){\n    graphlab::vertex_id_type other_vid;\n    strm >> other_vid;\n    strm >> edge_val;\n    if (strm.fail())\n      break;\n    graph.add_edge(vid, other_vid,edge_val);\n  }\n\n  return true;\n}\n\n/*\n * Algorithm class whose sole purpose is to reset launched and done booleans\n * for all id's in a PrestigeAnalysisNode\n */\nclass ClearBooleans :\n        public graphlab::ivertex_program,\n        public graphlab::IS_POD_TYPE {\npublic:\n  edge_dir_type gather_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n\n  gather_type gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n        GatherMultiTree g;\n        return g;\n  }\n\n  void apply(icontext_type& context, vertex_type& vertex, const gather_type& total) {\n      for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n          iter != vertex.data().djikstra_pieces.end(); ++iter){\n          long key = iter->first;\n          vertex.data().djikstra_pieces[key].launched = false;\n          vertex.data().djikstra_pieces[key].done = false;\n          vertex.data().djikstra_pieces[key].cost = 0.0;\n      }\n  }\n\n  edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n          return graphlab::NO_EDGES;\n  }\n\n  void scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n  }\n};\n\n/*\n * Djikstra Algorithm Class\n *\n * Starting from the starting nodes, create an id for this root and signal\n * all neighbors to start the calculations. Set launched when started, done\n * when all edges have been signaled.\n *\n * As a signal is receieved collect edges to determine if the best path has\n * changed.  If it has, update. If the first signal is receieved, marked\n * the node as launched and then mark the node done after signaling neighbors.\n *\n * The process terminates when all nodes active have no neighbors that are not done.\n */\nclass DjikstraAlgorithm :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\n    bool changed;\n\n  public:\n    edge_dir_type gather_edges(icontext_type& context, const vertex_type& vertex) const {\n      return graphlab::IN_EDGES;\n    }\n\n    gather_type gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n    Gather g;\n    GatherMultiTree tree;\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key=iter->first;\n            if((edge.source().data().djikstra_pieces[key].launched == true)&&\n                    (edge.source().data().djikstra_pieces[key].done == false)){\n                double c = edge.data() + edge.source().data().djikstra_pieces[key].cost;\n                g.cost = c;\n                g.id = edge.source().data().djikstra_pieces[key].id;\n                g.edge_count = 1;\n                tree.content[key] = g;\n            }else{\n                g.id=0;\n            }\n        }\n    return tree;\n    }\n\n    void apply(icontext_type& context, vertex_type& vertex, const gather_type& total) {\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key = iter->first;\n            if(vertex.data().djikstra_pieces[key].launched == false){\n                vertex.data().djikstra_pieces[key].launched = true;\n                vertex.data().edge_count = total.edge_count;\n                    if(vertex.data().djikstra_pieces[key].cost > total.content.find(key)->second.cost){\n                    vertex.data().djikstra_pieces[key].cost = total.content.find(key)->second.cost;\n                    vertex.data().djikstra_pieces[key].id = total.content.find(key)->second.id;\n                    }else{\n                    vertex.data().djikstra_pieces[key].done = true;\n                    }\n            }else{\n                vertex.data().djikstra_pieces[key].done = true;\n            }\n        }\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key = iter->first;\n            if(vertex.data().djikstra_pieces.find(key)==vertex.data().djikstra_pieces.end()){\n                vertex.data().djikstra_pieces[key].launched = true;\n                vertex.data().edge_count = total.edge_count;\n                vertex.data().djikstra_pieces[key].cost = total.content.find(key)->second.cost;\n                vertex.data().djikstra_pieces[key].id = total.content.find(key)->second.id;\n            }\n        }\n    }\n\n    edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n      // if vertex data changes, scatter to all edges.\n        bool done = true;\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key = iter->first;\n            if(vertex.data().djikstra_pieces.find(key)->second.launched &&\n                    !vertex.data().djikstra_pieces.find(key)->second.done){\n                done = false;\n            }\n        }\n        if(!done){\n            return graphlab::OUT_EDGES;\n        }else{\n            return graphlab::NO_EDGES;\n        }\n    }\n\n    void scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n    for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n        iter != vertex.data().djikstra_pieces.end(); ++iter){\n        long key = iter->first;\n        if((vertex.data().djikstra_pieces.find(key)->second.done == false) &&\n                (vertex.data().djikstra_pieces.find(key)->second.launched == true)){\n                context.signal(edge.target());\n            }\n    }\n  }\n};\n\nsize_t num_vertices = 3000;\nsize_t desired_vertices_count = 3000;\nsize_t selected_vertices_count = 0;\n\n/*\n * For every node, print the previous node in its spanning tree for all spanning trees this node is in.\n *\n */\nstruct betweeness_writer {\n  std::string save_vertex(graph_type::vertex_type v) {\n    std::stringstream strm;\n    strm << v.id();\n    double betweeness = 0.0;\n    for(std::map::const_iterator iter = v.data().djikstra_pieces.begin();\n        iter != v.data().djikstra_pieces.end(); ++iter){\n        betweeness += iter->second.cost;\n    }\n    betweeness /= selected_vertices_count;\n    strm << \"\\t\" << betweeness << std::endl;\n    return strm.str();\n  }\n  std::string save_edge (graph_type::edge_type e) { return \"\"; }\n};\n\n/*\n * Select ~3000 root nodes or an exact count which gives up around +/-3% accuracy\n * in prestige measures. It is a constant memory random selector.\n */\nbool selectVertices(const graph_type::vertex_type& vertex){\n    unsigned int r = random();\n    std::cout << \"Random seed is \" << r << std::endl;\n    if(r < (desired_vertices_count * RAND_MAX / num_vertices)){\n          selected_vertices_count++;\n          return true;\n    }\n    return false;\n}\n\n\n/*\n * Gather object that keeps track of betweeness counts for each spanning tree.\n *\n */\nclass BetweenessGather{\npublic:\n    std::map counts;\n    std::map edge_count;\n\t\n void save(graphlab::oarchive& oarc) const {\n    oarc << counts << edge_count;\n  }\n\n  void load(graphlab::iarchive& iarc) {\n    iarc >> counts >> edge_count;\n  }\n\n  BetweenessGather& operator+=(const BetweenessGather& gather){\n    for(std::map::const_iterator iter = this->counts.begin();\n        iter != this->counts.end(); ++iter ){\n                long key = iter->first;\n                this->counts[key] += gather.counts.find(key)->second;\n                this->edge_count[key] += gather.edge_count.find(key)->second;\n\t}\n    for(std::map::const_iterator iter = gather.counts.begin();\n            iter != gather.counts.end(); ++iter){\n                long key = iter->first;\n                if(this->counts.find(key) != this->counts.end()){\n                        this->counts[key] = gather.counts.find(key)->second;\n                        this->edge_count[key] = gather.edge_count.find(key)->second;\n                }\n    }\n    return *this;\n  }\n\n\n};\n\n/*\n * Walk backwards from leaf nodes (those that have no nodes pointing to them in\n * the gather step). Each signals the node referenced in its internal spanning tree\n * record. This is performed simultaneously for each spanning tree in the set.\n *\n * The betweeness score is cached in the cost field.\n *\n */\nclass BetweenessAlgorithm :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\n    bool changed;\n\n  public:\n    edge_dir_type gather_edges(icontext_type& context, const vertex_type& vertex) const {\n      return graphlab::IN_EDGES;\n    }\n\n    BetweenessGather gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n\tBetweenessGather g;\n    for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n        iter != vertex.data().djikstra_pieces.end(); ++iter){\n        long key= iter->first;\n        if(edge.target().data().djikstra_pieces[key].id == vertex.id()){\n            if(edge.source().data().djikstra_pieces[key].launched == true){\n                g.counts[key] = edge.source().data().djikstra_pieces[key].cost;\n                g.edge_count[key] = 1;\n            }\n        }\n\t}\n    return g;\n    }\n\n    void apply(icontext_type& context, vertex_type& vertex, const BetweenessGather& total) {\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n                iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key = iter->first;\n            if(total.edge_count.find(key)->second==0){\n                vertex.data().djikstra_pieces[key].launched = true;\n                vertex.data().djikstra_pieces[key].cost = 0.0;\n            }\n            if((vertex.data().djikstra_pieces[key].launched == true)&&\n                    (vertex.data().djikstra_pieces[key].done == false)&&\n                    (((long)vertex.data().djikstra_pieces[key].cost)==total.edge_count.find(key)->second)){\n                vertex.data().djikstra_pieces[key].done = true;\n                vertex.data().djikstra_pieces[key].cost = fmax(1.0,(double)total.edge_count.find(key)->second);\n            }\n        }\n    }\n\n    edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n      // if vertex data changes, scatter to all edges.\n     \tbool done = true;\n    for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n        iter != vertex.data().djikstra_pieces.end(); ++iter){\n        long key = iter->first;\n        if(vertex.data().djikstra_pieces.find(key)->second.launched && !vertex.data().djikstra_pieces.find(key)->second.done){\n        \t\tdone = false;\n      \t\t}\n        }\n        if(!done){\n                return graphlab::OUT_EDGES;\n        }else{\n                return graphlab::NO_EDGES;\n        }\n    }\n\n    void scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n                long key = iter->first;\n                if((vertex.data().djikstra_pieces.find(key)->second.done == false) &&\n                        (vertex.data().djikstra_pieces.find(key)->second.launched == true)){\n                        context.signal(edge.target());\n                }\n        }\n    }\n};\n\nint main (int argc, char** argv){\n    // Initialize control plain using mpi\n    graphlab::mpi_tools::init(argc, argv);\n    graphlab::distributed_control dc;\n    global_logger().set_log_level(LOG_INFO);\n\n    // Parse command line options -----------------------------------------------\n    graphlab::command_line_options clopts(\"Betweeness Algorithm.\");\n    std::string graph_dir;\n    clopts.attach_option(\"graph\", graph_dir, \"The graph file. Required \");\n    clopts.add_positional(\"graph\");\n    clopts.attach_option(\"samplesize\", desired_vertices_count, \"(Sample Size) Number of spanning trees to use\");\n\n    std::string saveprefix;\n    clopts.attach_option(\"saveprefix\", saveprefix,\n                         \"If set, will save the resultant betweness score to a \"\n                         \"sequence of files with prefix saveprefix\");\n\n    if(!clopts.parse(argc, argv)) {\n      dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n      return EXIT_FAILURE;\n    }\n    if (graph_dir == \"\") {\n      dc.cout() << \"Graph not specified. Cannot continue\";\n      return EXIT_FAILURE;\n    }\n\n    // Build the graph ----------------------------------------------------------\n    graph_type graph(dc);\n    dc.cout() << \"Loading graph using line parser\" << std::endl;\n    graph.load(graph_dir, line_parser);\n\n    dc.cout() << \"#vertices: \" << graph.num_vertices() << \" #edges:\" << graph.num_edges() << std::endl;\n\n    graphlab::omni_engine engine(dc, graph, \"asynchronous\", clopts);\n\n    num_vertices = graph.num_vertices();\n    graphlab::vertex_set start_set = graph.select(selectVertices);\n    engine.signal_vset(start_set);\n    engine.start();\n\n    const float runtime = engine.elapsed_seconds();\n    dc.cout() << \"Finished Djikstra engine in \" << runtime << \" seconds.\" << std::endl;\n\n    graphlab::omni_engine engine2(dc,graph,\"asynchronous\",clopts);\n    engine2.signal_all();\n    engine2.start();\n    const float runtime2 = engine.elapsed_seconds();\n    dc.cout() << \"Finished resetting graph engine in \" << runtime2 << \" seconds.\" << std::endl;\n\n    graphlab::omni_engine engine3(dc,graph,\"asynchronous\",clopts);\n    engine3.signal_all();\n    engine3.start();\n    const float runtime3 = engine.elapsed_seconds();\n    dc.cout() << \"Finished Betweeness engine in \" << runtime3 << \" seconds.\" << std::endl;\n\n    if (saveprefix != \"\") {\n      graph.save(saveprefix, betweeness_writer(),\n         false,  // do not gzip\n         true,   //save vertices\n         false); // do not save edges\n    }\n\n\n    graphlab::mpi_tools::finalize();\n    return EXIT_SUCCESS;\n}\n\n\n"
  },
  {
    "path": "toolkits/graph_algorithms/closeness.cpp",
    "content": "/*\n * Copyright (c) 2014 Daniel McEnnis.\n * portions of main Copyright (c) 2009 Carnegie Mellon\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n#include \n#include \n\n/*\n * Djikstra Graph Node Class\n *\n * This class contains the information about a single graphlab node.\n * id - current best path's previous node id - next node on path to root\n * cost - current cost of the path to route by the current route: Note - this\n *       can become inaccurate in the course of calculations and must be recalculated\n *       by traversing the shortest path tree to get an accurate result.\n * launched - has execution of this node been sheduled\n * done - has execution of this node been completed\n */\nclass DjikstraNode {\npublic:\n    unsigned long id;\n    double cost;\n    bool launched;\n    bool done;\n\n    DjikstraNode(){\n        id = 0;\n        cost = 1e100;\n        launched = false;\n        done=false;\n    }\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << id << cost << launched << done;\n  }\n\n  void load(graphlab::iarchive& iarc) {\n    iarc >> id >> cost >> launched >> done;\n  }\n};\n\n\n/*\n * PrestigeAnalysisNode\n * Graph Node class for running multiple djikstra tree algorithms simultaneously\n * Contains a map of node id's to DjikstraNode instances\n * bookkeeping components\n *\n */\nclass PrestigeAnalysisNode {\npublic:\n    std::map djikstra_pieces;\n    double local_value;\n    double total;\n    long count;\n    int edge_count;\n\n    PrestigeAnalysisNode(){\n        local_value=0.0;\n        total=0.0;\n        count=0;\n        edge_count=-1;\n    }\n\n    void save(graphlab::oarchive& oarc) const {\n        oarc << djikstra_pieces << local_value << total << count << edge_count;\n    }\n\n    void load(graphlab::iarchive& iarc) {\n        iarc >> djikstra_pieces >> local_value >> total >> count >> edge_count;\n    }\n};\n\n\n/*\n * Gather class for the Djikstra algorithm.\n * id: node id of the incoming edge's other end\n * cost: shortest path cost at the time this node gathers its edges\n * edge_count: a count of gathered edges\n *\n */\nclass Gather {\npublic:\n    long id;\n    double cost;\n    int edge_count;\n\n    Gather(){\n        id=0;\n        cost=0.0;\n        edge_count=1;\n    }\n\n    Gather& operator+=(const Gather& other){\n        if(other.id < 0){\n            return *this;\n        }\n        if(this->id < 0){\n            return *this;\n        }\n        if (cost <= other.cost){\n            this->edge_count++;\n            return *this;\n        }\n        this->edge_count += other.edge_count;\n        return *this;\n    }\n\n\n    void save(graphlab::oarchive& oarc) const {\n       oarc << id << cost << edge_count;\n     }\n\n     void load(graphlab::iarchive& iarc) {\n       iarc >> id >> cost >> edge_count;\n     }\n\n};\n\n\n/*\n * GatherMultiTree\n * map of djisktra root id's to their asociated content for that tree\n *\n */\nclass GatherMultiTree {\npublic:\n    std::map content;\n    int edge_count;\n\n    GatherMultiTree(){\n        edge_count=0;\n    }\n\n    GatherMultiTree& operator+=(const GatherMultiTree& other){\n        return *this;\n    }\n\n    void save(graphlab::oarchive& oarc) const {\n        oarc << content << edge_count;\n    }\n\n    void load(graphlab::iarchive& iarc) {\n        iarc >> content >> edge_count;\n    }\n};\n\ntypedef PrestigeAnalysisNode vertex_data_type;\n\ntypedef GatherMultiTree gather_type;\n\n// The graph type is determined by the vertex and edge data types\ntypedef graphlab::distributed_graph graph_type;\n\n\n/*\n * Loads graphs in the form 'id (id edge_strength)*'\n *\n */\nbool line_parser(graph_type& graph, const std::string& filename, const std::string& textline) {\n  std::stringstream strm(textline);\n  graphlab::vertex_id_type vid;\n  // first entry in the line is a vertex ID\n  strm >> vid;\n  PrestigeAnalysisNode node;\n  // insert this vertex with its label\n  graph.add_vertex(vid, node);\n  // while there are elements in the line, continue to read until we fail\n  double edge_val=1.0;\n  while(1){\n    graphlab::vertex_id_type other_vid;\n    strm >> other_vid;\n    strm >> edge_val;\n    if (strm.fail())\n      break;\n    graph.add_edge(vid, other_vid,edge_val);\n  }\n\n  return true;\n}\n\n\n/*\n * Algorithm class whose sole purpose is to reset launched and done booleans\n * for all id's in a PrestigeAnalysisNode\n */\nclass ClearBooleans :\n        public graphlab::ivertex_program,\n        public graphlab::IS_POD_TYPE {\npublic:\n  edge_dir_type gather_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n\n  gather_type gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n        GatherMultiTree g;\n        return g;\n  }\n\n  void apply(icontext_type& context, vertex_type& vertex, const gather_type& total) {\n      for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n          iter != vertex.data().djikstra_pieces.end(); ++iter){\n          long key = iter->first;\n          vertex.data().djikstra_pieces[key].launched = false;\n          vertex.data().djikstra_pieces[key].done = false;\n          vertex.data().djikstra_pieces[key].cost = 0.0;\n      }\n  }\n\n  edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n          return graphlab::NO_EDGES;\n  }\n\n  void scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n  }\n};\n\n\n/*\n * Djikstra Algorithm Class\n *\n * Starting from the starting nodes, create an id for this root and signal\n * all neighbors to start the calculations. Set launched when started, done\n * when all edges have been signaled.\n *\n * As a signal is receieved collect edges to determine if the best path has\n * changed.  If it has, update. If the first signal is receieved, marked\n * the node as launched and then mark the node done after signaling neighbors.\n *\n * The process terminates when all nodes active have no neighbors that are not done.\n *\n * This differs from djikstra in that the edge directions are reversed.\n */\nclass DjikstraAlgorithm :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\n    bool changed;\n\n  public:\n    edge_dir_type gather_edges(icontext_type& context, const vertex_type& vertex) const {\n      return graphlab::OUT_EDGES;\n    }\n\n    gather_type gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n    Gather g;\n    GatherMultiTree tree;\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key=iter->first;\n            if((edge.source().data().djikstra_pieces[key].launched == true)&&\n                    (edge.source().data().djikstra_pieces[key].done == false)){\n                double c = edge.data() + edge.source().data().djikstra_pieces[key].cost;\n                g.cost = c;\n                g.id = edge.source().data().djikstra_pieces[key].id;\n                g.edge_count = 1;\n                tree.content[key] = g;\n            }else{\n                g.id=0;\n            }\n        }\n    \treturn tree;\n\t}\n\n    void apply(icontext_type& context, vertex_type& vertex, const gather_type& total) {\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key = iter->first;\n            if(vertex.data().djikstra_pieces[key].launched == false){\n                vertex.data().djikstra_pieces[key].launched = true;\n                vertex.data().edge_count = total.edge_count;\n                    if(vertex.data().djikstra_pieces[key].cost > total.content.find(key)->second.cost){\n                    vertex.data().djikstra_pieces[key].cost = total.content.find(key)->second.cost;\n                    vertex.data().djikstra_pieces[key].id = total.content.find(key)->second.id;\n                    }else{\n                    vertex.data().djikstra_pieces[key].done = true;\n                    }\n            }else{\n                vertex.data().djikstra_pieces[key].done = true;\n            }\n        }\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key = iter->first;\n            if(vertex.data().djikstra_pieces.find(key)==vertex.data().djikstra_pieces.end()){\n                vertex.data().djikstra_pieces[key].launched = true;\n                vertex.data().edge_count = total.edge_count;\n                vertex.data().djikstra_pieces[key].cost = total.content.find(key)->second.cost;\n                vertex.data().djikstra_pieces[key].id = total.content.find(key)->second.id;\n            }\n        }\n    }\n\n    edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n      // if vertex data changes, scatter to all edges.\n        bool done = true;\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key = iter->first;\n            if(vertex.data().djikstra_pieces.find(key)->second.launched &&\n                    !vertex.data().djikstra_pieces.find(key)->second.done){\n                done = false;\n            }\n        }\n        if(!done){\n            return graphlab::IN_EDGES;\n        }else{\n            return graphlab::NO_EDGES;\n        }\n    }\n\n    void scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n    for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n        iter != vertex.data().djikstra_pieces.end(); ++iter){\n        long key = iter->first;\n        if((vertex.data().djikstra_pieces.find(key)->second.done == false) &&\n                (vertex.data().djikstra_pieces.find(key)->second.launched == true)){\n                context.signal(edge.target());\n            }\n    }\n  }\n};\n\nsize_t num_vertices = 3000;\nsize_t desired_sample_size = 3000;\nsize_t selected_sample_size = 0;\n\n/*\n * For every node, print the previous node in its spanning tree for all spanning trees this node is in.\n *\n */\nstruct closeness_writer {\n  std::string save_vertex(graph_type::vertex_type v) {\n    std::stringstream strm;\n    strm << v.id() << \"\\t\";\n    double value = 0.0;\n    for(std::map::const_iterator iter = v.data().djikstra_pieces.begin();\n        iter != v.data().djikstra_pieces.end();++iter){\n        value += iter->second.cost;\n    }\n    value /= selected_sample_size;\n    strm << value << std::endl;\n    return strm.str();\n  }\n  std::string save_edge (graph_type::edge_type e) { return \"\"; }\n};\n\n/*\n * Select ~3000 root nodes or an exact count which gives up around +/-3% accuracy\n * in prestige measures. It is a constant memory random selector.\n */\nbool selectVertices(const graph_type::vertex_type& vertex){\n    unsigned int r = random();\n    std::cout << \"Random seed is \" << r << std::endl;\n    if(r < (desired_sample_size * RAND_MAX / num_vertices)){\n          selected_sample_size++;\n          return true;\n    }\n    return false;\n}\n\n\n/*\n * Collects the current shortest path cost for each spanning tree\n * in its map of spanning trees.  The algorithm is responsible for\n * filtering out trees where the collecting node is not a node a step\n * further in the spanning tree.\n */\nclass ClosenessGather{\npublic:\n    std::map counts;\n    std::map edge_count;\n\t\n void save(graphlab::oarchive& oarc) const {\n    oarc << counts << edge_count;\n  }\n\n  void load(graphlab::iarchive& iarc) {\n    iarc >> counts >> edge_count;\n  }\n\n  ClosenessGather& operator+=(const ClosenessGather& gather){\n    for(std::map::const_iterator iter = this->counts.begin();\n        iter != this->counts.end(); ++iter ){\n                long key = iter->first;\n                this->counts[key] += gather.counts.find(key)->second;\n                this->edge_count[key] += gather.edge_count.find(key)->second;\n\t}\n    for(std::map::const_iterator iter = gather.counts.begin();\n            iter != gather.counts.end(); ++iter){\n                long key = iter->first;\n                if(this->counts.find(key) != this->counts.end()){\n                        this->counts[key] = gather.counts.find(key)->second;\n                        this->edge_count[key] = gather.edge_count.find(key)->second;\n                }\n    }\n    return *this;\n  }\n\n\n};\n\n/*\n * For every spanning tree root node in the graph node map, start from the root\n * and walk back towards the leafs.  Record the shortest path costs for every node\n * along the way, halting when no signaled node has a neighbor that is not done.\n *\n * This generates a random sample of the set of all spanning trees for the graph\n * of a about 3000 trees (or exact value if fewer than 3000 nodes) that gives the\n * closeness metric of each node to +/-3%. It is needed to sum the closeness for each\n * djikstra id to get the value without using the output function.\n *\n */\nclass ClosenessAlgorithm :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\n    bool changed;\n\n  public:\n    edge_dir_type gather_edges(icontext_type& context, const vertex_type& vertex) const {\n      return graphlab::OUT_EDGES;\n    }\n\n    ClosenessGather gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n    ClosenessGather g;\n    for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n        iter != vertex.data().djikstra_pieces.end(); ++iter){\n        long key= iter->first;\n        if(edge.target().data().djikstra_pieces[key].id == vertex.id()){\n            if(edge.source().data().djikstra_pieces[key].launched == true){\n                g.counts[key] = edge.source().data().djikstra_pieces[key].cost + edge.data();\n                g.edge_count[key] = 1;\n            }\n        }\n\t}\n    return g;\n    }\n\n    void apply(icontext_type& context, vertex_type& vertex, const ClosenessGather& total) {\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n                iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key = iter->first;\n            if(total.edge_count.find(key)->second==0){\n                vertex.data().djikstra_pieces[key].launched = true;\n            }\n            if((vertex.data().djikstra_pieces[key].launched == true)&&\n                    (vertex.data().djikstra_pieces[key].done == false)&&\n                    (((long)vertex.data().djikstra_pieces[key].cost)==total.edge_count.find(key)->second)){\n                vertex.data().djikstra_pieces[key].done = true;\n                vertex.data().djikstra_pieces[key].cost = (double)total.edge_count.find(key)->second;\n            }\n        }\n    }\n\n    edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n      // if vertex data changes, scatter to all edges.\n     \tbool done = true;\n    for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n        iter != vertex.data().djikstra_pieces.end(); ++iter){\n        long key = iter->first;\n        if(vertex.data().djikstra_pieces.find(key)->second.launched && !vertex.data().djikstra_pieces.find(key)->second.done){\n        \t\tdone = false;\n      \t\t}\n        }\n        if(!done){\n                return graphlab::IN_EDGES;\n        }else{\n                return graphlab::NO_EDGES;\n        }\n    }\n\n    void scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n                long key = iter->first;\n                if((vertex.data().djikstra_pieces.find(key)->second.done == false) &&\n                        (vertex.data().djikstra_pieces.find(key)->second.launched == true)){\n                        context.signal(edge.target());\n                }\n        }\n    }\n};\n\nint main (int argc, char** argv){\n    // Initialize control plain using mpi\n    graphlab::mpi_tools::init(argc, argv);\n    graphlab::distributed_control dc;\n    global_logger().set_log_level(LOG_INFO);\n\n    // Parse command line options -----------------------------------------------\n    graphlab::command_line_options clopts(\"Closeness Algorithm\");\n    std::string graph_dir;\n    clopts.attach_option(\"graph\", graph_dir, \"The graph file. Required \");\n    clopts.add_positional(\"graph\");\n    clopts.attach_option(\"samplesize\", desired_sample_size , \"(Sample size) the number of spanning trees to calculate\");\n\n    std::string saveprefix;\n    clopts.attach_option(\"saveprefix\", saveprefix,\n                         \"If set, will save the resultant closeness score to a \"\n                         \"sequence of files with prefix saveprefix\");\n\n    if(!clopts.parse(argc, argv)) {\n      dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n      return EXIT_FAILURE;\n    }\n    if (graph_dir == \"\") {\n      dc.cout() << \"Graph not specified. Cannot continue\";\n      return EXIT_FAILURE;\n    }\n\n    // Build the graph ----------------------------------------------------------\n    graph_type graph(dc);\n    dc.cout() << \"Loading graph using line parser\" << std::endl;\n    graph.load(graph_dir, line_parser);\n\n    dc.cout() << \"#vertices: \" << graph.num_vertices() << \" #edges:\" << graph.num_edges() << std::endl;\n\n    graphlab::omni_engine engine(dc, graph, \"asynchronous\", clopts);\n\n    num_vertices = graph.num_vertices();\n    graphlab::vertex_set start_set = graph.select(selectVertices);\n    engine.signal_vset(start_set);\n    engine.start();\n\n    const float runtime = engine.elapsed_seconds();\n    dc.cout() << \"Finished Djikstra engine in \" << runtime << \" seconds.\" << std::endl;\n\n    graphlab::omni_engine engine2(dc,graph,\"asynchronous\",clopts);\n    engine2.signal_all();\n    engine2.start();\n\n    const float runtime2 = engine.elapsed_seconds();\n    dc.cout() << \"Finished resetting the graph in \" << runtime2 << \" seconds.\" << std::endl;\n\n    graphlab::omni_engine engine3(dc,graph,\"asynchronous\",clopts);\n    engine3.signal_vset(start_set);\n    engine3.start();\n\n    const float runtime3 = engine.elapsed_seconds();\n    dc.cout() << \"Finished the closeness engine in \" << runtime3 << \" seconds.\" << std::endl;\n\n    if (saveprefix != \"\") {\n      graph.save(saveprefix, closeness_writer(),\n         false,  // do not gzip\n         true,   //save vertices\n         false); // do not save edges\n    }\n\n\n    graphlab::mpi_tools::finalize();\n    return EXIT_SUCCESS;\n}\n\n\n"
  },
  {
    "path": "toolkits/graph_algorithms/djikstra.cpp",
    "content": "/*  \n * Copyright (c) 2014 Daniel McEnnis.\n * portions of main Copyright (c) 2009 Carnegie Mellon\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n#include \n#include \n\n/*\n * Djikstra Graph Node Class\n *\n * This class contains the information about a single graphlab node.\n * id - current best path's previous node id - next node on path to root\n * cost - current cost of the path to route by the current route: Note - this\n *       can become inaccurate in the course of calculations and must be recalculated\n *       by traversing the shortest path tree to get an accurate result.\n * launched - has execution of this node been sheduled\n * done - has execution of this node been completed\n */\nclass DjikstraNode {\npublic:\n\tlong id;\n\tdouble cost;\n\tbool launched;\n\tbool done;\n\n\tDjikstraNode(){\n\t\tid = 0;\n\t\tcost = 1e100;\n\t\tlaunched = false;\n\t\tdone=false;\n\t}\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << id << cost << launched << done;\n  }\n\n  void load(graphlab::iarchive& iarc) {\n    iarc >> id >> cost >> launched >> done;\n  }\n};\n\n/*\n * PrestigeAnalysisNode\n * Graph Node class for running multiple djikstra tree algorithms simultaneously\n * Contains a map of node id's to DjikstraNode instances\n * bookkeeping components\n *\n */\nclass PrestigeAnalysisNode {\npublic:\n    std::map djikstra_pieces;\n\tdouble local_value;\n\tdouble total;\n\tlong count;\n\tint edge_count;\n\n\tPrestigeAnalysisNode(){\n\t\tlocal_value=0.0;\n\t\ttotal=0.0;\n\t\tcount=0;\n\t\tedge_count=-1;\n\t}\n\n    void save(graphlab::oarchive& oarc) const {\n        oarc << djikstra_pieces << local_value << total << count << edge_count;\n\t}\n\n\tvoid load(graphlab::iarchive& iarc) {\n        iarc >> djikstra_pieces >> local_value >> total >> count >> edge_count;\n\t}\n};\n\n/*\n * Gather class for the Djikstra algorithm.\n * id: node id of the incoming edge's other end\n * cost: shortest path cost at the time this node gathers its edges\n * edge_count: a count of gathered edges\n *\n */\nclass Gather {\npublic:\n\tunsigned long id;\n\tdouble cost;\n\tint edge_count;\n\t\n\tGather(){\n\t\tid=0;\n\t\tcost=0.0;\n\t\tedge_count=1;\n\t}\n\n\tGather& operator+=(const Gather& other){\n\t\tif(other.id < 0){\n\t\t\treturn *this;\n\t\t}\n        if(this->id < 0){\n\t\t\treturn *this;\n\t\t}\t\t\n\t\tif (cost <= other.cost){\n            this->edge_count++;\n\t\t\treturn *this;\n\t\t}\n        this->edge_count += other.edge_count;\n        return *this;\n\t}\n\n\n    void save(graphlab::oarchive& oarc) const {\n       oarc << id << cost << edge_count;\n     }\n\n     void load(graphlab::iarchive& iarc) {\n       iarc >> id >> cost >> edge_count;\n     }\n\n};\n\n/*\n * GatherMultiTree\n * map of djisktra root id's to their asociated content for that tree\n *\n */\nclass GatherMultiTree {\npublic:\n    std::map content;\n\tint edge_count;\n\n\tGatherMultiTree(){\n\t\tedge_count=0;\n\t}\n\n\tGatherMultiTree& operator+=(const GatherMultiTree& other){\n\t\treturn *this;\t\n\t}\n\n    void save(graphlab::oarchive& oarc) const {\n        oarc << content << edge_count;\n    }\n\n    void load(graphlab::iarchive& iarc) {\n        iarc >> content >> edge_count;\n    }\n};\n\n// The vertex data is its label \ntypedef PrestigeAnalysisNode vertex_data_type;\n\ntypedef GatherMultiTree gather_type;\n \n// The graph type is determined by the vertex and edge data types\ntypedef graphlab::distributed_graph graph_type;\n\n/*\n * Loads graphs in the form 'id (id edge_strength)*'\n *\n */\nbool line_parser(graph_type& graph, const std::string& filename, const std::string& textline) {\n  std::stringstream strm(textline);\n  graphlab::vertex_id_type vid;\n  // first entry in the line is a vertex ID\n  strm >> vid;\n  PrestigeAnalysisNode node;\n  // insert this vertex with its label\n  graph.add_vertex(vid, node);\n  // while there are elements in the line, continue to read until we fail\n  double edge_val=1.0;\n  while(1){\n    graphlab::vertex_id_type other_vid;\n    strm >> other_vid;\n    strm >> edge_val;\n    if (strm.fail())\n      break;\n    graph.add_edge(vid, other_vid,edge_val);\n  }\n\n  return true;\n}\n\n/*\n * Algorithm class whose sole purpose is to reset launched and done booleans\n * for all id's in a PrestigeAnalysisNode\n */\nclass ClearBooleans :\n        public graphlab::ivertex_program,\n        public graphlab::IS_POD_TYPE {\npublic:\n  edge_dir_type gather_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n\n  gather_type gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n        GatherMultiTree g;\n        return g;\n  }\n\n  void apply(icontext_type& context, vertex_type& vertex, const gather_type& total) {\n      for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n          iter != vertex.data().djikstra_pieces.end(); ++iter){\n          long key = iter->first;\n          vertex.data().djikstra_pieces[key].launched = false;\n          vertex.data().djikstra_pieces[key].done = false;\n          vertex.data().djikstra_pieces[key].cost = 0.0;\n      }\n  }\n\n  edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n          return graphlab::NO_EDGES;\n  }\n\n  void scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n  }\n};\n\n/*\n * Djikstra Algorithm Class\n *\n * Starting from the starting nodes, create an id for this root and signal\n * all neighbors to start the calculations. Set launched when started, done\n * when all edges have been signaled.\n *\n * As a signal is receieved collect edges to determine if the best path has\n * changed.  If it has, update. If the first signal is receieved, marked\n * the node as launched and then mark the node done after signaling neighbors.\n *\n * The process terminates when all nodes active have no neighbors that are not done.\n */\nclass DjikstraAlgorithm :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\n    bool changed;\n\n  public:\n    edge_dir_type gather_edges(icontext_type& context, const vertex_type& vertex) const {\n      return graphlab::IN_EDGES;\n    }\n\n    gather_type gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n    Gather g;\n    GatherMultiTree tree;\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key=iter->first;\n            if((edge.source().data().djikstra_pieces[key].launched == true)&&\n                    (edge.source().data().djikstra_pieces[key].done == false)){\n                double c = edge.data() + edge.source().data().djikstra_pieces[key].cost;\n                g.cost = c;\n                g.id = edge.source().data().djikstra_pieces[key].id;\n                g.edge_count = 1;\n                tree.content[key] = g;\n            }else{\n                g.id=0;\n            }\n        }\n\treturn tree;\n    }\n\n    void apply(icontext_type& context, vertex_type& vertex, const gather_type& total) {\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key = iter->first;\n            if(vertex.data().djikstra_pieces[key].launched == false){\n                vertex.data().djikstra_pieces[key].launched = true;\n                vertex.data().edge_count = total.edge_count;\n                    if(vertex.data().djikstra_pieces[key].cost > total.content.find(key)->second.cost){\n                    vertex.data().djikstra_pieces[key].cost = total.content.find(key)->second.cost;\n                    vertex.data().djikstra_pieces[key].id = total.content.find(key)->second.id;\n                    }else{\n                    vertex.data().djikstra_pieces[key].done = true;\n                    }\n      \t\t}else{\n                vertex.data().djikstra_pieces[key].done = true;\n            }\n        }\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key = iter->first;\n            if(vertex.data().djikstra_pieces.find(key)==vertex.data().djikstra_pieces.end()){\n                vertex.data().djikstra_pieces[key].launched = true;\n                vertex.data().edge_count = total.edge_count;\n                vertex.data().djikstra_pieces[key].cost = total.content.find(key)->second.cost;\n                vertex.data().djikstra_pieces[key].id = total.content.find(key)->second.id;\n            }\n        }\n    }\n\n    edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n      // if vertex data changes, scatter to all edges.\n     \tbool done = true;\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key = iter->first;\n            if(vertex.data().djikstra_pieces.find(key)->second.launched &&\n                    !vertex.data().djikstra_pieces.find(key)->second.done){\n        \t\tdone = false;\n      \t\t}\n        }\n        if(!done){\n            return graphlab::OUT_EDGES;\n        }else{\n            return graphlab::NO_EDGES;\n        }\n    }\n\n    void scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n    for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n        iter != vertex.data().djikstra_pieces.end(); ++iter){\n        long key = iter->first;\n        if((vertex.data().djikstra_pieces.find(key)->second.done == false) &&\n                (vertex.data().djikstra_pieces.find(key)->second.launched == true)){\n      \t\t\tcontext.signal(edge.target()); \n    \t\t}\n\t}\t\n  }\n};\n\n/*\n * For every node, print the previous node in its spanning tree for all spanning trees this node is in.\n *\n */\nstruct djikstra_writer {\n  std::string save_vertex(graph_type::vertex_type v) {\n    std::stringstream strm;\n    strm << v.id();\n    for(std::map::const_iterator iter = v.data().djikstra_pieces.begin();\n        iter != v.data().djikstra_pieces.end(); ++iter){\n        long key = iter->first;\n        strm << \"\\t\" << key << \"\\t\" << iter->second.id << std::endl;\n    }\n    return strm.str();\n  }\n  std::string save_edge (graph_type::edge_type e) { return \"\"; }\n};\n\nsize_t num_vertices = 3000;\nsize_t desired_vertices_count = 3000;\nsize_t selected_vertices_count = 0;\n\n/*\n * Select ~3000 root nodes or an exact count which gives up around +/-3% accuracy\n * in prestige measures. It is a constant memory random selector.\n */\nbool selectVertices(const graph_type::vertex_type& vertex){\n    unsigned int r = random();\n    std::cout << \"Random seed is \" << r << std::endl;\n    if(r < ((desired_vertices_count * RAND_MAX) / num_vertices)){\n        selected_vertices_count++;\n          return true;\n    }\n    return false;\n}\n\n\nint main(int argc, char** argv) {\n  // Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  global_logger().set_log_level(LOG_INFO);\n  \n  // Parse command line options -----------------------------------------------\n  graphlab::command_line_options clopts(\"Djikstra Algorithm.\");\n  std::string graph_dir;\n  clopts.attach_option(\"graph\", graph_dir, \"The graph file. Required \");\n  clopts.add_positional(\"graph\");\n  clopts.attach_option(\"samplesize\", desired_vertices_count, \"Target number of simultaneous spanning trees\");\n\n  std::string saveprefix;\n  clopts.attach_option(\"saveprefix\", saveprefix,\n                       \"If set, will save the spanning trees to a \"\n                       \"sequence of files with prefix saveprefix\");\n\n  if(!clopts.parse(argc, argv)) {\n    dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n  if (graph_dir == \"\") {\n    dc.cout() << \"Graph not specified. Cannot continue\";\n    return EXIT_FAILURE;\n  }\n \n  // Build the graph ----------------------------------------------------------\n  graph_type graph(dc);\n  dc.cout() << \"Loading graph using line parser\" << std::endl;\n  graph.load(graph_dir, line_parser);\n\n  dc.cout() << \"#vertices: \" << graph.num_vertices() << \" #edges:\" << graph.num_edges() << std::endl;\n\n  // Algorithm for creating the spanning trees\n  graphlab::omni_engine engine(dc, graph, \"asynchronous\", clopts);\n\n  num_vertices = graph.num_vertices();\n  // create the total number of djisktra spanning trees to create at once.\n  graphlab::vertex_set start_set = graph.select(selectVertices);\n  engine.signal_vset(start_set);\n  engine.start();\n\n  const float runtime = engine.elapsed_seconds();\n  dc.cout() << \"Finished Running engine in \" << runtime << \" seconds.\" << std::endl;\n\n  if (saveprefix != \"\") {\n    graph.save(saveprefix, djikstra_writer(),\n       false,  // do not gzip\n       true,   //save vertices\n       false); // do not save edges\n  }\n  \n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n}\n"
  },
  {
    "path": "toolkits/graph_algorithms/graph_analytics.dox",
    "content": "/**\n\n\\page graph_analytics Graph Analytics \n\n\\brief The graph analytics toolkit contains applications for performing graph \nanalytics and extracting patterns from the graph structure mostly from the Social Network Analysis Toolkit\n\nThe toolkit current contains:\n - \\ref djikstra \"Djisktra Algorithm Base\"\n - \\ref betweeness \"Betweeness Algorithm\"\n - \\ref closeness \"Closeness Algorithm\"\n - \\ref prestige \"Prestge Algoritm\"\n\nAll toolkits take any of the graph formats described in \\ref graph_formats . \n\n\n\\section djikstra \"Djikstra Algorithm Base\"\n\nThe input format for the djikstra algorithm is\n\n\\verbatim\n [ ]*\n\\endverbatim\n\nThe output format of the djikstra algorithm is\n\n\\verbatim\n [ ]*\n\\endverbatim\n\nroots do not list themselves\n\nRun this command with:\n\n\\verbatim\nmpiexec -n  --hostfile  ./djikstra --graph  [--saveprefix ] \n\\endverbatim\n\nThe output describes the spanning trees constructed starting at up to 3000 nodes\n\n\\subsection djikstra_imp \"Djikstra Algorithm Details\"\n\nDjikstra starts with a randomly selected (~3000) set of nodes using a constant time and space random process.  Each gather collects the earlier spanning tree members.  The apply step calculates the best path, storing it. The scatter step notifies all nodes other than nodes that have notifid it to execute now. This process runs concurrently for each starting node.  \n\nNote: this algorithm does not preserve correctness of the total cost at each node, only relative stregth between node choices. Re-walk the spanning tree to calculate this (see below for examples).\n\n\\section betweeness \"Betweeness Algorithms\"\n\nThe input format for the betweeness algorithm is:\n\n\\verbatim\n [ ]*\n\\endverbatim\n\nThe output format of the betweeness algorithm is\n\n\\verbatim\n \n\\endverbatim\n\nRun this command with:\n\n\\verbatim\nmpiexec -n  --hostfile  ./betweeness --graph  [--saveprefix ] \n\\endverbatim\n\nThe output estimates betweeness using ~3000 randomly selected spanning trees (typically +/-3% accuracy in the measure for each node.)\n\n\\subsection betweeness_imp \"Betweeness Algorithm Details\"\n\nDjikstra sanning trees are calculated first, then the datsa structure is reset, then betweeness scores are calculated by walking the spanning trees from leaves to roots. Finally, the betweeness scores are collated from the various samples of spanning trees.\n\nSee \\ref djiksra_imp for details on how spanning trees are calculated.\n\nThe next step resets all the bookkeeping on the spanning trees and sets costs to zero.\n\nFinally, the betweeness is calculated.  All nodes are started at first, but only nodes without another spanning tree node pointing to it have a non-null execution.  \n\nThe gather step checks if all nodes in the spanning tree pointing to it have been calculated yet, silently skipping if this is not true.  Otherwise, the betweeness scores are collected.\n\nThe apply step sums the betweeness score for this node and spanning tree.\n\nThe scatter step signals the next higher node in this spanning tree that a new betweeness score is ready.\n\nWhen the graph is saved, it outputs the sum of all betweeness scores across all calculated spanning trees and estimates the expected final betweeness score.\n\n\\section closeness \"Closeness Algorithm\"\n\nThe input format for the closeness algorithm is:\n\n\\verbatim\n [ ]*\n\\endverbatim\n\nThe output format of the betweeness algorithm is\n\n\\verbatim\n \n\\endverbatim\n\nRun this command with:\n\n\\verbatim\nmpiexec -n  --hostfile  ./closeness --graph  [--saveprefix ] \n\\endverbatim\n\nThe output estimates closeness using ~3000 randomly selected spanning trees (typically +/-3% accuracy in the measure for each node.)\n\n\\subsection closeness_imp \"Closeness Algorithm Details\"\n\nDjikstra spanning trees are calculated first, then the datsa structure is reset, then closeness scores are calculated by walking the spanning trees from leaves to roots. Finally, the closeness scores are collated from the various samples of spanning trees.\n\nSee \\ref djiksra_imp for details on how spanning trees are calculated, except the link direction is reversed.\n\nThe next step resets all the bookkeeping on the spanning trees and sets costs to zero.\n\nFinally, the closeness is calculated.  The starting node set is reused, and the spanning trees are all walked simultaneously from root to leaves.\n\nThe gather step collects the parent closeness score.\n\nThe apply step combines the parent's closeness score with the edge value and stores it.\n\nThe scatter step signals al child nodes.\n\nWhen the graph is saved, it outputs the sum of all closeness scores across all calculated spanning trees and estimates the expected final closeness score.\n\n\n\n\\section prestige \"Prestige Algorithm\" \n\nThe input format for the prestige algorithm is:\n\n\\verbatim\n [ ]*\n\\endverbatim\n\nThe output format of the betweeness algorithm is\n\n\\verbatim\n \n\\endverbatim\n\nRun this command with:\n\n\\verbatim\nmpiexec -n  --hostfile  ./prestige --graph  [--saveprefix ] \n\\endverbatim\n\nThe output estimates prestige using ~3000 randomly selected spanning trees (typically +/-3% accuracy in the measure for each node.)\n\n\n\\subsection prestige_imp \"Prestige Algorithm Details\"\n\nDjikstra spanning trees are calculated first, then the datsa structure is reset, then prestige scores are calculated by walking the spanning trees from leaves to roots. Finally, the prestige scores are collated from the various samples of spanning trees.\n\nSee \\ref djiksra_imp for details on how spanning trees are calculated.\n\nThe next step resets all the bookkeeping on the spanning trees and sets costs to zero.\n\nFinally, the prestige is calculated.  The starting node set is reused, and the spanning trees are all walked simultaneously from root to leaves.\n\nThe gather step collects the parent prestige score.\n\nThe apply step combines the parent's prestige score with the edge value and stores it.\n\nThe scatter step signals al child nodes.\n\nWhen the graph is saved, it outputs the sum of all prestige scores across all calculated spanning trees and estimates the expected final prestige score.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\\section graph_analytics_pagerank PageRank \n\nThe PageRank program computes the pagerank of each vertex. \nSee the Wikipedia article\nfor details of the algorithm.\n\n\\subsection Input Graph\nThe input to the system is a graph in any of the Portable graph format\ndescribed in \\ref graph_formats. \n\n\\verbatim\n> ./pagerank --graph=[graph prefix] --format=[format] \n\\endverbatim\n\nAlternatively, a synthetic power law graph of an arbitrary number of vertices\ncan be generated using:\n\\verbatim\n> ./pagerank --powerlaw=[nvertices]\n\\endverbatim\nThe resultant graph will have powerlaw out-degree, and nearly constant in-degree. \nThe actual generation process draws vertex degree from a truncated power-law\ndistribution with alpha=2.1. The distribution is truncated at maximum out-degree\n100M to avoid allocating massive amounts of memory for creating the sampling distribution.\n\n\\subsection Computation Type \nThere are several modes of computation that are supported. All will eventually\nobtain the same solutions. \n\n### Classical\nTo get classical PageRank iterations, adding the option\n\\verbatim\n> --iterations=[N Iterations]\n\\endverbatim\n\n### Dynamic Synchronous (default) \nThe dynamic synchronous computation only performs computation on vertices\nthat have not yet converged to the desired tolerance. The default tolerance\nis 0.001. This can be modified by adding the option\n\\verbatim\n>  --tol=[tolerance]\n\\endverbatim\n\n### Dynamic Asynchronous\nThe dynamic asynchronous computation only performs computation on vertices\nthat have not yet converged to the desired tolerance. This uses the \nasynchronous engine. The default tolerance is 0.001. \nThis can be modified by adding the option\n\\verbatim\n>  --tol=[tolerance]\n\\endverbatim\n\n\\note This is known to be slow! PageRank does not benefit from the consistency\nguaranteed by the asynchronous engine. A new engine is in development with \nweaker consistency semantics, but sufficient for pagerank. \n\n\n\\subsection Output\nTo save the resultant pagerank of each vertex, include the option\n\\verbatim\n> --saveprefix=[output prefix]\n\\endverbatim\n\nTne output prefix is where the output counts will be written. This\nmay be located on HDFS. For instance, if the output_prefix is \"v_out\",\nthe output files will be written to:\n\n\\verbatim\nv_out_1_of_16\nv_out_2_of_16\n...\nv_out_16_of_16\n\\endverbatim\n\nEach line in the output file contains two numbers: a Vertex ID, and the \ncomputed PageRank. Note that the output vector is NOT normalized, namely \ncomputed entries do not sum into one. \n\nThis program can also run distributed by using\n\\verbatim\n> mpiexec -n [N machines] --hostfile [host file] ./pagerank ....\n\\endverbatim\nSee your MPI documentation for details on how to launch this job.\nAll machines must have access to the input graph location and the output graph \nlocation. Graphs may be on HDFS. \nIf you have problems loading HDFS files, see the \\ref FAQ.\n\n\n\n\\subsection Options\nRelevant options are: \n\\li \\b --graph (Optional). The prefix from which to load the graph data\n\\li \\b --format (Optional). The format of the input graph \n\\li \\b --powerlaw (Optional. Default 0). If set, generates synthetic powerlaw graph with\n                        the specified number of vertices.\n\\li \\b --saveprefix (Optional. Default \"\"). If set, will write the output counts.\n\\li \\b --tol (Optional. Default=1E-3). Changes the convergence tolerance for the Dynamic\n                          computation modes.\n\\li \\b --iterations (Optional. Default 0). If set, runs classical PageRank iterations\n                      for the specified number of iterations.\n\\li \\b -–graph_opts (Optional, Default empty) Any additional graph options. See\n  graphlab::distributed_graph a list of options.\n\\li \\b --ncpus (Optional. Default 2) The number of processors that will be used\nfor computation.  \n\\li \\b -–engine (Optional, Default \"synchronous\") Sets the engine type. Must be\n                  either \"synchronous\" or \"asynchronous\"\n\\li \\b -–engine (Optional, Default \"synchronous\") Sets the engine options. Available\n                  options depend on the engine type. See\n                  graphlab::async_consistent_engine and\n                  graphlab::synchronous_engine for details.\n\n\n\n\n\\section graph_analytics_kcore KCore Decomposition \nThis program iteratively finds the KCore of the network.\n\n\\subsection Input Graph\nThe input to the system is a graph in any of the Portable graph format\ndescribed in \\ref graph_formats. \n\n\\verbatim\n> ./kcore --graph=[graph prefix] --format=[format] \n\\endverbatim\nOutput may look like:\n\\verbatim\nK=0:  #V = 875713   #E = 4322051\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 0\nK=1:  #V = 875713   #E = 4322051\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 153407\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nK=2:  #V = 711870   #E = 4160100\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 108715\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nK=3:  #V = 581712   #E = 3915291\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 69907\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nK=4:  #V = 492655   #E = 3668104\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 52123\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nK=5:  #V = 424155   #E = 3416251\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 41269\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nK=6:  #V = 367361   #E = 3158776\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 33444\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nK=7:  #V = 319194   #E = 2902138\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 29201\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nK=8:  #V = 274457   #E = 2629033\n......\n\\endverbatim\n\nTo just get the informative lines:\n\\verbatim\n> ./kcore --graph=[graph prefix] --format=[format] > k_out.txt\n  ...\n> cat k_out.txt\nComputes a k-core decomposition of a graph.\n\nNumber of vertices: 875713\nNumber of edges:    4322051\nK=0:  #V = 875713   #E = 4322051\nK=1:  #V = 875713   #E = 4322051\nK=2:  #V = 711870   #E = 4160100\nK=3:  #V = 581712   #E = 3915291\nK=4:  #V = 492655   #E = 3668104\nK=5:  #V = 424155   #E = 3416251\nK=6:  #V = 367361   #E = 3158776\nK=7:  #V = 319194   #E = 2902138\nK=8:  #V = 274457   #E = 2629033\nK=9:  #V = 231775   #E = 2335154\nK=10:  #V = 193406   #E = 2040738\nK=11:  #V = 159020   #E = 1753273\nK=12:  #V = 131362   #E = 1500517\nK=13:  #V = 106572   #E = 1256952\nK=14:  #V = 86302   #E = 1047053\nK=15:  #V = 68409   #E = 849471\nK=16:  #V = 53459   #E = 676076\nK=17:  #V = 40488   #E = 519077\n...\n\\endverbatim\n\n\nThe program can also save a copy of the graph at each stage by adding an\noption.\n\\verbatim\n> --savecores=[prefix]\n\\endverbatim\n\nThe resultant graphs will be saved with prefixes [prefix].K\nFor instance if prefix is out, \nThe 0-Core graph may be saved in \n\\verbatim\nout.0.1_of_4\nout.0.2_of_4\nout.0.3_of_4\nout.0.4_of_4\n\\endverbatim\n\nThe 5-Core graph will be saved in \n\\verbatim\nout.5.1_of_4\nout.5.2_of_4\nout.5.3_of_4\nout.5.4_of_4\n\\endverbatim\n\nand so on. \n\nThe range of k-Core graphs to compute can be controlled by the kmin\nand the kmax option described below.\n\nThis program can also run distributed by using\n\\verbatim\n> mpiexec -n [N machines] --hostfile [host file] ./kcore....\n\\endverbatim\nSee your MPI documentation for details on how to launch this job.\nAll machines must have access to the input graph location and the output graph \nlocation. Graphs may be on HDFS. \nIf you have problems loading HDFS files, see the \\ref FAQ.\n\n\n\\subsection Options\nRelevant options are: \n\\li \\b --graph (Required). The prefix from which to load the graph data\n\\li \\b --format (Required). The format of the input graph \n\\li \\b --ncpus (Optional. Default 2) The number of processors that will be used\nfor computation.  \n\\li \\b --savecores (Optional. Default \"\"). The target prefix to save \nthe resultant K-core graphs.\n\\li \\b --kmin (Optional. Default 0). Only output result for the K-core graph starting\n                        at K=kmin\n\\li \\b --kmax (Optional. Default Inf). Only output result for the K-core graph \n                        up to K=kmax\n\n\n\n\n\n\n\n\n\n\n\n\n\n\\section graph_analytics_triangle_coloring Graph Coloring \n\nThe graph coloring program implements a really simple graph coloring \nprocedure: each vertex reads the colors of its neighbors and takes on \nthe smallest possible color which does not conflict with its neighbors.\n\nThe procedure necessarily uses the asynchronous engine (it will never\nconverge with the synchronous engine).\n\nThe input to the system is a graph in any of the Portable graph format\ndescribed in \\ref graph_formats. It is important that the input be \"cleaned\"\nand that reverse edges are removed: i.e. if edge 1-->5 exists, edge 5-->1 should\nnot exist. (The program will run without these edge removed. But numbers\nmay be erroneous).\n\nTo color a graph, the minimal set of options required are:\n\\verbatim\n> ./simple_coloring --graph=[graph prefix] --format=[format] --output=[output prefix]\n\\endverbatim\nOutput looks like:\n\\verbatim\nNumber of vertices: 875713\nNumber of edges:    5105039\nColoring...\nCompleted Tasks: 875713\nIssued Tasks: 875713\nBlocked Issues: 0\n------------------\nJoined Tasks: 0\nColored in 42.3684 seconds\nMetrics server stopping.\n\\endverbatim\n\nObserve that the number of Completed Tasks is identical to the number of vertices.\nThis is a result of the consistency model which ensures that the entire\nvertex update is peformed \"atomically\".\n\nTne output prefix is where the output counts will be written. This\nmay be located on HDFS. For instance, if the output_prefix is \"v_out\",\nthe output files will be written to:\n\n\\verbatim\nv_out_1_of_16\nv_out_2_of_16\n...\nv_out_16_of_16\n\\endverbatim\n\nEach line in the output file contains two numbers: a Vertex ID, and the number\ncolor of the vertex.\n\nThis program can also run distributed by using\n\\verbatim\n> mpiexec -n [N machines] --hostfile [host file] ./simple_coloring ....\n\\endverbatim\nSee your MPI documentation for details on how to launch this job.\nAll machines must have access to the input graph location and the output graph \nlocation. Graphs may be on HDFS. \nIf you have problems loading HDFS files, see the \\ref FAQ.\n\n\n\\subsection Options\nRelevant options are: \n\\li \\b --graph (Required). The prefix from which to load the graph data\n\\li \\b --format (Required). The format of the input graph \n\\li \\b --ncpus (Optional. Default 2) The number of processors that will be used\nfor computation.  \n\\li \\b –-graph_opts (Optional, Default empty) Any additional graph options. See\n  --graph_help a list of options.\n\\li \\b –-engine_opts (Optional, Default empty) Any additional engine options. See\n  --engine_help a list of options.\n\nA particularly relevant option is \n\\verbatim\n--engine_opts=\"factorized=true\"\n\\endverbatim\n\nThis uses a weaker consistency setting which only guarantees that individual\n\"gather/apply/scatter\" operations are atomic, but does not guarantee atomicity\nof the entire update. As a result, this may require more updates to complete,\nbut could in practice run significantly faster.\n\n\n\n\n\n\n\n\n\n\n\\section graph_analytics_connected_component Connected Component\n\nThe connected component program can find all connected components in a \ngraph, and can also count the number of vertices (size) of each connected \ncomponent.\n\nThe input to the system is a graph in any of the Portable Graph formats\ndescribed in \\ref graph_formats.\n\nTo find connected components in a graph, the minimal set of options\nrequired are:\n\\verbatim\n> ./connected_component --graph=[graph prefix] --format=[format]\n\\endverbatim\n\nHere is a toy example, graph with 6 nodes and 5 edges:\n\\verbatim\n# example graph\n# vertices: 6 edges: 5\n1 2\n2 3\n4 5\n4 6\n5 6\n\\endverbatim\n\nAssume file name is toy_graph, the command used for running connected compnents is\n\\verbatim\n> ./connected_component --graph=toy_graph --format=tsv --saveprefix=out\n\\endverbatim\n\n\nWhen you set --saveprefix=output_prefix, the pairs of a Vertex ID and a \nComponent ID will be written to a sequence of files with prefix \noutput_prefix. This may be located on HDFS. For instance, if the \noutput_prefix is \"v_out\", the output files will be written to:\n\n\\verbatim\nout_1_of_4\nout_2_of_4\nout_3_of_4\nout_4_of_4\n\\endverbatim\n\nLet's examine the output. The first column is the node id, while the second column is it's assigned component number\n(which is also the lowest node id in this component). In our case:\n\\verbatim\n1,1\n2,1\n3,1\n4,4\n5,4\n6,4\n\\endverbatim\n\nThere are two components. The first compoent is 1,2,3 and the second component is 4,5,6 \n\nNote that this program can also run distributed by using\n\\verbatim\n> mpiexec -n [N machines] --hostfile [host file] ./connected_component ....\n\\endverbatim\nSee your MPI documentation for details on how to launch this job.\nAll machines must have access to the input graph location and the output graph \nlocation. Graphs may be on HDFS. \nIf you have problems loading HDFS files, see the \\ref FAQ.\n\n\\subsection Options\nRelevant options are: \n\\li \\b --graph (Required). The prefix from which to load the graph data\n\\li \\b --format (Required). The format of the input graph \n\\li \\b --saveprefix (Optional). If set, pairs of a Vertex ID and a Component \nID will be saved to a sequence of files with the given prefix.\n\\li \\b --ncpus (Optional. Default 2). The number of processors that will be used\nfor computation.\n\\li \\b --graph_opts (Optional, Default empty). Any additional graph options. See\n  graphlab::distributed_graph a list of options.\n  \nconnected_components_stats is a helper utility, which computes histogram of component \nsizes. \n\nUsing our toy example\n\\verbatim\n> ./connected_component_stats --graph=out\nConnected Component\n\nINFO:     mpi_tools.hpp(init:63): MPI Support was not compiled.\nINFO:     dc.cpp(init:573): Cluster of 1 instances created.\nINFO:     distributed_graph.hpp(set_ingress_method:3200): Automatically determine ingress method: grid\nLoading graph in format: adj\nINFO:     distributed_graph.hpp(load_from_posixfs:2189): Loading graph from file: ./out_1_of_4\nINFO:     distributed_graph.hpp(load_from_posixfs:2189): Loading graph from file: ./out_2_of_4\nINFO:     distributed_graph.hpp(load_from_posixfs:2189): Loading graph from file: ./out_3_of_4\nINFO:     distributed_graph.hpp(load_from_posixfs:2189): Loading graph from file: ./out_4_of_4\nINFO:     distributed_ingress_base.hpp(finalize:185): Finalizing Graph...\nINFO:     distributed_ingress_base.hpp(exchange_global_info:519): Graph info:\n\t nverts: 2\n\t nedges: 0\n\t nreplicas: 2\n\t replication factor: 1\nComplete Finalization in 0.001965\ngraph calculation time is 2.4e-05 sec\nRESULT:\nsize\tcount\n3\t2\n\\endverbatim\n\nAs expected, there are two components of size 3.\n\n\n\n\\section graph_analytics_approximate_diameter Approximate Diameter\n\nThe approximate diameter program can estimate a diameter of a graph. \nThe implemented algorithm is based on the work, \n\nU Kang, Charalampos Tsourakakis, Ana Paula Appel, Christos Faloutsos and Jure Leskovec, \nHADI: Fast Diameter Estimation and Mining in Massive Graphs with Hadoop (2008).\n\nThe input to the system is a graph in any of the Portable Graph formats\ndescribed in \\ref graph_formats.\n\nTo compute an approximate diameter of a graph, the minimal set of options\nrequired are:\n\\verbatim\n> ./approximate_diameter --graph=[graph prefix] --format=[format]\n\\endverbatim\nOutput looks like:\n\\verbatim\nApproximate graph diameter\nINFO:     synchronous_engine.hpp(start:1263): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1312):   Active vertices: 1271950\nINFO:     synchronous_engine.hpp(start:1361):    Running Aggregators\n1-th hop: 12895307 vertex pairs are reached\nINFO:     synchronous_engine.hpp(start:1263): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1312):   Active vertices: 1271950\nINFO:     synchronous_engine.hpp(start:1361):    Running Aggregators\n2-th hop: 319726269 vertex pairs are reached\nINFO:     synchronous_engine.hpp(start:1263): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1312):   Active vertices: 1271950\nINFO:     synchronous_engine.hpp(start:1361):    Running Aggregators\n3-th hop: 319769151 vertex pairs are reached\nconverge\ngraph calculation time is 40 sec\napproximate diameter is 2\n\\endverbatim\n\nThis program can also run distributed by using\n\\verbatim\n> mpiexec -n [N machines] --hostfile [host file] ./approximate_diameter ....\n\\endverbatim\nSee your MPI documentation for details on how to launch this job.\nAll machines must have access to the input graph location and the output graph \nlocation. Graphs may be on HDFS. \nIf you have problems loading HDFS files, see the \\ref FAQ.\n\n\\subsection Options\nRelevant options are: \n\\li \\b --graph (Required). The prefix from which to load the graph data\n\\li \\b --format (Required). The format of the input graph \n\\li \\b --tol (Optional. Default=1E-4). Changes the convergence tolerance for \nthe number of reached vertex pairs at each hop.\n\\li \\b --use-sketch (Optional. Default=1). If true, will use Flajolet & Martin \nbitmask to approximately count numbers of reached vertex pairs, and will require a \nsmaller memory. If false, will count exact numbers of reached vertex pairs. But \nthis will need a huge memory and be slow.\n\\li \\b --ncpus (Optional. Default 2). The number of processors that will be used\nfor computation.  \n\\li \\b --graph_opts (Optional, Default empty). Any additional graph options. See\n  graphlab::distributed_graph a list of options.\n\n\n\n\n\n\n\n\n\\section graph_analytics_partitioning Graph Partitioning \n\nThis program can partition a graph by using normalized cut.\n\nThe input to the system is a graph in any of the Portable Graph formats\ndescribed in \\ref graph_formats.\nYou can also give weights to edges with the weight format.\nFor instance in this weight format file, there are 5 edges:\n\n\\verbatim\n1 2 4.0\n2 3 1.0\n3 4 5.0\n4 5 2.0\n5 3 3.0\n\\endverbatim\n\nTo partition a graph, the minimal set of options required are:\n\n\\verbatim\n> ./partitioning --graph=[graph prefix] --format=[format]\n\\endverbatim\n\nThis program uses svd in Graphlab Collaborative Filtering Toolkit and \nkmeans in Graphlab Clustering Toolkit. The paths to the directories are \nspecified by --svd-dir and --kmeans-dir, respectively.\n\nThe program will create some intermediate files. The final partitioning\nresult is written in files named [graph prefix].result with suffix,\nfor example [graph prefix].result_1_of_4. The partitioning result \ndata consists of two columns: one for the ids and the other for the \nassigned partitions. For instance:\n\n\\verbatim\n1 0\n2 0\n3 1\n4 1\n5 1\n\\endverbatim\n\nNOTE: To run this program in a distributed setting, you must use the \n\"mpi-args\" option, not like other graphlab toolkits. \nThe graph partitioning calls other graphlab programs.\nWhen \"--mpi-args\" is set, these graphlab programs are called with \"mpiexec\" and the \nstring written after the \"mpi-args\" option.\nFor example, if you set --mpi-args=\"-n 4 --hostfile host\", the program calls the \nother graphlab programs with \"mpiexec -n 4 --hostfile host\".\n\n\\subsection Options\nRelevant options are:\n\\li \\b --graph (Required). The prefix from which to load the graph data\n\\li \\b --format (Required). The format of the input graph. If \"weight\" is \nset, the program will read the data file where each line holds [id1] [id2] \n[weight].\n\\li \\b --partitions (Optional. Default 2). The number of partitions\n\\li \\b --svd-dir (Optional. Default ../collaborative_filtering/).\nPath to the directory where Graphlab svd is located\n\\li \\b --kmeans-dir (Optional. Default ../clustering/). Path to the directory where \nGraphlab kmeans is located\n\\li \\b --ncpus (Optional. Default 2). The number of processors that will be used \nfor computation.\n\\li \\b --graph_opts (Optional, Default empty). Any additional graph options. See\n  graphlab::distributed_graph a list of options.\n\\li \\b --mpi-args (Optional, Default empty). If set, will execute mipexec with the given string.\n  \n  \n\\section graph_analytics_total_subgraph_centrality \"Total Subgraph Centrality\"\nTotal subgraph centrality was implemented by Jacob Kesinger, see additional\ndetails in his blog post.\nTotal Subgraph Communicability is a new centrality measure due to\nBenzi&Klymco [1].   For a directed graph with adjacenty matrix A, \n\n\\verbatim\nTSC_i = sum_j exp(A)_{ij} = (exp(A)*1)_i.\n\\endverbatim\n\nThis code calculates the TSC using an Arnoldi iteration on the Krylov\nsubspace {b, Ab,A*Ab, A*A*Ab, ...}  due to Saad[1], and using the new\nwarp engine from Graphlab 2.2 (without which this would have been, at\nbest, very challenging).\n\nSmall components of large graphs will have bogus answers due to\nfloating point issues.  To find the exact TSC for a particular node i,\nrun with \"--column i\" to find exp(A)*e_i; you will have to sum the\nresulting output yourself, however.\n\n\nSAMPLE INPUT:\n\\verbatim\n0\t1\n1\t2\n1\t3\n2\t4\n3\t4\n1\t0\n2\t1\n3\t1\n4\t2\n4\t3\n\\endverbatim\n\nOUTPUT:\n\\verbatim\n0 5.17784\n1 10.3319\n2 8.49789\n3 8.49789\n4 7.96807\n\\endverbatim\n\nYou can verify this in python as:\n\\verbatim\nimport scipy\nimport scipy.linalg\nA = scipy.array([[0,1,0,0,0],[1,0,1,1,0],[0,1,0,0,1],[0,1,0,0,1],[0,0,1,1,0]])\nscipy.linalg.expm2(A).sum(axis=1)\n\\endverbatim\n\n[1]: Benzi, Michele, and Christine Klymko. Total Communicability as a Centrality Measure. ArXiv e-print, February 27, 2013. arxiv\n\n[2]: Saad, Yousef. “Analysis of Some Krylov Subspace Approximations to the Matrix Exponential Operator.” SIAM Journal on Numerical Analysis 29, no. 1 (1992): 209–228.\n*/\n"
  },
  {
    "path": "toolkits/graph_algorithms/prestige.cpp",
    "content": "/*\n * Copyright (c) 2014 Daniel McEnnis.\n * portions of main Copyright (c) 2009 Carnegie Mellon\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n#include \n#include \n\n/*\n * Djikstra Graph Node Class\n *\n * This class contains the information about a single graphlab node.\n * id - current best path's previous node id - next node on path to root\n * cost - current cost of the path to route by the current route: Note - this\n *       can become inaccurate in the course of calculations and must be recalculated\n *       by traversing the shortest path tree to get an accurate result.\n * launched - has execution of this node been sheduled\n * done - has execution of this node been completed\n */\n\nclass DjikstraNode {\npublic:\n    unsigned long id;\n    double cost;\n    bool launched;\n    bool done;\n\n    DjikstraNode(){\n        id = 0;\n        cost = 1e100;\n        launched = false;\n        done=false;\n    }\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << id << cost << launched << done;\n  }\n\n  void load(graphlab::iarchive& iarc) {\n    iarc >> id >> cost >> launched >> done;\n  }\n};\n\n/*\n * PrestigeAnalysisNode\n * Graph Node class for running multiple djikstra tree algorithms simultaneously\n * Contains a map of node id's to DjikstraNode instances\n * bookkeeping components\n *\n */\nclass PrestigeAnalysisNode {\npublic:\n    std::map djikstra_pieces;\n    double local_value;\n    double total;\n    long count;\n    int edge_count;\n\n    PrestigeAnalysisNode(){\n        local_value=0.0;\n        total=0.0;\n        count=0;\n        edge_count=-1;\n    }\n\n    void save(graphlab::oarchive& oarc) const {\n        oarc << djikstra_pieces << local_value << total << count << edge_count;\n    }\n\n    void load(graphlab::iarchive& iarc) {\n        iarc >> djikstra_pieces >> local_value >> total >> count >> edge_count;\n    }\n};\n\n/*\n * Gather class for the Djikstra algorithm.\n * id: node id of the incoming edge's other end\n * cost: shortest path cost at the time this node gathers its edges\n * edge_count: a count of gathered edges\n *\n */\nclass Gather {\npublic:\n    long id;\n    double cost;\n    int edge_count;\n\n    Gather(){\n        id=0;\n        cost=0.0;\n        edge_count=1;\n    }\n\n    Gather& operator+=(const Gather& other){\n        if(other.id < 0){\n            return *this;\n        }\n        if(this->id < 0){\n            return *this;\n        }\n        if (cost <= other.cost){\n            this->edge_count++;\n            return *this;\n        }\n        this->edge_count += other.edge_count;\n        return *this;\n    }\n\n\n    void save(graphlab::oarchive& oarc) const {\n       oarc << id << cost << edge_count;\n     }\n\n     void load(graphlab::iarchive& iarc) {\n       iarc >> id >> cost >> edge_count;\n     }\n\n};\n\n/*\n * GatherMultiTree\n * map of djisktra root id's to their asociated content for that tree\n *\n */\nclass GatherMultiTree {\npublic:\n    std::map content;\n    int edge_count;\n\n    GatherMultiTree(){\n        edge_count=0;\n    }\n\n    GatherMultiTree& operator+=(const GatherMultiTree& other){\n        return *this;\n    }\n\n    void save(graphlab::oarchive& oarc) const {\n        oarc << content << edge_count;\n    }\n\n    void load(graphlab::iarchive& iarc) {\n        iarc >> content >> edge_count;\n    }\n};\n\ntypedef PrestigeAnalysisNode vertex_data_type;\n\ntypedef GatherMultiTree gather_type;\n\n// The graph type is determined by the vertex and edge data types\ntypedef graphlab::distributed_graph graph_type;\n\n/*\n * Loads graphs in the form 'id (id edge_strength)*'\n *\n */\nbool line_parser(graph_type& graph, const std::string& filename, const std::string& textline) {\n  std::stringstream strm(textline);\n  graphlab::vertex_id_type vid;\n  // first entry in the line is a vertex ID\n  strm >> vid;\n  PrestigeAnalysisNode node;\n  // insert this vertex with its label\n  graph.add_vertex(vid, node);\n  // while there are elements in the line, continue to read until we fail\n  double edge_val=1.0;\n  while(1){\n    graphlab::vertex_id_type other_vid;\n    strm >> other_vid;\n    strm >> edge_val;\n    if (strm.fail())\n      break;\n    graph.add_edge(vid, other_vid,edge_val);\n  }\n\n  return true;\n}\n\n/*\n * Algorithm class whose sole purpose is to reset launched and done booleans\n * for all id's in a PrestigeAnalysisNode\n */\nclass ClearBooleans :\n        public graphlab::ivertex_program,\n        public graphlab::IS_POD_TYPE {\npublic:\n  edge_dir_type gather_edges(icontext_type& context, const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n\n  gather_type gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n        GatherMultiTree g;\n        return g;\n  }\n\n  void apply(icontext_type& context, vertex_type& vertex, const gather_type& total) {\n      for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n          iter != vertex.data().djikstra_pieces.end(); ++iter){\n          long key = iter->first;\n          vertex.data().djikstra_pieces[key].launched = false;\n          vertex.data().djikstra_pieces[key].done = false;\n          vertex.data().djikstra_pieces[key].cost = 0.0;\n      }\n  }\n\n  edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n          return graphlab::NO_EDGES;\n  }\n\n  void scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n  }\n};\n\n\n/*\n * Djikstra Algorithm Class\n *\n * Starting from the starting nodes, create an id for this root and signal\n * all neighbors to start the calculations. Set launched when started, done\n * when all edges have been signaled.\n *\n * As a signal is receieved collect edges to determine if the best path has\n * changed.  If it has, update. If the first signal is receieved, marked\n * the node as launched and then mark the node done after signaling neighbors.\n *\n * The process terminates when all nodes active have no neighbors that are not done.\n */\nclass DjikstraAlgorithm :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\n    bool changed;\n\n  public:\n    edge_dir_type gather_edges(icontext_type& context, const vertex_type& vertex) const {\n      return graphlab::IN_EDGES;\n    }\n\n    gather_type gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n    Gather g;\n    GatherMultiTree tree;\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key=iter->first;\n            if((edge.source().data().djikstra_pieces[key].launched == true)&&\n                    (edge.source().data().djikstra_pieces[key].done == false)){\n                double c = edge.data() + edge.source().data().djikstra_pieces[key].cost;\n                g.cost = c;\n                g.id = edge.source().data().djikstra_pieces[key].id;\n                g.edge_count = 1;\n                tree.content[key] = g;\n            }else{\n                g.id=0;\n            }\n        }\n\treturn tree;\n    }\n\n    void apply(icontext_type& context, vertex_type& vertex, const gather_type& total) {\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key = iter->first;\n            if(vertex.data().djikstra_pieces[key].launched == false){\n                vertex.data().djikstra_pieces[key].launched = true;\n                vertex.data().edge_count = total.edge_count;\n                    if(vertex.data().djikstra_pieces[key].cost > total.content.find(key)->second.cost){\n                    vertex.data().djikstra_pieces[key].cost = total.content.find(key)->second.cost;\n                    vertex.data().djikstra_pieces[key].id = total.content.find(key)->second.id;\n                    }else{\n                    vertex.data().djikstra_pieces[key].done = true;\n                    }\n            }else{\n                vertex.data().djikstra_pieces[key].done = true;\n            }\n        }\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key = iter->first;\n            if(vertex.data().djikstra_pieces.find(key)==vertex.data().djikstra_pieces.end()){\n                vertex.data().djikstra_pieces[key].launched = true;\n                vertex.data().edge_count = total.edge_count;\n                vertex.data().djikstra_pieces[key].cost = total.content.find(key)->second.cost;\n                vertex.data().djikstra_pieces[key].id = total.content.find(key)->second.id;\n            }\n        }\n    }\n\n    edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n      // if vertex data changes, scatter to all edges.\n        bool done = true;\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key = iter->first;\n            if(vertex.data().djikstra_pieces.find(key)->second.launched &&\n                    !vertex.data().djikstra_pieces.find(key)->second.done){\n                done = false;\n            }\n        }\n        if(!done){\n            return graphlab::OUT_EDGES;\n        }else{\n            return graphlab::NO_EDGES;\n        }\n    }\n\n    void scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n    for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n        iter != vertex.data().djikstra_pieces.end(); ++iter){\n        long key = iter->first;\n        if((vertex.data().djikstra_pieces.find(key)->second.done == false) &&\n                (vertex.data().djikstra_pieces.find(key)->second.launched == true)){\n                context.signal(edge.target());\n            }\n    }\n  }\n};\n\nsize_t num_vertices = 3000;\nsize_t desired_sample_size = 3000;\nsize_t selected_sample_size = 0;\n\n/*\n * For every node, print the previous node in its spanning tree for all spanning trees this node is in.\n *\n */\nstruct djikstra_writer {\n  std::string save_vertex(graph_type::vertex_type v) {\n    std::stringstream strm;\n    strm << v.id() << \"\\t\";\n    double value = 0.0;\n    for(std::map::const_iterator iter = v.data().djikstra_pieces.begin();\n        iter != v.data().djikstra_pieces.end();++iter){\n        value += iter->second.cost;\n    }\n    value /= selected_sample_size;\n    strm << value << std::endl;\n    return strm.str();\n  }\n  std::string save_edge (graph_type::edge_type e) { return \"\"; }\n};\n\n/*\n * Select ~3000 root nodes or an exact count which gives up around +/-3% accuracy\n * in prestige measures. It is a constant memory random selector.\n */\nbool selectVertices(const graph_type::vertex_type& vertex){\n    unsigned int r = random();\n//    std::cout << \"Random seed is \" << r << std::endl;\n    if(r < (desired_sample_size * RAND_MAX / num_vertices)){\n        selected_sample_size++;\n        return true;\n    }\n    return false;\n}\n\n\n/*\n * Collect shortest path cost while traversing the spanning tree.\n * Every spanning tree is collected by default.  The algorithm is responsible\n * for filtering out entries where the collecting node is not next in the\n * spanning tree.\n */\nclass PrestigeGather{\npublic:\n    std::map counts;\n    std::map edge_count;\n\t\n void save(graphlab::oarchive& oarc) const {\n    oarc << counts << edge_count;\n  }\n\n  void load(graphlab::iarchive& iarc) {\n    iarc >> counts >> edge_count;\n  }\n\n  PrestigeGather& operator+=(const PrestigeGather& gather){\n    for(std::map::const_iterator iter = this->counts.begin();\n        iter != this->counts.end(); ++iter ){\n                long key = iter->first;\n                this->counts[key] += gather.counts.find(key)->second;\n                this->edge_count[key] += gather.edge_count.find(key)->second;\n\t}\n    for(std::map::const_iterator iter = gather.counts.begin();\n            iter != gather.counts.end(); ++iter){\n                long key = iter->first;\n                if(this->counts.find(key) != this->counts.end()){\n                        this->counts[key] = gather.counts.find(key)->second;\n                        this->edge_count[key] = gather.edge_count.find(key)->second;\n                }\n    }\n    return *this;\n  }\n\n\n};\n\n/*\n * Walk the spanning trees of a randomly selected subset of djikstra trees,\n * recording the shortest path cost at each node.  The sum represents a\n * sample that is within +/-3% of the real prestige score. Summing the scores\n * is done during output and must be independently aggregated if the graph is\n * not outputted.\n */\nclass PrestigeAlgorithm :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\n    bool changed;\n\n  public:\n    edge_dir_type gather_edges(icontext_type& context, const vertex_type& vertex) const {\n      return graphlab::IN_EDGES;\n    }\n\n    PrestigeGather gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n    PrestigeGather g;\n    for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n        iter != vertex.data().djikstra_pieces.end(); ++iter){\n        long key= iter->first;\n        if(edge.target().data().djikstra_pieces[key].id == vertex.id()){\n            if(edge.source().data().djikstra_pieces[key].launched == true){\n                g.counts[key] = edge.source().data().djikstra_pieces[key].cost + edge.data();\n                g.edge_count[key] = 1;\n            }\n        }\n\t}\n    return g;\n    }\n\n    void apply(icontext_type& context, vertex_type& vertex, const PrestigeGather& total) {\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n                iter != vertex.data().djikstra_pieces.end(); ++iter){\n            long key = iter->first;\n            if(total.edge_count.find(key)->second==0){\n                vertex.data().djikstra_pieces[key].launched = true;\n            }\n            if((vertex.data().djikstra_pieces[key].launched == true)&&\n                    (vertex.data().djikstra_pieces[key].done == false)&&\n                    (((long)vertex.data().djikstra_pieces[key].cost)==total.edge_count.find(key)->second)){\n                vertex.data().djikstra_pieces[key].done = true;\n                vertex.data().djikstra_pieces[key].cost = (double)total.edge_count.find(key)->second;\n            }\n        }\n    }\n\n    edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const {\n      // if vertex data changes, scatter to all edges.\n     \tbool done = true;\n    for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n        iter != vertex.data().djikstra_pieces.end(); ++iter){\n        long key = iter->first;\n        if(vertex.data().djikstra_pieces.find(key)->second.launched && !vertex.data().djikstra_pieces.find(key)->second.done){\n        \t\tdone = false;\n      \t\t}\n        }\n        if(!done){\n                return graphlab::OUT_EDGES;\n        }else{\n                return graphlab::NO_EDGES;\n        }\n    }\n\n    void scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const {\n        for(std::map::const_iterator iter = vertex.data().djikstra_pieces.begin();\n            iter != vertex.data().djikstra_pieces.end(); ++iter){\n                long key = iter->first;\n                if((vertex.data().djikstra_pieces.find(key)->second.done == false) &&\n                        (vertex.data().djikstra_pieces.find(key)->second.launched == true)){\n                        context.signal(edge.target());\n                }\n        }\n    }\n};\n\nint main (int argc, char** argv){\n    // Initialize control plain using mpi\n    graphlab::mpi_tools::init(argc, argv);\n    graphlab::distributed_control dc;\n    global_logger().set_log_level(LOG_INFO);\n\n    // Parse command line options -----------------------------------------------\n    graphlab::command_line_options clopts(\"Prestige Algorithm\");\n    std::string graph_dir;\n    clopts.attach_option(\"graph\", graph_dir, \"The graph file. Required \");\n    clopts.add_positional(\"graph\");\n    clopts.attach_option(\"samplesize\", desired_sample_size, \"(Sample size) the number of spanning trees to calculate\");\n\n    std::string saveprefix;\n    clopts.attach_option(\"saveprefix\", saveprefix,\n                         \"If set, will save the resultant prestige score to a \"\n                         \"sequence of files with prefix saveprefix\");\n\n    if(!clopts.parse(argc, argv)) {\n      dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n      return EXIT_FAILURE;\n    }\n    if (graph_dir == \"\") {\n      dc.cout() << \"Graph not specified. Cannot continue\";\n      return EXIT_FAILURE;\n    }\n\n    // Build the graph ----------------------------------------------------------\n    graph_type graph(dc);\n    dc.cout() << \"Loading graph using line parser\" << std::endl;\n    graph.load(graph_dir, line_parser);\n\n    dc.cout() << \"#vertices: \" << graph.num_vertices() << \" #edges:\" << graph.num_edges() << std::endl;\n\n    graphlab::omni_engine engine(dc, graph, \"asynchronous\", clopts);\n\n    num_vertices = graph.num_vertices();\n    graphlab::vertex_set start_set = graph.select(selectVertices);\n    engine.signal_vset(start_set);\n    engine.start();\n\n    const float runtime = engine.elapsed_seconds();\n    dc.cout() << \"Finished Djikstra engine in \" << runtime << \" seconds.\" << std::endl;\n\n    graphlab::omni_engine engine2(dc,graph,\"asynchronous\",clopts);\n    engine2.signal_all();\n    engine2.start();\n\n    const float runtime2 = engine.elapsed_seconds();\n    dc.cout() << \"Finished graph reset in \" << runtime2 << \" seconds.\" << std::endl;\n\n    graphlab::omni_engine engine3(dc,graph,\"asynchronous\",clopts);\n    engine3.signal_vset(start_set);\n    engine3.start();\n\n    const float runtime3 = engine.elapsed_seconds();\n    dc.cout() << \"Finished prestige calculations in \" << runtime3 << \" seconds.\" << std::endl;\n\n    if (saveprefix != \"\") {\n      graph.save(saveprefix, djikstra_writer(),\n         false,  // do not gzip\n         true,   //save vertices\n         false); // do not save edges\n    }\n\n\n    graphlab::mpi_tools::finalize();\n    return EXIT_SUCCESS;\n}\n\n\n"
  },
  {
    "path": "toolkits/graph_analytics/CMakeLists.txt",
    "content": "project(GraphProcessing)\n\nadd_graphlab_executable(simple_undirected_triangle_count simple_undirected_triangle_count)\nadd_graphlab_executable(undirected_triangle_count undirected_triangle_count.cpp)\nadd_graphlab_executable(directed_triangle_count directed_triangle_count.cpp)\nadd_graphlab_executable(pagerank pagerank.cpp)\nadd_graphlab_executable(kcore kcore.cpp)\nadd_graphlab_executable(format_convert format_convert.cpp)\nadd_graphlab_executable(sssp sssp.cpp)\nadd_graphlab_executable(simple_coloring simple_coloring.cpp)\nadd_graphlab_executable(degree_ordered_coloring degree_ordered_coloring.cpp)\nadd_graphlab_executable(saturation_ordered_coloring saturation_ordered_coloring.cpp)\nadd_graphlab_executable(connected_component connected_component.cpp)\nadd_graphlab_executable(connected_component_stats connected_component_stats.cpp)\nadd_graphlab_executable(approximate_diameter approximate_diameter.cpp)\nadd_graphlab_executable(eigen_vector_normalization eigen_vector_normalization.cpp)\nadd_graphlab_executable(graph_laplacian graph_laplacian.cpp)\nadd_graphlab_executable(partitioning partitioning.cpp)\n\n# add_graphlab_executable(warp_pagerank warp_pagerank.cpp)\n# add_graphlab_executable(warp_pagerank2 warp_pagerank2.cpp)\n# add_graphlab_executable(warp_coloring warp_coloring.cpp)\n# add_graphlab_executable(warp_bond_percolation warp_bond_percolation.cpp)\n# add_graphlab_executable(warp_pagerank_sweeps warp_pagerank_sweeps.cpp)\n\nadd_graphlab_executable(TSC TSC.cpp)\nrequires_eigen(TSC)\n"
  },
  {
    "path": "toolkits/graph_analytics/TSC.cpp",
    "content": "#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include  \n#include  \n#include \n\n#include \n\nint verbose;\nstd::vector Hleft;\n//#include \n\n/*\n\n  Total Subgraph Centrality.\n\n  For a graph G with adjacency matrix A, TSC(G) = exp(A)*b, where 1 is\n  the ones vector.\n\n\n  We're going to implement this with an Arnoldi solver, following Saad\n  (1992).\n\n  The algorithm works like this:\n  Choose a maximum iteration m.  Then make matrices V and H:\n  b = ones(A.nodecount)\n  V[0] = b/||b||\n  for j in 0..m:\n      w = A*V[j]\n      for i in 0..j:\n         H[i,j] = (w,V[i])\n\t w = w - H[i,j] * V[i]\n      H[j+1,j] = ||w||\n      V[j+1] =w/||w||\n\n  Then TSC = exp(A)*b ~= (V * exp(H) / ||b||)[:,0].  Stop when\n  successive approximations converge, or we run out of steps.  \n\n  We still have that matrix exponential, but it's small and dense, and\n  there's an implementation in Eigen. \n  \n\n  \n*/\n\nclass node {\npublic: \nstd::vector V;\ndouble w;\n  double TSC;\n  double prev;\n  node(): w(0.0),TSC(0.0),prev(0.0){};\n  void load(graphlab::iarchive& infile) {\n    infile>>V>>w>>TSC>>prev;\n  }\n  void save(graphlab::oarchive& outfile) const {\n    outfile<>weight; \n  }\n  \n  \n};\n\n\ntypedef node vertex_data_type;\ntypedef edge edge_data_type;\ntypedef graphlab::distributed_graph graph_type; \ntypedef graphlab::warp::warp_engine engine_type; \n\n\n// This is just a little class to be used to find the maximum change in TSC values. \nclass max_finder{\npublic:\n  double data;\n  max_finder& operator+=(const max_finder& other){\n    if (this->data data = other.data;\n    }\n    return *this;\n  }\n  \n  max_finder(): data(std::numeric_limits::max()) {};\n  max_finder(double x): data(x) {};\n  void load(graphlab::iarchive& infile) {\n    infile>>data;\n  }\n  void save(graphlab::oarchive& outfile) const {\n    outfile<graph.num_vertices()) {\n    m = graph.num_vertices();\n  }\n  engine_type engine(dc,graph,clopts);\n\n  engine.signal_all();\n  Eigen::MatrixXd H = Eigen::MatrixXd::Zero(m+1,m+1);\n  Hleft.resize(m);\n  for(int i=0;i=0) {\n    if (column>graph.num_vertices()){\n      column = 0;\n    }\n    graph.transform_vertices(boost::bind(initialize_column,_1,column,m));\n    beta = 1.0;\n  } else {\n    graph.transform_vertices(boost::bind(initialize_TSC,_1,m));\n    beta = sqrt(m);\n  }\n\n\n  // The first column of V is just w\n  graph.transform_vertices(w_to_v);\n\n\n  for(int j=0;j(boost::bind(w_dot_V,_1,i));\n      graph.transform_vertices(boost::bind(w_minus_hdot,_1,i,H(i,j)));\n    }\n    H(j+1,j)= sqrt(graph.map_reduce_vertices(sum_w));\n    // If we're in this case, it means we have a spanning set and we shouldn't\n    // prep for the next iteration.  Otherwise, make the data we'll need next time.\n    if ( ( !std::isnan(H(j+1,j))) && (H(j+1,j) > 0) ) {\n      graph.transform_vertices(boost::bind(scale_w, _1, H(j+1,j)));\n      graph.transform_vertices(w_to_v); \n    }\n\n    if (j>0) {\n      // Have we converged? \n      Eigen::MatrixXd EH(H);\n      EH = EH.exp();\n      Hleft.resize(j+1);\n      \n      for(int i=0;i<=j;i++) { Hleft[i] = EH(i,0) * beta;}\n      graph.transform_vertices(accumulate_hleft);\n      if (j>1) {\n \tmax_finder largest_error= graph.map_reduce_vertices(max_error);\n\tdouble all_error = graph.map_reduce_vertices(total_error);\n\tif (verbose){\n\t  std::cerr<<\"ARNOLDI STEP FINISHED \"<\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\n//helper function\nfloat myrand() {\n  return static_cast(rand() / (RAND_MAX + 1.0));\n}\n\n//helper function to return a hash value for Flajolet & Martin bitmask\nsize_t hash_value() {\n  size_t ret = 0;\n  while (myrand() < 0.5) {\n    ret++;\n  }\n  return ret;\n}\n\nconst size_t DUPULICATION_OF_BITMASKS = 10;\n\nstruct vdata {\n  //use two bitmasks for consistency\n  std::vector > bitmask1;\n  std::vector > bitmask2;\n  //indicate which is the bitmask for reading (or writing)\n  bool odd_iteration;\n  vdata() :\n      bitmask1(), bitmask2(), odd_iteration(true) {\n  }\n  //for exact counting (but needs large memory)\n  void create_bitmask(size_t id) {\n    std::vector mask1(id + 2, 0);\n    mask1[id] = 1;\n    bitmask1.push_back(mask1);\n    std::vector mask2(id + 2, 0);\n    mask2[id] = 1;\n    bitmask2.push_back(mask2);\n  }\n  //for approximate Flajolet & Martin counting\n  void create_hashed_bitmask(size_t id) {\n    for (size_t i = 0; i < DUPULICATION_OF_BITMASKS; ++i) {\n      size_t hash_val = hash_value();\n      std::vector mask1(hash_val + 2, 0);\n      mask1[hash_val] = 1;\n      bitmask1.push_back(mask1);\n      std::vector mask2(hash_val + 2, 0);\n      mask2[hash_val] = 1;\n      bitmask2.push_back(mask2);\n    }\n  }\n\n  void save(graphlab::oarchive& oarc) const {\n    size_t num = bitmask1.size();\n    oarc << num;\n    for (size_t a = 0; a < num; ++a) {\n      size_t size = bitmask1[a].size();\n      oarc << size;\n      for (size_t i = 0; i < size; ++i)\n        oarc << (bool)bitmask1[a][i];\n      for (size_t i = 0; i < size; ++i)\n        oarc << (bool)bitmask2[a][i];\n    }\n    oarc << odd_iteration;\n  }\n  void load(graphlab::iarchive& iarc) {\n    bitmask1.clear();\n    bitmask2.clear();\n    size_t num = 0;\n    iarc >> num;\n    for (size_t a = 0; a < num; ++a) {\n      size_t size = 0;\n      iarc >> size;\n      std::vector mask1;\n      for (size_t i = 0; i < size; ++i) {\n        bool element = true;\n        iarc >> element;\n        mask1.push_back(element);\n      }\n      bitmask1.push_back(mask1);\n      std::vector mask2;\n      for (size_t i = 0; i < size; ++i) {\n        bool element = true;\n        iarc >> element;\n        mask2.push_back(element);\n      }\n      bitmask2.push_back(mask2);\n    }\n    iarc >> odd_iteration;\n  }\n};\n\ntypedef graphlab::distributed_graph graph_type;\n\n//initialize bitmask\nvoid initialize_vertex(graph_type::vertex_type& v) {\n  v.data().create_bitmask(v.id());\n}\n//initialize bitmask\nvoid initialize_vertex_with_hash(graph_type::vertex_type& v) {\n  v.data().create_hashed_bitmask(v.id());\n}\n\n//helper function to compute bitwise-or\nvoid bitwise_or(std::vector >& v1,\n    const std::vector >& v2) {\n  for (size_t a = 0; a < v1.size(); ++a) {\n    while (v1[a].size() < v2[a].size()) {\n      v1[a].push_back(false);\n    }\n    for (size_t i = 0; i < v2[a].size(); ++i) {\n      v1[a][i] = v1[a][i] || v2[a][i];\n    }\n  }\n}\n\nstruct bitmask_gatherer {\n  std::vector > bitmask;\n\n  bitmask_gatherer() :\n    bitmask() {\n  }\n  explicit bitmask_gatherer(const std::vector > & in_b) :\n    bitmask(){\n    for(size_t i=0;i> num;\n    for (size_t a = 0; a < num; ++a) {\n      size_t size = 0;\n      iarc >> size;\n      std::vector mask1;\n      for (size_t i = 0; i < size; ++i) {\n        bool element = true;\n        iarc >> element;\n        mask1.push_back(element);\n      }\n      bitmask.push_back(mask1);\n    }\n  }\n};\n\n//The next bitmask b(h + 1; i) of i at the hop h + 1 is given as:\n//b(h + 1; i) = b(h; i) BITWISE-OR {b(h; k) | source = i & target = k}.\nclass one_hop: public graphlab::ivertex_program,\n    public graphlab::IS_POD_TYPE {\npublic:\n  //gather on out edges\n  edge_dir_type gather_edges(icontext_type& context,\n      const vertex_type& vertex) const {\n    return graphlab::OUT_EDGES;\n  }\n\n  //for each edge gather the bitmask of the edge\n  bitmask_gatherer gather(icontext_type& context, const vertex_type& vertex,\n      edge_type& edge) const {\n    if (vertex.data().odd_iteration) {\n      return bitmask_gatherer(edge.target().data().bitmask2);\n    } else {\n      return bitmask_gatherer(edge.target().data().bitmask1);\n    }\n  }\n\n  //get bitwise-ORed bitmask and switch bitmasks\n  void apply(icontext_type& context, vertex_type& vertex,\n      const gather_type& total) {\n    if (vertex.data().odd_iteration) {\n      if (total.bitmask.size() > 0)\n        bitwise_or(vertex.data().bitmask1, total.bitmask);\n      vertex.data().odd_iteration = false;\n    } else {\n      if (total.bitmask.size() > 0)\n        bitwise_or(vertex.data().bitmask2, total.bitmask);\n      vertex.data().odd_iteration = true;\n    }\n  }\n\n  edge_dir_type scatter_edges(icontext_type& context,\n      const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n  void scatter(icontext_type& context, const vertex_type& vertex,\n      edge_type& edge) const {\n  }\n};\n\n//copy the updated bitmask to the other\nvoid copy_bitmasks(graph_type::vertex_type& vdata) {\n  if (vdata.data().odd_iteration == false) { //odd_iteration has just finished\n    vdata.data().bitmask2 = vdata.data().bitmask1;\n  } else {\n    vdata.data().bitmask1 = vdata.data().bitmask2;\n  }\n}\n\n//count the number of vertices reached in the current hop\nsize_t absolute_vertex_data(const graph_type::vertex_type& vertex) {\n    size_t count = 0;\n    for (size_t i = 0; i < vertex.data().bitmask1[0].size(); ++i)\n      if (vertex.data().bitmask1[0][i])\n        count++;\n    return count;\n}\n\n//count the number of vertices reached in the current hop with Flajolet & Martin counting method\nsize_t approximate_pair_number(std::vector > bitmask) {\n  float sum = 0.0;\n  for (size_t a = 0; a < bitmask.size(); ++a) {\n    for (size_t i = 0; i < bitmask[a].size(); ++i) {\n      if (bitmask[a][i] == 0) {\n        sum += (float) i;\n        break;\n      }\n    }\n  }\n  return (size_t) (pow(2.0, sum / (float) (bitmask.size())) / 0.77351);\n}\n//count the number of notes reached in the current hop\nsize_t absolute_vertex_data_with_hash(\n    const graph_type::vertex_type& vertex) {\n    size_t count = approximate_pair_number(vertex.data().bitmask1);\n    return count;\n}\n\nint main(int argc, char** argv) {\n  std::cout << \"Approximate graph diameter\\n\\n\";\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n  std::string datafile;\n  float termination_criteria = 0.0001;\n  //parse command line\n  graphlab::command_line_options clopts(\n                \"Approximate graph diameter. \"\n                \"Directions of edges are considered.\");\n  std::string graph_dir;\n  std::string format = \"adj\";\n  bool use_sketch = true;\n  std::string exec_type = \"synchronous\";\n  clopts.attach_option(\"graph\", graph_dir,\n                       \"The graph file. This is not optional\");\n  clopts.add_positional(\"graph\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph file format\");\n  clopts.attach_option(\"tol\", termination_criteria,\n                       \"The permissible change at convergence.\");\n  clopts.attach_option(\"use-sketch\", use_sketch,\n                       \"If true, will use Flajolet & Martin bitmask, \"\n                       \"which is more compact and faster.\");\n\n  if (!clopts.parse(argc, argv)){\n    dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n  if (graph_dir == \"\") {\n    std::cout << \"--graph is not optional\\n\";\n    return EXIT_FAILURE;\n  }\n\n  //load graph\n  graph_type graph(dc, clopts);\n  dc.cout() << \"Loading graph in format: \"<< format << std::endl;\n  graph.load_format(graph_dir, format);\n  graph.finalize();\n\n  time_t start, end;\n  //initialize vertices\n  time(&start);\n  if (use_sketch == false)\n    graph.transform_vertices(initialize_vertex);\n  else\n    graph.transform_vertices(initialize_vertex_with_hash);\n\n  graphlab::omni_engine engine(dc, graph, exec_type, clopts);\n\n  //main iteration\n  size_t previous_count = 0;\n  size_t diameter = 0;\n  for (size_t iter = 0; iter < 100; ++iter) {\n    engine.signal_all();\n    engine.start();\n\n    graph.transform_vertices(copy_bitmasks);\n\n    size_t current_count = 0;\n    if (use_sketch == false)\n      current_count = graph.map_reduce_vertices(absolute_vertex_data);\n    else\n      current_count = graph.map_reduce_vertices(\n          absolute_vertex_data_with_hash);\n    dc.cout() << iter + 1 << \"-th hop: \" << current_count\n        << \" vertex pairs are reached\\n\";\n    if (iter > 0\n        && (float) current_count\n            < (float) previous_count * (1.0 + termination_criteria)) {\n      diameter = iter;\n      dc.cout() << \"converge\\n\";\n      break;\n    }\n    previous_count = current_count;\n  }\n  time(&end);\n\n  dc.cout() << \"graph calculation time is \" << (end - start) << \" sec\\n\";\n  dc.cout() << \"The approximate diameter is \" << diameter << \"\\n\";\n\n  graphlab::mpi_tools::finalize();\n\n  return EXIT_SUCCESS;\n}\n\n"
  },
  {
    "path": "toolkits/graph_analytics/connected_component.cpp",
    "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n\nstruct vdata {\n  uint64_t labelid;\n  vdata() :\n      labelid(0) {\n  }\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << labelid;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> labelid;\n  }\n};\n\ntypedef graphlab::distributed_graph graph_type;\n\n//set label id at vertex id\nvoid initialize_vertex(graph_type::vertex_type& v) {\n  v.data().labelid = v.id();\n}\n\n//message where summation means minimum\nstruct min_message {\n  uint64_t value;\n  explicit min_message(uint64_t v) :\n      value(v) {\n  }\n  min_message() :\n      value(std::numeric_limits::max()) {\n  }\n  min_message& operator+=(const min_message& other) {\n    value = std::min(value, other.value);\n    return *this;\n  }\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << value;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> value;\n  }\n};\n\nclass label_propagation: public graphlab::ivertex_program, public graphlab::IS_POD_TYPE {\nprivate:\n  size_t recieved_labelid;\n  bool perform_scatter;\npublic:\n  label_propagation() {\n    recieved_labelid = std::numeric_limits::max();\n    perform_scatter = false;\n  }\n\n  //receive messages\n  void init(icontext_type& context, const vertex_type& vertex,\n      const message_type& msg) {\n    recieved_labelid = msg.value;\n  }\n\n  //do not gather\n  edge_dir_type gather_edges(icontext_type& context,\n      const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n  size_t gather(icontext_type& context, const vertex_type& vertex,\n      edge_type& edge) const {\n    return 0;\n  }\n\n  //update label id. If updated, scatter messages\n  void apply(icontext_type& context, vertex_type& vertex,\n      const gather_type& total) {\n    if (recieved_labelid == std::numeric_limits::max()) {\n      perform_scatter = true;\n    } else if (vertex.data().labelid > recieved_labelid) {\n      perform_scatter = true;\n      vertex.data().labelid = recieved_labelid;\n    }\n  }\n\n  edge_dir_type scatter_edges(icontext_type& context,\n      const vertex_type& vertex) const {\n    if (perform_scatter)\n      return graphlab::ALL_EDGES;\n    else\n      return graphlab::NO_EDGES;\n  }\n\n  //If a neighbor vertex has a bigger label id, send a massage\n  void scatter(icontext_type& context, const vertex_type& vertex,\n      edge_type& edge) const {\n    if (edge.source().id() != vertex.id()\n        && edge.source().data().labelid > vertex.data().labelid) {\n      context.signal(edge.source(), min_message(vertex.data().labelid));\n    }\n    if (edge.target().id() != vertex.id()\n        && edge.target().data().labelid > vertex.data().labelid) {\n      context.signal(edge.target(), min_message(vertex.data().labelid));\n    }\n  }\n};\n\nclass graph_writer {\npublic:\n  std::string save_vertex(graph_type::vertex_type v) {\n    std::stringstream strm;\n    strm << v.id() << \",\" << v.data().labelid << \"\\n\";\n    return strm.str();\n  }\n  std::string save_edge(graph_type::edge_type e) {\n    return \"\";\n  }\n};\n\nint main(int argc, char** argv) {\n  std::cout << \"Connected Component\\n\\n\";\n\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  global_logger().set_log_level(LOG_DEBUG);\n  //parse options\n  graphlab::command_line_options clopts(\"Connected Component.\");\n  std::string graph_dir;\n  std::string saveprefix;\n  std::string format = \"adj\";\n  std::string exec_type = \"synchronous\";\n  clopts.attach_option(\"graph\", graph_dir,\n                       \"The graph file. This is not optional\");\n  clopts.add_positional(\"graph\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph file format\");\n  clopts.attach_option(\"saveprefix\", saveprefix,\n                       \"If set, will save the pairs of a vertex id and \"\n                       \"a component id to a sequence of files with prefix \"\n                       \"saveprefix\");\n  if (!clopts.parse(argc, argv)) {\n    dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n  if (graph_dir == \"\") {\n    std::cout << \"--graph is not optional\\n\";\n    return EXIT_FAILURE;\n  }\n\n  graph_type graph(dc, clopts);\n\n  //load graph\n  dc.cout() << \"Loading graph in format: \"<< format << std::endl;\n  graph.load_format(graph_dir, format);\n  graphlab::timer ti;\n  graph.finalize();\n  dc.cout() << \"Finalization in \" << ti.current_time() << std::endl;\n  graph.transform_vertices(initialize_vertex);\n\n  //running the engine\n  time_t start, end;\n  graphlab::omni_engine engine(dc, graph, exec_type, clopts);\n  engine.signal_all();\n  time(&start);\n  engine.start();\n\n  //write results\n  if (saveprefix.size() > 0) {\n    graph.save(saveprefix, graph_writer(),\n        false, //set to true if each output file is to be gzipped\n        true, //whether vertices are saved\n        false); //whether edges are saved\n  }\n\n  graphlab::mpi_tools::finalize();\n\n  return EXIT_SUCCESS;\n}\n\n"
  },
  {
    "path": "toolkits/graph_analytics/connected_component_stats.cpp",
    "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n\nstruct vdata {\n  std::vector vids;\n  void save(graphlab::oarchive& oarc) const {\n    oarc << vids;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> vids;\n  }\n};\n\ntypedef graphlab::distributed_graph graph_type;\n\n\nvoid vertex_combine(vdata& a, const vdata& b) {\n  for (size_t i = 0;i < b.vids.size(); ++i) a.vids.push_back(b.vids[i]);\n}\n\n\nbool ccoutput_parser(graph_type& graph, const std::string& filename, const std::string& textline) {\n  size_t split = textline.find_first_of(\",\");\n  if (split == std::string::npos) return true;\n  else {\n    std::string t = textline;\n    t[split] = 0;\n    vdata data;\n    data.vids.push_back(atol(t.c_str()));\n    graph.add_vertex(atol(t.c_str() + split + 1), data);\n    return true;\n  }\n}\n\nstruct size_counter {\n  // a map from size to count\n  boost::unordered_map counts;\n\n  size_counter() { }\n\n  explicit size_counter(size_t size) {\n    counts[size] = 1;\n  }\n\n  size_counter& operator+=(const size_counter& other) {\n    boost::unordered_map::const_iterator iter = other.counts.begin();\n    while(iter != other.counts.end()) {\n      counts[iter->first] += iter->second;\n      ++iter;\n    }\n    return *this;\n  }\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << counts;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> counts;\n  }\n};\n\nsize_counter absolute_vertex_data(const graph_type::vertex_type& vertex) {\n  return size_counter(vertex.data().vids.size());\n}\n\nclass graph_writer {\npublic:\n  std::string save_vertex(graph_type::vertex_type v) {\n    std::stringstream strm;\n    strm << v.id() << \":\";\n    for (size_t i = 0;i < v.data().vids.size(); ++i) {\n      strm << v.data().vids[i] << \" \";\n    }\n    strm << \"\\n\";\n    return strm.str();\n  }\n  std::string save_edge(graph_type::edge_type e) {\n    return \"\";\n  }\n};\n\n\n\n\nint main(int argc, char** argv) {\n  std::cout << \"Connected Component\\n\\n\";\n\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  //parse options\n  graphlab::command_line_options clopts(\"Connected Component Stats.\");\n  std::string graph_dir;\n  std::string saveprefix;\n  std::string format = \"adj\";\n  clopts.attach_option(\"graph\", graph_dir,\n                       \"The graph file. This is not optional\");\n  clopts.add_positional(\"graph\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph file format\");\n  clopts.attach_option(\"saveprefix\", saveprefix,\n                       \"save location\");\n  if (!clopts.parse(argc, argv)) {\n    dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n  if (graph_dir == \"\") {\n    std::cout << \"--graph is not optional\\n\";\n    return EXIT_FAILURE;\n  }\n\n  graph_type graph(dc, clopts);\n\n  //load graph\n  dc.cout() << \"Loading graph in format: \"<< format << std::endl;\n  graphlab::timer ti;\n  graph.set_duplicate_vertex_strategy(vertex_combine);\n  graph.load(graph_dir, ccoutput_parser);\n  graph.finalize();\n  dc.cout() << \"Complete Finalization in \" << ti.current_time() << std::endl;\n\n  ti.start();\n  //take statistics\n  size_counter stat = graph.map_reduce_vertices(\n      absolute_vertex_data);\n\n  dc.cout() << \"graph calculation time is \" << ti.current_time() << \" sec\\n\";\n  dc.cout() << \"RESULT:\\nsize\\tcount\\n\";\n  for (boost::unordered_map::const_iterator iter = stat.counts.begin();\n      iter != stat.counts.end(); iter++) {\n    dc.cout() << iter->first << \"\\t\" << iter->second << \"\\n\";\n  }\n  \n  //write results\n  if (saveprefix.size() > 0) {\n    graph.save(saveprefix, graph_writer(),\n        false, //set to true if each output file is to be gzipped\n        true, //whether vertices are saved\n        false); //whether edges are saved\n  }\n\n  graphlab::mpi_tools::finalize();\n\n  return EXIT_SUCCESS;\n}\n\n"
  },
  {
    "path": "toolkits/graph_analytics/degree_ordered_coloring.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n/*\n * Graph coloring algorithm, such that vertex programs are scheduled in \n * order of vertex degree. Includes trade-off featre for determining the\n * fraction of the graph to conduct ordered execution over random execution,\n * allowing user to specify run time - colour quality trade-off\n */\n\n#include \n#include \n#include \n#include \n#include   /* for std::abs(double) */\n\ntypedef graphlab::vertex_id_type color_type;\n\n/*\n * Vertex data: color and degree of node\n */\ntypedef struct {\n  int color;\n  int degree;\n\n   // serialize\n  void save(graphlab::oarchive& oarc) const {\n    oarc << color << degree;\n  }\n\n  // deserialize\n  void load(graphlab::iarchive& iarc) {\n    iarc >> color >> degree;\n  }\n\n} vertex_data_type;\n\n#define UNCOLORED -1\n/*\n * no edge data\n */\ntypedef graphlab::empty edge_data_type;\nbool EDGE_CONSISTENT = false;\nbool TRADE = false;\n\nsize_t graph_size = 0;\nsize_t fraction = 0;\nint max_degree = 0;\nint low_degree = INT_MAX;\nsigned int current_degree;\n\nsize_t already_signalled = 0;\nstd::set used_colors;\nstd::set degrees;\n/*\n * This is the gathering type which accumulates an (unordered) set of\n * all neighboring colors \n * It is a simple wrapper around a boost::unordered_set with\n * an operator+= which simply performs a set union.\n *\n * This struct can be significantly accelerated for small sets.\n * Small collections of vertex IDs should not require the overhead\n * of the unordered_set.\n */\nstruct set_union_gather {\n  boost::unordered_set colors;\n\n  /*\n   * Combining with another collection of vertices.\n   * Union it into the current set.\n   */\n  set_union_gather& operator+=(const set_union_gather& other) {\n    foreach(graphlab::vertex_id_type othervid, other.colors) {\n      colors.insert(othervid);\n    }\n    return *this;\n  }\n  \n  // serialize\n  void save(graphlab::oarchive& oarc) const {\n    oarc << colors;\n  }\n\n  // deserialize\n  void load(graphlab::iarchive& iarc) {\n    iarc >> colors;\n  }\n};\n/*\n * Define the type of the graph\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n/*\n * On gather, we accumulate a set of all adjacent colors.\n */\nclass graph_coloring:\n      public graphlab::ivertex_program,\n      /* I have no data. Just force it to POD */\n      public graphlab::IS_POD_TYPE  {\npublic:\n  // Gather on all edges\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  } \n\n  /*\n   * For each edge, figure out the ID of the \"other\" vertex\n   * and accumulate a set of the neighborhood vertex IDs.\n   */\n  gather_type gather(icontext_type& context,\n                     const vertex_type& vertex,\n                     edge_type& edge) const {\n    set_union_gather gather;\n    color_type other_color = edge.source().id() == vertex.id() ?\n                                 edge.target().data().color: edge.source().data().color;\n\n    gather.colors.insert(other_color);\n    return gather;\n  }\n\n  /*\n   * the gather result now contains the colors in the neighborhood.\n   * pick a different color and store it \n   */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& neighborhood) {\n    // find the smallest color not described in the neighborhood\n    size_t neighborhoodsize = neighborhood.colors.size();\n    for (color_type curcolor = 0; curcolor < neighborhoodsize + 1; ++curcolor) {\n      if (neighborhood.colors.count(curcolor) == 0) {\n        used_colors.insert(curcolor);\n        vertex.data().color = curcolor;\n        break;\n      }\n    }\n  }\n\n  edge_dir_type scatter_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    if (EDGE_CONSISTENT) return graphlab::NO_EDGES;\n    else return graphlab::ALL_EDGES;\n  } \n\n\n  /*\n   * For each edge, count the intersection of the neighborhood of the\n   * adjacent vertices. This is the number of triangles this edge is involved\n   * in.\n   */\n  void scatter(icontext_type& context,\n              const vertex_type& vertex,\n              edge_type& edge) const {\n    // both points have different colors!\n    if (edge.source().data().color == edge.target().data().color) {\n      context.signal(edge.source().id() == vertex.id() ? \n                      edge.target() : edge.source());\n    }\n  }\n};\n\nvoid initialize_vertex_values(graph_type::vertex_type& v) {\n  v.data().degree = v.num_out_edges();\n  degrees.insert(v.data().degree);\n  v.data().color = UNCOLORED;\n  if (v.data().degree > max_degree)\n    max_degree = v.data().degree;\n  if (v.data().degree < low_degree)\n    low_degree = v.data().degree;\n}\n\n\n/*\n * A saver which saves a file where each line is a vid / color pair\n */\nstruct save_colors{\n  std::string save_vertex(graph_type::vertex_type v) { \n    return graphlab::tostr(v.id()) + \"\\t\" +\n           graphlab::tostr(v.data().color) + \"\\n\";\n  }\n  std::string save_edge(graph_type::edge_type e) {\n    return \"\";\n  }\n};\n\ntypedef graphlab::async_consistent_engine engine_type;\n\ngraphlab::empty signal_vertices_at_degree(engine_type::icontext_type& ctx,\n                                     const graph_type::vertex_type& vertex) {\n  if (vertex.data().degree == current_degree) {\n    already_signalled++;\n    ctx.signal(vertex);\n  }\n  return graphlab::empty();\n}\n\ngraphlab::empty signal_uncolored(engine_type::icontext_type& ctx,\n                                     const graph_type::vertex_type& vertex) {\n  if (vertex.data().color == UNCOLORED) {\n    ctx.signal(vertex);\n  }\n  return graphlab::empty();\n}\n\nstruct max_deg_vertex_reducer: public graphlab::IS_POD_TYPE {\n  size_t degree;\n  graphlab::vertex_id_type vid;\n  max_deg_vertex_reducer& operator+=(const max_deg_vertex_reducer& other) {\n    if (degree < other.degree) {\n      (*this) = other;\n    }\n    return (*this);\n  }\n};\n\nmax_deg_vertex_reducer find_max_deg_vertex(const graph_type::vertex_type vtx) {\n  max_deg_vertex_reducer red;\n  red.degree = vtx.num_out_edges();\n  red.vid = vtx.id();\n  return red;\n}\n\n/**************************************************************************/\n/*                                                                        */\n/*                         Validation   Functions                         */\n/*                                                                        */\n/**************************************************************************/\nsize_t validate_conflict(graph_type::edge_type& edge) {\n  return edge.source().data().color == edge.target().data().color;\n}\n\ninline bool isEqual(double x, double y)\n{\n  const double epsilon = 1e-5;\n  return std::abs(x - y) <= epsilon * std::abs(x);\n}\n\nint main(int argc, char** argv) {\n\n  //global_logger().set_log_level(LOG_INFO);\n\n  // Initialize control plane using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n\n  dc.cout() << \"This program computes a simple graph coloring of a\"\n            \"provided graph.\\n\\n\";\n\n  graphlab::command_line_options clopts(\"Graph coloring. \"\n    \"Given a graph, this program computes a graph coloring of the graph.\"\n    \"The Asynchronous engine is used.\");\n  std::string prefix, format;\n  std::string output;\n  float alpha = 2.1;\n  size_t powerlaw = 0;\n  double trade = 0;\n  clopts.attach_option(\"graph\", prefix,\n                       \"Graph input. reads all graphs matching prefix*\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph format\");\n   clopts.attach_option(\"output\", output,\n                       \"A prefix to save the output.\");\n   clopts.attach_option(\"powerlaw\", powerlaw,\n                       \"Generate a synthetic powerlaw out-degree graph. \");\n   clopts.attach_option(\"alpha\", alpha,\n                       \"Alpha in powerlaw distrubution\");\n  clopts.attach_option(\"trade\", trade,\n                       \"Execute tradeoff version. Probability of degree execution for node (0.0 to 1.0)\");\n  clopts.attach_option(\"edgescope\", EDGE_CONSISTENT,\n                       \"Use Locking. \");\n\n  if(!clopts.parse(argc, argv)) return EXIT_FAILURE;\n  if (prefix.length() == 0 && powerlaw == 0) {\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  if (output == \"\") {\n    dc.cout() << \"Warning! Output will not be saved\\n\";\n  }\n\n  graphlab::launch_metric_server();\n  // load graph\n  graph_type graph(dc, clopts);\n\n  if (!isEqual(0.0, trade)) {\n    TRADE = true;\n  }\n  \n  if(powerlaw > 0) { // make a synthetic graph\n    dc.cout() << \"Loading synthetic Powerlaw graph.\" << std::endl;\n    graph.load_synthetic_powerlaw(powerlaw, false, alpha, 100000000);\n  } else { // Load the graph from a file\n    if (prefix == \"\") {\n      dc.cout() << \"--graph is not optional\\n\";\n      return EXIT_FAILURE;\n    }\n    else if (format == \"\") {\n      dc.cout() << \"--format is not optional\\n\";\n      return EXIT_FAILURE;\n    }\n    graph.load_format(prefix, format);\n  }\n  graph.finalize();\n\n  dc.cout() << \"Number of vertices: \" << graph.num_vertices() << std::endl\n    << \"Number of edges:    \" << graph.num_edges() << std::endl;\n\n  graphlab::timer ti;\n\n  \n  dc.cout() << \"Initialising vertex data...\" < engine(dc, graph, clopts);\n\n  //Tradeoff between ordered and random vertex execution\n  if (TRADE) {\n    graph_size = graph.num_vertices();\n    fraction = (int) graph_size * trade;\n    dc.cout() << \"Degree ordered coloring for \" << fraction << \" in \" << graph_size << \" vertices.\" << std::endl;\n  }\n  for (int x = max_degree; x >= low_degree; x--){\n    if (degrees.find(x) != degrees.end()) {\n      current_degree = x;\n      engine.map_reduce_vertices(signal_vertices_at_degree);  \n      if (TRADE) {\n        //Already signalled vertices for degree ordered execution\n        if(already_signalled >= fraction) {\n          engine.start();\n          //Signal remaining vertices randomly\n          engine.map_reduce_vertices(signal_uncolored);  \n          engine.start();\n          break;\n        }\n      }\n    }\n  }\n\n  if (!TRADE) {\n    engine.start();\n  }\n\n  size_t conflict_count = graph.map_reduce_edges(validate_conflict);\n  if (conflict_count > 0) {\n    dc.cout() << \"Still uncolored, finalising...\" << std::endl;\n    engine.map_reduce_vertices(signal_uncolored);\n    engine.start();\n    conflict_count = graph.map_reduce_edges(validate_conflict);\n  }\n\n  dc.cout() << \"Colored in \" << ti.current_time() << \" seconds\" << std::endl;\n  dc.cout() << \"Colored using \" << used_colors.size() << \" colors\" << std::endl;\n\n  dc.cout() << \"Num conflicts = \" << conflict_count << \"\\n\";\n  if (output != \"\") {\n    graph.save(output,\n              save_colors(),\n              false, /* no compression */\n              true, /* save vertex */\n              false, /* do not save edge */\n              1); /* one file per machine */\n  }\n  \n  graphlab::stop_metric_server();\n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // End of main"
  },
  {
    "path": "toolkits/graph_analytics/directed_triangle_count.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n#include \n/**\n This implements the exact counting procedure described in \n\n Efficient Algorithms for Large-Scale Local Triangle Counting\n Luca Becchetti, Paolo Boldi, Carlos Castillo, Aristides Gioni  \n\n  */\n   \n\n// Radix sort implementation from https://github.com/gorset/radix\n// Thanks to Erik Gorset\n//\n/*\nCopyright 2011 Erik Gorset. All rights reserved.\n\nRedistribution and use in source and binary forms, with or without modification, are\npermitted provided that the following conditions are met:\n\n1. Redistributions of source code must retain the above copyright notice, this list of\nconditions and the following disclaimer.\n\n2. Redistributions in binary form must reproduce the above copyright notice, this list\nof conditions and the following disclaimer in the documentation and/or other materials\nprovided with the distribution.\n\nTHIS SOFTWARE IS PROVIDED BY Erik Gorset ``AS IS'' AND ANY EXPRESS OR IMPLIED\nWARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND\nFITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL Erik Gorset OR\nCONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\nCONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\nSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON\nANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING\nNEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF\nADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\nThe views and conclusions contained in the software and documentation are those of the\nauthors and should not be interpreted as representing official policies, either expressed\nor implied, of Erik Gorset.\n*/\nvoid radix_sort(graphlab::vertex_id_type *array, int offset, int end, int shift) {\n    int x, y;\n    graphlab::vertex_id_type value, temp;\n    int last[256] = { 0 }, pointer[256];\n\n    for (x=offset; x> shift) & 0xFF];\n    }\n\n    last[0] += offset;\n    pointer[0] = offset;\n    for (x=1; x<256; ++x) {\n        pointer[x] = last[x-1];\n        last[x] += last[x-1];\n    }\n\n    for (x=0; x<256; ++x) {\n        while (pointer[x] != last[x]) {\n            value = array[pointer[x]];\n            y = (value >> shift) & 0xFF;\n            while (x != y) {\n                temp = array[pointer[y]];\n                array[pointer[y]++] = value;\n                value = temp;\n                y = (value >> shift) & 0xFF;\n            }\n            array[pointer[x]++] = value;\n        }\n    }\n\n    if (shift > 0) {\n        shift -= 8;\n        for (x=0; x<256; ++x) {\n            temp = x > 0 ? pointer[x] - pointer[x-1] : pointer[0] - offset;\n            if (temp > 64) {\n                radix_sort(array, pointer[x] - temp, pointer[x], shift);\n            } else if (temp > 1) {\n                std::sort(array + (pointer[x] - temp), array + pointer[x]);\n                //insertion_sort(array, pointer[x] - temp, pointer[x]);\n            }\n        }\n    }\n}\n\nsize_t HASH_THRESHOLD = 64;\n\n// We on each vertex, either a vector of sorted VIDs\n// or a hash set (cuckoo hash) of VIDs.\n// If the number of elements is greater than HASH_THRESHOLD,\n// the hash set is used. Otherwise the vector is used.\nstruct vid_vector{\n  std::vector vid_vec;\n  graphlab::cuckoo_set_pow2 *cset;\n  vid_vector(): cset(NULL) { }\n  vid_vector(const vid_vector& v):cset(NULL) {\n    (*this) = v;\n  }\n\n  vid_vector& operator=(const vid_vector& v) {\n    if (this == &v) return *this;\n    vid_vec = v.vid_vec;\n    if (v.cset != NULL) {\n      // allocate the cuckoo set if the other side is using a cuckoo set\n      // or clear if I alrady have one\n      if (cset == NULL) {\n        cset = new graphlab::cuckoo_set_pow2(-1, 0, 2 * v.cset->size());\n      }\n      else {\n        cset->clear();\n      }\n      (*cset) = *(v.cset);\n    }\n    else {\n      // if the other side is not using a cuckoo set, lets not use a cuckoo set\n      // either\n      if (cset != NULL) {\n        delete cset;\n        cset = NULL;\n      }\n    }\n    return *this;\n  }\n\n  ~vid_vector() {\n    if (cset != NULL) delete cset;\n  }\n\n  // assigns a vector of vertex IDs to this storage.\n  // this function will clear the contents of the vid_vector\n  // and reconstruct it.\n  // If the assigned values has length >= HASH_THRESHOLD,\n  // we will allocate a cuckoo set to store it. Otherwise,\n  // we just store a sorted vector\n  void assign(const std::vector& vec) {\n    clear();\n    if (vec.size() >= HASH_THRESHOLD) {\n        // move to cset\n        cset = new graphlab::cuckoo_set_pow2(-1, 0, 2 * vec.size());\n        foreach (graphlab::vertex_id_type v, vec) {\n          cset->insert(v);\n        }\n    }\n    else {\n      vid_vec = vec;\n      if (vid_vec.size() > 64) {\n        radix_sort(&(vid_vec[0]), 0, vid_vec.size(), 24);\n      }\n      else {\n        std::sort(vid_vec.begin(), vid_vec.end());\n      }\n      std::vector::iterator new_end = std::unique(vid_vec.begin(),\n                                               vid_vec.end());\n      vid_vec.erase(new_end, vid_vec.end());\n    }\n  }\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << (cset != NULL);\n    if (cset == NULL) oarc << vid_vec;\n    else oarc << (*cset);\n  }\n\n\n  void clear() {\n    vid_vec.clear();\n    if (cset != NULL) {\n      delete cset;\n      cset = NULL;\n    }\n  }\n\n  size_t size() const {\n    return cset == NULL ? vid_vec.size() : cset->size();\n  }\n\n  void load(graphlab::iarchive& iarc) {\n    clear();\n    bool hascset;\n    iarc >> hascset;\n    if (!hascset) iarc >> vid_vec;\n    else {\n      cset = new graphlab::cuckoo_set_pow2(-1, 0, 2);\n      iarc >> (*cset);\n    }\n  }\n};\n\n/*\n  A simple counting iterator which can be used as an insert iterator.\n  but only counts the number of elements inserted. Useful for\n  use with counting the size of an intersection using std::set_intersection\n*/\ntemplate \nstruct counting_inserter {\n  size_t* i;\n  counting_inserter(size_t* i):i(i) { }\n  counting_inserter& operator++() {\n    ++(*i);\n    return *this;\n  }\n  void operator++(int) {\n    ++(*i);\n  }\n\n  struct empty_val {\n    empty_val operator=(const T&) { return empty_val(); }\n  };\n\n  empty_val operator*() {\n    return empty_val();\n  }\n\n  typedef empty_val reference;\n};\n\n\n/*\n * Computes the size of the intersection of two vid_vector's\n */\nstatic uint32_t count_set_intersect(\n             const vid_vector& smaller_set,\n             const vid_vector& larger_set) {\n  if (smaller_set.size() > larger_set.size()) {\n    return count_set_intersect(larger_set, smaller_set);\n  }\n  if (smaller_set.cset == NULL && larger_set.cset == NULL) {\n    size_t i = 0;\n    counting_inserter iter(&i);\n    std::set_intersection(smaller_set.vid_vec.begin(), smaller_set.vid_vec.end(),\n                          larger_set.vid_vec.begin(), larger_set.vid_vec.end(),\n                          iter);\n    return i;\n  }\n  else if (smaller_set.cset == NULL && larger_set.cset != NULL) {\n    size_t i = 0;\n    foreach(graphlab::vertex_id_type vid, smaller_set.vid_vec) {\n      i += larger_set.cset->count(vid);\n    }\n    return i;\n  }\n  else if (smaller_set.cset != NULL && larger_set.cset == NULL) {\n    size_t i = 0;\n    foreach(graphlab::vertex_id_type vid, larger_set.vid_vec) {\n      i += smaller_set.cset->count(vid);\n    }\n    return i;\n  }\n  else {\n    size_t i = 0;\n    foreach(graphlab::vertex_id_type vid, *(smaller_set.cset)) {\n      i += larger_set.cset->count(vid);\n    }\n    return i;\n\n  }\n}\n\n\n// This structure is used to hold the final triangle counts \n// on each vertex\nstruct triangle_count: public graphlab::IS_POD_TYPE {\n  triangle_count(): out_triangles(0), in_triangles(0), \n                      through_triangles(0), cycle_triangles(0) { }\n  // A is the example below\n  /*\n    A ---> B\n    |   / \n    |  /\n    v /\n    C\n           diagonal edge direction does not matter\n  */\n  uint32_t out_triangles;\n\n/*\n    A<--- B\n    ^   / \n    |  /\n    | /\n    C\n          diagonal edge direction does not matter\n  */\n  uint32_t in_triangles;\n\n  /*\n    A---> B\n    ^   ^ \n    |  /\n    | /\n    C\n  */\n  uint32_t through_triangles;\n\n  /*\n    A---> B\n    ^   / \n    |  /\n    | v\n    C\n  */\n  uint32_t cycle_triangles;\n\n  triangle_count& operator+=(const triangle_count& other) {\n    out_triangles += other.out_triangles;\n    in_triangles += other.in_triangles;\n    through_triangles += other.through_triangles;\n    cycle_triangles += other.cycle_triangles;\n    return *this;\n  }\n};\n\n/*\n * Each vertex maintains a list of all its neighbors.\n * and a final count for the number of triangles it is involved in\n */\nstruct vertex_data_type {\n  vertex_data_type(){ }\n  // A list of all its neighbors\n  vid_vector in_vid_set;\n  vid_vector out_vid_set;\n  triangle_count count;\n // The number of triangles this vertex is involved it.\n  // only used if \"per vertex counting\" is used\n  void save(graphlab::oarchive &oarc) const {\n    oarc << in_vid_set << out_vid_set << count;\n  }\n  void load(graphlab::iarchive &iarc) {\n    iarc >> in_vid_set >> out_vid_set >> count;\n  }\n};\n\n\n\n\n// This structure is used to hold the final triangle counts \n// on each edge\nstruct edge_triangle_count: public graphlab::IS_POD_TYPE {\n  edge_triangle_count(): s_s(0), st_st(0), \n                      st_s(0) { }\n  // using notation from the paper\n  // s_s is the intersection between outgoing of source and outgoing of target\n  // st_st is intersection between incoming of source and incoming of target\n  // st_s is intersectino between incoming of source and outgoing of target\n  uint32_t s_s, st_st, st_s;\n\n  edge_triangle_count & operator+=(const edge_triangle_count& other) {\n    s_s += other.s_s;\n    st_st += other.st_st;\n    st_s += other.st_s;\n    return *this;\n  }\n};\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n/*\n * Each edge is simply a counter of triangles\n */\ntypedef edge_triangle_count edge_data_type;\n\n\n/*\n * This is the gathering type which accumulates an array of\n * all neighboring vertices.\n * It is a simple wrapper around a vector with\n * an operator+= which simply performs a  +=\n */\nstruct set_union_gather {\n  graphlab::vertex_id_type v;\n  std::vector vid_vec;\n\n  set_union_gather():v(-1) {\n  }\n\n  size_t size() const {\n    if (v == (graphlab::vertex_id_type)-1) return vid_vec.size();\n    else return 1;\n  }\n  /*\n   * Combining with another collection of vertices.\n   * Union it into the current set.\n   */\n  set_union_gather& operator+=(const set_union_gather& other) {\n    if (size() == 0) {\n      (*this) = other;\n      return (*this);\n    }\n    else if (other.size() == 0) {\n      return *this;\n    }\n\n    if (vid_vec.size() == 0) {\n      vid_vec.push_back(v);\n      v = (graphlab::vertex_id_type)(-1);\n    }\n    if (other.vid_vec.size() > 0) {\n      size_t ct = vid_vec.size();\n      vid_vec.resize(vid_vec.size() + other.vid_vec.size());\n      for (size_t i = 0; i < other.vid_vec.size(); ++i) {\n        vid_vec[ct + i] = other.vid_vec[i];\n      }\n    }\n    else if (other.v != (graphlab::vertex_id_type)-1) {\n      vid_vec.push_back(other.v);\n    }\n    return *this;\n  }\n  \n  // serialize\n  void save(graphlab::oarchive& oarc) const {\n    oarc << bool(vid_vec.size() == 0);\n    if (vid_vec.size() == 0) oarc << v;\n    else oarc << vid_vec;\n  }\n\n  // deserialize\n  void load(graphlab::iarchive& iarc) {\n    bool novvec;\n    v = (graphlab::vertex_id_type)(-1);\n    vid_vec.clear();\n    iarc >> novvec;\n    if (novvec) iarc >> v;\n    else iarc >> vid_vec;\n  }\n};\n\n\nstruct set_union_pair{\n  set_union_gather in_set;\n  set_union_gather out_set;\n\n  set_union_pair& operator+=(const set_union_pair& other) {\n    in_set += other.in_set;\n    out_set += other.out_set;\n    return (*this);\n  }\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << in_set << out_set;\n  }\n\n  // deserialize\n  void load(graphlab::iarchive& iarc) {\n    iarc >> in_set >> out_set;\n  }\n \n};\n\n/*\n * Define the type of the graph\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n\n/*\n * This class implements the triangle counting algorithm as described in\n * the header. On gather, we accumulate a set of all adjacent vertices.\n * If per_vertex output is not necessary, we can use the optimization\n * where each vertex only accumulates neighbors with greater vertex IDs.\n */\nclass triangle_count_program :\n      public graphlab::ivertex_program,\n      /* I have no data. Just force it to POD */\n      public graphlab::IS_POD_TYPE  {\npublic:\n  bool do_not_scatter;\n\n  // Gather on all edges\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  } \n\n  /*\n   * For each edge, figure out the ID of the \"other\" vertex\n   * and accumulate a set of the neighborhood vertex IDs.\n   */\n  gather_type gather(icontext_type& context,\n                     const vertex_type& vertex,\n                     edge_type& edge) const {\n    set_union_pair gather;\n    // check the edge direction\n    if (edge.source().id() == vertex.id()) {\n      // this is an out_edge\n      graphlab::vertex_id_type otherid = edge.target().id();\n      gather.out_set.v = otherid;\n    }\n    else {\n      // this is an in_edge\n      graphlab::vertex_id_type otherid = edge.source().id();\n      gather.in_set.v = otherid;\n    }\n    return gather;\n  }\n\n  /*\n   * the gather result now contains the vertex IDs in the neighborhood.\n   * store it on the vertex. \n   */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& neighborhood) {\n   do_not_scatter = false;\n   if (neighborhood.in_set.vid_vec.size() == 0) {\n     // neighborhood set may be empty or has only 1 element\n     vertex.data().in_vid_set.clear();\n     if (neighborhood.in_set.v != (graphlab::vertex_id_type(-1))) {\n       vertex.data().in_vid_set.vid_vec.push_back(neighborhood.in_set.v);\n     }\n   }\n   else {\n     vertex.data().in_vid_set.assign(neighborhood.in_set.vid_vec);\n   }\n\n\n   if (neighborhood.out_set.vid_vec.size() == 0) {\n     // neighborhood set may be empty or has only 1 element\n     vertex.data().out_vid_set.clear();\n     if (neighborhood.out_set.v != (graphlab::vertex_id_type(-1))) {\n       vertex.data().out_vid_set.vid_vec.push_back(neighborhood.out_set.v);\n     }\n   }\n   else {\n     vertex.data().out_vid_set.assign(neighborhood.out_set.vid_vec);\n   }\n\n   do_not_scatter = vertex.data().in_vid_set.size() == 0 && \n                    vertex.data().out_vid_set.size() == 0 ;\n  } // end of apply\n\n  /*\n   * Scatter over all edges to compute the intersection.\n   * I only need to touch each edge once, so if I scatter just on the\n   * out edges, that is sufficient.\n   */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const {\n    if (do_not_scatter) return graphlab::NO_EDGES;\n    else return graphlab::OUT_EDGES;\n  }\n\n\n  /*\n   * For each edge, count the intersection of the neighborhood of the\n   * adjacent vertices. This is the number of triangles this edge is involved\n   * in.\n   */\n  void scatter(icontext_type& context,\n              const vertex_type& vertex,\n              edge_type& edge) const {\n\n    //vertex_type othervtx = edge.target();\n    // ok. the real work happens here.\n    \n    const vertex_data_type& srclist = edge.source().data();\n    const vertex_data_type& targetlist = edge.target().data();\n\n    edge.data().s_s = count_set_intersect(srclist.out_vid_set,\n                                    targetlist.out_vid_set);\n    edge.data().st_st += count_set_intersect(srclist.in_vid_set,\n                                    targetlist.in_vid_set);\n    edge.data().st_s += count_set_intersect(srclist.in_vid_set,\n                                    targetlist.out_vid_set);\n\n  }\n};\n\n/*\n * This class is used in a second engine call if per vertex counts are needed.\n * The number of triangles a vertex is involved in can be computed easily\n * by summing over the number of triangles each adjacent edge is involved in\n * and dividing by 2. \n */\nclass get_per_vertex_count :\n      public graphlab::ivertex_program,\n      /* I have no data. Just force it to POD */\n      public graphlab::IS_POD_TYPE  {\npublic:\n  // Gather on all edges\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  }\n  // We gather the number of triangles each edge is involved in\n  triangle_count gather(icontext_type& context,\n                     const vertex_type& vertex,\n                     edge_type& edge) const {\n    triangle_count ret;\n    if (edge.source().id() == vertex.id()) {\n      ret.out_triangles += edge.data().s_s;\n      ret.through_triangles += edge.data().st_st;\n      ret.cycle_triangles += edge.data().st_s;\n    }\n    else {\n      ret.in_triangles += edge.data().st_st;\n    }\n    return ret;\n  }\n\n  /* the gather result is the total sum of the number of triangles\n   * each adjacent edge is involved in . Dividing by 2 gives the\n   * desired result.\n   */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& tc) {\n    vertex.data().in_vid_set.clear();\n    vertex.data().out_vid_set.clear();\n    vertex.data().count = tc;\n  }\n\n  // No scatter\n  edge_dir_type scatter_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n\n\n};\n\ntypedef graphlab::synchronous_engine engine_type;\n\n/* Used to sum over all the vertices in the graph in a\n * map_reduce_vertices call\n * to get the total number of triangles\n */\ntriangle_count get_vertex_counts(const graph_type::vertex_type& v) {\n  return v.data().count;\n}\n\n/*\n * A saver which saves a file where each line is a vid / # triangles pair\n */\nstruct save_triangle_count{\n  std::string save_vertex(graph_type::vertex_type v) { \n    triangle_count tc = v.data().count;\n    double n_followed = v.num_out_edges();\n    double n_following = v.num_in_edges();\n\n    return graphlab::tostr(v.id()) + \"\\t\" +\n           graphlab::tostr(tc.in_triangles) + \"\\t\" +\n           graphlab::tostr(tc.out_triangles) + \"\\t\" +\n           graphlab::tostr(tc.through_triangles) + \"\\t\" +\n           graphlab::tostr(tc.cycle_triangles) + \"\\t\" +\n           graphlab::tostr(n_followed) + \"\\t\" + \n           graphlab::tostr(n_following) + \"\\n\";\n  }\n  std::string save_edge(graph_type::edge_type e) {\n    return \"\";\n  }\n};\n\n\nint main(int argc, char** argv) {\n  std::cout << \"This program counts the exact number of triangles in the \"\n            \"provided graph.\\n\\n\";\n\n  graphlab::command_line_options clopts(\"Exact Triangle Counting. \"\n    \"Given a graph, this program computes the total number of triangles \"\n    \"in the graph. An option (per_vertex) is also provided which \"\n    \"computes for each vertex, the number of triangles it is involved in.\"\n    \"The algorithm assumes that each undirected edge appears exactly once \"\n    \"in the graph input. If edges may appear more than once, this procedure \"\n    \"will over count.\");\n  std::string prefix, format;\n  std::string per_vertex;\n  bool PER_VERTEX_COUNT = false;\n  clopts.attach_option(\"graph\", prefix,\n                       \"Graph input. reads all graphs matching prefix*\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph format\");\n  clopts.attach_option(\"ht\", HASH_THRESHOLD,\n                       \"Above this size, hash tables are used\");\n  clopts.attach_option(\"per_vertex\", per_vertex,\n                       \"If not empty, will count the number of \"\n                       \"triangles each vertex belongs to and \"\n                       \"save to file with prefix \\\"[per_vertex]\\\". \"\n                       \"The algorithm used is slightly different \"\n                       \"and thus will be a little slower\");\n  if(!clopts.parse(argc, argv)) return EXIT_FAILURE;\n  if (prefix == \"\") {\n    std::cout << \"--graph is not optional\\n\";\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  else if (format == \"\") {\n    std::cout << \"--format is not optional\\n\";\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n\n  if (per_vertex != \"\") PER_VERTEX_COUNT = true;\n  // Initialize control plane using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n  graphlab::launch_metric_server();\n  // load graph\n  graph_type graph(dc, clopts);\n  graph.load_format(prefix, format);\n  graph.finalize();\n  dc.cout() << \"Number of vertices: \" << graph.num_vertices() << std::endl\n            << \"Number of edges:    \" << graph.num_edges() << std::endl;\n\n  graphlab::timer ti;\n  \n  // create engine to count the number of triangles\n  dc.cout() << \"Counting Triangles...\" << std::endl;\n  engine_type engine(dc, graph, clopts);\n  engine.signal_all();\n  engine.start();\n\n  dc.cout() << \"Counted in \" << ti.current_time() << \" seconds\" << std::endl;\n  dc.cout() << \"Collecting results ... \" << std::endl;\n  graphlab::synchronous_engine engine2(dc, graph, clopts);\n  engine2.signal_all();\n  engine2.start();\n \n  if (PER_VERTEX_COUNT == false) {\n    triangle_count count = graph.map_reduce_vertices(get_vertex_counts);\n    dc.cout() << count.in_triangles << \" In triangles\\n\";\n    dc.cout() << count.out_triangles << \" Out triangles\\n\";\n    dc.cout() << count.through_triangles << \" Through triangles\\n\";\n    dc.cout() << count.cycle_triangles << \" Cycle triangles\\n\";\n  }\n  else {\n   dc.cout() << \"Saving Results...\\n\";\n   dc.cout() << \"Format is \\n\";\n   dc.cout() << \"   [vid]  [in triangles]  [out triangles]   [through triangles]  [cycle_triangles]  [#out edges] [#in edges]\" << std::endl;\n   graph.save(per_vertex,\n            save_triangle_count(),\n            false, /* no compression */\n            true, /* save vertex */\n            false, /* do not save edge */\n            clopts.get_ncpus()); /* one file per machine */\n\n  }\n  \n  graphlab::stop_metric_server();\n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // End of main\n\n"
  },
  {
    "path": "toolkits/graph_analytics/eigen_vector_normalization.cpp",
    "content": "#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\nsize_t num_data = 0;\nsize_t current_cluster = 0;\nsize_t last_updated_cluster = 0;\nsize_t num_clusters = 0;\n\n//helper function to normalize a vector;\nvoid normalize_eivec(std::vector& vec) {\n  float sum = 0.0;\n  for (size_t i = 0; i < vec.size(); ++i) {\n    sum += vec[i] * vec[i];\n  }\n  sum = sqrt(sum);\n  for (size_t i = 0; i < vec.size(); ++i) {\n    vec[i] /= sum;\n  }\n}\n\nstruct evec_vertex_data {\n  std::vector vec;\n  evec_vertex_data() :\n      vec(num_clusters, 0.0) {\n  }\n  void save(graphlab::oarchive& oarc) const {\n    oarc << vec;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> vec;\n  }\n};\n\nstruct evec_edge_data {\n  float val;\n  evec_edge_data() :\n      val(0.0) {\n  }\n  explicit evec_edge_data(float in_val) :\n      val(in_val) {\n  }\n  void save(graphlab::oarchive& oarc) const {\n    oarc << val;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> val;\n  }\n};\n\ntypedef graphlab::distributed_graph evec_graph_type;\n\n// Read a line from a file and add values to vertices\nbool evec_line_parser(evec_graph_type& graph, const std::string& filename,\n    const std::string& textline) {\n  std::stringstream strm(textline);\n  size_t vid = 0;\n  strm >> vid;\n  float val = 0.0;\n  size_t colcount = 0;\n  while(strm >> val) {\n    if (current_cluster + colcount >= num_clusters){\n      break;\n    }\n    graph.add_edge(vid, num_data + current_cluster + colcount + 1, evec_edge_data(val));\n    colcount++;\n  }\n  last_updated_cluster = current_cluster + colcount;\n\n  return true;\n}\n\n//gather values belonging to the same data\nstruct id_and_val {\n  std::vector ids;\n  std::vector vals;\n\n  id_and_val() :\n      ids(), vals() {\n  }\n  id_and_val(size_t id, float val) :\n      ids(), vals() {\n    ids.push_back(id);\n    vals.push_back(val);\n  }\n\n  id_and_val& operator+=(const id_and_val& other) {\n    for (size_t i = 0; i < other.ids.size(); ++i) {\n      ids.push_back(other.ids[i]);\n      vals.push_back(other.vals[i]);\n    }\n    return *this;\n  }\n\n  void save(graphlab::oarchive& oarc) const {\n    size_t num = ids.size();\n    oarc << num;\n    for (size_t a = 0; a < num; ++a)\n      oarc << ids[a];\n    for (size_t a = 0; a < num; ++a)\n      oarc << vals[a];\n  }\n  void load(graphlab::iarchive& iarc) {\n    ids.clear();\n    vals.clear();\n    size_t num = 0;\n    iarc >> num;\n    for (size_t a = 0; a < num; ++a) {\n      size_t id = 0;\n      iarc >> id;\n      ids.push_back(id);\n    }\n    for (size_t a = 0; a < num; ++a) {\n      float val = 0;\n      iarc >> val;\n      vals.push_back(val);\n    }\n  }\n};\n\n//gather values belonging to the same data\nclass aggregate_values: public graphlab::ivertex_program, public graphlab::IS_POD_TYPE {\npublic:\n  //gather on out edges\n  edge_dir_type gather_edges(icontext_type& context,\n      const vertex_type& vertex) const {\n    return graphlab::OUT_EDGES;\n  }\n  id_and_val gather(icontext_type& context, const vertex_type& vertex,\n      edge_type& edge) const {\n    return id_and_val(edge.target().id() - num_data - 1, edge.data().val);\n  }\n\n  //get values and make a vector\n  void apply(icontext_type& context, vertex_type& vertex,\n      const gather_type& total) {\n    const std::vector& ids = total.ids;\n    const std::vector& vals = total.vals;\n    for (size_t i = 0; i < ids.size(); ++i) {\n      vertex.data().vec[ids[i]] = vals[i];\n    }\n  }\n\n  edge_dir_type scatter_edges(icontext_type& context,\n      const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n  void scatter(icontext_type& context, const vertex_type& vertex,\n      edge_type& edge) const {\n  }\n};\n\nvoid normalize_eigen_vector(evec_graph_type::vertex_type& v) {\n  normalize_eivec(v.data().vec);\n}\n\nclass evec_graph_writer {\npublic:\n  std::string save_vertex(evec_graph_type::vertex_type v) {\n    if(v.id()>num_data) return\"\";\n\n    std::stringstream strm;\n    strm << v.id() << \" \";\n    const std::vector& vec = v.data().vec;\n    for (size_t i = 0; i < vec.size(); ++i) {\n      if (i != 0)\n        strm << \" \";\n      strm << vec[i];\n    }\n    strm << \"\\n\";\n\n    return strm.str();\n  }\n\n  std::string save_edge(evec_graph_type::edge_type e) {\n    return \"\";\n  }\n};\n\n//read and normalize eigen vectors\nint main(int argc, char** argv) {\n  std::string datafile;\n  size_t rank = 0;\n\n  //parse command line\n  graphlab::command_line_options clopts(\n      \"Normalize eigen vectors for graph partitioning\");\n  clopts.attach_option(\"data\", datafile, \"Input file prefix\");\n  clopts.attach_option(\"rank\", rank, \"Rank of Lanczos method\");\n  clopts.attach_option(\"clusters\", num_clusters, \"Number of clusters\");\n  clopts.attach_option(\"data-num\", num_data, \"Number of data points\");\n  if (!clopts.parse(argc, argv))\n    return EXIT_FAILURE;\n  if (datafile == \"\") {\n    std::cout << \"--data is not optional\\n\";\n    return EXIT_FAILURE;\n  }\n  if (rank == 0) {\n    std::cout << \"--rank is not optional\\n\";\n    return EXIT_FAILURE;\n  }\n  if (num_clusters == 0) {\n    std::cout << \"--clusters is not optional\\n\";\n    return EXIT_FAILURE;\n  }\n  if (num_data == 0) {\n    std::cout << \"--data-num is not optional\\n\";\n    return EXIT_FAILURE;\n  }\n\n  //load and normalize eigen vectors\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  evec_graph_type graph(dc, clopts);\n  //special vertices corresponding to clusters\n  for (size_t i = 0; i < num_clusters; ++i) {\n    graph.add_vertex(num_data + i + 1, evec_vertex_data());\n  }\n  \n  for (size_t i = 0; i < num_clusters; ++i) {\n    //current_cluster = i;\n    std::stringstream vec_filename;\n    vec_filename << datafile;\n    vec_filename << \".U.\";\n    vec_filename << i+1;\n    vec_filename << \"_\";\n    graph.load(vec_filename.str(), evec_line_parser);\n    current_cluster = last_updated_cluster + 1;\n    if (current_cluster >=  num_clusters) {\n      break;\n    }\n  }\n  graph.finalize();\n\n  graphlab::omni_engine engine(dc, graph, \"sync\", clopts);\n  engine.signal_all();\n  engine.start();\n\n  graph.transform_vertices(normalize_eigen_vector);\n\n  graph.save(datafile + \".compressed\", evec_graph_writer(), false, true, false,\n      1);\n\n  graphlab::mpi_tools::finalize();\n\n  return EXIT_SUCCESS;\n}\n\n"
  },
  {
    "path": "toolkits/graph_analytics/format_convert.cpp",
    "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n#include \n\nint main(int argc, char** argv) {\n  // Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  global_logger().set_log_level(LOG_INFO);\n\n  size_t powerlaw = 0;\n  std::string ingraph, informat;\n  std::string outgraph, outformat;\n\n  bool gzip = true;\n  // Parse command line options -----------------------------------------------\n  graphlab::command_line_options clopts(\"Graph Format Conversion.\", true);\n\n  clopts.attach_option(\"powerlaw\", powerlaw,\n                       \"Generates a synthetic powerlaw graph with this many \"\n                       \"vertices. If set, ingraph, and informat are ignored\");\n  clopts.attach_option(\"ingraph\", ingraph,\n                       \"The input graph file. Required \");\n  clopts.attach_option(\"informat\", informat,\n                       \"The input graph file format\");\n  clopts.attach_option(\"outgraph\", outgraph,\n                       \"The output graph file. Required \");\n  clopts.attach_option(\"outformat\", outformat,\n                       \"The output graph file format\");\n  clopts.attach_option(\"outgzip\", gzip,\n                       \"If output is to be gzip compressed\");\n\n  if(!clopts.parse(argc, argv)) {\n    dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n  if (powerlaw==0 && (ingraph.length() == 0 || outgraph.length() == 0)) {\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  typedef graphlab::distributed_graph graph_type;\n  graph_type graph(dc, clopts);\n\n  dc.cout() << \"Loading graph in format: \"<< ingraph << std::endl;\n  if (powerlaw) {\n    graph.load_synthetic_powerlaw(powerlaw, false, 2.1, 100000000 /*max degree*/);\n  } else {\n    graph.load_format(ingraph, informat);\n  }\n  graph.finalize();\n\n  dc.cout() << \"#vertices: \" << graph.num_vertices()\n            << \" #edges:\" << graph.num_edges() << std::endl;\n\n  graph.save_format(outgraph, outformat, gzip);\n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // End of main\n\n\n\n"
  },
  {
    "path": "toolkits/graph_analytics/graph_analytics.dox",
    "content": "/**\n\n\\page graph_analytics Graph Analytics \n\n\\brief The graph analytics toolkit contains applications for performing graph \nanalytics and extracting patterns from the graph structure.\n\nThe toolkit current contains:\n - \\ref graph_analytics_format_conversion \"Graph Format Conversion\"\n - \\ref graph_analytics_triangle_undirected \"Triangle Counting (undirected)\"\n - \\ref graph_analytics_triangle_directed \"Triangle Counting (directed)\"\n - \\ref graph_analytics_pagerank \"PageRank\"\n - \\ref graph_analytics_kcore \"KCore Decomposition\"\n - \\ref graph_analytics_connected_component \"Connected Component\"\n - \\ref graph_analytics_approximate_diameter \"Approximate Diameter\"\n - \\ref graph_analytics_partitioning \"Graph Partitioning\"\n - \\ref graph_coloring \"Graph Coloring\"\n - \\ref graph_analytics_total_subgraph_centrality \"Total Subgraph Centrality\"\n\nAll toolkits take any of the graph formats described in \\ref graph_formats . \n\n\n\n\\section graph_analytics_format_conversion Format Conversion\nThis is primarily a utility program, providing conversion between any of the\nPortable Graph formats described in \\ref graph_formats.\n\nTo run:\n\\verbatim\n> ./format_convert --ingraph=[input graph location] --informat=[input format type]\n                   --outgraph=[output graph location] --outformat[output format type]\n\\endverbatim\n\nThe output is by default gzip compressed.\nTo disable, add the option,\n\\verbatim\n --outgzip=0\n\\endverbatim\n\nThis program can also run distributed by using\n\\verbatim\n> mpiexec -n [N machines] --hostfile [host file] ./format_convert ....\n\\endverbatim\nSee your MPI documentation for details on how to launch this job.\nAll machines must have access to the input graph location and the output graph \nlocation. Graphs may be on HDFS.  \nIf you have problems loading HDFS files, see the \\ref FAQ.\n\n\n\n\n\n\n\n\n\n\n\\section graph_analytics_triangle_undirected Undirected Triangle Counting\n\nThe undirected triangle counting program can count the total number of \ntriangles in a graph, and can also, with little more time, count the number of\ntriangles passing through each vertex in the graph.\n\nIt implements the edge-iterator algorithm described in \n\nT. Schank. Algorithmic Aspects of Triangle-Based Network Analysis.\n    Phd in computer science, University Karlsruhe, 2007.\n\nwith several optimizations.\n\nThe input to the system is a graph in any of the Portable graph format\ndescribed in \\ref graph_formats. It is important that the input be \"cleaned\"\nand that reverse edges are removed: i.e. if edge 1-->5 exists, edge 5-->1 should\nnot exist. (The program will run without these edge removed. But numbers\nmay be erroneous).\n\nTo count the total number of triangles in a graph, the minimal set of options\nrequired are:\n\\verbatim\n> ./undirected_triangle_count --graph=[graph prefix] --format=[format]\n\\endverbatim\nOutput looks like:\n\\verbatim\nNumber of vertices: 875713\nNumber of edges:    4322051\nCounting Triangles...\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 875713\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nCounted in 1.16463 seconds\n13391903 Triangles\n\\endverbatim\n\n\n\nTo count the number of triangles on each vertex, the minimal set of options are:\n\n\\verbatim\n> ./undirected_triangle_count --graph=[graph prefix] --format=[format] --per_vertex=[output prefix]\n\\endverbatim\n\nTne output prefix is where the output counts will be written. This\nmay be located on HDFS. For instance, if the output_prefix is \"v_out\",\nthe output files will be written to:\n\n\\verbatim\nv_out_1_of_16\nv_out_2_of_16\n...\nv_out_16_of_16\n\\endverbatim\n\nEach line in the output file contains two numbers: a Vertex ID, and the number\nof triangles intersecting the vertex.\n\nThis program can also run distributed by using\n\\verbatim\n> mpiexec -n [N machines] --hostfile [host file] ./undirected_triangle_count ....\n\\endverbatim\nSee your MPI documentation for details on how to launch this job.\nAll machines must have access to the input graph location and the output graph \nlocation. Graphs may be on HDFS. \nIf you have problems loading HDFS files, see the \\ref FAQ.\n\n\n\\subsection Options\nRelevant options are: \n\\li \\b --graph (Required). The prefix from which to load the graph data\n\\li \\b --format (Required). The format of the input graph \n\\li \\b --per_vertex (Optional. Default \"\"). If set, will write the output counts.\n\\li \\b --ncpus (Optional. Default 2) The number of processors that will be used\nfor computation.  \n\\li \\b --ht (Optional. Default 64) The implementation uses a mix of vectors and\nhash sets to optimize set intersection computation. This parameter sets the capacity\nlimit below which, vectors are used, and above which, hash sets are used.\n\\li \\b –-graph_opts (Optional, Default empty) Any additional graph options. See\n  graphlab::distributed_graph a list of options.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\\section graph_analytics_triangle_directed Directed Triangle Counting\n\nThe directed triangle counting program counts the total number of \ndirected triangles in a graph of each type, and can also output the number of\ntriangles of each type passing through each vertex in the graph.\n\nWe show the 4 possible types of triangles here:\nIn each case, the vertex being evaluated is the green vertex labeled \"A\". \nA dotted edge means that the direction of the edge do not matter.\n\n\nTriangle Name     | Triangle Pattern\n------------------|----------------------------\nIn Triangle       | \\image html in_triangle.gif\nOut Triangle      | \\image html out_triangle.gif\nThrough Triangle  | \\image html through_triangle.gif\nCycle Triangle    | \\image html cycle_triangle.gif\n\n\n\nThe input to the system is a graph in any of the Portable graph format\ndescribed in \\ref graph_formats. \n\nTo count the total number of triangles in a graph, the minimal set of options\nrequired are:\n\\verbatim\n> ./directed_triangle_count --graph=[graph prefix] --format=[format]\n\\endverbatim\nOutput looks like this:\n\\verbatim\nNumber of vertices: 875713\nNumber of edges:    5105039\nCounting Triangles...\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 875713\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nCounted in 1.962 seconds\nCollecting results ... \nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 875713\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\n28198954 In triangles\n28198954 Out triangles\n28198954 Through triangles\n11669313 Cycle triangles\n\\endverbatim\nObserve that the number of In, Out and Through triangles are identical. This\nis because every In-triangle necessarily forms one Out and one Through triangle,\n(and similarly for the rest). Also the number of Cycle Triangles must be divisible\nby 3 since every cycle is counted 3 times, once on each vertex in the cycle.\n\nTo count the number of triangles on each vertex, the minimal set of options are:\n\n\\verbatim\n> ./directed_triangle_count --graph=[graph prefix] --format=[format] --per_vertex=[output prefix]\n\\endverbatim\n\nTne output prefix is where the output counts will be written. This\nmay be located on HDFS. For instance, if the output_prefix is \"v_out\",\nthe output files will be written to:\n\n\\verbatim\nv_out_1_of_16\nv_out_2_of_16\n...\nv_out_16_of_16\n\\endverbatim\n\nEach line in the output file has the following format:\n\\verbatim\n[vid]  [in triangles]  [out triangles]   [through triangles]  [cycle_triangles] [#out edges] [#in edges]\n\\endverbatim\n\nThis program can also run distributed by using\n\\verbatim\n> mpiexec -n [N machines] --hostfile [host file] ./directed_triangle_count ....\n\\endverbatim\nSee your MPI documentation for details on how to launch this job.\nAll machines must have access to the input graph location and the output graph \nlocation. Graphs may be on HDFS. \nIf you have problems loading HDFS files, see the \\ref FAQ.\n\n\n\n\\subsection Options\nRelevant options are: \n\\li \\b --graph (Required). The prefix from which to load the graph data\n\\li \\b --format (Required). The format of the input graph \n\\li \\b --per_vertex (Optional. Default \"\"). If set, will write the output counts.\n\\li \\b --ncpus (Optional. Default 2) The number of processors that will be used\nfor computation.  \n\\li \\b --ht (Optional. Default 64) The implementation uses a mix of vectors and\nhash sets to optimize set intersection computation. This parameter sets the capacity\nlimit below which, vectors are used, and above which, hash sets are used.\n\\li \\b -–graph_opts (Optional, Default empty) Any additional graph options. See\n  graphlab::distributed_graph a list of options.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\\section graph_analytics_pagerank PageRank \n\nThe PageRank program computes the pagerank of each vertex. \nSee the Wikipedia article\nfor details of the algorithm.\n\n\\subsection Input Graph\nThe input to the system is a graph in any of the Portable graph format\ndescribed in \\ref graph_formats. \n\n\\verbatim\n> ./pagerank --graph=[graph prefix] --format=[format] \n\\endverbatim\n\nAlternatively, a synthetic power law graph of an arbitrary number of vertices\ncan be generated using:\n\\verbatim\n> ./pagerank --powerlaw=[nvertices]\n\\endverbatim\nThe resultant graph will have powerlaw out-degree, and nearly constant in-degree. \nThe actual generation process draws vertex degree from a truncated power-law\ndistribution with alpha=2.1. The distribution is truncated at maximum out-degree\n100M to avoid allocating massive amounts of memory for creating the sampling distribution.\n\n\\subsection Computation Type \nThere are several modes of computation that are supported. All will eventually\nobtain the same solutions. \n\n### Classical\nTo get classical PageRank iterations, adding the option\n\\verbatim\n> --iterations=[N Iterations]\n\\endverbatim\n\n### Dynamic Synchronous (default) \nThe dynamic synchronous computation only performs computation on vertices\nthat have not yet converged to the desired tolerance. The default tolerance\nis 0.001. This can be modified by adding the option\n\\verbatim\n>  --tol=[tolerance]\n\\endverbatim\n\n### Dynamic Asynchronous\nThe dynamic asynchronous computation only performs computation on vertices\nthat have not yet converged to the desired tolerance. This uses the \nasynchronous engine. The default tolerance is 0.001. \nThis can be modified by adding the option\n\\verbatim\n>  --tol=[tolerance]\n\\endverbatim\n\n\\note This is known to be slow! PageRank does not benefit from the consistency\nguaranteed by the asynchronous engine. A new engine is in development with \nweaker consistency semantics, but sufficient for pagerank. \n\n\n\\subsection Output\nTo save the resultant pagerank of each vertex, include the option\n\\verbatim\n> --saveprefix=[output prefix]\n\\endverbatim\n\nTne output prefix is where the output counts will be written. This\nmay be located on HDFS. For instance, if the output_prefix is \"v_out\",\nthe output files will be written to:\n\n\\verbatim\nv_out_1_of_16\nv_out_2_of_16\n...\nv_out_16_of_16\n\\endverbatim\n\nEach line in the output file contains two numbers: a Vertex ID, and the \ncomputed PageRank. Note that the output vector is NOT normalized, namely \ncomputed entries do not sum into one. \n\nThis program can also run distributed by using\n\\verbatim\n> mpiexec -n [N machines] --hostfile [host file] ./pagerank ....\n\\endverbatim\nSee your MPI documentation for details on how to launch this job.\nAll machines must have access to the input graph location and the output graph \nlocation. Graphs may be on HDFS. \nIf you have problems loading HDFS files, see the \\ref FAQ.\n\n\n\n\\subsection Options\nRelevant options are: \n\\li \\b --graph (Optional). The prefix from which to load the graph data\n\\li \\b --format (Optional). The format of the input graph \n\\li \\b --powerlaw (Optional. Default 0). If set, generates synthetic powerlaw graph with\n                        the specified number of vertices.\n\\li \\b --saveprefix (Optional. Default \"\"). If set, will write the output counts.\n\\li \\b --tol (Optional. Default=1E-3). Changes the convergence tolerance for the Dynamic\n                          computation modes.\n\\li \\b --iterations (Optional. Default 0). If set, runs classical PageRank iterations\n                      for the specified number of iterations.\n\\li \\b -–graph_opts (Optional, Default empty) Any additional graph options. See\n  graphlab::distributed_graph a list of options.\n\\li \\b --ncpus (Optional. Default 2) The number of processors that will be used\nfor computation.  \n\\li \\b -–engine (Optional, Default \"synchronous\") Sets the engine type. Must be\n                  either \"synchronous\" or \"asynchronous\"\n\\li \\b -–engine (Optional, Default \"synchronous\") Sets the engine options. Available\n                  options depend on the engine type. See\n                  graphlab::async_consistent_engine and\n                  graphlab::synchronous_engine for details.\n\n\n\n\n\\section graph_analytics_kcore KCore Decomposition \nThis program iteratively finds the KCore of the network.\n\n\\subsection Input Graph\nThe input to the system is a graph in any of the Portable graph format\ndescribed in \\ref graph_formats. \n\n\\verbatim\n> ./kcore --graph=[graph prefix] --format=[format] \n\\endverbatim\nOutput may look like:\n\\verbatim\nK=0:  #V = 875713   #E = 4322051\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 0\nK=1:  #V = 875713   #E = 4322051\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 153407\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nK=2:  #V = 711870   #E = 4160100\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 108715\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nK=3:  #V = 581712   #E = 3915291\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 69907\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nK=4:  #V = 492655   #E = 3668104\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 52123\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nK=5:  #V = 424155   #E = 3416251\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 41269\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nK=6:  #V = 367361   #E = 3158776\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 33444\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nK=7:  #V = 319194   #E = 2902138\nINFO:     synchronous_engine.hpp(start:1213): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1257): \tActive vertices: 29201\nINFO:     synchronous_engine.hpp(start:1307): \t Running Aggregators\nK=8:  #V = 274457   #E = 2629033\n......\n\\endverbatim\n\nTo just get the informative lines:\n\\verbatim\n> ./kcore --graph=[graph prefix] --format=[format] > k_out.txt\n  ...\n> cat k_out.txt\nComputes a k-core decomposition of a graph.\n\nNumber of vertices: 875713\nNumber of edges:    4322051\nK=0:  #V = 875713   #E = 4322051\nK=1:  #V = 875713   #E = 4322051\nK=2:  #V = 711870   #E = 4160100\nK=3:  #V = 581712   #E = 3915291\nK=4:  #V = 492655   #E = 3668104\nK=5:  #V = 424155   #E = 3416251\nK=6:  #V = 367361   #E = 3158776\nK=7:  #V = 319194   #E = 2902138\nK=8:  #V = 274457   #E = 2629033\nK=9:  #V = 231775   #E = 2335154\nK=10:  #V = 193406   #E = 2040738\nK=11:  #V = 159020   #E = 1753273\nK=12:  #V = 131362   #E = 1500517\nK=13:  #V = 106572   #E = 1256952\nK=14:  #V = 86302   #E = 1047053\nK=15:  #V = 68409   #E = 849471\nK=16:  #V = 53459   #E = 676076\nK=17:  #V = 40488   #E = 519077\n...\n\\endverbatim\n\n\nThe program can also save a copy of the graph at each stage by adding an\noption.\n\\verbatim\n> --savecores=[prefix]\n\\endverbatim\n\nThe resultant graphs will be saved with prefixes [prefix].K\nFor instance if prefix is out, \nThe 0-Core graph may be saved in \n\\verbatim\nout.0.1_of_4\nout.0.2_of_4\nout.0.3_of_4\nout.0.4_of_4\n\\endverbatim\n\nThe 5-Core graph will be saved in \n\\verbatim\nout.5.1_of_4\nout.5.2_of_4\nout.5.3_of_4\nout.5.4_of_4\n\\endverbatim\n\nand so on. \n\nThe range of k-Core graphs to compute can be controlled by the kmin\nand the kmax option described below.\n\nThis program can also run distributed by using\n\\verbatim\n> mpiexec -n [N machines] --hostfile [host file] ./kcore....\n\\endverbatim\nSee your MPI documentation for details on how to launch this job.\nAll machines must have access to the input graph location and the output graph \nlocation. Graphs may be on HDFS. \nIf you have problems loading HDFS files, see the \\ref FAQ.\n\n\n\\subsection Options\nRelevant options are: \n\\li \\b --graph (Required). The prefix from which to load the graph data\n\\li \\b --format (Required). The format of the input graph \n\\li \\b --ncpus (Optional. Default 2) The number of processors that will be used\nfor computation.  \n\\li \\b --savecores (Optional. Default \"\"). The target prefix to save \nthe resultant K-core graphs.\n\\li \\b --kmin (Optional. Default 0). Only output result for the K-core graph starting\n                        at K=kmin\n\\li \\b --kmax (Optional. Default Inf). Only output result for the K-core graph \n                        up to K=kmax\n\n\n\n\n\n\n\n\n\n\n\n\n\n\\section graph_analytics_triangle_coloring Graph Coloring \n\nThe graph coloring program implements a really simple graph coloring \nprocedure: each vertex reads the colors of its neighbors and takes on \nthe smallest possible color which does not conflict with its neighbors.\n\nThe procedure necessarily uses the asynchronous engine (it will never\nconverge with the synchronous engine).\n\nThe input to the system is a graph in any of the Portable graph format\ndescribed in \\ref graph_formats. It is important that the input be \"cleaned\"\nand that reverse edges are removed: i.e. if edge 1-->5 exists, edge 5-->1 should\nnot exist. (The program will run without these edge removed. But numbers\nmay be erroneous).\n\nTo color a graph, the minimal set of options required are:\n\\verbatim\n> ./simple_coloring --graph=[graph prefix] --format=[format] --output=[output prefix]\n\\endverbatim\nOutput looks like:\n\\verbatim\nNumber of vertices: 875713\nNumber of edges:    5105039\nColoring...\nCompleted Tasks: 875713\nIssued Tasks: 875713\nBlocked Issues: 0\n------------------\nJoined Tasks: 0\nColored in 42.3684 seconds\nMetrics server stopping.\n\\endverbatim\n\nObserve that the number of Completed Tasks is identical to the number of vertices.\nThis is a result of the consistency model which ensures that the entire\nvertex update is peformed \"atomically\".\n\nTne output prefix is where the output counts will be written. This\nmay be located on HDFS. For instance, if the output_prefix is \"v_out\",\nthe output files will be written to:\n\n\\verbatim\nv_out_1_of_16\nv_out_2_of_16\n...\nv_out_16_of_16\n\\endverbatim\n\nEach line in the output file contains two numbers: a Vertex ID, and the number\ncolor of the vertex.\n\nThis program can also run distributed by using\n\\verbatim\n> mpiexec -n [N machines] --hostfile [host file] ./simple_coloring ....\n\\endverbatim\nSee your MPI documentation for details on how to launch this job.\nAll machines must have access to the input graph location and the output graph \nlocation. Graphs may be on HDFS. \nIf you have problems loading HDFS files, see the \\ref FAQ.\n\n\n\\subsection Options\nRelevant options are: \n\\li \\b --graph (Required). The prefix from which to load the graph data\n\\li \\b --format (Required). The format of the input graph \n\\li \\b --ncpus (Optional. Default 2) The number of processors that will be used\nfor computation.  \n\\li \\b –-graph_opts (Optional, Default empty) Any additional graph options. See\n  --graph_help a list of options.\n\\li \\b –-engine_opts (Optional, Default empty) Any additional engine options. See\n  --engine_help a list of options.\n\nA particularly relevant option is \n\\verbatim\n--engine_opts=\"factorized=true\"\n\\endverbatim\n\nThis uses a weaker consistency setting which only guarantees that individual\n\"gather/apply/scatter\" operations are atomic, but does not guarantee atomicity\nof the entire update. As a result, this may require more updates to complete,\nbut could in practice run significantly faster.\n\n\n\n\n\n\n\n\n\n\n\\section graph_analytics_connected_component Connected Component\n\nThe connected component program can find all connected components in a \ngraph, and can also count the number of vertices (size) of each connected \ncomponent.\n\nThe input to the system is a graph in any of the Portable Graph formats\ndescribed in \\ref graph_formats.\n\nTo find connected components in a graph, the minimal set of options\nrequired are:\n\\verbatim\n> ./connected_component --graph=[graph prefix] --format=[format]\n\\endverbatim\n\nHere is a toy example, graph with 6 nodes and 5 edges:\n\\verbatim\n# example graph\n# vertices: 6 edges: 5\n1 2\n2 3\n4 5\n4 6\n5 6\n\\endverbatim\n\nAssume file name is toy_graph, the command used for running connected compnents is\n\\verbatim\n> ./connected_component --graph=toy_graph --format=tsv --saveprefix=out\n\\endverbatim\n\n\nWhen you set --saveprefix=output_prefix, the pairs of a Vertex ID and a \nComponent ID will be written to a sequence of files with prefix \noutput_prefix. This may be located on HDFS. For instance, if the \noutput_prefix is \"v_out\", the output files will be written to:\n\n\\verbatim\nout_1_of_4\nout_2_of_4\nout_3_of_4\nout_4_of_4\n\\endverbatim\n\nLet's examine the output. The first column is the node id, while the second column is it's assigned component number\n(which is also the lowest node id in this component). In our case:\n\\verbatim\n1,1\n2,1\n3,1\n4,4\n5,4\n6,4\n\\endverbatim\n\nThere are two components. The first compoent is 1,2,3 and the second component is 4,5,6 \n\nNote that this program can also run distributed by using\n\\verbatim\n> mpiexec -n [N machines] --hostfile [host file] ./connected_component ....\n\\endverbatim\nSee your MPI documentation for details on how to launch this job.\nAll machines must have access to the input graph location and the output graph \nlocation. Graphs may be on HDFS. \nIf you have problems loading HDFS files, see the \\ref FAQ.\n\n\\subsection Options\nRelevant options are: \n\\li \\b --graph (Required). The prefix from which to load the graph data\n\\li \\b --format (Required). The format of the input graph \n\\li \\b --saveprefix (Optional). If set, pairs of a Vertex ID and a Component \nID will be saved to a sequence of files with the given prefix.\n\\li \\b --ncpus (Optional. Default 2). The number of processors that will be used\nfor computation.\n\\li \\b --graph_opts (Optional, Default empty). Any additional graph options. See\n  graphlab::distributed_graph a list of options.\n  \nconnected_components_stats is a helper utility, which computes histogram of component \nsizes. \n\nUsing our toy example\n\\verbatim\n> ./connected_component_stats --graph=out\nConnected Component\n\nINFO:     mpi_tools.hpp(init:63): MPI Support was not compiled.\nINFO:     dc.cpp(init:573): Cluster of 1 instances created.\nINFO:     distributed_graph.hpp(set_ingress_method:3200): Automatically determine ingress method: grid\nLoading graph in format: adj\nINFO:     distributed_graph.hpp(load_from_posixfs:2189): Loading graph from file: ./out_1_of_4\nINFO:     distributed_graph.hpp(load_from_posixfs:2189): Loading graph from file: ./out_2_of_4\nINFO:     distributed_graph.hpp(load_from_posixfs:2189): Loading graph from file: ./out_3_of_4\nINFO:     distributed_graph.hpp(load_from_posixfs:2189): Loading graph from file: ./out_4_of_4\nINFO:     distributed_ingress_base.hpp(finalize:185): Finalizing Graph...\nINFO:     distributed_ingress_base.hpp(exchange_global_info:519): Graph info:\n\t nverts: 2\n\t nedges: 0\n\t nreplicas: 2\n\t replication factor: 1\nComplete Finalization in 0.001965\ngraph calculation time is 2.4e-05 sec\nRESULT:\nsize\tcount\n3\t2\n\\endverbatim\n\nAs expected, there are two components of size 3.\n\n\n\n\\section graph_analytics_approximate_diameter Approximate Diameter\n\nThe approximate diameter program can estimate a diameter of a graph. \nThe implemented algorithm is based on the work, \n\nU Kang, Charalampos Tsourakakis, Ana Paula Appel, Christos Faloutsos and Jure Leskovec, \nHADI: Fast Diameter Estimation and Mining in Massive Graphs with Hadoop (2008).\n\nThe input to the system is a graph in any of the Portable Graph formats\ndescribed in \\ref graph_formats.\n\nTo compute an approximate diameter of a graph, the minimal set of options\nrequired are:\n\\verbatim\n> ./approximate_diameter --graph=[graph prefix] --format=[format]\n\\endverbatim\nOutput looks like:\n\\verbatim\nApproximate graph diameter\nINFO:     synchronous_engine.hpp(start:1263): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1312):   Active vertices: 1271950\nINFO:     synchronous_engine.hpp(start:1361):    Running Aggregators\n1-th hop: 12895307 vertex pairs are reached\nINFO:     synchronous_engine.hpp(start:1263): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1312):   Active vertices: 1271950\nINFO:     synchronous_engine.hpp(start:1361):    Running Aggregators\n2-th hop: 319726269 vertex pairs are reached\nINFO:     synchronous_engine.hpp(start:1263): 0: Starting iteration: 0\nINFO:     synchronous_engine.hpp(start:1312):   Active vertices: 1271950\nINFO:     synchronous_engine.hpp(start:1361):    Running Aggregators\n3-th hop: 319769151 vertex pairs are reached\nconverge\ngraph calculation time is 40 sec\napproximate diameter is 2\n\\endverbatim\n\nThis program can also run distributed by using\n\\verbatim\n> mpiexec -n [N machines] --hostfile [host file] ./approximate_diameter ....\n\\endverbatim\nSee your MPI documentation for details on how to launch this job.\nAll machines must have access to the input graph location and the output graph \nlocation. Graphs may be on HDFS. \nIf you have problems loading HDFS files, see the \\ref FAQ.\n\n\\subsection Options\nRelevant options are: \n\\li \\b --graph (Required). The prefix from which to load the graph data\n\\li \\b --format (Required). The format of the input graph \n\\li \\b --tol (Optional. Default=1E-4). Changes the convergence tolerance for \nthe number of reached vertex pairs at each hop.\n\\li \\b --use-sketch (Optional. Default=1). If true, will use Flajolet & Martin \nbitmask to approximately count numbers of reached vertex pairs, and will require a \nsmaller memory. If false, will count exact numbers of reached vertex pairs. But \nthis will need a huge memory and be slow.\n\\li \\b --ncpus (Optional. Default 2). The number of processors that will be used\nfor computation.  \n\\li \\b --graph_opts (Optional, Default empty). Any additional graph options. See\n  graphlab::distributed_graph a list of options.\n\n\n\n\n\n\n\n\n\\section graph_analytics_partitioning Graph Partitioning \n\nThis program can partition a graph by using normalized cut.\n\nThe input to the system is a graph in any of the Portable Graph formats\ndescribed in \\ref graph_formats.\nYou can also give weights to edges with the weight format.\nFor instance in this weight format file, there are 5 edges:\n\n\\verbatim\n1 2 4.0\n2 3 1.0\n3 4 5.0\n4 5 2.0\n5 3 3.0\n\\endverbatim\n\nTo partition a graph, the minimal set of options required are:\n\n\\verbatim\n> ./partitioning --graph=[graph prefix] --format=[format]\n\\endverbatim\n\nThis program uses svd in Graphlab Collaborative Filtering Toolkit and \nkmeans in Graphlab Clustering Toolkit. The paths to the directories are \nspecified by --svd-dir and --kmeans-dir, respectively.\n\nThe program will create some intermediate files. The final partitioning\nresult is written in files named [graph prefix].result with suffix,\nfor example [graph prefix].result_1_of_4. The partitioning result \ndata consists of two columns: one for the ids and the other for the \nassigned partitions. For instance:\n\n\\verbatim\n1 0\n2 0\n3 1\n4 1\n5 1\n\\endverbatim\n\nNOTE: To run this program in a distributed setting, you must use the \n\"mpi-args\" option, not like other graphlab toolkits. \nThe graph partitioning calls other graphlab programs.\nWhen \"--mpi-args\" is set, these graphlab programs are called with \"mpiexec\" and the \nstring written after the \"mpi-args\" option.\nFor example, if you set --mpi-args=\"-n 4 --hostfile host\", the program calls the \nother graphlab programs with \"mpiexec -n 4 --hostfile host\".\n\n\\subsection Options\nRelevant options are:\n\\li \\b --graph (Required). The prefix from which to load the graph data\n\\li \\b --format (Required). The format of the input graph. If \"weight\" is \nset, the program will read the data file where each line holds [id1] [id2] \n[weight].\n\\li \\b --partitions (Optional. Default 2). The number of partitions\n\\li \\b --svd-dir (Optional. Default ../collaborative_filtering/).\nPath to the directory where Graphlab svd is located\n\\li \\b --kmeans-dir (Optional. Default ../clustering/). Path to the directory where \nGraphlab kmeans is located\n\\li \\b --ncpus (Optional. Default 2). The number of processors that will be used \nfor computation.\n\\li \\b --graph_opts (Optional, Default empty). Any additional graph options. See\n  graphlab::distributed_graph a list of options.\n\\li \\b --mpi-args (Optional, Default empty). If set, will execute mipexec with the given string.\n  \n  \n\\section graph_analytics_total_subgraph_centrality \"Total Subgraph Centrality\"\nTotal subgraph centrality was implemented by Jacob Kesinger, see additional\ndetails in his blog post.\nTotal Subgraph Communicability is a new centrality measure due to\nBenzi&Klymco [1].   For a directed graph with adjacenty matrix A, \n\n\\verbatim\nTSC_i = sum_j exp(A)_{ij} = (exp(A)*1)_i.\n\\endverbatim\n\nThis code calculates the TSC using an Arnoldi iteration on the Krylov\nsubspace {b, Ab,A*Ab, A*A*Ab, ...}  due to Saad[1], and using the new\nwarp engine from Graphlab 2.2 (without which this would have been, at\nbest, very challenging).\n\nSmall components of large graphs will have bogus answers due to\nfloating point issues.  To find the exact TSC for a particular node i,\nrun with \"--column i\" to find exp(A)*e_i; you will have to sum the\nresulting output yourself, however.\n\n\nSAMPLE INPUT:\n\\verbatim\n0\t1\n1\t2\n1\t3\n2\t4\n3\t4\n1\t0\n2\t1\n3\t1\n4\t2\n4\t3\n\\endverbatim\n\nOUTPUT:\n\\verbatim\n0 5.17784\n1 10.3319\n2 8.49789\n3 8.49789\n4 7.96807\n\\endverbatim\n\nYou can verify this in python as:\n\\verbatim\nimport scipy\nimport scipy.linalg\nA = scipy.array([[0,1,0,0,0],[1,0,1,1,0],[0,1,0,0,1],[0,1,0,0,1],[0,0,1,1,0]])\nscipy.linalg.expm2(A).sum(axis=1)\n\\endverbatim\n\n[1]: Benzi, Michele, and Christine Klymko. Total Communicability as a Centrality Measure. ArXiv e-print, February 27, 2013. arxiv\n\n[2]: Saad, Yousef. “Analysis of Some Krylov Subspace Approximations to the Matrix Exponential Operator.” SIAM Journal on Numerical Analysis 29, no. 1 (1992): 209–228.\n*/\n"
  },
  {
    "path": "toolkits/graph_analytics/graph_laplacian.cpp",
    "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n\nstd::vector vertex_degrees;\n\nstruct vdata{\n  float degree;\n  vdata() : degree(0.0){}\n  void save(graphlab::oarchive& oarc) const {\n    oarc << degree;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> degree;\n  }\n};\n\nstruct edata{\n  float weight;\n  edata() : weight(1.0){}\n  explicit edata(const float in_w):\n    weight(in_w){}\n  void save(graphlab::oarchive& oarc) const {\n    oarc << weight;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> weight;\n  }\n};\n\ntypedef graphlab::distributed_graph graph_type;\n\n\n// [vertex_id1] [vertex_id2] [weight]\n// NOTE: vertex id should start from 1.\nbool line_parser(graph_type& graph, const std::string& filename,\n    const std::string& textline) {\n  std::stringstream strm(textline);\n  size_t source = 0;\n  size_t target = 0;\n  float weight = 0.0;\n  strm >> source;\n  strm.ignore(1);\n  strm >> target;\n  strm.ignore(1);\n  strm >> weight;\n  if(source != target)\n\t  graph.add_edge(source, target, edata(weight));\n\n  return true;\n}\n\n//calculate vertex degree\nclass add_rows: public graphlab::ivertex_program, public graphlab::IS_POD_TYPE {\npublic:\n\tadd_rows() {\n  }\n\n  void init(icontext_type& context, const vertex_type& vertex,\n      const message_type& msg) {\n  }\n\n  //gather on all the edges\n  edge_dir_type gather_edges(icontext_type& context,\n      const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  }\n\n  //for each edge gather the weighted of the edge\n  float gather(icontext_type& context, const vertex_type& vertex,\n      edge_type& edge) const {\n    return edge.data().weight;\n  }\n\n  //take inverse square root\n  void apply(icontext_type& context, vertex_type& vertex,\n      const gather_type& total) {\n    vertex.data().degree = total;\n  }\n\n  edge_dir_type scatter_edges(icontext_type& context,\n      const vertex_type& vertex) const {\n      return graphlab::NO_EDGES;\n  }\n  void scatter(icontext_type& context, const vertex_type& vertex,\n      edge_type& edge) const {\n  }\n};\n\n//take inverse square root\nvoid inverse_square_root(graph_type::vertex_type& v) {\n  v.data().degree = 1.0 / sqrt(v.data().degree);\n}\n\n//multiply D^-1/2\nvoid multiply_D(graph_type::edge_type& e) {\n  const float& d1 = e.source().data().degree;\n  const float& d2 = e.target().data().degree;\n  e.data().weight = e.data().weight * d1 * d2;\n}\n\n//needed for normalization for ratio cut\nstruct max_degree{\n  float degree;\n  max_degree(): degree(0.0){}\n  explicit max_degree(float in_degree): degree(in_degree){}\n\n  max_degree& operator+=(const max_degree& other){\n    degree = std::max(degree, other.degree);\n    return *this;\n  }\n  void save(graphlab::oarchive& oarc) const {\n    oarc << degree;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> degree;\n  }\n};\nmax_degree create_max_degree(const graph_type::vertex_type& vertex) {\n  return max_degree(vertex.data().degree);\n}\n\n//for normalization for ratio cut\nfloat normalize_factor = 1.0;\nvoid normalize_weight(graph_type::edge_type& e) {\n  e.data().weight /= normalize_factor;\n}\nvoid normalize_degree(graph_type::vertex_type& v) {\n  v.data().degree /= normalize_factor;\n}\n\nstruct max_vid{\n  size_t vid;\n  max_vid(): vid(0){}\n  explicit max_vid(size_t in_vid): vid(in_vid){}\n\n  max_vid& operator+=(const max_vid& other){\n    vid = std::max(vid, other.vid);\n    return *this;\n  }\n  void save(graphlab::oarchive& oarc) const {\n    oarc << vid;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> vid;\n  }\n};\nmax_vid absolute_vertex_data(const graph_type::vertex_type& vertex) {\n  return max_vid(vertex.id());\n}\n\nclass graph_writer_normalized_cut {\npublic:\n\tgraph_writer_normalized_cut(){}\n  std::string save_vertex(graph_type::vertex_type v) {\n    std::stringstream strm;\n    size_t vid = v.id();\n    if(vid == 0)\n    \treturn \"\";\n    strm << vid << \" \" << vid << \" 2.0\\n\";\n    return strm.str();\n  }\n\n  std::string save_edge(graph_type::edge_type e) {\n    std::stringstream strm;\n    size_t source = e.source().id();\n    size_t target = e.target().id();\n    float weight = e.data().weight;\n    strm << source << \" \" << target << \" \" << weight << \"\\n\";\n    strm << target << \" \" << source << \" \" << weight << \"\\n\";\n    return strm.str();\n  }\n};\n\nclass graph_writer_ratio_cut {\npublic:\n\tgraph_writer_ratio_cut(){}\n  std::string save_vertex(graph_type::vertex_type v) {\n    std::stringstream strm;\n    size_t vid = v.id();\n    if(vid == 0)\n    \treturn \"\";\n    strm << vid << \" \" << vid << \" \" << 5.0 - v.data().degree << \"\\n\";\n    return strm.str();\n  }\n\n  std::string save_edge(graph_type::edge_type e) {\n    std::stringstream strm;\n    size_t source = e.source().id();\n    size_t target = e.target().id();\n    float weight = e.data().weight;\n    strm << source << \" \" << target << \" \" << weight << \"\\n\";\n    strm << target << \" \" << source << \" \" << weight << \"\\n\";\n    return strm.str();\n  }\n};\n\n\nint main(int argc, char** argv) {\n  std::cout << \"Construct graph Laplacian for graph partitioning.\\n\\n\";\n\n  //parse command line\n  std::string graph_dir;\n  std::string format = \"adj\";\n  bool normalized_cut = true;\n  bool ratio_cut = false;\n  graphlab::command_line_options clopts\n\t(\"Constructing graph Laplacian for graph partitioning\");\n  clopts.attach_option(\"graph\", graph_dir,\n                       \"The graph file. This is not optional. Vertex ids must start from 1 \"\n                       \"and must not skip any numbers.\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph file format. If \\\"weight\\\" is set, the program will read \"\n                       \"the data file where each line holds [id1] [id2] [weight].\");\n//  clopts.attach_option(\"normalized-cut\", normalized_cut,\n//                       \"construct graph laplacian for normalized cut\");\n//  clopts.attach_option(\"ratio-cut\", ratio_cut,\n//                       \"construct graph laplacian for ratio cut\");\n  if(!clopts.parse(argc, argv)) return EXIT_FAILURE;\n  if (graph_dir == \"\") {\n\tstd::cout << \"--graph is not optional\\n\";\n\treturn EXIT_FAILURE;\n  }\n//  if(normalized_cut == true && ratio_cut == true){\n//    std::cout << \"Both normalized-cut and ratio-cut are true. Ratio cut is selected.\\n\";\n//    normalized_cut = false;\n//  }else if(normalized_cut == false && ratio_cut == false){\n//    std::cout << \"Both normalized-cut and ratio-cut are false. Ratio cut is selected.\\n\";\n//    ratio_cut = true;\n//  }\n\n  //load graph\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  graph_type graph(dc);\n  dc.cout() << \"Loading graph in format: \"<< format << std::endl;\n  if(format == \"weight\")\n    graph.load(graph_dir, line_parser);\n  else\n    graph.load_format(graph_dir, format);\n  graph.finalize();\n\n  time_t start, end;\n  graphlab::omni_engine engine(dc, graph, \"sync\", clopts);\n  engine.signal_all();\n  time(&start);\n  engine.start();\n  if(normalized_cut == true){\n\t  graph.transform_vertices(inverse_square_root);\n\t  graph.transform_edges(multiply_D);\n  }else if(ratio_cut == true){//normalize weight for ratio cut\n//    normalize_factor = graph.map_reduce_vertices(create_max_degree).degree;\n//    graph.transform_edges(normalize_weight);\n//    graph.transform_vertices(normalize_degree);\n  }\n  time(&end);\n\n  dc.cout() << \"graph calculation time is \" << (end - start) << \" sec\\n\";\n  dc.cout() << \"writing...\\n\";\n\n  const std::string outputname = graph_dir + \".glap\";\n  if(normalized_cut == true)\n    graph.save(\n\t  outputname,\n\t  graph_writer_normalized_cut(), false, //set to true if each output file is to be gzipped\n\t  true, //whether vertices are saved\n\t  true);//whether edges are saved\n  else if(ratio_cut == true)\n    graph.save(\n\t  outputname,\n\t  graph_writer_ratio_cut(), false, //set to true if each output file is to be gzipped\n\t  true, //whether vertices are saved\n\t  true);//whether edges are saved\n\n  size_t data_num = graph.map_reduce_vertices(absolute_vertex_data).vid;\n  //#graphlab::mpi_tools::finalize();\n\n  //write the number of data\n  if (graphlab::mpi_tools::rank()==0) {\n  const std::string datanum_filename = graph_dir + \".datanum\";\n  std::ofstream ofs(datanum_filename.c_str());\n  if(!ofs) {\n    std::cout << \"can't create file for number of data\" << std::endl;\n    graphlab::mpi_tools::finalize();\n    return EXIT_FAILURE;\n  }\n  ofs << data_num;\n  }\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n}\n\n"
  },
  {
    "path": "toolkits/graph_analytics/http/index.html",
    "content": "\n\n\n\n\n\n\n\n\n\n\n\n\nTwitter Triangle Counter Results\n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n\n
 Twitter Triangle Count Results 
\n\n\n\n\n
\n  
\n   
Vertices: 
\n   
42 Million
\n  
\n\n  
\n   
Edges: 
\n   
1.5 Billion
\n  
\n\n  
\n   
Triangles:
\n   
188 Billion
\n  
\n
\n\n
\n\n
\n\n\n\n\n\n\n"
  },
  {
    "path": "toolkits/graph_analytics/http/make_jsons.m",
    "content": "clear;\nraw = importdata('directed_triangles');\n%%\ncol.vid       = 1;\ncol.in        = 3; % Triangles among people following you\ncol.out       = 2; % Triangles among peopel you follow\ncol.through   = 4;\ncol.cycle     = 5;\ncol.followers = 6;\ncol.following = 7;\n\n\n%% compute top by cycle\n\ndegree = raw(:, col.followers) + raw(:, col.following);\n[~,ind.degree] = sort(degree, 'descend');\n\n[~,ind.followers] = sort(raw(:,col.followers), 'descend');\n\n[~,ind.cycle] = sort(raw(:,col.cycle), 'descend');\n[~,ind.in] = sort(raw(:,col.in), 'descend');\n[~,ind.out] = sort(raw(:,col.out), 'descend');\n[~,ind.through] = sort(raw(:,col.through), 'descend');\n\n%%\ncluster_coeff = sum(raw(:, [2,3,4,5]),2) ./ (degree + 1);\n[~,ind.cluster_coeff] = sort(cluster_coeff, 'descend');\n\n%%\n\ncluster_coeff2 = sum(raw(:, [2,3,4,5]),2) ./ (raw(:, col.following) +1);\n[~,ind.cluster_coeff2] = sort(cluster_coeff2, 'descend');\n\n%% \n\ncluster_coeff3 = raw(:, col.through) ./ (degree + 1);\n[~,ind.cluster_coeff3] = sort(cluster_coeff3, 'descend');\n\n\n\n%% \ncluster_coeff4 = raw(:, col.in) ./ (raw(:, col.followers) + 1);\n[~,ind.cluster_coeff4] = sort(cluster_coeff4, 'descend');\n\n\n%% Render json\nnusers = 10;\n\n\nfid = fopen('top_users.json', 'w');\nfprintf(fid, '[\\n');\n\nfprintf(fid, '\\t { \"name\": \"degree\", \"label\": \"Degree\", \"values\": [\\n');\nfor i = 1:nusers\n   sep = ',\\n';\n   if(i == nusers)\n        sep = '\\n';\n   end\n   fprintf(fid, ['\\t\\t [\"%d\", \"%d\"]', sep], ...\n        [raw(ind.degree(i), col.vid), degree(ind.degree(i))]); \nend\nfprintf(fid, '\\t]},\\n');\n\nfprintf(fid, '\\t { \"name\": \"followers\", \"label\": \"Followers\", \"values\": [\\n');\nfor i = 1:nusers\n   sep = ',\\n';\n   if(i == nusers)\n        sep = '\\n';\n   end\n   fprintf(fid, ['\\t\\t [\"%d\", \"%d\"]', sep], ...\n        raw(ind.followers(i), [col.vid, col.followers])); \nend\nfprintf(fid, '\\t]},\\n');\n\n\nfprintf(fid, '\\t { \"name\": \"cycle\", \"label\": \"Cycle Triangles\", \"values\": [\\n');\nfor i = 1:nusers\n   sep = ',\\n';\n   if(i == nusers)\n        sep = '\\n';\n   end\n   fprintf(fid, ['\\t\\t [\"%d\", \"%d\"]', sep], ...\n        raw(ind.cycle(i), [col.vid, col.cycle])); \nend\nfprintf(fid, '\\t]},\\n');\n\n% fprintf(fid, '\\t { \"name\": \"in\", \"label\": \"In Triangles\", \"values\": [\\n');\n% for i = 1:nusers\n%    sep = ',\\n';\n%    if(i == nusers)\n%         sep = '\\n';\n%    end\n%    fprintf(fid, ['\\t\\t [\"%d\", \"%d\"]', sep], ...\n%         raw(ind.in(i), [col.vid, col.in])); \n% end\n% fprintf(fid, '\\t]},\\n');\n% \n% \n% fprintf(fid, '\\t { \"name\": \"out\", \"label\": \"Out Triangles\", \"values\": [\\n');\n% for i = 1:nusers\n%    sep = ',\\n';\n%    if(i == nusers)\n%         sep = '\\n';\n%    end\n%    fprintf(fid, ['\\t\\t [\"%d\", \"%d\"]', sep], ...\n%         raw(ind.out(i), [col.vid, col.out])); \n% end\n% fprintf(fid, '\\t]},\\n');\n% \n% \n% fprintf(fid, '\\t { \"name\": \"through\", \"label\": \"Through Triangles\", \"values\": [\\n');\n% for i = 1:nusers\n%    sep = ',\\n';\n%    if(i == nusers)\n%         sep = '\\n';\n%    end\n%    fprintf(fid, ['\\t\\t [\"%d\", \"%d\"]', sep], ...\n%         raw(ind.through(i), [col.vid, col.through])); \n% end\n% fprintf(fid, '\\t]},\\n');\n% \n\nfprintf(fid, '\\t { \"name\": \"cluster\", \"label\": \"Triangles / Degree\", \"values\": [\\n');\nfor i = 1:nusers\n   sep = ',\\n';\n   if(i == nusers)\n        sep = '\\n';\n   end\n   fprintf(fid, ['\\t\\t [\"%d\", \"%d\"]', sep], ...\n        [raw(ind.cluster_coeff(i), col.vid), cluster_coeff(ind.cluster_coeff(i))]); \nend\nfprintf(fid, '\\t]},\\n');\n\n\nfprintf(fid, '\\t { \"name\": \"cluster2\", \"label\": \"Triangles / Following\", \"values\": [\\n');\nfor i = 1:nusers\n   sep = ',\\n';\n   if(i == nusers)\n        sep = '\\n';\n   end\n   fprintf(fid, ['\\t\\t [\"%d\", \"%d\"]', sep], ...\n        [raw(ind.cluster_coeff2(i), col.vid), cluster_coeff2(ind.cluster_coeff2(i))]); \nend\nfprintf(fid, '\\t]},\\n');\n\nfprintf(fid, '\\t { \"name\": \"cluster3\", \"label\": \"Through Triangles / Degree\", \"values\": [\\n');\nfor i = 1:nusers\n   sep = ',\\n';\n   if(i == nusers)\n        sep = '\\n';\n   end\n   fprintf(fid, ['\\t\\t [\"%d\", \"%d\"]', sep], ...\n        [raw(ind.cluster_coeff3(i), col.vid), cluster_coeff3(ind.cluster_coeff3(i))]); \nend\nfprintf(fid, '\\t]},\\n');\n\n\n\nfprintf(fid, '\\t { \"name\": \"cluster4\", \"label\": \"In Triangles / Followers\", \"values\": [\\n');\nfor i = 1:nusers\n   sep = ',\\n';\n   if(i == nusers)\n        sep = '\\n';\n   end\n   fprintf(fid, ['\\t\\t [\"%d\", \"%d\"]', sep], ...\n        [raw(ind.cluster_coeff4(i), col.vid), cluster_coeff4(ind.cluster_coeff4(i))]); \nend\nfprintf(fid, '\\t]}\\n');\n\n\n\nfprintf(fid, ']\\n');\nfclose(fid);\n"
  },
  {
    "path": "toolkits/graph_analytics/http/style.css",
    "content": "body {\n    text-align: center;\n}\n\n#main_page {\n    width: 900px;\n    margin-left: auto;\n    margin-right: auto;\n\n}\n\n#info {\n   width: auto;\n \n}\n\n.setting {\n    width: auto;\n    display: inline-block;\n}\n\n\n#results {\n \n}\n\n\n.user_list {\n    display: inline-block;\n    border-style: solid;\n    height: 600px;\n    width: 250px;\n    vertical-align: top;\n    text-align: left;\n    margin: 5px;\n}\n\n\n\n.title {\n    padding: 10px;\n    background: black;\n    color: white;\n    text-align: center;\n}\n\n\n.contents {\n   padding: 10px;\n}\n.user_image {\n    width: 50px;\n    height: 50px;\n}\n\n.user {\n\n}\n\n.user_info {\n  padding-left: 10px;\n    display: inline-block;\n  vertical-align: top;\n}\n\n.name {\n\n}\n\n.value {\n\n\n}\n\n"
  },
  {
    "path": "toolkits/graph_analytics/http/top_users.json",
    "content": "[\n\t { \"name\": \"degree\", \"label\": \"Degree\", \"values\": [\n\t\t [\"16409683\", \"3081108\"],\n\t\t [\"19058681\", \"2997653\"],\n\t\t [\"15846407\", \"2679666\"],\n\t\t [\"813286\", \"2653045\"],\n\t\t [\"428333\", \"2450768\"],\n\t\t [\"19397785\", \"1994945\"],\n\t\t [\"783214\", \"1959765\"],\n\t\t [\"16190898\", \"1885917\"],\n\t\t [\"19757371\", \"1844564\"],\n\t\t [\"17461978\", \"1844123\"]\n\t]},\n\t { \"name\": \"followers\", \"label\": \"Followers\", \"values\": [\n\t\t [\"19058681\", \"2997470\"],\n\t\t [\"15846407\", \"2679640\"],\n\t\t [\"16409683\", \"2674870\"],\n\t\t [\"428333\", \"2450750\"],\n\t\t [\"19397785\", \"1994930\"],\n\t\t [\"783214\", \"1959710\"],\n\t\t [\"16190898\", \"1885780\"],\n\t\t [\"813286\", \"1882890\"],\n\t\t [\"19757371\", \"1844500\"],\n\t\t [\"17461978\", \"1843560\"]\n\t]},\n\t { \"name\": \"cycle\", \"label\": \"Cycle Triangles\", \"values\": [\n\t\t [\"11915432\", \"104231123\"],\n\t\t [\"14389132\", \"102710493\"],\n\t\t [\"14669398\", \"97731164\"],\n\t\t [\"21836409\", \"95138730\"],\n\t\t [\"5210841\", \"94708939\"],\n\t\t [\"15117375\", \"91658795\"],\n\t\t [\"804455\", \"90986939\"],\n\t\t [\"17850012\", \"90488711\"],\n\t\t [\"15991049\", \"89806725\"],\n\t\t [\"11622712\", \"88501346\"]\n\t]},\n\t { \"name\": \"cluster\", \"label\": \"Triangles / Degree\", \"values\": [\n\t\t [\"19176053\", \"4.484443e+03\"],\n\t\t [\"21804494\", \"4.474835e+03\"],\n\t\t [\"22903720\", \"4.459627e+03\"],\n\t\t [\"15489652\", \"4.455816e+03\"],\n\t\t [\"21773417\", \"4.455429e+03\"],\n\t\t [\"22924805\", \"4.454160e+03\"],\n\t\t [\"22707565\", \"4.445663e+03\"],\n\t\t [\"22447669\", \"4.442444e+03\"],\n\t\t [\"21772831\", \"4.441429e+03\"],\n\t\t [\"23062614\", \"4.428422e+03\"]\n\t]},\n\t { \"name\": \"cluster2\", \"label\": \"Triangles / Following\", \"values\": [\n\t\t [\"19784831\", \"6449985\"],\n\t\t [\"18498684\", \"5.962666e+06\"],\n\t\t [\"16303106\", \"5607368\"],\n\t\t [\"24285686\", \"5272692\"],\n\t\t [\"23832022\", \"4483823\"],\n\t\t [\"780457\", \"3564740\"],\n\t\t [\"21499292\", \"3207562\"],\n\t\t [\"19397785\", \"2.936561e+06\"],\n\t\t [\"17220934\", \"2.488950e+06\"],\n\t\t [\"428333\", \"2.474015e+06\"]\n\t]},\n\t { \"name\": \"cluster3\", \"label\": \"Through Triangles / Degree\", \"values\": [\n\t\t [\"21804494\", \"1.183061e+03\"],\n\t\t [\"15489652\", \"1.177176e+03\"],\n\t\t [\"22903720\", \"1.175795e+03\"],\n\t\t [\"21773417\", \"1.175569e+03\"],\n\t\t [\"21772831\", \"1.173847e+03\"],\n\t\t [\"22447669\", \"1.173059e+03\"],\n\t\t [\"17232810\", \"1.165794e+03\"],\n\t\t [\"23062614\", \"1.165705e+03\"],\n\t\t [\"14995603\", \"1.164518e+03\"],\n\t\t [\"7967282\", \"1.162090e+03\"]\n\t]},\n\t { \"name\": \"cluster4\", \"label\": \"In Triangles / Followers\", \"values\": [\n\t\t [\"15143707\", \"2.230469e+03\"],\n\t\t [\"22903720\", \"2.202895e+03\"],\n\t\t [\"31083199\", \"2.201453e+03\"],\n\t\t [\"19955829\", \"2.198339e+03\"],\n\t\t [\"22708071\", \"2.195993e+03\"],\n\t\t [\"15135065\", \"2.177517e+03\"],\n\t\t [\"21772965\", \"2.176703e+03\"],\n\t\t [\"24953387\", \"2.171151e+03\"],\n\t\t [\"22447669\", \"2.164331e+03\"],\n\t\t [\"22343399\", \"2.161766e+03\"]\n\t]}\n]\n"
  },
  {
    "path": "toolkits/graph_analytics/http/twitter_triangles.js",
    "content": "google.load(\"jquery\", \"1.5\");\n\n\n\nvar domain_str = \"http://localhost:8090\";\nvar domain_str = \"\";\nvar page_str =  \"top_users.json\";\n\n// jsonp callback required\nvar twitter_addr = \"http://api.twitter.com/1/users/lookup.json\"\n\nvar current_results = [];\nvar user_profiles = {};\n\nfunction update_domain(form) {\n    domain_str = form.inputbox.value;\n    get_top_users();\n}\n\n\nfunction refresh() {\n    get_top_users();\n    reloadStylesheets();\n    render_page();\n}\n\n// Start the rendering of the UI\ngoogle.setOnLoadCallback(function() { \n    get_top_users();\n});\n\n\nfunction get_top_users() {\n    jQuery.getJSON(domain_str + page_str, get_user_profiles).error(function() { \n        console.log(\"Unable to access \" + domain_str + \" will try again.\");\n    });\n    // .complete(function() {\n    //             setTimeout(get_top_users, update_interval);\n    //         });\n}\n\n \n\n\nfunction get_user_profiles(data) {\n    // save the original results\n    current_results = data;\n    // compute the union of all the _missing_ profiles\n    jQuery.each(current_results, function(i, list) {\n        console.log(list.name);\n        jQuery.each(list.values, function(i, pair) {\n            var id = pair[0];\n            if(user_profiles[id] == undefined) { \n                user_profiles[id] = { queried: false, is_set: false, profile: {} }; \n            }\n        });\n    });\n\n    var id_list = \"\";\n    var id_list_len = 0;\n    // Grab all _missing_ profiles\n    jQuery.each(user_profiles, function(id, obj) {\n        console.log(id);\n        if(!user_profiles[id].queried) {\n            console.log(\"Requesting: \" + id);\n            user_profiles[id].queried = true;\n            id_list += id;\n            id_list_len++;\n            if(id_list_len >= 99) {\n                jQuery.getJSON(twitter_addr + \"?callback=?\", {user_id: id_list}, process_ids);\n                id = \"\";\n                id_list_len = 0;\n            } else { id_list += \",\"; }\n        } \n    });\n    if(id_list_len > 0) {\n        jQuery.getJSON(twitter_addr + \"?callback=?\", {user_id: id_list}, process_ids);\n        id = \"\";\n        id_list_len = 0;\n    }\n} // end of get user profiles\n\n\nfunction process_ids(data) {\n    jQuery.each(data, function(i, profile) {\n        var id = profile.id;\n        user_profiles[id].is_set = true;\n        user_profiles[id].profile = profile;\n    });;\n    render_page();\n}\n\nfunction render_page() {\n    var container = $(\"#results\");\n    container.empty();\n\n    // compute the union of all the profiles\n    jQuery.each(current_results, function(i, list) {\n        console.log(\"Creating div for: \" + list.name);\n        var div_str = \n            \"
\" +\n            \"
\" + list.label + \"
\" +\n            \"
\"\n        jQuery.each(list.values, function(i, pair) {\n            var id = pair[0];\n            var count = pair[1];\n            if(user_profiles[id].is_set) {\n                var profile = user_profiles[id].profile;\n                div_str += \"
\" +\n                    \"\" +\n                    \"
\" +\n                    \"
\" +\n                    \"\" + \n                    profile.name + \n                    \"\" +\n                    \"
\" +\n                    \"
\" + count + \"
\" +\n                    \"
\" + \"
\";\n            }\n        });\n        div_str += \"
\";\n        container.append(div_str);\n    });\n\n}\n\nfunction reloadStylesheets() {\n    var queryString = '?reload=' + new Date().getTime();\n    $('link[rel=\"stylesheet\"]').each(function () {\n        this.href = this.href.replace(/\\?.*|$/, queryString);\n    });\n}\n\n\n\n\n\n\n\n"
  },
  {
    "path": "toolkits/graph_analytics/kcore.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n/**\n *\n * In this program we implement the \"k-core\" decomposition algorithm.\n * We use a parallel variant of\n * \n * V. Batagelj and M. Zaversnik, An O(m) algorithm for cores\n * decomposition of networks,\n *\n *  - Essentially, recursively remove everything with degree 1\n *  - Then recursively remove everything with degree 2\n *  - etc.\n */\n\n/*\n * Each vertex maintains a \"degree\" count. If this value\n * is 0, the vertex is \"deleted\"\n */\ntypedef int vertex_data_type;\n\n/*\n * Don't need any edges\n */\ntypedef graphlab::empty edge_data_type;\n\n/*\n * Define the type of the graph\n */\ntypedef graphlab::distributed_graph graph_type;\n\n// The current K to compute\nsize_t CURRENT_K;\n\n/*\n * The core K-core implementation.\n * The basic concept is simple.\n * Each vertex maintains a count of the number of adjacent edges.\n * If a vertex receives a message, the message contains the number of\n * adjacent edges deleted. The vertex then updates its counter.\n * If the counter falls below K, it deletes itself\n * (set the adjacent count to 0) and signals each of its neighbors\n * with a message of 1.\n */\nclass k_core :\n  public graphlab::ivertex_program,   // messages are integral\n  public graphlab::IS_POD_TYPE  {\npublic:\n  // the last received message\n  int msg;\n  \n  /* Each vertex can only signal once. I set this flag\n   * if it is the first time this vertex falls below K, so I can\n   * initiate scattering\n   */\n  bool just_deleted;\n  \n  k_core():msg(0),just_deleted(false) { }\n\n  /* The message contains the number of adjacent edges deleted.\n   * Store the message in the program, and reset the just_deleted flag\n   */\n  void init(icontext_type& context, const vertex_type& vertex,\n            const message_type& message) {\n    msg = message;\n    just_deleted = false;\n  }\n\n  // gather is never invoked\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n\n  /* On apply, if the vertex has not yet been deleted,\n   * decrement the counter on the vertex.\n   * If the adjacency count of the vertex falls below K,\n   * the vertex shall be deleted.\n   * We set the vertex data to 0 to designate that it is deleted\n   * and Set the just_deleted flag to signal the neighbors in scatter\n   */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& unused) {\n    if (vertex.data() > 0) {\n      vertex.data() -= msg;\n      if (vertex.data() < CURRENT_K) {\n        just_deleted = true;\n        vertex.data() = 0;\n      }\n    }\n  } \n\n  /*\n   * If the vertex is deleted, we signal all neighbors on the scatter\n   */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const {\n    return just_deleted ?\n      graphlab::ALL_EDGES : graphlab::NO_EDGES;\n  }\n\n  /*\n   * For each neighboring vertex, if it is not yet deleted,\n   * signal it.\n   */\n  void scatter(icontext_type& context,\n               const vertex_type& vertex,\n               edge_type& edge) const {\n    vertex_type other = edge.source().id() == vertex.id() ?\n      edge.target() : edge.source();\n    if (other.data() > 0) {\n      context.signal(other, 1);\n    }\n  }\n  \n};\n\n// type of the synchronous_engine\ntypedef graphlab::synchronous_engine engine_type;\n\n/*\n * Called before any graph operation is performed.\n * Initializes all vertex data to the number of adjacent edges.\n * Can be called from a graph.transform_vertices()\n */\nvoid initialize_vertex_values(graph_type::vertex_type& v) {\n  v.data() = v.num_in_edges() + v.num_out_edges();\n}\n\n/*\n * Signals all non-deleted vertices with degree less than K.\n * Can be called from an engine.map_reduce_vertices()\n * We return empty since no reduction is performed. Only the map.\n */\ngraphlab::empty signal_vertices_at_k(engine_type::icontext_type& ctx,\n                                     const graph_type::vertex_type& vertex) {\n  if (vertex.data() > 0 && vertex.data() < CURRENT_K) {\n    ctx.signal(vertex, 0);\n  }\n  return graphlab::empty();\n}\n\n/*\n * Counts the number of un-deleted vertices.\n */\nsize_t count_active_vertices(const graph_type::vertex_type& vertex) {\n  return vertex.data() > 0;\n}\n\n/*\n * Counts the degree of each un-deleted vertex. Half of this\n * will be the size of the K-core graph.\n */\nsize_t double_count_active_edges(const graph_type::vertex_type& vertex) {\n  return (size_t) vertex.data();\n}\n\n\n\n/*\n * Saves the graph in a tsv format with the condition that\n * the adjacent vertices have not yet been deleted.\n * This allows saving of the k-core graph.\n */\nstruct save_core_at_k {\n  std::string save_vertex(graph_type::vertex_type) { return \"\"; }\n  std::string save_edge(graph_type::edge_type e) {\n    if (e.source().data() > 0 && e.target().data() > 0) {\n      return graphlab::tostr(e.source().id()) + \"\\t\" +\n        graphlab::tostr(e.target().id()) + \"\\n\";\n    }\n    else return \"\";\n  }\n};\n    \nint main(int argc, char** argv) {\n  std::cout << \"Computes a k-core decomposition of a graph.\\n\\n\";\n\n  graphlab::command_line_options clopts\n    (\"K-Core decomposition. This program \"\n     \"computes the K-Core decomposition of a graph, for K ranging from [kmin] \"\n     \"to [kmax]. The size of the remaining K-core graph at each K is printed. \"\n     \"The [savecores] allow the saving of each K-Core graph in a TSV format\"\n     );\n  std::string prefix, format;\n  size_t kmin = 0;\n  size_t kmax = (size_t)(-1);\n  std::string savecores;\n  clopts.attach_option(\"graph\", prefix,\n                       \"Graph input. reads all graphs matching prefix*\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph format\");\n  clopts.attach_option(\"kmin\", kmin,\n                       \"Compute the k-Core for k the range [kmin,kmax]\");\n  clopts.attach_option(\"kmax\", kmax,\n                       \"Compute the k-Core for k the range [kmin,kmax]\");\n  clopts.attach_option(\"savecores\", savecores,\n                       \"If non-empty, will save tsv of each core with prefix [savecores].K.\");\n\n  if(!clopts.parse(argc, argv)) return EXIT_FAILURE;\n  if (prefix == \"\") {\n    std::cout << \"--graph is not optional\\n\";\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  else if (format == \"\") {\n    std::cout << \"--format is not optional\\n\";\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  else if (kmax < kmin) {\n    std::cout << \"kmax must be at least as large as kmin\\n\";\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  // Initialize control plane using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  // load graph\n  graph_type graph(dc, clopts);\n  graph.load_format(prefix, format);\n  graph.finalize();\n  dc.cout() << \"Number of vertices: \" << graph.num_vertices() << std::endl\n            << \"Number of edges:    \" << graph.num_edges() << std::endl;\n\n  graphlab::timer ti;\n\n  graphlab::synchronous_engine engine(dc, graph, clopts);\n\n  // initialize the vertex data with the degree\n  graph.transform_vertices(initialize_vertex_values);\n\n  // for each K value\n  for (CURRENT_K = kmin; CURRENT_K <= kmax; CURRENT_K++) {\n    // signal all vertices with degree less than K\n    engine.map_reduce_vertices(signal_vertices_at_k);\n    // recursively delete all vertices with degree less than K\n    engine.start();\n    // count the number of vertices and edges remaining\n    size_t numv = graph.map_reduce_vertices(count_active_vertices);\n    size_t nume = graph.map_reduce_vertices(double_count_active_edges) / 2;\n    if (numv == 0) break;\n    // Output the size of the graph\n    dc.cout() << \"K=\" << CURRENT_K << \":  #V = \"\n              << numv << \"   #E = \" << nume << std::endl;\n\n    // Saves the result if requested\n    if (savecores != \"\") {\n      graph.save(savecores + \".\" + graphlab::tostr(CURRENT_K) + \".\",\n                 save_core_at_k(),\n                 false, /* no compression */ \n                 false, /* do not save vertex */\n                 true, /* save edge */ \n                 clopts.get_ncpus()); /* one file per machine */\n    }\n  }\n  \n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // End of main\n\n"
  },
  {
    "path": "toolkits/graph_analytics/pagerank.cpp",
    "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \n#include \n#include \n\n#include \n// #include \n\n// Global random reset probability\ndouble RESET_PROB = 0.15;\n\ndouble TOLERANCE = 1.0E-2;\n\nsize_t ITERATIONS = 0;\n\nbool USE_DELTA_CACHE = false;\n\n// The vertex data is just the pagerank value (a double)\ntypedef double vertex_data_type;\n\n// There is no edge data in the pagerank application\ntypedef graphlab::empty edge_data_type;\n\n// The graph type is determined by the vertex and edge data types\ntypedef graphlab::distributed_graph graph_type;\n\n/*\n * A simple function used by graph.transform_vertices(init_vertex);\n * to initialize the vertes data.\n */\nvoid init_vertex(graph_type::vertex_type& vertex) { vertex.data() = 1; }\n\n\n\n/*\n * The factorized page rank update function extends ivertex_program\n * specifying the:\n *\n *   1) graph_type\n *   2) gather_type: double (returned by the gather function). Note\n *      that the gather type is not strictly needed here since it is\n *      assumed to be the same as the vertex_data_type unless\n *      otherwise specified\n *\n * In addition ivertex program also takes a message type which is\n * assumed to be empty. Since we do not need messages no message type\n * is provided.\n *\n * pagerank also extends graphlab::IS_POD_TYPE (is plain old data type)\n * which tells graphlab that the pagerank program can be serialized\n * (converted to a byte stream) by directly reading its in memory\n * representation.  If a vertex program does not exted\n * graphlab::IS_POD_TYPE it must implement load and save functions.\n */\nclass pagerank :\n  public graphlab::ivertex_program {\n\n  double last_change;\npublic:\n\n  /**\n   * Gather only in edges.\n   */\n  edge_dir_type gather_edges(icontext_type& context,\n                              const vertex_type& vertex) const {\n    return graphlab::IN_EDGES;\n  } // end of Gather edges\n\n\n  /* Gather the weighted rank of the adjacent page   */\n  double gather(icontext_type& context, const vertex_type& vertex,\n               edge_type& edge) const {\n    return (edge.source().data() / edge.source().num_out_edges());\n  }\n\n  /* Use the total rank of adjacent pages to update this page */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& total) {\n\n    const double newval = (1.0 - RESET_PROB) * total + RESET_PROB;\n    last_change = (newval - vertex.data());\n    vertex.data() = newval;\n    if (ITERATIONS) context.signal(vertex);\n  }\n\n  /* The scatter edges depend on whether the pagerank has converged */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const {\n    // If an iteration counter is set then\n    if (ITERATIONS) return graphlab::NO_EDGES;\n    // In the dynamic case we run scatter on out edges if the we need\n    // to maintain the delta cache or the tolerance is above bound.\n    if(USE_DELTA_CACHE || std::fabs(last_change) > TOLERANCE ) {\n      return graphlab::OUT_EDGES;\n    } else {\n      return graphlab::NO_EDGES;\n    }\n  }\n\n  /* The scatter function just signal adjacent pages */\n  void scatter(icontext_type& context, const vertex_type& vertex,\n               edge_type& edge) const {\n    if(USE_DELTA_CACHE) {\n      context.post_delta(edge.target(), last_change);\n    }\n\n    if(last_change > TOLERANCE || last_change < -TOLERANCE) {\n        context.signal(edge.target());\n    } else {\n      context.signal(edge.target()); //, std::fabs(last_change));\n    }\n  }\n\n  void save(graphlab::oarchive& oarc) const {\n    // If we are using iterations as a counter then we do not need to\n    // move the last change in the vertex program along with the\n    // vertex data.\n    if (ITERATIONS == 0) oarc << last_change;\n  }\n\n  void load(graphlab::iarchive& iarc) {\n    if (ITERATIONS == 0) iarc >> last_change;\n  }\n\n}; // end of factorized_pagerank update functor\n\n\n/*\n * We want to save the final graph so we define a write which will be\n * used in graph.save(\"path/prefix\", pagerank_writer()) to save the graph.\n */\nstruct pagerank_writer {\n  std::string save_vertex(graph_type::vertex_type v) {\n    std::stringstream strm;\n    strm << v.id() << \"\\t\" << v.data() << \"\\n\";\n    return strm.str();\n  }\n  std::string save_edge(graph_type::edge_type e) { return \"\"; }\n}; // end of pagerank writer\n\n\ndouble map_rank(const graph_type::vertex_type& v) { return v.data(); }\n\n\ndouble pagerank_sum(graph_type::vertex_type v) {\n  return v.data();\n}\n\nint main(int argc, char** argv) {\n  // Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  global_logger().set_log_level(LOG_INFO);\n\n  // Parse command line options -----------------------------------------------\n  graphlab::command_line_options clopts(\"PageRank algorithm.\");\n  std::string graph_dir;\n  std::string format = \"adj\";\n  std::string exec_type = \"synchronous\";\n  clopts.attach_option(\"graph\", graph_dir,\n                       \"The graph file.  If none is provided \"\n                       \"then a toy graph will be created\");\n  clopts.add_positional(\"graph\");\n  clopts.attach_option(\"engine\", exec_type,\n                       \"The engine type synchronous or asynchronous\");\n  clopts.attach_option(\"tol\", TOLERANCE,\n                       \"The permissible change at convergence.\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph file format\");\n  size_t powerlaw = 0;\n  clopts.attach_option(\"powerlaw\", powerlaw,\n                       \"Generate a synthetic powerlaw out-degree graph. \");\n  clopts.attach_option(\"iterations\", ITERATIONS,\n                       \"If set, will force the use of the synchronous engine\"\n                       \"overriding any engine option set by the --engine parameter. \"\n                       \"Runs complete (non-dynamic) PageRank for a fixed \"\n                       \"number of iterations. Also overrides the iterations \"\n                       \"option in the engine\");\n  clopts.attach_option(\"use_delta\", USE_DELTA_CACHE,\n                       \"Use the delta cache to reduce time in gather.\");\n  std::string saveprefix;\n  clopts.attach_option(\"saveprefix\", saveprefix,\n                       \"If set, will save the resultant pagerank to a \"\n                       \"sequence of files with prefix saveprefix\");\n\n  if(!clopts.parse(argc, argv)) {\n    dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n\n  // Enable gather caching in the engine\n  clopts.get_engine_args().set_option(\"use_cache\", USE_DELTA_CACHE);\n\n  if (ITERATIONS) {\n    // make sure this is the synchronous engine\n    dc.cout() << \"--iterations set. Forcing Synchronous engine, and running \"\n              << \"for \" << ITERATIONS << \" iterations.\" << std::endl;\n    clopts.get_engine_args().set_option(\"type\", \"synchronous\");\n    clopts.get_engine_args().set_option(\"max_iterations\", ITERATIONS);\n    clopts.get_engine_args().set_option(\"sched_allv\", true);\n  }\n\n  // Build the graph ----------------------------------------------------------\n  graph_type graph(dc, clopts);\n  if(powerlaw > 0) { // make a synthetic graph\n    dc.cout() << \"Loading synthetic Powerlaw graph.\" << std::endl;\n    graph.load_synthetic_powerlaw(powerlaw, false, 2.1, 100000000);\n  }\n  else if (graph_dir.length() > 0) { // Load the graph from a file\n    dc.cout() << \"Loading graph in format: \"<< format << std::endl;\n    graph.load_format(graph_dir, format);\n  }\n  else {\n    dc.cout() << \"graph or powerlaw option must be specified\" << std::endl;\n    clopts.print_description();\n    return 0;\n  }\n  // must call finalize before querying the graph\n  graph.finalize();\n  dc.cout() << \"#vertices: \" << graph.num_vertices()\n            << \" #edges:\" << graph.num_edges() << std::endl;\n\n  // Initialize the vertex data\n  graph.transform_vertices(init_vertex);\n\n  // Running The Engine -------------------------------------------------------\n  graphlab::omni_engine engine(dc, graph, exec_type, clopts);\n  engine.signal_all();\n  engine.start();\n  const double runtime = engine.elapsed_seconds();\n  dc.cout() << \"Finished Running engine in \" << runtime\n            << \" seconds.\" << std::endl;\n\n\n  const double total_rank = graph.map_reduce_vertices(map_rank);\n  std::cout << \"Total rank: \" << total_rank << std::endl;\n\n  // Save the final graph -----------------------------------------------------\n  if (saveprefix != \"\") {\n    graph.save(saveprefix, pagerank_writer(),\n               false,    // do not gzip\n               true,     // save vertices\n               false);   // do not save edges\n  }\n\n  double totalpr = graph.map_reduce_vertices(pagerank_sum);\n  std::cout << \"Totalpr = \" << totalpr << \"\\n\";\n\n  // Tear-down communication layer and quit -----------------------------------\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // End of main\n\n\n// We render this entire program in the documentation\n\n\n"
  },
  {
    "path": "toolkits/graph_analytics/partitioning.cpp",
    "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\n//remove assigned options from arguments\nstd::string get_arg_str_without(int argc, char** argv,\n    std::vector remove_opts) {\n  std::stringstream strm;\n  bool skip_next = false;\n  for (int i = 1; i < argc; ++i) {\n    bool skip = false;\n    for (size_t j = 0; j < remove_opts.size(); ++j) {\n      std::string with_equal = remove_opts[j] + \"=\";\n      if (strncmp(with_equal.c_str(), argv[i], with_equal.size()) == 0) {\n        skip = true;\n      } else if (strncmp(remove_opts[j].c_str(), argv[i], remove_opts[j].size())\n          == 0) {\n        skip = true;\n        skip_next = true;\n      }\n    }\n    if (skip == false && skip_next == false) {\n      strm << argv[i] << \" \";\n    } else if (skip == false && skip_next == true) {\n      skip_next = false;\n    }\n  }\n  return strm.str();\n}\n\nbool call_graph_laplacian(const std::string& mpi_args,\n    const std::string& filename, const std::string& format,\n    const bool normalized_cut, const bool ratio_cut, const std::string& args) {\n  std::stringstream strm;\n  if(mpi_args.length() > 0)\n    strm << \"mpiexec \" << mpi_args << \" \";\n  strm << \"./graph_laplacian \";\n  strm << \" --graph=\" << filename;\n  strm << \" --format=\" << format;\n//  strm << \" --normalized-cut=\" << normalized_cut;\n//  strm << \" --ratio-cut=\" << ratio_cut;\n  strm << \" \" << args;\n  std::cout << \"CALLING >\" << strm.str() << std::endl;\n  int sys_ret = system(strm.str().c_str());\n  if (sys_ret != 0) {\n    std::cout << \"system call fails\" << std::endl;\n    return false;\n  }\n\n  return true;\n}\n\nvoid make_initial_vector_file(const std::string& filename, const size_t num_data){\n  std::ofstream ofs((filename + \".init\").c_str());\n  for(size_t i=0;i 0)\n    strm << \"mpiexec \" << mpi_args << \" \";\n  strm << svd_dir << \"svd \" + filename + \".glap\";\n  strm << \" --rows=\" << num_data+1;\n  strm << \" --cols=\" << num_data;\n  strm << \" --nsv=\" << num_clusters;\n  strm << \" --nv=\" << rank;\n  strm << \" --max_iter=4\";\n  strm << \" --quiet=1\";\n  strm << \" --save_vectors=1\";\n  strm << \" --ortho_repeats=3\";\n  //strm << \" --id=1\";\n  //strm << \" --prediction=\" << filename + \".\";\n  strm << \" --prediction=\" << filename;\n  strm << \" --initial_vector=\" << filename + \".init\";\n  strm << \" \" << args;\n  std::cout << \"CALLING >\" << strm.str() << std::endl;\n  int sys_ret = system(strm.str().c_str());\n  if (sys_ret != 0) {\n    std::cout << \"system call fails\" << std::endl;\n    return false;\n  }\n\n  return true;\n}\n\nbool call_eigen_vector_normalization(const std::string& mpi_args,\n    const std::string& filename, const size_t num_clusters, const size_t rank,\n    const size_t num_data, const std::string& args) {\n  std::stringstream strm;\n  if(mpi_args.length() > 0)\n    strm << \"mpiexec \" << mpi_args << \" \";\n  strm << \"./eigen_vector_normalization\";\n  strm << \" --data=\" << filename;\n  strm << \" --clusters=\" << num_clusters;\n  strm << \" --rank=\" << rank;\n  strm << \" --data-num=\" << num_data;\n  strm << \" \" << args;\n  std::cout << \"CALLING >\" << strm.str() << std::endl;\n  int sys_ret = system(strm.str().c_str());\n  if (sys_ret != 0) {\n    std::cout << \"system call fails\" << std::endl;\n    return false;\n  }\n\n  return true;\n}\n\nbool call_kmeans(const std::string& mpi_args, const std::string& filename,\n    const std::string& kmeans_dir, const size_t num_clusters,\n    const std::string& args) {\n  //call svd\n  std::stringstream strm;\n  if(mpi_args.length() > 0)\n    strm << \"mpiexec \" << mpi_args << \" \";\n  strm << kmeans_dir << \"kmeans \";\n  strm << \" --data \" << filename << \".compressed\";\n  strm << \" --clusters \" << num_clusters;\n  strm << \" --output-data \" << filename << \".result\";\n  strm << \" --id=1\";\n  strm << \" \" << args;\n  std::cout << \"CALLING >\" << strm.str() << std::endl;\n  int sys_ret = system(strm.str().c_str());\n  if (sys_ret != 0) {\n    std::cout << \"system call fails\" << std::endl;\n    return false;\n  }\n  return true;\n}\n\n//select good rank\nint get_lanczos_rank(const size_t num_clusters, const size_t num_data) {\n  size_t rank = 1;\n  if (num_data < 1000) {\n    if (num_clusters + 10 <= num_data)\n      rank = num_clusters + 10;\n    else\n      rank = num_data;\n  } else if (num_data < 10000) {\n    rank = num_clusters + 25;\n  } else if (num_data < 100000) {\n    rank = num_clusters + 50;\n  } else if (num_data < 1000000) {\n    rank = num_clusters + 80;\n  } else {\n    rank = num_clusters + 100;\n  }\n  return rank;\n//  return num_clusters + 1;\n}\n\nint main(int argc, char** argv) {\n  std::cout << \"Graph partitioning (normalized cut)\\n\\n\";\n\n  std::string graph_dir;\n  std::string format = \"adj\";\n  std::string svd_dir = \"../collaborative_filtering/\";\n  std::string kmeans_dir = \"../clustering/\";\n  std::string mpi_args;\n  size_t num_partitions = 2;\n  bool normalized_cut = true;\n  bool ratio_cut = false;\n  size_t sv = 0;\n  //parse command line\n  graphlab::command_line_options clopts(\n          \"Graph partitioning (normalized cut)\");\n  clopts.attach_option(\"graph\", graph_dir,\n                       \"The graph file. This is not optional. Vertex ids must start from 1 \"\n                       \"and must not skip any numbers.\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph file format. If \\\"weight\\\" is set, the program will read \"\n                       \"the data file where each line holds [id1] [id2] [weight].\");\n  clopts.attach_option(\"partitions\", num_partitions,\n                       \"The number of partitions to create\");\n  clopts.attach_option(\"svd-dir\", svd_dir,\n                       \"Path to the directory of Graphlab svd\");\n  clopts.attach_option(\"kmeans-dir\", kmeans_dir,\n                       \"Path to the directory of Graphlab kmeans\");\n  clopts.attach_option(\"mpi-args\", mpi_args,\n                       \"If set, will execute mipexec with the given arguments. \"\n                       \"For example, --mpi-args=\\\"-n [N machines] --hostfile [host file]\\\"\");\n  clopts.attach_option(\"sv\", sv,\n                       \"Number of vectors in each iteration in the Lanczos svd.\");\n//  clopts.attach_option(\"normalized-cut\", normalized_cut,\n//                       \"do normalized cut\");\n//  clopts.attach_option(\"ratio-cut\", ratio_cut,\n//                       \"do ratio cut\");\n  if (!clopts.parse(argc, argv))\n    return EXIT_FAILURE;\n  if (graph_dir == \"\") {\n    std::cout << \"--graph is not optional\\n\";\n    return EXIT_FAILURE;\n  }\n//  if(normalized_cut == true && ratio_cut == true){\n//    std::cout << \"Both normalized-cut and ratio-cut are true. Ratio cut is selected.\\n\";\n//    normalized_cut = false;\n//  }else if(normalized_cut == false && ratio_cut == false){\n//    std::cout << \"Both normalized-cut and ratio-cut are false. Ratio cut is selected.\\n\";\n//    ratio_cut = true;\n//  }\n  std::vector remove_opts;\n  remove_opts.push_back(\"--graph\");\n  remove_opts.push_back(\"--format\");\n  remove_opts.push_back(\"--svd-dir\");\n  remove_opts.push_back(\"--kmeans-dir\");\n  remove_opts.push_back(\"--partitions\");\n  remove_opts.push_back(\"--mpi-args\");\n  remove_opts.push_back(\"--sv\");\n//  remove_opts.push_back(\"--normalized-cut\");\n//  remove_opts.push_back(\"--ratio-cut\");\n  std::string other_args = get_arg_str_without(argc, argv, remove_opts);\n\n  //construct graph laplacian\n  if (call_graph_laplacian(mpi_args, graph_dir, format, normalized_cut,\n      ratio_cut, other_args) == false) {\n    return EXIT_FAILURE;\n  }\n\n  //eigen value decomposition\n  //read number of data\n  size_t num_data = 0;\n  const std::string datanum_filename = graph_dir + \".datanum\";\n  std::ifstream ifs(datanum_filename.c_str());\n  if (!ifs) {\n    std::cout << \"can't read number of data.\" << std::endl;\n    return false;\n  }\n  ifs >> num_data;\n  //determine the rank of Lanczos method\n  if(sv == 0){\n    sv = get_lanczos_rank(num_partitions, num_data);\n  }else{\n    if(sv < num_partitions)\n      sv = num_partitions;\n  }\n  if (call_svd(mpi_args, graph_dir, svd_dir, num_partitions, sv, num_data,\n      other_args) == false) {\n    return EXIT_FAILURE;\n  }\n  if (call_eigen_vector_normalization(mpi_args, graph_dir, num_partitions, sv,\n      num_data, other_args) == false) {\n    return EXIT_FAILURE;\n  }\n\n  //kmeans\n  if (call_kmeans(mpi_args, graph_dir, kmeans_dir, num_partitions, other_args)\n      == false) {\n    return EXIT_FAILURE;\n  }\n\n  return EXIT_SUCCESS;\n}\n\n"
  },
  {
    "path": "toolkits/graph_analytics/saturation_ordered_coloring.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n/*\n * Graph coloring algorithm, such that vertex programs are scheduled in \n * order of saturation (number of differently colored adjacent nodes),\n * in \"waves\" of execution\n */\n\n#include \n#include \n#include \n#include \n\n\ntypedef graphlab::vertex_id_type color_type;\n\n/*\n * Vertex data: color and degree of node\n */\ntypedef struct {\n  int color;\n  int saturation;\n  std::set adj_colors;\n   // serialize\n  void save(graphlab::oarchive& oarc) const {\n    oarc << color << saturation << adj_colors;\n  }\n\n  // deserialize\n  void load(graphlab::iarchive& iarc) {\n    iarc >> color >> saturation >> adj_colors;\n  }\n\n} vertex_data_type;\n\n\n/*\n * no edge data\n */\n\n#define UNCOLORED -1\n\ntypedef graphlab::empty edge_data_type;\nbool EDGE_CONSISTENT = false;\n\nsigned int max_saturation = 0;\nsigned int current_saturation = 0;\nbool still_uncolored = true;\nbool next_component = false;\nunsigned int uncoloredvs = 0;\nunsigned int uncoloredvs_old = 0;\nstd::set used_colors;\nstd::set sats;\n/*\n * This is the gathering type which accumulates an (unordered) set of\n * all neighboring colors \n * It is a simple wrapper around a boost::unordered_set with\n * an operator+= which simply performs a set union.\n *\n * This struct can be significantly accelerated for small sets.\n * Small collections of vertex IDs should not require the overhead\n * of the unordered_set.\n */\nstruct set_union_gather {\n  boost::unordered_set colors;\n\n  /*\n   * Combining with another collection of vertices.\n   * Union it into the current set.\n   */\n  set_union_gather& operator+=(const set_union_gather& other) {\n    foreach(graphlab::vertex_id_type othervid, other.colors) {\n      colors.insert(othervid);\n    }\n    return *this;\n  }\n  \n  // serialize\n  void save(graphlab::oarchive& oarc) const {\n    oarc << colors;\n  }\n\n  // deserialize\n  void load(graphlab::iarchive& iarc) {\n    iarc >> colors;\n  }\n};\n/*\n * Define the type of the graph\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n/*\n * On gather, we accumulate a set of all adjacent colors.\n */\nclass graph_coloring:\n      public graphlab::ivertex_program,\n      /* I have no data. Just force it to POD */\n      public graphlab::IS_POD_TYPE  {\npublic:\n  // Gather on all edges\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  } \n\n  /*\n   * For each edge, figure out the ID of the \"other\" vertex\n   * and accumulate a set of the neighborhood vertex IDs.\n   */\n  gather_type gather(icontext_type& context,\n                     const vertex_type& vertex,\n                     edge_type& edge) const {\n    set_union_gather gather;\n    color_type other_color = edge.source().id() == vertex.id() ?\n                                 edge.target().data().color: edge.source().data().color;\n    gather.colors.insert(other_color);\n\n    return gather;\n  }\n\n  /*\n   * the gather result now contains the colors in the neighborhood.\n   * pick a different color and store it \n   */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& neighborhood) {\n    // find the smallest color not described in the neighborhood\n    size_t neighborhoodsize = neighborhood.colors.size();\n    for (color_type curcolor = 0; curcolor < neighborhoodsize + 1; ++curcolor) {\n      if (neighborhood.colors.count(curcolor) == 0) {\n        used_colors.insert(curcolor);\n        vertex.data().color = curcolor;\n        break;\n      }\n    }\n  }\n\n  edge_dir_type scatter_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    if (EDGE_CONSISTENT) return graphlab::NO_EDGES;\n    else return graphlab::ALL_EDGES;\n  } \n\n\n  /*\n   * For each edge, count the intersection of the neighborhood of the\n   * adjacent vertices. This is the number of triangles this edge is involved\n   * in.\n   */\n  void scatter(icontext_type& context,\n              const vertex_type& vertex,\n              edge_type& edge) const {\n    // both points have different colors!\n    if (edge.source().data().color == edge.target().data().color) {\n      context.signal(edge.source().id() == vertex.id() ? \n                      edge.target() : edge.source());\n    }\n\n    else if (vertex.id() == edge.target().id()) {\n      edge.source().data().adj_colors.insert(vertex.data().color);\n      edge.source().data().saturation = edge.source().data().adj_colors.size();\n    }\n    else {\n      edge.target().data().adj_colors.insert(vertex.data().color);\n      edge.target().data().saturation = edge.target().data().adj_colors.size();\n    }\n      \n    //}\n  }\n};\n\nvoid initialize_vertex_values(graph_type::vertex_type& v) {\n  v.data().saturation = 0;\n  v.data().color = UNCOLORED;\n}\n\nvoid calculate_saturation(graph_type::vertex_type& v) {\n\n  if (v.data().saturation > 0 && v.data().color == UNCOLORED) {\n    next_component = false;\n    sats.insert(v.data().saturation);\n    if (v.data().saturation > max_saturation) {\n      max_saturation = v.data().saturation;\n    }\n  }\n  else if (v.data().color == UNCOLORED)\n    still_uncolored = true;\n}\n\n\n/*\n * A saver which saves a file where each line is a vid / color pair\n */\nstruct save_colors{\n  std::string save_vertex(graph_type::vertex_type v) { \n    return graphlab::tostr(v.id()) + \"\\t\" +\n           graphlab::tostr(v.data().color) + \"\\n\";\n  }\n  std::string save_edge(graph_type::edge_type e) {\n    return \"\";\n  }\n};\n\ntypedef graphlab::async_consistent_engine engine_type;\n\ngraphlab::empty signal_vertices_at_saturation (engine_type::icontext_type& ctx,\n                                     const graph_type::vertex_type& vertex) {\n  if (vertex.data().saturation == current_saturation && vertex.data().color == UNCOLORED) {\n    ctx.signal(vertex);\n  }\n  return graphlab::empty();\n}\n\ngraphlab::empty signal_uncolored (engine_type::icontext_type& ctx,\n                                     const graph_type::vertex_type& vertex) {\n  if (vertex.data().color == UNCOLORED) {\n    ctx.signal(vertex);\n  }\n  return graphlab::empty();\n}\n\nstruct max_deg_vertex_reducer: public graphlab::IS_POD_TYPE {\n  size_t degree;\n  graphlab::vertex_id_type vid;\n  max_deg_vertex_reducer& operator+=(const max_deg_vertex_reducer& other) {\n    if (degree < other.degree) {\n      (*this) = other;\n    }\n    return (*this);\n  }\n};\n\nmax_deg_vertex_reducer find_max_deg_vertex(const graph_type::vertex_type vtx) {\n  //we only want uncolored  \n  if (vtx.data().color == UNCOLORED) {\n    max_deg_vertex_reducer red;\n    red.degree = vtx.num_out_edges();\n    red.vid = vtx.id();\n    return red;\n  }\n}\n\n/**************************************************************************/\n/*                                                                        */\n/*                         Validation   Functions                         */\n/*                                                                        */\n/**************************************************************************/\nsize_t validate_conflict(graph_type::edge_type& edge) {\n  return edge.source().data().color == edge.target().data().color;\n}\n\n\nint main(int argc, char** argv) {\n\n  //global_logger().set_log_level(LOG_INFO);\n\n  // Initialize control plane using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n\n  dc.cout() << \"This program computes a simple graph coloring of a \"\n            \"provided graph.\\n\\n\";\n\n  graphlab::command_line_options clopts(\"Graph coloring. \"\n    \"Given a graph, this program computes a graph coloring of the graph.\"\n    \"The Asynchronous engine is used.\");\n  std::string prefix, format;\n  std::string output;\n  float alpha = 2.1;\n  size_t powerlaw = 0;\n  clopts.attach_option(\"graph\", prefix,\n                       \"Graph input. reads all graphs matching prefix*\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph format\");\n   clopts.attach_option(\"output\", output,\n                       \"A prefix to save the output.\");\n   clopts.attach_option(\"powerlaw\", powerlaw,\n                       \"Generate a synthetic powerlaw out-degree graph. \");\n   clopts.attach_option(\"alpha\", alpha,\n                       \"Alpha in powerlaw distrubution\");\n  clopts.attach_option(\"edgescope\", EDGE_CONSISTENT,\n                       \"Use Locking. \");\n    \n  if(!clopts.parse(argc, argv)) return EXIT_FAILURE;\n  if (prefix.length() == 0 && powerlaw == 0) {\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  if (output == \"\") {\n    dc.cout() << \"Warning! Output will not be saved\\n\";\n  }\n\n\n  graphlab::launch_metric_server();\n  // load graph\n  graph_type graph(dc, clopts);\n\n  if(powerlaw > 0) { // make a synthetic graph\n    dc.cout() << \"Loading synthetic Powerlaw graph.\" << std::endl;\n    graph.load_synthetic_powerlaw(powerlaw, false, alpha, 100000000);\n  } else { // Load the graph from a file\n    if (prefix == \"\") {\n      dc.cout() << \"--graph is not optional\\n\";\n      return EXIT_FAILURE;\n    }\n    else if (format == \"\") {\n      dc.cout() << \"--format is not optional\\n\";\n      return EXIT_FAILURE;\n    }\n    graph.load_format(prefix, format);\n  }\n  graph.finalize();\n\n  dc.cout() << \"Number of vertices: \" << graph.num_vertices() << std::endl\n    << \"Number of edges:    \" << graph.num_edges() << std::endl;\n\n  graphlab::timer ti;\n\n  \n  dc.cout() << \"Initialising vertex data...\" <(find_max_deg_vertex);\n  \n\n  // create engine to count the number of triangles\n  dc.cout() << \"Coloring...\" << std::endl;\n  if (EDGE_CONSISTENT) {\n    clopts.get_engine_args().set_option(\"factorized\", false);\n  } else {\n    clopts.get_engine_args().set_option(\"factorized\", true);\n  } \n  graphlab::async_consistent_engine engine(dc, graph, clopts);\n\n  engine.signal(v.vid);\n  engine.start();\n\n  //Continue execution until all nodes are colored\n  while (still_uncolored) {\n    still_uncolored = false;\n    next_component = true;\n    graph.transform_vertices(calculate_saturation); \n    for (int x = max_saturation; x > 0; x--) {    \n      if (sats.find(x) != sats.end()) {\n        current_saturation = x;\n        engine.map_reduce_vertices(signal_vertices_at_saturation);\n      }\n    }\n    engine.start();\n    \n    /*\n     *Colors the component with the highest degree and then colors all other components \n     *randomly if the graph has more than one component.\n     */\n    if (next_component) {\n      dc.cout() << \"Colouring other components...\" <(signal_uncolored);\n      engine.start();\n      still_uncolored = false;\n      break;\n    }\n\n    max_saturation = 0;\n    sats.clear();\n  } \n\n  size_t conflict_count = graph.map_reduce_edges(validate_conflict);\n\n  dc.cout() << \"Colored in \" << ti.current_time() << \" seconds\" << std::endl;\n  dc.cout() << \"Colored using \" << used_colors.size() << \" colors\" << std::endl;\n  dc.cout() << \"Num conflicts = \" << conflict_count << \"\\n\";\n  if (output != \"\") {\n    graph.save(output,\n              save_colors(),\n              false, /* no compression */\n              true, /* save vertex */\n              false, /* do not save edge */\n              1); /* one file per machine */\n  }\n  \n  graphlab::stop_metric_server();\n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // End of main\n"
  },
  {
    "path": "toolkits/graph_analytics/simple_coloring.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n\n\ntypedef graphlab::vertex_id_type color_type;\n\n/*\n * no edge data\n */\ntypedef graphlab::empty edge_data_type;\nbool EDGE_CONSISTENT = false;\n\nstd::set used_colors;\n/*\n * This is the gathering type which accumulates an (unordered) set of\n * all neighboring colors \n * It is a simple wrapper around a boost::unordered_set with\n * an operator+= which simply performs a set union.\n *\n * This struct can be significantly accelerated for small sets.\n * Small collections of vertex IDs should not require the overhead\n * of the unordered_set.\n */\nstruct set_union_gather {\n  boost::unordered_set colors;\n\n  /*\n   * Combining with another collection of vertices.\n   * Union it into the current set.\n   */\n  set_union_gather& operator+=(const set_union_gather& other) {\n    foreach(graphlab::vertex_id_type othervid, other.colors) {\n      colors.insert(othervid);\n    }\n    return *this;\n  }\n  \n  // serialize\n  void save(graphlab::oarchive& oarc) const {\n    oarc << colors;\n  }\n\n  // deserialize\n  void load(graphlab::iarchive& iarc) {\n    iarc >> colors;\n  }\n};\n\n/*\n * Define the type of the graph\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n/*\n * On gather, we accumulate a set of all adjacent colors.\n */\nclass graph_coloring:\n      public graphlab::ivertex_program,\n      /* I have no data. Just force it to POD */\n      public graphlab::IS_POD_TYPE  {\npublic:\n  // Gather on all edges\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  } \n\n  /*\n   * For each edge, figure out the ID of the \"other\" vertex\n   * and accumulate a set of the neighborhood vertex IDs.\n   */\n  gather_type gather(icontext_type& context,\n                     const vertex_type& vertex,\n                     edge_type& edge) const {\n    set_union_gather gather;\n    color_type other_color = edge.source().id() == vertex.id() ?\n                                 edge.target().data(): edge.source().data();\n    // vertex_id_type otherid= edge.source().id() == vertex.id() ?\n    //                              edge.target().id(): edge.source().id();\n     gather.colors.insert(other_color);\n    return gather;\n  }\n\n  /*\n   * the gather result now contains the colors in the neighborhood.\n   * pick a different color and store it \n   */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& neighborhood) {\n    // find the smallest color not described in the neighborhood\n    size_t neighborhoodsize = neighborhood.colors.size();\n    for (color_type curcolor = 0; curcolor < neighborhoodsize + 1; ++curcolor) {\n      if (neighborhood.colors.count(curcolor) == 0) {\n        used_colors.insert(curcolor);\n        vertex.data() = curcolor;\n        break;\n      }\n    }\n  }\n\n\n  edge_dir_type scatter_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    if (EDGE_CONSISTENT) return graphlab::NO_EDGES;\n    else return graphlab::ALL_EDGES;\n  } \n\n\n  /*\n   * For each edge, count the intersection of the neighborhood of the\n   * adjacent vertices. This is the number of triangles this edge is involved\n   * in.\n   */\n  void scatter(icontext_type& context,\n              const vertex_type& vertex,\n              edge_type& edge) const {\n    // both points have different colors!\n    if (edge.source().data() == edge.target().data()) {\n      context.signal(edge.source().id() == vertex.id() ? \n                      edge.target() : edge.source());\n    }\n  }\n};\n\n\n\n\n/*\n * A saver which saves a file where each line is a vid / color pair\n */\nstruct save_colors{\n  std::string save_vertex(graph_type::vertex_type v) { \n    return graphlab::tostr(v.id()) + \"\\t\" +\n           graphlab::tostr(v.data()) + \"\\n\";\n  }\n  std::string save_edge(graph_type::edge_type e) {\n    return \"\";\n  }\n};\n\n\n/**************************************************************************/\n/*                                                                        */\n/*                         Validation   Functions                         */\n/*                                                                        */\n/**************************************************************************/\nsize_t validate_conflict(graph_type::edge_type& edge) {\n  return edge.source().data() == edge.target().data();\n}\n\n\nint main(int argc, char** argv) {\n\n  //global_logger().set_log_level(LOG_INFO);\n\n  // Initialize control plane using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n\n  dc.cout() << \"This program computes a simple graph coloring of a\"\n            \"provided graph.\\n\\n\";\n\n  graphlab::command_line_options clopts(\"Graph coloring. \"\n    \"Given a graph, this program computes a graph coloring of the graph.\"\n    \"The Asynchronous engine is used.\");\n  std::string prefix, format;\n  std::string output;\n  float alpha = 2.1;\n  size_t powerlaw = 0;\n  clopts.attach_option(\"graph\", prefix,\n                       \"Graph input. reads all graphs matching prefix*\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph format\");\n   clopts.attach_option(\"output\", output,\n                       \"A prefix to save the output.\");\n   clopts.attach_option(\"powerlaw\", powerlaw,\n                       \"Generate a synthetic powerlaw out-degree graph. \");\n      clopts.attach_option(\"alpha\", alpha,\n                       \"Alpha in powerlaw distrubution\");\n  clopts.attach_option(\"edgescope\", EDGE_CONSISTENT,\n                       \"Use Locking. \");\n    \n  if(!clopts.parse(argc, argv)) return EXIT_FAILURE;\n  if (prefix.length() == 0 && powerlaw == 0) {\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  if (output == \"\") {\n    dc.cout() << \"Warning! Output will not be saved\\n\";\n  }\n\n\n  graphlab::launch_metric_server();\n  // load graph\n  graph_type graph(dc, clopts);\n\n  if(powerlaw > 0) { // make a synthetic graph\n    dc.cout() << \"Loading synthetic Powerlaw graph.\" << std::endl;\n    graph.load_synthetic_powerlaw(powerlaw, false, alpha, 100000000);\n  } else { // Load the graph from a file\n    if (prefix == \"\") {\n      dc.cout() << \"--graph is not optional\\n\";\n      return EXIT_FAILURE;\n    }\n    else if (format == \"\") {\n      dc.cout() << \"--format is not optional\\n\";\n      return EXIT_FAILURE;\n    }\n    graph.load_format(prefix, format);\n  }\n  graph.finalize();\n\n  dc.cout() << \"Number of vertices: \" << graph.num_vertices() << std::endl\n    << \"Number of edges:    \" << graph.num_edges() << std::endl;\n\n  graphlab::timer ti;\n  \n  // create engine to count the number of triangles\n  dc.cout() << \"Coloring...\" << std::endl;\n  if (EDGE_CONSISTENT) {\n    clopts.get_engine_args().set_option(\"factorized\", false);\n  } else {\n    clopts.get_engine_args().set_option(\"factorized\", true);\n  } \n  graphlab::async_consistent_engine engine(dc, graph, clopts);\n  engine.signal_all();\n  engine.start();\n\n\n  dc.cout() << \"Colored in \" << ti.current_time() << \" seconds\" << std::endl;\n  dc.cout() << \"Colored using \" << used_colors.size() << \" colors\" << std::endl;\n\t\t  \n  size_t conflict_count = graph.map_reduce_edges(validate_conflict);\n  dc.cout() << \"Num conflicts = \" << conflict_count << \"\\n\";\n  if (output != \"\") {\n    graph.save(output,\n              save_colors(),\n              false, /* no compression */\n              true, /* save vertex */\n              false, /* do not save edge */\n              1); /* one file per machine */\n  }\n  \n  graphlab::stop_metric_server();\n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // End of main\n\n"
  },
  {
    "path": "toolkits/graph_analytics/simple_undirected_triangle_count.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n/**\n *  \n * In this program we implement the \"hash-table\" version of the\n * \"edge-iterator\" algorithm described in\n * \n *    T. Schank. Algorithmic Aspects of Triangle-Based Network Analysis.\n *    Phd in computer science, University Karlsruhe, 2007.\n *\n * The procedure is quite straightforward:\n *   - each vertex maintains a list of all of its neighbors in a hash table.\n *   - For each edge (u,v) in the graph, count the number of intersections\n *     of the neighbor set on u and the neighbor set on v.\n *   - We store the size of the intersection on the edge.\n * \n * This will count every triangle exactly 3 times. Summing across all the\n * edges and dividing by 3 gives the desired result.\n *\n * The preprocessing stage take O(|E|) time, and it has been shown that this\n * algorithm takes $O(|E|^(3/2))$ time.\n *\n * If we only require total counts, we can introduce a optimization that is\n * similar to the \"forward\" algorithm\n * described in thesis above. Instead of maintaining a complete list of all\n * neighbors, each vertex only maintains a list of all neighbors with\n * ID greater than itself. This implicitly generates a topological sort\n * of the graph.\n *\n * Then you can see that each triangle\n *\n * \\verbatim\n  \n     A----->C\n     |     ^\n     |   /\n     v /\n     B\n   \n * \\endverbatim\n * Must be counted only once. (Only when processing edge AB, can one\n * observe that A and B have intersecting out-neighbor sets).\n *\n *\n * \\note The implementation here is built to be easy to understand\n * and not necessarily optimal. In particular the unordered_set is slow\n * for small number of entries. There is a much more efficient\n * (and substantially more complicated) version in undirected_triangle_count.cpp\n */\n\n/*\n * Each vertex maintains a list of all its neighbors.\n * and a final count for the number of triangles it is involved in\n */\nstruct vertex_data_type {\n  vertex_data_type():num_triangles(0) { }\n  // A list of all its neighbors\n  boost::unordered_set vid_set;\n  // The number of triangles this vertex is involved it.\n  // only used if \"per vertex counting\" is used\n  size_t num_triangles;\n  \n  void save(graphlab::oarchive &oarc) const {\n    oarc << vid_set << num_triangles;\n  }\n  void load(graphlab::iarchive &iarc) {\n    iarc >> vid_set >> num_triangles;\n  }\n};\n\n\n/*\n * Each edge is simply a counter of triangles\n */\ntypedef size_t edge_data_type;\n\n// To collect the set of neighbors, we need a message type which is\n// basically a set of vertex IDs\n\nbool PER_VERTEX_COUNT = false;\n\n\n/*\n * This is the gathering type which accumulates an (unordered) set of\n * all neighboring vertices.\n * It is a simple wrapper around a boost::unordered_set with\n * an operator+= which simply performs a set union.\n *\n * This struct can be significantly accelerated for small sets.\n * Small collections of vertex IDs should not require the overhead\n * of the unordered_set.\n */\nstruct set_union_gather {\n  boost::unordered_set vid_set;\n\n  /*\n   * Combining with another collection of vertices.\n   * Union it into the current set.\n   */\n  set_union_gather& operator+=(const set_union_gather& other) {\n    foreach(graphlab::vertex_id_type othervid, other.vid_set) {\n      vid_set.insert(othervid);\n    }\n    return *this;\n  }\n  \n  // serialize\n  void save(graphlab::oarchive& oarc) const {\n    oarc << vid_set;\n  }\n\n  // deserialize\n  void load(graphlab::iarchive& iarc) {\n    iarc >> vid_set;\n  }\n};\n\n/*\n * Define the type of the graph\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n/*\n * This class implements the triangle counting algorithm as described in\n * the header. On gather, we accumulate a set of all adjacent vertices.\n * If per_vertex output is not necessary, we can use the optimization\n * where each vertex only accumulates neighbors with greater vertex IDs.\n */\nclass triangle_count :\n      public graphlab::ivertex_program,\n      /* I have no data. Just force it to POD */\n      public graphlab::IS_POD_TYPE  {\npublic:\n  // Gather on all edges\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  } \n\n  /*\n   * For each edge, figure out the ID of the \"other\" vertex\n   * and accumulate a set of the neighborhood vertex IDs.\n   */\n  gather_type gather(icontext_type& context,\n                     const vertex_type& vertex,\n                     edge_type& edge) const {\n    set_union_gather gather;\n    // Insert the opposite end of the edge IF the opposite end has\n    // ID greater than the current vertex\n    // If we are getting per vertex counts, we need the entire neighborhood\n    vertex_id_type otherid = edge.source().id() == vertex.id() ?\n                             edge.target().id() : edge.source().id();\n    if (PER_VERTEX_COUNT ||\n        otherid > vertex.id()) gather.vid_set.insert(otherid);\n    return gather;\n  }\n\n  /*\n   * the gather result now contains the vertex IDs in the neighborhood.\n   * store it on the vertex. \n   */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& neighborhood) {\n    vertex.data().vid_set = neighborhood.vid_set;\n  } // end of apply\n\n  /*\n   * Scatter over all edges to compute the intersection.\n   * I only need to touch each edge once, so if I scatter just on the\n   * out edges, that is sufficient.\n   */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const {\n    return graphlab::OUT_EDGES;\n  }\n\n\n  /*\n   * Computes the size of the intersection of two unordered sets\n   */\n  static size_t count_set_intersect(\n               const boost::unordered_set& smaller_set,\n               const boost::unordered_set& larger_set) {\n    size_t count = 0;\n    foreach(vertex_id_type vid, smaller_set) {\n      count += larger_set.count(vid);\n    }\n    return count;\n  }\n\n  /*\n   * For each edge, count the intersection of the neighborhood of the\n   * adjacent vertices. This is the number of triangles this edge is involved\n   * in.\n   */\n  void scatter(icontext_type& context,\n              const vertex_type& vertex,\n              edge_type& edge) const {\n    const vertex_data_type& srclist = edge.source().data();\n    const vertex_data_type& targetlist = edge.target().data();\n    if (srclist.vid_set.size() >= targetlist.vid_set.size()) {\n      edge.data() = count_set_intersect(targetlist.vid_set, srclist.vid_set);\n    }\n    else {\n      edge.data() = count_set_intersect(srclist.vid_set, targetlist.vid_set);\n    }\n  }\n};\n\n\n\n/*\n * This class is used in a second engine call if per vertex counts are needed.\n * The number of triangles a vertex is involved in can be computed easily\n * by summing over the number of triangles each adjacent edge is involved in\n * and dividing by 2. \n */\nclass get_per_vertex_count :\n      public graphlab::ivertex_program,\n      /* I have no data. Just force it to POD */\n      public graphlab::IS_POD_TYPE  {\npublic:\n  // Gather on all edges\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  }\n  // We gather the number of triangles each edge is involved in\n  size_t gather(icontext_type& context,\n                     const vertex_type& vertex,\n                     edge_type& edge) const {\n    return edge.data();\n  }\n\n  /* the gather result is the total sum of the number of triangles\n   * each adjacent edge is involved in . Dividing by 2 gives the\n   * desired result.\n   */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& num_triangles) {\n    vertex.data().num_triangles = num_triangles / 2;\n  }\n\n  // No scatter\n  edge_dir_type scatter_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n\n\n};\n\n\n/* Used to sum over all the edges in the graph in a\n * map_reduce_edges call\n * to get the total number of triangles\n */\nsize_t get_edge_data(const graph_type::edge_type& e) {\n  return e.data();\n}\n\n\n\n/*\n * A saver which saves a file where each line is a vid / # triangles pair\n */\nstruct save_triangle_count{\n  std::string save_vertex(graph_type::vertex_type v) { \n    return graphlab::tostr(v.id()) + \"\\t\" +\n           graphlab::tostr(v.data().num_triangles) + \"\\n\";\n  }\n  std::string save_edge(graph_type::edge_type e) {\n    return \"\";\n  }\n};\n\n\nint main(int argc, char** argv) {\n  std::cout << \"This program counts the exact number of triangles in the \"\n            \"provided graph.\\n\\n\";\n\n  graphlab::command_line_options clopts(\"Exact Triangle Counting. \"\n    \"Given a graph, this program computes the total number of triangles \"\n    \"in the graph. An option (per_vertex) is also provided which \"\n    \"computes for each vertex, the number of triangles it is involved in.\"\n    \"The algorithm assumes that each undirected edge appears exactly once \"\n    \"in the graph input. If edges may appear more than once, this procedure \"\n    \"will over count.\");\n  std::string prefix, format;\n  std::string per_vertex;\n  clopts.attach_option(\"graph\", prefix,\n                       \"Graph input. reads all graphs matching prefix*\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph format\");\n  clopts.attach_option(\"per_vertex\", per_vertex,\n                       \"If not empty, will count the number of \"\n                       \"triangles each vertex belongs to and \"\n                       \"save to file with prefix \\\"[per_vertex]\\\". \"\n                       \"The algorithm used is slightly different \"\n                       \"and thus will be a little slower\");\n  \n  if(!clopts.parse(argc, argv)) return EXIT_FAILURE;\n  if (prefix == \"\") {\n    std::cout << \"--graph is not optional\\n\";\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  else if (format == \"\") {\n    std::cout << \"--format is not optional\\n\";\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n\n\n  if (per_vertex != \"\") PER_VERTEX_COUNT = true;\n  // Initialize control plane using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n  graphlab::launch_metric_server();\n  // load graph\n  graph_type graph(dc, clopts);\n  graph.load_format(prefix, format);\n  graph.finalize();\n  dc.cout() << \"Number of vertices: \" << graph.num_vertices() << std::endl\n            << \"Number of edges:    \" << graph.num_edges() << std::endl;\n\n  graphlab::timer ti;\n  \n  // create engine to count the number of triangles\n  dc.cout() << \"Counting Triangles...\" << std::endl;\n  graphlab::synchronous_engine engine(dc, graph, clopts);\n  engine.signal_all();\n  engine.start();\n\n  dc.cout() << \"Counted in \" << ti.current_time() << \" seconds\" << std::endl;\n\n  if (PER_VERTEX_COUNT == false) {\n    size_t count = graph.map_reduce_edges(get_edge_data);\n    dc.cout() << count << \" Triangles\"  << std::endl;\n  }\n  else {\n    graphlab::synchronous_engine engine(dc, graph, clopts);\n    engine.signal_all();\n    engine.start();\n    graph.save(per_vertex,\n            save_triangle_count(),\n            false, /* no compression */\n            true, /* save vertex */\n            false, /* do not save edge */\n            1); /* one file per machine */\n\n  }\n  \n  graphlab::stop_metric_server();\n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // End of main\n\n"
  },
  {
    "path": "toolkits/graph_analytics/sssp.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \n#include \n#include \n\n\n#include \n\n\n/**\n * \\brief The type used to measure distances in the graph.\n */\ntypedef float distance_type;\n\n/**\n * \\brief The current distance of the vertex.\n */\nstruct vertex_data : graphlab::IS_POD_TYPE {\n  distance_type dist;\n  vertex_data(distance_type dist = std::numeric_limits::max()) :\n    dist(dist) { }\n}; // end of vertex data\n\n\n\n/**\n * \\brief The distance associated with the edge.\n */\nstruct edge_data : graphlab::IS_POD_TYPE {\n  distance_type dist;\n  edge_data(distance_type dist = 1) : dist(dist) { }\n}; // end of edge data\n\n\n/**\n * \\brief The graph type encodes the distances between vertices and\n * edges\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n/**\n * \\brief Get the other vertex in the edge.\n */\ninline graph_type::vertex_type\nget_other_vertex(const graph_type::edge_type& edge,\n                 const graph_type::vertex_type& vertex) {\n  return vertex.id() == edge.source().id()? edge.target() : edge.source();\n}\n\n\n/**\n * \\brief Use directed or undireced edges.\n */\nbool DIRECTED_SSSP = false;\n\n\n/**\n * \\brief This class is used as the gather type.\n */\nstruct min_distance_type : graphlab::IS_POD_TYPE {\n  distance_type dist;\n  min_distance_type(distance_type dist = \n                    std::numeric_limits::max()) : dist(dist) { }\n  min_distance_type& operator+=(const min_distance_type& other) {\n    dist = std::min(dist, other.dist);\n    return *this;\n  }\n};\n\n\n/**\n * \\brief The single source shortest path vertex program.\n */\nclass sssp :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\n  distance_type min_dist;\n  bool changed;\npublic:\n\n\n  void init(icontext_type& context, const vertex_type& vertex,\n            const min_distance_type& msg) {\n    min_dist = msg.dist;\n  } \n\n  /**\n   * \\brief We use the messaging model to compute the SSSP update\n   */\n  edge_dir_type gather_edges(icontext_type& context, \n                             const vertex_type& vertex) const { \n    return graphlab::NO_EDGES;\n  }; // end of gather_edges \n\n\n  // /** \n  //  * \\brief Collect the distance to the neighbor\n  //  */\n  // min_distance_type gather(icontext_type& context, const vertex_type& vertex, \n  //                          edge_type& edge) const {\n  //   return min_distance_type(edge.data() + \n  //                            get_other_vertex(edge, vertex).data());\n  // } // end of gather function\n\n\n  /**\n   * \\brief If the distance is smaller then update\n   */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const graphlab::empty& empty) {\n    changed = false;\n    if(vertex.data().dist > min_dist) {\n      changed = true;\n      vertex.data().dist = min_dist;\n    }\n  }\n\n  /**\n   * \\brief Determine if SSSP should run on all edges or just in edges\n   */\n  edge_dir_type scatter_edges(icontext_type& context, \n                             const vertex_type& vertex) const {\n    if(changed)\n      return DIRECTED_SSSP? graphlab::OUT_EDGES : graphlab::ALL_EDGES; \n    else return graphlab::NO_EDGES;\n  }; // end of scatter_edges\n\n  /**\n   * \\brief The scatter function just signal adjacent pages \n   */\n  void scatter(icontext_type& context, const vertex_type& vertex,\n               edge_type& edge) const {\n    const vertex_type other = get_other_vertex(edge, vertex);\n    distance_type newd = vertex.data().dist + edge.data().dist;\n    if (other.data().dist > newd) {\n      const min_distance_type msg(newd);\n      context.signal(other, msg);\n    }\n  } // end of scatter\n\n}; // end of shortest path vertex program\n\n\n\n\n/**\n * \\brief We want to save the final graph so we define a write which will be\n * used in graph.save(\"path/prefix\", pagerank_writer()) to save the graph.\n */\nstruct shortest_path_writer {\n  std::string save_vertex(const graph_type::vertex_type& vtx) {\n    std::stringstream strm;\n    strm << vtx.id() << \"\\t\" << vtx.data().dist << \"\\n\";\n    return strm.str();\n  }\n  std::string save_edge(graph_type::edge_type e) { return \"\"; }\n}; // end of shortest_path_writer\n\n\n\nstruct max_deg_vertex_reducer: public graphlab::IS_POD_TYPE {\n  size_t degree;\n  graphlab::vertex_id_type vid;\n  max_deg_vertex_reducer& operator+=(const max_deg_vertex_reducer& other) {\n    if (degree < other.degree) {\n      (*this) = other;\n    }\n    return (*this);\n  }\n};\n\nmax_deg_vertex_reducer find_max_deg_vertex(const graph_type::vertex_type vtx) {\n  max_deg_vertex_reducer red;\n  red.degree = vtx.num_in_edges() + vtx.num_out_edges();\n  red.vid = vtx.id();\n  return red;\n}\n\nint main(int argc, char** argv) {\n  // Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  global_logger().set_log_level(LOG_INFO);\n\n  // Parse command line options -----------------------------------------------\n  graphlab::command_line_options \n    clopts(\"Single Source Shortest Path Algorithm.\");\n  std::string graph_dir;\n  std::string format = \"adj\";\n  std::string exec_type = \"synchronous\";\n  size_t powerlaw = 0;\n  std::vector sources;\n  bool max_degree_source = false;\n  clopts.attach_option(\"graph\", graph_dir,\n                       \"The graph file.  If none is provided \"\n                       \"then a toy graph will be created\");\n  clopts.add_positional(\"graph\");\n  clopts.attach_option(\"format\", format,\n                       \"graph format\");\n  clopts.attach_option(\"source\", sources,\n                       \"The source vertices\");\n  clopts.attach_option(\"max_degree_source\", max_degree_source,\n                       \"Add the vertex with maximum degree as a source\");\n\n  clopts.add_positional(\"source\");\n\n  clopts.attach_option(\"directed\", DIRECTED_SSSP,\n                       \"Treat edges as directed.\");\n\n  clopts.attach_option(\"engine\", exec_type, \n                       \"The engine type synchronous or asynchronous\");\n \n  \n  clopts.attach_option(\"powerlaw\", powerlaw,\n                       \"Generate a synthetic powerlaw out-degree graph. \");\n  std::string saveprefix;\n  clopts.attach_option(\"saveprefix\", saveprefix,\n                       \"If set, will save the resultant pagerank to a \"\n                       \"sequence of files with prefix saveprefix\");\n\n  if(!clopts.parse(argc, argv)) {\n    dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n\n  // Build the graph ----------------------------------------------------------\n  graph_type graph(dc, clopts);\n  if(powerlaw > 0) { // make a synthetic graph\n    dc.cout() << \"Loading synthetic Powerlaw graph.\" << std::endl;\n    graph.load_synthetic_powerlaw(powerlaw, false, 2, 100000000);\n  } else if (graph_dir.length() > 0) { // Load the graph from a file\n    dc.cout() << \"Loading graph in format: \"<< format << std::endl;\n    graph.load_format(graph_dir, format);\n  } else {\n    dc.cout() << \"graph or powerlaw option must be specified\" << std::endl;\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  // must call finalize before querying the graph\n  graph.finalize();\n  dc.cout() << \"#vertices:  \" << graph.num_vertices() << std::endl\n            << \"#edges:     \" << graph.num_edges() << std::endl;\n\n\n\n  if(sources.empty()) {\n    if (max_degree_source == false) {\n      dc.cout()\n        << \"No source vertex provided. Adding vertex 0 as source\" \n        << std::endl;\n      sources.push_back(0);\n    }\n  }\n\n  if (max_degree_source) {\n    max_deg_vertex_reducer v = graph.map_reduce_vertices(find_max_deg_vertex);\n    dc.cout()\n      << \"No source vertex provided.  Using highest degree vertex \" << v.vid << \" as source.\"\n      << std::endl;\n    sources.push_back(v.vid);\n  }\n\n\n\n  // Running The Engine -------------------------------------------------------\n  graphlab::omni_engine engine(dc, graph, exec_type, clopts);\n\n\n  \n  // Signal all the vertices in the source set\n  for(size_t i = 0; i < sources.size(); ++i) {\n    engine.signal(sources[i], min_distance_type(0));\n  }\n\n  engine.start();\n  const float runtime = engine.elapsed_seconds();\n  dc.cout() << \"Finished Running engine in \" << runtime\n            << \" seconds.\" << std::endl;\n\n\n  // Save the final graph -----------------------------------------------------\n  if (saveprefix != \"\") {\n    graph.save(saveprefix, shortest_path_writer(),\n               false,    // do not gzip\n               true,     // save vertices\n               false);   // do not save edges\n  }\n\n  // Tear-down communication layer and quit -----------------------------------\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // End of main\n\n\n// We render this entire program in the documentation\n\n\n\n"
  },
  {
    "path": "toolkits/graph_analytics/undirected_triangle_count.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n#include \n#include \n#include \n/**\n *  \n * In this program we implement the \"hash-set\" version of the\n * \"edge-iterator\" algorithm described in\n * \n *    T. Schank. Algorithmic Aspects of Triangle-Based Network Analysis.\n *    Phd in computer science, University Karlsruhe, 2007.\n *\n * The procedure is quite straightforward:\n *   - each vertex maintains a list of all of its neighbors in a hash set.\n *   - For each edge (u,v) in the graph, count the number of intersections\n *     of the neighbor set on u and the neighbor set on v.\n *   - We store the size of the intersection on the edge.\n * \n * This will count every triangle exactly 3 times. Summing across all the\n * edges and dividing by 3 gives the desired result.\n *\n * The preprocessing stage take O(|E|) time, and it has been shown that this\n * algorithm takes $O(|E|^(3/2))$ time.\n *\n * If we only require total counts, we can introduce a optimization that is\n * similar to the \"forward\" algorithm\n * described in thesis above. Instead of maintaining a complete list of all\n * neighbors, each vertex only maintains a list of all neighbors with\n * ID greater than itself. This implicitly generates a topological sort\n * of the graph.\n *\n * Then you can see that each triangle\n *\n * \\verbatim\n  \n     A----->C\n     |     ^\n     |   /\n     v /\n     B\n   \n * \\endverbatim\n * Must be counted only once. (Only when processing edge AB, can one\n * observe that A and B have intersecting out-neighbor sets).\n */\n \n\n// Radix sort implementation from https://github.com/gorset/radix\n// Thanks to Erik Gorset\n//\n/*\nCopyright 2011 Erik Gorset. All rights reserved.\n\nRedistribution and use in source and binary forms, with or without modification, are\npermitted provided that the following conditions are met:\n\n1. Redistributions of source code must retain the above copyright notice, this list of\nconditions and the following disclaimer.\n\n2. Redistributions in binary form must reproduce the above copyright notice, this list\nof conditions and the following disclaimer in the documentation and/or other materials\nprovided with the distribution.\n\nTHIS SOFTWARE IS PROVIDED BY Erik Gorset ``AS IS'' AND ANY EXPRESS OR IMPLIED\nWARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND\nFITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL Erik Gorset OR\nCONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\nCONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\nSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON\nANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING\nNEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF\nADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\nThe views and conclusions contained in the software and documentation are those of the\nauthors and should not be interpreted as representing official policies, either expressed\nor implied, of Erik Gorset.\n*/\nvoid radix_sort(graphlab::vertex_id_type *array, int offset, int end, int shift) {\n    int x, y;\n    graphlab::vertex_id_type value, temp;\n    int last[256] = { 0 }, pointer[256];\n\n    for (x=offset; x> shift) & 0xFF];\n    }\n\n    last[0] += offset;\n    pointer[0] = offset;\n    for (x=1; x<256; ++x) {\n        pointer[x] = last[x-1];\n        last[x] += last[x-1];\n    }\n\n    for (x=0; x<256; ++x) {\n        while (pointer[x] != last[x]) {\n            value = array[pointer[x]];\n            y = (value >> shift) & 0xFF;\n            while (x != y) {\n                temp = array[pointer[y]];\n                array[pointer[y]++] = value;\n                value = temp;\n                y = (value >> shift) & 0xFF;\n            }\n            array[pointer[x]++] = value;\n        }\n    }\n\n    if (shift > 0) {\n        shift -= 8;\n        for (x=0; x<256; ++x) {\n            temp = x > 0 ? pointer[x] - pointer[x-1] : pointer[0] - offset;\n            if (temp > 64) {\n                radix_sort(array, pointer[x] - temp, pointer[x], shift);\n            } else if (temp > 1) {\n                std::sort(array + (pointer[x] - temp), array + pointer[x]);\n                //insertion_sort(array, pointer[x] - temp, pointer[x]);\n            }\n        }\n    }\n}\n\nsize_t HASH_THRESHOLD = 64;\n\n// We on each vertex, either a vector of sorted VIDs\n// or a hash set (cuckoo hash) of VIDs.\n// If the number of elements is greater than HASH_THRESHOLD,\n// the hash set is used. Otherwise the vector is used.\nstruct vid_vector{\n  std::vector vid_vec;\n  graphlab::hopscotch_set *cset;\n  vid_vector(): cset(NULL) { }\n  vid_vector(const vid_vector& v):cset(NULL) {\n    (*this) = v;\n  }\n\n  vid_vector& operator=(const vid_vector& v) {\n    if (this == &v) return *this;\n    vid_vec = v.vid_vec;\n    if (v.cset != NULL) {\n      // allocate the cuckoo set if the other side is using a cuckoo set\n      // or clear if I alrady have one\n      if (cset == NULL) {\n        cset = new graphlab::hopscotch_set(HASH_THRESHOLD);\n      }\n      else {\n        cset->clear();\n      }\n      (*cset) = *(v.cset);\n    }\n    else {\n      // if the other side is not using a cuckoo set, lets not use a cuckoo set\n      // either\n      if (cset != NULL) {\n        delete cset;\n        cset = NULL;\n      }\n    }\n    return *this;\n  }\n\n  ~vid_vector() {\n    if (cset != NULL) delete cset;\n  }\n\n  // assigns a vector of vertex IDs to this storage.\n  // this function will clear the contents of the vid_vector\n  // and reconstruct it.\n  // If the assigned values has length >= HASH_THRESHOLD,\n  // we will allocate a cuckoo set to store it. Otherwise,\n  // we just store a sorted vector\n  void assign(const std::vector& vec) {\n    clear();\n    if (vec.size() >= HASH_THRESHOLD) {\n        // move to cset\n        cset = new graphlab::hopscotch_set(HASH_THRESHOLD);\n        foreach (graphlab::vertex_id_type v, vec) {\n          cset->insert(v);\n        }\n    }\n    else {\n      vid_vec = vec;\n      if (vid_vec.size() > 64) {\n        radix_sort(&(vid_vec[0]), 0, vid_vec.size(), 24);\n      }\n      else {\n        std::sort(vid_vec.begin(), vid_vec.end());\n      }\n      std::vector::iterator new_end = std::unique(vid_vec.begin(),\n                                               vid_vec.end());\n      vid_vec.erase(new_end, vid_vec.end());\n    }\n  }\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << (cset != NULL);\n    if (cset == NULL) oarc << vid_vec;\n    else oarc << (*cset);\n  }\n\n\n  void clear() {\n    vid_vec.clear();\n    if (cset != NULL) {\n      delete cset;\n      cset = NULL;\n    }\n  }\n\n  size_t size() const {\n    return cset == NULL ? vid_vec.size() : cset->size();\n  }\n\n  void load(graphlab::iarchive& iarc) {\n    clear();\n    bool hascset;\n    iarc >> hascset;\n    if (!hascset) iarc >> vid_vec;\n    else {\n      cset = new graphlab::hopscotch_set(HASH_THRESHOLD);\n      iarc >> (*cset);\n    }\n  }\n};\n\n/*\n  A simple counting iterator which can be used as an insert iterator.\n  but only counts the number of elements inserted. Useful for\n  use with counting the size of an intersection using std::set_intersection\n*/\ntemplate \nstruct counting_inserter {\n  size_t* i;\n  counting_inserter(size_t* i):i(i) { }\n  counting_inserter& operator++() {\n    ++(*i);\n    return *this;\n  }\n  void operator++(int) {\n    ++(*i);\n  }\n\n  struct empty_val {\n    empty_val operator=(const T&) { return empty_val(); }\n  };\n\n  empty_val operator*() {\n    return empty_val();\n  }\n\n  typedef empty_val reference;\n};\n\n\n/*\n * Computes the size of the intersection of two vid_vector's\n */\nstatic uint32_t count_set_intersect(\n             const vid_vector& smaller_set,\n             const vid_vector& larger_set) {\n\n  if (smaller_set.cset == NULL && larger_set.cset == NULL) {\n    size_t i = 0;\n    counting_inserter iter(&i);\n    std::set_intersection(smaller_set.vid_vec.begin(), smaller_set.vid_vec.end(),\n                          larger_set.vid_vec.begin(), larger_set.vid_vec.end(),\n                          iter);\n    return i;\n  }\n  else if (smaller_set.cset == NULL && larger_set.cset != NULL) {\n    size_t i = 0;\n    foreach(graphlab::vertex_id_type vid, smaller_set.vid_vec) {\n      i += larger_set.cset->count(vid);\n    }\n    return i;\n  }\n  else if (smaller_set.cset != NULL && larger_set.cset == NULL) {\n    size_t i = 0;\n    foreach(graphlab::vertex_id_type vid, larger_set.vid_vec) {\n      i += smaller_set.cset->count(vid);\n    }\n    return i;\n  }\n  else {\n    size_t i = 0;\n    foreach(graphlab::vertex_id_type vid, *(smaller_set.cset)) {\n      i += larger_set.cset->count(vid);\n    }\n    return i;\n\n  }\n}\n\n\n\n\n\n\n/*\n * Each vertex maintains a list of all its neighbors.\n * and a final count for the number of triangles it is involved in\n */\nstruct vertex_data_type {\n  vertex_data_type(): num_triangles(0){ }\n  // A list of all its neighbors\n  vid_vector vid_set;\n  // The number of triangles this vertex is involved it.\n  // only used if \"per vertex counting\" is used\n  uint32_t num_triangles;\n  void save(graphlab::oarchive &oarc) const {\n    oarc << vid_set << num_triangles;\n  }\n  void load(graphlab::iarchive &iarc) {\n    iarc >> vid_set >> num_triangles;\n  }\n};\n\n\n/*\n * Each edge is simply a counter of triangles\n */\ntypedef uint32_t edge_data_type;\n\n// To collect the set of neighbors, we need a message type which is\n// basically a set of vertex IDs\n\nbool PER_VERTEX_COUNT = false;\n\n\n/*\n * This is the gathering type which accumulates an array of\n * all neighboring vertices.\n * It is a simple wrapper around a vector with\n * an operator+= which simply performs a  +=\n */\nstruct set_union_gather {\n  graphlab::vertex_id_type v;\n  std::vector vid_vec;\n\n  set_union_gather():v(-1) {\n  }\n\n  size_t size() const {\n    if (v == (graphlab::vertex_id_type)-1) return vid_vec.size();\n    else return 1;\n  }\n  /*\n   * Combining with another collection of vertices.\n   * Union it into the current set.\n   */\n  set_union_gather& operator+=(const set_union_gather& other) {\n    if (size() == 0) {\n      (*this) = other;\n      return (*this);\n    }\n    else if (other.size() == 0) {\n      return *this;\n    }\n\n    if (vid_vec.size() == 0) {\n      vid_vec.push_back(v);\n      v = (graphlab::vertex_id_type)(-1);\n    }\n    if (other.vid_vec.size() > 0) {\n      size_t ct = vid_vec.size();\n      vid_vec.resize(vid_vec.size() + other.vid_vec.size());\n      for (size_t i = 0; i < other.vid_vec.size(); ++i) {\n        vid_vec[ct + i] = other.vid_vec[i];\n      }\n    }\n    else if (other.v != (graphlab::vertex_id_type)-1) {\n      vid_vec.push_back(other.v);\n    }\n    return *this;\n  }\n  \n  // serialize\n  void save(graphlab::oarchive& oarc) const {\n    oarc << bool(vid_vec.size() == 0);\n    if (vid_vec.size() == 0) oarc << v;\n    else oarc << vid_vec;\n  }\n\n  // deserialize\n  void load(graphlab::iarchive& iarc) {\n    bool novvec;\n    v = (graphlab::vertex_id_type)(-1);\n    vid_vec.clear();\n    iarc >> novvec;\n    if (novvec) iarc >> v;\n    else iarc >> vid_vec;\n  }\n};\n\n/*\n * Define the type of the graph\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n\n/*\n * This class implements the triangle counting algorithm as described in\n * the header. On gather, we accumulate a set of all adjacent vertices.\n * If per_vertex output is not necessary, we can use the optimization\n * where each vertex only accumulates neighbors with greater vertex IDs.\n */\nclass triangle_count :\n      public graphlab::ivertex_program,\n      /* I have no data. Just force it to POD */\n      public graphlab::IS_POD_TYPE  {\npublic:\n  bool do_not_scatter;\n\n  // Gather on all edges\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  } \n\n  /*\n   * For each edge, figure out the ID of the \"other\" vertex\n   * and accumulate a set of the neighborhood vertex IDs.\n   */\n  gather_type gather(icontext_type& context,\n                     const vertex_type& vertex,\n                     edge_type& edge) const {\n    set_union_gather gather;\n    graphlab::vertex_id_type otherid = edge.target().id() == vertex.id() ?\n                                       edge.source().id() : edge.target().id();\n\n    size_t other_nbrs = (edge.target().id() == vertex.id()) ?\n        (edge.source().num_in_edges() + edge.source().num_out_edges()): \n        (edge.target().num_in_edges() + edge.target().num_out_edges());\n\n    size_t my_nbrs = vertex.num_in_edges() + vertex.num_out_edges();\n\n    if (PER_VERTEX_COUNT || (other_nbrs > my_nbrs) || (other_nbrs == my_nbrs && otherid > vertex.id())) {\n    //if (PER_VERTEX_COUNT || otherid > vertex.id()) {\n     gather.v = otherid;\n    } \n    return gather;\n  }\n\n  /*\n   * the gather result now contains the vertex IDs in the neighborhood.\n   * store it on the vertex. \n   */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& neighborhood) {\n   do_not_scatter = false;\n   if (neighborhood.vid_vec.size() == 0) {\n     // neighborhood set may be empty or has only 1 element\n     vertex.data().vid_set.clear();\n     if (neighborhood.v != (graphlab::vertex_id_type(-1))) {\n       vertex.data().vid_set.vid_vec.push_back(neighborhood.v);\n     }\n   }\n   else {\n     vertex.data().vid_set.assign(neighborhood.vid_vec);\n   }\n   do_not_scatter = vertex.data().vid_set.size() == 0;\n  } // end of apply\n\n  /*\n   * Scatter over all edges to compute the intersection.\n   * I only need to touch each edge once, so if I scatter just on the\n   * out edges, that is sufficient.\n   */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const {\n    if (do_not_scatter) return graphlab::NO_EDGES;\n    else return graphlab::OUT_EDGES;\n  }\n\n\n  /*\n   * For each edge, count the intersection of the neighborhood of the\n   * adjacent vertices. This is the number of triangles this edge is involved\n   * in.\n   */\n  void scatter(icontext_type& context,\n              const vertex_type& vertex,\n              edge_type& edge) const {\n\n    //    vertex_type othervtx = edge.target();\n    const vertex_data_type& srclist = edge.source().data();\n    const vertex_data_type& targetlist = edge.target().data();\n    if (targetlist.vid_set.size() < srclist.vid_set.size()) {\n      edge.data() += count_set_intersect(targetlist.vid_set, srclist.vid_set);\n    }\n    else {\n      edge.data() += count_set_intersect(srclist.vid_set, targetlist.vid_set);\n    }\n  }\n};\n\n/*\n * This class is used in a second engine call if per vertex counts are needed.\n * The number of triangles a vertex is involved in can be computed easily\n * by summing over the number of triangles each adjacent edge is involved in\n * and dividing by 2. \n */\nclass get_per_vertex_count :\n      public graphlab::ivertex_program,\n      /* I have no data. Just force it to POD */\n      public graphlab::IS_POD_TYPE  {\npublic:\n  // Gather on all edges\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  }\n  // We gather the number of triangles each edge is involved in\n  size_t gather(icontext_type& context,\n                     const vertex_type& vertex,\n                     edge_type& edge) const {\n    return edge.data();\n  }\n\n  /* the gather result is the total sum of the number of triangles\n   * each adjacent edge is involved in . Dividing by 2 gives the\n   * desired result.\n   */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& num_triangles) {\n    vertex.data().vid_set.clear();\n    vertex.data().num_triangles = num_triangles / 2;\n  }\n\n  // No scatter\n  edge_dir_type scatter_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::NO_EDGES;\n  }\n\n\n};\n\ntypedef graphlab::synchronous_engine engine_type;\n\n/* Used to sum over all the edges in the graph in a\n * map_reduce_edges call\n * to get the total number of triangles\n */\nsize_t get_edge_data(const graph_type::edge_type& e) {\n  return e.data();\n}\n\n/*\n * A saver which saves a file where each line is a vid / # triangles pair\n */\nstruct save_triangle_count{\n  std::string save_vertex(graph_type::vertex_type v) { \n    double nt = v.data().num_triangles;\n    double n_followed = v.num_out_edges();\n    double n_following = v.num_in_edges();\n\n    return graphlab::tostr(v.id()) + \"\\t\" +\n           graphlab::tostr(nt) + \"\\t\" +\n           graphlab::tostr(n_followed) + \"\\t\" + \n           graphlab::tostr(n_following) + \"\\n\";\n  }\n  std::string save_edge(graph_type::edge_type e) {\n    return \"\";\n  }\n};\n\n\nint main(int argc, char** argv) {\n  std::cout << \"This program counts the exact number of triangles in the \"\n            \"provided graph.\\n\\n\";\n\n  graphlab::command_line_options clopts(\"Exact Triangle Counting. \"\n    \"Given a graph, this program computes the total number of triangles \"\n    \"in the graph. An option (per_vertex) is also provided which \"\n    \"computes for each vertex, the number of triangles it is involved in.\"\n    \"The algorithm assumes that each undirected edge appears exactly once \"\n    \"in the graph input. If edges may appear more than once, this procedure \"\n    \"will over count.\");\n  std::string prefix, format;\n  std::string per_vertex;\n  clopts.attach_option(\"graph\", prefix,\n                       \"Graph input. reads all graphs matching prefix*\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph format\");\n clopts.attach_option(\"ht\", HASH_THRESHOLD,\n                       \"Above this size, hash sets are used\");\n  clopts.attach_option(\"per_vertex\", per_vertex,\n                       \"If not empty, will count the number of \"\n                       \"triangles each vertex belongs to and \"\n                       \"save to file with prefix \\\"[per_vertex]\\\". \"\n                       \"The algorithm used is slightly different \"\n                       \"and thus will be a little slower\");\n  if(!clopts.parse(argc, argv)) return EXIT_FAILURE;\n  if (prefix == \"\") {\n    std::cout << \"--graph is not optional\\n\";\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  else if (format == \"\") {\n    std::cout << \"--format is not optional\\n\";\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n\n\n  if (per_vertex != \"\") PER_VERTEX_COUNT = true;\n  // Initialize control plane using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n  graphlab::launch_metric_server();\n  // load graph\n  graph_type graph(dc, clopts);\n  graph.load_format(prefix, format);\n  graph.finalize();\n  dc.cout() << \"Number of vertices: \" << graph.num_vertices() << std::endl\n            << \"Number of edges:    \" << graph.num_edges() << std::endl;\n\n  graphlab::timer ti;\n  \n  // create engine to count the number of triangles\n  dc.cout() << \"Counting Triangles...\" << std::endl;\n  engine_type engine(dc, graph, clopts);\n  engine.signal_all();\n  engine.start();\n\n  dc.cout() << \"Counted in \" << ti.current_time() << \" seconds\" << std::endl;\n\n  if (PER_VERTEX_COUNT == false) {\n    size_t count = graph.map_reduce_edges(get_edge_data);\n    dc.cout() << count << \" Triangles\"  << std::endl;\n  }\n  else {\n    graphlab::synchronous_engine engine(dc, graph, clopts);\n    engine.signal_all();\n    engine.start();\n    graph.save(per_vertex,\n            save_triangle_count(),\n            false, /* no compression */\n            true, /* save vertex */\n            false, /* do not save edge */\n            clopts.get_ncpus()); /* one file per machine */\n\n  }\n  \n  graphlab::stop_metric_server();\n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // End of main\n\n"
  },
  {
    "path": "toolkits/graph_analytics/warp_bond_percolation.cpp",
    "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n/**\n * \\file\n *\n * Written by Danny Bickson, CMU\n * See algorithm description in Wikipedia: http://en.wikipedia.org/wiki/Percolation_theory\n */\n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#define BOND_PERCOLATION_MAP_REDUCE 0\n#define SYNC\nbool debug;\nint max_iter = 100000;\nstd::string output_file;\nint n = 0;\nfloat p = 0;\n\nstruct vertex_data : public graphlab::IS_POD_TYPE{\n  unsigned int comp_id;\n  vertex_data(): comp_id(-1) {}\n};\n\nstd::size_t hash_value(vertex_data const& b) {\n  return b.comp_id;\n}\n\n\n/**\n * \\brief The edge data stores the entry in the matrix.\n *\n * In addition the edge data sgdo stores the most recent error estimate.\n */\nstruct edge_data : public graphlab::IS_POD_TYPE {\n   unsigned int id;\n   unsigned int comp_id;\n   edge_data(unsigned int id) : id(id) { comp_id = id; };\n   edge_data(){ id = comp_id = -1; }\n\n}; // end of edge data\n\nstd::size_t hash_value(edge_data const& b) {\n  return b.comp_id;\n}\n\n\n/**\n * \\brief The graph type is defined in terms of the vertex and edge\n * data.\n */\ntypedef graphlab::distributed_graph graph_type;\ntypedef graphlab::gl3engine engine_type;\n\n\n\n/**\n * \\brief The graph loader function is a line parser used for\n * distributed graph construction.\n */\ninline bool graph_loader(graph_type& graph,\n                         const std::string& filename,\n                         const std::string& line) {\n  ASSERT_FALSE(line.empty());\n  // Parse the line\n  std::stringstream strm(line);\n  graph_type::vertex_id_type source_id(-1), target_id(-1);\n  unsigned int edge_id(-1);\n  strm >> source_id >> target_id >> edge_id;\n  if (edge_id == (uint)-1)\n    logstream(LOG_FATAL)<<\"Input file must contains line with the following format: [from] [ to] [edge_id]\\n \" << std::endl << \" where edge_id is a consecutive integer \" << std::endl;\n\n  // Create an edge and add it to the graph\n  graph.add_edge(source_id, target_id, edge_data(edge_id));\n  return true; // successful load\n} // end of graph_loader\n\nsize_t count_component(const graph_type::edge_type & edge) {\n  int diff = (edge.source().data().comp_id != edge.target().data().comp_id);\n  if (debug && diff)\n    std::cout<<\"Adding diff between node: \" << edge.source().id() << \" to: \" << edge.target().id()<< \" compA: \" << edge.source().data().comp_id << \" compB: \" <<\n      edge.target().data().comp_id << std::endl;\n  return diff;\n}\n\nunsigned int bond_percolation_map(const graph_type::vertex_type& center,\n                         graph_type::edge_type& edge,\n                         const graph_type::vertex_type& other) {\n   if (debug)\n     std::cout<<\"Comparing: \" << center.data().comp_id << \" : \" << edge.data().id << \" : \" << other.data().comp_id << std::endl;\n   edge.data().comp_id =  std::min(std::min(center.data().comp_id, edge.data().id), other.data().comp_id);\n   if (debug)\n     std::cout<<\"Setting edge id to: \" << edge.data().comp_id << \" from: \" << center.id() << \" to: \" << other.id() << std::endl;\n   return edge.data().comp_id;\n}\n\n\n//find min component of two edges\nvoid bond_percolation_combine(unsigned int& v1, const unsigned int& v2) {\n    v1 = std::min(v1, v2);\n    if (debug)\n      std::cout<<\"Comparing two edge ids: \" << v1 << \" : \" << v2 << std::endl;\n}\n\n//the main update function\n\nvoid bond_percolation_function(engine_type::context_type& context,\n                  graph_type::vertex_type& vertex\n#ifndef SYNC\n                  , const engine_type::message_type& unused) {\n#else\n   ){\n#endif\n\n     uint comp_id = vertex.data().comp_id;\n     vertex.data().comp_id =  context.map_reduce(BOND_PERCOLATION_MAP_REDUCE, graphlab::ALL_EDGES);\n     if (debug && comp_id != vertex.data().comp_id)\n       std::cout<<\"node: \" << vertex.id() << \" min edge component found: \" << vertex.data().comp_id << std::endl;\n#ifndef SYNC\n     if (comp_id != vertex.data().comp_id)\n       context.broadcast_signal(graphlab::ALL_EDGES);\n#endif\n}\n\n\nstruct model_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n\n  std::string save_vertex(const vertex_type& vertex) const {\n    return \"\";\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return boost::lexical_cast(edge.data().id) + \" \" + boost::lexical_cast(edge.data().comp_id) + std::string(\"\\n\");\n  }\n};\n\n//reduce sizes of connected components\nstruct label_counter {\n  std::map counts;\n\n  label_counter() { }\n\n  explicit label_counter(size_t labelId) {\n    counts[labelId] = 1;\n  }\n\n  label_counter& operator+=(const label_counter& other) {\n    std::map::const_iterator iter = other.counts.begin();\n    while(iter != other.counts.end()) {\n      counts[iter->first] += iter->second;\n      ++iter;\n    }\n    return *this;\n  }\n\n  void save(graphlab::oarchive& oarc) const {\n    oarc << counts;\n  }\n  void load(graphlab::iarchive& iarc) {\n    iarc >> counts;\n  }\n};\n\n\nlabel_counter get_comp_ide(const graph_type::edge_type& edge) {\n  return label_counter(edge.data().comp_id);\n}\nlabel_counter get_comp_idv(const graph_type::vertex_type& vertex) {\n  return label_counter(vertex.data().comp_id);\n}\n\n\n\n\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n\n  // Parse command line options -----------------------------------------------\n  const std::string description =\n      \"Compute connected component - by edges\";\n  graphlab::command_line_options clopts(description);\n  std::string input_dir;\n  clopts.attach_option(\"graph\", input_dir,\n                       \"The directory containing the graph file\");\n  clopts.add_positional(\"graph\");\n  clopts.attach_option(\"output_file\", output_file,\n                       \"The prefix (folder and filename) to save output_file.\");\n  clopts.attach_option(\"max_iter\", max_iter, \"max number of iterations\");\n  clopts.attach_option(\"debug\", debug, \"debug (verbose) mode\");\n  clopts.attach_option(\"p\", p, \"percentage for active node (optional)\");\n  clopts.attach_option(\"n\", n, \"total number of nodes (optional)\");\n\n  if(!clopts.parse(argc, argv) || input_dir == \"\") {\n    std::cout << \"Error in parsing command line arguments.\" << std::endl;\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n  //dc.cout() << \"Loading graph.\" << std::endl;\n  graphlab::timer timer;\n  graph_type graph(dc, clopts);\n  graph.load(input_dir, graph_loader);\n  dc.cout() << \"Loading graph. Finished in \"\n            << timer.current_time() << std::endl;\n\n  //dc.cout() << \"Finalizing graph.\" << std::endl;\n  timer.start();\n  graph.finalize();\n  //dc.cout() << \"Finalizing graph. Finished in \"\n  //          << timer.current_time() << std::endl;\n\n\n  /*dc.cout()\n      << \"========== Graph statistics on proc \" << dc.procid()\n      << \" ===============\"\n      << \"\\n Num vertices: \" << graph.num_vertices()\n      << \"\\n Num edges: \" << graph.num_edges()\n      << \"\\n Num replica: \" << graph.num_replicas()\n      << \"\\n Replica to vertex ratio: \"\n      << float(graph.num_replicas())/graph.num_vertices()\n      << \"\\n --------------------------------------------\"\n      << \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n      << \"\\n Num local vertices: \" << graph.num_local_vertices()\n      << \"\\n Replica to own ratio: \"\n      << (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n      << \"\\n Num local edges: \" << graph.num_local_edges()\n      //<< \"\\n Begin edge id: \" << graph.global_eid(0)\n        << \"\\n Edge balance ratio: \"\n        << float(graph.num_local_edges())/graph.num_edges()\n        << std::endl;\n  */\n\n  if (debug)\n    //omp_set_num_threads(1);\n\n  /* THE MAIN LOOP */\n  dc.cout() << \"Creating engine\" << std::endl;\n  engine_type engine(dc, graph, clopts);\n#ifndef SYNC\n  engine.set_vertex_program(bond_percolation_function);\n#endif\n  engine.register_map_reduce(BOND_PERCOLATION_MAP_REDUCE, bond_percolation_map, bond_percolation_combine);\n\n  engine.signal_all();\n#ifndef SYNC\n  engine.wait();\n#endif\n\n#ifdef SYNC\n  /* FOR EACH ITERATION */\n  for (int i=0; i< max_iter; i++){\n     /* PERFORM UPDATE FUNCTION */\n     engine.parfor_all_local_vertices(bond_percolation_function);\n     /* WAIT UNTIL COMPLETION */\n     engine.wait();\n     /* CHECK FOR CONVERGENCE */\n     size_t diff = graph.map_reduce_edges(count_component);\n     dc.cout() << \"iter = \" << i << \" diff= \" << diff << std::endl;\n     if (diff == 0)\n       break;\n  }\n#endif\n\n  const double runtime = timer.current_time();\n  dc.cout() << \"----------------------------------------------------------\"\n            << std::endl\n            << \"Final Runtime (seconds):   \" << runtime;\n\n  dc.cout() << \" Num updates: \" << engine.num_updates() << std::endl;\n  // Make output_file ---------------------------------------------------------\n  if(!output_file.empty()) {\n    std::cout << \"Saving output_file\" << std::endl;\n    const bool gzip_output = false;\n    const bool save_vertices = false;\n    const bool save_edges = true;\n    const size_t threads_per_machine = 1;\n\n    //save the output\n    graph.save(output_file, model_saver(), gzip_output, save_vertices, save_edges, threads_per_machine);\n  }\n\n  //take statistics\n  label_counter state = graph.map_reduce_edges(get_comp_ide);\n  label_counter statv = graph.map_reduce_vertices(get_comp_idv);\n  if (p > 0)\n    dc.cout() << \"site fraction p= \" << p << std::endl;\n  if (n > 0){\n    dc.cout() << \"n=\" << n*p << std::endl;\n    dc.cout() << \"isolated sites: \" << p*n-graph.num_vertices() << std::endl;\n  }\n  dc.cout() << \"Number of sites: \" << graph.num_vertices() << std::endl;\n  dc.cout() << \"Number of bonds: \" << graph.num_edges() << std::endl;\n  if (n){\n  dc.cout() << \"Percentage of sites: \" << (double)graph.num_vertices() / n << std::endl;\n  dc.cout() << \"Percentage of bonds: \" << (double)graph.num_edges() / (2.0*n) << std::endl;\n  }\n  dc.cout() << \"SITES RESULT:\\nsize\\tcount\\n\";\n  std::map final_countsv;\n  uint total_sites = 0;\n  for (std::map::const_iterator iter = statv.counts.begin();\n      iter != statv.counts.end(); iter++) {\n      //dc.cout() << iter->first << \"\\t\" << iter->second << \"\\n\";\n      final_countsv[iter->second] += 1;\n      total_sites += iter->second;\n  }\n  for (std::map::const_iterator iter = final_countsv.begin();\n      iter != final_countsv.end(); iter++) {\n      dc.cout() << iter->first << \"\\t\" << iter->second << \"\\n\";\n  }\n\n  dc.cout() << \"BONDS RESULT:\\nsize\\tcount\\n\";\n  std::map final_countse;\n  uint total_bonds = 0;\n  for (std::map::const_iterator iter = state.counts.begin();\n      iter != state.counts.end(); iter++) {\n      //dc.cout() << iter->first << \"\\t\" << iter->second << \"\\n\";\n      final_countse[iter->second] += 1;\n      total_bonds += iter->second;\n  }\n  for (std::map::const_iterator iter = final_countse.begin();\n      iter != final_countse.end(); iter++) {\n      dc.cout() << iter->first << \"\\t\" << iter->second << \"\\n\";\n  }\n  assert(total_sites == graph.num_vertices());\n  assert(total_bonds == graph.num_edges());\n\n  //shutdown MPI\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // end of main\n\n\n\n"
  },
  {
    "path": "toolkits/graph_analytics/warp_coloring.cpp",
    "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \n#include \n#include \n\n#include \n#include \n#include \nusing namespace graphlab;\n\n#define UNIQUE_COLOR_MAP_REDUCE 0\n#define SIGNAL_IF_CHANGE 1\n\n// The vertex data is the color of the vertex\ntypedef graphlab::vertex_id_type color_type;\n\n\n\nstruct edge_data_type : public graphlab::IS_POD_TYPE {\n  bool dirty;\n  bool locked;\n  char owned_by; // 0 is source, 1 is other\n  bool requested; // 1 is requested by other party;\n};\n\nstd::ostream& operator<<(std::ostream& o, const edge_data_type& e) {\n  o << e.dirty << \" \" << e.locked << \" \" << (int)e.owned_by << \" \" << e.requested;\n  return o;\n}\n\n// The graph type is determined by the vertex and edge data types\ntypedef distributed_graph graph_type;\n\ntypedef gl3engine engine_type;\n\nengine_type* eng;\n\nbool EDGE_CONSISTENT = false;\n\n\n\n/*\n * This is the gathering type which accumulates an (unordered) set of\n * all neighboring colors\n * It is a simple wrapper around a boost::unordered_set with\n * an operator+= which simply performs a set union.\n *\n * This struct can be significantly accelerated for small sets.\n * Small collections of vertex IDs should not require the overhead\n * of the unordered_set.\n */\nstruct set_union_gather {\n  boost::unordered_set colors;\n\n  /*\n   * Combining with another collection of vertices.\n   * Union it into the current set.\n   */\n  set_union_gather& operator+=(const set_union_gather& other) {\n    foreach(graphlab::vertex_id_type othervid, other.colors) {\n      colors.insert(othervid);\n    }\n    return *this;\n  }\n\n  // serialize\n  void save(graphlab::oarchive& oarc) const {\n    oarc << colors;\n  }\n\n  // deserialize\n  void load(graphlab::iarchive& iarc) {\n    iarc >> colors;\n  }\n};\n\n/**************************************************************************/\n/*                                                                        */\n/*                           Coloring Functions                           */\n/*                                                                        */\n/**************************************************************************/\n\nset_union_gather unique_color_map(const graph_type::vertex_type& center,\n                                  graph_type::edge_type& e,\n                                  const graph_type::vertex_type& other) {\n  set_union_gather gather;\n  color_type other_color = other.data();\n  gather.colors.insert(other_color);\n  return gather;\n}\n\nvoid unique_color_combine(set_union_gather& v1, const set_union_gather& v2) {\n  v1 += v2;\n}\n\n\nvoid schedule_neighbors_if_change(const graph_type::vertex_type& center,\n                                  graph_type::edge_type& e,\n                                  const graph_type::vertex_type& other) {\n  if (center.data() == other.data()) {\n    eng->get_context().signal(other);\n  }\n}\n\n/**************************************************************************/\n/*                                                                        */\n/*                         Chandy Misra Functions                         */\n/*                                                                        */\n/**************************************************************************/\nvoid initialize_chandy_misra(graph_type::edge_type& edge) {\n  edge.data().dirty = true;\n  edge.data().locked = false;\n  edge.data().owned_by = edge.source().id() < edge.target().id() ? 0 : 1;\n  edge.data().requested = false;\n//   std::cout << edge.source().id() << \"->\" << edge.target().id() << \" \" << edge.data() << \"\\n\";\n}\n\n\n#define LOCK_IF_OWNED 3\nsize_t lock_if_owned_map(const graph_type::vertex_type& center,\n                         graph_type::edge_type& e,\n                         const graph_type::vertex_type& other) {\n  char m =  (e.source().id() == center.id()) ? 0 : 1;\n  if (e.data().owned_by == m) e.data().locked = true;\n  else if (e.data().dirty && e.data().locked == false) {\n    e.data().owned_by = m;\n    e.data().dirty = false;\n    e.data().requested = false;\n    e.data().locked = true;\n  } else {\n    e.data().requested = true;\n  }\n//   std::cout << \"Lock If Owned: \" << center.id() << \": Fork = \"<< e.source().id() << \"->\" << e.target().id() << \" \"  << e.data() << \"\\n\";\n  return (e.data().owned_by == m && e.data().locked);\n}\n\nvoid lock_if_owned_combine(size_t& v1, const size_t& v2) {\n  v1 += v2;\n}\n\n\n#define STOP_EATING 4\nvoid stop_eating(const graph_type::vertex_type& center,\n                  graph_type::edge_type& e,\n                  const graph_type::vertex_type& other) {\n  e.data().locked = false;\n  e.data().dirty = true;\n  if (e.data().requested) {\n    // switch owner\n    e.data().owned_by = !e.data().owned_by;\n    e.data().dirty = false;\n    e.data().requested = false;\n  }\n\n//   std::cout << \"Stop Eating: \" << center.id() << \": Fork = \"<< e.source().id() << \"->\" << e.target().id() << \" \"  << e.data() << \"\\n\";\n}\n\n\n#define UNLOCK_FORKS_MAINTAIN_REQUEST 5\nvoid unlock_forks_maintain_request(const graph_type::vertex_type& center,\n                                   graph_type::edge_type& e,\n                                   const graph_type::vertex_type& other) {\n  char m =  (e.source().id() == center.id()) ? 0 : 1;\n  if (e.data().owned_by == m && e.data().locked) {\n    e.data().locked = false;\n  }\n\n  if (e.data().owned_by == m && e.data().dirty && e.data().requested) {\n    e.data().owned_by = !e.data().owned_by;\n    e.data().dirty = false;\n    e.data().requested = true;\n  }\n\n//   std::cout << \"Release: \" << center.id() << \": Fork = \" << e.source().id() << \"->\" << e.target().id() << \" \" << e.data() << \"\\n\";\n}\n\n\n\n\n\nvoid update_function(engine_type::context_type& context,\n                     graph_type::vertex_type& vertex,\n                     const engine_type::message_type& unused) {\n\n  // acquire locks\n  if (EDGE_CONSISTENT) {\n    size_t expected_num_locks = vertex.num_in_edges() + vertex.num_out_edges();\n    while(1) {\n      size_t numnbr = context.map_reduce(LOCK_IF_OWNED, ALL_EDGES);\n      if (numnbr == expected_num_locks) {\n        break;\n      }\n      else {\n        context.edge_transform(UNLOCK_FORKS_MAINTAIN_REQUEST, ALL_EDGES);\n      }\n      graphlab::fiber_group::yield();\n    }\n  }\n  set_union_gather neighborhood =\n      context.map_reduce(UNIQUE_COLOR_MAP_REDUCE,\n                                           ALL_EDGES);\n\n  bool color_changed = false;\n\n  size_t neighborhoodsize = neighborhood.colors.size();\n  for (color_type curcolor = 0; curcolor < neighborhoodsize + 1; ++curcolor) {\n    if (neighborhood.colors.count(curcolor) == 0) {\n      vertex.data() = curcolor;\n      break;\n    }\n  }\n  if (EDGE_CONSISTENT) context.edge_transform(STOP_EATING, ALL_EDGES);\n\n  context.edge_transform(SIGNAL_IF_CHANGE, ALL_EDGES, false);\n}\n\n\n/**************************************************************************/\n/*                                                                        */\n/*                         Validation   Functions                         */\n/*                                                                        */\n/**************************************************************************/\nsize_t validate_conflict(graph_type::edge_type& edge) {\n  return edge.source().data() == edge.target().data();\n}\n\n\n\nint main(int argc, char** argv) {\n  // Initialize control plain using mpi\n  mpi_tools::init(argc, argv);\n  distributed_control dc;\n\n  // Parse command line options -----------------------------------------------\n  command_line_options clopts(\"PageRank algorithm.\");\n  std::string graph_dir;\n  std::string format = \"adj\";\n  clopts.set_scheduler_type(\"fifo\");\n  clopts.attach_option(\"graph\", graph_dir,\n                       \"The graph file.  If none is provided \"\n                       \"then a toy graph will be created\");\n  clopts.add_positional(\"graph\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph file format\");\n  clopts.attach_option(\"edgescope\", EDGE_CONSISTENT,\n                       \"Set to 1 if edge consistency is to be used\");\n  size_t powerlaw = 0;\n  clopts.attach_option(\"powerlaw\", powerlaw,\n                       \"Generate a synthetic powerlaw out-degree graph. \");\n  std::string saveprefix;\n  clopts.attach_option(\"saveprefix\", saveprefix,\n                       \"If set, will save the resultant pagerank to a \"\n                       \"sequence of files with prefix saveprefix\");\n\n  if(!clopts.parse(argc, argv)) {\n    dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  graphlab::launch_metric_server();\n  // Build the graph ----------------------------------------------------------\n  graph_type graph(dc, clopts);\n  if(powerlaw > 0) { // make a synthetic graph\n    dc.cout() << \"Loading synthetic Powerlaw graph.\" << std::endl;\n    graph.load_synthetic_powerlaw(powerlaw, false, 2.1, 100000000);\n  }\n  else if (graph_dir.length() > 0) { // Load the graph from a file\n    dc.cout() << \"Loading graph in format: \"<< format << std::endl;\n    graph.load_format(graph_dir, format);\n  }\n  else {\n    dc.cout() << \"graph or powerlaw option must be specified\" << std::endl;\n    clopts.print_description();\n    return 0;\n  }\n  // must call finalize before querying the graph\n  graph.finalize();\n\n  graph.transform_edges(initialize_chandy_misra);\n\n  dc.cout() << \"#vertices: \" << graph.num_vertices()\n            << \" #edges:\" << graph.num_edges() << std::endl;\n\n  // Running The Engine -------------------------------------------------------\n  engine_type engine(dc, graph, clopts);\n  eng = &engine;\n\n  engine.register_map_reduce(UNIQUE_COLOR_MAP_REDUCE,\n                             unique_color_map,\n                             unique_color_combine);\n\n  engine.register_edge_transform(SIGNAL_IF_CHANGE,\n                                 schedule_neighbors_if_change);\n\n\n  engine.register_map_reduce(LOCK_IF_OWNED,\n                             lock_if_owned_map,\n                             lock_if_owned_combine);\n\n  engine.register_edge_transform(STOP_EATING,\n                                 stop_eating);\n\n  engine.register_edge_transform(UNLOCK_FORKS_MAINTAIN_REQUEST,\n                                 unlock_forks_maintain_request);\n\n  engine.set_vertex_program(update_function);\n  timer ti; ti.start();\n  engine.signal_all();\n\n  engine.wait();\n  dc.cout() << \"Finished Running engine in \" << ti.current_time()\n            << \" seconds.\" << std::endl;\n  dc.cout() << engine.num_updates()\n            << \" updates.\" << std::endl;\n\n  size_t conflict_count = graph.map_reduce_edges(validate_conflict);\n  dc.cout() << \"Num conflicts = \" << conflict_count << \"\\n\";\n\n\n  graphlab::stop_metric_server();\n\n  mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // End of main\n\n\n// We render this entire program in the documentation\n\n\n"
  },
  {
    "path": "toolkits/graph_analytics/warp_pagerank.cpp",
    "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \n#include \n#include \n\n#include \n#include \n// #include \nusing namespace graphlab;\n\n#define PAGERANK_MAP_REDUCE 0\n\n// Global random reset probability\ndouble RESET_PROB = 0.15;\n\ndouble TOLERANCE = 1E-2;\n\n// The vertex data is just the pagerank value (a double)\ntypedef double vertex_data_type;\n\n// There is no edge data in the pagerank application\ntypedef empty edge_data_type;\n\n// The graph type is determined by the vertex and edge data types\ntypedef distributed_graph graph_type;\n\ntypedef gl3engine engine_type;\n/*\n * A simple function used by graph.transform_vertices(init_vertex);\n * to initialize the vertes data.\n */\nvoid init_vertex(graph_type::vertex_type& vertex) { vertex.data() = 1; }\n\n\n\n/*\n * We want to save the final graph so we define a write which will be\n * used in graph.save(\"path/prefix\", pagerank_writer()) to save the graph.\n */\nstruct pagerank_writer {\n  std::string save_vertex(graph_type::vertex_type v) {\n    std::stringstream strm;\n    strm << v.id() << \"\\t\" << v.data() << \"\\n\";\n    return strm.str();\n  }\n  std::string save_edge(graph_type::edge_type e) { return \"\"; }\n}; // end of pagerank writer\n\n\ndouble pagerank_map(const graph_type::vertex_type& v) {\n  return v.data() / v.num_out_edges();\n}\n\nvoid pagerank_combine(double& v1, const double& v2) {\n  v1 += v2;\n}\n\nvoid update_function(engine_type::context_type& context,\n                     graph_type::vertex_type& vertex,\n                     const engine_type::message_type& unused) {\n  double prev = vertex.data();\n  vertex.data() = 0.15 + 0.85 *\n      context.map_reduce(PAGERANK_MAP_REDUCE, IN_EDGES);\n\n  double last_change = std::fabs((vertex.data()- prev));// / vertex.num_out_edges());\n  if (last_change > TOLERANCE) {\n    context.broadcast_signal(OUT_EDGES);\n  }\n}\n\ndouble pagerank_sum(graph_type::vertex_type v) {\n  return v.data();\n}\n\n\nint main(int argc, char** argv) {\n  // Initialize control plain using mpi\n  mpi_tools::init(argc, argv);\n  distributed_control dc;\n\n  // Parse command line options -----------------------------------------------\n  command_line_options clopts(\"PageRank algorithm.\");\n  std::string graph_dir;\n  std::string format = \"adj\";\n  clopts.set_scheduler_type(\"fifo\");\n  clopts.attach_option(\"graph\", graph_dir,\n                       \"The graph file.  If none is provided \"\n                       \"then a toy graph will be created\");\n  clopts.add_positional(\"graph\");\n  clopts.attach_option(\"tol\", TOLERANCE,\n                       \"The permissible change at convergence.\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph file format\");\n  size_t powerlaw = 0;\n  clopts.attach_option(\"powerlaw\", powerlaw,\n                       \"Generate a synthetic powerlaw out-degree graph. \");\n  std::string saveprefix;\n  clopts.attach_option(\"saveprefix\", saveprefix,\n                       \"If set, will save the resultant pagerank to a \"\n                       \"sequence of files with prefix saveprefix\");\n\n  if(!clopts.parse(argc, argv)) {\n    dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  // Build the graph ----------------------------------------------------------\n  graphlab::launch_metric_server();\n  graph_type graph(dc, clopts);\n  if(powerlaw > 0) { // make a synthetic graph\n    dc.cout() << \"Loading synthetic Powerlaw graph.\" << std::endl;\n    graph.load_synthetic_powerlaw(powerlaw, false, 2.1, 100000000);\n  }\n  else if (graph_dir.length() > 0) { // Load the graph from a file\n    dc.cout() << \"Loading graph in format: \"<< format << std::endl;\n    graph.load_format(graph_dir, format);\n  }\n  else {\n    dc.cout() << \"graph or powerlaw option must be specified\" << std::endl;\n    clopts.print_description();\n    return 0;\n  }\n  // must call finalize before querying the graph\n  graph.finalize();\n  dc.cout() << \"#vertices: \" << graph.num_vertices()\n            << \" #edges:\" << graph.num_edges() << std::endl;\n\n  // Initialize the vertex data\n  graph.transform_vertices(init_vertex);\n\n  // Running The Engine -------------------------------------------------------\n  engine_type engine(dc, graph, clopts);\n  engine.register_map_reduce(PAGERANK_MAP_REDUCE,\n                             pagerank_map,\n                             pagerank_combine);\n\n  engine.set_vertex_program(update_function);\n  timer ti; ti.start();\n  engine.signal_all();\n\n  engine.wait();\n  dc.cout() << \"Finished Running engine in \" << ti.current_time()\n            << \" seconds.\" << std::endl;\n  dc.cout() << engine.num_updates()\n            << \" updates.\" << std::endl;\n\n\n  // Save the final graph -----------------------------------------------------\n  if (saveprefix != \"\") {\n    graph.save(saveprefix, pagerank_writer(),\n               false,    // do not gzip\n               true,     // save vertices\n               false);   // do not save edges\n  }\n\n  graphlab::stop_metric_server();\n  mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // End of main\n\n\n// We render this entire program in the documentation\n\n\n"
  },
  {
    "path": "toolkits/graph_analytics/warp_pagerank2.cpp",
    "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n#include \n#include \n#include \n\n#include \n#include \n// #include \nusing namespace graphlab;\n\n#define PAGERANK_MAP_REDUCE 0\n\n// Global random reset probability\nfloat RESET_PROB = 0.15;\n\nfloat TOLERANCE = 1E-2;\n\n// The vertex data is just the pagerank value (a float)\ntypedef float vertex_data_type;\n\n// There is no edge data in the pagerank application\ntypedef empty edge_data_type;\n\n// The graph type is determined by the vertex and edge data types\ntypedef distributed_graph graph_type;\n\ntypedef gl3engine engine_type;\n/*\n * A simple function used by graph.transform_vertices(init_vertex);\n * to initialize the vertes data.\n */\nvoid init_vertex(graph_type::vertex_type& vertex) { vertex.data() = 1; }\n\n\n\n/*\n * We want to save the final graph so we define a write which will be\n * used in graph.save(\"path/prefix\", pagerank_writer()) to save the graph.\n */\nstruct pagerank_writer {\n  std::string save_vertex(graph_type::vertex_type v) {\n    std::stringstream strm;\n    strm << v.id() << \"\\t\" << v.data() << \"\\n\";\n    return strm.str();\n  }\n  std::string save_edge(graph_type::edge_type e) { return \"\"; }\n}; // end of pagerank writer\n\n\nfloat pagerank_map(const graph_type::vertex_type& v) {\n  return v.data() / v.num_out_edges();\n}\n\nvoid pagerank_combine(float& v1, const float& v2) {\n  v1 += v2;\n}\n\nvoid update_function(engine_type::context_type& context,\n                     graph_type::vertex_type& vertex) {\n  vertex.data() = 0.15 + 0.85 *\n      context.map_reduce(PAGERANK_MAP_REDUCE, IN_EDGES);\n}\n\nfloat pagerank_sum(graph_type::vertex_type v) {\n  return v.data();\n}\n\n\nint main(int argc, char** argv) {\n  // Initialize control plain using mpi\n  mpi_tools::init(argc, argv);\n  distributed_control dc;\n  global_logger().set_log_level(LOG_INFO);\n\n  // Parse command line options -----------------------------------------------\n  command_line_options clopts(\"PageRank algorithm.\");\n  std::string graph_dir;\n  std::string format = \"adj\";\n  clopts.set_scheduler_type(\"fifo\");\n  clopts.attach_option(\"graph\", graph_dir,\n                       \"The graph file.  If none is provided \"\n                       \"then a toy graph will be created\");\n  clopts.add_positional(\"graph\");\n  clopts.attach_option(\"tol\", TOLERANCE,\n                       \"The permissible change at convergence.\");\n  clopts.attach_option(\"format\", format,\n                       \"The graph file format\");\n  size_t powerlaw = 0;\n  clopts.attach_option(\"powerlaw\", powerlaw,\n                       \"Generate a synthetic powerlaw out-degree graph. \");\n  size_t iterations = 10;\n  clopts.attach_option(\"iterations\", iterations,\n                       \"Number of asynchronous iterations to run\");\n  std::string saveprefix;\n  clopts.attach_option(\"saveprefix\", saveprefix,\n                       \"If set, will save the resultant pagerank to a \"\n                       \"sequence of files with prefix saveprefix\");\n\n  if(!clopts.parse(argc, argv)) {\n    dc.cout() << \"Error in parsing command line arguments.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  // Build the graph ----------------------------------------------------------\n  graph_type graph(dc, clopts);\n  if(powerlaw > 0) { // make a synthetic graph\n    dc.cout() << \"Loading synthetic Powerlaw graph.\" << std::endl;\n    graph.load_synthetic_powerlaw(powerlaw, false, 2.1, 100000000);\n  }\n  else if (graph_dir.length() > 0) { // Load the graph from a file\n    dc.cout() << \"Loading graph in format: \"<< format << std::endl;\n    graph.load_format(graph_dir, format);\n  }\n  else {\n    dc.cout() << \"graph or powerlaw option must be specified\" << std::endl;\n    clopts.print_description();\n    return 0;\n  }\n  // must call finalize before querying the graph\n  graph.finalize();\n  dc.cout() << \"#vertices: \" << graph.num_vertices()\n            << \" #edges:\" << graph.num_edges() << std::endl;\n\n  // Initialize the vertex data\n  graph.transform_vertices(init_vertex);\n\n  // Running The Engine -------------------------------------------------------\n  engine_type engine(dc, graph, clopts);\n  engine.register_map_reduce(PAGERANK_MAP_REDUCE,\n                             pagerank_map,\n                             pagerank_combine);\n\n  timer ti;\n  for (size_t i = 0;i < iterations; ++i) {\n    engine.parfor_all_local_vertices(update_function);\n    std::cout << \"Iteration \" << i << \" complete\\n\";\n    engine.wait();\n  }\n\n  dc.cout() << \"Finished Running engine in \" << ti.current_time()\n            << \" seconds.\" << std::endl;\n  dc.cout() << engine.num_updates()\n            << \" updates.\" << std::endl;\n\n\n  // Save the final graph -----------------------------------------------------\n  if (saveprefix != \"\") {\n    graph.save(saveprefix, pagerank_writer(),\n               false,    // do not gzip\n               true,     // save vertices\n               false);   // do not save edges\n  }\n\n  mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // End of main\n\n\n// We render this entire program in the documentation\n\n\n"
  },
  {
    "path": "toolkits/graphical_models/CMakeLists.txt",
    "content": "project(GraphicalModels)\n\n\nadd_graphlab_executable(dd dd_main.cpp)\nrequires_eigen(dd)\n\nadd_graphlab_executable(lbp_structured_prediction lbp_structured_prediction.cpp)\nrequires_eigen(lbp_structured_prediction)\n\nadd_graphlab_executable(profile_lbp_synthetic profile_lbp_synthetic.cpp)\nrequires_eigen(profile_lbp_synthetic)\n\n\nadd_graphlab_executable(profile_lbp_synthetic2 profile_lbp_synthetic2.cpp)\nrequires_eigen(profile_lbp_synthetic2)\n\n\nadd_graphlab_executable(synthetic_image_data synthetic_image_data.cpp)\nrequires_opencv(synthetic_image_data)\n\n\n\nadd_graphlab_executable(mplp_denoise mplp_denoise.cpp)\nrequires_eigen(mplp_denoise)\nrequires_opencv(mplp_denoise)\n\n\nsubdirs(factors)\n"
  },
  {
    "path": "toolkits/graphical_models/ad3_qp.hpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n *\n * \\brief This application performs MAP inference on Markov Nets \n * provided in standard UAI file format via Dual-Decomposition. \n *\n *\n *  \\authors Dhruv Batra, André Martins, Aroma Mahendru\n */\n\n\n\n#ifndef _AD3_QP_HPP_\n#define _AD3_QP_HPP_\n\n\n#include \n#include \n#include \n#include \"dd_grlab.hpp\"\n\n\n#define NEARLY_ZERO_TOL(a,tol) (((a)<=(tol)) && ((a)>=(-(tol))))\n#define NEARLY_EQ_TOL(a,b,tol) (((a)-(b))*((a)-(b))<=(tol))\n#define num_max_iterations_QP_ 10\n#define EXP 10\n\n\n\n////////////////////////////////////////////////////////////////////////////////\n// This class implements the Alternating Directions Dual Decompostion as \n// described in:\n//\n// André F. T. Martins, Mário A. T. Figueiredo, Pedro M. Q. Aguiar,\n// Noah A. Smith, and Eric P. Xing.\n// \"Alternating Directions Dual Decomposition\"\n// Arxiv preprint arXiv:1212.6550, 2012.\n/////////////////////////////////////////////////////////////////////////////////\n\nstruct ad3_vertex_program:public admm_vertex_program {\n\n/**\n * \\brief Maximize returns the maximum value and configuration with reference to \n * input additional and variable log potentials. addtional log potential corresponds \n * factor potentials and variable potential corresponds to sum of lagrange \n * multipliers and unary potentials divided by degree of the unary vertex.\n */\n\n\nvoid Maximize(vertex_type& vertex, vec additional_log_potentials, vec variable_log_potentials,\n                Configuration &configuration,\n                double *value) {\n          \n    vector  states(vertex.data().nvars,-1);\n    *value = -1e12;\n    for (int index = 0;\n         index < additional_log_potentials.size();\n         ++index) {\n      double score = additional_log_potentials[index];\n      get_configuration_states(vertex,index, &states);\n      int offset = 0;\n      for (int i = 0; i < vertex.data().nvars; ++i) {\n        score += variable_log_potentials[offset+states[i]];\n        offset = vertex.data().cards[i];\n        \n      }\n      \n      if (configuration < 0 || score > *value) {\n        configuration = index;\n        *value = score;\n      }\n    }\n    assert(configuration >= 0);\n    \n  }\n\n \n   // Delete configuration.\n void DeleteConfiguration(Configuration *configuration) {\n    Configuration *conf = configuration;\n    delete conf;\n  }\n  \nvoid DeleteConfiguration(vector  configuration) {\n    for(int i=0; i< configuration.size(); i++){\n    Configuration *conf = configuration[i];\n    delete conf;}\n  }\n\n/**\n * \\brief InvertAfterInsertion function is used to invert the matrix. Used in solveQP\n */\n\n bool InvertAfterInsertion(vertex_type& vertex, vector  & inverse_A_,\n        const vector &active_set, const Configuration &inserted_element) {\n\n  vector inverse_A = inverse_A_;\n  int size_A = active_set.size() + 1;\n  vector r(size_A);\n\n  r[0] = 1.0;\n  for (int i = 0; i < active_set.size(); ++i) {\n    // Count how many variable values the new assignment\n    // have in common with the i-th assignment.\n    int num_common_values = CountCommonValues(vertex, active_set[i], inserted_element);\n    r[i+1] = static_cast(num_common_values);\n  }\n\n  double r0 = static_cast(CountCommonValues(vertex,\n      inserted_element, inserted_element));\n  double s = r0;\n  for (int i = 0; i < size_A; ++i) {\n    if (r[i] == 0.0) continue;\n    s -= r[i] * r[i] * inverse_A[i * size_A + i];\n    for (int j = i+1; j < size_A; ++j) {\n      if (r[j] == 0.0) continue;\n      s -= 2 * r[i] * r[j] * inverse_A[i * size_A + j];\n    }\n  }\n\n    if (NEARLY_ZERO_TOL(s, 1e-9)) {\n         if (opts.verbose> 2) {\n      cout << \"Warning: updated matrix will become singular after insertion.\"\n           << endl;\n    }\n    return false;\n  }\n\n  double invs = 1.0 / s;\n  vector d(size_A, 0.0);\n  for (int i = 0; i < size_A; ++i) {\n    if (r[i] == 0.0) continue;\n    for (int j = 0; j < size_A; ++j) {\n      d[j] += inverse_A[i * size_A + j] * r[i];\n    }\n  }\n\n  int size_A_after = size_A + 1;\n  inverse_A_.resize(size_A_after * size_A_after);\n  for (int i = 0; i < size_A; ++i) {\n    for (int j = 0; j < size_A; ++j) {\n      inverse_A_[i * size_A_after + j] = inverse_A[i * size_A + j] +\n          invs * d[i] * d[j];\n    }\n    inverse_A_[i * size_A_after + size_A] = -invs * d[i];\n    inverse_A_[size_A * size_A_after + i] = -invs * d[i];\n  }\n  inverse_A_[size_A * size_A_after + size_A] = invs;\n\n  return true;\n}\n\n/**\n * \\brief InvertAfterRemoval function is used to invert the matrix. Used in solveQP\n */\nvoid InvertAfterRemoval(vector  &inverse_A_,const vector &active_set,\n                                       int removed_index) {\n  vector inverse_A = inverse_A_;\n  int size_A = active_set.size() + 1;\n  vector r(size_A);\n\n  ++removed_index; // Index in A has an offset of 1.\n  double invs = inverse_A[removed_index * size_A + removed_index];\n  assert(!NEARLY_ZERO_TOL(invs, 1e-12));\n  double s = 1.0 / invs;\n  vector d(size_A - 1, 0.0);\n  int k = 0;\n  for (int i = 0; i < size_A; ++i) {\n    if (i == removed_index) continue;\n    d[k] = -s * inverse_A[removed_index * size_A + i];\n    ++k;\n  }\n\n  int size_A_after = size_A - 1;\n  inverse_A_.resize(size_A_after * size_A_after);\n  k = 0;\n  for (int i = 0; i < size_A; ++i) {\n    if (i == removed_index) continue;\n    int l = 0;\n    for (int j = 0; j < size_A; ++j) {\n      if (j == removed_index) continue;\n      inverse_A_[k * size_A_after + l] = inverse_A[i * size_A + j] -\n          invs * d[k] * d[l];\n      ++l;\n    }\n    ++k;\n  }\n}\n\n/**\n * \\brief ComputeActiveSetSimilarities computes Mnz'*Mnz. Used in solveQP\n */\nvoid ComputeActiveSetSimilarities(vertex_type& vertex,\n    const vector &active_set,\n    vector *similarities) {\n  int size = active_set.size();\n\n  // Compute similarity matrix.\n  similarities->resize(size * size);\n  (*similarities)[0] = 0.0;\n  for (int i = 0; i < active_set.size(); ++i) {\n    (*similarities)[i*size + i] = static_cast(\n        CountCommonValues(vertex,active_set[i], active_set[i]) );\n    for (int j = i+1; j < active_set.size(); ++j) {\n      // Count how many variable values the i-th and j-th \n      // assignments have in common.\n      int num_common_values = CountCommonValues(vertex,active_set[i], active_set[j]);\n      (*similarities)[i*size + j] = num_common_values;\n      (*similarities)[j*size + i] = num_common_values;\n    }\n  }\n}\n\n/**\n * \\brief  ComputeMarginalsFromSparseDistribution computes marginalvalues for unary \n * factor from given factor distribution.\n */\n \nvoid ComputeMarginalsFromSparseDistribution( vertex_type& vertex, \n    const vector &active_set,\n    const vector &distribution,\n    vec  &variable_posteriors,\n    vec &additional_posteriors) {\n    variable_posteriors.setZero();           \n    additional_posteriors.setZero();  \n    for (int i = 0; i < active_set.size(); ++i) {\n    UpdateMarginalsFromConfiguration(vertex,active_set[i],\n                                       distribution[i],\n                                       variable_posteriors,\n                                       additional_posteriors);\n    }\n  }\n  \n  \n   // Given a configuration with a probability (weight), \n  // increment the vectors of variable and additional posteriors.\n  void UpdateMarginalsFromConfiguration(vertex_type& vertex,\n    const Configuration &configuration,\n    double weight,\n    vec &variable_posteriors,\n    vec &additional_posteriors) {\n    \n     vector  states(vertex.data().nvars, -1);\n     get_configuration_states(vertex, configuration, &states);\n     \n            int offset = 0;\n            \n            for (int k = 0; k < vertex.data().nvars; ++k) \n            {   variable_posteriors[offset + states[k]] += weight;\n                offset += vertex.data().cards[k];\n            }\n    additional_posteriors[configuration] += weight;\n \n  }\n  // Count how many common values two configurations have.\n  int CountCommonValues(vertex_type& vertex,Configuration configuration1,\n                        Configuration configuration2) {\n    \n    //assert(states1->size() == states2->size());\n    int count = 0;\n    vector  states1(vertex.data().nvars, -1); \n    vector  states2(vertex.data().nvars, -1);\n    get_configuration_states(vertex, configuration1, &states1);\n    get_configuration_states(vertex, configuration2, &states2);\n    for(int i = 0; i< vertex.data().nvars; i++)\n    {  if (states1[i] == states2[i])\n      { count++;} }\n    return count;\n  }\n  \n\n/**\n * \\brief Evaluate returns the maximum value  with reference to \n * input additional and variable log potentials and configuration. addtional \n * log potential corresponds factor potentials and variable potential corresponds \n * to sum of lagrange  * multipliers and unary potentials divided by degree of \n * the unary vertex.\n */\n  \n  \nvoid Evaluate(vertex_type& vertex, vec additional_log_potentials, vec variable_log_potentials,\n                const Configuration configuration,\n                double *value) {\n          \n    vector states(vertex.data().nvars, -1);\n    get_configuration_states(vertex, configuration, &states);\n    *value = 0.0;\n    int offset = 0;\n    for (int i = 0;i *similarities,\n                            vector *eigenvalues) {\n\n  int size = sqrt(similarities->size());\n\n  Eigen::SelfAdjointEigenSolver es;\n  Eigen::MatrixXd sim(size, size);\n  int t = 0;\n  for (int i = 0; i < size; ++i) {\n    for (int j = 0; j < size; ++j) {\n      sim(i, j) = (*similarities)[t];\n      ++t;\n    }\n  }\n  es.compute(sim);\n  const Eigen::VectorXd &eigvals = es.eigenvalues(); \n  eigenvalues->resize(size);\n  for (int i = 0; i < size; ++i) {\n    (*eigenvalues)[i] = eigvals[i];\n  }\n  const Eigen::MatrixXd &eigvectors = es.eigenvectors().transpose();\n  t = 0;\n  for (int i = 0; i < size; ++i) {\n    for (int j = 0; j < size; ++j) {\n      (*similarities)[t] = eigvectors(i, j);\n      ++t;\n    }\n  }\n\n}\n\n\n// Function to solve each quadratic sub problem for dense factors. \n// It uses active set method. Caching is deactivated\n// TODO: Activate caching feature\n\nvoid SolveQP_dense(vertex_type& vertex,const gather_type& total,\n            vec& variable_posteriors, vec& additional_posteriors) {\n   vertex_data& vdata = vertex.data();                        \n\n   vec additional_log_potentials = vdata.potentials;\n   vec variable_log_potentials = total.neighbor_distribution + total.messages;      \n   \n  // Initialize the active set.\n  \n   if (vdata.active_set_.size() == 0) {\n    variable_posteriors.resize(variable_log_potentials.size());     \n    additional_posteriors.resize(additional_log_potentials.size()); \n    vdata.distribution_.clear();\n    // Initialize by solving the LP, discarding the quadratic\n    // term.\n    Configuration configuration = -1;\n    double value;\n    Maximize(vertex, additional_log_potentials, variable_log_potentials,\n             configuration,\n             &value);\n    vdata.active_set_.push_back(configuration);\n    vdata.distribution_.push_back(1.0);\n\n    // Initialize inv(A) as [-M,1;1,0].\n    vdata.inverse_A_.resize(4);\n    vdata.inverse_A_[0] = static_cast(\n        -CountCommonValues(vertex,configuration, configuration));\n    vdata.inverse_A_[1] = 1;\n    vdata.inverse_A_[2] = 1;\n    vdata.inverse_A_[3] = 0;\n  }\n\n  bool changed_active_set = true;\n  vector z;\n  int num_max_iterations = num_max_iterations_QP_;\n  double tau = 0;\n  for (int iter = 0; iter < num_max_iterations; ++iter) {\n    bool same_as_before = true;\n    bool unbounded = false;\n    if (changed_active_set) {\n      // Recompute vector b.\n      vector b(vdata.active_set_.size() + 1, 0.0);\n      b[0] = 1.0;\n      for (int i = 0; i < vdata.active_set_.size(); ++i) {\n        const Configuration &configuration = vdata.active_set_[i];\n        double score;\n        Evaluate(vertex, additional_log_potentials, variable_log_potentials,\n                 configuration,\n                 &score);\n        b[i+1] = score;\n      }\n      // Solve the system Az = b.\n      z.resize(vdata.active_set_.size());\n      int size_A = vdata.active_set_.size() + 1;\n      for (int i = 0; i < vdata.active_set_.size(); ++i) {\n        z[i] = 0.0;\n        for (int j = 0; j < size_A; ++j) {\n          z[i] += vdata.inverse_A_[(i+1) * size_A + j] * b[j];\n        }\n      }\n      tau = 0.0;\n      for (int j = 0; j < size_A; ++j) {\n        tau += vdata.inverse_A_[j] * b[j];\n      }\n\n      same_as_before = false;\n    }\n\n    if (same_as_before) {\n      // Compute the variable marginals from the full distribution\n      // stored in z.\n      ComputeMarginalsFromSparseDistribution(vertex, vdata.active_set_,\n                                             z,\n                                             variable_posteriors,\n                                             additional_posteriors);\n      // Get the most violated constraint\n      // (by calling the black box that computes the MAP).\n      vec scores = variable_log_potentials;               \n      for (int i = 0; i < scores.size(); ++i) {\n        scores[i] -= variable_posteriors[i];\n      }\n      Configuration configuration = -1;\n      double value = 0.0;\n      \n      Maximize(vertex,\n                additional_log_potentials, scores,\n               configuration,\n               &value);\n      double very_small_threshold = 1e-9;\n      if (value <= tau + very_small_threshold) { // value <= tau.\n        // We have found the solution;\n        // the distribution, active set, and inv(A) are cached for the next round.\n        //DeleteConfiguration(configuration);\n        return;\n      } else {\n        for (int k = 0; k < vdata.active_set_.size(); ++k) {\n          // This is expensive and should just be a sanity check.\n          // However, in practice, numerical issues force an already existing\n          // configuration to try to be added. Therefore, we always check\n          // if a configuration already exists before inserting it.\n          // If it does, that means the active set method converged to a\n          // solution (but numerical issues had prevented us to see it.)\n          if (vdata.active_set_[k] == configuration) {                         \n            if (opts.verbose > 2) {\n              cout << \"Warning: value - tau = \"\n                   << value - tau << \" \" << value << \" \" << tau\n                   << endl;\n            }\n            // We have found the solution;\n            // the distribution, active set, and inv(A)\n            // are cached for the next round.\n            //DeleteConfiguration(configuration);\n\n            // Just in case, clean the cache.\n            // This may prevent eventual numerical problems in the future.\n            for (int j = 0; j < vdata.active_set_.size(); ++j) {\n              if (j == k) continue; // This configuration was deleted already.\n              //DeleteConfiguration(active_set_[j]);\n            }\n            vdata.active_set_.clear();\n            vdata.inverse_A_.clear();\n            vdata.distribution_.clear();\n\n            // Return.\n            return;\n          }\n        }\n        z.push_back(0.0);\n        vdata.distribution_ = z;\n\n        // Update inv(A).\n        bool singular = !InvertAfterInsertion(vertex, vdata.inverse_A_, vdata.active_set_, configuration);\n        if (singular) {\n          // If adding a new configuration causes the matrix to be singular,\n          // don't just add it. Instead, look for a configuration in the null\n          // space and remove it before inserting the new one.\n          // Right now, if more than one such configuration exists, we just\n          // remove the first one we find. There's a chance this could cause\n          // some cyclic behaviour. If that is the case, we should randomize\n          // this choice.\n          // Note: This step is expensive and requires an eigendecomposition.\n          // TODO: I think there is a graph interpretation for this problem.\n          // Maybe some specialized graph algorithm is cheaper than doing\n          // the eigendecomposition.\n          vector similarities(vdata.active_set_.size() * vdata.active_set_.size());\n          ComputeActiveSetSimilarities(vertex,vdata.active_set_, &similarities);\n          \n          //cout<<\"compute active similarities in solveQP ..\"< padded_similarities((vdata.active_set_.size()+2) * \n                                             (vdata.active_set_.size()+2), 1.0);\n          for (int i = 0; i < vdata.active_set_.size(); ++i) {\n            for (int j = 0; j < vdata.active_set_.size(); ++j) {\n              padded_similarities[(i+1)*(vdata.active_set_.size()+2) + (j+1)] =\n                  similarities[i*vdata.active_set_.size() + j];\n            }\n          }\n          padded_similarities[0] = 0.0;\n          for (int i = 0; i < vdata.active_set_.size(); ++i) {\n            double value = static_cast(\n                CountCommonValues(vertex, configuration, vdata.active_set_[i]));\n            padded_similarities[(i+1)*(vdata.active_set_.size()+2) +\n                                (vdata.active_set_.size()+1)] = value;\n            padded_similarities[(vdata.active_set_.size()+1)*(vdata.active_set_.size()+2) +\n                                (i+1)] = value;\n          }\n          double value = static_cast(\n              CountCommonValues(vertex, configuration, configuration));\n          padded_similarities[(vdata.active_set_.size()+1)*(vdata.active_set_.size()+2) +\n                              (vdata.active_set_.size()+1)] = value;\n\n          vector eigenvalues(vdata.active_set_.size()+2);\n          EigenDecompose(&padded_similarities, &eigenvalues);\n          int zero_eigenvalue = -1;\n          for (int i = 0; i < vdata.active_set_.size()+2; ++i) {\n            if (NEARLY_EQ_TOL(eigenvalues[i], 0.0, 1e-9)) {\n              if (zero_eigenvalue >= 0) {\n                // If this happens, something failed. Maybe a numerical problem\n                // may cause this. In that case, just give up, clean the cache\n                // and return. Hopefully the next iteration will fix it.\n                cout << \"Multiple zero eigenvalues: \"\n                     << eigenvalues[zero_eigenvalue] << \" and \"\n                     << eigenvalues[i] << endl;\n                cout << \"Warning: Giving up.\" << endl;\n                // Clean the cache.\n                for (int j = 0; j < vdata.active_set_.size(); ++j) {\n                  //DeleteConfiguration(active_set_[j]);\n                }\n                vdata.active_set_.clear();\n                vdata.inverse_A_.clear();\n                vdata.distribution_.clear();\n                return;\n              }\n              zero_eigenvalue = i;\n            }\n          }\n          assert(zero_eigenvalue >= 0);\n          vector configurations_to_remove;\n          for (int j = 1; j < vdata.active_set_.size()+1; ++j) {\n            double value = padded_similarities[zero_eigenvalue*(vdata.active_set_.size()+2) + j];\n            if (!NEARLY_EQ_TOL(value, 0.0, 1e-9)) {\n              configurations_to_remove.push_back(j-1);\n            }\n          }\n          if (opts.verbose > 2) {\n            cout << \"Pick a configuration to remove (\" << configurations_to_remove.size()\n                 << \" out of \" << vdata.active_set_.size() << \").\" << endl;\n          }\n\n          assert(configurations_to_remove.size() >= 1);\n          int j = configurations_to_remove[0];\n\n          // Update inv(A).\n          InvertAfterRemoval(vdata.inverse_A_, vdata.active_set_, j);\n\n          // Remove blocking constraint from the active set.\n          //DeleteConfiguration(active_set_[j]); // Delete configutation.\n          vdata.active_set_.erase(vdata.active_set_.begin() + j);\n\n          singular = !InvertAfterInsertion(vertex, vdata.inverse_A_, vdata.active_set_, configuration);\n          assert(!singular);\n        }\n\n        // Insert configuration to active set.\n        if (opts.verbose > 2) {\n          cout << \"Inserted one element to the active set (iteration \"\n               << iter << \").\" << endl;\n        }\n        vdata.active_set_.push_back(configuration);\n        changed_active_set = true;\n      }      \n    } else {\n      // Solution has changed from the previous iteration.\n      // Look for blocking constraints.\n      int blocking = -1;\n      bool exist_blocking = false;\n      double alpha = 1.0;\n      for (int i = 0; i < vdata.active_set_.size(); ++i) {\n        assert(vdata.distribution_[i] >= -1e-12);\n        if (z[i] >= vdata.distribution_[i]) continue;\n        if (z[i] < 0) exist_blocking = true;\n        double tmp = vdata.distribution_[i] / (vdata.distribution_[i] - z[i]);\n        if (blocking < 0 || tmp < alpha) {\n          alpha = tmp;\n          blocking = i;\n        }\n      }\n\n      if (!exist_blocking) {\n        // No blocking constraints.\n        assert(!unbounded);\n        vdata.distribution_ = z;\n        alpha = 1.0;\n        changed_active_set = false;\n      } else {\n        if (alpha > 1.0 && !unbounded) alpha = 1.0;\n        // Interpolate between factor_posteriors_[i] and z.\n        if (alpha == 1.0) {\n          vdata.distribution_ = z;\n        } else {\n          for (int i = 0; i < vdata.active_set_.size(); ++i) {\n            z[i] = (1 - alpha) * vdata.distribution_[i] + alpha * z[i];\n            vdata.distribution_[i] = z[i];\n          }\n        }\n\n        // Update inv(A).\n        InvertAfterRemoval(vdata.inverse_A_, vdata.active_set_, blocking);\n\n        // Remove blocking constraint from the active set.\n        if (opts.verbose > 2) {\n          cout << \"Removed one element to the active set (iteration \"\n               << iter << \").\" << endl;\n        }\n\n        //DeleteConfiguration(active_set_[blocking]); // Delete configutation.\n        vdata.active_set_.erase(vdata.active_set_.begin() + blocking);\n\n        z.erase(z.begin() + blocking);\n        vdata.distribution_.erase(vdata.distribution_.begin() + blocking);\n        changed_active_set = true;\n        for (int i = 0; i < vdata.distribution_.size(); ++i) {\n          assert(vdata.distribution_[i] > -1e-16);\n        }\n      }\n    }\n  }\n\n  // Maximum number of iterations reached.\n  // Return the best existing solution by computing the variable marginals \n  // from the full distribution stored in z.\n  //assert(false);\n  ComputeMarginalsFromSparseDistribution(vertex, vdata.active_set_,\n                                         z,\n                                         variable_posteriors,\n                                         additional_posteriors); \n\n  }\n  \n  void InsertionSort(pair arr[], int length) {\n  int i, j;\n  pair tmp;\n\n  for (i = 1; i < length; i++) {\n    j = i;\n    while (j > 0 && arr[j - 1].first > arr[j].first) {\n      tmp = arr[j];\n      arr[j] = arr[j - 1];\n      arr[j - 1] = tmp;\n      j--;\n    }\n  }\n}\n\n  \n  int project_onto_budget_constraint_cached(vec& x,\n                                          int d,\n                                          double budget, \n                                          vector >& y) {\n  int j, k, l, level;\n  double s = 0.0;\n  double tau = 0.0, tightsum;\n  double left, right = -std::numeric_limits::infinity();\n\n  // Load x into a reordered y (the reordering is cached).\n  if (y.size() != d) {\n    y.resize(d);\n    for (j = 0; j < d; j++) {\n      s -= x[j];\n      y[j].first = -x[j];\n      y[j].second = j;\n    }\n    sort(y.begin(), y.end());\n  } else {\n    for (j = 0; j < d; j++) {\n      s -= x[j];\n      y[j].first = -x[y[j].second];\n    }\n    // If reordering is cached, use a sorting algorithm \n    // which is fast when the vector is almost sorted.\n    InsertionSort(&y[0], d);\n  }\n\n  tightsum = s;\n  s += budget;\n  \n  k = l = level = 0;\n  bool found = false;\n  double val_a, val_b;\n  while (k < d && l < d) {\n    if (level != 0) {\n      tau = (s - tightsum) / static_cast(level);\n    }\n    if (k < d) val_a = y[k].first;\n    val_b = 1.0 + y[l].first;\n    left = right;\n    if (k == d || val_b <= val_a) {\n      right = val_b;\n    } else {\n      right = val_a;\n    }\n    if ((level == 0 && s == tightsum) || (level != 0 && tau <= right)) {\n      // Found the right split-point!\n      found = true;\n      break;\n    }\n    if (k == d || val_b <= val_a) {\n      tightsum += val_b;\n      --level;\n      ++l;\n    } else {\n      tightsum -= val_a;\n      ++level;\n      ++k;\n    }\n  }\n\n  if (!found) {\n    left = right;\n    right = std::numeric_limits::infinity();\n  }\n      \n  for (j = 0; j < d; j++) {\n    if (-x[j] >= right) {\n      x[j] = 0.0;\n    } else if (1.0 - x[j] <= left) {\n      x[j] = 1.0;\n    } else {\n      x[j] += tau;\n    }\n  }\n\n  return 0;\n}\n\n  \n  int project_onto_budget_constraint(vec& x, int d, double budget) {\n  int j, k, l, level;\n  double s = 0.0;\n  double tau = 0.0, tightsum;\n  double left, right = -std::numeric_limits::infinity();\n  vector y(d, 0.0);\n\n  for (j = 0; j < d; j++) {\n    s -= x[j];\n    y[j] = -x[j];\n  }\n  sort(y.begin(), y.end());\n  tightsum = s;\n  s += budget;\n  \n  k = l = level = 0;\n  bool found = false;\n  double val_a, val_b;\n  while (k < d && l < d) {\n    if (level != 0) {\n      tau = (s - tightsum) / static_cast(level);\n    }\n    if (k < d) val_a = y[k];\n    val_b = 1.0 + y[l];\n    left = right;\n    if (k == d || val_b <= val_a) {\n      right = val_b;\n    } else {\n      right = val_a;\n    }\n    if ((level == 0 && s == tightsum) || (level != 0 && tau <= right)) {\n      // Found the right split-point!\n      found = true;\n      break;\n    }\n    if (k == d || val_b <= val_a) {\n      tightsum += val_b;\n      --level;\n      ++l;\n    } else {\n      tightsum -= val_a;\n      ++level;\n      ++k;\n    }\n  }\n\n  if (!found) {\n    left = right;\n    right = std::numeric_limits::infinity();\n  }\n      \n  for (j = 0; j < d; j++) {\n    if (-x[j] >= right) {\n      x[j] = 0.0;\n    } else if (1.0 - x[j] <= left) {\n      x[j] = 1.0;\n    } else {\n      x[j] += tau;\n    }\n  }\n\n  return 0;\n}\n  \n  // Solve the QP subproblem for budget factor.\n  // TODO Enable caching\nvoid SolveQP_budget(vertex_type& vertex,const gather_type& total,\n            vec& variable_posteriors, vec& additional_posteriors){\n            \n  vertex_data& vdata =  vertex.data();\n  vec variable_log_potentials = total.neighbor_distribution + total.messages;\n  for (int f = 0; f < variable_log_potentials.size(); ++f) {\n    variable_posteriors[f] = variable_log_potentials[f];\n    if (variable_posteriors[f] < 0.0) {\n      variable_posteriors[f] = 0.0;\n    } else if (variable_posteriors[f] > 1.0) {\n      variable_posteriors[f] = 1.0;\n    }\n  }\n\n  double s = 0.0;\n  for (int f = 0; f < vdata.nvars; ++f) {\n    s += variable_posteriors[f];\n  }\n\n  if (s > static_cast(vdata.budget)) {\n    for (int f = 0; f < variable_log_potentials.size(); ++f) {\n      variable_posteriors[f] = variable_log_potentials[f];\n    }\n    project_onto_budget_constraint_cached(variable_posteriors, \n                                          variable_log_potentials.size(), \n                                          static_cast(vdata.budget), vdata.last_sort_);\n  }\n}\n\n// Finds best configuration of budget factors\nvoid SolveMAP_budget(vertex_type& vertex,const gather_type& total,\n            vec& variable_posteriors, vec& additional_posteriors, double& value) {\n \n   vertex_data& vdata = vertex.data();\n  // Create a local copy of the log potentials.\n   vec log_potentials(total.messages); \n  double valaux;\n \n  value = 0.0;\n  \n  int num_active = 0;\n  double sum = 0.0;\n\n  for (int f = 0; f < vdata.nvars; ++f) {\n    valaux = log_potentials[f];\n    if (valaux < 0.0) {\n      variable_posteriors[f] =  0.0;\n    } else {\n      sum += valaux;\n      variable_posteriors[f] = 1.0;\n    }\n    ++num_active;\n  }\n  if (num_active > vdata.budget) {\n    vector > scores(vdata.nvars);\n    for (int f = 0; f < vdata.nvars; ++f) {\n      scores[f].first = -log_potentials[f];\n      scores[f].second = f;\n    }\n\n    sort(scores.begin(), scores.end());\n    num_active = 0;\n    sum = 0.0;\n    for (int k = 0; k < vdata.budget; ++k) {\n      valaux = -scores[k].first;\n      if (valaux < 0.0) break;\n      int f = scores[k].second;\n      variable_posteriors[f] = 1.0;\n      sum += valaux;\n      ++num_active;      \n    }\n\n    for (int k = num_active; k < vdata.nvars; ++k) {\n      int f = scores[k].second;\n      variable_posteriors[f] = 0.0;\n    }   \n  }\n  \n  value += sum;\n  \n  \n}\n\n// Finds best configuration for dense factors\nvoid SolveMAP_dense(vertex_type& vertex,const gather_type& total,\n            vec& variable_posteriors, vec& additional_posteriors, double& value){\n     vertex_data& vdata = vertex.data();\n     vec beliefs = vdata.potentials;         \n     int num_configurations = vdata.potentials.size();\n     for (int index_configuration = 0;\n           index_configuration < num_configurations;\n            ++index_configuration) {\n        vector states(vdata.nvars, -1);\n        get_configuration_states(vertex, index_configuration, &states);\n        int offset = 0;\n        for (int k = 0; k < vdata.nvars; ++k) {\n             beliefs[index_configuration] += total.messages[offset + states[k]];\n             offset += vdata.cards[k];}\n    } \n            \n        value = beliefs.maxCoeff();\n \n }\n\nint project_onto_simplex_cached(vec& x,\tint d, double r, \n\t\t\t\tvector >& y) {\n  int j;\n  double s = 0.0;\n  double tau;\n\n  // Load x into a reordered y (the reordering is cached).\n  if (y.size() != d) {\n    y.resize(d);\n    for (j = 0; j < d; j++) {\n      s += x[j];\n      y[j].first = x[j];\n      y[j].second = j;\n    }\n    sort(y.begin(), y.end());\n  } else {\n    for (j = 0; j < d; j++) {\n      s += x[j];\n      y[j].first = x[y[j].second];\n    }\n    // If reordering is cached, use a sorting algorithm \n    // which is fast when the vector is almost sorted.\n    InsertionSort(&y[0], d);\n  }\n\n  for (j = 0; j < d; j++) {\n    tau = (s - r) / ((double) (d - j));\n    if (y[j].first > tau) break;\n    s -= y[j].first;\n  }\n\n  for (j = 0; j < d; j++) {\n    if (x[j] < tau) {\n      x[j] = 0.0;\n    } else {\n      x[j] -= tau;\n    }\n  }\n\n  return 0;\n}\n\nvoid SolveQP_xor(vertex_type& vertex,const gather_type& total,\n            vec& variable_posteriors, vec& additional_posteriors) {\n  vertex_data& vdata =  vertex.data();\n  //vec variable_posteriors = total.neighbor_distribution + total.messages;\n  for (int f = 0; f < variable_posteriors.size(); ++f) {\n    variable_posteriors[f] = total.neighbor_distribution[f] + total.messages[f]; \n    //cout< log_potentials[first]) first = f;\n  }\n  value += log_potentials[first];\n  variable_posteriors.setZero();\n  variable_posteriors[first] = 1.0;\n};\n\n// Finds beliefs using dense and budget factors\nvoid compute_beliefs(vertex_type& vertex,const gather_type& total,\n            vec& variable_posteriors, vec& additional_posteriors){\nswitch(vertex.data().factor_type){\n\ncase DENSE: SolveQP_dense(vertex,total, variable_posteriors, additional_posteriors);\n        break;\ncase BUDGET: SolveQP_budget(vertex,total, variable_posteriors, additional_posteriors);\n         break;\n\ncase XOR: SolveQP_xor(vertex,total, variable_posteriors, additional_posteriors);\n\n}\n};\n\n// General solveMAP function\nvoid SolveMAP(vertex_type& vertex,const gather_type& total,\n            vec& variable_posteriors, vec& additional_posteriors, double& value){\nswitch(vertex.data().factor_type){\ncase XOR: SolveMAP_xor(vertex,total, variable_posteriors, additional_posteriors, value);\n          break;\ncase DENSE: SolveMAP_dense(vertex,total, variable_posteriors, additional_posteriors, value);\n        break;\ncase BUDGET: SolveMAP_budget(vertex,total, variable_posteriors, additional_posteriors, value);\n  }\n };\n\n};\n/* end of ad3_vertex_program*/\n\n\n\n////////////////////////////////////////////////////////////////////////////////\n// This class implements the Bethe-ADMM as  described in:\n//\n//   Qiang Fu, Huahua Wang and Arindam Banerjee.\n// \"Bethe-ADMM for Tree Decomposition based Parallel MAP Inference\"\n//  Conference on Uncertainty in Artificial Intelligence (UAI), 2013\n//\n//////////////////////////////////////////////////////////////////////////////// \n\nstruct bethe_admm_vertex_program:public admm_vertex_program {\n\n/* compute_grad_phi computes the gradient of bethe entropy for the factor */\n  \n       void compute_grad_phi(vertex_type& vertex,vec& unary_beliefs, \n                   vec& factor_beliefs, vec& unary_grad,vec& factor_grad) {\n\n             vertex_data& vdata = vertex.data();  \n            // computation for variable beliefs  \n             for(int i=0; i< unary_beliefs.size(); i++){\n                 unary_grad[i] *= EXP * (unary_beliefs[i]);\n             }\n             // computation for factor beliefs\n             if(vdata.factor_type == DENSE){\n             vector states(vdata.nvars);\n             for(int i=0; i< factor_beliefs.size(); i++){\n                 factor_grad[i] *= factor_beliefs[i] / EXP;\n                 get_configuration_states(vertex, i, &states);\n                 int offset =0;\n                 for(int j=0; j< vdata.nvars; j++){\n                     factor_grad[i] /= unary_beliefs[offset + states[j]];\n                     offset += vdata.cards[j];\n                 }\n             }\n           }\n        }\n\n\n/* run_bp computes marginal beliefs using sum-product belief propagation */\n       void run_bp(vertex_type& vertex, vec& unary_pots, vec& factor_pots, \n             vec& unary_margs, vec& factor_margs) {\n\n            vertex_data& vdata = vertex.data();        \n            unary_margs.resize(unary_pots.size());\n            factor_margs = factor_pots;\n            vector states(vdata.nvars, -1);\n           // computing messages\n           for(int i=0; i < vdata.nvars; i++) {\n               \n               vec messages = factor_pots;\n               for(int j=0; j < factor_pots.size(); j++) {\n                  get_configuration_states(vertex, j, &states);\n                  int offset = 0;\n                  for(int k = 0; k < vdata.nvars; k++) {\n                     if( k != i)\n                     messages[j] *= unary_pots[offset + states[k]];\n                     offset += vdata.cards[k];\n                  }\n               }\n               vector  marg_messages(vdata.cards[i], 0);\n               for(int j=0; j < factor_pots.size(); j++)  {\n                   get_configuration_states(vertex, j, &states);\n                   marg_messages[states[i]] += messages[j];\n                   }\n               int var_offset =0;\n               for(int j=0; j < i; j++) {          \n                   var_offset += vdata.cards[j];\n               }\n               double sum = 0;\n               // computing marginal beliefs for variables\n               for(int j=0; j < marg_messages.size(); j++) { \n                  unary_margs[var_offset + j] = marg_messages[j] \n                                           * unary_pots[var_offset + j];\n                  sum += unary_margs[var_offset + j];\n               }\n               for(int j=0; j < marg_messages.size(); j++) { \n                   unary_margs[var_offset + j] /= sum ; \n               }       \n          }\n          // compuitng factor beliefs\n          double fact_sum = 0;\n          for(int i=0; i < factor_pots.size(); i++) {\n              get_configuration_states(vertex, i, &states);\n              int offset = 0;\n              for(int j =0; j < vdata.nvars; j++) {\n                 factor_margs[i] *= unary_pots[offset + states[j]];\n                 offset += vdata.cards[j];\n              }\n              fact_sum += factor_margs[i];\n          }\n        \n          for(int i=0; i < factor_pots.size(); i++) {\n              factor_margs[i] /= fact_sum;\n          }\n         \n       }\n\n\n/* adjust_beliefs prevents overflow/ underflow of belief variable */\n       void adjust_beliefs(vertex_type& vertex){\n            \n            vertex_data& vdata = vertex.data();\n            for(int i=0; i< vdata.beliefs.size(); i++){\n                if(vdata.beliefs[i] < 10e-100)\n                   vdata.beliefs[i] = 10e-100;\n            }  \n            \n            if(vdata.factor_type == DENSE){\n            for(int i=0; i< vdata.factor_beliefs.size(); i++){\n                if(vdata.factor_beliefs[i] < 10e-100)\n                   vdata.factor_beliefs[i] = 10e-100;\n            }\n           } \n       }\n       \n/* exponentiates potentials for bp. TODO: use faster approximation of pow */\n       void exponentiate(vec& potential_vector){\n       \n            for(int i=0; i< potential_vector.size(); i++){\n                potential_vector[i] = pow(EXP, potential_vector[i]);\n            }\n       }\n       \n       void softmax(vertex_type& vertex, vec& unary_pots, vec& unary_margs){\n                   unary_margs = unary_pots/unary_pots.sum();}\n         /* solves QP for factor vertices using bp */\n       void compute_beliefs(vertex_type& vertex,const gather_type& total,\n            vec& variable_posteriors, vec& additional_posteriors){\n\n             vertex_data& vdata = vertex.data();\n             vec unary_eta, factor_eta;\n             vec unary_grad, factor_grad;\n             // computing eta  \n             \n             unary_eta = total.messages + opts.step_size * (total.neighbor_distribution - vdata.beliefs); \n             unary_eta = (unary_eta)/opts.alpha;        \n             exponentiate(unary_eta);\n             unary_grad.resize(unary_eta.size());\n             \n             if(vdata.factor_type == DENSE){\n             factor_eta = (vdata.potentials)/opts.alpha; \n             exponentiate(factor_eta);\n\t     factor_grad.resize(factor_eta.size());\n             }\n             compute_grad_phi(vertex, vdata.beliefs, vdata.factor_beliefs, unary_eta, factor_eta);\n             //running bp on eta\n             switch(vdata.factor_type){\n             case DENSE: run_bp(vertex, unary_eta, factor_eta, vdata.beliefs, vdata.factor_beliefs); \n                         break;\n             case XOR: softmax(vertex, unary_eta, vdata.beliefs);\n             }\n             //adjust beliefs for overflow/underflow          \n             adjust_beliefs(vertex);             \n        };\n /* solves MAP for factor vertices */      \n        void SolveMAP_dense(vertex_type& vertex,const gather_type& total,\n            vec& variable_posteriors, vec& additional_posteriors, double& value){\n             vertex_data& vdata = vertex.data();\n             vec beliefs = vdata.potentials;\n             vector states(vdata.nvars);\n             for(int i=0; i< vdata.potentials.size(); i++) {\n                 get_configuration_states(vertex, i, &states);\n                 int offset = 0;\n                 for(int j=0; j< vdata.nvars; j++){\n                     beliefs[i] += total.messages[offset + states[j]];\n                     offset += vdata.cards[j];\n                 }\n             }\n             value = beliefs.maxCoeff();  \n         };\n \n         void SolveMAP_xor(vertex_type& vertex,const gather_type& total,\n            vec& variable_posteriors, vec& additional_posteriors, double& value) {\n            vertex_data& vdata =  vertex.data();\n            vec log_potentials = total.messages;\n\n            int first = -1;\n            value = 0.0;\n\n            for (int f = 0; f < log_potentials.size(); ++f) {\n            if (first < 0 || log_potentials[f] > log_potentials[first]) first = f;\n             }\n            value += log_potentials[first];\n            variable_posteriors.setZero();\n            variable_posteriors[first] = 1.0;\n        };\n\n       void SolveMAP(vertex_type& vertex,const gather_type& total,\n            vec& variable_posteriors, vec& additional_posteriors, double& value){\n             switch(vertex.data().factor_type){\n             case XOR: SolveMAP_xor(vertex,total, variable_posteriors, additional_posteriors, value);\n                       break;\n             case DENSE: SolveMAP_dense(vertex,total, variable_posteriors, additional_posteriors, value);\n             }\n       };\n\n\n};\n/* end of  bethe_admm_vertex_program */\n\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/dd_grlab.hpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n *\n * \\brief This application performs MAP inference on Markov Nets \n * provided in standard UAI file format via Dual-Decomposition. \n *\n *\n *  \\authors Dhruv Batra, André Martins, Aroma Mahendru\n */\n\n\n#ifndef __DD_GRLAB_HPP__\n#define __DD_GRLAB_HPP__\n\n#include \n#include \n#include \n#include \n#include \n\n\n#include \n#include \"eigen_serialization.hpp\"\n\n#include \"dd_opts.hpp\"\n#include \"utils.hpp\"\n\n#include \n#include \n\n\nusing namespace std;\n\n\n/**\n * \\brief Eigen library vectors are used to store the potentials (log-space)\n */\ntypedef Eigen::VectorXd factor_type;\ntypedef Eigen::VectorXd vec;\ntypedef Eigen::MatrixXd mat;\n\n\n\nstruct dd_global_vars {\n\ndouble old_dual ;        // stores the value of dual objective for the previous iteration\ndouble primal_best ;     //  stores the value of best primal objective found so far.\nbool converged ;         // true if dual objective value has converged to required tolerance level, otherwise false\nint dual_inc_count ;     // keeps track of the number of times the value of dual objective increased\nvector < vector > history ; // stores dual and primal objective values\nint sq_norm_g ;   //  stores the value of the square of the norm of the subgradient vector\nint iter_at_aggregate ;  //  iteration number at the time of aggregate\ngraphlab::timer timer ; //  time object. Helps in finding the time elapsed.\ndd_global_vars(): old_dual(200), primal_best(-1e10),\n                  converged(false), dual_inc_count(1),\n                  history(4,vector()), \n                  sq_norm_g(100), \n                  iter_at_aggregate(0){};\n} global_vars;\n\n/* end of struct dd_global_vars */\n\n\n\n\n\n\n\n/////////////////////////////////////////////////////////////////////////\n// Edge and Vertex data and Graph Type\n/**\n * \\brief There is a vertex for each factor in the graph AND each singleton\n */\nstruct vertex_data \n{\n    int nvars;                 // Number of variables in this factor.\n    int factor_type;           //type of  factor : dense(0), budget(1)\n    int degree;                // Degree of this factor (same as nvars for higher-order factors).\n    \n    vector cards;         // Cardinality of each variable.\n    vector neighbors;     // Vertex ids of the neighbors.    \n    vec potentials;            // Potentials for each configuration of the factor.\n    int budget;                // Only for Budget factors\n    vector  bound_states;  // Only for Budget Factors\n    vector unary_degree;   // Only for unary vertices\n    \n    int best_configuration;    // Index of the best configuration at a subgradient step.\n                               // TODO: Maybe replace best_configuration by beliefs for the high order variables?\n                               // In which case, beliefs would be vector beliefs.\n\n    double dual_contrib;       // Contribution of this factor to the dual. We can compute this during the applies. \n    double primal_contrib;     // Contribution of this factor to the primal. We can compute this during the applies. \n                               // NOTENOTE: ONLY true for sync engine. For async, we need to write an aggregator function. \n    double dual_res_contrib;   // Contribution of this factor to the dual residual. (Used only for ADMM)                    \n    double primal_res_contrib; // Contribution of this factor to the primal residual. (Used only for ADMM)   \n    double primal_rel_contrib; // Contribution of this factor to the relaxed primal. (Used only for ADMM)   \n    \n    vec beliefs;               // Posterior values for the configurations after averaging (projected DD, unary variables only).\n    vec factor_beliefs;        // Posterior value for factor variables\n    int apply_count;           // No. of times apply has been called on this vertex\n    int sum_sq_norm_g;         // Sum of square of norm of subgradient for each vertex (used only for factor vertices)\n    \n    bool schedule_vertex;      // Decides if vertex is to be scheduled for further iterations or not\n    \n    vector distribution_;\n    vector active_set_;\n    vector inverse_A_;\n    vector > last_sort_;\n\n    vertex_data(): \n    nvars(0), factor_type(0), degree(0),\n    budget(0),\n    best_configuration(0),\n    dual_contrib(0), primal_contrib(0),\n    dual_res_contrib(0), primal_res_contrib(0), primal_rel_contrib(0),\n    sum_sq_norm_g(0),\n    apply_count(0),\n    schedule_vertex(true)\n    {}\n    \n    void load(graphlab::iarchive& arc) \n    {\n        arc >> nvars >> degree \n            >> cards >> neighbors >> potentials \n            >> dual_contrib >> primal_contrib\n            >> best_configuration >> beliefs \n            >> apply_count>>factor_beliefs\n            >>sum_sq_norm_g>>primal_res_contrib\n            >>primal_rel_contrib>>dual_res_contrib\n            >>schedule_vertex>>factor_type\n            >>budget>>bound_states\n            >>unary_degree>>active_set_\n            >>inverse_A_>>last_sort_ ;\n    }\n    void save(graphlab::oarchive& arc) const \n    {\n        arc << nvars << degree \n            << cards << neighbors << potentials \n            << dual_contrib << primal_contrib\n            << best_configuration << beliefs \n            << apply_count <> potentials >> multiplier_messages >> local_messages;\n    }\n    void save(graphlab::oarchive& arc) const {\n        arc << potentials << multiplier_messages << local_messages;\n    }\n};  //end of edge_data\n\n\n/**\n * \\brief gather_type is a structure that will be used as the return type of gather function. It includes \n *  messages (used both for unary and factor vertices), neighbor_best_conf, neighbor_distribution (used only for \n *  factor vertices) and sq_norm_g (for storing square of norm of subgradient) for each edge.\n */\n\nstruct gather_type\n{ factor_type messages;\n  factor_type multipliers;\n  vector  neighbor_conf;\n  vec neighbor_distribution;\n  int sq_norm_g;\n     \n    gather_type():sq_norm_g(0){};\n    \n    \n    gather_type(factor_type f, vector  nc = vector  (), int sg = 0, \n                             vec nd = vec() ): messages(f), neighbor_conf(nc),\n                                   neighbor_distribution(nd), sq_norm_g(sg){};\n    void load(graphlab::iarchive& arc) {\n        arc >>messages>>neighbor_conf\n            >>sq_norm_g>>neighbor_distribution\n            >>multipliers;\n    }\n    void save(graphlab::oarchive& arc) const {\n        arc <>dual>>primal>>sum_sq_norm_g\n            >>primal_rel>>total_confs\n            >>dual_res>>primal_res;\n    }\n    void save(graphlab::oarchive& arc) const {\n        arc < graph_type;\n\n\n/** \n * \\brief The Dual Decomposition Vertex Program.\n */\nstruct dd_vertex_program : \npublic graphlab::ivertex_program< graph_type, gather_type,\ngraphlab::messages::sum_priority >,\npublic graphlab::IS_POD_TYPE \n{\n    \n    /////////////////////////////////////////////////////////////////////////\n    // Find the configuration index of a factor given the array of states.\n    /////////////////////////////////////////////////////////////////////////\n    int get_configuration_index(const graph_type::vertex_type& vertex,\n                                const std::vector& states) const \n    {\n        const vertex_data& vdata = vertex.data();\n        int index = states[0];\n        for (size_t i = 1; i < states.size(); ++i) \n        {\n            index *= vdata.cards[i];\n            index += states[i];\n        }\n        return index;\n    }\n    \n    /////////////////////////////////////////////////////////////////////////\n    // Find the array of states corresponding to a factor configuration index.\n    /////////////////////////////////////////////////////////////////////////\n    void get_configuration_states(const graph_type::vertex_type& vertex,\n                                  int index, std::vector* states) const \n    {   \n        const vertex_data& vdata = vertex.data();\n        int tmp = 1;\n        for (size_t i = 1; i < states->size(); ++i) \n            tmp *= vdata.cards[i];\n        \n        (*states)[0] = index / tmp;\n        for (size_t i = 1; i < states->size(); ++i) \n        {   \n            index = index % tmp;\n            tmp /= vdata.cards[i];\n            (*states)[i] = index / tmp;\n        }\n       \n    }\n\n    ///////////////////////////////////////////////////////////\n    // Updates stepsize according to different stepsize rules. \n    ///////////////////////////////////////////////////////////\n\n    double update_stepsize(icontext_type& context, int type, double old_dual, double primal_best,int norm_g_sq,\n                                                                             int dual_inc_count,int iter_since_aggregate) const\n   {  switch (type) {\n      case 0: return opts.step_size;\n              break;\n      case 1: return(opts.step_size/(context.iteration()+2));\n              break;\n      case 2: return(2* opts.step_size *(old_dual-primal_best)/((norm_g_sq+1) * (iter_since_aggregate + dual_inc_count + 1)));\n              break;\n      case 3: return(opts.step_size/dual_inc_count + 1);\n              break;\n      case 4: if(context.iteration()+2 < 300)\n              return( opts.step_size/(context.iteration()+2));\n              else return(opts.step_size/(300));\n                     }\n   }\n    \n    /**\n     * \\brief Given an edge and a vertex return the other vertex along\n     * that edge. \n     */\n    inline vertex_type get_other_vertex(edge_type& edge, \n                                        const vertex_type& vertex) const \n    {\n        return vertex.id() == edge.source().id()? edge.target() : edge.source();\n    }; // end of other_vertex\n    \n    \n    virtual edge_dir_type gather_edges(icontext_type& context,\n                                       const vertex_type& vertex) const = 0;\n    virtual gather_type gather(icontext_type& context, const vertex_type& vertex, \n                               edge_type& edge) const = 0;\n    virtual void apply(icontext_type& context, vertex_type& vertex, \n                       const gather_type& total) = 0;\n    virtual edge_dir_type scatter_edges(icontext_type& context,\n                                        const vertex_type& vertex) const = 0; \n    virtual void scatter(icontext_type& context, const vertex_type& vertex, \n                         edge_type& edge) const = 0;\n}; // end of class bp_vertex_program\n\n\n\n\n////////////////////////////////////////////////////////////////////////////////\n// This class implements the \"symmetric\" version of dual decomposition described\n// in:\n// D. Sontag, A. Globerson, T. Jaakkola.\n// Introduction to Dual Decomposition for Inference.\n// Optimization for Machine Learning, editors S. Sra, S. Nowozin, and S. J.\n// Wright: MIT Press, 2011\n////////////////////////////////////////////////////////////////////////////////\n\nstruct dd_vertex_program_symmetric : public dd_vertex_program {\n    /**\n     * \\brief Since the MRF is undirected we will use all edges for gather and\n     * scatter\n     */\n    edge_dir_type gather_edges(icontext_type& context,\n                               const vertex_type& vertex) const \n    {   if(!opts.debug){\n        return graphlab::ALL_EDGES; }\n        else \n        return graphlab::NO_EDGES;\n    }; // end of gather_edges \n    \n    /**\n     * \\brief The gather function takes a vertex and an edge as inputs and outputs \n     a vector of numeric values. Vectors of numeric values will later be summed \n     over all edges incident in this vertex. So, if the vertex is a unary factor, \n     we can just return the vector of Lagrange multipliers stored in \"edge.messages\". \n     Otherwise (if vertex is a general factor), things are a little more tricky. \n     Suppose the factor is linked to K variables, with cardinalities C_1, ..., C_K. \n     Suppose this edge is with respect to the k-th variable. Then, we return a \n     vector of size C_1 + ... + C_K which is zero everywhere except in the \n     k-th slot, where the Lagrange multipliers in \"edge.messages\" will be copied \n     to. This way, when the \"gather sum\" takes place, and since all these slots \n     are disjoint, we will just get the Lagrange multipliers of all the variables.\n     It also gathers the best_configuration of neighbors for the factors and norm\n     of subgradient value for each edge.    \n     */\n     gather_type gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const \n     {         \n        if (opts.verbose > 1)\n            cout << \"gather begin\" << endl;\n        \n        const vertex_type other_vertex = get_other_vertex(edge, vertex);\n        const vertex_data& vdata = vertex.data();\n        edge_data& edata = edge.data();\n        \n        if (vdata.nvars == 1) \n        {\n            // Unary factor.\n            if (opts.verbose > 1)\n                cout << \"This unary factor has \" << vertex.num_in_edges() << \n                \" in edges and \" << vertex.num_out_edges() << \" out edges\" << endl;\n\n            if (opts.verbose>1)\n            {\n                cout << \"Gather on (\" << vertex.id() << \",\" << other_vertex.id() \n                                     << \") called from \" << vertex.id() << \"\\n\";\n                cout << \"vdata.neighbours = \" << vdata.neighbors << \"\\n\";\n                cout << \"Message: \" << edata.multiplier_messages << \"\\n---\\n\";\n            }\n            \n            gather_type gather_data(edata.multiplier_messages);\n            return gather_data;\n        } \n        else \n        {\n            // General factor.\n            factor_type messages;\n            messages.setZero(vdata.potentials.size());\n            int offset = 0;\n            int index_neighbor = -1;\n            for (int k = 0; k < vdata.nvars; ++k) \n            {\n                int vertex_id = vdata.neighbors[k];\n                if (vertex_id == other_vertex.id()) \n                {\n                    index_neighbor = k;\n                    break;\n                }\n                offset += vdata.cards[k];\n            }\n            CHECK_GE(index_neighbor, 0);\n            vector  neighbor_conf(vdata.nvars, 0);\n            neighbor_conf[index_neighbor] = other_vertex.data().best_configuration;\n            \n\n            for (int state = 0; state < vdata.cards[index_neighbor]; ++state) \n            {\n                messages[offset + state] = -edata.multiplier_messages[state];\n            }\n\n     \n            if (opts.verbose>1) \n            {\n                cout << \"Gather on (\" << vertex.id() << \",\" << other_vertex.id() << \") called from \" << vertex.id() << \"\\n\";\n                cout << \"vdata.neighbours = \" << vdata.neighbors << \"\\n\";\n                cout << \"estimated offset = \" << offset << \"\\n\";\n                cout << \"Message: \" << messages << \"\\n---\\n\";\n            }\n            vector states(vdata.nvars, -1);\n            get_configuration_states(vertex, vdata.best_configuration, &states);\n            int sq_norm_g = (states[index_neighbor] == other_vertex.data().best_configuration)?0:2;\n            gather_type gather_data(messages, neighbor_conf);\n            gather_data.sq_norm_g = sq_norm_g;\n            return gather_data;\n        }\n        if (opts.verbose > 2)\n            cout << \"gather end\" << endl;\n    }; // end of gather function\n    \n    /**\n     * \\brief The apply function takes a vertex and a vector of numeric values \n     (a total) as input. For unary vertices, this will be the sum of Lagrange \n     multipliers, and we just need to sum that to the vertex potential and compute \n     the argmax. For general factors, the vector of numeric values, as stated above, \n     will contain all the Lagrange multipliers of the neighboring variables. \n     So we need to loop through all possible factor configurations, get the \n     sequence of states of each configuration, fetch the Lagrange multipliers for \n     those states, and add them to the factor potential. Then we compute the argmax. \n     It also computes dual and primal contribution for finding dual and primal\n     objective values.\n     */\n    void apply(icontext_type& context, vertex_type& vertex, const gather_type& total) \n    {   if (!opts.debug){     \n        vertex_data& vdata = vertex.data();\n                \n        if (vdata.nvars == 1) \n        {\n            // Unary factor.\n            ASSERT_EQ(vdata.potentials.size(), total.messages.size());\n            \n            vec belief = vdata.potentials + total.messages;\n             // Find primal contrib\n            vdata.primal_contrib = vdata.potentials[vdata.best_configuration];\n             // Save the best configuration for this vertex and find dual contrib\n            vdata.dual_contrib = belief.maxCoeff(&vdata.best_configuration);\n            \n            if (opts.verbose > 1)\n            {\n                cout << \"Vertex: \" << vertex.id() << \"\\n\";\n                cout << \"Potential: \" << vdata.potentials << \"\\n\";\n                cout << \"incomming message: \" << total.messages << \"\\n\";\n                cout << \"belief: \" << belief << \"\\n\";\n                cout << \"dual contrib: \" << vdata.dual_contrib << \"\\n\";                \n                cout << \"vdata.best_configuration = \" << vdata.best_configuration << \"\\n---\\n\";\n            }\n        } \n        else \n        {\n            // General factor.\n            vec belief = vdata.potentials;\n            int num_configurations = vdata.potentials.size();\n            for (int index_configuration = 0;\n                 index_configuration < num_configurations;\n                 ++index_configuration) \n            {\n                vector states(vdata.nvars, -1);\n                // This could be made more efficient by defining an iterator over factor\n                // configurations.\n                get_configuration_states(vertex, index_configuration, &states);\n                int offset = 0;\n                for (int k = 0; k < vdata.nvars; ++k) \n                {\n                    belief[index_configuration] += total.messages[offset + states[k]];\n                    offset += vdata.cards[k];\n                }\n            }\n            // Save the best configuration for this factor and find dual contrib\n            vdata.dual_contrib = belief.maxCoeff(&vdata.best_configuration);\n            //Find primal contrib\n            int conf_index = get_configuration_index(vertex, total.neighbor_conf);\n            vdata.primal_contrib = vdata.potentials[conf_index];\n            //Find contribution fir sum of square of gradient\n            vdata.sum_sq_norm_g = total.sq_norm_g;\n            \n            if (opts.verbose > 1)\n            {\n                cout << \"Vertex: \" << vertex.id() << \"\\n\";\n                cout << \"Potential: \" << vdata.potentials << \"\\n\";\n                cout << \"incomming message: \" << total.messages << \"\\n\";\n                cout << \"belief: \" << belief << \"\\n\";\n                cout << \"dual contrib: \" << vdata.dual_contrib << \"\\n\";\n                cout << \"vdata.best_configuration = \" << vdata.best_configuration << \"\\n---\\n\";\n            }\n        }\n       }\n      else usleep(1);\n\n      if (opts.verbose > 1)\n            cout << \"end apply\" << endl;\n    }; // end of apply\n    \n    /**\n     * \\brief Since the MRF is undirected we will use all edges for gather and\n     * scatter\n     */\n    edge_dir_type scatter_edges(icontext_type& context, const vertex_type& vertex) const \n    { \n         return graphlab::ALL_EDGES; \n        // NOTENOTE: This assumes a sync engine. \n    }; // end of scatter edges\n    \n    /**\n     * \\brief The scatter function takes a vertex and an edge as input. \n     We just need to update the messages (Lagrange multipliers) by looking at the \n     saved argmaxes.\n     */\n    void scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const \n    {  \n        const vertex_type other_vertex = get_other_vertex(edge, vertex);\n        const vertex_type *unary_vertex;\n        const vertex_type *factor_vertex;\n     if (!opts.debug){       \n        if (vertex.data().nvars == 1) \n        {\n            // Unary factor.\n            unary_vertex = &vertex;\n            factor_vertex = &other_vertex;\n        } \n        else \n        {\n            // General factor.\n            unary_vertex = &other_vertex;\n            factor_vertex = &vertex;\n        }\n        const vertex_data& vdata = unary_vertex->data();\n        const vertex_data& other_vdata = factor_vertex->data();\n        edge_data& edata = edge.data();\n        \n        if (opts.verbose > 1)\n            cout << \"begin scatter\" << endl;\n                        \n        int iter_since_aggregate = (context.iteration()+2) - global_vars.iter_at_aggregate ;\n        double stepsize = update_stepsize(context, 1, global_vars.old_dual, global_vars.primal_best, \n                                      global_vars.sq_norm_g, global_vars.dual_inc_count, iter_since_aggregate);\n         \n        CHECK_GE(vdata.best_configuration, 0);                                                            \n        CHECK_LT(vdata.best_configuration, vdata.cards[0]);    \n        \n        // Negative subgradient\n       \n        edata.multiplier_messages[vdata.best_configuration] -= stepsize; \n        \n        vector states(other_vdata.nvars, -1);\n        get_configuration_states(*factor_vertex, other_vdata.best_configuration, &states);\n        int offset = 0;\n        int index_neighbor = -1;\n        for (int k = 0; k < other_vdata.nvars; ++k) {\n            int vertex_id = other_vdata.neighbors[k];\n            if (vertex_id == unary_vertex->id()) {\n                index_neighbor = k;\n                break;\n            }\n            offset += other_vdata.cards[k];\n        }\n        CHECK_GE(index_neighbor, 0);\n        CHECK_GE(states[index_neighbor], 0);\n        CHECK_LT(states[index_neighbor], other_vdata.cards[index_neighbor]);\n        CHECK_EQ(other_vdata.cards[index_neighbor], vdata.cards[0]);\n\n        // Negative subgradient\n        edata.multiplier_messages[states[index_neighbor]] += stepsize;\n        \n        //if (opts.verbose > 1)\n        if (opts.verbose>1) \n        {\n            cout << \"Scatter on (\" << unary_vertex->id() << \",\" << factor_vertex->id() << \") called from \" << vertex.id() << \"\\n\";\n            cout << \"unary best config = \" << vdata.best_configuration << \"\\n\"\n                 << \"factor best config = \" << states[index_neighbor] << \"\\n\"; \n            cout << \"Message: \" << edata.multiplier_messages << \"\\n---\\n\";\n        }\n       }\n        if (opts.verbose > 1)\n            cout << \"end scatter\" << endl;\n        \n        // Signalling the other vertex and yourself to start. \n        if ((context.iteration()+1) < opts.maxiter && global_vars.converged == false)\n        {\n            context.signal(vertex);\n            context.signal(other_vertex);\n        }\n\n    }; // end of scatter\n\n}; // end of class dd_vertex_program_symmetric\n\n\n\n\n////////////////////////////////////////////////////////////////////////////////\n// This class implements the \"projected\" version of dual decomposition described\n// in:\n// Komodakis, N., Paragios, N., and Tziritas, G. (2007).\n// \"MRF optimization via dual decomposition: Message-passing revisited\"\n// In Proc. of International Conference on Computer Vision.\n// \n// The formulation used is the one in Algorithm 1 of:\n//\n// André F. T. Martins, Mário A. T. Figueiredo, Pedro M. Q. Aguiar,\n// Noah A. Smith, and Eric P. Xing.\n// \"An Augmented Lagrangian Approach to Constrained MAP Inference.\"\n// International Conference on Machine Learning (ICML), 2011.\n////////////////////////////////////////////////////////////////////////////////\n\nstruct dd_vertex_program_projected : public dd_vertex_program {\n\n     graphlab::timer vertex_timer;\n    /**\n     * \\brief Since the MRF is undirected we will use all edges for gather and\n     * scatter\n     */\n    edge_dir_type gather_edges(icontext_type& context,\n                               const vertex_type& vertex) const { \n        if(!opts.debug){\n        return graphlab::ALL_EDGES; }\n        else \n        return graphlab::NO_EDGES;\n    }; // end of gather_edges \n    \n    /**\n     * \\brief The gather function takes a vertex and an edge as inputs and outputs \n     a vector of numeric values. Vectors of numeric values will later be summed \n     over all edges incident in this vertex. \n     If the vertex is a unary factor, compute the sum of all the local MAP variables,\n     which in the \"apply\" function will serve to compute the global MAP. \n     Otherwise (if vertex is a general factor), things are a little more tricky. \n     Suppose the factor is linked to K variables, with cardinalities C_1, ..., C_K. \n     Suppose this edge is with respect to the k-th variable. Then, we return a \n     vector of size C_1 + ... + C_K which is zero everywhere except in the \n     k-th slot, where the Lagrange multipliers in \"edge.messages\" will be copied \n     to. This way, when the \"gather sum\" takes place, and since all these slots \n     are disjoint, we will just get the Lagrange multipliers of all the variables.    \n     */\n    gather_type gather(icontext_type& context, const vertex_type& vertex, \n                       edge_type& edge) const {\n        const vertex_type other_vertex = get_other_vertex(edge, vertex);\n        const vertex_data& vdata = vertex.data();\n        edge_data& edata = edge.data();\n       \n        if (vdata.factor_type == VAR ) \n        {   // Unary factor.\n            if (opts.verbose > 1) {\n            cout << \"This unary factor has \" << vertex.num_in_edges() << \n            \" in edges and \" << vertex.num_out_edges() << \" out edges\" << endl; }\n            gather_type gatherdata(edata.local_messages);\n            return gatherdata; \n        } \n        else \n        {   \n            // General factor.\n            factor_type messages;\n            //messages.resize(vdata.potentials.size());\n            messages.setZero(vdata.potentials.size());\n            int offset = 0;\n            int index_neighbor = -1;\n            for (int k = 0; k < vdata.nvars; ++k) {\n                int vertex_id = vdata.neighbors[k];\n                if (vertex_id == other_vertex.id()) {\n                    index_neighbor = k;\n                    break;\n                }\n                offset += vdata.cards[k];\n            }\n    \n            CHECK_GE(index_neighbor, 0);\n            vector  neighbor_conf(vdata.nvars, 0);\n            neighbor_conf[index_neighbor] = other_vertex.data().best_configuration;\n            for (int state = 0; state < vdata.cards[index_neighbor]; ++state) {\n                messages[offset + state] = edata.multiplier_messages[state] + edata.potentials[state];               \n            }\n            gather_type gather_data(messages,neighbor_conf);\n            return gather_data;\n        }\n    }; // end of gather function\n    \n    \n    \n    /**\n     * \\brief The apply function takes a vertex and a vector of numeric values \n     (a total) as input. \n     For a unary vertex, \"total\" will be the sum of local MAP vectors, and we \n     just need to divide by the vertex degree and save the result as global MAP.\n     For higher-order factors, \"total\" will contain all the Lagrange multipliers \n     of the neighboring variables. So we need to loop through all possible factor \n     configurations, get the sequence of states of each configuration, fetch the \n     Lagrange multipliers for those states, and add them to the factor potential. \n     Then we compute the argmax and save result to local MAP for each variable \n     connected to the factor. \n     Note that since global MAP is computed from local MAP , locla MAP is needed \n     to be computed before global MAP. Hence in even iterations local MAP is \n     computed ( in scatter step) and in the subsequent iteration (which is hence \n     odd) global MAP (in apply step) and multiplier messages (in scatter)\n     are updated.\n     */\n    void apply(icontext_type& context, vertex_type& vertex, \n               const gather_type& total) {\n        vertex_data& vdata = vertex.data();\n       \n     if (!opts.debug){\n        if (vdata.factor_type == VAR ) {\n           if (context.iteration()%2 != 0) {   \n            vdata.beliefs = total.messages / static_cast(vdata.degree);\n            vdata.beliefs.maxCoeff(&vdata.best_configuration);\n            //Find primal contrib\n            vdata.primal_contrib = vdata.potentials[vdata.best_configuration];\n            }\n        } \n        else {\n          if(context.iteration()%2 == 0){\n            // General factor.\n            vec beliefs = vdata.potentials;\n            int num_configurations = vdata.potentials.size();\n            for (int index_configuration = 0;\n                 index_configuration < num_configurations;\n                 ++index_configuration) {\n                vector states(vdata.nvars, -1);\n                // This could be made more efficient by defining an iterator over factor\n                // configurations.\n                get_configuration_states(vertex, index_configuration, &states);\n                int offset = 0;\n                for (int k = 0; k < vdata.nvars; ++k) {\n                    beliefs[index_configuration] += total.messages[offset + states[k]];\n                    offset += vdata.cards[k];\n                }\n            }\n            // Save the best configuration for this factor and find dual contrib             \n            vdata.dual_contrib = beliefs.maxCoeff(&vdata.best_configuration);\n            //Find primal contrib\n            int conf_index = get_configuration_index(vertex, total.neighbor_conf);\n            vdata.primal_contrib = vdata.potentials[conf_index];       \n         }\n        }\n      }\n      else usleep(1);\n    }; // end of apply\n    \n  \n        /**\n     * \\brief Since the MRF is undirected we will use all edges for gather and\n     * scatter\n     */\n    edge_dir_type scatter_edges(icontext_type& context,\n                                const vertex_type& vertex) const { \n        return graphlab::ALL_EDGES; \n    }; // end of scatter edges\n    \n    /**\n     * \\brief The scatter function takes a vertex and an edge as input. \n     (1) If the vertex is a unary factor, we update the messages (Lagrange multipliers)\n     by subtracting the global MAP by the local MAP.\n     (2) If the vertex is a higher order factor, this function will take the best\n     configuration (obtained at the apply function) and save the local MAP \n     at the corresponding edge.\n     */\n    void scatter(icontext_type& context, const vertex_type& vertex, \n                 edge_type& edge) const {  \n       \n        const vertex_data& vdata = vertex.data();\n        edge_data& edata = edge.data();\n        const vertex_type other_vertex = get_other_vertex(edge, vertex);\n        if(!opts.debug){\n        if (vdata.factor_type == VAR ) {\n            if (context.iteration()%2 != 0) {\n\n            // Unary factor. Update the messages (Lagrange multipliers).      \n             double stepsize = update_stepsize(context, 1, global_vars.old_dual, global_vars.primal_best, \n                                              global_vars.sq_norm_g, global_vars.dual_inc_count, 0);\n            \n             edata.multiplier_messages += (vdata.beliefs - edata.local_messages) * stepsize;\n          }  \n        } \n       else \n       {   if (context.iteration()%2 == 0) {\n            //General factor. Update the local MAPs.\n            const vertex_type &unary_vertex = get_other_vertex(edge, vertex);\n            vector states(vdata.nvars, -1);\n            get_configuration_states(vertex, vdata.best_configuration, &states);\n            int offset = 0;\n            int index_neighbor = -1;\n            for (int k = 0; k < vdata.nvars; ++k) {\n                int vertex_id = vdata.neighbors[k];\n                if (vertex_id == unary_vertex.id()) {\n                    index_neighbor = k;\n                    break;\n                }\n                offset += vdata.cards[k];\n            }\n            CHECK_GE(index_neighbor, 0);\n            CHECK_GE(states[index_neighbor], 0);\n            CHECK_LT(states[index_neighbor], vdata.cards[index_neighbor]);\n            //CHECK_EQ(vdata.cards[index_neighbor], unary_vertex.data().cards[0]);\n            edata.local_messages.setZero();\n            edata.local_messages[states[index_neighbor]] += 1.0; \n          }\n        }\n      }\n        if ((context.iteration()+1) < opts.maxiter && global_vars.converged == false)\n        {\n            context.signal(vertex);\n            context.signal(other_vertex);\n        }\n    }; // end of scatter\n}; // end of class dd_vertex_program_projected\n\n\n\n\n///////////////////////////////////////////////////////////////////////////////////\n// This class implements the general Alternating Directions Method of Multipliers.\n//  \n//  The formulation used is the one in Algorithm 2 of:\n//\n// André F. T. Martins, Mário A. T. Figueiredo, Pedro M. Q. Aguiar,\n// Noah A. Smith, and Eric P. Xing.\n// \"Alternating Directions Dual Decomposition\"\n// Arxiv preprint arXiv:1212.6550, 2012.\n\n///////////////////////////////////////////////////////////////////////////////////\n \nstruct admm_vertex_program:public dd_vertex_program {\n  \n  typedef int Configuration;\n  \n  // Function to solve each quadratic programming sub problem \n  virtual void compute_beliefs(vertex_type& vertex,const gather_type& total,\n                 vec& variable_posteriors, vec& additional_posteriors) = 0;\n  virtual void SolveMAP(vertex_type& vertex,const gather_type& total,\n                 vec& variable_posteriors, vec& additional_posteriors, double& value) = 0;\n                                 \n                                 \n   /**\n     * \\brief Since the MRF is undirected we will use all edges for gather and\n     * scatter\n     */\n    edge_dir_type gather_edges(icontext_type& context,\n                               const vertex_type& vertex) const { \n        if(!opts.debug){\n        return graphlab::ALL_EDGES; }\n        else \n        return graphlab::NO_EDGES;\n    }; // end of gather_edges \n    \n    /**\n     * \\brief The gather function takes a vertex and an edge as inputs and outputs \n     a vector of numeric values. Vectors of numeric values will later be summed \n     over all edges incident in this vertex. \n     If the vertex is a unary factor, compute the sum of all the local MAP variables,\n     which in the \"apply\" function will serve to compute the global MAP. \n     Otherwise (if vertex is a general factor), things are a little more tricky. \n     Suppose the factor is linked to K variables, with cardinalities C_1, ..., C_K. \n     Suppose this edge is with respect to the k-th variable. Then, we return a \n     vector of size C_1 + ... + C_K which is zero everywhere except in the \n     k-th slot, where the Lagrange multipliers in \"edge.messages\" will be copied \n     to. This way, when the \"gather sum\" takes place, and since all these slots \n     are disjoint, we will just get the Lagrange multipliers of all the variables.    \n     */\n\n   gather_type gather(icontext_type& context, const vertex_type& vertex, \n                       edge_type& edge) const {\n        const vertex_type other_vertex = get_other_vertex(edge, vertex);\n        const vertex_data& vdata = vertex.data();\n        edge_data& edata = edge.data();\n        if (vdata.factor_type == VAR ) {  \n            // Unary factor.\n            if (opts.verbose > 1){\n            cout << \"This unary factor has \" << vertex.num_in_edges() << \n            \" in edges and \" << vertex.num_out_edges() << \" out edges\" << endl; \n            }\n            gather_type gatherdata(edata.local_messages);\n            return gatherdata; \n        } \n        else if(vdata.factor_type != BUDGET){  \n            // Dense factor.\n            factor_type messages, neighbor_distribution, multipliers;        \n            messages.setZero(vdata.potentials.size());\n            neighbor_distribution.setZero(vdata.potentials.size());\n            multipliers.setZero(vdata.potentials.size());\n            int offset = 0;\n            int index_neighbor = -1;\n            for (int k = 0; k < vdata.nvars; ++k) {\n                int vertex_id = vdata.neighbors[k];\n                if (vertex_id == other_vertex.id()) {\n                    index_neighbor = k;\n                    break;}\n                offset += vdata.cards[k];\n            }\n            CHECK_GE(index_neighbor, 0);\n            \n            vector  neighbor_conf(vdata.nvars, 0);\n            neighbor_conf[index_neighbor] = other_vertex.data().best_configuration;\n           \n\n            for (int state = 0; state < vdata.cards[index_neighbor]; ++state) {\n                messages[offset + state] = edata.multiplier_messages[state] + edata.potentials[state]; \n                multipliers[offset + state] = edata.multiplier_messages[state]; \n                neighbor_distribution[offset + state] = other_vertex.data().beliefs[state];\n            }\n            \n            gather_type gather_data(messages,neighbor_conf);\n            gather_data.neighbor_distribution = neighbor_distribution;\n            gather_data.multipliers = multipliers;\n            return gather_data;\n        }\n            \n        else if(vdata.factor_type == BUDGET) {\n            //Budget factor\n            factor_type messages, neighbor_distribution;        \n            messages.setZero(vdata.nvars);\n            neighbor_distribution.setZero(vdata.nvars);\n            int index_neighbor = -1;\n            for (int k = 0; k < vdata.nvars; ++k) {\n                int vertex_id = vdata.neighbors[k];\n                if (vertex_id == other_vertex.id()) {\n                    index_neighbor = k;\n                    break;}\n            }\n           messages[index_neighbor] =  edata.multiplier_messages[0] + edata.potentials[vdata.bound_states[index_neighbor]];\n\n           neighbor_distribution[index_neighbor] = other_vertex.data().beliefs[vdata.bound_states[index_neighbor]];\n           \n           gather_type gather_data(messages);\n           gather_data.neighbor_distribution = neighbor_distribution;\n           return gather_data;\n          \n        }\n    }; // end of gather function\n    \n    \n   /**\n     * \\brief The apply function takes a vertex and a vector of numeric values \n     (a total) as input. \n     For a unary vertex, \"total\" will be the sum of local MAP vectors, and we \n     just need to divide by the vertex degree and save the result as global MAP.\n     For higher-order factors, \"total\" will contain all the Lagrange multipliers \n     of the neighboring variables. So we need to loop through all possible factor \n     configurations, get the sequence of states of each configuration, fetch the \n     Lagrange multipliers for those states, and add them to the factor potential. \n     Then we compute solution of Quadratic subproblem and save result to local \n     MAP for each variable connected to the factor. \n     Note that since global MAP is computed from local MAP , locla MAP is needed \n     to be computed before global MAP. Hence in even iterations local MAP is \n     computed ( in scatter step) and in the subsequent iteration (which is hence \n     odd) global MAP (in apply step) and multiplier messages (in scatter)\n     are updated.\n     */  \n  \n  void apply(icontext_type& context, vertex_type& vertex, \n               const gather_type& total) {\n        vertex_data& vdata = vertex.data();\n     if (!opts.debug){\n        if (vdata.factor_type == VAR ) {   \n             if (context.iteration()%2 != 0) {\n            // Unary factor. \n            //Find dual residual contrib\n                vec dual_res_contrib;\n                dual_res_contrib.setZero(vdata.cards[0]);\n                for(int i=0; i(vdata.unary_degree[i]))\n                                                                                - vdata.beliefs[i] ;\n                   dual_res_contrib[i] = dual_res_contrib[i] * dual_res_contrib[i]; \n            // update global MAP     \n                   vdata.beliefs[i] = total.messages[i] / static_cast(vdata.unary_degree[i]);\n                }\n                vdata.dual_res_contrib = dual_res_contrib.sum(); \n            // Find best configuration\n                vdata.beliefs.maxCoeff(&vdata.best_configuration);\n            //Find relaxed primal contribution\n                vdata.primal_rel_contrib = vdata.potentials.dot(vdata.beliefs);\n            // Find primal contribution\n                vdata.primal_contrib = vdata.potentials[vdata.best_configuration];\n                \n            }\n        } \n        else{  \n            if(context.iteration()%2 == 0){\n            // Dense and Budget factors\n               vec additional_posteriors, variable_posteriors;\n               additional_posteriors.setZero(vdata.potentials.size());\n               variable_posteriors.setZero(vdata.potentials.size());\n               if(vdata.factor_type == BUDGET){\n                 additional_posteriors.setZero(vdata.nvars);\n                 variable_posteriors.setZero(vdata.nvars); \n                }\n            //Find dual contrib\n                SolveMAP(vertex, total, variable_posteriors, additional_posteriors, vdata.dual_contrib);\n            // Find relaxed primal contribution\n                if(vdata.factor_type == DENSE) {\n                  vdata.primal_rel_contrib = vdata.potentials.dot(vdata.factor_beliefs);\n                }\n                else if(vdata.factor_type == XOR){\n                  vdata.primal_rel_contrib = vdata.potentials.dot(vdata.beliefs);\n                }\n            //Find primal residual contribution\n                vec primal_res_contrib = vdata.beliefs - total.neighbor_distribution;\n                for(int i=0;i< vdata.beliefs.size(); i++){\n                   primal_res_contrib[i] = primal_res_contrib[i]*primal_res_contrib[i];\n                }\n                vdata.primal_res_contrib = primal_res_contrib.sum();\n            // Compute QP subproblem solution\n                compute_beliefs(vertex, total, vdata.beliefs, vdata.factor_beliefs);\n               //cout< global_vars.old_dual){\n             global_vars.dual_inc_count ++;\n         }\n     \n         if (total.primal> global_vars.primal_best) {\n            global_vars.primal_best = total.primal;\n         }\n\n         if(opts.verbose >0) {\n            cout<<\"iteration: \"<\n#include \n#include \n#include \n#include \n#include \n//#include \n\n//#include \"utils.h\"\n#include \"utils.hpp\"\n#include \"dd_grlab.hpp\"\n#include \"ad3_qp.hpp\"\n\n\n/////////////////////////////////////////////////////////////////////////\n// Load the UAI file. Each factor as a different vertex\nvoid loadUAIfile(graphlab::distributed_control& dc, graph_type& graph, string graph_file, int& nodes) \n{  \n    // Not sure why this is needed\n    dc.barrier();\n    // Open file\n    ifstream in(graph_file.c_str());\n     \n    //CHECK(in.good(),\"Could not open file: \"+graph_file);\n    CHECK(in.good());\n    \n    // Read type of network\n    string name; \n    \n    in >> name; \n    //CHECK(name.compare(\"MARKOV\")==0, \"Only Markov networks are supported. Are you sure this is a typeUAI energy file?\");\n    CHECK(name.compare(\"MARKOV\")==0);\n    \n    // Read size of graph\n    int nnodes, nfactors;\n    in >> nnodes;\n    nodes = nnodes;\n    //CHECK(nnodes>0, \"No. of nodes can't be negative. Are you sure this is a typeUAI energy file?\");\n    CHECK(nnodes>0);    \n    // Read node cardinalities\n    vector cardinalities(nnodes,0);\n    int cardinality_i, sum_of_cardinalities = 0;\n    for (int i = 0; i != nnodes; ++i) \n    {\n        in >> cardinality_i;\n        cardinalities[i] = cardinality_i;\n        sum_of_cardinalities += cardinality_i;\n       \n        //CHECK(in.good(), \"Could not finish reading cardinalities. Are you sure this is a typeUAI energy file?\");\n        CHECK(in.good());\n    }\n\n    int vid = 0;\n    if(opts.algorithm != 0){\n       for(int i = 0; i < nnodes; i++){                      //temporary .. put condition\n           vertex_data vdata;\n           vdata.factor_type = VAR; \n           vdata.nvars = 1;\n           vdata.cards.resize(1, cardinalities[i]);\n           vdata.potentials.setZero(cardinalities[i]);\n           vdata.beliefs.setConstant(cardinalities[i], 0.5);\n           graph.add_vertex(vid, vdata);\n           vid++;\n       }\n    }\n    // Read no. of factors\n    in >> nfactors;\n    \n    //factor_size.resize(nfactors); factor_id.resize(nfactors);\n    vector factor_size(nfactors,0); //vector factor_id(nfactors,0); \n    vector< vector > factor_memb; factor_memb.resize(nfactors);\n    int temp1, temp2;\n    \n    // Loop and read factor members\n    for (int i=0; i!=nfactors; ++i) \n    {\n        in >> temp1;\n        factor_size[i] = temp1; \n        \n        factor_memb[i].resize(temp1);\n        for (int j=0; j!=temp1; ++j) \n        {\n            in >> temp2;\n            factor_memb[i][j] = temp2;\n        }\n        \n        //CHECK(in.good(), \"Could not finish reading cardinalities. Are you sure this is a typeUAI energy file?\");\n        CHECK(in.good());\n    }\n    \n    if (opts.verbose > 1)\n        cout \n        << \"Finished Reading UAI-Preamble:\"\n        << \" #Nodes = \" << nnodes \n        << \", #Factors = \"<< nfactors \n        << \", Average Cardinality = \" << double(sum_of_cardinalities)/nfactors\n        << \"\\n\";\n        \n        \n    // Now read factor potentials\n    for (int i=0; i!=nfactors; ++i) \n    {\n        int cardprod; double potential_value; //, energy;\n        in >> cardprod;\n        \n        vertex_data vdata;        \n        vdata.nvars = factor_size[i];\n        if (vdata.nvars > 1) {\n          vdata.degree = vdata.nvars; // Factor degree.\n          vdata.factor_type = DENSE;\n        }\n        else {\n          vdata.degree = 1; // Factor degree.\n          vdata.factor_type = XOR;\n        }\n        vdata.cards.resize(factor_size[i]);\n        vdata.neighbors.resize(factor_size[i]);\n        \n        vector edata(factor_size[i]);\n        vector varid(factor_size[i]);\n        vector card(factor_size[i]);\n        \n        int cardprod2 = 1;\n        for (int j=0; j!=factor_size[i]; ++j) \n        {\n            vdata.cards[j] = cardinalities[factor_memb[i][j]];\n            vdata.neighbors[j] = factor_memb[i][j]; // afm (check if this was intended!)\n            cardprod2 *= vdata.cards[j];\n                      \n            // Also create edge structs here\n            //if (factor_size[i]>1)\n           // {\n                varid[j] = factor_memb[i][j];\n                card[j] = cardinalities[varid[j]];\n                edata[j].multiplier_messages.setZero(card[j]);\n                edata[j].local_messages.setZero(card[j]);\n                edata[j].potentials.setZero(card[j]);\n          //  }\n        }\n        \n        //CHECK_EQ(cardprod, cardprod2, \"Incorrectly sized factor\");\n        CHECK_EQ(cardprod, cardprod2);\n        \n        // Read factor potentials\n        vdata.potentials.resize(cardprod);\n        vdata.beliefs.resize(cardprod);\n        int x_offset = 0;\n        for(int x=0; x< vdata.nvars; x++){\n            for(int y=0; y> potential_value;\n            //energy = Potential2Energy(potential_value);\n            \n            vdata.potentials[k] = log10(potential_value) ;\n        }\n        \n        //CHECK(in.good(), \"Could not finish reading factor tables. Are you sure this is a typeUAI energy file?\");\n\n        CHECK(in.good());\n         \n\n         vdata.potentials.maxCoeff(&vdata.best_configuration);\n        // allocate factors evenly to different machines.\n        if (i%dc.numprocs() != dc.procid()) \n            continue;\n        \n        // If all is well, add vertex and edge\n        graph.add_vertex(vid ,vdata);\n\n        if (factor_size[i] > 1 || opts.algorithm > 0) // if not a unary, add edges to unaries\n        for (int j=0; j!=factor_size[i]; ++j) \n            graph.add_edge(vid,varid[j],edata[j]);\n        \n        //after adding everything increment vertex id\n        vid++; \n        \n        if (opts.verbose > 1)\n        {\n            cout << \"Machine #\" << dc.procid() << \", Vertex Id = \" << i\n            << \" with \" << vdata.nvars << \" variables.\"; \n            if (factor_size[i] > 1)\n            {\n                cout << \", Edges = \";\n                for (int j=0; j!=factor_size[i]; ++j)             \n                    cout << \", (\" << i << \",\" << varid[j] << \")\";\n            }\n            cout << \"\\n\";\n            cout << \"potential: \" << vdata.potentials << \"\\n\";\n        }\n        \n    } // End of reading factors     \n   \n    dc.barrier();\n} // end of loading UAI file\n\n/////////////////////////////////////////////////////////////////////////\n// Load the distributed UAI file\nbool line_parser(graph_type& graph, const std::string& filename, const std::string& textline) {\n    std::stringstream strm(textline);\n    graphlab::vertex_id_type vid;\n    vertex_data vdata;\n    vdata.dual_contrib = 0.0;\n    string type;\n    strm >> type;\n \n     if(type == \"v\") { \n      vdata.factor_type = VAR;\n      vdata.nvars = 1;\n      vdata.cards.resize(1);\n      strm>>vid;\n      strm >> vdata.cards[0];\n      vdata.potentials.resize(vdata.cards[0]);\n      //vdata.beliefs.setOnes(vdata.cards[0]);\n      //vdata.beliefs /= vdata.cards[0];\n      vdata.beliefs.setConstant(vdata.cards[0], 0.5);\n      vdata.unary_degree.resize(vdata.cards[0], 0);\n      //for(int i=0; i< vdata.cards[0]; i++){\n      // strm>>vdata.potentials[i];\n      //   vdata.potentials[i] = log10(vdata.potentials[i]);\n         \n      //   }\n     //    vdata.potentials.maxCoeff(&vdata.best_configuration);\n      graph.add_vertex(vid,vdata);\n    }\n   else if(type == \"d\" || type == \"u\") {\n     vdata.factor_type = (type==\"d\")?DENSE:XOR;\n     if(vdata.factor_type == DENSE)\n     strm>>vdata.nvars;\n     else \n     vdata.nvars = 1;\n     strm>>vid;\n     vdata.neighbors.resize(vdata.nvars);\n     vdata.cards.resize(vdata.nvars);\n     int cardprod = 1;\n     int cardsum =0;\n     for(int i=0; i>vdata.neighbors[i]; \n        }\n     for(int i=0; i>vdata.cards[i]; \n        cardprod *=vdata.cards[i];\n        cardsum +=vdata.cards[i];}\n     vdata.potentials.setZero(cardprod);\n     vdata.beliefs.setOnes(cardprod);\n     vdata.beliefs /=cardsum;\n     //vdata.beliefs.setConstant(cardprod, 0.5);\n     vdata.factor_beliefs.setOnes(cardprod);\n     vdata.factor_beliefs /= cardprod;\n     for(int i=0; i>vdata.potentials[i]; \n        vdata.potentials[i] = log10(vdata.potentials[i]);\n        }\n        vdata.potentials.maxCoeff(&vdata.best_configuration);\n     graph.add_vertex(vid, vdata);\n     edge_data edata;\n     for(int i=0; i>vdata.nvars;\n     strm>>vid;\n     vdata.neighbors.resize(vdata.nvars);\n     vdata.bound_states.resize(vdata.nvars);\n     vdata.cards.resize(vdata.nvars);\n     vdata.beliefs.setZero(vdata.nvars);\n     for(int i=0; i>vdata.neighbors[i]; }\n        for(int i=0; i>vdata.cards[i]; }\n     for(int i=0; i>vdata.bound_states[i]; }\n     strm>>vdata.budget;\n     graph.add_vertex(vid, vdata);\n     edge_data edata;\n     for(int i=0; i degree;\n   vec potentials;\n\n    void load(graphlab::iarchive& arc) {\n        arc >>degree>>potentials;\n    }\n    void save(graphlab::oarchive& arc) const {\n        arc <,\npublic graphlab::IS_POD_TYPE {\n\nedge_dir_type gather_edges(icontext_type& context,\n                               const vertex_type& vertex) const \n    {  return graphlab::ALL_EDGES; \n     };\n\ngather_potentials gather(icontext_type& context, const vertex_type& vertex, edge_type& edge) const \n     { \n       const vertex_data& vdata = vertex.data();\n       const vertex_type& other_vertex = (edge.source().id() == vertex.id())?edge.target():edge.source();\n       vector  degree;\n       vec potentials;\n       \n       if(vdata.factor_type == VAR)  {\n          potentials.resize(vdata.cards[0]);\n          potentials.setZero();\n          switch(other_vertex.data().factor_type){\n          case XOR : potentials = other_vertex.data().potentials;\n       \n          case DENSE : degree.resize(vdata.potentials.size(),1);\n       \n                       break;\n          \n          case BUDGET : degree.resize(vdata.potentials.size(), 0);\n                        int index_neighbor = -1;\n                        for(int i=0; i< other_vertex.data().nvars; i++){\n                            if(other_vertex.data().neighbors[i] == vertex.id()){\n                               index_neighbor = i;\n                               break;\n                              }\n                        }\n       \n                       degree[other_vertex.data().bound_states[index_neighbor]] = 1;\n          }\n       }\n       else {\n         degree.resize(1);\n         potentials.resize(1);\n       \n       }   \n       gather_potentials gather_data;\n       gather_data.degree  = degree;\n       gather_data.potentials = potentials;\n       return gather_data; \n     };\n       \nvoid apply(icontext_type& context, vertex_type& vertex, const gather_potentials& total)\n     { vertex_data& vdata =  vertex.data();\n       if(vdata.factor_type == VAR) {\n          vdata.unary_degree = total.degree;\n          vdata.potentials = total.potentials;\n         } \n    \n      };\n\nedge_dir_type scatter_edges(icontext_type& context,\n                               const vertex_type& vertex) const \n    { \n     return graphlab::ALL_EDGES; };\n\nvoid scatter(icontext_type& context, const vertex_type& vertex, edge_type& edge) const \n     { const vertex_data& vdata = vertex.data();\n       const vertex_type& other_vertex = (edge.source().id() == vertex.id())?edge.target():edge.source();\n       if(vdata.factor_type == VAR) {  \n          if(other_vertex.data().factor_type != BUDGET) {\n             for(int i =0; i< vdata.potentials.size(); i++){\n               edge.data().potentials[i] = vdata.potentials[i]/vdata.unary_degree[i];\n               }\n           }\n          else if(other_vertex.data().factor_type == BUDGET) {\n               int index_neighbor = -1;\n               edge.data().potentials.setZero();\n               for(int i=0; i< other_vertex.data().nvars; i++){\n                  if(other_vertex.data().neighbors[i] == vertex.id()){\n                     index_neighbor = i;\n                      break;}\n                }\n               int state_index = other_vertex.data().bound_states[index_neighbor];\n               edge.data().potentials[state_index]  = vdata.potentials[state_index]/vdata.unary_degree[state_index];\n              }\n            }  \n           //cout<<\"complete scatter\"< engine_type;\n    // Instantiate the engine object  \n    engine_type engine(dc, graph, opts.exec_type, clopts);\n    engine.signal_all();\n    graphlab::timer timer;\n    // Attach an aggregator to compute primal/dual objective, with periodic interval specified in cmdline argument.\n    engine.add_vertex_aggregator(\"pd_obj\",sum, print_obj); \n    if(!opts.debug){\n     engine.aggregate_periodic(\"pd_obj\",opts.agg_time); }\n    //The main command. Run graphlab\n    engine.start();  \n    engine.aggregate_now(\"pd_obj\");\n    const double runtime = timer.current_time();    \n    dc.cout() \n    << \"----------------------------------------------------------\" << std::endl\n    << \"Final Runtime (seconds):   \" << runtime \n    << std::endl\n    << \"Updates executed: \" << engine.num_updates() << std::endl\n    << \"Update Rate (updates/second): \" \n    << engine.num_updates() / runtime << std::endl;\n    }\n    /* end of run_dd_symmetric */\n    \n    void run_dd_projected(graphlab::distributed_control& dc, graph_type& graph, \n                   std::string exec_type, graphlab::command_line_options clopts){\n   \n    // Instantiate the engine object\n    graph.transform_vertices(compute_degree);\n    graph.transform_edges(dist_unary_potentials);\n     // Define the engine.    \n    typedef graphlab::omni_engine engine_type;\n    engine_type engine(dc, graph, opts.exec_type, clopts);\n    engine.signal_all();\n    graphlab::timer timer;    \n    // Attach an aggregator to compute primal/dual objective, with periodic interval specified in cmdline argument.\n    engine.add_vertex_aggregator(\"pd_obj\",sum, print_obj); \n    if (!opts.debug){\n     engine.aggregate_periodic(\"pd_obj\", opts.agg_time);}\n    // The main command. Run graphlab\n    engine.start();  \n    engine.aggregate_now(\"pd_obj\");\n    const double runtime = timer.current_time();    \n    dc.cout() \n    << \"----------------------------------------------------------\" << std::endl\n    << \"Final Runtime (seconds):   \" << runtime \n    << std::endl\n    << \"Updates executed: \" << engine.num_updates() << std::endl\n    << \"Update Rate (updates/second): \" \n    << engine.num_updates() / runtime << std::endl;\n    }\n     /* end of run_dd_projected */\n    \n    void run_ad3(graphlab::distributed_control& dc, graph_type& graph, \n              std::string exec_type, graphlab::command_line_options clopts){\n    // Define the engine.\n    typedef  graphlab::omni_engine transform_engine;\n    transform_engine distribute_potentials(dc, graph, opts.exec_type, clopts);\n    distribute_potentials.signal_all();\n    distribute_potentials.start();\n     typedef graphlab::omni_engine engine_type;\n    // Instantiate the engine object\n    engine_type engine(dc, graph, opts.exec_type, clopts);\n    engine.signal_all();\n    graphlab::timer timer;\n    // Attach an aggregator to compute primal/dual objective, with periodic interval specified in cmdline argument.\n    engine.add_vertex_aggregator(\"pd_obj\",sum, print_obj);\n    if(!opts.debug){ \n     engine.aggregate_periodic(\"pd_obj\",opts.agg_time); }\n    // The main command. Run graphlab\n    engine.start();  \n    engine.aggregate_now(\"pd_obj\");\n    const double runtime = timer.current_time();    \n    dc.cout() \n    << \"----------------------------------------------------------\" << std::endl\n    << \"Final Runtime (seconds):   \" << runtime \n    << std::endl\n    << \"Updates executed: \" << engine.num_updates() << std::endl\n    << \"Update Rate (updates/second): \" \n    << engine.num_updates() / runtime << std::endl;\n    \n    }\n    /* end of run_admm */\n\n void run_bethe_admm(graphlab::distributed_control& dc, graph_type& graph, \n              std::string exec_type, graphlab::command_line_options clopts){\n    // Define the engine.\n    typedef  graphlab::omni_engine transform_engine;\n    transform_engine distribute_potentials(dc, graph, opts.exec_type, clopts);\n    distribute_potentials.signal_all();\n    distribute_potentials.start();\n    typedef graphlab::omni_engine engine_type;\n    // Instantiate the engine object\n    engine_type engine(dc, graph, opts.exec_type, clopts);\n    engine.signal_all();\n    graphlab::timer timer;\n    // Attach an aggregator to compute primal/dual objective, with periodic interval specified in cmdline argument.\n    engine.add_vertex_aggregator(\"pd_obj\",sum, print_obj);\n    if(!opts.debug){ \n     engine.aggregate_periodic(\"pd_obj\",opts.agg_time); }\n    // The main command. Run graphlab\n    engine.start();  \n    engine.aggregate_now(\"pd_obj\");\n    const double runtime = timer.current_time();    \n    dc.cout() \n    << \"----------------------------------------------------------\" << std::endl\n    << \"Final Runtime (seconds):   \" << runtime \n    << std::endl\n    << \"Updates executed: \" << engine.num_updates() << std::endl\n    << \"Update Rate (updates/second): \" \n    << engine.num_updates() / runtime << std::endl;\n    }\n    /* end of run_bethe_admm */\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/dd_opts.hpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n *\n * \\brief This application performs MAP inference on Markov Nets \n * provided in standard UAI file format via Dual-Decomposition. \n *\n *\n *  \\authors Dhruv Batra, André Martins, Aroma Mahendru\n */\n\n\n#ifndef __DD_OPTS_HPP__\n#define __DD_OPTS_HPP__\n\n#include \n#define VAR 0\n#define XOR 1\n#define DENSE 2\n#define BUDGET 3\n/////////////////////////////////////////////////////////////////////////\n// Option Struct\nstruct Options \n{\n    // graphlab options\n    std::string exec_type;\n    \n    // input output dirs\n    std::string graph_file;\n    std::string output_dir;\n    std::string history_file;\n    std::string file_format;\n    std::string output_file;\n\n    int verbose;\n    int algorithm;  \n    int maxiter;\n\n    double dualimprovthres;\n    double pdgapthres;\n    double alpha;\n    double step_size; \n    double agg_time; \n    \n    bool debug;\n    \n    // Default values\n    Options(): \n    exec_type(\"sync\"),\n    output_dir(\"./\"),\n    history_file(\"\\0\"),\n    file_format(\"uai\"),\n    output_file(\"output\"),\n    verbose(0),\n    algorithm(0),\n    maxiter(10000),\n    dualimprovthres(1e-12),\n    pdgapthres(1e-1),\n    alpha(1),\n    step_size(1.0),\n    agg_time(1e-4),\n    debug(false)\n    {}\n};\n\nextern Options opts;\n\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/factors/CMakeLists.txt",
    "content": "project(GraphicalModels)\n# add_library(factors STATIC\n#   binary_factor.cpp\n#   discrete_variable.cpp\n#   unary_factor.cpp)"
  },
  {
    "path": "toolkits/graphical_models/deprecated/factors/binary_factor.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \n\nstd::ostream& operator<<(std::ostream& out, \n                         const graphlab::binary_factor& fact) {\n  out << \"Binary Factor(v_\" << fact.var1() << \" in {1...\"\n      << fact.arity1() << \"}, \" \n      << \", v_ \" << fact.var2() << \" in {1...\" \n      << fact.arity2() << \"})\" << std::endl;\n  for(uint16_t i = 0; i < fact.arity1(); ++i) {\n    for(uint16_t j = 0; j < fact.arity2(); ++j) {\n      out << fact.logP(i,j) << \" \";\n    }\n    out << std::endl;\n  }\n  return out;\n} // end of operator<<\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/factors/binary_factor.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n */\n\n\n#ifndef BINARY_FACTOR_HPP\n#define BINARY_FACTOR_HPP\n\n\n/**\n * This file contains the definitions of some of the basic factor\n * types needed for loopy belief propagation.  This is demo code and\n * is intentionally kept as simple as possible.\n *\n *  \\author Joseph Gonzalez\n */\n\n\n\n// Including Standard Libraries\n#include \n#include \n#include \n#include \n#include \n\n// Random number generation\n#include \n\n#include \n\n// Include the macro for the for each operation\n#include \n\n\n\n/**\n * A binary factor is a table over a pair of variables and is\n * assocaited with each edge in a pairwise markov random field.  All\n * data is represented in log form.\n */\nclass binary_factor {\n  \npublic:\n    \n  binary_factor(uint32_t var1 = 0,\n                uint16_t arity1 = 0,\n                uint32_t var2 = 0,\n                uint16_t arity2 = 0) :\n    _var1(var1), _arity1(arity1), _var2(var2), _arity2(arity2),\n    _data(arity1 * arity2) { }\n\n  binary_factor(const binary_factor& other) :\n    _var1(other._var1), _arity1(other._arity1),\n    _var2(other._var2), _arity2(other._arity2),\n    _data(other._data) { }\n\n  binary_factor& operator=(const binary_factor& other) {\n    _var1 = other._var1;\n    _arity1 = other._arity1;\n    _var2 = other._var2;\n    _arity2 = other._arity2;\n    _data = other._data;\n    return *this;\n  }\n\n  void resize(uint16_t arity1, uint16_t arity2) {\n    _arity1 = arity1;\n    _arity2 = arity2;\n    _data.resize(_arity1 * _arity2);\n  }\n  \n  uint32_t& var1() { return _var1;  } \n  const uint32_t& var1() const { return _var1; }\n  uint32_t& var2() { return _var2;  } \n  const uint32_t& var2() const { return _var2; } \n\n  const uint16_t& arity1() const { return _arity1; }\n  const uint16_t& arity2() const { return _arity2; } \n\n  /** Get the value of the factor.  In var1 == var2 the variables\n      are ignored. */\n  double& logP(uint32_t x1, uint16_t asg1,\n               uint32_t x2, uint16_t asg2) {\n    // If the factor is not symmetric then we may have to match the\n    // arguments\n    if( _var1 != _var2 ) {\n      assert((x1 == var1() && x2 == var2()) ||\n             (x2 == var1() && x1 == var2()));\n      if(x1 == var2() && x2 == var1()) std::swap(asg1, asg2);\n    }\n    assert( asg1 < arity1() );\n    assert( asg2 < arity2() );\n    // return value\n    return _data[asg1 + asg2 * arity1()];\n  } // end of logP for a binary factor\n\n\n  const double& logP(uint32_t x1, uint16_t asg1,\n                     uint32_t x2, uint16_t asg2) const {\n    // If the factor is not symmetric then we may have to match the\n    // arguments\n    if( _var1 != _var2 ) {\n      assert((x1 == var1() && x2 == var2()) ||\n             (x2 == var1() && x1 == var2()));\n      if(x1 == var2() && x2 == var1()) std::swap(asg1, asg2);\n    }\n    ASSERT_LT( asg1 , arity1() );\n    ASSERT_LT( asg2 , arity2() );\n    // return value\n    return _data[asg1 + asg2 * arity1()];\n  } // end of logP for a binary factor\n\n  \n  double& logP(uint16_t asg1, uint16_t asg2) {\n    ASSERT_LT( asg1 , arity1() );\n    ASSERT_LT( asg2 , arity2() );\n    return _data[asg1 + asg2 * arity1()];\n  } // end of logP for a binary factor\n\n\n  const double& logP(uint16_t asg1, uint16_t asg2) const {\n    assert( asg1 < arity1() );\n    assert( asg2 < arity2() );\n    return _data[asg1 + asg2 * arity1()];\n  } // end of logP for a binary factor\n\n    /** ensure that sum_x this(x) = 1 */\n  inline void normalize() {\n    assert(arity1() > 0);\n    assert(arity2() > 0);\n    // Compute the max value\n    double max_value = logP(0,0);\n    for(uint16_t asg1 = 0; asg1 < arity1(); ++asg1) \n      for(uint16_t asg2 = 0; asg2 < arity2(); ++asg2)  \n        max_value = std::max(max_value, \n                             logP(asg1, asg2));\n    assert( !std::isinf(max_value) );\n    assert( !std::isnan(max_value) );\n    // scale and compute normalizing constant\n    double Z = 0.0;\n    for(uint16_t asg1 = 0; asg1 < arity1(); ++asg1) \n      for(uint16_t asg2 = 0; asg2 < arity2(); ++asg2)  \n        Z += std::exp(logP(asg1, asg2) -= max_value);\n    assert( !std::isinf(Z) );\n    assert( !std::isnan(Z) );\n    assert( Z > 0.0);\n    double logZ = std::log(Z);\n    // Normalize\n    for(uint16_t asg1 = 0; asg1 < arity1(); ++asg1) \n      for(uint16_t asg2 = 0; asg2 < arity2(); ++asg2)  \n        logP(asg1, asg2) -= logZ;\n  } // End of normalize\n\n  \n  \n  void set_as_agreement(double lambda) {\n    for(uint16_t i = 0; i < arity1(); ++i) { \n      for(uint16_t j = 0; j < arity2(); ++j) { \n        if( i != j) logP(i,j) = -lambda;\n        else logP(i,j) = 0;\n      }\n    }\n  } // end of set_as_agreement\n  \n  void set_as_laplace(double lambda) {\n    for(uint16_t i = 0; i < arity1(); ++i) { \n      for(uint16_t j = 0; j < arity2(); ++j) { \n        logP(i,j) = -std::abs(double(i) - double(j)) * lambda;\n      }\n    }\n  } // end of set_as_laplace\n\n\n\n    /**\n     * Compute the Mooji Kappen Message derivative. \n     */\n  double mk_derivative() const {\n    double max_value = -std::numeric_limits::max();\n    for(uint16_t a = 0; a < arity1(); ++a) {\n      for(uint16_t b = 0; b < arity2(); ++b) {\n        for(uint16_t x = 0; x < arity1(); ++x) {\n          for(uint16_t y = 0; y < arity2(); ++y) {\n            if(a != x && b != y) {\n              double value =\n                (logP(a,b) + logP(x,y) - (logP(x,b) + logP(a,y)))/4.0;\n              value = std::tanh(value);\n              max_value = std::max(max_value, value);\n            }\n          }\n        }\n      }\n    }\n    return max_value;\n  }\n\n  //! Compute the Ihler dynamic range\n  double ihler_dynamic_range() const {\n    double min_value = *std::min_element(_data.begin(), _data.end());\n    double max_value = *std::max_element(_data.begin(), _data.end());\n    return std::exp((max_value - min_value)/2);\n  }\n\n    \n\n  //! Print the factor description\n  void printP(std::ostream& out) const {\n    out << \"Binary Factor(v_\" << var1() << \" in {1...\"\n        << arity1() << \"}, \" \n        << \", v_ \" << var2() << \" in {1...\" \n        << arity2() << \"})\" << std::endl;\n    for(uint16_t i = 0; i < arity1(); ++i) {\n      for(uint16_t j = 0; j < arity2(); ++j) {\n        out << std::exp(logP(i,j)) << \" \";\n      }\n      out << std::endl;\n    }\n  }\n\n  //! Save the factor to a file\n  void save(graphlab::oarchive &oarc) const {\n    oarc << _var1 << _arity1 \n         << _var2 << _arity2\n         << _data;\n  }\n\n  //! Load the factor from a file\n  void load(graphlab::iarchive &iarc) {\n    iarc >> _var1 >> _arity1 \n         >> _var2 >> _arity2\n         >> _data;\n  }\n\n\n    \nprivate:\n  uint32_t _var1;\n  uint16_t _arity1;\n  uint32_t _var2;\n  uint16_t _arity2;\n  std::vector _data;\n    \n}; // end of class binary_factor\n\n\n\ninline std::ostream& operator<<(std::ostream& out, \n                                const binary_factor& fact) {\n  out << \"Binary Factor(v_\" << fact.var1() << \" in {1...\"\n      << fact.arity1() << \"}, \" \n      << \", v_ \" << fact.var2() << \" in {1...\" \n      << fact.arity2() << \"})\" << std::endl;\n  for(uint16_t i = 0; i < fact.arity1(); ++i) {\n    for(uint16_t j = 0; j < fact.arity2(); ++j) {\n      out << fact.logP(i,j) << \" \";\n    }\n    out << std::endl;\n  }\n  return out;\n} // end of operator<<\n\n\n#include \n#endif\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/factors/discrete_variable.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \n\nstd::ostream& operator<<(std::ostream& out, \n                         const graphlab::discrete_variable& var) {\n  // return out << \"v_\" << var.id()\n  //            << \" in {0:\" << var.size()-1 << \"}\";\n  return out << var.id();\n}\n\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/factors/factor_includes.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \"discrete_variable.hpp\"\n#include \"unary_factor.hpp\"\n#include \"binary_factor.hpp\"\n#include \"table_factor.hpp\"\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/factors/factor_test.cxx",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n#include \n#include \n#include \n\n\n#include \n\n\n#include \n\n#include \"factors/factor_includes.hpp\"\n\n\n\nusing namespace graphlab;\n\n\nclass factor_tests : public CxxTest::TestSuite {\npublic:\n\n  void test_variables() {\n    std::cout << \"Test Variables\" << std::endl;\n    discrete_variable v1(1, 3);\n    std::cout << v1 << std::endl;\n\n    discrete_variable v2(2, 4);\n    std::cout << v2 << std::endl;\n  \n    discrete_variable v3(3, 2);\n    std::cout << v3 << std::endl;\n\n    TS_ASSERT_EQUALS( v1, v1 );\n    TS_ASSERT_DIFFERS( v1, v2 );\n    TS_ASSERT_DIFFERS( v1, v3 );\n    TS_ASSERT_LESS_THAN(v1, v2);\n    TS_ASSERT_LESS_THAN(v2, v3);       \n  }\n\n  void test_domain() {\n    std::cout << \"Test domain\" << std::endl;\n    const size_t max_dim = 5;\n    typedef discrete_domain domain_type;\n    \n    discrete_variable v1(1, 3);\n    discrete_variable v2(2, 4);\n    discrete_variable v3(3, 2);\n    discrete_variable v4(4, 2);\n    domain_type dom0;\n    domain_type dom1(v1);\n    domain_type dom2(v1,v3);\n    domain_type dom3(v1,v2,v3);\n    std::vector vec;\n    vec.push_back(v3); vec.push_back(v1); vec.push_back(v2);\n    domain_type dom4(vec);\n    TS_ASSERT_EQUALS(dom3, dom4);\n    domain_type dom5 = dom0 + dom2;\n    TS_ASSERT_EQUALS(dom5, dom2);\n    domain_type dom6 = dom2 + dom2;\n    TS_ASSERT_EQUALS(dom6, dom2);\n    domain_type dom7(v3);\n    TS_ASSERT_EQUALS(dom2 - dom1, dom7);\n    TS_ASSERT_EQUALS(dom2 - dom7, dom1);\n    TS_ASSERT_EQUALS(dom3 - v2, dom2);\n    TS_ASSERT_EQUALS(dom2 + dom1 + dom3, dom3);\n\n    TS_ASSERT_EQUALS(dom2.intersect(dom3), dom2);\n    domain_type dom9(v2, v3);\n    TS_ASSERT_EQUALS(dom2.intersect(dom3), dom2);\n    TS_ASSERT_EQUALS(dom2.intersect(dom7), dom7);\n    TS_ASSERT_EQUALS(dom0.intersect(dom3), dom0);\n    TS_ASSERT_EQUALS(dom2.intersect(dom9), dom7);\n  }\n\n  void test_assignment() {\n    std::cout << \"Test domain: \" << std::endl;\n    const size_t max_dim = 5;\n    typedef discrete_domain domain_type;\n    typedef discrete_assignment assignment_type;\n    \n    discrete_variable v1(1,11), v2(2,10), v3(3,8), v4(4,3), v5(5,2);\n    domain_type dom(v1,v2,v3);\n    domain_type sub_dom(v1,v3);\n    size_t i = 0;\n    size_t val[3];\n    for(size_t j = 0; j < 3; ++j) val[j] = 0;\n    for(assignment_type asg = dom.begin();\n        asg < dom.end(); ++asg, ++i) {\n      TS_ASSERT_EQUALS(asg.linear_index(), i);\n      TS_ASSERT_EQUALS(asg.asg_at(0), asg.asg(1));\n      TS_ASSERT_EQUALS(asg.asg_at(1), asg.asg(2));\n      TS_ASSERT_EQUALS(asg.asg_at(2), asg.asg(3));\n      TS_ASSERT_EQUALS(val[0], asg.asg(1));\n      TS_ASSERT_EQUALS(val[1], asg.asg(2));\n      TS_ASSERT_EQUALS(val[2], asg.asg(3));\n      for(size_t j = 0; j < 3; ++j) {\n        if(val[j] < dom.var(j).size() - 1) {\n          val[j]++;\n          break;\n        } else val[j] = 0;\n      }\n      assignment_type rev_asg(dom, asg.linear_index());\n      TS_ASSERT_EQUALS(rev_asg.asg(1), asg.asg(1));\n      TS_ASSERT_EQUALS(rev_asg.asg(2), asg.asg(2));\n      TS_ASSERT_EQUALS(rev_asg.asg(3), asg.asg(3));\n      TS_ASSERT_EQUALS(rev_asg.linear_index(), asg.linear_index());\n      assignment_type other_asg = asg.restrict(sub_dom);\n      TS_ASSERT_EQUALS(other_asg.asg(1), asg.asg(1));\n      TS_ASSERT_EQUALS(other_asg.asg(3), asg.asg(3));\n      assignment_type asg3(dom);\n      asg3.set_asg(1, asg.asg(1));\n      asg3.set_asg(2, asg.asg(2));\n      asg3.set_asg(3, asg.asg(3));\n      TS_ASSERT_EQUALS(asg3.asg(1), asg.asg(1));\n      TS_ASSERT_EQUALS(asg3.asg(2), asg.asg(2));\n      TS_ASSERT_EQUALS(asg3.asg(3), asg.asg(3));\n      domain_type dom2(v4, v5);\n      for(assignment_type asg2 = dom2.begin();  asg2 < dom2.end(); ++asg2) {\n        assignment_type joint = asg & asg2;\n        TS_ASSERT_EQUALS(joint.asg(4), asg2.asg(4));\n        TS_ASSERT_EQUALS(joint.asg(5), asg2.asg(5));\n      }\n\n      assignment_type asg4(sub_dom);\n      asg4.update(asg);\n      TS_ASSERT_EQUALS(asg4, asg.restrict(sub_dom));\n      \n    }\n\n    \n    \n  }\n\n  \n  \n  void test_table_factor() {\n    std::cout << \"Testing factors\" << std::endl;\n    const size_t max_dim = 5;\n    typedef table_factor factor_type;\n    typedef factor_type::domain_type domain_type;\n    typedef factor_type::assignment_type assignment_type;\n\n    \n    discrete_variable v1(1,3), v2(2,2), v3(3,2), v4(4,3), v5(5,2);\n    domain_type dom(v1,v2,v3);\n\n    // Create a factor over the domain\n    factor_type factor(dom);\n    factor.uniform();\n    double sum = 0;\n    double sum2 = 0;\n    for(assignment_type asg = dom.begin(); asg < dom.end(); ++asg) {\n      sum += std::exp(factor.logP(asg));\n      sum2 += std::exp(factor.logP(asg.linear_index()));\n    }\n    TS_ASSERT_EQUALS(sum, sum2);\n    TS_ASSERT_LESS_THAN(std::abs(sum - 1), 1E-10);\n   \n    for(assignment_type asg = dom.begin(); asg < dom.end(); ++asg) {\n      double val = double(rand()) / RAND_MAX;\n      factor.logP(asg) = val;\n      TS_ASSERT_EQUALS(factor.logP(asg), \n                       factor.logP(asg.linear_index()));\n    }\n\n    factor /= factor;\n\n    for(assignment_type asg = dom.begin(); asg < dom.end(); ++asg) {\n      TS_ASSERT_EQUALS(factor.logP(asg), 0);\n    }\n\n    factor *= factor;\n\n    for(assignment_type asg = dom.begin(); asg < dom.end(); ++asg) {\n      TS_ASSERT_EQUALS(factor.logP(asg), 0);\n    }\n\n    for(assignment_type asg = dom.begin(); asg < dom.end(); ++asg) {\n      double val = double(rand()) / RAND_MAX;\n      factor.logP(asg) = val;\n      TS_ASSERT_EQUALS(factor.logP(asg), \n                       factor.logP(asg.linear_index()));\n    }\n\n    factor_type factor2 = factor * factor;\n    factor_type factor3 = factor2 / factor;\n\n    for(assignment_type asg = dom.begin(); asg < dom.end(); ++asg) {\n      TS_ASSERT_EQUALS(factor2.logP(asg)/2, factor.logP(asg));\n      TS_ASSERT_EQUALS(factor3.logP(asg), factor.logP(asg));\n      TS_ASSERT_EQUALS(factor2.logP(asg)/2, factor.logP(asg.linear_index()));\n      TS_ASSERT_EQUALS(factor3.logP(asg), factor.logP(asg.linear_index()));\n    }\n    \n    factor.normalize();\n\n\n    factor_type factor4( domain_type(v1, v3) );\n    factor4.marginalize(factor);\n    \n\n    factor_type factor5( domain_type(v1, v3) );\n    domain_type d2(v2);\n    factor5.zero();\n\n    for(assignment_type asg = d2.begin(); asg < d2.end(); ++asg) {\n      factor_type tmp(domain_type(v1,v3));\n      tmp.condition(factor, asg);\n      for(assignment_type asg2 = tmp.args().begin(); \n          asg2 < tmp.args().end(); ++asg2) {\n        factor5.logP(asg2) += std::exp(tmp.logP(asg2));\n      }\n    } \n\n    for(size_t i = 0; i < factor5.size(); ++i) {\n      factor5.logP(i) = std::log(factor5.logP(i));\n      TS_ASSERT_EQUALS(factor5.logP(i), factor4.logP(i));\n    }\n\n    factor.logP(0) +=2;\n    factor.logP(2) +=3;\n    factor.logP(4) +=4;\n    factor.logP(6) +=1;\n\n    factor_type counts(factor.args());\n    factor.normalize();\n    size_t num_samples = 10000000;\n    for(size_t i = 0; i < num_samples; ++i) {\n      assignment_type asg = factor.sample();\n      ++counts.logP(asg);\n    }\n    sum = 0;\n    for(size_t i = 0; i < counts.size(); ++i)\n      sum += counts.logP(i);\n    for(size_t i = 0; i < counts.size(); ++i) {\n      counts.logP(i) = std::log( counts.logP(i) / sum );\n    }\n    std::cout << \"True Factor: \" << factor << std::endl;\n    std::cout << \"Sampled: \" << counts << std::endl;\n    for(size_t i = 0; i < counts.size(); ++i) {\n      TS_ASSERT_LESS_THAN(std::abs(factor.logP(i) -\n                                   counts.logP(i)) , 1E-2);\n    }\n\n  }\n\n  void test_unary_binary_factors() {\n\n    unary_factor a(0, 5);\n    unary_factor b(1, 7);\n  \n    binary_factor bin(0,5, 1,7);\n    \n    // initialize factors\n    for(size_t i = 0; i < a.arity(); ++i) a.logP(i) = i;\n    //  a.normalize();\n    for(size_t i = 0; i < a.arity(); ++i)\n      for(size_t j = 0; j < b.arity(); ++j)\n        bin.logP(0, i, 1, j) = i + j  + 1;\n  \n\n    // try some math\n    b.convolve(bin,a);\n    \n    std::cout << a << std::endl;\n    std::cout << b << std::endl;\n    std::cout << bin << std::endl;\n  }\n  \n  void test_bench_marginalize() {\n    // create variables\n    std::vector v(5);\n    for (size_t i = 0;i < 5; ++i) {\n      v[i].id() = i;\n      v[i].size() = 3;\n    }\n    // create base domain\n    discrete_domain<5> alldomain(v);\n    table_factor<5> joint(alldomain);\n    joint.uniform();\n    // create test marginalization domains\n    std::vector > testfactors;\n    for (size_t i = 0;i < 5; ++i) {\n      for (size_t j = i + 1; j < 5; ++j) {\n        testfactors.push_back(table_factor<5>(discrete_domain<5>(v[i],v[j])));\n      }\n    }\n    \n    timer ti;\n    ti.start();\n    const size_t iterations = 10000;\n    for (size_t i = 0;i < iterations; ++i) {\n      for (size_t j = 0; j < testfactors.size(); ++j) {\n        testfactors[j].marginalize(joint);\n      }\n    }\n    std::cout << iterations * testfactors.size() \n              << \" marginalize ops of 3^5 --> 3^2 done in \" \n                 << ti.current_time() << \" seconds\" << std::endl;\n  }\n  \n  \n  void test_bench_condition() {\n    // create variables\n    std::vector v(5);\n    for (size_t i = 0;i < 5; ++i) {\n      v[i].id() = i;\n      v[i].size() = 3;\n    }\n    // create base domain\n    discrete_domain<5> alldomain(v);\n    table_factor<5> joint(alldomain);\n    joint.uniform();\n    // create test marginalization assignments\n    std::vector > testasg;\n    std::vector > testfactors;\n    for (size_t i = 0;i < 5; ++i) {\n      for (size_t j = i + 1; j < 5; ++j) {\n        testasg.push_back(discrete_assignment<5>(v[i], j % 3,v[j], i % 3));\n        testfactors.push_back(table_factor<5>(alldomain - testasg.rbegin()->args()));\n      }\n    }\n    \n    timer ti;\n    ti.start();\n    const size_t iterations = 1000;\n    for (size_t i = 0; i < iterations; ++i) {\n      for (size_t j = 0; j < testasg.size(); ++j) {\n        testfactors[j].condition(joint, testasg[j]);\n      }\n    }\n    std::cout << iterations * testfactors.size() \n              << \" condition ops of 3^5 --> 3^3 done in \" \n                 << ti.current_time() << \" seconds\" << std::endl;\n  }\n};\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/factors/unary_factor.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \n\nstd::ostream& operator<<(std::ostream& out, \n                         const graphlab::unary_factor& fact) {\n  out << \"Unary Factor(\" << fact.arity()  << \")\"\n      << std::endl;\n  for(size_t i = 0; i < fact.arity(); ++i) {\n    out << fact.logP(i) << \" \";\n  }\n  out << std::endl;\n  return out;\n} // end of operator<<\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/factors/unary_factor.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef UNARY_FACTOR_HPP\n#define UNARY_FACTOR_HPP\n\n\n/**\n * This file contains the definitions of some of the basic factor\n * types needed for loopy belief propagation.  This is demo code and\n * is intentionally kept as simple as possible.\n *\n *  \\author Joseph Gonzalez\n */\n\n\n// Including Standard Libraries\n#include \n#include \n#include \n#include \n#include \n\n// Random number generation\n#include \n#include \n\n// Basic unary factor\n#include \"binary_factor.hpp\"\n\n\n// Include the macro for the for each operation\n#include \n\n\n\n  \n/**\n * A unary factor is a table over a single variable and is associated\n * with edge variable in the pairwise markov random field.  Unary\n * factors are also used to represent messages. All data is\n * represented in log form.\n */\nclass unary_factor {  \npublic:\n\n  /**\n   * Construct a unary factor with a given variable and arrity.\n   */\n  unary_factor(uint32_t var = 0, uint16_t arity = 0) :\n    _var(var), _data(arity) {}\n\n  // /**\n  //  * Copy constructor for a unary factor\n  //  */\n  // unary_factor(const unary_factor& other) :\n  //   _var(other._var), _data(other._data) { }\n\n  // unary_factor& operator=(const unary_factor& other) {\n  //   _var = other._var;\n  //   _data = other._data;\n  //   return *this;\n  // }\n\n  /**\n   * Increase the size of the factor\n   */\n  inline void resize(uint16_t arity) {\n    _data.resize(arity);\n  }\n        \n  uint32_t& var() { return _var;  }\n  const uint32_t& var() const { return _var; }\n  uint16_t arity() const { return (uint16_t)_data.size(); }\n  size_t size() const { return _data.size(); }\n  bool empty() const { return _data.empty(); }\n\n\n  inline double& logP(size_t asg) {\n    assert(asg < arity()); return _data[asg];\n  } // end of logP for a unary factor\n\n  inline const double& logP(size_t asg) const {\n    assert(asg < arity()); return _data[asg];\n  } // end of logP for a unary factor \n\n    /** zero out the factor */\n    // inline void zero() {\n    //   for(size_t asg = 0; asg < arity(); ++asg) logP(asg) = 0;\n    // }\n\n  inline void uniform(double value = 0) {\n    for(uint16_t asg = 0; asg < arity(); ++asg) logP(asg) = value;\n  }\n\n\n  /** ensure that sum_x this(x) = 1 */\n  inline void normalize() {\n    assert(arity() > 0);\n    // Compute the max value\n    double max_value = logP(0);\n    for(uint16_t asg = 0; asg < arity(); ++asg) \n      max_value = std::max(max_value, logP(asg));\n    assert( !std::isinf(max_value) );\n    assert( !std::isnan(max_value) );\n    // scale and compute normalizing constant\n    double Z = 0.0;\n    for(uint16_t asg = 0; asg < arity(); ++asg) \n      Z += std::exp(logP(asg) -= max_value);\n    assert( !std::isinf(Z) );\n    assert( !std::isnan(Z) );\n    assert( Z > 0.0);\n    double logZ = std::log(Z);\n    // Normalize\n    for(uint16_t asg = 0; asg < arity(); ++asg)\n      logP(asg) -= logZ;\n  } // End of normalize\n\n  inline unary_factor& operator*=(const unary_factor& other) {\n    ASSERT_EQ(arity(), other.arity());\n    for(uint16_t asg = 0; asg < arity(); ++asg)\n      logP(asg) += other.logP(asg);\n    return *this;\n  }\n\n    /** this(x) += other(x); */\n  inline unary_factor& operator+=(const unary_factor& other) {\n    ASSERT_EQ(arity(), other.arity());\n    for(uint16_t asg = 0; asg < arity(); ++asg)\n      logP(asg) = log(exp(logP(asg)) + exp(other.logP(asg)));\n    return *this;\n  } // end plus\n\n\n    /** this(x) /= other(x); */\n  inline unary_factor& operator/=(const unary_factor& other) {\n    ASSERT_EQ(arity(), other.arity());\n    for(uint16_t asg = 0; asg < arity(); ++asg)\n      logP(asg) -= other.logP(asg);\n    return *this;\n  } // end of divide\n  \n    /** this(x) = sum_y fact(x,y) * other(y) */\n  inline void convolve(const binary_factor& bin_fact,\n                       const unary_factor& other) {\n    // Compute C(x) = Sum_y A(x,y) B(y)\n    for(uint16_t x = 0; x < arity(); ++x) {\n      double sum = 0.0;\n      for(uint16_t y = 0; y < other.arity(); ++y) {          \n        sum += std::exp(bin_fact.logP(var(), x, other.var(), y) +\n                        other.logP(y));\n      }\n      assert( !(sum < 0.0) );\n      // Gaurd against zeros\n      if(sum == 0) sum = std::numeric_limits::min();\n      logP(x) = std::log(sum);\n    }\n  }\n  \n  /** this(x) = this(x) * fact(x, asg) */\n  inline void condition(const binary_factor& bin_fact,\n                        uint16_t asg) {\n    uint32_t other_var =\n      var() != bin_fact.var1()? bin_fact.var1() : bin_fact.var2();\n    for(uint16_t x = 0; x < arity(); ++x) \n      logP(x) += bin_fact.logP(var(), x, other_var, asg);    \n  } // end of condition\n  \n  \n    /** This = other * damping + this * (1-damping) */\n  inline void damp(const unary_factor& other, double damping) {\n    assert(arity() == other.arity());\n    if(damping == 0) return;\n    assert(damping >= 0.0);\n    assert(damping < 1.0);\n    for(uint16_t asg = 0; asg < arity(); ++asg) \n      logP(asg) = std::log(damping * std::exp(other.logP(asg)) + \n                           (1.0 - damping) * std::exp(logP(asg)));  \n  } // end of damp\n  \n  \n    /** Compute the residual between two unary factors */\n  inline double residual(const unary_factor& other) const {  \n    assert(arity() == other.arity());\n    double residual = 0;\n    for(uint16_t asg = 0; asg < arity(); ++asg) \n      residual += std::abs(std::exp(logP(asg)) - \n                           std::exp(other.logP(asg)));\n    return residual / arity();\n  } // end of residual\n  \n  \n    /** get the max assignment*/\n  inline size_t max_asg() const {  \n    size_t max_asg = 0;\n    double max_value = logP(0);\n    for(uint16_t asg = 0; asg < arity(); ++asg) { \n      if(logP(asg) > max_value) {\n        max_value = logP(asg);\n        max_asg = asg;\n      }\n    }\n    return max_asg;\n  } // end of max asg\n  \n    /** Get the expected assignment */\n  inline double expectation() const {  \n    double sum = 0;\n    double s2 = 0;\n    for(uint16_t asg = 0; asg < arity(); ++asg)  {\n      sum += asg * std::exp(logP(asg));       \n      s2 += std::exp(logP(asg));       \n    }\n    return sum / s2;;\n  } // end of expectation\n  \n    /** Draw a random sample from the factor */\n  inline size_t sample() const {  \n    // Using the cdf method to generate a random sample\n    assert(arity() > 0);\n    // double t = static_cast(rand()) / RAND_MAX;\n    double t = graphlab::random::rand01();\n    assert( t >= 0);\n    assert(t < 1);\n    double sum = 0.0;\n    for(uint16_t asg = 0; asg < arity(); ++asg) {\n      sum += exp(logP(asg));\n      if(t <= sum) return asg;\n      assert(sum < 1);\n    }\n    // We were unable to draw a sample;\n    assert(false);\n  } // end of sample\n\n  void save(graphlab::oarchive &oarc) const {\n    oarc << _var << _data;\n  }\n  void load(graphlab::iarchive &iarc) {\n    iarc >> _var >> _data;\n  }\nprivate:\n  uint32_t _var;\n  std::vector _data;\n};  // End of unary factor\n\n\n\ninline std::ostream& operator<<(std::ostream& out, \n                                const unary_factor& fact) {\n  out << \"Unary Factor(\" << fact.arity()  << \")\"\n      << std::endl;\n  for(size_t i = 0; i < fact.arity(); ++i) {\n    out << fact.logP(i) << \" \";\n  }\n  out << std::endl;\n  return out;\n} // end of operator<<\n\n\n#include \n#endif\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/CMakeLists.txt",
    "content": "project(GraphLab)\n\n\n\n\n# add_library(pgibbs STATIC\n#     image.cpp\n#     chromatic_sampler.cpp \n#     util.cpp \n#     factorized_model.cpp\n#     mrf.cpp\n#     junction_tree.cpp\n#     jt_splash_sampler.cpp\n#     pgibbs_tls.cpp\n#     global_variables.cpp\n#     )\n\n# add_library(pgibbs_pic STATIC\n#     image.cpp\n#     chromatic_sampler.cpp \n#     util.cpp \n#     factorized_model.cpp\n#     mrf.cpp\n#     junction_tree.cpp\n#     jt_splash_sampler.cpp\n#     pgibbs_tls.cpp\n#     global_variables.cpp\n#     )\n\n# target_link_libraries(pgibbs_pic \n#   graphlab_pic)\n\n# target_link_libraries(pgibbs \n#   graphlab)\n\n# get_property(pgibbs_flags TARGET pgibbs PROPERTY COMPILE_FLAGS)\n# set_target_properties(pgibbs_pic\n#   PROPERTIES COMPILE_FLAGS \"${pgibbs_flags} -fPIC\")\n\n\n\n\n# add_graphlab_executable(make_denoise_alchemy \n#   make_denoise_alchemy.cpp \n#   )\n# target_link_libraries(make_denoise_alchemy pgibbs)\n\n\n\n\n\n# add_graphlab_executable(sampler sampler.cpp)\n# target_link_libraries(sampler pgibbs)\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/README",
    "content": "The pgibbs folder contains the implementations of the parallel Gibbs\nsamplers described in:\n\n  Parallel Gibbs Sampling: From Colored Fields to Think Junction Trees\n   by Joseph Gonzalez, Yucheng Low, Arthur Gretton, and Carlos Guestrin\n\nBecause the sampler.cpp is currently taylored to the Alchemy\nintermediate factor format we recommend using the precompiled matlab\nbinaries in the matlab folder.  The matlab interface uses a very\nsimple discrete factorized model representation.\n\nThis project has to main files the sampler.cpp which defines the main\nsampler binary and the make_denoise_alchemy.cpp which creates a\nsynthetic denoising problem in Alchemy format:\n\n  sampler.cpp: This is the main point of entry for the Gibbs sampler\n    binary.  To learn more about how to use this run ./sampler --help\n    The sampler binary operates on alchemy factor-graph files with the\n    format defined in Section\n\n  make_denoise_alchemy.cpp: This file is used to create a simple\n    synthetic image denoising problem.  To learn more about this file\n    run ./make_denoise_alchemy.\n\nAlchemy Factor Graph Format:\n\nThe Alchemy factor graph form was developed in collaboration with the\nAlchemy team (http://alchemy.cs.washington.edu/) at the University of\nWashington.  The format describes a discrete factorized model as a\ntext file with the following form:\n\n   variables:\n   \\t\n    ...\n   factors:\n    /  / ... /  //   ...\n\nThe remaining files are described below:\n\n  factorized_model.hpp/cpp: This file is used to read and parse\n    Alchemy files into GraphLab data structures to represent the\n    factorized model.  Note this file also contains the key\n    definitions of many of the various types (vertex_id_t, variable_t,\n    ...) used in the Gibbs sampler. \n\n  mrf.hpp/cpp: This file defines the key data-structures needed to\n    assemble a GraphLab graph representing a Markov Random Field. In\n    addition this file defines routines to construct an MRF from a\n    factorized model. \n\n  global_variable.hpp/cpp: This file defines the few global variables\n    which are used to access the factors from within the threads\n    (without needing to make copies) as well as the GraphLab GLShared\n    objects which are GraphLab managed global variables (formerly part\n    of the Shared Data Table). \n  \n\n  chromatic_sampler.hpp/cpp: This file defines the key parts of the\n    chromatic sampler algorithm including the update function as well\n    as a helper routine used to launch the chromatic sampler for a\n    series of timed experiments.\n  \n  junction_tree.hpp/cpp: This file defines helper routine to construct\n    junction_trees from a subset of the variables in the MRF.  In\n    particular this file contains the extend() routine which is\n    described in the original paper.  Finally, this file contains\n    routines to build a GraphLab graph representation of a junction\n    tree to run parallel calibration and sampling on the junction\n    tree.\n\n  jt_splash_sampler.hpp/cpp: This rather long file contains the bulk\n    of the junction tree Splash sampler.  The key parts of which are\n    the jt_splash_sampler object and its dependent jt_worker object.\n    The jt_splash_sampler maintains ncpus jt_workers.  Each jt_worker\n    has access to the shared MRF and asynchronously constructs\n    Splashes. In addition the splash_settings object describes the\n    settings for the jt_splash_sampler.\n\n  pgibbs_tls.hpp/cpp: This file defines thread local storage objects\n    used to reduce memory allocations when running the various\n    sampling algorithms.  \n\n\nFor questions or comments please contact me at jegonzal@cs.cmu.edu.\n\nThanks,\nJoey\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/chromatic_sampler.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n\n#include \"util.hpp\"\n#include \"chromatic_sampler.hpp\"\n#include \"run_statistics.hpp\"\n#include \"global_variables.hpp\"\n\n// Include the macro for the foreach operation\n#include \n\n\n\nvoid gibbs_update::operator()(base::icontext_type& context) {\n  mrf_vertex_data& vdata = context.vertex_data();\n  //TODO: switch to use tls\n  factor_t belief(vdata.variable);\n  belief.uniform();\n  foreach(const factor_id_t factor_id, vdata.factor_ids) {\n    //const factor_t& factor(SHARED_FACTORS.get()[factor_id]);\n    const factor_t& factor((*SHARED_FACTORS_PTR)[factor_id]);\n    // build the conditional\n    assignment_t conditional_asg = factor.args() - vdata.variable;\n    for(size_t i = 0; i < conditional_asg.num_vars(); ++i) {\n      const mrf_vertex_data& other_vdata = \n\tcontext.const_neighbor_vertex_data(conditional_asg.args().var(i).id());\n      assert(conditional_asg.args().var(i) == other_vdata.variable);\n      conditional_asg.set_asg_at(i, other_vdata.asg);\n    }\n    belief.times_condition(factor, conditional_asg);\n  }\n  belief.normalize();\n  size_t new_asg = belief.sample().asg_at(0);\n  vdata.nchanges += (new_asg != vdata.asg);\n  vdata.asg = new_asg;\n  vdata.belief += belief;\n  vdata.nsamples++;  \n}\n\n\n\n// bool nsamples_terminator(const mrf_gl::ishared_data* shared_data) {\n//   const size_t nsamples = n_samples.get_val();\n//   bool terminate = nsamples >= MAX_NSAMPLES.get();\n//   if(terminate) {\n//     //     std::cout << \"Termination condition reached\" << std::endl;\n//   }\n//   return terminate;\n// }\n\n\nvoid run_chromatic_sampler(graphlab::core& core,\n                           const std::string& chromatic_results_fn,\n                           const std::vector& runtimes,\n                           const bool draw_images) {\n  // Initialize scheduler\n  core.set_scheduler_type(\"chromatic\");\n  core.set_scope_type(\"null\");\n\n  const size_t ncpus = core.get_options().get_ncpus();\n\n  // Use fixed update function\n  gibbs_update ufun;\n  core.schedule_all( ufun );\n  \n  double total_runtime = 0;\n  double actual_total_runtime = 0;\n  foreach(const double experiment_runtime, runtimes) {\n    total_runtime += experiment_runtime;\n    // get the experiment id\n    size_t experiment_id = file_line_count(chromatic_results_fn);\n    std::cout << \"Running chromatic sampler experiment \" << experiment_id\n              << \" for \" << experiment_runtime << \" seconds.\" << std::endl;\n    // set the termination time\n    core.engine().set_timeout(experiment_runtime);\n    // Run the engine\n    graphlab::timer timer;\n    timer.start();\n    core.start();\n    double actual_experiment_runtime = timer.current_time();\n    actual_total_runtime += actual_experiment_runtime;\n    /// ==================================================================\n    // Compute final statistics of the mode\n    run_statistics stats(core.graph());\n    stats.print();\n    // Save the beliefs\n    save_beliefs(core.graph(),\n                 make_filename(\"chromatic_blfs_\", \".tsv\", experiment_id));\n    // // Save the current assignments\n    save_asg(core.graph(),\n             make_filename(\"chromatic_asg_\", \".asg\", experiment_id));\n    // Save the experiment\n    std::ofstream fout(chromatic_results_fn.c_str(), std::ios::app);\n    fout.precision(16);\n    fout << experiment_id << '\\t'\n         << total_runtime << '\\t'\n         << actual_total_runtime << '\\t'\n         << ncpus << '\\t'\n         << stats.nsamples << '\\t'\n         << stats.nchanges << '\\t'\n         << stats.loglik << '\\t'\n         << stats.min_samples << '\\t'\n         << stats.max_samples << std::endl;\n    fout.close();\n    // Plot images if desired\n    if(draw_images) draw_mrf(experiment_id, \"chromatic\", core.graph());\n  }\n} // end run_chromatic sampler\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/chromatic_sampler.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef CHROMATIC_SAMPLER_HPP\n#define CHROMATIC_SAMPLER_HPP\n\n#include \n\n#include \"mrf.hpp\"\n\nclass gibbs_update :\n  public graphlab::iupdate_functor {\n  typedef graphlab::iupdate_functor base;\n  void operator()(base::icontext_type& context);\n}; // end of class gibbs update\n\n/** Get the update counts for a vertex */\ninline size_t get_nsamples(const mrf_vertex_data& vdata) { \n  return vdata.nsamples; \n}\n\n\n//! Run the chromatic sampler for a fixed ammount of time\nvoid run_chromatic_sampler(graphlab::core& core, \n                           const std::string& chromatic_results_fn,\n                           const std::vector& runtime,\n                           const bool draw_images);\n\n\n\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/factorized_model.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \"factorized_model.hpp\"\n\n\n\n\n#include \n\nvoid factorized_model::reserve(const size_t num_factors) {\n  _factors.reserve(num_factors);\n}\n\nfactor_t& factorized_model::add_factor(const factor_t& factor) {\n  _factors.push_back(factor);\n  // // normalize the factor\n  // _factors.back().normalize();\n  factor_id_t factor_id = _factors.size() - 1;\n  for(size_t i = 0; i < factor.num_vars(); ++i) {\n    variable_t var = factor.args().var(i); \n    _variables.insert(var);\n    // add factor to reverse map\n    _var_to_factor[var].insert(factor_id);\n  }\n  return _factors.back();\n}\n\nfactor_t& factorized_model::add_factor(const domain_t& vars) {\n  _factors.push_back(factor_t());\n  factor_t& factor = _factors.back();\n  factor.set_args(vars);\n  // // normalize the factor\n  // _factors.back().normalize();\n  factor_id_t factor_id = _factors.size() - 1;\n  for(size_t i = 0; i < factor.num_vars(); ++i) {\n    variable_t var = factor.args().var(i); \n    _variables.insert(var);\n    // add factor to reverse map\n    _var_to_factor[var].insert(factor_id);\n  }\n  return _factors.back();\n}\n\n\nconst std::set& \nfactorized_model::factor_ids(const variable_t& var) const {\n  typedef std::map >::const_iterator iterator;\n  iterator iter = _var_to_factor.find(var);\n  ASSERT_TRUE(iter != _var_to_factor.end());\n  return iter->second;\n}\n\n\n\n\nvoid factorized_model::load_alchemy(const std::string& filename) {\n  // Open an input file stream\n  std::ifstream fin(filename.c_str());\n  ASSERT_TRUE(fin.good());\n  std::string line;\n  size_t line_number = 0;\n  // Read the first line which should be \"variable:\"\n  const bool success = getline(fin,line,line_number++);\n  ASSERT_TRUE(success);\n  line = trim(line);\n  {\n    const std::string variables_str(\"variables:\");\n    ASSERT_EQ(line, variables_str);\n  }\n  // Read all the variables and create a map from the variable name\n  // (string) to the variable* prl variable pointer.\n  typedef std::map var_map_type;\n  typedef var_map_type::iterator var_map_iter_type;\n  var_map_type var_map;\n  size_t unique_var_id = 0;\n  while(fin.good() &&\n        getline(fin, line, line_number++) &&\n        trim(line) != \"factors:\") {\n    // Separate into name and size\n    line = trim(line);\n    ASSERT_GT(line.length(), 0);\n    size_t namelen = line.find_last_of('\\t');\n    size_t varsize = 2;\n    // if their is a '\\t' character then the variable size follows it\n    if(namelen != std::string::npos) {\n      std::stringstream istrm(trim(line.substr(namelen)));\n      istrm >> varsize;\n    }\n    // Get the variable name\n    std::string var_name = trim(line.substr(0, namelen));\n    ASSERT_GT(varsize, 0);\n    // Create a new finite variable in the universe\n    variable_t variable(unique_var_id++, varsize);\n    // Store the variable in the local variable map\n    var_map[var_name] = variable;\n    _var_name.push_back(var_name);\n    ASSERT_EQ(_var_name.size(), unique_var_id);\n  }\n\n  // Starting to read factors\n  {\n    const std::string factors_string(\"factors:\");\n    ASSERT_EQ(trim(line), factors_string);\n  }\n\n  while(fin.good() && getline(fin, line, line_number++)) {\n    /// if the line is empty skip it\n    if(trim(line).length() == 0) continue;\n    //      std::cout << \"Line: \" << line << std::endl;\n    // the factor being read may contain the same variable multiple\n    // times to account for that, we first read a temporary factors,\n    // making every variable unique artificially, and then convert\n    // it to the factor we actually need\n\n    // Process the arguments\n    size_t end_of_variables = line.find(\"//\")-1;\n    std::vector args;\n    std::set args_set;\n\n    // Read in all the variables in the factor and store them\n    for(size_t i = 0; i < end_of_variables;\n        i = line.find_first_of('/', i) + 1) {\n      // Read the next variable as a string\n      std::string variable_name =\n        trim(line.substr(i, line.find_first_of('/',i) - i));\n      //        std::cout << \"Variable Name: \" << variable_name << std::endl;\n      // Look up the variable in the variable map\n      var_map_iter_type iter = var_map.find(variable_name);\n      ASSERT_TRUE(iter != var_map.end());\n      variable_t var = iter->second;\n      // This argument must be unique\n      if(args_set.count(var) > 0) {\n        std::cout << \"Line Number: \" << line_number << std::endl;\n        ASSERT_EQ(args_set.count(var), 0);\n      }\n\n      args_set.insert(var);\n      // Save the arguments read from the file\n      args.push_back(var);\n    } // end of first pass through variable\n\n      // Construct the arguments (which will remap the domain)\n    domain_t domain(args);\n    //      std::cout << \"domain: \" << domain << std::endl;\n    // Build the factor\n    factor_t factor(domain);\n      \n    // Now for the tricky part we need an assignment in the original\n    // order\n    domain_t orig_domain;\n    for(size_t i = 0; i < args.size(); ++i) {\n      orig_domain += variable_t(i, args[i].size());\n    }\n\n\n    // Advance to the correct location in the line\n    std::istringstream tbl_values;\n    size_t weightpos = line.find(\"///\");\n    if (weightpos == std::string::npos) {\n      tbl_values.str(line.substr(line.find(\"//\") + 2));\n    } else {\n      size_t startpos = line.find(\"//\") + 2;\n      tbl_values.str(line.substr(startpos, weightpos - startpos));\n    }\n      \n    // Read in the weights\n    for(assignment_t orig_asg = orig_domain.begin();\n        orig_asg < orig_domain.end(); ++orig_asg) {\n      assignment_t asg(domain);\n      // Translate the original assignment into the sorted factor assignment\n      for(size_t i = 0; i < domain.num_vars(); ++i) {\n        size_t variable_id = args[i].id();\n        asg.set_asg(variable_id, orig_asg.asg(i));\n      }\n      // Read then next value\n      ASSERT_TRUE(tbl_values.good());\n      double value = 0;\n      tbl_values >> value;\n      // Values are stored in log form      \n      factor.logP(asg.linear_index()) = value;                \n    }\n    // Save the factor to the factor graph\n    add_factor(factor);                \n  } // End of outer while loop over factors should be end of file\n\n  ASSERT_FALSE(fin.good());\n  fin.close();\n} // end of load alchemy\n\n\n\n  //! Save the factor to a file\nvoid factorized_model::save(graphlab::oarchive &arc) const {\n  arc << _variables\n      << _factors\n      << _var_to_factor\n      << _var_name;\n}\n\n\n//! Load the factor from a file\nvoid factorized_model::load(graphlab::iarchive &arc) {\n  arc >> _variables\n      >> _factors\n      >> _var_to_factor\n      >> _var_name;\n}  \n\n\n//! save the alchemy file\nvoid factorized_model::save_alchemy(const std::string& filename) const {\n  std::ofstream fout(filename.c_str());\n  ASSERT_TRUE(fout.good());\n  fout << \"variables:\" << std::endl;\n  foreach(variable_t var, _variables) {\n    fout << var.id() << '\\t' << var.size() << \"\\n\";\n  }\n  fout << \"factors:\" << std::endl;\n  foreach(const factor_t& factor, _factors) {\n    domain_t domain = factor.args();\n    for(size_t i = 0; i < domain.num_vars(); ++i) {\n      fout << domain.var(i).id();\n      if(i + 1 < domain.num_vars()) fout << \" / \";\n    }\n    fout << \" // \";\n    for(size_t i = 0; i < factor.size(); ++i) {\n      fout << factor.logP(i);\n      if(i + 1 < factor.size()) fout << ' ';\n    }\n    fout << '\\n';\n  }\n  fout.flush();\n  fout.close();\n}\n\n\n#include \n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/factorized_model.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef PGIBBS_FACTORIZED_MODEL_HPP\n#define PGIBBS_FACTORIZED_MODEL_HPP\n\n/**\n *\n * Represents a factorized model in alchemy format\n *\n *  \\author Joseph Gonzalez\n */\n\n// INCLUDES ===================================================================>\n\n\n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n\n\n\n#include \n\n\n\n\n// The maximum number of dimensions in a factor table\nconst size_t MAX_TREEWIDTH = 32;\nconst size_t MAX_DIM = MAX_TREEWIDTH + 1;\n\n// Basic graphical model typedefs\ntypedef uint32_t                        factor_id_t;\ntypedef graphlab::discrete_variable     variable_t;\ntypedef graphlab::table_factor factor_t;\ntypedef factor_t::domain_type           domain_t;\ntypedef factor_t::assignment_type       assignment_t;\n\n\n\n// A class which represents a factorized distribution as a collection\n// of factors.\nclass factorized_model {\nprivate:\n  std::set _variables;\n  std::vector _factors;\n  std::map< variable_t, std::set > _var_to_factor;\n  std::vector _var_name;\n\n  /**\n   * same as the stl string get line but this also increments a line\n   * counter which is useful for debugging purposes\n   */\n  inline bool getline(std::ifstream& fin,\n                      std::string& line,\n                      size_t line_number) {\n    return std::getline(fin, line).good();\n  }\n\n  /**\n   * Removes trailing and leading white space from a string\n   */\n  inline std::string trim(const std::string& str) {\n    std::string::size_type pos1 = str.find_first_not_of(\" \\t\\r\");\n    std::string::size_type pos2 = str.find_last_not_of(\" \\t\\r\");\n    return str.substr(pos1 == std::string::npos ? 0 : pos1,\n                      pos2 == std::string::npos ? str.size()-1 : pos2-pos1+1);\n  }\n\npublic:\n\n  typedef std::vector factor_map_t;\n\n\n  void reserve(const size_t num_factors);\n\n  //! add a factor to the factorized model\n  factor_t& add_factor(const factor_t& factor);\n\n  //! add a factor to the factorized model\n  factor_t& add_factor(const domain_t& dom);\n\n  \n  const factor_map_t& factors() const { return _factors; }\n  const std::set& variables() const { return _variables; }\n\n  const std::set& factor_ids(const variable_t& var) const;\n\n  const std::string& var_name(size_t id) const {\n    ASSERT_LT(id, _var_name.size());\n    return _var_name[id];\n  }\n\n  \n  void load_alchemy(const std::string& filename);\n\n  //! Save the factor to a file\n  void save(graphlab::oarchive &arc) const;\n\n  //! Load the factor from a file\n  void load(graphlab::iarchive &arc);\n\n  //! save the alchemy file\n  void save_alchemy(const std::string& filename) const;\n \n}; //end of factorized model\n\n\n#endif\n\n\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/global_variables.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \"global_variables.hpp\"\n\n// Global Shared Varaibles ====================================================>\n//graphlab::glshared_const SHARED_FACTORS;\nconst factorized_model::factor_map_t* SHARED_FACTORS_PTR = NULL;\ngraphlab::glshared_const MAX_NSAMPLES;\ngraphlab::glshared n_samples;\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/global_variables.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef  PGIBBS_GLOBAL_VARIABLES\n#define  PGIBBS_GLOBAL_VARIABLES\n\n\n#include \n\n#include \"factorized_model.hpp\"\n\n// Global Shared Varaibles ====================================================>\n//extern graphlab::glshared_const SHARED_FACTORS;\nextern const factorized_model::factor_map_t* SHARED_FACTORS_PTR;\nextern graphlab::glshared_const MAX_NSAMPLES;\nextern graphlab::glshared n_samples;\n\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/image.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\n\n#include \n\n\n#include \"image.hpp\"\n\n\n#include \n\n\n\nvoid image::save_vec(const char* filename) const {\n  std::ofstream os(filename);\n  ASSERT_TRUE(os.good());\n  for(size_t i = 0; i < pixels(); ++i) {\n    os << pixel(i) << \"\\n\";\n  }\n  os.flush();\n  os.close();\n}\n\n\ndouble image::min() const {\n  return *std::min_element(data.begin(), data.end());\n}\n\n\ndouble image::max() const {\n  return *std::max_element(data.begin(), data.end());\n}\n\n\nvoid image::save(graphlab::oarchive &oarc) const {\n  oarc << _rows;\n  oarc << _cols;\n  oarc << data;\n}\n\n\nvoid image::load(graphlab::iarchive &iarc) {\n  iarc >> _rows;\n  iarc >> _cols;\n  iarc >> data;\n}\n\n\n/** Generate a normally distributed random number N(mu, sigma^2) */\n// std::pair randn(double mu = 0, double sigma = 1 ); \n\n\n// IMPLEMENTATION =============================================================>\n\n\nvoid image::resize(size_t rows, size_t cols) {\n  _rows = rows;\n  _cols = cols;\n  data.resize(rows * cols, 0);\n}\n  \n\n\n/** Get the vertex id of a pixel */\nsize_t image::vertid(size_t i, size_t j) const {\n  ASSERT_LT(i, _rows);\n  ASSERT_LT(j, _cols);    \n  return i * _cols + j; \n}\n\n// static size_t image::vertid(size_t rows, size_t cols, size_t i, size_t j)  {\n//   assert(i < rows);\n//   assert(j < cols);    \n//   return i * cols + j; \n// }\n\n\n/** Get the vertex id of a pixel */\nstd::pair image::loc(size_t vertexid) const {\n  ASSERT_LT(vertexid, _rows * _cols);\n  return std::make_pair( vertexid / _cols, vertexid % _cols);\n}\n\n\nvoid image::save(const char* filename) const {\n  std::ofstream os(filename);\n  os << \"P2\" << std::endl\n     << _cols << \" \" << _rows << std::endl\n     << 255 << std::endl;\n  // Compute min and max pixel intensities\n  double min = data[0]; double max = data[0];\n  for(size_t i = 0; i < _rows * _cols; ++i) {\n    min = std::min(min, data[i]);\n    max = std::max(max, data[i]);\n  }\n\n  // Save the image (rescaled)\n  for(size_t r = 0; r < _rows; ++r) {\n    for(size_t c = 0; c < _cols; c++) {\n      if(min != max) {\n        int color = \n          static_cast(255.0 * (pixel(r,c) - min)/(max-min));\n        os << color;\n      } else { os << min; }\n      if(c != _cols-1) os << \"\\t\";\n    }\n    os << std::endl;\n  } \n  os.flush();\n  os.close();\n} // end of save\n\n\n\nvoid image::paint_sunset(size_t states) {\n  const double center_r = rows() / 2.0;\n  const double center_c = cols() / 2.0;\n  const double max_radius = std::min(rows(), cols()) / 2.0;\n  // Fill out the image\n  for(size_t r = 0; r < rows(); ++r) {\n    for(size_t c = 0; c < cols(); ++c) {\n      double distance = sqrt((r-center_r)*(r-center_r) + \n                             (c-center_c)*(c-center_c));\n      // If on top of image\n      if(r < rows() / 2) {\n        // Compute ring of sunset\n        size_t ring = \n          static_cast(std::floor(std::min(1.0, distance/max_radius)\n                                         * (states - 1) ) );\n        pixel(r,c) = ring;\n      } else {\n        size_t blockx = r / 20;\n        size_t blocky = (c + 20 * sin(10.0*r/rows())) / 20;\n        size_t index = blockx + 2*blocky;\n        pixel(r,c) = index % states;\n\n      }\n    }\n  }\n} // end of paint_beatiful_sunset\n\n\nvoid image::paint_checkerboard(size_t states, size_t blocks) {\n  size_t block_size = std::min(rows(), cols() ) / blocks;\n  // Fill out the image\n  for(size_t r = 0; r < rows(); ++r) {\n    for(size_t c = 0; c < cols(); ++c) {\n      size_t blockx = r / block_size;\n      size_t blocky = c / block_size;\n      size_t index = blockx + blocky * block_size;\n      pixel(r,c) = index % states;\n    }\n  }\n} // end of paint_beatiful_sunset\n\n\n\n\n\n/** corrupt the image with gaussian noise */\nvoid image::gaussian_corrupt(double sigma) {\n  //  boost::mt19937 rng;\n  boost::lagged_fibonacci607 rng;\n  boost::normal_distribution noise_model(0, sigma);\n  for(size_t i = 0; i < rows() * cols();  ) {\n    // Corrupt two pixels at a time.\n    pixel(i++) += noise_model(rng);\n  }\n} // end of corrupt_image\n\n/** flip_corrupt */\nvoid image::flip_corrupt(size_t states, double prob_flip) {\n  boost::mt19937 rng;\n  boost::uniform_real dist01;\n\n  for(size_t i = 0; i < rows() * cols();  ++i) {\n    double p = dist01(rng);\n    if(p < prob_flip) pixel(i) = rand() % states;\n  }\n} // end of corrupt_image\n\n\n\n// /** generate a normally distributed iid pair */\n// std::pair randn(double mu , double sigma ) {\n//   // Generate a N(0,1) from a Unif(0,1) using Box-Muller generator:\n//   double u1 = static_cast(rand()) / RAND_MAX;\n//   double u2 = static_cast(rand()) / RAND_MAX;\n//   double coeff = std::sqrt(-2.0 * std::log(u1));\n//   double n1 = coeff * std::cos(2.0 * M_PI * u2) ;\n//   double n2 = coeff * std::sin(2.0 * M_PI * u2) ;\n//   // Adjust for mean and variance\n//   n1 = sigma * n1 + mu;\n//   n2 = sigma * n2 + mu; \n//   return std::make_pair(n1, n2);\n// } // end of randn\n\n#include \n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/image.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef PGIBBS_IMAGE_HPP\n#define PGIBBS_IMAGE_HPP\n\n#include \n#include \n#include \n\n#include \n\n\n/** A simple struct represent a gray scale image */\nclass image {\n  size_t _rows, _cols;\n  std::vector data;\npublic:\n\n  /** Create an empty image */\n  image() : _rows(0), _cols(0), data(0,0) { }\n  \n  /** Create an image of a fixed size */\n  image(size_t rows, size_t cols) : \n    _rows(rows), _cols(cols), data(rows * cols, 0) { }\n\n  void resize(size_t rows, size_t cols);\n\n  /** Get the number of rows */\n  size_t rows() const { return _rows; }\n\n  /** Get the number of columns */\n  size_t cols() const { return _cols; }\n\n  /** get the number of pixels */\n  size_t pixels() const { return _rows * _cols; }\n\n  /** A function to read a pixel */\n  double& pixel(size_t i, size_t j) { return data[vertid(i,j)]; }\n  double pixel(size_t i, size_t j) const { return data[vertid(i,j)]; }\n  \n  /** Linear indexing */\n  double& pixel(size_t i) { return data.at(i); }\n  double pixel(size_t i) const { return data.at(i); }\n\n  /** Get the vertex id of a pixel */\n  size_t vertid(size_t i, size_t j) const;\n\n  static size_t vertid(size_t rows, size_t cols, size_t i, size_t j) {\n    ASSERT_LT(i, rows);\n    ASSERT_LT(j, cols);    \n    return i * cols + j; \n  }\n  \n  \n  /** Get the pixel address from the vertex id */\n  std::pair loc(size_t vertex) const;\n\n  \n  /** A function to save the image to a file in pgm format */\n  void save(const char* filename) const;\n\n  void save_vec(const char* filename) const;\n  \n  /** paint a beautiful sunset */\n  void paint_sunset(size_t states);\n  \n  void paint_checkerboard(size_t states, size_t blocks = 10);\n  \n  /** Add random noise to the image */\n  void gaussian_corrupt(double sigma);\n\n  void flip_corrupt(size_t states, double flip_prob);\n\n  double min() const;\n\n  double max() const;\n\n  void save(graphlab::oarchive &oarc) const;\n  \n  void load(graphlab::iarchive &iarc);\n\n};\n\n\n#endif\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/jt_splash_sampler.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n\n#include \"util.hpp\"\n#include \"jt_splash_sampler.hpp\"\n#include \"pgibbs_tls.hpp\"\n#include \"run_statistics.hpp\"\n#include \"global_variables.hpp\"\n\n\n\n\n#include \n\n\n\n\n\n\n\n\n\n\n\nvoid run_jtsplash_sampler(mrf_graph_type& mrf_graph,\n                          const std::string& jtsplash_results_fn,\n                          const std::vector& runtimes,\n                          const bool draw_images,\n                          const splash_settings& settings) {\n\n  //  size_t ncpus = core.engine().get_ncpus();\n  // bool affinities = \n  //   core.get_engine_options().get_cpu_affinities();\n  // Initialize the jtsplash sampler\n  jt_splash_sampler jtsplash_sampler(mrf_graph, settings);\n\n  double total_runtime = 0;\n  double actual_total_runtime = 0;\n  foreach(const double experiment_runtime, runtimes) {\n    total_runtime += experiment_runtime;\n    // get the experiment id\n    size_t experiment_id = file_line_count(jtsplash_results_fn);\n    std::cout << \"Running JTSplash sampler experiment \" << experiment_id\n              << \" for \" << experiment_runtime << \" seconds.\" << std::endl;\n\n    std::cout << \"Settings: ======================\" << std::endl\n              << \"Experiment:    \" << experiment_id << std::endl\n              << \"runtime:       \" << experiment_runtime << std::endl\n              << \"treesize:      \" << settings.max_tree_size << std::endl\n              << \"treewidth:     \" << settings.max_tree_width << std::endl\n              << \"treeheight:    \" << settings.max_tree_height << std::endl\n              << \"factorsize:    \" << settings.max_factor_size << std::endl\n              << \"subthreads:    \" << settings.subthreads << std::endl\n              << \"priorities:    \" << settings.priorities << std::endl   \n              << \"vanish:        \" << settings.vanish_updates << std::endl;   \n\n\n    graphlab::timer timer;\n    timer.start();\n\n    // run the sampler once\n    jtsplash_sampler.sample_seconds(experiment_runtime);\n\n    double actual_experiment_runtime = timer.current_time();\n    std::cout << \"Actual Experiment Runtime:\" \n              << actual_experiment_runtime << std::endl;        \n    actual_total_runtime += actual_experiment_runtime;\n    std::cout << \"Total Experiment Runtime (actual): \"\n              << total_runtime \n              << \"(\" << actual_total_runtime << \")\" \n              << std::endl;\n\n\n    // check mrf graph\n    for(size_t i = 0; i < mrf_graph.num_vertices(); ++i) {\n      ASSERT_EQ(mrf_graph.vertex_data(i).tree_info.tree_id, NULL_VID);\n    }\n\n\n\n    /// ==================================================================\n    // Compute final statistics of the mode\n    run_statistics stats(mrf_graph);\n    stats.print();\n    // Save the beliefs\n    save_beliefs(mrf_graph,\n                 make_filename(\"jtsplash_blfs_\", \".tsv\", experiment_id));\n    // // Save the current assignments\n    save_asg(mrf_graph,\n             make_filename(\"jtsplash_asg_\", \".asg\", experiment_id));\n    // Save the experiment\n    std::ofstream fout(jtsplash_results_fn.c_str(), std::ios::app);\n    fout.precision(8);\n    fout << experiment_id << '\\t'\n         << total_runtime << '\\t'\n         << actual_total_runtime << '\\t'\n         << settings.ntrees << '\\t'\n         << stats.nsamples << '\\t'\n         << stats.nchanges << '\\t'\n         << stats.loglik << '\\t'\n         << stats.min_samples << '\\t'\n         << stats.max_samples << '\\t'\n         << settings.max_tree_size << '\\t'\n         << settings.max_tree_width << '\\t'\n         << settings.max_factor_size << '\\t'\n         << settings.max_tree_height << '\\t'\n         << settings.subthreads << '\\t'\n         << settings.priorities << '\\t'\n         << jtsplash_sampler.total_trees() << '\\t'  \n         << jtsplash_sampler.total_collisions() << '\\t'\n         << std::endl;\n    fout.close();\n\n    // Plot images if desired\n    if(draw_images) draw_mrf(experiment_id, \"jtsplash\", mrf_graph);\n\n  } // end of for loop over runtimes\n\n} // end of run_jtsplash_sampler\n\n\n\n\n\nvoid jtree_update::operator()(base::icontext_type& context) {\n  typedef factorized_model::factor_map_t factor_map_t;\n  ASSERT_NE(mrf_ptr, NULL);\n  mrf_graph_type& mrf = *mrf_ptr;\n    \n  // get the vertex data\n  jtree_vertex_data& vdata = context.vertex_data();\n    \n  // get thread local storage to reduce hit on allocator\n  pgibbs_tls& tls = get_pgibbs_tls();\n\n  //////////////////////////////////////////////////////////////////\n  // Initialize factor    \n    \n  // If the factor args have not been set then we need to initialize\n  // the local factor by setting the args and taking the product of\n  // all factors associated with the clique.  Some of these factors\n  // may depend on variables not in the clique and are therefore\n  // sliced (conditioned) on the current assignment to those\n  // variables.\n  if(vdata.factor.args() != vdata.variables) {\n    // Resize the factor for the variables in the clique\n    vdata.factor.set_args(vdata.variables);\n    vdata.factor.uniform();\n\n    // We now build up the factor by iteratoring over the dependent\n    // factors conditioning if necessary into the conditional_factor\n    // and then multiplying.\n    factor_t& conditional_factor(tls.conditional_factor);\n    // Iterate over the factors and multiply each into this factor\n    foreach(size_t factor_id, vdata.factor_ids) {\n      //const factor_t& factor = SHARED_FACTORS.get()[factor_id];\n      const factor_t& factor = (*SHARED_FACTORS_PTR)[factor_id];\n      // Build up an assignment for the conditional\n      domain_t conditional_args = factor.args() - vdata.variables;\n      assignment_t conditional_asg;\n      for(size_t i = 0; i < conditional_args.num_vars(); ++i) {\n        const mrf_vertex_data& mrf_vdata = \n          mrf.vertex_data(conditional_args.var(i).id());\n        ASSERT_EQ(mrf_vdata.tree_info.tree_id, NULL_VID);\n        //        ASSERT_FALSE(mrf_vdata.tree_info.in_tree);\n        conditional_asg &= \n          assignment_t(mrf_vdata.variable, mrf_vdata.asg);         \n      }\n      // set the factor arguments\n      conditional_factor.set_args(factor.args() - conditional_args);\n      conditional_factor.condition(factor, conditional_asg);        \n      // Multiply the conditional factor in\n      vdata.factor *= conditional_factor;\n    }\n    // Extra normalization for stability on the table factors\n    vdata.factor.normalize();\n    // vdata.belief = vdata.factor;\n  }\n\n  //////////////////////////////////////////////////////////////////\n  // receive any unreceived messages\n  size_t received_neighbors = 0;\n  if(!vdata.calibrated) {\n    foreach(edge_id_t in_eid, context.in_edge_ids()) {\n      jtree_edge_data& in_edata = context.edge_data(in_eid);\n      // if the message has been calibrated but not received\n      if(in_edata.calibrated && !in_edata.received) {\n        // receive message and mark as calibrated\n        vdata.factor *= in_edata.message;\n        vdata.factor.normalize();\n        in_edata.received = true;\n      }\n      // track total received neighbors\n      if(in_edata.received) received_neighbors++;\n    } // end of receive all in messages\n      // if all messages have been received then set as calibrated\n    vdata.calibrated = \n      received_neighbors == context.in_edge_ids().size();\n  } else {\n    received_neighbors = context.in_edge_ids().size();\n  }\n\n\n  //////////////////////////////////////////////////////////////////\n  // send any unset messages \n  // if we have recieve enough in messages\n  if(received_neighbors + 1 >= context.in_edge_ids().size()) {\n    factor_t& cavity(tls.cavity);\n    foreach(edge_id_t out_eid, context.out_edge_ids()) {\n      jtree_edge_data& out_edata = context.edge_data(out_eid);\n      edge_id_t rev_eid = context.reverse_edge(out_eid);\n      // if the out message is not calibrated try to calibrate it:\n      if(!out_edata.calibrated) {\n        bool ready_to_send = true;\n        // check that all in messages (except the one we want to\n        // send) have been recieved\n        foreach(edge_id_t in_eid, context.in_edge_ids()) {\n          const jtree_edge_data& in_edata = context.const_edge_data(in_eid);\n          // if the in edge has not been received and is not from\n          // the destination of the out edge then we cannot send\n          if(!in_edata.received && rev_eid != in_eid) {\n            ready_to_send = false;\n            break;\n          }\n        } // check all neighbors are go for send\n\n          // if we are ready to send then compute message\n        if(ready_to_send) {\n          cavity = vdata.factor;\n          const jtree_edge_data& in_edata = context.const_edge_data(rev_eid);\n          // construct cavity if necessary\n          if(in_edata.received) {\n            cavity /= in_edata.message;\n            cavity.normalize();\n          }\n          // compute actual message\n          out_edata.message.set_args(out_edata.variables);\n          out_edata.message.marginalize(cavity);\n          out_edata.message.normalize();\n          out_edata.calibrated = true;\n          // schedule the reception of the message\n          callback.add_task(context.target(out_eid), jtree_sample_update, 1.0);      \n        } // end of if ready to send\n      } // end of if not calibrated\n    } // end of loop over outbound messages\n  } // of send all out messages\n\n\n  //////////////////////////////////////////////////////////////////\n  // Construct RB estimate and Sample if calibrated but not yet\n  // sampled\n  if(vdata.calibrated && !vdata.sampled) {\n    // check that the parent is sampled and also determine which\n    // variables are going to be sampled at this clique.  This is\n    // done by finding the parent assignment if there is one\n    assignment_t parent_asg;\n    edge_id_t to_parent_eid = NULL_EID;\n\n    // find the parent\n    bool parent_found = false;\n    foreach(edge_id_t out_eid, context.out_edge_ids()) {       \n      const jtree_vertex_data& parent_vdata = \n        context.const_neighbor_vertex_data(context.target(out_eid));\n      if(parent_vdata.sampled) {\n        ASSERT_TRUE(parent_vdata.calibrated);\n        ASSERT_FALSE(parent_found);\n        parent_found = true;\n        to_parent_eid = out_eid;\n        const jtree_edge_data& parent_edata = \n          context.const_edge_data(to_parent_eid);\n        parent_asg = \n          parent_vdata.asg.restrict(parent_edata.variables);\n        ASSERT_EQ(parent_asg.args(), parent_edata.variables);            \n        // break;\n      }\n    }\n\n      \n    // Determine the remaining variables for which we will need to\n    // sample and construct RB estimates\n    domain_t unsampled_variables = \n      vdata.variables - parent_asg.args();\n    vdata.asg = parent_asg;\n\n    // if there are unsampled variables then sample them\n    if(unsampled_variables.num_vars() > 0) {\n      // First update all the RB estimates for the unsampled\n      // variables in the mrf graph\n      factor_t& tmp_belief(tls.tmp_belief);\n      for(size_t i = 0; i < unsampled_variables.num_vars(); ++i) {\n        variable_t var = unsampled_variables.var(i);\n        // Construct the RB belief estimate\n        tmp_belief.set_args(var);\n        tmp_belief.marginalize(vdata.factor);\n        tmp_belief.normalize();\n        // Update the MRF\n        mrf_vertex_data& mrf_vdata = get_mrf_vdata(var.id());\n        mrf_vdata.belief += tmp_belief;\n      } \n\n      // Condition the belief on the parent assignmnet\n      tmp_belief.set_args(unsampled_variables);\n      tmp_belief.condition(vdata.factor, parent_asg);\n      tmp_belief.normalize();\n\n      // Sample the remaining variables from the belief\n      assignment_t sample_asg = tmp_belief.sample();\n      // Set the local assignment\n      vdata.asg = sample_asg & parent_asg;\n      // the assignment should exacty cover the variables\n      ASSERT_EQ(vdata.asg.args(), vdata.variables);\n\n      \n      //// Fill out the MRF with the sampled variables\n      for(size_t i = 0; i < sample_asg.num_vars(); ++i) {\n        variable_t var = sample_asg.args().var(i);\n        mrf_vertex_data& mrf_vdata = get_mrf_vdata(var.id());\n        assignment_t local_asg = sample_asg.restrict(var);\n        if(mrf_vdata.asg != local_asg.asg_at(0)) {\n          mrf_vdata.nchanges++;\n        }\n        mrf_vdata.asg = local_asg.asg_at(0);\n        mrf_vdata.nsamples++;\n        // std::cout << graphlab::thread::thread_id()\n        //           << \": sampling \" << mrf_vdata.variable << std::endl;\n        // remove the vertex from any trees\n        mrf_vdata.tree_info.tree_id = NULL_VID;\n        //        mrf_vdata.tree_info.in_tree = false;\n        mrf_vdata.tree_info.height = 0;\n          \n        // double& logP = mrf_vdata.belief.logP(mrf_vdata.asg.asg_at(0));\n        // logP = std::log( std::exp(logP) + 1.0 );          \n      } \n    } // end of sampling unsampled variables\n\n      // mark as sampled\n    vdata.sampled = true;\n\n    // Reschedule unsampled neighbors\n    foreach(edge_id_t out_eid, context.out_edge_ids()) {\n      if(out_eid != to_parent_eid) {\n        const vertex_id_t neighbor_vid = context.target(out_eid);\n        ASSERT_LT(neighbor_vid, context.num_vertices());\n        callback.add_task(neighbor_vid, \n                          jtree_sample_update, \n                          1.0);\n      }\n    }\n  } // End of if(!sampled) sampling procedure\n\n} // End of update function\n\n\n\n\ntermination_condition::termination_condition() : \n  error(false), finish_time_seconds(-1), target_nsamples(0), target_ntrees(0),\n  atomic_nsamples(0), atomic_ntrees(0) { }\n\nbool termination_condition::finished() const {\n  return\n    error ||\n    (finish_time_seconds > 0 &&\n     finish_time_seconds < graphlab::lowres_time_seconds()) ||\n    (target_nsamples > 0 && atomic_nsamples.value > target_nsamples) ||\n    (target_ntrees > 0 && atomic_ntrees.value > target_ntrees);\n}\n\nvoid termination_condition::reset() {\n  error = false;\n  finish_time_seconds = -1;\n  target_nsamples = 0;\n  atomic_nsamples.value = 0;\n  target_ntrees = 0;\n  atomic_ntrees.value = 0;\n}\n\n\n\n\n\n\n\n\n\n\njt_worker::jt_worker(size_t worker_id, \n                     const splash_settings& settings,\n                     scope_factory_type& scope_factory, \n                     const std::vector& root_perm,\n                     termination_condition& terminator) :\n  worker_id(worker_id),\n  settings(settings),\n  scope_factory_ptr(&scope_factory),\n  root_index(root_perm.size()),\n  root_perm_ptr(&root_perm),\n  current_root(root_perm.at(worker_id)),\n  terminator_ptr(&terminator),\n  ncollisions(0) {  \n  // Initialize local jtcore\n  if(settings.subthreads > 1) {\n    jt_core.set_scheduler_type(\"multiqueue_fifo\");\n    jt_core.set_scope_type(\"edge\");\n    jt_core.set_ncpus(settings.subthreads);\n    jt_core.set_engine_type(\"async\");\n  } else {\n    jt_core.set_scheduler_type(\"fifo\");\n    jt_core.set_scope_type(\"none\");\n    jt_core.set_ncpus(1);\n    jt_core.set_engine_type(\"async_sim\");\n  }\n} // end of init\n\n\n\n\n\n\n\n\n// get a root\nvoid jt_worker::run() {   \n  // looup until runtime is reached\n  while(!terminator_ptr->finished()) {\n    /////////////////////////////////////////////////////////\n    // Construct one tree (we must succeed in order to count a tree)\n    advance_root();\n    //    std::cout << \"Worker \" << worker_id << \": \" << current_root << std::endl;\n    // here we loop until the current root is sampled\n    size_t sampled_variables = 0;\n    while(sampled_variables == 0 &&  !terminator_ptr->finished()) {\n      sampled_variables = splash_once();\n      // If sample once failed due to collision record a collision event\n      if(sampled_variables == 0) {\n        ncollisions++;\n        // sched_yield();\n      }\n    }\n    // if variables where sampled in the splash increment the atomic\n    // counters.\n    if(sampled_variables > 0) {\n      terminator_ptr->atomic_nsamples.inc(sampled_variables);\n      terminator_ptr->atomic_ntrees.inc();\n    }\n  } \n  //  std::cout << \"N Collisions: \" << ncollisions << std::endl;\n} // end of run\n\n\n\n\nvoid jt_worker::advance_root() {  \n  root_index += settings.ntrees;\n  if(root_index >= root_perm_ptr->size()) root_index = worker_id;\n  current_root = root_perm_ptr->at(root_index);\n}\n\n\n\n\n///////////////////////////////////////////////////////////////////////\n/// Markov Blanket Locking Helper functions\n\n\n/**\n * See if the vertex can be grabbed into this workers tree. If true we\n * still need to actually grab the vertex (which could still\n * fail). However if the vertex is currently unavailable we could save\n * time by not even trying (although it may become available later).\n */\nbool jt_worker::is_vertex_available(vertex_id_t vid) {\n  ASSERT_NE(scope_factory_ptr, NULL);\n  const mrf_graph_type& mrf(scope_factory_ptr->get_graph());\n  const mrf_vertex_data& vdata = mrf.vertex_data(vid);\n  // Check that this vertex is not already in a tree\n  bool in_tree = vdata.tree_info.tree_id != NULL_VID;\n  if(in_tree) return false;\n  // check that the neighbors are not in any other trees than this\n  // one\n  const mrf_gl::edge_list& in_eids = mrf.in_edge_ids(vid);\n  foreach(edge_id_t in_eid, in_eids) {\n    vertex_id_t neighbor_vid = mrf.source(in_eid);\n    const mrf_vertex_data& vdata = mrf.vertex_data(neighbor_vid);\n    bool in_tree = vdata.tree_info.tree_id != NULL_VID;\n    // if the neighbor is in a tree other than this one quit\n    if(in_tree && worker_id != vdata.tree_info.tree_id) return false;\n  }\n  return true;\n} // end of try grab vertex\n\n\n\n\n/**\n * Grab this vertex into the tree owned by worker id.  If this returns\n * true than the vertex is marked as grabbed. This must be called\n * within the context of an edge scope.\n */\nbool jt_worker::try_grab_vertex(iscope_type& scope) {\n  // Check that this vertex is not already in a tree\n  bool in_tree = scope.vertex_data().tree_info.tree_id != NULL_VID;\n  if(in_tree) return false;\n\n  // check that the neighbors are not in any other trees than this\n  // one\n  vertex_id_t min_height(std::numeric_limits::max());\n  foreach(edge_id_t in_eid, scope.in_edge_ids()) {\n    vertex_id_t neighbor_vid = scope.source(in_eid);\n    const mrf_vertex_data& vdata = \n      scope.const_neighbor_vertex_data(neighbor_vid);\n    bool in_tree = vdata.tree_info.tree_id != NULL_VID;\n    // if the neighbor is in a tree other than this one quit\n    if(in_tree && worker_id != vdata.tree_info.tree_id) return false;\n    if(in_tree) min_height = std::min(min_height, vdata.tree_info.height);\n  }\n  // Assert that this vertex is not in a tree and that none of the\n  // neighbors are in other trees This vertex does not neighbor any\n  // other trees than this one\n  scope.vertex_data().tree_info.tree_id = worker_id;\n  //  scope.vertex_data().tree_info.in_tree = true;\n  scope.vertex_data().tree_info.height = min_height + 1;\n  return true;\n} // end of try grab vertex\n\n\n\n\n/**\n * Release the vertex\n */\nvoid jt_worker::release_vertex(iscope_type& scope) {\n  // This vertex does not neighbor any other trees than this one\n  scope.vertex_data().tree_info.tree_id = NULL_VID;\n  //  scope.vertex_data().tree_info.in_tree = false;\n  scope.vertex_data().tree_info.height = 0;\n} // release the vertex\n\n\n\n\n\n\n\n\n\n\n\n///////////////////////////////////////////////////////////////////////\n/// Scoring helper functions\n\n/**\n * This function returns the priority of a particular vertex.\n */\ndouble jt_worker::score_vertex(vertex_id_t vid) {\n  ASSERT_NE(scope_factory_ptr, NULL);\n  mrf_graph_type& mrf(scope_factory_ptr->get_graph());\n  mrf_vertex_data& vdata = mrf.vertex_data(vid);\n\n  if (vdata.nsamples < settings.vanish_updates || \n      vdata.tree_info.priority < 0) {\n    vdata.tree_info.priority = score_vertex_log_odds(vid); \n  }\n  return vdata.tree_info.priority;\n}\n\n\n\n\n\n\ndouble jt_worker::score_vertex_l1_diff(vertex_id_t vid) {\n  // Get the scope factory\n  ASSERT_NE(scope_factory_ptr, NULL);\n  const mrf_graph_type& mrf(scope_factory_ptr->get_graph());\n  const mrf_vertex_data& vdata = mrf.vertex_data(vid);\n\n  // Construct the domain of neighbors that are already in the tree\n  domain_t vars = vdata.variable;\n  foreach(edge_id_t ineid, mrf.in_edge_ids(vid)) {\n    const vertex_id_t neighbor_vid = mrf.source(ineid);\n    const mrf_vertex_data& neighbor = mrf.vertex_data(neighbor_vid);\n    // test to see if the neighbor is in the tree by checking the\n    // elimination time map\n    if(jt_list.contains(neighbor_vid)) {\n      vars += neighbor.variable;\n      // If this vertex has too many tree neighbor than the priority\n      // is set to -1;\n      if(vars.num_vars() > settings.max_tree_width + 1) return -1;\n      if(vars.size() > settings.max_factor_size) return -1;\n    } \n  }\n\n  // Compute the clique factor\n  clique_factor.set_args(vars);\n  clique_factor.uniform();\n  // get all the factors const factorized_model::factor_map_t&\n  // factors(SHARED_FACTORS.get());\n  const factorized_model::factor_map_t& factors(*SHARED_FACTORS_PTR);\n  // Iterate over the factors and multiply each into this factor\n  foreach(size_t factor_id, vdata.factor_ids) {\n    const factor_t& factor = factors[factor_id];      \n    // Build up an assignment for the conditional\n    domain_t conditional_args = factor.args() - vars;\n    if(conditional_args.num_vars() > 0) {\n      assignment_t conditional_asg;\n      for(size_t i = 0; i < conditional_args.num_vars(); ++i) {\n        const mrf_vertex_data& neighbor_vdata = \n          mrf.vertex_data(conditional_args.var(i).id());\n        conditional_asg &= \n          assignment_t(neighbor_vdata.variable, neighbor_vdata.asg);\n      }\n      // set the factor arguments\n      conditional_factor.set_args(factor.args() - conditional_args);\n      conditional_factor.condition(factor, conditional_asg);        \n      // Multiply the conditional factor in\n      clique_factor *= conditional_factor;\n      //       clique_factor.normalize();\n    } else {\n      clique_factor *= factor;\n    }\n  } // end of loop over factors\n  clique_factor.normalize();\n\n\n  // Compute the product of marginals\n  product_of_marginals_factor.set_args(vars);\n  product_of_marginals_factor.uniform();\n  for(size_t i = 0; i < vars.num_vars(); ++i) {\n    marginal_factor.set_args(vars.var(i));\n    marginal_factor.marginalize(clique_factor);\n    marginal_factor.normalize();\n    product_of_marginals_factor *= marginal_factor;\n  }\n  product_of_marginals_factor.normalize();\n\n  // Compute the residual\n  double residual = clique_factor.l1_diff(product_of_marginals_factor);\n\n  ASSERT_GE( residual, 0);\n  ASSERT_FALSE( std::isnan(residual) );\n  ASSERT_TRUE( std::isfinite(residual) );\n\n  // ensure score is bounded\n  //    residual = std::tanh(residual);\n\n  return residual;\n\n} // end of score l1 diff\n\n\n\n\n\n\n\n\ndouble jt_worker::score_vertex_log_odds(vertex_id_t vid) {\n  // Get the scope factory\n  const mrf_graph_type& mrf(scope_factory_ptr->get_graph());\n  const mrf_vertex_data& vdata(mrf.vertex_data(vid));\n\n  // Construct the domain of neighbors that are already in the tree\n  domain_t vars = vdata.variable;\n  foreach(edge_id_t ineid, mrf.in_edge_ids(vid)) {\n    const vertex_id_t neighbor_vid = mrf.source(ineid);\n    const mrf_vertex_data& neighbor = mrf.vertex_data(neighbor_vid);\n    // test to see if the neighbor is in the tree by checking the\n    // elimination time map\n    if(jt_list.contains(neighbor_vid)) {\n      vars += neighbor.variable;\n      // If this vertex has too many tree neighbor than the priority\n      // is set to 0;\n      if(vars.num_vars() > settings.max_tree_width + 1) return -1;\n      if(vars.size() > settings.max_factor_size) return -1;\n    } \n  }\n    \n  ASSERT_EQ(vars.num_vars(),  2);\n\n\n  // Compute the clique factor\n  clique_factor.set_args(vars);\n  clique_factor.uniform();\n  // get all the factors\n  // const factorized_model::factor_map_t& factors(SHARED_FACTORS.get());\n  const factorized_model::factor_map_t& factors(*SHARED_FACTORS_PTR);\n\n  // Iterate over the factors and multiply each into this factor\n  foreach(size_t factor_id, vdata.factor_ids) {\n    const factor_t& factor = factors[factor_id];      \n    // Build up an assignment for the conditional\n    domain_t conditional_args = factor.args() - vars;\n    if(conditional_args.num_vars() > 0) {\n      assignment_t conditional_asg;\n      for(size_t i = 0; i < conditional_args.num_vars(); ++i) {\n        const mrf_vertex_data& neighbor_vdata = \n          mrf.vertex_data(conditional_args.var(i).id());\n        conditional_asg &= \n          assignment_t(neighbor_vdata.variable, neighbor_vdata.asg);\n      }\n      // set the factor arguments\n      conditional_factor.set_args(factor.args() - conditional_args);\n      conditional_factor.condition(factor, conditional_asg);        \n      // Multiply the conditional factor in\n      clique_factor *= conditional_factor;\n      //        clique_factor.normalize();\n    } else {\n      clique_factor *= factor;\n    }\n  } // end of loop over factors\n    // Compute the conditional factor and marginal factors\n  conditional_factor.set_args(vars - vdata.variable);\n  conditional_factor.condition(clique_factor, \n                               assignment_t(vdata.variable, vdata.asg));  \n  marginal_factor.set_args(vars - vdata.variable);\n  marginal_factor.marginalize(clique_factor);\n    \n  // Compute metric\n  conditional_factor.normalize();\n  marginal_factor.normalize();\n  // double residual = conditional_factor.l1_logdiff(marginal_factor);\n  double residual = conditional_factor.l1_diff(marginal_factor);\n\n  // rescale by updates\n  //    residual = residual / (vdata.updates + 1);\n\n  ASSERT_GE( residual, 0);\n  ASSERT_FALSE( std::isnan(residual) );\n  ASSERT_TRUE( std::isfinite(residual) );\n\n  // ensure score is bounded\n  //    residual = std::tanh(residual);\n\n  return residual;\n} // end of score vertex\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\ndouble jt_worker::score_vertex_lik(vertex_id_t vid) {\n  // Get the scope factory\n  const mrf_graph_type& mrf(scope_factory_ptr->get_graph());\n  const mrf_vertex_data& vdata(mrf.vertex_data(vid));\n\n  // Construct the domain of neighbors that are already in the tree\n  domain_t vars = vdata.variable;\n  foreach(edge_id_t ineid, mrf.in_edge_ids(vid)) {\n    const vertex_id_t neighbor_vid = mrf.source(ineid);\n    const mrf_vertex_data& neighbor = mrf.vertex_data(neighbor_vid);\n    // test to see if the neighbor is in the tree by checking the\n    // elimination time map\n    if(jt_list.contains(neighbor_vid)) {\n      vars += neighbor.variable;\n      // If this vertex has too many tree neighbor than the priority\n      // is set to 0;\n      if(vars.num_vars() > settings.max_tree_width + 1) return -1;\n      if(vars.size() > settings.max_factor_size) return -1;\n    } \n  }\n    \n  // Compute the clique factor\n  clique_factor.set_args(vars);\n  clique_factor.uniform();\n  // get all the factors\n  // const factorized_model::factor_map_t& factors(SHARED_FACTORS.get());\n  const factorized_model::factor_map_t& factors(*SHARED_FACTORS_PTR);\n\n  // Iterate over the factors and multiply each into this factor\n  foreach(size_t factor_id, vdata.factor_ids) {\n    const factor_t& factor = factors[factor_id];      \n    // Build up an assignment for the conditional\n    domain_t conditional_args = factor.args() - vars;\n    if(conditional_args.num_vars() > 0) {\n      assignment_t conditional_asg;\n      for(size_t i = 0; i < conditional_args.num_vars(); ++i) {\n        const mrf_vertex_data& neighbor_vdata = \n          mrf.vertex_data(conditional_args.var(i).id());\n        conditional_asg &= \n          assignment_t(neighbor_vdata.variable, neighbor_vdata.asg);\n      }\n      // set the factor arguments\n      conditional_factor.set_args(factor.args() - conditional_args);\n      conditional_factor.condition(factor, conditional_asg);        \n      // Multiply the conditional factor in\n      clique_factor *= conditional_factor;\n      //        clique_factor.normalize();\n    } else {\n      clique_factor *= factor;\n    }\n  } // end of loop over factors\n\n    // Compute the conditional factor and marginal factors\n  marginal_factor.set_args(vdata.variable);\n  marginal_factor.marginalize(clique_factor);\n  marginal_factor.normalize();\n  double residual =  1.0 - exp(marginal_factor.logP(vdata.asg));\n\n  ASSERT_GE( residual, 0);\n  ASSERT_FALSE( std::isnan(residual) );\n  ASSERT_TRUE(  std::isfinite(residual) );\n\n  // // ensure score is bounded\n  // residual = std::tanh(residual);\n\n\n  return residual;\n} // end of max lik\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n///////////////////////////////////////////////////////////////////////\n/// Tree Growing helper functions\n\n\nvoid jt_worker::grow_bfs_jtree() {\n  ASSERT_NE(scope_factory_ptr, NULL);\n  // Get the scope factory\n  mrf_graph_type& mrf = scope_factory_ptr->get_graph();\n  // Clear local data structures\n  jt_list.clear();\n  bfs_queue.clear();\n  visited.clear();\n     \n  // add the root\n  bfs_queue.push_back(current_root);\n  visited.insert(current_root);\n\n  while(!bfs_queue.empty() && \n        jt_list.cliques.size() < settings.max_tree_size) {\n    // Take the top element\n    const vertex_id_t next_vertex = bfs_queue.front();\n    bfs_queue.pop_front();\n\n    // pretest that the vertex is available before trying to get it\n    if(!is_vertex_available(next_vertex)) continue;\n\n    // Maybe we can get the vertex so actually try to get it by first\n    // grabbing the lock (scope) for the vertex\n    iscope_type* scope_ptr = \n      scope_factory_ptr->get_edge_scope(worker_id, next_vertex);\n    ASSERT_NE(scope_ptr, NULL);\n    iscope_type& scope(*scope_ptr);\n\n    // See if we can get the vertex for this tree\n    if(!try_grab_vertex(scope)) {\n      // release the scope and move on\n      scope_factory_ptr->release_scope(&scope);        \n      continue;\n    }\n\n    // Assert that we own the vertex at this point\n    ASSERT_EQ(scope.vertex_data().tree_info.tree_id, worker_id);\n    \n    // Determine if this is the root (it is the root if there are no\n    // cliques in the junction tree).\n    bool is_root = jt_list.cliques.empty();   \n    // Set the height of the root to be zero explicity\n    if(is_root) scope.vertex_data().tree_info.height = 0;\n \n    // Check if it is safe to extend to the tree to include next variable\n    bool extended_jtree =\n      scope.vertex_data().tree_info.height < settings.max_tree_height\n      &&\n      jt_list.extend(next_vertex,\n                     mrf,\n                     settings.max_tree_width,\n                     settings.max_factor_size);\n\n    // If we were unable to extend the tree then release the vertex\n    if(!extended_jtree) {\n      release_vertex(scope);     \n      scope_factory_ptr->release_scope(&scope);        \n      continue;\n    } \n\n\n    // add the neighbors to the search queue\n    foreach(edge_id_t eid, mrf.out_edge_ids(next_vertex)) {\n      vertex_id_t neighbor_vid = mrf.target(eid);\n      if(visited.count(neighbor_vid) == 0) {\n        bfs_queue.push_back(neighbor_vid);\n        visited.insert(neighbor_vid);\n      }\n    }    \n\n    // Release the scope and let neighbors start to run\n    scope_factory_ptr->release_scope(&scope);        \n  } // end of while loop\n\n} // end grow_bfs_jtree\n\n\n\n\n\n\n\nvoid jt_worker::grow_prioritized_jtree() {\n  ASSERT_NE(scope_factory_ptr, NULL);\n  // Get the scope factory\n  mrf_graph_type& mrf = scope_factory_ptr->get_graph();\n  // Clear local data structures\n  jt_list.clear();\n  priority_queue.clear();\n  visited.clear();\n     \n  // add the root\n  priority_queue.push(current_root, 1.0);\n  visited.insert(current_root);\n\n  while(!priority_queue.empty() && \n        jt_list.cliques.size() < settings.max_tree_size) {\n    // Take the top element\n    const vertex_id_t next_vertex = priority_queue.pop().first;\n\n    // pretest that the vertex is available before trying to get it\n    if(!is_vertex_available(next_vertex)) continue;\n\n    // Maybe we can get the vertex so actually try to get it by first\n    // grabbing the lock (scope) for the vertex\n    iscope_type* scope_ptr = \n      scope_factory_ptr->get_edge_scope(worker_id, next_vertex);\n    ASSERT_NE(scope_ptr, NULL);\n    iscope_type& scope(*scope_ptr);\n\n    // See if we can get the vertex for this tree\n    if(!try_grab_vertex(scope)) {\n      // release the scope and move on\n      scope_factory_ptr->release_scope(&scope);        \n      continue;\n    }\n\n    // Assert that we own the vertex at this point\n    ASSERT_EQ(scope.vertex_data().tree_info.tree_id, worker_id);\n\n    // Determine if this is the root (it is the root if there are no\n    // cliques in the junction tree).\n    bool is_root = jt_list.cliques.empty();   \n    // Set the height of the root to be zero explicity\n    if(is_root) scope.vertex_data().tree_info.height = 0;\n          \n    // test the \n    bool extended_jtree =\n      scope.vertex_data().tree_info.height < settings.max_tree_height\n      &&\n      jt_list.extend(next_vertex,\n                     mrf,\n                     settings.max_tree_width,\n                     settings.max_factor_size);                    \n\n    // If we were unable to extend the tree then release the vertex\n    if(!extended_jtree) {\n      release_vertex(scope);\n      scope_factory_ptr->release_scope(&scope);\n      continue;\n    }\n\n    // If the tree was extended, extend the boundary by adding the\n    // neighbors of the newly added vertex to the tree\n\n    // add the neighbors to the search queue or update their priority\n    foreach(edge_id_t eid, mrf.out_edge_ids(next_vertex)) {\n      vertex_id_t neighbor_vid = mrf.target(eid);          \n      if(visited.count(neighbor_vid) == 0) {\n        visited.insert(neighbor_vid);\n        // Vertex has not yet been visited\n        double score = score_vertex(neighbor_vid);\n        // if the score is greater than zero then add the neighbor\n        // to the priority queue.  The score is zero if there is no\n        // advantage or the treewidth is already too large\n        if(score >= 0) priority_queue.push(neighbor_vid, score);       \n      } else if(priority_queue.contains(neighbor_vid)) {\n        // vertex is still in queue we may need to recompute\n        // score\n        double score = score_vertex(neighbor_vid);\n        if(score >= 0) {\n          // update the priority queue with the new score\n          priority_queue.update(neighbor_vid, score);\n        } else {\n          // The score computation revealed that the clique would be\n          // too large so simply remove the vertex from the priority\n          // queue\n          priority_queue.remove(neighbor_vid);\n        }\n      } // otherwise the vertex has been visited and processed\n    } // end of foreach    \n\n    // Release the scope and let neighbors start to run\n    scope_factory_ptr->release_scope(&scope);        \n\n  } // end of while loop\n    \n} // end grow_prioritized_jtree\n\n\n\n\n\n\n\n\n\n\n\n\n\nsize_t jt_worker::splash_once() {\n  if(settings.priorities) {\n    // grow the prioritized junction tree data structure\n    grow_prioritized_jtree();\n  } else {\n    // grow the bfs junction tree data structure\n    grow_bfs_jtree();\n  }\n\n  ASSERT_NE(scope_factory_ptr, NULL);\n  // Get the scope factory\n  mrf_graph_type& mrf = scope_factory_ptr->get_graph();\n    \n  // If we failed to build a tree return failure\n  if(jt_list.cliques.empty()) return 0;\n\n  //  std::cout << \"Varcount: \" << jt_list.cliques.size() << std::endl;  \n\n  // ///////////////////////////////////\n  // // plot the graph\n  // if(worker_id == 0) {\n  //   std::cout << \"Saving treeImage:\" << std::endl;\n  //   size_t rows = std::sqrt(mrf.num_vertices());\n  //   image img(rows, rows);\n  //   for(vertex_id_t vid = 0; vid < mrf.num_vertices(); ++vid) {\n  //     vertex_id_t tree_id = mrf.vertex_data(vid).tree_id;\n  //     img.pixel(vid) = \n  //         tree_id == vertex_id_t(-1)? 0 : tree_id + worker_count;\n  //   }\n  //   img.save(make_filename(\"tree\", \".pgm\", tree_count).c_str());\n  // }\n\n  // Build the junction tree and sample\n  jt_core.graph().clear();\n  size_t num_factors = (*SHARED_FACTORS_PTR).size();\n\n  // jt_list.validate();\n\n  jt_list.load_graph(mrf, num_factors, jt_core.graph());\n\n  // Rebuild the engine (clear the old scheduler)\n  jt_core.rebuild_engine();\n  \n  // add tasks to all vertices\n  jt_core.add_task_to_all(jtree_sample_update, 1.0);\n\n  // Run the core\n  jt_core.start();\n\n\n\n  // Check that the junction tree is sampled\n  size_t actual_tree_width = 0;\n  for(vertex_id_t vid = 0; \n      vid < jt_core.graph().num_vertices(); ++vid) {\n    const jtree_vertex_data& vdata = jt_core.graph().vertex_data(vid);\n    ASSERT_TRUE(vdata.sampled);\n    ASSERT_TRUE(vdata.calibrated);\n    ASSERT_GT(vdata.variables.num_vars(), 0);\n    actual_tree_width = \n      std::max(vdata.variables.num_vars() - 1, actual_tree_width); \n  }    \n  //  std::cout << \"Treewidth: \" << actual_tree_width << std::endl;\n\n  // Return the number of variables in the tree\n  return jt_list.elim_time.size();\n} // end of sample once\n\n\n\n\n\n\n\n\n\n\n\njt_splash_sampler::\njt_splash_sampler(mrf_graph_type& mrf, \n                  const splash_settings& settings) :\n  workers(settings.ntrees, NULL),\n  scope_factory(mrf, settings.ntrees, \n                graphlab::scope_range::EDGE_CONSISTENCY),\n  root_perm(mrf.num_vertices()) { \n  ASSERT_LE(settings.ntrees, mrf.num_vertices());\n  \n  // Set the shared graph pointer\n  shared_mrf_ptr = &mrf;\n\n  // Shuffle the root ordering \n  for(vertex_id_t vid = 0; vid < mrf.num_vertices(); ++vid)\n    root_perm[vid] = vid;\n  graphlab::random::shuffle(root_perm.begin(), root_perm.end());\n  \n  // initialize the worker thread objects\n  for(size_t i = 0; i < workers.size(); ++i) {\n    workers[i] = \n      new jt_worker(i, settings, scope_factory, root_perm, terminator);\n  }\n} // end of constructor\n\njt_splash_sampler::~jt_splash_sampler() {\n  for(size_t i = 0; i < workers.size(); ++i) {\n    if(workers[i] != NULL) {\n      delete workers[i];\n      workers[i] = NULL;\n    }\n  }\n}\n\nsize_t jt_splash_sampler::total_collisions() const {\n  size_t total_collisions = 0;\n  foreach(const jt_worker* worker, workers)  {\n    ASSERT_NE(worker, NULL);\n    total_collisions += worker->ncollisions;\n  }\n  return total_collisions;\n}\n\n\nsize_t jt_splash_sampler::total_trees() const {\n  return terminator.atomic_ntrees.value;\n}\n\nsize_t jt_splash_sampler::total_samples() const {\n  return terminator.atomic_nsamples.value;\n}\n\n  \nvoid jt_splash_sampler::sample_seconds(float runtime_secs) {\n  // Set the termination condition\n  terminator.reset();\n  terminator.finish_time_seconds = \n    graphlab::lowres_time_seconds() + runtime_secs;\n  run();\n}                   \n\nvoid jt_splash_sampler::sample_trees(size_t total_trees) {\n  // Set the termination condition\n  terminator.reset();\n  terminator.target_ntrees = total_trees;\n  run();\n}\n\nvoid jt_splash_sampler::sample_updates(size_t total_updates) {\n  // Set the termination condition\n  terminator.reset();\n  terminator.target_nsamples = total_updates;\n  run();\n}\n\n\n\n\n\n\n\n\nvoid jt_splash_sampler::run() {\n  // create worker threads\n  graphlab::thread_group threads;\n  if(workers.size() == 1) {\n    ASSERT_NE(workers[0], NULL);\n    workers[0]->run();\n  } else {\n    // Launch the threads\n    for(size_t i = 0; i < workers.size(); ++i) {   \n      ASSERT_NE(workers[i], NULL);\n      // if(use_cpu_affinity) \n      //   threads.launch(boost::bind(&jt_worker::run, &(workers[i])), i);\n      // else \n      threads.launch(boost::bind(&jt_worker::run, workers[i]));\n    }\n    const char* exception_message = \"Exception!\";\n    // Wait for all threads to finish\n    while (threads.running_threads() > 0) {\n      try {\n        threads.join();\n      } catch(const char* error) {\n        logstream(LOG_ERROR) << \"Exception Caught:\\n\\t\" << error << std::endl;\n        exception_message = error;\n        // killall the running threads\n        terminator.error = true;\n      }\n    }\n    if(terminator.error) {\n      throw exception_message;\n    }\n  }\n}\n\n\n\n\n\n\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/jt_splash_sampler.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef PGIBBS_JT_SPLASH_SAMPLER_HPP\n#define PGIBBS_JT_SPLASH_SAMPLER_HPP\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\n\n// Including Standard Libraries\n\n\n#include \n#include \n\n#include \n\n#include \"factorized_model.hpp\"\n#include \"mrf.hpp\"\n#include \"junction_tree.hpp\"\n\n\n/**\n * The settings for the jt_splash_sampler.  Originally these formed a\n * long list of arguments but since the order can easily introduce\n * bugs we switched to a struct.\n */\nstruct splash_settings {\n  size_t ntrees;\n  size_t max_tree_size;\n  size_t max_tree_height;\n  size_t max_tree_width;\n  size_t max_factor_size;\n  bool   priorities;\n  size_t vanish_updates; \n  size_t subthreads;\n\n  splash_settings() : \n    ntrees(2), \n    max_tree_size(std::numeric_limits::max()),\n    max_tree_height(std::numeric_limits::max()),\n    max_tree_width(2),\n    max_factor_size(std::numeric_limits::max()),\n    priorities(false),\n    vanish_updates(10),\n    subthreads(1) { }\n};\n\n\n\n/**\n * Run the jtsplash sampler\n *\n */\nvoid run_jtsplash_sampler(mrf_graph_type& mrf_graph,\n                          const std::string& jtsplash_results_fn,\n                          const std::vector& runtimes,\n                          const bool draw_images,\n                          const splash_settings& settings);\n\n\n\n\n\n\n\n/**\n * This fairly complex update function assembles the clique factors by\n * conditioning on variables not in the tree.  Then it computes\n * messages at each clique to calibrate the junction tree.  Finally,\n * using the messages and the conditioned parents, it samples each\n * clique constructing new assignments to each variable.\n */\nclass jtree_update :\n  public graphlab::iupdate_functor {\npublic:  \n  typedef graphlab::iupdate_functor base;\n  jtree_update(mrf_graph_type* mrf_ptr = NULL) : mrf_ptr(mrf_ptr) { }\n  mrf_graph_type* mrf_ptr;\n  void operator()(base::icontext_type& context);\n}; // end of class jtree_update\n\n\n\n// Termination management\nstruct termination_condition {\n  bool   error;\n  float  finish_time_seconds;\n  size_t target_nsamples;\n  size_t target_ntrees;\n  graphlab::atomic atomic_nsamples;\n  graphlab::atomic atomic_ntrees;\n  termination_condition();\n  bool finished() const;\n  void reset();\n};\n\n\n\n\n\n\n\n//! Predecleration \n//! The jt worker executes splashes sequential within each thread.\nclass jt_builder : \n  public graphlab::iupdate_functor{\npublic:\n  typedef graphlab::iupdate_functor base;\n\n  struct splash_state {\n    size_t worker_id;\n    splash_settings settings;\n    // Tree building data structures \n    size_t root_index;\n    const std::vector* root_perm_ptr;\n    vertex_id_t current_root;\n    //! track termination\n    termination_condition* terminator_ptr;\n    mrf_graph_type* graph_ptr;\n    //! Track the collisions with the roots\n    size_t ncollisions;\n    //! Local junction tree graphlab core\n    jtree_gl::core jt_core;\n    /**\n     * Local jt list used to build on the structure of the\n     * jt_core.graph()\n     */\n    jtree_list jt_list;\n    /**\n     * Local data structures to reduce thread contention\n     */\n    std::deque bfs_queue;\n    graphlab::mutable_queue priority_queue;\n    boost::unordered_set visited;\n    factor_t clique_factor;\n    factor_t product_of_marginals_factor;\n    factor_t conditional_factor;\n    factor_t marginal_factor;\n  };\n\n  std::set state_set;\n  jt_worker(splash_state* state_ptr = NULL);\n  void operator+=(const jt_builder& other);\n\n\n\n\n\n\n  //! The main loop\n  void run();\n  \nprivate:\n  //! Construct a single splash\n  size_t splash_once();\n  //! advance the root\n  void advance_root();\n  /**\n   * Grab this vertex into the tree owned by worker id\n   */\n  bool is_vertex_available(vertex_id_t vid);\n  /**\n   * Grab this vertex into the tree owned by worker id\n   */\n  bool try_grab_vertex(iscope_type& scope);\n  /**\n   * Release the vertex\n   */\n  void release_vertex(iscope_type& scope);\n\n\n  double score_vertex(vertex_id_t vid);\n  double score_vertex_l1_diff(vertex_id_t vid);\n  double score_vertex_log_odds(vertex_id_t vid);\n  double score_vertex_lik(vertex_id_t vid);\n\n  void grow_bfs_jtree();\n  void grow_prioritized_jtree();\n};  // End of JT worker\n\n\n\n\n\n/**\n * The jt_splash_sampler implements the junction tree based Gibbs\n * sampler defined in:\n *\n *  Parallel Gibbs Sampling: From Colored Fields to Think Junction Trees\n *   by Joseph Gonzalez, Yucheng Low, Arthur Gretton, and Carlos Guestrin\n *  \n */\nclass jt_splash_sampler {\npublic:\n  typedef graphlab::general_scope_factory\n  scope_factory_type;\n\nprivate:\n  std::vector     workers;\n  scope_factory_type          scope_factory;\n  std::vector< vertex_id_t >  root_perm;\n  termination_condition       terminator;\npublic:\n  jt_splash_sampler(mrf_graph_type& mrf_core,\n                    const splash_settings& settings);\n  ~jt_splash_sampler();\n\n\n  /**\n   * Get the number of times the splash sampler collided on a root.\n   * This minor race event can lead to wasted cpu cycles but does not\n   * affect the quality of the samples.\n   */\n  size_t total_collisions() const;\n\n  /**\n   * Get the total number of trees constructed on the last run\n   */\n  size_t total_trees() const;\n\n  /**\n   * Get the total number of single variable updates on the last run.\n   */\n  size_t total_samples() const;\n\n  /** Run the splash sampler for a fixed number of seconds */\n  void sample_seconds(float runtime_secs);\n  /** Run the splash sampler for a fixed number of trees */\n  void sample_trees(size_t total_trees);\n  /**  \n   * Run the splash sampler for a fixed number of single variable *\n   * updates\n   */\n  void sample_updates(size_t total_updates);\nprivate:\n  void run();\n};\n\n\n\n\n\n\n\n\n\n\n\n\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/junction_tree.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n\n#include \"junction_tree.hpp\"\n\n\n#include \n\n#include \n\n/**\n * Extend the jtree_list data structure by eliminating the vertex.  If\n * the jtree list can be extended then it is extended and this\n * function returns true.\n *\n **/\nbool jtree_list::\nextend(const mrf_graph_type::vertex_id_type elim_vertex,\n       const mrf_graph_type& mrf,\n       const size_t max_tree_width,\n       const size_t max_factor_size) {\n  typedef mrf_graph_type::edge_id_type edge_id_type;\n  typedef mrf_graph_type::vertex_id_type vertex_id_type;\n  // sanity check: The vertex to eliminate should not have already\n  // been eliminated\n  ASSERT_FALSE( contains(elim_vertex) );\n  /// =====================================================================\n  // Construct the elimination clique for the new vertex\n  // 1) Fill out clique\n  // 2) Track the corresponding factor size and treewidth\n  // 3) Find the parent of this clique\n  jtree_list::elim_clique clique;\n  clique.elim_vertex = elim_vertex;\n  // the factor must at least have the eliminated vertex\n  size_t factor_size = \n    std::max(mrf.vertex_data(elim_vertex).variable.size(),\n             uint32_t(1));\n  foreach(const edge_id_type ineid, mrf.in_edge_ids(elim_vertex)) {\n    const vertex_id_type vid = mrf.source(ineid);\n    const bool is_in_jtree = contains(vid);\n    // if the neighbor is in the set of vertices being eliminated\n    if(is_in_jtree) {      \n      clique.vertices += vid;\n      factor_size *= \n        std::max(mrf.vertex_data(vid).variable.size(), uint32_t(1) );\n    }\n    // if the clique ever gets too large then teminate\n    // the + 1 is because we need to include the elim vertex\n    if((clique.vertices.size() > max_tree_width) || \n       (max_factor_size > 0 && factor_size > max_factor_size)) \n      return false;\n  }\n\n  // Determine the parent of this clique -------------------------\n  vertex_id_type parent_id = 0;\n  foreach(vertex_id_type vid, clique.vertices)\n    parent_id = std::max(parent_id, elim_time_lookup(vid));\n  clique.parent = parent_id;\n\n  /// =====================================================================\n  // Simulate injecting vertices in parent cliques back to when RIP is\n  // satisfied\n  vertex_set rip_verts = clique.vertices;\n  for(vertex_id_t parent_vid = clique.parent; \n      !rip_verts.empty() && parent_vid < cliques.size(); ) {\n    const jtree_list::elim_clique& parent_clique = cliques[parent_vid];    \n    \n    // Remove the parent vertex\n    rip_verts -= vertex_set(parent_clique.elim_vertex);\n\n    // Construct the new vertices that would normally be stored at\n    // this vertes\n    const vertex_set tmp_verts = rip_verts + parent_clique.vertices;\n\n    // Check that the expanded clique is still within tree width\n    if(tmp_verts.size()  > max_tree_width) return false;\n\n    // If we care about the maximum factor size Compute the factor\n    // size and fail if the factor is too large\n    if(max_factor_size > 0) {\n      size_t factor_size = \n        std::max(mrf.vertex_data(parent_clique.elim_vertex).variable.size(),\n                 uint32_t(1));\n      foreach(vertex_id_t vid, tmp_verts) {\n        factor_size *= \n          std::max(mrf.vertex_data(vid).variable.size(), uint32_t(1));\n      }\n      if(factor_size > max_factor_size) return false;\n    }\n\n    // Find the new parent\n    vertex_id_t new_parent_vid = 0;\n    foreach(vertex_id_t vid, tmp_verts) {\n      new_parent_vid = \n        std::max(new_parent_vid, elim_time_lookup(vid));\n    }\n    // if the parent changes then we may need to update RIP with\n    // tmp_verts otherwise we use rip_verts\n    if(new_parent_vid != parent_clique.parent) \n      rip_verts = tmp_verts;\n    else\n      rip_verts -= parent_clique.vertices;\n    \n    // move up the tree\n    parent_vid = new_parent_vid;\n  }\n\n\n  /// =====================================================================\n  // Assert that if we reached this point RIP can be satisfied safely\n  // so proceed to update local data structures\n  const size_t new_elim_time = cliques.size();\n  cliques.push_back(clique);\n  elim_time[clique.elim_vertex] = new_elim_time;\n\n  /// =====================================================================\n  // Satisfy RIP\n  rip_verts = clique.vertices;\n  for(vertex_id_t parent_vid = clique.parent; \n      !rip_verts.empty() && parent_vid < cliques.size(); ) {\n    // get the parent clique\n    jtree_list::elim_clique& parent_clique = cliques[parent_vid];       \n\n    // otherwise update that the rip_verts\n    rip_verts -= vertex_set(parent_clique.elim_vertex);\n\n\n    // Construct the new vertices that would normally be stored at\n    // this vertes\n    const vertex_set tmp_verts = \n      rip_verts + parent_clique.vertices;\n\n    // Determine the new parent (except first vertex)\n    vertex_id_t new_parent_vid = 0;\n    foreach(vertex_id_t vid, tmp_verts) {\n      new_parent_vid = \n        std::max(new_parent_vid, elim_time_lookup(vid));\n    }\n\n    //! if the parent changes we must update the rip_verts and the parent value\n    if(new_parent_vid != parent_clique.parent) { \n      rip_verts = tmp_verts;\n      parent_clique.parent = new_parent_vid;\n    } else {\n      // If the parent is unchanged then we can remove all the\n      // variables stored locally from the rip_verts since they all\n      // already satisfy RIP.\n      rip_verts -= parent_clique.vertices;\n    }\n    // update the local vertices\n    parent_clique.vertices = tmp_verts;\n\n    // Move up tree\n    parent_vid = new_parent_vid;\n  }\n\n  // Ensure that the parent of the first clique is the null VID\n  cliques.front().parent = NULL_VID;\n  // Add successfully\n  return true;\n} // end of extend clique list\n\n\n\n\n/**\n * Convert a jtree_list into a jtree_graph \n */\nvoid jtree_list::\nload_graph(const mrf_graph_type& mrf,\n           const size_t num_factors,\n           jtree_graph_type& jt_graph) const {\n  //! Todo: Turn this into stack allocated boolean vector\n  std::vector assigned_factors(num_factors, false);\n  \n\n  {  // Construct the junction tree\n    // Ensure that the parent of the root is identifiable\n    ASSERT_EQ(cliques.front().parent, NULL_VID); \n    foreach(const jtree_list::elim_clique& clique, cliques) {      \n      const mrf_vertex_data& elim_vertex_vdata = \n        mrf.vertex_data(clique.elim_vertex);\n      // Create the vertex data\n      jtree_vertex_data vdata;\n      // set the vertex parent\n      vdata.parent = clique.parent;\n      // add the eliminated vertex\n      vdata.variables = elim_vertex_vdata.variable;\n      // add all the other variables in the clique\n      foreach(vertex_id_t vid, clique.vertices) \n        vdata.variables += mrf.vertex_data(vid).variable;      \n      // Add the vertex to the junction tree\n      vertex_id_t child_id = jt_graph.add_vertex(vdata);\n      // get the cliques parent\n      vertex_id_t parent_id = clique.parent;\n      // Add the edge to parent if not root\n      if(parent_id != NULL_VID) {\n        // Get the parent vertex data\n        const jtree_vertex_data& parent_vdata =\n          jt_graph.vertex_data(parent_id);\n        jtree_edge_data edata;\n        edata.variables = \n          vdata.variables.intersect(parent_vdata.variables);\n        // Add the actual edges\n        jt_graph.add_edge(child_id, parent_id, edata);\n        jt_graph.add_edge(parent_id, child_id, edata);\n      }\n    } // end of for each\n  } // End of construct cliques\n\n\n  { // Assign factors \n    // Very important that these be assigned in reverse order\n    size_t jt_vid = jt_graph.num_vertices() - 1;\n    rev_foreach(const jtree_list::elim_clique& clique, cliques) {\n      ASSERT_LT(jt_vid, jt_graph.num_vertices());\n      jtree_vertex_data& jt_vdata = jt_graph.vertex_data(jt_vid--);\n      const mrf_vertex_data& mrf_vdata = mrf.vertex_data(clique.elim_vertex);\n      foreach(factor_id_t fid, mrf_vdata.factor_ids) {\n        if(!assigned_factors[fid]) {\n          jt_vdata.factor_ids.push_back(fid);\n          assigned_factors[fid] = true;\n        }\n      }\n    }\n  }\n} // end of build junction tree\n\n\n\nstruct jtgraph_node {\n  vertex_id_t vid;\n  std::set vars;\n  std::set neighbors;\n};\n\ntypedef std::vector jtgraph_type;\n\n\nstd::ostream& \noperator<<(std::ostream& out, const std::set& set) {\n  using namespace graphlab;\n  return out << set;\n  // out << \"{\";\n  // size_t i = 0; \n  // foreach(const vertex_id_t vid, set) {\n  //   out << vid;\n  //   if(i + 1 < set.size()) out << \", \";\n  //   i++;\n  // }\n  // out << \"}\";\n  // return out;\n}\n\nvoid test_rip(const jtgraph_type& graph) {\n  namespace gl = graphlab;\n  std::map > > edgedata;\n  std::vector< std::set > reachable(graph.size());\n  foreach(const jtgraph_node& node, graph) {\n    reachable[node.vid] = node.vars;\n    // initiailize the out edges;\n    foreach(vertex_id_t nvid, node.neighbors) {\n      ASSERT_NE(nvid, node.vid);\n      edgedata[node.vid][nvid] = node.vars;\n    }\n  }\n  rev_foreach(const jtgraph_node& node, graph) {\n    // Receive in reachable\n    foreach(vertex_id_t nvid, node.neighbors) {\n      reachable[node.vid].insert(edgedata[nvid][node.vid].begin(),\n                                 edgedata[nvid][node.vid].end());\n    }\n    // write out reachable\n    foreach(vertex_id_t nvid, node.neighbors) {\n      std::set tmpset =         \n        gl::set_difference(reachable[node.vid], edgedata[nvid][node.vid]);\n      edgedata[node.vid][nvid].insert(tmpset.begin(), tmpset.end());\n\n    }\n  }\n  foreach(const jtgraph_node& node, graph) {\n    // Receive in reachable\n    foreach(vertex_id_t nvid, node.neighbors) {\n      reachable[node.vid].insert(edgedata[nvid][node.vid].begin(),\n                                 edgedata[nvid][node.vid].end());\n    }\n    // write out reachable\n    foreach(vertex_id_t nvid, node.neighbors) {\n      std::set tmpset =         \n        gl::set_difference(reachable[node.vid], edgedata[nvid][node.vid]);\n      edgedata[node.vid][nvid].insert(tmpset.begin(), tmpset.end());\n    }\n  }\n\n  // Check the running intersection property\n  foreach(const jtgraph_node& node, graph) {\n    // std::cout << node.vid << \": \" << node.vars << std::endl     \n    //           << \"\\t\" << node.neighbors << std::endl;\n    std::set local_sep_set;\n    foreach(const vertex_id_t n1, node.neighbors) {\n      //      std::cout << \"\\t\" << n1 << \"--\" << edgedata[n1][node.vid] << std::endl;\n      foreach(const vertex_id_t n2, node.neighbors) {\n        if(n1 != n2) {\n          local_sep_set =\n            gl::set_union(local_sep_set,\n                          gl::set_intersect(edgedata[n1][node.vid],\n                                            edgedata[n2][node.vid]));\n        }\n      }\n    }\n    //    std::cout << \"\\t\" << local_sep_set << std::endl;\n    ASSERT_TRUE(gl::is_subset(local_sep_set, node.vars));\n  }\n  //  getchar();\n\n}\n\n\n/**\n * Scan the junction tree list to ensure that all invariants hold.\n */\nvoid jtree_list::\nvalidate() const {\n  jtgraph_type jtgraph(cliques.size());\n  // validate the junction tree list data structure\n  for(size_t i = 0; i < cliques.size(); ++i) {\n    const elim_clique& clique = cliques[i];\n    ASSERT_EQ(i, elim_time_lookup(clique.elim_vertex));\n    ASSERT_FALSE(clique.vertices.contains(clique.elim_vertex));\n    const bool is_root = (clique.parent == NULL_VID);\n    if(is_root) {\n      ASSERT_EQ(i, 0);\n    } else {\n      ASSERT_GT(i, 0);\n      // ensure that the parent is elimated later\n      ASSERT_LT(clique.parent, jtgraph.size());\n      const elim_clique& parent_clique = cliques[clique.parent];\n      ASSERT_TRUE(clique.vertices.contains(parent_clique.elim_vertex));\n      ASSERT_LE(clique.vertices, \n                parent_clique.vertices + parent_clique.elim_vertex);\n    }\n    // populate the jtgraph node\n    jtgraph_node& node(jtgraph[i]);\n    node.vid = i;\n    node.vars.insert(clique.elim_vertex);\n    node.vars.insert(clique.vertices.begin(), clique.vertices.end());\n    if(!is_root) {\n      node.neighbors.insert(clique.parent);   \n      jtgraph[clique.parent].neighbors.insert(i);\n    }\n  }\n  // test running intersection property.  \n  test_rip(jtgraph);\n}\n\n\n\n\n\n\n\n#include \n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/junction_tree.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef PGIBBS_JUNCTION_TREE_HPP\n#define PGIBBS_JUNCTION_TREE_HPP\n\n\n/**\n *\n * Represents a junction tree\n *\n *  \\author Joseph Gonzalez\n */\n\n// INCLUDES ===================================================================>\n\n\n\n\n\n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n\n\n#include \n\n\n#include \n#include \n\n\n#include \"factorized_model.hpp\"\n#include \"mrf.hpp\"\n\n\n\n\nstruct jtree_vertex_data; \nstruct jtree_edge_data;\n\ntypedef graphlab::graph< jtree_vertex_data, jtree_edge_data> jtree_graph_type;\n\n\nstruct jtree_vertex_data {\n  jtree_graph_type::vertex_id_type parent;\n  domain_t variables;\n  bool calibrated;\n  bool sampled;\n  std::vector factor_ids;\n  factor_t factor;\n  assignment_t asg;\n  \n  jtree_vertex_data() : parent(-1), calibrated(false), sampled(false) { }\n}; // End of vertex data\n\n\nstruct jtree_edge_data {\n  domain_t variables;\n  factor_t message;\n  bool calibrated;\n  bool received;\n  jtree_edge_data() : \n    calibrated(false), \n    received(false) { }\n}; // End of edge data\n\n\n\n//// Junction tree construction code\n//// =====================================================================>\n\n//! The fast set used in junction tree construction\ntypedef graphlab::small_set<2*MAX_DIM, jtree_graph_type::vertex_id_type> \nvertex_set;\n\n\n\n\nstruct jtree_list {\n  struct elim_clique {\n    //! The parent of this elim clique in the jtree_list\n    jtree_graph_type::vertex_id_type parent;\n    //! The vertex eliminated when this clique was created \n    mrf_graph_type::vertex_id_type elim_vertex;\n    //! The vertices created in this clique EXCLUDING elim_vertex\n    vertex_set vertices; \n    elim_clique() : parent(-1) { }\n  };\n  typedef std::vector clique_list_type;\n  typedef boost::unordered_map \n  elim_time_type;\n  \n  //! The collection of cliques\n  clique_list_type cliques;\n  //! the time variable i was eliminated\n  elim_time_type   elim_time;\n\n  inline bool contains(const mrf_graph_type::vertex_id_type vid) const {\n    return elim_time.find(vid) != elim_time.end();\n  }\n\n  inline mrf_graph_type::vertex_id_type\n  elim_time_lookup(const mrf_graph_type::vertex_id_type vid) const {\n    elim_time_type::const_iterator iter(elim_time.find(vid));\n    ASSERT_TRUE(iter != elim_time.end());\n    return iter->second;\n  }\n\n\n  inline void clear() {\n    cliques.clear();\n    elim_time.clear();\n  }\n\n  /**\n   * Extend the jtree_list data structure by eliminating the vertex.\n   * If the jtree list can be extended then it is extended and this\n   * function returns true.\n   *\n   **/\n  bool extend(const mrf_graph_type::vertex_id_type elim_vertex,\n              const mrf_graph_type& mrf,\n              const size_t max_tree_width,\n              const size_t max_factor_size);\n\n  /**\n   * Convert a jtree_list into a jtree_graph \n   */\n  void load_graph(const mrf_graph_type& mrf,\n                  const size_t num_factors,\n                  jtree_graph_type& jt_graph) const;\n\n  /**\n   * Check internal data structures\n   */\n  void validate() const;\n\n};\n\n\n\n\n\n\n// /**\n//  * Extend the jtree_list data structure by eliminating the vertex.  If\n//  * the jtree list can be extended then it is extended and this\n//  * function returns true.\n//  *\n//  **/\n// bool extend_jtree_list(const vertex_id_t elim_vertex,\n//                        const mrf_graph_type& mrf,\n//                        const size_t max_tree_width,\n//                        const size_t max_factor_size,\n//                        jtree_list& jt_list);\n\n\n\n\n\n\n\n// /**\n//  * Convert a jtree_list into a jtree_graph \n//  */\n// void jtree_list_to_jtree_graph(const jtree_list& jt_list,\n//                                const mrf_graph_type& mrf,\n//                                const size_t num_factors,\n//                                jtree_graph_type& jt_graph);\n\n\n\n// /**\n//  * Scan the junction tree list to ensure that all invariants hold.\n//  */\n// bool validate_jtree_list(const jtree_list& jt_list);\n\n\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/make_denoise_alchemy.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * Run parallel junction tree gibbs sampling on a factorized model\n */\n\n#include \n#include \n\n\n#include \n\n#include \"image.hpp\"\n#include \"factorized_model.hpp\"\n\n#include \n\n\n\n/** Construct denoising ising model based on the image */\nvoid construct_denoise_graph(image& img,\n                             size_t num_rings,\n                             double sigma,\n                             const std::string& corruption,\n                             factor_t edge_factor,\n                             factorized_model& model) {\n \n} // End of construct graph\n\n\n\n\nint main(int argc, char** argv) {\n  std::cout << \"make the image denoising alchemy problem\"\n            << std::endl;\n\n  std::string model_filename = \"image\";\n  std::string drawing = \"sunset\";\n  std::string corruption = \"gaussian\";\n  std::string smoothing = \"square\";\n  double lambda = 3;\n  double sigma = 1;\n  size_t rows = 200;\n  size_t rings = 7;\n  \n\n\n  \n\n  // Command line parsing\n  graphlab::command_line_options clopts(\"Make the alchemy image\", true);\n  clopts.attach_option(\"model\", \n                       &model_filename, model_filename,\n                       \"Alchemy formatted model file\");\n  clopts.attach_option(\"drawing\", \n                       &drawing, drawing,\n                       \"drawing type\");\n  clopts.attach_option(\"corruption\", \n                       &corruption, corruption,\n                       \"corruption type\");\n  clopts.attach_option(\"smoothing\", \n                       &smoothing, smoothing,\n                       \"smoothing type\");\n  clopts.attach_option(\"lambda\", \n                       &lambda, lambda,\n                       \"edge parameter\");\n  clopts.attach_option(\"sigma\", \n                       &sigma, sigma,\n                       \"noise parameter\");\n  clopts.attach_option(\"rows\", \n                       &rows, rows,\n                       \"number of rows and cols\");\n  clopts.attach_option(\"rings\", \n                       &rings, rings,\n                       \"number of rings\");\n\n  if( !clopts.parse(argc, argv) ) { \n    std::cout << \"Error parsing command line arguments!\"\n              << std::endl;\n    return EXIT_FAILURE;\n  }\n\n\n  \n  std::cout << \"Creating a synethic image.\" << std::endl;\n  image original(rows, rows);\n  if(drawing == \"sunset\") \n    original.paint_sunset(rings);\n  else if(drawing == \"checkerboard\")\n    original.paint_checkerboard(rings);\n  else {\n    std::cout << \"Invalid drawing type!\" << std::endl;\n    exit(1);\n  }\n  std::cout << \"Saving original image. \" << std::endl;\n  original.save(\"original.pgm\");    \n\n    \n  std::cout << \"Corrupting Image. \" << std::endl;\n  image noisy = original;\n  if(corruption == \"gaussian\") \n    noisy.gaussian_corrupt(sigma);\n  else if(corruption == \"flip\")\n    noisy.flip_corrupt(rings, 0.75);\n  else if(corruption == \"ising\") \n    noisy = image(rows, rows);\n  else {\n    std::cout << \"Invalid corruption type!\" << std::endl;\n    exit(1);\n  }\n  std::cout << \"Saving corrupted image. \" << std::endl;\n  noisy.save(\"corrupted.pgm\");\n  \n\n  // dummy variables 0 and 1 and num_rings by num_rings\n  std::cout << \"Creating edge factor\" << std::endl;\n  factor_t edge_factor(domain_t(variable_t(0, rings), variable_t(1, rings)));\n  // Set the smoothing type\n  if(smoothing == \"square\") {\n    edge_factor.set_as_agreement(lambda);\n  } else if (smoothing == \"laplace\") {\n    edge_factor.set_as_laplace(lambda);\n  } else  {\n    std::cout << \"Invalid smoothing stype!\" << std::endl;\n    assert(false);\n  }\n  std::cout << edge_factor << std::endl;\n  \n  std::cout << \"Constructing factor graph.\" << std::endl;\n  factorized_model model;\n  // Add all the node factors\n  double sigmaSq = sigma*sigma;\n  for(size_t i = 0; i < noisy.rows(); ++i) {\n    for(size_t j = 0; j < noisy.cols(); ++j) {\n      // initialize the potential and belief\n      uint32_t pixel_id = noisy.vertid(i, j);\n      variable_t var(pixel_id, rings);\n      factor_t factor(var);\n      // Set the node potential\n      double obs = noisy.pixel(i, j);\n      if(corruption == \"gaussian\") {\n        for(size_t pred = 0; pred < rings; ++pred) {\n          factor.logP(pred) = \n            -(obs - pred)*(obs - pred) / (2.0 * sigmaSq);\n        }\n      } else if(corruption == \"flip\") {\n        for(size_t pred = 0; pred < rings; ++pred) {\n          factor.logP(pred) = obs == pred? 0 : -sigma;\n        }\n      } else if(corruption == \"ising\") {\n        // Do nothing since we want a uniform node potential\n        factor.uniform();\n      } else {\n        std::cout << \"Invalid corruption!\" << std::endl;\n        exit(1);\n      }\n      factor.normalize();\n      model.add_factor(factor);\n    } // end of for j in cols\n  } // end of for i in rows\n\n  // Construct edge_factors  \n  for(size_t i = 0; i < noisy.rows(); ++i) {\n    for(size_t j = 0; j < noisy.cols(); ++j) {\n      size_t source = noisy.vertid(i,j);\n      variable_t source_var(source, rings);\n      if(i+1 < noisy.rows()) {\n        vertex_id_t target = noisy.vertid(i+1, j);\n        variable_t target_var(target, rings);\n        domain_t dom(source_var, target_var);\n        edge_factor.set_args(dom);\n        model.add_factor(edge_factor);\n      }\n      if(j+1 < noisy.cols()) {\n        vertex_id_t target = noisy.vertid(i, j+1);\n        variable_t target_var(target, rings);\n        domain_t dom(source_var, target_var);\n        edge_factor.set_args(dom);\n        model.add_factor(edge_factor);\n      }\n    } // end of for j in cols\n  } // end of for i in rows\n\n  std::cout << \"Saving model in alchemy format\" << std::endl;\n  model.save_alchemy(model_filename + \".alchemy\");\n\n\n  return EXIT_SUCCESS;\n} // end of main\n\n\n\n\n\n\n\n\n\n\n\n\n\n#include \n\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/matlab/Makefile",
    "content": "\ngibbs_sampler_impl: gibbs_sampler_impl.cpp matwrap.hpp\n\tmex -largeArrayDims \\\n\tCXXFLAGS=\"-g -fPIC -Wall -O3 -pthread -fexceptions -fno-omit-frame-pointer -fopenmp\" \\\n\tgibbs_sampler_impl.cpp \\\n\t-I../../../src \\\n\t-L../../../release/src/graphlab \\\n\t-L../../../release/src/graphlab/extern/metis/GKlib \\\n\t-L../../../release/src/graphlab/extern/metis/libmetis \\\n\t-L../../../release/demoapps/pgibbs \\\n\t-lpgibbs_pic \\\n\t-lgraphlab_pic \\\n\t-lgomp\t\n\n\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/matlab/asg2ind.m",
    "content": "function ndx = asg2ind(siz, asg)\nmultiple = [1, cumprod(siz(1:end-1))];\nassert(isempty(find(asg > siz, 1)));\nndx = sum(multiple .* (asg - 1)) + 1;\nend\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/matlab/compile_gibbs_sampler.m",
    "content": "%% Compile all mex files\n% This script compiles the mex files needed to run the parallel sampling\n% algorithms.\n\ngraphlab_path='../../..';\ngraphlab_bin_path=[graphlab_path, '/release'];\npgibbs_bin_path=[graphlab_bin_path, '/demoapps/pgibbs'];\ngraphlab_include_path = [graphlab_path, '/src'];\ngraphlab_link_path = [graphlab_bin_path, '/src/graphlab'];\ncxx_flags = ['CXXFLAGS=', ...\n   '\"-fPIC -Wall -O3 -pthread -fexceptions -fno-omit-frame-pointer ', ...\n   '-fopenmp\"'];\n\n \n%% If release folder does not exist run configure\nif(~exist(graphlab_bin_path, 'dir')) \n  disp('Configure was not yet run running config now.');\n  [errorstatus, result] = ...\n    system(['cd ', graphlab_path, ';', ' ./configure'])\n  if(errorstatus) \n    error('Error running config!');\n  end\nend\n\n%% Compile the pgibbs library needed for the mex file\n[errorstatus, result] = ...\n  system(['cd ', pgibbs_bin_path, ';', ' make -j2'])\nif(errorstatus) \n  error('Error compiling pgibbs!');\nend\n\n  \n%% Do the compilation\ncompiler_type_flags = '';\nif(ismac()) \n  disp('We require gcc 4.2 on mac');\n  compiler_type_flags = 'LD=gcc-4.2 CC=gcc-4.2 CXX=g++-4.2';\nend \n\ncompilestr = ...\n   ['mex ', compiler_type_flags, ' ', ...\n    '-largeArrayDims', ' ', ...\n    cxx_flags, ' ', ...\n    'gibbs_sampler_impl.cpp', ' ', ...\n    '-I', graphlab_include_path, ' ', ...\n    '-L', graphlab_link_path, ' ', ...\n    '-L', graphlab_link_path, '/extern/metis/GKlib', ' ', ...\n    '-L', graphlab_link_path, '/extern/metis/libmetis', ' ', ...\n    '-L', pgibbs_bin_path, ' ', ...\n    '-lpgibbs_pic', ' ', ...\n    '-lgraphlab_pic ', ' ', ...\n    '-lgomp'];\n\ndisp(compilestr);\neval(compilestr);\ndisp('Finished!');\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/matlab/gibbs_sampler.m",
    "content": "%% Parallel Gibbs sampler\n% The parallel gibbs sampler is an optimized a c++ implementation of\n% the discrete Gibbs samplers which uses multiple threads to\n% accelerate the generation of a single sampling chain.  The parallel\n% Gibbs sampler implements two algorithms described in the paper:\n%\n%   Parallel Gibbs Sampling: From Colored Fields to Think Junction Trees\n%     by Joseph Gonzalez, Yucheng Low, Arthur Gretton, and Carlos Guestrin\n%\n% The first algorithm is the Chromatic sampler which is a direct\n% parallelization of the classic Gibbs sampler.  The second algorithm\n% is the Splash Gibbs sampler which incrementally builds thin junction\n% trees.\n%\n% To use this function you must first construct a discrete factor\n% graph which is simply a cell array of table factors:\n%\n%   factor{1} = table_factor( [1,2], log(rand(3,4)) );\n%   factor{2} = table_factor( [2,3], log(rand(4,2)) );\n%\n% This creates a factorized model (with random tables) over the\n% variables 1, 2, and 3. We can then run the CHROMATIC sampler by\n% calling:\n%\n%   options.alg_type = 'CHROMATIC';\n%   options.nsamples = 100;\n%   options.nskip    = 10;\n%   [samples, nupdates, nchanges, marginals] = ...\n%      gibbs_sampler(factors, options); \n%\n%\n% \n% Arguments:\n%   factors: a cell array of factors constructed using the table_factor \n%     function.\n%   options: a struct with the following fields:\n%    * alg_type: [Default: 'CHROMATIC'] A string either 'CHROMATIC' or\n%        'SPLASH'.  For relatively fast mixing models the 'CHORMATIC'\n%        algorithm is simpler and faster.  For slowly mixing models\n%        use the 'SPLASH' algorithm.  In this case additional options\n%        will need to be set.\n%    * nsamples: [Default: 10] The number of joint samples to collect.\n%    * nskip: [Default: 10] The number of samples to skip between\n%        joint samples.  Because of the asynchronous nature of the\n%        algorithms more or than nskip samples may actually be skipped\n%        in practice.  In the 'SPLASH' algorithm nskip * nvariables\n%        single variable updates are computed before the next joint\n%        sample is constructed.\n%    * ncpus: [Default: 2] The number of threads to use when running\n%        the inference algorithm. The number of cpus should be less\n%        than the number of variables and ideally not much larger than\n%        the number of processors.\n%    * treewidth: [Default: 3] The treewidth of the junction trees\n%        constructed using the Splash sampler.\n%    * treeheight: [Default: maxint] The largest height of a tree\n%    * treesize: [Default: maxint] The largest possible size of a\n%        tree\n%    * priorities: [Default: false] Use priorities when\n%       constructing the splash trees\n%    * checkargs: [Default: True] Determines if the arguments are\n%        checked before calling the C++ code.  While we do additional\n%        argument checking withing the C++ code it is often easier to\n%        debug broken factors from within the matlab code.  However\n%        for the fastest performance disable checkargs (set to false).\n%\n% Return Arguments:\n%   samples: nvars * nsamples matrix of joint assingments\n%   nupdates: nvars * nsamples number of times each variable was updated.\n%   nchanges: nvars * nsamples the number of times the variable's \n%      assignment changed values\n%   beliefs: nvars * nsamples cell array of vectors represent the \n%      Rao-Blackwellized marginal estimates for each variable.\n%\n% See Also: table_factor\n%\n% This actual c++ mex function is provided in gibbs_sampler_impl.cpp\n% which can be compiled by running compile_gibbs_sampler.m.\n%\nfunction [samples, nupdates, nchanges, marginals] = ...\n      gibbs_sampler(factors, options)    \n\n  %% Check the arguments\n  if(~iscell(factors))\n    error('The factors argument must be a cell array of table_factors');\n  end\n  % Define default options\n  if(~exist('options', 'var'))\n    options.alg_type = 'CHROMATIC';\n  end\n  if(~isfield(options, 'alg_type'))\n    options.alg_type = 'CHROMATIC';\n  end\n  if(~isfield(options, 'nsamples'))\n    options.nsamples = 10;\n  end\n  if(~isfield(options, 'nskip'))\n    options.nskip = 10;\n  end\n  if(~isfield(options, 'ncpus'))\n    options.ncpus = 2;\n  end\n  if(~isfield(options, 'treewidth'))\n    options.treewidth = 3;\n  end\n  if(~isfield(options, 'treeheight'))\n    options.treeheight = double(intmax());\n  end\n  if(~isfield(options, 'treesize'))\n    options.treesize = double(intmax());\n  end\n  if(~isfield(options, 'priorities'))\n    options.priorities = false;\n  end\n  if(~isfield(options, 'vanish'))\n    options.vanish = 10;\n  end\n  if(~isfield(options, 'checkargs'))\n    options.checkargs = true;\n  end\n  if(~isfield(options, 'save_alchemy'))\n    options.save_alchemy = false;\n  end\n\n  options.treewidth = double(options.treewidth);\n  if(options.treewidth > 32) \n     error('Treewidth must be less than 32');\n  end\n  if(options.treewidth < 1) \n     error('Treewidth must be at least 1');\n  end\n\n  if(options.checkargs) \n    max_var = 0;\n    %% Check the factors data structure\n    for i = 1:length(factors)\n      if(~isfield(factors{i}, 'vars'))\n        disp(factors{i});\n        error('Factor %d does not contain the field vars', i);\n      end\n      if(~strcmp(class(factors{i}.vars), 'uint32'))\n        disp(factors{i});\n        error(['Factor ', num2str(i), ...\n               ' has variables of type ', ...\n               class(factors{i}.vars), ...\n               ' when they should be of type uint32.']);    \n      end\n      if(~isfield(factors{i}, 'logP'))\n        disp(factors{i});\n        error('Factor %d does not contain the field logP', i);\n      end\n      if(~strcmp(class(factors{i}.logP), 'double'))\n        disp(factors{i});\n        error(['Factor ', num2str(i), ...\n               ' has logP of type ', ...\n               class(factors{i}.logP), ...\n               ' when they should be of type double.']);    \n      end    \n      % Get the maximum variables\n      max_var = max(max(factors{i}.vars(:)), max_var);\n      if(min(factors{i}.vars) <= 0) \n        disp(factors{i});\n        error('Factor %d has 0 valued variables', i);\n      end   \n    end\n\n    %% check all the variables have consistent sizes;\n    vars = 1:max_var;\n    var_sizes = zeros(max_var, 1);\n    for i = 1:length(factors)\n      current_sizes = var_sizes(factors{i}.vars);\n      dims = size(factors{i}.logP)';\n      dims = dims(dims > 1);\n      if(length(dims) ~= length(current_sizes))\n        error(['The number of variables %d in factor %d does not match ' ...\n               'the number of dimensions %d.'], ...\n              length(current_sizes), i, length(dims));\n      end   \n      ind = current_sizes > 0;\n      % all elements in ind have been set and should just match\n      if( ~isempty(find(current_sizes(ind) ~= dims(ind), 1)) ) \n        errorind = find(current_sizes(ind) ~= dims(ind), 1);\n        error(['Variable %d has already been seen having size %d ', ...\n               'but was just now observed to have size %d in factor %d.'], ...\n              factors{i}.variables(ind(errorind)), ...\n              current_sizes(ind(errorind)), ...\n              dims(ind(errorind)), ...\n              i);\n      end\n      var_sizes(factors{i}.vars(~ind)) = dims(~ind);\n    end\n    unset_vars = find(var_sizes(:) == 0);\n    if(~isempty(unset_vars)) \n      error(['The following variables were not set correctly: ', ...\n             mat2str(unset_vars)]);   \n    end\n  end\n  \n  %% Call the sampler\n  if(nargout() <= 1)\n    samples = gibbs_sampler_impl(factors, options);\n  elseif(nargout() == 2)\n    [samples, nupdates] = gibbs_sampler_impl(factors, options);\n  elseif(nargout() == 3)\n    [samples, nupdates, nchanges] = ...\n      gibbs_sampler_impl(factors, options);\n  elseif(nargout() == 4)\n    [samples, nupdates, nchanges, marginals] = ...\n      gibbs_sampler_impl(factors, options);\n  end\n\nend\n    \n    \n    \n    "
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/matlab/gibbs_sampler_impl.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n/**\n * This file contains the code used to run the chromatic gibbs sampler\n * from matlab\n */\n\n#include \"mex.h\"\n\n#include \n\n#include \n\n\n\n#include \"../factorized_model.hpp\"\n#include \"../mrf.hpp\"\n#include \"../global_variables.hpp\"\n#include \"../junction_tree.hpp\"\n\n#include \"../chromatic_sampler.hpp\"\n#include \"../jt_splash_sampler.hpp\"\n\n\n\n#include \"matwrap.hpp\"\n\nusing namespace std;\n\n///////////////////////////////////////////////////////////////////////////\n// Struct fieldnames\n\nconst char* vars_field_name = \"vars\";\nconst char* logP_field_name = \"logP\";\nint vars_field_id = -1;\nint logP_field_id = -1;\n\n\n\n\n\n///////////////////////////////////////////////////////////////////////////\n\n\nstruct options {\n  enum { CHROMATIC, SPLASH } alg_type;\n  size_t nsamples;\n  //  double tskip;\n  size_t nskip;\n  size_t ncpus;\n\n  //  size_t ntrees;\n  size_t treewidth;\n  size_t treeheight;\n  size_t treesize;\n  bool priorities;\n  size_t vanish;\n  bool save_alchemy;\n  size_t ncpus_per_splash;\n\n  options(matwrap args =  matwrap(NULL)) :\n    alg_type(CHROMATIC),  nsamples(10), nskip(10), \n    // tskip(5), \n    ncpus(2),\n    // ntrees(ncpus),\n    treewidth(3), \n    treeheight(std::numeric_limits::max()), \n    treesize(std::numeric_limits::max()), \n    priorities(false),  \n    vanish(10),\n    save_alchemy(false),\n    ncpus_per_splash(1) {\n    if(args.is_null()) return;\n    safe_assert(args.is_struct(), \n                \"Additional arguments must be in a struct\");\n    { // parse the sampler algorithm type\n      matwrap arg(args.get_field(\"alg_type\"));\n      if(!arg.is_null()) {\n        const size_t str_len(255);\n        char sampler_type_c_str[str_len];\n        arg.as_string(sampler_type_c_str, str_len);\n        std::string sampler_type_str(sampler_type_c_str, arg.size());\n\n        if(sampler_type_str == \"CHROMATIC\") alg_type = CHROMATIC;\n        else if(sampler_type_str == \"SPLASH\") alg_type = SPLASH;\n        else {\n          char error_str[2*str_len];\n          std::sprintf(error_str, \"Invalid sampler type: (%s)\\n\", \n                       sampler_type_c_str);\n          mexErrMsgTxt(error_str);\n        }        \n      }\n    } // end of parse field name\n    { // parse the number of samples\n      matwrap arg(args.get_field(\"nsamples\"));\n      if(!arg.is_null()) {\n        nsamples = size_t(arg.get_double_array()[0]);    \n      }\n    } // end of parse field name\n    { // parse the number of skipped samples\n      matwrap arg(args.get_field(\"nskip\"));\n      if(!arg.is_null()) {\n        nskip = size_t(arg.get_double_array()[0]);    \n      }\n    } // end of parse field name\n    // { // parse the number of skipped samples\n    //   matwrap arg(args.get_field(\"tskip\"));\n    //   if(!arg.is_null()) {\n    //     tskip = arg.get_double_array()[0];    \n    //   }\n    // } // end of parse field name                        \n    { // parse the number of cpus\n      matwrap arg(args.get_field(\"ncpus\"));\n      if(!arg.is_null()) {\n        ncpus = size_t(arg.get_double_array()[0]);    \n      }\n    } // end of parse field name\n    // { // parse the number of trees\n    //   matwrap arg(args.get_field(\"ntrees\"));\n    //   if(!arg.is_null()) {\n    //     ntrees = size_t(arg.get_double_array()[0]);    \n    //   }\n    // } // end of parse field name\n    { // parse the treewidth\n      matwrap arg(args.get_field(\"treewidth\"));\n      if(!arg.is_null()) {\n        treewidth = size_t(arg.get_double_array()[0]);    \n      }\n    } // end of parse field name\n    { // parse the treeheight\n      matwrap arg(args.get_field(\"treeheight\"));\n      if(!arg.is_null()) {\n        treeheight = size_t(arg.get_double_array()[0]);    \n      }\n      ASSERT_GT(treeheight, 0);\n    } // end of parse field name\n    { // parse the treesize\n      matwrap arg(args.get_field(\"treesize\"));\n      if(!arg.is_null()) {\n        treesize = size_t(arg.get_double_array()[0]);    \n      }\n      ASSERT_GT(treesize, 0);\n    } // end of parse field name\n    { // parse the priorities\n      matwrap arg(args.get_field(\"priorities\"));\n      if(!arg.is_null()) {\n        priorities = bool(arg.get_double_array()[0]);    \n      }\n    } // end of parse field name                         \n    { // parse the vanish\n      matwrap arg(args.get_field(\"vanish\"));\n      if(!arg.is_null()) {\n        vanish = size_t(arg.get_double_array()[0]);    \n      }\n    } // end of parse field name\n    { // parse the ncpus_per_splash\n      matwrap arg(args.get_field(\"npcus_per_splash\"));\n      if(!arg.is_null()) {\n        ncpus_per_splash = size_t(arg.get_double_array()[0]);    \n      }\n    } // end of parse field name                         \n    { // parse the ncpus_per_splash\n      matwrap arg(args.get_field(\"save_alchemy\"));\n      if(!arg.is_null()) {\n        save_alchemy = bool(arg.get_double_array()[0]);    \n      }\n    } // end of parse field name                         \n\n  } // end of constructor\n\n  void print() {\n    std::cout << \"Generating \"\n              << nsamples << \" samples \" << std::endl;\n    std::cout << \"Skipping every \" << nskip << \" samples.\" << std::endl;\n    switch(alg_type) {\n    case CHROMATIC: mexPrintf(\"Using the chromatic sampler.\\n\"); break;\n    case SPLASH: mexPrintf(\"Using the splash sampler.\\n\"); \n      std::cout << \"ncpus:        \"  << ncpus << std::endl\n        //                << \"ntrees:       \"  << ntrees << std::endl\n                << \"treewidth:    \"  << treewidth << std::endl\n                << \"treeheight:   \" << treeheight << std::endl\n                << \"prioritizeds:   \" << (priorities ? \"enabled\" : \"disabled\")\n                << std::endl\n                << \"ncpus/splash: \" << ncpus_per_splash << std::endl;\n\n      break;\n    default: mexErrMsgTxt(\"No algorithm selected!\\n\"); break;\n    }\n    flush_screen();\n  }\n\n}; // end of options struct\n\n\n\n\n\n\n\n\n\n///////////////////////////////////////////////////////////////////////////\n// Factor Graph generation code\n\n/**\n * Build a graphlab table factor from the matlab table factor\n */\nvoid add_factor(factorized_model& model,\n                matwrap matlab_factor) {                \n  if(matlab_factor.is_null()) {\n    mexErrMsgTxt(\"Null factor argument to build factor!\\n\");    \n  }\n  if(!matlab_factor.is_struct()) {\n    mexErrMsgTxt(\"Invalid factor type\");\n  }\n  // Get the field numbers\n  if(vars_field_id < 0) {\n    vars_field_id = matlab_factor.get_field_number(vars_field_name);\n  }\n  if(logP_field_id < 0) {\n    logP_field_id = matlab_factor.get_field_number(logP_field_name);\n  }\n  safe_assert(vars_field_id >= 0, \"No field named vars in factor struct!\");\n  safe_assert(logP_field_id >= 0, \"No field named logP in factor struct!\");\n\n  // Load the members from the factor\n  matwrap variables = matlab_factor.get_field(vars_field_id);\n  matwrap logP = matlab_factor.get_field(logP_field_id);\n\n  // Check the fields \n  safe_assert(variables.is_uint32(), \"Variables are not uint32t\");\n  safe_assert(logP.is_double(), \"logP must be of type double\");\n\n  // Get information about each field\n  const mwSize num_vars = variables.size();\n  const mwSize* var_dims = logP.get_dimensions();\n  const mwSize num_dims = logP.get_num_dimensions();\n  safe_assert(num_vars <= MAX_DIM,\n              \"Too many (>32) variables in factor.\");\n\n\n  // Build up the domain\n  domain_t dom;\n  const uint32_t* varids = variables.get_data();\n  for(size_t i = 0, j = 0; i < num_vars && j < num_dims; ++i, ++j) {\n    // Skip all the empty dimensions\n    while(var_dims[j] <= 1 && j < num_dims) j++;\n    safe_assert(j < num_dims, \"Factor dimensions do not match variables\");\n    safe_assert(varids[i] > 0, \"Variabile ids must start at 1\");\n    variable_t var(varids[i]-1, var_dims[j]);\n    dom += var;\n  }\n\n  //  Create the factor\n  factor_t& factor(model.add_factor(dom));\n  const size_t num_logP(logP.size());\n  const double* data(logP.get_double_array());\n  safe_assert(num_logP == dom.size(), \n          \"Insufficient factor data to match variables\");\n  for(size_t i = 0; i < num_logP; ++i) {\n    factor.logP(i) = data[i];\n  }\n  factor.normalize();\n\n}\n\n\nvoid build_factorized_model(factorized_model& model, matwrap factors) {\n  const size_t num_factors(factors.size());\n  model.reserve(num_factors);\n  // Load all the factors\n  for(size_t i = 0; i < num_factors; ++i) {\n    add_factor(model, factors.get_cell(i));   \n  }\n}\n\n\n\n\n\n\n\n///////////////////////////////////////////////////////////////////////////\n// Code for sample collection\nstruct result_collector {\n  size_t sample_id;\n  matwrap samples;\n  matwrap beliefs;\n  matwrap nsamples; \n  matwrap nchanges; \n \n  result_collector(matwrap samples = NULL, \n                   matwrap beliefs = NULL,\n                   matwrap nsamples = NULL,\n                   matwrap nchanges = NULL) :\n    sample_id(0), samples(samples), beliefs(beliefs), \n    nsamples(nsamples), nchanges(nchanges)  { }\n};\n\ngraphlab::glshared glshared_collector;\n\nvoid collector_sync(mrf_gl::iscope& scope,\n                    graphlab::any& accumulator) {\n  result_collector collector = glshared_collector.get_val();\n  const size_t& sample_id = collector.sample_id;\n  const mrf_vertex_data& vdata(scope.const_vertex_data());\n  if(!collector.samples.is_null() && \n     sample_id < collector.samples.cols()) {\n    collector.samples.mat_index2d(vdata.variable.id(), \n                                  sample_id) = vdata.asg + 1;     \n  }\n  if(!collector.beliefs.is_null() && \n     sample_id < collector.beliefs.cols()) {\n    matwrap blf(collector.beliefs.\n                       get_cell_index2d(vdata.variable.id(), sample_id));\n    double sum = 0;\n    for(size_t i = 0; i < vdata.variable.size(); ++i) {\n      sum += (blf.get_double_array()[i] = exp(vdata.belief.logP(i)));\n    }\n    for(size_t i = 0; i < vdata.variable.size(); ++i) {\n      blf.get_double_array()[i] /= sum;\n    }\n  }\n  if(!collector.nsamples.is_null() && \n     sample_id < collector.nsamples.cols()) {\n    collector.nsamples.mat_index2d(vdata.variable.id(),\n                                   sample_id) = vdata.nsamples;\n  }\n  if(!collector.nchanges.is_null() && \n     sample_id < collector.nchanges.cols()) {\n    collector.nchanges.mat_index2d(vdata.variable.id(),\n                                   sample_id) = vdata.nchanges;\n  }\n}\n\nvoid collector_apply(graphlab::any& current_data, \n                     const graphlab::any& param) {\n  result_collector& collector = current_data.as();\n  collector.sample_id++;\n}\n\nvoid collector_merge(graphlab::any& merge_dest, \n                     const graphlab::any& merge_src) {\n  // nop\n}\n\n\nsize_t global_termination_nsamples = 0;\nbool nsamples_terminator() {\n  return glshared_collector.get_val().sample_id >= \n    global_termination_nsamples;\n}\n\n\n\n\n\n\n\n\n\nvoid run_chromatic_sampler(mrf_gl::core& core, \n                           const options& opts) {\n  \n  core.sched_options().add_option(\"update_function\", \n                                  single_gibbs_update);\n  // core.sched_options().add_option(\"max_iterations\", \n  //                                 opts.nsamples * opts.nskip);\n  global_termination_nsamples = opts.nsamples;\n  core.engine().add_terminator(nsamples_terminator);\n  std::cout << \"Running.\" << std::endl;\n  flush_screen();\n  double runtime = core.start();    \n  std::cout << \"Runtime: \" << runtime << std::endl;\n  flush_screen();\n}  // end of run_chromatic_sampler\n\n\n\nvoid run_jt_splash_sampler(mrf_gl::core& core, \n                           const options& opts) {\n\n  std::cout << \"Starting Splash Sampler.\" << std::endl;\n  flush_screen();\n  splash_settings settings;\n  settings.ntrees           = opts.ncpus;\n  settings.max_tree_width   = opts.treewidth;\n  settings.max_tree_height  = opts.treeheight;\n  settings.max_tree_size    = opts.treesize;\n  settings.priorities       = opts.priorities;\n  settings.subthreads       = opts.ncpus_per_splash;\n\n  jt_splash_sampler jtsplash_sampler(core.graph(),\n                                     settings);\n  \n  const size_t samples_per_iteration = \n    core.graph().num_vertices() * opts.nskip;\n  for(size_t i = 0; i < opts.nsamples; ++i) {\n    //    std::cout << \"Running an iteration\" << std::endl;\n    flush_screen();\n    // run the splash sampler\n    jtsplash_sampler.sample_updates(samples_per_iteration);\n    // jtsplash_sampler.sample_seconds(opts.tskip);\n    // compute the sync\n    core.sync_now(glshared_collector);\n    // std::cout << \"Ntrees: \" << jtsplash_sampler.total_trees() \n    //           << std::endl\n    //           << \"Nsamples: \" << jtsplash_sampler.total_samples()\n    //           << std::endl\n    //           << \"NCollisions: \" << jtsplash_sampler.total_collisions()\n    //           << std::endl;\n    \n  }\n  \n} // end of run splash sampler\n\n\n\n\n\n/**\n * See parallel_gibbs.m for identical arguments\n * \n *   [samples, nupdates, nchanges, marginals] = ...\n *     gibbs_sampler(factors, options); \n * \n */\nvoid mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) {\n  safe_assert(nrhs > 0, \n              \"Invalid number of arguments. See documentation for details\");  \n\n  { // Seed the random number generator using rand from within matlab\n    mxArray* plhs[1];\n    mxArray* prhs[4];\n    prhs[0] = mxCreateNumericMatrix(1,1, mxUINT32_CLASS, mxREAL);\n    prhs[1] = mxCreateNumericMatrix(1,1, mxUINT32_CLASS, mxREAL);\n    prhs[2] = mxCreateNumericMatrix(1,1, mxUINT32_CLASS, mxREAL);\n    matwrap(prhs[0]).get_data()[0] = \n      std::numeric_limits::max();\n    matwrap(prhs[1]).get_data()[0] = 1;\n    matwrap(prhs[2]).get_data()[0] = 2;\n    prhs[3] = mxCreateString(\"uint32\");\n    mexCallMATLAB(1, plhs, 4, prhs, \"randi\");\n    const size_t seed_value = \n      matwrap(plhs[0]).get_data()[0];\n    //    std::cout << \"Seed value: \" << seed_value << std::endl;\n    mxDestroyArray(plhs[0]);\n    mxDestroyArray(prhs[0]); \n    mxDestroyArray(prhs[1]); \n    mxDestroyArray(prhs[2]);\n    mxDestroyArray(prhs[3]);\n    graphlab::random::seed(seed_value);\n  }\n\n\n  // Get first argument the cell array of factors\n  const matwrap matlab_factors(const_cast(prhs[0]));\n  if(matlab_factors.is_null()) { \n    mexErrMsgTxt(\"No factors provided!\\n\"); \n  }\n  if(!matlab_factors.is_cell()) { \n    mexErrMsgTxt(\"Factors must be in cell array form!\\n\"); \n  }\n  \n  options opts;\n  if(nrhs > 1) opts = options(const_cast(prhs[1]));\n  opts.print();\n \n\n\n  // Load the factorized model\n  factorized_model model;\n  build_factorized_model(model, matlab_factors);\n  std::cout << \"Finished Loading Factors\" << std::endl;\n  flush_screen();\n  if(opts.save_alchemy) {\n    std::cout << \"Saving Alchemy file \\\"problem.alchemy\\\"\" << std::endl;\n    model.save_alchemy(\"problem.alchemy\");\n  }\n\n  // mexPrintf(\"Finished Saving Alchemy File\\n\");\n\n  // Set the global factors\n  SHARED_FACTORS_PTR = &(model.factors());\n\n\n  // Construct the markov random field\n  mrf_gl::core mrf_core;\n  mrf_from_factorized_model(model, mrf_core.graph());    \n\n  if(opts.alg_type == options::CHROMATIC) {\n    const size_t ncolors(mrf_core.graph().compute_coloring());  \n    std::cout << \"Finished coloring graph with \" << ncolors \n              << \" colors.\" << std::endl;\n    flush_screen();\n    mrf_core.set_ncpus(opts.ncpus);\n    mrf_core.set_scheduler_type(\"chromatic\");\n    mrf_core.set_scope_type(\"null\");\n    mrf_core.engine().set_sched_yield(false);\n  }\n\n\n  // allocate the return matrix for samples\n  matwrap matlab_samples;\n  if(nlhs > 0) {\n    matlab_samples = \n      matwrap::create_matrix(model.variables().size(),\n                                    opts.nsamples);\n    plhs[0] = matlab_samples.array;\n    safe_assert(!matlab_samples.is_null(), \n                \"Error initializing return samples\");\n  }\n  if( !matlab_samples.is_null() ) {\n    double* entries(matlab_samples.get_double_array());\n     const size_t num_entries(matlab_samples.size());\n    for(size_t i = 0; i < num_entries; ++i) {\n      entries[i] = std::numeric_limits::quiet_NaN();\n    }\n  }\n\n\n  // allocate the return matrix for nsamples\n  matwrap matlab_nsamples;\n  if(nlhs > 1) {\n    matlab_nsamples = \n      matwrap::create_matrix(model.variables().size(),\n                                    opts.nsamples);\n    plhs[1] = matlab_nsamples.array;\n    safe_assert(!matlab_nsamples.is_null(), \n                \"Error initializing return nsamples\");\n  }\n  if( !matlab_nsamples.is_null() ) {\n    double* entries(matlab_nsamples.get_double_array());\n     const size_t num_entries(matlab_nsamples.size());\n    for(size_t i = 0; i < num_entries; ++i) {\n      entries[i] = std::numeric_limits::quiet_NaN();\n    }\n  }\n\n  // allocate the return matrix for nsamples\n  matwrap matlab_nchanges;\n  if(nlhs > 2) {\n    matlab_nchanges = \n      matwrap::create_matrix(model.variables().size(),\n                                    opts.nsamples);\n    plhs[2] = matlab_nchanges.array;\n    safe_assert(!matlab_nchanges.is_null(), \n                \"Error initializing return nchanges\");\n  }\n  if( !matlab_nchanges.is_null() ) {\n    double* entries(matlab_nchanges.get_double_array());\n     const size_t num_entries(matlab_nchanges.size());\n    for(size_t i = 0; i < num_entries; ++i) {\n      entries[i] = std::numeric_limits::quiet_NaN();\n    }\n  }\n\n\n  // allocate the return cell array for beliefs\n  matwrap matlab_beliefs;\n  if(nlhs > 3) {\n    matlab_beliefs = \n      matwrap::create_cell(model.variables().size(),\n                                  opts.nsamples);\n    plhs[3] = matlab_beliefs.array;\n    safe_assert(!matlab_beliefs.is_null(), \n                \"Error initializing return beliefs\");\n  }\n  if( !matlab_beliefs.is_null() ) {\n    //populate each of the entries\n    for(size_t i = 0; i < mrf_core.graph().num_vertices(); ++i) {\n      const mrf_vertex_data& vdata = \n        mrf_core.graph().vertex_data(i);      \n      for(size_t j = 0; j < opts.nsamples; ++j) {\n        matwrap blf = \n          matwrap::create_matrix(vdata.variable.size(),\n                                        1);\n        safe_assert(!blf.is_null(), \"Unable to allocate beliefs\");\n        matlab_beliefs.set_cell_index2d(i, j, blf);\n      }\n    }\n  }\n\n\n\n\n  //! initialize results collection\n  glshared_collector.set(result_collector(matlab_samples, \n                                          matlab_beliefs, \n                                          matlab_nsamples, \n                                          matlab_nchanges));\n\n\n  // if(!matlab_samples.is_null()) {\n  //   std::cout << \"Enabling sample collection.\" << std::endl;\n  //   flush_screen();\n    //    last_tic = graphlab::lowres_time_seconds();\n  const size_t sync_interval = \n    opts.nskip * mrf_core.graph().num_vertices();\n  mrf_core.set_sync(glshared_collector,\n                    collector_sync,\n                    collector_apply,\n                    graphlab::any(size_t(0)),\n                    sync_interval,\n                    collector_merge);\n    //}\n\n\n\n  if(opts.alg_type == options::CHROMATIC) {\n    run_chromatic_sampler(mrf_core, opts);\n  } else if( opts.alg_type == options::SPLASH) {\n    run_jt_splash_sampler(mrf_core, opts);\n  } else {\n    mexErrMsgTxt(\"Invalid algorithm type.\\n\");\n  }\n\n\n   \n\n} // end of main\n\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/matlab/ind2asg.m",
    "content": "function asg = ind2asg(siz, ndx)\nn = length(siz);\nasg = zeros(1,n);\nndx = ndx - 1;\nfor i = 1:n\n  asg(i) = mod(ndx, siz(i));\n  ndx = floor(ndx / siz(i));\nend\nasg = asg + 1;\nend\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/matlab/matwrap.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef MATWRAP\n#define MATWRAP\n\n#include \n#include \n\n#include \"mex.h\"\n\n\n\n \n/**\n * A safe assertion for matlab\n */\nvoid safe_assert(const bool value, const char* msg) {\n  if(!value) { mexErrMsgTxt(msg); }\n  //  if( __builtin_expect(!value, 0) ) { mexErrMsgTxt(msg); }\n};\n\n\nvoid flush_screen() {\n  mexEvalString(\"drawnow\");\n}\n\n\n/**\n * A convenient wrapper around mxArray objects\n */\nstruct matwrap {\n\n  mxArray* array;\n  \n  matwrap(mxArray* array = NULL) : array(array) {  }\n\n\n\n  bool is_null() const { return array == NULL; }\n\n  matwrap get_property(const char* property) const { \n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    mxArray* result(mxGetProperty(array, 0, property));\n    if(result == NULL) {\n      char buffer[256];\n      sprintf(buffer, \"Invalid property %s\\n\", property);\n      mexErrMsgTxt(buffer);\n    }\n    return matwrap(result);\n  } // end of get property\n\n  \n  template  T* get_data() {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    return reinterpret_cast(mxGetData(array));\n  }\n\n  double* get_double_array() {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    return mxGetPr(array);\n  }\n\n  double& mat_index2d(const size_t i, const size_t j) {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    return mxGetPr(array)[ i + j*rows()];    \n  }\n\n\n  void set_cell_index2d(const size_t i, const size_t j,\n                        matwrap contents) {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    mxSetCell(array, i + j*rows(), contents.array);    \n  }\n\n  matwrap get_cell_index2d(const size_t i, const size_t j) {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    return mxGetCell(array, i + j*rows());\n  }\n\n\n  matwrap get_field(const char* fieldname) const {\n    safe_assert(is_struct(),\n                \"Attempted to access field of a non-struct element.\");\n    return mxGetField(array,0,fieldname);\n  }\n\n  int get_number_of_fields() const { return  mxGetNumberOfFields(array); }\n\n  matwrap get_field(const int field_id) const {\n    safe_assert(is_struct(),\n                \"Attempted to access field of a non-struct element.\");\n    safe_assert(field_id < get_number_of_fields(), \"Invalid field id!\");\n    return mxGetFieldByNumber(array,0,field_id);\n  }\n\n  int get_field_number(const char* field_name) const {\n    safe_assert(is_struct(),\n                \"Attempted to access field of a non-struct element.\");\n    return mxGetFieldNumber(array, field_name);\n  }\n\n  bool is_class(const char* classname) const {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    safe_assert(classname != NULL, \"Invalid classname argument\");\n    return mxIsClass(array, classname);     \n  } // end of is class\n\n  const char* get_classname() const {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    return mxGetClassName(array);\n  } // end of is class\n\n\n  size_t size() const {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    return mxGetNumberOfElements(array);\n  }\n\n  matwrap get_cell(size_t index) const {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    safe_assert(mxGetClassID(array) == mxCELL_CLASS,\n                \"Attempted to access a cell in a non-cell array.\");\n    return mxGetCell(array, index);\n  }  \n\n  bool is_cell() const {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    return mxIsCell(array);\n  }\n\n  bool is_struct() const {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    return mxIsStruct(array);\n  }\n\n  bool is_double() const {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    return mxGetClassID(array) == mxDOUBLE_CLASS;\n  }\n\n\n  bool is_uint32() const {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    return mxGetClassID(array) == mxUINT32_CLASS;\n  }\n\n  bool is_string() const {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    return mxGetClassID(array) == mxCHAR_CLASS;\n  }\n\n\n  void as_string(char* str_buffer, size_t buffer_len) {\n    safe_assert(str_buffer != NULL, \"NULL string buffer\");\n    int error = mxGetString(array, str_buffer, buffer_len);\n    safe_assert(!error, \"Error processing string!\");\n  }\n\n  const mwSize* get_dimensions() const {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    return mxGetDimensions(array);\n  }\n\n  size_t rows() const {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    return mxGetM(array);\n  }\n\n  size_t cols() const {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    return mxGetN(array);\n  }\n\n\n  const mwSize get_num_dimensions() const {\n    safe_assert(array != NULL, \"dereferenced null mxArray\");\n    return mxGetNumberOfDimensions(array);\n\n  }\n\n\n\n\n\n\n\n\n\n\n\n  static matwrap create_matrix(size_t m, size_t n) {\n    return matwrap(mxCreateDoubleMatrix(m, n, mxREAL));\n  }\n\n  static matwrap create_cell(size_t m, size_t n) {\n    return matwrap(mxCreateCellMatrix(m, n));\n  }\n\n\n\n\n  \n};\n\n\n\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/matlab/table_factor.m",
    "content": "%% Construct a discrete table factor\n%   \n%   factor = table_factor(vars, logP)\n%\n% vars: array of variable ids (e.g., [1,2,4] )\n% logP: tensor representing the log potential values (e.g., ones(3,7,2)\n%    where variable 1 takes on 3 states variable 2 takes on 7 states and\n%    variable 4 takes on 2 states.\n%   \n% A table factor represents a factor or potential over a small set of\n% discrete variables.  Forexample if we wanted to encode a similarity\n% funciton over a pair of variables x1 and x2 we could define the table\n% factor:\n%\n%    psi(x1,x2) = exp( |x1 - x2| )\n% \n% Assuming x1 and x2 take on 4 and 3 values respectively we could define\n% the matrix (table) representation of psi:\n%    \n%              0  1  2 \n%  tbl = exp(  1  0  1  )\n%              2  1  0\n%              3  2  1\n% \n% We can build a table factor representing this as:\n%\n%  factor = table_factor([1, 2], tbl);\n%\nfunction factor = table_factor(vars, data) \nfactor.vars = sort(uint32(vars));\nfactor.logP = double(data);\nend\n\n%%\n% Originally I had hoped to used a matlab class but unfortunatley mex\n% support for classes is limited resulting in substantial performance\n% penalties when accessing fields.\n\n% classdef table_factor\n%   properties (SetAccess = private)\n%     variables;\n%   end\n%   properties\n%     logP;\n%   end\n%   methods\n%     %% the variables should be a d dimensional array\n%     function obj = table_factor(vars, data)\n%       obj.variables = uint32(vars);\n%       obj.logP = double(data);\n%     end\n%     \n%   end\n%   \n% end\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/matlab/tests/denoise_test.m",
    "content": "clear;\n%% Build the factors\nrows = 100;\ncols = 100;\nstates = 5;\nlambdaSmooth = 1.5; % Laplace smoothing parameter\nnoiseP = 0.3; % proportion of randomly sampled values\n[factors, img, noisy_img] = ...\n  make_grid_model(rows, cols, states, lambdaSmooth, noiseP);\n\n%%\noptions.alg_type = 'CHROMATIC';\noptions.nsamples = 50;\noptions.nskip = 10;\noptions.ncpus = 4;\ndisp('---------------');\n[samples, nupdates, nchanges, marginals] = ...\n  gibbs_sampler(factors, options);\n\n%% Take the last set of beliefs and compute the exected value\nfor i = 1:options.nsamples\n  %   pred_img = reshape(cellfun(@(x) (1:length(x)) * x, marginals(:,i)), ...\n  %     rows, cols);\n  pred_img = reshape(arrayfun(@(x) (1:length(x)) * x, samples(:,i)), ...\n    rows, cols);\n  \n  figure(1);subplot(1,3,3); colormap('gray');\n  imagesc(pred_img); title(['Sample Image ', num2str(i)]);\nend\n\n%% Render the final marginal expectations\n\npred_img = reshape(cellfun(@(x) (1:length(x)) * x, marginals(:,i)), ...\n  rows, cols);\nfigure(2); colormap('gray');\nimagesc(pred_img); title('Expected Pixel Marginals');\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/matlab/tests/make_grid_model.m",
    "content": "%% This code generates a grid model\nfunction [factors, img, noisy_img] = make_grid_model(rows, cols, states, ...\n   lambdaSmooth, noiseP)\n\n% Create a virtual image\n[u,v] = meshgrid(linspace(0,1,rows), linspace(0,1,cols));\nimg = (1 + cos(1./sqrt((u-.5).^2 + (v-.5).^2)) )/2 + u.^2;\nimg = (img - min(img(:)))/(max(img(:)) - min(img(:)));\nimg = (states - 1) * img;\nimg = round(img) + 1;\nfigure(1); clf();subplot(1,3,1); colormap('gray');\nimagesc(img);\ntitle('Original Image');\n\n% add noise\nmask = rand(rows,cols) < noiseP;\nnoise = ceil(states*rand(rows,cols));\nnoisy_img = mask .* noise + ~mask .* img;\nfigure(1); subplot(1,3,2); colormap('gray');\nimagesc(noisy_img);\ntitle('Noisy Image');\n% Build the edge factor table (in log form)\n[u,v] = meshgrid(1:states, 1:states);\nedgetbl = -lambdaSmooth * abs(u - v);\n\n\n% Build the node factor tableS based on the noise model\nnodetbls = zeros(rows*cols, states) + ...\n   noiseP/(states - 1);\nind = sub2ind([rows * cols, states], (1:(rows*cols))', noisy_img(:));\nnodetbls(ind) = 1-noiseP;\nnodetbls = log(nodetbls);\n\n\n% Get all the edges and variables\nvars = 1:(rows*cols);\ngridvars = reshape(vars, rows, cols);\nedges = ...\n   [reshape(gridvars(1:(end-1),:), (rows-1) * cols,1), ...\n   reshape(gridvars(2:end,:), (rows-1) * cols,1); ...\n   reshape(gridvars(:,1:(end-1)), rows * (cols-1), 1), ...\n   reshape(gridvars(:,2:end), rows * (cols-1),1)];\n\n\n% construct the actual factors\nfactors = cell(length(vars) + length(edges), 1);\nindex = 1;\nfor i = 1:length(vars)\n   factors{index} = table_factor(vars(i), nodetbls(vars(i),:));   \n   index = index + 1;\nend\nindex = length(vars)+1;\nfor i = 1:length(edges)\n   factors{index} = table_factor(sort(edges(i,:)), edgetbl);   \n   index = index + 1;\nend\n\n\n\nend"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/matlab/tests/small_test.m",
    "content": "clear;\nRandStream.setDefaultStream(RandStream.create('mt19937ar', 'seed', 5849))\n\nsmoothing = 0.000001;\n\nvar_sizes = [3,5,3,2,4];\nvars = [1,2,3];\nfactors{1} = table_factor(vars, log(rand(var_sizes(vars)) + smoothing));\n\nvars = [2,3,4];\nfactors{2} = table_factor(vars, log(rand(var_sizes(vars)) + smoothing));\n\nvars = [1,3,5];\nfactors{3} = table_factor(vars, log(rand(var_sizes(vars)) + smoothing));\n\nvars = [4,5];\nfactors{4} = table_factor(vars, log(rand(var_sizes(vars)) + smoothing));\n\n\nmaxasg = prod(var_sizes);\n\njoint = zeros(var_sizes);\n\n%% compute joint\nfor i = 1:maxasg\n  asg = ind2asg(var_sizes, i);\n  for j = 1:length(factors)\n    subi = asg2ind(var_sizes(factors{j}.vars), asg(factors{j}.vars));\n    joint(i) = joint(i)  + factors{j}.logP(subi);\n  end\nend\n\nP = exp(joint) / sum(exp(joint(:)));\n\n\n%% run the Chromatic Sampler\noptions.alg_type = 'CHROMATIC';\noptions.nsamples = 1000;\noptions.nskip = 100;\noptions.ncpus = 1;\n\nsamples = gibbs_sampler(factors, options);\n\nP_est = zeros(var_sizes);\nfor i = 1:options.nsamples\n  ind = asg2ind(var_sizes, samples(:,i)');\n  P_est(ind) = P_est(ind) + 1;\nend\nP_est = P_est ./ sum(P_est(:));\n\n\nerror = abs(P_est - P);\ndisp(['Chromatic error: ', num2str(max(error(:)))]);\n\n%% run the sampler\noptions.alg_type = 'SPLASH';\noptions.nsamples = 1000;\noptions.nskip = 100;\noptions.ncpus = 1;\noptions.treewidth=5;\nsamples = gibbs_sampler(factors, options);\n\n\nP_est = zeros(var_sizes);\nfor i = 1:options.nsamples\n  ind = asg2ind(var_sizes, samples(:,i)');\n  P_est(ind) = P_est(ind) + 1;\nend\nP_est = P_est ./ sum(P_est(:));\n\n\nerror = abs(P_est - P);\ndisp(['Splash error: ', num2str(max(error(:)))]);"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/matlab/tests/small_test2.m",
    "content": "clear;\nRandStream.setDefaultStream(RandStream.create('mt19937ar', 'seed', 5849))\n\n% Construct strong agreement potentials\nnstates = 5;\nvar_sizes = ones(1,5)*nstates;\ntbl = log(eye(nstates) + 0.001);\n\n% Construct a set of factors\nfactors{1} = table_factor([1,2], tbl);\nfactors{2} = table_factor([1,3], tbl);\nfactors{3} = table_factor([1,4], tbl);\nfactors{4} = table_factor([2,4], tbl);\nfactors{5} = table_factor([2,5], tbl);\nfactors{6} = table_factor([3,4], tbl);\nfactors{7} = table_factor([4,5], tbl);\n\nmaxasg = prod(var_sizes);\njoint = zeros(var_sizes);\n\n%% compute joint\nfor i = 1:maxasg\n  asg = ind2asg(var_sizes, i);\n  for j = 1:length(factors)\n    subi = asg2ind(var_sizes(factors{j}.vars), asg(factors{j}.vars));\n    joint(i) = joint(i)  + factors{j}.logP(subi);\n  end\nend\n\nP = exp(joint) / sum(exp(joint(:)));\n\n\n%% run the Chromatic Sampler\noptions.alg_type = 'CHROMATIC';\noptions.nsamples = 1000;\noptions.nskip = 100;\noptions.ncpus = 1;\n\nsamples =  gibbs_sampler(factors, options);\n\nP_est = zeros(var_sizes);\nfor i = 1:options.nsamples\n  ind = asg2ind(var_sizes, samples(:,i)');\n  P_est(ind) = P_est(ind) + 1;\nend\nP_est = P_est ./ sum(P_est(:));\n\n\nerror = abs(P_est - P);\ndisp(['Chromatic error: ', num2str(max(error(:)))]);\n\n%% run the sampler\noptions.alg_type = 'SPLASH';\noptions.nsamples = 1000;\noptions.nskip = 10;\noptions.ncpus = 1;\noptions.treewidth=5;\n\nsamples = gibbs_sampler(factors, options);\n\n\nP_est = zeros(var_sizes);\nfor i = 1:options.nsamples\n  ind = asg2ind(var_sizes, samples(:,i)');\n  P_est(ind) = P_est(ind) + 1;\nend\nP_est = P_est ./ sum(P_est(:));\n\n\nerror = abs(P_est - P);\ndisp(['Splash error: ', num2str(max(error(:)))]);"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/mrf.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n\n#include \"mrf.hpp\"\n\n#include \"util.hpp\"\n#include \"image.hpp\"\n\n#include \"global_variables.hpp\"\n\n\n#include \n\n\n\n\n\n/** Save the beliefs stored in the graph */\nvoid save_beliefs(const mrf_graph_type& mrf,\n                  const std::string& filename) {\n  std::ofstream fout(filename.c_str());\n  fout.precision(16);\n  factor_t marginal;\n  for(size_t v = 0; v < mrf.num_vertices(); ++v) {\n    const mrf_vertex_data& vdata = mrf.vertex_data(v);\n    marginal = vdata.belief;\n    marginal.normalize();\n    fout << vdata.nsamples << '\\t';\n    size_t arity = marginal.args().var(0).size();\n    for(size_t asg = 0; asg < arity; ++asg) {\n      fout << std::exp( marginal.logP(asg) );\n      if(asg + 1 < arity ) fout << '\\t';      \n    }\n    fout << '\\n';\n  } \n  fout.close();\n} // End of save beliefs\n\n\n\n\n\n\n\n\n\n\nvoid save_asg(const mrf_graph_type& mrf,\n              const std::string& filename) {\n  std::ofstream fout(filename.c_str());\n  for(size_t v = 0; v < mrf.num_vertices(); ++v) \n    fout << mrf.vertex_data(v).asg << '\\n';\n  fout.close();\n} // End of save beliefs\n\n\n\n\n\n\n\n\n\n/** Construct an MRF from the factorized model */\nvoid mrf_from_factorized_model(const factorized_model& model,\n                               mrf_graph_type& mrf) {\n  typedef mrf_graph_type::vertex_id_type vertex_id_type;\n  typedef mrf_graph_type::edge_id_type   edge_id_type;\n  ///======================================================================\n  // Add all the variables\n  factor_t conditional, belief;\n  foreach(variable_t variable, model.variables()) {\n    mrf_vertex_data vdata(variable, model.factor_ids(variable));\n    { // Construct a uniformly random initial assignment\n      assignment_t asg(vdata.variable);\n      asg.uniform_sample();\n      vdata.asg = asg.asg_at(0);\n      double& logP = vdata.belief.logP(vdata.asg);\n      logP = log(exp(logP) + 1.0);\n    }\n    // { // construct mode center initial assignment\n    //   belief.set_args(variable);\n    //   belief.uniform();\n    //   conditional.set_args(variable);\n    //   const std::set& factor_ids = model.factor_ids(variable);\n    //   foreach(vertex_id_t fid, factor_ids) {\n    //     conditional.marginalize(model.factors()[fid]);\n    // \t   belief *= conditional;\n    //   }\n    //   belief.normalize();\n    //   assignment_t asg = belief.sample();\n    //   vdata.asg = asg.asg_at(0);\n    //   double& logP = vdata.belief.logP(vdata.asg);\n    //   logP = log(exp(logP) + 1.0);\n    // }\n    const vertex_id_type vid = mrf.add_vertex(vdata);\n    // We require variable ids to match vertex id (this simplifies a\n    // lot of stuff).\n    ASSERT_EQ(vid, variable.id());\n  }  \n  ASSERT_EQ(mrf.num_vertices(), model.variables().size());\n\n  ///======================================================================\n  // Add all the edges\n  const factorized_model::factor_map_t& factors(model.factors());\n  for(vertex_id_type vid = 0; vid < mrf.num_vertices(); ++vid) {\n    const mrf_vertex_data& vdata = mrf.vertex_data(vid);\n    // Compute all the neighbors of this vertex by looping over all\n    // the variables in all the factors that contain this vertex\n    std::set neighbors;\n    foreach(const factor_id_t fid, vdata.factor_ids) {\n      const domain_t& args = factors[fid].args();\n      for(size_t n = 0; n < args.num_vars(); ++n) {\n        variable_t neighbor_var = args.var(n);\n        if(vdata.variable != neighbor_var )\n          neighbors.insert(neighbor_var);\n      }\n    }\n    // For each of those variables add an edge from this varaible to\n    // that variable\n    foreach(const variable_t neighbor_variable, neighbors) {\n      const vertex_id_type neighbor_vid = neighbor_variable.id();\n      mrf_edge_data edata;\n      mrf.add_edge(vid, neighbor_vid, edata);      \n    }\n  } // loop over factors\n  mrf.finalize();\n} // End of construct_mrf\n\n\n\n\n\n\n\n\n\n\n\n\n\n//! Compute the unormalized likelihood of the current assignment\ndouble unnormalized_loglikelihood(const mrf_graph_type& mrf) {\n  typedef mrf_graph_type::vertex_id_type vertex_id_type;\n  double sum = 0;\n  //  size_t num_factors = SHARED_FACTORS.get().size();\n  size_t num_factors = SHARED_FACTORS_PTR->size();\n  // Sum the logprob of each factor\n  for(factor_id_t fid = 0; fid < num_factors; ++fid) {\n    // const factor_t& factor(SHARED_FACTORS.get()[fid]);\n    const factor_t& factor((*SHARED_FACTORS_PTR)[fid]);\n    // Accumulate the assignments \n    domain_t dom = factor.args();\n    assignment_t asg;\n    for(size_t i = 0; i < dom.num_vars(); ++i) {\n      const vertex_id_type vid = dom.var(i).id();\n      const mrf_vertex_data& vdata = mrf.vertex_data(vid);\n      ASSERT_EQ(vdata.variable, dom.var(i));\n      asg &= assignment_t(vdata.variable, vdata.asg);\n    }\n    sum += factor.logP(asg);\n  }\n  return sum;\n}\n\n\n\n\n\n\n\n\n\n\n\n\n\nvoid draw_mrf(const size_t experiment_id,\n              const std::string& base_name, \n              const mrf_graph_type& mrf) {\n  typedef mrf_graph_type::vertex_id_type vertex_id_type;\n  size_t rows = std::sqrt(mrf.num_vertices());\n  std::cout << \"Rows: \" << rows << std::endl;\n  image img(rows, rows);\n  std::vector values(1);\n  factor_t belief;\n  for(vertex_id_type vid = 0; vid < mrf.num_vertices(); ++vid) {\n    const mrf_vertex_data& vdata = mrf.vertex_data(vid);\n    belief = vdata.belief;\n    belief.normalize();\n    belief.expectation(values);\n    img.pixel(vid) = values[0];\n  }\n  img.pixel(0) = 0;\n  img.pixel(1) = mrf.vertex_data(0).variable.size()-1;\n  img.save(make_filename(base_name + \"_pred_\", \".pgm\", experiment_id).c_str());\n  \n  for(vertex_id_type vid = 0; vid < mrf.num_vertices(); ++vid) {   \n    img.pixel(vid) = mrf.vertex_data(vid).nsamples;\n  }\n  img.save(make_filename(base_name + \"_updates_\", \".pgm\", experiment_id).c_str());\n  \n  for(vertex_id_type vid = 0; vid < mrf.num_vertices(); ++vid) {   \n    img.pixel(vid) = mrf.vertex_data(vid).nsamples == 0;\n  }\n  img.save(make_filename(base_name + \"_unsampled_\", \".pgm\", experiment_id).c_str());\n  \n  for(vertex_id_type vid = 0; vid < mrf.num_vertices(); ++vid) {   \n    img.pixel(vid) = mrf.vertex_data(vid).asg;\n  }\n  img.pixel(0) = 0;\n  img.pixel(1) = mrf.vertex_data(0).variable.size()-1;\n  img.save(make_filename(base_name + \"_final_sample_\", \".pgm\", experiment_id).c_str());\n} // end of draw_mrf\n\n#include \n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/mrf.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef PGIBBS_MRF_HPP\n#define PGIBBS_MRF_HPP\n\n/**\n *\n * This code is ued to represent a markov random field\n *\n *  \\author Joseph Gonzalez\n */\n\n// INCLUDES ===================================================================>\n\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n\n#include \n\n#include \"factorized_model.hpp\"\n\n\nstruct mrf_vertex_data {\n  //! Problem specific variables\n  variable_t               variable;\n  //! current assignment\n  size_t                   asg;\n  //! The vector of factor_ids associated with this vertex\n  std::vector factor_ids;\n  //! Current belief estimate\n  factor_t                 belief;\n  //! The number of times this vertex has been sampled\n  size_t                   nsamples;\n  //! The number of itmes this vertex has changed its value\n  size_t                   nchanges;\n  //! Properties associated with the tree\n  struct tree_info_type {\n    double         priority;\n    size_t         tree_id;\n    size_t         height;\n    bool           in_tree; \n    tree_info_type () : \n      priority(-1), tree_id(-1), height(0), in_tree(false) { }\n    void save(graphlab::oarchive& arc) const {\n      arc << in_tree << tree_id << height << priority;\n    }\n    void load(graphlab::iarchive& arc) {\n      arc >> in_tree >> tree_id >> height >> priority;\n    }\n  };\n  //! tree info\n  tree_info_type tree_info;\n  mrf_vertex_data() : asg(0), nsamples(0), nchanges(0) { }\n  mrf_vertex_data(const variable_t& variable,\n                  const std::set& factor_ids_set) :\n    variable(variable),\n    asg(0),\n    factor_ids(factor_ids_set.begin(), factor_ids_set.end()),\n    belief(domain_t(variable)),\n    nsamples(0),\n    nchanges(0) {\n    // Initialize the belief to \"0\"\n    belief.uniform(-std::numeric_limits::max());\n    // Require that factor ids be non empty\n    ASSERT_FALSE(factor_ids.empty());\n  }\n  void save(graphlab::oarchive& arc) const {\n    arc << variable << asg << factor_ids << belief << nsamples\n        << nchanges << tree_info;\n  }\n  void load(graphlab::iarchive& arc) {\n    arc >> variable >> asg >> factor_ids >> belief >> nsamples\n        >> nchanges >> tree_info;\n  }  \n}; // End of mrf vertex data\n\n\n/**\n * The data associated with each directed edge in the pairwise markov\n * random field\n */\nstruct mrf_edge_data { \n  // Currently empty\n  void save(graphlab::oarchive &arc) const {  }\n  void load(graphlab::iarchive &arc) { }\n};\n\ntypedef graphlab::graph< mrf_vertex_data, mrf_edge_data> mrf_graph_type;\n\n/** Save the beliefs stored in the graph */\nvoid save_beliefs(const mrf_graph_type& mrf,\n                  const std::string& filename);\n\nvoid save_asg(const mrf_graph_type& mrf,\n              const std::string& filename);\n\n/** Construct an MRF from the factorized model */\nvoid mrf_from_factorized_model(const factorized_model& model,\n                               mrf_graph_type& mrf);\n\n//! Compute the unormalized likelihood of the current assignment\ndouble unnormalized_loglikelihood(const mrf_graph_type& mrf);\n\nvoid draw_mrf(const size_t experiment_id,\n              const std::string& base_name, \n              const mrf_graph_type& mrf);\n\n\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/pgibbs_tls.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \"pgibbs_tls.hpp\"\n\npthread_key_t pgibbs_tls_key;\n\npgibbs_tls* create_pgibbs_tls() {\n  ASSERT_EQ(pthread_getspecific(pgibbs_tls_key), NULL);\n  pgibbs_tls* data = new pgibbs_tls();\n  ASSERT_NE(data, NULL);\n  pthread_setspecific(pgibbs_tls_key, data);\n  return data;\n}\n\npgibbs_tls& get_pgibbs_tls() {\n  pgibbs_tls* tls =\n    reinterpret_cast\n    (pthread_getspecific(pgibbs_tls_key) );\n  // If no tsd be has been associated, create one\n  if(tls == NULL) tls = create_pgibbs_tls();\n  ASSERT_NE(tls, NULL);\n  return *tls;\n}\n\nvoid destroy_pgibbs_tls(void* ptr) {\n  pgibbs_tls* tls = \n    reinterpret_cast(ptr);\n  if(tls != NULL) delete tls;\n\n}\n\n\nstruct pgibbs_tls_key_creater {\n  pgibbs_tls_key_creater( )  {\n    pthread_key_create(&pgibbs_tls_key,\n                       destroy_pgibbs_tls);\n  }\n};\nstatic const pgibbs_tls_key_creater make_pgibbs_tls_key;\n\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/pgibbs_tls.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef PGIBBS_TLS_HPP\n#define PGIBBS_TLS_HPP\n\n\n\n/**\n *\n * This code is used to represent thread local storage needed in some\n * of the sampler code\n *\n *  \\author Joseph Gonzalez\n */\n\n// INCLUDES ===================================================================>\n\n#include \n\n\n#include \"factorized_model.hpp\"\n\n\n// //! Key used to get the pgibbs tls\n// extern pthread_key_t pgibbs_tls_key;\n\n//! Local state available to each thread\nstruct pgibbs_tls {\n  factor_t cavity;\n  factor_t conditional_factor;\n  factor_t belief;\n  factor_t tmp_belief;\n};\n\n\npgibbs_tls& get_pgibbs_tls();\n\n\n\n\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/run_statistics.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef PGIBBS_RUN_STATISTICS_HPP\n#define PGIBBS_RUN_STATISTICS_HPP\n\n#include \"mrf.hpp\"\n\n//! Statistics associated with a run\nstruct run_statistics {\n  size_t nsamples;\n  size_t nchanges;\n  double loglik;\n  size_t min_samples;\n  size_t max_samples;\n  run_statistics() :\n    nsamples(0), nchanges(0), loglik(0.0),\n    min_samples(std::numeric_limits::max()), max_samples(0) { }\n  run_statistics(const mrf_graph_type& mrf) :\n    nsamples(0), nchanges(0), loglik(0.0),\n    min_samples(std::numeric_limits::max()), max_samples(0) {\n    typedef mrf_graph_type::vertex_id_type vertex_id_type;\n    // Compute the unnormalized log likelihood\n    loglik = unnormalized_loglikelihood(mrf);\n    for(vertex_id_type vid = 0; vid < mrf.num_vertices(); ++vid) {\n      const mrf_vertex_data& vdata = mrf.vertex_data(vid);\n      nsamples += vdata.nsamples;\n      nchanges += vdata.nchanges;\n      min_samples = std::min(min_samples, vdata.nsamples);\n      max_samples = std::max(max_samples, vdata.nsamples);\n    } // end of for loop\n  } // end of compute run statistics\n  \n  void print() const {\n    std::cout << \"nsamples:        \" << nsamples << std::endl\n              << \"nchanges:        \" << nchanges << std::endl\n              << \"loglik:          \" << loglik   << std::endl\n              << \"min_samples:     \" << min_samples << std::endl\n              << \"max_samples:     \" << max_samples << std::endl;\n  }\n};\n\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/sampler.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n *\n * This program runs the various gibbs samplers\n *\n *  \\author Joseph Gonzalez\n */\n\n// INCLUDES ===================================================================>\n\n// Including Standard Libraries\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include  \n#include \n#include \n#include \n\n\n\n#include \n#include \n\n\n\n#include \n\n\n// Image reading/writing code\n#include \"util.hpp\"\n#include \"factorized_model.hpp\"\n#include \"mrf.hpp\"\n#include \"junction_tree.hpp\"\n\n#include \"chromatic_sampler.hpp\"\n#include \"jt_splash_sampler.hpp\"\n#include \"global_variables.hpp\"\n\n\n// Include the macro for the foreach operation\n#include \n\n\n// Results files =============================================================>\nconst std::string chromatic_results_fn = \"chromatic_results.tsv\";\nconst std::string async_results_fn     = \"async_results.tsv\";\nconst std::string splash_results_fn    = \"splash_results.tsv\";\nconst std::string jtsplash_results_fn  = \"jtsplash_results.tsv\";\n\n\n\n\n// Command Line Arguments  ====================================================>\nstd::string model_filename; \nstd::string experiment_type = \"chromatic\";\nstd::vector runtimes(1, 10);  \nbool draw_images = false;\n\nsize_t treesize = 1000;\nsize_t treewidth = 3;\nsize_t treeheight = std::numeric_limits::max();\nsize_t factorsize = std::numeric_limits::max();\nsize_t subthreads = 1;\nbool   priorities = false;\n\n\n\n\n\n\n\n\n// MAIN =======================================================================>\nint main(int argc, char** argv) { \n\n  // set the global logger\n  global_logger().set_log_level(LOG_WARNING);\n  global_logger().set_log_to_console(true);\n\n\n  // std::srand ( graphlab::timer::usec_of_day() );\n  // graphlab::random::seed();\n  \n  // Parse command line arguments --------------------------------------------->\n  graphlab::command_line_options clopts;\n\n  clopts.attach_option(\"model\",\n                       &model_filename, model_filename,\n                       \"model file name\");\n  clopts.add_positional(\"model\");\n\n  clopts.attach_option(\"experiment\", \n                       &experiment_type, experiment_type,\n                       \"the type of experiment to run \"\n                       \"{chromatic, jtsplash}\");\n  clopts.add_positional(\"experiment\");\n\n  clopts.attach_option(\"runtimes\", \n                       &runtimes, runtimes,\n                       \"total runtime in seconds\");\n\n\n  clopts.attach_option(\"draw_images\", \n                       &draw_images, draw_images,\n                       \"draw pictures (assume sqrt(numvert) rows)\");\n\n\n  clopts.attach_option(\"treesize\", \n                       &treesize, treesize,\n                       \"The maximum number of variables in a junction tree\");\n\n  clopts.attach_option(\"treewidth\", \n                       &treewidth, treewidth,\n                       \"The maximum treewidth.\");\n\n  clopts.attach_option(\"treeheight\", \n                       &treeheight, treeheight,\n                       \"The maximum height of the trees. \");\n\n  clopts.attach_option(\"factorsize\", \n                       &factorsize, factorsize,\n                       \"The maximum factorsize\");\n\n  clopts.attach_option(\"subthreads\", \n                       &subthreads, subthreads,\n                       \"The number of threads to use inside each tree \"\n                       \"(zero means not used)\");\n\n  clopts.attach_option(\"priorities\",\n                       &priorities, priorities,\n                       \"Use priorities?\");\n\n  // Set defaults for scope and scheduler\n  clopts.set_scheduler_type(\"fifo\");\n  clopts.set_scope_type(\"edge\");\n\n  if( !clopts.parse(argc, argv) ) { \n    std::cout << \"Error parsing command line arguments!\"\n              << std::endl;\n    return EXIT_FAILURE;\n  }\n\n\n  std::cout << \"Application Options\" << std::endl;\n  std::cout \n    << \"model:          \" << model_filename << std::endl\n    << \"experiment:     \" << experiment_type << std::endl \n    << \"runtime:        \" \n    << boost::lexical_cast(runtimes) << std::endl;\n   \n  std::cout << \"Graphlab Options\" << std::endl;\n  clopts.print();\n\n\n  // create model filename\n  std::cout << \"Load alchemy file.\" << std::endl;\n  factorized_model factor_graph;\n  factor_graph.load_alchemy(model_filename);\n\n  // Set the global factors\n  //SHARED_FACTORS.set(factor_graph.factors());\n  SHARED_FACTORS_PTR = &(factor_graph.factors());\n  \n\n  std::cout << \"Building graphlab MRF GraphLab core.\" << std::endl;\n  mrf_gl::core mrf_core;\n  mrf_from_factorized_model(factor_graph, mrf_core.graph());\n  mrf_core.set_engine_options(clopts);\n\n  std::cout << \"Computing coloring.\" << std::endl;\n  size_t colors = mrf_core.graph().compute_coloring();\n  std::cout << \"Colors: \" << colors << std::endl;\n  \n  // Create synthetic images -------------------------------------------------->\n  if(experiment_type == \"chromatic\") {\n    run_chromatic_sampler(mrf_core, \n                          chromatic_results_fn, \n                          runtimes,\n                          draw_images);\n  } if(experiment_type == \"jtsplash\") {\n    splash_settings settings; \n    settings.ntrees = mrf_core.get_engine_options().get_ncpus();\n    settings.max_tree_size = treesize;\n    settings.max_tree_height = treeheight;\n    settings.max_tree_width = treewidth;\n    settings.max_tree_height = treeheight;\n    settings.priorities = priorities;\n    settings.subthreads = subthreads;\n    \n    run_jtsplash_sampler(mrf_core.graph(),\n                         jtsplash_results_fn,\n                         runtimes,\n                         draw_images,\n                         settings);\n\n  } else {\n    std::cout << \"Invalid experiment type!\" << std::endl;\n    return EXIT_FAILURE;\n  }\n  \n  std::cout << \"Done!\" << std::endl;\n  return EXIT_SUCCESS;  \n} // End of main\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n#include \n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/util.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n\n#include \"util.hpp\"\n\n\n\nsize_t file_line_count(const std::string& experiment_file) {\n  std::ifstream fin(experiment_file.c_str());\n  size_t lines = 0;\n  std::string line;\n  while(getline(fin, line)) lines++;\n  fin.close();\n  return lines;\n}\n\n\nstd::string make_filename(const std::string& base,\n                          const std::string& suffix,\n                          const size_t number) {\n  std::stringstream strm;\n  strm << base\n       << std::setw(10) << std::setfill('0')\n       << number\n       << suffix;\n  std::cout << strm.str() << std::endl;\n  return strm.str();\n}\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/gibbs_sampling/util.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef PGIBBS_UTIL_HPP\n#define PGIBBS_UTIL_HPP\n\n#include \n\n//! Get the number of lines in the file\nsize_t file_line_count(const std::string& experiment_file);\n\n\n//! make a filename from base sufix and number\nstd::string make_filename(const std::string& base,\n                          const std::string& suffix,\n                          const size_t number);\n\n\n\n\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/kernelbp/CMakeLists.txt",
    "content": "project(GraphLab)\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/kernelbp/old/BallTreeDensity.cpp",
    "content": "/* Copyright (c) 2003 Alexander Ihler\r\n * Original code from: http://www.ics.uci.edu/~ihler/code/index.html\r\n *\r\n *     All rights reserved.\r\n *\r\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\r\n *  you may not use this file except in compliance with the License.\r\n *  You may obtain a copy of the License at\r\n *\r\n *      http://www.apache.org/licenses/LICENSE-2.0\r\n *\r\n *  Unless required by applicable law or agreed to in writing,\r\n *  software distributed under the License is distributed on an \"AS\r\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\r\n *  express or implied.  See the License for the specific language\r\n *  governing permissions and limitations under the License. */\r\n\r\n //\r\n// Matlab MEX interface for KD-tree C++ functions\r\n//\r\n// Written by Alex Ihler and Mike Mandel\r\n// Copyright (C) 2003 Alexander Ihler\r\n//\r\n\r\n//#define MEX\r\n#include \"cpp/BallTreeDensity.h\"\r\n#ifdef MEX\r\n#include \"mex.h\"\r\n#endif\r\n\r\n#ifdef MEX\r\nvoid mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])\r\n{\r\n\r\n  // check for the right number of arguments\r\n  if((nrhs < 3)||(nrhs > 4))\r\n    mexErrMsgTxt(\"Takes 3-4 input arguments\");\r\n  if(nlhs != 1)\r\n    mexErrMsgTxt(\"Outputs one result (a structure)\");\r\n\r\n  if (nrhs == 3) //                          points, weights, bandwidths\r\n    plhs[0] = BallTreeDensity::createInMatlab(prhs[0],prhs[1],prhs[2]);\r\n  else {          //                          points, weights, bandwidths,type\r\n    int ktype = (int) mxGetScalar(prhs[3]);\r\n    plhs[0] = BallTreeDensity::createInMatlab(prhs[0],prhs[1],prhs[2],(BallTreeDensity::KernelType) ktype);\r\n  }\r\n}\r\n#else\r\n\r\n\r\n#endif\r\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/kernelbp/old/CMakeLists.txt",
    "content": "project(GraphLab)\n\n# if (ITPP-FOUND)\n#   add_graphlab_executable(nbp denoise.cpp BallTreeDensity.cpp cpp/BallTreeClass.cc cpp/BallTreeDensityClass.cc)\n# endif()\n\n\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/kernelbp/old/cpp/BallTree.h",
    "content": "/* Copyright (c) 2003 Alexander Ihler\r\n * Original code from: http://www.ics.uci.edu/~ihler/code/index.html\r\n *\r\n *     All rights reserved.\r\n *\r\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\r\n *  you may not use this file except in compliance with the License.\r\n *  You may obtain a copy of the License at\r\n *\r\n *      http://www.apache.org/licenses/LICENSE-2.0\r\n *\r\n *  Unless required by applicable law or agreed to in writing,\r\n *  software distributed under the License is distributed on an \"AS\r\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\r\n *  express or implied.  See the License for the specific language\r\n *  governing permissions and limitations under the License. */\r\n\r\n //////////////////////////////////////////////////////////////////////////////////////\r\n// BallTree.h  --  class definition for a BallTree (actually KD-tree) object\r\n//\r\n// A few functions are defined only for MEX calls (construction & load from matlab)\r\n// Most others can be used more generally.\r\n// \r\n//////////////////////////////////////////////////////////////////////////////////////\r\n//\r\n// Written by Alex Ihler and Mike Mandel\r\n// Copyright (C) 2003 Alexander Ihler\r\n//\r\n//////////////////////////////////////////////////////////////////////////////////////\r\n#ifndef __BALL_TREE_H\r\n#define __BALL_TREE_H\r\n\r\n#ifdef MEX\r\n#include \"mex.h\"\r\n#endif\r\n\r\n#include \r\n#include \r\n#include \r\n#include \"../kde.h\"\r\n\r\n#define FALSE 0\r\n#define TRUE 1\r\n\r\ndouble log(double);\r\ndouble exp(double);\r\ndouble sqrt(double);\r\ndouble pow(double , double);\r\ndouble fabs(double);\r\n#define PI 3.141592653589\r\n\r\ninline double * vec2vec(const itpp::vec * _vec){\r\n\tdouble * ret = new double[_vec->size()];\r\n        memcpy(ret, _vec->_data(), _vec->size() * sizeof(double)); \r\n       return ret;\r\n}\r\ninline double * vec2vec(const itpp::mat * _mat){\r\n\tdouble * ret = new double[_mat->size()];\r\n        memcpy(ret, _mat->_data(), _mat->size() * sizeof(double)); \r\n       return ret;\r\n}\r\n\r\n\r\n\r\nclass BallTree {\r\n public:\r\n  //typedef unsigned int index;              // define \"index\" type (long)\r\n  typedef uint32_t index;              // define \"index\" type (long)\r\n  const static BallTree::index NO_CHILD = (index) -1;  // indicates no further children\r\n\r\n  /////////////////////////////\r\n  // Constructors\r\n  /////////////////////////////\r\n  \r\n  //BallTree( unsigned int d, index N, double* centers_,\r\n  //     double* ranges_, double* weights_ );\r\n  BallTree(){\r\n    lowest_leaf = NULL;\r\n    highest_leaf = NULL;\r\n    left_child = NULL;\r\n    right_child = NULL;\r\n    permutation = NULL;\r\n    ranges = NULL;\r\n    centers = NULL;\r\n    weights = NULL;\r\n    dims = 0; num_points = 0; next = 1;\r\n  }\r\n  virtual ~BallTree();\r\n#ifdef MEX\r\n  BallTree(const mxArray* structure);     // for loading ball trees from matlab\r\n  \r\n  // For creating BallTree structures in matlab:\r\n  static mxArray*  createInMatlab(const mxArray* pts, const mxArray* wts);\r\n#else\r\n  BallTree(const kde & pkde);     // for loading ball trees from matlab\r\n#endif\r\n\r\n  /////////////////////////////\r\n  // Accessor Functions  \r\n  /////////////////////////////\r\n  BallTree::index root() const              { return 0; }\r\n  unsigned int    Ndim() const              { return dims; }\r\n  index Npts()                    const { return num_points; }\r\n  index Npts(BallTree::index i)   const { return highest_leaf[i]-lowest_leaf[i]+1; }\r\n  const double* center(BallTree::index i)   const { return centers+i*dims; }\r\n  const double* range(BallTree::index i)    const { return ranges +i*dims; }\r\n  double  weight(BallTree::index i)         const { return *(weights+i); }\r\n  bool isLeaf(BallTree::index ind)          const { return ind >= num_points; }\r\n  bool validIndex(BallTree::index ind)      const { return ((0<=ind) && (ind < 2*num_points)); }\r\n  BallTree::index left(BallTree::index i)   const { return left_child[i]; }\r\n  BallTree::index right(BallTree::index i)  const { return right_child[i]; }\r\n  BallTree::index leafFirst(BallTree::index i) const { return lowest_leaf[i]; }\r\n  BallTree::index leafLast(BallTree::index i)  const { return highest_leaf[i]; }\r\n  void clean();\r\n\r\n  // Convert a BallTree::index to the numeric index in the original data\r\n  index getIndexOf(BallTree::index i) const { return permutation[i]; }\r\n\r\n  void movePoints(double*);\r\n  void changeWeights(const double *);\r\n\r\n  // Test two sub-trees to see which is nearer another BallTree\r\n  BallTree::index closer(BallTree::index, BallTree::index, const BallTree&,BallTree::index) const;  \r\n  BallTree::index closer(BallTree::index i, BallTree::index j, const BallTree& other_tree) const\r\n      { return closer(i,j,other_tree,other_tree.root()); };\r\n\r\n  void kNearestNeighbors(index *, double *, const double *, int, int) const;\r\n\r\n  /////////////////////////////\r\n  // Private class f'ns\r\n  /////////////////////////////\r\n protected:\r\n#ifdef MEX\r\n  static mxArray*  matlabMakeStruct(const mxArray* pts, const mxArray* wts);\r\n#endif\r\n  virtual void calcStats(BallTree::index);     // construction recursion\r\n\r\n  unsigned int dims;             // dimension of data \r\n  BallTree::index num_points;     // # of points \r\n  double *centers;                // ball centers, dims numbers per ball \r\n  double *ranges ;                 // bounding box ranges, dims per ball, dist from center to one side\r\n  double *weights;                // total weight in each ball \r\n  \r\n  BallTree::index *left_child ,  *right_child ;  // left, right children; no parent indices\r\n  BallTree::index *lowest_leaf , *highest_leaf; // lower & upper leaf indices for each ball\r\n  BallTree::index *permutation;                // point's position in the original data\r\n\r\n  BallTree::index next;                        // internal var for placing the non-leaf nodes \r\n\r\n  static const char *FIELD_NAMES[];            // list of matlab structure fields\r\n  static const int nfields;\r\n\r\n  // for building the ball tree\r\n  void buildBall(BallTree::index firstLeaf, BallTree::index lastLeaf, BallTree::index root);\r\n  BallTree::index most_spread_coord(BallTree::index, BallTree::index) const;\r\n  BallTree::index partition(unsigned int dim, BallTree::index low, BallTree::index high);\r\n  virtual void swap(BallTree::index, BallTree::index);         // leaf-swapping function\r\n\r\n  void select(unsigned int dimension, index position, \r\n\t\t       index low, index high);\r\n\r\n  double minDist(index, const double*) const;\r\n  double maxDist(index, const double*) const;\r\n\r\n  // build the non-leaf nodes from the leaves\r\n  void buildTree();\r\n};\r\n\r\n#endif\r\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/kernelbp/old/cpp/BallTreeClass.cc",
    "content": "/* Copyright (c) 2003 Alexander Ihler\n * Original code from: http://www.ics.uci.edu/~ihler/code/index.html\n *\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License. */\n\n //////////////////////////////////////////////////////////////////////////////////////\n// BallTreeClass  --  class definitions for a BallTree (actually KD-tree) \n//                    object, primarily for use in matlab MEX files.\n//\n// See BallTree.h for the class definition.\n//\n//////////////////////////////////////////////////////////////////////////////////////\n//\n// Written by Alex Ihler and Mike Mandel\n// Copyright (C) 2003 Alexander Ihler\n//\n//////////////////////////////////////////////////////////////////////////////////////\n\n//#define MEX\n#include \n#include \"BallTree.h\"\n#include \n#include \n#include \n\n#include \"../kde.h\"\n\nconst char* BallTree::FIELD_NAMES[] = {\"D\", \"N\", \"centers\", \"ranges\", \"weights\",\n            \"lower\", \"upper\", \"leftch\", \"rightch\", \"perm\"};\nconst int BallTree::nfields = 10;\n\n// Given the leaves, build the rest of the tree from the top down.\n// Split the leaves along the most spread coordinate, build two balls\n// out of those, and then build a ball around those two children.\nvoid BallTree::buildBall(BallTree::index low, BallTree::index high, BallTree::index root)\n{\n  // special case for N=1 trees\n  if(low == high) {\n    lowest_leaf[root] = low;\n    highest_leaf[root] = high;\n    left_child[root] = low;\n\n    // point right child to the same as left for calc stats, and then\n    // point it to the correct NO_CHILD afterwards.  kinda kludgey\n    right_child[root] = high;\n    calcStats(root);\n    right_child[root] = NO_CHILD;\n\n    return;\n  }\n\n  BallTree::index coord, split, left, right;\n  coord = most_spread_coord(low, high);    // find dimension of widest spread\n  \n  // split the current leaves into two groups, to build balls on them.\n  // Chose the most spread coordinate to split them on, and make sure\n  // there are the same number of points in each (+-1 for round off\n  // error).\n  split = (low + high) / 2;\n  select(coord, split, low, high);\n\n  // an alternative is to use partition, but that doesn't deal well\n  // with repeated numbers and it doesn't split into balanced sets.\n//   split = partition(coord, low, high);\n\n  // if the left sub-tree is just one leaf, don't make a new non-leaf\n  // node for it, just point left_idx directly to the leaf itself.\n  if(split <= low)    left = low;\n  else                left = next++;\n\n  // same for the right\n  if(split+1 >= high) right = high;\n  else                right = next++;\n\n  lowest_leaf[root]  = low;\n  highest_leaf[root] = high;\n  left_child[root]   = left;\n  right_child[root]  = right;\n\n  // build sub-trees if necessary\n  if(left != low)    buildBall(low, split, left);\n  if(right != high)  buildBall(split+1, high, right);\n\n  calcStats(root);\n}\n\n// Find the dimension along which the leaves between low and high\n// inclusive have the greatest variance\nBallTree::index BallTree::most_spread_coord(BallTree::index low, BallTree::index high) const\n{\n  BallTree::index dimension, point, max_dim;\n  double mean, variance, max_variance;\n\n  max_variance = 0;\n  max_dim = 0;\n  assert(dims >0);\n\n  for(dimension = 0; dimension max_variance) {\n      max_variance = variance;\n      max_dim = dimension;\n    }\n  }\n\n  return max_dim;\n}\n\n\n// straight from CLR, the unrandomized partition algorithm for\n// quicksort.  Partitions the leaves from low to high inclusive around\n// a random pivot in the given dimension.  Does not affect non-leaf\n// nodes, but does relabel the leaves from low to high.\nBallTree::index BallTree::partition(unsigned int dimension, BallTree::index low, \n\t\t\t\t  BallTree::index high) \n{\n  BallTree::index pivot;\n\n  pivot = low;  // not randomized, could set pivot to a random element\n\n  while(low < high) {\n    while(centers[dims*high + dimension] >= centers[dims*pivot + dimension])\n      high--;\n    while(centers[dims*low + dimension] < centers[dims*pivot + dimension])\n      low++;\n    \n    swap(low, high);\n    pivot = high;\n  }\n\n  return high;\n}\n\n\n// Function to partition the data into two (equal-sized or near as possible)\n//   sets, one of which is uniformly greater than the other in the given\n//   dimension.\nvoid BallTree::select(unsigned int dimension, BallTree::index position,\n\t\t      BallTree::index low, BallTree::index high)\n{\n  BallTree::index m,r,i;\n  \n  while (low < high) {\n    r = (low + high)/2; \n    swap(r,low);\n    m = low;\n    for (i=low+1; i<=high; i++) {\n      if (centers[dimension+dims*i] < centers[dimension+dims*low]) {\n        m++;\n        swap(m,i);\n      } \n    }\n    swap(low,m);\n    if (m <= position) low=m+1;\n    if (m >= position) high=m-1;\n  }    \n}\n\n\n// Swap the ith leaf with the jth leaf.  Actually, only swap the\n// weights, permutation, and centers, so only for swapping\n// leaves. Will not swap ranges correctly and will not swap children\n// correctly.\nvoid BallTree::swap(BallTree::index i, BallTree::index j) \n{\n  BallTree::index k;\n  double tmp;\n\n  if (i==j) return;\n\n  // swap weights\n  tmp = weights[i];    weights[i] = weights[j];          weights[j] = tmp;\n\n  // swap perm\n  k = permutation[i];  permutation[i] = permutation[j];  permutation[j] = k;\n\n  // swap centers\n  i *= dims;   j *= dims;\n  for(k=0; k 0);\n\n  // figure out the center and ranges of this ball based on it's children\n  double max, min;\n  for(d=0; d center(rightI)[d] + range(rightI)[d])\n      max = center(leftI)[d] + range(leftI)[d];\n    else\n      max = center(rightI)[d] + range(rightI)[d];\n\n    if (center(leftI)[d] - range(leftI)[d] < center(rightI)[d] - range(rightI)[d])\n      min = center(leftI)[d] - range(leftI)[d];\n    else\n      min = center(rightI)[d] - range(rightI)[d];\n\n    centers[root*dims+d] = (max+min) / 2;\n    ranges[root*dims+d] = (max-min) / 2;\n  }    \n  \n  // if the left ball is the same as the right ball (should only\n  // happen when calling the function directly with the same argument\n  // twice), don't count the weight twice\n  if(leftI != rightI)\n    weights[root] = weights[leftI] + weights[rightI];\n  else\n    weights[root] = weights[leftI];\n}\n  \n\n// Public method to build the tree, just calls the private method with\n// the proper starting arguments.\nvoid BallTree::buildTree()\n{\n  BallTree::index i,j;\n  for (j=0, i=num_points; j0);\n \n  for(int i=0; i0);\n  index i;\n  for (i=leafFirst(root());i<=leafLast(root());i++)\n    for (unsigned int k=0;k0);\n  for(index i=num_points, j=0; i0);\n  for(index i=0; i= 0)\n      dist += tmp*tmp;\n  }\n  \n  return dist;\n}\ndouble BallTree::maxDist(index myBall, const double* point) const\n{\n  double dist = 0, tmp;\n\n  assert(dims >0);\n  for(index i=0; i myMap;\n\n// nns is a K x N matrix\n// dists is a 1 x N vector\n// points is a D x N matrix\nvoid BallTree::kNearestNeighbors(index *nns, double *dists, const double *points, \n\t\t\t\t int N, int k) \n  const\n{\n  myMap m;\n  int leavesDone = 0;\n  index lastBall;\n  assert(dims >0);\n  assert(N>0);\n\n  for(index target = 0; target < N*dims; target += dims, ++leavesDone) {  \n\n    int nnsSoFar = 0;\n    double leastDist = maxDist(root(), points+target);\n\n    m.insert(myMap::value_type(minDist(root(), points+target), root()));\n    // examining points in order of min dist means that when you see a\n    // point, it is the closest point left\n    \n    while(! m.empty() && nnsSoFar < k) {\n      index current = (*m.begin()).second;\n      m.erase(m.begin());\n      \n      if(isLeaf(current)) {\n\t// since the nodes are sorted by minDist, a leaf at the front of\n\t// the pq must be the next nearest neighbor\n\tnns[leavesDone*k + nnsSoFar++] = current;\n\tlastBall = current;\n      } else {  // not a leaf\n\t// push both children\n\tm.insert(myMap::value_type(minDist(left_child[current], points+target), \n\t\t\t\t   left_child[current]));\n\tm.insert(myMap::value_type(minDist(right_child[current], points+target), \n\t\t\t\t   right_child[current]));\n      }\n      \n      bool keepInnerPruning = true;\n      myMap::iterator it = m.begin(), last;\n      // get rid of ineligible balls and find the min distance\n      while(it != m.end()) {\n\tcurrent = (*it).second;\n\tdouble max = maxDist(current, points+target);\n\tif(max < leastDist && Npts(current) >= k - nnsSoFar)\n\t  leastDist = max;\n\t\n//\tif((*it).first > leastDist) {\n//\t  // we see the points in order of minDist, so once we see one\n//\t  // that's too big, the rest will also be too big\n//\t  m.erase(it, m.end());\n//\t  break;\n//\t}\n\t\n\tmyMap::iterator last = it++;\n\tif(keepInnerPruning) {\n\t  if(it != m.end() && max < (*it).first && Npts(current) < k - nnsSoFar) {\n\t    // if the closest ball doesn't have too many points and all of\n\t    // them are nearer than any other ball, include them all\n\t    //   note \"<\" not \"<=\" so that *dist will be right\n\t    lastBall = current;\n\t    for(index i=leafFirst(current); i <= leafLast(current); i++)\n\t      nns[leavesDone*k + nnsSoFar++] = i;\n\t    m.erase(last);\n\t  } else {\n\t    keepInnerPruning = false;\n\t  }\n\t}\n      } // end pruning\n    } // end single nearest neighbor\n\n    // clear out the remaining points\n    m.clear();\n\n    dists[leavesDone] = sqrt(maxDist(lastBall, points+target));\n    \n    index i;\n    for(i=leavesDone*k; i < leavesDone*k+nnsSoFar; i++)\n      nns[i] = getIndexOf(nns[i]);\n    for(i=leavesDone*k+nnsSoFar; i<(leavesDone+1)*k; i++)\n      nns[i] = NO_CHILD;\n  } // end all nearest neighbors\n}\n\n/////////////////////////////// matlab functions ////////////////////////////\n\n// Constructor that doesn't initialize members, so that they can be\n// set by the loadFromMatlab and createInMatlab functions.\n//BallTree::BallTree() : next(1) {}\n\n\n#ifdef MEX\n\n// Load the arrays already allocated in matlab from the given\n// structure.\nBallTree::BallTree(const mxArray* structure) \n{\n  dims       = (unsigned int) mxGetScalar(mxGetField(structure,0,\"D\")); // get the dimensions\n  num_points = (BallTree::index) mxGetScalar(mxGetField(structure,0,\"N\")); //\n  \n  centers = (double*) mxGetPr(mxGetField(structure,0,\"centers\"));\n  ranges  = (double*) mxGetPr(mxGetField(structure,0,\"ranges\"));\n  weights = (double*) mxGetPr(mxGetField(structure,0,\"weights\"));\n\n  lowest_leaf = (BallTree::index*) mxGetData(mxGetField(structure,0,\"lower\"));\n  highest_leaf= (BallTree::index*) mxGetData(mxGetField(structure,0,\"upper\"));\n  left_child  = (BallTree::index*) mxGetData(mxGetField(structure,0,\"leftch\"));\n  right_child = (BallTree::index*) mxGetData(mxGetField(structure,0,\"rightch\"));\n  permutation = (BallTree::index*) mxGetData(mxGetField(structure,0,\"perm\"));\n\n  next = 1;    // unimportant\n}\n\n// Create new matlab arrays and put them in the given structure.\nmxArray* BallTree::createInMatlab(const mxArray* _pointsMatrix, const mxArray* _weightsMatrix)\n{\n  mxArray* structure;\n  structure = matlabMakeStruct(_pointsMatrix,_weightsMatrix);\n  BallTree bt(structure);\n  if (bt.Npts() > 0) bt.buildTree();\n\n  return structure;\n}\n\n// Create new matlab arrays and put them in the given structure.\nmxArray* BallTree::matlabMakeStruct(const mxArray* _pointsMatrix, const mxArray* _weightsMatrix)\n{\n  mxArray* structure;\n  BallTree::index i, j;\n  double *_points, *_weights;\n  \n  // get fields from input arguments\n  unsigned int Nd = mxGetM(_pointsMatrix);\n  BallTree::index Np = mxGetN(_pointsMatrix);\n  _points  = (double*)mxGetData(_pointsMatrix);\n  _weights = (double*)mxGetData(_weightsMatrix);\n\n  // create structure, populate it, and get handles to the arrays\n  structure = mxCreateStructMatrix(1, 1, nfields, FIELD_NAMES);\n  \n  mxSetField(structure, 0, \"D\",       mxCreateDoubleScalar((double) Nd));\n  mxSetField(structure, 0, \"N\",       mxCreateDoubleScalar((double) Np));\n\n  mxSetField(structure, 0, \"centers\", mxCreateDoubleMatrix(Nd, 2*Np, mxREAL));\n  mxSetField(structure, 0, \"ranges\",  mxCreateDoubleMatrix(Nd, 2*Np, mxREAL));\n  mxSetField(structure, 0, \"weights\", mxCreateDoubleMatrix(1, 2*Np, mxREAL));\n\n  mxSetField(structure, 0, \"lower\",   mxCreateNumericMatrix(1, 2*Np, mxUINT32_CLASS, mxREAL));\n  mxSetField(structure, 0, \"upper\",   mxCreateNumericMatrix(1, 2*Np, mxUINT32_CLASS, mxREAL));\n  mxSetField(structure, 0, \"leftch\",  mxCreateNumericMatrix(1, 2*Np, mxUINT32_CLASS, mxREAL));\n  mxSetField(structure, 0, \"rightch\", mxCreateNumericMatrix(1, 2*Np, mxUINT32_CLASS, mxREAL));\n  mxSetField(structure, 0, \"perm\",    mxCreateNumericMatrix(1, 2*Np, mxUINT32_CLASS, mxREAL));\n\n  // initialize arrays\n  double* centers = (double *) mxGetData(mxGetField(structure, 0, \"centers\"));\n  double* weights = (double *) mxGetData(mxGetField(structure, 0, \"weights\"));\n  for (j=0,i=Nd*Np; j= 1); \n \n  //centers = (double*) mxGetPr(mxGetField(structure,0,\"centers\"));\n  itpp::vec temp = structure.centers.get_row(0);\n  centers = new double[dims*2*num_points];\n  for (int i=0; i< num_points; i++)\n     centers[i+dims*num_points] = temp[i];\n  //ranges  = (double*) mxGetPr(mxGetField(structure,0,\"ranges\"));\n\n  ranges = new double[2 * dims*num_points];\n  //weights = (double*) mxGetPr(mxGetField(structure,0,\"weights\"));\n  //weights = vec2vec(&structure.weights);\n  weights = new double[2*dims*num_points];\n  for (int i=0; i< num_points; i++)\n     weights[i+dims*num_points] = structure.weights(i);\n\n  lowest_leaf = new unsigned int[2*num_points]; //(BallTree::index*) mxGetData(mxGetField(structure,0,\"lower\"));\n  highest_leaf=  new unsigned int[2*num_points];//(BallTree::index*) mxGetData(mxGetField(structure,0,\"upper\"));\n  left_child  =  new unsigned int[2*num_points];//(BallTree::index*) mxGetData(mxGetField(structure,0,\"leftch\"));\n  right_child =  new unsigned int [2*num_points];//BallTree::index*) mxGetData(mxGetField(structure,0,\"rightch\"));\n  permutation =  new unsigned int [2*num_points];//BallTree::index*) mxGetData(mxGetField(structure,0,\"perm\"));\n   \n  next = 1;    // unimportant\n\n  //buildTree();\n}\n\nvoid BallTree::clean(){\n  if (lowest_leaf != NULL)\n       delete[] lowest_leaf;\n  if (highest_leaf != NULL)\n       delete[] highest_leaf;\n  if (left_child != NULL)\n       delete[] left_child;\n  if (right_child != NULL)\n       delete[] right_child;\n  if (permutation != NULL)\n       delete[] permutation;\n  if (centers != NULL)\n       delete[] centers;\n  if (weights!= NULL)\n       delete[] weights;\n  dims = num_points = 0;\n}\n\nBallTree::~BallTree(){\n/*  if (lowest_leaf != NULL)\n       delete[] lowest_leaf;\n  if (highest_leaf != NULL)\n       delete[] highest_leaf;\n  if (left_child != NULL)\n       delete[] left_child;\n  if (right_child != NULL)\n       delete[] right_child;\n  if (permutation != NULL)\n       delete[] permutation;\n  if (centers != NULL)\n       delete[] centers;\n  if (weights!= NULL)\n       delete[] weights;\n*/\n//  printf(\"was in destructor\\n\");\n}\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/kernelbp/old/cpp/BallTreeDensity.h",
    "content": "/* Copyright (c) 2003 Alexander Ihler\r\n * Original code from: http://www.ics.uci.edu/~ihler/code/index.html\r\n *\r\n *     All rights reserved.\r\n *\r\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\r\n *  you may not use this file except in compliance with the License.\r\n *  You may obtain a copy of the License at\r\n *\r\n *      http://www.apache.org/licenses/LICENSE-2.0\r\n *\r\n *  Unless required by applicable law or agreed to in writing,\r\n *  software distributed under the License is distributed on an \"AS\r\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\r\n *  express or implied.  See the License for the specific language\r\n *  governing permissions and limitations under the License. */\r\n\r\n //////////////////////////////////////////////////////////////////////////////////////\r\n// BallTreeDensity.h  --  class definition for a tree-based kernel density estimate\r\n//\r\n// A few functions are defined only for MEX calls (construction & load from matlab)\r\n// Most others can be used more generally.\r\n// \r\n//////////////////////////////////////////////////////////////////////////////////////\r\n//\r\n// Written by Alex Ihler and Mike Mandel\r\n// Copyright (C) 2003 Alexander Ihler\r\n//\r\n//////////////////////////////////////////////////////////////////////////////////////\r\n#ifndef __BALL_TREE_DENSITY_H\r\n#define __BALL_TREE_DENSITY_H\r\n\r\n#include \"BallTree.h\"\r\n#include \r\n#include \r\n#include \"../kde.h\"\r\n\r\n#ifndef NULL\r\n#define NULL 0\r\n#endif\r\n\r\nclass BallTreeDensity : public BallTree {\r\n public:\r\n  enum KernelType { Gaussian, Epanetchnikov, Laplacian };\r\n  KernelType getType(void) const { return type; };\r\n\r\n  enum Gradient { WRTMean, WRTVariance, WRTWeight };\r\n\r\n  /////////////////////////////\r\n  // Constructors\r\n  /////////////////////////////\r\n\r\n  //BallTreeDensity( unsigned int d, index N, double* points_,\r\n  //     double* weights_, double* bandwidths_);\r\n  BallTreeDensity() : BallTree() { bandwidth = bandwidthMax = bandwidthMin = NULL; }\r\n#ifdef MEX             // for loading ball trees from matlab\r\n  BallTreeDensity(const mxArray* structure);\r\n  static mxArray* createInMatlab(const mxArray* pts, const mxArray* wts, const mxArray* bw, BallTreeDensity::KernelType _type=Gaussian);\r\n#else\r\n  BallTreeDensity(const kde & pkde);\r\n  BallTreeDensity* createInMatlab(const itpp::mat* pts, const itpp::vec* wts, const itpp::vec* bw, BallTreeDensity::KernelType _type=Gaussian);\r\n#endif\r\n\r\n  /////////////////////////////\r\n  // Accessor Functions  \r\n  /////////////////////////////\r\n  const double* mean(BallTree::index i)     const { return means+i*dims; }\r\n  const double* variance(BallTree::index i) const { return bandwidth+i*dims; } // !!! only works for Gaussian\r\n\r\n  const double* bw(BallTree::index i)     const { return bandwidth   +i*dims; }\r\n  const double* bwMax(BallTree::index i)  const { return bandwidthMax+i*dims*multibandwidth; }\r\n  const double* bwMin(BallTree::index i)  const { return bandwidthMin+i*dims*multibandwidth; }\r\n  bool bwUniform(void) const { return multibandwidth==0; };\r\n  //   -- Others inherited from BallTree --\r\n\r\n///////////////////////////////\r\n//\r\n// Evaluation of the density at a set of points:\r\n//   pre-constructed balltree version\r\n//   array of doubles version\r\n//   leave-one-out cross-validation version\r\n//\r\n  void evaluate(const BallTree& atPoints, double* values, double maxErr=0) const;\r\n//  void evaluate(index Npts, const double* atPoints, double* values, double maxErr=0) const;\r\n  void evaluate(double* p, double maxErr) const {  evaluate(*this,p,maxErr); }\r\n\r\n  void llGrad(const BallTree& locations, double* gradDens, double* gradAt, double tolEval, double tolGrad, Gradient) const;\r\n//  void llGrad(index Npts, const double* atPoints, double* gradDens, double* gradAt, double tolEval, double tolGrad) const;\r\n \r\n  bool updateBW(const double*, index);\r\n\r\n\r\n\r\n\r\n  /////////////////////////////\r\n  // Private object functions\r\n  /////////////////////////////\r\n protected:\r\n#ifdef MEX\r\n  static mxArray* matlabMakeStruct(const mxArray* pts, const mxArray* wts, const mxArray* bw, BallTreeDensity::KernelType type);\r\n#endif\r\n  virtual void swap(BallTree::index, BallTree::index);// leaf-swapping function\r\n  virtual void   calcStats(BallTree::index root);     // recursion for computing BW ranges\r\n\r\n  KernelType type;\r\n  unsigned int multibandwidth;    // flag: is bandwidth uniform?\r\n\r\n  double *means;                  // Weighted mean of points from this level down     \r\n  double *bandwidth;              // Variance or other multiscale bandwidth\r\n  double *bandwidthMax,*bandwidthMin; // Bounds on BW in non-uniform case\r\n  \r\n  // Internal evaluate functions:\r\n  //   Recursive tree evaluation\r\n  const static index DirectSize = 100;        // if N*M is less than this, just compute.\r\n\r\n  void evaluate(BallTree::index myRoot, const BallTree& atTree, BallTree::index aRoot, double maxErr) const;\r\n  void evalDirect(BallTree::index myRoot, const BallTree& atTree, BallTree::index aRoot) const;\r\n\r\n  void llGradDirect(BallTree::index dRoot, const BallTree& atTree, BallTree::index aRoot, Gradient) const;\r\n  void llGradRecurse(BallTree::index dRoot,const BallTree& atTree, BallTree::index aRoot, double tolGrad, Gradient) const;\r\n  void llGradWDirect(index dRoot, const BallTree& atTree, index aRoot) const;\r\n  void llGradWRecurse(index dRoot,const BallTree& atTree, index aRoot, double tolGrad) const;\r\n\r\n\r\n  //   Bounds on kernel values between points in this subtree & another\r\n  double maxDistKer(BallTree::index dRoot, const BallTree& atTree, BallTree::index aRoot) const { \r\n    switch(getType()) \r\n      { case Gaussian:       return maxDistGauss(dRoot,atTree,aRoot);\r\n        case Laplacian:      return maxDistLaplace(dRoot,atTree,aRoot); \r\n        case Epanetchnikov:  return maxDistEpanetch(dRoot,atTree,aRoot); \r\n        default: assert(false);\r\n      }\r\n      return -1;\r\n    };\r\n  double minDistKer(BallTree::index dRoot, const BallTree& atTree, BallTree::index aRoot) const {\r\n    switch(getType()) \r\n      { case Gaussian:       return minDistGauss(dRoot,atTree,aRoot);\r\n        case Laplacian:      return minDistLaplace(dRoot,atTree,aRoot); \r\n        case Epanetchnikov:  return minDistEpanetch(dRoot,atTree,aRoot); \r\n        default: assert(false);\r\n      }\r\n      return -1;\r\n     \r\n    };\r\n\r\n  // Types of kernels supported\r\n  double maxDistLaplace(BallTree::index dRoot, const BallTree& atTree, BallTree::index aRoot) const;\r\n  double minDistLaplace(BallTree::index dRoot, const BallTree& atTree, BallTree::index aRoot) const;\r\n  double maxDistGauss(BallTree::index dRoot, const BallTree& atTree, BallTree::index aRoot) const;\r\n  double minDistGauss(BallTree::index dRoot, const BallTree& atTree, BallTree::index aRoot) const;\r\n  double maxDistEpanetch(BallTree::index dRoot, const BallTree& atTree, BallTree::index aRoot, int dim=-1) const;\r\n  double minDistEpanetch(BallTree::index dRoot, const BallTree& atTree, BallTree::index aRoot, int dim=-1) const;\r\n\r\n  void  dKdX_p(BallTree::index dRoot,const BallTree& atTree, BallTree::index aRoot, bool bothLeaves, Gradient) const;\r\n\r\n};\r\n\r\n#endif\r\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/kernelbp/old/cpp/BallTreeDensityClass.cc",
    "content": "/* Copyright (c) 2003 Alexander Ihler\r\n * Original code from: http://www.ics.uci.edu/~ihler/code/index.html\r\n *\r\n *     All rights reserved.\r\n *\r\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\r\n *  you may not use this file except in compliance with the License.\r\n *  You may obtain a copy of the License at\r\n *\r\n *      http://www.apache.org/licenses/LICENSE-2.0\r\n *\r\n *  Unless required by applicable law or agreed to in writing,\r\n *  software distributed under the License is distributed on an \"AS\r\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\r\n *  express or implied.  See the License for the specific language\r\n *  governing permissions and limitations under the License. */\r\n\r\n //////////////////////////////////////////////////////////////////////////////////////\r\n// KD-tree code extended for use in kernel density estimation\r\n//////////////////////////////////////////////////////////////////////////////////////\r\n//\r\n// Written by Alex Ihler and Mike Mandel\r\n// Copyright (C) 2003 Alexander Ihler\r\n//\r\n//////////////////////////////////////////////////////////////////////////////////////\r\n//#define MEX\r\n//#define NEWVERSION\r\n#include \r\n#include \r\n\r\n#ifdef MEX\r\n#include \"mex.h\"\r\n#endif\r\n#include \"BallTreeDensity.h\"\r\n\r\n\r\n\r\ndouble *pMin, *pMax;                // need to declare these here, for kernel \r\ndouble **pAdd, *pErr;\r\ndouble *min, *max;                  //   derivative functions in kernel.h\r\n\r\n#include \"kernels.h\"                // min&max kernel bounds for various kernels\r\n\r\n///////////////////////////////////////////////////////////////////////////////\r\n///////////////////////////////////////////////////////////////////////////////\r\n//\r\n// EVALUATION\r\n//\r\n///////////////////////////////////////////////////////////////////////////////\r\n///////////////////////////////////////////////////////////////////////////////\r\n\r\nvoid pushDownLocal(const BallTree& atTree, const BallTree::index aRoot)\r\n{\r\n  BallTree::index close;\r\n    if (!atTree.isLeaf(aRoot)) {\r\n      close = atTree.left(aRoot); \r\n      if (close != BallTree::NO_CHILD) pAdd[0][close] += pAdd[0][aRoot];\r\n      close = atTree.right(aRoot); \r\n      if (close != BallTree::NO_CHILD) pAdd[0][close] += pAdd[0][aRoot];\r\n      pAdd[0][aRoot] = 0;\r\n    }\r\n}\r\nvoid pushDownAll(const BallTree& locations)\r\n{\r\n  BallTree::index j;\r\n  for (j=locations.root(); j pMin[r]) pMin[aRoot] = pMin[r];\r\n  if (pMax[aRoot] < pMax[r]) pMax[aRoot] = pMax[r];\r\n}\r\n/////////////////////////////////////////////////////////////////////\r\n// Recursively evaluate the density implied by the samples of the \r\n// subtree (rooted at dRoot) of densTree at the locations given by\r\n// the subtree (rooted at aRoot) of *this, to within the error \r\n// percentage \"maxErr\"\r\n/////////////////////////////////////////////////////////////////////\r\n\r\nvoid BallTreeDensity::evaluate(BallTree::index dRoot,\r\n              const BallTree& atTree, BallTree::index aRoot, \r\n              double maxErr) const\r\n{\r\n  BallTree::index k, close, far;\r\n  double Kmin,Kmax,add,total;\r\n\r\n  // find the minimum and maximum effect of these two balls on each other\r\n  Kmax = minDistKer(dRoot, atTree, aRoot);\r\n  Kmin = maxDistKer(dRoot, atTree, aRoot);\r\n\r\n  total = pMin[ aRoot ];\t\t   \t     // take pmin of data below this level\r\n#ifdef NEWVERSION\r\n  total += pAdd[0][aRoot] - pErr[aRoot]; // add lower bound from local expansion\r\n#endif\r\n  total += weight(dRoot)*Kmin;           // also add minimum for this block\r\n\r\n  // if the weighted contribution of this multiply is below the\r\n  //    threshold, no need to recurse; just treat as constant\r\n  //// //if ( Kmax - Kmin <= maxErr) {                    // APPROXIMATE: ABSOLUTE\r\n  if ( Kmax - Kmin <= maxErr * total) {                    // APPROXIMATE: PERCENT\r\n    Kmin *= weight(dRoot); Kmax *= weight(dRoot);\r\n\r\n    if (this == &atTree && aRoot==dRoot) {                 // LEAVE-ONE-OUT (and same subtree)\r\n      for (k=atTree.leafFirst(aRoot); k<=atTree.leafLast(aRoot); k++){\r\n        pMin[k] += Kmin * (1 - weight(k)/weight(dRoot));   // leave our weight out of it\r\n        pMax[k] += Kmax * (1 - weight(k)/weight(dRoot));   // \r\n      }\r\n      recurseMinMax(atTree,aRoot);\r\n    } else {                                               //     NO L-O-O => just add away\r\n#ifdef NEWVERSION\r\n      pAdd[0][aRoot] += (Kmin + Kmax)/2; pErr[aRoot] = (Kmax-Kmin)/2;\r\n#else\r\n      // !!! Should *not* do this -- instead add to local expansion (constant term)\r\n      for (k=atTree.leafFirst(aRoot); k<=atTree.leafLast(aRoot); k++) {\r\n        pMin[k] += Kmin;\r\n        pMax[k] += Kmax;\r\n      }\r\n#endif\r\n      if (!atTree.isLeaf(aRoot)) { pMin[aRoot] += Kmin; pMax[aRoot] += Kmax; }\r\n    }\r\n\r\n  } else if (Npts(dRoot)*atTree.Npts(aRoot)<=DirectSize){  // DIRECT EVALUATION\r\n    evalDirect(dRoot,atTree,aRoot);\r\n  } else if (0) {                                          // FAST GAUSS APPROX\r\n    // if FGTTerms > 0 : have computed Hermite expansions of densTree (sigma uniform)\r\n    // if FGTError(dRoot->Nterms,minDistDtoA,sigma) < maxError * total\r\n    //  (if maxError, sigma, Nterms known, compute R0 & check >= minDist)\r\n    //   translate dRoot's hermite expansion to a local expansion around aRoot\r\n    //   Need to iterate over aRoot's leaves & evaluate?  (N log N)\r\n    //   Update pMin structure...\r\n  } else {                                                 // RECURSE ON SUBTREES\r\n\r\n#ifdef NEWVERSION\r\n    // push any local expansion\r\n    pushDownLocal(atTree,aRoot);\r\n#endif\r\n\r\n    // Find the subtree in closest to the other tree's left child and do \r\n    // that first so that the values are higher and there is a better\r\n    // chance of being able to skip a recursion.\r\n    close = atTree.closer( atTree.left(aRoot), atTree.right(aRoot), *this, left(dRoot)); \r\n    if (left(dRoot) != NO_CHILD && close != NO_CHILD)\r\n      evaluate(left(dRoot), atTree, close, maxErr); \r\n    far   = (close == atTree.left(aRoot)) ? atTree.right(aRoot) : atTree.left(aRoot);\r\n    if (left(dRoot) != NO_CHILD && far != NO_CHILD)\r\n      evaluate(left(dRoot), atTree, far, maxErr); \r\n\r\n    // Now the same thing for the density's right child    \r\n    close = atTree.closer( atTree.left(aRoot), atTree.right(aRoot), *this, right(dRoot)); \r\n    if (right(dRoot) != NO_CHILD && close != NO_CHILD) \r\n      evaluate(right(dRoot), atTree, close, maxErr); \r\n    far   = (close == atTree.left(aRoot)) ? atTree.right(aRoot) : atTree.left(aRoot);\r\n    if (right(dRoot) != NO_CHILD && far != NO_CHILD) \r\n      evaluate(right(dRoot), atTree, far, maxErr); \r\n\r\n    // Propogate additions in children's minimum value to this node\r\n    if (!atTree.isLeaf(aRoot)) {\r\n      pMin[aRoot] = pMin[ atTree.left(aRoot) ]; \r\n      pMax[aRoot] = pMax[ atTree.left(aRoot) ];\r\n      if (atTree.right(aRoot) != NO_CHILD) {\r\n        if (pMin[aRoot] > pMin[ atTree.right(aRoot) ])\r\n          pMin[aRoot] = pMin[ atTree.right(aRoot) ];\r\n        if (pMax[aRoot] < pMax[ atTree.right(aRoot) ])\r\n          pMax[aRoot] = pMax[ atTree.right(aRoot) ];\r\n      }\r\n    }\r\n\r\n  }\r\n}\r\n\r\n///////////////////////////////////////////\r\n// Maybe we just want to evaluate this stuff directly.\r\n///////////////////////////////////////////\r\nvoid BallTreeDensity::evalDirect(BallTree::index dRoot, const BallTree& atTree, BallTree::index aRoot) const\r\n{\r\n  BallTree::index i,j;\r\n  bool firstFlag = true;\r\n  double minVal=2e22, maxVal=0;\r\n\r\n  for (j=atTree.leafFirst(aRoot); j<=atTree.leafLast(aRoot); j++) {\r\n    for (i=leafFirst(dRoot); i<=leafLast(dRoot); i++) {\r\n      if (this != &atTree || i!=j) {               // Check leave-one-out condition;\r\n        double d = weight(i) * maxDistKer(i,atTree,j);  //  Do direct N^2 kernel evaluation\r\n        //if (this == &atTree) d /= 1-weight(j);     // leave-one-out => renormalize weights\r\n        pMin[j] += d;\r\n        pMax[j] += d;\r\n      }\r\n    }\r\n#ifdef NEWVERSION\r\n  }\r\n  recurseMinMax(atTree,aRoot);              // pass up min (& max) value for pruning\r\n#else\r\n  if (pMin[j] < minVal) minVal = pMin[j];   // determine min & max value in this block\r\n  if (pMax[j] > maxVal) maxVal = pMax[j];   \r\n  }\r\n  pMin[aRoot] = minVal; pMax[aRoot] = maxVal;\r\n#endif\r\n}\r\n\r\n/////////////////////////////////////////////////////////////////////\r\n// Dual Tree evaluation: estimate the values at this ball tree's\r\n// points given the other tree as the samples from a distribution.\r\n/////////////////////////////////////////////////////////////////////\r\nvoid BallTreeDensity::evaluate(const BallTree& locations, double* p, double maxErr) const\r\n{\r\n  BallTree::index j;\r\n  \r\n  assert(Ndim() == locations.Ndim());\r\n  assert(p != NULL);\r\n\r\n  pMin = new double[2*locations.Npts()];\r\n  pMax = new double[2*locations.Npts()];\r\n  for (j=0;j<2*locations.Npts();j++) pMin[j] = pMax[j] = 0;\r\n#ifdef NEWVERSION\r\n  pAdd = new double*[1]; pAdd[0] = new double[2*locations.Npts()];\r\n  pErr = new double[2*locations.Npts()];\r\n  for (j=0;j<2*locations.Npts();j++) pAdd[0][j] = pErr[j] = 0;\r\n#endif\r\n  \r\n  evaluate(root(), locations, locations.root(), 2*maxErr);\r\n\r\n  // Compute & account for the kernel f'ns normalization constant\r\n  double norm = 1;\r\n  switch(getType()) {\r\n    case Gaussian:  norm = pow(2*PI, ((double)Ndim())/2 );\r\n                    if (bwUniform()) \r\n                      for (unsigned int i=0;i  d(log L)/dxj[k] = - sum_i wi 1/p(yi) wj K'(xj-yi)\r\n//      d(log L)/dyi[k] = wi 1/p(yi) sum_j wj K'(xj-yi)     (same K')\r\n//\r\n////////////////////////////////////////////////////////////////////////////////////\r\nvoid BallTreeDensity::llGrad(const BallTree& locations, double* _gradD, double* _gradA, double tolEval, double tolGrad, Gradient gradWRT) const\r\n{\r\n  BallTree::index j, k;\r\n  gradD = _gradD; gradA = _gradA;\r\n\r\n  min = new double[locations.Ndim()]; max = new double[locations.Ndim()];\r\n  pMin = new double[2*locations.Npts()];\r\n  pMax = new double[2*locations.Npts()];\r\n  for (j=0;j<2*locations.Npts();j++) pMin[j] = pMax[j] = 0;\r\n#ifdef NEWVERSION\r\n  pAdd = new double*[1]; pAdd[0] = new double[2*locations.Npts()];\r\n  pErr = new double[2*locations.Npts()];\r\n  for (j=0;j<2*locations.Npts();j++) pAdd[0][j] = pErr[j] = 0;\r\n#endif\r\n  evaluate(root(), locations, locations.root(), 2*tolEval);\r\n#ifdef NEWVERSION\r\n  pushDownAll(locations);  \r\n#endif\r\n  if (this == &locations) {                          // fix leave-one-out normalization\r\n    for (j=leafFirst(root()); j<=leafLast(root()); j++)\r\n      pMax[j] /= (1-weight(j)); pMin[j] /= (1-weight(j));\r\n  }\r\n\r\n  if(gradWRT == WRTWeight)\r\n    llGradWRecurse(root(),locations,locations.root(), tolGrad*tolGrad);\r\n  else\r\n    llGradRecurse(root(),locations,locations.root(), tolGrad*tolGrad, gradWRT);\r\n\r\n  if (this == &locations) {                          // fix leave-one-out normalization\r\n    for (j=leafFirst(root()); j<=leafLast(root()); j++) {\r\n      index Nj = Ndim() * getIndexOf(j);\r\n      for (k=0;k\r\n    bandwidthMin = bandwidthMax + 2*num_points*dims;    //   track min/max vals\r\n  } else {                                              // all the same => min = max\r\n    multibandwidth = 0;                                 //         = any leaf node\r\n    bandwidthMax = bandwidthMin = bandwidth + num_points*dims;\r\n  }\r\n}\r\n\r\n\r\n// Create new matlab arrays and put them in the given structure\r\nmxArray* BallTreeDensity::createInMatlab(const mxArray* _pointsMatrix, const mxArray* _weightsMatrix,\r\n                                         const mxArray* _bwMatrix,BallTreeDensity::KernelType _type)\r\n{\r\n  mxArray* structure = matlabMakeStruct(_pointsMatrix, _weightsMatrix,_bwMatrix,_type);\r\n  BallTreeDensity dens(structure);\r\n  if (dens.Npts() > 0) dens.buildTree();\r\n\r\n  return structure;\r\n}\r\n\r\n// Create new matlab arrays and put them in the given structure.\r\nmxArray* BallTreeDensity::matlabMakeStruct(const mxArray* _pointsMatrix, const mxArray* _weightsMatrix,\r\n                                           const mxArray* _bwMatrix,BallTreeDensity::KernelType _type)\r\n{\r\n  BallTree::index i,j;\r\n\r\n  mxArray* structure = BallTree::matlabMakeStruct(_pointsMatrix, _weightsMatrix);\r\n\r\n  unsigned int Nd = (unsigned int) mxGetScalar(mxGetField(structure,0,\"D\"));\r\n  index Np = (BallTree::index) mxGetScalar(mxGetField(structure,0,\"N\"));\r\n\r\n  mxAddField(structure, \"means\");\r\n  mxSetField(structure, 0, \"means\", mxCreateDoubleMatrix(Nd, 2*Np, mxREAL));\r\n\r\n  mxAddField(structure, \"bandwidth\");\r\n  if (mxGetN(_bwMatrix) == 1)\r\n    mxSetField(structure, 0, \"bandwidth\",     mxCreateDoubleMatrix(Nd, 2*Np, mxREAL));\r\n  else\r\n    mxSetField(structure, 0, \"bandwidth\",     mxCreateDoubleMatrix(Nd, 6*Np, mxREAL));\r\n\r\n  mxAddField(structure, \"type\");\r\n    mxSetField(structure, 0, \"type\",          mxCreateDoubleScalar((double)_type));\r\n\r\n  // initialize arrays\r\n  double* means = (double *) mxGetData(mxGetField(structure, 0, \"means\"));\r\n  double* points = (double *) mxGetData(mxGetField(structure, 0, \"centers\"));\r\n  for (j=0,i=Nd*Np; j 1){\r\n    itpp::Sort mysort;\r\n    mysort.sort(0,tmp.size()-1,tmp);\r\n  }\r\n\r\n  //if (mxGetN(mxGetField(structure,0,\"bandwidth\")) == 6*num_points) {\r\n  if (structure.bw.size() > 1 && tmp(0) < tmp(tmp.size()-1)){ \r\n    assert(false);\r\n    multibandwidth = 1;\r\n    bandwidthMax = bandwidth + 2*num_points*dims;       // not all the same =>\r\n    bandwidthMin = bandwidthMax + 2*num_points*dims;    //   track min/max vals\r\n  } else {                                              // all the same => min = max\r\n     bandwidth = new double[2*dims*num_points];\r\n     for (int i=0; i bandwidthMax[NiR+k]) \r\n                              ? bandwidthMax[NiL+k] : bandwidthMax[NiR+k];\r\n      bandwidthMin[Ni+k] = (bandwidthMin[NiL+k] < bandwidthMin[NiR+k]) \r\n                              ? bandwidthMin[NiL+k] : bandwidthMin[NiR+k];\r\n  } }\r\n\r\n  switch(type) {\r\n  case Gaussian:\r\n    for(unsigned int k=0; k < dims; k++) {\r\n      means[Ni+k]     = wtL * means[NiL+k] + wtR * means[NiR+k];\r\n      bandwidth[Ni+k] = wtL* (bandwidth[NiL+k] + means[NiL+k]*means[NiL+k]) +\r\n                        wtR* (bandwidth[NiR+k] + means[NiR+k]*means[NiR+k]) -\r\n                        means[Ni+k]*means[Ni+k];\r\n    }; break;\r\n  case Laplacian:\r\n    for(unsigned int k=0; k < dims; k++) {\r\n      means[Ni+k]     = wtL * means[NiL+k] + wtR * means[NiR+k];\r\n      bandwidth[Ni+k] = wtL* (2*bandwidth[NiL+k]*bandwidth[NiL+k] + means[NiL+k]*means[NiL+k]) +\r\n                        wtR* (2*bandwidth[NiR+k]*bandwidth[NiR+k] + means[NiR+k]*means[NiR+k]) -\r\n                        means[Ni+k]*means[Ni+k];     // compute in terms of variance\r\n      bandwidth[Ni+k] = sqrt(.5*bandwidth[Ni+k]);    //  then convert back to normal BW rep.\r\n    }; break;\r\n  case Epanetchnikov:\r\n    for(unsigned int k=0; k < dims; k++) {\r\n      means[Ni+k]     = wtL * means[NiL+k] + wtR * means[NiR+k];\r\n      bandwidth[Ni+k] = wtL* (.2*bandwidth[NiL+k]*bandwidth[NiL+k] + means[NiL+k]*means[NiL+k]) +\r\n                        wtR* (.2*bandwidth[NiR+k]*bandwidth[NiR+k] + means[NiR+k]*means[NiR+k]) -\r\n                        means[Ni+k]*means[Ni+k];     // compute in terms of variance\r\n      bandwidth[Ni+k] = sqrt(5*bandwidth[Ni+k]);     //  then convert back to normal BW rep.\r\n    }; break; \r\n }\r\n\r\n}\r\n\r\n// Swap the ith leaf with the jth leaf.\r\nvoid BallTreeDensity::swap(BallTree::index i, BallTree::index j) \r\n{\r\n  if (i==j) return;\r\n\r\n  BallTree::swap(i,j);\r\n\r\n  i *= dims;  j *= dims;\r\n  for(unsigned int k=0; k 0) ? tmp : 0;\r\n    if (bwUniform()) result -= (tmp*tmp)/bw[k];\r\n    else             result -= (tmp*tmp)/bw[k] + log(bwMin(dRoot)[k]);\r\n  }\r\n  result = exp(result/2);\r\n  return result;\r\n}\r\n\r\ndouble BallTreeDensity::maxDistGauss(BallTree::index dRoot,\r\n                  const BallTree& atTree, BallTree::index aRoot) const\r\n{\r\n  unsigned int k;\r\n  double tmp,result=0;\r\n  const double *atCenter, *densCenter, *bw;\r\n\r\n  atCenter = atTree.center(aRoot); densCenter = center(dRoot);\r\n  bw = bwMin(dRoot);\r\n  for (k=0; k 0) ? tmp : 0;\r\n    if (bwUniform()) result -= tmp/bw[k];\r\n    else             result -= tmp/bw[k] + log(bwMin(dRoot)[k]);\r\n  }\r\n  result = exp(result);\r\n  return result;\r\n}\r\n\r\ndouble BallTreeDensity::maxDistLaplace(BallTree::index dRoot,\r\n                  const BallTree& atTree, BallTree::index aRoot) const\r\n{\r\n  unsigned int k;\r\n  double tmp,result=0;\r\n  const double *atCenter, *densCenter, *bw;\r\n\r\n  atCenter = atTree.center(aRoot); densCenter = center(dRoot);\r\n  bw = bwMin(dRoot);\r\n  for (k=0; k 0) ? tmp : 0;\r\n    tmp = (tmp > bw[k]) ? bw[k] : tmp;\r\n    if (k==dim) { if (tmp==bw[k]) result=0; continue;}\r\n    if (bwUniform()) result *= 1-(tmp/bw[k])*(tmp/bw[k]);\r\n    else             result *= (1-(tmp/bw[k])*(tmp/bw[k]))/bwMin(dRoot)[k];\r\n  }\r\n  return result;\r\n}\r\n\r\ndouble BallTreeDensity::maxDistEpanetch(BallTree::index dRoot,\r\n                  const BallTree& atTree, BallTree::index aRoot, int dim) const\r\n{\r\n  unsigned int k;\r\n  double tmp,result=1;\r\n  const double *atCenter, *densCenter, *bw;\r\n\r\n  atCenter = atTree.center(aRoot); densCenter = center(dRoot);\r\n  bw = bwMin(dRoot);\r\n  for (k=0; k bw[k]) ? bw[k] : tmp;\r\n    if (k==dim) { if (tmp==bw[k]) result=0; continue;}\r\n    if (bwUniform()) result *= 1-(tmp/bw[k])*(tmp/bw[k]);\r\n    else             result *= (1-(tmp/bw[k])*(tmp/bw[k]))/bwMax(dRoot)[k];\r\n  }\r\n  return result;\r\n}\r\n\r\n\r\n\r\n/////////////////////////////////////////////////////////////////////\r\n// Find upper and lower bounds on 1/p(yj) K'(xi-yj)\r\n//   for any points yj in the aRoot-th ball of atTree \r\n//              and xi in the dRoot-th ball of densTree \r\n// 3 Possible Kernels : Gaussian, Laplacian, Epanetchnikov\r\n//   Takes account of possible non-uniform bandwidth values\r\n/////////////////////////////////////////////////////////////////////\r\n\r\nvoid BallTreeDensity::dKdX_p(BallTree::index dRoot,const BallTree& atTree, \r\n\t\t\t     BallTree::index aRoot, bool bothLeaves, \r\n\t\t\t     Gradient gradType) const\r\n{\r\n  // Compute a maximum value of K'(yi-xj) for any pair: xj in dRoot, yi in aRoot\r\n  //\r\n  // e.g. Gaussian:  <----- K ------------------->  <------- D ------->\r\n  //  K'(x) = exp(-sum( .5*(a[m]-d[m])^2/bw[m] ) ) * (a[k]-d[k])/bw[k]\r\n  //\r\n  // Crappy bound is  [ min(Kmax*Dmin,Kmin*Dmin), max(Kmax*Dmax,Kmin*Dmin) ]\r\n  //\r\n  const double *atCenter, *densCenter;\r\n  double Kmin,Kmax;\r\n  atCenter = atTree.center(aRoot); densCenter = center(dRoot);\r\n\r\n//  printf(\"%d:%d \\n\",dRoot,aRoot);\r\n\r\n  if (getType()!=Epanetchnikov) {                // for the exponential forms:\r\n    if (!bothLeaves) {\r\n      Kmin = maxDistKer(dRoot,atTree,aRoot);     // if non-leaf node need both\r\n      Kmax = minDistKer(dRoot,atTree,aRoot);     //  values;\r\n    } else                                         // leaf nodes, we know they're equal\r\n      Kmax = Kmin = maxDistKer(dRoot,atTree,aRoot);//  so don't double-compute\r\n  }\r\n  \r\n  for(unsigned int k=0;k leave out k^th dim\r\n      Kmax = 2*minDistEpanetch(dRoot,atTree,aRoot,k);        //  when calculating\r\n      if (!bwUniform()) { Kmax /= bwMin(dRoot)[k]; Kmin /= bwMax(dRoot)[k]; }\r\n    }    \r\n\r\n    if (getType() == Laplacian) {                              // non-quadratic form\r\n      if (Dmin < 0) Dmin = -1;  if (Dmax < 0) Dmax = -1;       //  => sign(x-y) instead\r\n      if (Dmin > 0) Dmin = +1;  if (Dmax > 0) Dmax = +1;       //  of (x-y)\r\n    }                                                          \r\n\r\n    double bwmax = bwMax(dRoot)[k], bwmin = bwMin(dRoot)[k];\r\n    if (getType() == Epanetchnikov) {\r\n      bwmax *= bwmax; bwmin *= bwmin;\r\n    }\r\n\r\n    if (gradType == WRTMean) {\r\n      if (Dmin < 0)\r\n\tmax[k] = -Kmax*Dmin/bwmin/pMin[aRoot];\r\n      else\r\n\tmax[k] = -Kmin*Dmin/bwmax/pMax[aRoot]; \r\n      \r\n      if (Dmax < 0)\r\n\tmin[k] = -Kmin*Dmax/bwmax/pMax[aRoot];\r\n      else\r\n\tmin[k] = -Kmax*Dmax/bwmin/pMin[aRoot];\r\n\r\n    } else if(gradType == WRTWeight) {\r\n      max[k] = -Kmax / pMin[aRoot];\r\n      min[k] = -Kmin / pMax[aRoot];\r\n      break;  // only need to do for the first dimension\r\n\r\n    } else if(gradType == WRTVariance) {\r\n      max[k] = -Kmax / pMin[aRoot] * (0.5 / bwmin) * (Dmax * Dmax / bwmin - 1);\r\n      min[k] = -Kmin / pMax[aRoot] * (0.5 / bwmax) * (Dmin * Dmin / bwmax - 1);\r\n\r\n    } else {\r\n      max[k] = min[k] = 0;\r\n    }\r\n//    printf(\"  %d -- %f %f  -> %f / %f\\n\",k,Kmin,Kmax,min[k],max[k]);\r\n  }\r\n}  \r\n\r\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/kernelbp/old/denoise.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *  \n *  Any changes to the code must include this original license notice in full.\n * Original code by Yucheng Low, CMU\n * Modified to Gaussian Mixture by DAnny Bickson, CMU\n * Based on Matlab code by Alex Ihler, UC Irvine\n * See the paper: Nonparametric Belief Propagation. E. Sudderth, A. Ihler, W. Freeman, and A. Willsky. CVPR, June 2003.\n */\n\n\n#include \n#include \n#include \n#include \n#include \"kde.h\"\n#include \n#include \"image.hpp\"\n#include \"prodSampleEpsilon.hpp\"\n#include \n\n#include \n#include \n\n#include \n\n#define PROPOSAL_STDEV 20\n\nusing namespace itpp;\nusing namespace std;\n\n\nint NSAMP =12; //number of samples\ndouble EPSILON =1e-5; //epsilon (accuracy of product sampling)\nint MAX_ITERATIONS = 10;\nint iiter = 0;\n\n// STRUCTS (Edge and Vertex data) =============================================>\nstruct edge_data: public graphlab::unsupported_serialize {\n  kde msg; //the NBP message sent along this edge\n  kde edge_pot; //edge potential of this edge\n  int update_count;\n}; \n\nstruct vertex_data: public graphlab::unsupported_serialize {\n  kde obs; //ovservation\n  kde bel; //belief\n  int rounds;\n  vertex_data(){ rounds = 0;}\n};\n\ntypedef graphlab::graph graph_type;\ntypedef graphlab::types gl_types;\n\n\n/**\n * compare MAE (mean average error) of the true image vs. inferred image\n */\ndouble image_compare_mae(image &trueimg, image &infered) {\n    assert(trueimg.rows() == infered.rows());\n    assert(trueimg.cols() == infered.cols());\n    // get the set of colors in the trueimg\n    std::set colors;\n    for (size_t i = 0; i < trueimg.rows(); ++i) {\n      for (size_t j = 0; j < trueimg.cols(); ++j) {\n        colors.insert(size_t(trueimg.pixel(i,j)));\n      }\n    }\n    \n    // fill a rounding color map\n    int colormap[256];\n    int previval = -256;\n    std::set::iterator curi = colors.begin();\n    std::set::iterator nexti = curi;\n    nexti++;\n    int nextival = (nexti != colors.end())?*nexti:512;\n    while (curi != colors.end()) {\n      int low = (previval + (*curi)) / 2; if (low < 0) low = 0;\n      int high = (nextival + (*curi)) / 2; if (high > 256) high = 256;\n      \n      for (int i = low; i < high; ++i) {\n          colormap[i] = (*curi);\n      }\n      previval = (*curi);\n      curi++;\n      nexti++;\n      nextival = (nexti != colors.end())?*nexti:512;\n    }\n    \n    // compute absolute difference\n    double err = 0;\n    for (size_t i = 0; i < infered.rows(); ++i) {\n      for (size_t j = 0; j < infered.cols(); ++j) {\n        //err  += (infered.pixel(i,j) - trueimg.pixel(i,j)) * (infered.pixel(i,j) - trueimg.pixel(i,j)) ;\n        err += fabs(infered.pixel(i,j) - trueimg.pixel(i,j));\n      }\n    }\n    err  /= (infered.rows() * infered.cols());\n    return err;\n}\n\n/**\n * compare RMSE (root mean square error) of true image vs. inferred image\n */\ndouble image_compare_rmse(image &trueimg, image &infered) {\n    assert(trueimg.rows() == infered.rows());\n    assert(trueimg.cols() == infered.cols());\n    // get the set of colors in the trueimg\n    std::set colors;\n    for (size_t i = 0; i < trueimg.rows(); ++i) {\n      for (size_t j = 0; j < trueimg.cols(); ++j) {\n        colors.insert(size_t(trueimg.pixel(i,j)));\n      }\n    }\n    \n    // fill a rounding color map\n    int colormap[256];\n    int previval = -256;\n    std::set::iterator curi = colors.begin();\n    std::set::iterator nexti = curi;\n    nexti++;\n    int nextival = (nexti != colors.end())?*nexti:512;\n    while (curi != colors.end()) {\n      int low = (previval + (*curi)) / 2; if (low < 0) low = 0;\n      int high = (nextival + (*curi)) / 2; if (high > 256) high = 256;\n      \n      for (int i = low; i < high; ++i) {\n          colormap[i] = (*curi);\n      }\n      previval = (*curi);\n      curi++;\n      nexti++;\n      nextival = (nexti != colors.end())?*nexti:512;\n    }\n    \n    // compute absolute difference\n    double err = 0;\n    for (size_t i = 0; i < infered.rows(); ++i) {\n      for (size_t j = 0; j < infered.cols(); ++j) {\n        err  += (infered.pixel(i,j) - trueimg.pixel(i,j)) * (infered.pixel(i,j) - trueimg.pixel(i,j)) ;\n//        err += fabs(infered.pixel(i,j) - trueimg.pixel(i,j));\n      }\n    }\n    err  /= (infered.rows() * infered.cols());\n    return err;\n}\n\n\n/**\n * Non-parametric BP update function\n */\nvoid nbp_update(gl_types::iscope& scope,\n               gl_types::icallback& scheduler) {\n\n\n  bool debug = true;\n\n  vertex_data& v_data = scope.vertex_data();\n  graphlab::vertex_id_t vid = scope.vertex();\n  if (debug && vid%1000000 == 0){\n     std::cout<<\"Entering node \" << (int)vid << \" obs: \";\n     v_data.obs.matlab_print();\n     std::cout << std::endl;\n  }\n\n  v_data.rounds++;\n\n  if ((int)vid == 0)\n      iiter++;\n\n  gl_types::edge_list in_edges = scope.in_edge_ids();\n  gl_types::edge_list out_edges = scope.out_edge_ids();\n  assert(in_edges.size() == out_edges.size()); // Sanity check\n\n  //for each incoming message\n  for (size_t j = 0; j < in_edges.size(); ++j){\n   \n     std::vector kdes;\n     for(size_t i = 0; i < in_edges.size(); ++i) {\n    \n      graphlab::edge_id_t ineid = in_edges[i];\n      edge_data& in_edge = scope.edge_data(ineid);\n      if (i != j){\n         in_edge.msg.verify();\n         //add the message into the mixture list\n         kdes.push_back(in_edge.msg);\n      }\n     }\n\n     kdes.push_back(v_data.obs);\n     \n     graphlab::edge_id_t outeid = out_edges[j];\n     edge_data& out_edge = scope.edge_data(outeid);\n     kde marg = out_edge.edge_pot.marginal(0);  \n      //insert the marginal of this dimension as the first item of the mixture\n     kdes.insert(kdes.begin(), marg);//important: has to be first!     \n\n      //compute the mixtures product\n     prodSampleEpsilon producter; \n     kde m = producter.prodSampleEpsilonRun(kdes.size(), NSAMP, EPSILON, kdes);\n     \n     m.verify();\n     kde mar2 = out_edge.edge_pot.marginal(1);\n     mar2.verify();\n     imat firstrowind = m.indices(0,0,0,m.indices.cols()-1);\n      //sample from marginal, using indices taken from product\n     kde outmsg = mar2.sample(firstrowind,m.weights);\n     outmsg.verify(); \n     out_edge.msg = outmsg;\n\n  }\n\n   //compute belief by multiplying self potential with all incoming message\n   if (v_data.rounds == MAX_ITERATIONS){\n\tif (debug && vid%100000 == 0)\n\t   printf(\"computing belief node %d\\n\", vid);\n\n      std::vector kdes;\n      for (size_t j = 0; j < in_edges.size(); ++j){\n        graphlab::edge_id_t ineid = in_edges[j];\n        edge_data& in_edge = scope.edge_data(ineid);\n        in_edge.msg.verify();\n        //add all incoming message to mixture list\n        kdes.push_back(in_edge.msg);\n      }\n\n      //add self potential\n      kdes.push_back(v_data.obs);\n      //compute the product\n      prodSampleEpsilon prod;\n      kde m = prod.prodSampleEpsilonRun(kdes.size(), NSAMP, EPSILON, kdes);\n     \n      m.verify();\n      //store the result\n      v_data.bel = m;\n      if (debug && vid == 0){\n\t   printf(\"belief node %d is\\n\", vid);\n           m.matlab_print(); printf(\"\\n\");\n      }\n   }\n\n} // end of nbp_update\n\n\n\n\n\nvoid construct_graph(image& img,\n                     kde & edge_pot,\n                     gl_types::graph& graph) {\n\n  \n  for(size_t i = 0; i < img.rows(); ++i) {\n    for(size_t j = 0; j < img.cols(); ++j) {\n      vertex_data vdat;\n      vdat.rounds = 0;\n\n      // Set the node potentials\n      vec cent = zeros(2);\n      //center of mixture component is around pixel color\n      cent[0] = img.pixel(i,j);\n      cent[1] = img.pixel(i,j);\n      mat cent2 = cent; cent2 = transpose(cent2);\n      vec bw = \"30 30\";\n      mat bw2 = bw; bw2 = transpose(bw2);\n      vec wght = \"1 1\";\n      //create a mixture\n      vdat.obs = kde(cent2, bw2, wght);\n      vdat.bel = vdat.obs;\n      if (i == 0 && j == 0)\n       vdat.obs.matlab_print();\n      graph.add_vertex(vdat);\n      vdat.obs.verify();\n      vdat.bel.verify();\n\n    } // end of for j in cols\n  } // end of for i in rows\n\n  edge_data edata;\n  edata.edge_pot = edge_pot;\n  edata.edge_pot.matlab_print();\n\n  //add the edges to the grid graph\n  for(size_t i = 0; i < img.rows(); ++i) {\n    for(size_t j = 0; j < img.cols(); ++j) {\n      \n      size_t vertid = img.vertid(i,j);\n      if(i-1 < img.rows()) {\n        edata.msg = graph.vertex_data(img.vertid(i-1, j)).bel;\n        graph.add_edge(vertid, img.vertid(i-1, j), edata);\n      }\n      if(i+1 < img.rows()) {\n        edata.msg = graph.vertex_data(img.vertid(i+1, j)).bel;\n        graph.add_edge(vertid, img.vertid(i+1, j), edata);\n      }\n      if(j-1 < img.cols()) {\n        edata.msg = graph.vertex_data(img.vertid(i, j-1)).bel;\n        graph.add_edge(vertid, img.vertid(i, j-1), edata);\n      }\n      if(j+1 < img.cols()) {\n        edata.msg = graph.vertex_data(img.vertid(i, j+1)).bel;\n        graph.add_edge(vertid, img.vertid(i, j+1), edata);\n      }\n    } // end of for j in cols\n  } // end of for i in rows\n  graph.finalize();\n} // End of construct graph\n\n\n\n// MAIN =======================================================================>\nint main(int argc, char** argv) {\n  // set the global logger\n  global_logger().set_log_level(LOG_WARNING);\n  global_logger().set_log_to_console(true);\n\n  std::string gmmfile= \"\";\n  std::string inputfile = \"\";\n\n    // Parse command line arguments --------------------------------------------->\n  graphlab::command_line_options clopts(\"NBP image denoising\");\n  clopts.attach_option(\"epsilon\",\n                       &EPSILON, EPSILON,\n                       \"epsilon - product accuracy\");\n  clopts.attach_option(\"gmmfile\",\n                       &gmmfile, std::string(\"\"),\n                       \"true image + self and edge potential file\");\n  clopts.attach_option(\"inputfile\",\n                       &inputfile, std::string(\"\"),\n                       \"the input noisy image\");\n  clopts.attach_option(\"max_iter\", &MAX_ITERATIONS, MAX_ITERATIONS, \"maximum number of iterations. In this round the belief is compted\");\n\n  clopts.set_scheduler_type(\"round_robin\");\n\n  bool success = clopts.parse(argc, argv);\n  if(!success) {\n    return EXIT_FAILURE;\n  }\n\n  // load the potentials mixture components\n  it_ifile f(gmmfile.c_str());\n\n  mat edgecenter, edgesigma, edgeweight;\n  mat nodecenter, nodesigma, nodeweight;\n  ivec truedata;\n  ivec imgsize;\n  imat integermat;\n  vec doublevec;\n  //read edge potentials\n  f >> Name(\"edge_ce\") >> integermat;   edgecenter = to_mat(integermat);\n  f >> Name(\"edge_alpha\") >> doublevec; edgeweight = doublevec;\n  f >> Name(\"edge_sigma\") >> doublevec; edgesigma = doublevec;\n\n  //read self potential\n  f >> Name(\"like_ce\") >> nodecenter;\n  f >> Name(\"like_alpha\") >> doublevec; nodeweight = doublevec;\n  f >> Name(\"like_sigma\") >> doublevec; nodesigma = doublevec;\n  \n  //read true image\n  f >> Name(\"img1\") >> truedata;\n  //read image size\n  f >> Name(\"isize\") >> imgsize;\n\n  size_t rows = imgsize(0);\n  size_t cols = imgsize(1);\n  std::cout << \"Image size is \" << rows << \" x \" << cols << std::endl;\n  mat edgesigma2 = edgesigma;\n  edgesigma2 = transpose(edgesigma2);\n  mat edgeweight2 = edgeweight;\n  edgeweight2 = transpose(edgeweight2); \n  if (edgesigma2.cols() > edgecenter.cols())\n\tedgesigma2 = edgesigma2(0,0,0,edgecenter.cols()-1);\n  kde edge_pot = kde(edgecenter, edgesigma2, edgeweight2);\n\n// convert the true image to an image\n  image trueimg(rows, cols);\n  for (size_t i = 0; i < size_t(truedata.size()); ++i) {\n    trueimg.pixel(i) = truedata(i);\n  }\n\n  //read noisy image\n  it_ifile imgfile(inputfile.c_str());\n  vec observations;\n  imgfile >> Name(\"obs2\") >> observations;\n  // convert observations to an image\n  image img(rows, cols);\n  for (size_t i = 0;i < size_t(observations.size()); ++i) {\n    img.pixel(i) = observations(i);\n  }\n  img.save(\"noisy.pgm\");\n  trueimg.save(\"source_img.pgm\");\n\n  // Create the graph --------------------------------------------------------->\n  gl_types::core core;\n  // Set the engine options\n  core.set_engine_options(clopts);\n\n  std::cout << \"Constructing pairwise Markov Random Field. \" << std::endl;\n  construct_graph(img, edge_pot, core.graph());\n\n  // Running the engine ------------------------------------------------------->\n  core.sched_options().add_option(\"update_function\", nbp_update);\n  std::cout << \"Running the engine. \" << std::endl;\n\n  // Add the bp update to all vertices\n  core.add_task_to_all(nbp_update, 100.0);\n  // Starte the engine\n  const double runtime = core.start();\n\n  // Saving the output -------------------------------------------------------->\n  std::cout << \"Rendering the cleaned image. \" << std::endl;\n  \n  //parse belief to find the reconstructed image \n  for(size_t v = 0; v < core.graph().num_vertices(); ++v) {\n    const vertex_data& vdata = core.graph().vertex_data(v);\n    float a = vdata.bel.max();\n    if (a < 0) a = 0;\n    if (a > 255) a = 255;\n    img.pixel(v) = size_t(a);\n  }\n  double err = sqrt(image_compare_rmse(trueimg, img));\n  double err2 = image_compare_mae(trueimg, img);\n  img.save(\"inferred.pgm\");\n  std::cout << \"RMSE: \" << err << \" MAE: \"<< err2<\n#include \n\n#define mxMalloc malloc\n#define mxFree free\n#define mexErrMsgTxt(a) {printf(a); assert(false); }\n#define mxDestroyArray\n\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/kernelbp/old/image.hpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef IMAGE_HPP\n#define IMAGE_HPP\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\n\n#include \n\n#include \n\n/** A simple struct represent a gray scale image */\nclass image {\n  size_t _rows, _cols;\n  std::vector data;\npublic:\n\n  /** Create an empty image */\n  image() : _rows(0), _cols(0), data(0,0) { }\n  \n  /** Create an image of a fixed size */\n  image(size_t rows, size_t cols) : \n    _rows(rows), _cols(cols), data(rows * cols, 0) { }\n\n  inline void resize(size_t rows, size_t cols);\n\n  /** Get the number of rows */\n  inline size_t rows() const { return _rows; }\n\n  /** Get the number of columns */\n  inline size_t cols() const { return _cols; }\n\n  /** get the number of pixels */\n  inline size_t pixels() const { return _rows * _cols; }\n\n  /** A function to read a pixel */\n  inline double& pixel(size_t i, size_t j) { return data[vertid(i,j)]; }\n  inline double pixel(size_t i, size_t j) const { return data[vertid(i,j)]; }\n  \n  /** Linear indexing */\n  inline double& pixel(size_t i) { return data.at(i); }\n  inline double pixel(size_t i) const { return data.at(i); }\n\n  /** Get the vertex id of a pixel */\n  inline size_t vertid(size_t i, size_t j) const;\n  inline static size_t vertid(size_t rows, size_t cols, size_t i, size_t j) {\n    assert(i < rows);\n    assert(j < cols);    \n    return i * cols + j; \n  }\n  \n  \n  /** Get the pixel address from the vertex id */\n  std::pair loc(size_t vertex) const;\n\n  \n  /** A function to save the image to a file in pgm format */\n  inline void save(const char* filename, bool autorescale = true, double min=0, double max=255) const;\n\n  inline void save_vec(const char* filename) const {\n    std::ofstream os(filename);\n    assert(os.good());\n    for(size_t i = 0; i < pixels(); ++i) {\n      os << pixel(i) << \"\\n\";\n    }\n    os.flush();\n    os.close();\n  }\n\n  \n  /** paint a beautiful sunset */\n  void paint_sunset(size_t num_rings);\n  \n  /** Add random noise to the image */\n  void corrupt(double sigma);\n\n  inline double min() {\n    return *std::min_element(data.begin(), data.end());\n  }\n\n  inline double max() {\n    return *std::max_element(data.begin(), data.end());\n  }\n\n  inline void save(graphlab::oarchive &oarc) const {\n    oarc << _rows;\n    oarc << _cols;\n    oarc << data;\n  }\n  \n  inline void load(graphlab::iarchive &iarc) {\n    iarc >> _rows;\n    iarc >> _cols;\n    iarc >> data;\n  }\n\n\n};\n\n\n\n/** Generate a normally distributed random number N(mu, sigma^2) */\n// std::pair randn(double mu = 0, double sigma = 1 ); \n\n\n// IMPLEMENTATION =============================================================>\n\n\ninline void image::resize(size_t rows, size_t cols) {\n  _rows = rows;\n  _cols = cols;\n  data.resize(rows * cols, 0);\n}\n  \n\n\n/** Get the vertex id of a pixel */\ninline size_t image::vertid(size_t i, size_t j) const {\n  assert(i < _rows);\n  assert(j < _cols);    \n  return i * _cols + j; \n}\n\n// static size_t image::vertid(size_t rows, size_t cols, size_t i, size_t j)  {\n//   assert(i < rows);\n//   assert(j < cols);    \n//   return i * cols + j; \n// }\n\n\n/** Get the vertex id of a pixel */\ninline std::pair image::loc(size_t vertexid) const {\n  assert(vertexid < _rows * _cols);\n  return std::make_pair( vertexid / _cols, vertexid % _cols);\n}\n\n\ninline void image::save(const char* filename, bool autorescale, double min_, double max_) const {\n  assert(_rows > 0 && _cols > 0);\n  std::ofstream os(filename);\n  os << \"P2\" << std::endl\n     << _cols << \" \" << _rows << std::endl\n     << 255 << std::endl;\n  // Compute min and max pixel intensities\n  double min = data[0]; double max = data[0];\n  if (autorescale) {\n    for(size_t i = 0; i < _rows * _cols; ++i) {\n      min = std::min(min, data[i]);\n      max = std::max(max, data[i]);\n    }\n  }\n  else {\n    min = min_;\n    max = max_;\n  }\n  // Save the image (rescaled)\n  for(size_t r = 0; r < _rows; ++r) {\n    for(size_t c = 0; c < _cols; c++) {\n      if(min != max) {\n        int color = \n          static_cast(255.0 * (pixel(r,c) - min)/(max-min));\n        if (color < 0) color = 0;\n        if (color > 255) color = 255;\n        os << color;\n      } else { os << min; }\n      if(c != _cols-1) os << \"\\t\";\n    }\n    os << std::endl;\n  } \n  os.flush();\n  os.close();\n} // end of save\n\n\n\ninline void image::paint_sunset(size_t num_rings) {\n  const double center_r = rows() / 2.0;\n  const double center_c = cols() / 2.0;\n  const double max_radius = std::min(rows(), cols()) / 2.0;\n  // Fill out the image\n  for(size_t r = 0; r < rows(); ++r) {\n    for(size_t c = 0; c < cols(); ++c) {\n      double distance = sqrt((r-center_r)*(r-center_r) + \n                             (c-center_c)*(c-center_c));\n      // If on top of image\n      if(r < rows() / 2) {\n        // Compute ring of sunset\n        size_t ring = \n          static_cast(std::floor(std::min(1.0, distance/max_radius)\n                                         * (num_rings - 1) ) );\n        pixel(r,c) = ring;\n      } else {\n        pixel(r,c) = 0;\n      }\n    }\n  }\n} // end of paint_beatiful_sunset\n\n\n/** corrupt the image with gaussian noise */\ninline void image::corrupt(double sigma) {\n  //  boost::mt19937 rng;\n  boost::lagged_fibonacci607 rng;\n  boost::normal_distribution noise_model(0, sigma);\n  for(size_t i = 0; i < rows() * cols();  ) {\n    // Corrupt two pixels at a time.\n    pixel(i++) += noise_model(rng);\n  }\n} // end of corrupt_image\n\n\n// /** generate a normally distributed iid pair */\n// std::pair randn(double mu , double sigma ) {\n//   // Generate a N(0,1) from a Unif(0,1) using Box-Muller generator:\n//   double u1 = static_cast(rand()) / RAND_MAX;\n//   double u2 = static_cast(rand()) / RAND_MAX;\n//   double coeff = std::sqrt(-2.0 * std::log(u1));\n//   double n1 = coeff * std::cos(2.0 * M_PI * u2) ;\n//   double n2 = coeff * std::sin(2.0 * M_PI * u2) ;\n//   // Adjust for mean and variance\n//   n1 = sigma * n1 + mu;\n//   n2 = sigma * n2 + mu; \n//   return std::make_pair(n1, n2);\n// } // end of randn\n\n#include \n\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/kernelbp/old/kde.h",
    "content": "#ifndef __KDE_H\n#define __KDE_H\n\n\n#include \n#include \n#include \"assert.h\"\n#include \n\n/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *  \n *  Any changes to the code must include this original license notice in full.\n  * KDE code written by Danny Bickson, CMU\n *  Based on Matlab code by Alex Ihler, UC Irvine\n */\n\ntypedef itpp::Mat uimat;\n\ninline double square(const double v) {\n  return v * v;\n}\n\n/**\n * function for computing the log likelihood of N(u, sigma^2) at location x \n * */\ninline double log_likelihood(double u, double sigma, double x){\n  return log(1.0/sigma) - square(x - u) / (2 * sigma * sigma);\n}\n\n/**\n * class for storing a KDE\n */\nclass kde{\npublic:\n        //mixture centers\n\titpp::mat centers;\n        //mixture bandwidths\n\titpp::mat bw; \n        //mixture weights\n        itpp::vec weights;\n        //selected mixture indices\n        itpp::imat indices;\n\n        kde(){};\n\n\tkde(itpp::mat &_centers, itpp::mat &_bw, itpp::vec &_weights){\n\t    centers = _centers;\n            bw = _bw;\n            weights = _weights;\n            normalize_weights();\n        }\n\tkde(double _center, double _bw, double _weight){\n\t    centers = itpp::zeros(1,1);\n            centers.set(0,0,_center);\n            bw = itpp::zeros(1,1);\n            bw.set(0,0,_bw);\n            weights = itpp::vec(1);\n            weights[0] = _weight;\n            normalize_weights();\n        }\n \tkde(const char * _centers, const char * _bw, const char * _weights){\n\t    centers = itpp::mat(_centers);\n            bw = itpp::mat(_bw);\n            weights = itpp::vec(_weights);\n            normalize_weights();\n        }\n        kde(itpp::mat &_centers, itpp::mat &_bw, itpp::mat &_weights){\n\t    centers = _centers;\n            bw = _bw;\n            assert(_weights.rows() == 1);\n            weights = _weights.get_row(0);\n            normalize_weights();\n        }\n\tkde(itpp::mat &_centers, itpp::mat &_bw){\n\t    centers = _centers;\n            bw = _bw;\n        }\n\t/*\n \t* compute marginal along a certain dimension \t\n \t */\n        kde marginal(int dim){\n\t    assert(dim < centers.rows());\n            assert(dim >= 0);\n            itpp::mat slice = centers(dim, dim, 0, centers.cols()-1);\n            itpp::mat bwslice = bw;\n            if (bw.rows() > 1)\n                 bwslice = bw(dim, dim, 0, bw.cols() - 1); \n            return kde(slice, bwslice, weights); \n        }\n\n      /**\n        * normalize mixture weights to sum into 1\n        */\n       void normalize_weights(){\n           double sum = itpp::sum(weights);\n           assert(sum > 0);\n           weights = weights/ sum;\n       }\n\n\n        /*\n         * find the maximal mixture component\n         */\n        double max() const{\n           assert(getDim() == 1);\n           double max = -1e100;\n           int pos = -1;\n           for (int i=0; i= 0);\n           return centers(pos);\n        }\n\n       \n        /*\n         * compute likelihood of a mixutre which is composed from a weighted sum of likelihood of mixture components \n         */\n        double likelihood(const double &d) const{\n           double ret = 0;\n           for (size_t j = 0;j < (size_t)getPoints(); ++j) {\n             double curll = log_likelihood(centers(j), bw(j), d);\n             ret += weights(j) * exp(curll);\n           }\n           return ret;\n       }\n\n \n        /*\n         * verify mixture params\n         */\n        void verify() const{\n\t    assert(sum(weights) > 0);\n            assert(sumsum(bw) > 0);\n            assert(itpp::min(itpp::min(bw))>0);\n            assert(centers.rows() > 0);\n            assert(centers.cols() > 0);\n            assert(itpp::max(itpp::max(centers)) < 1e10);\n            assert(itpp::max(itpp::max(bw)) < 1e10);\n            assert(centers.size() < 1000);\n            assert(weights.size() == getPoints());\n            assert(bw.cols() == centers.cols()); \n            assert(weights.size() == centers.cols());\n            if (indices.size() > 0){\n                assert(itpp::max(indices.get_row(0))< 10*getPoints());//TODO more careful checkiung;\n                assert(itpp::min(indices.get_row(0))>=0);\n             }\n                             \n       }\n    /*\n    * return the number of dimensions \n    */\n        int getDim() const{\n            return centers.rows();\n\n     /* return the number of mixture components */\n        }\n\n        int getPoints() const{\n            return centers.cols();\n        }\n\n        /* sample from a mixture using the specified indices */\n        // points = pts(:,ind) + getBW(npd,ind).*randKernel(getDim(npd),length(ind),getType(npd));\n         kde sample(itpp::imat & ind,itpp::vec & _weights){\n             assert(sum(_weights)>0);\n             assert(itpp::max(itpp::max(ind)) < centers.cols());\n             assert(itpp::min(itpp::min(ind)) >= 0);\n             assert(itpp::max(itpp::max(ind)) < getPoints());\n             itpp::mat randN; \n             itpp::randn(getDim(), ind.size(), randN);\n             itpp::mat pts = itpp::zeros(centers.rows(), ind.size());\n             itpp::mat pbw = itpp::zeros(centers.rows(), ind.size());\n             for (int i=0; i< centers.rows(); i++){\n                for (int j=0; j< ind.size(); j++){\n                   pts.set(i,j,centers(i,ind(j)));\n                   pbw.set(i,j,bw(ind(j)));\n                }\n             }\n             itpp::mat points = pts + elem_mult(pbw, randN);\n             return kde(points, pbw, _weights);\n                    \n         }\n\n         /* sample from a mixture using random indices */\n         kde sample(){\n            itpp::ivec ind2 = itpp::randi(getPoints(), 0, getPoints() -1);\n            itpp::imat mind2(1,getPoints());\n            for (int i=0; i< getPoints(); i++)\n               mind2(0,i) = ind2(i);\n            itpp::vec weights2 = itpp::ones(getPoints());\n            return sample(mind2, weights2);\n         }\n\n\n      /* debugging functions */ \n      static void matlab_print(const itpp::ivec & data){\n          for (int i=0; i< data.size(); i++)\n           \tstd::cout<<\" \"< 0)\n            std::cout<<\"indices=[\";\n            matlab_print(indices);\n            std::cout< 1); //no meaning to compute variance over one point\n            double prop = 1.0;\n            double sig = sqrt(itpp::variance(centers.get_row(0)));\n            assert(!std::isnan(sig));\n            assert(sig > 0);\n            double h = prop*sig*powf(getPoints(),(-1.0/(4.0+getDim())));\n            bw = itpp::ones(1,getPoints()) * h; \n        }\n\n  };\n\n\n      /* unit testing */\n      /****************/\n      inline void test_marginal(){ \n          printf(\"testing marginal..\\n\");\n          itpp::mat mcenters = \"1 2 3; 3 2 1\";\n          itpp::mat mbw = \"0.5 0.5 0.2; 0.5 0.5 0.2\";\n          itpp::vec weights = \"0.2 0.3 0.4\";\n          kde k = kde(mcenters, mbw, weights);\n          assert(k.getDim() == 2);\n          assert(k.getPoints() == 3); \n          k.verify();\n          k.matlab_print();\n          kde k1 = k.marginal(0);\n          k1.matlab_print();\n          assert(k1.centers.get_row(0) == itpp::vec(\" 1 2 3\"));\n          assert(k1.bw.get_row(0) == itpp::vec(\".5 .5 .2\"));\n          assert(square(k1.weights(0) - 0.22222) < 1e-8);\n \n          kde k2 = k.marginal(1);\n          assert(k2.centers.get_row(0) == itpp::vec(\" 3 2 1\"));\n          assert(k2.bw.get_row(0) == itpp::vec(\".5 .5 .2\"));\n          assert(square(k2.weights(0) - 0.22222) < 1e-8);\n          k2.matlab_print();\n      }\n\n       inline void test_max(){ \n          printf(\"testing max..\\n\");\n          itpp::mat mcenters = \" 1  2   3    -1  -2  3    2   1\";\n          itpp::mat mbw =      \"0.5 0.5 0.2  0.5 0.5 0.2  3   2\";\n          itpp::vec weights =  \"0.2 0.3 0.4  0.1 0.05 0.05 0.05 0.05\";\n          kde k = kde(mcenters, mbw, weights);\n          k.matlab_print();\n          std::cout<\n//#include \n//#include \n#define pi 3.14152965\n\n#include \"graphlab/util/random.hpp\"\n\nusing namespace itpp;\nusing namespace std;\n\n\nvoid randv(int n, vec & ret){\n   assert(n>=1);\n   for (int i=0; i< n; i++)\n       //ret[i] = drand48();\n       ret[i] = graphlab::random::rand01();\n}\nmat randn1(int Dx, int Dy){\n  if (Dx == 0)\n    Dx = 1;\n  assert(Dy>=1);\n  mat ret = zeros(Dx,Dy);\n  vec us = zeros(ceil(Dx*Dy/2.0)*2); \n  randv(ceil(Dx*Dy/2.0)*2, us);\n  int k=0;\n  for (int i=0; i\n#include \n#include \n#include \"kde.h\"\n\n#include \"cpp/BallTreeDensity.h\"\n#include \"prob.hpp\"\n\nclass prodSampleEpsilon{\n\npublic:\n\n\n  // a little addressing formula: \n  //   to access a^th dimension of density pair (b,c)'s constant\n#define SIGVALSMAX(a,b,c) (SigValsMax + a+Ndim*b+Ndim*Ndens*c)\n#define SIGVALSMIN(a,b,c) (SigValsMin + a+Ndim*b+Ndim*Ndens*c)\n  double *SigValsMax, *SigValsMin;\n\n  //BallTreeDensity *trees;    // structure of all trees\n  std::vector trees;\n  BallTree::index *ind;      // indices of this level of the trees\n\n  double *C,*sC,*M;\n\n  double *randunif1, *randunif2, *randnorm;  // required random numbers\n  double *samples;\n  BallTree::index* indices;    // return data\n\n  double maxErr;                 // epsilon tolerance (%) of algorithm\n  double total, soFar, soFarMin; // partition f'n and accumulation\n\n  unsigned int Ndim,Ndens;   // useful constants\n  unsigned long Nsamp;\n  bool bwUniform ;\n\n  prodSampleEpsilon(){\n    SigValsMin = SigValsMax = 0;\n    ind = 0; \n    C = sC = M = 0;\n    randunif2 = randunif1 = randnorm = 0;\n    samples = 0; indices = 0;\n    maxErr = 0; total = 0; soFarMin =0; soFar = 0;\n    Ndim = 0; Ndens = 0; Nsamp = 0; bwUniform = true;\n  }\n\n  ~prodSampleEpsilon(){\n\n    mxFree(C); mxFree(sC); mxFree(M); mxFree(SigValsMin); mxFree(SigValsMax);\n  }\n\n#ifdef MEX\n  //////////////////////////////////////////////////////////////////////\n  // MEX WRAPPER\n  //////////////////////////////////////////////////////////////////////\n  void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])\n  {\n    mxArray *rNorm, *rUnif1, *rUnif2, *rsize;\n    unsigned int i,j;\n  \n    /*********************************************************************\n     ** Verify arguments and initialize variables\n     *********************************************************************/\n\n    if (nrhs != 3)\n      mexErrMsgTxt(\"Takes 3 input arguments\");\n    if (nlhs >  2)\n      mexErrMsgTxt(\"Outputs 2 results\");\n\n    Ndens = mxGetN(prhs[0]);                               // get # of densities\n    //  trees = new BallTreeDensity[Ndens];\n    trees = (BallTreeDensity*) mxMalloc(Ndens*sizeof(BallTreeDensity));\n    bwUniform = true;\n    bool allGaussians = true;\n    for (i=0;i& kdes)\n  {\n    unsigned int i;//,j;\n  \n \n\n    /*********************************************************************\n     ** Verify arguments and initialize variables\n     *********************************************************************/\n\n    bool debug = false;\n\n    Ndens = _Ndens;\n    Nsamp = _Nsamp;\n    maxErr = _maxErr;\n    assert(Ndens >= 1);\n    assert(Nsamp>= 1);\n    // get # of densities\n    //trees = new BallTreeDensity[Ndens];\n    //trees = (BallTreeDensity*) mxMalloc(Ndens*sizeof(BallTreeDensity));\n    bwUniform = true;\n    bool allGaussians = true;\n    for (i=0;i mysort;\n    mysort.sort(0,Nsamp,rUnif1);\n    randunif1 = vec2vec(&rUnif1); \n    \n    itpp::vec rUnif2 = zeros(Nsamp*Nsamp);\n    randv(Ndens*Nsamp, rUnif2);\n    randunif2 = vec2vec(&rUnif2);\n  \n    itpp::mat rNorm = zeros(Ndim*Nsamp);\n    rNorm = randn1(Ndim, Nsamp);\n    randnorm = vec2vec(&rNorm);\n\n    kde out;\n    out.centers = itpp::zeros(Ndim, Nsamp); \n    samples = out.centers._data();\n    out.indices = itpp::imat(Ndens, Nsamp);\n    indices = (BallTree::index*) out.indices._data();\n\n    SigValsMax = (double*) mxMalloc(Ndim*Ndens*Ndens*sizeof(double));  // precalc'd constants\n    SigValsMin = (double*) mxMalloc(Ndim*Ndens*Ndens*sizeof(double));  // precalc'd constants\n    C       = (double*) mxMalloc(Ndim*sizeof(double));\n    sC      = (double*) mxMalloc(Ndim*sizeof(double));\n    M       = (double*) mxMalloc(Ndim*sizeof(double));\n  \n    total =    -1; soFar = soFarMin = 0;   multiEval();  // calculate total weight\n    total = soFar; soFar = soFarMin = 0;   multiEval();  //   then sample\n\n    out.ROT();\n    out.weights = itpp::ones(1, out.getPoints())/(double)out.getPoints();\n    if (debug)\n      out.matlab_print();\n    //else {printf(\".\"); fflush(NULL);}\n    out.indices = out.indices - 1; //c++ count starts from zero\n    out.verify();\n \n \n    for (size_t i=0; i < trees.size(); i++)\n      trees[i].clean();\n    trees.clear(); \n \n    return out;\n  }\n\n#endif\n\n\n  double normConstant(void) {\n    unsigned int i,j;\n    double tmp, normConst;\n    //const double pi = 3.141592653589;\n  \n    normConst = 1;                               // precalculate influence of normalization\n    tmp = pow(2*pi,((double)Ndim)/2);\n    for (i=0;i 0);\n    double *SigNormMin = (double*) mxMalloc(Ndim*sizeof(double));\n    double *SigNormMax = (double*) mxMalloc(Ndim*sizeof(double));\n    for (i=0;i maxDiscrep) {           // also find which pair\n          maxDiscrep = maxValT - minValT;                 //   has the largest\n          maxInd0=i; maxInd1=j;                           //   discrepancy (A/B)\n        }\n      }\n      allLeaves = allLeaves && trees[i].isLeaf(ind[i]);\n    }\n    maxVal = exp(maxVal); minVal = exp(minVal);\n\n    // If the approximation is good enough,\n    if (allLeaves || fabs(maxVal - minVal) <= maxErr * (soFarMin+minVal) ) {  // APPROXIMATE\n      double add = (maxVal + minVal)/2;                   // compute contribution\n      for (i=0;i *randunif2) break;\n          }\n          randunif2++;\n          for (j=0;j size0)\n          size0 = trees[maxInd0].range(ind[maxInd0])[k];\n      for(BallTree::index k=0; k size1)\n          size1 = trees[maxInd1].range(ind[maxInd1])[k];    \n\n      split = (size0 > size1) ? maxInd0 : maxInd1;        // take the largest.\n    \n      BallTree::index current = ind[split];\n      if (!trees[split].isLeaf(current)) {\n        ind[split] = trees[split].left(current);  \n        multiEvalRecursive();                             // recurse left \n        ind[split] = trees[split].right(current);         //   and right tree\n        multiEvalRecursive();                             // restore indices \n        ind[split] = current;                             //   for calling f'n\n      }                                                   \n    }\n  }\n\n\n  void multiEval(void) {\n    unsigned int i,j;//,k;\n    //  ind = new BallTree::index[Ndens];               // construct index array  \n\n    assert(Ndens>0);\n    ind = (BallTree::index*) mxMalloc(Ndens*sizeof(BallTree::index));    // construct index array  \n    memset(ind, 0, Ndens * sizeof(BallTree::index));\n    for (i=0;i vecs;\n  vecs.push_back(k);\n  vecs.push_back(j);\n  prodSampleEpsilon prod;\n  kde out = prod.prodSampleEpsilonRun(2,48,1e-5,vecs);\n  out.matlab_print();\n  out.verify();\n}\n\n\n\n\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/deprecated/loopybp_denoise.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * This file contains an example of graphlab used for discrete loopy\n * belief propagation in a pairwise markov random field to denoise a\n * synthetic noisy image.\n *\n *  \\author Joseph Gonzalez\n */\n\n// INCLUDES ===================================================================>\n\n// Including Standard Libraries\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n\n#include \n\n// #include \"image.hpp\"\n#include \"factors/factor_includes.hpp\"\n\n\n#include \n#include   \n\n\n// Include the macro for the for each operation\n#include \n\n\n// Global variables\nbinary_factor EDGE_FACTOR;\nsize_t NCOLORS;\ndouble SIGMA;\ndouble BOUND;\ndouble DAMPING;\n\n\n\n// STRUCTS (Edge and Vertex data) =============================================>\n\n/**\n * The data associated with each variable in the pairwise markov\n * random field\n */\nstruct vertex_data : graphlab::IS_POD_TYPE {\n  float obs_color;\n  uint16_t true_color, pred_color;\n  vertex_data(float obs_color = 0, uint16_t true_color = 0) : \n    obs_color(obs_color), true_color(true_color), pred_color(obs_color) { }\n}; // End of vertex data\n\n\n/**\n * The data associated with each directed edge in the pairwise markov\n * random field\n */\nclass edge_data {\n  unary_factor messages[4];\n  size_t message_idx(size_t source_id, size_t target_id, bool is_new) {\n    return size_t(source_id < target_id)  + 2 * size_t(is_new);\n  }\npublic:\n  edge_data() { }\n  edge_data(graphlab::vertex_id_type v1,\n            graphlab::vertex_id_type v2,\n            size_t ncolors) {\n    for(size_t i = 0; i < 4; ++i) {\n      messages[i].resize(ncolors);\n      messages[i].uniform();\n    }\n    message(v1, v2).var() = v2;\n    old_message(v1, v2).var() = v2;\n    message(v2, v1).var() = v1;\n    old_message(v2, v1).var() = v1;\n  } // end of constructor\n\n  unary_factor& message(size_t source_id, size_t target_id) { \n    return messages[message_idx(source_id, target_id, true)];\n  }\n  unary_factor& old_message(size_t source_id, size_t target_id) { \n    return messages[message_idx(source_id, target_id, false)];\n  }\n  void update_old(size_t source_id, size_t target_id) { \n    old_message(source_id, target_id) = message(source_id, target_id);\n  }\n  void save(graphlab::oarchive& arc) const {\n    for(size_t i = 0; i < 4; ++i) arc << messages[i];\n  }\n  void load(graphlab::iarchive& arc) {\n    for(size_t i = 0; i < 4; ++i) arc >> messages[i];\n  }\n}; // End of edge data\n\n\n\n/**\n * The type of the distributed graph representing the MRF.\n */\ntypedef graphlab::distributed_graph graph_type;\n\n/** \n * The gather_type for the vertex program needs to compute *= in place\n * of += so we create a new type which convertes computes *= for +=.\n */\nstruct factor_product {\n  unary_factor factor;\n  factor_product(const unary_factor& factor = unary_factor()) : \n    factor(factor) { }\n  factor_product& operator+=(const factor_product& other) {\n    ASSERT_EQ(factor.arity(), other.factor.arity());\n    factor *= other.factor;\n    return *this;\n  }\n  void save(graphlab::oarchive& arc) const { arc << factor; }\n  void load(graphlab::iarchive& arc) { arc >> factor; }\n}; // end of struct factor product\n\n\n/** \n * Belief Propagation Vertex Program\n *\n */\nclass bp_vertex_program : \n  public graphlab::ivertex_program< graph_type, factor_product,\n                                    graphlab::messages::sum_priority > {\nprivate:\n  /**\n   * The belief estimate for this vertex program\n   */\n  unary_factor belief;\npublic:\n  void save(graphlab::oarchive& arc) const { arc << belief; }\n  void load(graphlab::iarchive& arc) { arc >> belief; }\n\n  /**\n   * Since the MRF is undirected we will use all edges for gather and\n   * scatter\n   */\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of gather_edges \n\n  /**\n   * Update the old message to be the new message and collect the\n   * message value.\n   */\n  gather_type gather(icontext_type& context, \n                     const vertex_type& vertex, \n                     edge_type& edge) const {\n    const vertex_type other_vertex = get_other_vertex(edge, vertex);\n    edge_data& edata = edge.data();\n    edata.update_old(other_vertex.id(), vertex.id());\n    return factor_product(edata.old_message(other_vertex.id(), vertex.id()));\n  }; // end of gather function\n\n  /**\n   * Multiply message product by node potential and update the belief.\n   */\n  void apply(icontext_type& context, vertex_type& vertex, \n             const gather_type& total) {\n    // construct the node potential\n    belief = make_potential(vertex);\n    ASSERT_EQ(belief.arity(), total.factor.arity());\n    // multiply in the rest of the message product;\n    belief *= total.factor;\n    belief.normalize();\n    // compute the predicted value\n    vertex.data().pred_color = belief.max_asg();\n  }; // end of apply\n\n  /**\n   * Since the MRF is undirected we will use all edges for gather and\n   * scatter\n   */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of scatter edges\n\n  /**\n   * Compute new message value for each edge.\n   */\n  void scatter(icontext_type& context, const vertex_type& vertex, \n               edge_type& edge) const {  \n    const vertex_type other_vertex = get_other_vertex(edge, vertex);\n    edge_data& edata = edge.data();\n    // construct the cavity\n    unary_factor cavity = belief;\n    cavity /= edata.old_message(other_vertex.id(), vertex.id());\n    cavity.normalize();\n    // compute the new message\n    unary_factor& new_message = \n      edata.message(vertex.id(), other_vertex.id());\n    const unary_factor& old_message = \n      edata.old_message(vertex.id(), other_vertex.id());\n    ASSERT_NE(&new_message, &old_message);\n    new_message.convolve(EDGE_FACTOR, cavity);\n    new_message.normalize();\n    new_message.damp(old_message, DAMPING);\n    // Compute message residual\n    const double residual = new_message.residual(old_message);  \n    context.clear_gather_cache(other_vertex);\n    // Schedule the adjacent vertex\n    if(residual > BOUND) context.signal(other_vertex, residual);\n }; // end of scatter\n\nprivate:\n  /**\n   * Construct the unary evidence potential\n   */\n  unary_factor make_potential(const vertex_type& vertex) const {\n    unary_factor potential(vertex.id(), NCOLORS);\n    const double obs = vertex.data().obs_color;\n    const double sigmaSq = SIGMA*SIGMA;\n    for(size_t pred = 0; pred < potential.arity(); ++pred) {\n      potential.logP(pred) = \n        -(obs - pred)*(obs - pred) / (2.0 * sigmaSq);\n    }\n    potential.normalize();\n    return potential;\n  } // end of make_potentail\n\n  /**\n   * Return the other vertex\n   */\n  vertex_type get_other_vertex(edge_type& edge, \n                               const vertex_type& vertex) const {\n    return vertex.id() == edge.source().id()? edge.target() : edge.source();\n  }; // end of other_vertex\n\n}; // end of class bp_vertex_program\n\n/**\n * Define the engine type\n */\n//typedef graphlab::synchronous_engine engine_type;\ntypedef graphlab::async_consistent_engine engine_type;\n\n\n/**\n * construct the synthetic image graph.\n */\nvoid create_synthetic_mrf(graphlab::distributed_control& dc,\n                          graph_type& graph, \n                          const size_t rows, const size_t cols); \n\n               \ntemplate\nstruct merge_reduce {\n  std::vector values;\n  void save(graphlab::oarchive& arc) const { arc << values; }\n  void load(graphlab::iarchive& arc) { arc >> values; }\n  merge_reduce& operator+=(const merge_reduce& other) {\n    values.insert(values.end(), other.values.begin(), \n                  other.values.end());\n    return *this;\n  }\n}; // end of merge_reduce\n\ntypedef std::pair pred_pair_type; \ntypedef merge_reduce merge_reduce_type;\n\nmerge_reduce_type pred_map_function(graph_type::vertex_type vertex) {\n  merge_reduce ret;\n  ret.values.push_back(pred_pair_type(vertex.id(), vertex.data().pred_color));\n  return ret;\n} // end of pred_map_function\n\nmerge_reduce_type obs_map_function(graph_type::vertex_type vertex) {\n  merge_reduce ret;\n  ret.values.push_back(pred_pair_type(vertex.id(), vertex.data().obs_color));\n  return ret;\n} // end of obs_map_function\n\n\n/**\n * Save the image data in the vector of pairs to an image file\n */\nvoid save_image(const size_t rows, const size_t cols,\n                const std::vector& values,\n                const std::string& fname);\n\n\n\n// MAIN =======================================================================>\nint main(int argc, char** argv) {\n  std::cout << \"This program creates and denoises a synthetic \" << std::endl\n            << \"image using loopy belief propagation inside \" << std::endl\n            << \"the graphlab framework.\" << std::endl;\n\n  // // set the global logger\n  // global_logger().set_log_level(LOG_WARNING);\n  // global_logger().set_log_to_console(true);\n\n  // Set initial values for members ------------------------------------------->\n  NCOLORS = 5;\n  SIGMA = 2;\n  BOUND = 1E-4;\n  DAMPING = 0.1;\n \n  size_t nrows = 200;\n  size_t ncols = 200;\n  double lambda = 2;\n\n  std::string smoothing = \"laplace\";\n  std::string orig_fn = \"source_img.jpeg\";\n  std::string noisy_fn = \"noisy_img.jpeg\";\n  std::string pred_fn = \"pred_img.jpeg\";\n\n  // std::string orig_fn = \"source_img.pgm\";\n  // std::string noisy_fn = \"noisy_img.pgm\";\n  // std::string pred_fn = \"pred_img.pgm\";\n\n\n\n  // Parse command line arguments --------------------------------------------->\n  graphlab::command_line_options clopts(\"Loopy BP image denoising\");\n  clopts.attach_option(\"bound\",\n                       &BOUND, BOUND,\n                       \"Residual termination bound\");\n  clopts.attach_option(\"damping\",\n                       &DAMPING, DAMPING,\n                       \"The amount of message damping (higher = more damping)\");\n  clopts.attach_option(\"ncolors\",\n                       &NCOLORS, NCOLORS,\n                       \"The number of colors in the noisy image\");\n  clopts.attach_option(\"sigma\",\n                       &SIGMA, SIGMA,\n                       \"Standard deviation of noise.\");\n  clopts.attach_option(\"nrows\",\n                       &nrows, nrows,\n                       \"The number of rows in the noisy image\");\n  clopts.attach_option(\"ncols\",\n                       &ncols, ncols,\n                       \"The number of columns in the noisy image\");\n  clopts.attach_option(\"lambda\",\n                       &lambda, lambda,\n                       \"Smoothness parameter (larger => smoother).\");\n  clopts.attach_option(\"smoothing\",\n                       &smoothing, smoothing,\n                       \"Options are {square, laplace}\");\n  clopts.attach_option(\"orig\",\n                       &orig_fn, orig_fn,\n                       \"Original image file name.\");\n  clopts.attach_option(\"noisy\",\n                       &noisy_fn, noisy_fn,\n                       \"Noisy image file name.\");\n  clopts.attach_option(\"pred\",\n                       &pred_fn, pred_fn,\n                       \"Predicted image file name.\");\n    \n\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  const bool success = clopts.parse(argc, argv);\n  if(!success) {\n    graphlab::mpi_tools::finalize();\n    return EXIT_FAILURE;\n  }\n  ///! Create a distributed control object \n  graphlab::distributed_control dc;\n  ///! display settings  \n  if(dc.procid() == 0) {\n    std::cout << \"ncpus:          \" << clopts.get_ncpus() << std::endl\n              << \"bound:          \" << BOUND << std::endl\n              << \"damping:        \" << DAMPING << std::endl\n              << \"colors:         \" << NCOLORS << std::endl\n              << \"nrows:           \" << nrows << std::endl\n              << \"ncols:           \" << ncols << std::endl\n              << \"sigma:          \" << SIGMA << std::endl\n              << \"lambda:         \" << lambda << std::endl\n              << \"smoothing:      \" << smoothing << std::endl\n              << \"scheduler:      \" << clopts.get_scheduler_type() << std::endl\n              << \"orig_fn:        \" << orig_fn << std::endl\n              << \"noisy_fn:       \" << noisy_fn << std::endl\n              << \"pred_fn:        \" << pred_fn << std::endl;\n  }\n\n  \n  \n\n  // Create synthetic images -------------------------------------------------->\n  std::cout << \"Creating a synthetic noisy image.\" << std::endl;\n  graph_type graph(dc, clopts);\n  create_synthetic_mrf(dc, graph, nrows, ncols);\n  std::cout << \"Finalizing the graph.\" << std::endl;\n  graph.finalize();\n  if(dc.procid() == 0) {\n    std::cout << \"Number of vertices: \" << graph.num_vertices() << std::endl\n              << \"Number of edges:    \" << graph.num_edges() << std::endl;\n  }\n\n  std::cout << \"Collect the noisy image. \" << std::endl;\n  merge_reduce_type obs_image = \n    graph.map_reduce_vertices(obs_map_function);\n  std::cout << \"saving the noisy image.\" << std::endl;\n  if(dc.procid() == 0) {\n    save_image(nrows, ncols, obs_image.values, noisy_fn);\n  }\n\n  // Initialze the edge factor ----------------------------------------------->\n  std::cout << \"Initializing shared edge agreement factor. \" << std::endl;\n  // dummy variables 0 and 1 and num_rings by num_rings\n  EDGE_FACTOR = binary_factor(0, NCOLORS, 0, NCOLORS);\n  // Set the smoothing type\n  if(smoothing == \"square\") {\n    EDGE_FACTOR.set_as_agreement(lambda);\n  } else {\n    EDGE_FACTOR.set_as_laplace(lambda);\n  } \n  if(dc.procid() == 0)\n    std::cout << EDGE_FACTOR << std::endl;\n\n  // Create the engine -------------------------------------------------------->\n  std::cout << \"Creating the engine. \" << std::endl;\n  engine_type engine(dc, graph, clopts);\n\n  std::cout << \"Scheduling all vertices\" << std::endl;\n  engine.signal_all();\n  std::cout << \"Starting the engine\" << std::endl;\n  engine.start();\n  const float runtime = engine.elapsed_seconds();\n  size_t update_count = engine.num_updates();\n  std::cout << \"Finished Running engine in \" << runtime \n            << \" seconds.\" << std::endl\n            << \"Total updates: \" << update_count << std::endl\n            << \"Efficiency: \" << (double(update_count) / runtime)\n            << \" updates per second \"\n            << std::endl;  \n\n\n  // Saving the output -------------------------------------------------------->\n  std::cout << \"Saving the predicted image\" << std::endl;\n  std::cout << \"Collect the noisy image. \" << std::endl;\n  merge_reduce_type pred_image = \n    graph.map_reduce_vertices(pred_map_function);\n  std::cout << \"saving the pred image.\" << std::endl;\n  if(dc.procid() == 0) {\n    save_image(nrows, ncols, pred_image.values, pred_fn);\n  }\n\n  std::cout << \"Done!\" << std::endl;\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // End of main\n\n\n\n\ngraphlab::vertex_id_type sub2ind(size_t rows, size_t cols,\n                                 size_t r, size_t c) {\n  return r * cols + c;\n}; // end of sub2ind\n\nstd::pair ind2sub(size_t rows, size_t cols,\n                           size_t ind) {\n  return std::make_pair(ind / cols, ind % cols);\n}; // end of sub2ind\n\n\nvoid create_synthetic_mrf(graphlab::distributed_control& dc,\n                          graph_type& graph,\n                          const size_t rows, const size_t cols) {\n  dc.barrier();\n  const double center_r = rows / 2.0;\n  const double center_c = cols / 2.0;\n  const double max_radius = std::min(rows, cols) / 2.0;\n \n  for(size_t r = dc.procid(); r < rows; r += dc.numprocs()) {\n    for(size_t c = 0; c < cols; ++c) {\n      // Compute the true pixel value\n      const double distance = sqrt((r-center_r)*(r-center_r) + \n                                   (c-center_c)*(c-center_c));\n      // Compute ring of sunset\n      const uint16_t ring_color =  \n        std::floor(std::min(1.0, distance/max_radius) * (NCOLORS - 1) );\n      // Compute the true pixel color by masking with the horizon\n      const uint16_t true_color = r < rows/2 ? ring_color : 0;\n      // compute the predicted color\n      const float obs_color = true_color + graphlab::random::normal(0, SIGMA);\n      // determine the true pixel id\n      const graphlab::vertex_id_type vid = sub2ind(rows,cols,r,c);\n      const vertex_data vdata(obs_color, true_color);\n      graph.add_vertex(vid, vdata);\n      // Add the edges\n      if(r + 1 < rows) \n        graph.add_edge(vid, sub2ind(rows,cols,r+1,c),\n                       edge_data(vid, sub2ind(rows,cols,r+1,c), NCOLORS));\n      if(c + 1 < cols) \n        graph.add_edge(vid, sub2ind(rows,cols,r,c+1),\n                       edge_data(vid, sub2ind(rows,cols,r,c+1), NCOLORS));\n    } // end of loop over cols\n  } // end of loop over rows\n  dc.barrier();\n}; // end of create synthetic mrf\n\n\n// void save_image(const size_t rows, const size_t cols,\n//                 const std::vector& values,\n//                 const std::string& fname) {\n//   std::cout << \"NPixels: \" << values.size() << std::endl;\n//   image img(rows, cols);\n//   foreach(pred_pair_type pair, values) \n//     img.pixel(pair.first) = pair.second;\n//   img.save(fname);\n// }\n\n\nvoid save_image(const size_t rows, const size_t cols,\n                const std::vector& values,\n                const std::string& fname) {\n  std::cout << \"NPixels: \" << values.size() << std::endl;\n  // determine the max and min colors\n  float max_color = -std::numeric_limits::max();\n  float min_color =  std::numeric_limits::max();\n  foreach(pred_pair_type pair, values) {\n    max_color = std::max(max_color, pair.second);\n    min_color = std::min(min_color, pair.second);\n  }\n\n  cv::Mat img(cols, rows, CV_8UC1);\n  foreach(pred_pair_type pair, values) {\n    std::pair coords = ind2sub(rows,cols, pair.first);\n    float value = (pair.second - min_color) / (max_color - min_color);\n    int color = 255 * value > 255 ? 255 : 255 * value;\n    img.at(coords.first, coords.second) = color;\n  }\n  cv::imwrite(fname, img);\n}\n"
  },
  {
    "path": "toolkits/graphical_models/eigen_serialization.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n#include \"eigen_serialization.hpp\"\n\n\n\ngraphlab::oarchive& operator<<(graphlab::oarchive& arc, const Eigen::VectorXd& vec) {\n  typedef Eigen::VectorXd::Index index_type;\n  typedef Eigen::VectorXd::Scalar scalar_type;\n  const index_type size = vec.size();\n  arc << size;\n  graphlab::serialize(arc, vec.data(), size * sizeof(scalar_type));\n  return arc;\n} // end of save vector\n\ngraphlab::iarchive& operator>>(graphlab::iarchive& arc, Eigen::VectorXd& vec) {\n  typedef Eigen::VectorXd::Index index_type;\n  typedef Eigen::VectorXd::Scalar scalar_type;\n  index_type size = 0;\n  arc >> size;\n  vec.resize(size);\n  graphlab::deserialize(arc, vec.data(), size * sizeof(scalar_type));\n  return arc;\n} // end of save vector\n\n\ngraphlab::oarchive& operator<<(graphlab::oarchive& arc, const Eigen::MatrixXd& mat) {\n  typedef Eigen::MatrixXd::Index index_type;\n  typedef Eigen::MatrixXd::Scalar scalar_type;\n  const index_type rows = mat.rows();\n  const index_type cols = mat.cols();\n  arc << rows << cols;\n  graphlab::serialize(arc, mat.data(), rows*cols*sizeof(scalar_type));\n  return arc;\n} // end of save matrix\n\ngraphlab::iarchive& operator>>(graphlab::iarchive& arc,  Eigen::MatrixXd& mat) {\n  typedef Eigen::MatrixXd::Index index_type; \n  typedef Eigen::MatrixXd::Scalar scalar_type;\n  index_type rows=0, cols=0;\n  arc >> rows >> cols;\n  mat.resize(rows,cols);\n  graphlab::deserialize(arc, mat.data(), rows*cols*sizeof(scalar_type));\n  return arc;\n} // end of load matrix\n"
  },
  {
    "path": "toolkits/graphical_models/eigen_serialization.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n#ifndef EIGEN_SERIALIZATION_HPP\n#define EIGEN_SERIALIZATION_HPP\n\n\n\n#include \n\n#include \n\n\n\nBEGIN_OUT_OF_PLACE_SAVE(arc, Eigen::VectorXd, vec) {\n  typedef Eigen::VectorXd::Index index_type;\n  typedef Eigen::VectorXd::Scalar scalar_type;\n  const index_type size = vec.size();\n  arc << size;\n  graphlab::serialize(arc, vec.data(), size * sizeof(scalar_type));\n} END_OUT_OF_PLACE_SAVE()\n\n\nBEGIN_OUT_OF_PLACE_LOAD(arc, Eigen::VectorXd, vec) {\n  typedef Eigen::VectorXd::Index index_type;\n  typedef Eigen::VectorXd::Scalar scalar_type;\n  index_type size = 0;\n  arc >> size;\n  vec.resize(size);\n  graphlab::deserialize(arc, vec.data(), size * sizeof(scalar_type));\n} END_OUT_OF_PLACE_LOAD()\n\n\nBEGIN_OUT_OF_PLACE_SAVE(arc, Eigen::MatrixXd, mat) {\n  typedef Eigen::MatrixXd::Index index_type;\n  typedef Eigen::MatrixXd::Scalar scalar_type;\n  const index_type rows = mat.rows();\n  const index_type cols = mat.cols();\n  arc << rows << cols;\n  graphlab::serialize(arc, mat.data(), rows*cols*sizeof(scalar_type));\n} END_OUT_OF_PLACE_SAVE()\n\n\nBEGIN_OUT_OF_PLACE_LOAD(arc, Eigen::MatrixXd, mat) {\n  typedef Eigen::MatrixXd::Index index_type; \n  typedef Eigen::MatrixXd::Scalar scalar_type;\n  index_type rows=0, cols=0;\n  arc >> rows >> cols;\n  mat.resize(rows,cols);\n  graphlab::deserialize(arc, mat.data(), rows*cols*sizeof(scalar_type));\n} END_OUT_OF_PLACE_LOAD()\n\n\n\n\n\n// inline graphlab::oarchive& operator<<(graphlab::oarchive& arc, const Eigen::VectorXd& vec) {\n//   typedef Eigen::VectorXd::Index index_type;\n//   typedef Eigen::VectorXd::Scalar scalar_type;\n//   const index_type size = vec.size();\n//   arc << size;\n//   graphlab::serialize(arc, vec.data(), size * sizeof(scalar_type));\n//   return arc;\n// } // end of save vector\n\n// inline graphlab::iarchive& operator>>(graphlab::iarchive& arc, Eigen::VectorXd& vec) {\n//   typedef Eigen::VectorXd::Index index_type;\n//   typedef Eigen::VectorXd::Scalar scalar_type;\n//   index_type size = 0;\n//   arc >> size;\n//   vec.resize(size);\n//   graphlab::deserialize(arc, vec.data(), size * sizeof(scalar_type));\n//   return arc;\n// } // end of save vector\n\n\n// inline graphlab::oarchive& operator<<(graphlab::oarchive& arc, const Eigen::MatrixXd& mat) {\n//   typedef Eigen::MatrixXd::Index index_type;\n//   typedef Eigen::MatrixXd::Scalar scalar_type;\n//   const index_type rows = mat.rows();\n//   const index_type cols = mat.cols();\n//   arc << rows << cols;\n//   graphlab::serialize(arc, mat.data(), rows*cols*sizeof(scalar_type));\n//   return arc;\n// } // end of save matrix\n\n// inline graphlab::iarchive& operator>>(graphlab::iarchive& arc,  Eigen::MatrixXd& mat) {\n//   typedef Eigen::MatrixXd::Index index_type; \n//   typedef Eigen::MatrixXd::Scalar scalar_type;\n//   index_type rows=0, cols=0;\n//   arc >> rows >> cols;\n//   mat.resize(rows,cols);\n//   graphlab::deserialize(arc, mat.data(), rows*cols*sizeof(scalar_type));\n//   return arc;\n// } // end of load matrix\n\n\n#endif\n"
  },
  {
    "path": "toolkits/graphical_models/factors/CMakeLists.txt",
    "content": "project(GraphLab)\n# add_library(factors STATIC\n#   binary_factor.cpp\n#   discrete_variable.cpp\n#   unary_factor.cpp)\n\nsubdirs(tests)\n"
  },
  {
    "path": "toolkits/graphical_models/factors/bp_graph_data.h",
    "content": "/**  \n *  Software submitted by \n *  Systems & Technology Research / Vision Systems Inc., 2013\n *\n *  Approved for public release; distribution is unlimited. [DISTAR Case #21428]\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef VSI_BP_GRAPH_DATA_H\n#define VSI_BP_GRAPH_DATA_H\n\n#include \n\n#include \n#include \n#include \n\n#include \n#include \n\n#include \"table_factor.hpp\"\n\n\nnamespace belief_prop {\n\n\n/**\n * The type of the distributed graph representing the Factor Graph.\n */\ntemplate class vertex_data;\ntemplate class edge_data;\n\ntemplate\nstruct graph_type {\n  typedef graphlab::distributed_graph, edge_data > type;\n};\n\n\n// Edge and Vertex data =============================================>\n\n/**\n * The data associated with each variable in the factor graph\n */\ntemplate\nclass vertex_data {\n  typedef graphlab::table_factor factor_type;\n\npublic:\n  double DAMPING;\n  double BOUND;\n  double REGULARIZATION;\n\n  std::string name;\n  // might be nice to be able to use different datatypes, however any\n  // such class would be required to implement the same protocols as \n  // table_factor (and table_base if it was to use bp_vertex_program) \n  factor_type potential;\n  factor_type belief;\n  bool        isVariable; // is the vertex a variable or a factor \n\n  vertex_data() : \n      DAMPING(0.0),\n      BOUND(0.0),\n      REGULARIZATION(0.0),\n      name(\"\"),\n      potential(factor_type::nil), \n      belief(factor_type::nil),\n      isVariable(false) { } \n\n  vertex_data(const factor_type& potential_, \n              const factor_type& belief_, \n              bool isVariable_,\n              std::string name_ = \"\") : \n      DAMPING(0.0), \n      BOUND(0.0), \n      REGULARIZATION(0.0),\n      name(name_), \n      potential(potential_), \n      belief(belief_),\n      isVariable(isVariable_) { }\n  \n  void load(graphlab::iarchive& arc) {\n    arc >> DAMPING;\n    arc >> BOUND;\n    arc >> REGULARIZATION;\n    arc >> name;\n    arc >> potential;\n    arc >> belief;\n    arc >> isVariable;\n  }\n  void save(graphlab::oarchive& arc) const {\n    arc << DAMPING;\n    arc << BOUND;\n    arc << REGULARIZATION;\n    arc << name;\n    arc << potential;\n    arc << belief;\n    arc << isVariable;\n  }\n}; // End of vertex data\n\n\n/**\n * The data associated with each edge in the factor graph\n */\ntemplate\nclass edge_data {\n  // REVIEW this could be a dense_table<1>, but not sure how operations\n  // on a dense_table<16> would work\n  typedef graphlab::dense_table msg_type;\n\n  msg_type messages[4];\n\n  size_t message_idx(size_t source_id, size_t target_id, bool is_new) {\n    return size_t(source_id < target_id) + 2 * size_t(is_new);\n  }\npublic:\n  edge_data() { \n    for(size_t i = 0; i < 4; ++i) {\n      messages[i] = msg_type();\n    }\n  } // end of constructor\n\n  edge_data(const msg_type& msg) {\n    for(size_t i = 0; i < 4; ++i) {\n      messages[i] = msg;\n    }\n  } // end of constructor\n\n  msg_type& message(size_t source_id, size_t target_id) { \n    return messages[message_idx(source_id, target_id, true)];\n  }\n  msg_type& old_message(size_t source_id, size_t target_id) { \n    return messages[message_idx(source_id, target_id, false)];\n  }\n  void update_old(size_t source_id, size_t target_id) { \n    old_message(source_id, target_id) = message(source_id, target_id);\n  }\n  void save(graphlab::oarchive& arc) const {\n    for(size_t i = 0; i < 4; ++i) arc << messages[i];\n  }\n  void load(graphlab::iarchive& arc) {\n    for(size_t i = 0; i < 4; ++i) arc >> messages[i];\n  }\n}; // End of edge data\n\n\n} // end of namespace belief_prop\n\n#endif // VSI_BP_GRAPH_DATA_H\n"
  },
  {
    "path": "toolkits/graphical_models/factors/bp_vertex_program.hpp",
    "content": "/**  \n *  Software submitted by \n *  Systems & Technology Research / Vision Systems Inc., 2013\n *\n *  Approved for public release; distribution is unlimited. [DISTAR Case #21428]\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef VSI_BP_VERTEX_PROGRAM_HPP\n#define VSI_BP_VERTEX_PROGRAM_HPP\n\n/**\n * This file defines the max-sum vertex program for beleif propagation.\n *\n * \\author Scott Richardson     10/2012\n */\n\n#include \n\n#include \n#include \n#include \n\n#include \n#include \n\n#include \"table_factor.hpp\"\n#include \"bp_graph_data.h\"\n\n\nnamespace belief_prop {\n\n\n/** \n * The gather_type for the vertex program needs to compute *= in place\n * of += so we create a new type which computes *= for +=.\n */\ntemplate\nclass factor_product {\n  typedef graphlab::table_factor    factor_type;\n\npublic:\n  factor_type factor;\n  // REVIEW deep copying the factor around could get expensive.\n  //   after profiling, it seems most of this is either negligible or gets\n  //   optimized out\n  factor_product(const factor_type& factor = factor_type()) : \n      factor(factor) { }\n  factor_product& operator+=(const factor_product& other) {\n    DCHECK_EQ(factor.table()->numel(), other.factor.table()->numel());\n    factor *= other.factor;\n    return *this;\n  }\n  void save(graphlab::oarchive& arc) const { arc << factor; }\n  void load(graphlab::iarchive& arc) { arc >> factor; }\n}; // end of struct factor product\n\n\n/** \n * Belief Propagation Vertex Program. As implemented, this \"program\" \n * performs the max-sum algorithm. GraphLab runs this program at \n * every vertex.\n * \n * \\author Scott Richardson\n */\ntemplate\nclass bp_vertex_program : \n    public graphlab::ivertex_program< typename graph_type::type, \n                                      factor_product,\n                                      graphlab::messages::sum_priority >\n{\n  // unfortunately this is necessary...from C++ Standard 14.6.2/3:\n  // \"In the definition of a class template or a member of a class template, if a\n  // base class of the class template depends on a template-parameter, the base \n  // class scope is not examined during unqualified name lookup either at the \n  // point of definition of the class template or member or during an instantiation \n  // of the class template or member.\n  typedef graphlab::ivertex_program< typename graph_type::type, \n                                     factor_product,\n                                     graphlab::messages::sum_priority > ivertex_program_t;\n  // NOTE there is a bug in GCC < 4.7 which prevents these using declarations from \n  // compiling (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=14258)\n  //using typename ivertex_program_t::edge_dir_type;\n  //using typename ivertex_program_t::vertex_type;\n  //using typename ivertex_program_t::edge_type;\n  typedef typename ivertex_program_t::edge_dir_type edge_dir_type;\n  typedef typename ivertex_program_t::vertex_type   vertex_type;\n  typedef typename ivertex_program_t::edge_type     edge_type;\n\n  typedef vertex_data             vertex_data_t;\n  typedef edge_data               edge_data_t;\n  typedef graphlab::table_factor  factor_type; // vertex_data_t::factor_type\n  typedef graphlab::dense_table   msg_type;    // edge_data_t::msg_type\n\npublic:\n  //using typename ivertex_program_t::gather_type;\n  //using typename ivertex_program_t::icontext_type;\n  typedef typename ivertex_program_t::gather_type gather_type;\n  typedef typename ivertex_program_t::icontext_type icontext_type;\n\npublic:\n  bp_vertex_program() { }\n\n  /**\n   * Since we are handling edge direction ourselves, we will use all edges for \n   * gather and scatter\n   */\n  edge_dir_type gather_edges(icontext_type& context,\n                       const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of gather_edges \n\n  /**\n   * Update the old message to be the new message and collect the\n   * message value.\n   */\n  gather_type gather(icontext_type& context, \n               const vertex_type& vertex, edge_type& edge) const {\n    // NOTE While gather() technically has a non-const reference to the  \n    // source and target vertex data through edge.source() and edge.target(), \n    // it should not modify them. The data on the edge (accessible through \n    // edge.data()) is modifiable however.\n    const vertex_data_t&  vdata = vertex.data();\n    const vertex_type     other_vertex = get_other_vertex(edge, vertex);\n    const vertex_data_t&  ovdata = other_vertex.data();\n    edge_data_t& edata = edge.data();\n    logstream(LOG_DEBUG) << \"in bp_vertex_program::gather(): compute message to '\" \n        << vdata.name << \"' from vertex '\" << ovdata.name << \"'\" << std::endl;\n\n    // Update the old message with the value of the new message. We\n    // then receive the old message during gather and then compute the\n    // \"cavity\" during scatter (again using the old message).\n    edata.update_old(other_vertex.id(), vertex.id());\n    msg_type& msg = edata.old_message(other_vertex.id(), vertex.id());\n    logstream(LOG_DEBUG) << \"edata=\" << msg << std::endl;\n\n    gather_type& rep = repmat(msg, vdata);\n    logstream(LOG_EVERYTHING) << \"repmat-ed msg=\" << rep.factor << std::endl;\n\n    logstream(LOG_DEBUG) << \"END bp_vertex_program::gather()\" << std::endl; \n    return rep;\n  }; // end of gather function\n\n  /**\n   * Multiply message product by node potential and update the belief.\n   */\n  void apply(icontext_type& context, vertex_type& vertex, \n       const gather_type& total) {\n    if(vertex.num_in_edges() + vertex.num_out_edges() == 0) return; \n\n    // factor_type knows it is in log space (so it adds)\n    vertex_data_t& vdata = vertex.data();\n\n    logstream(LOG_DEBUG) << \"in bp_vertex_program::apply(): vertex = '\" \n        << vdata.name << \"'\" << std::endl;\n\n    //vdata.belief = vdata.potential * total.factor;\n    vdata.belief = vdata.potential;\n    //vdata.belief.table()->copy_onto(*(vdata.potential.table())); // should be faster...\n    vdata.belief *= total.factor;\n    logstream(LOG_EVERYTHING) << \"vdata.potential=\" << vdata.potential << std::endl;\n    logstream(LOG_EVERYTHING) << \"total.factor=\" << total.factor << std::endl;\n    logstream(LOG_EVERYTHING) << \"vdata.belief=vdata.potential * total.factor = \" \n        << vdata.belief << std::endl;\n    if(vdata.isVariable == true) {\n      logstream(LOG_INFO) << \"belief-prop variable state = '\" \n          << vdata.name << \"' \"\n          << vdata.belief << std::endl;\n    }\n\n    DCHECK_GT(vdata.belief.table()->numel(), 0);\n    // Rescale the belief to ensure numerical stability. (This is\n    // essentially normalization in log-space.)\n    // REVIEW is this needed to match belief_prop\n    //vdata.belief.table()->shift_normalize();\n    //logstream(LOG_DEBUG) << \"vdata.shift_normalized=\" << vdata.belief << std::endl;\n    logstream(LOG_DEBUG) << \"END bp_vertex_program::apply()\" << std::endl;\n  }; // end of apply\n\n  /**\n   * Since we are handling edge direction ourselves, we will use all edges for \n   * gather and scatter\n   */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of scatter edges\n\n  /**\n   * Compute new message value for each edge.\n   */\n  void scatter(icontext_type& context, const vertex_type& vertex, \n               edge_type& edge) const {\n    const vertex_data_t&  vdata = vertex.data();\n    const vertex_type     other_vertex = get_other_vertex(edge, vertex);\n    const vertex_data_t&  ovdata = other_vertex.data();\n    edge_data_t& edata = edge.data();\n    logstream(LOG_DEBUG)\n        << \"in bp_vertex_program::scatter(): compute message from '\" << vdata.name \n        << \"' to vertex '\" << ovdata.name << \"'\" << std::endl;\n\n    // construct the cavity\n    //factor_type cavity = vdata.belief;\n    factor_type& cavity = init_factor(vdata.belief).factor; // initilizes the cache factor\n    logstream(LOG_EVERYTHING) << \"factor=\" << cavity << std::endl;\n    const msg_type& incoming_message = edata.old_message(other_vertex.id(), vertex.id());\n    cavity /= incoming_message;\n    logstream(LOG_DEBUG) << \"incoming_message=\" << incoming_message << std::endl;\n    logstream(LOG_EVERYTHING) << \"cavity=\" << cavity << std::endl;\n    //cavity.table()->normalize();\n\n    // compute the new outgoing message\n    const msg_type& old_message = \n        edata.old_message(vertex.id(), other_vertex.id());\n    msg_type& new_message = \n        edata.message(vertex.id(), other_vertex.id());\n    DCHECK_NE(&new_message, &old_message);\n\n    // max-product algorithm \n    cavity.table()->MAP(new_message);\n    // sum-product algorithm\n    //cavity.table()->marginalize(new_message);\n\n\n    if(vdata.REGULARIZATION > 0.0) {\n      // shift normalize\n      new_message.shift_normalize(); \n      //logstream(LOG_DEBUG) << \"normalized=\" << new_message << std::endl;\n      // regularize\n      logstream(LOG_DEBUG) << \"regularization_value=\" << vdata.REGULARIZATION << std::endl;\n      msg_type reg(new_message.domain());\n      reg.uniform(1.0);\n      new_message.damp(reg, vdata.REGULARIZATION);\n      logstream(LOG_DEBUG) << \"regularized=\" << new_message << std::endl;\n    }\n\n    // shift normalize\n    new_message.shift_normalize(); \n    logstream(LOG_DEBUG) << \"normalized=\" << new_message << std::endl;\n    msg_type raw_message(new_message);\n\n    // dampen\n    new_message.damp(old_message, vdata.DAMPING);\n    logstream(LOG_DEBUG) << \"damped=\" << new_message << std::endl;\n\n    // Compute message residual\n    //const double residual = new_message.l1_diff(old_message);\n    // dec: The literature seems to indicate that the l_inf norm is better, so lets try that.\n    const double residual = new_message.linf_diff(old_message);\n    logstream(LOG_INFO) << \"residual=\" << residual << std::endl;\n\n    context.clear_gather_cache(other_vertex);\n    // to prevent drift, we may want to only update the new_message   \n    // if the residual is greater than the BOUND because we only \n    // signal the neighboring node when this is true. however this\n    // doesnt work as well as expected...\n    logstream(LOG_INFO) << \"belief-prop message from '\" \n        << vdata.name << \"' to vertex '\" << ovdata.name << \"':\" \n        << \" raw=\" << new_message // NOTE newlines here can break atomicity...\n        << \" damped=\" << raw_message \n        << std::endl;\n    // Schedule the adjacent vertex\n    if(residual > vdata.BOUND) {\n      context.signal(other_vertex, residual);\n    }\n//    else {\n//      new_message = old_message; \n//    } \n\n    logstream(LOG_DEBUG) << \"END bp_vertex_program::scatter()\" << std::endl;\n  }; // end of scatter\n\n  /** Save the values to a binary archive */\n  // NOTE no need to serialize the contents of cache, although it is not POD, \n  // so we must serialize something\n  void save(graphlab::oarchive& arc) const { arc << gather_type(); }\n\n  /** Read the values from a binary archive */\n  void load(graphlab::iarchive& arc) { arc >> cache; }  \n\nprivate:\n  /**\n   * Initilize 'gather_type cache'---a factor that is re-used to avoid costly \n   * data-structure construction.\n   */\n  // REVIEW not sure if this would work in async mode\n  gather_type& init_factor(const factor_type& other) const {\n    if(cache.factor.table_storage() == factor_type::nil) {\n      cache.factor = other;\n    } else {\n      cache.factor.table()->copy_onto(*(other.table()));\n    }\n\n    return cache;\n  }\n  /** \n   * Return msg copied (broadcasted) across the domain defined by vdata\n   */\n  // you cant multiply (or add) two edge-messages because the intersection of\n  // their domains is null (cf. dense_table::operator*() => dense_table::logP(asg) => \n  // discrete_assignment::restrict() fails), so i repmat the message to cover the domain\n  gather_type& repmat(const msg_type& msg, const vertex_data_t& vdata) const {\n    gather_type& ones = init_factor(vdata.belief);\n    //ones = vdata.belief;\n    ones.factor.table()->zero();\n    // NOTE implicit broadcasting\n    // NOTE factor_type knows it is in log space (so it adds)\n    ones.factor *= msg;\n\n    return ones;\n  }\n  /**\n   * Return the other vertex\n   */\n  const vertex_type get_other_vertex(edge_type& edge, \n                               const vertex_type& vertex) const {\n    return vertex.id() == edge.source().id() ? edge.target() : edge.source();\n  }\n\nprivate:\n  // keeping a cache here has very good storage requirements as opposed to in vector_data. \n  // NOTE no need to serialize.\n  mutable gather_type cache;\n}; // end of class bp_vertex_program\n\n\n} // end of namespace belief_prop\n\n#endif // VSI_BP_VERTEX_PROGRAM_HPP\n"
  },
  {
    "path": "toolkits/graphical_models/factors/dense_table.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef DENSE_TABLE_HPP\n#define DENSE_TABLE_HPP\n\n\n/**\n * This file contains the definitions of some of the basic factor\n * types needed for loopy belief propagation. \n *\n *  \\author Joseph Gonzalez\n *  \\author Scott Richardson     09/2012\n *          \n */\n\n\n\n// INCLUDES ===================================================================>\n\n// Including Standard Libraries\n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n\n\n// Random number generation\n#include \n\n#include \n\n#include \"discrete_variable.hpp\"\n#include \"discrete_domain.hpp\"\n#include \"discrete_assignment.hpp\"\n#include \"fast_discrete_assignment.hpp\"\n#include \"table_base.hpp\"\n\n\n// Include the macro for the for each operation\n#include \nnamespace graphlab {\n\n  /**\n   * An n-D table up to max_dim dimensions. \n   * NOTE this table stores the data in log-space, although this \n   * implementation detail is abstracted away. E.g., operator* actually\n   * adds values in log-space.\n   * NOTE you can use begin()/end() to iterate over all assignments\n   * in the domain. for example, a domain {var1[0,4), var2[0,5)} has two \n   * dimensions, var1 and var2, with four and five labels respectively.\n   * logP()/set_logP() provide access to the underlying data via an \n   * assignment, e.g., [0,3]. the data in the table is serialized \n   * according to the linear indexing of the domain, which is ordered \n   * such that the variable with the lowest id iterates fastest.\n   */\n  template\n  class dense_table_impl : public table_base {\n  public:\n    typedef discrete_variable            variable_type;\n    typedef discrete_domain     domain_type;\n    typedef discrete_assignment assignment_type;\n    typedef table_base          table_base_t;\n\n\n    /** Construct an empty table */\n    dense_table_impl() { }\n    \n    /** Construct a table over the given domain */\n    // dom : the domain over which the table is defined \n    dense_table_impl(const domain_type& dom) :\n      _args(dom), _data(dom.size()) {  }\n\n    /** Construct a dense table over the given domain */\n    dense_table_impl(const std::vector &args) { \n      // Construct the arguments (which will remap the domain)\n      set_domain(domain_type(args));\n    }\n\n    /** Construct a dense table over the given domain \n     * dom  : the domain over which the table is defined \n     * data : a vector of values serialized according to dom; that is, such\n     *        that the variable with the smallest id iterates fastest \n     */\n    dense_table_impl(const domain_type& dom, const std::vector &data) :\n      _args(dom), _data(dom.size()) \n    {\n      set_data(data);\n    }\n\n    /** Construct a dense table over the given domain \n     * vars : a vector of variables that compose the domain\n     * data : a vector of values serialized such that the first \n     *        variable in vars iterates the fastest\n     * NOTE this is a convenience constructor. the entries in the \n     * vector are re-sorted such that the variable with the smallest\n     * id iterates fastest\n     * REVIEW make these static factory methods:\n     * e.g., static dense_table& table_from_serialized_data(...)\n     */\n    dense_table_impl(const std::vector& vars, \n        const std::vector& data) \n    { \n      // Construct the arguments (which will remap the domain)\n      set_domain(domain_type(vars));\n\n      // create a faux domain with the size of the dimensions ordered correctly. this\n      // is essentially a permute operation.\n      domain_type dom;\n      for(size_t i=0; i asgs(asg.begin(), asg.end());\n        assignment_type fast_asg(vars, asgs);\n        set_logP(fast_asg, data.at(i));\n      }\n    }\n\n    /** Construct a unary table factor over the given var */\n    dense_table_impl(const variable_type &var) { \n      // Construct the arguments (which will remap the domain)\n      set_domain(domain_type(var));\n    }\n\n    /** Construct a unary dense table over the given var */\n    dense_table_impl(const variable_type& var, std::vector& logd) { \n      // Construct the arguments (which will remap the domain)\n      set_domain(domain_type(var));\n\n      set_data(logd);\n    }\n\n    /** Construct a unary dense table over the given var */\n    dense_table_impl(const variable_type& var, double const* const begin, \n        double const* const end) \n    { \n      // Construct the arguments (which will remap the domain)\n      set_domain(domain_type(var));\n\n      set_data(begin, end - begin);\n    }\n\n  // NOTE currently, implementing the (big) three isnt strictly necessary\n    /** Construct a copy */\n    dense_table_impl(const dense_table_impl& other) :\n      _args(other._args), _data(other._data) { }\n\n    /** Destructor */\n    virtual ~dense_table_impl() { }\n\n    /** Standard assignment operator */\n    dense_table_impl& operator=(const dense_table_impl& other) {\n      if(this == &other) \n        return *this;\n\n      _args = other._args;\n      _data = other._data;\n      return *this;\n    }\n\n  private:\n    void set_data(const std::vector &data) {\n      DCHECK_EQ(_data.size(), data.size());\n      DCHECK_EQ(_data.size(), _args.size());\n      // i need this for copy_onto()\n      //ASSERT_EQ(_args.num_vars(), 1);\n\n      std::replace_copy_if(data.begin(), data.end(), _data.begin(), \n          isless(APPROX_LOG_ZERO()), APPROX_LOG_ZERO());\n      //ASSERT_TRUE(is_finite());\n    }\n    void set_data(const double data[], const size_t n) {\n      DCHECK_EQ(_data.size(), n);\n      DCHECK_EQ(_data.size(), _args.size());\n      // i need this for copy_onto()\n      //ASSERT_EQ(_args.num_vars(), 1);\n\n      std::replace_copy_if(data, data+n, _data.begin(), \n          isless(APPROX_LOG_ZERO()), APPROX_LOG_ZERO());\n      //ASSERT_TRUE(is_finite());\n    }\n\n    struct isless {\n      double _val;\n      isless(const double& val) : _val(val) { }\n      bool operator() (double number) { return number < _val; }\n    };\n\n  public:\n    using table_base_t::APPROX_LOG_ZERO;\n\n    // TODO implement operator== and operator!=\n\n    using table_base_t::copy_onto;\n\n    dense_table_impl& copy_onto(const dense_table_impl& other) {\n      if(this == &other)\n        return *this;\n\n      DCHECK_EQ(args(), other.args());\n\n      set_data(other._data);\n      return *this;\n    }\n\n    void set_args(const domain_type& args) {\n      _args = args;\n      _data.resize(args.size());\n    }\n    void set_domain(const domain_type& args) {\n      set_args(args);\n    }\n\n    inline const domain_type& args() const {\n      return _args;\n    }\n    inline const domain_type& domain() const {\n      return args();\n    } \n    \n    // NOTE index is serialized according to the linear indexing of the domain\n    // TODO can i make this private\n    inline const double& logP(const size_t index) const {\n      DCHECK_LT(index, size());\n      return _data[index];\n    }\n    const double& logP(const assignment_type& asg) const {\n      if(asg.args() == args()) {\n        // if the assignment index matches\n        const size_t index(asg.linear_index());\n        DCHECK_LT(index, size());\n        return _data[index];\n      } else {\n        // Restrict the assignment to this domain\n        const assignment_type sub_asg = asg.restrict(_args);\n        DCHECK_LT(sub_asg.linear_index(), size());\n        return _data[sub_asg.linear_index()];\n      }\n    }\n\n  private:\n    // clip values to be greater than or equal to APPROX_LOG_ZERO\n    // NOTE not as efficient as exposing the value by reference, but safer.\n    inline void set_logP(const size_t index, const double& val) {\n      DCHECK_LT(index, size());\n      _data[index] = std::max(val, APPROX_LOG_ZERO());\n    }\n\n  public:\n    // NOTE be careful not to replace instances of logP(assignment_type) with \n    // logP(assignment_type.linear_index()) as these are not the same thing \n    // when the domains are different. \n    void set_logP(const assignment_type& asg, const double& val) {\n      if(asg.args() == args()) {\n        // if the assignment index matches\n        const size_t index(asg.linear_index());\n        DCHECK_LT(index, size());\n        _data[index] = std::max(val, APPROX_LOG_ZERO());\n      } else {\n        // Restrict the assignment to this domain\n        const assignment_type sub_asg = asg.restrict(_args);\n        DCHECK_LT(sub_asg.linear_index(), size());\n        _data[sub_asg.linear_index()] = std::max(val, APPROX_LOG_ZERO());\n      }\n    } // end of logP\n\n    size_t size() const { \n      DCHECK_EQ(_args.size(), _data.size()); \n      return _args.size(); \n    }\n    virtual size_t numel() const { return size(); } \n\n    size_t num_vars() const { return _args.num_vars(); }\n    virtual size_t ndims() const { return num_vars(); } \n\n    virtual void zero() { std::fill(_data.begin(), _data.end(), 0); }\n\n    void uniform() {\n      std::fill(_data.begin(), _data.end(), log(1.0/size()));\n    }\n   \n    void uniform(double value) {\n      std::fill(_data.begin(), _data.end(), value);\n    } \n  \n    //! ensure that sum_x this(x) = 1 \n    void normalize() {\n      //ASSERT_TRUE(is_finite());\n      // Compute the max value\n      double max_value = logP(0);\n      for(size_t i = 0; i < size(); ++i) {\n        max_value = std::max(max_value, logP(i) );\n      }\n      // scale and compute normalizing constant\n      double Z = 0.0;\n      for(size_t i = 0; i < size(); ++i) {\n        double val = logP(i) - max_value;\n        set_logP(i, val);\n        Z += exp(val);\n      }\n      // assert( !std::isinf(Z) );\n      // assert( !std::isnan(Z) );\n      // assert( Z > 0.0);\n      const double logZ(log(Z));\n      DASSERT_FALSE( std::isinf(logZ) );\n      DASSERT_FALSE( std::isnan(logZ) );\n      // Normalize\n      for(size_t i = 0; i < size(); ++i) { set_logP( i, logP(i) - logZ ); }\n      //ASSERT_TRUE(is_finite());\n    } // End of normalize\n    \n    /** \n     * Ensure that the largest value in log form is zero.  This\n     * prevents overflows on normalization. \n     */\n    void shift_normalize() {\n      //ASSERT_TRUE(is_finite());\n      // Compute the max value\n      double max_value = logP(0);\n      for(size_t i = 0; i < size(); ++i) {\n        max_value = std::max(max_value, logP(i));\n      }\n      for(size_t i = 0; i < size(); ++i) { set_logP( i, logP(i) - max_value ); }\n      //ASSERT_TRUE(is_finite());\n    }\n\n    /**\n     * Return false if any of the entries are not finite \n     */\n    bool is_finite() const { \n      for(size_t i = 0; i < size(); ++i) {\n        const bool is_inf( std::isinf( logP(i) ) );\n        const bool is_nan( std::isnan( logP(i) ) );\n        if( __builtin_expect( is_inf || is_nan, 0) ) return false;\n      }\n      return true;\n    }\n\n\n  public: \n    //! this(x) *= other(x);\n    dense_table_impl& operator*=(const dense_table_impl& other) {\n      return for_each_assignment(other, multiplies());\n    }\n\n//  //! Create a dense table on the fly\n//  dense_table_impl operator*(const dense_table_impl& other) const {\n//    dense_table_impl tbl = *this;\n//    return tbl *= other;\n//  }\n\n    //! this(x) += other(x);\n    // supports broadcasting of a sub-domain across the full domain \n    dense_table_impl& operator+=(const dense_table_impl& other) {\n      return for_each_assignment(other, plus());\n    }\n\n    //! this(x) /= other(x);\n    // supports broadcasting of a sub-domain across the full domain \n    dense_table_impl& operator/=(const dense_table_impl& other) {\n      return for_each_assignment(other, divides());\n    }\n\n//  //! Create a dense table on the fly\n//  dense_table_impl operator/(const dense_table_impl& other) const {\n//    dense_table_impl tbl = *this;\n//    return tbl /= other;\n//  }\n\n  private:\n    // NOTE we assume we are in log space\n    struct divides {\n      double operator()(const double& a, const double& b) const {\n        return a - b; \n      }\n    };\n    struct multiplies {\n      double operator()(const double& a, const double& b) const {\n        return a + b; \n      }\n    };\n    struct plus {\n      double operator()(const double& a, const double& b) const {\n        double out = log( exp(a) + exp(b) );\n        DASSERT_FALSE(std::isinf( out ));\n        DASSERT_FALSE(std::isnan( out ));\n        return out;\n      }\n    };\n\n    template\n    inline dense_table_impl& for_each_assignment(const dense_table_impl& other, \n        const Func& f) {\n      //ASSERT_TRUE(is_finite());\n      if(args() == other.args()) {\n        DCHECK_EQ(size(), other.size());\n        // More verctorizable version\n        for(size_t i = 0; i < size(); ++i) {\n          double val = f(logP(i), other.logP(i));\n          set_logP( i, val );\n          //logP(i) -= other.logP(i);\n        }\n      } else { \n        // other domain must be a subset of this domain\n        DCHECK_EQ((args() + other.args()).num_vars(), num_vars());\n\n        typename domain_type::const_iterator asg = args().begin();\n        typename domain_type::const_iterator end = args().end();\n        for( ; asg != end; ++asg) {\n          double val = f(logP(asg->linear_index()), other.logP(*asg));\n          set_logP( asg->linear_index(), val );\n        }\n      }\n      //ASSERT_TRUE(is_finite());\n      return *this;\n    }\n\n  public:\n    // Currently unused\n    //! this(x) = sum_y joint(x,y) * other(y) \n    void convolve(const dense_table_impl& joint,\n                         const dense_table_impl& other) {\n      // ensure that both tables have the same domain\n      DCHECK_EQ(args() + other.args(), joint.args());\n      // Initialize the table to zero so we can use it as an accumulator\n      uniform(0);\n      typename domain_type::const_iterator asg = joint.args().begin();\n      typename domain_type::const_iterator end = joint.args().end();\n      for( ; asg != end; ++asg) {\n        const double value =\n            exp(joint.logP(asg->linear_index()) + other.logP(*asg));\n        DASSERT_FALSE(std::isinf( value ));\n        DASSERT_FALSE(std::isnan( value ));\n        // NOTE durring this accumulation, the table is not in log space\n        //logP(*asg) += value;\n        set_logP(*asg, logP(*asg) + value);\n      }\n\n      for(size_t i = 0; i < size(); ++i) {\n        double sum = logP(i);\n        DCHECK_GE(sum, 0.0);\n        if(sum == 0) { set_logP( i, APPROX_LOG_ZERO() ); }\n        else { set_logP( i, log(sum) ); }\n      }\n    }\n    \n\n    //! this(x) = other(x, y = asg) \n    void condition(const dense_table_impl& other,\n                   const assignment_type& asg) {\n      DCHECK_EQ(args(), other.args() - asg.args());\n        \n      // create a fast assignment starting from the '0' assignment\n      // of args() and the conditioning assignment of asg\n      fast_discrete_assignment fastyasg(assignment_type(args()) & asg);\n      // transpose the remaining assignments to the start\n      fastyasg.transpose_to_start(args());\n      \n      typename domain_type::const_iterator xasg = args().begin(); \n      typename domain_type::const_iterator end = args().end();\n      for( ; xasg != end; ++xasg) {\n        // REVIEW should this be other.logP(fastasg)? since other and fastyasg \n        // dont have the same domain? (would still need to convert fastasg to \n        // a discrete_assignment)\n        set_logP( xasg->linear_index(), other.logP(fastyasg.linear_index()) );\n        ++fastyasg;\n      }\n    }\n\n\n    //! this(x) = this(x) other(x, y = asg) \n    void times_condition(const dense_table_impl& other,\n                         const assignment_type& asg) {\n      //assert(args() == other.args() - asg.args());\n        \n      // create a fast assignment starting from the '0' assignment\n      // of args() and the conditioning assignment of asg\n      fast_discrete_assignment fastyasg(assignment_type(args()) & asg);\n      // transpose the remaining assignments to the start\n      fastyasg.transpose_to_start(args());\n      if(asg.num_vars() == 0) {\n        *this *= other;\n      } else {\n        typename domain_type::const_iterator xasg = args().begin(); \n        typename domain_type::const_iterator end = args().end();\n        for( ; xasg != end; ++xasg) {\n          // REVIEW should this be other.logP(fastasg)? since other and fastyasg \n          // dont have the same domain? (would still need to convert fastasg to \n          // a discrete_assignment)\n          double val = logP(xasg->linear_index()) + other.logP(fastyasg.linear_index());\n          set_logP( xasg->linear_index(), val );\n          ++fastyasg;\n        }\n      }\n    }\n    \n    using table_base_t::marginalize;\n    \n    //! msg(x) = sum_y this(x,y) \n    void marginalize(dense_table_impl& msg) const {\n      // No need to marginalize\n      if(args() == msg.args()) {\n        // Just copy and return\n        msg = *this;\n        return;\n      }\n      // Compute the domain to remove\n      domain_type ydom = args() - msg.args();\n      DCHECK_GT(ydom.num_vars(), 0);\n          \n      fast_discrete_assignment fastyasg(args());\n      fastyasg.transpose_to_start(ydom);\n      // count the number of elements in ydom\n      size_t numel = ydom.size();\n      // Loop over x\n      typename domain_type::const_iterator xasg = msg.args().begin(); \n      typename domain_type::const_iterator end = msg.args().end();\n      for( ; xasg != end; ++xasg) {\n        double sum = 0;\n        for(size_t i = 0;i < numel; ++i) {\n          sum += exp(logP(fastyasg.linear_index()));\n          ++fastyasg;\n        }\n        DASSERT_FALSE( std::isinf(sum) );\n        DASSERT_FALSE( std::isnan(sum) );\n        DCHECK_GE(sum, 0.0);\n        if(sum == 0) \n          msg.set_logP( xasg->linear_index(), APPROX_LOG_ZERO() );\n        else \n          msg.set_logP( xasg->linear_index(), log(sum) );\n      }\n    }\n      \n    using table_base_t::MAP;\n\n    //! msg(x) = max_y this(x,y)\n    void MAP(dense_table_impl& msg) const {\n      //ASSERT_TRUE(is_finite());\n      // No need to marginalize\n      if(args() == msg.args()) {\n        // Just copy and return\n        msg = *this;\n        return;\n      }\n      // Compute the domain to remove\n      domain_type ydom = args() - msg.args();\n      DCHECK_GT(ydom.num_vars(), 0);\n        \n      fast_discrete_assignment fastyasg(args());\n      fastyasg.transpose_to_start(ydom);\n      // count the number of elements in ydom\n      size_t numel = ydom.size();\n      // Loop over x\n      typename domain_type::const_iterator xasg = msg.args().begin();\n      typename domain_type::const_iterator end = msg.args().end();\n      for( ; xasg != end; ++xasg) {\n        double maxval = APPROX_LOG_ZERO();\n        for(size_t i = 0;i < numel; ++i) {\n          maxval = std::max(maxval, logP(fastyasg.linear_index()));\n          ++fastyasg;\n        }\n        msg.set_logP( xasg->linear_index(), maxval );\n      }\n      //ASSERT_TRUE(is_finite());\n    }\n\n    //! This = other * damping + this * (1-damping) \n    void damp(const dense_table_impl& other, const double& damping) {\n      //ASSERT_TRUE(is_finite());\n      // This table must be over the same domain as the other\n      if(damping == 0) return;\n      DCHECK_EQ(args(), other.args());\n      DCHECK_GT(damping, 0.0);\n      DCHECK_LT(damping, 1.0);\n      for(size_t i = 0; i < size(); ++i) {\n        double val = damping * exp(other.logP(i)) + (1-damping) * exp(logP(i));\n        DCHECK_GE(val, 0);\n        if(val == 0) { set_logP( i, APPROX_LOG_ZERO() ); }\n        else { set_logP( i, log(val) ); }\n        DASSERT_FALSE( std::isinf(logP(i)) );\n        DASSERT_FALSE( std::isnan(logP(i)) );\n      }\n      //ASSERT_TRUE(is_finite());\n    }\n\n\n    //! compute the l_inf norm (i.e. the max diff) between two tables\n    double linf_diff(const dense_table_impl& other) const {\n      //ASSERT_TRUE(is_finite());\n      // This table must be over the same domain as the other\n      DCHECK_EQ(args(), other.args());  \n      double max_diff = 0;\n      for(size_t i = 0; i < size(); ++i) {\n        double diff = fabs(exp(other.logP(i)) - exp(logP(i)));\n        if (diff > max_diff) {\n          max_diff = diff;\n        }\n      }\n      //ASSERT_TRUE(is_finite());\n      return max_diff;\n    }\n\n    //! compute the average l1 norm between two tables\n    double l1_diff(const dense_table_impl& other) const {\n      //ASSERT_TRUE(is_finite());\n      // This table must be over the same domain as the other\n      DCHECK_EQ(args(), other.args());  \n      double sum = 0;\n      for(size_t i = 0; i < size(); ++i) {\n        sum += fabs(exp(other.logP(i)) - exp(logP(i)));\n      }\n      //ASSERT_TRUE(is_finite());\n      return sum / size(); // TODO the l1 norm is not normalized\n    }\n\n    //! compute the l1 norm in log space\n    double l1_logdiff(const dense_table_impl& other) const {\n      DCHECK_EQ(args(), other.args());\n      double sum = 0; \n      for(size_t i = 0; i < size(); ++i) {\n        sum += fabs(other.logP(i) - logP(i));\n      }\n      return sum / size(); // TODO the l1 norm is not normalized\n    }\n\n    //! argmax(): return the assignment of the largest value\n    assignment_type max_asg() const {\n      typename domain_type::iterator max_asg = args().begin();\n      double max_value = logP(max_asg->linear_index());\n\n      typename domain_type::const_iterator asg = args().begin(); \n      typename domain_type::const_iterator end = args().end();\n      for( ; asg != end; ++asg) {\n        if(logP(asg->linear_index()) > max_value) {\n          max_value = logP(asg->linear_index());\n          *max_asg = *asg;\n        }\n      }\n      return *max_asg;\n    }\n\n    //! return the linear index of the largest value\n    size_t max_index() const {\n      return max_asg().linear_index();\n    }\n\n    /**\n     * Compute the expectation of the dense table\n     */\n    inline void expectation(std::vector& values) const {\n      values.clear();\n      values.resize(num_vars(), 0);\n      double sum = 0;\n      typename domain_type::const_iterator asg = args().begin(); \n      typename domain_type::const_iterator end = args().end();\n      for( ; asg != end; ++asg) {\n      const double scale = exp(logP(asg->linear_index()));\n        sum += scale;\n        typename assignment_type::const_iterator asg_it = asg.begin();\n        for(size_t i = 0; i < num_vars(); ++i) {\n          values[i] += asg_it[i] * scale;\n        }\n      }\n      // Rescale for normalization\n      for(size_t i = 0; i < num_vars(); ++i)  values[i] /= sum;\n    } // end of expectation\n\n    /**\n     * Draw a sample from the dense table\n     */\n    inline assignment_type sample() const {\n      DCHECK_GT(size(), 0);\n      // This table must be normalized\n      const double t = graphlab::random::rand01();\n      DCHECK_GE( t, 0 );\n      DCHECK_LT( t, 1 );\n      double sum = 0;\n      for(size_t i = 0; i < size(); ++i) {\n        sum += exp( logP(i) );\n        if(t <=sum) { return assignment_type(args(), i); }\n        DCHECK_LT(sum, 1);\n      }\n      // Unreachable\n      throw(\"Invalid state reached in sample()\");\n      return assignment_type();\n    } // end of sample\n    \n\n    /**\n     * Construct a binary agreement factor\n     */\n    void set_as_agreement(const double& lambda) {\n      DCHECK_EQ(num_vars(), 2);\n\n      typename domain_type::const_iterator asg = args().begin();\n      typename domain_type::const_iterator end = args().end();\n      for( ; asg != end; ++asg) {\n        typename assignment_type::const_iterator asg_it = asg->begin();\n        const int diff = abs( int(asg_it[0]) - int(asg_it[1]) );\n        if( diff > 0) { set_logP( asg->linear_index(), -lambda ); }\n        else { set_logP( asg->linear_index(), 0 ); }\n      }\n    } // end of set_as_agreement\n    \n    void set_as_laplace(const double& lambda) {\n      DCHECK_EQ(num_vars(), 2);\n      typename domain_type::const_iterator asg = args().begin();\n      typename domain_type::const_iterator end = args().end();\n      for( ; asg != end; ++asg) {\n        typename assignment_type::const_iterator asg_it = asg->begin();\n        const int diff = abs( int(asg_it[0]) - int(asg_it[1]) );\n        set_logP( asg->linear_index(), -diff * lambda );\n      }\n    } // end of set_as_laplace\n\n  public:\n    void load(graphlab::iarchive& arc) {\n      arc >> _args;\n      arc >> _data;\n    }\n\n    void save(graphlab::oarchive& arc) const {\n      arc << _args;\n      arc << _data;\n    }\n    \n  private:\n  //! The domain of the table (the arity of the table, along with its cardinality) \n    domain_type _args;\n    // NOTE _data is ordered according to the linear indexing of our domain, \n    // which is ordered such that the variable with the lowest id iterates \n    // fastest. weird! i know!\n    std::vector _data;\n  }; // End of dense table\n\n\n\n// REVIEW could move the these methods into dense_table_impl (which \n// would become dense_table again) like in sparse_table \n  template \n  class dense_table : public dense_table_impl {\n    typedef dense_table_impl  dense_table_impl_t;\n\n    // \"using typename\" parses but doesn't work in GCC < 4.7. Use \"typedef typename\" instead.\n    //using typename dense_table_impl_t::table_base_t;\n    //using typename dense_table_impl_t::variable_type;\n    //using typename dense_table_impl_t::domain_type;\n    //using typename dense_table_impl_t::assignment_type;\n    typedef typename dense_table_impl_t::table_base_t     table_base_t;\n    typedef typename dense_table_impl_t::variable_type    variable_type;\n    typedef typename dense_table_impl_t::domain_type      domain_type;\n    typedef typename dense_table_impl_t::assignment_type  assignment_type;\n\n  public:\n    /** Construct an empty dense table */\n    dense_table() : dense_table_impl_t() { }\n    \n    /** Construct a dense table over the given domain */\n    dense_table(const domain_type& dom) :\n        dense_table_impl_t(dom) { }\n\n    /** Construct a dense table over the given domain and distribution */\n    dense_table(const domain_type& dom, const std::vector &data) :\n        dense_table_impl_t(dom, data) { }\n\n    /** Construct a dense table over the given domain */\n    dense_table(const std::vector &args) : \n        dense_table_impl_t(args) { }\n\n    /** Construct a dense table over the given domain */\n    dense_table(const std::vector &args, \n        const std::vector &data) : dense_table_impl_t(args, data) { }\n\n    // REVIEW make these static factory methods: \n    // static dense_table& unary_table_node(...) or use the virtual\n    // constructor idiom or something\n    /** Construct a unary dense table over the given var */\n    dense_table(const variable_type &var) : dense_table_impl_t(var) { }\n\n    /** Construct a unary dense table over the given var */\n    dense_table(const variable_type& var, std::vector& logd) : \n        dense_table_impl_t(var, logd) { }\n\n    /** Construct a unary dense table over the given var */\n    dense_table(const variable_type& var, \n        double const* const begin, double const* const end) :\n        dense_table_impl_t(var, begin, end) { }\n\n    /** Construct a copy */\n    dense_table(const dense_table& other) :\n        dense_table_impl_t(other) { }\n\n    virtual ~dense_table() { }\n\n    // REVIEW currently, this isnt necessary\n    /** Standard assignment operator */\n    dense_table& operator=(const dense_table& other) {\n      if(this == &other) \n        return *this;\n\n      dense_table_impl_t::operator=(other);\n\n      return *this;\n    }\n\n    friend std::ostream& operator<<(std::ostream& out,\n                              const dense_table& tbl) {\n      out << \"Dense Table: \" << tbl.args() << \"{\" << std::endl;\n      typename domain_type::const_iterator asg = tbl.args().begin();\n      typename domain_type::const_iterator end = tbl.args().end();\n      for( ; asg != end; ++asg) {\n        out << \"\\tLogP(\" << *asg << \")=\" << tbl.logP(*asg) << std::endl;\n      }\n      out << \"}\";\n\n//      dense_table::const_iterator asg = tbl.args().begin();\n//      dense_table::const_iterator end = tbl.args().end();\n//      for( ; asg != end; ++asg) {\n//        out << tbl.logP(*asg) << \" \";\n//      }\n\n      return out;\n    }\n\n  // virtual methods\n  public:\n    typedef dense_table const *const const_ptr;\n\n    virtual dense_table& deep_copy(const table_base_t& base) {\n      if(this == &base) return *this;\n\n      // ensure we are dealing with a dense_table\n      const_ptr other = dynamic_cast(&base);\n      if(other == NULL) {\n        std::cout << \"ERROR: std::bad_cast\" << std::endl;\n        // REVIEW should probably raise an exception\n        ASSERT_TRUE(false);\n      }\n\n      *this = *other;\n      return *this;\n    }\n\n    using dense_table_impl_t::copy_onto;\n\n    virtual dense_table& copy_onto(const table_base_t& base) {\n      if(this == &base) return *this;\n\n      // ensure we are dealing with a dense_table\n      const_ptr other = dynamic_cast(&base);\n      if(other == NULL) {\n        std::cout << \"ERROR: std::bad_cast\" << std::endl;\n        // REVIEW should probably raise an exception\n        ASSERT_TRUE(false);\n      }\n\n      dense_table_impl_t::copy_onto(*other);\n      return *this;\n    }\n\n    virtual const variable_type& var(const size_t index) const {\n      return dense_table_impl_t::domain().var(index);\n    }\n/*\n    //! this(x) += other(x);\n    virtual dense_table& plus_equals(const table_base_t& base) {\n      // ensure we are dealing with a dense_table\n      const_ptr other = dynamic_cast(&base);\n      if(other == NULL) {\n        std::cout << \"ERROR: std::bad_cast\" << std::endl;\n        // REVIEW should probably raise an exception\n        ASSERT_TRUE(false);\n      }\n\n      *this += *other;\n      return *this;\n    }\n*/\n    //! this(x) *= other(x);\n    virtual dense_table& times_equals(const table_base_t& base) {\n      // ensure we are dealing with a dense_table\n      const_ptr other = dynamic_cast(&base);\n      if(other == NULL) {\n        std::cout << \"ERROR: std::bad_cast\" << std::endl;\n        // REVIEW should probably raise an exception\n        ASSERT_TRUE(false);\n      }\n\n      *this *= *other;\n\n      return *this;\n    }\n\n    //! this(x) /= other(x);\n    virtual dense_table& divide_equals(const table_base_t& base) {\n      // ensure we are dealing with a dense_table\n      const_ptr other = dynamic_cast(&base);\n      if(other == NULL) {\n        std::cout << \"ERROR: std::bad_cast\" << std::endl;\n        // REVIEW should probably raise an exception\n        ASSERT_TRUE(false);\n      }\n\n      *this /= *other;\n\n      return *this;\n    }\n\n    //! (out(x) = this(x)) * other(x);\n    virtual void times(const table_base_t& base, \n        table_base_t& out_base) const {\n\n      // ensure we are dealing with a dense_table\n      dense_table *const out = \n          dynamic_cast(&out_base);\n      if(out == NULL) {\n        std::cout << \"ERROR: std::bad_cast\" << std::endl;\n        // REVIEW should probably raise an exception\n        ASSERT_TRUE(false);\n      }\n\n      *out = *this; // deep copy\n      out->times_equals(base);\n    }\n\n    //! (out(x) = this(x)) / other(x);\n    virtual void divide(const table_base_t& base, \n        table_base_t& out_base) const {\n\n      // ensure we are dealing with a dense_table\n      dense_table *const out = \n          dynamic_cast(&out_base);\n      if(out == NULL) {\n        std::cout << \"ERROR: std::bad_cast\" << std::endl;\n        // REVIEW should probably raise an exception\n        ASSERT_TRUE(false);\n      }\n\n      *out = *this; // deep copy\n      out->divide_equals(base);\n    }\n\n    using dense_table_impl_t::marginalize;\n\n    virtual void marginalize(table_base_t& base) const {\n      // ensure we are dealing with a dense_table\n      dense_table* msg = dynamic_cast(&base);\n      if(msg == NULL) {\n        std::cout << \"ERROR: std::bad_cast\" << std::endl;\n        // REVIEW should probably raise an exception\n        ASSERT_TRUE(false);\n      }\n\n      dense_table_impl_t::marginalize(*msg);\n    }\n\n    using dense_table_impl_t::MAP;\n\n    virtual void MAP(table_base_t& base) const {\n      // ensure we are dealing with a dense_table\n      dense_table* msg = dynamic_cast(&base);\n      if(msg == NULL) {\n        std::cout << \"ERROR: std::bad_cast\" << std::endl;\n        // REVIEW should probably raise an exception\n        ASSERT_TRUE(false);\n      }\n\n      dense_table_impl_t::MAP(*msg);\n    }\n    \n  public:\n    virtual void load(graphlab::iarchive& arc) { \n      dense_table_impl_t::load(arc); \n    }\n\n    virtual void save(graphlab::oarchive& arc) const {\n      dense_table_impl_t::save(arc); \n    }\n\n    virtual std::ostream& print(std::ostream& out = std::cout) const {\n      // ensure we are dealing with a dense_table\n      const_ptr tbl = dynamic_cast(this);\n      if(tbl == NULL) {\n        std::cout << \"ERROR: std::bad_cast\" << std::endl;\n        // REVIEW should probably raise an exception\n        ASSERT_TRUE(false);\n      }\n\n      out << *tbl;\n      return out;\n    }\n  };\n\n\n}; // end of namespace graphlab\n\n\n\n#include \n#endif // DENSE_TABLE_HPP\n\n"
  },
  {
    "path": "toolkits/graphical_models/factors/discrete_assignment.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n */\n\n\n#ifndef DISCRETE_ASSIGNMENT_HPP\n#define DISCRETE_ASSIGNMENT_HPP\n\n#include \n\n// Random number generation\n#include \n\n#include \"discrete_variable.hpp\"\n#include \"discrete_bounds.hpp\"\n\n\n#include \nnamespace graphlab {\n\n/**\n * This class respresents a discrete assignment on a domain, i.e, an\n * index into a domain. E.g., element [3,1] in the domain [0,5) x [0x5).\n *\n *  \\author Joseph Gonzalez\n *  \\author Scott Richardson     09/2012\n *\n */\ntemplate\nclass discrete_assignment {\n  typedef uint16_t  subasg_type;\n\npublic:\n  typedef subasg_type*         iterator;\n  typedef const subasg_type*   const_iterator;\n\npublic:\n  typedef discrete_bounds domain_type;\n  typedef discrete_variable        variable_type;\n\n  //! Construct an empty discrete_assignment\n  discrete_assignment() : _index(0) { }  \n\n  //! Construct a zero discrete_assignment over the domain\n  explicit discrete_assignment(const domain_type& args) :\n      _args(args), _index(0) {\n    for(size_t i = 0; i < args.num_vars(); ++i) \n      _asgs[i] = 0;\n  }\n  \n  //! construct an discrete_assignment from one variable\n  discrete_assignment(const variable_type& v1, size_t asg1) :\n      _args(v1), _index(asg1) {\n    assert(asg1 < v1.size());\n    _asgs[0] = asg1;\n  }\n  \n  //! construct an discrete_assignment from two variables\n  discrete_assignment(const variable_type& v1, size_t asg1, \n                      const variable_type& v2, size_t asg2) :  \n      _args(v1, v2), _index(0) {\n    set_asg(v1.id(), asg1);\n    set_asg(v2.id(), asg2);\n  }\n\n  //! construct an discrete_assignment from multiple variables\n  // NOTE each element of asg indexes into the dimension defined by the \n  // corresponding element (variable) of vars\n  discrete_assignment(const std::vector& vars, \n                      const std::vector& asg) :\n      _args(domain_type(vars)), _index(0) {\n    assert(vars.size() == asg.size());\n    // map variables to their assignment in the domain\n    for(size_t j=0; j& asg) :\n      _args(args), _index(0) {\n    for(size_t i = 0; i < _args.num_vars(); ++i) {\n      assert(asg[i] < args.var(i).size());\n      _asgs[i] = asg[i];\n    }\n    recompute_linear_index();\n  }\n\n  iterator begin() { return &(_asgs[0]); }\n  iterator end() { return &(_asgs[_args.num_vars()]); }\n  const_iterator begin() const { return &(_asgs[0]); }\n  const_iterator end() const { return &(_asgs[_args.num_vars()]); }\n\n  // //! Construct the union of two discrete_assignments\n  // inline discrete_assignment& operator&=(const discrete_assignment& asg2) {\n  //   discrete_assignment asg1 = *this;\n  //   const domain_type& dom1 = asg1.args();\n  //   const domain_type& dom2 = asg2.args();\n  //   _args = dom1 + dom2;\n  //   _index = 0;\n  //   size_t i = 0, j1 = 0, j2 = 0;\n  //   for( ; i < _args.num_vars() && \n  //          (j1 < dom1.num_vars() || j2 < dom2.num_vars()); \n  //        ++i) {\n  //     // If the the two discrete_assignments share a same variable\n  //     if(j1 < dom1.num_vars() && \n  //        _args.var(i) == dom1.var(j1) && \n  //        j2 < dom2.num_vars() &&\n  //        _args.var(i) == dom2.var(j2)) {\n  //       // Then they must have the same discrete_assignment\n  //       //          assert(asg1._asgs[j1] == asg2._asgs[j2]);\n  //       _asgs[i] = asg1._asgs[j1];\n  //       ++j1; ++j2;\n  //     } else if(j1 < dom1.num_vars() &&\n  //               _args.var(i) == dom1.var(j1) ) {\n  //       _asgs[i] = asg1._asgs[j1];\n  //       ++j1;\n  //     } else if(j2 < dom2.num_vars() &&\n  //               _args.var(i) == dom2.var(j2) ) {\n  //       _asgs[i] = asg2._asgs[j2];\n  //       ++j2;\n  //     } else {\n  //       // Unreachable state\n  //       assert(false);\n  //     }\n  //   }\n  //   assert(i == _args.num_vars());\n  //   assert(j1 == dom1.num_vars());\n  //   assert(j2 == dom2.num_vars());\n  //   recompute_linear_index();\n  //   return *this;\n  // }\n  // // Construct the union of two discrete_assignments\n  // discrete_assignment operator&(const discrete_assignment& other) const {\n  //   discrete_assignment new_asg = *this;\n  //   return new_asg &= other;\n  // }  \n\n  //! Construct the union of two discrete_assignments\n  inline discrete_assignment operator&(const discrete_assignment& other) const {\n    discrete_assignment result(args() + other.args());\n    // Require disjoint discrete_assignments\n    //      assert(args().size() + other.args().size() == result.size());\n    size_t i = 0, j = 0, k = 0;\n    while(i < num_vars() && j < other.num_vars()) {        \n      // extra increment if necessary\n      assert(k < result.num_vars());\n      result._asgs[k] = \n        (result.args().var(k) == args().var(i))?\n        asg_at(i) : other.asg_at(j);\n      // if the variables are the same then the discrete_assignments must\n      // also be the same\n      assert(!(args().var(i) == other.args().var(j)) ||\n             (asg_at(i) == other.asg_at(j)));               \n      // move indexs\n      i += (args().var(i) == result.args().var(k));\n      j += (other.args().var(j) == result.args().var(k));\n      k++;\n    }\n    while(i < num_vars()) \n      result._asgs[k++] = asg_at(i++);\n    while(j < other.num_vars()) \n      result._asgs[k++] = other.asg_at(j++); \n    // recompute the linear index of the result\n    result.recompute_linear_index();\n    return result;\n  }\n    \n  // Construct the union of two discrete_assignments\n  inline discrete_assignment& operator&=(const discrete_assignment& other) {\n    discrete_assignment tmp = *this & other;\n    *this = tmp;\n    return *this;\n  }\n  \n  //! Get the next discrete_assignment\n  discrete_assignment& operator++() {\n    assert(_index < _args.size());\n    // Increment the index\n    ++_index;\n    // Update the discrete_assignments\n    for(size_t i = 0; i < _args.num_vars(); ++i) {\n      _asgs[i]= ((_asgs[i] + 1) % _args.var(i).size());\n      if(_asgs[i] > 0) { return *this; }\n    }\n    // Reached end\n    make_end();\n    return *this;\n  }\n\n  //! Make this an ending discrete_assignment\n  const discrete_assignment& make_end() {\n    _index = -1;\n    return *this;\n    // for(size_t i = 0; i < _args.num_vars(); ++i)\n    //   _asgs[i] = _args.var(i).size();\n  }\n\n  //! Uniformly sample a new index value\n  void uniform_sample() {\n    set_index( graphlab::random::fast_uniform(size_t(0), size() - 1)  );\n  }\n  \n\n  //! get the domain\n  inline const domain_type& args() const { return _args; }\n\n  //! get the number of variables\n  inline size_t num_vars() const { return _args.num_vars(); }\n\n  //! get the size of the discrete_assignment\n  inline size_t size() const { return _args.size(); }\n\n\n  size_t asg(const variable_type& var) const {\n    return asg(var.id());\n  }\n  void set_asg(const variable_type& var, size_t value) {\n    set_asg(var.id(), value);\n  }\n\n private:\n  size_t asg(size_t var_id) const {\n    size_t index = _args.var_location(var_id);\n    assert(index < _args.num_vars());\n    return _asgs[index];\n  }\n\n  size_t asg_at(size_t index) const {\n    assert(index < _args.num_vars());\n    return _asgs[index];\n  }\n\n  void set_asg(size_t var_id, size_t value, bool recompute=true) {\n    size_t index = _args.var_location(var_id);\n    assert(index < _args.num_vars());\n    assert(value < _args.var(index).size());\n    _asgs[index] = value;\n    if(recompute) recompute_linear_index();\n  }\n\n  void set_asg_at(size_t index, size_t value, bool recompute=true) {\n    assert(index < _args.num_vars());\n    assert(value < _args.var(index).size());\n    _asgs[index] = value;\n    if(recompute) recompute_linear_index();\n  }\n\n public:\n  //! Get the index of this discrete_assignment\n  // NOTE index is serialized according to the linear indexing of the domain\n  inline size_t linear_index() const { return _index; }\n\n  //! Set the index of this discrete_assignment\n  // NOTE index is serialized according to the linear indexing of the domain\n  void set_index(size_t index) {\n    assert(index < _args.size());\n    _index = index;\n    recompute_asgs();\n  }\n\n  //! Tests whether two discrete_assignments are equal\n  bool operator==(const discrete_assignment& other) const {\n    return _index == other._index;\n  }\n  //! Tests whether two discrete_assignments are not equal\n  bool operator!=(const discrete_assignment& other) const {\n    return !this->operator==(other);\n  }\n  //! Tests whether this discrete_assignment is < other\n  bool operator<(const discrete_assignment& other) const {\n    return _index < other._index;\n  }\n  //! Tests whether this discrete_assignment is > other\n  bool operator>(const discrete_assignment& other) const {\n    return other.operator<(*this);\n  }\n  //! Tests whether this discrete_assignment is <= other\n  bool operator<=(const discrete_assignment& other) const {\n    return !this->operator>(other);\n  }\n  //! Tests whether this discrete_assignment is >= other\n  bool operator>=(const discrete_assignment& other) const {\n    return !this->operator<(other);\n  }\n  \n  \n  //! Restrict the discrete_assignment to a discrete_assignment over the sub-domain\n  discrete_assignment restrict(const domain_type& sub_domain) const {\n    // sub_domain must be a subset of this domain\n    DCHECK_EQ((_args + sub_domain).num_vars(), num_vars());\n\n    discrete_assignment other_asg(sub_domain);\n    size_t index = 0;\n    // Map the variables \n    // NOTE this depends on the list of variables in both domains being sorted\n    for(size_t i = 0; i < _args.num_vars() && \n        index < sub_domain.num_vars(); ++i) {\n      if(sub_domain.var(index) == _args.var(i)) {\n        other_asg._asgs[index] = _asgs[i];\n        index++;\n      }\n    }\n    assert(index == sub_domain.num_vars());\n    // Recompute the index\n    other_asg.recompute_linear_index();\n    return other_asg;\n  } // end of restrict\n\n  //! Update the variables in this discrete_assignment with the values in the\n  //! other discrete_assignment\n  void update(const discrete_assignment& other) {\n    // REVIEW should this domain be a subset of the other domain?\n\n    // NOTE this depends on the list of variables in both domains being sorted\n    for(size_t i = 0, j = 0;\n        i < num_vars() && j < other.num_vars(); ) {\n      if(_args.var(i) == other._args.var(j)) {\n        _asgs[i] = other._asgs[j]; i++; j++;\n      }\n      while(i < num_vars() &&\n            _args.var(i) < other.args().var(j)) i++;\n      while(j < other.num_vars() && \n            other.args().var(j) < _args.var(i)) j++;\n    }\n    recompute_linear_index();\n  }\n\n  void load(graphlab::iarchive& arc) {\n    arc >> _args;\n    arc >> _index;\n    recompute_asgs();\n  }\n  \n  void save(graphlab::oarchive& arc) const {\n    arc << _args;\n    arc << _index;\n  }\n\n  friend std::ostream& operator<<(std::ostream& out,\n                            const discrete_assignment& asg) {\n    out << \"{\";\n    for(size_t i = 0; i < asg.args().num_vars(); ++i) {\n      // TODO the v_varId is redundant. remove it\n      //out << \"v_\" << asg.args().var(i).id() << \"=\";\n      out << asg.asg_at(i);\n      if(i < asg.args().num_vars() - 1) out << \", \";\n    }\n    out << \"}=\" << asg.linear_index();\n    return out;\n  }\n\nprivate:\n  //! Recompute the index from the discrete_assignment\n  void recompute_linear_index() {\n    size_t multiple = 1;\n    // Clear the index\n    _index = 0;\n    for(size_t i = 0; i < _args.num_vars(); ++i) {\n      _index += multiple * _asgs[i];\n      // assert(_args.var(i).nasgs > 0);\n      multiple *= _args.var(i).size();\n    }\n  }\n    \n  //! Recompute the discrete_assignments from the index\n  void recompute_asgs() {\n    assert(_index < _args.size());\n    size_t quotient = _index;\n    for(size_t i = 0; i < _args.num_vars(); ++i) {\n      _asgs[i] = quotient % _args.var(i).size();\n      quotient /= _args.var(i).size();\n      // assert(_asgs[i] < _args.var(i).size());\n    }\n  }\n\n  // a discrete domain over a set of variables, e.g., {v1, v2}\n  domain_type _args;\n  // an assignment on the domain, e.g., [3,1] (recomputed anytime\n  // _index is updated with recompute_asgs())\n  subasg_type _asgs[MAX_DIM];\n  // the linear index of the assignment _asgs (recomputed anytime _asgs is \n  // updated with recompute_linear_index())\n  uint32_t _index;\n};\n\n\n}; // end of namespace graphlab\n\n#include \n#endif // DISCRETE_ASSIGNMENT_HPP\n\n"
  },
  {
    "path": "toolkits/graphical_models/factors/discrete_bounds.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n */\n\n\n#ifndef DISCRETE_BOUNDS_HPP\n#define DISCRETE_BOUNDS_HPP\n\n#include \n\n#include \"discrete_variable.hpp\"\n\n\n#include \nnamespace graphlab {\n\n\n/**\n * This class respresents a discrete domain over a set of variables.\n */\ntemplate\nclass discrete_bounds {\npublic:\n  //! Make an empty domain\n  discrete_bounds() : _num_vars(0) { }\n\n  //! Make a single variable discrete_bounds\n  discrete_bounds(const discrete_variable& v1) :\n      _num_vars(1) \n  {\n    DCHECK_LE(_num_vars, MAX_DIM);\n    _vars[0] = v1;\n  }\n\n  //! Make a two variable discrete_bounds\n  discrete_bounds(const discrete_variable& v1, const discrete_variable& v2) :\n      _num_vars(2) \n  {\n    DCHECK_LE(_num_vars, MAX_DIM);\n    assert(v1 != v2);\n    if(v1 < v2) {\n      _vars[0] = v1;\n      _vars[1] = v2;\n    } else {\n      _vars[0] = v2;\n      _vars[1] = v1;\n    }\n  }\n\n  //! Make a three variable discrete_bounds\n  discrete_bounds(const discrete_variable& v1,\n                  const discrete_variable& v2,\n                  const discrete_variable& v3) :\n      _num_vars(3) \n  {\n    DCHECK_LE(_num_vars, MAX_DIM);\n    DCHECK_NE(v1, v2);\n    DCHECK_NE(v2, v3);\n    DCHECK_NE(v1, v3);\n      \n    if(v1 < v2 && v2 < v3) {\n      _vars[0] = v1;\n      _vars[1] = v2;\n      _vars[2] = v3;\n    } else if( v1 < v3 && v3 < v2) {\n      _vars[0] = v1;\n      _vars[1] = v3;\n      _vars[2] = v2;\n    } else if( v2 < v1 && v1 < v3) {\n      _vars[0] = v2;\n      _vars[1] = v1;\n      _vars[2] = v3;\n    } else if( v2 < v3 && v3 < v1) {\n      _vars[0] = v2;\n      _vars[1] = v3;\n      _vars[2] = v1;\n    } else if( v3 < v1 && v1 < v2) {\n      _vars[0] = v3;\n      _vars[1] = v1;\n      _vars[2] = v2;\n    } else if( v3 < v2 && v2 < v1) {\n      _vars[0] = v3;\n      _vars[1] = v2;\n      _vars[2] = v1;\n    } else { throw(\"Invalid Case!\"); }\n  }\n\n  //! Make a discrete_bounds from a vector of variables\n  explicit discrete_bounds(const std::vector& vars) :\n      _num_vars(vars.size()) \n  {\n    DCHECK_LE(_num_vars, MAX_DIM);\n    for(size_t i = 0; i < _num_vars; ++i) {\n      _vars[i] = vars[i];\n    }\n    std::sort(_vars, _vars + std::min(MAX_DIM, _num_vars) );\n  }\n\n  //! Make a discrete_bounds from a set of variables\n  explicit discrete_bounds(const std::set& vars) :\n      _num_vars(vars.size()) \n  {\n    DCHECK_LE(_num_vars, MAX_DIM); \n    size_t i = 0; \n    foreach(const discrete_variable& var, vars) _vars[i++] = var;\n  }\n  \n  discrete_bounds(const discrete_bounds& other) :\n      _num_vars(other._num_vars) \n  {\n    *this = other;\n  }\n\n  virtual ~discrete_bounds() { }\n\n  discrete_bounds& operator=(const discrete_bounds& other) {\n    if(this == &other) \n      return *this;\n    \n    _num_vars = other._num_vars;\n    DCHECK_LE(_num_vars, MAX_DIM);\n    for(size_t i = 0; i < _num_vars; ++i) {\n      _vars[i] = other.var(i);\n    }\n    return *this;\n  }\n\n  //! test whether two discrete_boundss are equal\n  bool operator==(const discrete_bounds& other) const {\n    if( num_vars() != other.num_vars() ) return false;  \n    for(size_t i = 0; i < num_vars(); ++i) {\n      if(var(i) != other.var(i)) return false;\n    }\n    return true;\n  }\n    \n  //! test whether two discrete_boundss are not equal\n  bool operator!=(const discrete_bounds& other) const {\n    return !(*this == other);\n  }\n\n  //! add the other discrete_bounds to this discrete_bounds\n  discrete_bounds& operator+=(const discrete_variable& var) {\n    if(_vars[_num_vars - 1] < var) {\n      _vars[_num_vars] = var;\n      _num_vars++;\n      return *this;\n    }\n    return operator+=(discrete_bounds(var));\n  }\n\n  //! add the discrete_bounds to this discrete_bounds\n  discrete_bounds& operator+=(const discrete_bounds& other) {\n    if(other.num_vars() == 0) return *this;\n    discrete_bounds backup = *this;\n    _num_vars = 0;\n    for(size_t i = 0, j = 0; \n        i < backup.num_vars() || j < other.num_vars(); ) {\n      DCHECK_LE(_num_vars, MAX_DIM);\n      // Both \n      if(i < backup.num_vars() && j < other.num_vars() \n         && _num_vars < MAX_DIM) {\n        if(backup.var(i) < other.var(j))  \n          _vars[_num_vars++] = backup.var(i++);\n        else if(other.var(j) < backup.var(i))  \n          _vars[_num_vars++] = other.var(j++);\n        else { _vars[_num_vars++] = backup.var(i++); j++; }\n      } else if(i < backup.num_vars() && _num_vars < MAX_DIM) {\n        _vars[_num_vars++] = backup.var(i++);\n      } else if(j < other.num_vars() && _num_vars < MAX_DIM) {\n        _vars[_num_vars++] = other.var(j++);\n      } else {\n        *this = backup;\n        // Unreachable\n        throw(\"Unreachable case in domain operator+=\");\n      }\n    }\n    return *this;\n  }\n  \n  //! add two discrete_boundss together\n  discrete_bounds operator+(const discrete_variable& var) const {\n    discrete_bounds dom = *this;\n    return dom += var;\n  }\n\n  //! add the other discrete_bounds to this discrete_bounds\n  discrete_bounds operator+(const discrete_bounds& other) const {\n    discrete_bounds dom = *this;\n    return dom += other;\n  }\n\n  \n  //! subtract the other discrete_bounds from this discrete_bounds\n  discrete_bounds& operator-=(const discrete_bounds& other) {\n    if(other.num_vars() == 0) return *this;\n      \n    size_t tmp_num_vars = 0;\n    for(size_t i = 0, j = 0; i < _num_vars; ++i ) {\n      // advance the other index\n      for( ; j < other._num_vars && _vars[i].id() > other._vars[j].id(); ++j) { }\n\n      if(!(j < other._num_vars && _vars[i].id() == other._vars[j].id())) {\n        _vars[tmp_num_vars++] = _vars[i];\n      }\n    }\n    _num_vars = tmp_num_vars;\n    return *this;\n  }\n\n  //! subtract the other discrete_bounds from this discrete_bounds\n  discrete_bounds operator-(const discrete_bounds& other) const {\n    discrete_bounds dom(*this);\n    return dom -= other;\n  }\n\n  discrete_bounds intersect(const discrete_bounds& other) const {\n    discrete_bounds new_dom;\n    new_dom._num_vars = 0;\n    for(size_t i = 0, j = 0;\n        i < num_vars() && j < other.num_vars(); ) {\n      if(_vars[i] == other._vars[j]) {\n        // new discrete_bounds gets the variable\n        new_dom._vars[new_dom.num_vars()] = _vars[i];\n        // Everyone advances\n        new_dom._num_vars++;  i++; j++;\n      } else {\n        // otherwise increment one of the variables          \n        if(_vars[i] < other._vars[j]) i++; else j++;\n      }\n    }\n    return new_dom;\n  }\n\n  //! Get the number of variables\n  inline size_t num_vars() const { return _num_vars; }\n\n  //! Get the ith variable\n  inline const discrete_variable& var(size_t index) const {\n    DCHECK_LT(index, _num_vars);\n    return _vars[index];\n  }\n\n  /** get the index of the variable or returns number of variables\n      if the index is not found */\n  size_t var_location(discrete_variable var) const {\n    return var_location(var.id());\n  }\n\n  size_t var_location(size_t var_id) const {\n    size_t location = _num_vars;\n    for(size_t i = 0; i < _num_vars && !(location < _num_vars); ++i) {\n      if(_vars[i].id() == var_id) location = i;\n    }\n    return location;\n  }\n\n  // get the index within our domain of each variable in other.\n  // other must be a subset \n  std::vector vars_location(const discrete_bounds& other) const {\n    // ensure that the other domain is a subset of our domain\n    DCHECK_EQ((*this + other).num_vars(), num_vars());\n    \n    std::vector locations(other.num_vars());\n    // NOTE this depends on the list of variables in both domains being sorted\n    size_t subdomain_idx = 0;\n    for(size_t i=0; i vars_location(const std::vector& other) const {\n    // ensure that the other domain is a subset of our domain\n    DCHECK_EQ((*this + discrete_bounds(other)).num_vars(), num_vars());\n    \n    std::vector locations(other.size());\n    for(size_t i=0; i 0) {\n      sum = 1;\n      for(size_t i = 0; i < num_vars(); ++i) {\n        // ensure variables to be sorted order\n        if(i > 0) { DCHECK_LT( _vars[ i-1], _vars[i] ); }\n        // and have positive arity\n        DCHECK_GT(_vars[i].size(), 0);\n        sum *= _vars[i].size();\n      }\n    }\n    return sum;\n  }\n\n  void load(graphlab::iarchive& arc) {\n    arc >> _num_vars;\n    DCHECK_LE(_num_vars, MAX_DIM);\n    for(size_t i = 0; i < _num_vars; ++i) arc >> _vars[i];\n  }\n    \n  void save(graphlab::oarchive& arc) const {\n    arc << _num_vars;\n    for(size_t i = 0; i < _num_vars; ++i) arc << _vars[i];\n  }\n\nprivate:\n  //mutable size_t _size_cached;\n  //mutable size_t _last_num_vars;\n  size_t _num_vars;\n  // REVIEW C style array is space inefficient. a vector might be \n  // better. it would have to be resized if the domain was modified; \n  // however, it would remove the need for the template parameter.\n  //   i tried this (37fef6f16b6d). creating the vector on the heap is \n  //   far too slow \n  discrete_variable _vars[MAX_DIM];\n};\n\n\ntemplate\nstd::ostream& operator<<(std::ostream& out,\n                   const discrete_bounds& dom) {\n  out << \"{\";\n  for(size_t i = 0; i < dom.num_vars(); ++i) {\n    out << dom.var(i) << \"[0:\" << dom.var(i).size()-1 << \"]\";\n    if( i < dom.num_vars()-1 ) out << \", \";\n  }\n  return out << \"} \";  \n}\n\n\n}; // end of namespace graphlab\n\n\n\n#include \n\n#endif // DISCRETE_BOUNDS_HPP\n"
  },
  {
    "path": "toolkits/graphical_models/factors/discrete_domain.hpp",
    "content": "/**  \n *  Software submitted by \n *  Systems & Technology Research / Vision Systems Inc., 2013\n *\n *  Approved for public release; distribution is unlimited. [DISTAR Case #21428]\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef DISCRETE_DOMAIN_HPP\n#define DISCRETE_DOMAIN_HPP\n\n#include \n\n#include \"discrete_variable.hpp\"\n#include \"discrete_bounds.hpp\"\n#include \"discrete_assignment.hpp\"\n\n#include \nnamespace graphlab {\n\n\n/**\n * This class respresents a discrete domain over a set of variables.\n *\n * \\author Scott Richardson     4/2013\n */\ntemplate\nclass discrete_domain : public discrete_bounds {\n  typedef discrete_bounds     bounds_type;\n  typedef discrete_assignment assignment_type;\n\npublic:\n  //! Make an empty domain\n  discrete_domain() : \n      bounds_type() { }\n\n  //! Make a single variable discrete_domain\n  discrete_domain(const discrete_variable& v1) :\n      bounds_type(v1) { }\n\n  //! Make a two variable discrete_domain\n  discrete_domain(const discrete_variable& v1, const discrete_variable& v2) :\n      bounds_type(v1, v2) { }\n\n  //! Make a three variable discrete_domain\n  discrete_domain(const discrete_variable& v1,\n                  const discrete_variable& v2,\n                  const discrete_variable& v3) :\n      bounds_type(v1, v2, v3) { }\n\n  //! Make a discrete_domain from a vector of variables\n  explicit discrete_domain(const std::vector& variables) :\n      bounds_type(variables) { }\n\n  //! Make a discrete_domain from a set of variables\n  explicit discrete_domain(const std::set& variables) :\n      bounds_type(variables) { }\n  \n  discrete_domain(const discrete_domain& other) : \n      bounds_type(other) { }\n\n  discrete_domain(const bounds_type& other) : \n      bounds_type(other) { }\n\n  virtual ~discrete_domain() { } \n\n  /** Standard assignment operator */\n  discrete_domain& operator=(const discrete_domain& other) {\n    if(this == &other) \n      return *this;\n    \n    bounds_type::operator=(other);\n    return *this;\n  }\n\n  class ConstIterator;\n\n  // Iterators \n  // from http://www.oreillynet.com/pub/a/network/2005/11/21/what-is-iterator-in-c-plus-plus-part2.html\n  // and http://www.cs.helsinki.fi/u/tpkarkka/alglib/k06/lectures/Iterators.html\n  // although i think i should use this reference: \n  // http://www.drdobbs.com/the-standard-librarian-defining-iterato/184401331?pgno=3\n  class Iterator : \n      public std::iterator\n  {\n  private:\n    friend class ConstIterator;\n    typedef discrete_domain domain_type;\n\n    typedef std::iterator iterator_t;\n    // \"using typename\" doesn't actually work in GCC < 4.7, which we don't have installed\n    // everywhere. The \"typedef typename\" construct seems to, so stick with that for now.\n    //using typename iterator_t::value_type;\n    //using typename iterator_t::reference;\n    //using typename iterator_t::pointer;\n    typedef typename iterator_t::value_type   value_type_custom;\n    typedef typename iterator_t::reference    reference_custom;\n    typedef typename iterator_t::pointer      pointer_custom;\n\n    assignment_type _asg;\n\n  public:\n    explicit Iterator(const domain_type& dom) {\n      // initilize a new assignment\n      _asg = assignment_type(dom);\n    }\n    explicit Iterator(const assignment_type& asg) : _asg(asg) { }\n    Iterator& operator=(const Iterator& other) {\n      if(this == &other) return *this;\n\n      _asg = other._asg;\n      return *this;\n    }\n    // implicit copy constructor, copy assignment and destructor\n    bool operator==(const Iterator& other) const {\n      return _asg == other._asg;\n    }\n    bool operator!=(const Iterator& other) const {\n      return !(*this == other);\n    }\n    reference_custom operator*() {\n      return _asg;\n    }\n    pointer_custom operator->() {\n      // this may be more correct, but less clear\n      //return &*(domain_type::Iterator)*this;\n      return &_asg;\n    }\n    Iterator& operator++() {\n      ++_asg;\n      return *this;\n    }\n    Iterator operator++(int) {\n      Iterator orig = *this; \n      ++(*this); \n      return orig;\n    }\n  };\n\n  class ConstIterator : \n      public std::iterator \n  {\n    typedef discrete_domain domain_type;\n\n    typedef std::iterator const_iterator_t;\n    //using typename const_iterator_t::value_type;\n    //using typename const_iterator_t::reference;\n    //using typename const_iterator_t::pointer;\n    typedef typename const_iterator_t::value_type   value_type_custom;\n    typedef typename const_iterator_t::reference    reference_custom;\n    typedef typename const_iterator_t::pointer      pointer_custom;\n\n    assignment_type _asg;\n\n  public:\n    explicit ConstIterator(const domain_type& dom) {\n      // initilize a new assignment\n      _asg = assignment_type(dom);\n    }\n    explicit ConstIterator(const assignment_type& asg) : _asg(asg) { }\n    ConstIterator(const Iterator& other) : _asg(other._asg) { }\n    // implicit copy constructor and destructor\n    ConstIterator& operator=(const ConstIterator& other) {\n      if(this == &other) return *this;\n\n      _asg = other._asg;\n      return *this;\n    }\n    bool operator==(const ConstIterator& other) const {\n      return _asg == other._asg;\n    }\n    bool operator!=(const ConstIterator& other) const {\n      return !(*this == other);\n    }\n    reference_custom operator*() const {\n      return _asg;\n    }\n    pointer_custom operator->() const {\n      return &_asg;\n    }\n    ConstIterator& operator++() {\n      ++_asg;\n      return *this;\n    }\n    ConstIterator operator++(int) {\n      ConstIterator orig = *this; \n      ++(*this); \n      return orig;\n    }\n  };\n\n  Iterator begin() const { \n    return Iterator(*this); \n  }\n\n  Iterator end() const { \n    Iterator ret(*this);\n    ret->make_end();\n    return ret;\n  }\n\npublic:\n  typedef Iterator       iterator;\n  typedef ConstIterator  const_iterator;\n};\n\n}; // end of namespace graphlab\n\n\n\n#include \n\n#endif // DISCRETE_DOMAIN_HPP\n"
  },
  {
    "path": "toolkits/graphical_models/factors/discrete_variable.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef DISCRETE_VARIABLE_HPP\n#define DISCRETE_VARIABLE_HPP\n/**\n * This file contains the definitions of a simple discrete variable.\n *\n *  \\author Joseph Gonzalez\n */\n\n#include \n#include \n\n#include \n#include \n\n\n// Include the macro for the for each operation\n//#include \nnamespace graphlab {\n\n\n/** represents a discrete variable */\nclass discrete_variable {\npublic:\n  //! The type used to store the variable id\n  typedef uint32_t id_type;\n  //! the type used to index the variable assignments\n  typedef uint32_t index_type;\n\n  discrete_variable() : id_(0), nasgs_(0) { }\n\n  /** construct a discrte variable with a given id and number of\n      assignments */\n  discrete_variable(id_type id, index_type nasgs) : \n    id_(id), \n    nasgs_(nasgs) { }\n  discrete_variable(const discrete_variable& other) :\n    id_(other.id_), \n    nasgs_(other.nasgs_) { }\n  discrete_variable& operator=(const discrete_variable& other) {\n    if(this == &other) \n      return *this;\n\n    id_ = other.id_;\n    nasgs_ = other.nasgs_;\n    return *this;\n  }\n  //! get the variable id\n  inline id_type& id() { return id_; }\n  //! get the variable id\n  inline const id_type& id() const { return id_; }\n  //! get the number of assignments the variable can take\n  inline index_type& size() { return nasgs_; }   \n  //! get the number of assignments the variable can take\n  inline const index_type& size() const { return nasgs_; }\n  //! Compare two variables\n  inline bool operator<(const discrete_variable& other) const { \n    return id_ < other.id_; \n  }\n  //! test equality between two varaibles\n  inline bool operator==(const discrete_variable& other) const {\n    return id_ == other.id_;     \n  }\n  //! Test inequality between two variables\n  inline bool operator!=(const discrete_variable& other) const { \n    return id_ != other.id_; \n  }\n\n  //! load the variable from an archive\n  void load(graphlab::iarchive& arc) { \n    arc >> id_ >> nasgs_;\n  }\n  //! save the variable to an archive\n  void save(graphlab::oarchive& arc) const { \n    arc << id_ << nasgs_; \n  }\n  \nprivate:\n  //! The variable id\n  id_type id_;\n  //! The number of assignments the variable takes\n  index_type nasgs_;\n};\n\n\ninline std::ostream& operator<<(std::ostream& out, \n                   const graphlab::discrete_variable& var) {\n  return out << \"v_\" << var.id();\n  //           << \" in {0:\" << var.size()-1 << \"}\";\n}\n\n}; // end of namespace graphlab\n\n\n\n//#include \n#endif\n\n"
  },
  {
    "path": "toolkits/graphical_models/factors/factor_graph.hpp",
    "content": "/**  \n *  Software submitted by \n *  Systems & Technology Research / Vision Systems Inc., 2013\n *\n *  Approved for public release; distribution is unlimited. [DISTAR Case #21428]\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n\n#ifndef VSI_FACTOR_GRAPH_HPP\n#define VSI_FACTOR_GRAPH_HPP\n\n#include \n\n#include \n#include \n#include \n#include \n#include \n\n//#include \n#include \n#include \n#include \n\n\n#include \"table_factor.hpp\"\n#include \"bp_graph_data.h\"\n\n\n\n//#include \nnamespace belief_prop {\n\n\n/** \n * Defines a factor_graph; i.e., a bipartite graph whose verticies can \n * be divide into two disjoint sets: a set of variables V and a set of \n * factors F. Undirected edges connect factors to variables. An edge \n * exists between f and v if v is a member of the factor's domain.\n * \n * A variable specifies a unary discrete probabiltiy mass function over \n * a set of labels. A variable's PMF is assumed to be defined by a \n * dense_table in which each label has a corresponding probability. \n * \n * A factor specifies a discrete joint probability mass function over \n * a set of variables. The n-D PMF is defined by either a dense_table  \n * or a sparse_table. The values in the table are stored such that the \n * first variable added to the domain iterates the fastest. \n *\n * The typical usasge of this interface would consist of 1) adding a set\n * of variables by using add_variable(), 2) defining the prior distribution\n * over each variable by using one of the prior related methods (such as \n * set_prior_for_variable() ), 3) adding a set of factors by using \n * add_factor(), 4) constructing the distributed graph using make_bp_graph(), \n * 5) running belief propagation on the distributed graph to propagate the \n * evidence across the graph (outside the scope of this interface), and \n * 6) loading the results using pull_beliefs_for_variables(). \n * \n * The resulting belief for a variable can be queried using \n * belief_for_variable().logP() (once the evidence has been propagated \n * across the distributed graph and the results pulled back into the  \n * factor_graph using pull_beliefs_for_variables() ). \n *\n * \\author Scott Richardson     10/2012\n *\n */\n// NOTE a variable has a domain that spans only itself. a factor has\n// a domain that spans its neighboring variables\ntemplate\nclass factor_graph {\n  typedef typename graph_type::type graph_type_t;\n  typedef vertex_data               vertex_data_t;\n  typedef edge_data                 edge_data_t;\n  // dense_table is used to define the distributions over variables.  \n  // *this shouldnt need to know specifically about sparse_table\n  typedef graphlab::dense_table     dense_table_t;\n  typedef graphlab::table_base      table_base_t;\n  typedef graphlab::table_factor    factor_type;\n  typedef graphlab::discrete_variable        variable_type;\n\n// variable related methods\npublic: \n  factor_graph() : _unique_var_id(0) { }\n\npublic: \n  /** Add a new discrete variable to the factor graph */\n  // TODO rename to create_variable()\n  variable_type add_variable(const size_t n_labels, \n      const std::string& default_var_name=\"\") \n  {\n    size_t id = _unique_var_id++;\n    // assert that the id can be used to index into _factors\n    DCHECK_EQ(id, num_factors());\n\n    // Only store the variable in the local variable map if the variable (key) \n    // does not exist in the graph (map) already\n//    var_map_const_iter_type it = _var_map.find(var_name);\n//    if(it != _var_map.end()) {\n//      std::cout << \"WARNING: variable already exists for that name\" << std::endl;\n//      return it->second;\n//    }\n\n    // Create a new variable\n    variable_type variable(id, n_labels);\n\n    std::string var_name(default_var_name);\n    if(var_name.empty()) {\n      std::stringstream ss; ss << id;\n      var_name = ss.str();\n    } \n    // Save the factor to the factor graph\n    vertex_data_t vert = add_vertex(variable, var_name);\n    logstream(LOG_INFO) << \"var_id=\" << id \n                        << \" description='\" << vert.name << \"'\" << std::endl;\n\n    return variable; \n  }\n\n  /** \n   * Direct access to the variable's belief distribution.  \n   * useful for initialization \n   */\n  dense_table_t& belief_for_variable(const variable_type& var) {\n    DCHECK_LT(var.id(), num_factors());\n\n    // NOTE the variable's prior distribution is always dense\n    dense_table_t* belief = \n        dynamic_cast(factors()[var.id()].belief.table());\n    DCHECK_NE(belief, NULL); \n\n    return *belief;\n  }\n  dense_table_t& belief_for_var(const variable_type& var) {\n    return belief_for_variable(var);\n  }\n  /** \n   * Direct access to the variable's belief distribution.  \n   */\n  const dense_table_t& belief_for_variable(const variable_type& var) const {\n    DCHECK_LT(var.id(), num_factors());\n\n    // NOTE the variable's prior distribution is always dense\n    dense_table_t const *const belief = \n        dynamic_cast(\n        factors()[var.id()].belief.table());\n    DCHECK_NE(belief, NULL); \n\n    return *belief;\n  }\n  const dense_table_t& belief_for_var(const variable_type& var) const {\n    return belief_for_variable(var);\n  }\n\n  /** \n   * Direct access to the variable's prior distribution.  \n   * useful for initialization \n   */\n  dense_table_t& prior_for_variable(const variable_type& var) {\n    DCHECK_LT(var.id(), num_factors());\n\n    // NOTE the variable's prior distribution is always dense\n    dense_table_t* potential = \n        dynamic_cast(factors()[var.id()].potential.table());\n    DCHECK_NE(potential, NULL); \n\n    return *potential;\n  }\n  /** \n   * Direct access to the variable's prior distribution.  \n   */\n  const dense_table_t& prior_for_variable(const variable_type& var) const {\n    DCHECK_LT(var.id(), num_factors());\n\n    // NOTE the variable's prior distribution is always dense\n    dense_table_t const *const potential = \n        dynamic_cast(factors()[var.id()].potential.table());\n    DCHECK_NE(potential, NULL); \n\n    return *potential;\n  }\n\n  // TODO rename to stage_prior_for_variable, etc. \n  void set_prior_for_variable(const variable_type& var, \n      const std::vector& data) \n  {\n    DCHECK_LT(var.id(), num_factors());\n    factor_type& potential = factors()[var.id()].potential;\n    // NOTE the variable's prior distribution is always dense\n    dense_table_t* table = dynamic_cast(potential.table());\n    DCHECK_NE(table, NULL);\n\n    *table = dense_table_t(table->domain(), data);\n  }\n\n  void set_belief_for_variable(const variable_type& var, \n      const std::vector& data) \n  {\n    DCHECK_LT(var.id(), num_factors());\n    factor_type& belief = factors()[var.id()].belief;\n    // NOTE the variable's prior distribution is always dense\n    dense_table_t* table = dynamic_cast(belief.table());\n    DCHECK_NE(table, NULL);\n\n    *table = dense_table_t(table->domain(), data);\n  }\n\n  void set_prior_for_variable(const variable_type& var, \n      const dense_table_t& table) \n  {\n    DCHECK_LT(var.id(), num_factors());\n\n    factors()[var.id()].potential = factor_type(factor_type::DENSE_TABLE, table);\n  }\n\n  void set_belief_for_variable(const variable_type& var, \n      const dense_table_t& table) \n  {\n    DCHECK_LT(var.id(), num_factors());\n\n    factors()[var.id()].belief = factor_type(factor_type::DENSE_TABLE, table);\n  }\n\n  variable_type get_variable(const size_t id) {\n    DCHECK_LT(id, num_factors());\n    DCHECK_EQ(factors()[id].potential.table()->ndims(), 1);\n    return factors()[id].potential.table()->var(0);\n  }\n\nprivate:\n  // REVIEW prevent a variable from being double added\n  const vertex_data_t& add_vertex(const variable_type &variable, \n      const std::string& var_name) \n  {\n    // assert that the id can be used to index into _factors\n    DCHECK_EQ(variable.id(), num_factors());\n\n    // Define a unary factor (the concept not the class) over the var to \n    // support a prior. \n    // NOTE the variable's prior distribution is always dense\n    factor_type prior(factor_type::DENSE_TABLE, dense_table_t(variable));\n    // using shift normalization, this is equivalent to uniform()\n    prior.table()->zero();\n\n    // NOTE the variable's prior distribution is always dense\n    factor_type belief(factor_type::DENSE_TABLE, dense_table_t(variable));\n    // using shift normalization, this is equivalent to uniform()\n    belief.table()->zero(); \n    // assert that prior and belief have the same domain\n    //ASSERT_TRUE(belief.domain() == prior.domain());\n\n    // catalog the variable and associated metadata\n    vertex_data_t vertex(prior, belief, true, var_name);\n    _factors.push_back(vertex);\n    //ASSERT_EQ(_factors[variable.id()].belief, vertex.belief);\n    //ASSERT_EQ(_factors[variable.id()].potential, vertex.potential);\n\n    return _factors.back();\n  }\n\n  factor_type uniform_factor_from_factor(const factor_type& factor) {\n    factor_type belief = factor;\n    // using shift normalization, this is equivalent to uniform()\n    belief.table()->zero(); \n\n    return belief;\n  }\n\n// factor related methods\npublic:\n  /** Add a new discrete factor to the factor graph */\n  // REVIEW prevent a factor from being double added\n  void add_factor(const table_base_t& factor, \n      const std::string& default_factor_name = \"\") \n  {\n    size_t id = _unique_var_id++;\n    DCHECK_NE(factor.ndims(), 0);\n    // assert that the id can be used to index into _factors\n    DCHECK_EQ(id, num_factors());\n\n    std::string factor_name; \n    if(default_factor_name.empty()) {\n      std::stringstream ss;\n      ss << id;\n      factor_name = ss.str();\n    } else { \n      factor_name = default_factor_name;\n    }\n\n    // assert all variables have already been added to the graph\n    logstream(LOG_INFO) << \"ndims=\" << factor.ndims() << \" id=\" << id \n        << \" description='\" << factor_name << \"'\" << std::endl;\n    for(size_t i = 0; i < factor.ndims(); ++i) {\n      logstream(LOG_INFO) << \"  factor.var(\" << i << \").id()=\" << factor.var(i).id() << std::endl;\n      DCHECK_LT(factor.var(i).id(), num_factors());\n      DCHECK_EQ(factor.var(i).id(), get_variable(factor.var(i).id()).id());\n    }\n\n    factor_type node(factor);\n    factor_type uniform = uniform_factor_from_factor(node);\n    vertex_data_t vertex(node, uniform, false, factor_name);\n    _factors.push_back(vertex);\n    //ASSERT_EQ(_factors[id], vertex);\n  }\n\n// utils\npublic:\n  size_t num_factors() const {\n    return factors().size();\n  }\n\n  // FIXME O(n)\n  size_t num_variables() const {\n    size_t ndims = 0;\n    for(typename std::vector::const_iterator factor = factors().begin(); \n        factor != factors().end(); ++factor) {\n      // any vertex that has a domain with only a single dimension is a variable ...\n      if(factor->potential.table()->ndims() == 1) {\n        ndims++;\n      }\n    }\n    return ndims;\n  }\n\n  const std::string& name(const size_t id) const {\n    DCHECK_LT(id, num_factors());\n    return factors()[id].name;\n  }\n\n  /**\n   * write a dot file which can be loaded into graphviz\n   */\n  void save_graph_summary(const std::string& filename) {\n    DCHECK_EQ(_unique_var_id, num_factors());\n\n    std::ofstream fout(filename.c_str());\n    if(fout.is_open() == false) {\n      std::cerr << \"ERROR: \" << filename << \" not opened.\" << std::endl;\n      return;\n    }\n\n    fout << \"graph G {\" << std::endl;\n    fout << \"layout=sfdp;\" << std::endl;\n    fout << \"overlap=false;\" << std::endl; \n    //fout << \"sccmap;\" << std::endl;\n    fout << \"K=2;\" << std::endl;\n    //fout << \"clusterrank=local;\" << std::endl;\n\n    // Iterate all the factors and all the edges. NOTE all variables are also factors\n    typename std::vector::const_iterator factor;\n    size_t factor_idx = 0;\n    for(factor = factors().begin(); \n        factor != factors().end(); ++factor_idx, ++factor) \n    {\n      //fout << \"subgraph cluster_\" << factor_idx << \" {\" << std::endl;\n      //fout << \"color=none;\" << std::endl;\n      // Iterate edges for a factor\n      for(size_t i = 0; i < factor->potential.table()->ndims(); ++i) {\n        variable_type variable = factor->potential.table()->var(i);\n        // all variables are also factors, dont link a variable to itself\n        if(variable.id() == factor_idx) continue; \n\n        fout << \"\\\"\" << factor->name << \" {\" << factor_idx << \"}\" \n               << \"\\\" -- \\\"\" << name(variable.id()) << \"{\" << variable.id() << \"}\" \n               << \"\\\";\" << std::endl; \n      }\n      //fout << \"}\" << std::endl;\n    }\n    fout << \"}\" << std::endl;\n  } // end of save_graph_summary\n\n  /**\n   * Construct a belief propagation graph from a factor graph\n   */\n  // NOTE could rewrite this function to construct the graph in parallel using \n  // the graphlab::distributed_control obj\n  // TODO rename to finalize_distributed_graph()\n  // REVIEW because bound, damping and regularization are now an attribute of a \n  // factor, perhaps they should be set in add_factor()\n  void make_bp_graph(graph_type_t& graph, \n      double bound, double damping, double regularization=0.0) \n  {\n    DCHECK_NE(num_factors(), 0);\n    DCHECK_EQ(_unique_var_id, num_factors());\n\n    // TODO clear the graph\n    \n    graphlab::timer timer; \n\n    if (graph.dc().procid() == 0) \n    {\n      // Add all the factors and all the edges. NOTE all variables are also factors\n      typename std::vector::iterator factor = factors().begin();\n      typename std::vector::const_iterator end = factors().end();\n      size_t factor_idx = 0;\n      for( ; factor != end; ++factor_idx, ++factor) {\n        // Add the factor to the graph\n        factor->BOUND = bound; \n        factor->DAMPING = damping;\n        factor->REGULARIZATION = regularization;\n        graph.add_vertex(factor_idx, *factor);\n        // TODO does the order in which i add variables and factors matter? \n\n        // Attach all the edges\n        for(size_t i = 0; i < factor->potential.table()->ndims(); ++i) {\n          variable_type variable = factor->potential.table()->var(i);\n          // all variables are also factors, dont link a variable to itself\n          if(variable.id() == factor_idx) continue; \n\n          // NOTE from graph::add_edge() - An edge can only be added if both the  \n          // source and target vertex id's are already in the graph. Duplicate  \n          // edges are not supported and may result in undefined behavior\n          // NOTE messages are always dense\n          dense_table_t msg(variable);\n          msg.zero(); // using shift normalization, this is equivalent to uniform()\n          graph.add_edge(factor_idx, variable.id(), edge_data_t(msg));\n        }\n      }\n      graph.dc().cout() << \"Loading graph. Finished in \" \n          << timer.current_time() << std::endl;\n    }\n    timer.start();\n    graph.finalize();\n    graph.dc().cout() << \"Finalizing graph. Finished in \" \n        << timer.current_time() << std::endl;\n\n    graph.dc().cout() \n      << \"================ \"\n      << \"Graph statistics on proc \" << graph.dc().procid() << \" of \" << graph.dc().numprocs()\n      << \" ===============\"\n      << \"\\n Num vertices: \" << graph.num_vertices()\n      << \"\\n Num edges: \" << graph.num_edges()\n      << \"\\n Num replica: \" << graph.num_replicas()\n      << \"\\n Replica to vertex ratio: \" \n      << float(graph.num_replicas())/graph.num_vertices()\n      << \"\\n --------------------------------------------\" \n      << \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n      << \"\\n Num local vertices: \" << graph.num_local_vertices()\n      << \"\\n Replica to own ratio: \" \n      << (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n      << \"\\n Num local edges: \" << graph.num_local_edges()\n      //<< \"\\n Begin edge id: \" << graph.global_eid(0)\n      << \"\\n Edge balance ratio: \" \n      << float(graph.num_local_edges())/graph.num_edges()\n      << std::endl;\n  } // end of make_bp_graph\n\n  /** \n   * Update the value of the variables' beliefs given the propegated \n   * graphlab distributed-graph. \n   */ \n  // TODO rename to pull_beliefs\n  void pull_beliefs_for_variables(graph_type_t& graph) {\n    typedef dense_table_t const *const const_ptr;\n\n    // aggregate (reduce) the variable verticies into a vector\n    aggregate_vertex_data agg = \n        graph.template map_reduce_vertices(\n          aggregate_vertex_data()\n        ); // wow\n\n    for(unsigned i=0; i < agg.size(); ++i) {\n      const vertex_data_t& ovdata = agg.agg[i];\n      DCHECK_EQ(ovdata.belief.table()->ndims(), 1);\n\n      const_ptr other = dynamic_cast(ovdata.belief.table());\n      if(other == NULL) {\n        std::cout << \"ERROR: std::bad_cast\" << std::endl;\n        // REVIEW should probably raise an exception\n        ASSERT_TRUE(false);\n      }\n\n      vertex_data_t& vdata = factors()[other->args().var(0).id()];\n      DCHECK_EQ(vdata.belief.table()->ndims(), 1);\n\n      const_ptr tbl = dynamic_cast(vdata.belief.table());\n      if(tbl == NULL) {\n        std::cout << \"ERROR: std::bad_cast\" << std::endl;\n        // REVIEW should probably raise an exception\n        ASSERT_TRUE(false);\n      }\n      ASSERT_EQ(tbl->args().var(0).id(), other->args().var(0).id());\n\n      vdata.belief = ovdata.belief;\n    }\n  }\n\nprivate:\n  struct aggregate_vertex_data {\n    std::vector agg;\n\n    aggregate_vertex_data() { }\n\n    aggregate_vertex_data operator()(const typename graph_type_t::vertex_type& vertex) {\n      const vertex_data_t& ovdata = vertex.data();\n\n      aggregate_vertex_data out;\n      // variables are always dense and 1D, although not all 1D vertices are variables...\n      if(ovdata.isVariable == true) {\n        out.agg.push_back(ovdata);\n      }\n\n      return out;\n    }\n\n    aggregate_vertex_data& operator+=(const aggregate_vertex_data& other) {\n      agg.insert(agg.end(), other.agg.begin(), other.agg.end());\n      return *this;\n    }\n\n    unsigned size() { return agg.size(); }\n\n    void save(graphlab::oarchive& arc) const { arc << agg; }\n    void load(graphlab::iarchive& arc) { arc >> agg; }\n  };\n\npublic:\n  void print_variable(const variable_type& var, const std::vector& labels,\n      std::ostream& out = std::cout) \n  {\n    const dense_table_t& table = belief_for_variable(var);\n    DCHECK_EQ(table.size(), labels.size());\n\n    //std::ios::fmtflags f(out.flags()); // i cant believe this is how you do this\n\n    out << \"var_\" << var.id() << \": \" \n              << factors()[var.id()].name << std::endl;\n    out << std::setw(8) << \"index\" << std::setw(16) << \"logP\" << std::setw(16) << \"label\" << std::endl;\n    size_t end = table.size();\n    for(size_t i=0; i < end; ++i) {\n      out << std::setw(8) << i \n          << std::setw(16) << table.logP(i) \n          << std::setw(16) << labels[i] << std::endl;\n    }\n\n    //out.flags(f);\n  }\n\n// accessors\nprivate:\n  const std::vector& factors() const { return _factors; }\n  std::vector& factors() { return _factors; }\n\nprivate:\n  // NOTE if ever multithreaded, this requires atomic access \n  size_t _unique_var_id;\n  // REVIEW deep-copying data into std::vectors is slow \n  std::vector _factors; \n};\n\n} // end of namespace belief_prop\n\n//#include \n#endif // VSI_FACTOR_GRAPH_HPP\n\n"
  },
  {
    "path": "toolkits/graphical_models/factors/factor_graphs.dox",
    "content": "/**\n\n\\page factor_graphs Factor Graph Toolkit\n\n\\brief \nThe Factor Graph toolkit (defined in toolkits/graphical_models/factors/factor_graph.hpp) \nis an abstraction layer which is able to translate a factor graph into a \ngraphlab distributed-graph. \n\n\\image html factor_graph_to_distributed_graph.png\n\nA factor graph\nis a bipartite graph composed of two types of vertices: variable nodes \nand factor nodes. \n\nA variable specifies a unary discrete \nprobabiltiy mass function \n(PFM) over a set of labels. A variable's PMF is assumed to be defined by a \nclass dense_table in which each label has a corresponding \nprobability. \n\nA factor specifies a discrete joint probability mass function over \na set of neighboring variables which form the factor's domain. The \nn-D PMF is defined by either a class dense_table or \nclass sparse_table (which is basically just an N-d matrix and is \ndefined in toolkits/graphical_models/factors/dense_table.hpp and sparse_table.hpp), \nin which each label has a corresponding log probability (we use the log of the \nbelief value for numerical stability).\nNOTE When constructing an n-D table, take care to understand the sorting order \nof the values in the table and that expected by the various accessors/constructors \nof the class.\n\nInference in a factor graph is commonly performed using \nloopy belief \npropagation (LBP), a well-known message-passing algorithm. The vertex program \nspecified in bp_vertex_program.cpp defines the max-sum variant of \nbelief propagation (although sum-product is also supported). Messages between \nvariables and factors are always dense 1-d tables. \n\nBelief propagation and factor graphs are general tools that have applications in a \nvariety of domains. \n\nWhile GraphLab proper supports several common portable graph formats, \nsuch as tsv, the Factor Graph toolkit instead defines a factor graph \nprogramatically. The typical usasge of the class factor_graph \ninterface would consist of:\n\n-# adding a set of variables to the factor graph by using add_variable(), \n\n-# defining the prior distribution over each variable by using one \nof the prior related methods (such as set_prior_for_variable(...) )\n\n-# adding a set of factors to the graph by using add_factor(...), \n\n-# constructing the distributed graph using make_bp_graph(), \n\n-# running belief propagation on the distributed graph to propagate the \nevidence across the graph (outside the scope of this interface), and \n\n-# loading the results using pull_beliefs_for_variables(...). \n\nThe resulting belief for a variable can be queried using \nbelief_for_variable(var).logP() (i.e., once the evidence has been propagated \nacross the distributed graph and the results pulled back into the\nfactor_graph using pull_beliefs_for_variables() ). \n\n\\section factor_graph_structured_prediction A Factor Graph for Structured Prediction\n\nWe have implemented the \\ref structured_prediction \"Structured Prediction\" example \nusing a factor graph. We can run the structured prediction application on the \nsynthetic image like this:\n\\verbatim\n> ./denoise --damping=.3 \n\\endverbatim\n\nThe structure prediction application applies the Loopy Belief propagation \nalgorithm to a factor graph encoding encoding the classic \nPotts Model. \n\nOnce the application terminates, the final predictions will be stored in denoised.png.\n\n\\image html denoised.jpeg\n\nNot bad! Given the synthetic noisy image (noisy_img.png), denoised.png is very \nsimilar to the true underlying image (orig_img.png) that we would like to recover.\n\n\n\\subsection structured_predictions_options Options\n\n\\li --help Display the help message describing the list of\noptions.\n\n\\li --damping (Optional, Default 0.1) The amount of damping to\nuse. Damping can help ensure that the algorithm converges.  Larger\ndamping values lead to slower but more reliable convergence.\n\n\n\\li --engine (Optional, Default: asynchronous) The engine type to\nuse when executing the vertex-programs\n       - synchronous: All LoopyBP updates are run at the same\n         time (Synchronous BP). This engine exposes greater parallelism but is less\n         computationally efficient.\n       - asynchronous: LoopyBP updates are run asynchronous\n         with priorities (Residual BP). This engine is has greater\n         overhead and exposes less parallelism but can substantially\n         improve the rate over convergence.\n\n\\li --ncpus (Optional, Default 2) The number of local computation \nthreads to use on each machine. This should typically match the number \nof physical cores. \n\n\\li --scheduler (Optional, Default sweep) The scheduler to use when \nrunning with the asynchronous engine. The default is typically sufficient. \n\n\\li --engine_opts (Optional, Default empty) Any additional engine\noptions. See --engine_help for a list of options.\n\n\n\\li --graph_opts (Optional, Default empty) Any additional graph\noptions. See --graph_help for a list of options.\n\n\\li --scheduler_opts (Optional, Default empty) Any additional scheduler\noptions. See --scheduler_help for a list of options.\n\n\\section Example\nThe test file toolkits/graphical_models/factors/tests/test_bool_var/test_cat_bool_var.cpp\ncreates two variables, foo and bool_var_b, each with two states, connected to a 2x2 factor (there is also a unary evidence factor (prior) attached to bool_var_b). Visually, this looks like\n\n\\verbatim\n bool_obs (factor)\n     |\n     |\n bool_var_b (variable : {false, true})\n     |\n     |\n factor\n cat/fp-tp |  false | true |\n ----------|--------|------|\n state1    |  0.1   |  0.9 |  ---  foo (variable : {state1, false pos})\n ----------|--------|------|\n false pos |  0.8   |  0.2 |\n ---------------------------\n joint belief values (cbj)\n\\endverbatim\n\nThis factor graph, creates two variables, foo and bool_var_b, each with two states. The variables are related to each other through the factor node, cbj. (bool_obs is also a unary evidence factor (prior) attached to bool_var_b.)\n\nA factor or variable's probability mass function (PMF) is specified by a dense_table (which is basically just an N-d matrix and is defined in toolkits/graphical_models/factors/dense_table.hpp) in which each label has a corresponding log probability (we use the log of the belief value for numerical stability). A variable is always a 1D dense_table. Factor cbj is a 2D dense_table with a domain that spans the cross product of domains' the two variables.)\n\nThe nlog belief values in the table for this example are arbitrary. You can set them to whatever your discrete probability mass function is using one of dense_table::logP(size_t), dense_table::logP(discrete_assignment), or one of the specialized constructors. (A discrete_assignment is a subindex over a domain (in Matlab, for example, to access element (1,2) in an array, you would do myArray(1,2); analogously, a discrete_assignment specifies this assignment. I can explain these in more detail if you'l like)). \n\nOnce the evidence has been propagated across the distributed graph and the results pulled back into the factor graph using factor_graph.hpp::pull_beliefs_for_variables(), the resulting marginal distribution for the variable var can be queried using factor_graph.hpp::belief_for_variable(var), which returns a 1D dense_table. Then, you can find the most likely value in the distribution using dense_table::max_index(), which returns the linear index of the largest value.\n\n*/\n"
  },
  {
    "path": "toolkits/graphical_models/factors/fast_discrete_assignment.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef FAST_DISCRETE_ASSIGNMENT_HPP\n#define FAST_DISCRETE_ASSIGNMENT_HPP\n\n#include \"discrete_variable.hpp\"\n#include \"discrete_domain.hpp\"\n#include \"discrete_assignment.hpp\"\n\n\n\n\n//#include \nnamespace graphlab {\n\n\n  /**\n   * A limited version of discrete_assignment which supports\n   * a smaller (and different) set of operations\n   */\n  template\n  class fast_discrete_assignment {\n      \n  public:\n\n    fast_discrete_assignment(const discrete_domain& args) :\n        _num_vars(args.num_vars()), _index(0)\n    { \n      transposed = false;\n      for (size_t i = 0;i < _num_vars; ++i) {\n        _vars[i] = args.var(i);\n        _asgs[i] = 0;\n      }\n      size_t multiple = 1;\n      for (size_t i = 0;i < _num_vars; ++i) {\n        _increment_step[i] = multiple;\n        multiple *= _vars[i].size();\n      }\n    }\n\n    //! Construct a fast_discrete_assignment from a discrete_assignment\n    fast_discrete_assignment(const discrete_assignment& asg) : \n      _num_vars(asg.args().num_vars()), _index(asg.linear_index()) { \n      transposed = false;\n      typename discrete_assignment::const_iterator asg_it = asg.begin();\n      for (size_t i = 0; i < _num_vars; ++i) {\n        _vars[i] = asg.args().var(i);\n        _asgs[i] = asg_it[i];\n      }\n        \n      size_t multiple = 1;\n      for (size_t i = 0;i < _num_vars; ++i) {\n        _increment_step[i] = multiple;\n        multiple *= _vars[i].size();\n      }\n    }\n    \n    //! get the number of variables\n    size_t num_vars() const { return _num_vars; }\n\n    //! get the number of variables\n    discrete_variable var(size_t i) const { return _vars[i]; }\n\n\n    size_t linear_index() const { return _index; }\n      \n    //! Get the next fast_discrete_assignment\n    fast_discrete_assignment& operator++() {\n      // Update the discrete_assignments\n      for(size_t i = 0; i < num_vars(); ++i) {\n        if (_asgs[i] < (_vars[i].size() - 1)) {\n          _asgs[i] = (_asgs[i] + 1);\n          _index += _increment_step[i];\n          return *this;\n        }\n        else {\n          _index -= _asgs[i] * _increment_step[i];\n          _asgs[i] = 0;\n        }\n      }\n      // Reached end\n      make_end();\n      return *this;\n    }\n\n    void set_index(size_t index) {\n      _index = index;\n      recompute_asgs();\n    }\n\n    size_t asg(size_t var_id) const {\n      size_t idx = var_location(var_id);\n      assert(idx < num_vars());\n      return _asgs[idx];\n    }\n\n    void set_asg(size_t var_id, size_t value) {\n      size_t idx = var_location(var_id);\n      assert(idx < num_vars());\n      assert(value < var(idx).size());\n      _asgs[idx] = value;\n      recompute_linear_index();\n    }\n\n    //! Tests whether two fast_discrete_assignments are equal\n    bool operator==(const fast_discrete_assignment& other) const {\n      return _index == other._index;\n    }\n    //! Tests whether two fast_discrete_assignments are not equal\n    bool operator!=(const fast_discrete_assignment& other) const {\n      return _index != other._index;\n    }\n\n    //! Make this an ending fast_discrete_assignment\n    void make_end() {\n      _index = -1;\n    }\n\n    /** Makes the sub_domain the first set of variables to be incremented over\n     * Can only be called once\n     */\n    void transpose_to_start(const discrete_domain& sub_domain) {\n      ASSERT_FALSE(transposed);\n      transposed = true;\n        \n      size_t reorder_map[MAX_DIM];\n      size_t cursubdomain_idx = 0;\n      size_t remainder_idx = sub_domain.num_vars();\n      for (size_t i = 0;i < num_vars(); ++i) {\n        if (cursubdomain_idx < sub_domain.num_vars() && \n            _vars[i].id() == sub_domain.var(cursubdomain_idx).id()) {\n          reorder_map[cursubdomain_idx] = i;\n          ++cursubdomain_idx;\n        }\n        else {\n          reorder_map[remainder_idx] = i;\n          ++remainder_idx;\n        }\n      }\n      //move the asg around\n\n        \n      uint16_t newasgs[MAX_DIM];\n      size_t newincrement_step[MAX_DIM]; \n      discrete_variable newvars[MAX_DIM];\n      for (size_t i = 0;i < num_vars() ; ++i) {\n        newincrement_step[i] = _increment_step[reorder_map[i]];\n        newasgs[i] = _asgs[reorder_map[i]];\n        newvars[i] = _vars[reorder_map[i]];\n      }\n      // copyback\n      for (size_t i = 0;i < num_vars(); ++i) {\n        _asgs[i] = newasgs[i];\n        _vars[i] = newvars[i];\n        _increment_step[i] = newincrement_step[i];\n      }\n    } // end of transpose_to_start\n\n\n\n  private:\n    //! Recompute the index from the discrete_assignment\n    void recompute_linear_index() {\n      size_t multiple = 1;\n      // Clear the index\n      _index = 0;\n      for(size_t i = 0; i < num_vars(); ++i) {\n        _index += multiple * _asgs[i];\n        //        assert(_args.var(i).nasgs > 0);\n        multiple *= _vars[i].size();\n      }\n    }\n      \n    //! Recompute the discrete_assignments from the index\n    void recompute_asgs() {\n      size_t quotient = _index;\n      for(size_t i = 0; i < num_vars(); ++i) {\n        _asgs[i] = quotient % _vars[i].size();\n        quotient /= _vars[i].size();\n        // assert(_asgs[i] < _args.var(i).size());\n      }\n    }\n\n    /** get the index of the variable or returns number of variables\n        if the index is not found */\n    size_t var_location(size_t var_id) const {\n      size_t location = num_vars();\n      for(size_t i = 0; i < num_vars() && !(location < num_vars()); ++i) {\n        if(_vars[i].id() == var_id) location = i;\n      }\n      return location;\n    }\n\n\n    size_t _num_vars;\n    uint32_t _index;    \n    discrete_variable _vars[MAX_DIM]; // actual ordering of the assignments\n    size_t _increment_step[MAX_DIM];  //increment ordering according to _vars\n    uint16_t _asgs[MAX_DIM];          // assignments with respect to _vars\n    bool transposed;\n  };\n}; // end of namespace graphlab\n\n\n\n\n\ntemplate\nstd::ostream& operator<<(std::ostream& out,\n                         const graphlab::fast_discrete_assignment& asg) {\n  out << \"{\";\n  for(size_t i = 0; i < asg.num_vars(); ++i) {\n    out << \"v_\" << asg.var(i).id();\n    if(i < asg.num_vars() - 1) out << \", \";\n  }\n  out << \"}=\" << asg.linear_index();\n  return out;\n}\n\n\n//#include \n#endif // FAST_DISCRETE_ASSIGNMENT_HPP\n\n"
  },
  {
    "path": "toolkits/graphical_models/factors/sparse_index.hpp",
    "content": "/**  \n *  Software submitted by \n *  Systems & Technology Research / Vision Systems Inc., 2013\n *\n *  Approved for public release; distribution is unlimited. [DISTAR Case #21428]\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef SPARSE_INDEX_HPP\n#define SPARSE_INDEX_HPP\n\n\n#include \n#include \n\n\nnamespace graphlab {\n\n\n// some version of this could be shared with discrete_assignment.\ntemplate\nclass sparse_index {\npublic:\n  typedef uint16_t*        iterator;\n  typedef const uint16_t*  const_iterator;\n\npublic: \n  sparse_index() : _num_vars(0) { } \n  explicit sparse_index(size_t num_vars) : \n      _num_vars(num_vars) \n  { \n    DCHECK_LE(_num_vars, MAX_DIM);\n    for(size_t i=0; i<_num_vars; ++i)\n      _asg[i] = 0;\n  }\n  sparse_index(size_t const* const begin, size_t const* const end) {\n    _num_vars = end - begin;\n    DCHECK_LE(_num_vars, MAX_DIM);\n    for(size_t i=0; i<_num_vars; ++i)\n      _asg[i] = begin[i];\n  }\n  explicit sparse_index(std::vector asg) : \n      _num_vars(asg.size()) \n  {\n    DCHECK_LE(_num_vars, MAX_DIM);     \n    for(size_t i=0; i<_num_vars; ++i)\n      _asg[i] = asg[i];\n  }\n\npublic: \n  // SEE http://stackoverflow.com/questions/4421706/operator-overloading\n  //! Tests whether two sparse_index are equal \n  bool operator==(const sparse_index& other) const {\n    DCHECK_EQ(num_vars(), other.num_vars());\n    \n    for(size_t i=0; i<_num_vars; ++i) \n      if(_asg[i] != other._asg[i]) return false; \n    return true;\n  }\n  //! Tests whether this sparse_index is < other\n  bool operator<(const sparse_index& other) const {\n    DCHECK_EQ(num_vars(), other.num_vars());\n  \n    for(size_t i=0; i<_num_vars; ++i) {\n      if(_asg[i] > other._asg[i]) return false;\n      else if(_asg[i] < other._asg[i]) return true;\n    }\n    return false;\n  }\n  //! Tests whether two sparse_indexs are not equal\n  bool operator!=(const sparse_index& other) const {\n    return !this->operator==(other);\n  }\n  //! Tests whether this sparse_index is > other\n  bool operator>(const sparse_index& other) const {\n    return other.operator<(*this);\n  }\n  //! Tests whether this sparse_index is <= other\n  bool operator<=(const sparse_index& other) const {\n    return !this->operator>(other);\n  }\n  //! Tests whether this sparse_index is >= other\n  bool operator>=(const sparse_index& other) const {\n    return !this->operator<(other);\n  }\n\n  iterator begin() { return &_asg[0]; }\n  iterator end() { return &_asg[_num_vars]; }\n  const_iterator begin() const { return &_asg[0]; }\n  const_iterator end() const { return &_asg[_num_vars]; }\n\n// TODO i dont like these methods being public, but there is little i \n// can do about it since sparse_index doesnt know about the domain\n  //! Return the assignment at the specified index \n  // NOTE this is not the assignment for a given variable id; \n  // be mindful of variable reordering \n  inline size_t asg_at(const size_t index) const {\n    DCHECK_LT(index, _num_vars);\n    return _asg[index];\n  }\n\n  //! Set the assignment at the specified index \n  // NOTE this is the assignment for a given index, not for a given variable id; \n  // be mindful of variable reordering \n  inline void set_asg_at(const size_t index, const size_t value) {\n    DCHECK_LT(index, _num_vars);\n    _asg[index] = value;\n  }\n  \n  inline size_t num_vars() const { return _num_vars; }\n\n  void load(graphlab::iarchive& arc) {\n    arc >> _num_vars;\n    ASSERT_LE(_num_vars, MAX_DIM);\n    for(size_t i = 0; i < _num_vars; ++i) arc >> _asg[i];\n  }\n  \n  void save(graphlab::oarchive& arc) const {\n    arc << _num_vars;\n    for(size_t i = 0; i < _num_vars; ++i) arc << _asg[i];\n  }\n\n  friend std::ostream& operator<<(std::ostream& out, \n      const sparse_index& sa) {\n    for(size_t i=0; i < sa._num_vars; ++i) {\n      out << sa._asg[i]; \n      if(i < sa._num_vars - 1) out << \", \";\n    }\n    return out;\n  }\n\nprivate:\n  size_t   _num_vars;\n  uint16_t _asg[MAX_DIM];\n};\n\n} // end of namespace graphlab\n\n#endif // SPARSE_INDEX_HPP\n"
  },
  {
    "path": "toolkits/graphical_models/factors/sparse_table.hpp",
    "content": "/**  \n *  Software submitted by \n *  Systems & Technology Research / Vision Systems Inc., 2013\n *\n *  Approved for public release; distribution is unlimited. [DISTAR Case #21428]\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef SPARSE_TABLE_HPP\n#define SPARSE_TABLE_HPP\n\n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n\n#include \"table_base.hpp\"\n#include \"dense_table.hpp\"\n#include \"sparse_index.hpp\"\n\n\nnamespace graphlab {\n\n\n/**\n * An n-D sparse table up to max_dim dimensions. \n * SEE dense_table.hpp for more detail\n *\n * \\author Scott Richardson     10/2012\n */\ntemplate\nclass sparse_table : public table_base {\nprivate:\n  typedef sparse_table const *const    const_ptr;\n  typedef table_base          table_base_t;\n\n  typedef discrete_variable            variable_t;\n  typedef discrete_domain     domain_t;\n  typedef discrete_assignment assignment_t;\n  typedef dense_table         dense_table_t;\n\npublic:\n  typedef sparse_index sparse_index_t;\n  typedef std::vector > compact_data_t;\nprivate:\n  typedef std::map sparse_data_t;\n  typedef std::vector >       compact_view_t;\n  typedef std::vector > compact_const_view_t;\n\npublic:\n  /** Construct an empty table */\n  sparse_table() { } \n\n  /** Construct a table over the given domain */\n  sparse_table(const domain_t& dom) {\n    set_domain(dom);\n  }\n\n  /** Construct a table over the given domain \n   * dom  : the domain over which the table is defined \n   * data : a vector of assignment-value pairs. the assignment must\n   *        be sorted according to dom; that is, such that the \n   *        variable with the smallest id iterates fastest \n   */\n  sparse_table(const domain_t& dom, const sparse_data_t& data) {\n    set_domain(dom);\n    _dataAtAsg = data;\n  }\n  \n  /** Construct a table over the given domain \n   * vars : a vector of variables that compose the domain\n   * data : a vector of values serialized such that the first \n   *        variable in vars iterates the fastest\n   * NOTE this is a convenience constructor. the entries in the \n   * vector are re-sorted such that the variable with the smallest\n   * id iterates fastest\n   */\n  sparse_table(const std::vector& vars, \n      const std::vector >& data) \n  {\n    // Construct the arguments (which will remap the domain)\n    set_domain(domain_t(vars));\n\n    // create a faux domain with the size of the dimensions ordered correctly. this\n    // is essentially a permute operation.\n    domain_t dom;\n    for(size_t i=0; i asgs(asg.begin(), asg.end());\n      assignment_t fast_asg(vars, asgs);\n      set_logP(fast_asg, data[i].second);\n    }\n  }\n\n  /** Construct an empty table over the given variable */\n  sparse_table(const variable_t& args) { \n    // Construct the arguments (which will remap the domain)\n    set_domain(domain_t(args));\n  }\n  \n  /** Construct an empty table over the given domain */\n  sparse_table(const std::vector& args) { \n    // Construct the arguments (which will remap the domain)\n    set_domain(domain_t(args));\n  }\n\n// NOTE currently, implementing the (big) three isnt strictly necessary\n  /** Construct a copy */\n  sparse_table(const sparse_table& other) : \n      _args(other._args), _dataAtAsg(other._dataAtAsg) { }\n\n  /** Destructor */\n  virtual ~sparse_table() { }\n\n  // REVIEW currently, this isnt necessary\n  /** Standard assignment operator */\n  sparse_table& operator=(const sparse_table& other) {\n    if(this == &other) \n      return *this;\n  \n    _args = other._args;\n    //_dataAtAsg.insert(other._dataAtAsg.begin(), other._dataAtAsg.end());\n    _dataAtAsg = other._dataAtAsg;\n    return *this;\n  }\n\npublic: \n  using table_base_t::APPROX_LOG_ZERO;\n\n  // if the data structures between the two tables is equivilent, this is faster\n  sparse_table& copy_onto(const sparse_table& other) {\n    if(this == &other) \n      return *this;\n  \n    // ensure the domains are the same \n    DCHECK_EQ(args(), other.args());\n    // ensure the number of non-zero entries are the same (sanity check)\n    DCHECK_EQ(_dataAtAsg.size(), other._dataAtAsg.size());\n\n    typename sparse_data_t::iterator entry     = _dataAtAsg.begin();\n    typename sparse_data_t::const_iterator end = _dataAtAsg.end();\n    typename sparse_data_t::const_iterator oentry = other._dataAtAsg.begin();\n    for( ; entry != end; ++entry, ++oentry) {\n      // ensure the two assignments are equivilent (std::map should sort them similarly)\n      DCHECK_EQ(entry->first, oentry->first);\n      __set_logP(entry->second, oentry->second);\n    }\n    // slower\n    //_dataAtAsg.insert(other._dataAtAsg.begin(), other._dataAtAsg.end());\n\n    return *this;\n  }\n\n  /** \n   * Reset the domain for the table. A domain is defined by a vector of\n   * variables, and an assignment is defined over that domain. \n   */\n  void set_domain(const domain_t& args) {\n    _args = args;\n    _dataAtAsg.clear();\n  }\n  const domain_t& domain() const {\n    return args();\n  }\n\n  bool operator==(const sparse_table& other) {\n    // are the two domains equal\n    if(args() != other.args()) return false;\n    // are there the same number of non-zero elements in the two tables\n    if(_dataAtAsg.size() != other._dataAtAsg.size()) return false;\n\n    typename sparse_data_t::iterator entry     = _dataAtAsg.begin();\n    typename sparse_data_t::const_iterator end = _dataAtAsg.end();\n    typename sparse_data_t::const_iterator oentry = other._dataAtAsg.begin();\n    for( ; entry != end; ++entry, ++oentry) {\n      // is the assignment the same (std::map should sort them similarly)\n      if(entry->first != oentry->first) return false;\n      // is the value the same\n      if(entry->second != oentry->second) return false;\n    }\n    return true;\n  }\n  bool operator!=(const sparse_table& other) {\n    return !this->operator==(other);\n  }\n\n  /** \n   * Return the variable at the given index within the domain. (var(i) \n   * specifies the dimension associated with the i'th element of an \n   * assignment, i.e., sparse_index::_asg[i])\n   */\n  virtual const variable_t& var(const size_t index) const {\n    return args().var(index);\n  }\n  /** \n   * Return the index for a given variable within the domain (as well as\n   * into sparse_index::_asg[]).\n   */\n  size_t var_location(const variable_t& var) {\n    return args().var_location(var);\n  }\n\n  /** Return the number of dimensions in the domain */\n  virtual size_t ndims() const { return args().num_vars(); }\n  /** Return the number of elements in the domain: prod(size(table)) */\n  virtual size_t numel() const { return args().size(); } \n  /** Return the number of non-zero elements in the table */\n  size_t nnz() const { return _dataAtAsg.size(); }\n\n  /** Zero existing entries in the table */\n  virtual void zero() {\n    typename sparse_data_t::iterator entry     = _dataAtAsg.begin();\n    typename sparse_data_t::const_iterator end = _dataAtAsg.end();\n    for( ; entry != end; ++entry) {\n      entry->second = 0.0;\n    }\n  }\n  \nprivate:\n\n  inline void remove_logP(const sparse_index_t& asg) {\n    // the assignment must be within the domain\n    DASSERT_TRUE(validate_asg(asg)); \n\n    if(_dataAtAsg.count(asg) == 0) return;\n\n    _dataAtAsg.erase(asg);\n  }\n\n  // REVIEW i dont love this method, but it does afford me some bounds checking. \n  // set_logP(sparse_index_t&, double), seems cleaner, but it has to \n  // re-lookup the pointer. very slow. \n  inline void __set_logP(double& tbl_ref, const double& val) {\n    tbl_ref = std::max(val, APPROX_LOG_ZERO());\n  }\n\n  inline void set_logP(const size_t linear_index, const double& val) {\n    set_logP(compute_asg(linear_index), val);\n  }\n  inline void set_logP(const std::vector& asg, const double& val) {\n    set_logP(sparse_index_t(asg), val); \n  }\n  inline void set_logP(const sparse_index_t& asg, const double& val) {\n    // the assignment must be within the domain\n    DASSERT_TRUE(validate_asg(asg)); \n\n    _dataAtAsg[asg] = std::max(val, APPROX_LOG_ZERO());\n  }\n\n  inline double logP(const sparse_index_t& asg) const {\n    DASSERT_TRUE(validate_asg(asg)); \n\n    // O(log(n))\n    typename sparse_data_t::const_iterator val = _dataAtAsg.find(asg);\n    return val == _dataAtAsg.end() ? APPROX_LOG_ZERO() : val->second;\n  }\n\npublic:\n  /** \n   *  Add an entry to the sparse table (indexed by its sparse_index).\n   *  Clip values to be greater than or equal to APPROX_LOG_ZERO.\n   */\n  // NOTE the assignment is not removed from the domain if val is APPROX_LOG_ZERO. \n  // in the future, if these values are removed, it could invalidate any iterator\n  // over the list of sparse assignments.\n  inline void set_logP(const assignment_t& asg, const double& val) {\n    DCHECK_EQ(asg.args(), args());\n    set_logP(as_sparse_index(asg), val);\n  }\n  /** Remove an entry from the sparse table and its corresponding sparse_index) */ \n  inline void remove_logP(const assignment_t& asg) {\n    DCHECK_EQ(asg.args(), args());\n    remove_logP(as_sparse_index(asg));\n  }\n  // NOTE index is serialized according to the linear indexing of the domain\n  // TODO can i make this private? \n  inline double logP(const size_t linear_index) const {\n    return logP(compute_asg(linear_index));\n  }\n  /** Return an entry from the sparse table (indexed by its sparse_index) */ \n  inline double logP(const assignment_t& asg) const {\n    DCHECK_EQ(asg.args(), args());\n    return logP(as_sparse_index(asg));\n  }\n\n  //! this(x) /= other(x);\n  // supports broadcasting of a sub-domain across the full domain \n  sparse_table& operator/=(const dense_table_t& other) {\n    return for_each_assignment(other, divides());\n    return *this;\n  }\n\n  //! this(x) *= other(x);\n  // supports broadcasting of a sub-domain across the full domain \n  sparse_table& operator*=(const dense_table_t& other) {\n    return for_each_assignment(other, multiplies());\n  }\n\n  //! this(x) /= other(x);\n  // supports broadcasting of a sub-domain across the full domain \n  sparse_table& operator/=(const sparse_table& other) {\n    return for_each_assignment(other, divides());\n  }\n    //! this(x) *= other(x);\n  // supports broadcasting of a sub-domain across the full domain \n  sparse_table& operator*=(const sparse_table& other) {\n    return for_each_assignment(other, multiplies());\n  }\n\nprivate:\n  struct divides {\n    inline double operator()(const double& a, const double& b) const {\n      return a - b; \n    }\n  };\n  struct multiplies {\n    inline double operator()(const double& a, const double& b) const {\n      return a + b; \n    }\n  };\n\n  template\n  inline sparse_table& for_each_assignment(const dense_table_t& other, const Func& f) {\n    // other domain must be a subset of this domain\n    DCHECK_EQ((args() + other.args()).num_vars(), args().num_vars());\n\n    assignment_t dense_asg(args()); \n    // only need to operate on the the assignments in the sparse table\n    // (equivalently, the intersection of the sparse and dense assignments)\n    typename sparse_data_t::iterator it        = _dataAtAsg.begin();\n    typename sparse_data_t::const_iterator end = _dataAtAsg.end();\n    for( ; it != end; ++it) {\n      dense_asg.set_index(linear_index(it->first));\n      //double val = it->second + other.logP(dense_asg));\n      //it->second = val;\n      double val = f(it->second, other.logP(dense_asg));\n      __set_logP(it->second, val);\n    }\n    return *this;\n  }\n\n  template\n  sparse_table& for_each_assignment(const sparse_table& other, const Func& f) {\n\n    // if the tables span the same domain\n    if(args() == other.args()) {\n      DCHECK_EQ(numel(), other.numel());\n      \n      // NOTE the assignments NOT in the intersection of the two sparse tables \n      // will be APPROX_LOG_ZERO() and are removed in *this\n      intersect(other);\n      typename sparse_data_t::iterator it        = _dataAtAsg.begin();\n      typename sparse_data_t::const_iterator end = _dataAtAsg.end();\n      typename sparse_data_t::const_iterator other_it = other._dataAtAsg.begin();\n      for( ; it != end; ++it, ++other_it) {\n        //double val = it->second + other_it->second);\n        //it->second = val;\n        double val = f(it->second, other_it->second);\n        __set_logP(it->second, val);\n      }\n    }\n    // else, broadcast the sub-domain across the full domain\n    else { \n      // other domain must be a subset of this domain\n      DCHECK_EQ((args() + other.args()).num_vars(), args().num_vars());\n      \n      compact_view_t compact_view = as_vector_view();\n      compact_const_view_t other_compact_view = other.as_vector_view();\n      // define the one-to-one mapping from other's domain to our's\n      std::vector sorting_inds = args().vars_location(other.args());\n      // reorder the assignments so they can be quickly iterated over\n      permute(sorting_inds, compact_view);\n      other.permute(other_compact_view);\n      \n      // Loop over x\n      // NOTE the assignments are sorted the same. ie. our assignments share the same \n      // ordering over the sub-domain spaned by msg as the assignments in msg.\n      typename compact_const_view_t::const_iterator x_fastasg = other_compact_view.begin();\n      typename compact_const_view_t::const_iterator x_end     = other_compact_view.end();\n      typename compact_view_t::iterator       y_fastasg = compact_view.begin();\n      typename compact_view_t::const_iterator y_end     = compact_view.end();\n      sparse_index_t yasg;\n      for( ; x_fastasg < x_end; ++x_fastasg) {\n        while(y_fastasg != y_end) {\n          yasg = restrict(*(y_fastasg->first), other.args());\n          if(*(x_fastasg->first) > yasg) { \n            ++y_fastasg;\n            continue;\n          } \n          else if(*(x_fastasg->first) < yasg) {\n            ++x_fastasg;\n            break;\n          }\n          // else the sub-assignments are equal\n          else {\n            //double val = *(y_fastasg->second) + *(x_fastasg->second);\n            //*(y_fastasg->second) = val;\n            double val = f(*(y_fastasg->second), *(x_fastasg->second));\n            __set_logP(*(y_fastasg->second), val);\n            ++y_fastasg;\n          }\n        }\n      }\n    }\n    return *this;\n  }\n\npublic:\n  using table_base_t::marginalize;\n\n  // since the message is always a unary distribution, this is basically \n  // >> sum(\n  //      reshape(\n  //        permute(cavity, circshift(1:ndims(cavity), [0, -msg.dim])), \n  //      [], msg.numel),\n  //    [], 1)\n  // or more generally, \n  // >> sum(\n  //      reshape(\n  //        permute(cavity, [setdiff(1:ndims(cavity), msg.dims), msg.dims]),  \n  //      [], msg.numel),\n  //    [], 1)\n  void marginalize(dense_table_t& msg) const {\n    // No need to marginalize\n    if(args() == msg.args()) {\n      // Just copy and return\n      as_dense_table(msg);\n      return;\n    }\n    \n    // the domains cannot be disjoint\n    DCHECK_GT((args() - msg.args()).num_vars(), 0);\n  \n    compact_const_view_t fast_view = as_vector_view();\n    // define the one-to-one mapping from the msg's domain to our's\n    std::vector sorting_inds = args().vars_location(msg.args());\n    // reorder the assignments so they can be quickly iterated over\n    permute(sorting_inds, fast_view); \n\n    assignment_t yasg(args());\n    // Loop over x\n    // NOTE our assignments have been reordered so we can index assignments in \n    // the two domains consecutively. e.g., if the domain of msg, {v1,v2}, is \n    // sorted in ascending order, then our assignments must also be sorted in \n    // assending order over {v1,v2} (although these sub-domains need not be \n    // be sorted the same.)\n    typename compact_const_view_t::const_iterator fastyasg = fast_view.begin();\n    typename compact_const_view_t::const_iterator yend     = fast_view.end();\n    typename domain_t::const_iterator xasg = msg.args().begin();\n    typename domain_t::const_iterator xend = msg.args().end();\n    for( ; xasg != xend; ++xasg) {\n      double sum = 0;\n\n      // loop over y\n      while(fastyasg != yend) {\n        yasg.set_index(linear_index(*(fastyasg->first))); \n        if(*xasg != yasg.restrict(xasg->args())) break;\n        \n        //maxval = std::sum(maxval, _dataAtAsg[*fastyasg]);\n        sum += exp(*(fastyasg->second));\n        ++fastyasg;\n      }\n      DASSERT_FALSE( std::isinf(sum) );\n      DASSERT_FALSE( std::isnan(sum) );\n      DCHECK_GE(sum, 0.0);\n      if(sum == 0) \n        msg.set_logP( *xasg, APPROX_LOG_ZERO() );\n      else \n        msg.set_logP( *xasg, log(sum) );\n    }\n  }\n\n  using table_base_t::MAP;\n\n  // since the message is always a unary distribution, this is basically \n  // >> max(\n  //      reshape(\n  //        permute(cavity, circshift(1:ndims(cavity), [0, -msg.dim])), \n  //      [], msg.numel),\n  //    [], 1)\n  // or more generally, \n  // >> max(\n  //      reshape(\n  //        permute(cavity, [setdiff(1:ndims(cavity), msg.dims), msg.dims]),  \n  //      [], msg.numel),\n  //    [], 1)\n  void MAP(dense_table_t& msg) const {\n    // No need to marginalize\n    if(args() == msg.args()) {\n      // Just copy and return\n      as_dense_table(msg);\n      return;\n    }\n    \n    // the domains cannot be disjoint\n    DCHECK_GT((args() - msg.args()).num_vars(), 0);\n  \n    compact_const_view_t fast_view = as_vector_view();\n    // define the one-to-one mapping from the msg's domain to our's\n    std::vector sorting_inds = args().vars_location(msg.args());\n    // reorder the assignments so they can be quickly iterated over\n    permute(sorting_inds, fast_view); \n\n    assignment_t yasg(args());\n    // Loop over x\n    // NOTE our assignments have been reordered so we can index assignments in \n    // the two domains consecutively. e.g., if the domain of msg, {v1,v2}, is \n    // sorted in ascending order, then our assignments must also be sorted in \n    // assending order over {v1,v2} (although these sub-domains need not be \n    // be sorted the same.)\n    typename compact_const_view_t::const_iterator fastyasg = fast_view.begin();\n    typename compact_const_view_t::const_iterator yend     = fast_view.end();\n    typename domain_t::const_iterator xasg = msg.args().begin();\n    typename domain_t::const_iterator xend = msg.args().end();\n    for( ; xasg != xend; ++xasg) {\n      double maxval = APPROX_LOG_ZERO();\n\n      // loop over y\n      while(fastyasg != yend) {\n        yasg.set_index(linear_index(*(fastyasg->first))); \n        if(*xasg != yasg.restrict(xasg->args())) break;\n        \n        //maxval = std::max(maxval, _dataAtAsg[*fastyasg]);\n        maxval = std::max(maxval, *(fastyasg->second));\n        ++fastyasg;\n      }\n      msg.set_logP( *xasg, maxval );\n    }\n  }\n\n  void intersect(const sparse_table& other) {\n    map_left_intersection(_dataAtAsg, other._dataAtAsg);\n  }\n\nprivate:\n  //! Compute the index from the sparse_index\n  // NOTE index is serialized according to the linear indexing of the domain\n  size_t linear_index(const sparse_index_t& asg) const {\n    size_t multiple = 1;\n    // Clear the index\n    size_t index = 0;\n    for(size_t i = 0; i < args().num_vars(); ++i) {\n      index += multiple * asg.asg_at(i);\n      // assert(args().var(i).nasgs > 0);\n      multiple *= args().var(i).size();\n    }\n    return index;\n  }\n\n  /** Ensure that an asg falls within the domain */\n  bool validate_asg(const sparse_index_t& asg) const {\n    // no index can be larger than the number of labels in that dimension\n    //return asg <= end_asg();\n    for(size_t i=0; i= args().var(i).size()) return false;\n    return true;\n  }\n\n  //! Compute the sparse_index from the index\n  // NOTE index is serialized according to the linear indexing of the domain\n  sparse_index_t compute_asg(const size_t index) const {\n    DCHECK_LT(index, args().size());\n  \n    sparse_index_t asg(args().num_vars());\n    size_t quotient = index;\n    for(size_t i = 0; i < args().num_vars(); ++i) {\n      asg.set_asg_at(i, quotient % args().var(i).size());\n      quotient /= args().var(i).size();\n      // assert(asg.asg_at(i) < args().var(i).size());\n    }\n    return asg;\n  }\n\npublic:\n  //! Compute the largest assignment possible\n  assignment_t end_asg() {\n    sparse_index_t asg;\n    for(size_t i=0; i keyset() const {\n    std::vector keys;\n  \n    typename sparse_data_t::const_iterator it  = _dataAtAsg.begin();\n    typename sparse_data_t::const_iterator end = _dataAtAsg.end();\n    for( ; it != end; ++it) {\n      keys.push_back(it->first);\n    }\n    return keys;\n  }\n\n//: virtual methods\npublic:\n  virtual sparse_table& deep_copy(const table_base_t& base) {\n    if(this == &base) return *this;\n  \n    // ensure we are dealing with a sparse_table\n    const_ptr other = dynamic_cast(&base);\n    if(other == NULL) {\n      std::cout << \"ERROR: std::bad_cast\" << std::endl;\n      // REVIEW should probably raise an exception\n      ASSERT_TRUE(false);\n    }\n  \n    *this = *other;\n    return *this;\n  }\n  virtual sparse_table& copy_onto(const table_base_t& base) {\n    if(this == &base) return *this;\n  \n    // ensure we are dealing with a sparse_table\n    const_ptr other = dynamic_cast(&base);\n    if(other == NULL) {\n      std::cout << \"ERROR: std::bad_cast\" << std::endl;\n      // REVIEW should probably raise an exception\n      ASSERT_TRUE(false);\n    }\n  \n    this->copy_onto(*other);\n    return *this;\n  }\n/*  \n  // NOTE this operation would turn a sparse table into a dense table\n  //! this(x) += other(x);\n  virtual sparse_table& plus_equals(const table_base_t& base) {\n    // ensure we are dealing with a sparse_table\n    const_ptr other = dynamic_cast(&base);\n    if(other == NULL) {\n      std::cout << \"ERROR: std::bad_cast\" << std::endl;\n      // REVIEW should probably raise an exception\n      ASSERT_TRUE(false);\n    }\n  \n    // TODO implement operator\n    *this += *other;\n  \n    return *this;\n  }\n*/\n  //! this(x) *= other(x);\n  virtual sparse_table& times_equals(const table_base_t& base) {\n    // ensure we are dealing with a sparse_table\n    {\n      sparse_table const* const other = \n        dynamic_cast(&base);\n      if( NULL !=  other) {\n        *this *= *other;\n        return *this;\n      }\n    } {\n      dense_table_t const* const other = \n        dynamic_cast(&base);\n      if( NULL != other ) { \n        *this *= *other;\n        return *this;\n      }\n    }\n    std::cout << \"ERROR: std::bad_cast\" << std::endl;\n    // REVIEW should probably raise an exception\n    ASSERT_TRUE(false);\n  }\n  \n  //! this(x) /= other(x);\n  virtual sparse_table& divide_equals(const table_base_t& base) {\n    // ensure we are dealing with a sparse_table\n    {\n      sparse_table const* const other = \n          dynamic_cast(&base);\n      if( NULL !=  other) {\n        *this /= *other;\n        return *this;\n      }\n    } {\n      dense_table_t const* const other = \n          dynamic_cast(&base);\n      if( NULL != other ) { \n        *this /= *other;\n        return *this;\n      }\n   }\n    std::cout << \"ERROR: std::bad_cast\" << std::endl;\n    // REVIEW should probably raise an exception\n    ASSERT_TRUE(false);\n  }\n  \n  //! (out(x) = this(x)) * other(x);\n  virtual void times(const table_base_t& base, \n      table_base_t& out_base) const {\n  \n    // ensure we are dealing with a sparse_table\n    sparse_table *const out = \n        dynamic_cast(&out_base);\n    if(out == NULL) {\n      std::cout << \"ERROR: std::bad_cast\" << std::endl;\n      // REVIEW should probably raise an exception\n      ASSERT_TRUE(false);\n    }\n  \n    *out = *this; // deep copy\n    out->times_equals(base);\n  }\n  \n  //! (out(x) = this(x)) / other(x);\n  virtual void divide(const table_base_t& base, \n      table_base_t& out_base) const {\n  \n    // ensure we are dealing with a sparse_table\n    sparse_table *const out = \n        dynamic_cast(&out_base);\n    if(out == NULL) {\n      std::cout << \"ERROR: std::bad_cast\" << std::endl;\n      // REVIEW should probably raise an exception\n      ASSERT_TRUE(false);\n    }\n  \n    *out = *this; // deep copy\n    out->divide_equals(base);\n  }\n  \n  virtual void marginalize(table_base_t& base) const {\n    // ensure we are dealing with a dense_table\n    dense_table_t* msg = dynamic_cast(&base);\n    if(msg == NULL) {\n      std::cout << \"ERROR: std::bad_cast\" << std::endl;\n      // REVIEW should probably raise an exception\n      ASSERT_TRUE(false);\n    }\n  \n    marginalize(*msg);\n  }\n\n  virtual void MAP(table_base_t& base) const {\n    // ensure we are dealing with a dense_table\n    dense_table_t* msg = dynamic_cast(&base);\n    if(msg == NULL) {\n      std::cout << \"ERROR: std::bad_cast\" << std::endl;\n      // REVIEW should probably raise an exception\n      ASSERT_TRUE(false);\n    }\n  \n    MAP(*msg);\n  }\n\n  virtual std::ostream& print(std::ostream& out = std::cout) const {\n    // ensure we are dealing with a sparse_table\n    const_ptr tbl = dynamic_cast(this);\n    if(tbl == NULL) {\n      std::cout << \"ERROR: std::bad_cast\" << std::endl;\n      // REVIEW should probably raise an exception\n      ASSERT_TRUE(false);\n    }\n  \n    out << *tbl;\n    return out;\n  }\n\n  friend std::ostream& operator<<(std::ostream& out,\n      const sparse_table& tbl) \n  {\n    out << \"Sparse Table: \" << tbl.args() << \"{\" << std::endl;\n    typename sparse_data_t::const_iterator val = tbl._dataAtAsg.begin();\n    typename sparse_data_t::const_iterator end = tbl._dataAtAsg.end();\n    for( ; val != end; ++val) {\n      out << \"\\tLogP({\" << val->first << \"}=\" << tbl.linear_index(val->first) << \")=\" << val->second << std::endl;\n    }\n    out << \"}\";\n    return out;\n  }\n  \n  virtual void load(graphlab::iarchive& arc) {\n    arc >> _args;\n    arc >> _dataAtAsg; // uses graphlab serialization operator\n  }\n  virtual void save(graphlab::oarchive& arc) const {\n    arc << _args;\n    arc << _dataAtAsg; // uses graphlab serialization operator\n  }\n\nprivate:\n  //! Tests whether one sparse_index's i'th index is < another's \n  struct less_than_by_index {\n    less_than_by_index(size_t ind) : _sorting_ind(ind) { } \n  \n    inline bool operator()(\n        const std::pair& a, \n        const std::pair& b) const\n    {\n      return less_than(a.first, b.first);\n    }\n    inline bool operator()(\n        const std::pair& a, \n        const std::pair& b) const\n    {\n      return less_than(*(a.first), *(b.first));\n    }\n    inline bool operator()(\n        const std::pair& a, \n        const std::pair& b) const\n    {\n      return less_than(*(a.first), *(b.first));\n    }\n  private: \n    inline bool less_than(const sparse_index_t& a, \n        const sparse_index_t& b) const \n    {\n      return a.asg_at(_sorting_ind) < b.asg_at(_sorting_ind);\n    }\n    size_t _sorting_ind;\n  };\n  template\n  void permute(T& data) const {\n    std::vector sorting_inds;\n    for(size_t i=0; i\n  void permute( const size_t sorting_ind, T& data ) const {\n    std::vector sorting_inds;\n    sorting_inds.push_back(sorting_ind);\n    permute(sorting_inds, data);\n  }\n  // REVIEW i might be able to copy _dataAtAsg and sort it with a new predicate\n  template\n  void permute( const std::vector& sorting_inds, T& data ) const {\n    std::vector::const_reverse_iterator s    = sorting_inds.rbegin();\n    std::vector::const_reverse_iterator rend = sorting_inds.rend();\n    for( ; s != rend; ++s) {\n      DCHECK_LT(*s, args().num_vars());\n  \n      std::stable_sort(data.begin(), data.end(), less_than_by_index(*s));\n    }\n  }\n\n  //! Restrict the sparse_index to a sparse_index over the subdomain\n  sparse_index_t restrict(const sparse_index_t& asg, \n      const domain_t& sub_domain) const \n  {\n    sparse_index_t other_asg(sub_domain.num_vars());\n    size_t index = 0;\n    // Map the variables \n    for(size_t i = 0; i < args().num_vars() && \n        index < sub_domain.num_vars(); ++i) {\n      if(sub_domain.var(index) == args().var(i)) {\n        other_asg.set_asg_at(index, asg.asg_at(i));\n        index++;\n      }\n    }\n    DCHECK_EQ(index, sub_domain.num_vars());\n    return other_asg;\n  } // end of restrict\n  \n  // O(n)\n  // from http://stackoverflow.com/questions/3772664/intersection-of-two-stl-maps \n  // and http://stackoverflow.com/questions/1773526/in-place-c-set-intersection\n  // REVIEW is there any problem with invalidated iterators? \n  template\n  void map_left_intersection(\n      std::map& left, \n      const std::map& right) const\n  {\n    typename std::map::iterator    il = left.begin();\n    typename std::map::iterator l_end = left.end();\n    typename std::map::const_iterator ir    = right.begin();\n    typename std::map::const_iterator r_end = right.end();\n    while (il != l_end && ir != r_end) {\n      if (il->first < ir->first) {\n        left.erase(il);\n        ++il;\n      }\n      else if (ir->first < il->first) {\n        ++ir;\n      }\n      else {\n        ++il;\n        ++ir;\n      }\n    }\n    left.erase(il, l_end);\n  }\n\n  compact_view_t as_vector_view() {\n    compact_view_t compact_view;\n    compact_view.resize(_dataAtAsg.size());\n\n    typename sparse_data_t::iterator it = _dataAtAsg.begin();\n    for(size_t i = 0; i < _dataAtAsg.size(); ++i, ++it) {\n      compact_view.at(i) = std::make_pair(&(it->first), &(it->second));\n    }\n    //std::copy(_dataAtAsg.begin(), _dataAtAsg.end(), compact_view.begin());\n    return compact_view;\n  }\n  compact_const_view_t as_vector_view() const {\n    compact_const_view_t compact_view;\n    compact_view.resize(_dataAtAsg.size());\n\n    typename sparse_data_t::const_iterator it = _dataAtAsg.begin();\n    for(size_t i = 0; i < _dataAtAsg.size(); ++i, ++it) {\n      compact_view.at(i) = std::make_pair(&(it->first), &(it->second));\n    }\n    //std::copy(_dataAtAsg.begin(), _dataAtAsg.end(), compact_view.begin());\n    return compact_view;\n  }\n\nprivate:\n  inline const domain_t& args() const { return _args; }\n\n  inline assignment_t as_assignment(const sparse_index_t &asg) const {\n    std::vector asgs(asg.begin(), asg.end());\n    return assignment_t(args(), asgs);\n  }\n\n  inline sparse_index_t as_sparse_index(const assignment_t &asg) const {\n    std::vector asgs(asg.begin(), asg.end());\n    return sparse_index_t(asgs);\n  }\n\nprivate:\n  //! The indicies in an assignment are mapped (one-to-one and in-order) to the \n  //  variables in a domain.\n  domain_t _args;\n\n  //! Map between the sparse assignment in the domain and its value. Sorted by\n  //  assignment.\n  sparse_data_t  _dataAtAsg;\n};\n\n\n} // end of namespace graphlab\n\n#endif // SPARSE_TABLE_HPP\n"
  },
  {
    "path": "toolkits/graphical_models/factors/table_base.hpp",
    "content": "/**  \n *  Software submitted by \n *  Systems & Technology Research / Vision Systems Inc., 2013\n *\n *  Approved for public release; distribution is unlimited. [DISTAR Case #21428]\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef TABLE_BASE_HPP\n#define TABLE_BASE_HPP\n\n/**\n * This file defines the root of the table hierarchy for\n * the various table types.\n *\n *  \\author Scott Richardson     10/2012\n */\n\n// INCLUDES ===================================================================>\n\n// Including Standard Libraries\n#include \n#include \n\n#include \n\n#include \"discrete_variable.hpp\"\n\n\nnamespace graphlab {\n\n\ntemplate\nclass table_base {\npublic:\n  typedef table_base const *const const_ptr;\n\n  virtual ~table_base() { }\n\n  static inline double APPROX_LOG_ZERO() { \n    //return -std::numeric_limits::max();\n    return -1e6; \n  }\n\n  virtual table_base& deep_copy(const table_base& base) = 0;\n  virtual table_base& copy_onto(const table_base& base) = 0;\n\n  //virtual table_base&   plus_equals(const table_base& base) = 0;\n  virtual table_base&  times_equals(const table_base& base) = 0;\n  virtual table_base& divide_equals(const table_base& base) = 0;\n  //virtual void    plus(const table_base& base, table_base& out) const = 0;\n  virtual void   times(const table_base& base, table_base& out) const = 0;\n  virtual void  divide(const table_base& base, table_base& out) const = 0;\n\n  virtual const discrete_variable& var(size_t index) const = 0;\n  \n  // REVIEW these are always dense_tables. should they be in here?\n  virtual void     MAP(table_base& msg) const = 0;\n  virtual void     marginalize(table_base& msg) const = 0;\n\n  virtual void    zero() = 0;\n  virtual size_t numel() const = 0; // REVIEW might not be necessary\n  virtual size_t ndims() const = 0;\n  virtual void    load(graphlab::iarchive& arc) = 0;\n  virtual void    save(graphlab::oarchive& arc) const = 0;\n\n  friend std::ostream& operator<<(std::ostream& out,\n                            const table_base& factor) {\n    factor.print(out);\n    return out;\n  }\n\nprivate:\n  virtual std::ostream& print(std::ostream& out = std::cout) const = 0;\n}; // end of table_base\n\n} // end of namespace graphlab \n\n#endif // TABLE_BASE_HPP\n"
  },
  {
    "path": "toolkits/graphical_models/factors/table_factor.hpp",
    "content": "/**  \n *  Software submitted by \n *  Systems & Technology Research / Vision Systems Inc., 2013\n *\n *  Approved for public release; distribution is unlimited. [DISTAR Case #21428]\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#ifndef TABLE_FACTOR_HPP\n#define TABLE_FACTOR_HPP\n\n/**\n * This file defines an opaque interface (or facade) for the various table types.\n *\n *  \\author Scott Richardson     10/2012\n */\n\n// INCLUDES ===================================================================>\n\n// Including Standard Libraries\n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \"dense_table.hpp\"\n#include \"sparse_table.hpp\"\n\n\n\nnamespace graphlab {\n\n\n\n/**\n * An abstraction implemented to manage the creation and deletion \n * of the sparse/dense tables; so that RAII\n */\n // NOTE im not 100% this class is necessary, except to provide \n // convinient access to certian operators like operator*(). if i\n // made table_base concrete, i might be able to use it wherever\n // i currently use this class. (it's not like this class is \n // providing conversions between dense and sparse tables or anything.)\ntemplate\nclass table_factor {\n  typedef dense_table    dense_table_t;\n  typedef sparse_table   sparse_table_t;\n  typedef table_base     table_base_t;\n\npublic:\n  //static const size_t MAX_DIM_ = MAX_DIM; \n \n\n  // REVIEW accessing this requires constructing a table_factor\n  enum table_storage_t { nil, DENSE_TABLE, SPARSE_TABLE };\n\n  table_factor() : _table_storage(table_factor::nil), _table(NULL) { }\n\n  table_factor(table_storage_t storage) : \n      _table_storage(storage), _table(NULL) \n  { \n    alloc();\n  }\n\n  table_factor(const table_factor& other) :\n      _table_storage(other._table_storage), _table(NULL) \n  {\n    if(other._table == NULL) { // integrity check\n      DCHECK_EQ(other._table_storage, table_factor::nil);\n    } else {\n      alloc();\n\n      _table->deep_copy(*(other._table));\n    }\n  }\n\n  table_factor(table_storage_t storage, typename table_base_t::const_ptr base) : \n      _table_storage(storage), _table(NULL) \n  {\n    DCHECK_NE(base, NULL);\n    DCHECK_NE(_table_storage, table_factor::nil);\n\n    alloc();\n    _table->deep_copy(*base);\n  }\n\n  table_factor(table_storage_t storage, const table_base_t& base) : \n      _table_storage(storage), _table(NULL) \n  {\n    DCHECK_NE(_table_storage, table_factor::nil);\n\n    alloc();\n    _table->deep_copy(base);\n  }\n\n  table_factor(typename table_base_t::const_ptr base) :\n      _table_storage(table_factor::nil), _table(NULL) \n  {  \n    DCHECK_NE(base, NULL);\n\n    determine_storage_t(*base);\n    alloc();\n\n    _table->deep_copy(*base);\n  }\n\n  table_factor(const table_base_t& base) : \n      _table_storage(table_factor::nil), _table(NULL) {\n\n    determine_storage_t(base);\n    alloc();\n\n    _table->deep_copy(base);\n  }\n\n  ~table_factor() {\n    delete _table;\n  }\n\n  // TODO provide iterators\n\npublic:\n  table_factor& operator=(const table_factor& other) {\n    if(this == &other) return *this;\n\n    table_base_t* base = NULL;\n    if(other._table == NULL) { // integrity check\n      DCHECK_EQ(other._table_storage, table_factor::nil);\n    } else {\n      try {\n        alloc_table(other._table_storage, &base);\n        base->deep_copy(*(other._table));\n      } catch ( ... ) {\n        delete base;\n        throw;\n      }\n    }\n    _table_storage = other._table_storage;\n    delete _table;\n    _table = base;\n\n    return *this;\n  }\n/*\n  table_factor& operator+=(const table_factor& other) {\n    DCHECK_NE(_table, NULL);\n    DCHECK_NE(other._table, NULL);\n\n    _table->plus_equals(*(other._table));\n    return *this;\n  }\n*/\n  table_factor& operator*=(const table_factor& other) {\n    DCHECK_NE(_table, NULL);\n    DCHECK_NE(other._table, NULL);\n\n    _table->times_equals(*(other._table));\n    return *this;\n  }\n\n  table_factor& operator/=(const table_factor& other) {\n    DCHECK_NE(_table, NULL);\n    DCHECK_NE(other._table, NULL);\n\n    _table->divide_equals(*(other._table));\n    return *this;\n  }\n\n  table_factor operator*(const table_factor& other) const {\n    DCHECK_NE(_table, NULL);\n    DCHECK_NE(other._table, NULL);\n    \n    // deep copy\n    table_factor out(_table_storage);\n    _table->times(*(other._table), *(out._table));\n    \n    return out;\n  }\n\n  table_factor operator/(const table_factor& other) const {\n    DCHECK_NE(_table, NULL);\n    DCHECK_NE(other._table, NULL);\n    \n    // deep copy\n    table_factor out(_table_storage);\n    _table->divide(*(other._table), *(out._table));\n    \n    return out;\n  }\n\n  table_base_t const * table() const { \n    DCHECK_NE(_table, NULL);\n\n    return _table; \n  }\n\n  table_base_t* table() { \n    DCHECK_NE(_table, NULL);\n\n    return _table; \n  }\n\n  void load(graphlab::iarchive& arc) {\n    arc >> _table_storage; \n    alloc();\n    if(_table_storage != table_factor::nil) arc >> *_table;\n  }\n\n  void save(graphlab::oarchive& arc) const {\n    arc << _table_storage;\n    if(_table_storage != table_factor::nil) arc << *_table;\n  }\n\nprivate:\n  // NOTE take ownership of base\n  table_factor(table_storage_t storage, table_base_t *const *const base) : \n      _table_storage(storage) {\n    if(base != NULL) _table = *base;\n    DCHECK_NE(_table, NULL);\n  }\n\n  void determine_storage_t(const table_base_t& base) {\n    if( typeid(base) == typeid(dense_table_t) ) \n      _table_storage = table_factor::DENSE_TABLE;\n    else if( typeid(base) == typeid(sparse_table_t) ) \n      _table_storage = table_factor::SPARSE_TABLE;\n    else {\n      _table_storage = table_factor::nil;\n      // REVIEW should probably raise an exception\n      std::cout << \"ERROR: unknown table storage type. \" << std::endl;\n      ASSERT_TRUE(false);\n    }\n  }\n\n  void alloc() {\n    alloc_table(_table_storage, &(this->_table));\n  }\n\n  // REVIEW could probably use virtual constructors to avoid some of this\n  void alloc_table(table_storage_t storage, table_base_t** base) {\n    // try to avoid memory leaks\n    DCHECK_EQ(*base, NULL);\n\n    switch(storage) {\n      case table_factor::DENSE_TABLE:\n        *base = new dense_table_t();\n        break;\n      case table_factor::SPARSE_TABLE:\n        *base = new sparse_table_t();\n        break;\n      case table_factor::nil:\n      default:\n        // i allow *this to be constructed with a table_factor::nil table storage \n        // type, so this path is possible \n        break;\n    }\n  }\n\npublic:\n  table_storage_t table_storage() const { return _table_storage; } \n\n  friend std::ostream& operator<<(std::ostream& out,\n                            const table_factor& factor) {\n    out << \"Table Factor(\" << factor._table_storage << \"): \" << *(factor._table);\n    return out;\n  }\n\nprivate: \n  table_storage_t _table_storage; \n  table_base_t* _table;\n\n}; // end of table_factor\n\n\n\n} // end of namespace graphlab \n\n#endif // TABLE_FACTOR_HPP\n"
  },
  {
    "path": "toolkits/graphical_models/factors/tests/CMakeLists.txt",
    "content": "project(GraphLab)\n\n# link_libraries(${Boost_LIBRARIES})\n# link_libraries(${GraphLab_LIBRARIES})\n\n\n# set include path. doesn't seem like the best way to set this...\ninclude_directories(\n  ${GraphLab_SOURCE_DIR}/../..)\n\n\nmacro(add_all_subdirectories retval curdir)\n  file(GLOB sub-dir RELATIVE ${curdir} *)\n  set(list_of_dirs \"\")\n  foreach(dir ${sub-dir})\n    if(IS_DIRECTORY ${curdir}/${dir})\n    STRING(SUBSTRING ${dir} 0 1 firstchar)\n        if(${firstchar} STREQUAL \".\" OR ${firstchar} STREQUAL \"_\" )\n        else(${firstchar} STREQUAL \".\" OR ${firstchar} STREQUAL \"_\")\n          set(list_of_dirs ${list_of_dirs} ${dir})\n          message(STATUS \"Detected App: \" ${dir})\n          add_subdirectory(${dir})\n        endif()\n    endif()\n  endforeach()\n  set(${retval} ${list_of_dirs})\nendmacro()\n\nadd_all_subdirectories(retval, ${CMAKE_CURRENT_SOURCE_DIR})\n\n"
  },
  {
    "path": "toolkits/graphical_models/factors/tests/denoise/CMakeLists.txt",
    "content": "project(GraphLab)\n\nadd_graphlab_executable(denoise denoise.cpp)\nrequires_opencv(denoise)\n\nfile(COPY noisy_img.png \n     DESTINATION ${CMAKE_CURRENT_BINARY_DIR}/)\nfile(COPY denoised_gm.png\n     DESTINATION ${CMAKE_CURRENT_BINARY_DIR}/)\n\n"
  },
  {
    "path": "toolkits/graphical_models/factors/tests/denoise/denoise.cpp",
    "content": "/**  \n *  Software submitted by \n *  Systems & Technology Research / Vision Systems Inc., 2013\n *\n *  Approved for public release; distribution is unlimited. [DISTAR Case #21428]\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * This file contains an example of graphlab used for discrete loopy\n * belief propagation in a factor graph to denoise a synthetic noisy image.\n *\n * ./denoise --damping=.3 --ncpus=4 \n *\n *  \\author Scott Richardson \n *          based on toolkits/graphical_models/deprecated/loopybp_denoise.cpp\n */\n\n// INCLUDES ===================================================================>\n\n// Including Standard Libraries\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\n#include \n\n// #include \"image.hpp\"\n#include \n#include \n\n// Include the macro for each operation\n#include \n\n// Global variables\n//static size_t NCOLORS;\n//static double SIGMA;\n\n\nconst size_t MAX_DIM = 4;\ntypedef graphlab::dense_table       dense_table_t;\ntypedef graphlab::discrete_domain   domain_t;\ntypedef graphlab::discrete_variable          variable_t;\n\n\nstruct clopts_vals { \n  clopts_vals(double bound = 1E-4, double damping = 0.3, std::string exec_t=\"sync\") : \n      BOUND(bound), DAMPING(damping), exec_type(exec_t) { }\n\n  double BOUND;\n  double DAMPING;\n  std::string exec_type;\n};\n\n\nint setup_cli(graphlab::command_line_options& clopts, clopts_vals& clvals, \n    int argc, char** argv);\ntemplate\nvoid run_engine(graphlab::distributed_control& dc, \n    typename belief_prop::graph_type::type& graph, \n    const std::string& exec_type, \n    const graphlab::command_line_options& clopts);\n\n// MAIN\n// ============================================================================>\nint main(int argc, char** argv) {\n  std::cout << \"This program solves the sum task.\"\n            << std::endl;\n\n  // Parse command line arguments --------------------------------------------->\n  std::string beliefs_filename = \"beliefs.txt\";\n  \n  graphlab::mpi_tools::init(argc, argv);\n  ///! Create a distributed control object (must come after mpi_tools::init())\n  graphlab::distributed_control dc; \n\n  graphlab::command_line_options clopts(\"Run Loopy BP on a Network\");\n  clopts_vals clvals;\n  if( setup_cli(clopts, clvals, argc, argv) != EXIT_SUCCESS ) return EXIT_FAILURE;\n  \n  ///! Create a distributed graph object \n  belief_prop::graph_type::type graph(dc, clopts);\n\n\n  \n  // Create the factor graph ------------------------------------------>\n  std::cout << \"Loading Factor Graph\" << std::endl;\n  belief_prop::factor_graph fgraph;\n\n  cv::Mat_< uchar > image = cv::imread(\"noisy_img.png\", 0); // force to grayscale with '0'\n  assert( image.data != 0x0 );\n\n  unsigned ni = image.cols;\n  unsigned nj = image.rows;\n  std::cout << \"ni = \" << ni << \" nj = \" << nj << std::endl;\n\n\n  // get the colorspace\n  unsigned n_levels = 5;\n  std::cout << \"pixel values = \";\n  std::vector values(n_levels);\n  for (unsigned i=0; i0?1:0);\n    std::cout << (int)values[i] << \" \";\n  }\n  std::cout << std::endl;\n\n\n  // create variables and prior factor nodes\n  std::vector< std::vector< variable_t > > var_ids;\n  var_ids.resize( nj, std::vector( ni, variable_t() ) );\n  for (unsigned i=0; ilinear_index() < n_levels);\n        double p;\n        if(asg->linear_index() == idx_og)\n          p = same_prob;\n        else\n          p = diff_prob;\n        // Values are stored in log form\n        prior.set_logP( *asg, log(p) );\n      }\n    }\n  }\n\n\n  // create factors and connect the variables\n  float neighbor_same_prob = 0.5f;\n  float neighbor_diff_prob = (1.0f - neighbor_same_prob)/(n_levels - 1);\n  const float cost_scale = 10.0f;\n  for (unsigned i=0; i args;\n        // connect vertical neighbors\n        args.push_back(var_ids[ j-1 ][ i ]);\n        args.push_back(var_ids[ j ][ i ]);\n\n        // Construct the arguments (which will remap the domain)\n        // std::cout << \"domain: \" << domain << std::endl;\n        // Build the factor\n        domain_t domain(args);\n        dense_table_t factor(domain);\n\n        // Set the weights\n        domain_t::const_iterator end = factor.domain().end();\n        for(domain_t::const_iterator asg = factor.domain().begin(); \n            asg != end; ++asg) {\n          assert(asg->linear_index() < n_levels*n_levels);\n          double err;\n          if(asg->asg(var_ids[j-1][i]) == asg->asg(var_ids[j][i]))\n            err = neighbor_same_prob;\n          else\n            err = neighbor_diff_prob;\n\n          // Values are stored in log form\n          factor.set_logP( *asg, log(err*cost_scale) );\n        }\n        // Save the factor to the factor graph\n        fgraph.add_factor(factor);\n      }\n      if (i != 0) {\n        // Create the 2-way factor\n        std::vector args;\n        // connect horizontal neighbors\n        args.push_back(var_ids[ j ][ i-1 ]);\n        args.push_back(var_ids[ j ][ i ]);\n\n        // Construct the arguments (which will remap the domain)\n        // std::cout << \"domain: \" << domain << std::endl;\n        // Build the factor\n        domain_t domain(args);\n        dense_table_t factor(domain);\n\n        // Set the weights\n        domain_t::const_iterator end = factor.domain().end();\n        for(domain_t::const_iterator asg = factor.domain().begin(); \n            asg != end; ++asg) {\n          assert(asg->linear_index() < n_levels*n_levels);\n          double err;\n          if(asg->asg(var_ids[j][i-1]) == asg->asg(var_ids[j][i]))\n            err = neighbor_same_prob;\n          else\n            err = neighbor_diff_prob;\n\n          // Values are stored in log form\n          factor.set_logP( *asg, log(err*cost_scale) );\n        }\n        // Save the factor to the factor graph\n        fgraph.add_factor(factor);\n      }\n    }\n  }\n  assert(fgraph.num_factors() == nj*ni + (nj-1)*ni + nj*(ni-1));\n\n  const size_t num_variables = fgraph.num_variables();\n  const size_t num_factors = fgraph.num_factors();\n  std::cout << \"num_variables: \" << num_variables << \" \"\n            << \"num_factors: \" << num_factors << std::endl;\n  std::cout << \"Finished!\" << std::endl;\n\n\n  // Build the BP graph from the factor graph---------------------------------->\n  std::cout << \"Building BP graph from the factor graph\" << std::endl;\n  fgraph.make_bp_graph( graph, clvals.BOUND, clvals.DAMPING ); \n  run_engine(dc, graph, clvals.exec_type, clopts);\n  fgraph.pull_beliefs_for_variables( graph );\n\n\n  // Saving the output -------------------------------------------------------->\n  // NOTE: this can be done better. see loopybp_denoise.cpp\n  std::cout << \"Saving the predicted image\" << std::endl;\n  std::cout << \"Collect the noisy image. \" << std::endl;\n//  merge_reduce_type pred_image = \n//    graph.map_reduce_vertices(pred_map_function);\n  std::cout << \"saving the pred image.\" << std::endl;\n  if(dc.procid() == 0) {\n    // Fill in output image------------------------------------------------------>\n    cv::Mat_< uchar > output( nj, ni );\n    for (unsigned i=0; i gm = cv::imread(\"denoised_gm.png\", 0); // force to grayscale with '0'\n    cv::Scalar err = cv::sum(cv::abs(gm - output));\n    ASSERT_LT(err(0), ni*nj*1e-3);\n  }\n\n  std::cout << \"All tests passed\" << std::endl;\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n} // end of main\n\n\n// UTILS\n// ============================================================================>\nint setup_cli(graphlab::command_line_options& clopts, clopts_vals& clvals,\n    int argc, char** argv) {\n\n  clopts.attach_option(\"bound\", clvals.BOUND,\n                       \"Residual termination bound\");\n  clopts.attach_option(\"damping\", clvals.DAMPING,\n                       \"The amount of message damping (higher = more damping)\");\n//  clopts.attach_option(\"beliefs\", &beliefs_filename,\n//                       \"The file to save the belief predictions\"); \n  clopts.attach_option(\"engine\", clvals.exec_type,\n                       \"The type of engine to use {async, sync}.\");\n  clopts.set_scheduler_type(\"fifo\");\n\n\n  bool success = clopts.parse(argc, argv);\n  if(!success) {    \n    std::cout << \"Error parsing command line arguments!\"\n              << std::endl;\n    graphlab::mpi_tools::finalize();\n    return EXIT_FAILURE;\n  }\n  return EXIT_SUCCESS;\n}\n\ntemplate\nvoid run_engine(graphlab::distributed_control& dc, \n    typename belief_prop::graph_type::type& graph, \n    const std::string& exec_type, \n    const graphlab::command_line_options& clopts) \n{\n  size_t num_vertices = graph.num_vertices();\n  size_t num_edges = graph.num_edges();\n  std::cout << \"Loaded: \" << num_vertices << \" vertices \"\n            << \"and \" << num_edges << \" edges.\" << std::endl;\n  std::cout << \"Finished!\" << std::endl;\n\n  // Create the engine -------------------------------------------------------->\n  std::cout << \"Creating the engine. \" << std::endl;\n  typedef graphlab::omni_engine > engine_type;\n  engine_type engine(dc, graph, exec_type, clopts);\n\n  std::cout << \"Scheduling all vertices\" << std::endl;\n  engine.signal_all();\n  std::cout << \"Starting the engine\" << std::endl;\n  engine.start();\n  const float runtime = engine.elapsed_seconds();\n  size_t update_count = engine.num_updates();\n  std::cout << \"Finished Running engine in \" << runtime \n            << \" seconds with \" << clopts.get_ncpus() << \" cpus.\" << std::endl\n            << \"Total updates: \" << update_count << std::endl\n            << \"Efficiency: \" << (double(update_count) / runtime)\n            << \" updates per second \"\n            << std::endl;\n}\n\n\n"
  },
  {
    "path": "toolkits/graphical_models/factors/tests/test_MAD_relation/CMakeLists.txt",
    "content": "project(GraphLab)\n\nadd_graphlab_executable(test_MAD_relation test_MAD_relation.cpp)\n"
  },
  {
    "path": "toolkits/graphical_models/factors/tests/test_MAD_relation/test_MAD_relation.cpp",
    "content": "/**  \n *  Software submitted by \n *  Systems & Technology Research / Vision Systems Inc., 2013\n *\n *  Approved for public release; distribution is unlimited. [DISTAR Case #21428]\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * This file contains an example of graphlab belief propagation on\n * a factor node designed to constrain a MAD (multiply and add)\n * relation (a + b/c) = d\n *\n *  \\author Scott Richardson \n */\n\n// INCLUDES ===================================================================>\n\n// Including Standard Libraries\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\n#include \n#include \n\n\n// Include the macro for each operation\n#include \n\n\nconst size_t MAX_DIM = 4;\ntypedef graphlab::dense_table     dense_table_t;\ntypedef graphlab::discrete_domain domain_t;\ntypedef graphlab::discrete_variable        variable_t;\n\n\nstruct clopts_vals { \n  clopts_vals(double bound = 1E-4, double damping = 0.0, std::string exec_t=\"sync\") : \n      BOUND(bound), DAMPING(damping), exec_type(exec_t) { }\n\n  double BOUND;\n  double DAMPING;\n  std::string exec_type;\n};\n\nint setup_cli(graphlab::command_line_options& clopts, clopts_vals& clvals, \n    int argc, char** argv);\ntemplate\nvoid run_engine(graphlab::distributed_control& dc, \n    typename belief_prop::graph_type::type& graph, \n    const std::string& exec_type, \n    const graphlab::command_line_options& clopts);\nstd::vector compute_labels(size_t n_labels, \n    double max_range, double min_range);\nvoid compute_normal_dist(double mean, double std_dev, \n    const std::vector& labels, dense_table_t& prior);\n\n// MAIN\n// ============================================================================>\nint main(int argc, char** argv) {\n  std::cout << \"This program solves the sum task.\"\n            << std::endl;\n\n  global_logger().set_log_level(LOG_DEBUG);\n\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc; // must come after mpi_tools::init()\n\n  // Parse command line arguments --------------------------------------------->\n  graphlab::command_line_options clopts(\"Run Loopy BP on a Network\");\n  clopts_vals clvals;\n  if( setup_cli(clopts, clvals, argc, argv) != EXIT_SUCCESS ) return EXIT_FAILURE;\n  ///! Create a distributed control object \n  belief_prop::graph_type::type graph(dc, clopts);\n\n\n  // Create the factor graph ------------------------------------------>\n  std::cout << \"Loading Factor Graph\" << std::endl;\n  belief_prop::factor_graph fgraph;\n\n  // Create the variables\n  size_t n_labels = 20; \n  double cost_scale = 100;\n\n  double min_range = 0.0; double max_range = 3.0; \n  std::vector labels = compute_labels(n_labels, max_range, min_range);\n\n  variable_t var_a = fgraph.add_variable(n_labels, \"var_a\");\n  {\n  double mean = 2.5; double std_dev = 0.1; \n  dense_table_t& prior = fgraph.prior_for_variable(var_a);\n  compute_normal_dist(mean, std_dev, labels, prior);\n  std::cout << \"var_a_prior=\" << prior << std::endl;\n  }\n\n  variable_t var_b = fgraph.add_variable(n_labels, \"var_b\");\n  {\n  double mean = 1.1; double std_dev = 0.1; \n  dense_table_t& prior = fgraph.prior_for_variable(var_b);\n  compute_normal_dist(mean, std_dev, labels, prior);\n  std::cout << \"var_b_prior=\" << prior << std::endl;\n  }\n\n  // scale\n  float c_scale = -0.5;\n\n  std::vector d_labels = compute_labels(2*n_labels, max_range, min_range);\n  // REVIEW this should probably have n_labels*2\n  variable_t var_d = fgraph.add_variable(2*n_labels, \"var_d\");\n  {\n  dense_table_t& prior = fgraph.prior_for_variable(var_d);\n  prior.zero();\n  std::cout << \"var_d_prior=\" << prior << std::endl;\n  }\n\n\n  // Create a factor\n  std::vector args;\n  // connect vertical neighbors\n  args.push_back(var_a);\n  args.push_back(var_b);\n  args.push_back(var_d);\n  // Build the factor\n  dense_table_t mad(args);\n  // Set the weights\n  domain_t::const_iterator end = mad.domain().end();\n  for(domain_t::const_iterator asg = mad.domain().begin(); asg != end; ++asg) { \n    double a = labels[asg->asg(var_a)];\n    double b = labels[asg->asg(var_b)];\n    double c = c_scale;\n    double d = d_labels[asg->asg(var_d)];\n    double d_prime = (a + c*b);\n    double err = d_prime - d;\n    //std::cout << \"a = \" << a << \" b = \" << b << \" d = \" << d << \" err = \" << err << std::endl;\n    mad.set_logP( *asg, -1*(cost_scale*err*err) );\n  }\n  // Save the factor to the factor graph\n  fgraph.add_factor(mad, \"mad\");\n\n\n  const size_t num_variables = fgraph.num_variables();\n  const size_t num_factors = fgraph.num_factors();\n  std::cout << \"num_variables: \" << num_variables << \" num_factors: \" << num_factors << std::endl;\n  std::cout << \"Finished!\" << std::endl;\n\n\n  // Build the BP graph from the factor graph---------------------------------->\n  std::cout << \"Building BP graph from the factor graph\" << std::endl;\n  fgraph.make_bp_graph( graph, clvals.BOUND, clvals.DAMPING ); \n  run_engine(dc, graph, clvals.exec_type, clopts);\n  fgraph.pull_beliefs_for_variables( graph );\n\n\n  // Saving the output -------------------------------------------------------->\n  //fgraph.print_variable(var_a, labels);\n  //fgraph.print_variable(var_b, labels);\n  //fgraph.print_variable(var_d, d_labels);\n\n  double a = labels[fgraph.belief_for_variable(var_a).max_index()];\n  double b = labels[fgraph.belief_for_variable(var_b).max_index()];\n  double d = d_labels[fgraph.belief_for_variable(var_d).max_index()];\n  std::cout << \"var_a: \" << a << std::endl;\n  std::cout << \"var_b: \" << b << std::endl;\n  std::cout << \"var_d: \" << d << std::endl;\n\n  double d_prime = a + (c_scale * b);\n  double err = std::abs(d_prime - d);\n  std::cout << \"d: \" << d << \" d_prime: \" << d_prime << \" err: \" << err << std::endl;\n  ASSERT_LT(err, (max_range - min_range)/n_labels);\n  std::cout << \"All tests passed\" << std::endl;\n} // end of main\n\n\n// UTILS\n// ============================================================================>\nint setup_cli(graphlab::command_line_options& clopts, clopts_vals& clvals,\n    int argc, char** argv) {\n\n  clopts.attach_option(\"bound\", clvals.BOUND,\n                       \"Residual termination bound\");\n  clopts.attach_option(\"damping\", clvals.DAMPING,\n                       \"The amount of message damping (higher = more damping)\");\n//  clopts.attach_option(\"beliefs\", &beliefs_filename,\n//                       \"The file to save the belief predictions\"); \n  clopts.attach_option(\"engine\", clvals.exec_type,\n                       \"The type of engine to use {async, sync}.\");\n  clopts.set_scheduler_type(\"fifo\");\n\n\n  bool success = clopts.parse(argc, argv);\n  if(!success) {    \n    std::cout << \"Error parsing command line arguments!\"\n              << std::endl;\n    graphlab::mpi_tools::finalize();\n    return EXIT_FAILURE;\n  }\n  return EXIT_SUCCESS;\n}\n\ntemplate\nvoid run_engine(graphlab::distributed_control& dc, \n    typename belief_prop::graph_type::type& graph, \n    const std::string& exec_type, \n    const graphlab::command_line_options& clopts) \n{\n  size_t num_vertices = graph.num_vertices();\n  size_t num_edges = graph.num_edges();\n  std::cout << \"Loaded: \" << num_vertices << \" vertices \"\n            << \"and \" << num_edges << \" edges.\" << std::endl;\n  std::cout << \"Finished!\" << std::endl;\n\n  // Create the engine -------------------------------------------------------->\n  std::cout << \"Creating the engine. \" << std::endl;\n  typedef graphlab::omni_engine > engine_type;\n  engine_type engine(dc, graph, exec_type, clopts);\n\n  std::cout << \"Scheduling all vertices\" << std::endl;\n  engine.signal_all();\n  std::cout << \"Starting the engine\" << std::endl;\n  engine.start();\n  const float runtime = engine.elapsed_seconds();\n  size_t update_count = engine.num_updates();\n  std::cout << \"Finished Running engine in \" << runtime \n            << \" seconds.\" << std::endl\n            << \"Total updates: \" << update_count << std::endl\n            << \"Efficiency: \" << (double(update_count) / runtime)\n            << \" updates per second \"\n            << std::endl;\n}\n\nstd::vector compute_labels(size_t n_labels, \n    double max_range, double min_range) \n{\n  std::vector labels(n_labels, 0.0);\n\n  double step = (max_range - min_range)/(n_labels-1);\n  for(unsigned i = 0; i < n_labels; ++i) {\n    labels[i] = min_range + i*step;\n  }\n  return labels;\n}\n\nvoid compute_normal_dist(double mean, double std_dev, \n    const std::vector& labels, dense_table_t& prior) \n{\n  domain_t::const_iterator asg = prior.domain().begin();\n  domain_t::const_iterator end = prior.domain().end();\n  std::vector::const_iterator label = labels.begin();\n  for( ; asg != end; ++asg, ++label) {\n    double nv = (*label-mean)/std_dev;\n    prior.set_logP( *asg, -1*nv*nv );\n  }\n}\n\n"
  },
  {
    "path": "toolkits/graphical_models/factors/tests/test_bool_var/CMakeLists.txt",
    "content": "project(GraphLab)\n\nadd_graphlab_executable(test_bool_var test_bool_var.cpp)\nadd_graphlab_executable(test_cat_bool_joint test_cat_bool_joint.cpp)\n"
  },
  {
    "path": "toolkits/graphical_models/factors/tests/test_bool_var/test_bool_var.cpp",
    "content": "/**  \n *  Software submitted by \n *  Systems & Technology Research / Vision Systems Inc., 2013\n *\n *  Approved for public release; distribution is unlimited. [DISTAR Case #21428]\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * This file contains an example of graphlab belief propagation on\n * a factor graph designed to test a unary factor node.\n *\n *  \\author Scott Richardson \n */\n\n// INCLUDES ===================================================================>\n\n// Including Standard Libraries\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\n#include \n#include \n\n\n// Include the macro for each operation\n#include \n\n\nconst size_t MAX_DIM = 4;\ntypedef graphlab::dense_table   dense_table_t;\ntypedef graphlab::discrete_variable      variable_t;\n\n\nstruct clopts_vals { \n  clopts_vals(double bound = 1E-4, double damping = 0.0, std::string exec_t=\"sync\") : \n      BOUND(bound), DAMPING(damping), exec_type(exec_t) { }\n\n  double BOUND;\n  double DAMPING;\n  std::string exec_type;\n};\n\nint setup_cli(graphlab::command_line_options& clopts, clopts_vals& clvals, \n    int argc, char** argv);\ntemplate\nvoid run_engine(graphlab::distributed_control& dc, \n    typename belief_prop::graph_type::type& graph, \n    const std::string& exec_type, const graphlab::command_line_options& clopts);\n\n// MAIN\n// ============================================================================>\nint main(int argc, char** argv) {\n  std::cout << \"This program solves the sum task.\"\n            << std::endl;\n\n  graphlab::mpi_tools::init(argc, argv);\n  ///! Create a distributed control object (must come after mpi_tools::init())\n  graphlab::distributed_control dc; \n\n  // Parse command line arguments --------------------------------------------->\n  graphlab::command_line_options clopts(\"Run Loopy BP on a Network\");\n  clopts_vals clvals;\n  if( setup_cli(clopts, clvals, argc, argv) != EXIT_SUCCESS ) return EXIT_FAILURE;\n\n  ///! Create a distributed graph object \n  belief_prop::graph_type::type graph(dc, clopts);\n\n\n\n  // Create the factor graph ------------------------------------------>\n  std::cout << \"Loading Factor Graph\" << std::endl;\n  belief_prop::factor_graph fgraph;\n\n  // Create the variable\n  size_t nlabels = 2;\n  variable_t bool_var_b = fgraph.add_variable(nlabels, \"bool_var_b\");\n  dense_table_t& prior = fgraph.prior_for_variable(bool_var_b);\n  prior.zero();\n\n\n  // Set the weights\n  std::vector logf(2); logf[0] = std::log(0.6); logf[1] = std::log(0.4);\n  // Build a unary factor\n  dense_table_t bool_obs( bool_var_b, logf );\n  // Save the factor to the factor graph\n  fgraph.add_factor(bool_obs, \"bool_obs\");\n\n\n  const size_t num_variables = fgraph.num_variables();\n  const size_t num_factors = fgraph.num_factors();\n  std::cout << \"num_variables: \" << num_variables << \" num_factors: \" << num_factors << std::endl;\n  std::cout << \"Finished!\" << std::endl;\n\n\n  // Build the BP graph from the factor graph---------------------------------->\n  std::cout << \"Building BP graph from the factor graph\" << std::endl;\n  fgraph.make_bp_graph( graph, clvals.BOUND, clvals.DAMPING ); \n  run_engine(dc, graph, clvals.exec_type, clopts);\n  fgraph.pull_beliefs_for_variables( graph );\n\n\n  // Saving the output -------------------------------------------------------->\n  std::cout << fgraph.belief_for_variable(bool_var_b) << std::endl;\n  double bobs = fgraph.belief_for_variable(bool_var_b).logP(1);\n  double err = abs(bobs - .405465);\n  ASSERT_LT(err, .01);\n  std::cout << \"All tests passed\" << std::endl;\n} // end of main\n\n\n// UTILS\n// ============================================================================>\nint setup_cli(graphlab::command_line_options& clopts, clopts_vals& clvals,\n    int argc, char** argv) {\n\n  clopts.attach_option(\"bound\", clvals.BOUND,\n                       \"Residual termination bound\");\n  clopts.attach_option(\"damping\", clvals.DAMPING,\n                       \"The amount of message damping (higher = more damping)\");\n//  clopts.attach_option(\"beliefs\", &beliefs_filename,\n//                       \"The file to save the belief predictions\"); \n  clopts.attach_option(\"engine\", clvals.exec_type,\n                       \"The type of engine to use {async, sync}.\");\n  clopts.set_scheduler_type(\"fifo\");\n\n  bool success = clopts.parse(argc, argv);\n  if(!success) {    \n    std::cout << \"Error parsing command line arguments!\"\n              << std::endl;\n    graphlab::mpi_tools::finalize();\n    return EXIT_FAILURE;\n  }\n  return EXIT_SUCCESS;\n}\n\ntemplate\nvoid run_engine(graphlab::distributed_control& dc, \n    typename belief_prop::graph_type::type& graph, \n    const std::string& exec_type, \n    const graphlab::command_line_options& clopts) \n{\n  size_t num_vertices = graph.num_vertices();\n  size_t num_edges = graph.num_edges();\n  std::cout << \"Loaded: \" << num_vertices << \" vertices \"\n            << \"and \" << num_edges << \" edges.\" << std::endl;\n  std::cout << \"Finished!\" << std::endl;\n\n  // Create the engine -------------------------------------------------------->\n  std::cout << \"Creating the engine. \" << std::endl;\n  typedef graphlab::omni_engine > engine_type;\n  engine_type engine(dc, graph, exec_type, clopts);\n\n  std::cout << \"Scheduling all vertices\" << std::endl;\n  engine.signal_all();\n  std::cout << \"Starting the engine\" << std::endl;\n  engine.start();\n  const float runtime = engine.elapsed_seconds();\n  size_t update_count = engine.num_updates();\n  std::cout << \"Finished Running engine in \" << runtime \n            << \" seconds.\" << std::endl\n            << \"Total updates: \" << update_count << std::endl\n            << \"Efficiency: \" << (double(update_count) / runtime)\n            << \" updates per second \"\n            << std::endl;\n}\n\n"
  },
  {
    "path": "toolkits/graphical_models/factors/tests/test_bool_var/test_cat_bool_joint.cpp",
    "content": "/**  \n *  Software submitted by \n *  Systems & Technology Research / Vision Systems Inc., 2013\n *\n *  Approved for public release; distribution is unlimited. [DISTAR Case #21428]\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * This file contains an example of graphlab belief propagation on\n * a factor graph designed to ignore false positives.\n *\n *  \\author Scott Richardson \n */\n\n// INCLUDES ===================================================================>\n\n// Including Standard Libraries\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n// annoyingly, this must be done before graphlab/logger/logger.hpp is called\n#define OUTPUTLEVEL LOG_EMPH\n#include \n\n#include \n#include \n\n\n// Include the macro for each operation\n#include \n\n\nconst size_t MAX_DIM = 4;\ntypedef graphlab::dense_table   dense_table_t;\ntypedef graphlab::discrete_variable      variable_t;\n\n\nstruct clopts_vals { \n  clopts_vals(double bound = 1E-4, double damping = 0.0, \n      std::string exec_type=\"sync\", int verbose = LOG_EMPH) : \n      BOUND(bound), DAMPING(damping), EXEC_TYPE(exec_type), VERBOSE(verbose) { }\n\n  double BOUND;\n  double DAMPING;\n  std::string EXEC_TYPE;\n  int VERBOSE;\n};\n\nint setup_cli(graphlab::command_line_options& clopts, clopts_vals& clvals, \n    int argc, char** argv);\ntemplate\nvoid run_engine(graphlab::distributed_control& dc, \n    typename belief_prop::graph_type::type& graph, \n    const std::string& exec_type, \n    const graphlab::command_line_options& clopts);\n\n// MAIN\n// ============================================================================>\nint main(int argc, char** argv) {\n  std::cout << \"This program solves the sum task.\"\n            << std::endl;\n\n  graphlab::mpi_tools::init(argc, argv);\n  ///! Create a distributed control object (must come after mpi_tools::init())\n  graphlab::distributed_control dc; \n\n  // Parse command line arguments --------------------------------------------->\n  graphlab::command_line_options clopts(\"Run Loopy BP on a Network\");\n  clopts_vals clvals;\n  if( setup_cli(clopts, clvals, argc, argv) != EXIT_SUCCESS ) return EXIT_FAILURE;\n  \n  ///! Create a distributed graph object \n  belief_prop::graph_type::type graph(dc, clopts);\n\n\n\n  // Create the factor graph ------------------------------------------>\n  std::cout << \"Loading Factor Graph\" << std::endl;\n  belief_prop::factor_graph fgraph;\n\n  // Create the variables\n  size_t nlabels = 2;\n  variable_t foo = fgraph.add_variable(nlabels, \"foo\");\n  std::vector logf(nlabels, 0.0); logf[0] = -1.0;\n  fgraph.set_prior_for_variable(foo, logf);\n\n  variable_t bool_var_b = fgraph.add_variable(nlabels, \"bool_var_b\");\n  dense_table_t& bool_var_b_prior = fgraph.prior_for_variable(bool_var_b);\n  bool_var_b_prior.zero();\n  //std::vector logb(nlabels, std::log(0.5));\n  //fgraph.set_prior_for_variable(bool_var_b, logb);\n\n  // add joint nlog belief values\n  //  cat/fp-tp|  false | true |\n  // ----------|--------|------|\n  // foo       |  0.1   |  0.9 |\n  // ----------|--------|------|\n  // false pos |  0.8   |  0.2 |\n  // ---------------------------\n  //\n\n\n  // Create a factor\n  std::vector args;\n  // connect vertical neighbors\n  args.push_back(foo);\n  args.push_back(bool_var_b);\n  // Set the weights\n  std::vector logc(nlabels*nlabels);\n  logc[0] = std::log(0.1); logc[2] = std::log(0.9);\n  logc[1] = std::log(0.8); logc[3] = std::log(0.2);\n  // Build the factor\n  dense_table_t cbj(args, logc);\n  // Save the factor to the factor graph\n  fgraph.add_factor(cbj, \"cbj\");\n\n  // Build the unary factor\n  logf[0] = std::log(0.1); logf[1] = std::log(0.9);\n  dense_table_t bool_obs(bool_var_b, logf);\n  // Save the factor to the factor graph\n  fgraph.add_factor(bool_obs, \"bool_obs\");\n\n\n  const size_t num_variables = fgraph.num_variables();\n  const size_t num_factors = fgraph.num_factors();\n  std::cout << \"num_variables: \" << num_variables << \" num_factors: \" << num_factors << std::endl;\n  std::cout << \"Finished!\" << std::endl;\n\n\n  // Build the BP graph from the factor graph---------------------------------->\n  std::cout << \"Building BP graph from the factor graph\" << std::endl;\n  fgraph.make_bp_graph( graph, clvals.BOUND, clvals.DAMPING ); \n  run_engine(dc, graph, clvals.EXEC_TYPE, clopts);\n  fgraph.pull_beliefs_for_variables( graph );\n\n\n  // Saving the output -------------------------------------------------------->\n  double bobs_t = fgraph.belief_for_variable(bool_var_b).logP(1);\n  double bobs_f = fgraph.belief_for_variable(bool_var_b).logP(0);\n  double p_true = std::exp(bobs_t) / (std::exp(bobs_t) + std::exp(bobs_f));\n  std::cout << \"p_true = \" << p_true << std::endl;\n  double err = abs(p_true - .788);\n  ASSERT_LT(err, .01);\n\n//  std::cout << fgraph.belief(graph, bool_var_b.id()). << std::endl;\n//  std::cout << fgraph.belief(graph, foo.id()). << std::endl;\n  std::cout << \"All tests passed\" << std::endl;\n} // end of main\n\n\n// UTILS\n// ============================================================================>\nint setup_cli(graphlab::command_line_options& clopts, clopts_vals& opts,\n    int argc, char** argv) {\n\n  clopts.attach_option(\"bound\", opts.BOUND,\n                       \"Residual termination bound\");\n  clopts.attach_option(\"damping\", opts.DAMPING,\n                       \"The amount of message damping (higher = more damping)\");\n  clopts.attach_option(\"verbose\", opts.VERBOSE,\n                       \"Verbosity of Printing: 0 (lots), 2 (default), 6 (no printing).\");\n//  clopts.attach_option(\"beliefs\", &beliefs_filename,\n//                       \"The file to save the belief predictions\"); \n  clopts.attach_option(\"engine\", opts.EXEC_TYPE,\n                       \"The type of engine to use {async, sync}.\");\n  clopts.set_scheduler_type(\"fifo\");\n\n  bool success = clopts.parse(argc, argv);\n  if(!success) {    \n    std::cout << \"Error parsing command line arguments!\"\n              << std::endl;\n    graphlab::mpi_tools::finalize();\n    return EXIT_FAILURE;\n  }\n\n  std::cout << \"logging level: \" << std::max(opts.VERBOSE, OUTPUTLEVEL) << std::endl;\n  global_logger().set_log_level(opts.VERBOSE);\n\n  return EXIT_SUCCESS;\n}\n\ntemplate\nvoid run_engine(graphlab::distributed_control& dc, \n    typename belief_prop::graph_type::type& graph, \n    const std::string& exec_type, \n    const graphlab::command_line_options& clopts) \n{\n  size_t num_vertices = graph.num_vertices();\n  size_t num_edges = graph.num_edges();\n  std::cout << \"Loaded: \" << num_vertices << \" vertices \"\n            << \"and \" << num_edges << \" edges.\" << std::endl;\n  std::cout << \"Finished!\" << std::endl;\n\n  // Create the engine -------------------------------------------------------->\n  std::cout << \"Creating the engine. \" << std::endl;\n  typedef graphlab::omni_engine > engine_type;\n  engine_type engine(dc, graph, exec_type, clopts);\n\n  std::cout << \"Scheduling all vertices\" << std::endl;\n  engine.signal_all();\n  std::cout << \"Starting the engine\" << std::endl;\n  engine.start();\n  const float runtime = engine.elapsed_seconds();\n  size_t update_count = engine.num_updates();\n  std::cout << \"Finished Running engine in \" << runtime \n            << \" seconds.\" << std::endl\n            << \"Total updates: \" << update_count << std::endl\n            << \"Efficiency: \" << (double(update_count) / runtime)\n            << \" updates per second \"\n            << std::endl;\n}\n\n"
  },
  {
    "path": "toolkits/graphical_models/factors/tests/test_dense_table/CMakeLists.txt",
    "content": "project(GraphLab)\n\nadd_graphlab_executable(test_dense_table test_dense_table.cpp)\n"
  },
  {
    "path": "toolkits/graphical_models/factors/tests/test_dense_table/test_dense_table.cpp",
    "content": "/**  \n *  Software submitted by \n *  Systems & Technology Research / Vision Systems Inc., 2013\n *\n *  Approved for public release; distribution is unlimited. [DISTAR Case #21428]\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * Test various functions of a dense table\n *\n *  \\author Scott Richardson \n */\n\n#include \n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\n\nconst size_t MAX_DIM = 4;\ntypedef graphlab::dense_table          dense_table_t;\ntypedef graphlab::discrete_domain      domain_t;\ntypedef graphlab::discrete_assignment  assignment_t;\ntypedef graphlab::discrete_variable             variable_t;\n\n\ndense_table_t create_rand_dense_table(unsigned v0_id, unsigned v1_id, unsigned v2_id) \n{\n  variable_t v0(v0_id, 4);\n  variable_t v1(v1_id, 3);\n  variable_t v2(v2_id, 2);\n\n  std::vector vars;\n  vars.push_back(v0);\n  vars.push_back(v1);\n  vars.push_back(v2);\n  domain_t domain(vars); \n\n  dense_table_t dt(domain);\n  assignment_t da(domain);\n\n  for(int i=0; i < domain.var(0).size(); ++i) {\n    da.set_asg(v0, i);\n    for(int j=0; j < domain.var(1).size(); ++j) {\n      da.set_asg(v1, j);\n      for(int k=0; k < domain.var(2).size(); ++k) {\n        da.set_asg(v2, k);\n\n        if(rand() % 100 <= 20) {\n          dt.set_logP(da, -1 * (rand() % 100));\n        }\n      }\n    }\n  }\n\n  return dt;\n}\n\nstd::vector create_rand_data_vector(size_t d0, size_t d1, size_t d2) {\n  std::vector v(d0*d1*d2);\n  for(size_t i=0; i < v.size(); ++i) {\n    v[i] = -1 * (rand() % 100);\n  }\n  return v;\n}\n\n// [-Inf -Inf -3   -Inf]\n// [-15  -4   -Inf -23 ]\n// [-Inf -Inf -Inf -Inf]\n//\n// [-Inf -Inf -Inf -Inf]\n// [-20  -12  -19  -78 ]\n// [-Inf -Inf -Inf -32 ]\nvoid create_data_vector(std::vector &data) \n{\n  data.resize(4*3*2);\n  data[0] = -1000; data[1] = -1000; data[2]  = -3;    data[3]  = -1000;\n  data[4] = -15;   data[5] = -4;    data[6]  = -1000; data[7]  = -23;\n  data[8] = -1000; data[9] = -1000; data[10] = -1000; data[11] = -1000;\n\n  data[12] = -1000; data[13] = -1000; data[14] = -1000; data[15] = -1000;\n  data[16] = -20;   data[17] = -12;   data[18] = -19;   data[19] = -78;\n  data[20] = -1000; data[21] = -1000; data[22] = -1000; data[23] = -32;\n}\n\nvoid testDataReorder(unsigned v0_id, unsigned v1_id, unsigned v2_id) {\n  // setup the data as if its from belief prop\n  std::vector data = create_rand_data_vector(4, 4, 4);\n\n  // create a table with variables that are not resorted\n  variable_t v0(0, 4);\n  variable_t v1(1, 4);\n  variable_t v2(2, 4);\n  std::vector vars;\n  vars.push_back(v0);\n  vars.push_back(v1);\n  vars.push_back(v2);\n  dense_table_t dt_gm(vars, data);\n  //std::cout << \"dt_gm \" << dt_gm << std::endl;\n\n  //std::cout << \"v0_id \" << v0_id << \" v1_id \" << v1_id << \" v2_id \" << v2_id << std::endl;\n  variable_t r0(v0_id, 4);\n  variable_t r1(v1_id, 4);\n  variable_t r2(v2_id, 4);\n  std::vector reordered_vars;\n  reordered_vars.push_back(r0);\n  reordered_vars.push_back(r1);\n  reordered_vars.push_back(r2);\n  dense_table_t dt(reordered_vars, data);\n  //std::cout << \"dt \" << dt << std::endl;\n\n\n  for(int i=0; i < dt.numel(); ++i) {\n    assignment_t da_gm(domain_t(vars), i);\n    // permute assignment\n    std::vector asgs(da_gm.begin(), da_gm.end());\n    assignment_t da(reordered_vars, asgs);\n    \n    //std::cout << \"dt_gm{\" << da_gm << \"}=\" << dt_gm.logP(da_gm) << \"; \"\n    //          << \"dt{\" << da << \"}=\" << dt.logP(da) << std::endl;\n    ASSERT_TRUE(dt_gm.logP(da_gm) == dt.logP(da));\n  }\n}\n\ndense_table_t create_dense_table(unsigned v0_id, unsigned v1_id, unsigned v2_id) {\n  // create a table with dimensions ordered like the original table (e.g., from \n  // belief prop) to make computing the linear index easier\n  std::vector data;\n  create_data_vector(data);\n\n  variable_t v0(v0_id, 4);\n  variable_t v1(v1_id, 3);\n  variable_t v2(v2_id, 2);\n\n  std::vector vars;\n  vars.push_back(v0);\n  vars.push_back(v1);\n  vars.push_back(v2);\n\n  dense_table_t dt(vars, data);\n\n  return dt;\n}\n\n\nvoid multiplyTest(unsigned v0_id, unsigned v1_id, unsigned v2_id) \n{\n  dense_table_t dt = create_dense_table(v0_id, v1_id, v2_id);\n  dense_table_t dt_gm = dt;\n  //std::cout << \"dt \" << dt << std::endl;\n\n  dt *= dt;\n  //std::cout << \"dt *= dt \" << dt << std::endl;\n  {\n  domain_t::const_iterator asg = dt.domain().begin();\n  domain_t::const_iterator end = dt.domain().end();\n  for( ; asg != end; ++asg)\n    ASSERT_EQ(dt.logP(*asg), dt_gm.logP(*asg)+dt_gm.logP(*asg));\n  }\n\n  dt = dt_gm;\n  unsigned msg_length = dt.var(dt.domain().var_location(v0_id)).size();\n  variable_t v0(v0_id, msg_length);\n  dense_table_t msg(v0);\n  domain_t::const_iterator asg = msg.domain().begin();\n  domain_t::const_iterator end = msg.domain().end();\n  for( ; asg != end; ++asg) {\n    msg.set_logP( *asg, -1*(rand() % 100) );\n  }\n\n  dt *= msg;\n  //std::cout << \"msg = \" << msg << std::endl;\n  //std::cout << \"dt *= msg \" << dt << std::endl;\n\n  for(size_t i=0; i < dt.size(); ++i) {\n    assignment_t dt_asg(dt.domain(), i);\n    assignment_t msg_asg = dt_asg.restrict(msg.domain());\n    ASSERT_EQ(dt.logP(dt_asg), dt_gm.logP(dt_asg)+msg.logP(msg_asg));\n  }\n}\n\nint main() {\n  // create a table \n  dense_table_t dt_gm = create_dense_table(2, 0, 1);\n  //std::cout << \"dt_gm \" << dt_gm << std::endl;  \n\n  double err;\n\n  // equals test\n  dense_table_t dt = create_dense_table(2, 0, 1);\n  //std::cout << \"dt \" << dt << std::endl;\n  err = dt.l1_diff(dt_gm);\n  //std::cout << \"err: \" << err << std::endl;\n  ASSERT_LT(err, 1e-4);\n\n  // copy test\n  dense_table_t dt_copy;\n  dt_copy = dt;\n  err = dt.l1_diff(dt_copy);\n  //std::cout << \"err: \" << err << std::endl;\n  ASSERT_LT(err, 1e-4);\n\n  // test dense table data reorder\n  testDataReorder(2, 3, 4);\n  testDataReorder(2, 4, 3);\n  testDataReorder(3, 2, 4);\n  testDataReorder(3, 4, 2);\n  testDataReorder(4, 2, 3);\n  testDataReorder(4, 3, 2);\n\n  // multiply test - compare a (pre-ordered) dense table to the dense table \n  multiplyTest(2, 3, 4);\n  multiplyTest(2, 4, 3);\n  multiplyTest(3, 2, 4);\n  multiplyTest(3, 4, 2);\n  multiplyTest(4, 2, 3);\n  multiplyTest(4, 3, 2);\n\n  std::cout << \"All tests passed\" << std::endl;\n}\n"
  },
  {
    "path": "toolkits/graphical_models/factors/tests/test_sparse_table/CMakeLists.txt",
    "content": "project(GraphLab)\n\nadd_graphlab_executable(test_sparse_table test_sparse_table.cpp)\nadd_graphlab_executable(test_neg_relation test_neg_relation.cpp)\n"
  },
  {
    "path": "toolkits/graphical_models/factors/tests/test_sparse_table/test_neg_relation.cpp",
    "content": "/**  \n *  Software submitted by \n *  Systems & Technology Research / Vision Systems Inc., 2013\n *\n *  Approved for public release; distribution is unlimited. [DISTAR Case #21428]\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * This file contains an example of graphlab belief propagation on\n * a factor node to constrain a negation relation a = -b\n *\n *  \\author Scott Richardson \n */\n\n// INCLUDES ===================================================================>\n\n// Including Standard Libraries\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\n#include \n#include \n\n\n// Include the macro for each operation\n#include \n\n\nconst size_t MAX_DIM = 4;\ntypedef graphlab::dense_table          dense_table_t;\ntypedef graphlab::sparse_table         sparse_table_t;\ntypedef graphlab::discrete_assignment  assignment_t;\ntypedef graphlab::discrete_domain      domain_t;\ntypedef graphlab::discrete_variable             variable_t;\n\n\nstruct clopts_vals { \n  clopts_vals(double bound = 1E-4, double damping = 0.0, std::string exec_t=\"sync\") : \n      BOUND(bound), DAMPING(damping), exec_type(exec_t) { }\n\n  double BOUND;\n  double DAMPING;\n  std::string exec_type;\n};\n\nint setup_cli(graphlab::command_line_options& clopts, clopts_vals& clvals, \n    int argc, char** argv);\ntemplate\nvoid run_engine(graphlab::distributed_control& dc, \n    typename belief_prop::graph_type::type& graph, \n    const std::string& exec_type, const graphlab::command_line_options& clopts);\nstd::vector compute_labels(size_t n_labels, \n    double max_range, double min_range);\nvoid compute_normal_dist(double mean, double std_dev, \n    const std::vector& labels, dense_table_t& prior);\n\n\n// MAIN\n// ============================================================================>\nint main(int argc, char** argv) {\n  std::cout << \"This program solves the sum task.\"\n            << std::endl;\n\n  global_logger().set_log_level(LOG_DEBUG);\n\n  graphlab::mpi_tools::init(argc, argv);\n  ///! Create a distributed control object (must come after mpi_tools::init())\n  graphlab::distributed_control dc; \n\n  // Parse command line arguments --------------------------------------------->\n  graphlab::command_line_options clopts(\"Run Loopy BP on a Network\");\n  clopts_vals clvals;\n  if( setup_cli(clopts, clvals, argc, argv) != EXIT_SUCCESS ) return EXIT_FAILURE;\n\n  ///! Create a distributed graph object \n  belief_prop::graph_type::type graph(dc, clopts);\n\n\n  // Create the factor graph ------------------------------------------>\n  std::cout << \"Loading Factor Graph\" << std::endl;\n  belief_prop::factor_graph fgraph;\n\n  // Create the variables\n  size_t n_labels = 10; \n  double cost_scale = 100;\n\n  double min_range = -3.0; double max_range = 3.0; \n  std::vector labels = compute_labels(n_labels, max_range, min_range);\n\n  variable_t var_a = fgraph.add_variable(n_labels, \"var_a\");\n  {\n  double mean = 2.3; double std_dev = 0.1; \n  dense_table_t& prior = fgraph.prior_for_variable(var_a);\n  compute_normal_dist(mean, std_dev, labels, prior);\n  std::cout << \"var_a_prior=\" << prior << std::endl;\n  }\n\n  variable_t var_b = fgraph.add_variable(n_labels, \"var_b\");\n  {\n  dense_table_t& prior = fgraph.prior_for_variable(var_b);\n  prior.zero();\n  std::cout << \"var_b_prior=\" << prior << std::endl;\n  }\n\n\n  // Create a factor\n  std::vector args;\n  // connect vertical neighbors\n  args.push_back(var_a);\n  args.push_back(var_b);\n  // Build the factor\n  domain_t dom(args);\n  sparse_table_t neg(dom);\n  // Set the weights\n  assignment_t asg(dom);\n  for(size_t i=0; i\n  std::cout << \"Building BP graph from the factor graph\" << std::endl;\n  fgraph.make_bp_graph( graph, clvals.BOUND, clvals.DAMPING ); \n  run_engine(dc, graph, clvals.exec_type, clopts);\n  fgraph.pull_beliefs_for_variables( graph );\n\n\n  // Saving the output -------------------------------------------------------->\n  fgraph.print_variable(var_a, labels);\n  fgraph.print_variable(var_b, labels);\n\n  double a = labels[fgraph.belief_for_variable(var_a).max_index()];\n  double b = labels[fgraph.belief_for_variable(var_b).max_index()];\n  std::cout << \"var_a: \" << a << std::endl;\n  std::cout << \"var_b: \" << b << std::endl;\n\n  double b_prime = -1*a;\n  double err = std::abs(b - b_prime);\n  std::cout << \"b: \" << b << \" b_prime: \" << b_prime << \" err: \" << err << std::endl;\n  ASSERT_LT(err, 1E-4);\n  std::cout << \"All tests passed\" << std::endl;\n} // end of main\n\n\n// UTILS\n// ============================================================================>\nint setup_cli(graphlab::command_line_options& clopts, clopts_vals& clvals,\n    int argc, char** argv) {\n\n  clopts.attach_option(\"bound\", clvals.BOUND,\n                       \"Residual termination bound\");\n  clopts.attach_option(\"damping\", clvals.DAMPING,\n                       \"The amount of message damping (higher = more damping)\");\n//  clopts.attach_option(\"beliefs\", &beliefs_filename,\n//                       \"The file to save the belief predictions\"); \n  clopts.attach_option(\"engine\", clvals.exec_type,\n                       \"The type of engine to use {async, sync}.\");\n  clopts.set_scheduler_type(\"fifo\");\n\n\n  bool success = clopts.parse(argc, argv);\n  if(!success) {    \n    std::cout << \"Error parsing command line arguments!\"\n              << std::endl;\n    graphlab::mpi_tools::finalize();\n    return EXIT_FAILURE;\n  }\n  return EXIT_SUCCESS;\n}\n\ntemplate\nvoid run_engine(graphlab::distributed_control& dc, \n    typename belief_prop::graph_type::type& graph, \n    const std::string& exec_type, \n    const graphlab::command_line_options& clopts) \n{\n  size_t num_vertices = graph.num_vertices();\n  size_t num_edges = graph.num_edges();\n  std::cout << \"Loaded: \" << num_vertices << \" vertices \"\n            << \"and \" << num_edges << \" edges.\" << std::endl;\n  std::cout << \"Finished!\" << std::endl;\n\n  // Create the engine -------------------------------------------------------->\n  std::cout << \"Creating the engine. \" << std::endl;\n  typedef graphlab::omni_engine > engine_type;\n  engine_type engine(dc, graph, exec_type, clopts);\n\n  std::cout << \"Scheduling all vertices\" << std::endl;\n  engine.signal_all();\n  std::cout << \"Starting the engine\" << std::endl;\n  engine.start();\n  const float runtime = engine.elapsed_seconds();\n  size_t update_count = engine.num_updates();\n  std::cout << \"Finished Running engine in \" << runtime \n            << \" seconds.\" << std::endl\n            << \"Total updates: \" << update_count << std::endl\n            << \"Efficiency: \" << (double(update_count) / runtime)\n            << \" updates per second \"\n            << std::endl;\n}\n\nstd::vector compute_labels(size_t n_labels, \n    double max_range, double min_range) \n{\n  std::vector labels(n_labels, 0.0);\n\n  double step = (max_range - min_range)/(n_labels-1);\n  for(unsigned i = 0; i < n_labels; ++i) {\n    labels[i] = min_range + i*step;\n  }\n  return labels;\n}\n\nvoid compute_normal_dist(double mean, double std_dev, \n    const std::vector& labels, dense_table_t& prior) \n{\n  domain_t::const_iterator asg = prior.domain().begin();\n  domain_t::const_iterator end = prior.domain().end();\n  std::vector::const_iterator label = labels.begin();\n  for( ; asg != end; ++asg, ++label) {\n    double nv = (*label-mean)/std_dev;\n    prior.set_logP( *asg, -1*nv*nv );\n  }\n}\n"
  },
  {
    "path": "toolkits/graphical_models/factors/tests/test_sparse_table/test_sparse_table.cpp",
    "content": "/**  \n *  Software submitted by \n *  Systems & Technology Research / Vision Systems Inc., 2013\n *\n *  Approved for public release; distribution is unlimited. [DISTAR Case #21428]\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * Test various functions of a sparse table\n *\n *  \\author Scott Richardson \n */\n\n#include \n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n\n\nconst size_t MAX_DIM = 4;\ntypedef graphlab::dense_table          dense_table_t;\ntypedef graphlab::sparse_table         sparse_table_t;\ntypedef graphlab::discrete_assignment  assignment_t;\ntypedef graphlab::discrete_domain      domain_t;\ntypedef graphlab::discrete_variable             variable_t;\n\n\nsparse_table_t create_rand_sparse_table(unsigned v0_id, unsigned v1_id, unsigned v2_id) \n{\n  variable_t v0(v0_id, 4);\n  variable_t v1(v1_id, 3);\n  variable_t v2(v2_id, 2);\n\n  std::vector args;\n  args.push_back(v0);\n  args.push_back(v1);\n  args.push_back(v2);\n\n  domain_t dom(args); \n  sparse_table_t st(dom);\n\n  domain_t::const_iterator asg_it = dom.begin();\n  domain_t::const_iterator end = dom.end();\n  for( ; asg_it != end; ++asg_it) {\n    if(rand() % 100 <= 20) {\n      st.set_logP( *asg_it, -1 * (rand() % 100) );\n    }\n  }\n\n  return st;\n}\n\n// [-Inf -Inf -3   -Inf]\n// [-15  -4   -Inf -23 ]\n// [-Inf -Inf -Inf -Inf]\n//\n// [-Inf -Inf -Inf -Inf]\n// [-20  -12  -19  -78 ]\n// [-Inf -Inf -Inf -32 ]\nvoid create_data_vector(std::vector > &data, \n    const domain_t &dom) \n{\n  {\n  size_t sa[] = {2,1,1}; \n  assignment_t da(dom, std::vector(sa, sa+3));\n  data.push_back(std::make_pair(da.linear_index(), -19));\n  } {  \n  size_t sa[] = {3,2,1}; \n  assignment_t da(dom, std::vector(sa, sa+3));\n  data.push_back(std::make_pair(da.linear_index(), -32));\n  } {\n  size_t sa[] = {2,0,0}; \n  assignment_t da(dom, std::vector(sa, sa+3));\n  data.push_back(std::make_pair(da.linear_index(), -3));\n  } {\n  size_t sa[] = {3,1,0}; \n  assignment_t da(dom, std::vector(sa, sa+3));\n  data.push_back(std::make_pair(da.linear_index(), -23));\n  } {\n  size_t sa[] = {3,1,1}; \n  assignment_t da(dom, std::vector(sa, sa+3));\n  data.push_back(std::make_pair(da.linear_index(), -78));\n  } {\n  size_t sa[] = {1,1,1}; \n  assignment_t da(dom, std::vector(sa, sa+3));\n  data.push_back(std::make_pair(da.linear_index(), -12));\n  } {\n  size_t sa[] = {1,1,0}; \n  assignment_t da(dom, std::vector(sa, sa+3));\n  data.push_back(std::make_pair(da.linear_index(), -4));\n  } {\n  size_t sa[] = {0,1,0}; \n  assignment_t da(dom, std::vector(sa, sa+3));\n  data.push_back(std::make_pair(da.linear_index(), -15));\n  } {  \n  size_t sa[] = {0,1,1}; \n  assignment_t da(dom, std::vector(sa, sa+3));\n  data.push_back(std::make_pair(da.linear_index(), -20));\n  }\n}\n\nvoid testDataReorder(unsigned v0_id, unsigned v1_id, unsigned v2_id) {\n  assert(v0_id != v1_id && v1_id != v2_id && v0_id != v2_id);\n\n  // setup the data as if its from belief prop\n  std::vector > data;\n  variable_t v0(0, 4);\n  variable_t v1(1, 3);\n  variable_t v2(2, 2);\n\n  std::vector args;\n  args.push_back(v0);\n  args.push_back(v1);\n  args.push_back(v2);\n  domain_t dom(args); \n\n  create_data_vector(data, dom);\n\n  // create a table with variables that are not resorted\n  sparse_table_t st_gm(args, data);\n  //std::cout << \"st_gm \" << st_gm << std::endl;\n\n\n  variable_t r0(v0_id, 4);\n  variable_t r1(v1_id, 3);\n  variable_t r2(v2_id, 2);\n  std::vector reordered_args;\n  reordered_args.push_back(r0);\n  reordered_args.push_back(r1);\n  reordered_args.push_back(r2);\n  sparse_table_t st(reordered_args, data);\n  //std::cout << \"st \" << st << std::endl;\n\n  assignment_t asg_gm(st_gm.domain());\n  domain_t::const_iterator asg_it = dom.begin();\n  domain_t::const_iterator end = dom.begin();\n  for( ; asg_it != end; ++asg_it) {\n    asg_gm.set_asg(v0, asg_it->asg(r0));\n    asg_gm.set_asg(v1, asg_it->asg(r1));\n    asg_gm.set_asg(v2, asg_it->asg(r2));\n    \n    //std::cout << \"st_gm{\" << asg_gm << \"}=\" << st_gm.logP(asg_gm) << \"; \"\n    //          << \"st{\" << asg_it << \"}=\" << st.logP(asg_it) << std::endl;\n    ASSERT_TRUE(st_gm.logP(asg_gm) == st.logP(*asg_it));\n  }\n}\n\nsparse_table_t create_sparse_table(unsigned v0_id, unsigned v1_id, unsigned v2_id) {\n  // create a table with dimensions ordered like the original table (e.g., from \n  // belief prop) to make computing the linear index easier\n  std::vector > data;\n  {\n  variable_t v0(0, 4);\n  variable_t v1(1, 3);\n  variable_t v2(2, 2);\n\n  std::vector args;\n  args.push_back(v0);\n  args.push_back(v1);\n  args.push_back(v2);\n  domain_t dom(args); \n\n  create_data_vector(data, dom);\n  }\n\n  variable_t v0(v0_id, 4);\n  variable_t v1(v1_id, 3);\n  variable_t v2(v2_id, 2);\n\n  std::vector args;\n  args.push_back(v0);\n  args.push_back(v1);\n  args.push_back(v2);\n\n  sparse_table_t st(args, data);\n\n  return st;\n}\n\n\nvoid multiplyTest(unsigned v0_id, unsigned v1_id, unsigned v2_id, dense_table_t& dt) \n{\n  dense_table_t  dt_gm = dt;\n  sparse_table_t st = create_sparse_table(v0_id, v1_id, v2_id);\n  sparse_table_t st_gm = st;\n  //std::cout << \"st \" << st << std::endl;\n\n  dt *= dt;\n\n  st *= st;\n  //std::cout << \"st *= st \" << st << std::endl;\n\n  {\n  dense_table_t st_as_dt(st.domain());\n  st.as_dense_table(st_as_dt); \n  double err = dt.l1_diff(st_as_dt);\n  //std::cout << \"err: \" << err << std::endl;\n  ASSERT_LT(err, 1e-4);\n  }\n\n\n  dt = dt_gm;\n  st = st_gm;\n  unsigned msg_length = st.var(st.domain().var_location(v0_id)).size();\n  variable_t v0(v0_id, msg_length);\n  dense_table_t msg(v0);\n\n  domain_t::const_iterator asg = msg.domain().begin();\n  domain_t::const_iterator end = msg.domain().end();\n  for( ; asg != end; ++asg) {\n    msg.set_logP( *asg, -1*(rand() % 100) );\n  }\n\n  dt *= msg;\n\n  st *= msg;\n  //std::cout << \"st *= msg \" << st << std::endl;\n\n  {\n  dense_table_t st_as_dt(st.domain());\n  st.as_dense_table(st_as_dt);\n  double err = dt.l1_diff(st_as_dt);\n  ASSERT_LT(err, 1e-4);\n  }\n}\n\nint main() {\n  // create a table \n  sparse_table_t st_gm = create_sparse_table(2, 0, 1);\n  //std::cout << \"st_gm \" << st_gm << std::endl;  \n  dense_table_t dt(st_gm.domain());\n  st_gm.as_dense_table(dt);\n\n  // equals test\n  sparse_table_t st = create_sparse_table(2, 0, 1);\n  //std::cout << \"st \" << st << std::endl;\n  ASSERT_TRUE(st == st_gm);\n\n  // copy test\n  sparse_table_t st_copy;\n  st_copy = st;\n  ASSERT_TRUE(st == st_copy);\n  \n  // test sparse table data reorder\n  testDataReorder(2, 3, 4);\n  testDataReorder(2, 4, 3);\n  testDataReorder(3, 2, 4);\n  testDataReorder(3, 4, 2);\n  testDataReorder(4, 2, 3);\n  testDataReorder(4, 3, 2);\n\n  // multiply test - compare a dense table to the sparse table. \n  // (i've already compared a pre-ordered dense table to the \n  // re-ordered ones.)\n  multiplyTest(2, 0, 1, dt);\n\n  std::cout << \"All tests passed\" << std::endl;\n}\n"
  },
  {
    "path": "toolkits/graphical_models/graphical_models.dox",
    "content": "/**\n\n\\page graphical_models Graphical Models\n\n\\brief The Graphical Models toolkit contains a collection of\napplications for reasoning about structured noisy data.  Graphical models provide a compact interpretable\nrepresentation of complex statistical phenomena by encoding random\nvariables as vertices in a graph and relationships between those\nvariables as edges. Given a graphical model representation, we can\nthen apply Bayes rule to quantitatively infer\nproperties of some variables given observations about others.\nGraphical models also provide the unique ability to quantify\nuncertainty in our prediction.\n\n\\section distributed_dual_decomposition Distributed Dual Decomposition\n\nDual Decomposition (DD), also called Lagrangian Relaxation, is a powerful technique \nwith a rich history in Operations Research. DD solves a relaxation of difficult optimization\nproblems by decomposing them into simpler subproblems, solving these simpler subproblems\nindependently and then combining these solutions into an approximate global solution. \n\nMore details about DD for solving Maximum A Posteriori (MAP) inference problems in Markov Random\nFields (MRFs) can be found in the following:\n\n\\verbatim\nD. Sontag, A. Globerson, T. Jaakkola. \nIntroduction to Dual Decomposition for Inference. \nOptimization for Machine Learning, editors S. Sra, S. Nowozin, and S. J. Wright: MIT Press, 2011.\n\\endverbatim\n\nImplemented by Andre' F. T. Martins and Dhruv Batra.\n\n\\subsection running_ddd Running DDD\n\nThe input MRF graph is assumed to be in the standard UAI file format. \nFor example a 3x3 grid MRF can be found here: grid3x3.uai.\n\nThe program can be run like this:\n\n\\verbatim\n> ./dd --graph grid3x3.uai \n\\endverbatim\n\nOther arguments are:\n\n\\li --help Display the help message describing the list of\noptions.\n\n\\li --output The output directory in which to save\nthe final predictions.\n\n\\li --dualimprovthres (Optional, default 0.00001) The amount of change in\ndual objective (in log-space) that will be tolerated at convergence.\n\n\\li --pdgapthres (Optional, default 0.1) The tolerance level for zero primal-dual gap.\n\n\\li --maxiter (Optional, default 10000) The maximum no. of dual update iterations.\n\n\\li --engine (Optional, Default: asynchronous) The engine type to\nuse when executing the vertex-programs\n       - synchronous: All LoopyBP updates are run at the same\n         time (Synchronous BP). This engine exposes greater parallelism but is less\n         computationally efficient.\n       - asynchronous: LoopyBP updates are run asynchronous\n         with priorities (Residual BP).  This engine is has greater\n         overhead and exposes less parallelism but can substantially\n         improve the rate over convergence.\n\n\\li --ncpus (Optional, Default 2) The number of local computation \nthreads to use on each machine.  This should typically match the number \nof physical cores. \n\n\\li --scheduler (Optional, Default sweep) The scheduler to use when \nrunning with the asynchronous engine.  The default is typically sufficient. \n\n\\li --engine_opts (Optional, Default empty) Any additional engine\noptions. See --engine_help for a list of options.\n\n\n\\li --graph_opts (Optional, Default empty) Any additional graph\noptions. See --graph_help for a list of options.\n\n\\li --scheduler_opts (Optional, Default empty) Any additional scheduler\noptions. See --scheduler_help for a list of options.\n\n\\section structured_prediction Structured Prediction\n\nCurrently the Graphical Models toolkit contains a discrete structured\nprediction application which can be applied to a wide range of\nprediction tasks where we have prior noisy predictions for a large\nnumber of variables (e.g., political inclination of each user or\narticle) and a graph encoding similarity or dissimilarity\nrelationships between those variables (e.g., friends share similar\npolitical inclinations).  The structured prediction application then\ninfers the posterior distribution for each random variable improving\nupon the prior prediction and providing a measure of uncertainty.\n\n\\subsection structured_prediction_example Structured Prediction Example\n\nFor example, supposed we had the recent posts for each user in a large\nsocial network.  Based on frequency each user mentions a conservative\nor liberal news item we might be able to construct a noisy prior\nestimate of their political inclination.  A user with no posts may\nhave a prior of 0.5 conservative and 0.5 liberal while another user\nthat frequently mentions a conservative pundit might have a prior of\n0.8 conservative and 0.2 liberal.  If a user with no posts is friends\nwith a user that frequently mentions conservative news items, then it\nis more likely that the user with no posts is also conservative. More\ngenerally we can leverage the social network to improve our prediction\nfor each user by examining not only their immediate friends but also\nthe community around each user.  This exactly what the structured\nprediction application accomplishes.  The output of the structured\nprediction application is the posterior estimates for each user.\n\n\n\n\\subsection structured_prediction_model The Structured Prediction Model\n\nThe structure prediction application applies the Loopy\nBelief propagation (LBP) algorithm to a pair-wise Markov Random Field encoding the classic Potts\nModel.  The joint probability mass function is given by:\n\n\\image html potts_model.png\n\n\nThe edge weight \\c w is obtained from the graph file but defaults to\nw=1 if no edge weight is provided.  The smoothing paramater \\c\nSMOOTHING can be set as a command line argument and controls the\ngeneral smoothing.\n\n\\subsection loopy_bp_algorithm Loopy BP Algorithm \n\nThe structured prediction application uses an Loopy BP\napproximate inference algorithm to estimate the posterior marginals.\nThe Loopy BP algorithm iteratively estimates a set of edge parameters\ncommonly referred to as \"messages.\"  The structured prediction\napplication uses the asynchronous residual variant of the Loopy BP\nalgorithm.\n\n\n\\subsection structured_prediction_data Synthetic Data\n\nTo demonstrate the power of the structured prediction application we\nhave provided a synthetic dataset generator.  To use the synthetic\ngenerator simply build and run:\n\n\\verbatim\n./synthetic_image_data \n\\endverbatim\n\nThis will create the synthetic noisy image:\n\n\\image html noisy_img.jpeg\n\nas well as the true underlying image that we would like to recover:\n\n\\image html orig_img.jpeg\n\nEach pixel in the image corresponds to a random variable whose unknown\nis the true pixel color.  The goal is to use the neighborhood of each\npixel to improve our estimate and resolve the original image.  The \\c\nsynthetic_image_data application will also create the two input files\nneeded to run the structured prediction application. The first is\nsynthetic prior estimates for each pixel.  Each row begins with the\nrandom variable id followed by the prior probability distribution for\nthat random variable.  Notice that the prior assigns half of the mass\nto the observed pixel value and the remaining mass to the other\ncandidate pixel values.\n\n\\verbatim\n> head synth_vdata.tsv \n0\t0.125\t0.5\t0.125\t0.125\t0.125\n1\t0.125\t0.125\t0.125\t0.125\t0.125\n2\t0.125\t0.125\t0.5\t0.125\t0.125\n3\t0.125\t0.125\t0.125\t0.125\t0.5\n4\t0.125\t0.125\t0.125\t0.125\t0.5\n\\endverbatim\n\nThe second \\c synth_edata.tsv file contains the graph structure with\neach line corresponding to an edge.  Here we do not assign edge\nweights (and so the default weight of 1) will be used on all edges.\nHad we wanted to use weighted edges we would have added the weight\nvalue after each edge.\n\n\\verbatim\n> head synth_edata.tsv \n0\t65536\n0\t1\n1\t65537\n1\t2\n2\t65538\n2\t3\n\\endverbatim\n\nWe can now run the structured prediction application on the synthetic\nimage.\n\n\\verbatim\n> ./lbp_structured_prediction --prior synth_vdata.tsv --graph synth_edata.tsv \\\n                          --output posterior_vdata.tsv\n\\endverbatim\n\nOnce the application terminates the final predictions will be stored\nin the sequence of files \\c posterior_vdata.tsv_X_of_X in exactly the\nsame format as the prior \\c synth_vdata.tsv.  \n\n\\verbatim\n> ls -l posterior_vdata.tsv_*\nposterior_vdata.tsv_1_of_2\nposterior_vdata.tsv_2_of_2\n\\endverbatim\n\nin the format:\n\n\\verbatim\n> head posterior_vdata.tsv_1_of_2 \n0\t0.0237064\t0.0947784\t0.0245065\t0.0323516\t0.824657\n1\t0.00886895\t0.0176509\t0.0114683\t0.0112453\t0.950767\n2\t0.00402855\t0.00489077\t0.0161093\t0.00426689\t0.970705\n3\t0.00088747\t0.00091284\t0.00124409\t0.000894688\t0.996061\n4\t0.000696577\t0.000695895\t0.000706134\t0.000695375\t0.997206\n5\t0.000740404\t0.000705437\t0.000706437\t0.000705451\t0.997142\n\\endverbatim\n\nTo visualize the predictions for the synthetic application we run:\n\n\\verbatim\n> cat posterior_vdata.tsv_* | ./synthetic_image_data --pred pred_image.jpeg\nCreate a synthetic noisy image.\nReading in predictions\nnrows: 200\nncols: 200\nminp:  0\nmaxp:  4\n\\endverbatim\n\nIf we then open \\c pred_image.jpeg we get:\n\n\\image html pred_img.jpeg\n\nNot bad!\n\n\\subsection structured_predictions_options Options\n\n\\li --help Display the help message describing the list of\noptions.\n\n\\li --prior The prior vertex data file.\n\n\\li --output The output directory/file_prefix in which to save\nthe final predictions.\n\n\\li --graph The graph describing the random variable dependency\nstructure as well as optional weights.\n\n\\li --smoothing (Optional, Default 2) The default smoothing\nparameter. Larger values imply stronger relationships between adjacent\nrandom variables in the graph.\n\n\\li --damping (Optional, Default 0.1) The amount of damping to\nuse.  Damping can help ensure that the algorithm converges.  Larger\ndamping values lead to slower but more reliable convergence.\n\n\\li --tol (Optional, default 0.01) The amount of change in\nparameter values (in log-space) that will be tolerated at convergence.\n\n\\li --map (Optional, default false) If set to true the\nmaximizing assignment will be returned in the output instead of the\ndistribution. \n\n\n\\li --engine (Optional, Default: asynchronous) The engine type to\nuse when executing the vertex-programs\n       - synchronous: All LoopyBP updates are run at the same\n         time (Synchronous BP). This engine exposes greater parallelism but is less\n         computationally efficient.\n       - asynchronous: LoopyBP updates are run asynchronous\n         with priorities (Residual BP).  This engine is has greater\n         overhead and exposes less parallelism but can substantially\n         improve the rate over convergence.\n\n\\li --ncpus (Optional, Default 2) The number of local computation \nthreads to use on each machine.  This should typically match the number \nof physical cores. \n\n\\li --scheduler (Optional, Default sweep) The scheduler to use when \nrunning with the asynchronous engine.  The default is typically sufficient. \n\n\\li --engine_opts (Optional, Default empty) Any additional engine\noptions. See --engine_help for a list of options.\n\n\n\\li --graph_opts (Optional, Default empty) Any additional graph\noptions. See --graph_help for a list of options.\n\n\\li --scheduler_opts (Optional, Default empty) Any additional scheduler\noptions. See --scheduler_help for a list of options.\n\n\n\n*/\n\n\n\n// \n// P(x_1, \\ldots, x_n) \\propto \n// \\exp \\left(\n// - \\sum_{(i,j) \\in E} I[x_i \\neq x_j ] w_{(i,j)} * \\mathtt{SMOOTHING}\n// \\right) \\prod_{i \\in V} \\mathtt{prior}_i(x_i)\n// \n"
  },
  {
    "path": "toolkits/graphical_models/lbp_structured_prediction.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n *\n * \\brief This application used for structured prediction on a graph.\n * For example, suppose you want to model the interests of users in a\n * social network.  \n *\n * Overview and Usage\n * ======================\n *\n * For simplicity lets suppose you want to know the users interest in\n * the categories movies, sports, and music.  After analyzing each\n * users profile you might be able to estimate a crude distribution\n * over her interests.  However you would like to leverage similarity\n * among friends to improve your estimates.  This application is\n * designed to do exactly that.\n *\n * As an input you provide two folders (or files) the first contains\n * the prior probabilities for each vertex in the form:\n *\n *    \\t  \\t  ... \\n\n *\n * For example:\n *   \n *   1    0.2   0.2   0.6\n *   2    0.3   0.6   0.1\n *   3    0.3   0.3   0.4\n *           ... \n * \n * The second folder contains the graph structure in the form:\n *\n *    \\t  \\t [Optional Weight]\n *\n * For example:\n *\n *   1   2\n *   1   3  1.7\n *   3   2  0.3\n *\n * The default weight value is 1 (times the smoothing parameter passed\n * in as a command line argument).  Larger weight values imply\n * stronger relationships.  A negative weight implies a \"repulsive\"\n * relationship in which neighboring vertices would like to have\n * different assignments.\n *\n * We have provided a synthetic data generator which creates a\n * synthetic dataset for an simulated image denoising task. See the\n * synthetic_image_data application for details.\n *\n * As output the application produces another set files with a format\n * identical to the vertex prior file with each weight (probability)\n * corresponding to the posterior predictions.\n *\n * Technical Explanation\n * ========================\n *\n * This application creates a pair-wise Markov Random Field with\n * Ising-Potts edge factors and then uses residual loopy belief\n * propagation to compute posterior belief estimates for each vertex.\n *\n *\n *  \\author Joseph Gonzalez\n */\n\n\n#include \n#include \n#include \n#include \n#include \n\n\n#include \n#include \"eigen_serialization.hpp\"\n\n\n\n#include \n#include \n\n\n\n\n/**\n * \\brief Eigen library vectors are used to store factor in _LOG\n * SPACE_.\n */\ntypedef Eigen::VectorXd factor_type;\n\n/**\n * \\brief The Ising smoothing parameter which controls the coupling\n * between adjacent predictions in the graph.  Larger values imply\n * greater smoothing (stronger coupling). \n *\n * \\code\n * edge_factor(xi, xj) = exp( (xi == xj)? 0 : -SMOOTHING * edge_weight ); \n * \\endcode\n *\n * Not that the default edge weight is 1 however the graph file can\n * contain an additional edge weight column which allows per edge\n * control of the smoothing parameter.\n *\n * This parameter is set as a command line argument.\n */\ndouble SMOOTHING = 2;\n\n/**\n * \\brief The Damping parameter which helps ensure stable convergence.\n * Larger damping values lead to slower but more stable convergence.\n *\n * Currently damping is implemented in log-space in the following\n * equation:\n *\n * \\code\n * log(new_message) = DAMPING * log(old_message) + \n *                         (1-DAMPING) * log(new_message);\n * \\endcode\n *\n * This parameter is set as a command line argument.\n */\ndouble DAMPING = 0.1;\n\n/**\n * \\brief The convergence threshold for each message.  Smaller values\n * imply tighter convergence but slower execution.\n *\n *\n * The algorithm convergence when:\n *   \n * \\code\n * sum(abs(log(old_message) - log(new_message))) < TOLERANCE\n * \\endcode\n *\n * The parameter is set as a command line argument\n */\ndouble TOLERANCE = 0.01;\n\n\n/**\n * \\brief The vertex data contains the vertex potential as well as the\n * current belief estimate and represents a random variable in the\n * Markov Random Field.\n *\n * The vertex potential represents the prior and is obtained from the\n * vertex prior file (stored in log form).\n *\n * The belief represents the current posterior estimate.\n */\nstruct vertex_data {\n  factor_type belief;\n  factor_type potential;\n  void load(graphlab::iarchive& arc) { arc >> belief >> potential; }\n  void save(graphlab::oarchive& arc) const { arc << belief << potential; }\n}; // end of vertex_data\n\n\n/**\n * \\brief The edge data represents an edge in the Markov Random Field\n * and contains the loopy belief propagation message in both\n * directions along that edge as well as the old message in each\n * direction.  In addition each edge contains the weight parameter\n * used to set edge specific smoothing (default value is 1).\n */\nclass edge_data {\n  /**\n   * \\brief We store old and new messages in both directions as an\n   * array of messages.  The particular message index is then computed\n   * using the \\ref message_idx function.\n   */\n  factor_type messages_[4];\n  /**\n   * \\brief The weight associated with the edge (used to scale the\n   * smoothing parameter)\n   */\n  double weight_;\n  /**\n   * \\brief The function used to compute the message index in the edge\n   * message array.\n   */\n  size_t message_idx(size_t source_id, size_t target_id, bool is_new) {\n    return size_t(source_id < target_id)  + 2 * size_t(is_new);\n  }\n\npublic:\n\n  edge_data(const double w = 1) : weight_(w) { }\n  const double& weight() const { return weight_; }\n\n  /**\n   * \\brief Get the new message value from source_id to target_id\n   */\n  factor_type& message(size_t source_id, size_t target_id) { \n    return messages_[message_idx(source_id, target_id, true)];\n  }\n  /**\n   * \\brief Get the old message value from source_id to target_id\n   */\n  factor_type& old_message(size_t source_id, size_t target_id) { \n     return messages_[message_idx(source_id, target_id, false)];\n  }\n\n  /**\n   * \\brief Set the old message value equal to the new message value\n   */\n  void update_old(size_t source_id, size_t target_id) { \n    old_message(source_id, target_id) = message(source_id, target_id);\n  }\n  \n  /**\n   * \\brief Initialize the edge data with source and target having the\n   * appropriate number of states.\n   *\n   * \\param source_id the vertex id of the source\n   * \\param nsource the number of states the source vertex takes\n   * \\param target_id the vertex id of the target\n   * \\param ntarget the number of states the target vertex takes\n   */\n  void initialize(size_t source_id, size_t nsource, size_t target_id, size_t ntarget) {\n    ASSERT_GT(nsource, 0); ASSERT_GT(ntarget, 0);\n    message(source_id, target_id).setZero(ntarget);\n    old_message(source_id, target_id).setZero(ntarget);\n    message(target_id, source_id).setZero(nsource);\n    old_message(target_id, source_id).setZero(nsource);\n  }\n  void save(graphlab::oarchive& arc) const {\n    for(size_t i = 0; i < 4; ++i) arc << messages_[i];\n    arc << weight_;\n  }\n  void load(graphlab::iarchive& arc) {\n    for(size_t i = 0; i < 4; ++i) arc >> messages_[i];\n    arc >> weight_;\n  }\n}; // End of edge data\n\n\n\n/**\n * \\brief The graph type used to store the Markov Random Field with\n * vertex data containing node potentials and beliefs and edge data\n * containing messages and weights.\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n\n/** \n * \\brief The Loopy Belief Propagation Vertex Program which computes\n * the product of the inbound messages during the gather phase,\n * updates the belief during the apply phase, and then computes the\n * new out-bound messages during the scatter phase.\n *\n * Since the gather phase is computing the product of the inbound\n * messages and the messages are stored in log form the resulting sum\n * operation is actually a vector sum and so the gather type is simply\n * the factor type and the operator+= operation for the factor type is\n * sufficient.\n *\n */\nstruct bp_vertex_program : \n  public graphlab::ivertex_program< graph_type, factor_type,\n                                    graphlab::messages::sum_priority >,\n  public graphlab::IS_POD_TYPE {\n\n  /**\n   * \\brief Since the MRF is undirected we will use all edges for gather and\n   * scatter\n   */\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of gather_edges \n\n  /**\n   * \\brief Update the old message to be the new message and collect the\n   * message value.\n   */\n  factor_type gather(icontext_type& context, const vertex_type& vertex, \n                     edge_type& edge) const {\n    const vertex_type other_vertex = get_other_vertex(edge, vertex);\n    edge_data& edata = edge.data();\n    // Update the old message with the value of the new Message.  We\n    // then receive the old message during gather and then compute the\n    // \"cavity\" during scatter (again using the old message).\n    edata.update_old(other_vertex.id(), vertex.id());\n    const factor_type& recv_message = \n      edata.old_message(other_vertex.id(), vertex.id());\n    // Ensure that the received message has the correct size\n    ASSERT_EQ(recv_message.size(), vertex.data().potential.size());\n    return recv_message;\n  }; // end of gather function\n\n  /**\n   * \\brief Multiply message product by node potential and update the\n   * belief.\n   */\n  void apply(icontext_type& context, vertex_type& vertex, \n             const factor_type& total) {\n    // If we have no neighbors than the belief is equal to the\n    // potential so simply update the belief\n    if(vertex.num_in_edges() + vertex.num_out_edges() == 0) {\n      vertex.data().belief = vertex.data().potential;\n    } else {\n      vertex_data& vdata = vertex.data();\n      ASSERT_EQ(vdata.potential.size(), total.size());\n      // Multiply (add in log space) the potential to compute the belief\n      vdata.belief = vdata.potential + total;\n      ASSERT_GT(vdata.belief.size(), 0);\n      // Rescale the belief to ensure numerical stability.  (This is\n      // essentially normalization in log-space.)\n      vdata.belief.array() -= vdata.belief.maxCoeff();\n    }\n  }; // end of apply\n\n  /**\n   * \\brief Since the MRF is undirected we will use all edges for gather and\n   * scatter\n   */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of scatter edges\n\n  /**\n   * \\brief Compute new message value for each edge.\n   */\n  void scatter(icontext_type& context, const vertex_type& vertex, \n               edge_type& edge) const {  \n    const vertex_type other_vertex = get_other_vertex(edge, vertex);\n    edge_data& edata = edge.data();\n    // Divide (subtract in log space) out of the belief the old in\n    // message to construct the cavity\n    const factor_type& old_in_message = \n      edata.old_message(other_vertex.id(), vertex.id());\n    ASSERT_EQ(old_in_message.size(), vertex.data().belief.size());\n    factor_type cavity = vertex.data().belief - old_in_message;\n    // compute the new message by convolving with the Ising-Potts Edge\n    // factor.\n    factor_type& new_out_message = \n      edata.message(vertex.id(), other_vertex.id());\n    const factor_type& old_out_message = \n      edata.old_message(vertex.id(), other_vertex.id());\n    convolve(cavity, edata.weight(), new_out_message);\n    // Renormalize (done in log space)\n    new_out_message.array() -= new_out_message.maxCoeff();\n    // Apply damping to the message to stabilize convergence.\n    new_out_message = DAMPING * old_out_message + \n      (1-DAMPING) * new_out_message;\n    // Compute message residual\n    const double residual = \n      (new_out_message - old_out_message).cwiseAbs().sum();\n    context.clear_gather_cache(other_vertex);\n    // Schedule the adjacent vertex\n    if(residual > TOLERANCE) context.signal(other_vertex, residual);\n }; // end of scatter\n\nprivate:\n\n  /**\n   * \\brief Compute the convolution of the cavity with the Ising-Potts\n   * edge potential and store the result in the message\n   *\n   * \\param cavity the belief minus the in-bound message\n   * \\param weight the edge weight used to scale the smoothing parameter\n   * \\param [out] message The message in which to store the result of\n   * the convolution.\n   */\n  inline void convolve(const factor_type& cavity, const double& weight, \n                       factor_type& message) const {\n    for(int i = 0; i < message.size(); ++i) {\n      double sum = 0;\n      for(int j = 0; j < cavity.size(); ++j) {\n        sum += std::exp( cavity(j)  + ( i == j? 0 : -(SMOOTHING*weight) ) ); \n      }\n      // To try and ensure numerical stability we do not allow\n      // messages to underflow in log-space\n      message(i) = (sum > 0)? std::log(sum) : \n        std::numeric_limits::min();\n    }\n  } // end of convolve\n  \n  /**\n   * \\brief Given an edge and a vertex return the other vertex along\n   * that edge. \n   */\n  inline vertex_type get_other_vertex(edge_type& edge, \n                                      const vertex_type& vertex) const {\n    return vertex.id() == edge.source().id()? edge.target() : edge.source();\n  }; // end of other_vertex\n\n}; // end of class bp_vertex_program\n\n\n\n\n\n\n\n/**\n * \\brief The vertex load is used by the graph loading API to parse\n * the lines of prior data in the vertex data file.\n *\n * This parser uses the boost::spirit library to parse the vertex data\n * file. As a consequence it is fairly flexible allowing both comma\n * and tab delimited files as well as vertices with different numbers\n * of states.\n */\nbool vertex_loader(graph_type& graph, const std::string& fname, \n                   const std::string& line) {\n  // If the line is empty simply skip it\n  if(line.empty()) return true;\n  // We use the boost spirit parser which requires (too) many separate\n  // namespaces so to make things clear we shorten them here.\n  namespace qi = boost::spirit::qi;\n  namespace ascii = boost::spirit::ascii;\n  namespace phoenix = boost::phoenix;\n  graphlab::vertex_id_type vid(-1);\n  std::vector values;\n  const bool success = qi::phrase_parse\n    (line.begin(), line.end(),       \n     //  Begin grammar\n     (\n      qi::ulong_[phoenix::ref(vid) = qi::_1] >> -qi::char_(\",\") >>\n      (qi::double_[phoenix::push_back(phoenix::ref(values), qi::_1)] % -qi::char_(\",\") )\n      )\n     ,\n     //  End grammar\n     ascii::space); \n  // Test to see if the boost parser was able to parse the line\n  if(!success) {\n    logstream(LOG_ERROR) << \"Parse error in vertex prior parser.\" << std::endl;\n    return false;\n  }\n\n  // Ensure that a prior was provided.  Technically this should not be\n  // reached since the parser requires at least one prior entry\n  if(values.empty()) {\n    logstream(LOG_ERROR) << \"Vertex has no prior.\" << std::endl;\n    return false;\n  }\n\n  // Renormalize the vertex data. We require positive probabilities.\n  double sum = 0;\n  for(size_t i = 0; i < values.size(); ++i) {\n    if(values[i] < 0) { \n      logstream(LOG_ERROR) << \"Encountered negative probability.\" << std::endl;\n      return false;\n    }\n    if(values[i] == 0) { \n      logstream(LOG_ERROR) \n        << \"Zero probability assignments are not currently supported.\" << std::endl;\n      return false;\n    }\n    sum += values[i]; \n  }\n  ASSERT_GT(sum, 0);\n  for(size_t i = 0; i < values.size(); ++i) values[i] /= sum;\n\n  vertex_data vdata;\n  vdata.potential.resize(values.size());\n  for(size_t i = 0; i < values.size(); ++i) {\n    ASSERT_GT(values[i], 0);\n    vdata.potential(i) = std::log(values[i]);\n  }\n  graph.add_vertex(vid, vdata);\n  return true;\n} // end of vertex_loader;\n\n\n/**\n * \\brief The edge data loader is used by the GraphLab graph loading\n * API to parse lines in the edge data file. \n */\nbool edge_loader(graph_type& graph, const std::string& fname, \n                 const std::string& line) {\n  ASSERT_FALSE(line.empty()); \n  namespace qi = boost::spirit::qi;\n  namespace ascii = boost::spirit::ascii;\n  namespace phoenix = boost::phoenix;\n  graphlab::vertex_id_type source(-1), target(-1);\n  double weight = 1;\n  const bool success = qi::phrase_parse\n    (line.begin(), line.end(),       \n     //  Begin grammar\n     (\n      qi::ulong_[phoenix::ref(source) = qi::_1] >>  -qi::char_(',') \n      >> qi::ulong_[phoenix::ref(target) = qi::_1] >>  \n      -(-qi::char_(',') >> qi::double_[phoenix::ref(weight) = qi::_1])\n      )\n     ,\n     //  End grammar\n     ascii::space); \n  if(!success) return false;  \n  graph.add_edge(source, target, edge_data(weight));\n  return true;\n} // end of edge loader\n\n\n\n/**\n * \\brief The edge initializer is used to allocate the messages along\n * each edge based on the number of states of the source and target\n * vertex.\n */\nvoid edge_initializer(graph_type::edge_type& edge) {\n  edge_data& edata = edge.data();\n  const graphlab::vertex_id_type source_id = edge.source().id();\n  const size_t nsource = edge.source().data().potential.size(); \n  const graphlab::vertex_id_type target_id = edge.target().id();\n  const size_t ntarget = edge.target().data().potential.size();\n  edata.initialize(source_id, nsource, target_id, ntarget);\n} // end of edge initializer\n\n\n\n\n/**\n * \\brief The belief prediction saver is used to save the belief\n * predictions for each vertex.\n */\nstruct belief_prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    std::stringstream strm;\n    strm << vertex.id() << '\\t';\n    factor_type pred = vertex.data().belief;\n    double sum = 0;\n    for(int i = 0; i < pred.size(); ++i) \n      sum += (pred(i) = std::exp(pred(i)));\n    pred.array() /= sum;\n    for(int i = 0; i < pred.size(); ++i) \n      strm << pred(i) << (i+1 < pred.size()? '\\t' : '\\n');\n    return strm.str();\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\"; // nop\n  }\n}; // end of belief_prediction_saver\n\n\n/**\n * \\brief The MAP prediction saver is used to save the map estimated\n * for each vertex.  The MAP estimate is the most likely assignment\n */\nstruct map_prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    std::stringstream strm;\n    size_t prediction = 0;\n    vertex.data().belief.maxCoeff(&prediction);\n    strm << vertex.id() << '\\t' << prediction << '\\n';\n    return strm.str();\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\"; // nop\n  }\n}; // end of map prediction_saver\n\n\n\n\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  // Parse command line options -----------------------------------------------\n  // \\todo update description string\n  const std::string description = \"Structure prediction solver\";\n  graphlab::command_line_options clopts(description);\n  std::string prior_dir; \n  std::string graph_dir;\n  std::string output_dir = \"pred\";\n  std::string exec_type = \"async\";\n  bool map = false;\n  clopts.attach_option(\"prior\", prior_dir,\n                       \"The directory containing the prior\");\n  clopts.add_positional(\"prior\");\n  clopts.attach_option(\"graph\", graph_dir,\n                       \"The directory containing the adjacency graph\");\n  clopts.add_positional(\"graph\");\n  clopts.attach_option(\"output\", output_dir,\n                       \"The directory in which to save the predictions\");\n  clopts.add_positional(\"output\");\n  clopts.attach_option(\"smoothing\", SMOOTHING,\n                       \"The amount of smoothing (larger = more)\");\n  clopts.attach_option(\"damping\", DAMPING,\n                       \"The amount of damping (0 -> no damping and 1 -> no progress)\");\n  clopts.attach_option(\"tol\", TOLERANCE,\n                       \"The tolerance level for convergence.\");\n  clopts.attach_option(\"map\", map,\n                       \"Return maximizing assignment instead of the posterior distribution.\");\n  clopts.attach_option(\"engine\", exec_type,\n                       \"The type of engine to use {async, sync}.\");\n  if(!clopts.parse(argc, argv)) {\n    graphlab::mpi_tools::finalize();\n    return clopts.is_set(\"help\")? EXIT_SUCCESS : EXIT_FAILURE;\n  }\n\n  if(prior_dir.empty()) {\n    logstream(LOG_ERROR) << \"No prior was provided.\" << std::endl;\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n\n  if(graph_dir.empty()) {\n    logstream(LOG_ERROR) << \"No graph was provided.\" << std::endl;\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n\n  // Start the webserver\n  graphlab::launch_metric_server();\n\n  ///! load the graph\n  graph_type graph(dc, clopts);  \n\n\n  ///! load the graph\n  graph.load(prior_dir, vertex_loader);\n  graph.load(graph_dir, edge_loader);\n  graph.finalize();\n  graph.transform_edges(edge_initializer);\n\n  typedef graphlab::omni_engine engine_type;\n  engine_type engine(dc, graph, exec_type, clopts);\n  engine.signal_all();\n  graphlab::timer timer;\n  engine.start();  \n  const double runtime = timer.current_time();\n    dc.cout() \n    << \"----------------------------------------------------------\" << std::endl\n    << \"Final Runtime (seconds):   \" << runtime \n    << std::endl\n    << \"Updates executed: \" << engine.num_updates() << std::endl\n    << \"Update Rate (updates/second): \" \n    << engine.num_updates() / runtime << std::endl;\n    \n    \n  std::cout << \"Saving predictions\" << std::endl;\n  const bool gzip_output = false;\n  const bool save_vertices = true;\n  const bool save_edges = false;\n  const size_t threads_per_machine = 2;\n  if(map) {\n    graph.save(output_dir, map_prediction_saver(),\n               gzip_output, save_vertices, \n               save_edges, threads_per_machine);\n  } else { \n    graph.save(output_dir, belief_prediction_saver(),\n               gzip_output, save_vertices, \n               save_edges, threads_per_machine);\n  }\n\n\n    \n  //  graphlab::stop_metric_server_on_eof();\n  graphlab::stop_metric_server();\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n\n\n} // end of main\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
  },
  {
    "path": "toolkits/graphical_models/mplp_denoise.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n/**\n * This file contains an example of graphlab used for MAP inference \n * in a discrete graphical model (pairwise MRF). The algorithm\n * implemented is the MPLP LP-Relaxation scheme of Globerson & Jaakkola. \n *\n *  \\author Dhruv Batra\n */\n\n\n#include \n#include \n#include \n\n\n#include \n\n#include \n#include   \n\n\n\n#include \n\n#include \"eigen_serialization.hpp\"\n\n#include \n\ntypedef Eigen::VectorXd vector;\ntypedef Eigen::MatrixXd matrix;\n\ntemplate \ninline std::ostream& operator<<(std::ostream& os, std::vector& x)\n{\n    typename std::vector::const_iterator i(x.begin());\n    while(i != x.end()) os << *i++ << ' ';\n    return os;\n}\n\n// Global variables\nsize_t NCOLORS;\ndouble SIGMA;\ndouble BOUND;\n\n// LP-based upper-bound on MAP\ngraphlab::mutex mutex;\n//mutex.lock();\ndouble LPval = 0;\ndouble MAPval = 0;\ndouble MAPrepval = 0;\n//mutex.unlock();\n\n// Shared base edge potential\nmatrix THETA_ij; \n\n// keep track of predictions at each node\nvector PRED_COLOR;\n\n// check if all nodes are visited\n//Eigen::Matrix, Eigen::Dynamic,1> vinit;\n//Eigen::Matrix, Eigen::Dynamic,1> vapply;\n//vector vinit;\n//vector vapply;\n//std::vector > vinit;\n//std::vector > vapply;\nstd::vector vinit;\nstd::vector vapply;\n\n\n\n// STRUCTS (Edge and Vertex data) =============================================>\n\n/**\n * Each GraphLab vertex is a (pairwise) factor from the MRF\n */\nstruct vertex_data {\n    /** variable ids */\n    int i, j; \n    \n    // degree of these nodes in the MRF\n    int deg_i, deg_j; \n    \n    /** observed color for each variable */\n    float obs_color_i, obs_color_j;\n    /** predicted color for each variable */\n    float pred_color_i, pred_color_j;\n    \n    // current maximizers of reparameterized theta_i, theta_j and theta_IJ\n    int maxI, maxJ, maxIJ_i, maxIJ_j;\n    \n    // current contribution to LP dual value\n    double vali, valj, valij;\n    // current contribution to MAP value\n    double pvali, pvalj, pvalij;\n    // current contribution to MAPrep value\n    double prvali, prvalj, prvalij;\n    \n    // since variables i and j are present in multiple factors, this determines who owns them\n    bool iowner, jowner; \n    \n    /** dual variables being optimized (or messages) */\n    vector delf_i, delf_j;  \n\n    // constructor\n    vertex_data(): i(-1), j(-1), deg_i(0), deg_j(0), \n    obs_color_i(-1), obs_color_j(-1), \n    pred_color_i(0), pred_color_j(0),\n    vali(0), valj(0), valij(0),\n    pvali(0), pvalj(0), pvalij(0), \n    prvali(0), prvalj(0), prvalij(0), \n    iowner(false), jowner(false)\n    { }\n    \n    void save(graphlab::oarchive& arc) const \n    {\n        arc << i << j \n        << deg_i << deg_j \n        << obs_color_i << obs_color_j \n        << pred_color_i << pred_color_j \n        << maxI << maxJ << maxIJ_i << maxIJ_j\n        << vali << valj << valij \n        << pvali << pvalj << pvalij \n        << prvali << prvalj << prvalij \n        << iowner << jowner\n        << delf_i << delf_j;\n    }\n    void load(graphlab::iarchive& arc) \n    {\n        arc >> i >> j \n        >> deg_i >> deg_j\n        >> obs_color_i >> obs_color_j \n        >> pred_color_i >> pred_color_j \n        >> maxI >> maxJ >> maxIJ_i >> maxIJ_j\n        >> vali >> valj >> valij\n        >> pvali >> pvalj >> pvalij\n        >> prvali >> prvalj >> prvalij\n        >> iowner >> jowner\n        >> delf_i >> delf_j;\n    }\n}; // End of vertex data\n\n\n// /**\n//  * The data associated with a pair of factors in a pairwise MRF\n//  */\n//struct edge_data : public graphlab::IS_POD_TYPE \n//{\n//    // primal labelling; We assume pairwise factors, so intersection has\n//    // a single node\n//    int pred_color;\n//    \n//    // current contribution to LP dual value\n//    double dval;\n//    // current contribution to MAP value\n//    double pval;\n//    // current contribution to MAPrep value\n//    double prval;\n//    \n//    edge_data():\n//    pred_color(0),\n//    dval(0), pval(0), prval(0)\n//    {}\n//     \n//    void save(graphlab::oarchive& arc) const \n//    {\n//        arc << pred_color\n//        << dval << pval << prval; \n//    }\n//    void load(graphlab::iarchive& arc) \n//    {\n//        arc >> pred_color\n//        >> dval >> pval >> prval;\n//    }\n//}; // End of edge data\ntypedef graphlab::empty edge_data;\n\n/**\n * The graph type\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n// GraphLab Vertex Program ====================================================\n/**\n * The type passed around during the gather phase\n */\nstruct gather_type \n{\n    vector delf_i, delf_j;\n    \n    gather_type& operator+=(const gather_type& other) \n    {\n        if(!other.delf_i.size() == 0) \n        {\n            if(delf_i.size() == 0) delf_i = other.delf_i;\n            else delf_i += other.delf_i;\n        }\n        if(!other.delf_j.size() == 0) \n        {\n            if(delf_j.size() == 0) delf_j = other.delf_j;\n            else delf_j += other.delf_j;\n        }\n        return *this;\n    } // end of operator +=\n    void save(graphlab::oarchive& arc) const \n    {\n        arc << delf_i << delf_j;\n    }\n    void load(graphlab::iarchive& arc) \n    {\n        arc >> delf_i >> delf_j;\n    }\n}; // end of gather type\n\n\n\n/** \n * The core belief propagation update function.  This update satisfies\n * the graphlab update_function interface.  \n */\nclass mplp_vertex_program : \npublic graphlab::ivertex_program,\npublic graphlab::IS_POD_TYPE {\nprivate:\n    double priority;\npublic:\n    \n    mplp_vertex_program() : priority(0) { }\n    \n    // void save(graphlab::oarchive& arc) const { /** save members */ }\n    // void load(graphlab::iarchive& arc) { /** load members */ }\n    \n    /**\n     * This function is now called in the main by invoking:\n     * engine.transform_vertices(mplp_vertex_program::init)\n     */\n    static void init_vertex_data(icontext_type& context, vertex_type& vertex)\n    { \n        vertex_data& vdata = vertex.data();\n        \n        // Create zero messages\n        vdata.delf_i = vector::Zero(NCOLORS);\n        vdata.delf_j = vector::Zero(NCOLORS);\n        \n        // create temporary node potentials\n        vector theta_i = make_unary_potential(vertex, 'i');\n        vector theta_j = make_unary_potential(vertex, 'j');\n\n        // if we own i\n        if (vdata.iowner) \n        {\n            // get dual contribution\n            vdata.vali = theta_i.maxCoeff(&vdata.maxI); \n\n            // get primal contribution\n            vdata.pred_color_i = vdata.maxI;\n            vdata.pvali = vdata.vali;\n            \n            // also update the global copy\n            PRED_COLOR[vdata.i] = vdata.pred_color_i;\n\n            // get rep primal contribution\n            vdata.prvali = vdata.pvali;\n        }\n        else // if we don't own then just copy over the global predicted color\n            vdata.pred_color_i = PRED_COLOR[vdata.i];\n\n        // if we own j\n        if (vdata.jowner) \n        {\n            // get dual contribution\n            vdata.valj = theta_j.maxCoeff(&vdata.maxJ); \n            \n            // get primal contribution\n            vdata.pred_color_j = vdata.maxJ;\n            vdata.pvalj = vdata.valj;\n            \n            // also update the global copy\n            PRED_COLOR[vdata.j] = vdata.pred_color_j;\n\n            // get rep primal contribution\n            vdata.prvalj = vdata.pvalj;\n        }\n        else \n            vdata.pred_color_j = PRED_COLOR[vdata.j];\n        \n        // we always own edge i,j\n        vdata.valij = THETA_ij.maxCoeff(&vdata.maxIJ_i,&vdata.maxIJ_j); \n        vdata.pvalij = THETA_ij(vdata.pred_color_i, vdata.pred_color_j);\n        vdata.prvalij = vdata.pvalij;\n             \n        mutex.lock();\n        LPval += vdata.vali; LPval += vdata.valj; LPval += vdata.valij;\n        MAPval += vdata.pvali; MAPval += vdata.pvalj; MAPval += vdata.pvalij;\n        MAPrepval += vdata.prvali; MAPrepval += vdata.prvalj; MAPrepval += vdata.prvalij;\n        mutex.unlock();\n\n        // debug code to check in all nodes are inited\n        if (vinit[vertex.id()] == 0)\n            vinit[vertex.id()] = 1;\n    }\n    \n    /**\n     * Recv message is called by the engine to receive a message to this\n     * vertex program.  The vertex program can use this to initialize\n     * any state before entering the gather phase.  If the vertex\n     * program does not implement this function then the default\n     * implementation (NOP) is used.\n     */\n    // void init(icontext_type& context, const vertex_type& vertex, \n    //                   const message_type& msg) { /** NOP */ }\n    \n    /**\n     * Since the MRF is undirected we will use all edges for gather and\n     * scatter\n     */\n    edge_dir_type gather_edges(icontext_type& context,\n                               const vertex_type& vertex) const { \n        return graphlab::ALL_EDGES; \n    }; // end of gather_edges \n    \n    \n    // Run the gather operation over all in edges\n    gather_type gather(icontext_type& context, const vertex_type& target_vertex, \n                       edge_type& edge) const \n    {\n        const vertex_type source_vertex = get_other_vertex(edge, target_vertex);   \n        const vertex_data& source_vdata = source_vertex.data();\n        const vertex_data& target_vdata = target_vertex.data();\n        \n        // Accumulate message\n        gather_type ret_value;\n        if (target_vdata.i == source_vdata.i)\n            ret_value.delf_i = source_vdata.delf_i;\n        else if (target_vdata.j == source_vdata.i)\n            ret_value.delf_j = source_vdata.delf_i;\n        else if (target_vdata.i == source_vdata.j)\n            ret_value.delf_i = source_vdata.delf_j;\n        else if (target_vdata.j == source_vdata.j)\n            ret_value.delf_j = source_vdata.delf_j;\n        else assert(false); // invalid state\n        \n        return ret_value;\n    } // end of gather\n    \n    /** Update the dual parameters */\n    void apply(icontext_type& context, vertex_type& vertex, \n               const gather_type& sum) \n    {\n        // Make sure this vertex has neighbours. Everyone should have neighbours\n        ASSERT_GT(vertex.num_in_edges() + vertex.num_out_edges(), 0);\n        \n        vertex_data& vdata = vertex.data();  \n        vector theta_i = make_unary_potential(vertex, 'i');\n        vector theta_j = make_unary_potential(vertex, 'j');\n                \n        ASSERT_EQ(THETA_ij.rows(), theta_i.size());\n        ASSERT_EQ(THETA_ij.rows(), sum.delf_i.size());\n        ASSERT_EQ(THETA_ij.cols(), theta_j.size());\n        ASSERT_EQ(THETA_ij.cols(), sum.delf_j.size());   \n        \n        // debug code to check in all nodes are applied\n        if (vapply[vertex.id()] == 0)\n            vapply[vertex.id()] = 1;\n        \n        \n        ////////////////////////////////////////////\n        // Update outgoing messages (coordinate descent)\n        \n        // Backup the old prediction\n        const vector old_delf_i = vdata.delf_i;\n        const vector old_delf_j = vdata.delf_j;\n        \n        // Update del fi\n        vdata.delf_i = -(theta_i + sum.delf_i)/2 + \n        (THETA_ij + (theta_j + sum.delf_j).transpose().replicate(THETA_ij.rows(),1)).\n        rowwise().maxCoeff()/2;\n        // Update del fj\n        vdata.delf_j = -(theta_j + sum.delf_j)/2 + \n        ((THETA_ij + (theta_i + sum.delf_i).replicate(1,THETA_ij.cols())).\n         colwise().maxCoeff()).transpose()/2;\n        \n        ////////////////////////////////////////////\n        // Compute contributions to dual, primal and rep primal\n        \n        // Remove contribution of old labels from LPval\n        double LPremove=0, MAPremove=0, MAPrepremove=0;\n        LPremove += vdata.vali; LPremove += vdata.valj; LPremove += vdata.valij;\n        MAPremove += vdata.pvali; MAPremove += vdata.pvalj; MAPremove += vdata.pvalij;\n        MAPrepremove += vdata.prvali; MAPrepremove += vdata.prvalj; MAPrepremove += vdata.prvalij;\n\n        // Update dual, primal and rep primal contributions. \n        // TODO: if primal labelling changes at a node we own, update it's edge potential too\n        if (vdata.iowner)\n        {\n            // reparameterized node potential\n            vector thetarep_i = theta_i + sum.delf_i + vdata.delf_i;\n            \n            vdata.vali = thetarep_i.maxCoeff(&vdata.maxI);\n            \n            vdata.pred_color_i = vdata.maxI;\n            vdata.pvali = theta_i[vdata.pred_color_i];\n            \n            PRED_COLOR[vdata.i] = vdata.pred_color_i;\n            \n            vdata.prvali = thetarep_i[vdata.pred_color_i];\n        } \n        else\n            vdata.pred_color_i = PRED_COLOR[vdata.i];\n        if (vdata.jowner)\n        {\n            // reparameterized node potential\n            vector thetarep_j = theta_j + sum.delf_j + vdata.delf_j;\n            \n            vdata.valj = thetarep_j.maxCoeff(&vdata.maxJ);\n            \n            vdata.pred_color_j = vdata.maxJ;\n            vdata.pvalj = theta_j[vdata.pred_color_j];\n            \n            PRED_COLOR[vdata.j] = vdata.pred_color_j;\n\n            vdata.prvalj = thetarep_j[vdata.pred_color_j];\n        }\n        else\n            vdata.pred_color_j = PRED_COLOR[vdata.j];\n        \n        // We always own edge i,j\n        matrix thetarep_ij = THETA_ij - (vdata.delf_i.replicate(1,THETA_ij.cols()))\n                            - (vdata.delf_j.transpose().replicate(THETA_ij.rows(),1));\n        \n        vdata.valij = thetarep_ij.maxCoeff(&vdata.maxIJ_i, &vdata.maxIJ_j);\n        vdata.pvalij = THETA_ij(vdata.pred_color_i, vdata.pred_color_j);\n        vdata.prvalij = thetarep_ij(vdata.pred_color_i, vdata.pred_color_j);\n        \n        mutex.lock();\n        LPval -= LPremove; MAPval -= MAPremove; MAPrepval -= MAPrepremove;\n        LPval += vdata.vali; LPval += vdata.valj; LPval += vdata.valij;\n        MAPval += vdata.pvali; MAPval += vdata.pvalj; MAPval += vdata.pvalij;\n        MAPrepval += vdata.prvali; MAPrepval += vdata.prvalj; MAPrepval += vdata.prvalij;\n        mutex.unlock();\n        \n        ////////////////////////////////////////////\n        // Debugging printing and residuals\n        \n        //std::cout << vertex.id() << \": \" << vdata.i << \",\" << vdata.j << \"\\n\";\n        if (0) // (vdata.i == 0 )\n        {\n            mutex.lock();\n            std::cout << \"Applying at vertex: \" << vertex.id() << \"(\" << vdata.i << \",\" << vdata.j << \")\\n\";\n            \n            std::cout << LPval << \",\" << MAPval << \",\" << MAPrepval << \"\\t\" ;        \n            int vinitsum = 0, vapplysum = 0;\n            //std::vector >::iterator it;\n            std::vector::iterator it;\n            for (it = vinit.begin(); it < vinit.end(); ++it)\n                vinitsum += *it;\n            for (it = vapply.begin(); it < vapply.end(); ++it)\n                vapplysum += *it;\n            \n            // if all vertices have been visited start counting again\n            if (vapply.size() == vapplysum)\n                for (int i=0; i!=vapply.size(); ++i)\n                    vapply[i] = 0;\n            \n            std::cout << \"Verted Id: \" << vertex.id() << \" \"  << vdata.i << \",\" << vdata.j << \" \"; \n            std::cout << \"Inited: \" << vinitsum << \" Applied: \" << vapplysum << \"\\n\";\n            if (vinit.size() == vinitsum)\n                std::cout << \"Restarting counting of apply\\n\";\n            std::cout.flush();\n            mutex.unlock();\n        }\n        \n        if (0)//vdata.i == 1) \n        {\n            std::cout << \"\\n\\n\";\n            \n            std::cout << \"Pairwise Potential\\n\" << THETA_ij << \"\\n\\n\";\n            std::cout << \"theta_ij reparameterized: \\n\" <<         \n            (THETA_ij - vdata.delf_i.replicate(1,THETA_ij.cols()) \n             - vdata.delf_j.transpose().replicate(THETA_ij.rows(),1)   )  << \"\\n\\n\";\n            std::cout << \"maxIJ_i: \" << vdata.maxIJ_i << \" maxIJ_j: \" << vdata.maxIJ_j << \"\\n\\n\";\n            \n            std::cout << \"thetai \\n\" << theta_i << \"\\n\\n\";\n            std::cout << \"sum of incomming messages into i\\n\" << sum.delf_i << \"\\n\\n\";\n            std::cout << \"outgoing message to i\\n\" << vdata.delf_i << \"\\n\\n\";\n            std::cout << \" Reparamterized thetai\\n\" << (theta_i + sum.delf_i + vdata.delf_i) << \"\\n\\n\";\n            std::cout << \"maxI: \" << vdata.maxI << \"\\n\\n\";\n            \n            std::cout << \"thetaj \\n\" << theta_j << \"\\n\\n\";\n            std::cout << \"sum of incomming messages into j\\n\" << sum.delf_j << \"\\n\\n\";\n            std::cout << \"outgoing message to j\\n\" << vdata.delf_j << \"\\n\\n\";\n            std::cout << \" Reparamterized thetaj\\n\" << (theta_j + sum.delf_j + vdata.delf_j) << \"\\n\\n\";\n            std::cout << \"maxJ: \" << vdata.maxJ << \"\\n\\n\";\n            \n            std::cout << \"thetaij + j message\\n\" << (THETA_ij + sum.delf_j.transpose().replicate(THETA_ij.rows(),1))/2 << \"\\n\\n\";\n            \n            std::cout << (THETA_ij + sum.delf_j.transpose().replicate(THETA_ij.rows(),1)).\n            rowwise().maxCoeff()/2 << std::endl << std::endl;\n            \n            std::cout << \"thetaij + i message\\n\" << (THETA_ij + sum.delf_i.replicate(1,THETA_ij.cols()))/2 << \"\\n\\n\";\n            std::cout << \t  ((THETA_ij + sum.delf_i.replicate(1,THETA_ij.cols())).\n                               colwise().maxCoeff()).transpose()/2 << \"\\n\\n\";\n            \n            std::cout << \"Old del_fi\\n\" << vdata.delf_i << \"\\n\\n\";\n            std::cout << \"Old del_fj\\n\" << vdata.delf_j << \"\\n\\n\";\n            std::cout << \"New del_fi\\n\" << -(theta_i + sum.delf_i)/2 + \n            (THETA_ij + (theta_j + sum.delf_j).transpose().replicate(THETA_ij.rows(),1)).\n            rowwise().maxCoeff()/2 << \"\\n\\n\";\n            \n            std::cout << \"New del_fj\\n\" << -(theta_j + sum.delf_j)/2 + \n            ((THETA_ij + (theta_i + sum.delf_i).replicate(1,THETA_ij.cols())).\n             colwise().maxCoeff()).transpose()/2 << \"\\n\\n\";\n            \n            getchar();\n        }\n        \n        // const double residual = (vdata.delf_i - old_delf_i).cwiseAbs().sum() +\n        // (vdata.delf_j - old_delf_j).cwiseAbs().sum();\n        \n        //priority = residual;\n        priority = LPval - MAPval;\n        //std::cout << \"priority: \" << priority << std::endl;\n        //std::cout << LPval << std::endl;\n        //test code; for now, only run 1 iteration\n        //priority = 0;\n    } // end of apply\n    \n    /**\n     * Since the MRF is undirected we will use all edges for gather and\n     * scatter\n     */\n    edge_dir_type scatter_edges(icontext_type& context,\n                                const vertex_type& vertex) const { \n        //return priority < BOUND? graphlab::NO_EDGES : graphlab::ALL_EDGES; \n        return graphlab::ALL_EDGES;\n    }; // end of gather_edges \n    \n    \n    /** reschedule neighbors with a given priority and updated\n     predictions on each edge*/\n    void scatter(icontext_type& context, const vertex_type& vertex, \n                 edge_type& edge) const {  \n        context.signal(get_other_vertex(edge, vertex), priority);\n    } // end of scatter\n    \nprivate:\n    \n    /**\n     * Construct the unary evidence potential\n     */\n    static vector make_unary_potential(const vertex_type& vertex, \n                                       const char varid) {\n        vector potential(NCOLORS);\n        const double obs = varid == 'i'? \n        vertex.data().obs_color_i : vertex.data().obs_color_j;\n        const double sigmaSq = SIGMA*SIGMA;\n        for(int pred = 0; pred < potential.size(); ++pred) {\n            potential(pred) = -(obs - pred)*(obs - pred) / (2.0 * sigmaSq);\n        }\n        //potential /= std::abs(potential.sum());\n        \n        //float tmp = potential.minCoeff();\n        //potential.array() -= tmp; // (float) potential.minCoeff();\n        return potential;\n    } // end of make_potentail\n    \n    /**\n     * Return the other vertex\n     */\n    vertex_type get_other_vertex(edge_type& edge, \n                                 const vertex_type& vertex) const {\n        return vertex.id() == edge.source().id()? edge.target() : edge.source();\n    } // end of other_vertex\n    \n}; // end of MPLP vertex program\n\n\n\n\n/**\n * Define the engine type\n */\n//typedef graphlab::synchronous_engine engine_type;\ntypedef graphlab::async_consistent_engine engine_type;\n//typedef graphlab::asynchronous_consistent_engine engine_type;\n\n\n\n\n/////////////////////////////////////////////////////////////////////////////////////\n// Aggregator functions to compute primal & dual values\ndouble get_energy_fun(mplp_vertex_program::icontext_type& context, const mplp_vertex_program::vertex_type& vertex) \n{\n    double tmp = 0;\n    \n    const vertex_data &vdata = vertex.data();\n    \n    if (vdata.iowner)\n        tmp += vdata.vali;\n    if (vdata.jowner)\n        tmp += vdata.valj;\n    tmp += vdata.valij;\n\n    return tmp;\n}\n\nvoid finalize_fun(mplp_vertex_program::icontext_type& context, double total) \n{\n    if(context.procid() == 0) \n        std::cout << \"Dual value: \" << total << std::endl;\n}\n\n\n\n// Helper functions ===========================================================>\ngraphlab::vertex_id_type pixel_ind(size_t rows, size_t cols,\n                                   size_t r, size_t c) {\n    return r * cols + c;\n}; // end of pixel_ind\n\n\ngraphlab::vertex_id_type factor_ind(size_t rows, size_t cols,\n                                    size_t i, size_t j) {\n    if(i > j) std::swap(i,j);\n    return i * (rows * cols) + j;\n}; // end of factor_ind\n\ngraphlab::vertex_id_type factor_ind2(size_t rows, size_t cols,\n                                    size_t i, size_t j) {\n    if(i > j) std::swap(i,j);\n    \n    if (j == (i+1)) // horizontal edge\n        return i - std::floor(i/cols);\n    else if (j == (i+cols))\n        return rows*(cols-1) + i;\n    else\n      std::cout << \"Problem \";\n      //ASSERT_TRUE(false);\n    return 0;\n}; // end of factor_ind\n\n\nvoid create_synthetic_cluster_graph(graphlab::distributed_control& dc,\n                                    graph_type& graph,\n                                    const size_t rows, const size_t cols) {\n    dc.barrier();\n    // Generate the image on all machines --------------------------------------->\n    // Need to ensure that all machines generate the same noisy image\n    graphlab::random::generator gen; gen.seed(314);\n    std::vector    obs_pixels(rows * cols);\n    std::vector true_pixels(rows * cols);\n    const double center_r = rows / 2.0;\n    const double center_c = cols / 2.0;\n    const double max_radius = std::min(rows, cols) / 2.0;\n    for(size_t r = 0; r < rows; ++r) \n    {\n        for(size_t c = 0; c < cols; ++c) \n        {\n            // Compute the true pixel value\n            const double distance = sqrt((r-center_r)*(r-center_r) + \n                                         (c-center_c)*(c-center_c));\n            // Compute ring of sunset\n            const uint16_t ring_color =  \n            std::floor(std::min(1.0, distance/max_radius) * (NCOLORS - 1) );\n            // Compute the true pixel color by masking with the horizon\n            const uint16_t true_color = r < rows/2 ? ring_color : 0;\n            // compute the predicted color\n            const float obs_color = true_color + gen.normal(0, SIGMA);\n            // determine the true pixel id\n            const size_t pixel = pixel_ind(rows,cols,r,c);\n            true_pixels[pixel] = true_color; obs_pixels[pixel] = obs_color;\n        } // end of loop over cols\n    } // end of loop over rows\n    \n    if(dc.procid() == 0) \n    {\n        //int nedges = 2*rows*cols -rows-cols;\n        int nedges = factor_ind2(rows,cols,rows*cols-cols,rows*cols);\n        //vinit = vector::Zero(2*rows*cols -rows-cols);\n        //vapply = vector::Zero(2*rows*cols -rows-cols);\n        vinit.clear(); vinit.resize(nedges, 0);\n        vapply.clear(); vapply.resize(nedges,0);\n\n        int max_vid = 0; \n        \n        PRED_COLOR = vector::Zero(rows*cols);\n        int ownercount = 0; \n        \n        // temp code \n        std::ofstream ne, ee; \n        ne.open(\"./node_en.txt\");\n        //ee.open(\"./edge_en.txt\");\n        \n        ne << rows*cols << \" \" << NCOLORS << \" \"\n        << 0 << \" \" << rows << \" \" << cols << std::endl;\n        // end temp\n        \n        std::vector nbrs;\n        // load the graph\n        for(size_t r = 0; r < rows; ++r) \n        {\n            for(size_t c = 0; c < cols; ++c) \n            {\n                // temp code to write out potential to file:\n                std::vector potential(NCOLORS);\n                const double obs = obs_pixels[pixel_ind(rows,cols,r,c)];\n                const double sigmaSq = SIGMA*SIGMA;\n                double sum = 0;\n                for(int pred = 0; pred < potential.size(); ++pred) \n                {\n                    potential[pred] = +(obs - pred)*(obs - pred) / (2.0 * sigmaSq);\n                    sum += potential[pred];\n                }\n                //                for(int pred = 0; pred < potential.size(); ++pred)                 \n                //                    potential[pred] /= sum;\n                //                ne << true_pixels[pixel_ind(rows,cols,r,c)] << \" \" <<  obs << \" \" << potential << std::endl;\n                //                ne << 4 - int((r==0)||(r==(rows-1))) - int((c==0)||(c==(cols-1))) << \" \" << potential << std::endl;\n                ne << potential << std::endl;\n                // end temp\n                \n                // Add the two vertices (factors to the right and below this\n                // pixel)\n                if(r + 1 < rows) \n                {\n                    vertex_data vdata;\n                    vdata.i = pixel_ind(rows,cols,r,c);\n                    vdata.j = pixel_ind(rows,cols,r+1,c);\n                    vdata.deg_i = 4 - int((r==0)||(r==(rows-1))) - int((c==0)||(c==(cols-1)));\n                    vdata.deg_j = 4 - int(((r+1)==0)||((r+1)==(rows-1))) - int((c==0)||(c==(cols-1)));\n                    vdata.obs_color_i = obs_pixels[vdata.i];\n                    vdata.obs_color_j = obs_pixels[vdata.j];\n                    graph.add_vertex(factor_ind2(rows,cols,vdata.i,vdata.j), vdata);\n                    \n                    // temp code\n                    max_vid = std::max((int)factor_ind2(rows,cols,vdata.i,vdata.j), max_vid); \n                }\n                if(c + 1 < cols) \n                {\n                    vertex_data vdata;\n                    vdata.i = pixel_ind(rows,cols,r,c);\n                    vdata.j = pixel_ind(rows,cols,r,c+1);\n                    vdata.deg_i = 4 - int((r==0)||(r==(rows-1))) - int((c==0)||(c==(cols-1)));\n                    vdata.deg_j = 4 - int((r==0)||(r==(rows-1))) - int(((c+1)==0)||((c+1)==(cols-1)));\n                    vdata.obs_color_i = obs_pixels[vdata.i];\n                    vdata.obs_color_j = obs_pixels[vdata.j];\n\n                    vdata.iowner = true; // give i-ownership to horizontal edges\n                    ++ownercount;\n                    if ((c+1)==(cols-1)) // and j-ownership too if last node in this row\n                    {\n                        vdata.jowner = true;\n                        ++ownercount;\n                    }\n                    graph.add_vertex(factor_ind2(rows,cols,vdata.i,vdata.j), vdata);\n                    \n                    // temp code\n                    max_vid = std::max((int)factor_ind2(rows,cols,vdata.i,vdata.j), max_vid); \n                }\n                // Compute all the factors that contain this pixel\n                nbrs.clear();\n                if(r+1 < rows)\n                    nbrs.push_back(factor_ind2(rows,cols,\n                                              pixel_ind(rows,cols,r,c),\n                                              pixel_ind(rows,cols,r+1,c)));\n                if(r-1 < rows)\n                    //if(r-1 >= 0)\n                    nbrs.push_back(factor_ind2(rows,cols,\n                                              pixel_ind(rows,cols,r-1,c),\n                                              pixel_ind(rows,cols,r,c)));\n                if(c+1 < cols)\n                    nbrs.push_back(factor_ind2(rows,cols,\n                                              pixel_ind(rows,cols,r,c),\n                                              pixel_ind(rows,cols,r,c+1)));\n                if(c-1 < cols)\n                    //if(c-1 >= 0)\n                    nbrs.push_back(factor_ind2(rows,cols,\n                                              pixel_ind(rows,cols,r,c-1),\n                                              pixel_ind(rows,cols,r,c)));\n                // construct the clique over the factors\n                for(size_t i = 0; i < nbrs.size(); ++i) \n                {\n                    for(size_t j = i+1; j < nbrs.size(); ++j) \n                    {\n                        graph.add_edge(nbrs[i], nbrs[j]);\n                    }\n                }\n            } // end of for cols\n        } // end of for rows\n        \n        // temp code\n        ne.close(); //ee.close();\n        std::cout << \"Max vid fed into graphlab: \" << max_vid << \"\\n\";\n        std::cout << \"No. of owners: \" << ownercount << \"\\n\";\n    } // end of if proc 0\n    dc.barrier();\n} // end of create synthetic cluster graph\n\n\n\n\n\nvoid initialize_theta_ij(const std::string& smoothing,\n                         const double lambda) \n{\n    THETA_ij.resize(NCOLORS, NCOLORS);\n    // Set the smoothing type\n    if(smoothing == \"laplace\") \n    {\n        for(int i = 0; i < THETA_ij.rows(); ++i) \n            for(int j = 0; j < THETA_ij.cols(); ++j) \n                THETA_ij(i,j) = -std::abs(double(i) - double(j)) * lambda;\n    } \n    else \n    {   \n        for(int i = 0; i < THETA_ij.rows(); ++i) \n            for(int j = 0; j < THETA_ij.cols(); ++j) \n                THETA_ij(i,j) = -(i == j? 0 : lambda);\n    } \n} // end of initialize_theta_ij\n\n\n\ntemplate\nstruct merge_reduce {\n    std::set values;\n    void save(graphlab::oarchive& arc) const { arc << values; }\n    void load(graphlab::iarchive& arc) { arc >> values; }\n    merge_reduce& operator+=(const merge_reduce& other) {\n        values.insert(other.values.begin(), other.values.end());\n        return *this;\n    }\n}; // end of merge_reduce\n\ntypedef std::pair pred_pair_type; \ntypedef merge_reduce merge_reduce_type;\n\nmerge_reduce_type pred_map_function(graph_type::vertex_type vertex) {\n    merge_reduce ret;\n    ret.values.insert(pred_pair_type(vertex.data().i, vertex.data().pred_color_i));\n    ret.values.insert(pred_pair_type(vertex.data().j, vertex.data().pred_color_j));\n    return ret;\n} // end of pred_map_function\n\nmerge_reduce_type obs_map_function(graph_type::vertex_type vertex) {\n    merge_reduce ret;\n    ret.values.insert(pred_pair_type(vertex.data().i, vertex.data().obs_color_i));\n    ret.values.insert(pred_pair_type(vertex.data().j, vertex.data().obs_color_j));\n    return ret;\n} // end of obs_map_function\n\n\n\n\nstd::pair ind2sub(size_t rows, size_t cols,\n                           size_t ind) {\n    return std::make_pair(ind / cols, ind % cols);\n}; // end of sub2ind\n\n\n// /**\n//  * Saving an image as a pgm file.\n//  */\n// void save_image(const size_t rows, const size_t cols,\n//                 const std::set& values,\n//                 const std::string& fname) {\n//     std::cout << \"NPixels: \" << values.size() << std::endl;\n//     image img(rows, cols);\n//     foreach(pred_pair_type pair, values) \n//     img.pixel(pair.first) = pair.second;\n//     img.save(fname);\n// } // end of save_image\n\n\n/**\n * Saving an image as a pgm file.\n */\n\nvoid save_image(const size_t rows, const size_t cols,\n                const std::set& values,\n                const std::string& fname) {\n  std::cout << \"NPixels: \" << values.size() << std::endl;\n  // determine the max and min colors\n  float max_color = -std::numeric_limits::max();\n  float min_color =  std::numeric_limits::max();\n  foreach(pred_pair_type pair, values) {\n    max_color = std::max(max_color, pair.second);\n    min_color = std::min(min_color, pair.second);\n  }\n\n  cv::Mat img(cols, rows, CV_8UC1);\n  foreach(pred_pair_type pair, values) {\n    std::pair coords = ind2sub(rows,cols, pair.first);\n    float value = (pair.second - min_color) / (max_color - min_color);\n    int color = 255 * value > 255 ? 255 : 255 * value;\n    img.at(coords.first, coords.second) = color;\n  }\n  cv::imwrite(fname, img);\n}\n\n\n\n\n// MAIN =======================================================================>\nint main(int argc, char** argv) {\n    std::cout << \"This program creates and denoises a synthetic \" << std::endl\n    << \"image using loopy belief propagation inside \" << std::endl\n    << \"the graphlab framework.\" << std::endl;\n    \n    // // set the global logger\n    // global_logger().set_log_level(LOG_WARNING);\n    // global_logger().set_log_to_console(true);\n    \n    // Set initial values for members ------------------------------------------->\n    NCOLORS = 5;\n    SIGMA = 2;\n    BOUND = 1E-4;\n    \n    \n//    size_t nrows = 200;\n//    size_t ncols = 200;\n    size_t nrows = 20;\n    size_t ncols = 20;\n    double lambda = 0.2;\n    \n    std::string smoothing = \"square\";\n    \n    std::string orig_fn =  \"source_img.jpeg\";\n    std::string noisy_fn = \"noisy_img.jpeg\";\n    std::string pred_fn = \"pred_img.jpeg\";\n    \n    // std::string orig_fn =  \"source_img.pgm\";\n    // std::string noisy_fn = \"noisy_img.pgm\";\n    // std::string pred_fn = \"pred_img.pgm\";\n    \n    \n    \n    // Parse command line arguments --------------------------------------------->\n    graphlab::command_line_options clopts(\"Loopy BP image denoising\");\n    clopts.attach_option(\"bound\", BOUND,\n                         \"Residual termination bound\");\n    clopts.attach_option(\"ncolors\", NCOLORS,\n                         \"The number of colors in the noisy image\");\n    clopts.attach_option(\"sigma\", SIGMA,\n                         \"Standard deviation of noise.\");\n    clopts.attach_option(\"nrows\", nrows,\n                         \"The number of rows in the noisy image\");\n    clopts.attach_option(\"ncols\", ncols,\n                         \"The number of columns in the noisy image\");\n    clopts.attach_option(\"lambda\", lambda,\n                         \"Smoothness parameter (larger => smoother).\");\n    clopts.attach_option(\"smoothing\", smoothing,\n                         \"Options are {square, laplace}\");\n    clopts.attach_option(\"orig\", orig_fn,\n                         \"Original image file name.\");\n    clopts.attach_option(\"noisy\", noisy_fn,\n                         \"Noisy image file name.\");\n    clopts.attach_option(\"pred\", pred_fn,\n                         \"Predicted image file name.\");\n    \n    ///! Initialize control plain using mpi\n    graphlab::mpi_tools::init(argc, argv);\n    const bool success = clopts.parse(argc, argv);\n    if(!success) {\n        clopts.print_description();\n        graphlab::mpi_tools::finalize();\n        return EXIT_FAILURE;\n    }\n    \n    ///! Create a distributed control object \n    graphlab::distributed_control dc;\n    ///! display settings  \n    if(dc.procid() == 0) {\n        std::cout << \"ncpus:          \" << clopts.get_ncpus() << std::endl\n        << \"bound:          \" << BOUND << std::endl\n        << \"colors:         \" << NCOLORS << std::endl\n        << \"nrows:           \" << nrows << std::endl\n        << \"ncols:           \" << ncols << std::endl\n        << \"sigma:          \" << SIGMA << std::endl\n        << \"lambda:         \" << lambda << std::endl\n        << \"smoothing:      \" << smoothing << std::endl\n        << \"scheduler:      \" << clopts.get_scheduler_type() << std::endl\n        << \"orig_fn:        \" << orig_fn << std::endl\n        << \"noisy_fn:       \" << noisy_fn << std::endl\n        << \"pred_fn:        \" << pred_fn << std::endl;\n    }\n    \n    \n    \n    \n    // Create synthetic images -------------------------------------------------->\n    std::cout << \"Creating a synthetic noisy image.\" << std::endl;\n    graph_type graph(dc, clopts);\n    create_synthetic_cluster_graph(dc, graph, nrows, ncols);\n    std::cout << \"Finalizing the graph.\" << std::endl;\n    graph.finalize();\n    \n    std::cout << \"Collect the noisy image. \" << std::endl;\n    merge_reduce_type obs_image = \n    graph.map_reduce_vertices(obs_map_function);\n    std::cout << \"saving the noisy image.\" << std::endl;\n    if(dc.procid() == 0) {\n        save_image(nrows, ncols, obs_image.values, noisy_fn);\n    }\n    \n    // Initialze the edge factor ----------------------------------------------->\n    std::cout << \"Initializing shared edge factor. \" << std::endl;\n    // dummy variables 0 and 1 and num_rings by num_rings\n    initialize_theta_ij(smoothing, lambda);\n    if(dc.procid() == 0) std::cout << THETA_ij << std::endl;\n    \n    // Create the engine -------------------------------------------------------->\n    std::cout << \"Creating the engine. \" << std::endl;\n    engine_type engine(dc, graph, clopts);\n\n    engine.add_vertex_aggregator(\"energy\", get_energy_fun, finalize_fun);\n    engine.aggregate_periodic(\"energy\", 3); // run every 3 seconds\n\n    engine.transform_vertices(mplp_vertex_program::init_vertex_data);\n\n    std::cout << \"Scheduling all vertices\" << std::endl;\n    engine.signal_all();\n    std::cout << \"Starting the engine\" << std::endl;\n    engine.start();\n    const float runtime = engine.elapsed_seconds();\n    size_t update_count = engine.num_updates();\n    std::cout << \"Finished Running engine in \" << runtime \n    << \" seconds.\" << std::endl\n    << \"Total updates: \" << update_count << std::endl\n    << \"Efficiency: \" << (double(update_count) / runtime)\n    << \" updates per second \"\n    << std::endl;  \n    \n    \n    // Saving the output -------------------------------------------------------->\n    std::cout << \"Saving the predicted image\" << std::endl;\n    std::cout << \"Collect the noisy image. \" << std::endl;\n    merge_reduce_type pred_image = \n    graph.map_reduce_vertices(pred_map_function);\n    std::cout << \"saving the pred image.\" << std::endl;\n    if(dc.procid() == 0) {\n        save_image(nrows, ncols, pred_image.values, pred_fn);\n    }\n    \n    std::cout << \"Done!\" << std::endl;\n    graphlab::mpi_tools::finalize();\n    return EXIT_SUCCESS;\n} // End of main\n\n\n\n"
  },
  {
    "path": "toolkits/graphical_models/mplp_structured_prediction.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n *\n * \\brief This application used for structured prediction on a graph.\n * For example, suppose you want to model the interests of users in a\n * social network.  \n *\n * Overview and Usage\n * ======================\n *\n * For simplicity lets suppose you want to know the users interest in\n * the categories movies, sports, and music.  After analyzing each\n * users profile you might be able to estimate a crude distribution\n * over her interests.  However you would like to leverage similarity\n * among friends to improve your estimates.  This application is\n * designed to do exactly that.\n *\n * As an input you provide two folders (or files) the first contains\n * the prior probabilities for each vertex in the form:\n *\n *    \\t  \\t  ... \\n\n *\n * For example:\n *   \n *   1    0.2   0.2   0.6\n *   2    0.3   0.6   0.1\n *   3    0.3   0.3   0.4\n *           ... \n * \n * The second folder contains the graph structure in the form:\n *\n *    \\t  \\t [Optional Weight]\n *\n * For example:\n *\n *   1   2\n *   1   3  1.7\n *   3   2  0.3\n *\n * The default weight value is 1 (times the smoothing parameter passed\n * in as a command line argument).  Larger weight values imply\n * stronger relationships.  A negative weight implies a \"repulsive\"\n * relationship in which neighboring vertices would like to have\n * different assignments.\n *\n * We have provided a synthetic data generator which creates a\n * synthetic dataset for an simulated image denoising task. See the\n * synthetic_image_data application for details.\n *\n * As output the application produces another set files with a format\n * identical to the vertex prior file with each weight (probability)\n * corresponding to the posterior predictions.\n *\n * Technical Explanation\n * ========================\n *\n * This application creates a pair-wise Markov Random Field with\n * Ising-Potts edge factors and then uses residual loopy belief\n * propagation to compute posterior belief estimates for each vertex.\n *\n *\n *  \\author Dhruv Batra \n */\n\n\n#include \n#include \n#include \n#include \n#include \n\n\n#include \n#include \"eigen_serialization.hpp\"\n\n\n\n#include \n#include \n\n\n\n\n\ntypedef Eigen::VectorXd vector;\ntypedef Eigen::MatrixXd matrix;\n\n\n/**\n * \\brief The Ising smoothing parameter which controls the coupling\n * between adjacent predictions in the graph.  Larger values imply\n * greater smoothing (stronger coupling). \n *\n * \\code\n * edge_factor(xi, xj) = exp( (xi == xj)? 0 : -SMOOTHING * edge_weight ); \n * \\endcode\n *\n * Not that the default edge weight is 1 however the graph file can\n * contain an additional edge weight column which allows per edge\n * control of the smoothing parameter.\n *\n * This parameter is set as a command line argument.\n */\ndouble SMOOTHING = 2;\n\n\n/**\n * \\brief The convergence threshold for each message.  Smaller values\n * imply tighter convergence but slower execution.\n *\n *\n * The algorithm convergence when:\n *   \n * \\code\n * sum(abs(log(old_message) - log(new_message))) < TOLERANCE\n * \\endcode\n *\n * The parameter is set as a command line argument\n */\ndouble TOLERANCE = 0.01;\n\n\n/**\n * \\brief The vertex data contains the vertex potential as well as the\n * current belief estimate and represents a random variable in the\n * Markov Random Field.\n *\n * The vertex potential represents the prior and is obtained from the\n * vertex prior file (stored in log form).\n *\n * The belief represents the current posterior estimate.\n */\nstruct vertex_data {\n  factor_type belief;\n  factor_type potential;\n  void load(graphlab::iarchive& arc) { arc >> belief >> potential; }\n  void save(graphlab::oarchive& arc) const { arc << belief << potential; }\n}; // end of vertex_data\n\n\n/**\n * \\brief The edge data represents an edge in the Markov Random Field\n * and contains the loopy belief propagation message in both\n * directions along that edge as well as the old message in each\n * direction.  In addition each edge contains the weight parameter\n * used to set edge specific smoothing (default value is 1).\n */\nclass edge_data {\n  /**\n   * \\brief We store old and new messages in both directions as an\n   * array of messages.  The particular message index is then computed\n   * using the \\ref message_idx function.\n   */\n  factor_type messages_[4];\n  /**\n   * \\brief The weight associated with the edge (used to scale the\n   * smoothing parameter)\n   */\n  double weight_;\n  /**\n   * \\brief The function used to compute the message index in the edge\n   * message array.\n   */\n  size_t message_idx(size_t source_id, size_t target_id, bool is_new) {\n    return size_t(source_id < target_id)  + 2 * size_t(is_new);\n  }\n\npublic:\n\n  edge_data(const double w = 1) : weight_(w) { }\n  const double& weight() const { return weight_; }\n\n  /**\n   * \\brief Get the new message value from source_id to target_id\n   */\n  factor_type& message(size_t source_id, size_t target_id) { \n    return messages_[message_idx(source_id, target_id, true)];\n  }\n  /**\n   * \\brief Get the old message value from source_id to target_id\n   */\n  factor_type& old_message(size_t source_id, size_t target_id) { \n     return messages_[message_idx(source_id, target_id, false)];\n  }\n\n  /**\n   * \\brief Set the old message value equal to the new message value\n   */\n  void update_old(size_t source_id, size_t target_id) { \n    old_message(source_id, target_id) = message(source_id, target_id);\n  }\n  \n  /**\n   * \\brief Initialize the edge data with source and target having the\n   * appropriate number of states.\n   *\n   * \\param source_id the vertex id of the source\n   * \\param nsource the number of states the source vertex takes\n   * \\param target_id the vertex id of the target\n   * \\param ntarget the number of states the target vertex takes\n   */\n  void initialize(size_t source_id, size_t nsource, size_t target_id, size_t ntarget) {\n    ASSERT_GT(nsource, 0); ASSERT_GT(ntarget, 0);\n    message(source_id, target_id).resize(ntarget);\n    old_message(source_id, target_id).resize(ntarget);\n    message(target_id, source_id).resize(nsource);\n    old_message(target_id, source_id).resize(nsource);\n  }\n  void save(graphlab::oarchive& arc) const {\n    for(size_t i = 0; i < 4; ++i) arc << messages_[i];\n    arc << weight_;\n  }\n  void load(graphlab::iarchive& arc) {\n    for(size_t i = 0; i < 4; ++i) arc >> messages_[i];\n    arc >> weight_;\n  }\n}; // End of edge data\n\n\n\n/**\n * \\brief The graph type used to store the Markov Random Field with\n * vertex data containing node potentials and beliefs and edge data\n * containing messages and weights.\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n\n/** \n * \\brief The Loopy Belief Propagation Vertex Program which computes\n * the product of the inbound messages during the gather phase,\n * updates the belief during the apply phase, and then computes the\n * new out-bound messages during the scatter phase.\n *\n * Since the gather phase is computing the product of the inbound\n * messages and the messages are stored in log form the resulting sum\n * operation is actually a vector sum and so the gather type is simply\n * the factor type and the operator+= operation for the factor type is\n * sufficient.\n *\n */\nstruct bp_vertex_program : \n  public graphlab::ivertex_program< graph_type, factor_type,\n                                    graphlab::messages::sum_priority >,\n  public graphlab::IS_POD_TYPE {\n\n  /**\n   * \\brief Since the MRF is undirected we will use all edges for gather and\n   * scatter\n   */\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of gather_edges \n\n  /**\n   * \\brief Update the old message to be the new message and collect the\n   * message value.\n   */\n  factor_type gather(icontext_type& context, const vertex_type& vertex, \n                     edge_type& edge) const {\n    const vertex_type other_vertex = get_other_vertex(edge, vertex);\n    edge_data& edata = edge.data();\n    // Update the old message with the value of the new Message.  We\n    // then receive the old message during gather and then compute the\n    // \"cavity\" during scatter (again using the old message).\n    edata.update_old(other_vertex.id(), vertex.id());\n    const factor_type& recv_message = \n      edata.old_message(other_vertex.id(), vertex.id());\n    // Ensure that the received message has the correct size\n    ASSERT_EQ(recv_message.size(), vertex.data().potential.size());\n    return recv_message;\n  }; // end of gather function\n\n  /**\n   * \\brief Multiply message product by node potential and update the\n   * belief.\n   */\n  void apply(icontext_type& context, vertex_type& vertex, \n             const factor_type& total) {\n    // If we have no neighbors than the belief is equal to the\n    // potential so simply update the belief\n    if(vertex.num_in_edges() + vertex.num_out_edges() == 0) {\n      vertex.data().belief = vertex.data().potential;\n    } else {\n      vertex_data& vdata = vertex.data();\n      ASSERT_EQ(vdata.potential.size(), total.size());\n      // Multiply (add in log space) the potential to compute the belief\n      vdata.belief = vdata.potential + total;\n      ASSERT_GT(vdata.belief.size(), 0);\n      // Rescale the belief to ensure numerical stability.  (This is\n      // essentially normalization in log-space.)\n      vdata.belief.array() -= vdata.belief.maxCoeff();\n    }\n  }; // end of apply\n\n  /**\n   * \\brief Since the MRF is undirected we will use all edges for gather and\n   * scatter\n   */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of scatter edges\n\n  /**\n   * \\brief Compute new message value for each edge.\n   */\n  void scatter(icontext_type& context, const vertex_type& vertex, \n               edge_type& edge) const {  \n    const vertex_type other_vertex = get_other_vertex(edge, vertex);\n    edge_data& edata = edge.data();\n    // Divide (subtract in log space) out of the belief the old in\n    // message to construct the cavity\n    const factor_type& old_in_message = \n      edata.old_message(other_vertex.id(), vertex.id());\n    ASSERT_EQ(old_in_message.size(), vertex.data().belief.size());\n    factor_type cavity = vertex.data().belief - old_in_message;\n    // compute the new message by convolving with the Ising-Potts Edge\n    // factor.\n    factor_type& new_out_message = \n      edata.message(vertex.id(), other_vertex.id());\n    const factor_type& old_out_message = \n      edata.old_message(vertex.id(), other_vertex.id());\n    convolve(cavity, edata.weight(), new_out_message);\n    // Renormalize (done in log space)\n    new_out_message.array() -= new_out_message.maxCoeff();\n    // Apply damping to the message to stabilize convergence.\n    new_out_message = DAMPING * old_out_message + \n      (1-DAMPING) * new_out_message;\n    // Compute message residual\n    const double residual = \n      (new_out_message - old_out_message).cwiseAbs().sum();\n    context.clear_gather_cache(other_vertex);\n    // Schedule the adjacent vertex\n    if(residual > TOLERANCE) context.signal(other_vertex, residual);\n }; // end of scatter\n\nprivate:\n\n  /**\n   * \\brief Compute the convolution of the cavity with the Ising-Potts\n   * edge potential and store the result in the message\n   *\n   * \\param cavity the belief minus the in-bound message\n   * \\param weight the edge weight used to scale the smoothing parameter\n   * \\param [out] message The message in which to store the result of\n   * the convolution.\n   */\n  inline void convolve(const factor_type& cavity, const double& weight, \n                       factor_type& message) const {\n    for(size_t i = 0; i < message.size(); ++i) {\n      double sum = 0;\n      for(size_t j = 0; j < cavity.size(); ++j) {\n        sum += std::exp( cavity(j)  + ( i == j? 0 : -(SMOOTHING*weight) ) ); \n      }\n      // To try and ensure numerical stability we do not allow\n      // messages to underflow in log-space\n      message(i) = (sum > 0)? std::log(sum) : \n        std::numeric_limits::min();\n    }\n  } // end of convolve\n  \n  /**\n   * \\brief Given an edge and a vertex return the other vertex along\n   * that edge. \n   */\n  inline vertex_type get_other_vertex(edge_type& edge, \n                                      const vertex_type& vertex) const {\n    return vertex.id() == edge.source().id()? edge.target() : edge.source();\n  }; // end of other_vertex\n\n}; // end of class bp_vertex_program\n\n\n\n\n\n\n\n/**\n * \\brief The vertex load is used by the graph loading API to parse\n * the lines of prior data in the vertex data file.\n *\n * This parser uses the boost::spirit library to parse the vertex data\n * file. As a consequence it is fairly flexible allowing both comma\n * and tab delimited files as well as vertices with different numbers\n * of states.\n */\nbool vertex_loader(graph_type& graph, const std::string& fname, \n                   const std::string& line) {\n  // If the line is empty simply skip it\n  if(line.empty()) return true;\n  // We use the boost spirit parser which requires (too) many separate\n  // namespaces so to make things clear we shorten them here.\n  namespace qi = boost::spirit::qi;\n  namespace ascii = boost::spirit::ascii;\n  namespace phoenix = boost::phoenix;\n  graphlab::vertex_id_type vid(-1);\n  std::vector values;\n  const bool success = qi::phrase_parse\n    (line.begin(), line.end(),       \n     //  Begin grammar\n     (\n      qi::ulong_[phoenix::ref(vid) = qi::_1] >> -qi::char_(\",\") >>\n      (qi::double_[phoenix::push_back(phoenix::ref(values), qi::_1)] % -qi::char_(\",\") )\n      )\n     ,\n     //  End grammar\n     ascii::space); \n  // Test to see if the boost parser was able to parse the line\n  if(!success) {\n    logstream(LOG_ERROR) << \"Parse error in vertex prior parser.\" << std::endl;\n    return false;\n  }\n\n  // Ensure that a prior was provided.  Technically this should not be\n  // reached since the parser requires at least one prior entry\n  if(values.empty()) {\n    logstream(LOG_ERROR) << \"Vertex has no prior.\" << std::endl;\n    return false;\n  }\n\n  // Renormalize the vertex data. We require positive probabilities.\n  double sum = 0;\n  for(size_t i = 0; i < values.size(); ++i) {\n    if(values[i] < 0) { \n      logstream(LOG_ERROR) << \"Encountered negative probability.\" << std::endl;\n      return false;\n    }\n    if(values[i] == 0) { \n      logstream(LOG_ERROR) \n        << \"Zero probability assignments are not currently supported.\" << std::endl;\n      return false;\n    }\n    sum += values[i]; \n  }\n  ASSERT_GT(sum, 0);\n  for(size_t i = 0; i < values.size(); ++i) values[i] /= sum;\n\n  vertex_data vdata;\n  vdata.potential.resize(values.size());\n  for(size_t i = 0; i < values.size(); ++i) {\n    ASSERT_GT(values[i], 0);\n    vdata.potential(i) = std::log(values[i]);\n  }\n  graph.add_vertex(vid, vdata);\n  return true;\n} // end of vertex_loader;\n\n\n/**\n * \\brief The edge data loader is used by the GraphLab graph loading\n * API to parse lines in the edge data file. \n */\nbool edge_loader(graph_type& graph, const std::string& fname, \n                 const std::string& line) {\n  ASSERT_FALSE(line.empty()); \n  namespace qi = boost::spirit::qi;\n  namespace ascii = boost::spirit::ascii;\n  namespace phoenix = boost::phoenix;\n  graphlab::vertex_id_type source(-1), target(-1);\n  double weight = 1;\n  const bool success = qi::phrase_parse\n    (line.begin(), line.end(),       \n     //  Begin grammar\n     (\n      qi::ulong_[phoenix::ref(source) = qi::_1] >>  -qi::char_(',') \n      >> qi::ulong_[phoenix::ref(target) = qi::_1] >>  \n      -(-qi::char_(',') >> qi::double_[phoenix::ref(weight) = qi::_1])\n      )\n     ,\n     //  End grammar\n     ascii::space); \n  if(!success) return false;  \n  graph.add_edge(source, target, edge_data(weight));\n  return true;\n} // end of edge loader\n\n\n\n/**\n * \\brief The edge initializer is used to allocate the messages along\n * each edge based on the number of states of the source and target\n * vertex.\n */\nvoid edge_initializer(graph_type::edge_type& edge) {\n  edge_data& edata = edge.data();\n  const graphlab::vertex_id_type source_id = edge.source().id();\n  const size_t nsource = edge.source().data().potential.size(); \n  const graphlab::vertex_id_type target_id = edge.target().id();\n  const size_t ntarget = edge.target().data().potential.size();\n  edata.initialize(source_id, nsource, target_id, ntarget);\n} // end of edge initializer\n\n\n\n\n/**\n * \\brief The belief prediction saver is used to save the belief\n * predictions for each vertex.\n */\nstruct belief_prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    std::stringstream strm;\n    strm << vertex.id() << '\\t';\n    factor_type pred = vertex.data().belief;\n    double sum = 0;\n    for(size_t i = 0; i < pred.size(); ++i) \n      sum += (pred(i) = std::exp(pred(i)));\n    pred.array() /= sum;\n    for(size_t i = 0; i < pred.size(); ++i) \n      strm << pred(i) << (i+1 < pred.size()? '\\t' : '\\n');\n    return strm.str();\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\"; // nop\n  }\n}; // end of belief_prediction_saver\n\n\n/**\n * \\brief The MAP prediction saver is used to save the map estimated\n * for each vertex.  The MAP estimate is the most likely assignment\n */\nstruct map_prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    std::stringstream strm;\n    size_t prediction = 0;\n    vertex.data().belief.maxCoeff(&prediction);\n    strm << vertex.id() << '\\t' << prediction << '\\n';\n    return strm.str();\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\"; // nop\n  }\n}; // end of map prediction_saver\n\n\n\n\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  // Parse command line options -----------------------------------------------\n  // \\todo update description string\n  const std::string description = \"Structure prediction solver\";\n  graphlab::command_line_options clopts(description);\n  std::string prior_dir; \n  std::string graph_dir;\n  std::string output_dir = \"pred\";\n  bool map = false;\n  clopts.attach_option(\"prior\", &prior_dir, prior_dir,\n                       \"The directory containing the prior\");\n  clopts.add_positional(\"prior\");\n  clopts.attach_option(\"graph\", &graph_dir, graph_dir,\n                       \"The directory containing the adjacency graph\");\n  clopts.add_positional(\"graph\");\n  clopts.attach_option(\"smoothing\", &SMOOTHING, SMOOTHING,\n                       \"The amount of smoothing (larger = more)\");\n  clopts.attach_option(\"damping\", &DAMPING, DAMPING,\n                       \"The amount of damping (0 -> no damping and 1 -> no progress)\");\n  clopts.attach_option(\"tol\", &TOLERANCE, TOLERANCE,\n                       \"The tolerance level for convergence.\");\n  clopts.attach_option(\"output\", &output_dir, output_dir,\n                       \"The directory in which to save the predictions\");\n  clopts.attach_option(\"map\", &map, map,\n                       \"Return maximizing assignment instead of the posterior distribution.\");\n  if(!clopts.parse(argc, argv)) {\n    graphlab::mpi_tools::finalize();\n    return clopts.is_set(\"help\")? EXIT_SUCCESS : EXIT_FAILURE;\n  }\n\n  if(prior_dir.empty()) {\n    logstream(LOG_ERROR) << \"No prior was provided.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  if(graph_dir.empty()) {\n    logstream(LOG_ERROR) << \"No graph was provided.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  // Start the webserver\n  graphlab::launch_metric_server();\n\n  ///! load the graph\n  graph_type graph(dc, clopts);  \n\n\n  ///! load the graph\n  graph.load(prior_dir, vertex_loader);\n  graph.load(graph_dir, edge_loader);\n  graph.finalize();\n  graph.transform_edges(edge_initializer);\n\n  typedef graphlab::omni_engine engine_type;\n  engine_type engine(dc, graph, clopts, \"asynchronous\");\n  engine.signal_all();\n  graphlab::timer timer;\n  engine.start();  \n  const double runtime = timer.current_time();\n    dc.cout() \n    << \"----------------------------------------------------------\" << std::endl\n    << \"Final Runtime (seconds):   \" << runtime \n    << std::endl\n    << \"Updates executed: \" << engine.num_updates() << std::endl\n    << \"Update Rate (updates/second): \" \n    << engine.num_updates() / runtime << std::endl;\n    \n    \n  std::cout << \"Saving predictions\" << std::endl;\n  const bool gzip_output = false;\n  const bool save_vertices = true;\n  const bool save_edges = false;\n  const size_t threads_per_machine = 2;\n  if(map) {\n    graph.save(output_dir, map_prediction_saver(),\n               gzip_output, save_vertices, \n               save_edges, threads_per_machine);\n  } else { \n    graph.save(output_dir, belief_prediction_saver(),\n               gzip_output, save_vertices, \n               save_edges, threads_per_machine);\n  }\n\n\n    \n  //  graphlab::stop_metric_server_on_eof();\n  graphlab::stop_metric_server();\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n\n\n} // end of main\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
  },
  {
    "path": "toolkits/graphical_models/profile_lbp_synthetic.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n *\n * \\brief This application is almost identical to LBP structured\n * prediction except that it generates an artificial field for every\n * vertex enabling the studying of distributed LBP on various graph\n * structures without the need for actual data.  \n *\n *\n * Technical Explanation\n * ========================\n *\n * This application creates a pair-wise Markov Random Field with\n * Ising-Potts edge factors and then uses residual loopy belief\n * propagation to compute posterior belief estimates for each vertex.\n *\n *\n *  \\author Joseph Gonzalez\n */\n\n\n#include \n#include \n#include \n#include \n#include \n\n\n#include \n#include \"eigen_serialization.hpp\"\n\n\n\n#include \n#include \n\n\n\n\n/**\n * \\brief Eigen library vectors are used to store factor in _LOG\n * SPACE_.\n */\ntypedef Eigen::VectorXd factor_type;\n\n/**\n * \\brief The Ising smoothing parameter which controls the coupling\n * between adjacent predictions in the graph.  Larger values imply\n * greater smoothing (stronger coupling). \n *\n * \\code\n * edge_factor(xi, xj) = exp( (xi == xj)? 0 : -SMOOTHING * edge_weight ); \n * \\endcode\n *\n * Not that the default edge weight is 1 however the graph file can\n * contain an additional edge weight column which allows per edge\n * control of the smoothing parameter.\n *\n * This parameter is set as a command line argument.\n */\ndouble SMOOTHING = 2;\n\n\ndouble FIELD = 2;\nsize_t NSTATES = 5;\n\n/**\n * \\brief The Damping parameter which helps ensure stable convergence.\n * Larger damping values lead to slower but more stable convergence.\n *\n * Currently damping is implemented in log-space in the following\n * equation:\n *\n * \\code\n * log(new_message) = DAMPING * log(old_message) + \n *                         (1-DAMPING) * log(new_message);\n * \\endcode\n *\n * This parameter is set as a command line argument.\n */\ndouble DAMPING = 0.1;\n\n/**\n * \\brief The convergence threshold for each message.  Smaller values\n * imply tighter convergence but slower execution.\n *\n *\n * The algorithm convergence when:\n *   \n * \\code\n * sum(abs(log(old_message) - log(new_message))) < TOLERANCE\n * \\endcode\n *\n * The parameter is set as a command line argument\n */\ndouble TOLERANCE = 0.01;\n\n\n\nbool USE_CACHE = false;\n\n\n/**\n * Make a synthetic node potential\n */\nfactor_type make_node_potential(size_t vid) {\n  // const size_t obs = vid % NSTATES;\n  const size_t obs = 0;\n  factor_type factor;\n  factor.setZero(NSTATES);\n  if(vid % 101 < 1) {\n    for(size_t i = 0; i <  NSTATES; ++i) {\n      factor(i) = (i == obs)? 0 : -FIELD;\n    }\n  } \n  return factor;\n}\n\n/**\n * \\brief The vertex data contains the vertex potential as well as the\n * current belief estimate and represents a random variable in the\n * Markov Random Field.\n *\n * The vertex potential represents the prior and is obtained from the\n * vertex prior file (stored in log form).\n *\n * The belief represents the current posterior estimate.\n */\nstruct vertex_data {\n  factor_type belief;\n  void load(graphlab::iarchive& arc) { arc >> belief; }\n  void save(graphlab::oarchive& arc) const { arc << belief; }\n}; // end of vertex_data\n\n\n/**\n * \\brief The edge data represents an edge in the Markov Random Field\n * and contains the loopy belief propagation message in both\n * directions along that edge as well as the old message in each\n * direction.  In addition each edge contains the weight parameter\n * used to set edge specific smoothing (default value is 1).\n */\nclass edge_data {\n  /**\n   * \\brief We store old and new messages in both directions as an\n   * array of messages.  The particular message index is then computed\n   * using the \\ref message_idx function.\n   */\n  factor_type messages_[4];\n  /**\n   * \\brief The weight associated with the edge (used to scale the\n   * smoothing parameter)\n   */\n  double weight_;\n  /**\n   * \\brief The function used to compute the message index in the edge\n   * message array.\n   */\n  size_t message_idx(size_t source_id, size_t target_id, bool is_new) {\n    return size_t(source_id < target_id)  + 2 * size_t(is_new);\n  }\n\npublic:\n\n  edge_data(const double w = 1) : weight_(w) { }\n  const double& weight() const { return weight_; }\n\n  /**\n   * \\brief Get the new message value from source_id to target_id\n   */\n  factor_type& message(size_t source_id, size_t target_id) { \n    return messages_[message_idx(source_id, target_id, true)];\n  }\n  /**\n   * \\brief Get the old message value from source_id to target_id\n   */\n  factor_type& old_message(size_t source_id, size_t target_id) { \n     return messages_[message_idx(source_id, target_id, false)];\n  }\n\n  /**\n   * \\brief Set the old message value equal to the new message value\n   */\n  void update_old(size_t source_id, size_t target_id) { \n    old_message(source_id, target_id) = message(source_id, target_id);\n  }\n  \n  /**\n   * \\brief Initialize the edge data with source and target having the\n   * appropriate number of states.\n   *\n   * \\param source_id the vertex id of the source\n   * \\param nsource the number of states the source vertex takes\n   * \\param target_id the vertex id of the target\n   * \\param ntarget the number of states the target vertex takes\n   */\n  void initialize(size_t source_id, size_t nsource, size_t target_id, size_t ntarget) {\n    ASSERT_GT(nsource, 0); ASSERT_GT(ntarget, 0);\n    message(source_id, target_id).setZero(ntarget);\n    old_message(source_id, target_id).setZero(ntarget);\n    message(target_id, source_id).setZero(nsource);\n    old_message(target_id, source_id).setZero(nsource);\n  }\n  void save(graphlab::oarchive& arc) const {\n    for(size_t i = 0; i < 4; ++i) arc << messages_[i];\n    arc << weight_;\n  }\n  void load(graphlab::iarchive& arc) {\n    for(size_t i = 0; i < 4; ++i) arc >> messages_[i];\n    arc >> weight_;\n  }\n}; // End of edge data\n\n\n\n/**\n * \\brief The graph type used to store the Markov Random Field with\n * vertex data containing node potentials and beliefs and edge data\n * containing messages and weights.\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n\n\n\n\n/** \n * \\brief The Loopy Belief Propagation Vertex Program which computes\n * the product of the inbound messages during the gather phase,\n * updates the belief during the apply phase, and then computes the\n * new out-bound messages during the scatter phase.\n *\n * Since the gather phase is computing the product of the inbound\n * messages and the messages are stored in log form the resulting sum\n * operation is actually a vector sum and so the gather type is simply\n * the factor type and the operator+= operation for the factor type is\n * sufficient.\n *\n */\nstruct bp_vertex_program : \n  public graphlab::ivertex_program< graph_type, factor_type,\n                                    graphlab::messages::sum_priority >,\n  public graphlab::IS_POD_TYPE {\n\n  /**\n   * \\brief Since the MRF is undirected we will use all edges for gather and\n   * scatter\n   */\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of gather_edges \n\n  /**\n   * \\brief Update the old message to be the new message and collect the\n   * message value.\n   */\n  factor_type gather(icontext_type& context, const vertex_type& vertex, \n                     edge_type& edge) const {\n    const vertex_type other_vertex = get_other_vertex(edge, vertex);\n    edge_data& edata = edge.data();\n    // Update the old message with the value of the new Message.  We\n    // then receive the old message during gather and then compute the\n    // \"cavity\" during scatter (again using the old message).\n    edata.update_old(other_vertex.id(), vertex.id());\n    const factor_type& recv_message = \n      edata.old_message(other_vertex.id(), vertex.id());\n    return recv_message;\n\n  }; // end of gather function\n\n  /**\n   * \\brief Multiply message product by node potential and update the\n   * belief.\n   */\n  void apply(icontext_type& context, vertex_type& vertex, \n             const factor_type& total) {\n    // If we have no neighbors than the belief is equal to the\n    // potential so simply update the belief\n    if(vertex.num_in_edges() + vertex.num_out_edges() == 0) {\n      vertex.data().belief = make_node_potential(vertex.id());\n    } else {\n      vertex_data& vdata = vertex.data();\n      // Multiply (add in log space) the potential to compute the belief\n      vdata.belief = make_node_potential(vertex.id()) + total;\n      ASSERT_GT(vdata.belief.size(), 0);\n      // Rescale the belief to ensure numerical stability.  (This is\n      // essentially normalization in log-space.)\n      vdata.belief.array() -= vdata.belief.maxCoeff();\n    }\n  }; // end of apply\n\n  /**\n   * \\brief Since the MRF is undirected we will use all edges for gather and\n   * scatter\n   */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of scatter edges\n\n  /**\n   * \\brief Compute new message value for each edge.\n   */\n  void scatter(icontext_type& context, const vertex_type& vertex, \n               edge_type& edge) const {  \n    const vertex_type other_vertex = get_other_vertex(edge, vertex);\n    edge_data& edata = edge.data();\n    // Divide (subtract in log space) out of the belief the old in\n    // message to construct the cavity\n    const factor_type& old_in_message = \n      edata.old_message(other_vertex.id(), vertex.id());\n    ASSERT_EQ(old_in_message.size(), vertex.data().belief.size());\n    factor_type cavity = vertex.data().belief - old_in_message;\n    // compute the new message by convolving with the Ising-Potts Edge\n    // factor.\n    factor_type& new_out_message = \n      edata.message(vertex.id(), other_vertex.id());\n    // Make a backup of the last sent message which we will use to\n    // maintain the cache\n    const factor_type last_sent_message = new_out_message;\n    const factor_type& last_recv_message = \n      edata.old_message(vertex.id(), other_vertex.id());\n    convolve(cavity, edata.weight(), new_out_message);\n    // Renormalize (done in log space)\n    new_out_message.array() -= new_out_message.maxCoeff();\n    // // Apply damping to the message to stabilize convergence.\n    new_out_message = DAMPING * last_sent_message + \n      (1-DAMPING) * new_out_message;\n    // Compute message residual\n    const double residual = \n      (new_out_message - last_recv_message).cwiseAbs().sum();\n    if(USE_CACHE) {\n      // context.clear_gather_cache(other_vertex);\n      context.post_delta(other_vertex, new_out_message - last_sent_message);\n      edata.update_old(vertex.id(), other_vertex.id());\n    }\n    // Schedule the adjacent vertex\n    if(residual > TOLERANCE) context.signal(other_vertex, residual);\n }; // end of scatter\n\nprivate:\n\n  /**\n   * \\brief Compute the convolution of the cavity with the Ising-Potts\n   * edge potential and store the result in the message\n   *\n   * \\param cavity the belief minus the in-bound message\n   * \\param weight the edge weight used to scale the smoothing parameter\n   * \\param [out] message The message in which to store the result of\n   * the convolution.\n   */\n  inline void convolve(const factor_type& cavity, const double& weight, \n                       factor_type& message) const {\n    for(int i = 0; i < message.size(); ++i) {\n      double sum = 0;\n      for(int j = 0; j < cavity.size(); ++j) {\n        sum += std::exp( cavity(j)  + ( i == j? 0 : -(SMOOTHING*weight) ) ); \n      }\n      // To try and ensure numerical stability we do not allow\n      // messages to underflow in log-space\n      message(i) = (sum > 0)? std::log(sum) : \n        std::numeric_limits::min();\n    }\n  } // end of convolve\n  \n  /**\n   * \\brief Given an edge and a vertex return the other vertex along\n   * that edge. \n   */\n  inline vertex_type get_other_vertex(edge_type& edge, \n                                      const vertex_type& vertex) const {\n    return vertex.id() == edge.source().id()? edge.target() : edge.source();\n  }; // end of other_vertex\n\n}; // end of class bp_vertex_program\n\n\n\n\n \n\n\n\n\n\n/**\n * \\brief The edge data loader is used by the GraphLab graph loading\n * API to parse lines in the edge data file. \n */\nbool edge_loader(graph_type& graph, const std::string& fname, \n                 const std::string& line) {\n  ASSERT_FALSE(line.empty()); \n  namespace qi = boost::spirit::qi;\n  namespace ascii = boost::spirit::ascii;\n  namespace phoenix = boost::phoenix;\n  graphlab::vertex_id_type source(-1), target(-1);\n  double weight = 1;\n  const bool success = qi::phrase_parse\n    (line.begin(), line.end(),       \n     //  Begin grammar\n     (\n      qi::ulong_[phoenix::ref(source) = qi::_1] >>  -qi::char_(',') \n      >> qi::ulong_[phoenix::ref(target) = qi::_1] >>  \n      -(-qi::char_(',') >> qi::double_[phoenix::ref(weight) = qi::_1])\n      )\n     ,\n     //  End grammar\n     ascii::space);\n  if(!success) return false;\n  if(source == target) return true;\n  else {\n    graph.add_edge(source, target, edge_data(weight));\n    return true;\n  }\n} // end of edge loader\n\n\n\n/**\n * \\brief The edge initializer is used to allocate the messages along\n * each edge based on the number of states of the source and target\n * vertex.\n */\nvoid edge_initializer(graph_type::edge_type& edge) {\n  edge_data& edata = edge.data();\n  const graphlab::vertex_id_type source_id = edge.source().id();\n  const size_t nsource = NSTATES;\n  const graphlab::vertex_id_type target_id = edge.target().id();\n  const size_t ntarget = NSTATES;\n  edata.initialize(source_id, nsource, target_id, ntarget);\n} // end of edge initializer\n\n\n\n\n/**\n * \\brief The belief prediction saver is used to save the belief\n * predictions for each vertex.\n */\nstruct belief_prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    std::stringstream strm;\n    strm << vertex.id() << '\\t';\n    factor_type pred = vertex.data().belief;\n    double sum = 0;\n    for(int i = 0; i < pred.size(); ++i) \n      sum += (pred(i) = std::exp(pred(i)));\n    pred.array() /= sum;\n    for(int i = 0; i < pred.size(); ++i) \n      strm << pred(i) << (i+1 < pred.size()? '\\t' : '\\n');\n    return strm.str();\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\"; // nop\n  }\n}; // end of belief_prediction_saver\n\n\n/**\n * \\brief The MAP prediction saver is used to save the map estimated\n * for each vertex.  The MAP estimate is the most likely assignment\n */\nstruct map_prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    std::stringstream strm;\n    size_t prediction = 0;\n    vertex.data().belief.maxCoeff(&prediction);\n    strm << vertex.id() << '\\t' << prediction << '\\n';\n    return strm.str();\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\"; // nop\n  }\n}; // end of map prediction_saver\n\n\n\n\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  // Parse command line options -----------------------------------------------\n  // \\todo update description string\n  const std::string description = \"Structure prediction solver\";\n  graphlab::command_line_options clopts(description);\n \n  std::string graph_dir;\n  std::string output_dir = \"pred\";\n  std::string exec_type = \"async\";\n  std::string format = \"tsv\";\n  bool map = false;\n  clopts.attach_option(\"graph\", graph_dir,\n                       \"The directory containing the adjacency graph\");\n  clopts.add_positional(\"graph\"); \n  clopts.attach_option(\"field\", FIELD, \n                       \"The background field used to construct the node potentials\");\n  clopts.attach_option(\"nstates\", NSTATES, \n                       \"The number of states for each variable\");\n  clopts.attach_option(\"cache\", USE_CACHE, \"use gather caching\");\n  clopts.attach_option(\"output\", output_dir,\n                       \"The directory in which to save the predictions\");\n  clopts.attach_option(\"format\", format, \"The graph file format.\");\n  clopts.add_positional(\"output\");\n  clopts.attach_option(\"smoothing\", SMOOTHING,\n                       \"The amount of smoothing (larger = more)\");\n  clopts.attach_option(\"damping\", DAMPING,\n                       \"The amount of damping (0 -> no damping and 1 -> no progress)\");\n  clopts.attach_option(\"tol\", TOLERANCE,\n                       \"The tolerance level for convergence.\");\n  clopts.attach_option(\"map\", map,\n                       \"Return maximizing assignment instead of the posterior distribution.\");\n  clopts.attach_option(\"engine\", exec_type,\n                       \"The type of engine to use {async, sync}.\");\n  if(!clopts.parse(argc, argv)) {\n    graphlab::mpi_tools::finalize();\n    return clopts.is_set(\"help\")? EXIT_SUCCESS : EXIT_FAILURE;\n  }\n\n  clopts.get_engine_args().set_option(\"use_cache\", USE_CACHE);\n\n  if(graph_dir.empty()) {\n    logstream(LOG_ERROR) << \"No graph was provided.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  // Start the webserver\n  graphlab::launch_metric_server();\n\n  ///! load the graph\n  graph_type graph(dc, clopts);  \n\n\n  ///! load the graph\n  graph.load_format(graph_dir, format);\n  graph.finalize();\n  dc.cout() << \"Initializing edge data\" << std::endl;\n  graph.transform_edges(edge_initializer);\n\n  typedef graphlab::omni_engine engine_type;\n  engine_type engine(dc, graph, exec_type, clopts);\n  engine.signal_all();\n  graphlab::timer timer;\n  dc.cout() << \"Running engine\" << std::endl;\n  engine.start();  \n  const double runtime = timer.current_time();\n    dc.cout() \n    << \"----------------------------------------------------------\" << std::endl\n    << \"Final Runtime (seconds):   \" << runtime \n    << std::endl\n    << \"Updates executed: \" << engine.num_updates() << std::endl\n    << \"Update Rate (updates/second): \" \n    << engine.num_updates() / runtime << std::endl;\n    \n    \n  std::cout << \"Saving predictions\" << std::endl;\n  const bool gzip_output = false;\n  const bool save_vertices = true;\n  const bool save_edges = false;\n  const size_t threads_per_machine = 2;\n  if(map) {\n    graph.save(output_dir, map_prediction_saver(),\n               gzip_output, save_vertices, \n               save_edges, threads_per_machine);\n  } else { \n    graph.save(output_dir, belief_prediction_saver(),\n               gzip_output, save_vertices, \n               save_edges, threads_per_machine);\n  }\n\n\n    \n  //  graphlab::stop_metric_server_on_eof();\n  graphlab::stop_metric_server();\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n\n\n} // end of main\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
  },
  {
    "path": "toolkits/graphical_models/profile_lbp_synthetic2.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n *\n * \\brief This application is almost identical to LBP structured\n * prediction except that it generates an artificial field for every\n * vertex enabling the studying of distributed LBP on various graph\n * structures without the need for actual data.  \n *\n *\n * Technical Explanation\n * ========================\n *\n * This application creates a pair-wise Markov Random Field with\n * Ising-Potts edge factors and then uses residual loopy belief\n * propagation to compute posterior belief estimates for each vertex.\n *\n *\n *  \\author Joseph Gonzalez\n */\n\n\n#include \n#include \n#include \n#include \n#include \n\n\n#include \n#include \"eigen_serialization.hpp\"\n\n\n\n#include \n#include \n\n\n\n\n/**\n * \\brief Eigen library vectors are used to store factor in _LOG\n * SPACE_.\n */\ntypedef Eigen::VectorXd factor_type;\n\n/**\n * \\brief The Ising smoothing parameter which controls the coupling\n * between adjacent predictions in the graph.  Larger values imply\n * greater smoothing (stronger coupling). \n *\n * \\code\n * edge_factor(xi, xj) = exp( (xi == xj)? 0 : -SMOOTHING * edge_weight ); \n * \\endcode\n *\n * Not that the default edge weight is 1 however the graph file can\n * contain an additional edge weight column which allows per edge\n * control of the smoothing parameter.\n *\n * This parameter is set as a command line argument.\n */\ndouble SMOOTHING = 2;\n\n\ndouble FIELD = 2;\nsize_t NSTATES = 5;\n\n/**\n * \\brief The Damping parameter which helps ensure stable convergence.\n * Larger damping values lead to slower but more stable convergence.\n *\n * Currently damping is implemented in log-space in the following\n * equation:\n *\n * \\code\n * log(new_message) = DAMPING * log(old_message) + \n *                         (1-DAMPING) * log(new_message);\n * \\endcode\n *\n * This parameter is set as a command line argument.\n */\ndouble DAMPING = 0.1;\n\n/**\n * \\brief The convergence threshold for each message.  Smaller values\n * imply tighter convergence but slower execution.\n *\n *\n * The algorithm convergence when:\n *   \n * \\code\n * sum(abs(log(old_message) - log(new_message))) < TOLERANCE\n * \\endcode\n *\n * The parameter is set as a command line argument\n */\ndouble TOLERANCE = 0.01;\n\n\n\nbool USE_CACHE = false;\n\n\n/**\n * Make a synthetic node potential\n */\nfactor_type make_node_potential(size_t vid) {\n  // const size_t obs = vid % NSTATES;\n  const size_t obs = 0;\n  factor_type factor;\n  factor.setZero(NSTATES);\n  if(vid % 101 < 1) {\n    for(size_t i = 0; i <  NSTATES; ++i) {\n      factor(i) = (i == obs)? 0 : -FIELD;\n    }\n  } \n  return factor;\n}\n\n/**\n * \\brief The vertex data contains the vertex potential as well as the\n * current belief estimate and represents a random variable in the\n * Markov Random Field.\n *\n * The vertex potential represents the prior and is obtained from the\n * vertex prior file (stored in log form).\n *\n * The belief represents the current posterior estimate.\n */\nstruct vertex_data {\n  factor_type belief;\n  void load(graphlab::iarchive& arc) { arc >> belief; }\n  void save(graphlab::oarchive& arc) const { arc << belief; }\n}; // end of vertex_data\n\n\n/**\n * \\brief The edge data represents an edge in the Markov Random Field\n * and contains the loopy belief propagation message in both\n * directions along that edge as well as the old message in each\n * direction.  In addition each edge contains the weight parameter\n * used to set edge specific smoothing (default value is 1).\n */\nclass edge_data {\n  /**\n   * \\brief We store old and new messages in both directions as an\n   * array of messages.  The particular message index is then computed\n   * using the \\ref message_idx function.\n   */\n  factor_type messages_[2];\n  /**\n   * \\brief The weight associated with the edge (used to scale the\n   * smoothing parameter)\n   */\n  double weight_;\n  /**\n   * \\brief The function used to compute the message index in the edge\n   * message array.\n   */\n  size_t message_idx(size_t source_id, size_t target_id) {\n    return size_t(source_id < target_id);\n  }\n\npublic:\n\n  edge_data(const double w = 1) : weight_(w) { }\n  const double& weight() const { return weight_; }\n\n  /**\n   * \\brief Get the new message value from source_id to target_id\n   */\n  factor_type& message(size_t source_id, size_t target_id) { \n    return messages_[message_idx(source_id, target_id)];\n  }\n  \n  /**\n   * \\brief Initialize the edge data with source and target having the\n   * appropriate number of states.\n   *\n   * \\param source_id the vertex id of the source\n   * \\param nsource the number of states the source vertex takes\n   * \\param target_id the vertex id of the target\n   * \\param ntarget the number of states the target vertex takes\n   */\n  void initialize(size_t source_id, size_t nsource, size_t target_id, size_t ntarget) {\n    ASSERT_GT(nsource, 0); ASSERT_GT(ntarget, 0);\n    message(source_id, target_id).setZero(ntarget);\n    message(target_id, source_id).setZero(nsource);\n  }\n  void save(graphlab::oarchive& arc) const {\n    for(size_t i = 0; i < 2; ++i) arc << messages_[i];\n    arc << weight_;\n  }\n  void load(graphlab::iarchive& arc) {\n    for(size_t i = 0; i < 2; ++i) arc >> messages_[i];\n    arc >> weight_;\n  }\n}; // End of edge data\n\n\n\n/**\n * \\brief The graph type used to store the Markov Random Field with\n * vertex data containing node potentials and beliefs and edge data\n * containing messages and weights.\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n\n\n\n\n/** \n * \\brief The Loopy Belief Propagation Vertex Program which computes\n * the product of the inbound messages during the gather phase,\n * updates the belief during the apply phase, and then computes the\n * new out-bound messages during the scatter phase.\n *\n * Since the gather phase is computing the product of the inbound\n * messages and the messages are stored in log form the resulting sum\n * operation is actually a vector sum and so the gather type is simply\n * the factor type and the operator+= operation for the factor type is\n * sufficient.\n *\n */\nstruct bp_vertex_program : \n  public graphlab::ivertex_program< graph_type, factor_type,\n                                    graphlab::messages::sum_priority >,\n  public graphlab::IS_POD_TYPE {\n  float residual;\n  bp_vertex_program() : residual(0) { }\n\n  /**\n   * \\brief Since the MRF is undirected we will use all edges for gather and\n   * scatter\n   */\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of gather_edges \n\n  /**\n   * \\brief Update the old message to be the new message and collect the\n   * message value.\n   */\n  factor_type gather(icontext_type& context, const vertex_type& vertex, \n                     edge_type& edge) const {\n    const vertex_type other_vertex = get_other_vertex(edge, vertex);\n    edge_data& edata = edge.data();\n    const factor_type old_out_message = edata.message(vertex.id(), other_vertex.id());\n    const factor_type old_in_message = edata.message(other_vertex.id(), vertex.id());\n    factor_type cavity;\n    if(other_vertex.data().belief.size()  == old_out_message.size()) {\n      cavity = other_vertex.data().belief - old_out_message;\n    } else { cavity = make_node_potential(other_vertex.id()); }\n    factor_type new_in_message(old_in_message.size()); \n    convolve(cavity, edata.weight(), new_in_message);\n    new_in_message.array() -= new_in_message.maxCoeff();\n    new_in_message = DAMPING * old_in_message + (1-DAMPING) * new_in_message;\n    new_in_message.array() -= new_in_message.maxCoeff();\n    edata.message(other_vertex.id(), vertex.id()) = new_in_message;\n    return new_in_message;\n  }; // end of gather function\n\n  /**\n   * \\brief Multiply message product by node potential and update the\n   * belief.\n   */\n  void apply(icontext_type& context, vertex_type& vertex, \n             const factor_type& total) {\n    // If we have no neighbors than the belief is equal to the\n    // potential so simply update the belief\n    if(vertex.num_in_edges() + vertex.num_out_edges() == 0) {\n      vertex.data().belief = make_node_potential(vertex.id());\n    } else {\n      vertex_data& vdata = vertex.data();\n      // Multiply (add in log space) the potential to compute the belief\n      factor_type new_belief = make_node_potential(vertex.id()) + total;\n      ASSERT_GT(new_belief.size(), 0);\n      // Rescale the belief to ensure numerical stability.  (This is\n      // essentially normalization in log-space.)\n      new_belief.array() -= new_belief.maxCoeff();\n      if(vdata.belief.size() != new_belief.size()) { residual = 1; }\n      else { residual = (new_belief - vdata.belief).cwiseAbs().sum();}\n      vdata.belief = new_belief;\n    }\n  }; // end of apply\n\n  /**\n   * \\brief Since the MRF is undirected we will use all edges for gather and\n   * scatter\n   */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const {\n    if(USE_CACHE || residual > TOLERANCE)\n      return graphlab::ALL_EDGES; \n    else return graphlab::NO_EDGES;\n  }; // end of scatter edges\n\n  /**\n   * \\brief Compute new message value for each edge.\n   */\n  void scatter(icontext_type& context, const vertex_type& vertex, \n               edge_type& edge) const {  \n    const vertex_type other_vertex = get_other_vertex(edge, vertex);\n    if(USE_CACHE) {\n      context.clear_gather_cache(other_vertex);\n    }\n    // Schedule the adjacent vertex\n    if(residual > TOLERANCE) context.signal(other_vertex, residual);\n }; // end of scatter\n\nprivate:\n\n  /**\n   * \\brief Compute the convolution of the cavity with the Ising-Potts\n   * edge potential and store the result in the message\n   *\n   * \\param cavity the belief minus the in-bound message\n   * \\param weight the edge weight used to scale the smoothing parameter\n   * \\param [out] message The message in which to store the result of\n   * the convolution.\n   */\n  inline void convolve(const factor_type& cavity, const double& weight, \n                       factor_type& message) const {\n    for(int i = 0; i < message.size(); ++i) {\n      double sum = 0;\n      for(int j = 0; j < cavity.size(); ++j) {\n        sum += std::exp( cavity(j)  + ( i == j? 0 : -(SMOOTHING*weight) ) ); \n      }\n      // To try and ensure numerical stability we do not allow\n      // messages to underflow in log-space\n      message(i) = (sum > 0)? std::log(sum) : \n        std::numeric_limits::min();\n    }\n  } // end of convolve\n  \n  /**\n   * \\brief Given an edge and a vertex return the other vertex along\n   * that edge. \n   */\n  inline vertex_type get_other_vertex(edge_type& edge, \n                                      const vertex_type& vertex) const {\n    return vertex.id() == edge.source().id()? edge.target() : edge.source();\n  }; // end of other_vertex\n\n}; // end of class bp_vertex_program\n\n\n\n\n \n\n\n\n\n\n/**\n * \\brief The edge data loader is used by the GraphLab graph loading\n * API to parse lines in the edge data file. \n */\nbool edge_loader(graph_type& graph, const std::string& fname, \n                 const std::string& line) {\n  ASSERT_FALSE(line.empty()); \n  namespace qi = boost::spirit::qi;\n  namespace ascii = boost::spirit::ascii;\n  namespace phoenix = boost::phoenix;\n  graphlab::vertex_id_type source(-1), target(-1);\n  double weight = 1;\n  const bool success = qi::phrase_parse\n    (line.begin(), line.end(),       \n     //  Begin grammar\n     (\n      qi::ulong_[phoenix::ref(source) = qi::_1] >>  -qi::char_(',') \n      >> qi::ulong_[phoenix::ref(target) = qi::_1] >>  \n      -(-qi::char_(',') >> qi::double_[phoenix::ref(weight) = qi::_1])\n      )\n     ,\n     //  End grammar\n     ascii::space);\n  if(!success) return false;\n  if(source == target) return true;\n  else {\n    graph.add_edge(source, target, edge_data(weight));\n    return true;\n  }\n} // end of edge loader\n\n\n\n/**\n * \\brief The edge initializer is used to allocate the messages along\n * each edge based on the number of states of the source and target\n * vertex.\n */\nvoid edge_initializer(graph_type::edge_type& edge) {\n  edge_data& edata = edge.data();\n  const graphlab::vertex_id_type source_id = edge.source().id();\n  const size_t nsource = NSTATES;\n  const graphlab::vertex_id_type target_id = edge.target().id();\n  const size_t ntarget = NSTATES;\n  edata.initialize(source_id, nsource, target_id, ntarget);\n} // end of edge initializer\n\n\n\n\n/**\n * \\brief The belief prediction saver is used to save the belief\n * predictions for each vertex.\n */\nstruct belief_prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    std::stringstream strm;\n    strm << vertex.id() << '\\t';\n    factor_type pred = vertex.data().belief;\n    double sum = 0;\n    for(int i = 0; i < pred.size(); ++i) \n      sum += (pred(i) = std::exp(pred(i)));\n    pred.array() /= sum;\n    for(int i = 0; i < pred.size(); ++i) \n      strm << pred(i) << (i+1 < pred.size()? '\\t' : '\\n');\n    return strm.str();\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\"; // nop\n  }\n}; // end of belief_prediction_saver\n\n\n/**\n * \\brief The MAP prediction saver is used to save the map estimated\n * for each vertex.  The MAP estimate is the most likely assignment\n */\nstruct map_prediction_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    std::stringstream strm;\n    size_t prediction = 0;\n    vertex.data().belief.maxCoeff(&prediction);\n    strm << vertex.id() << '\\t' << prediction << '\\n';\n    return strm.str();\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\"; // nop\n  }\n}; // end of map prediction_saver\n\n\n\n\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  // Parse command line options -----------------------------------------------\n  // \\todo update description string\n  const std::string description = \"Structure prediction solver\";\n  graphlab::command_line_options clopts(description);\n \n  std::string graph_dir;\n  std::string output_dir = \"pred\";\n  std::string exec_type = \"async\";\n  std::string format = \"tsv\";\n  bool map = false;\n  clopts.attach_option(\"graph\", graph_dir,\n                       \"The directory containing the adjacency graph\");\n  clopts.add_positional(\"graph\"); \n  clopts.attach_option(\"field\", FIELD, \n                       \"The background field used to construct the node potentials\");\n  clopts.attach_option(\"nstates\", NSTATES, \n                       \"The number of states for each variable\");\n  clopts.attach_option(\"cache\", USE_CACHE, \"use gather caching\");\n  clopts.attach_option(\"output\", output_dir,\n                       \"The directory in which to save the predictions\");\n  clopts.attach_option(\"format\", format, \"The graph file format.\");\n  clopts.add_positional(\"output\");\n  clopts.attach_option(\"smoothing\", SMOOTHING,\n                       \"The amount of smoothing (larger = more)\");\n  clopts.attach_option(\"damping\", DAMPING,\n                       \"The amount of damping (0 -> no damping and 1 -> no progress)\");\n  clopts.attach_option(\"tol\", TOLERANCE,\n                       \"The tolerance level for convergence.\");\n  clopts.attach_option(\"map\", map,\n                       \"Return maximizing assignment instead of the posterior distribution.\");\n  clopts.attach_option(\"engine\", exec_type,\n                       \"The type of engine to use {async, sync}.\");\n  if(!clopts.parse(argc, argv)) {\n    graphlab::mpi_tools::finalize();\n    return clopts.is_set(\"help\")? EXIT_SUCCESS : EXIT_FAILURE;\n  }\n\n  clopts.get_engine_args().set_option(\"use_cache\", USE_CACHE);\n\n  if(graph_dir.empty()) {\n    logstream(LOG_ERROR) << \"No graph was provided.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  // Start the webserver\n  graphlab::launch_metric_server();\n\n  ///! load the graph\n  graph_type graph(dc, clopts);  \n\n\n  ///! load the graph\n  graph.load_format(graph_dir, format);\n  graph.finalize();\n  dc.cout() << \"Initializing edge data\" << std::endl;\n  graph.transform_edges(edge_initializer);\n\n  typedef graphlab::omni_engine engine_type;\n  engine_type engine(dc, graph, exec_type, clopts);\n  engine.signal_all();\n  graphlab::timer timer;\n  dc.cout() << \"Running engine\" << std::endl;\n  engine.start();  \n  const double runtime = timer.current_time();\n    dc.cout() \n    << \"----------------------------------------------------------\" << std::endl\n    << \"Final Runtime (seconds):   \" << runtime \n    << std::endl\n    << \"Updates executed: \" << engine.num_updates() << std::endl\n    << \"Update Rate (updates/second): \" \n    << engine.num_updates() / runtime << std::endl;\n    \n    \n  std::cout << \"Saving predictions\" << std::endl;\n  const bool gzip_output = false;\n  const bool save_vertices = true;\n  const bool save_edges = false;\n  const size_t threads_per_machine = 2;\n  if(map) {\n    graph.save(output_dir, map_prediction_saver(),\n               gzip_output, save_vertices, \n               save_edges, threads_per_machine);\n  } else { \n    graph.save(output_dir, belief_prediction_saver(),\n               gzip_output, save_vertices, \n               save_edges, threads_per_machine);\n  }\n\n\n    \n  //  graphlab::stop_metric_server_on_eof();\n  graphlab::stop_metric_server();\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n\n\n} // end of main\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
  },
  {
    "path": "toolkits/graphical_models/synthetic_image_data.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * This application creates a synthetic data to test and demonstrate\n * the structred prediction applications.  The synthetic task is to\n * remove noise from a synthetic noisy image.\n *\n * In addition this application can be used to take the output of\n * the structured prediction tools and rendering the predicted noise\n * free image.\n *\n *\n * \n *  \\author Joseph Gonzalez\n */\n\n#include \n#include \n#include \n#include \n#include \n\n\n#include \n#include   \n\n#include \n\nusing namespace cv;\n\n\n/**\n * The pixel struct encodes a pixel location and value.\n *\n * Because each pixel corresponds to vertex in the graph we need a\n * mapping between integers and coordinats.  This is accomplished by\n * using the first two lowest order bytes to encode the column and the\n * two highest order bytes to encode the row.\n */\nstruct pixel {\n  uint16_t row, col;\n  double value;\n  pixel(uint32_t ind = 0, double value = 0) :\n    row(ind >> 16), col( ind & ((1 << 16)-1)), value(value) {  }\n}; // end of sub2ind\n\n\n\ngraphlab::vertex_id_type sub2ind(uint16_t r, uint16_t c) {\n  ASSERT_LT(r, ((1 << 16)-1));\n  ASSERT_LT(c, ((1 << 16)-1));\n  return (r << 16) | c;\n}; // end of sub2ind\n\n\n\n\nvoid make_data(const uint16_t rows, const uint16_t cols,\n               const size_t ncolors, const double error_rate,\n               const std::string& vdata_fn,\n               const std::string& edata_fn,\n               const std::string& orig_img_fn,\n               const std::string& noisy_img_fn) {\n  const double center_r = rows / 2.0;\n  const double center_c = cols / 2.0;\n  const double max_radius = std::min(rows, cols) / 2.0;\n\n  Mat orig_img(cols, rows, CV_8UC1);\n  Mat noisy_img(cols, rows, CV_8UC1);\n  std::ofstream vdata_fout(vdata_fn.c_str());\n  std::ofstream edata_fout(edata_fn.c_str());\n\n  for(size_t r = 0; r < rows; ++r) {\n    for(size_t c = 0; c < cols; ++c) {\n      // determine the true pixel id\n      const graphlab::vertex_id_type vid = sub2ind(r,c);\n      // Compute the true pixel value\n      const double distance = sqrt((r-center_r)*(r-center_r) + \n                                   (c-center_c)*(c-center_c));\n      // Compute ring of sunset\n      const uint16_t ring_color =  \n        std::floor(std::min(1.0, distance/max_radius) * (ncolors - 1) );\n      // Compute the true pixel color by masking with the horizon\n      const uint16_t true_color = r < rows/2 ? ring_color : 0;\n      // compute the predicted color\n      const uint16_t obs_color = graphlab::random::rand01() < error_rate?\n        graphlab::random::fast_uniform(0, ncolors-1) : true_color;\n\n      const double c1p = double(true_color)/(ncolors-1);\n      unsigned char c1 = (unsigned char)(255 * c1p > 255 ? 255 : 255 * c1p);\n      orig_img.at(r, c) = c1;\n      const double c2p = double(obs_color)/(ncolors-1);\n      unsigned char c2 = (unsigned char)(255 * c2p > 255 ? 255 : 255 * c2p);\n      noisy_img.at(r, c) = c2;\n\n      // Save the prior\n      vdata_fout << vid << '\\t';\n      for(size_t pred = 0; pred < ncolors; ++pred) {\n        const double prior = obs_color == pred? error_rate : (error_rate) /(ncolors - 1);\n        vdata_fout << prior << (pred+1 < ncolors? '\\t' : '\\n');\n      }\n\n      // Add the edges\n      if(r + 1 < rows) \n        edata_fout << vid << '\\t' << sub2ind(r+1,c) << '\\n';\n      if(c + 1 < cols) \n        edata_fout << vid << '\\t' << sub2ind(r,c+1) << '\\n';\n    } // end of loop over cols\n  } // end of loop over rows\n\n  vdata_fout.close();\n  edata_fout.close();\n  imwrite(orig_img_fn, orig_img);\n  imwrite(noisy_img_fn, noisy_img);\n} // end of make data\n\n\n\n\nvoid read_data(const std::string& pred_img_fn) {\n  namespace qi = boost::spirit::qi;\n  namespace ascii = boost::spirit::ascii;\n  namespace phoenix = boost::phoenix;\n\n  std::vector pixels;\n\n  std::string line;\n  size_t line_counter = 0;\n  uint16_t nrows = 0, ncols = 0;\n  size_t min_pixel(-1);\n  size_t max_pixel(0);\n  while (std::getline(std::cin, line)) {\n    graphlab::vertex_id_type vid(-1);\n    std::vector values;\n    const bool success = qi::phrase_parse\n      (line.begin(), line.end(),       \n       //  Begin grammar\n       (\n        qi::ulong_[phoenix::ref(vid) = qi::_1] >> -qi::char_(\",\") >>\n        (qi::double_[phoenix::push_back(phoenix::ref(values), qi::_1)] % -qi::char_(\",\") )\n        )\n       ,\n       //  End grammar\n       ascii::space); \n    if(!success) {\n      logstream(LOG_ERROR) << \"Error parsing line: \" << line_counter << std::endl\n                           << \"\\t\\\"\" << line << \"\\\"\";\n    }\n    ASSERT_GT(values.size(), 0);\n    const size_t pred = \n      std::max_element(values.begin(), values.end()) - values.begin();\n    min_pixel = std::min(min_pixel, pred);\n    max_pixel = std::max(max_pixel, pred);\n    const pixel pix(vid, double(pred) / (values.size() - 1) );\n    pixels.push_back(pix);  \n    nrows = std::max(nrows, pix.row);\n    ncols = std::max(ncols, pix.col);\n  }\n  nrows++; ncols++;\n  std::cout << \"nrows: \" << nrows << std::endl\n            << \"ncols: \" << ncols << std::endl\n            << \"minp:  \" << min_pixel << std::endl\n            << \"maxp:  \" << max_pixel << std::endl;\n  Mat pred_img(ncols, nrows, CV_8UC1);\n  for(size_t i = 0; i < pixels.size(); ++i) {\n    int s = 255 * pixels[i].value;\n    pred_img.at(pixels[i].row, pixels[i].col) = \n                                     (unsigned char)(s >= 255 ? 255 : s);\n  }\n  imwrite(pred_img_fn, pred_img);\n} // end of make data\n\n\n\nint main(int argc, char** argv) {\n  std::cout << \"Create a synthetic noisy image.\" << std::endl;\n\n  // Set initial values for members ------------------------------------------->\n  size_t ncolors = 5;\n  double error_rate = 0.5;\n  uint16_t nrows = 200;\n  uint16_t ncols = 200;\n \n  std::string vdata_fn = \"synth_vdata.tsv\";\n  std::string edata_fn = \"synth_edata.tsv\";\n\n  std::string orig_img_fn = \"orig_img.jpeg\";\n  std::string noisy_img_fn = \"noisy_img.jpeg\";\n  std::string pred_img_fn;\n\n \n\n\n\n  // Parse command line arguments --------------------------------------------->\n  graphlab::command_line_options clopts(\"Create synthetic prediction\", false);\n \n  clopts.attach_option(\"vdata\", vdata_fn,\n                       \"Vertex prior filename\");\n  clopts.attach_option(\"edata\", edata_fn,\n                       \"Adjacency information\");\n  clopts.attach_option(\"ncolors\", ncolors,\n                       \"The number of colors in the noisy image\");\n  clopts.attach_option(\"error_rate\", error_rate,\n                       \"Standard deviation of noise.\");\n  clopts.attach_option(\"nrows\", nrows,\n                       \"The number of rows in the noisy image\");\n  clopts.attach_option(\"ncols\", ncols,\n                       \"The number of columns in the noisy image\");\n  clopts.attach_option(\"orig\", orig_img_fn,\n                       \"Original image file name.\");\n  clopts.attach_option(\"noisy\", noisy_img_fn,\n                       \"Noisy image file name.\");\n  clopts.attach_option(\"pred\", pred_img_fn,\n                       \"Predicted image file name.\");\n    \n  ///! Initialize control plain using mpi\n  const bool success = clopts.parse(argc, argv);\n  if(!success) {\n    return EXIT_FAILURE;\n  }\n\n  if(!pred_img_fn.empty()) {\n    std::cout << \"Reading in predictions\" << std::endl;\n    read_data(pred_img_fn);\n  } else {\n    std::cout << \"Generating synthetic data\" << std::endl;\n    make_data(nrows, ncols, ncolors, error_rate,\n              vdata_fn, edata_fn,\n              orig_img_fn, noisy_img_fn);\n  }\n  return EXIT_SUCCESS;\n} // End of main\n\n\n\n\n\n// void save_image(const size_t rows, const size_t cols,\n//                 const std::vector& values,\n//                 const std::string& fname) {\n//   using namespace Magick;\n//   std::cout << \"NPixels: \" << values.size() << std::endl;\n//   // determine the max and min colors\n//   float max_color = -std::numeric_limits::max();\n//   float min_color =  std::numeric_limits::max();\n//   foreach(pred_pair_type pair, values) {\n//     max_color = std::max(max_color, pair.second);\n//     min_color = std::min(min_color, pair.second);\n//   }\n//   Image img(Magick::Geometry(rows, cols), \"white\");\n//   // img.modifyImage();\n//   // Pixels img_cache(img);\n//   // PixelPackets* pixels = img_cache.\n//   foreach(pred_pair_type pair, values) {\n//     std::pair coords = ind2sub(rows,cols, pair.first);\n//     float value = (pair.second - min_color) / (max_color - min_color);\n//     Color color(MaxRGB * value, MaxRGB * value, MaxRGB * value, 0);\n//     img.pixelColor(coords.second, coords.first, color);\n//   }\n//   img.write(fname);\n// }\n"
  },
  {
    "path": "toolkits/graphical_models/utils.hpp",
    "content": "\n// utils.hpp - miscellaneous utilities \n// Originally from Nicol N. Schraudolph's isinf package\n// Later expanded by Dhruv Batra\n\n#ifndef UTILS_HPP\n#define UTILS_HPP\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n// row-major array access\n#define ARR_RM(arr, r_ind, c_ind, ncols) (*(arr + r_ind*ncols + c_ind))\n// col-major array access\n#define ARR_CM(arr, r_ind, c_ind, nrows) (*(arr + c_ind*nrows + r_ind))\n\n// row-major ind2sub\n#define IND2SUB_RM(ind,r,c,ncols) \\\n        r = floor(ind/ncols);     \\\n        c = ind % ncols;\n// column-major ind2sub\n#define IND2SUB_CM(ind,r,c,nrows) \\\n        c = floor(ind/nrows);     \\\n        r = ind % nrows;     \n\n// row-major sub2ind\n#define SUB2IND_RM(r,c,ncols) r*ncols + c\n\n// col-major sub2ind\n#define SUB2IND_CM(r,c,nrows) c*nrows + r\n\n// operators & formatted I/O for vectors\n// inner product\ntemplate \ninline T operator*(const std::vector& a, const std::vector& b)\n{\n    assert(a.size() == b.size());\n    T sum(0);\n    for (size_t i = 0; i < a.size(); ++i)\n        sum += a[i]*b[i];\n    return sum;\n}\n\n// element-wise sum\ntemplate \ninline std::vector& operator+(const std::vector& a, const std::vector& b)\n{\n    assert(a.size() == b.size());\n    std::vector sum(a.size());\n    for (size_t i = 0; i < a.size(); ++i)\n        sum[i] = a[i]+b[i];\n    return sum;\n}\n\ntemplate \ninline std::vector& operator+=(std::vector& a, const T& b)\n{\n    typename std::vector::iterator i(a.begin());\n    while (i != a.end()) *i++ += b;\n    return a;\n}\n\ntemplate \ninline std::vector& operator-=(std::vector& a, const T& b)\n{\n    typename std::vector::iterator i(a.begin());\n    while (i != a.end()) *i++ -= b;\n    return a;\n}\n\ntemplate \ninline std::vector& operator*=(std::vector& a, const T& b)\n{\n    typename std::vector::iterator i(a.begin());\n    while (i != a.end()) *i++ *= b;\n    return a;\n}\n\ntemplate \ninline std::vector& operator/=(std::vector& a, const T& b)\n{\n    typename std::vector::iterator i(a.begin());\n    while (i != a.end()) *i++ /= b;\n    return a;\n}\n\ntemplate \ninline std::vector& operator+=(std::vector& a, const std::vector& b)\n{\n    assert(a.size() == b.size());\n    typename std::vector::iterator i(a.begin());\n    typename std::vector::const_iterator j(b.begin());\n    while (i != a.end()) *i++ += *j++;\n    return a;\n}\n\ntemplate \ninline std::vector& operator-=(std::vector& a, const std::vector& b)\n{\n    assert(a.size() == b.size());\n    typename std::vector::iterator i(a.begin());\n    typename std::vector::const_iterator j(b.begin());\n    while (i != a.end()) *i++ -= *j++;\n    return a;\n}\n\ntemplate \ninline std::vector& operator*=(std::vector& a, const std::vector& b)\n{\n    assert(a.size() == b.size());\n    typename std::vector::iterator i(a.begin());\n    typename std::vector::const_iterator j(b.begin());\n    while (i != a.end()) *i++ *= *j++;\n    return a;\n}\n\ntemplate \ninline std::vector& operator/=(std::vector& a, const std::vector& b)\n{\n    assert(a.size() == b.size());\n    typename std::vector::iterator i(a.begin());\n    typename std::vector::const_iterator j(b.begin());\n    while (i != a.end()) *i++ /= *j++;\n    return a;\n}\n\ntemplate \ninline std::ostream& operator<<(std::ostream& os, const std::vector& x)\n{\n    typename std::vector::const_iterator i(x.begin());\n    while(i != x.end()) os << *i++ << ' ';\n    return os;\n}\n\ntemplate \ninline std::istream& operator>>(std::istream& is, std::vector& x)\n{\n    std::string s;\n    const size_t n = x.size();\n\n    while (x.size() == n)\n    {\n        getline(is, s);\n        if (is.fail()) break;\n\n        std::istringstream iss(s);\n        T item;\n\n        iss >> item;\n        while (iss.good())\n        {\n            x.push_back(item);\n            iss >> item;\n        }\n        if (!iss.fail())\n            x.push_back(item);\n    }\n\n    return is;\n}\n\n// Function to write a vector to file (Assumes << is defined for type T)\n// CHECK_NULL is provided by Danny Tarlow's Nymph Utils\ntemplate  void WriteToFile(std::string fname, std::vector vecx)\n{\n    std::ofstream fout; \n    fout.open(fname.c_str());\n    \n    //CHECK_NULL(fout.fail(),\"Could not open file for writing results\\n\");\n    \n    fout << vecx;\n    \n    fout.close();\n}\n\n\n#endif\n\n"
  },
  {
    "path": "toolkits/linear_solvers/CMakeLists.txt",
    "content": "project(GraphLab)\n# include(CheckCXXSourceCompiles)\n\n\n\n# Build als\nadd_graphlab_executable(jacobi jacobi.cpp)\nrequires_eigen(jacobi) # build and attach eigen\n\n\n"
  },
  {
    "path": "toolkits/linear_solvers/jacobi.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * Functionality: The code solves the linear system Ax = b using\n * The Jacobi algorithm. (A is a square matrix). \n * A assumed to be full column rank.  Algorithm is described\n * http://en.wikipedia.org/wiki/Jacobi_method\n * Written by Danny Bickson \n */\n#include \"../collaborative_filtering/eigen_wrapper.hpp\"\n#include \"../collaborative_filtering/types.hpp\"\n#include \"../collaborative_filtering/eigen_serialization.hpp\"\n#include \n#include \n\n\n\nenum jacobi_fields{\n  JACOBI_X = 0,\n  JACOBI_REAL_X = 1,\n  JACOBI_Y = 2,\n  JACOBI_PREV_X = 3,\n  JACOBI_PREC = 4\n};\n\nint actual_vector_len = 5;\nint data_size = 5;\nbool final_residual = true;\nbool zero = false;  //allow for zero entries in sparse matrix market format\nbool update_function = false;\ndouble ortho_repeats = 3;\nstd::string vecfile;\nint rows = 0, cols = 0;\nint max_iter = 10;\ndouble tol = 1e-5;\nint quiet = 0;\nint unittest = 0;\n\nstruct vertex_data {\n  vec pvec;\n  double A_ii;\n  //real_type y, Aii;\n  //real_type pvec[JACOBI_X], pvec[JACOBI_REAL_X], pvec[JACOBI_PREV_X];\n  vertex_data(): A_ii(1) { //: y(0), Aii(1), pvec[JACOBI_X](0), pvec[JACOBI_REAL_X](0), \n                 // pvec[JACOBI_PREV_X](-1) \n     pvec = zeros(data_size);\n     pvec[JACOBI_PREV_X] = -1;\n  }\n  void save(graphlab::oarchive& arc) const { \n    arc << pvec << A_ii;\n  }\n  /** \\brief Load the vertex data from a binary archive */\n  void load(graphlab::iarchive& arc) { \n    arc >> pvec >> A_ii;\n  }\n}; // end of vertex_data\n\nclass gather_type {\n  public:\n    vec pvec;\n    double training_rmse;\n    double validation_rmse;\n    gather_type() { training_rmse= validation_rmse = 0; }\n    void save(graphlab::oarchive& arc) const { arc << pvec << training_rmse << validation_rmse; }\n    void load(graphlab::iarchive& arc) { arc >> pvec >> training_rmse >> validation_rmse; }  \n    gather_type& operator+=(const gather_type& other) {\n      pvec += other.pvec;\n      training_rmse += other.training_rmse;\n      validation_rmse += other.validation_rmse;\n      return *this;\n    } \n\n};\n\n//gather_type ret;\n\n\n\nstruct edge_data : public graphlab::IS_POD_TYPE {\n  double obs;\n  int role;\n  enum data_role_type { TRAIN, VALIDATE, PREDICT  };\n\n  edge_data(double obs = 1, data_role_type role = TRAIN) :\n    obs(obs), role(role) { }\n\n}; // end of edge data\n\n\n/**\n * \\brief The graph type is defined in terms of the vertex and edge\n * data.\n */ \ntypedef graphlab::distributed_graph graph_type;\ngraph_type * pgraph;\n\n/**\n * \\brief Given a vertex and an edge return the other vertex in the\n * edge.\n */\ninline graph_type::vertex_type\nget_other_vertex(graph_type::edge_type& edge, \n    const graph_type::vertex_type& vertex) {\n  return vertex.id() == edge.source().id()? edge.target() : edge.source();\n}; // end of get_other_vertex\n\n//typedef double gather_type;\ntypedef double message_type;\n\nvoid start_engine();\n#include \"../collaborative_filtering/math.hpp\"\n\n\nvoid verify_values(int unittest, double residual){\n   if (unittest == 1)\n     assert(residual < 1e-5);\n}\n/**\n * \\brief The graph loader function is a line parser used for\n * distributed graph construction.\n */\ninline bool graph_loader(graph_type& graph, \n    const std::string& filename,\n    const std::string& line) {\n\n  //no need to parse\n  if (boost::algorithm::ends_with(filename ,vecfile))\n    return true;\n\n  ASSERT_FALSE(line.empty()); \n  // Determine the role of the data\n  edge_data::data_role_type role = edge_data::TRAIN;\n  \n  // Parse the line\n  std::stringstream strm(line);\n  graph_type::vertex_id_type source_id(-1), target_id(-1);\n  float obs(0);\n  strm >> source_id >> target_id;\n  source_id--; target_id--;\n  if (source_id >= (uint)rows)\n    logstream(LOG_FATAL)<<\"Row number: \" << source_id << \" sould be < rows \" << rows << \" [ line: \" << line << \" ] \" << std::endl;\n  if (target_id >= (uint)cols)\n    logstream(LOG_FATAL)<<\"Col number: \" << target_id << \" sould be < cols \" << cols << \" [ line: \" << line << \" ] \" << std::endl;\n  strm >> obs;\n  if (!info.is_square())\n  target_id = rows + target_id;\n\n  if (source_id == target_id){\n      vertex_data data;\n      data.A_ii = obs;\n      data.pvec[JACOBI_PREC] = obs;\n      graph.add_vertex(source_id, data);\n  }\n  // Create an edge and add it to the graph\n  else graph.add_edge(source_id, target_id, edge_data(obs, role)); \n  return true; // successful load\n} // end of graph_loader\n\n\n#include \"../collaborative_filtering/math.hpp\" //uses vertex_data and edge_data so has to be included here\n#include \"../collaborative_filtering/printouts.hpp\" // the same\ntypedef graphlab::omni_engine engine_type;\nengine_type * pengine = NULL;\n\nstruct linear_model_saver {\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n\n  int pos;\n  linear_model_saver(int pos): pos(pos) {}\n\n  std::string save_vertex(const vertex_type& vertex) const {\n     assert(pos >= 0 && pos < vertex.data().pvec.size());\n     std::string ret;\n     ret = boost::lexical_cast(vertex.id() + 1) + \" \";\n     ret += boost::lexical_cast(vertex.data().pvec[pos]) + \"\\n\";\n     return ret;\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\";\n  }\n}; \n\n\nvoid start_engine(){\n  vertex_set nodes = pgraph->select(selected_node);\n  pengine->signal_vset(nodes);\n  pengine->start();\n}\n\nint main(int argc, char** argv) {\n  global_logger().set_log_to_console(true);\n\n  // Parse command line options -----------------------------------------------\n  const std::string description = \n    \"Solve a linear system using Jacobi method\";\n  graphlab::command_line_options clopts(description);\n  std::string input_dir, output_dir;\n  std::string exec_type = \"synchronous\";\n  clopts.attach_option(\"matrix\", input_dir,\n      \"The directory containing the matrix file\");\n  clopts.add_positional(\"matrix\");\n  clopts.attach_option(\"initial_vector\", vecfile,\"optional initial vector\");\n  clopts.attach_option(\"debug\", debug, \"Display debug output.\");\n  clopts.attach_option(\"unittest\", unittest,  \n      \"unit testing 0=None, 1=3x3 matrix\");\n  clopts.attach_option(\"max_iter\", max_iter, \"max iterations\");\n  clopts.attach_option(\"regularization\", regularization, \"regularization\");\n  clopts.attach_option(\"tol\", tol, \"convergence threshold\");\n  clopts.attach_option(\"rows\", rows, \"number of rows\");\n  clopts.attach_option(\"cols\", cols, \"number of cols\");\n  clopts.attach_option(\"quiet\", quiet, \"quiet mode (less verbose)\");\n  if(!clopts.parse(argc, argv) || input_dir == \"\") {\n    std::cout << \"Error in parsing command line arguments.\" << std::endl;\n    clopts.print_description();\n    return EXIT_FAILURE;\n  }\n  if (quiet){\n    global_logger().set_log_level(LOG_ERROR);\n    debug = false;\n  }\n\n  if (rows <= 0 || cols <= 0 || rows != cols)\n    logstream(LOG_FATAL)<<\"Please specify number of rows/cols of the input matrix\" << std::endl;\n    \n \n  info.rows = rows;\n  info.cols = cols;\n\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n  dc.cout() << \"Loading graph.\" << std::endl;\n  graphlab::timer timer; \n  graph_type graph(dc, clopts);  \n  graph.load(input_dir, graph_loader); \n  pgraph = &graph;\n  dc.cout() << \"Loading graph. Finished in \" \n    << timer.current_time() << std::endl;\n  dc.cout() << \"Finalizing graph.\" << std::endl;\n  timer.start();\n  graph.finalize();\n  dc.cout() << \"Finalizing graph. Finished in \" \n    << timer.current_time() << std::endl;\n\n\n  dc.cout() \n    << \"========== Graph statistics on proc \" << dc.procid() \n    << \" ===============\"\n    << \"\\n Num vertices: \" << graph.num_vertices()\n    << \"\\n Num edges: \" << graph.num_edges()\n    << \"\\n Num replica: \" << graph.num_replicas()\n    << \"\\n Replica to vertex ratio: \" \n    << float(graph.num_replicas())/graph.num_vertices()\n    << \"\\n --------------------------------------------\" \n    << \"\\n Num local own vertices: \" << graph.num_local_own_vertices()\n    << \"\\n Num local vertices: \" << graph.num_local_vertices()\n    << \"\\n Replica to own ratio: \" \n    << (float)graph.num_local_vertices()/graph.num_local_own_vertices()\n    << \"\\n Num local edges: \" << graph.num_local_edges()\n    //<< \"\\n Begin edge id: \" << graph.global_eid(0)\n    << \"\\n Edge balance ratio: \" \n    << float(graph.num_local_edges())/graph.num_edges()\n    << std::endl;\n\n  dc.cout() << \"Creating engine\" << std::endl;\n  engine_type engine(dc, graph, exec_type, clopts);\n  pengine = &engine;\n\n  init_math(&graph, info, ortho_repeats, update_function);\n\n  if (vecfile.size() > 0){\n    std::cout << \"Load b vector from file\" << input_dir << vecfile << std::endl;\n    FILE * file = fopen((input_dir + vecfile).c_str(), \"r\");\n    if (file == NULL)\n      logstream(LOG_FATAL)<<\"Failed to open initial vector\"<< std::endl;\n    vec input = vec::Zero(rows);\n    double val = 0;\n    for (int i=0; i< rows; i++){\n      int rc = fscanf(file, \"%lg\\n\", &val);\n      if (rc != 1)\n        logstream(LOG_FATAL)<<\"Failed to open initial vector\"<< std::endl;\n      input[i] = val;\n    }\n    fclose(file);\n    DistVec v0(info, JACOBI_Y, false, \"v0\");\n    v0 = input;\n  }  \n\n  dc.cout() << \"Running Jacobi\" << std::endl;\n  dc.cout() << \"(C) Code by Danny Bickson, CMU \" << std::endl;\n  dc.cout() << \"Please send bug reports to danny.bickson@gmail.com\" << std::endl;\n  timer.start();\n\n  DistMat A(info);\n  DistVec b(info, JACOBI_Y,true, \"b\");\n  DistVec x(info, JACOBI_X,true, \"x\", JACOBI_PREV_X);\n  DistVec A_ii(info, JACOBI_PREC, true, \"A_ii\");\n\n  PRINT_VEC(b);\n  PRINT_VEC(x);\n  PRINT_VEC(A_ii);\n  for (int i=0; i < max_iter; i++){\n    mi.use_diag = false;\n    x = (b - A*x)/A_ii;\n    PRINT_VEC(x);\n  }\n \n  dc.cout() << \"Jacobi finished in \" << runtime << std::endl;\n  dc.cout() << \"\\t Updates: \" << engine.num_updates() << std::endl;\n\n    DistVec p(info, JACOBI_PREV_X, true, \"p\");\n    mi.use_diag = true;\n    p = A*x -b;\n    PRINT_VEC(p);\n    DistDouble ret = norm(p);\n    dc.cout() << \"Solution converged to residual: \" << ret.toDouble() << std::endl;\n \n  //vec ret = fill_output(&core.graph(), info, JACOBI_X);\n  //write_output_vector(datafile + \"x.out\", format, ret, false);\n  const double runtime = timer.current_time();\n  dc.cout() << \"----------------------------------------------------------\"\n    << std::endl\n    << \"Final Runtime (seconds):   \" << runtime \n                                        << std::endl\n                                        << \"Updates executed: \" << engine.num_updates() << std::endl\n                                        << \"Update Rate (updates/second): \" \n                                          << engine.num_updates() / runtime << std::endl;\n\n  graph.save(\"x.out\", linear_model_saver(JACOBI_X), false, true, false, 1);\n  graphlab::mpi_tools::finalize();\n\n   return EXIT_SUCCESS;\n}\n\n\n\n\n\n\n\n\n\n\n"
  },
  {
    "path": "toolkits/linear_solvers/linear_solvers.dox",
    "content": "/** \n\n\\page linear_solvers Linear iterative solver\n\n\\brief GraphLab linear solver library is used for solving the linear system Ax = b.\n\n\\section Jacobi\nThe Jacobi algorithm is one of the simplest methods, it works by applying repeatedly the update rule:\n\\verbatim\nx = (b-(A-diag(diag(A))*x) ./ diag(A)\n\\endverbatim\n\n\\section Input\nThe input folder is given using the command line --matrix=folder_name. Inside this folder should have a sparse matrix A file with the format, in each line.\n\\verbatim\nrow col val\n\\endverbatim\n\nAdditional input vector b is given using the command --input_vector=filename, this file should be found inside the folder given by --matrix. \n\n\\section Output\nThe output of the computation is a solution vector x. File name is x.out_1_of_1.\n\n\\section Jacobi example.\n\nAssume we have the linear system\n\\verbatim\nA=[  1.8147    0.9134    0.2785\n     0.9058    1.6324    0.5469\n     0.1270    0.0975    1.9575 ];\nb= [ 0.9649    0.1576    0.9706 ]';\n\\endverbatim\nThe solution will be\n\\verbatim\nx = A \\ b = [ 0.6803   -0.4396    0.4736 ]';\n\\endverbatim\n\nTo run it in GraphLab we preper a folder named\njacobi_testA, inside it we have the input file for A\nnamed A:\n\\verbatim\n1 1 1.8147\n1 2 0.9134\n1 3 0.2785\n2 1 0.9058\n2 2 1.6324\n2 3 0.5469\n3 1 0.127\n3 2 0.0975\n3 3 1.9575\n\\endverbatim\n\nAnd the input file for b (called vecB)\n\\verbatim\n0.9649\n0.1576\n0.9706\n\\endverbatim\n\nNote: both A and vecB files are found under jacobi_testA/ folder.\n\nNow we run:\n\\verbatim\n./jacobi --matrix=jacobi_testA/ --initial_vec=vecB --rows=3 --cols=3 --debug=1 --max_iter=10\n\nRunning Jacobi\n(C) Code by Danny Bickson, CMU \nPlease send bug reports to danny.bickson@gmail.com\nSolution converged to residual: 0.00507232\n----------------------------------------------------------\nFinal Runtime (seconds):   0.649031\nUpdates executed: 33\nUpdate Rate (updates/second): 50.845\n\\endverbatim\n\nWe examine the output:\n\\verbatim\n$ cat x.out_1_of_1\n1 0.67806771486642969\n2 -0.43984914015767995\n3 0.47337333903573475\n\\endverbatim\n\n*/\n\n"
  },
  {
    "path": "toolkits/toolkits.dox",
    "content": "/**\n\n\\page toolkits GraphLab Toolkits\n\n\nTo enable users to use GraphLab out-of-the-box and to demonstrate the\npower of the GraphLab API we have implemented a collection of\napplications to address a wide range of standard tasks in large-scale\ngraph computation.  \n\nWe have implemented the following toolkits\n\n\\li \\subpage topic_modeling  contains applications like LDA which can be\nused to cluster documents and extract topical representations.\n\n\\li \\subpage graph_algorithms contains algorithms mostly from the Social Network Analysis Handbook algorithms set.\n\n\\li \\subpage graph_analytics contains application like pagerank and\ntriangle counting which can be applied to general graphs to estimate\ncommunity structure.\n\n\\li \\subpage clustering contains standard data clustering tools such as \nKmeans\n\n\\li \\subpage collaborative_filtering contains a collection of\napplications used to make predictions about users interests and\nfactorize large matrices.\n\n\\li \\subpage graphical_models contains tools for making joint predictions\nabout collections of related random variables.\n\n\\li \\subpage factor_graphs contains Belief Propagation impelementation for\nfactor graphs\n\n\\li \\subpage linear_solvers contains solvers for linear systems of equations - currently the Jacobi algorithm is implemented\n\n\\li \\subpage computer_vision contains a collection of tools for\nreasoning about images.\n\n\n\n\n\n\n*/\n"
  },
  {
    "path": "toolkits/topic_modeling/CMakeLists.txt",
    "content": "project(GraphLab)\n\n# Primary executable\nadd_graphlab_executable(lda_sequential_cgs lda_sequential_cgs.cpp)\nadd_graphlab_executable(cgs_lda cgs_lda.cpp)\nadd_graphlab_executable(cgs_lda_mimno_experimental cgs_lda_mimno_experimental.cpp)\n"
  },
  {
    "path": "toolkits/topic_modeling/cgs_lda.cpp",
    "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * \\file cgs_lda.cpp\n *\n * \\brief This file contains a GraphLab based implementation of the\n * Collapsed Gibbs Sampler (CGS) for the Latent Dirichlet Allocation\n * (LDA) model.\n *\n * \n *\n * \\author Joseph Gonzalez, Diana Hu\n */\n\n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n\n\n// Global Types\n// ============================================================================\ntypedef long count_type;\n\n\n/**\n * \\brief The factor type is used to store the counts of tokens in\n * each topic for words, documents, and assignments.\n *\n * Atomic counts are used because we violate the abstraction by\n * modifying adjacent vertex data on scatter.  As a consequence\n * multiple threads on the same machine may try to update the same\n * vertex data at the same time.  The graphlab::atomic type ensures\n * that multiple increments are serially consistent.\n */\ntypedef std::vector< graphlab::atomic > factor_type;\n\n\n/**\n * \\brief We use the factor type in accumulators and so we define an\n * operator+=\n */\ninline factor_type& operator+=(factor_type& lvalue,\n                               const factor_type& rvalue) {\n  if(!rvalue.empty()) {\n    if(lvalue.empty()) lvalue = rvalue;\n    else {\n      for(size_t t = 0; t < lvalue.size(); ++t) lvalue[t] += rvalue[t];\n    }\n  }\n  return lvalue;\n} // end of operator +=\n\n// We include the rest of GraphLab after we define the operator+= for\n// vector.\n#include \n#include \n\n\n\n\n/**\n * \\brief The latent topic id of a token is the smallest reasonable\n * type.\n */\ntypedef uint16_t topic_id_type;\n\n// We require a null topic to represent the topic assignment for\n// tokens that have not yet been assigned.\n#define NULL_TOPIC (topic_id_type(-1))\n\n\n\n/**\n * \\brief The assignment type is used on each edge to store the\n * assignments of each token.  There can be several occurrences of the\n * same word in a given document and so a vector is used to store the\n * assignments of each occurrence.\n */\ntypedef std::vector< topic_id_type > assignment_type;\n\n\n// Global Variables\n// ============================================================================\n\n/**\n * \\brief The alpha parameter determines the sparsity of topics for\n * each document.\n */\ndouble ALPHA = 1;\n\n/**\n * \\brief the Beta parameter determines the sparsity of words in each\n * document.\n */\ndouble BETA = 0.1;\n\n/**\n * \\brief the total number of topics to uses\n */\nsize_t NTOPICS = 50;\n\n/**\n * \\brief The total number of words in the dataset.\n */\nsize_t NWORDS = 0;\n\n/**\n * \\brief The total number of docs in the dataset.\n */\nsize_t NDOCS = 0;\n\n/**\n * \\brief The total number of tokens in the corpus\n */\nsize_t NTOKENS = 0;\n\n\n/**\n * \\brief The number of top words to display during execution (from\n * each topic).\n */\nsize_t TOPK = 5;\n\n/**\n * \\brief The interval to display topics during execution.\n */\nsize_t INTERVAL = 10;\n\n\n/**\n * \\brief The interval to compute & display the likelihood\n */\nsize_t LIK_INTERVAL = 5;\n\n/**\n * \\brief The global variable storing the global topic count across\n * all machines.  This is maintained periodically using aggregation.\n */\nfactor_type GLOBAL_TOPIC_COUNT;\n\n/**\n * \\brief A dictionary of words used to print the top words during\n * execution.\n */\nstd::vector DICTIONARY;\n\n/**\n * \\brief The maximum occurences allowed for an individual term-doc\n * pair. (edge data)\n */\nsize_t MAX_COUNT = 100;\n\n\n/**\n * \\brief The time to run until the first sample is taken.  If less\n * than zero then the sampler will run indefinitely.\n */\nfloat BURNIN = -1;\n\n/**\n * \\brief The json top word struct contains the current set of top\n * words for each topic encoded in the form of a json string.\n */\nstruct top_words_type {\n  graphlab::mutex lock;\n  std::string json_string;\n  top_words_type() : \n    json_string(\"{\\n\" + json_header_string() + \"\\tvalues: [] \\n }\") { }\n  inline std::string json_header_string() const {\n    return\n      \"\\t\\\"ntopics\\\": \" + graphlab::tostr(NTOPICS) + \",\\n\" +\n      \"\\t\\\"nwords\\\":  \" + graphlab::tostr(NWORDS) + \",\\n\" +\n      \"\\t\\\"ndocs\\\":   \" + graphlab::tostr(NDOCS) + \",\\n\" +\n      \"\\t\\\"ntokens\\\": \" + graphlab::tostr(NTOKENS) + \",\\n\" +\n      \"\\t\\\"alpha\\\":   \" + graphlab::tostr(ALPHA) + \",\\n\" +\n      \"\\t\\\"beta\\\":    \" + graphlab::tostr(BETA) + \",\\n\";\n  } // end of json header string\n} TOP_WORDS;\n\n\n\n/**\n * \\brief This method is called by the web interface to construct and\n * return the word clouds.\n */\nstd::pair\nword_cloud_callback(std::map& varmap) {\n  TOP_WORDS.lock.lock();\n  const std::pair\n    pair(\"text/html\",TOP_WORDS.json_string);\n  TOP_WORDS.lock.unlock();\n  return pair;\n}\n\n\n\n\n/**\n * \\brief Create a token changes event tracker which is reported in\n * the GraphLab metrics dashboard.\n */\nDECLARE_EVENT(TOKEN_CHANGES);\n\n\n// Graph Types\n// ============================================================================\n\n/**\n * \\brief The vertex data represents each term and document in the\n * corpus and contains the counts of tokens in each topic.\n */\nstruct vertex_data {\n  ///! The total number of updates\n  uint32_t nupdates;\n  ///! The total number of changes to adjacent tokens\n  uint32_t nchanges;\n  ///! The count of tokens in each topic\n  factor_type factor;\n  vertex_data() : nupdates(0), nchanges(0), factor(NTOPICS) { }\n  void save(graphlab::oarchive& arc) const {\n    arc << nupdates << nchanges << factor;\n  }\n  void load(graphlab::iarchive& arc) {\n    arc >> nupdates >> nchanges >> factor;\n  }\n}; // end of vertex_data\n\n\n/**\n * \\brief The edge data represents the individual tokens (word,doc)\n * pairs and their assignment to topics.\n */\nstruct edge_data {\n  ///! The number of changes on the last update\n  uint16_t nchanges;\n  ///! The assignment of all tokens\n  assignment_type assignment;\n  edge_data(size_t ntokens = 0) : nchanges(0), assignment(ntokens, NULL_TOPIC) { }\n  void save(graphlab::oarchive& arc) const { arc << nchanges << assignment; }\n  void load(graphlab::iarchive& arc) { arc >> nchanges >> assignment; }\n}; // end of edge_data\n\n\n/**\n * \\brief The LDA graph is a bipartite graph with docs connected to\n * terms if the term occurs in the document.\n *\n * The edges store the number of occurrences of the term in the\n * document as a vector of the assignments of that term in that\n * document to topics.\n *\n * The vertices store the total topic counts.\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n/**\n * \\brief Edge data parser used in graph.load_json\n *\n * Make sure that the edge file list\n * has docids from -2 to -(total #docid) and wordids 0 to (total #words -1)\n */\nbool eparser(edge_data& ed, const std::string& line){\n  const int BASE = 10;\n  char* next_char_ptr = NULL;\n  size_t count = strtoul(line.c_str(), &next_char_ptr, BASE);\n  if(next_char_ptr ==NULL) return false;\n\n  //threshold count\n  count = std::min(count, MAX_COUNT);\n  ed = (edge_data(count));\n  return true;\n}\n\n/**\n * \\brief Vertex data parser used in graph.load_json\n */\nbool vparser(vertex_data& vd, const std::string& line){\n  vd = vertex_data();\n  return true;\n}\n\n\n/**\n * \\brief The graph loader is used by graph.load to parse lines of the\n * text data file.\n *\n * The global variable MAX_COUNT limits the number of tokens that can\n * be constructed on a particular edge.\n *\n * We use the relativley fast boost::spirit parser to parse each line.\n */\nbool graph_loader(graph_type& graph, const std::string& fname,\n                  const std::string& line) {\n  ASSERT_FALSE(line.empty());\n  namespace qi = boost::spirit::qi;\n  namespace ascii = boost::spirit::ascii;\n  namespace phoenix = boost::phoenix;\n\n  graphlab::vertex_id_type doc_id(-1), word_id(-1);\n  size_t count = 0;\n  const bool success = qi::phrase_parse\n    (line.begin(), line.end(),       \n     //  Begin grammar\n     (\n      qi::ulong_[phoenix::ref(doc_id) = qi::_1] >> -qi::char_(',') >>\n      qi::ulong_[phoenix::ref(word_id) = qi::_1] >> -qi::char_(',') >>\n      qi::ulong_[phoenix::ref(count) = qi::_1]\n      )\n     ,\n     //  End grammar\n     ascii::space); \n  if(!success) return false;  \n  // Threshold the count\n  count = std::min(count, MAX_COUNT);\n  // since this is a bipartite graph I need a method to number the\n  // left and right vertices differently.  To accomplish I make sure\n  // all vertices have non-zero ids and then negate the right vertex.\n  // Unfortunatley graphlab reserves -1 and so we add 2 and negate.\n  doc_id += 2;\n  ASSERT_GT(doc_id, 1);\n  doc_id = -doc_id;\n  ASSERT_NE(doc_id, word_id);\n  // Create an edge and add it to the graph\n  graph.add_edge(doc_id, word_id, edge_data(count));\n  return true; // successful load\n}; // end of graph loader\n\n\n\n\n\n/**\n * \\brief Determine if the given vertex is a word vertex or a doc\n * vertex.\n *\n * For simplicity we connect docs --> words and therefore if a vertex\n * has in edges then it is a word.\n */\ninline bool is_word(const graph_type::vertex_type& vertex) {\n  return vertex.num_in_edges() > 0 ? 1 : 0;\n}\n\n\n/**\n * \\brief Determine if the given vertex is a doc vertex\n *\n * For simplicity we connect docs --> words and therefore if a vertex\n * has out edges then it is a doc\n */\ninline bool is_doc(const graph_type::vertex_type& vertex) {\n  return vertex.num_out_edges() > 0 ? 1 : 0;\n}\n\n/**\n * \\brief return the number of tokens on a particular edge.\n */\ninline size_t count_tokens(const graph_type::edge_type& edge) {\n  return edge.data().assignment.size();\n}\n\n\n/**\n * \\brief Get the other vertex in the edge.\n */\ninline graph_type::vertex_type\nget_other_vertex(const graph_type::edge_type& edge,\n                 const graph_type::vertex_type& vertex) {\n  return vertex.id() == edge.source().id()? edge.target() : edge.source();\n}\n\n\n\n// ========================================================\n// The Collapsed Gibbs Sampler Function\n\n\n\n/**\n * \\brief The gather type for the collapsed Gibbs sampler is used to\n * collect the topic counts on adjacent edges so that the apply\n * function can compute the correct topic counts for the center\n * vertex.\n *\n */\nstruct gather_type {\n  factor_type factor;\n  uint32_t nchanges;\n  gather_type() : nchanges(0) { };\n  gather_type(uint32_t nchanges) : factor(NTOPICS), nchanges(nchanges) { };\n  void save(graphlab::oarchive& arc) const { arc << factor << nchanges; }\n  void load(graphlab::iarchive& arc) { arc >> factor >> nchanges; }\n  gather_type& operator+=(const gather_type& other) {\n    factor += other.factor;\n    nchanges += other.nchanges;\n    return *this;\n  }\n}; // end of gather type\n\n\n\n\n\n\n\n/**\n * \\brief The collapsed Gibbs sampler vertex program updates the topic\n * counts for the center vertex and then draws new topic assignments\n * for each edge durring the scatter phase.\n * \n */\nclass cgs_lda_vertex_program :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\npublic:\n\n  /**\n   * \\brief At termination we want to disable sampling to allow the\n   * correct final counts to be computed.\n   */\n  static bool DISABLE_SAMPLING; \n\n  /** \\brief gather on all edges */\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  } // end of gather_edges\n\n  /**\n   * \\brief Collect the current topic count on each edge.\n   */\n  gather_type gather(icontext_type& context, const vertex_type& vertex,\n                     edge_type& edge) const {\n    gather_type ret(edge.data().nchanges);\n    const assignment_type& assignment = edge.data().assignment;\n    foreach(topic_id_type asg, assignment) {\n      if(asg != NULL_TOPIC) ++ret.factor[asg];\n    }\n    return ret;\n  } // end of gather\n\n\n  /**\n   * \\brief Update the topic count for the center vertex.  This\n   * ensures that the center vertex has the correct topic count before\n   * resampling the topics for each token along each edge.\n   */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& sum) {\n    const size_t num_neighbors = vertex.num_in_edges() + vertex.num_out_edges();\n    ASSERT_GT(num_neighbors, 0);\n    // There should be no new edge data since the vertex program has been cleared\n    vertex_data& vdata = vertex.data();\n    ASSERT_EQ(sum.factor.size(), NTOPICS);\n    ASSERT_EQ(vdata.factor.size(), NTOPICS);\n    vdata.nupdates++;\n    vdata.nchanges = sum.nchanges;\n    vdata.factor = sum.factor;\n  } // end of apply\n\n\n  /**\n   * \\brief Scatter on all edges if the computation is on-going.\n   * Computation stops after bunrin or when disable sampling is set to\n   * true.\n   */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const {\n    return (DISABLE_SAMPLING || (BURNIN > 0 && context.elapsed_seconds() > BURNIN))? \n      graphlab::NO_EDGES : graphlab::ALL_EDGES;\n  }; // end of scatter edges\n\n\n  /**\n   * \\brief Draw new topic assignments for each edge token.\n   *\n   * Note that we exploit the GraphLab caching model here by DIRECTLY\n   * modifying the topic counts of adjacent vertices.  Making the\n   * changes immediately visible to any adjacent vertex programs\n   * running on the same machine.  However, these changes will be\n   * overwritten during the apply step and are only used to accelerate\n   * sampling.  This is a potentially dangerous violation of the\n   * abstraction and should be taken with caution.  In our case all\n   * vertex topic counts are preallocated and atomic operations are\n   * used.  In addition during the sampling phase we must be careful\n   * to guard against potentially negative temporary counts.\n   */\n  void scatter(icontext_type& context, const vertex_type& vertex,\n               edge_type& edge) const {\n    factor_type& doc_topic_count =  is_doc(edge.source()) ?\n      edge.source().data().factor : edge.target().data().factor;\n    factor_type& word_topic_count = is_word(edge.source()) ?\n      edge.source().data().factor : edge.target().data().factor;\n    ASSERT_EQ(doc_topic_count.size(), NTOPICS);\n    ASSERT_EQ(word_topic_count.size(), NTOPICS);\n    // run the actual gibbs sampling\n    std::vector prob(NTOPICS);\n    assignment_type& assignment = edge.data().assignment;\n    edge.data().nchanges = 0;\n    foreach(topic_id_type& asg, assignment) {\n      const topic_id_type old_asg = asg;\n      if(asg != NULL_TOPIC) { // construct the cavity\n        --doc_topic_count[asg];\n        --word_topic_count[asg];\n        --GLOBAL_TOPIC_COUNT[asg];\n      }\n      for(size_t t = 0; t < NTOPICS; ++t) {\n        const double n_dt =\n          std::max(count_type(doc_topic_count[t]), count_type(0));\n        const double n_wt =\n          std::max(count_type(word_topic_count[t]), count_type(0));\n        const double n_t  =\n          std::max(count_type(GLOBAL_TOPIC_COUNT[t]), count_type(0));\n        prob[t] = (ALPHA + n_dt) * (BETA + n_wt) / (BETA * NWORDS + n_t);\n      }\n      asg = graphlab::random::multinomial(prob);\n      // asg = std::max_element(prob.begin(), prob.end()) - prob.begin();\n      ++doc_topic_count[asg];\n      ++word_topic_count[asg];\n      ++GLOBAL_TOPIC_COUNT[asg];\n      if(asg != old_asg) {\n        ++edge.data().nchanges;\n        INCREMENT_EVENT(TOKEN_CHANGES,1);\n      }\n    } // End of loop over each token\n    // singla the other vertex\n    context.signal(get_other_vertex(edge, vertex));\n  } // end of scatter function\n\n}; // end of cgs_lda_vertex_program\n\n\nbool cgs_lda_vertex_program::DISABLE_SAMPLING = false;\n\n\n/**\n * \\brief The icontext type associated with the cgs_lda_vertex program\n * is needed for all aggregators.\n */\ntypedef cgs_lda_vertex_program::icontext_type icontext_type;\n\n\n// ========================================================\n// Aggregators\n\n\n/**\n * \\brief The topk aggregator is used to periodically compute and\n * display the topk most common words in each topic.\n *\n * The number of words is determined by the global variable \\ref TOPK\n * and the interval is determined by the global variable \\ref INTERVAL.\n *\n */\nclass topk_aggregator {\n  typedef std::pair cw_pair_type;\nprivate:\n  std::vector< std::set > top_words;\n  size_t nchanges, nupdates;\npublic:\n  topk_aggregator(size_t nchanges = 0, size_t nupdates = 0) :\n    nchanges(nchanges), nupdates(nupdates) { }\n\n  void save(graphlab::oarchive& arc) const { arc << top_words << nchanges; }\n  void load(graphlab::iarchive& arc) { arc >> top_words >> nchanges; }\n\n\n  topk_aggregator& operator+=(const topk_aggregator& other) {\n    nchanges += other.nchanges;\n    nupdates += other.nupdates;\n    if(other.top_words.empty()) return *this;\n    if(top_words.empty()) top_words.resize(NTOPICS);\n    for(size_t i = 0; i < top_words.size(); ++i) {\n      // Merge the topk\n      top_words[i].insert(other.top_words[i].begin(),\n                          other.top_words[i].end());\n      // Remove excess elements\n      while(top_words[i].size() > TOPK)\n        top_words[i].erase(top_words[i].begin());\n    }\n    return *this;\n  } // end of operator +=\n\n  static topk_aggregator map(icontext_type& context,\n                             const graph_type::vertex_type& vertex) {\n    topk_aggregator ret_value;\n    const vertex_data& vdata = vertex.data();\n    ret_value.nchanges = vdata.nchanges;\n    ret_value.nupdates = vdata.nupdates;\n    if(is_word(vertex)) {\n      const graphlab::vertex_id_type wordid = vertex.id();\n      ret_value.top_words.resize(vdata.factor.size());\n      for(size_t i = 0; i < vdata.factor.size(); ++i) {\n        const cw_pair_type pair(vdata.factor[i], wordid);\n        ret_value.top_words[i].insert(pair);\n      }\n    }\n    return ret_value;\n  } // end of map function\n\n\n  static void finalize(icontext_type& context,\n                       const topk_aggregator& total) {\n    if(context.procid() != 0) return;\n    std::string json = \"{\\n\"+ TOP_WORDS.json_header_string() +\n      \"\\t\\\"values\\\": [\\n\";\n    for(size_t i = 0; i < total.top_words.size(); ++i) {\n      std::cout << \"Topic \" << i << \": \";\n      json += \"\\t[\\n\";\n      size_t counter = 0;\n      rev_foreach(cw_pair_type pair, total.top_words[i])  {\n      ASSERT_LT(pair.second, DICTIONARY.size());\n        json += \"\\t\\t[\\\"\" + DICTIONARY[pair.second] + \"\\\", \" +\n          graphlab::tostr(pair.first) + \"]\";\n        if(++counter < total.top_words[i].size()) json += \", \";\n        json += '\\n';\n        std::cout << DICTIONARY[pair.second]\n                  << \"(\" << pair.first << \")\" << \", \";\n        // std::cout << DICTIONARY[pair.second] << \",  \";\n      }\n      json += \"\\t]\";\n      if(i+1 < total.top_words.size()) json += \", \";\n      json += '\\n';\n      std::cout << std::endl;\n    }\n    json += \"]}\";\n    // Post the change to the global variable\n    TOP_WORDS.lock.lock();\n    TOP_WORDS.json_string.swap(json);\n    TOP_WORDS.lock.unlock();\n\n    std::cout << \"\\nNumber of token changes: \" << total.nchanges << std::endl;\n    std::cout << \"\\nNumber of updates:       \" << total.nupdates << std::endl;\n  } // end of finalize\n}; // end of topk_aggregator struct\n\n\n\n/**\n * \\brief The global counts aggregator computes the total number of\n * tokens in each topic across all words and documents and then\n * updates the \\ref GLOBAL_TOPIC_COUNT variable.\n *\n */\nstruct global_counts_aggregator {\n  typedef graph_type::vertex_type vertex_type;\n  static factor_type map(icontext_type& context, const vertex_type& vertex) {\n    return vertex.data().factor;\n  } // end of map function\n\n  static void finalize(icontext_type& context, const factor_type& total) {\n    size_t sum = 0;\n    for(size_t t = 0; t < total.size(); ++t) {\n      GLOBAL_TOPIC_COUNT[t] =\n        std::max(count_type(total[t]/2), count_type(0));\n      sum += GLOBAL_TOPIC_COUNT[t];\n    }\n    context.cout() << \"Total Tokens: \" << sum << std::endl;\n  } // end of finalize\n}; // end of global_counts_aggregator struct\n\n\n/**\n * Computing log_gamma can be a bit slow so this class precomptues \n * log gamma for a subset of values.\n */\nclass log_gamma {\n  double offset;\n  std::vector values;\npublic:\n  log_gamma(): offset(1.0) {}\n\n  void init(const double& new_offset, const size_t& buckets) {\n    using boost::math::lgamma;\n    ASSERT_GT(offset, 0.0);\n    values.resize(buckets);\n    offset = new_offset;\n    for(size_t i = 0; i < values.size(); ++i) {\n      values[i] = lgamma(i + offset);\n    }\n  }\n\n  double operator()(const count_type& index) const {\n    using boost::math::lgamma;\n    if(index < values.size() && index >= 0) { return values[index]; }\n    else { return lgamma(index + offset); }\n  }\n\n};\n\nlog_gamma ALPHA_LGAMMA;\nlog_gamma BETA_LGAMMA;\n\n/**\n * \\brief The Likelihood aggregators maintains the current estimate of\n * the log-likelihood of the current token assignments.\n *\n *  llik_words_given_topics = ...\n *    ntopics * (gammaln(nwords * beta) - nwords * gammaln(beta)) - ...\n *    sum_t(gammaln( n_t + nwords * beta)) +\n *    sum_w(sum_t(gammaln(n_wt + beta)));\n *\n *  llik_topics = ...\n *    ndocs * (gammaln(ntopics * alpha) - ntopics * gammaln(alpha)) + ...\n *    sum_d(sum_t(gammaln(n_td + alpha)) - gammaln(sum_t(n_td) + ntopics * alpha));\n *\n * Latex formulation:\n *\n    \\mathcal{L}( w | z) & = T * \\left( \\log\\Gamma(W * \\beta) - W * \\log\\Gamma(\\beta) \\right) + \\\\\n    & \\sum_{t} \\left( \\left(\\sum_{w} \\log\\Gamma(N_{wt} + \\beta)\\right) - \n           \\log\\Gamma\\left( W * \\beta + \\sum_{w} N_{wt}  \\right) \\right) \\\\\n    & = T * \\left( \\log\\Gamma(W * \\beta) - W * \\log\\Gamma(\\beta) \\right) - \n        \\sum_{t} \\log\\Gamma\\left( W * \\beta + N_{t}  \\right) + \\\\\n    & \\sum_{w} \\sum_{t} \\log\\Gamma(N_{wt} + \\beta)   \\\\\n    \\\\\n    \\mathcal{L}(z) & = D * \\left(\\log\\Gamma(T * \\alpha) - T * \\log\\Gamma(\\alpha) \\right) + \\\\\n    & \\sum_{d} \\left( \\left(\\sum_{t}\\log\\Gamma(N_{td} + \\alpha)\\right) -  \n        \\log\\Gamma\\left( T * \\alpha + \\sum_{t} N_{td} \\right) \\right) \\\\\n    \\\\\n    \\mathcal{L}(w,z) & = \\mathcal{L}(w | z) + \\mathcal{L}(z)\n *\n */\nclass likelihood_aggregator : public graphlab::IS_POD_TYPE {\n  typedef graph_type::vertex_type vertex_type;\n  double lik_words_given_topics;\n  double lik_topics;\npublic:\n  likelihood_aggregator() : lik_words_given_topics(0), lik_topics(0) { }\n\n  likelihood_aggregator& operator+=(const likelihood_aggregator& other) {\n    lik_words_given_topics += other.lik_words_given_topics;\n    lik_topics += other.lik_topics;\n    return *this;\n  } // end of operator +=\n\n  static likelihood_aggregator\n  map(icontext_type& context, const vertex_type& vertex) {\n    // using boost::math::lgamma;\n    const factor_type& factor = vertex.data().factor;\n    ASSERT_EQ(factor.size(), NTOPICS);\n    likelihood_aggregator ret;\n    if(is_word(vertex)) {\n      for(size_t t = 0; t < NTOPICS; ++t) {\n        const count_type value = std::max(count_type(factor[t]), count_type(0));\n        //ret.lik_words_given_topics += lgamma(value + BETA);\n        ret.lik_words_given_topics += BETA_LGAMMA(value);\n      }\n    } else {  ASSERT_TRUE(is_doc(vertex));\n      double ntokens_in_doc = 0;\n      for(size_t t = 0; t < NTOPICS; ++t) {\n        const count_type value = std::max(count_type(factor[t]), count_type(0));\n        //ret.lik_topics += lgamma(value + ALPHA);\n        ret.lik_topics += ALPHA_LGAMMA(value);\n        ntokens_in_doc += value;\n      }\n      ret.lik_topics -= lgamma(ntokens_in_doc + NTOPICS * ALPHA);\n    }\n    return ret;\n  } // end of map function\n\n  static void finalize(icontext_type& context, const likelihood_aggregator& total) {\n    using boost::math::lgamma;\n    // Address the global sum terms\n    double denominator = 0;\n    for(size_t t = 0; t < NTOPICS; ++t) {\n      const count_type value = \n        std::max(count_type(GLOBAL_TOPIC_COUNT[t]), count_type(0));\n      denominator += lgamma(value + NWORDS * BETA);\n    } // end of for loop\n\n    const double lik_words_given_topics =\n      NTOPICS * (lgamma(NWORDS * BETA) - NWORDS * lgamma(BETA)) -\n      denominator + total.lik_words_given_topics;\n\n    const double lik_topics =\n      NDOCS * (lgamma(NTOPICS * ALPHA) - NTOPICS * lgamma(ALPHA)) +\n      total.lik_topics;\n\n    const double lik = lik_words_given_topics + lik_topics;\n    context.cout() << \"Likelihood: \" << lik << std::endl;\n  } // end of finalize\n}; // end of likelihood_aggregator struct\n\n\n\n/**\n * \\brief The selective signal functions are used to signal only the\n * vertices corresponding to words or documents.  This is done by\n * using the iengine::map_reduce_vertices function.\n */\nstruct signal_only {\n  /**\n   * \\brief Signal only the document vertices and skip the word\n   * vertices.\n   */ \n  static graphlab::empty\n  docs(icontext_type& context, const graph_type::vertex_type& vertex) {\n    if(is_doc(vertex)) context.signal(vertex);\n    return graphlab::empty();\n  } // end of signal_docs\n \n /**\n  * \\brief Signal only the word vertices and skip the document\n  * vertices.\n  */\n  static graphlab::empty\n  words(icontext_type& context, const graph_type::vertex_type& vertex) {\n    if(is_word(vertex)) context.signal(vertex);\n    return graphlab::empty();\n  } // end of signal_words\n}; // end of selective_only\n\n\n\n\n\n/**\n * \\brief This function is used to load and then initialize the data\n * graph (corpus) from a folder or file.\n * \n * The graph can be in either json form constructed using the graph\n * builder tools or in raw text form.  The raw text format contains a\n * token on each line of each file in the format:\n *\n \\verbatim\n   \n          ...\n \\endverbatim\n *\n * for example:\n \\verbatim\n    0    0     2\n    0    4     1\n    0    2     3\n \\endverbatim\n * \n * implies that document zero contains word zero twice, word 4 once,\n * and word two three times.\n *\n * If a dictionary is used it is important that each word id\n * correspond to the index in the dictionary file (starting at zero).\n *\n * Once loaded the total number of words, documents, and tokens is\n * counted and saved to global variables which are read during the\n * execution of the sampler.\n *\n * \\param [in] dc The distributed control object used to coordinate\n * between machines.\n *\n * \\param [in,out] graph The graph object that is initialized.\n * \n * \\param [in] corpus_dir The directory or file containing the graph\n * data.  The corpus directory can reside on hdfs in which case the\n * path should begin with \"hdfs://namenode\".  In addition the file(s)\n * may be gzipped and therefore must end in \".gz\".\n *\n * \\param [in] load_json Whether the graph data is in text format or\n * preprocessed json format using the graph builder tools.\n */\nbool load_and_initialize_graph(graphlab::distributed_control& dc,\n                               graph_type& graph,\n                               const std::string& corpus_dir,\n                               const std::string& format\t\t\t       \n\t\t\t       ) {\n  dc.cout() << \"Loading graph.\" << std::endl;\n  graphlab::timer timer; timer.start();\n\n  if(format==\"matrix\"){\n      dc.cout() << \"matrix format\" << std::endl;\n      graph.load(corpus_dir, graph_loader);\n  // } else if(format==\"json\"){\n  //     dc.cout() << \"json format\" << std::endl;\n  //     graph.load_json(corpus_dir, false, eparser, vparser);\n  // } else if(format==\"json-gzip\"){\n  //     dc.cout() <<\"json gzip format\" << std::endl;\n  //     graph.load_json(corpus_dir, true, eparser, vparser);\n  }else{\n      dc.cout() << \"Non supported format. See --help\" << std::endl;\n      return false;\n  }\n\n  dc.cout() << \"Finalizing graph.\" << std::endl;\n  timer.start();\n  graph.finalize();\n  dc.cout() << \"Finalizing graph. Finished in \"\n            << timer.current_time() << \" seconds.\" << std::endl;\n\n  dc.cout() << \"Computing number of words and documents.\" << std::endl;\n  NWORDS = graph.map_reduce_vertices(is_word);\n  NDOCS = graph.map_reduce_vertices(is_doc);\n  NTOKENS = graph.map_reduce_edges(count_tokens);\n\n\n  dc.cout() << \"Number of words:     \" << NWORDS  << std::endl;\n  dc.cout() << \"Number of docs:      \" << NDOCS   << std::endl;\n  dc.cout() << \"Number of tokens:    \" << NTOKENS << std::endl;\n\n  ASSERT_GT(NWORDS, 0);\n  ASSERT_GT(NDOCS, 0);\n  ASSERT_GT(NTOKENS, 0);\n\n\n  // Prepare the json struct with the word counts\n  TOP_WORDS.lock.lock();\n  TOP_WORDS.json_string = \"{\\n\" + TOP_WORDS.json_header_string() +\n    \"\\t\\\"values\\\": [] \\n }\";\n  TOP_WORDS.lock.unlock();\n  return true;\n} // end of load and initialize graph\n\n\n\n/**\n * \\brief Load the dictionary global variable from the file containing\n * the terms (one term per line).\n *\n * Note that while graphs can be loaded from multiple files the\n * dictionary must be in a single file.  The dictionary is loaded\n * entirely into memory and used to display word clouds and the top\n * terms in each topic.\n *\n * \\param [in] fname the file containing the dictionary data.  The\n * data can be located on HDFS and can also be gzipped (must end in\n * \".gz\").\n * \n */\nbool load_dictionary(const std::string& fname)  {\n  // std::cout << \"staring load on: \"\n  //           << graphlab::get_local_ip_as_str() << std::endl;\n  const bool gzip = boost::ends_with(fname, \".gz\");\n  // test to see if the graph_dir is an hadoop path\n  if(boost::starts_with(fname, \"hdfs://\")) {\n    graphlab::hdfs hdfs;\n    graphlab::hdfs::fstream in_file(hdfs, fname);\n    boost::iostreams::filtering_stream fin;\n    fin.set_auto_close(false);\n    if(gzip) fin.push(boost::iostreams::gzip_decompressor());\n    fin.push(in_file);\n    if(!fin.good()) {\n      logstream(LOG_ERROR) << \"Error loading dictionary: \"\n                           << fname << std::endl;\n      return false;\n    }\n    std::string term;\n    while(std::getline(fin,term).good()) DICTIONARY.push_back(term);\n    if (gzip) fin.pop();\n    fin.pop();\n    in_file.close();\n  } else {\n    std::cout << \"opening: \" << fname << std::endl;\n    std::ifstream in_file(fname.c_str(),\n                          std::ios_base::in | std::ios_base::binary);\n    boost::iostreams::filtering_stream fin;\n    if (gzip) fin.push(boost::iostreams::gzip_decompressor());\n    fin.push(in_file);\n    if(!fin.good() || !fin.good()) {\n      logstream(LOG_ERROR) << \"Error loading dictionary: \"\n                           << fname << std::endl;\n      return false;\n    }\n    std::string term;\n    std::cout << \"Loooping\" << std::endl;\n    while(std::getline(fin, term).good()) DICTIONARY.push_back(term);\n    if (gzip) fin.pop();\n    fin.pop();\n    in_file.close();\n  } // end of else\n  // std::cout << \"Finished load on: \"\n  //           << graphlab::get_local_ip_as_str() << std::endl;\n  std::cout << \"Dictionary Size: \" << DICTIONARY.size() << std::endl;\n  return true;\n} // end of load dictionary\n\n\n\n\nstruct count_saver {\n  bool save_words;\n  count_saver(bool save_words) : save_words(save_words) { }\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    // Skip saving vertex data if the vertex type is not consistent\n    // with the save type\n    if((save_words && is_doc(vertex)) ||\n       (!save_words && is_word(vertex))) return \"\";\n    // Proceed to save\n    std::stringstream strm;\n    if(save_words) {\n      const graphlab::vertex_id_type vid = vertex.id();\n      strm << vid << '\\t';\n    } else { // save documents\n      const graphlab::vertex_id_type vid = (-vertex.id()) - 2;\n      strm << vid << '\\t';\n    }\n    const factor_type& factor = vertex.data().factor;\n    for(size_t i = 0; i < factor.size(); ++i) { \n      strm << factor[i];\n      if(i+1 < factor.size()) strm << '\\t';\n    }\n    strm << '\\n';\n    return strm.str();\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\"; //nop\n  }\n}; // end of prediction_saver\n\n\n\n\n\n\n\n/**\n * \\brief The omni engine type is used to allow switching between\n * synchronous and asynchronous computation. \n */\ntypedef graphlab::omni_engine engine_type;\n\n\n\n\n\n\n\n\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  //  INITIALIZE_EVENT_LOG(dc);\n  ADD_CUMULATIVE_EVENT(TOKEN_CHANGES, \"Token Changes\", \"Changes\");\n\n  // Parse command line options -----------------------------------------------\n  const std::string description =\n    \"\\n=========================================================================\\n\"\n    \"The Collapsed Gibbs Sampler for the LDA model implements\\n\"\n    \"a highly asynchronous version of parallel LDA in which document\\n\"\n    \"and word counts are maintained in an eventually consistent\\n\"\n    \"manner.\\n\"\n    \"\\n\"\n    \"The standard usage is: \\n\"\n    \"\\t./cgs_lda --dictionary dictionary.txt --corpus doc_word_count.tsv\\n\"\n    \"where dictionary.txt contains: \\n\"\n    \"\\taaa \\n\\taaai \\n\\tabalone \\n\\t   ... \\n\"\n    \"each line number corresponds to wordid (i.e aaa has wordid=0)\\n\\n\"\n    \"and doc_word_count.tsv is formatted   :\\n\"\n    \"(where wordid is indexed starting from zero and docid are positive integers)\\n\"\n    \"\\t0\\t0\\t3\\n\"\n    \"\\t0\\t5\\t1\\n\"\n    \"\\t ...\\n\\n\"\n    \"For JSON format, make sure docid are negative integers index starting from -2 \\n\\n\"\n    \"To learn more about the NLP package and its applications visit\\n\\n\"\n    \"\\t\\t http://graphlab.org \\n\\n\"\n    \"Additional Options\";\n  graphlab::command_line_options clopts(description);\n  std::string corpus_dir;\n  std::string dictionary_fname;\n  std::string doc_dir;\n  std::string word_dir;\n  std::string exec_type = \"asynchronous\";\n  std::string format = \"matrix\";\n  \n  clopts.attach_option(\"dictionary\", dictionary_fname,\n                       \"The file containing the list of unique words\");\n  clopts.attach_option(\"engine\", exec_type, \n                       \"The engine type synchronous or asynchronous\");\n  clopts.attach_option(\"corpus\", corpus_dir,\n                       \"The directory or file containing the corpus data.\");\n  clopts.add_positional(\"corpus\");\n  clopts.attach_option(\"ntopics\", NTOPICS,\n                       \"Number of topics to use.\");\n  clopts.attach_option(\"alpha\", ALPHA,\n                       \"The document hyper-prior\");\n  clopts.attach_option(\"beta\", BETA,\n                       \"The word hyper-prior\");\n  clopts.attach_option(\"topk\", TOPK,\n                       \"The number of words to report\");\n  clopts.attach_option(\"interval\", INTERVAL,\n                       \"statistics reporting interval (in seconds)\");\n  clopts.attach_option(\"lik_interval\", LIK_INTERVAL,\n                       \"likelihood reporting interval (in seconds)\");\n  clopts.attach_option(\"max_count\", MAX_COUNT,\n                       \"The maximum number of occurences of a word in a document.\");\n  clopts.attach_option(\"format\", format,\n                       \"Formats: matrix,json,json-gzip\");\n  clopts.attach_option(\"burnin\", BURNIN, \n                       \"The time in second to run until a sample is collected. \"\n                       \"If less than zero the sampler runs indefinitely.\");\n  clopts.attach_option(\"doc_dir\", doc_dir,\n                       \"The output directory to save the final document counts.\");\n  clopts.attach_option(\"word_dir\", word_dir,\n                       \"The output directory to save the final words counts.\");\n\n\n  if(!clopts.parse(argc, argv)) {\n    graphlab::mpi_tools::finalize();\n    return clopts.is_set(\"help\")? EXIT_SUCCESS : EXIT_FAILURE;\n  }\n\n  if(dictionary_fname.empty()) {\n    logstream(LOG_WARNING) << \"No dictionary file was provided.\" << std::endl\n                           << \"Top k words will not be estimated.\" << std::endl;\n  }\n\n  if(corpus_dir.empty()) {\n    logstream(LOG_ERROR) << \"No corpus file was provided.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  // Start the webserver\n  graphlab::launch_metric_server();\n  graphlab::add_metric_server_callback(\"wordclouds\", word_cloud_callback);\n\n\n  ///! Initialize global variables\n  GLOBAL_TOPIC_COUNT.resize(NTOPICS);\n  if(!dictionary_fname.empty()) {\n    const bool success = load_dictionary(dictionary_fname);\n    if(!success) {\n      logstream(LOG_ERROR) << \"Error loading dictionary.\" << std::endl;\n      return EXIT_FAILURE;\n    }\n  }\n\n  if(ALPHA <= 0) {\n    logstream(LOG_ERROR) \n      << \"Alpha must be positive (alpha=\" << ALPHA << \")!\"  << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  if(BETA <= 0) {\n    logstream(LOG_ERROR) \n      << \"Beta must be positive (beta=\" << BETA << \")!\"  << std::endl;\n    return EXIT_FAILURE;\n  }\n   \n  /// Initialize the log_gamma precached calculations.\n  ALPHA_LGAMMA.init(ALPHA, 100000);\n  BETA_LGAMMA.init(BETA, 1000000);\n\n\n  ///! load the graph\n  graph_type graph(dc, clopts);\n  {\n    const bool success = \n      load_and_initialize_graph(dc, graph, corpus_dir, format);\n    if(!success) {\n      logstream(LOG_ERROR) << \"Error loading graph.\" << std::endl;\n      return EXIT_FAILURE;\n    }\n  }\n\n\n  const size_t ntokens = graph.map_reduce_edges(count_tokens);\n  dc.cout() << \"Total tokens: \" << ntokens << std::endl;\n\n\n\n  engine_type engine(dc, graph, exec_type, clopts);\n  ///! Add an aggregator\n  if(!DICTIONARY.empty()) {\n    const bool success =\n      engine.add_vertex_aggregator\n      (\"topk\", topk_aggregator::map, topk_aggregator::finalize) &&\n      engine.aggregate_periodic(\"topk\", INTERVAL);\n    ASSERT_TRUE(success);\n  }\n\n  { // Add the Global counts aggregator\n    const bool success =\n      engine.add_vertex_aggregator\n      (\"global_counts\", \n       global_counts_aggregator::map, \n       global_counts_aggregator::finalize) &&\n      engine.aggregate_periodic(\"global_counts\", 5);\n    ASSERT_TRUE(success);\n  }\n  \n  { // Add the likelihood aggregator\n    const bool success =\n      engine.add_vertex_aggregator\n      (\"likelihood\", \n       likelihood_aggregator::map, \n       likelihood_aggregator::finalize) &&\n      engine.aggregate_periodic(\"likelihood\", LIK_INTERVAL);\n    ASSERT_TRUE(success);\n  }\n\n  ///! schedule only documents\n  dc.cout() << \"Running The Collapsed Gibbs Sampler\" << std::endl;\n  engine.map_reduce_vertices(signal_only::docs);\n  graphlab::timer timer;\n  // Enable sampling\n  cgs_lda_vertex_program::DISABLE_SAMPLING = false;\n  // Run the engine\n  engine.start();\n  // Finalize the counts\n  cgs_lda_vertex_program::DISABLE_SAMPLING = true;\n  engine.signal_all();\n  engine.start();\n  \n  const double runtime = timer.current_time();\n  dc.cout()\n    << \"----------------------------------------------------------\" << std::endl\n    << \"Final Runtime (seconds):   \" << runtime\n    << std::endl\n    << \"Updates executed: \" << engine.num_updates() << std::endl\n    << \"Update Rate (updates/second): \"\n    << engine.num_updates() / runtime << std::endl;\n  \n  \n  \n  if(!word_dir.empty()) {\n    // save word topic counts\n    const bool gzip_output = false;\n    const bool save_vertices = true;\n    const bool save_edges = false;\n    const size_t threads_per_machine = 2;\n    const bool save_words = true;\n    graph.save(word_dir, count_saver(save_words),\n               gzip_output, save_vertices, \n               save_edges, threads_per_machine);\n  }\n\n  \n  if(!doc_dir.empty()) {\n    // save doc topic counts\n    const bool gzip_output = false;\n    const bool save_vertices = true;\n    const bool save_edges = false;\n    const size_t threads_per_machine = 2;\n    const bool save_words = false;\n    graph.save(doc_dir, count_saver(save_words),\n               gzip_output, save_vertices, \n               save_edges, threads_per_machine);\n\n  }\n\n\n  graphlab::stop_metric_server_on_eof();\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n\n\n} // end of main\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
  },
  {
    "path": "toolkits/topic_modeling/cgs_lda_mimno_experimental.cpp",
    "content": "/*\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n/**\n * \\file cgs_lda.cpp\n *\n * \\brief This file contains a GraphLab based implementation of the\n * Collapsed Gibbs Sampler (CGS) for the Latent Dirichlet Allocation\n * (LDA) model.\n *\n * \n *\n * \\author Joseph Gonzalez, Diana Hu\n */\n\n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n\n\n// Global Types\n// ============================================================================\ntypedef int count_type;\n\n\n/**\n * \\brief The factor type is used to store the counts of tokens in\n * each topic for words, documents, and assignments.\n *\n * Atomic counts are used because we violate the abstraction by\n * modifying adjacent vertex data on scatter.  As a consequence\n * multiple threads on the same machine may try to update the same\n * vertex data at the same time.  The graphlab::atomic type ensures\n * that multiple increments are serially consistent.\n */\ntypedef std::vector factor_type;\n\n\n/**\n * \\brief We use the factor type in accumulators and so we define an\n * operator+=\n */\ninline factor_type& operator+=(factor_type& lvalue,\n                               const factor_type& rvalue) {\n  if(!rvalue.empty()) {\n    if(lvalue.empty()) lvalue = rvalue;\n    else {\n      for(size_t t = 0; t < lvalue.size(); ++t) lvalue[t] += rvalue[t];\n    }\n  }\n  return lvalue;\n} // end of operator +=\n\n// We include the rest of GraphLab after we define the operator+= for\n// vector.\n#include \n#include \n\n\n\n\n/**\n * \\brief The latent topic id of a token is the smallest reasonable\n * type.\n */\ntypedef uint16_t topic_id_type;\n\n// We require a null topic to represent the topic assignment for\n// tokens that have not yet been assigned.\n#define NULL_TOPIC (topic_id_type(-1))\n\n\n\n/**\n * \\brief The assignment type is used on each edge to store the\n * assignments of each token.  There can be several occurrences of the\n * same word in a given document and so a vector is used to store the\n * assignments of each occurrence.\n */\ntypedef std::vector< topic_id_type > assignment_type;\n\n\n// Global Variables\n// ============================================================================\n\n/**\n * \\brief The alpha parameter determines the sparsity of topics for\n * each document.\n */\nfloat ALPHA = 1;\n\n/**\n * \\brief the Beta parameter determines the sparsity of words in each\n * document.\n */\nfloat BETA = 0.1;\n\n/**\n * \\brief the total number of topics to uses\n */\nsize_t NTOPICS = 50;\n\n/**\n * \\brief The total number of words in the dataset.\n */\nsize_t NWORDS = 0;\n\n/**\n * \\brief The total number of docs in the dataset.\n */\nsize_t NDOCS = 0;\n\n/**\n * \\brief The total number of tokens in the corpus\n */\nsize_t NTOKENS = 0;\n\n\n/**\n * \\brief The number of top words to display during execution (from\n * each topic).\n */\nsize_t TOPK = 5;\n\n/**\n * \\brief The interval to display topics during execution.\n */\nsize_t INTERVAL = 10;\n\n/**\n * \\brief The global variable storing the global topic count across\n * all machines.  This is maintained periodically using aggregation.\n */\nfactor_type GLOBAL_TOPIC_COUNT;\n\n/**\n * \\brief A dictionary of words used to print the top words during\n * execution.\n */\nstd::vector DICTIONARY;\n\n/**\n * \\brief The maximum occurences allowed for an individual term-doc\n * pair. (edge data)\n */\nsize_t MAX_COUNT = 100;\n\n\n/**\n * \\brief The time to run until the first sample is taken.  If less\n * than zero then the sampler will run indefinitely.\n */\nfloat BURNIN = -1;\n\n\nfloat MIMNO_S;\n\n/**\n * \\brief The json top word struct contains the current set of top\n * words for each topic encoded in the form of a json string.\n */\nstruct top_words_type {\n  graphlab::mutex lock;\n  std::string json_string;\n  top_words_type() : \n    json_string(\"{\\n\" + json_header_string() + \"\\tvalues: [] \\n }\") { }\n  inline std::string json_header_string() const {\n    return\n      \"\\t\\\"ntopics\\\": \" + graphlab::tostr(NTOPICS) + \",\\n\" +\n      \"\\t\\\"nwords\\\":  \" + graphlab::tostr(NWORDS) + \",\\n\" +\n      \"\\t\\\"ndocs\\\":   \" + graphlab::tostr(NDOCS) + \",\\n\" +\n      \"\\t\\\"ntokens\\\": \" + graphlab::tostr(NTOKENS) + \",\\n\" +\n      \"\\t\\\"alpha\\\":   \" + graphlab::tostr(ALPHA) + \",\\n\" +\n      \"\\t\\\"beta\\\":    \" + graphlab::tostr(BETA) + \",\\n\";\n  } // end of json header string\n} TOP_WORDS;\n\n\n\n/**\n * \\brief This method is called by the web interface to construct and\n * return the word clouds.\n */\nstd::pair\nword_cloud_callback(std::map& varmap) {\n  TOP_WORDS.lock.lock();\n  const std::pair\n    pair(\"text/html\",TOP_WORDS.json_string);\n  TOP_WORDS.lock.unlock();\n  return pair;\n}\n\n\n\n\n/**\n * \\brief Create a token changes event tracker which is reported in\n * the GraphLab metrics dashboard.\n */\nDECLARE_EVENT(TOKEN_CHANGES);\n\n\n// Graph Types\n// ============================================================================\n\n/**\n * \\brief The vertex data represents each term and document in the\n * corpus and contains the counts of tokens in each topic.\n */\nstruct vertex_data {\n  ///! The total number of updates\n  uint32_t nupdates;\n  ///! The total number of changes to adjacent tokens\n  uint32_t nchanges;\n  ///! The count of tokens in each topic\n  factor_type factor;\n  float MIMNO_R;\n\n   vertex_data() : nupdates(0), nchanges(0), factor(NTOPICS),MIMNO_R(0) { }\n  void save(graphlab::oarchive& arc) const {\n\n    arc << nupdates << nchanges << MIMNO_R; \n    uint16_t ni = 0;\n    for (size_t i = 0;i < factor.size(); ++i) {\n      ni += (factor[i] > 0);\n    }\n    arc << ni;\n    for (size_t i = 0;i < factor.size(); ++i) {\n      if (factor[i] > 0) {\n        arc << uint16_t(i) << factor[i];\n      }\n    }\n  }\n  void load(graphlab::iarchive& arc) {\n    arc >> nupdates >> nchanges >> MIMNO_R; \n    for (size_t i = 0;i < factor.size(); ++i) factor[i] = 0;\n    uint16_t ni;\n    arc >> ni; \n    for (uint16_t i = 0;i < ni; ++i) {\n      uint16_t u; arc >> u;\n      arc >> factor[u];\n    }\n\n  }\n}; // end of vertex_data\n\n\n/**\n * \\brief The edge data represents the individual tokens (word,doc)\n * pairs and their assignment to topics.\n */\nstruct edge_data {\n  ///! The number of changes on the last update\n  uint16_t nchanges;\n  ///! The assignment of all tokens\n  assignment_type assignment;\n  edge_data(size_t ntokens = 0) : nchanges(0), assignment(ntokens, NULL_TOPIC) { }\n  void save(graphlab::oarchive& arc) const { arc << nchanges << assignment; }\n  void load(graphlab::iarchive& arc) { arc >> nchanges >> assignment; }\n}; // end of edge_data\n\n\n/**\n * \\brief The LDA graph is a bipartite graph with docs connected to\n * terms if the term occurs in the document.\n *\n * The edges store the number of occurrences of the term in the\n * document as a vector of the assignments of that term in that\n * document to topics.\n *\n * The vertices store the total topic counts.\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n/**\n * \\brief Edge data parser used in graph.load_json\n *\n * Make sure that the edge file list\n * has docids from -2 to -(total #docid) and wordids 0 to (total #words -1)\n */\nbool eparser(edge_data& ed, const std::string& line){\n  const int BASE = 10;\n  char* next_char_ptr = NULL;\n  size_t count = strtoul(line.c_str(), &next_char_ptr, BASE);\n  if(next_char_ptr ==NULL) return false;\n\n  //threshold count\n  count = std::min(count, MAX_COUNT);\n  ed = (edge_data(count));\n  return true;\n}\n\n/**\n * \\brief Vertex data parser used in graph.load_json\n */\nbool vparser(vertex_data& vd, const std::string& line){\n  vd = vertex_data();\n  return true;\n}\n\n\n/**\n * \\brief The graph loader is used by graph.load to parse lines of the\n * text data file.\n *\n * The global variable MAX_COUNT limits the number of tokens that can\n * be constructed on a particular edge.\n *\n * We use the relativley fast boost::spirit parser to parse each line.\n */\nbool graph_loader(graph_type& graph, const std::string& fname,\n                  const std::string& line) {\n  ASSERT_FALSE(line.empty());\n  namespace qi = boost::spirit::qi;\n  namespace ascii = boost::spirit::ascii;\n  namespace phoenix = boost::phoenix;\n\n  graphlab::vertex_id_type doc_id(-1), word_id(-1);\n  size_t count = 0;\n  const bool success = qi::phrase_parse\n    (line.begin(), line.end(),       \n     //  Begin grammar\n     (\n      qi::ulong_[phoenix::ref(doc_id) = qi::_1] >> -qi::char_(',') >>\n      qi::ulong_[phoenix::ref(word_id) = qi::_1] >> -qi::char_(',') >>\n      qi::ulong_[phoenix::ref(count) = qi::_1]\n      )\n     ,\n     //  End grammar\n     ascii::space); \n  if(!success) return false;  \n  // Threshold the count\n  count = std::min(count, MAX_COUNT);\n  // since this is a bipartite graph I need a method to number the\n  // left and right vertices differently.  To accomplish I make sure\n  // all vertices have non-zero ids and then negate the right vertex.\n  // Unfortunatley graphlab reserves -1 and so we add 2 and negate.\n  doc_id += 2;\n  ASSERT_GT(doc_id, 1);\n  doc_id = -doc_id;\n  ASSERT_NE(doc_id, word_id);\n  // Create an edge and add it to the graph\n  graph.add_edge(doc_id, word_id, edge_data(count));\n  return true; // successful load\n}; // end of graph loader\n\n\n\n\n\n/**\n * \\brief Determine if the given vertex is a word vertex or a doc\n * vertex.\n *\n * For simplicity we connect docs --> words and therefore if a vertex\n * has in edges then it is a word.\n */\ninline bool is_word(const graph_type::vertex_type& vertex) {\n  return vertex.num_in_edges() > 0 ? 1 : 0;\n}\n\n\n/**\n * \\brief Determine if the given vertex is a doc vertex\n *\n * For simplicity we connect docs --> words and therefore if a vertex\n * has out edges then it is a doc\n */\ninline bool is_doc(const graph_type::vertex_type& vertex) {\n  return vertex.num_out_edges() > 0 ? 1 : 0;\n}\n\n/**\n * \\brief return the number of tokens on a particular edge.\n */\ninline size_t count_tokens(const graph_type::edge_type& edge) {\n  return edge.data().assignment.size();\n}\n\n\n/**\n * \\brief Get the other vertex in the edge.\n */\ninline graph_type::vertex_type\nget_other_vertex(const graph_type::edge_type& edge,\n                 const graph_type::vertex_type& vertex) {\n  return vertex.id() == edge.source().id()? edge.target() : edge.source();\n}\n\n\n\n// ========================================================\n// The Collapsed Gibbs Sampler Function\n\n\n\n/**\n * \\brief The gather type for the collapsed Gibbs sampler is used to\n * collect the topic counts on adjacent edges so that the apply\n * function can compute the correct topic counts for the center\n * vertex.\n *\n */\nstruct gather_type {\n  factor_type factor;\n  uint32_t nchanges;\n  gather_type() : nchanges(0) { };\n  gather_type(uint32_t nchanges) : factor(NTOPICS), nchanges(nchanges) { };\n  void save(graphlab::oarchive& arc) const { \n    arc << nchanges; \n    uint16_t ni = 0;\n    for (size_t i = 0;i < factor.size(); ++i) {\n      ni += (factor[i] > 0);\n    }\n    arc << ni;\n    for (size_t i = 0;i < factor.size(); ++i) {\n      if (factor[i] > 0) {\n        arc << uint16_t(i) << factor[i];\n      }\n    }\n\n  }\n  void load(graphlab::iarchive& arc) { \n    arc >> nchanges;\n    factor.resize(NTOPICS);\n    for (size_t i = 0;i < factor.size(); ++i) factor[i] = 0;\n    uint16_t ni;\n    arc >> ni; \n    for (uint16_t i = 0;i < ni; ++i) {\n      uint16_t u; arc >> u;\n      arc >> factor[u];\n    }\n  }\n  gather_type& operator+=(const gather_type& other) {\n    factor += other.factor;\n    nchanges += other.nchanges;\n    return *this;\n  }\n}; // end of gather type\n\n\n\n\n\n\n\n/**\n * \\brief The collapsed Gibbs sampler vertex program updates the topic\n * counts for the center vertex and then draws new topic assignments\n * for each edge durring the scatter phase.\n * \n */\nclass cgs_lda_vertex_program :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE\n   {\npublic:\n\n  /**\n   * \\brief At termination we want to disable sampling to allow the\n   * correct final counts to be computed.\n   */\n  static bool DISABLE_SAMPLING; \n\n   /** \\brief gather on all edges */\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  } // end of gather_edges\n\n  /**\n   * \\brief Collect the current topic count on each edge.\n   */\n  gather_type gather(icontext_type& context, const vertex_type& vertex,\n                     edge_type& edge) const {\n    gather_type ret(edge.data().nchanges);\n    const assignment_type& assignment = edge.data().assignment;\n    foreach(topic_id_type asg, assignment) {\n      if(asg != NULL_TOPIC) ++ret.factor[asg];\n    }\n    return ret;\n  } // end of gather\n\n\n  /**\n   * \\brief Update the topic count for the center vertex.  This\n   * ensures that the center vertex has the correct topic count before\n   * resampling the topics for each token along each edge.\n   */\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& sum) {\n    const size_t num_neighbors = vertex.num_in_edges() + vertex.num_out_edges();\n    ASSERT_GT(num_neighbors, 0);\n    // There should be no new edge data since the vertex program has been cleared\n    vertex_data& vdata = vertex.data();\n    ASSERT_EQ(sum.factor.size(), NTOPICS);\n    ASSERT_EQ(vdata.factor.size(), NTOPICS);\n    vdata.nupdates++;\n    vdata.nchanges = sum.nchanges;\n    vdata.factor = sum.factor;\n    if (is_doc(vertex)) {\n      float MIMNO_R = 0.0;\n      for (size_t i = 0;i < vdata.factor.size(); ++i) {\n        MIMNO_R += vdata.factor[i] * BETA / (BETA * NWORDS + GLOBAL_TOPIC_COUNT[i]);\n      }\n      vdata.MIMNO_R = MIMNO_R;\n    }\n } // end of apply\n\n\n  /**\n   * \\brief Scatter on all edges if the computation is on-going.\n   * Computation stops after bunrin or when disable sampling is set to\n   * true.\n   */\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const {\n    return (DISABLE_SAMPLING || (BURNIN > 0 && context.elapsed_seconds() > BURNIN))? \n      graphlab::NO_EDGES : graphlab::ALL_EDGES;\n  }; // end of scatter edges\n\n\n  /**\n   * \\brief Draw new topic assignments for each edge token.\n   *\n   * Note that we exploit the GraphLab caching model here by DIRECTLY\n   * modifying the topic counts of adjacent vertices.  Making the\n   * changes immediately visible to any adjacent vertex programs\n   * running on the same machine.  However, these changes will be\n   * overwritten during the apply step and are only used to accelerate\n   * sampling.  This is a potentially dangerous violation of the\n   * abstraction and should be taken with caution.  In our case all\n   * vertex topic counts are preallocated and atomic operations are\n   * used.  In addition during the sampling phase we must be careful\n   * to guard against potentially negative temporary counts.\n   */\n  void scatter(icontext_type& context, const vertex_type& vertex,\n               edge_type& edge) const {\n    factor_type& doc_topic_count =  is_doc(edge.source()) ?\n      edge.source().data().factor : edge.target().data().factor;\n    factor_type& word_topic_count = is_word(edge.source()) ?\n      edge.source().data().factor : edge.target().data().factor;\n    ASSERT_EQ(doc_topic_count.size(), NTOPICS);\n    ASSERT_EQ(word_topic_count.size(), NTOPICS);\n    float MIMNO_R = is_doc(edge.source()) ? edge.source().data().MIMNO_R :\n                      edge.target().data().MIMNO_R;\n    float MIMNO_Q = 0.0;\n    std::vector MIMNO_Q_CACHE(NTOPICS);\n\n    for (size_t t = 0; t < NTOPICS; ++t) {\n      const float n_wt  =\n        std::max(count_type(word_topic_count[t]), count_type(0));\n     if (n_wt > 0) {\n      const float n_dt =\n          std::max(count_type(doc_topic_count[t]), count_type(0));\n      const float n_t  =\n        std::max(count_type(GLOBAL_TOPIC_COUNT[t]), count_type(0));\n       MIMNO_Q_CACHE[t] = (ALPHA + n_dt)/(BETA * NWORDS + n_t); \n       MIMNO_Q_CACHE[t] = MIMNO_Q_CACHE[t] * n_wt; \n       MIMNO_Q += MIMNO_Q_CACHE[t]; \n     }\n    }\n\n    // run the actual gibbs sampling\n    std::vector prob(NTOPICS);\n    assignment_type& assignment = edge.data().assignment;\n    edge.data().nchanges = 0;\n    foreach(topic_id_type& asg, assignment) {\n      const topic_id_type old_asg = asg;\n      if(asg != NULL_TOPIC) { // construct the cavity\n        --doc_topic_count[asg];\n        --word_topic_count[asg];\n        --GLOBAL_TOPIC_COUNT[asg];\n      const float n_dt =\n          std::max(count_type(doc_topic_count[asg]), count_type(0));\n      const float n_t  =\n        std::max(count_type(GLOBAL_TOPIC_COUNT[asg]), count_type(0));\n      const float n_wt  =\n        std::max(count_type(word_topic_count[asg]), count_type(0));\n\n\n        MIMNO_Q -= MIMNO_Q_CACHE[asg];\n        MIMNO_Q_CACHE[asg] = (ALPHA + n_dt)/(BETA * NWORDS + n_t) * n_wt;\n        MIMNO_Q += MIMNO_Q_CACHE[asg]; \n      }\n      asg = 0; \n      ASSERT_GE(MIMNO_S, 0);\n      ASSERT_GE(MIMNO_R, 0);\n      ASSERT_GE(MIMNO_Q, 0);\n      float f = graphlab::random::uniform(0, MIMNO_S + MIMNO_R + MIMNO_Q);\n      if (f < MIMNO_S) {\n        float ctr = 0;\n        \n        for (size_t t = 0; t < NTOPICS; ++t) {\n          ctr += ALPHA * BETA / (BETA * NWORDS + GLOBAL_TOPIC_COUNT[t]);\n          if (ctr >= f) {\n            asg = t;\n            break;\n          }\n        }\n      }\n      else if (f < MIMNO_S + MIMNO_R) {\n        float ctr = 0;\n        f = f - MIMNO_S;\n        for(size_t t = 0; t < NTOPICS; ++t) {\n          if (doc_topic_count[t] > 0) {\n            ctr += doc_topic_count[t] * BETA / (BETA * NWORDS + GLOBAL_TOPIC_COUNT[t]);\n            if (ctr >= f) {\n              asg = t;\n              break;\n            }\n          }\n        }\n      }\n      else {\n        f = f - MIMNO_S - MIMNO_R;\n        float ctr = 0;\n        for(size_t t = 0; t < NTOPICS; ++t) {\n          if (word_topic_count[t] > 0) {\n            ctr += MIMNO_Q_CACHE[t]; \n            if (ctr >= f) {\n              asg = t;\n              break;\n            }\n          }\n        }\n      }\n      // asg = std::max_element(prob.begin(), prob.end()) - prob.begin();\n      ++doc_topic_count[asg];\n      ++word_topic_count[asg];\n      ++GLOBAL_TOPIC_COUNT[asg];\n      MIMNO_Q -= MIMNO_Q_CACHE[asg];\n{\n      const float n_dt =\n          std::max(count_type(doc_topic_count[asg]), count_type(0));\n      const float n_t  =\n        std::max(count_type(GLOBAL_TOPIC_COUNT[asg]), count_type(0));\n      const float n_wt  =\n        std::max(count_type(word_topic_count[asg]), count_type(0));\n\n      MIMNO_Q_CACHE[asg] = (ALPHA + n_dt)/(BETA * NWORDS + n_t) * n_wt;\n      MIMNO_Q += MIMNO_Q_CACHE[asg]; \n}\n      if(asg != old_asg) {\n        ++edge.data().nchanges;\n      }\n      INCREMENT_EVENT(TOKEN_CHANGES,1);\n    } // End of loop over each token\n    // singla the other vertex\n    context.signal(get_other_vertex(edge, vertex));\n  } // end of scatter function\n\n}; // end of cgs_lda_vertex_program\n\n\nbool cgs_lda_vertex_program::DISABLE_SAMPLING = false;\n\n\n/**\n * \\brief The icontext type associated with the cgs_lda_vertex program\n * is needed for all aggregators.\n */\ntypedef cgs_lda_vertex_program::icontext_type icontext_type;\n\n\n// ========================================================\n// Aggregators\n\n\n/**\n * \\brief The topk aggregator is used to periodically compute and\n * display the topk most common words in each topic.\n *\n * The number of words is determined by the global variable \\ref TOPK\n * and the interval is determined by the global variable \\ref INTERVAL.\n *\n */\nclass topk_aggregator {\n  typedef std::pair cw_pair_type;\nprivate:\n  std::vector< std::set > top_words;\n  size_t nchanges, nupdates;\npublic:\n  topk_aggregator(size_t nchanges = 0, size_t nupdates = 0) :\n    nchanges(nchanges), nupdates(nupdates) { }\n\n  void save(graphlab::oarchive& arc) const { arc << top_words << nchanges; }\n  void load(graphlab::iarchive& arc) { arc >> top_words >> nchanges; }\n\n\n  topk_aggregator& operator+=(const topk_aggregator& other) {\n    nchanges += other.nchanges;\n    nupdates += other.nupdates;\n    if(other.top_words.empty()) return *this;\n    if(top_words.empty()) top_words.resize(NTOPICS);\n    for(size_t i = 0; i < top_words.size(); ++i) {\n      // Merge the topk\n      top_words[i].insert(other.top_words[i].begin(),\n                          other.top_words[i].end());\n      // Remove excess elements\n      while(top_words[i].size() > TOPK)\n        top_words[i].erase(top_words[i].begin());\n    }\n    return *this;\n  } // end of operator +=\n\n  static topk_aggregator map(icontext_type& context,\n                             const graph_type::vertex_type& vertex) {\n    topk_aggregator ret_value;\n    const vertex_data& vdata = vertex.data();\n    ret_value.nchanges = vdata.nchanges;\n    ret_value.nupdates = vdata.nupdates;\n    if(is_word(vertex)) {\n      const graphlab::vertex_id_type wordid = vertex.id();\n      ret_value.top_words.resize(vdata.factor.size());\n      for(size_t i = 0; i < vdata.factor.size(); ++i) {\n        const cw_pair_type pair(vdata.factor[i], wordid);\n        ret_value.top_words[i].insert(pair);\n      }\n    }\n    return ret_value;\n  } // end of map function\n\n\n  static void finalize(icontext_type& context,\n                       const topk_aggregator& total) {\n    if(context.procid() != 0) return;\n    std::string json = \"{\\n\"+ TOP_WORDS.json_header_string() +\n      \"\\t\\\"values\\\": [\\n\";\n    for(size_t i = 0; i < total.top_words.size(); ++i) {\n      std::cout << \"Topic \" << i << \": \";\n      json += \"\\t[\\n\";\n      size_t counter = 0;\n      rev_foreach(cw_pair_type pair, total.top_words[i])  {\n      ASSERT_LT(pair.second, DICTIONARY.size());\n        json += \"\\t\\t[\\\"\" + DICTIONARY[pair.second] + \"\\\", \" +\n          graphlab::tostr(pair.first) + \"]\";\n        if(++counter < total.top_words[i].size()) json += \", \";\n        json += '\\n';\n        std::cout << DICTIONARY[pair.second]\n                  << \"(\" << pair.first << \")\" << \", \";\n        // std::cout << DICTIONARY[pair.second] << \",  \";\n      }\n      json += \"\\t]\";\n      if(i+1 < total.top_words.size()) json += \", \";\n      json += '\\n';\n      std::cout << std::endl;\n    }\n    json += \"]}\";\n    // Post the change to the global variable\n    TOP_WORDS.lock.lock();\n    TOP_WORDS.json_string.swap(json);\n    TOP_WORDS.lock.unlock();\n\n    std::cout << \"\\nNumber of token changes: \" << total.nchanges << std::endl;\n    std::cout << \"\\nNumber of updates:       \" << total.nupdates << std::endl;\n  } // end of finalize\n}; // end of topk_aggregator struct\n\n\n\n/**\n * \\brief The global counts aggregator computes the total number of\n * tokens in each topic across all words and documents and then\n * updates the \\ref GLOBAL_TOPIC_COUNT variable.\n *\n */\nstruct global_counts_aggregator {\n  typedef graph_type::vertex_type vertex_type;\n  static factor_type map(icontext_type& context, const vertex_type& vertex) {\n    return vertex.data().factor;\n  } // end of map function\n\n  static void finalize(icontext_type& context, const factor_type& total) {\n    size_t sum = 0;\n    float NEW_MIMNO_S = 0;\n    for(size_t t = 0; t < total.size(); ++t) {\n      GLOBAL_TOPIC_COUNT[t] =\n        std::max(count_type(total[t]/2), count_type(0));\n      sum += GLOBAL_TOPIC_COUNT[t];\n      NEW_MIMNO_S += ALPHA * BETA / (BETA * NWORDS + (GLOBAL_TOPIC_COUNT[t] > 0 ? GLOBAL_TOPIC_COUNT[t] : 0));\n    }\n    MIMNO_S = NEW_MIMNO_S;\n    context.cout() << \"Total Tokens: \" << sum << std::endl;\n  } // end of finalize\n}; // end of global_counts_aggregator struct\n\n\n\n/**\n * \\brief The Likelihood aggregators maintains the current estimate of\n * the log-likelihood of the current token assignments.\n *\n *  llik_words_given_topics = ...\n *    ntopics * (gammaln(nwords * beta) - nwords * gammaln(beta)) - ...\n *    sum_t(gammaln( n_t + nwords * beta)) +\n *    sum_w(sum_t(gammaln(n_wt + beta)));\n *\n *  llik_topics = ...\n *    ndocs * (gammaln(ntopics * alpha) - ntopics * gammaln(alpha)) + ...\n *    sum_d(sum_t(gammaln(n_td + alpha)) - gammaln(sum_t(n_td) + ntopics * alpha));\n */\nclass likelihood_aggregator : public graphlab::IS_POD_TYPE {\n  typedef graph_type::vertex_type vertex_type;\n  float lik_words_given_topics;\n  float lik_topics;\npublic:\n  likelihood_aggregator() : lik_words_given_topics(0), lik_topics(0) { }\n\n  likelihood_aggregator& operator+=(const likelihood_aggregator& other) {\n    lik_words_given_topics += other.lik_words_given_topics;\n    lik_topics += other.lik_topics;\n    return *this;\n  } // end of operator +=\n\n  static likelihood_aggregator\n  map(icontext_type& context, const vertex_type& vertex) {\n    using boost::math::lgamma;\n    const factor_type& factor = vertex.data().factor;\n    ASSERT_EQ(factor.size(), NTOPICS);\n   likelihood_aggregator ret;\n    if(is_word(vertex)) {\n      for(size_t t = 0; t < NTOPICS; ++t) {\n        const float value = std::max(count_type(factor[t]), count_type(0));\n        ret.lik_words_given_topics += lgamma(value + BETA);\n      }\n    } else {  ASSERT_TRUE(is_doc(vertex));\n      float ntokens_in_doc = 0;\n      for(size_t t = 0; t < NTOPICS; ++t) {\n        const float value = std::max(count_type(factor[t]), count_type(0));\n        ret.lik_topics += lgamma(value + ALPHA);\n        ntokens_in_doc += factor[t];\n      }\n      ret.lik_topics -= lgamma(ntokens_in_doc + NTOPICS * ALPHA);\n    }\n    return ret;\n  } // end of map function\n\n  static void finalize(icontext_type& context, const likelihood_aggregator& total) {\n    using boost::math::lgamma;\n    // Address the global sum terms\n    float denominator = 0;\n    for(size_t t = 0; t < NTOPICS; ++t) {\n      denominator += lgamma(GLOBAL_TOPIC_COUNT[t] + NWORDS * BETA);\n    } // end of for loop\n\n    const float lik_words_given_topics =\n      NTOPICS * (lgamma(NWORDS * BETA) - NWORDS * lgamma(BETA)) -\n      denominator + total.lik_words_given_topics;\n\n    const float lik_topics =\n      NDOCS * (lgamma(NTOPICS * ALPHA) - NTOPICS * lgamma(ALPHA)) +\n      total.lik_topics;\n\n    const float lik = lik_words_given_topics + lik_topics;\n    context.cout() << \"Likelihood: \" << lik << std::endl;\n  } // end of finalize\n}; // end of likelihood_aggregator struct\n\n\n\n/**\n * \\brief The selective signal functions are used to signal only the\n * vertices corresponding to words or documents.  This is done by\n * using the iengine::map_reduce_vertices function.\n */\nstruct signal_only {\n  /**\n   * \\brief Signal only the document vertices and skip the word\n   * vertices.\n   */ \n  static graphlab::empty\n  docs(icontext_type& context, const graph_type::vertex_type& vertex) {\n    if(is_doc(vertex)) context.signal(vertex);\n    return graphlab::empty();\n  } // end of signal_docs\n \n /**\n  * \\brief Signal only the word vertices and skip the document\n  * vertices.\n  */\n  static graphlab::empty\n  words(icontext_type& context, const graph_type::vertex_type& vertex) {\n    if(is_word(vertex)) context.signal(vertex);\n    return graphlab::empty();\n  } // end of signal_words\n}; // end of selective_only\n\n\n\n\n\n/**\n * \\brief This function is used to load and then initialize the data\n * graph (corpus) from a folder or file.\n * \n * The graph can be in either json form constructed using the graph\n * builder tools or in raw text form.  The raw text format contains a\n * token on each line of each file in the format:\n *\n \\verbatim\n   \n          ...\n \\endverbatim\n *\n * for example:\n \\verbatim\n    0    0     2\n    0    4     1\n    0    2     3\n \\endverbatim\n * \n * implies that document zero contains word zero twice, word 4 once,\n * and word two three times.\n *\n * If a dictionary is used it is important that each word id\n * correspond to the index in the dictionary file (starting at zero).\n *\n * Once loaded the total number of words, documents, and tokens is\n * counted and saved to global variables which are read during the\n * execution of the sampler.\n *\n * \\param [in] dc The distributed control object used to coordinate\n * between machines.\n *\n * \\param [in,out] graph The graph object that is initialized.\n * \n * \\param [in] corpus_dir The directory or file containing the graph\n * data.  The corpus directory can reside on hdfs in which case the\n * path should begin with \"hdfs://namenode\".  In addition the file(s)\n * may be gzipped and therefore must end in \".gz\".\n *\n * \\param [in] load_json Whether the graph data is in text format or\n * preprocessed json format using the graph builder tools.\n */\nbool load_and_initialize_graph(graphlab::distributed_control& dc,\n                               graph_type& graph,\n                               const std::string& corpus_dir,\n                               const std::string& format\t\t\t       \n\t\t\t       ) {\n  dc.cout() << \"Loading graph.\" << std::endl;\n  graphlab::timer timer; timer.start();\n\n  if(format==\"matrix\"){\n      dc.cout() << \"matrix format\" << std::endl;\n      graph.load(corpus_dir, graph_loader);\n  // }else if(format==\"json\"){\n  //     dc.cout() << \"json format\" << std::endl;\n  //     graph.load_json(corpus_dir, false, eparser, vparser);\n  // }else if(format==\"json-gzip\"){\n  //     dc.cout() <<\"json gzip format\" << std::endl;\n  //     graph.load_json(corpus_dir, true, eparser, vparser);\n  }else{\n      dc.cout() << \"Non supported format. See --help\" << std::endl;\n      return false;\n  }\n\n  dc.cout() << \"Finalizing graph.\" << std::endl;\n  timer.start();\n  graph.finalize();\n  dc.cout() << \"Finalizing graph. Finished in \"\n            << timer.current_time() << \" seconds.\" << std::endl;\n\n  dc.cout() << \"Computing number of words and documents.\" << std::endl;\n  NWORDS = graph.map_reduce_vertices(is_word);\n  NDOCS = graph.map_reduce_vertices(is_doc);\n  NTOKENS = graph.map_reduce_edges(count_tokens);\n  dc.cout() << \"Number of words:     \" << NWORDS  << std::endl;\n  dc.cout() << \"Number of docs:      \" << NDOCS   << std::endl;\n  dc.cout() << \"Number of tokens:    \" << NTOKENS << std::endl;\n  // Prepare the json struct with the word counts\n  TOP_WORDS.lock.lock();\n  TOP_WORDS.json_string = \"{\\n\" + TOP_WORDS.json_header_string() +\n    \"\\t\\\"values\\\": [] \\n }\";\n  TOP_WORDS.lock.unlock();\n  return true;\n} // end of load and initialize graph\n\n\n\n/**\n * \\brief Load the dictionary global variable from the file containing\n * the terms (one term per line).\n *\n * Note that while graphs can be loaded from multiple files the\n * dictionary must be in a single file.  The dictionary is loaded\n * entirely into memory and used to display word clouds and the top\n * terms in each topic.\n *\n * \\param [in] fname the file containing the dictionary data.  The\n * data can be located on HDFS and can also be gzipped (must end in\n * \".gz\").\n * \n */\nbool load_dictionary(const std::string& fname)  {\n  // std::cout << \"staring load on: \"\n  //           << graphlab::get_local_ip_as_str() << std::endl;\n  const bool gzip = boost::ends_with(fname, \".gz\");\n  // test to see if the graph_dir is an hadoop path\n  if(boost::starts_with(fname, \"hdfs://\")) {\n    graphlab::hdfs hdfs;\n    graphlab::hdfs::fstream in_file(hdfs, fname);\n    boost::iostreams::filtering_stream fin;\n    fin.set_auto_close(false);\n    if(gzip) fin.push(boost::iostreams::gzip_decompressor());\n    fin.push(in_file);\n    if(!fin.good()) {\n      logstream(LOG_ERROR) << \"Error loading dictionary: \"\n                           << fname << std::endl;\n      return false;\n    }\n    std::string term;\n    while(std::getline(fin,term).good()) DICTIONARY.push_back(term);\n    if (gzip) fin.pop();\n    fin.pop();\n    in_file.close();\n  } else {\n    std::cout << \"opening: \" << fname << std::endl;\n    std::ifstream in_file(fname.c_str(),\n                          std::ios_base::in | std::ios_base::binary);\n    boost::iostreams::filtering_stream fin;\n    if (gzip) fin.push(boost::iostreams::gzip_decompressor());\n    fin.push(in_file);\n    if(!fin.good() || !fin.good()) {\n      logstream(LOG_ERROR) << \"Error loading dictionary: \"\n                           << fname << std::endl;\n      return false;\n    }\n    std::string term;\n    std::cout << \"Loooping\" << std::endl;\n    while(std::getline(fin, term).good()) DICTIONARY.push_back(term);\n    if (gzip) fin.pop();\n    fin.pop();\n    in_file.close();\n  } // end of else\n  // std::cout << \"Finished load on: \"\n  //           << graphlab::get_local_ip_as_str() << std::endl;\n  std::cout << \"Dictionary Size: \" << DICTIONARY.size() << std::endl;\n  return true;\n} // end of load dictionary\n\n\n\n\nstruct count_saver {\n  bool save_words;\n  count_saver(bool save_words) : save_words(save_words) { }\n  typedef graph_type::vertex_type vertex_type;\n  typedef graph_type::edge_type   edge_type;\n  std::string save_vertex(const vertex_type& vertex) const {\n    // Skip saving vertex data if the vertex type is not consistent\n    // with the save type\n    if((save_words && is_doc(vertex)) ||\n       (!save_words && is_word(vertex))) return \"\";\n    // Proceed to save\n    std::stringstream strm;\n    if(save_words) {\n      const graphlab::vertex_id_type vid = vertex.id();\n      strm << vid << '\\t';\n    } else { // save documents\n      const graphlab::vertex_id_type vid = (-vertex.id()) - 2;\n      strm << vid << '\\t';\n    }\n    const factor_type& factor = vertex.data().factor;\n    for(size_t i = 0; i < factor.size(); ++i) { \n      strm << factor[i];\n      if(i+1 < factor.size()) strm << '\\t';\n    }\n    strm << '\\n';\n    return strm.str();\n  }\n  std::string save_edge(const edge_type& edge) const {\n    return \"\"; //nop\n  }\n}; // end of prediction_saver\n\n\n\n\n\n\n\n/**\n * \\brief The omni engine type is used to allow switching between\n * synchronous and asynchronous computation. \n */\ntypedef graphlab::omni_engine engine_type;\n\n\n\n\n\n\n\n\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n  //  INITIALIZE_EVENT_LOG(dc);\n  ADD_CUMULATIVE_EVENT(TOKEN_CHANGES, \"Token Changes\", \"Changes\");\n\n  // Parse command line options -----------------------------------------------\n  const std::string description =\n    \"\\n=========================================================================\\n\"\n    \"The Collapsed Gibbs Sampler for the LDA model implements\\n\"\n    \"a highly asynchronous version of parallel LDA in which document\\n\"\n    \"and word counts are maintained in an eventually consistent\\n\"\n    \"manner.\\n\"\n    \"\\n\"\n    \"The standard usage is: \\n\"\n    \"\\t./cgs_lda --dictionary dictionary.txt --corpus doc_word_count.tsv\\n\"\n    \"where dictionary.txt contains: \\n\"\n    \"\\taaa \\n\\taaai \\n\\tabalone \\n\\t   ... \\n\"\n    \"each line number corresponds to wordid (i.e aaa has wordid=0)\\n\\n\"\n    \"and doc_word_count.tsv is formatted   :\\n\"\n    \"(where wordid is indexed starting from zero and docid are positive integers)\\n\"\n    \"\\t0\\t0\\t3\\n\"\n    \"\\t0\\t5\\t1\\n\"\n    \"\\t ...\\n\\n\"\n    \"For JSON format, make sure docid are negative integers index starting from -2 \\n\\n\"\n    \"To learn more about the NLP package and its applications visit\\n\\n\"\n    \"\\t\\t http://graphlab.org \\n\\n\"\n    \"Additional Options\";\n  graphlab::command_line_options clopts(description);\n  std::string corpus_dir;\n  std::string dictionary_fname;\n  std::string doc_dir;\n  std::string word_dir;\n  std::string exec_type = \"asynchronous\";\n  std::string format = \"matrix\";\n  clopts.attach_option(\"dictionary\", dictionary_fname,\n                       \"The file containing the list of unique words\");\n  clopts.attach_option(\"engine\", exec_type, \n                       \"The engine type synchronous or asynchronous\");\n  clopts.attach_option(\"corpus\", corpus_dir,\n                       \"The directory or file containing the corpus data.\");\n  clopts.add_positional(\"corpus\");\n  clopts.attach_option(\"ntopics\", NTOPICS,\n                       \"Number of topics to use.\");\n  clopts.attach_option(\"alpha\", ALPHA,\n                       \"The document hyper-prior\");\n  clopts.attach_option(\"beta\", BETA,\n                       \"The word hyper-prior\");\n  clopts.attach_option(\"topk\", TOPK,\n                       \"The number of words to report\");\n  clopts.attach_option(\"interval\", INTERVAL,\n                       \"statistics reporting interval\");\n  clopts.attach_option(\"max_count\", MAX_COUNT,\n                       \"The maximum number of occurences of a word in a document.\");\n  clopts.attach_option(\"format\", format,\n                       \"Formats: matrix,json,json-gzip\");\n  clopts.attach_option(\"burnin\", BURNIN, \n                       \"The time in second to run until a sample is collected. \"\n                       \"If less than zero the sampler runs indefinitely.\");\n  clopts.attach_option(\"doc_dir\", doc_dir,\n                       \"The output directory to save the final document counts.\");\n  clopts.attach_option(\"word_dir\", word_dir,\n                       \"The output directory to save the final words counts.\");\n\n\n  if(!clopts.parse(argc, argv)) {\n    graphlab::mpi_tools::finalize();\n    return clopts.is_set(\"help\")? EXIT_SUCCESS : EXIT_FAILURE;\n  }\n\n  if(dictionary_fname.empty()) {\n    logstream(LOG_WARNING) << \"No dictionary file was provided.\" << std::endl\n                           << \"Top k words will not be estimated.\" << std::endl;\n  }\n\n  if(corpus_dir.empty()) {\n    logstream(LOG_ERROR) << \"No corpus file was provided.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  // Start the webserver\n  graphlab::launch_metric_server();\n  graphlab::add_metric_server_callback(\"wordclouds\", word_cloud_callback);\n\n\n  ///! Initialize global variables\n  GLOBAL_TOPIC_COUNT.resize(NTOPICS);\n  if(!dictionary_fname.empty()) {\n    const bool success = load_dictionary(dictionary_fname);\n    if(!success) {\n      logstream(LOG_ERROR) << \"Error loading dictionary.\" << std::endl;\n      return EXIT_FAILURE;\n    }\n  }\n\n  ///! load the graph\n  graph_type graph(dc, clopts);\n  {\n    const bool success = \n      load_and_initialize_graph(dc, graph, corpus_dir, format);\n    if(!success) {\n      logstream(LOG_ERROR) << \"Error loading graph.\" << std::endl;\n      return EXIT_FAILURE;\n    }\n  }\n\n\n  const size_t ntokens = graph.map_reduce_edges(count_tokens);\n  dc.cout() << \"Total tokens: \" << ntokens << std::endl;\n\n\n\n  engine_type engine(dc, graph, exec_type, clopts);\n  ///! Add an aggregator\n  if(!DICTIONARY.empty()) {\n    const bool success =\n      engine.add_vertex_aggregator\n      (\"topk\", topk_aggregator::map, topk_aggregator::finalize) &&\n      engine.aggregate_periodic(\"topk\", INTERVAL);\n    ASSERT_TRUE(success);\n  }\n\n  { // Add the Global counts aggregator\n    const bool success =\n      engine.add_vertex_aggregator\n      (\"global_counts\", \n       global_counts_aggregator::map, \n       global_counts_aggregator::finalize) &&\n      engine.aggregate_periodic(\"global_counts\", 5);\n    ASSERT_TRUE(success);\n  }\n  \n/*  { // Add the likelihood aggregator\n    const bool success =\n      engine.add_vertex_aggregator\n      (\"likelihood\", \n       likelihood_aggregator::map, \n       likelihood_aggregator::finalize) &&\n      engine.aggregate_periodic(\"likelihood\", 10);\n    ASSERT_TRUE(success);\n  }*/\n\n  engine.aggregate_now(\"global_counts\");\n\n  ///! schedule only documents\n  dc.cout() << \"Running The Collapsed Gibbs Sampler\" << std::endl;\n  engine.map_reduce_vertices(signal_only::docs);\n  graphlab::timer timer;\n  // Enable sampling\n  cgs_lda_vertex_program::DISABLE_SAMPLING = false;\n  // Run the engine\n  engine.start();\n  // Finalize the counts\n  cgs_lda_vertex_program::DISABLE_SAMPLING = true;\n  engine.signal_all();\n  engine.start();\n  \n  const double runtime = timer.current_time();\n  dc.cout()\n    << \"----------------------------------------------------------\" << std::endl\n    << \"Final Runtime (seconds):   \" << runtime\n    << std::endl\n    << \"Updates executed: \" << engine.num_updates() << std::endl\n    << \"Update Rate (updates/second): \"\n    << engine.num_updates() / runtime << std::endl;\n  \n  \n  \n  if(!word_dir.empty()) {\n    // save word topic counts\n    const bool gzip_output = false;\n    const bool save_vertices = true;\n    const bool save_edges = false;\n    const size_t threads_per_machine = 2;\n    const bool save_words = true;\n    graph.save(word_dir, count_saver(save_words),\n               gzip_output, save_vertices, \n               save_edges, threads_per_machine);\n  }\n\n  \n  if(!doc_dir.empty()) {\n    // save doc topic counts\n    const bool gzip_output = false;\n    const bool save_vertices = true;\n    const bool save_edges = false;\n    const size_t threads_per_machine = 2;\n    const bool save_words = false;\n    graph.save(doc_dir, count_saver(save_words),\n               gzip_output, save_vertices, \n               save_edges, threads_per_machine);\n\n  }\n\n\n  graphlab::stop_metric_server_on_eof();\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n\n\n} // end of main\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
  },
  {
    "path": "toolkits/topic_modeling/deprecated/cgs_lda.cpp",
    "content": "/**\n * Copyright (c) 2009 Carnegie Mellon University.\n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n * For more about this software visit:\n *\n *      http://www.graphlab.ml.cmu.edu\n *\n */\n\n\n#include \n#include \n\n#include \"cgs_lda_common.hpp\"\n\n#include \n\n\n\n/**\n * \\brief The gather type used to accumulate information about the\n * words in a document\n */\nstruct gather_type {\n  typedef std::pair edge_pair_type;\n  typedef std::map\n  neighborhood_map_type;\n  neighborhood_map_type neighborhood_map;\n\n  gather_type() { }\n\n  gather_type(const graph_type::edge_type& edge,\n              const graph_type::vertex_type& vertex) {\n    const graph_type::vertex_type other_vertex =\n      get_other_vertex(edge, vertex);\n    neighborhood_map[other_vertex.id()] =\n      edge_pair_type(other_vertex.data().factor, edge.data().assignment);\n  }\n\n  void save(graphlab::oarchive& arc) const { arc << neighborhood_map; }\n\n  void load(graphlab::iarchive& arc) { arc >> neighborhood_map; }\n\n  gather_type& operator+=(const gather_type& other) {\n    neighborhood_map.insert(other.neighborhood_map.begin(),\n                            other.neighborhood_map.end());\n    return *this;\n  } // end of operator +=\n\n}; // end of gather type\n\n\n\n\nclass cgs_lda_vertex_program :\n  public graphlab::ivertex_program {\nprivate:\n  typedef std::map edge_data_map_type;\n  edge_data_map_type new_edge_data;\npublic:\n  void save(graphlab::oarchive& arc) const {\n    arc << new_edge_data;\n  } // end of save cgs_lda\n\n  void load(graphlab::iarchive& arc) {\n    arc >> new_edge_data;\n  } // end of load cgs_lda\n\n\n  edge_dir_type gather_edges(icontext_type& context,\n                             const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  } // end of gather_edges\n\n  gather_type gather(icontext_type& context, const vertex_type& vertex,\n                     edge_type& edge) const {\n    return gather_type(edge, vertex);\n  } // end of gather\n\n\n  void apply(icontext_type& context, vertex_type& vertex,\n             const gather_type& sum) {\n    const size_t num_neighbors = vertex.num_in_edges() + vertex.num_out_edges();\n    ASSERT_GT(num_neighbors, 0);\n    ASSERT_EQ(new_edge_data.size(), 0);\n    vertex_data& vdata = vertex.data();\n    factor_type& factor = vdata.factor;\n    ASSERT_EQ(factor.size(), NTOPICS);\n\n    // first update the factor count for this vertex\n    typedef gather_type::neighborhood_map_type::value_type pair_type;\n    foreach(const pair_type& nbr_pair, sum.neighborhood_map) {\n      const assignment_type& assignment = nbr_pair.second.second;\n      foreach(const topic_id_type& asg, assignment) {\n        if(asg != NULL_TOPIC) ++factor[asg];\n      } // end of loop over assignments\n    } // end of loop over neighborhood\n\n    // Resample the vertex\n    vdata.nchanges = 0;\n    // run the actual gibbs sampling\n    std::vector prob(NTOPICS);\n    typedef gather_type::neighborhood_map_type::value_type pair_type;\n    foreach(const pair_type& nbr_pair, sum.neighborhood_map) {\n      const graphlab::vertex_id_type other_id = nbr_pair.first;\n      factor_type other_factor = nbr_pair.second.first;\n      assignment_type assignment = nbr_pair.second.second;\n      factor_type& doc_topic_count  = is_doc(vertex)? factor : other_factor;\n      factor_type& word_topic_count = is_word(vertex)? factor : other_factor;\n      ASSERT_EQ(word_topic_count.size(), NTOPICS);\n      ASSERT_EQ(doc_topic_count.size(), NTOPICS);\n      // Resample the topics\n      foreach(topic_id_type& asg, assignment) {\n        const topic_id_type old_asg = asg;\n        if(asg != NULL_TOPIC) { // construct the cavity\n          --doc_topic_count[asg];\n          --word_topic_count[asg];\n          --GLOBAL_TOPIC_COUNT[asg];\n        }\n        for(size_t t = 0; t < NTOPICS; ++t) {\n          const double n_dt =\n            std::max(count_type(doc_topic_count[t]), count_type(0));\n          ASSERT_GE(n_dt, 0);\n          const double n_wt =\n            std::max(count_type(word_topic_count[t]), count_type(0));\n          ASSERT_GE(n_wt, 0);\n          const double n_t  =\n            std::max(count_type(GLOBAL_TOPIC_COUNT[t]), count_type(0));\n          ASSERT_GE(n_t, 0);\n          prob[t] = (ALPHA + n_dt) * (BETA + n_wt) / (BETA * NWORDS + n_t);\n        }\n        asg = graphlab::random::multinomial(prob);\n        ++doc_topic_count[asg];\n        ++word_topic_count[asg];\n        ++GLOBAL_TOPIC_COUNT[asg];\n        // record a change if one occurs\n        if(old_asg != asg) vdata.nchanges++;\n      } // End of loop over each token\n      // test to see if the topic assignments have change\n      // sort the topic assignment to be in a \"canonical order\"\n      std::sort(assignment.begin(), assignment.end());\n      const assignment_type& old_assignment = nbr_pair.second.second;\n      bool is_same = (old_assignment.size() == assignment.size());\n      for(size_t i = 0; i < assignment.size() && is_same; ++i)\n        is_same = (assignment[i] == old_assignment[i]);\n      if(!is_same) new_edge_data[other_id] = assignment;\n    } // end of loop over neighbors\n  } // end of apply\n\n\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  }; // end of scatter edges\n\n  void scatter(icontext_type& context, const vertex_type& vertex,\n               edge_type& edge) const  {\n    const vertex_type other_vertex = get_other_vertex(edge, vertex);\n    edge_data_map_type::const_iterator iter =\n      new_edge_data.find(other_vertex.id());\n    // If there is an assignment then something changed\n    if(iter != new_edge_data.end()) {\n      const assignment_type& new_topic_assignment = iter->second;\n      ASSERT_EQ(new_topic_assignment.size(),\n                edge.data().assignment.size());\n      edge.data().assignment = new_topic_assignment;\n    }\n    context.signal(get_other_vertex(edge, vertex));\n  } // end of scatter function\n\n}; // end of cgs_lda_vertex_program\n\n\n\n\n\n\ntypedef graphlab::omni_engine engine_type;\ntypedef cgs_lda_vertex_program::icontext_type icontext_type;\ntypedef topk_aggregator topk_type;\ntypedef selective_signal signal_only;\ntypedef global_counts_aggregator global_counts_agg;\n\n\n\n\n\n\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n  // Parse command line options -----------------------------------------------\n  const std::string description =\n    \"\\n=========================================================================\\n\"\n    \"The fast Collapsed Gibbs Sampler for the LDA model implements\\n\"\n    \"a highly asynchronous version of parallel LDA in which document\\n\"\n    \"and word counts are maintained in an eventually consistent\\n\"\n    \"manner.\\n\"\n    \"\\n\"\n    \"The standard usage is: \\n\"\n    \"\\t./fast_cgs_lda --dictionary dictionary.txt --matrix doc_word_count.tsv\\n\"\n    \"where dictionary.txt contains: \\n\"\n    \"\\taaa \\n\\taaai \\n\\tabalone \\n\\t   ... \\n\"\n    \"each line number corresponds to wordid (i.e aaa has wordid=0)\\n\\n\"\n    \"and doc_word_count.tsv is formatted   :\\n\"\n    \"(where wordid is indexed starting from zero and docid are positive integers)\\n\"\n    \"\\t0\\t0\\t3\\n\"\n    \"\\t0\\t5\\t1\\n\"\n    \"\\t ...\\n\\n\"\n    \"For JSON format, make sure docid are negative integers index starting from -2 \\n\\n\"\n    \"To learn more about the NLP package and its applications visit\\n\\n\"\n    \"\\t\\t http://graphlab.org \\n\\n\"\n    \"Additional Options\";\n  graphlab::command_line_options clopts(description);\n  std::string matrix_dir;\n  std::string dictionary_fname;\n  bool loadjson = false;\n  clopts.attach_option(\"dictionary\", &dictionary_fname, dictionary_fname,\n                       \"The file containing the list of unique words\");\n  clopts.add_positional(\"dictionary\");\n  clopts.attach_option(\"matrix\", &matrix_dir, matrix_dir,\n                       \"The directory containing the matrix file\");\n  clopts.add_positional(\"matrix\");\n  clopts.attach_option(\"ntopics\", &NTOPICS, NTOPICS,\n                       \"Number of topics to use.\");\n  clopts.attach_option(\"alpha\", &ALPHA, ALPHA,\n                       \"The document hyper-prior\");\n  clopts.attach_option(\"beta\", &BETA, BETA,\n                       \"The word hyper-prior\");\n  clopts.attach_option(\"topk\", &TOPK, TOPK,\n                       \"The number of words to report\");\n  clopts.attach_option(\"interval\", &INTERVAL, INTERVAL,\n                       \"statistics reporting interval\");\n  clopts.attach_option(\"loadjson\",&loadjson,loadjson,\n                        \"Boolean if in json format (matrix arg is dir or gzip file)\");\n\n  if(!clopts.parse(argc, argv)) {\n    graphlab::mpi_tools::finalize();\n    return clopts.is_set(\"help\")? EXIT_SUCCESS : EXIT_FAILURE;\n  }\n  if(dictionary_fname.empty()) {\n    logstream(LOG_ERROR) << \"No dictionary file was provided.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n  if(matrix_dir.empty()) {\n    logstream(LOG_ERROR) << \"No matrix file was provided.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n\n\n  ///! Initialize global variables\n  GLOBAL_TOPIC_COUNT.resize(NTOPICS);\n  bool success = load_dictionary(dictionary_fname);\n  if(!success) {\n    logstream(LOG_ERROR) << \"Error loading dictionary.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  ///! load the graph\n  graph_type graph(dc, clopts);\n  success = load_and_initialize_graph(dc, graph, matrix_dir,loadjson);\n  if(!success) {\n    logstream(LOG_ERROR) << \"Error loading graph.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n\n\n  engine_type engine(dc, graph, clopts, \"synchronous\");\n  ///! Add an aggregator\n  success =\n    engine.add_vertex_aggregator\n    (\"topk\", topk_type::map, topk_type::finalize) &&\n    engine.aggregate_periodic(\"topk\", INTERVAL);\n  ASSERT_TRUE(success);\n  // success =\n  //   engine.add_vertex_aggregator\n  //   (\"global_counts\", global_counts_agg::map, global_counts_agg::finalize) &&\n  //   engine.aggregate_periodic(\"global_counts\", 5);\n  // ASSERT_TRUE(success);\n\n\n  ///! schedule only documents\n  dc.cout() << \"Running The Collapsed Gibbs Sampler\" << std::endl;\n  engine.map_reduce_vertices(signal_only::docs);\n  graphlab::timer timer;\n  engine.start();\n  const double runtime = timer.current_time();\n  dc.cout()\n    << \"----------------------------------------------------------\" << std::endl\n    << \"Final Runtime (seconds):   \" << runtime\n    << std::endl\n    << \"Updates executed: \" << engine.num_updates() << std::endl\n    << \"Update Rate (updates/second): \"\n    << engine.num_updates() / runtime << std::endl;\n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n\n\n} // end of main\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
  },
  {
    "path": "toolkits/topic_modeling/deprecated/cvb0_lda_common.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \n#include \n#include \n\n\n#include \"cvb0_lda_common.hpp\"\n\n#include \n\ndouble ALPHA    = 0.1;\ndouble BETA     = 0.1;\nsize_t NTOPICS  = 50;\nsize_t NWORDS   = 0;\nsize_t TOPK     = 5;\nsize_t INTERVAL = 10;\nfactor_type GLOBAL_TOPIC_COUNT;\nstd::vector DICTIONARY;\nsize_t MAX_COUNT = 100;\n\nbool graph_loader(graph_type& graph, const std::string& fname, \n                  const std::string& line) {\n  ASSERT_FALSE(line.empty()); \n  const int BASE = 10;\n  char* next_char_ptr = NULL;\n  graph_type::vertex_id_type doc_id = \n    strtoul(line.c_str(), &next_char_ptr, BASE);\n  if(next_char_ptr == NULL) return false;\n  const graph_type::vertex_id_type word_id = \n    strtoul(next_char_ptr, &next_char_ptr, BASE);\n  if(next_char_ptr == NULL) return false;\n  size_t count = \n    strtoul(next_char_ptr, &next_char_ptr, BASE);\n  if(next_char_ptr == NULL) return false;\n  \n  count = std::min(count, MAX_COUNT);\n\n  // since this is a bipartite graph I need a method to number the\n  // left and right vertices differently.  To accomplish I make sure\n  // all vertices have non-zero ids and then negate the right vertex.\n  doc_id += 2; \n  ASSERT_GT(doc_id, 1); \n  doc_id = -doc_id;\n  ASSERT_NE(doc_id, word_id);\n\n  // Create an edge and add it to the graph\n  graph.add_edge(doc_id, word_id, edge_data(count));\n  return true; // successful load\n}; // end of graph loader\n\n\n\n/** populate the global dictionary */\nbool load_dictionary(const std::string& fname) {\n  const bool gzip = boost::ends_with(fname, \".gz\");\n  // test to see if the graph_dir is an hadoop path\n  if(boost::starts_with(fname, \"hdfs://\")) {\n    graphlab::hdfs hdfs;\n    graphlab::hdfs::fstream in_file(hdfs, fname);\n    boost::iostreams::filtering_stream fin;  \n    fin.set_auto_close(false);\n    if(gzip) fin.push(boost::iostreams::gzip_decompressor());\n    fin.push(in_file);\n    if(!fin.good()) {\n      logstream(LOG_ERROR) << \"Error loading dictionary: \"\n                           << fname << std::endl;\n      return false;\n    }\n    std::string term;\n    while(std::getline(fin,term).good()) DICTIONARY.push_back(term);\n    if (gzip) fin.pop();\n    fin.pop();\n    in_file.close();\n  } else {\n    std::cout << \"opening: \" << fname << std::endl;\n    std::ifstream in_file(fname.c_str(), \n                          std::ios_base::in | std::ios_base::binary);\n    boost::iostreams::filtering_stream fin;  \n    if (gzip) fin.push(boost::iostreams::gzip_decompressor());\n    fin.push(in_file);\n    if(!fin.good()) {\n      logstream(LOG_ERROR) << \"Error loading dictionary: \"\n                           << fname << std::endl;\n      return false;\n    }\n    std::string term;\n    std::cout << \"Loooping\" << std::endl;\n    while(std::getline(fin, term).good()) DICTIONARY.push_back(term);\n    if (gzip) fin.pop();\n    fin.pop();\n    in_file.close();\n  } // end of else\n  std::cout << \"Dictionary Size: \" << DICTIONARY.size() << std::endl;\n  return true;\n} // end of load dictionary\n\n\n\nbool load_and_initialize_graph(graphlab::distributed_control& dc,\n                               graph_type& graph,\n                               const std::string& matrix_dir) {  \n  dc.cout() << \"Loading graph.\" << std::endl;\n  graphlab::timer timer; timer.start();\n  graph.load(matrix_dir, graph_loader); \n  dc.cout() << \": Loading graph. Finished in \" \n            << timer.current_time() << \" seconds.\" << std::endl;\n\n  dc.cout() << \"Finalizing graph.\" << std::endl;\n  timer.start();\n  graph.finalize();\n  dc.cout() << \"Finalizing graph. Finished in \" \n            << timer.current_time() << \" seconds.\" << std::endl;\n\n  dc.cout() << \"Initializing Vertex Data\" << std::endl;\n  timer.start();\n  graph.transform_vertices(initialize_vertex_data);\n  dc.cout() << \"Finished initializing Vertex Data in \" \n            << timer.current_time() << \" seconds.\" << std::endl;\n\n  dc.cout() << \"Verivying dictionary size.\" << std::endl;\n  NWORDS = graph.map_reduce_vertices(is_word);\n  dc.cout()  << \"Number of words: \" << NWORDS;\n  //ASSERT_LT(NWORDS, DICTIONARY.size());\n\n  return true;\n} // end of load and initialize graph\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
  },
  {
    "path": "toolkits/topic_modeling/deprecated/cvb0_lda_common.hpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n#ifndef CVB0_LDA_HPP\n#define CVB0_LDA_HPP\n \n#include \n#include \n#include \n\n#include \ntypedef float count_type;\ntypedef uint16_t topic_id_type;\n#define NULL_TOPIC (topic_id_type(-1))\n\ntypedef std::vector< count_type > factor_type;\ntypedef std::vector< topic_id_type > assignment_type;\n\nextern double ALPHA;\nextern double BETA;\nextern size_t NTOPICS;\nextern size_t NWORDS;\nextern size_t TOPK;\nextern size_t INTERVAL;\nextern factor_type GLOBAL_TOPIC_COUNT;\nextern std::vector DICTIONARY;\nextern size_t MAX_COUNT;\n\n\n\ninline factor_type& operator+=(factor_type& lvalue, const factor_type& rvalue) {\n  if(!rvalue.empty()) {\n    if(lvalue.size() != NTOPICS) lvalue = rvalue;\n    else {\n      for(size_t t = 0; t < lvalue.size(); ++t) lvalue[t] += rvalue[t];\n    }\n  }\n  return lvalue;\n} // end of operator +=\n\n\n\n/**\n * The vertex data type\n */\nstruct vertex_data {\n  factor_type factor;\n  size_t nupdates;\n  float nchanges;\n  vertex_data() : nupdates(0), nchanges(0) { }\n  void save(graphlab::oarchive& arc) const { \n    arc << factor << nupdates << nchanges; \n  }\n  void load(graphlab::iarchive& arc) { \n    arc >> factor >> nupdates >> nchanges; \n  } \n}; // end of vertex_data\n\n\n/**\n * The edge data type\n */\nstruct edge_data {\n  uint32_t count; factor_type belief;\n  edge_data(uint32_t count = 0) : count(count), belief(NTOPICS) { }\n  void save(graphlab::oarchive& arc) const { \n    arc << count << belief;\n  }\n  void load(graphlab::iarchive& arc) { \n    arc >> count >> belief;\n  }\n}; // end of edge data\n\n\n\n/**\n * \\brief The graph type;\n */\ntypedef graphlab::distributed_graph graph_type;\n\n\n\n\nbool graph_loader(graph_type& graph, const std::string& fname, \n                  const std::string& line);\n\ninline void initialize_vertex_data(graph_type::vertex_type& vertex) {\n  vertex.data().factor.resize(NTOPICS);\n}\n\n\nbool load_and_initialize_graph(graphlab::distributed_control& dc,\n                               graph_type& graph,\n                               const std::string& matrix_dir);\n\n\n\n\n/** populate the global dictionary */\nbool load_dictionary(const std::string& fname);\n\n\ninline bool is_word(const graph_type::vertex_type& vertex) {\n  return vertex.num_in_edges() > 0 ? 1 : 0;\n}\n\ninline bool is_doc(const graph_type::vertex_type& vertex) {\n  return vertex.num_out_edges() > 0 ? 1 : 0;\n}\n\ninline graph_type::vertex_type \nget_other_vertex(graph_type::edge_type& edge, \n                 const graph_type::vertex_type& vertex) {\n  return vertex.id() == edge.source().id()? edge.target() : edge.source();\n}\n\n\ntemplate\nclass topk_aggregator {\n  typedef IContext icontext_type;\n  typedef std::pair cw_pair_type;\nprivate:\n  std::vector< std::set > top_words;\n  float nchanges, nupdates;\npublic:\n  topk_aggregator(size_t nchanges = 0, size_t nupdates = 0) : \n    nchanges(nchanges), nupdates(nupdates) { }\n\n  void save(graphlab::oarchive& arc) const { \n    arc << top_words << nchanges << nupdates; \n  }\n  void load(graphlab::iarchive& arc) { \n    arc >> top_words >> nchanges >> nupdates;\n  }\n\n  topk_aggregator& operator+=(const topk_aggregator& other) {\n    nchanges += other.nchanges;\n    nupdates += other.nupdates;\n    if(other.top_words.empty()) return *this;\n    if(top_words.empty()) top_words.resize(NTOPICS);\n    for(size_t i = 0; i < top_words.size(); ++i) {\n      // Merge the topk\n      top_words[i].insert(other.top_words[i].begin(), \n                          other.top_words[i].end());\n      // Remove excess elements        \n      while(top_words[i].size() > TOPK) \n        top_words[i].erase(top_words[i].begin());\n    }\n    return *this;\n  } // end of operator +=\n  \n  static topk_aggregator map(icontext_type& context, \n                             const graph_type::vertex_type& vertex) {\n    topk_aggregator ret_value;\n    const vertex_data& vdata = vertex.data();\n    ret_value.nchanges = vdata.nchanges;\n    ret_value.nupdates = vdata.nupdates;\n    if(is_word(vertex)) {\n      const graphlab::vertex_id_type wordid = vertex.id();    \n      ret_value.top_words.resize(vdata.factor.size());\n      for(size_t i = 0; i < vdata.factor.size(); ++i) {\n        const cw_pair_type pair(vdata.factor[i], wordid);\n        ret_value.top_words[i].insert(pair);\n      }\n    } \n    return ret_value;\n  } // end of map function\n\n  static void finalize(icontext_type& context,\n                       const topk_aggregator& total) {\n    if(context.procid() != 0) return;\n    for(size_t i = 0; i < total.top_words.size(); ++i) {\n      std::cout << \"Topic \" << i << \": \";\n      rev_foreach(cw_pair_type pair, total.top_words[i])  {\n        ASSERT_LT(pair.second, DICTIONARY.size());\n        // std::cout << DICTIONARY[pair.second] \n        //           << \"(\" << pair.first << \")\" << \", \"; \n        std::cout << DICTIONARY[pair.second] << \",  \";\n      }\n      std::cout << std::endl;\n    }\n    std::cout << \"\\nNumber of token changes: \" << total.nchanges << std::endl;\n    std::cout << \"\\nNumber of updates:       \" << total.nupdates << std::endl;\n  } // end of finalize\n}; // end of topk_aggregator struct\n\n\n\n\ntemplate\nstruct global_counts_aggregator {\n  typedef graph_type::vertex_type vertex_type;\n  static factor_type map(IContext& context, const vertex_type& vertex) {\n    return vertex.data().factor;\n  } // end of map function\n\n  static void finalize(IContext& context, const factor_type& total) {\n    for(size_t t = 0; t < total.size(); ++t)\n      GLOBAL_TOPIC_COUNT[t] =\n        std::max(count_type(total[t]/2), count_type(0));\n  } // end of finalize\n}; // end of global_counts_aggregator struct\n\n\n\n\ntemplate\nstruct selective_signal {\n static graphlab::empty \n docs(IContext& context, graph_type::vertex_type& vertex) {\n   if(is_doc(vertex)) context.signal(vertex);\n   return graphlab::empty();\n } // end of signal_docs\n static graphlab::empty \n words(IContext& context, graph_type::vertex_type& vertex) {\n    if(is_word(vertex)) context.signal(vertex);\n    return graphlab::empty();\n  } // end of signal_words\n}; // end of selective_signal\n\n\n\n\n\n\n\n\n#include \n#endif\n"
  },
  {
    "path": "toolkits/topic_modeling/deprecated/fast_cvb0_lda.cpp",
    "content": "/**  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \n#include \n#include \n\n\n#include \"cvb0_lda_common.hpp\"\n\n\n#include \n\n\nclass cvb0_lda_vertex_program :\n  public graphlab::ivertex_program,\n  public graphlab::IS_POD_TYPE {\npublic:\n\n  edge_dir_type gather_edges(icontext_type& context, \n                             const vertex_type& vertex) const {\n    return graphlab::ALL_EDGES;\n  } // end of gather_edges \n\n  factor_type gather(icontext_type& context, const vertex_type& vertex, \n                     edge_type& edge) const {\n    vertex_type other_vertex = get_other_vertex(edge, vertex);\n    // VIOLATING THE ABSTRACTION!\n    vertex_data& vdata = graph_type::vertex_type(vertex).data();\n    // VIOLATING THE ABSTRACTION!\n    vertex_data& other_vdata = other_vertex.data();\n    factor_type& doc_topic_count = \n      is_doc(vertex) ? vdata.factor : other_vdata.factor;\n    factor_type& word_topic_count = \n      is_word(vertex) ? vdata.factor : other_vdata.factor;\n    ASSERT_EQ(doc_topic_count.size(), NTOPICS);\n    ASSERT_EQ(word_topic_count.size(), NTOPICS);\n    // run the actual gibbs sampling \n    factor_type& belief = edge.data().belief;\n    const uint32_t count = edge.data().count;\n    // Resample the topics\n    double sum = 0, old_sum = 0;\n    for(size_t t = 0; t < NTOPICS; ++t) {\n      old_sum += belief[t];\n      doc_topic_count[t] -= belief[t];\n      word_topic_count[t] -= belief[t];\n      GLOBAL_TOPIC_COUNT[t] -= belief[t];\n      const double n_dt = \n        std::max(count_type(doc_topic_count[t]), count_type(0));\n      ASSERT_GE(n_dt, 0);\n      const double n_wt = \n        std::max(count_type(word_topic_count[t]), count_type(0)); \n      ASSERT_GE(n_wt, 0);\n      const double n_t  = \n        std::max(count_type(GLOBAL_TOPIC_COUNT[t]), count_type(0)); \n      ASSERT_GE(n_t, 0);\n      belief[t] = (ALPHA + n_dt) * (BETA + n_wt) / (BETA * NWORDS + n_t);\n      sum += belief[t];\n    } // End of loop over each token\n    ASSERT_GT(sum, 0);\n    if(old_sum == 0) {\n      size_t asg = graphlab::random::multinomial(belief);\n      for(size_t i = 0; i < NTOPICS; ++i) belief[i] = 0;\n      belief[asg] = count;\n      return belief;\n    }\n    for(size_t t = 0; t < NTOPICS; ++t) {\n      belief[t] = count * (belief[t]/sum);\n      doc_topic_count[t] += belief[t];\n      word_topic_count[t] += belief[t];\n      GLOBAL_TOPIC_COUNT[t] += belief[t];\n    }\n    return belief;\n  } // end of gather\n\n\n  void apply(icontext_type& context, vertex_type& vertex,\n             const factor_type& sum) {\n    const size_t num_neighbors = vertex.num_in_edges() + vertex.num_out_edges();\n    ASSERT_GT(num_neighbors, 0);\n    // There should be no new edge data since the vertex program has been cleared\n    vertex_data& vdata = vertex.data();\n    ASSERT_EQ(sum.size(), NTOPICS);\n    ASSERT_EQ(vdata.factor.size(), NTOPICS);\n    vdata.nupdates++; vdata.nchanges = 0; \n    vdata.factor = sum;\n  } // end of apply\n\n  edge_dir_type scatter_edges(icontext_type& context,\n                              const vertex_type& vertex) const { \n    return graphlab::ALL_EDGES; \n  }; // end of scatter edges\n\n  void scatter(icontext_type& context, const vertex_type& vertex, \n               edge_type& edge) const {\n    const vertex_type other_vertex = get_other_vertex(edge, vertex);\n    context.signal(other_vertex);\n  } // end of scatter function\n\n}; // end of cvb0_lda_vertex_program\n\n\n\n\n\ntypedef graphlab::omni_engine engine_type;\ntypedef cvb0_lda_vertex_program::icontext_type icontext_type;\ntypedef topk_aggregator topk_type;\ntypedef selective_signal signal_only;\ntypedef global_counts_aggregator global_counts_agg;\n\n\n\n\n\nint main(int argc, char** argv) {\n  global_logger().set_log_level(LOG_INFO);\n  global_logger().set_log_to_console(true);\n  ///! Initialize control plain using mpi\n  graphlab::mpi_tools::init(argc, argv);\n  graphlab::distributed_control dc;\n\n  // Parse command line options -----------------------------------------------\n  const std::string description = \n    \"\\n=========================================================================\\n\"\n    \"The fast Collapsed Variational Bayes Alg for the LDA model implements\\n\" \n    \"a highly asynchronous version of parallel CVB0 in which document\\n\"\n    \"and word counts are maintained in an eventually consistent\\n\"\n    \"manner.\\n\"\n    \"\\n\"\n    \"The standard usage is: \\n\"\n    \"\\t./fast_cvb0_lda --dictionary dictionary.txt --matrix doc_word_count.tsv\\n\"\n    \"where dictionary.txt contains: \\n\"\n    \"\\taaa \\n\\taaai \\n\\tabalone \\n\\t   ... \\n\\n\"\n    \"and doc_word_count.tsv is formatted   :\\n\"\n    \"\\t0\\t0\\t3\\n\"\n    \"\\t0\\t5\\t1\\n\"\n    \"\\t ...\\n\\n\"\n    \"To learn more about the NLP package and its applications visit\\n\\n\"\n    \"\\t\\t http://graphlab.org \\n\\n\"\n    \"Additional Options\";\n  graphlab::command_line_options clopts(description);\n  std::string matrix_dir; \n  std::string dictionary_fname;\n  clopts.attach_option(\"dictionary\", &dictionary_fname, dictionary_fname,\n                       \"The file containing the list of unique words\");\n  clopts.add_positional(\"dictionary\");\n  clopts.attach_option(\"matrix\", &matrix_dir, matrix_dir,\n                       \"The directory or file containing the matrix data.\");\n  clopts.add_positional(\"matrix\");\n  clopts.attach_option(\"ntopics\", &NTOPICS, NTOPICS,\n                       \"Number of topics to use.\");\n  clopts.attach_option(\"alpha\", &ALPHA, ALPHA,\n                       \"The document hyper-prior\");\n  clopts.attach_option(\"beta\", &BETA, BETA,                       \n                       \"The word hyper-prior\");\n  clopts.attach_option(\"topk\", &TOPK, TOPK,\n                       \"The number of words to report\");\n  clopts.attach_option(\"interval\", &INTERVAL, INTERVAL,\n                       \"statistics reporting interval\");\n  clopts.attach_option(\"max_count\", &MAX_COUNT, MAX_COUNT,\n                       \"The maximum number of occurences of a word in a document.\");\n  if(!clopts.parse(argc, argv)) {\n    graphlab::mpi_tools::finalize();\n    return clopts.is_set(\"help\")? EXIT_SUCCESS : EXIT_FAILURE;\n  }\n\n  if(dictionary_fname.empty()) {\n    logstream(LOG_ERROR) << \"No dictionary file was provided.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  if(matrix_dir.empty()) {\n    logstream(LOG_ERROR) << \"No matrix file was provided.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n  ///! Initialize global variables\n  GLOBAL_TOPIC_COUNT.resize(NTOPICS);\n  bool success = load_dictionary(dictionary_fname); \n  if(!success) {\n    logstream(LOG_ERROR) << \"Error loading dictionary.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n  \n  ///! load the graph\n  graph_type graph(dc, clopts);  \n  success = load_and_initialize_graph(dc, graph, matrix_dir);\n  if(!success) {\n    logstream(LOG_ERROR) << \"Error loading graph.\" << std::endl;\n    return EXIT_FAILURE;\n  }\n\n\n  \n  engine_type engine(dc, graph, clopts, \"asynchronous\");\n  ///! Add an aggregator\n  success = \n    engine.add_vertex_aggregator\n    (\"topk\", topk_type::map, topk_type::finalize) &&\n    engine.aggregate_periodic(\"topk\", INTERVAL);\n  ASSERT_TRUE(success);\n  success = \n    engine.add_vertex_aggregator\n    (\"global_counts\", global_counts_agg::map, global_counts_agg::finalize) &&\n    engine.aggregate_periodic(\"global_counts\", 5);\n  ASSERT_TRUE(success);\n\n\n  ///! schedule only documents\n  dc.cout() << \"Running The Collapsed Gibbs Sampler\" << std::endl;\n  engine.map_reduce_vertices(signal_only::docs);\n  graphlab::timer timer;\n  engine.start();  \n  const double runtime = timer.current_time();\n    dc.cout() \n    << \"----------------------------------------------------------\" << std::endl\n    << \"Final Runtime (seconds):   \" << runtime \n    << std::endl\n    << \"Updates executed: \" << engine.num_updates() << std::endl\n    << \"Update Rate (updates/second): \" \n    << engine.num_updates() / runtime << std::endl;\n\n\n\n  graphlab::mpi_tools::finalize();\n  return EXIT_SUCCESS;\n\n\n} // end of main\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
  },
  {
    "path": "toolkits/topic_modeling/http/index.html",
    "content": "\n\n\n\n\n\n\n\n\n\n\n\n\nLDA WordCloud Viewer \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
 WordCloud Visualizer 
\n
\n\n\n
\n\n
\n  
 \n   
NTopics
 \n   
\n  
\n\n  
 \n   
NWords
 \n   
\n  
\n\n  
 \n   
NDocs
 \n   
\n  
\n  \n  
 \n   
NTokens
   \n   
\n  
\n\n  
 \n   
Alpha
 \n   
\n  
\n\n  
 \n   
Beta
 \n   
\n  \n\n
\n\n\n\n\n\n\n\n"
  },
  {
    "path": "toolkits/topic_modeling/http/lda_visualizer.js",
    "content": "google.load(\"jquery\", \"1.5\");\ngoogle.load(\"jqueryui\", \"1.7.2\");\ngoogle.load(\"visualization\", \"1\");\n\n\nvar domain_str = \"http://localhost:8090\";\nvar page_str = \"/wordclouds\";\nvar update_interval = 2000;\n\nfunction update_domain(form) {\n    domain_str = form.inputbox.value;\n    get_top_words();\n}\n\nvar term_clouds = [];\n\n// Start the rendering of the UI\ngoogle.setOnLoadCallback(function() { \n    get_top_words();\n});\n\n\nfunction get_top_words() {\n    jQuery.getJSON(domain_str + page_str, process_top_words).error(function() { \n            console.log(\"Unable to access \" + domain_str + \" will try again.\");\n        }).complete(function() {\n            setTimeout(get_top_words, update_interval);\n        });\n}\n\n\n\n\n\nfunction process_top_words(data) {\n    // Load summary info\n    $(\"#ntopics\").text(data.ntopics);\n    $(\"#nwords\").text(data.nwords);\n    $(\"#ndocs\").text(data.ndocs);\n    $(\"#ntokens\").text(data.ntokens);\n    $(\"#alpha\").text(data.alpha);\n    $(\"#beta\").text(data.beta);   \n\n    // Render all the current values\n    var container = $(\"#word_clouds\");\n    \n    jQuery.each(data.values, function(i, term_count_table) {        \n        if(term_clouds[i] == undefined) {\n            var div_name = \"term_cloud_\" + i;            \n            container.append(\n                \"
\");\n            var div = container.children(\"#\" + div_name);            \n            var cloud = new TermCloud(div[0]);\n            term_clouds[i] = { div: div, cloud: cloud };\n        }\n        var labels = [[\"String\", \"Value\"]];        \n        var table_data =  labels.concat(term_count_table);\n        var table = google.visualization.arrayToDataTable(table_data);\n        table.addColumn(\"string\", \"URL\");\n        //        console.log(table);\n        term_clouds[i].cloud.draw(table, null );\n    });\n    // Get the job info again\n} // end of process top words\n\n\n"
  },
  {
    "path": "toolkits/topic_modeling/http/style.css",
    "content": ".body {\n    text-align: center;\n\n}\n.cloud {\n    display: inline-block;\n    width: 200px;\n    height: 200px;\n}\n\n.setting {\n    display: inline-block;\n    width: 100px;\n}\n\n.label {\n    font-size: 14pt;\n}\n"
  },
  {
    "path": "toolkits/topic_modeling/http/wordclouds",
    "content": "{\n\t\"ntopics\": 50,\n\t\"nwords\":  8349328,\n\t\"ndocs\":   2601668,\n\t\"ntokens\": 479126338,\n\t\"alpha\":   0.5,\n\t\"beta\":    0.1,\n\t\"values\": [\n\t[\n\t\t[\"river\", 260462], \n\t\t[\"water\", 154470], \n\t\t[\"lake\", 137116], \n\t\t[\"mountain\", 87756], \n\t\t[\"area\", 80303], \n\t\t[\"north\", 68834], \n\t\t[\"sea\", 68785], \n\t\t[\"valley\", 65516], \n\t\t[\"park\", 59788], \n\t\t[\"south\", 52968], \n\t\t[\"forest\", 51247], \n\t\t[\"mountains\", 50866], \n\t\t[\"creek\", 42652], \n\t\t[\"rock\", 38748], \n\t\t[\"ice\", 37717], \n\t\t[\"range\", 36985], \n\t\t[\"west\", 34535], \n\t\t[\"high\", 34438], \n\t\t[\"dam\", 34158], \n\t\t[\"located\", 33700], \n\t\t[\"island\", 33345], \n\t\t[\"fish\", 33186], \n\t\t[\"national\", 32635], \n\t\t[\"bay\", 32511], \n\t\t[\"trail\", 31835], \n\t\t[\"ocean\", 31464], \n\t\t[\"east\", 31423], \n\t\t[\"winter\", 31241], \n\t\t[\"rivers\", 30637], \n\t\t[\"northern\", 30126], \n\t\t[\"areas\", 30011], \n\t\t[\"food\", 29762], \n\t\t[\"beach\", 29407], \n\t\t[\"wine\", 29129], \n\t\t[\"flows\", 28804], \n\t\t[\"southern\", 28628], \n\t\t[\"mount\", 28387], \n\t\t[\"fishing\", 28031], \n\t\t[\"summer\", 27191], \n\t\t[\"long\", 26993], \n\t\t[\"region\", 26712], \n\t\t[\"small\", 26610], \n\t\t[\"large\", 26522], \n\t\t[\"coast\", 26439], \n\t\t[\"lakes\", 26206], \n\t\t[\"years\", 25754], \n\t\t[\"natural\", 25746], \n\t\t[\"point\", 25608], \n\t\t[\"basin\", 24581], \n\t\t[\"great\", 24348]\n\t], \n\t[\n\t\t[\"number\", 59354], \n\t\t[\"space\", 47824], \n\t\t[\"set\", 47815], \n\t\t[\"function\", 46299], \n\t\t[\"theory\", 46070], \n\t\t[\"example\", 41449], \n\t\t[\"time\", 41122], \n\t\t[\"1\", 39395], \n\t\t[\"point\", 38591], \n\t\t[\"field\", 38161], \n\t\t[\"system\", 35163], \n\t\t[\"will\", 33158], \n\t\t[\"called\", 32081], \n\t\t[\"group\", 30044], \n\t\t[\"case\", 29285], \n\t\t[\"model\", 28084], \n\t\t[\"form\", 27220], \n\t\t[\"defined\", 26408], \n\t\t[\"numbers\", 25599], \n\t\t[\"problem\", 24195], \n\t\t[\"0\", 23684], \n\t\t[\"method\", 23448], \n\t\t[\"order\", 22210], \n\t\t[\"points\", 21109], \n\t\t[\"values\", 20471], \n\t\t[\"general\", 20341], \n\t\t[\"functions\", 19980], \n\t\t[\"equation\", 19596], \n\t\t[\"light\", 19328], \n\t\t[\"distance\", 19263], \n\t\t[\"2\", 19161], \n\t\t[\"linear\", 18590], \n\t\t[\"complex\", 17839], \n\t\t[\"data\", 17699], \n\t\t[\"mathematical\", 17674], \n\t\t[\"quantum\", 17593], \n\t\t[\"theorem\", 17420], \n\t\t[\"constant\", 17157], \n\t\t[\"sequence\", 16984], \n\t\t[\"result\", 16901], \n\t\t[\"equal\", 16793], \n\t\t[\"terms\", 16791], \n\t\t[\"mass\", 16653], \n\t\t[\"frequency\", 16355], \n\t\t[\"elements\", 16339], \n\t\t[\"real\", 16260], \n\t\t[\"object\", 16235], \n\t\t[\"measure\", 16193], \n\t\t[\"surface\", 16155], \n\t\t[\"distribution\", 16118]\n\t], \n\t[\n\t\t[\"chinese\", 117128], \n\t\t[\"china\", 108771], \n\t\t[\"hong\", 49992], \n\t\t[\"kong\", 48754], \n\t\t[\"emperor\", 40718], \n\t\t[\"swedish\", 40242], \n\t\t[\"norwegian\", 35603], \n\t\t[\"sweden\", 35590], \n\t\t[\"norway\", 31796], \n\t\t[\"li\", 30072], \n\t\t[\"singapore\", 27479], \n\t\t[\"danish\", 25759], \n\t\t[\"taiwan\", 21623], \n\t\t[\"denmark\", 21541], \n\t\t[\"vietnam\", 20716], \n\t\t[\"province\", 18312], \n\t\t[\"dynasty\", 18155], \n\t\t[\"thailand\", 17941], \n\t\t[\"wang\", 17617], \n\t\t[\"finland\", 17472], \n\t\t[\"finnish\", 16520], \n\t\t[\"beijing\", 15304], \n\t\t[\"liu\", 15226], \n\t\t[\"modern\", 15080], \n\t\t[\"han\", 14879], \n\t\t[\"vietnamese\", 14198], \n\t\t[\"shanghai\", 14088], \n\t\t[\"wu\", 13748], \n\t\t[\"prince\", 13740], \n\t\t[\"thai\", 13254], \n\t\t[\"asia\", 12640], \n\t\t[\"yang\", 12389], \n\t\t[\"zhang\", 12382], \n\t\t[\"government\", 12224], \n\t\t[\"son\", 12155], \n\t\t[\"imperial\", 12104], \n\t\t[\"stockholm\", 11894], \n\t\t[\"oslo\", 11566], \n\t\t[\"yuan\", 11455], \n\t\t[\"hawaii\", 11328], \n\t\t[\"asian\", 11198], \n\t\t[\"chen\", 11143], \n\t\t[\"sun\", 11118], \n\t\t[\"tang\", 10570], \n\t\t[\"general\", 10416], \n\t\t[\"wei\", 10070], \n\t\t[\"capital\", 9970], \n\t\t[\"northern\", 9943], \n\t\t[\"copenhagen\", 9879], \n\t\t[\"peoples\", 9820]\n\t], \n\t[\n\t\t[\"station\", 218615], \n\t\t[\"road\", 211136], \n\t\t[\"route\", 166646], \n\t\t[\"railway\", 162110], \n\t\t[\"bridge\", 117385], \n\t\t[\"north\", 93174], \n\t\t[\"highway\", 90290], \n\t\t[\"service\", 89930], \n\t\t[\"east\", 73950], \n\t\t[\"south\", 70309], \n\t\t[\"train\", 69541], \n\t\t[\"trains\", 67179], \n\t\t[\"west\", 66155], \n\t\t[\"bus\", 63332], \n\t\t[\"street\", 63221], \n\t\t[\"rail\", 61975], \n\t\t[\"traffic\", 61103], \n\t\t[\"city\", 55517], \n\t\t[\"built\", 51505], \n\t\t[\"opened\", 51184], \n\t\t[\"services\", 50737], \n\t\t[\"lines\", 47599], \n\t\t[\"railroad\", 46751], \n\t\t[\"main\", 43720], \n\t\t[\"construction\", 42361], \n\t\t[\"river\", 41844], \n\t\t[\"routes\", 38736], \n\t\t[\"junction\", 38702], \n\t\t[\"stations\", 37598], \n\t\t[\"canal\", 37036], \n\t\t[\"passenger\", 36131], \n\t\t[\"system\", 35540], \n\t\t[\"transport\", 35523], \n\t\t[\"operated\", 35131], \n\t\t[\"avenue\", 33983], \n\t\t[\"branch\", 33422], \n\t\t[\"track\", 31377], \n\t\t[\"runs\", 31158], \n\t\t[\"tunnel\", 30271], \n\t\t[\"central\", 30257], \n\t\t[\"roads\", 30175], \n\t\t[\"side\", 27881], \n\t\t[\"railways\", 26943], \n\t\t[\"transit\", 26668], \n\t\t[\"buses\", 26533], \n\t\t[\"area\", 26351], \n\t\t[\"passengers\", 25829], \n\t\t[\"located\", 25769], \n\t\t[\"southern\", 25737], \n\t\t[\"crossing\", 25642]\n\t], \n\t[\n\t\t[\"africa\", 72052], \n\t\t[\"african\", 58981], \n\t\t[\"israel\", 45272], \n\t\t[\"south\", 38017], \n\t\t[\"islamic\", 36077], \n\t\t[\"arab\", 34470], \n\t\t[\"israeli\", 32276], \n\t\t[\"muslim\", 32113], \n\t\t[\"al\", 31321], \n\t\t[\"iran\", 30762], \n\t\t[\"iraq\", 30647], \n\t\t[\"government\", 29202], \n\t\t[\"turkish\", 24919], \n\t\t[\"international\", 24117], \n\t\t[\"turkey\", 22596], \n\t\t[\"egypt\", 21825], \n\t\t[\"islam\", 21774], \n\t\t[\"united\", 20860], \n\t\t[\"ali\", 20493], \n\t\t[\"afghanistan\", 20104], \n\t\t[\"country\", 20003], \n\t\t[\"muslims\", 18215], \n\t\t[\"iranian\", 17513], \n\t\t[\"ibn\", 17238], \n\t\t[\"muhammad\", 17186], \n\t\t[\"nations\", 16723], \n\t\t[\"military\", 16531], \n\t\t[\"british\", 16381], \n\t\t[\"group\", 16381], \n\t\t[\"east\", 16358], \n\t\t[\"palestinian\", 16130], \n\t\t[\"arabic\", 16037], \n\t\t[\"lebanon\", 15922], \n\t\t[\"persian\", 15509], \n\t\t[\"iraqi\", 15374], \n\t\t[\"security\", 15325], \n\t\t[\"armenian\", 15050], \n\t\t[\"saudi\", 15009], \n\t\t[\"countries\", 14815], \n\t\t[\"war\", 14277], \n\t\t[\"nigeria\", 14011], \n\t\t[\"abu\", 13779], \n\t\t[\"egyptian\", 13540], \n\t\t[\"palestine\", 13334], \n\t\t[\"republic\", 13297], \n\t\t[\"jordan\", 13088], \n\t\t[\"foreign\", 12990], \n\t\t[\"national\", 12911], \n\t\t[\"middle\", 12839], \n\t\t[\"arabia\", 12801]\n\t], \n\t[\n\t\t[\"god\", 93661], \n\t\t[\"jewish\", 69449], \n\t\t[\"jews\", 35911], \n\t\t[\"religious\", 34013], \n\t\t[\"temple\", 31507], \n\t\t[\"jesus\", 29290], \n\t\t[\"gods\", 27329], \n\t\t[\"day\", 26164], \n\t\t[\"religion\", 25225], \n\t\t[\"rabbi\", 24187], \n\t\t[\"christian\", 22783], \n\t\t[\"life\", 22730], \n\t\t[\"spiritual\", 22025], \n\t\t[\"will\", 21864], \n\t\t[\"tradition\", 20805], \n\t\t[\"people\", 20758], \n\t\t[\"text\", 20618], \n\t\t[\"called\", 20473], \n\t\t[\"hebrew\", 20082], \n\t\t[\"bible\", 19857], \n\t\t[\"man\", 18481], \n\t\t[\"israel\", 17916], \n\t\t[\"spirit\", 17889], \n\t\t[\"divine\", 17375], \n\t\t[\"book\", 17297], \n\t\t[\"great\", 16997], \n\t\t[\"three\", 16652], \n\t\t[\"faith\", 15460], \n\t\t[\"death\", 15099], \n\t\t[\"ancient\", 15076], \n\t\t[\"buddhist\", 14703], \n\t\t[\"time\", 14341], \n\t\t[\"testament\", 13416], \n\t\t[\"jerusalem\", 13349], \n\t\t[\"scholars\", 13341], \n\t\t[\"century\", 12900], \n\t\t[\"biblical\", 12711], \n\t\t[\"prayer\", 12398], \n\t\t[\"practice\", 12319], \n\t\t[\"word\", 12218], \n\t\t[\"ritual\", 12205], \n\t\t[\"written\", 12073], \n\t\t[\"sacred\", 11998], \n\t\t[\"worship\", 11720], \n\t\t[\"judaism\", 11636], \n\t\t[\"buddhism\", 11563], \n\t\t[\"earth\", 11533], \n\t\t[\"son\", 11524], \n\t\t[\"soul\", 11523], \n\t\t[\"lord\", 11432]\n\t], \n\t[\n\t\t[\"will\", 94707], \n\t\t[\"father\", 90397], \n\t\t[\"family\", 89269], \n\t\t[\"mother\", 85666], \n\t\t[\"man\", 84192], \n\t\t[\"love\", 80756], \n\t\t[\"tells\", 72378], \n\t\t[\"life\", 71707], \n\t\t[\"house\", 69787], \n\t\t[\"time\", 60776], \n\t\t[\"find\", 58992], \n\t\t[\"woman\", 57520], \n\t\t[\"friends\", 53698], \n\t\t[\"story\", 51044], \n\t\t[\"day\", 50993], \n\t\t[\"young\", 49865], \n\t\t[\"friend\", 49143], \n\t\t[\"help\", 48819], \n\t\t[\"wife\", 48554], \n\t\t[\"girl\", 47015], \n\t\t[\"night\", 46902], \n\t\t[\"takes\", 45610], \n\t\t[\"finds\", 44474], \n\t\t[\"relationship\", 43821], \n\t\t[\"episode\", 41882], \n\t\t[\"child\", 40538], \n\t\t[\"daughter\", 38850], \n\t\t[\"son\", 38504], \n\t\t[\"brother\", 38075], \n\t\t[\"room\", 38034], \n\t\t[\"money\", 37618], \n\t\t[\"children\", 37584], \n\t\t[\"parents\", 37200], \n\t\t[\"death\", 36815], \n\t\t[\"leave\", 35373], \n\t\t[\"police\", 35105], \n\t\t[\"going\", 34058], \n\t\t[\"eventually\", 33450], \n\t\t[\"boy\", 33386], \n\t\t[\"jack\", 32434], \n\t\t[\"car\", 32434], \n\t\t[\"sister\", 32160], \n\t\t[\"told\", 31823], \n\t\t[\"asks\", 31298], \n\t\t[\"husband\", 29418], \n\t\t[\"left\", 29167], \n\t\t[\"job\", 29145], \n\t\t[\"girls\", 29071], \n\t\t[\"named\", 28463], \n\t\t[\"michael\", 28428]\n\t], \n\t[\n\t\t[\"japan\", 101530], \n\t\t[\"japanese\", 101059], \n\t\t[\"fight\", 34890], \n\t\t[\"korea\", 34406], \n\t\t[\"korean\", 30948], \n\t\t[\"tokyo\", 28879], \n\t\t[\"title\", 23513], \n\t\t[\"anime\", 17698], \n\t\t[\"manga\", 16643], \n\t\t[\"round\", 16027], \n\t\t[\"boxing\", 14448], \n\t\t[\"series\", 12374], \n\t\t[\"decision\", 11629], \n\t\t[\"kim\", 11563], \n\t\t[\"martial\", 10966], \n\t\t[\"lost\", 8766], \n\t\t[\"champion\", 8642], \n\t\t[\"fighting\", 8642], \n\t\t[\"prefecture\", 8376], \n\t\t[\"defeated\", 7824], \n\t\t[\"japans\", 7758], \n\t\t[\"three\", 7702], \n\t\t[\"second\", 7457], \n\t\t[\"bout\", 7450], \n\t\t[\"seoul\", 7438], \n\t\t[\"fought\", 7420], \n\t\t[\"time\", 7114], \n\t\t[\"period\", 7095], \n\t\t[\"heavyweight\", 7078], \n\t\t[\"ufc\", 6874], \n\t\t[\"fights\", 6863], \n\t\t[\"called\", 6344], \n\t\t[\"imperial\", 6192], \n\t\t[\"fighter\", 6034], \n\t\t[\"south\", 6021], \n\t\t[\"clan\", 5980], \n\t\t[\"osaka\", 5833], \n\t\t[\"belt\", 5807], \n\t\t[\"main\", 5709], \n\t\t[\"training\", 5633], \n\t\t[\"super\", 5459], \n\t\t[\"samurai\", 5436], \n\t\t[\"fighters\", 5355], \n\t\t[\"lee\", 5331], \n\t\t[\"karate\", 5221], \n\t\t[\"arts\", 5206], \n\t\t[\"named\", 5186], \n\t\t[\"knockout\", 5155], \n\t\t[\"kyoto\", 5121], \n\t\t[\"boxer\", 5121]\n\t], \n\t[\n\t\t[\"company\", 285451], \n\t\t[\"business\", 155572], \n\t\t[\"services\", 109733], \n\t\t[\"companies\", 97954], \n\t\t[\"market\", 92489], \n\t\t[\"international\", 91872], \n\t\t[\"development\", 91070], \n\t\t[\"management\", 86497], \n\t\t[\"government\", 84136], \n\t\t[\"bank\", 83545], \n\t\t[\"financial\", 79530], \n\t\t[\"industry\", 77810], \n\t\t[\"united\", 74349], \n\t\t[\"public\", 69466], \n\t\t[\"will\", 65405], \n\t\t[\"group\", 64362], \n\t\t[\"economic\", 62847], \n\t\t[\"service\", 60081], \n\t\t[\"countries\", 59365], \n\t\t[\"trade\", 56501], \n\t\t[\"products\", 54904], \n\t\t[\"including\", 49870], \n\t\t[\"private\", 49795], \n\t\t[\"national\", 49680], \n\t\t[\"project\", 49414], \n\t\t[\"tax\", 47754], \n\t\t[\"corporation\", 46149], \n\t\t[\"price\", 44329], \n\t\t[\"money\", 43774], \n\t\t[\"largest\", 42667], \n\t\t[\"year\", 42205], \n\t\t[\"billion\", 41753], \n\t\t[\"investment\", 41661], \n\t\t[\"sold\", 40125], \n\t\t[\"capital\", 38712], \n\t\t[\"years\", 38620], \n\t\t[\"global\", 38366], \n\t\t[\"employees\", 38014], \n\t\t[\"production\", 37815], \n\t\t[\"exchange\", 37379], \n\t\t[\"health\", 37288], \n\t\t[\"provide\", 37122], \n\t\t[\"cost\", 37070], \n\t\t[\"projects\", 36951], \n\t\t[\"major\", 36395], \n\t\t[\"sales\", 36384], \n\t\t[\"established\", 36025], \n\t\t[\"policy\", 35937], \n\t\t[\"agency\", 35357], \n\t\t[\"stock\", 35202]\n\t], \n\t[\n\t\t[\"air\", 233008], \n\t\t[\"aircraft\", 159233], \n\t\t[\"flight\", 71583], \n\t\t[\"force\", 69627], \n\t\t[\"airport\", 67388], \n\t\t[\"flying\", 47659], \n\t\t[\"wing\", 45639], \n\t\t[\"base\", 40380], \n\t\t[\"storm\", 37845], \n\t\t[\"squadron\", 37632], \n\t\t[\"space\", 37140], \n\t\t[\"pilot\", 34437], \n\t\t[\"aviation\", 33698], \n\t\t[\"mission\", 32739], \n\t\t[\"operations\", 31167], \n\t\t[\"fighter\", 29713], \n\t\t[\"united\", 29583], \n\t\t[\"training\", 29079], \n\t\t[\"system\", 27502], \n\t\t[\"airlines\", 26439], \n\t\t[\"tropical\", 26179], \n\t\t[\"hurricane\", 25897], \n\t\t[\"service\", 25349], \n\t\t[\"pilots\", 23754], \n\t\t[\"flights\", 23114], \n\t\t[\"missile\", 22827], \n\t\t[\"crew\", 21962], \n\t\t[\"raf\", 21869], \n\t\t[\"control\", 21739], \n\t\t[\"group\", 21450], \n\t\t[\"ground\", 21325], \n\t\t[\"flew\", 21136], \n\t\t[\"landing\", 20774], \n\t\t[\"launch\", 20689], \n\t\t[\"radar\", 20574], \n\t\t[\"airline\", 19017], \n\t\t[\"plane\", 18911], \n\t\t[\"missions\", 18601], \n\t\t[\"time\", 18264], \n\t\t[\"three\", 17917], \n\t\t[\"damage\", 17260], \n\t\t[\"rocket\", 16729], \n\t\t[\"jet\", 16662], \n\t\t[\"military\", 16334], \n\t\t[\"center\", 16317], \n\t\t[\"international\", 15841], \n\t\t[\"unit\", 15820], \n\t\t[\"test\", 15699], \n\t\t[\"support\", 15617], \n\t\t[\"early\", 15519]\n\t], \n\t[\n\t\t[\"social\", 117849], \n\t\t[\"people\", 104088], \n\t\t[\"human\", 96367], \n\t\t[\"work\", 76026], \n\t\t[\"example\", 75910], \n\t\t[\"will\", 75046], \n\t\t[\"society\", 70991], \n\t\t[\"women\", 66998], \n\t\t[\"theory\", 62441], \n\t\t[\"life\", 53499], \n\t\t[\"nature\", 47164], \n\t\t[\"groups\", 46975], \n\t\t[\"movement\", 46907], \n\t\t[\"based\", 46219], \n\t\t[\"study\", 45798], \n\t\t[\"individual\", 45676], \n\t\t[\"time\", 44975], \n\t\t[\"culture\", 43671], \n\t\t[\"term\", 43671], \n\t\t[\"knowledge\", 43549], \n\t\t[\"view\", 43517], \n\t\t[\"process\", 42888], \n\t\t[\"political\", 42806], \n\t\t[\"form\", 41585], \n\t\t[\"common\", 41406], \n\t\t[\"concept\", 40469], \n\t\t[\"considered\", 40453], \n\t\t[\"development\", 39545], \n\t\t[\"person\", 38333], \n\t\t[\"modern\", 37511], \n\t\t[\"system\", 37313], \n\t\t[\"idea\", 37139], \n\t\t[\"change\", 37119], \n\t\t[\"ideas\", 35671], \n\t\t[\"approach\", 35582], \n\t\t[\"developed\", 34599], \n\t\t[\"individuals\", 33850], \n\t\t[\"group\", 33834], \n\t\t[\"sense\", 33568], \n\t\t[\"sexual\", 33500], \n\t\t[\"philosophy\", 33365], \n\t\t[\"practice\", 32248], \n\t\t[\"history\", 32151], \n\t\t[\"generally\", 31362], \n\t\t[\"experience\", 30769], \n\t\t[\"subject\", 30594], \n\t\t[\"cultural\", 30274], \n\t\t[\"influence\", 30003], \n\t\t[\"american\", 29215], \n\t\t[\"scientific\", 29152]\n\t], \n\t[\n\t\t[\"series\", 65608], \n\t\t[\"earth\", 53229], \n\t\t[\"time\", 50167], \n\t\t[\"power\", 47313], \n\t\t[\"battle\", 39450], \n\t\t[\"story\", 36420], \n\t\t[\"powers\", 32912], \n\t\t[\"planet\", 32725], \n\t\t[\"character\", 32060], \n\t\t[\"human\", 31386], \n\t\t[\"magic\", 29620], \n\t\t[\"appears\", 29023], \n\t\t[\"doctor\", 27237], \n\t\t[\"will\", 27222], \n\t\t[\"dark\", 26562], \n\t\t[\"man\", 25171], \n\t\t[\"evil\", 24866], \n\t\t[\"attack\", 24804], \n\t\t[\"characters\", 24769], \n\t\t[\"black\", 24738], \n\t\t[\"death\", 24664], \n\t\t[\"universe\", 24600], \n\t\t[\"form\", 23619], \n\t\t[\"body\", 23146], \n\t\t[\"team\", 22832], \n\t\t[\"killed\", 22507], \n\t\t[\"comics\", 22430], \n\t\t[\"kill\", 22355], \n\t\t[\"eventually\", 22240], \n\t\t[\"original\", 21975], \n\t\t[\"war\", 21342], \n\t\t[\"fight\", 21247], \n\t\t[\"help\", 20763], \n\t\t[\"dragon\", 20753], \n\t\t[\"created\", 20708], \n\t\t[\"life\", 20669], \n\t\t[\"revealed\", 20266], \n\t\t[\"space\", 20079], \n\t\t[\"named\", 20034], \n\t\t[\"star\", 19931], \n\t\t[\"ship\", 19925], \n\t\t[\"called\", 19653], \n\t\t[\"control\", 19524], \n\t\t[\"find\", 19494], \n\t\t[\"group\", 19049], \n\t\t[\"alien\", 18945], \n\t\t[\"captain\", 18606], \n\t\t[\"powerful\", 18572], \n\t\t[\"order\", 18050], \n\t\t[\"return\", 17992]\n\t], \n\t[\n\t\t[\"india\", 148568], \n\t\t[\"indian\", 91540], \n\t\t[\"pakistan\", 43810], \n\t\t[\"temple\", 37909], \n\t\t[\"sri\", 35298], \n\t\t[\"khan\", 30039], \n\t\t[\"singh\", 24930], \n\t\t[\"district\", 22364], \n\t\t[\"tamil\", 21134], \n\t\t[\"delhi\", 20166], \n\t\t[\"hindu\", 20009], \n\t\t[\"famous\", 19594], \n\t\t[\"british\", 19013], \n\t\t[\"village\", 18641], \n\t\t[\"people\", 17676], \n\t\t[\"lanka\", 17074], \n\t\t[\"bangladesh\", 15089], \n\t\t[\"place\", 14848], \n\t\t[\"bengal\", 14839], \n\t\t[\"called\", 14839], \n\t\t[\"started\", 14216], \n\t\t[\"pradesh\", 13552], \n\t\t[\"punjab\", 13256], \n\t\t[\"kerala\", 12910], \n\t\t[\"government\", 12611], \n\t\t[\"mumbai\", 12480], \n\t\t[\"family\", 12424], \n\t\t[\"lord\", 12266], \n\t\t[\"shah\", 11677], \n\t\t[\"muslim\", 10955], \n\t\t[\"raja\", 10531], \n\t\t[\"guru\", 10127], \n\t\t[\"nepal\", 10126], \n\t\t[\"krishna\", 9927], \n\t\t[\"ali\", 9782], \n\t\t[\"indias\", 9363], \n\t\t[\"hindi\", 9250], \n\t\t[\"temples\", 9131], \n\t\t[\"bombay\", 8845], \n\t\t[\"son\", 8751], \n\t\t[\"pakistani\", 8686], \n\t\t[\"day\", 8570], \n\t\t[\"till\", 8389], \n\t\t[\"main\", 8222], \n\t\t[\"madras\", 8119], \n\t\t[\"major\", 8119], \n\t\t[\"gandhi\", 8095], \n\t\t[\"shiva\", 8084], \n\t\t[\"time\", 7964], \n\t\t[\"sabha\", 7914]\n\t], \n\t[\n\t\t[\"church\", 379972], \n\t\t[\"st\", 171048], \n\t\t[\"catholic\", 107879], \n\t\t[\"bishop\", 80819], \n\t\t[\"christian\", 68886], \n\t\t[\"saint\", 68356], \n\t\t[\"pope\", 50786], \n\t\t[\"churches\", 50319], \n\t\t[\"parish\", 49519], \n\t\t[\"roman\", 48606], \n\t\t[\"religious\", 41677], \n\t\t[\"cathedral\", 41340], \n\t\t[\"holy\", 39495], \n\t\t[\"john\", 36213], \n\t\t[\"diocese\", 32853], \n\t\t[\"christ\", 31852], \n\t\t[\"century\", 30111], \n\t\t[\"chapel\", 29020], \n\t\t[\"year\", 27677], \n\t\t[\"priest\", 26000], \n\t\t[\"paul\", 25944], \n\t\t[\"archbishop\", 25759], \n\t\t[\"bishops\", 24389], \n\t\t[\"monastery\", 23467], \n\t\t[\"mary\", 23176], \n\t\t[\"abbey\", 23106], \n\t\t[\"order\", 23035], \n\t\t[\"mission\", 21333], \n\t\t[\"orthodox\", 21196], \n\t\t[\"council\", 21051], \n\t\t[\"founded\", 19932], \n\t\t[\"saints\", 19454], \n\t\t[\"faith\", 19425], \n\t\t[\"theology\", 18786], \n\t\t[\"rome\", 18611], \n\t\t[\"congregation\", 18387], \n\t\t[\"baptist\", 18365], \n\t\t[\"cardinal\", 18137], \n\t\t[\"seminary\", 18002], \n\t\t[\"protestant\", 17006], \n\t\t[\"jesus\", 16969], \n\t\t[\"time\", 16921], \n\t\t[\"years\", 16178], \n\t\t[\"cross\", 16090], \n\t\t[\"ordained\", 15762], \n\t\t[\"life\", 15652], \n\t\t[\"rev\", 15449], \n\t\t[\"theological\", 15308], \n\t\t[\"priests\", 14920], \n\t\t[\"missionary\", 14743]\n\t], \n\t[\n\t\t[\"time\", 253091], \n\t\t[\"years\", 155361], \n\t\t[\"people\", 104756], \n\t\t[\"day\", 104464], \n\t\t[\"death\", 98956], \n\t\t[\"began\", 88619], \n\t\t[\"police\", 87021], \n\t\t[\"left\", 86479], \n\t\t[\"led\", 81651], \n\t\t[\"days\", 80684], \n\t\t[\"claimed\", 71065], \n\t\t[\"despite\", 70647], \n\t\t[\"months\", 65961], \n\t\t[\"decided\", 64082], \n\t\t[\"case\", 60374], \n\t\t[\"men\", 59711], \n\t\t[\"continued\", 59195], \n\t\t[\"three\", 59073], \n\t\t[\"reported\", 58914], \n\t\t[\"stated\", 57336], \n\t\t[\"asked\", 56990], \n\t\t[\"prison\", 56931], \n\t\t[\"return\", 55748], \n\t\t[\"eventually\", 54135], \n\t\t[\"forced\", 53807], \n\t\t[\"refused\", 53627], \n\t\t[\"called\", 52745], \n\t\t[\"early\", 52195], \n\t\t[\"brought\", 50917], \n\t\t[\"set\", 50116], \n\t\t[\"met\", 48529], \n\t\t[\"result\", 48261], \n\t\t[\"told\", 48232], \n\t\t[\"trial\", 47790], \n\t\t[\"caused\", 47731], \n\t\t[\"life\", 47473], \n\t\t[\"killed\", 46955], \n\t\t[\"order\", 46905], \n\t\t[\"allowed\", 46225], \n\t\t[\"long\", 46022], \n\t\t[\"report\", 45690], \n\t\t[\"wanted\", 45532], \n\t\t[\"failed\", 44595], \n\t\t[\"fact\", 44577], \n\t\t[\"attempt\", 44387], \n\t\t[\"man\", 43862], \n\t\t[\"agreed\", 43792], \n\t\t[\"believed\", 43738], \n\t\t[\"received\", 43719], \n\t\t[\"returned\", 42847]\n\t], \n\t[\n\t\t[\"2009\", 366418], \n\t\t[\"2008\", 363976], \n\t\t[\"2007\", 340817], \n\t\t[\"2010\", 340449], \n\t\t[\"2006\", 275341], \n\t\t[\"1\", 270064], \n\t\t[\"september\", 231466], \n\t\t[\"june\", 224598], \n\t\t[\"october\", 223378], \n\t\t[\"january\", 221942], \n\t\t[\"july\", 221320], \n\t\t[\"march\", 214129], \n\t\t[\"november\", 210514], \n\t\t[\"december\", 208958], \n\t\t[\"april\", 206148], \n\t\t[\"2\", 204552], \n\t\t[\"august\", 202367], \n\t\t[\"2005\", 196752], \n\t\t[\"3\", 175170], \n\t\t[\"february\", 172170], \n\t\t[\"4\", 157279], \n\t\t[\"5\", 154334], \n\t\t[\"10\", 150965], \n\t\t[\"2004\", 141553], \n\t\t[\"announced\", 132186], \n\t\t[\"year\", 127087], \n\t\t[\"6\", 124446], \n\t\t[\"12\", 121249], \n\t\t[\"7\", 120417], \n\t\t[\"will\", 119843], \n\t\t[\"20\", 113723], \n\t\t[\"8\", 112973], \n\t\t[\"15\", 112740], \n\t\t[\"2003\", 112621], \n\t\t[\"11\", 108367], \n\t\t[\"9\", 99739], \n\t\t[\"30\", 97553], \n\t\t[\"16\", 94235], \n\t\t[\"13\", 94047], \n\t\t[\"14\", 92821], \n\t\t[\"2002\", 88687], \n\t\t[\"2001\", 82892], \n\t\t[\"25\", 82620], \n\t\t[\"17\", 82304], \n\t\t[\"21\", 78303], \n\t\t[\"22\", 76483], \n\t\t[\"19\", 74498], \n\t\t[\"23\", 74081], \n\t\t[\"24\", 70794], \n\t\t[\"26\", 69395]\n\t], \n\t[\n\t\t[\"energy\", 89905], \n\t\t[\"water\", 82925], \n\t\t[\"power\", 66159], \n\t\t[\"gas\", 56708], \n\t\t[\"high\", 45724], \n\t\t[\"oil\", 44152], \n\t\t[\"chemical\", 39157], \n\t\t[\"process\", 38258], \n\t\t[\"nuclear\", 37878], \n\t\t[\"will\", 35298], \n\t\t[\"temperature\", 34434], \n\t\t[\"material\", 33868], \n\t\t[\"heat\", 33328], \n\t\t[\"plant\", 32528], \n\t\t[\"surface\", 31283], \n\t\t[\"pressure\", 30483], \n\t\t[\"materials\", 28993], \n\t\t[\"production\", 27183], \n\t\t[\"carbon\", 26924], \n\t\t[\"reaction\", 26304], \n\t\t[\"low\", 24192], \n\t\t[\"acid\", 23823], \n\t\t[\"light\", 22965], \n\t\t[\"current\", 22556], \n\t\t[\"form\", 21664], \n\t\t[\"produced\", 20971], \n\t\t[\"air\", 20933], \n\t\t[\"metal\", 20628], \n\t\t[\"fuel\", 20144], \n\t\t[\"system\", 19647], \n\t\t[\"mine\", 19430], \n\t\t[\"electric\", 19429], \n\t\t[\"produce\", 19316], \n\t\t[\"products\", 19214], \n\t\t[\"liquid\", 19082], \n\t\t[\"systems\", 19012], \n\t\t[\"hydrogen\", 18912], \n\t\t[\"large\", 18659], \n\t\t[\"enzyme\", 18634], \n\t\t[\"electricity\", 18610], \n\t\t[\"electrical\", 18563], \n\t\t[\"solar\", 17620], \n\t\t[\"iron\", 17574], \n\t\t[\"plants\", 17297], \n\t\t[\"example\", 17011], \n\t\t[\"natural\", 16722], \n\t\t[\"oxygen\", 16647], \n\t\t[\"higher\", 16408], \n\t\t[\"organic\", 16400], \n\t\t[\"flow\", 15931]\n\t], \n\t[\n\t\t[\"scotland\", 62224], \n\t\t[\"list\", 55368], \n\t\t[\"scottish\", 52947], \n\t\t[\"national\", 51865], \n\t\t[\"register\", 33796], \n\t\t[\"places\", 30371], \n\t\t[\"historic\", 28995], \n\t\t[\"listed\", 28810], \n\t\t[\"james\", 24496], \n\t\t[\"edinburgh\", 23219], \n\t\t[\"glasgow\", 22937], \n\t\t[\"hamilton\", 19534], \n\t\t[\"ross\", 17636], \n\t\t[\"david\", 17374], \n\t\t[\"campbell\", 17112], \n\t\t[\"scott\", 17073], \n\t\t[\"john\", 15620], \n\t\t[\"douglas\", 15137], \n\t\t[\"gordon\", 14794], \n\t\t[\"murray\", 14600], \n\t\t[\"stewart\", 13584], \n\t\t[\"alexander\", 13207], \n\t\t[\"robert\", 13187], \n\t\t[\"anderson\", 12835], \n\t\t[\"bruce\", 11325], \n\t\t[\"stamps\", 10801], \n\t\t[\"graham\", 10195], \n\t\t[\"aberdeen\", 9808], \n\t\t[\"andrews\", 9476], \n\t\t[\"watson\", 9182], \n\t\t[\"properties\", 9056], \n\t\t[\"post\", 8765], \n\t\t[\"clan\", 8395], \n\t\t[\"postal\", 8385], \n\t\t[\"scots\", 8275], \n\t\t[\"duncan\", 8237], \n\t\t[\"united\", 7759], \n\t\t[\"county\", 7752], \n\t\t[\"districts\", 7548], \n\t\t[\"highland\", 7423], \n\t\t[\"andrew\", 7356], \n\t\t[\"isle\", 7263], \n\t\t[\"glen\", 7159], \n\t\t[\"stamp\", 7116], \n\t\t[\"wallace\", 6982], \n\t\t[\"cameron\", 6955], \n\t\t[\"burns\", 6911], \n\t\t[\"macdonald\", 6876], \n\t\t[\"year\", 6845], \n\t\t[\"included\", 6723]\n\t], \n\t[\n\t\t[\"system\", 112467], \n\t\t[\"data\", 98808], \n\t\t[\"software\", 80744], \n\t\t[\"systems\", 80637], \n\t\t[\"computer\", 69581], \n\t\t[\"users\", 50524], \n\t\t[\"code\", 49057], \n\t\t[\"network\", 43536], \n\t\t[\"internet\", 43063], \n\t\t[\"digital\", 42273], \n\t\t[\"user\", 41382], \n\t\t[\"technology\", 41067], \n\t\t[\"web\", 39579], \n\t\t[\"will\", 37937], \n\t\t[\"support\", 36380], \n\t\t[\"version\", 35618], \n\t\t[\"windows\", 35469], \n\t\t[\"access\", 35283], \n\t\t[\"applications\", 33364], \n\t\t[\"file\", 31566], \n\t\t[\"memory\", 31465], \n\t\t[\"application\", 30358], \n\t\t[\"mobile\", 30310], \n\t\t[\"control\", 29927], \n\t\t[\"standard\", 29568], \n\t\t[\"time\", 29370], \n\t\t[\"program\", 29227], \n\t\t[\"based\", 28953], \n\t\t[\"design\", 27497], \n\t\t[\"video\", 27447], \n\t\t[\"number\", 26619], \n\t\t[\"features\", 26351], \n\t\t[\"devices\", 25250], \n\t\t[\"service\", 24458], \n\t\t[\"project\", 24424], \n\t\t[\"device\", 24401], \n\t\t[\"allows\", 24250], \n\t\t[\"source\", 23965], \n\t\t[\"operating\", 23881], \n\t\t[\"computers\", 23293], \n\t\t[\"include\", 23255], \n\t\t[\"process\", 22808], \n\t\t[\"developed\", 22475], \n\t\t[\"development\", 22002], \n\t\t[\"content\", 21905], \n\t\t[\"microsoft\", 21682], \n\t\t[\"files\", 21588], \n\t\t[\"server\", 21270], \n\t\t[\"hardware\", 21055], \n\t\t[\"example\", 21011]\n\t], \n\t[\n\t\t[\"de\", 212319], \n\t\t[\"spanish\", 94530], \n\t\t[\"la\", 75775], \n\t\t[\"mexico\", 64655], \n\t\t[\"spain\", 55565], \n\t\t[\"el\", 55072], \n\t\t[\"san\", 47334], \n\t\t[\"del\", 39534], \n\t\t[\"brazil\", 39453], \n\t\t[\"wrestling\", 32528], \n\t\t[\"puerto\", 32448], \n\t\t[\"mexican\", 32360], \n\t\t[\"portuguese\", 32195], \n\t\t[\"juan\", 30728], \n\t\t[\"match\", 26829], \n\t\t[\"argentina\", 26466], \n\t\t[\"santa\", 25341], \n\t\t[\"jos\", 24102], \n\t\t[\"portugal\", 23298], \n\t\t[\"rio\", 22341], \n\t\t[\"brazilian\", 22309], \n\t\t[\"rico\", 21972], \n\t\t[\"city\", 21453], \n\t\t[\"los\", 20305], \n\t\t[\"latin\", 20200], \n\t\t[\"carlos\", 20015], \n\t\t[\"chile\", 19658], \n\t\t[\"america\", 19504], \n\t\t[\"antonio\", 18782], \n\t\t[\"da\", 17048], \n\t\t[\"madrid\", 17022], \n\t\t[\"tag\", 16607], \n\t\t[\"cuba\", 16558], \n\t\t[\"luis\", 16150], \n\t\t[\"las\", 16136], \n\t\t[\"american\", 15534], \n\t\t[\"pedro\", 15490], \n\t\t[\"colombia\", 15458], \n\t\t[\"peru\", 15195], \n\t\t[\"argentine\", 14541], \n\t\t[\"buenos\", 14207], \n\t\t[\"aires\", 14185], \n\t\t[\"francisco\", 13820], \n\t\t[\"manuel\", 13669], \n\t\t[\"title\", 13498], \n\t\t[\"venezuela\", 13358], \n\t\t[\"costa\", 13326], \n\t\t[\"began\", 12172], \n\t\t[\"main\", 11831], \n\t\t[\"cuban\", 11672]\n\t], \n\t[\n\t\t[\"de\", 236835], \n\t\t[\"french\", 202365], \n\t\t[\"france\", 138650], \n\t\t[\"la\", 86432], \n\t\t[\"paris\", 83054], \n\t\t[\"italian\", 80684], \n\t\t[\"italy\", 60821], \n\t\t[\"le\", 55637], \n\t\t[\"dutch\", 50800], \n\t\t[\"van\", 43969], \n\t\t[\"des\", 36817], \n\t\t[\"du\", 35568], \n\t\t[\"louis\", 34582], \n\t\t[\"netherlands\", 34128], \n\t\t[\"di\", 32325], \n\t\t[\"belgium\", 22104], \n\t\t[\"les\", 21407], \n\t\t[\"jean\", 19444], \n\t\t[\"rome\", 18344], \n\t\t[\"belgian\", 15469], \n\t\t[\"pierre\", 15268], \n\t\t[\"milan\", 14440], \n\t\t[\"amsterdam\", 14430], \n\t\t[\"en\", 13437], \n\t\t[\"european\", 13300], \n\t\t[\"charles\", 12309], \n\t\t[\"il\", 11477], \n\t\t[\"jacques\", 10905], \n\t\t[\"napoleon\", 10840], \n\t\t[\"brussels\", 10782], \n\t\t[\"florence\", 10540], \n\t\t[\"naples\", 10539], \n\t\t[\"born\", 10526], \n\t\t[\"europe\", 10480], \n\t\t[\"grand\", 9970], \n\t\t[\"giovanni\", 9592], \n\t\t[\"spain\", 9456], \n\t\t[\"venice\", 9372], \n\t\t[\"holland\", 8956], \n\t\t[\"left\", 8898], \n\t\t[\"marie\", 8530], \n\t\t[\"famous\", 8381], \n\t\t[\"franois\", 8293], \n\t\t[\"san\", 8095], \n\t\t[\"del\", 7911], \n\t\t[\"della\", 7890], \n\t\t[\"henri\", 7863], \n\t\t[\"commune\", 7857], \n\t\t[\"lyon\", 7575], \n\t\t[\"named\", 7506]\n\t], \n\t[\n\t\t[\"won\", 181752], \n\t\t[\"team\", 122397], \n\t\t[\"race\", 116450], \n\t\t[\"event\", 104584], \n\t\t[\"championship\", 81216], \n\t\t[\"held\", 72452], \n\t\t[\"championships\", 70070], \n\t\t[\"second\", 66475], \n\t\t[\"competition\", 63669], \n\t\t[\"place\", 58087], \n\t\t[\"racing\", 55343], \n\t\t[\"final\", 53941], \n\t\t[\"three\", 51909], \n\t\t[\"grand\", 51844], \n\t\t[\"medal\", 51297], \n\t\t[\"series\", 51006], \n\t\t[\"tour\", 50173], \n\t\t[\"finished\", 50073], \n\t\t[\"olympic\", 49302], \n\t\t[\"gold\", 48741], \n\t\t[\"win\", 48719], \n\t\t[\"olympics\", 48532], \n\t\t[\"time\", 48188], \n\t\t[\"season\", 45041], \n\t\t[\"events\", 44912], \n\t\t[\"year\", 44752], \n\t\t[\"champion\", 43662], \n\t\t[\"best\", 41662], \n\t\t[\"international\", 36648], \n\t\t[\"test\", 36112], \n\t\t[\"winning\", 36109], \n\t\t[\"tournament\", 35647], \n\t\t[\"winner\", 35552], \n\t\t[\"third\", 35337], \n\t\t[\"competed\", 34801], \n\t\t[\"points\", 34729], \n\t\t[\"races\", 34514], \n\t\t[\"record\", 33922], \n\t\t[\"top\", 32812], \n\t\t[\"cricket\", 32031], \n\t\t[\"womens\", 31763], \n\t\t[\"summer\", 31668], \n\t\t[\"open\", 30495], \n\t\t[\"prix\", 30425], \n\t\t[\"games\", 30369], \n\t\t[\"silver\", 30071], \n\t\t[\"track\", 30009], \n\t\t[\"cup\", 29733], \n\t\t[\"teams\", 28898], \n\t\t[\"round\", 28471]\n\t], \n\t[\n\t\t[\"london\", 188616], \n\t\t[\"australia\", 125600], \n\t\t[\"england\", 117346], \n\t\t[\"australian\", 106100], \n\t\t[\"british\", 100209], \n\t\t[\"south\", 96541], \n\t\t[\"wales\", 77076], \n\t\t[\"zealand\", 68704], \n\t\t[\"st\", 53938], \n\t\t[\"sydney\", 52194], \n\t\t[\"john\", 48656], \n\t\t[\"centre\", 48005], \n\t\t[\"royal\", 47826], \n\t\t[\"west\", 45883], \n\t\t[\"victoria\", 43149], \n\t\t[\"uk\", 42222], \n\t\t[\"melbourne\", 39022], \n\t\t[\"north\", 36808], \n\t\t[\"britain\", 36585], \n\t\t[\"council\", 33376], \n\t\t[\"local\", 32867], \n\t\t[\"oxford\", 31555], \n\t\t[\"manchester\", 31278], \n\t\t[\"sir\", 29660], \n\t\t[\"school\", 29069], \n\t\t[\"great\", 28144], \n\t\t[\"welsh\", 27981], \n\t\t[\"hill\", 26377], \n\t\t[\"war\", 25474], \n\t\t[\"east\", 25095], \n\t\t[\"george\", 24434], \n\t\t[\"years\", 23911], \n\t\t[\"cambridge\", 23777], \n\t\t[\"including\", 23407], \n\t\t[\"county\", 23177], \n\t\t[\"birmingham\", 22708], \n\t\t[\"yorkshire\", 22380], \n\t\t[\"english\", 22041], \n\t\t[\"borough\", 21981], \n\t\t[\"queensland\", 21650], \n\t\t[\"town\", 21525], \n\t\t[\"road\", 21134], \n\t\t[\"liverpool\", 20615], \n\t\t[\"kent\", 20207], \n\t\t[\"bristol\", 19115], \n\t\t[\"hall\", 18610], \n\t\t[\"formed\", 18197], \n\t\t[\"william\", 17697], \n\t\t[\"early\", 17039], \n\t\t[\"victorian\", 16810]\n\t], \n\t[\n\t\t[\"university\", 463429], \n\t\t[\"member\", 137345], \n\t\t[\"born\", 134363], \n\t\t[\"college\", 121666], \n\t\t[\"served\", 117173], \n\t\t[\"received\", 109053], \n\t\t[\"director\", 105592], \n\t\t[\"award\", 103422], \n\t\t[\"worked\", 103347], \n\t\t[\"years\", 100862], \n\t\t[\"professor\", 95568], \n\t\t[\"national\", 95436], \n\t\t[\"president\", 89049], \n\t\t[\"school\", 87300], \n\t\t[\"degree\", 85310], \n\t\t[\"institute\", 78974], \n\t\t[\"international\", 66953], \n\t\t[\"graduated\", 65260], \n\t\t[\"awarded\", 63288], \n\t\t[\"dr\", 59801], \n\t\t[\"association\", 58365], \n\t\t[\"career\", 56992], \n\t\t[\"attended\", 56058], \n\t\t[\"work\", 55240], \n\t\t[\"joined\", 54174], \n\t\t[\"studies\", 51526], \n\t\t[\"science\", 51371], \n\t\t[\"society\", 51074], \n\t\t[\"assistant\", 49970], \n\t\t[\"department\", 48537], \n\t\t[\"american\", 48345], \n\t\t[\"1972\", 47973], \n\t\t[\"1970\", 47967], \n\t\t[\"1980\", 47806], \n\t\t[\"1976\", 47304], \n\t\t[\"board\", 46827], \n\t\t[\"1968\", 46734], \n\t\t[\"appointed\", 46721], \n\t\t[\"1975\", 46700], \n\t\t[\"1979\", 46583], \n\t\t[\"1990\", 46556], \n\t\t[\"1973\", 46504], \n\t\t[\"studied\", 46201], \n\t\t[\"1978\", 46106], \n\t\t[\"1974\", 46030], \n\t\t[\"academy\", 45998], \n\t\t[\"1977\", 45926], \n\t\t[\"1985\", 45800], \n\t\t[\"1969\", 45510], \n\t\t[\"1984\", 45365]\n\t], \n\t[\n\t\t[\"area\", 291810], \n\t\t[\"town\", 228390], \n\t\t[\"city\", 196667], \n\t\t[\"population\", 194023], \n\t\t[\"village\", 166037], \n\t\t[\"district\", 141399], \n\t\t[\"people\", 134300], \n\t\t[\"land\", 134236], \n\t\t[\"region\", 127491], \n\t\t[\"island\", 126985], \n\t\t[\"local\", 121356], \n\t\t[\"south\", 120239], \n\t\t[\"north\", 116976], \n\t\t[\"century\", 98568], \n\t\t[\"east\", 88909], \n\t\t[\"west\", 87533], \n\t\t[\"areas\", 84905], \n\t\t[\"located\", 80980], \n\t\t[\"community\", 75808], \n\t\t[\"towns\", 68451], \n\t\t[\"islands\", 62252], \n\t\t[\"small\", 62094], \n\t\t[\"municipality\", 58992], \n\t\t[\"river\", 58226], \n\t\t[\"settlement\", 54627], \n\t\t[\"cities\", 53679], \n\t\t[\"northern\", 52990], \n\t\t[\"large\", 52005], \n\t\t[\"largest\", 51995], \n\t\t[\"villages\", 49592], \n\t\t[\"government\", 49564], \n\t\t[\"established\", 48278], \n\t\t[\"rural\", 44343], \n\t\t[\"southern\", 42714], \n\t\t[\"residents\", 42432], \n\t\t[\"territory\", 41947], \n\t\t[\"major\", 41053], \n\t\t[\"county\", 40317], \n\t\t[\"number\", 39993], \n\t\t[\"urban\", 39422], \n\t\t[\"inhabitants\", 38785], \n\t\t[\"agricultural\", 38404], \n\t\t[\"country\", 38281], \n\t\t[\"communities\", 36513], \n\t\t[\"capital\", 36254], \n\t\t[\"main\", 36179], \n\t\t[\"western\", 35873], \n\t\t[\"province\", 34868], \n\t\t[\"districts\", 34175], \n\t\t[\"early\", 33552]\n\t], \n\t[\n\t\t[\"party\", 282718], \n\t\t[\"law\", 243059], \n\t\t[\"government\", 211086], \n\t\t[\"election\", 206291], \n\t\t[\"court\", 184981], \n\t\t[\"president\", 156133], \n\t\t[\"elected\", 146881], \n\t\t[\"council\", 130785], \n\t\t[\"general\", 127154], \n\t\t[\"minister\", 123014], \n\t\t[\"political\", 107021], \n\t\t[\"national\", 104419], \n\t\t[\"members\", 101221], \n\t\t[\"committee\", 98626], \n\t\t[\"united\", 97465], \n\t\t[\"office\", 95109], \n\t\t[\"federal\", 87667], \n\t\t[\"member\", 85523], \n\t\t[\"house\", 84878], \n\t\t[\"parliament\", 82624], \n\t\t[\"vote\", 80810], \n\t\t[\"public\", 80383], \n\t\t[\"elections\", 76578], \n\t\t[\"democratic\", 73790], \n\t\t[\"held\", 72992], \n\t\t[\"assembly\", 72587], \n\t\t[\"justice\", 70258], \n\t\t[\"rights\", 69036], \n\t\t[\"candidate\", 68445], \n\t\t[\"congress\", 66559], \n\t\t[\"senate\", 66305], \n\t\t[\"district\", 63603], \n\t\t[\"seat\", 63331], \n\t\t[\"constitution\", 60225], \n\t\t[\"secretary\", 60148], \n\t\t[\"republican\", 59644], \n\t\t[\"campaign\", 57992], \n\t\t[\"governor\", 57466], \n\t\t[\"legal\", 57439], \n\t\t[\"appointed\", 56932], \n\t\t[\"term\", 56705], \n\t\t[\"liberal\", 56525], \n\t\t[\"commission\", 56499], \n\t\t[\"supreme\", 55591], \n\t\t[\"votes\", 53420], \n\t\t[\"conservative\", 50926], \n\t\t[\"leader\", 49861], \n\t\t[\"parties\", 49491], \n\t\t[\"bill\", 48521], \n\t\t[\"police\", 47747]\n\t], \n\t[\n\t\t[\"york\", 204770], \n\t\t[\"county\", 170218], \n\t\t[\"american\", 159428], \n\t\t[\"united\", 124701], \n\t\t[\"city\", 115571], \n\t\t[\"washington\", 102216], \n\t\t[\"john\", 93869], \n\t\t[\"texas\", 86119], \n\t\t[\"served\", 85731], \n\t\t[\"virginia\", 82839], \n\t\t[\"pennsylvania\", 64529], \n\t\t[\"war\", 64076], \n\t\t[\"moved\", 62540], \n\t\t[\"ohio\", 61422], \n\t\t[\"chicago\", 60577], \n\t\t[\"william\", 59053], \n\t\t[\"carolina\", 57294], \n\t\t[\"north\", 56089], \n\t\t[\"florida\", 55904], \n\t\t[\"illinois\", 50975], \n\t\t[\"george\", 50905], \n\t\t[\"james\", 49954], \n\t\t[\"died\", 49153], \n\t\t[\"massachusetts\", 48767], \n\t\t[\"president\", 48574], \n\t\t[\"named\", 48408], \n\t\t[\"jersey\", 47805], \n\t\t[\"born\", 47060], \n\t\t[\"boston\", 46249], \n\t\t[\"south\", 45861], \n\t\t[\"union\", 44879], \n\t\t[\"west\", 43781], \n\t\t[\"company\", 43299], \n\t\t[\"georgia\", 41809], \n\t\t[\"smith\", 41778], \n\t\t[\"began\", 41438], \n\t\t[\"michigan\", 40164], \n\t\t[\"fort\", 39219], \n\t\t[\"years\", 38873], \n\t\t[\"philadelphia\", 38871], \n\t\t[\"white\", 38815], \n\t\t[\"missouri\", 35834], \n\t\t[\"maryland\", 35661], \n\t\t[\"america\", 34825], \n\t\t[\"black\", 33697], \n\t\t[\"tennessee\", 33559], \n\t\t[\"indiana\", 32860], \n\t\t[\"st\", 32376], \n\t\t[\"jr\", 32009], \n\t\t[\"johnson\", 31714]\n\t], \n\t[\n\t\t[\"century\", 124400], \n\t\t[\"king\", 111104], \n\t\t[\"roman\", 78855], \n\t\t[\"empire\", 73842], \n\t\t[\"greek\", 67390], \n\t\t[\"bc\", 58607], \n\t\t[\"ancient\", 49778], \n\t\t[\"emperor\", 49372], \n\t\t[\"ii\", 48992], \n\t\t[\"kingdom\", 48498], \n\t\t[\"period\", 45001], \n\t\t[\"battle\", 39897], \n\t\t[\"city\", 39562], \n\t\t[\"time\", 35244], \n\t\t[\"great\", 34437], \n\t\t[\"war\", 31828], \n\t\t[\"ad\", 30236], \n\t\t[\"early\", 27546], \n\t\t[\"reign\", 27366], \n\t\t[\"kings\", 27205], \n\t\t[\"iii\", 27189], \n\t\t[\"son\", 26791], \n\t\t[\"rule\", 25766], \n\t\t[\"power\", 25517], \n\t\t[\"greece\", 25443], \n\t\t[\"army\", 25238], \n\t\t[\"centuries\", 24801], \n\t\t[\"dynasty\", 24065], \n\t\t[\"rome\", 23997], \n\t\t[\"modern\", 23165], \n\t\t[\"history\", 20624], \n\t\t[\"imperial\", 20453], \n\t\t[\"medieval\", 19869], \n\t\t[\"death\", 19778], \n\t\t[\"ottoman\", 19660], \n\t\t[\"years\", 19434], \n\t\t[\"led\", 19132], \n\t\t[\"byzantine\", 18137], \n\t\t[\"defeated\", 17534], \n\t\t[\"ruled\", 17140], \n\t\t[\"year\", 17062], \n\t\t[\"throne\", 16521], \n\t\t[\"athens\", 16421], \n\t\t[\"capital\", 16348], \n\t\t[\"castle\", 16320], \n\t\t[\"military\", 16163], \n\t\t[\"late\", 15897], \n\t\t[\"iv\", 15474], \n\t\t[\"middle\", 15394], \n\t\t[\"control\", 14896]\n\t], \n\t[\n\t\t[\"engine\", 106344], \n\t\t[\"car\", 86013], \n\t\t[\"design\", 71867], \n\t\t[\"model\", 61157], \n\t\t[\"cars\", 60135], \n\t\t[\"production\", 51957], \n\t\t[\"built\", 51884], \n\t\t[\"engines\", 47808], \n\t\t[\"vehicle\", 45526], \n\t\t[\"class\", 44905], \n\t\t[\"models\", 42450], \n\t\t[\"speed\", 41574], \n\t\t[\"vehicles\", 41047], \n\t\t[\"designed\", 40748], \n\t\t[\"produced\", 37386], \n\t\t[\"power\", 33408], \n\t\t[\"front\", 32909], \n\t\t[\"system\", 32136], \n\t\t[\"version\", 32103], \n\t\t[\"type\", 31726], \n\t\t[\"series\", 31444], \n\t\t[\"motor\", 30334], \n\t\t[\"rear\", 30329], \n\t\t[\"standard\", 29624], \n\t\t[\"gun\", 28957], \n\t\t[\"company\", 28028], \n\t\t[\"introduced\", 25959], \n\t\t[\"range\", 25069], \n\t\t[\"ford\", 25015], \n\t\t[\"sold\", 24536], \n\t\t[\"fuel\", 21898], \n\t\t[\"drive\", 21660], \n\t\t[\"wheel\", 21509], \n\t\t[\"tank\", 21361], \n\t\t[\"fitted\", 20254], \n\t\t[\"factory\", 20253], \n\t\t[\"machine\", 19589], \n\t\t[\"developed\", 19383], \n\t\t[\"based\", 19316], \n\t\t[\"replaced\", 18844], \n\t\t[\"wheels\", 18649], \n\t\t[\"time\", 18463], \n\t\t[\"powered\", 18070], \n\t\t[\"small\", 17800], \n\t\t[\"high\", 17446], \n\t\t[\"weight\", 17020], \n\t\t[\"electric\", 16986], \n\t\t[\"body\", 16628], \n\t\t[\"mounted\", 16372], \n\t\t[\"early\", 16244]\n\t], \n\t[\n\t\t[\"war\", 295245], \n\t\t[\"army\", 266723], \n\t\t[\"military\", 143905], \n\t\t[\"forces\", 142815], \n\t\t[\"battle\", 134711], \n\t\t[\"force\", 111038], \n\t\t[\"british\", 110228], \n\t\t[\"command\", 105739], \n\t\t[\"general\", 103690], \n\t\t[\"navy\", 101416], \n\t\t[\"ship\", 100403], \n\t\t[\"division\", 99686], \n\t\t[\"ships\", 95391], \n\t\t[\"troops\", 82337], \n\t\t[\"corps\", 80884], \n\t\t[\"service\", 80655], \n\t\t[\"naval\", 75172], \n\t\t[\"regiment\", 74964], \n\t\t[\"commander\", 72783], \n\t\t[\"infantry\", 68989], \n\t\t[\"attack\", 66213], \n\t\t[\"men\", 64022], \n\t\t[\"officer\", 62600], \n\t\t[\"fleet\", 59858], \n\t\t[\"soldiers\", 58431], \n\t\t[\"units\", 55498], \n\t\t[\"officers\", 54634], \n\t\t[\"operations\", 53533], \n\t\t[\"unit\", 53380], \n\t\t[\"june\", 51211], \n\t\t[\"august\", 50803], \n\t\t[\"brigade\", 50271], \n\t\t[\"july\", 50067], \n\t\t[\"fire\", 49368], \n\t\t[\"training\", 49331], \n\t\t[\"march\", 48720], \n\t\t[\"battalion\", 48554], \n\t\t[\"april\", 48485], \n\t\t[\"operation\", 48099], \n\t\t[\"captain\", 47971], \n\t\t[\"september\", 47352], \n\t\t[\"three\", 45886], \n\t\t[\"enemy\", 45104], \n\t\t[\"united\", 43013], \n\t\t[\"october\", 42618], \n\t\t[\"sea\", 41631], \n\t\t[\"royal\", 41548], \n\t\t[\"german\", 41104], \n\t\t[\"marine\", 40895], \n\t\t[\"major\", 40763]\n\t], \n\t[\n\t\t[\"son\", 167725], \n\t\t[\"died\", 146431], \n\t\t[\"married\", 128307], \n\t\t[\"family\", 123107], \n\t\t[\"king\", 111969], \n\t\t[\"daughter\", 105251], \n\t\t[\"john\", 93901], \n\t\t[\"death\", 87804], \n\t\t[\"william\", 86856], \n\t\t[\"father\", 85020], \n\t\t[\"born\", 81388], \n\t\t[\"wife\", 80860], \n\t\t[\"royal\", 79052], \n\t\t[\"ireland\", 78714], \n\t\t[\"irish\", 76756], \n\t\t[\"henry\", 75696], \n\t\t[\"house\", 73119], \n\t\t[\"lord\", 68289], \n\t\t[\"charles\", 68138], \n\t\t[\"sir\", 67619], \n\t\t[\"prince\", 64874], \n\t\t[\"brother\", 64076], \n\t\t[\"children\", 61342], \n\t\t[\"england\", 59207], \n\t\t[\"queen\", 57108], \n\t\t[\"duke\", 56660], \n\t\t[\"thomas\", 52106], \n\t\t[\"years\", 49872], \n\t\t[\"marriage\", 45259], \n\t\t[\"george\", 44609], \n\t\t[\"earl\", 43289], \n\t\t[\"edward\", 42847], \n\t\t[\"english\", 42269], \n\t\t[\"second\", 39571], \n\t\t[\"elizabeth\", 38506], \n\t\t[\"sons\", 36878], \n\t\t[\"mary\", 36632], \n\t\t[\"james\", 36472], \n\t\t[\"mother\", 36003], \n\t\t[\"appointed\", 35157], \n\t\t[\"year\", 31700], \n\t\t[\"dublin\", 31063], \n\t\t[\"lady\", 30127], \n\t\t[\"title\", 29992], \n\t\t[\"great\", 29772], \n\t\t[\"succeeded\", 29558], \n\t\t[\"robert\", 29173], \n\t\t[\"ii\", 28901], \n\t\t[\"member\", 28894], \n\t\t[\"castle\", 28571]\n\t], \n\t[\n\t\t[\"season\", 256538], \n\t\t[\"team\", 216040], \n\t\t[\"game\", 178379], \n\t\t[\"league\", 134385], \n\t\t[\"games\", 132121], \n\t\t[\"played\", 101956], \n\t\t[\"coach\", 92610], \n\t\t[\"football\", 87033], \n\t\t[\"record\", 78102], \n\t\t[\"teams\", 74960], \n\t\t[\"baseball\", 65873], \n\t\t[\"field\", 62567], \n\t\t[\"year\", 62495], \n\t\t[\"second\", 59618], \n\t\t[\"career\", 58813], \n\t\t[\"play\", 57694], \n\t\t[\"basketball\", 56620], \n\t\t[\"hockey\", 56094], \n\t\t[\"three\", 54372], \n\t\t[\"yards\", 51744], \n\t\t[\"won\", 48262], \n\t\t[\"bowl\", 47535], \n\t\t[\"points\", 44995], \n\t\t[\"win\", 44803], \n\t\t[\"series\", 44746], \n\t\t[\"player\", 44170], \n\t\t[\"head\", 43803], \n\t\t[\"conference\", 43753], \n\t\t[\"championship\", 43112], \n\t\t[\"seasons\", 42888], \n\t\t[\"players\", 41676], \n\t\t[\"draft\", 41300], \n\t\t[\"high\", 40701], \n\t\t[\"time\", 39616], \n\t\t[\"named\", 37883], \n\t\t[\"national\", 37642], \n\t\t[\"led\", 37545], \n\t\t[\"nfl\", 37529], \n\t\t[\"third\", 36419], \n\t\t[\"major\", 36305], \n\t\t[\"finished\", 33991], \n\t\t[\"stadium\", 32681], \n\t\t[\"division\", 32508], \n\t\t[\"lead\", 31697], \n\t\t[\"playing\", 30923], \n\t\t[\"ncaa\", 29967], \n\t\t[\"history\", 29672], \n\t\t[\"runs\", 29525], \n\t\t[\"touchdown\", 29347], \n\t\t[\"signed\", 29002]\n\t], \n\t[\n\t\t[\"species\", 218072], \n\t\t[\"family\", 65492], \n\t\t[\"birds\", 55986], \n\t\t[\"small\", 48097], \n\t\t[\"long\", 43737], \n\t\t[\"large\", 38060], \n\t\t[\"animals\", 35579], \n\t\t[\"bird\", 34415], \n\t\t[\"plants\", 33186], \n\t\t[\"genus\", 32612], \n\t\t[\"plant\", 30935], \n\t\t[\"natural\", 30686], \n\t\t[\"habitat\", 30079], \n\t\t[\"tree\", 30041], \n\t\t[\"fish\", 28678], \n\t\t[\"tropical\", 27964], \n\t\t[\"white\", 26524], \n\t\t[\"black\", 25570], \n\t\t[\"order\", 25511], \n\t\t[\"leaves\", 24934], \n\t\t[\"brown\", 23957], \n\t\t[\"common\", 21978], \n\t\t[\"forests\", 21929], \n\t\t[\"trees\", 21513], \n\t\t[\"animal\", 20610], \n\t\t[\"flowers\", 20003], \n\t\t[\"eggs\", 19524], \n\t\t[\"worldwide\", 19021], \n\t\t[\"feed\", 18813], \n\t\t[\"occur\", 18520], \n\t\t[\"subtropical\", 18249], \n\t\t[\"wild\", 17362], \n\t\t[\"length\", 16602], \n\t\t[\"male\", 16511], \n\t\t[\"breeding\", 16462], \n\t\t[\"habitats\", 16348], \n\t\t[\"range\", 16272], \n\t\t[\"food\", 16219], \n\t\t[\"female\", 15961], \n\t\t[\"fruit\", 15662], \n\t\t[\"short\", 15442], \n\t\t[\"insects\", 15343], \n\t\t[\"endemic\", 15230], \n\t\t[\"forest\", 14960], \n\t\t[\"group\", 14842], \n\t\t[\"including\", 14680], \n\t\t[\"include\", 14587], \n\t\t[\"moist\", 14567], \n\t\t[\"threatened\", 14443], \n\t\t[\"tail\", 14414]\n\t], \n\t[\n\t\t[\"art\", 221966], \n\t\t[\"museum\", 124132], \n\t\t[\"work\", 100651], \n\t\t[\"works\", 65369], \n\t\t[\"artists\", 60161], \n\t\t[\"collection\", 56623], \n\t\t[\"design\", 55177], \n\t\t[\"arts\", 54633], \n\t\t[\"painting\", 49171], \n\t\t[\"artist\", 43694], \n\t\t[\"gallery\", 42139], \n\t\t[\"paintings\", 36760], \n\t\t[\"exhibition\", 34067], \n\t\t[\"style\", 32498], \n\t\t[\"fine\", 26393], \n\t\t[\"including\", 26002], \n\t\t[\"painted\", 24375], \n\t\t[\"architecture\", 24160], \n\t\t[\"york\", 21498], \n\t\t[\"fashion\", 20628], \n\t\t[\"painter\", 20611], \n\t\t[\"life\", 20555], \n\t\t[\"early\", 20091], \n\t\t[\"created\", 19733], \n\t\t[\"sculpture\", 19691], \n\t\t[\"artistic\", 19141], \n\t\t[\"history\", 19083], \n\t\t[\"contemporary\", 19063], \n\t\t[\"collections\", 18987], \n\t\t[\"years\", 18082], \n\t\t[\"museums\", 18003], \n\t\t[\"worked\", 17574], \n\t\t[\"images\", 17546], \n\t\t[\"time\", 17525], \n\t\t[\"photography\", 17482], \n\t\t[\"figures\", 17133], \n\t\t[\"academy\", 16423], \n\t\t[\"exhibitions\", 16098], \n\t\t[\"modern\", 15538], \n\t\t[\"portrait\", 15478], \n\t\t[\"photographs\", 15308], \n\t\t[\"began\", 15089], \n\t\t[\"studio\", 14463], \n\t\t[\"drawing\", 14370], \n\t\t[\"include\", 13926], \n\t\t[\"exhibited\", 13866], \n\t\t[\"produced\", 13707], \n\t\t[\"designed\", 13658], \n\t\t[\"period\", 13429], \n\t\t[\"visual\", 13335]\n\t], \n\t[\n\t\t[\"white\", 96740], \n\t\t[\"red\", 81635], \n\t\t[\"black\", 78110], \n\t\t[\"blue\", 65174], \n\t\t[\"called\", 42832], \n\t\t[\"color\", 41713], \n\t\t[\"will\", 36576], \n\t\t[\"head\", 36007], \n\t\t[\"green\", 35511], \n\t\t[\"gold\", 35229], \n\t\t[\"side\", 34744], \n\t\t[\"small\", 34412], \n\t\t[\"hand\", 33632], \n\t\t[\"long\", 32262], \n\t\t[\"arms\", 30112], \n\t\t[\"top\", 29926], \n\t\t[\"flag\", 29799], \n\t\t[\"horse\", 29723], \n\t\t[\"wear\", 29421], \n\t\t[\"silver\", 28948], \n\t\t[\"common\", 28709], \n\t\t[\"light\", 27650], \n\t\t[\"dog\", 27358], \n\t\t[\"wood\", 25393], \n\t\t[\"body\", 25251], \n\t\t[\"type\", 25059], \n\t\t[\"large\", 25005], \n\t\t[\"yellow\", 24524], \n\t\t[\"form\", 23338], \n\t\t[\"worn\", 22795], \n\t\t[\"dogs\", 22590], \n\t\t[\"cut\", 22197], \n\t\t[\"popular\", 22133], \n\t\t[\"left\", 21901], \n\t\t[\"generally\", 21826], \n\t\t[\"traditional\", 21543], \n\t\t[\"ball\", 21220], \n\t\t[\"front\", 20991], \n\t\t[\"horses\", 20849], \n\t\t[\"shape\", 20445], \n\t\t[\"hair\", 20397], \n\t\t[\"feet\", 19597], \n\t\t[\"colors\", 19549], \n\t\t[\"time\", 18692], \n\t\t[\"coat\", 18489], \n\t\t[\"three\", 18123], \n\t\t[\"typically\", 17728], \n\t\t[\"modern\", 17695], \n\t\t[\"face\", 17646], \n\t\t[\"cross\", 17455]\n\t], \n\t[\n\t\t[\"school\", 797694], \n\t\t[\"students\", 322211], \n\t\t[\"university\", 223060], \n\t\t[\"high\", 215654], \n\t\t[\"college\", 215126], \n\t\t[\"schools\", 213090], \n\t\t[\"education\", 209030], \n\t\t[\"year\", 120280], \n\t\t[\"program\", 119598], \n\t\t[\"student\", 111419], \n\t\t[\"campus\", 91134], \n\t\t[\"community\", 78216], \n\t\t[\"programs\", 74522], \n\t\t[\"training\", 68184], \n\t\t[\"center\", 65221], \n\t\t[\"members\", 64165], \n\t\t[\"science\", 63324], \n\t\t[\"national\", 62197], \n\t\t[\"years\", 61502], \n\t\t[\"public\", 60020], \n\t\t[\"academic\", 57457], \n\t\t[\"association\", 53148], \n\t\t[\"courses\", 48179], \n\t\t[\"arts\", 47300], \n\t\t[\"educational\", 47263], \n\t\t[\"include\", 46719], \n\t\t[\"class\", 46284], \n\t\t[\"institute\", 46084], \n\t\t[\"department\", 45032], \n\t\t[\"teachers\", 44371], \n\t\t[\"colleges\", 42472], \n\t\t[\"classes\", 42407], \n\t\t[\"offers\", 41769], \n\t\t[\"activities\", 41247], \n\t\t[\"universities\", 40298], \n\t\t[\"district\", 39492], \n\t\t[\"engineering\", 38781], \n\t\t[\"learning\", 38515], \n\t\t[\"founded\", 38485], \n\t\t[\"faculty\", 38255], \n\t\t[\"girls\", 38064], \n\t\t[\"sports\", 37555], \n\t\t[\"children\", 37484], \n\t\t[\"boys\", 37462], \n\t\t[\"international\", 37262], \n\t\t[\"board\", 36855], \n\t\t[\"teaching\", 36291], \n\t\t[\"academy\", 36168], \n\t\t[\"secondary\", 36151], \n\t\t[\"established\", 35640]\n\t], \n\t[\n\t\t[\"album\", 446364], \n\t\t[\"band\", 330220], \n\t\t[\"song\", 286320], \n\t\t[\"released\", 257516], \n\t\t[\"music\", 220855], \n\t\t[\"songs\", 160722], \n\t\t[\"single\", 137848], \n\t\t[\"records\", 125041], \n\t\t[\"recorded\", 118398], \n\t\t[\"rock\", 105713], \n\t\t[\"bands\", 93876], \n\t\t[\"release\", 89818], \n\t\t[\"live\", 88207], \n\t\t[\"tour\", 87917], \n\t\t[\"video\", 86282], \n\t\t[\"record\", 84378], \n\t\t[\"albums\", 83031], \n\t\t[\"label\", 80581], \n\t\t[\"group\", 80131], \n\t\t[\"recording\", 73915], \n\t\t[\"guitar\", 66461], \n\t\t[\"track\", 62417], \n\t\t[\"cover\", 61590], \n\t\t[\"version\", 61032], \n\t\t[\"tracks\", 60467], \n\t\t[\"number\", 59460], \n\t\t[\"featured\", 59052], \n\t\t[\"time\", 59033], \n\t\t[\"chart\", 57046], \n\t\t[\"hit\", 56365], \n\t\t[\"uk\", 54638], \n\t\t[\"top\", 53980], \n\t\t[\"performed\", 53958], \n\t\t[\"studio\", 53541], \n\t\t[\"played\", 53173], \n\t\t[\"singles\", 50390], \n\t\t[\"sound\", 47823], \n\t\t[\"love\", 46189], \n\t\t[\"pop\", 44820], \n\t\t[\"artist\", 44742], \n\t\t[\"solo\", 43783], \n\t\t[\"cd\", 43642], \n\t\t[\"debut\", 43475], \n\t\t[\"singer\", 42952], \n\t\t[\"artists\", 42801], \n\t\t[\"members\", 40864], \n\t\t[\"included\", 39588], \n\t\t[\"early\", 38715], \n\t\t[\"second\", 38353], \n\t\t[\"bass\", 37997]\n\t], \n\t[\n\t\t[\"radio\", 183618], \n\t\t[\"station\", 137115], \n\t\t[\"news\", 111480], \n\t\t[\"television\", 93354], \n\t\t[\"channel\", 88974], \n\t\t[\"broadcast\", 70195], \n\t\t[\"stations\", 68950], \n\t\t[\"network\", 65774], \n\t\t[\"media\", 60717], \n\t\t[\"tv\", 58102], \n\t\t[\"broadcasting\", 46294], \n\t\t[\"time\", 40693], \n\t\t[\"format\", 40519], \n\t\t[\"local\", 40496], \n\t\t[\"program\", 35684], \n\t\t[\"bbc\", 35218], \n\t\t[\"programming\", 33265], \n\t\t[\"live\", 32987], \n\t\t[\"fm\", 30534], \n\t\t[\"morning\", 29192], \n\t\t[\"host\", 29155], \n\t\t[\"began\", 28818], \n\t\t[\"sports\", 27645], \n\t\t[\"fox\", 26656], \n\t\t[\"air\", 26478], \n\t\t[\"cable\", 26270], \n\t\t[\"call\", 25763], \n\t\t[\"hosted\", 24746], \n\t\t[\"coverage\", 24088], \n\t\t[\"music\", 23877], \n\t\t[\"pm\", 22354], \n\t\t[\"sunday\", 21791], \n\t\t[\"daily\", 21756], \n\t\t[\"channels\", 21636], \n\t\t[\"digital\", 21065], \n\t\t[\"abc\", 20952], \n\t\t[\"aired\", 20886], \n\t\t[\"changed\", 20429], \n\t\t[\"current\", 20200], \n\t\t[\"launched\", 20098], \n\t\t[\"communications\", 20097], \n\t\t[\"programme\", 19888], \n\t\t[\"day\", 19757], \n\t\t[\"broadcasts\", 19295], \n\t\t[\"moved\", 19072], \n\t\t[\"cbs\", 18958], \n\t\t[\"years\", 18620], \n\t\t[\"saturday\", 18126], \n\t\t[\"talk\", 17806], \n\t\t[\"night\", 17600]\n\t], \n\t[\n\t\t[\"age\", 386369], \n\t\t[\"18\", 228156], \n\t\t[\"population\", 222305], \n\t\t[\"income\", 172578], \n\t\t[\"average\", 168517], \n\t\t[\"years\", 167212], \n\t\t[\"median\", 154814], \n\t\t[\"living\", 149471], \n\t\t[\"65\", 125600], \n\t\t[\"males\", 124300], \n\t\t[\"females\", 122617], \n\t\t[\"households\", 121379], \n\t\t[\"100\", 109446], \n\t\t[\"family\", 105354], \n\t\t[\"people\", 99715], \n\t\t[\"families\", 98724], \n\t\t[\"older\", 97900], \n\t\t[\"town\", 95590], \n\t\t[\"size\", 95179], \n\t\t[\"city\", 88956], \n\t\t[\"household\", 88380], \n\t\t[\"miles\", 88082], \n\t\t[\"density\", 86183], \n\t\t[\"american\", 73743], \n\t\t[\"township\", 72828], \n\t\t[\"total\", 71709], \n\t\t[\"area\", 71054], \n\t\t[\"county\", 69734], \n\t\t[\"races\", 67846], \n\t\t[\"census\", 62004], \n\t\t[\"2000\", 59614], \n\t\t[\"square\", 55762], \n\t\t[\"45\", 55584], \n\t\t[\"25\", 55308], \n\t\t[\"64\", 54827], \n\t\t[\"children\", 54479], \n\t\t[\"24\", 52937], \n\t\t[\"44\", 52845], \n\t\t[\"white\", 52734], \n\t\t[\"female\", 49373], \n\t\t[\"land\", 48571], \n\t\t[\"including\", 48458], \n\t\t[\"units\", 48242], \n\t\t[\"housing\", 48034], \n\t\t[\"bureau\", 47910], \n\t\t[\"individuals\", 47648], \n\t\t[\"located\", 47004], \n\t\t[\"poverty\", 46279], \n\t\t[\"united\", 45975], \n\t\t[\"village\", 44825]\n\t], \n\t[\n\t\t[\"music\", 246693], \n\t\t[\"musical\", 63951], \n\t\t[\"opera\", 63270], \n\t\t[\"festival\", 59822], \n\t\t[\"orchestra\", 59570], \n\t\t[\"dance\", 55452], \n\t\t[\"performed\", 52677], \n\t\t[\"jazz\", 43862], \n\t\t[\"piano\", 41471], \n\t\t[\"theatre\", 37764], \n\t\t[\"performance\", 34873], \n\t\t[\"works\", 34382], \n\t\t[\"concert\", 32349], \n\t\t[\"symphony\", 30688], \n\t\t[\"composer\", 29121], \n\t\t[\"played\", 26756], \n\t\t[\"performances\", 26516], \n\t\t[\"instruments\", 26372], \n\t\t[\"musicians\", 24682], \n\t\t[\"classical\", 24321], \n\t\t[\"including\", 24052], \n\t\t[\"work\", 23426], \n\t\t[\"composed\", 23187], \n\t\t[\"major\", 21898], \n\t\t[\"singing\", 21731], \n\t\t[\"songs\", 20068], \n\t\t[\"folk\", 20029], \n\t\t[\"instrument\", 19937], \n\t\t[\"ballet\", 19630], \n\t\t[\"composition\", 19456], \n\t\t[\"composers\", 19333], \n\t\t[\"play\", 19138], \n\t\t[\"performing\", 18902], \n\t\t[\"concerts\", 18852], \n\t\t[\"playing\", 18293], \n\t\t[\"stage\", 18078], \n\t\t[\"years\", 17992], \n\t\t[\"include\", 17931], \n\t\t[\"popular\", 17256], \n\t\t[\"choir\", 17243], \n\t\t[\"ensemble\", 17238], \n\t\t[\"sound\", 17231], \n\t\t[\"style\", 17011], \n\t\t[\"time\", 16361], \n\t\t[\"violin\", 15833], \n\t\t[\"hall\", 15628], \n\t\t[\"piece\", 15272], \n\t\t[\"chamber\", 15186], \n\t\t[\"recordings\", 15120], \n\t\t[\"string\", 15028]\n\t], \n\t[\n\t\t[\"war\", 139497], \n\t\t[\"russian\", 109040], \n\t\t[\"soviet\", 103128], \n\t\t[\"union\", 102373], \n\t\t[\"russia\", 68357], \n\t\t[\"government\", 66993], \n\t\t[\"republic\", 62877], \n\t\t[\"political\", 58416], \n\t\t[\"party\", 58152], \n\t\t[\"polish\", 57559], \n\t\t[\"communist\", 51884], \n\t\t[\"national\", 48571], \n\t\t[\"european\", 45873], \n\t\t[\"poland\", 45447], \n\t\t[\"military\", 44126], \n\t\t[\"international\", 41758], \n\t\t[\"foreign\", 38543], \n\t\t[\"moscow\", 37234], \n\t\t[\"united\", 36736], \n\t\t[\"movement\", 36530], \n\t\t[\"countries\", 34988], \n\t\t[\"europe\", 33844], \n\t\t[\"country\", 31205], \n\t\t[\"treaty\", 30290], \n\t\t[\"socialist\", 29581], \n\t\t[\"revolution\", 29252], \n\t\t[\"group\", 28381], \n\t\t[\"independence\", 26686], \n\t\t[\"nations\", 26537], \n\t\t[\"members\", 25364], \n\t\t[\"workers\", 24975], \n\t\t[\"germany\", 23727], \n\t\t[\"peoples\", 23038], \n\t\t[\"army\", 22667], \n\t\t[\"central\", 22540], \n\t\t[\"peace\", 21535], \n\t\t[\"romania\", 21501], \n\t\t[\"ukraine\", 21310], \n\t\t[\"forces\", 20784], \n\t\t[\"ii\", 20712], \n\t\t[\"ukrainian\", 20523], \n\t\t[\"relations\", 20441], \n\t\t[\"german\", 19947], \n\t\t[\"organization\", 19877], \n\t\t[\"romanian\", 19848], \n\t\t[\"led\", 19485], \n\t\t[\"formed\", 19175], \n\t\t[\"serbian\", 17856], \n\t\t[\"revolutionary\", 17565], \n\t\t[\"groups\", 17469]\n\t], \n\t[\n\t\t[\"canada\", 151475], \n\t\t[\"canadian\", 112362], \n\t\t[\"california\", 111394], \n\t\t[\"san\", 109557], \n\t\t[\"ontario\", 52500], \n\t\t[\"toronto\", 48493], \n\t\t[\"los\", 44998], \n\t\t[\"francisco\", 43748], \n\t\t[\"angeles\", 43445], \n\t\t[\"quebec\", 33237], \n\t\t[\"city\", 32182], \n\t\t[\"philippines\", 31211], \n\t\t[\"montreal\", 28186], \n\t\t[\"provincial\", 25292], \n\t\t[\"vancouver\", 23755], \n\t\t[\"british\", 23011], \n\t\t[\"columbia\", 22401], \n\t\t[\"bay\", 22286], \n\t\t[\"santa\", 21405], \n\t\t[\"alberta\", 20061], \n\t\t[\"malaysia\", 19542], \n\t\t[\"pacific\", 18844], \n\t\t[\"ottawa\", 18720], \n\t\t[\"diego\", 17902], \n\t\t[\"province\", 17861], \n\t\t[\"nova\", 17821], \n\t\t[\"philippine\", 17469], \n\t\t[\"indonesia\", 16269], \n\t\t[\"named\", 15705], \n\t\t[\"riding\", 14119], \n\t\t[\"canadas\", 13937], \n\t\t[\"manila\", 13401], \n\t\t[\"manitoba\", 12963], \n\t\t[\"scotia\", 12505], \n\t\t[\"saskatchewan\", 11114], \n\t\t[\"winnipeg\", 10871], \n\t\t[\"calgary\", 10425], \n\t\t[\"indonesian\", 10407], \n\t\t[\"brunswick\", 9661], \n\t\t[\"edmonton\", 9244], \n\t\t[\"filipino\", 9193], \n\t\t[\"newfoundland\", 8506], \n\t\t[\"jose\", 8345], \n\t\t[\"centre\", 8332], \n\t\t[\"malaysian\", 8203], \n\t\t[\"north\", 8086], \n\t\t[\"halifax\", 7827], \n\t\t[\"currently\", 7430], \n\t\t[\"malay\", 7351], \n\t\t[\"provinces\", 7191]\n\t], \n\t[\n\t\t[\"building\", 281354], \n\t\t[\"built\", 220409], \n\t\t[\"house\", 166902], \n\t\t[\"park\", 166415], \n\t\t[\"city\", 156863], \n\t\t[\"site\", 147300], \n\t\t[\"street\", 138217], \n\t\t[\"located\", 124267], \n\t\t[\"buildings\", 113201], \n\t\t[\"area\", 95684], \n\t\t[\"construction\", 87967], \n\t\t[\"hall\", 83058], \n\t\t[\"opened\", 79477], \n\t\t[\"center\", 71073], \n\t\t[\"tower\", 69313], \n\t\t[\"main\", 67615], \n\t\t[\"houses\", 66295], \n\t\t[\"large\", 63898], \n\t\t[\"original\", 63313], \n\t\t[\"historic\", 61372], \n\t\t[\"public\", 59065], \n\t\t[\"hotel\", 57774], \n\t\t[\"stone\", 51095], \n\t\t[\"designed\", 50777], \n\t\t[\"centre\", 50729], \n\t\t[\"century\", 50349], \n\t\t[\"place\", 49442], \n\t\t[\"constructed\", 48599], \n\t\t[\"room\", 47931], \n\t\t[\"square\", 47850], \n\t\t[\"fire\", 45655], \n\t\t[\"location\", 44816], \n\t\t[\"side\", 44128], \n\t\t[\"floor\", 43495], \n\t\t[\"open\", 40587], \n\t\t[\"garden\", 39694], \n\t\t[\"local\", 39407], \n\t\t[\"small\", 39012], \n\t\t[\"office\", 38976], \n\t\t[\"originally\", 38476], \n\t\t[\"hill\", 38361], \n\t\t[\"wall\", 37613], \n\t\t[\"store\", 37573], \n\t\t[\"structure\", 36982], \n\t\t[\"completed\", 36398], \n\t\t[\"property\", 36232], \n\t\t[\"closed\", 35914], \n\t\t[\"including\", 34877], \n\t\t[\"today\", 34082], \n\t\t[\"nearby\", 33618]\n\t], \n\t[\n\t\t[\"film\", 469340], \n\t\t[\"series\", 222043], \n\t\t[\"films\", 134231], \n\t\t[\"television\", 116992], \n\t\t[\"role\", 105450], \n\t\t[\"movie\", 99630], \n\t\t[\"appeared\", 97475], \n\t\t[\"best\", 88219], \n\t\t[\"episode\", 87383], \n\t\t[\"theatre\", 73971], \n\t\t[\"production\", 73493], \n\t\t[\"award\", 71857], \n\t\t[\"played\", 68464], \n\t\t[\"tv\", 61091], \n\t\t[\"episodes\", 59692], \n\t\t[\"character\", 58376], \n\t\t[\"directed\", 57375], \n\t\t[\"actor\", 56600], \n\t\t[\"director\", 51701], \n\t\t[\"cast\", 51028], \n\t\t[\"star\", 50060], \n\t\t[\"comedy\", 49963], \n\t\t[\"play\", 47832], \n\t\t[\"awards\", 47593], \n\t\t[\"including\", 47518], \n\t\t[\"produced\", 45696], \n\t\t[\"season\", 42907], \n\t\t[\"featured\", 42380], \n\t\t[\"festival\", 42374], \n\t\t[\"drama\", 40069], \n\t\t[\"york\", 39971], \n\t\t[\"original\", 39771], \n\t\t[\"actress\", 38614], \n\t\t[\"stage\", 38445], \n\t\t[\"movies\", 36284], \n\t\t[\"roles\", 35660], \n\t\t[\"acting\", 35436], \n\t\t[\"released\", 35406], \n\t\t[\"hollywood\", 35077], \n\t\t[\"characters\", 34844], \n\t\t[\"actors\", 34108], \n\t\t[\"won\", 34068], \n\t\t[\"career\", 33768], \n\t\t[\"starred\", 33640], \n\t\t[\"starring\", 33618], \n\t\t[\"feature\", 33386], \n\t\t[\"company\", 33374], \n\t\t[\"john\", 32625], \n\t\t[\"dvd\", 31721], \n\t\t[\"producer\", 31058]\n\t], \n\t[\n\t\t[\"game\", 242557], \n\t\t[\"player\", 112707], \n\t\t[\"games\", 105728], \n\t\t[\"players\", 80679], \n\t\t[\"will\", 52340], \n\t\t[\"play\", 42400], \n\t\t[\"version\", 38410], \n\t\t[\"cards\", 35149], \n\t\t[\"card\", 34870], \n\t\t[\"released\", 33341], \n\t\t[\"2\", 28870], \n\t\t[\"points\", 26591], \n\t\t[\"time\", 25853], \n\t\t[\"three\", 25500], \n\t\t[\"level\", 24051], \n\t\t[\"series\", 23436], \n\t\t[\"number\", 23335], \n\t\t[\"characters\", 23140], \n\t\t[\"character\", 21261], \n\t\t[\"video\", 20544], \n\t\t[\"original\", 20520], \n\t\t[\"chess\", 19964], \n\t\t[\"set\", 19842], \n\t\t[\"3\", 18092], \n\t\t[\"mode\", 17844], \n\t\t[\"features\", 16443], \n\t\t[\"played\", 16115], \n\t\t[\"move\", 15657], \n\t\t[\"based\", 15613], \n\t\t[\"called\", 15605], \n\t\t[\"rules\", 15571], \n\t\t[\"score\", 15249], \n\t\t[\"release\", 14968], \n\t\t[\"final\", 14367], \n\t\t[\"main\", 13939], \n\t\t[\"levels\", 13663], \n\t\t[\"items\", 13360], \n\t\t[\"special\", 13332], \n\t\t[\"board\", 13288], \n\t\t[\"online\", 12861], \n\t\t[\"system\", 12849], \n\t\t[\"playing\", 12531], \n\t\t[\"beer\", 12230], \n\t\t[\"super\", 12025], \n\t\t[\"order\", 11532], \n\t\t[\"nintendo\", 11363], \n\t\t[\"playstation\", 11247], \n\t\t[\"include\", 11131], \n\t\t[\"gameplay\", 11009], \n\t\t[\"versions\", 10914]\n\t], \n\t[\n\t\t[\"language\", 149361], \n\t\t[\"english\", 92811], \n\t\t[\"word\", 79375], \n\t\t[\"languages\", 68906], \n\t\t[\"names\", 56480], \n\t\t[\"term\", 47898], \n\t\t[\"example\", 46912], \n\t\t[\"form\", 43898], \n\t\t[\"meaning\", 38166], \n\t\t[\"common\", 36463], \n\t\t[\"number\", 32288], \n\t\t[\"latin\", 27480], \n\t\t[\"written\", 26917], \n\t\t[\"list\", 25071], \n\t\t[\"called\", 23330], \n\t\t[\"spoken\", 23085], \n\t\t[\"text\", 22672], \n\t\t[\"forms\", 22410], \n\t\t[\"refer\", 21825], \n\t\t[\"modern\", 20685], \n\t\t[\"letters\", 20633], \n\t\t[\"letter\", 20288], \n\t\t[\"standard\", 19483], \n\t\t[\"speakers\", 17581], \n\t\t[\"include\", 16572], \n\t\t[\"case\", 16026], \n\t\t[\"derived\", 15405], \n\t\t[\"referred\", 14955], \n\t\t[\"dialect\", 14896], \n\t\t[\"note\", 14794], \n\t\t[\"system\", 14469], \n\t\t[\"refers\", 14460], \n\t\t[\"considered\", 14424], \n\t\t[\"numbers\", 14305], \n\t\t[\"generally\", 14191], \n\t\t[\"person\", 13218], \n\t\t[\"official\", 13202], \n\t\t[\"origin\", 12983], \n\t\t[\"phrase\", 12851], \n\t\t[\"characters\", 12821], \n\t\t[\"three\", 12738], \n\t\t[\"examples\", 12645], \n\t\t[\"dialects\", 12625], \n\t\t[\"commonly\", 12220], \n\t\t[\"based\", 12212], \n\t\t[\"original\", 11929], \n\t\t[\"alphabet\", 11877], \n\t\t[\"century\", 11845], \n\t\t[\"second\", 11649], \n\t\t[\"usage\", 11514]\n\t], \n\t[\n\t\t[\"medical\", 81136], \n\t\t[\"health\", 62222], \n\t\t[\"patients\", 52027], \n\t\t[\"hospital\", 50570], \n\t\t[\"disease\", 50131], \n\t\t[\"treatment\", 47234], \n\t\t[\"cells\", 44762], \n\t\t[\"blood\", 41010], \n\t\t[\"medicine\", 35521], \n\t\t[\"cell\", 35353], \n\t\t[\"cancer\", 33582], \n\t\t[\"drug\", 31790], \n\t\t[\"care\", 29153], \n\t\t[\"brain\", 27701], \n\t\t[\"protein\", 25276], \n\t\t[\"clinical\", 24232], \n\t\t[\"patient\", 23703], \n\t\t[\"human\", 23609], \n\t\t[\"body\", 23564], \n\t\t[\"gene\", 23558], \n\t\t[\"surgery\", 23230], \n\t\t[\"effects\", 22509], \n\t\t[\"dna\", 21575], \n\t\t[\"study\", 21470], \n\t\t[\"cases\", 21286], \n\t\t[\"drugs\", 20795], \n\t\t[\"symptoms\", 20597], \n\t\t[\"include\", 20222], \n\t\t[\"studies\", 19473], \n\t\t[\"risk\", 18956], \n\t\t[\"heart\", 18831], \n\t\t[\"system\", 18779], \n\t\t[\"therapy\", 18259], \n\t\t[\"associated\", 18166], \n\t\t[\"diseases\", 17413], \n\t\t[\"pain\", 16328], \n\t\t[\"syndrome\", 16059], \n\t\t[\"including\", 15603], \n\t\t[\"common\", 15528], \n\t\t[\"proteins\", 15371], \n\t\t[\"levels\", 14738], \n\t\t[\"tissue\", 14680], \n\t\t[\"skin\", 14437], \n\t\t[\"muscle\", 14436], \n\t\t[\"infection\", 13986], \n\t\t[\"genetic\", 13698], \n\t\t[\"activity\", 13262], \n\t\t[\"bone\", 13248], \n\t\t[\"genes\", 12996], \n\t\t[\"called\", 12992]\n\t], \n\t[\n\t\t[\"club\", 291842], \n\t\t[\"team\", 241041], \n\t\t[\"league\", 239635], \n\t\t[\"season\", 219303], \n\t\t[\"played\", 197371], \n\t\t[\"cup\", 178327], \n\t\t[\"football\", 159512], \n\t\t[\"match\", 114696], \n\t\t[\"final\", 108423], \n\t\t[\"won\", 100308], \n\t\t[\"division\", 87234], \n\t\t[\"clubs\", 81631], \n\t\t[\"teams\", 81039], \n\t\t[\"second\", 80340], \n\t\t[\"matches\", 75647], \n\t\t[\"championship\", 68096], \n\t\t[\"scored\", 67743], \n\t\t[\"goals\", 65912], \n\t\t[\"play\", 65408], \n\t\t[\"playing\", 63517], \n\t\t[\"round\", 63266], \n\t\t[\"players\", 62152], \n\t\t[\"win\", 60486], \n\t\t[\"game\", 59595], \n\t\t[\"side\", 59044], \n\t\t[\"united\", 56338], \n\t\t[\"national\", 55634], \n\t\t[\"games\", 55024], \n\t\t[\"fc\", 54618], \n\t\t[\"goal\", 54103], \n\t\t[\"time\", 53022], \n\t\t[\"career\", 52535], \n\t\t[\"debut\", 50146], \n\t\t[\"year\", 48994], \n\t\t[\"player\", 48653], \n\t\t[\"rugby\", 47338], \n\t\t[\"three\", 47010], \n\t\t[\"tournament\", 45088], \n\t\t[\"years\", 43663], \n\t\t[\"manager\", 41932], \n\t\t[\"stadium\", 40446], \n\t\t[\"top\", 39879], \n\t\t[\"champions\", 39613], \n\t\t[\"joined\", 38914], \n\t\t[\"lost\", 38881], \n\t\t[\"signed\", 38388], \n\t\t[\"city\", 35640], \n\t\t[\"competition\", 34291], \n\t\t[\"premier\", 33871], \n\t\t[\"winning\", 33843]\n\t], \n\t[\n\t\t[\"published\", 256420], \n\t\t[\"book\", 250232], \n\t\t[\"books\", 140892], \n\t\t[\"work\", 110416], \n\t\t[\"wrote\", 107171], \n\t\t[\"magazine\", 85501], \n\t\t[\"writing\", 77030], \n\t\t[\"written\", 76898], \n\t\t[\"works\", 75120], \n\t\t[\"novel\", 74155], \n\t\t[\"history\", 66926], \n\t\t[\"author\", 61923], \n\t\t[\"story\", 60417], \n\t\t[\"stories\", 60383], \n\t\t[\"life\", 55012], \n\t\t[\"editor\", 53731], \n\t\t[\"literature\", 51738], \n\t\t[\"edition\", 51666], \n\t\t[\"press\", 51634], \n\t\t[\"poetry\", 50407], \n\t\t[\"literary\", 47623], \n\t\t[\"journal\", 46512], \n\t\t[\"publication\", 46430], \n\t\t[\"writer\", 45833], \n\t\t[\"times\", 42528], \n\t\t[\"articles\", 42371], \n\t\t[\"newspaper\", 41887], \n\t\t[\"publishing\", 40501], \n\t\t[\"english\", 40033], \n\t\t[\"writers\", 39788], \n\t\t[\"library\", 38689], \n\t\t[\"years\", 37018], \n\t\t[\"paper\", 36787], \n\t\t[\"collection\", 35651], \n\t\t[\"short\", 34783], \n\t\t[\"series\", 34601], \n\t\t[\"fiction\", 33783], \n\t\t[\"novels\", 33268], \n\t\t[\"review\", 32437], \n\t\t[\"time\", 32015], \n\t\t[\"poems\", 31090], \n\t\t[\"including\", 29809], \n\t\t[\"poem\", 28743], \n\t\t[\"early\", 28501], \n\t\t[\"issue\", 27897], \n\t\t[\"authors\", 27552], \n\t\t[\"poet\", 27213], \n\t\t[\"publications\", 26996], \n\t\t[\"volume\", 26726], \n\t\t[\"article\", 26539]\n\t], \n\t[\n\t\t[\"german\", 172197], \n\t\t[\"germany\", 121131], \n\t\t[\"von\", 58106], \n\t\t[\"berlin\", 49291], \n\t\t[\"war\", 41302], \n\t\t[\"der\", 36579], \n\t\t[\"1945\", 29360], \n\t\t[\"austria\", 27860], \n\t\t[\"vienna\", 25456], \n\t\t[\"ii\", 23359], \n\t\t[\"austrian\", 21004], \n\t\t[\"die\", 20967], \n\t\t[\"swiss\", 20006], \n\t\t[\"switzerland\", 18646], \n\t\t[\"nazi\", 18459], \n\t\t[\"und\", 18176], \n\t\t[\"hungarian\", 18163], \n\t\t[\"czech\", 17813], \n\t\t[\"munich\", 16825], \n\t\t[\"1933\", 15681], \n\t\t[\"hitler\", 15400], \n\t\t[\"prague\", 15064], \n\t\t[\"hungary\", 15010], \n\t\t[\"prussia\", 14830], \n\t\t[\"1938\", 14611], \n\t\t[\"1939\", 14100], \n\t\t[\"karl\", 13856], \n\t\t[\"hamburg\", 13457], \n\t\t[\"1936\", 12530], \n\t\t[\"founded\", 12390], \n\t\t[\"wilhelm\", 12008], \n\t\t[\"otto\", 11766], \n\t\t[\"died\", 11724], \n\t\t[\"1934\", 11525], \n\t\t[\"history\", 11496], \n\t\t[\"1935\", 11044], \n\t\t[\"prussian\", 11032], \n\t\t[\"ss\", 11017], \n\t\t[\"hans\", 10920], \n\t\t[\"friedrich\", 10895], \n\t\t[\"born\", 10669], \n\t\t[\"frankfurt\", 10594], \n\t\t[\"johann\", 10121], \n\t\t[\"franz\", 10023], \n\t\t[\"1930\", 9937], \n\t\t[\"1932\", 9806], \n\t\t[\"east\", 9756], \n\t\t[\"1937\", 9748], \n\t\t[\"1940\", 9535], \n\t\t[\"pomerania\", 9424]\n\t]\n]}"
  },
  {
    "path": "toolkits/topic_modeling/lda_sequential_cgs.cpp",
    "content": "/*  \n * Copyright (c) 2009 Carnegie Mellon University. \n *     All rights reserved.\n *\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\n *  you may not use this file except in compliance with the License.\n *  You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n *  Unless required by applicable law or agreed to in writing,\n *  software distributed under the License is distributed on an \"AS\n *  IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n *  express or implied.  See the License for the specific language\n *  governing permissions and limitations under the License.\n *\n *\n */\n\n\n#include \n#include \n#include \n#include \n\n\n#include \n#include \n#include \n\n#include \n#include \n\n\n#include \n\n\n#include \n\ntypedef uint32_t word_id_type;\ntypedef uint32_t doc_id_type;\ntypedef uint16_t topic_id_type;\ntypedef uint32_t count_type;\n#define NULL_TOPIC topic_id_type(-1)\n\nstruct token_type {\n  word_id_type word;\n  doc_id_type doc;\n  token_type(const word_id_type& word = 0, const doc_id_type& doc = 0) : \n    word(word), doc(doc) { }\n};\nstd::ostream& operator<<(std::ostream& out, const token_type& tok) {\n  return out << \"(\" << tok.word << \", \" << tok.doc << \")\";\n}\n\n\nstruct corpus_type {\n  size_t nwords, ndocs, ntokens;\n  std::vector< token_type > tokens;\n  std::vector dictionary;\n  std::vector< word_id_type > ntokens_in_doc;\n  \n  corpus_type(const std::string& dictionary_fname, \n              const std::string& counts_fname ) : \n    nwords(0), ndocs(0), ntokens(0) {\n    dictionary.reserve(20000);\n    ntokens_in_doc.reserve(5000);\n    tokens.reserve(100000);\n    load_dictionary(dictionary_fname);\n    load_counts(counts_fname);\n  }\n  void load_dictionary(const std::string& fname) {\n    std::ifstream fin(fname.c_str());\n    std::string str;\n    while(fin.good()) {\n      std::getline(fin, str);\n      if(fin.good()) { dictionary.push_back(str); nwords++; }\n    }\n    fin.close();\n  }\n\n  void load_counts(const std::string& fname)  {\n    std::ifstream fin(fname.c_str());    \n    while(fin.good()) {\n      // Read a collection of tokens\n      const size_t NULL_VALUE(-1);\n      size_t word = NULL_VALUE, doc = NULL_VALUE, count = NULL_VALUE;\n      fin >> doc >> word >> count;\n      if(fin.good()) {\n        assert(word != NULL_VALUE && doc != NULL_VALUE && count != NULL_VALUE);\n        // update the doc counter\n        ndocs = std::max(ndocs, doc + 1);\n        // Assert valid word\n        assert(word < nwords);\n        // Update the words in document counter\n        if(doc >= ntokens_in_doc.size())\n          ntokens_in_doc.resize(doc+1, 0);\n        ntokens_in_doc[doc] += count;\n        // Add all the tokens\n        token_type tok; tok.word = word; tok.doc = doc;\n        for(size_t i = 0; i < count; ++i) tokens.push_back(tok);\n      }\n    }\n    fin.close();\n    ntokens = tokens.size();\n  } // end of load counts\n\n  void shuffle_tokens() { graphlab::random::shuffle(tokens); }\n}; // end of corpus\n\n\n\ntemplate\nclass matrix {\nprivate:\n  size_t _rows, _cols;\n  std::vector data;\n\n  const size_t linear_index(const size_t& i, const size_t& j) const {\n    assert(i < _rows && j < _cols);\n    return i + j * _rows;\n  }\n\npublic:\n  matrix(const size_t& rows, const size_t& cols, const T& zero = T(0)) :\n    _rows(rows), _cols(cols), data(rows*cols, zero) { };\n  const T& operator()(const size_t& i, const size_t& j) const {\n    return data[linear_index(i,j)];\n  }\n  size_t rows() const { return _rows; }\n  size_t cols() const { return _cols; }\n  T& operator()(const size_t& i, const size_t& j) {\n    return data[linear_index(i,j)];\n  }\n  const T& operator()(const size_t& i) const {\n    assert(i < data.size());\n    return data[i];\n  }\n  T& operator()(const size_t& i) {\n    assert(i < data.size());\n    return data[i];\n  }\n  void zeros() {\n    std::fill(data.begin(), data.end(), T(0));\n  }\n  void operator+=(const matrix& other) {\n    assert(_rows == other._rows);\n    assert(_cols == other._cols);\n    for(size_t i = 0; i < data.size(); ++i) data[i] += other.data[i];\n  }\n  T sum() const {\n    T z(0);\n    for(size_t i = 0; i < data.size(); ++i) z += data[i];\n    return z;\n  }\n}; // end of matrix \ntypedef matrix mat_type;\n\n\nclass collapsed_gibbs {\npublic:\n  const corpus_type* corpus_ptr;\n  const size_t ntopics;\n  const double alpha, beta;\n \n  std::vector< topic_id_type > topics;\n  //! n_td(t,d) Number of occurences of topic t in document d\n  mat_type n_td;\n  //! n_wt(w,t) Number of occurences of word w in topic t\n  mat_type n_wt;\n  //! n_t(t) The total number of words assigned to topic t\n  mat_type n_t;\n  //! number of times a token was assigned to a new topic\n  size_t nchanges;\n\n  collapsed_gibbs(const corpus_type& corpus, \n                  const size_t& ntopics,\n                  const double& alpha,\n                  const double& beta) : \n    corpus_ptr(&corpus), ntopics(ntopics), alpha(alpha),\n    beta(beta), topics(corpus.ntokens, NULL_TOPIC),\n    n_td(ntopics, corpus.ndocs, 0),\n    n_wt(corpus.nwords, ntopics, 0),\n    n_t(ntopics, 1, 0),\n    nchanges(0) { }\n  \n  \n  void iterate() {\n    assert(corpus_ptr != NULL);\n    const corpus_type& corpus = *corpus_ptr;\n    // Reset the number of changes\n    nchanges = 0;\n    std::vector conditional(ntopics);\n\n    // Loop over all the tokens\n    for(size_t i = 0; i < corpus.ntokens; ++i) {\n      // Get the word and document for the ith token\n      const word_id_type w = corpus.tokens[i].word;\n      const doc_id_type d = corpus.tokens[i].doc;\n      const topic_id_type old_topic = topics[i];\n\n      // Remove the word from the current counters\n      if(old_topic != NULL_TOPIC) {\n        --n_td(old_topic, d); --n_wt(w, old_topic), --n_t(old_topic);\n      }\n\n      // Construct the conditional\n      double normalizer = 0;\n      for(size_t t = 0; t < ntopics; ++t) {\n        conditional[t] = (alpha + n_td(t,d)) * (beta + n_wt(w,t)) /\n          (beta * corpus.nwords + n_t(t)); \n        normalizer += conditional[t];\n      }\n      assert(normalizer > 0);\n\n      // Draw a new value\n      topic_id_type new_topic = 0;\n      // normalize and then sample\n      for(size_t t = 0; t < ntopics; ++t) conditional[t] /= normalizer;\n      new_topic = graphlab::random::multinomial(conditional);\n\n      // Update the topic assignment and counters\n      topics[i] = new_topic;\n      if(new_topic != old_topic) nchanges++;\n      ++n_td(new_topic, d); ++n_wt(w, new_topic), ++n_t(new_topic);\n    } // end of loop over tokens\n\n    const size_t n_td_sum = n_td.sum();\n    const size_t n_wt_sum = n_wt.sum();\n    const size_t n_t_sum = n_t.sum();\n    assert(n_td_sum == corpus.ntokens);\n    assert(n_wt_sum == corpus.ntokens);\n    assert(n_t_sum == corpus.ntokens);\n  }\n};\n\n\ndouble log_likelihood(const double& alpha, const double& beta,\n                      const mat_type& n_td, const mat_type& n_wt) {\n  using boost::math::lgamma;\n  const size_t ndocs  = n_td.cols();\n  const size_t ntopics = n_td.rows();\n  const size_t nwords = n_wt.rows();\n\n  mat_type n_t(ntopics,1, 0);\n  for(size_t t = 0; t < n_wt.cols(); ++t) \n    for(size_t w = 0; w < n_wt.rows(); ++w) n_t(t) += n_wt(w,t);\n      \n  \n  // Matlab Functions:\n  //\n  //  llik_w_given_z = ...\n  //    ntopics * (gammaln(nwords * beta) - nwords * gammaln(beta)) + ...\n  //    sum((sum(gammaln(n_wt + beta)) - gammaln( sum(n_wt) + nwords*beta)));\n  //\n  //  llik_z = ...\n  //    ndocs * (gammaln(ntopics * alpha) - ntopics * gammaln(alpha)) + ...\n  //    sum(sum(gammaln(n_td + alpha)) - gammaln(sum(n_td) + ntopics * alpha));\n\n  double llik_words_given_topics = \n    ntopics * (lgamma(nwords * beta) - nwords * lgamma(beta));\n  for(size_t t = 0; t < ntopics; ++t) {\n    for(size_t w = 0; w < nwords; ++w) {\n      llik_words_given_topics += lgamma(n_wt(w,t) + beta);\n    }\n    llik_words_given_topics -= lgamma(n_t(t) + nwords * beta);\n  }\n  double llik_topics = ndocs * (lgamma(ntopics * alpha) - ntopics * lgamma(alpha));\n  for(size_t d = 0; d < ndocs; ++d) {\n    size_t ntokens_in_doc = 0;\n    for(size_t t = 0; t < ntopics; ++t) {\n      llik_topics += lgamma(n_td(t,d) + alpha);\n      ntokens_in_doc += n_td(t,d); \n    }\n    llik_topics -= lgamma(ntokens_in_doc + ntopics * alpha);\n  }\n  return llik_words_given_topics + llik_topics;\n} // end of log_likelihood\n\n\nvoid display_top(const corpus_type& corpus,\n                 const mat_type& n_wt,\n                 const size_t& ntop) {\n  assert(ntop > 0);\n  const size_t nwords = n_wt.rows();\n  const size_t ntopics = n_wt.cols();\n  typedef std::pair cw_pair_type;\n  for(size_t t = 0; t < ntopics; ++t) {\n    std::set< cw_pair_type > top_words;\n    for(size_t w = 0; w < nwords; ++w) {\n      if(top_words.size() < ntop || n_wt(w,t) > top_words.begin()->first) {\n        top_words.insert(std::make_pair(n_wt(w,t), w));\n        if(top_words.size() > ntop) top_words.erase(top_words.begin());\n      }\n    }\n    std::cout << std::endl;\n    rev_foreach(const cw_pair_type& pair, top_words) {\n      std::cout << corpus.dictionary.at(pair.second) << \", \";\n    }\n    std::cout << std::endl;\n  }\n} // end of display top\n\n\n\n\n\n\n\n\nint main(int argc, char** argv) {\n\n  std::string dictionary_fname(\"dictionary.txt\");\n  std::string counts_fname(\"counts.tsv\");\n  size_t ntopics(50);\n  size_t nburnin(50);\n  size_t nsamples(10);\n  double alpha(50.0/double(ntopics));\n  double beta(0.1);\n  size_t topk(20);\n  std::string llik_fname(\"llik.txt\");\n  std::string doctop_fname(\"doctop.txt\");\n  std::string wordtop_fname(\"wordtop.txt\");\n\n\n  // Parse command line options\n  namespace po = boost::program_options;\n  po::options_description desc(\"LDA sampler code\");\n  desc.add_options()\n    (\"help\", \"produce help message\")\n    (\"dictionary\", po::value(&dictionary_fname)->\n     default_value(dictionary_fname), \"Dictionary file\")\n    (\"counts\", po::value(&counts_fname)->\n     default_value(counts_fname), \"Counts file\")\n    (\"ntopics\", po::value(&ntopics)->\n     default_value(ntopics), \"Number of topics\")\n    (\"nburnin\", po::value(&nburnin)->\n     default_value(nburnin), \"Number of iterations\")\n    (\"nsamples\", po::value(&nsamples)->\n     default_value(nsamples), \"Number of iterations\")\n    (\"alpha\", po::value(&alpha)->\n     default_value(alpha), \"Alpha prior\")\n    (\"beta\", po::value(&beta)->\n     default_value(beta), \"Beta prior\")\n    (\"doctop_fname\", po::value(&doctop_fname)->\n     default_value(doctop_fname), \"doctop_fname\")\n    (\"wordtop_fname\", po::value(&wordtop_fname)->\n     default_value(wordtop_fname), \"wordtop_fname\")\n    (\"topk\", po::value(&topk)->\n     default_value(topk), \"number of top k to show\");\n  \n  po::variables_map vm;\n  po::store(po::parse_command_line(argc, argv, desc), vm);\n  po::notify(vm);    \n  if (vm.count(\"help\")) {\n    std::cout << desc << \"\\n\";\n    return EXIT_FAILURE;\n  }\n\n  if (dictionary_fname.length() == 0 || counts_fname.length() == 0) {\n    std::cout << \"Both counts and dictionary must be specified\" << std::endl;\n    std::cout << desc << \"\\n\";\n    return EXIT_FAILURE; \n  }\n  \n  std::cout << \"Loading the corpus.\" << std::endl;\n  corpus_type corpus(dictionary_fname, counts_fname);\n\n  std::cout << \"Number of words:   \" << corpus.nwords << std::endl\n            << \"Number of docs:    \" << corpus.ndocs << std::endl\n            << \"Number of tokens:  \" << corpus.ntokens << std::endl\n            << \"Ntopics:           \" << ntopics << std::endl\n            << \"Alpha:             \" << alpha   << std::endl\n            << \"Beta:              \" << beta    << std::endl;\n\n  std::cout << \"Seeding Generator: \" << std::endl;\n  graphlab::random::nondet_seed();\n  std::cout << \"Shuffling corpus: \" << std::endl;\n  corpus.shuffle_tokens();\n\n\n  std::cout << \"Constructing Gibbs Sampler: \" << std::endl;\n  collapsed_gibbs gibbs(corpus, ntopics, alpha, beta);\n  \n  \n  std::ofstream llik_fout(llik_fname.c_str());\n  llik_fout.precision(16);\n\n  std::cout << \"Starting Burnin\" << std::endl;\n  for(size_t i = 0; i < nburnin; ++i) {\n    std::cout << \"Burnin iteration: \" << i << std::endl;\n    gibbs.iterate();\n    std::cout << \"Computing top \" << topk << \" of each topic\" << std::endl;\n    display_top(corpus, gibbs.n_wt, topk);\n    std::cout << \"Number of changes: \" << gibbs.nchanges << std::endl\n              << \"Prop. Changes:     \" \n              << double(gibbs.nchanges)/ corpus.ntokens << std::endl;\n    double llik = log_likelihood(gibbs.alpha, gibbs.beta, gibbs.n_td, gibbs.n_wt);\n    std::cout << \"Log-likelihood:    \" // std::setprecision(8) <<\n              <<  llik << std::endl;\n    llik_fout << llik << '\\t' << gibbs.nchanges << std::endl;\n\n  }\n  std::cout << \"Finished burnin.  Preparing final sample set.\" << std::endl;\n \n  mat_type n_td(ntopics, corpus.ndocs, 0);\n  mat_type n_wt(corpus.nwords, ntopics, 0);\n  mat_type n_t(ntopics, 1, 0);\n\n  for(size_t i = 0; i < nsamples; ++i) {\n    std::cout << \"Sampling iteration: \" << i << std::endl;\n    gibbs.iterate();\n    std::cout << \"Number of changes: \" << gibbs.nchanges << std::endl\n              << \"Prop. Changes:     \" \n              << double(gibbs.nchanges)/ corpus.ntokens << std::endl;\n    std::cout << \"Accumulating sample\" << std::endl;\n    n_td += gibbs.n_td;\n    n_wt += gibbs.n_wt;\n    n_t  += gibbs.n_t;\n\n    std::cout << \"Computing top \" << topk << \" of each topic\" << std::endl;\n    display_top(corpus, n_wt, topk);    \n    std::cout << \"Number of changes: \" << gibbs.nchanges << std::endl\n              << \"Prop. Changes:     \" \n              << double(gibbs.nchanges)/ corpus.ntokens << std::endl;\n    double llik = log_likelihood(gibbs.alpha, gibbs.beta, gibbs.n_td, gibbs.n_wt);\n    std::cout << \"Log-likelihood:    \" // std::setprecision(8) <<\n              <<  llik << std::endl;\n    llik_fout << llik << '\\t' << gibbs.nchanges << std::endl;\n\n  }\n  llik_fout.close();\n\n  std::cout << \"Saving doctop: \" << doctop_fname << std::endl;\n  std::ofstream doctop_fout(doctop_fname.c_str());\n  for(size_t d = 0; d < corpus.ndocs; ++d) {\n    double normalizer = ntopics * alpha; \n    for(size_t t = 0; t < ntopics; ++t) \n      normalizer += double(n_td(t,d)) / double(nsamples);\n    for(size_t t = 0; t < ntopics; ++t) {\n      const double value = \n        (double(n_td(t,d))/double(nsamples) + alpha) / normalizer;\n      doctop_fout << value << ((t+1 < ntopics)? '\\t' : '\\n');\n    }\n  }\n  doctop_fout.close();\n  \n  std::cout << \"Saving wordtop: \" << wordtop_fname << std::endl;\n\n\n\n  return EXIT_SUCCESS;\n}\n"
  },
  {
    "path": "toolkits/topic_modeling/topic_modeling.dox",
    "content": "/**\n\n\\page topic_modeling Topic Modeling\n\n\n\\brief The topic modeling toolkit contains a collection of\napplications targeted at clustering documents and extracting topical\nrepresentations.  The resulting topical representation can be used as\na feature space in information retrieval tasks and to group topically\nrelated words and documents.\n\nCurrently the text modeling toolkit implements a fast asynchronous\ncollapsed Gibbs sampler for the widely used Latent Dirichlet\nAllocation (LDA) model. In the near future we plan to add a Collapsed\nVariational Bayesian inference algorithm for the LDA model as well as\nsome more general topic models.\n\n\n\\section lda_model  The LDA Model \n\nThe LDA\nmodel  associates a topic id with each token (word) in each\ndocument in the input corpus.  Conceptually, topic ids correspond to\nsemantic groups like \"foods\", \"colors\", and \"politics\" however the\nassociation between the id 1, 2, ..., N and the particular topic\nmeaning \"foods\", \"colors\", ... is not know in advance and can be\nresolved by running the approximate inference algorithm.  In addition\nthe LDA model assigns a distribution over topics to each document and\na distribution over term to each topic.  The the topic id for each\ntoken is drawn from the topic distribution for each document.  The\nactual word is then drawn from the term distribution for that topic.\nAt a high-level the LDA model encodes the following intuitive\nassumptions:\n\n\\li Words in the same document are topically related.\n\n\\li Documents that share common terms are topically related. \n\nSolving for the latent topic assignments of each token as well as the\ntopic distribution for each document and the term distribution for\neach topic is a challenging (NP-Hard) task. Fortunately there are\nseveral approximate inference algorithms that typically can resolve\ncoherent posterior estimates for the LDA model.\n\n\n\\subsection collapsed_gibbs The Collapsed Gibbs Sampler\n\n\nThe topic modeling toolkit currently implements an asynchronous\nvariant of the Collapsed Gibbs Sampler described by Griffiths and\nSteyvers in their landmark paper Finding\nScientific Topics.  The collapsed Gibbs sampler is a Markov Chain\nMonte Carlo (MCMC) algorithm which generates a sequence of topic\nassignments for each token that in the limit converge to a sequence of\nsamples drawn from the posterior distribution.  In practice the\nalgorithm is run for a sufficient long time to allow the topics to\n\"converge\" (sometimes referred to as burn-in) and then the last few\nsamples are used to estimate the posterior distribution over topics\nfor each document and the posterior distribution over words for each\ntopic.\n\n\n\n\n\\subsection parallel_collapsed_gibbs The Parallel Collapsed Gibbs Sampler\n\n\nThe parallelization of the Collapsed Gibbs Sampler is achieved by\ndrawing new assignments for multiple tokens simultaneously using a\nmethod that is similar to that described by Ahmed et al. \n(Paper).  \nUnfortunately, the collapsed LDA model used to accelerate mixing of\nthe Gibbs sampler also eliminates any conditional independence\nstructure needed to obtain a parallel ergodic sampler as described by \nGonzalez et al. (Paper)\n\n\nHowever, by mapping the collapsed Gibbs sampler into the GraphLab\nabstraction we obtain a statistically more efficient algorithm.  To\nimplement the collapsed Gibbs sampler in GraphLab we construct a\nbipartite graph connecting each document with terms that occur in that\ndocument.  Each edge contains the token count and latent topic\nassignments for that token.  The GraphLab update function maintains\nthe term and document counts during the gather and apply phases and\nthen samples new values for the tokens on the scatter phase.  We\nexploit local atomic integer operations and the GraphLab caching model\nto immediately propagate changes.  The asynchronous consistency model\nensures that only one token per document term pair is sampled at a\ntime improving upon the original formulation of the asynchronous Gibbs\nsampler described by Ahmed et al. (Paper) or the\nsampler described by Asuncion et al. (Paper).\n\n\n\n\\section cgs_lda_usage Usage \n==============\n\nThe collapsed Gibbs sampler application (\\ref cgs_lda.cpp) takes as an\ninput a text corpus represented as one or more token files.  Each\ntoken consists of lines in the form:\n\n\\verbatim\n  \n\\endverbatim\n\nfor example a file containing:\n\n\\verbatim\n0    0     2\n0    4     1\n1    2     3\n\\endverbatim\n \nimplies that the word with id 0 occurs twice in document 0, the word\nwith id 4 occurs once in document 0, and the word with id 2 occurs\nthree times in document 1.\n\nOn termination the system outputs for each term the number of\noccurrences of that term that have been assigned to each topic and for\neach document the number of tokens assigned to each document.\n\n\n\\subsection cgs_lda_example Example\n\nTo demonstrate how the CGS LDA application works we have obtained a\ncopy of the Daily Kos bag-of-words data from the  UCI\nRepository  and reformatted it for the cgs_lda application.  You\ncan download the reformatted data from here.\nOnce extracted the folder contains:\n\n\\verbatim\n> ls -lR daily_kos\ntotal 120\n-rw-r--r--  1 jegonzal  staff    904 Jul  1 22:37 README\n-rw-r--r--@ 1 jegonzal  staff  55467 Jul  1 22:21 dictionary.txt\ndrwxr-xr-x  3 jegonzal  staff    102 Jul  1 22:21 tokens\n\n./tokens:\ntotal 7960\n-rw-r--r--  1 jegonzal  staff  4074516 Jul  1 22:21 doc_word_count.tsv\n\\endverbatim\n\nTo run the CGS_LDA GraphLab application on a single machine we simply run:\n\\verbatim\n> ./cgs_lda --corpus ./daily_kos/tokens --dictionary ./daily_kos/dictionary.txt\n\\endverbatim\n\nThis will run indefinitley display the top words in each topic every\n10 seconds.  To help visualize the output if you open the webpage\n\\verbatim\ngraphlab/toolkits/topic_modeling/http/index.html\n\\endverbatim\nWe render a word cloud webpage that connects directly to the cgs_lda \nappliction's internal web-server running on localhost port 8090. \n\nIn most cases we will also be interested in collecting the final\nassignments. This can be done by running:\n\\verbatim\n> ./cgs_lda --corpus ./daily_kos/tokens --dictionary ./daily_kos/dictionary.txt \\\n   --word_dir word_counts --doc_dir doc_counts --burnin=60 \n\\endverbatim\n\nThis will run the cgs_lda sampler for roughly 60 seconds and the save\nthe counts of tokens in each topic for the words and documents in the\nfiles \\c word_counts_x_of_x and \\c doc_counts_x_of_x .  If instead you\nwould like to save the counts to seperate folders you can prepend the\nfolder path.\n\nBy default GraphLab runs with two threads.  However we can increase the\nparallelism on a single machine by increasing the number of threads:\n\\verbatim\n> ./cgs_lda --corpus ./daily_kos/tokens --dictionary ./daily_kos/dictionary.txt \\\n  --ncpus=8\n\\endverbatim\n\nThe cgs_lda application can run in the distributed setting as well simply by\nusing MPI to launch it:\n\\verbatim\n> mpiexec --hostfile machine_list.txt -n 16 ./cgs_lda \\\n  --corpus ./daily_kos/tokens --dictionary ./daily_kos/dictionary.txt \\\n  --ncpus=8\n\\endverbatim\n\nThis will run 16 instances each consuming 8 on the machines in the \n\\c machine_list.txt file.  Each of these instances will automatically\ncommunicate splitting the work as well as the memory requirements.  It\nis really that easy!\n\nSince we are running in the distributed setting it is convenient to be\nable to read and write to a distributed filesystem.  We have built HDFS \nsupport into the GraphLab abstraction therefore we can simply change \nthe arguments to be:\n\\verbatim\n> mpiexec --hostfile machine_list.txt -n 16 ./cgs_lda \\\n  --corpus hdfs://bros.ml.cmu.edu/daily_kos/tokens \\\n  --dictionary hdfs://bros.ml.cmu.edu/daily_kos/dictionary.txt \\\n  --ncpus=8\n\\endverbatim\n\n\n\\section convergence Tracking progress\n\nWe have added a Likelihood printout every interval seconds (defined using the --interval command line argument)\nwhich captures the convergence progress of the LDA algorithm. The Likelihood should go up every interval.\n\n\\section cgs_lda_options Command Line Options\n\nThere are a wide range of options available when calling the cgs_lda\nprogram:\n\n\n\\li --corpus _Required_ The path to the token file(s).  This\ncan be a folder, a file, or a folder plus a file prefix. In addition\nthis can be a path in hdfs (e.g., hdfs://namenode/tokens/).\n\n\\li --help Display the help screen listing the available options\n\n\\li --dictionary (Optional) The path to the dictionary\nfile. This can be a local path or a path on hdfs \n(e.g., hdfs://namenode/dictionary.txt).  If no dictionary is provided then\nincremental topk words lists will not be generated and the word list webpage \nwill not be available.\n\n\\li --engine (Optional, Default: asynchronous) The engine type to\nuse when executing the vertex program.  Accepted values are:\n       - synchronous: All tokens are sampled simultaneously leading to\n           faster computation but slower convergence.\n       - asynchronous: Tokens are sampled as resources become available\n           and token conditionals are updated immediately after sampling.  This\n           reduces parallelism and sampling speed but increases the rate of\n           convergence.\n\n\\li --ntopics (Optional, Default: 50) The number of topics to use in the\nLDA model.  Using fewer topics will increase speed and result in more \"mixed\"\nconcepts.  Using more topics will slow down sampling but lead to more specific\ntopics.\n\n\\li --alpha (Optional, Default 1) The hyper-parameter for the topic \ndistribution for each document.  Larger values imply documents have a more \nuniform mix of topics.  Smaller values (less than one) imply that documents\nare more focused on a small subsets of topics.  Note that smaller values \nalso slow down convergence of the sampler.\n\n\\li --beta (Optional, Default 0.1) The hyper-parameter for the\nword distribution in each topic.  Larger values imply that topics contain\nall words equally and smaller values (less than one) imply that topics are\nfocused on a small set of words.  Note that smaller values also slow down \nconvergence of the sampler.\n\n\\li --topk (Optional, Default 5) The number of words to show in\neach topic when incrementally listing the top words in each topic. \nThis also affects the word cloud viewer. \n\n\\li --interval (Optional, Default 10) The time in seconds between\nwhen the incremental listing of top words is presented.  \n\n\\li --max_count (Optional, Default 100) The maximum number of \noccurrences of a token in a document.  If a token occurs more than \n\\c max_count then it is reported as occurring \\c max_count times. \nThis ensures that overly frequent words do not dominate documents. \n\n\\li --loadjson (Optional, Default false) This flag is used to turn\non the experimental JSON graph loader that reads graphs constructed using\nexternal graph builder libraries.  If set to true then the \\c --corpus\nargument must point to the JSON files.\n\n\\li --burnin (Optional, Default -1) The time in seconds to run \nthe sampler before the sample is saved to file (and the sampler terminates). \nIf the value is less than zero then the sample will run indefinitely.\n\n\\li --doc_dir (Optional, Default empty) The location (path/prefix)\nto save the final topic counts for each document after burnin.  This can \nalso be an hdfs path (e.g., hdfs://namenode/folder/prefix).  If this is\nnot set then the per document topic counts are not saved.\n\n\\li --word_dir (Optional, Default empty) The location (path/prefix)\nto save the final topic counts for each word after burnin.  This can \nalso be an hdfs path (e.g., hdfs://namenode/folder/prefix).  If this is\nnot set then the per word topic counts are not saved.\n\n\\li --ncpus (Optional, Default 2) The number of local computation \nthreads to use on each machine.  This should typically match the number \nof physical cores. \n\n\\li --scheduler (Optional, Default sweep) The scheduler to use when \nrunning with the asynchronous engine.  The default is typically sufficient. \n\n\\li --engine_opts (Optional, Default empty) Any additional engine\noptions. See --engine_help for a list of options.\n\n\n\\li --graph_opts (Optional, Default empty) Any additional graph\noptions. See --graph_help for a list of options.\n\n\\li --scheduler_opts (Optional, Default empty) Any additional scheduler\noptions. See --scheduler_help for a list of options.\n\n\n\n*/\n\n\n"
  }
]