Repository: dspinellis/dgsh Branch: master Commit: e51fb999119b Files: 460 Total size: 1.1 MB Directory structure: gitextract_fo8a72ky/ ├── .gitignore ├── .gitmodules ├── .travis/ │ ├── linux-ubuntu-trusty.install.sh │ └── macosx-xcode8.3.install.sh ├── .travis.yml ├── CITATION.cff ├── LICENSE ├── Makefile ├── README.md ├── core-tools/ │ ├── .gitignore │ ├── Makefile.am │ ├── configure.ac │ ├── src/ │ │ ├── .gitignore │ │ ├── Makefile.am │ │ ├── debug.h │ │ ├── dgsh-conc.1 │ │ ├── dgsh-conc.c │ │ ├── dgsh-debug.h │ │ ├── dgsh-elf.s │ │ ├── dgsh-enumerate.1 │ │ ├── dgsh-enumerate.c │ │ ├── dgsh-fft-input.c │ │ ├── dgsh-httpval.1 │ │ ├── dgsh-httpval.c │ │ ├── dgsh-macho.s │ │ ├── dgsh-merge-sum.1 │ │ ├── dgsh-merge-sum.pl │ │ ├── dgsh-monitor.1 │ │ ├── dgsh-monitor.c │ │ ├── dgsh-parallel.1 │ │ ├── dgsh-parallel.sh │ │ ├── dgsh-pecho.c │ │ ├── dgsh-readval.1 │ │ ├── dgsh-readval.c │ │ ├── dgsh-tee.1 │ │ ├── dgsh-tee.c │ │ ├── dgsh-w.c │ │ ├── dgsh-wrap.1 │ │ ├── dgsh-wrap.c │ │ ├── dgsh-writeval.1 │ │ ├── dgsh-writeval.c │ │ ├── dgsh.1 │ │ ├── dgsh.h │ │ ├── dgsh_negotiate.3 │ │ ├── kvstore.c │ │ ├── kvstore.h │ │ ├── minmax.h │ │ ├── negotiate.c │ │ ├── negotiate.h │ │ ├── perm.1 │ │ ├── perm.sh │ │ └── run-built-dgsh.sh │ ├── tests/ │ │ ├── .gitignore │ │ ├── Makefile.am │ │ ├── Makefile.patch │ │ ├── check.hack │ │ └── check_negotiate.c │ └── tests-regression/ │ ├── .gitignore │ ├── author-compare/ │ │ └── out.ok │ ├── bin/ │ │ └── gdate │ ├── code-metrics/ │ │ ├── in/ │ │ │ └── date/ │ │ │ ├── date.c │ │ │ ├── extern.h │ │ │ ├── netdate.c │ │ │ ├── vary.c │ │ │ └── vary.h │ │ └── out.ok │ ├── commit-stats/ │ │ └── out.ok │ ├── compress-compare/ │ │ └── out.ok │ ├── dgsh-wrap/ │ │ ├── .gitignore │ │ ├── dd-args.ok │ │ ├── echo-deaf.ok │ │ ├── echo-s.ok │ │ ├── echo-s_caps.ok │ │ ├── paste1.ok │ │ ├── paste2.ok │ │ ├── paste3.ok │ │ ├── paste4.ok │ │ ├── tee1.ok │ │ └── tee2.ok │ ├── duplicate-files/ │ │ ├── in/ │ │ │ ├── another-same-1 │ │ │ ├── another-same-2 │ │ │ ├── different-file-1 │ │ │ ├── different-file-2 │ │ │ ├── different-file-3 │ │ │ ├── same-file-1 │ │ │ ├── same-file-2 │ │ │ └── same-file-3 │ │ └── out.ok │ ├── map-hierarchy/ │ │ ├── in/ │ │ │ ├── a/ │ │ │ │ └── date/ │ │ │ │ ├── date.c │ │ │ │ ├── vary.c │ │ │ │ └── vary.h │ │ │ └── b/ │ │ │ └── bin/ │ │ │ └── date/ │ │ │ ├── date.c │ │ │ ├── headers/ │ │ │ │ ├── extern.h │ │ │ │ └── vary.h │ │ │ ├── netdate.c │ │ │ └── vary.c │ │ └── out.ok/ │ │ └── map-hierarchy/ │ │ └── in/ │ │ └── b/ │ │ └── bin/ │ │ └── date/ │ │ ├── date.c │ │ ├── headers/ │ │ │ └── vary.h │ │ └── vary.c │ ├── parallel-word-count/ │ │ └── out.ok │ ├── regression/ │ │ ├── errors/ │ │ │ ├── stream-scatter-cycle.ok │ │ │ ├── stream-scatter-cycle.sh │ │ │ ├── unsafe-gather.ok │ │ │ ├── unsafe-gather.sh │ │ │ ├── unsafe-gather2.ok │ │ │ └── unsafe-gather2.sh │ │ ├── graphs/ │ │ │ ├── NMRPipe.ok │ │ │ ├── code-metrics.ok │ │ │ ├── commit-stats.ok │ │ │ ├── committer-plot.ok │ │ │ ├── compress-compare.ok │ │ │ ├── dir.ok │ │ │ ├── duplicate-files.ok │ │ │ ├── ft2d.ok │ │ │ ├── map-hierarchy.ok │ │ │ ├── parallel-logresolve.ok │ │ │ ├── spell-highlight.ok │ │ │ ├── text-properties.ok │ │ │ ├── web-log-report.ok │ │ │ ├── web-log-stats.ok │ │ │ └── word-properties.ok │ │ ├── scripts/ │ │ │ ├── NMRPipe.ok │ │ │ ├── code-metrics.ok │ │ │ ├── commit-stats.ok │ │ │ ├── committer-plot.ok │ │ │ ├── compress-compare.ok │ │ │ ├── dir.ok │ │ │ ├── duplicate-files.ok │ │ │ ├── ft2d.ok │ │ │ ├── map-hierarchy.ok │ │ │ ├── parallel-logresolve.ok │ │ │ ├── spell-highlight.ok │ │ │ ├── text-properties.ok │ │ │ ├── web-log-report.ok │ │ │ ├── web-log-stats.ok │ │ │ └── word-properties.ok │ │ └── warnings/ │ │ ├── single-target.ok │ │ ├── single-target.sh │ │ ├── unsafe-scatter.ok │ │ └── unsafe-scatter.sh │ ├── spell-highlight/ │ │ └── out.ok │ ├── tee/ │ │ ├── oom.err │ │ ├── perm.ok │ │ ├── tee-fastout-I-l.ok │ │ ├── tee-fastout-I-m 2k -f.ok │ │ ├── tee-fastout-I-m 2k.ok │ │ ├── tee-fastout-I.ok │ │ ├── tee-fastout-l.ok │ │ ├── tee-fastout-m 2k -f.ok │ │ ├── tee-fastout-m 2k.ok │ │ ├── tee-fastout.ok │ │ ├── tee-lagout-I-l.ok │ │ ├── tee-lagout-I-m 2k -f.ok │ │ ├── tee-lagout-I-m 2k.ok │ │ ├── tee-lagout-I.ok │ │ ├── tee-lagout-l.ok │ │ ├── tee-lagout-m 2k -f.ok │ │ ├── tee-lagout-m 2k.ok │ │ ├── tee-lagout.ok │ │ └── tee-lahout.ok │ ├── test-dgsh.sh │ ├── test-kvstore.sh │ ├── test-merge-sum.sh │ ├── test-tee.sh │ ├── test-wrap.sh │ ├── text-properties/ │ │ └── out.ok/ │ │ ├── character.txt │ │ ├── digram.txt │ │ ├── trigram.txt │ │ └── words.txt │ ├── web-log-report/ │ │ ├── logfile │ │ └── out.ok │ └── word-properties/ │ ├── LostWorldChap1-3 │ └── out.ok ├── eval/ │ ├── .gitignore │ ├── Makefile │ ├── README.md │ ├── SConstruct │ ├── TextProperties.java │ ├── WebStats.java │ ├── eval-lib.sh │ ├── ft2d.sh │ ├── log-grow.pl │ ├── perf-eval.sh │ ├── web-log-report.pl │ └── webeval.sh ├── example/ │ ├── NMRPipe.sh │ ├── author-compare.sh │ ├── code-metrics.sh │ ├── commit-stats.sh │ ├── committer-plot.sh │ ├── compress-compare.sh │ ├── dir.sh │ ├── duplicate-files.sh │ ├── fft-block8.sh │ ├── ft2d.sh │ ├── map-hierarchy.sh │ ├── parallel-word-count.sh │ ├── reorder-columns.sh │ ├── spell-highlight.sh │ ├── static-functions.sh │ ├── text-properties.sh │ ├── uniform-5x5.sh │ ├── web-log-report.sh │ └── word-properties.sh ├── png/ │ └── README ├── simple-shell/ │ ├── comm_paste.dgsh │ ├── comm_paste.success │ ├── comm_paste_join_diff.dgsh │ ├── comm_paste_join_diff.success │ ├── comm_sort.dgsh │ ├── comm_sort.success │ ├── dir-plain.dgsh │ ├── grep_comm.dgsh │ ├── grep_comm.success │ ├── grep_comm.success-bash │ ├── grep_diff.dgsh │ ├── grep_diff.success │ ├── grep_diff.success-bash │ ├── grep_diff_comm.dgsh │ ├── grep_diff_comm.success1 │ ├── grep_diff_comm.success1-bash │ ├── grep_diff_comm.success2 │ ├── grep_diff_comm.success2-bash │ ├── grep_diff_comm.success3 │ ├── grep_diff_comm.success3-bash │ ├── grep_diff_comm.success4 │ ├── grep_diff_comm.success4-bash │ ├── join_sort.dgsh │ ├── join_sort.success │ ├── join_sort_diff.dgsh │ ├── join_sort_diff.success │ ├── ls_wc.dgsh │ ├── paste_diff.dgsh │ ├── paste_diff.success │ ├── secho_paste.dgsh │ ├── secho_paste.success │ ├── secho_secho_fgrep.dgsh │ ├── secho_secho_fgrep.success │ ├── simple-shell.py │ ├── sort_sort_comm.dgsh │ ├── sort_sort_comm.success │ ├── sort_sort_comm_paste_join_diff.dgsh │ ├── sort_sort_comm_paste_join_diff.success │ ├── tee-copy_diff_comm.dgsh │ ├── tee-copy_diff_comm.success │ ├── tee-scatter_diff_comm.dgsh │ ├── tee-scatter_diff_comm.success1 │ ├── tee-scatter_diff_comm.success2 │ ├── wrap-cat_comm_sort.dgsh │ └── wrap-cat_comm_sort.success ├── test-data/ │ ├── .gitignore │ ├── F │ ├── access.log │ ├── cmp0.success │ ├── cmp1-same1.success │ ├── cmp1-same2.success │ ├── cmp2-diff.success │ ├── cmp2-same.success │ ├── comm_paste.success │ ├── comm_paste_join_diff.success │ ├── comm_sort.success │ ├── d3 │ ├── data.csv │ ├── diff0-noin.success │ ├── diff0-stdin1.success │ ├── diff0-stdin2.success │ ├── diff0.success │ ├── diff1-stdin.success │ ├── diff1.success │ ├── diff2.success │ ├── diff3-0.success │ ├── diff3-1.success │ ├── diff3-2-stdin1.success │ ├── diff3-2-stdin2.success │ ├── diff3-2.success │ ├── diff3-3.success │ ├── diff4.success │ ├── dir-plain.sh │ ├── f1s │ ├── f2s │ ├── f3s │ ├── f4s │ ├── f4ss │ ├── f5s │ ├── f5ss │ ├── ff │ ├── function_bash_tools.success │ ├── function_dgsh_tools.success │ ├── grep-Lcap-c-l-matching-lines-nomatch.success │ ├── grep-Lcap-c-l-matching-lines.success │ ├── grep-Lcap-c-l-nomatch.success │ ├── grep-Lcap-c-l.success │ ├── grep-Lcap-c-matching-lines-l-nomatch.success │ ├── grep-Lcap-c-matching-lines-l.success │ ├── grep-Lcap-c-matching-lines-nomatch.success │ ├── grep-Lcap-c-matching-lines.success │ ├── grep-Lcap-c-nomatch.success │ ├── grep-Lcap-c.success │ ├── grep-Lcap-cat-nomatch.success │ ├── grep-Lcap-cat.success │ ├── grep-Lcap-l-c-matching-lines-nomatch.success │ ├── grep-Lcap-l-c-matching-lines.success │ ├── grep-Lcap-l-c-nomatch.success │ ├── grep-Lcap-l-c.success │ ├── grep-Lcap-l-matching-lines-c-nomatch.success │ ├── grep-Lcap-l-matching-lines-c.success │ ├── grep-Lcap-l-matching-lines-nomatch.success │ ├── grep-Lcap-l-matching-lines.success │ ├── grep-Lcap-l-nomatch.success │ ├── grep-Lcap-l.success │ ├── grep-Lcap-matching-lines-c-l-nomatch.success │ ├── grep-Lcap-matching-lines-c-l.success │ ├── grep-Lcap-matching-lines-c-nomatch.success │ ├── grep-Lcap-matching-lines-c.success │ ├── grep-Lcap-matching-lines-l-c-nomatch.success │ ├── grep-Lcap-matching-lines-l-c.success │ ├── grep-Lcap-matching-lines-l-nomatch.success │ ├── grep-Lcap-matching-lines-l.success │ ├── grep-Lcap-matching-lines-nomatch.success │ ├── grep-Lcap-matching-lines.success │ ├── grep-Lcap-nomatch.success │ ├── grep-Lcap.success │ ├── grep-c-Lcap-l-matching-lines-nomatch.success │ ├── grep-c-Lcap-l-matching-lines.success │ ├── grep-c-Lcap-l-nomatch.success │ ├── grep-c-Lcap-l.success │ ├── grep-c-Lcap-matching-lines-l-nomatch.success │ ├── grep-c-Lcap-matching-lines-l.success │ ├── grep-c-Lcap-matching-lines-nomatch.success │ ├── grep-c-Lcap-matching-lines.success │ ├── grep-c-Lcap-nomatch.success │ ├── grep-c-Lcap.success │ ├── grep-c-cat.success │ ├── grep-c-l-Lcap-matching-lines-nomatch.success │ ├── grep-c-l-Lcap-matching-lines.success │ ├── grep-c-l-Lcap-nomatch.success │ ├── grep-c-l-Lcap.success │ ├── grep-c-l-matching-lines-Lcap-nomatch.success │ ├── grep-c-l-matching-lines-Lcap.success │ ├── grep-c-l-matching-lines.success │ ├── grep-c-l.success │ ├── grep-c-matching-lines-Lcap-l-nomatch.success │ ├── grep-c-matching-lines-Lcap-l.success │ ├── grep-c-matching-lines-Lcap-nomatch.success │ ├── grep-c-matching-lines-Lcap.success │ ├── grep-c-matching-lines-l-Lcap-nomatch.success │ ├── grep-c-matching-lines-l-Lcap.success │ ├── grep-c-matching-lines-l.success │ ├── grep-c-matching-lines.success │ ├── grep-c.success │ ├── grep-f-cat.success │ ├── grep-f.success │ ├── grep-l-Lcap-c-matching-lines-nomatch.success │ ├── grep-l-Lcap-c-matching-lines.success │ ├── grep-l-Lcap-c-nomatch.success │ ├── grep-l-Lcap-c.success │ ├── grep-l-Lcap-matching-lines-c-nomatch.success │ ├── grep-l-Lcap-matching-lines-c.success │ ├── grep-l-Lcap-matching-lines-nomatch.success │ ├── grep-l-Lcap-matching-lines.success │ ├── grep-l-Lcap-nomatch.success │ ├── grep-l-Lcap.success │ ├── grep-l-c-Lcap-matching-lines-nomatch.success │ ├── grep-l-c-Lcap-matching-lines.success │ ├── grep-l-c-Lcap-nomatch.success │ ├── grep-l-c-Lcap.success │ ├── grep-l-c-matching-lines-Lcap-nomatch.success │ ├── grep-l-c-matching-lines-Lcap.success │ ├── grep-l-c-matching-lines.success │ ├── grep-l-c.success │ ├── grep-l-cat.success │ ├── grep-l-matching-lines-Lcap-c-nomatch.success │ ├── grep-l-matching-lines-Lcap-c.success │ ├── grep-l-matching-lines-Lcap-nomatch.success │ ├── grep-l-matching-lines-Lcap.success │ ├── grep-l-matching-lines-c-Lcap-nomatch.success │ ├── grep-l-matching-lines-c-Lcap.success │ ├── grep-l-matching-lines-c.success │ ├── grep-l-matching-lines.success │ ├── grep-l.success │ ├── grep-matching-lines-Lcap-c-l-nomatch.success │ ├── grep-matching-lines-Lcap-c-l.success │ ├── grep-matching-lines-Lcap-c-nomatch.success │ ├── grep-matching-lines-Lcap-c.success │ ├── grep-matching-lines-Lcap-l-c-nomatch.success │ ├── grep-matching-lines-Lcap-l-c.success │ ├── grep-matching-lines-Lcap-l-nomatch.success │ ├── grep-matching-lines-Lcap-l.success │ ├── grep-matching-lines-Lcap-nomatch.success │ ├── grep-matching-lines-Lcap.success │ ├── grep-matching-lines-c-Lcap-l-nomatch.success │ ├── grep-matching-lines-c-Lcap-l.success │ ├── grep-matching-lines-c-Lcap-nomatch.success │ ├── grep-matching-lines-c-Lcap.success │ ├── grep-matching-lines-c-l-Lcap-nomatch.success │ ├── grep-matching-lines-c-l-Lcap.success │ ├── grep-matching-lines-c-l.success │ ├── grep-matching-lines-c.success │ ├── grep-matching-lines-cat.success │ ├── grep-matching-lines-l-Lcap-c-nomatch.success │ ├── grep-matching-lines-l-Lcap-c.success │ ├── grep-matching-lines-l-Lcap-nomatch.success │ ├── grep-matching-lines-l-Lcap.success │ ├── grep-matching-lines-l-c-Lcap-nomatch.success │ ├── grep-matching-lines-l-c-Lcap.success │ ├── grep-matching-lines-l-c.success │ ├── grep-matching-lines-l.success │ ├── grep-matching-lines.success │ ├── grep-noargs-cat.success │ ├── grep-noargs.success │ ├── grep-o-cat.success │ ├── grep-o.success │ ├── grep-v-cat.success │ ├── grep-v.success │ ├── grep_comm.success │ ├── group.success │ ├── hello │ ├── j2 │ ├── join_sort.success │ ├── join_sort_diff.success │ ├── last │ ├── multipipe_one_last.success │ ├── multipipe_one_start.success │ ├── nondgsh.success │ ├── p1 │ ├── paste_diff.success │ ├── read_while.success │ ├── recursive_multipipe_oneline_end.success │ ├── recursive_multipipe_oneline_start.success │ ├── results │ ├── secho_paste.success │ ├── secho_secho_fgrep.success │ ├── sort_sort_comm.success │ ├── sort_sort_comm_paste_join_diff.success │ ├── subshell.success │ ├── tee-copy_diff_comm.success │ ├── top │ ├── world │ └── wrap-cat_comm_sort.success ├── unix-tools/ │ ├── .gitignore │ ├── Makefile │ ├── Readme.md │ ├── cat.sh │ ├── cmp.sh │ ├── cpow.c │ ├── cygwin-sys-select-patch.sh │ ├── diff.sh │ ├── diff3.sh │ ├── echo_echo_dgsh-tee.sh │ ├── install-wrapped.sh │ ├── run_all_simple_tests.sh │ ├── run_simple_test.sh │ ├── run_test.sh │ ├── tee.sh │ ├── test-compat.sh │ ├── wrapped-commands-posix │ └── wrapped-commands-tests └── web/ ├── format-eg.sh ├── format-syntax.sh └── index.html ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ *.dot *.exe *.o *.out *.outb *.pdf *.swp *.test .deps .config .gdb_history /build /png/*.png tags unix-tools/sf* ================================================ FILE: .gitmodules ================================================ [submodule "unix-tools/coreutils"] path = unix-tools/coreutils url = https://github.com/mfragkoulis/coreutils branch = master ignore=dirty [submodule "unix-tools/grep"] path = unix-tools/grep url = https://github.com/mfragkoulis/grep branch = master ignore=dirty [submodule "unix-tools/bash"] path = unix-tools/bash url = https://github.com/mfragkoulis/bash.git branch = master ignore=dirty ================================================ FILE: .travis/linux-ubuntu-trusty.install.sh ================================================ #!/bin/bash sudo apt-get -qq update # For installation sudo apt-get install -y make automake gcc libtool pkg-config texinfo help2man autopoint bison check gperf # For testing sudo apt-get install -y wbritish wamerican libfftw3-dev csh wget http://ftp.gnu.org/gnu/gettext/gettext-0.19.5.tar.xz tar Jxvf gettext-0.19.5.tar.xz cd gettext-0.19.5 && ./configure && make && sudo make install cd .. git clone --depth=1 -b madagascar-devel-2016 https://github.com/ahay/src.git madagascar cd madagascar && rm -rf trip sudo ./configure --prefix=/usr/local && sudo make && sudo make install cd .. mkdir nmrpipe && cd nmrpipe wget https://www.ibbr.umd.edu/nmrpipe/install.com wget https://www.ibbr.umd.edu/nmrpipe/binval.com wget https://www.ibbr.umd.edu/nmrpipe/NMRPipeX.tZ wget https://www.ibbr.umd.edu/nmrpipe/s.tZ wget https://www.ibbr.umd.edu/nmrpipe/dyn.tZ wget https://www.ibbr.umd.edu/nmrpipe/talos.tZ wget http://spin.niddk.nih.gov/bax/software/smile/plugin.smile.tZ chmod a+rx *.com && ./install.com sudo install nmrbin.linux212_64/var2pipe nmrbin.linux212_64/nmrPipe /usr/local/bin sudo install nmrbin.linux212_64/addNMR /usr/bin cd .. make config && make ================================================ FILE: .travis/macosx-xcode8.3.install.sh ================================================ #!/bin/bash set -x brew update brew install autoconf check xz texinfo help2man gettext libelf export PATH="/usr/local/opt/texinfo/bin:$PATH" brew link gettext --force #wget http://ftp.gnu.org/gnu/gettext/gettext-0.19.5.tar.xz #tar Jxvf gettext-0.19.5.tar.xz >/dev/null #cd gettext-0.19.5 && ./configure && make && sudo make install >/dev/null #cd .. #git clone --depth=1 -b madagascar-devel-2016 https://github.com/ahay/src.git madagascar #cd madagascar && rm -rf trip #sudo ./configure --prefix=/usr/local && sudo make && sudo make install #cd .. #mkdir nmrpipe && cd nmrpipe #wget https://www.ibbr.umd.edu/nmrpipe/install.com #wget https://www.ibbr.umd.edu/nmrpipe/binval.com #wget https://www.ibbr.umd.edu/nmrpipe/NMRPipeX.tZ #wget https://www.ibbr.umd.edu/nmrpipe/s.tZ #wget https://www.ibbr.umd.edu/nmrpipe/dyn.tZ #wget https://www.ibbr.umd.edu/nmrpipe/talos.tZ #wget http://spin.niddk.nih.gov/bax/software/smile/plugin.smile.tZ #chmod a+rx *.com && ./install.com >/dev/null #sudo install nmrbin.linux212_64/var2pipe nmrbin.linux212_64/nmrPipe /usr/local/bin >/dev/null #sudo install nmrbin.linux212_64/addNMR /usr/bin >/dev/null #cd .. make config && make ================================================ FILE: .travis.yml ================================================ matrix: include: - os: linux dist: trusty install: - ./.travis/linux-ubuntu-trusty.install.sh - os: osx osx_image: xcode8.3 install: - ./.travis/macosx-xcode8.3.install.sh language: C compiler: gcc script: make test && sudo make install ================================================ FILE: CITATION.cff ================================================ cff-version: 1.2.0 title: dgsh: The Directed Graph Shell message: >- If you use this software, please cite, using the metadata from this file, both the article from preferred-citation and the software itself. preferred-citation: date-released: "2017-09-01" doi: "10.1109/TC.2017.2695447" title: "Extending Unix Pipelines to DAGs" authors: - family-names: "Spinellis" given-names: "Diomidis" - family-names: "Fragkoulis" given-names: "Marios" type: "article" volume: 66 issue: 9 journal: "IEEE Transactions on Computers" start: 1547 end: 1561 type: software authors: - given-names: Diomidis family-names: Spinellis email: dds@aueb.gr affiliation: Athens University of Economics and Business orcid: 'https://orcid.org/0000-0003-4231-1897' - given-names: "Marios" family-names: "Fragkoulis" repository-code: 'https://github.com/dspinellis/dgsh' keywords: - Unix - shell - dataflow programming license: Apache-2.0 ================================================ FILE: LICENSE ================================================ Copyright 2012-2013 Diomidis Spinellis Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ------------------------------------------------------------------------ jquery.js is licensed as follows. Copyright 2013 jQuery Foundation and other contributors http://jquery.com/ Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ------------------------------------------------------------------------ dgsh-httpval is based on micro_httpd, which is licensed as follows. micro_httpd: Copyright (c) 1999,2005 by Jef Poskanzer . All rights reserved. Dgsh modifications: Copyright 2013 Diomidis Spinellis Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ================================================ FILE: Makefile ================================================ # # Copyright 2012-2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # -include .config export PREFIX?=/usr/local ifdef DEBUG CPPFLAGS=-DDEBUG CXXFLAGS=-g -Wall -O0 else CXXFLAGS=-O -Wall endif ifdef TIME CFLAGS+=-DTIME endif DOTFLAGS=-Nfontname=Arial -Ngradientangle=90 -Nstyle=filled -Nshape=ellipse -Nfillcolor=yellow:white # Manual pages MAN1SRC=$(wildcard core-tools/src/*.1) MANPDF=$(patsubst %.1,%.pdf,$(MAN1SRC)) core-tools/src/dgsh_negotiate.pdf MANHTML=$(patsubst %.1,%.html,$(MAN1SRC)) core-tools/src/dgsh_negotiate.html # Web files EXAMPLES=$(patsubst example/%,%,$(wildcard example/*.sh)) EGPNG=$(patsubst %.sh,png/%-pretty.png,$(EXAMPLES)) EGDOT=$(patsubst %.sh,graphdot/%.dot,$(EXAMPLES)) WEBPNG=$(EGPNG) WEBDIST=../../../pubs/web/home/sw/dgsh/ png/%-pretty.png: graphdot/%.dot gvpr 'BEG_G { graph_t L = cloneG($$G,"last")} END {write(L)}' $< | \ dot $(DOTFLAGS) -Tpng >$@ %.pdf: %.1 groff -man -Tps $< | ps2pdf - $@ %.pdf: %.3 groff -man -Tps $< | ps2pdf - $@ %.html: %.1 groff -man -Thtml $< >$@ %.html: %.3 groff -man -Thtml $< >$@ graphdot/%.dot: example/%.sh mkdir -p graphdot rm -f graphdot/$*.dot DGSH_DRAW_EXIT=1 DGSH_DOT_DRAW=graphdot/$* ./unix-tools/bash/bash --dgsh $< .PHONY: all tools core-tools unix-tools export-prefix \ config config-core-tools \ test test-dgsh test-merge-sum test-tee test-negotiate \ test-unix-tools test-wrap test-kvstore \ clean install webfiles dist pull commit uninstall dotfiles all: tools tools: core-tools unix-tools core-tools: $(MAKE) -C core-tools CFLAGS="$(CFLAGS)" cd core-tools/src && $(MAKE) build-install unix-tools: core-tools $(MAKE) -C unix-tools make MAKEFLAGS= $(MAKE) -C unix-tools build-install export-prefix: echo "export PREFIX?=$(PREFIX)" >.config config: export-prefix config-core-tools $(MAKE) -C unix-tools configure config-core-tools: core-tools/configure.ac core-tools/Makefile.am core-tools/src/Makefile.am core-tools/tests/Makefile.am -mkdir core-tools/m4 cd core-tools && \ autoreconf --install && \ ./configure --prefix=$(PREFIX) \ --bindir=$(PREFIX)/bin && \ cd tests && \ patch Makefile /) { system("$$1"); } else { print; }' web/index.html >$(WEBDIST)/index.html cp $(MANHTML) $(MANPDF) $(WEBDIST) cp $(WEBPNG) $(WEBDIST) # Obtain dot files from Unix host rsync-graphdot: ssh stereo 'cd src/dgsh && git pull && make webfiles' rsync -a stereo:src/dgsh/graphdot/ graphdot/ rsync -a stereo:src/dgsh/example/ example/ dotfiles: $(EGDOT) pull: git pull # Pull master on all sub-repositories. # Note that the gnulib ones get detached by by builds specifying # a specific gnulib version. Through this pull repos on master # stay on master; detached repos (gnulib) stay in the version they # were detached. git submodule status --recursive | awk '{print $$2}' | sort -r | while read d ; do ( echo "Pulling $$d" && cd $$d && old=$$(if [ $$(git rev-parse master) = $$(git rev-parse HEAD) ] ; then echo master ; else git rev-parse HEAD ; fi) && git checkout master && git pull && git checkout -q $$old ) ; done push: git push --recurse-submodules=on-demand commit: # Commit -a including submodules printf '\n\n# Please enter the commit message for your changes.\n#\n' >.git/COMMIT_EDITMSG git status --ignore-submodules=untracked | sed '/./s/^/# /;s/^$$/#/' >>.git/COMMIT_EDITMSG $${VISUAL-vi} .git/COMMIT_EDITMSG for i in $$(echo unix-tools/*/.git | sed 's/\.git//g') . ; do grep -v '^#' .git/COMMIT_EDITMSG | (cd $$i && git commit -a -F -) ; done rm -f .git/COMMIT_EDITMSG zip: cd .. && zip -r dgsh.zip dgsh -x *.git* # Rough uninstall rule to verify that tests pick up correct files uninstall: rm -rf $(PREFIX)/bin/dgsh-* $(PREFIX)/libexec/dgsh \ $(PREFIX)/lib/libdgsh.a ================================================ FILE: README.md ================================================ ## dgsh: The Directed Graph Shell [![Build Status](https://travis-ci.org/dspinellis/dgsh.svg?branch=master)](https://travis-ci.org/dspinellis/dgsh) The directed graph shell, *dgsh*, allows the expressive expression of efficient big data set and streams processing pipelines using existing Unix tools as well as custom-built components. It is a Unix-style shell allowing the specification of pipelines with non-linear scatter-gather operations. These form a directed acyclic process graph, which is typically executed by multiple processor cores, thus increasing the operation's processing throughput. You can find a complete introduction, reference documentation, and illustrated examples in the suite's [web site](http://www.spinellis.gr/sw/dgsh/). See also, a [quick video overview](https://youtu.be/crqzO4YanwA) and the associated (open access) paper, [Extending Unix pipelines to DAGs](http://dx.doi.org/10.1109/TC.2017.2695447), published in the *IEEE Transactions on Computers*, 66(9):1547–1561, 2017. ================================================ FILE: core-tools/.gitignore ================================================ Makefile Makefile.in aclocal.m4 autom4te.cache/ build-aux/ config.h config.h.in config.log config.status configure libtool m4/ stamp-h1 ================================================ FILE: core-tools/Makefile.am ================================================ ## Process this file with automake to produce Makefile.in ACLOCAL_AMFLAGS = -I m4 SUBDIRS = src . tests ================================================ FILE: core-tools/configure.ac ================================================ # Process this file with autoconf to produce a configure script. # Prelude. AC_PREREQ([2.59]) AC_INIT([DGSH Negotiate], [1.0], [check-devel AT lists.sourceforge.net]) # unique source file --- primitive safety check AC_CONFIG_SRCDIR([src/negotiate.c]) AC_CONFIG_MACRO_DIR([m4]) # place to put some extra build scripts installed AC_CONFIG_AUX_DIR([build-aux]) # fairly severe build strictness # change foreign to gnu or gnits to comply with gnu standards AM_INIT_AUTOMAKE([-Wall -Werror subdir-objects foreign 1.9.6]) # Checks for programs. AM_PROG_AR AM_PROG_AS AC_PROG_CC AC_PROG_LIBTOOL # Checks for libraries. # This macro is defined in check.m4 and tests if check.h and # libcheck.a are installed in your system. It sets CHECK_CFLAGS and # CHECK_LIBS accordingly. # AM_PATH_CHECK([MINIMUM-VERSION, # [ACTION-IF-FOUND [, ACTION-IF-NOT-FOUND]]]) PKG_CHECK_MODULES([CHECK], [check >= 0.9.4]) # Checks for header files. AC_HEADER_STDC AC_CHECK_HEADERS([stdlib.h]) # Checks for typedefs, structures, and compiler characteristics. # Checks for library functions. AC_FUNC_MALLOC AC_CHECK_FUNCS(cpow) # Output files AC_CONFIG_HEADERS([config.h]) AC_CONFIG_FILES([Makefile src/Makefile tests/Makefile]) AM_CONDITIONAL([LINUX], [test $(uname) = "Linux"]) AM_CONDITIONAL([DARWIN], [test $(uname) = "Darwin"]) AC_OUTPUT ================================================ FILE: core-tools/src/.gitignore ================================================ dgsh-conc dgsh-conc.html dgsh-enumerate dgsh-enumerate.html dgsh-fft-input dgsh.html dgsh-httpval dgsh-httpval.html dgsh-merge-sum dgsh-merge-sum.html dgsh-monitor dgsh-monitor.html dgsh_negotiate.html dgsh-parallel dgsh-parallel.html dgsh-pecho dgsh-ps dgsh-readval dgsh-readval.html dgsh-tee dgsh-tee.html dgsh-w dgsh-wrap dgsh-wrap.html dgsh-writeval dgsh-writeval.html libdgsh.a negotiate perm perm.html ================================================ FILE: core-tools/src/Makefile.am ================================================ include ../../.config if LINUX DGSH_ASSEMBLY_FILE=dgsh-elf.s else if DARWIN DGSH_ASSEMBLY_FILE=dgsh-macho.s endif endif lib_LIBRARIES = libdgsh.a libdgsh_a_SOURCES = negotiate.c $(DGSH_ASSEMBLY_FILE) include_HEADERS = dgsh.h bin_PROGRAMS = dgsh-monitor dgsh-httpval dgsh-readval bin_SCRIPTS = dgsh-merge-sum man1_MANS = dgsh.1 dgsh-conc.1 dgsh-enumerate.1 dgsh-httpval.1 \ dgsh-merge-sum.1 dgsh-monitor.1 \ dgsh-parallel.1 dgsh-readval.1 dgsh-tee.1 dgsh-wrap.1 \ dgsh-writeval.1 perm.1 man3_MANS = dgsh_negotiate.3 libexec_PROGRAMS = dgsh-tee dgsh-writeval dgsh-readval dgsh-monitor \ dgsh-conc dgsh-wrap dgsh-enumerate dgsh-pecho \ dgsh-fft-input dgsh-w libexec_SCRIPTS = dgsh-parallel perm libexecdir = $(prefix)/libexec/dgsh dgsh_monitor_SOURCES = dgsh-monitor.c dgsh_httpval_SOURCES = dgsh-httpval.c kvstore.c dgsh_readval_SOURCES = dgsh-readval.c kvstore.c dgsh_tee_SOURCES = dgsh-tee.c dgsh_writeval_SOURCES = dgsh-writeval.c dgsh_conc_SOURCES = dgsh-conc.c dgsh_wrap_SOURCES = dgsh-wrap.c dgsh_enumerate_SOURCES = dgsh-enumerate.c dgsh_pecho_SOURCES = dgsh-pecho.c dgsh_fft_input_SOURCES = dgsh-fft-input.c dgsh_w_SOURCES = dgsh-w.c $(CPOW) dgsh_readval_LDADD = libdgsh.a dgsh_writeval_LDADD = libdgsh.a dgsh_conc_LDADD = libdgsh.a dgsh_wrap_LDADD = libdgsh.a dgsh_tee_LDADD = libdgsh.a dgsh_enumerate_LDADD = libdgsh.a dgsh_pecho_LDADD = libdgsh.a dgsh_fft_input_LDADD = libdgsh.a dgsh_w_LDADD = libdgsh.a -lm dgsh-parallel: dgsh-parallel.sh install $? $@ perm: perm.sh install $? $@ dgsh-merge-sum: dgsh-merge-sum.pl install $? $@ clean-local: -rm -rf dgsh-parallel perm degsh-merge-sum build-install: mkdir -p ../../build/bin ../../build/libexec/dgsh cp $(bin_PROGRAMS) $(bin_SCRIPTS) ../../build/bin/ cp $(libexec_PROGRAMS) $(libexec_SCRIPTS) ../../build/libexec/dgsh/ ================================================ FILE: core-tools/src/debug.h ================================================ /* * Copyright 2013-2017 Diomidis Spinellis * * Debug macros * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ #ifndef DEBUG_H #define DEBUG_H #ifdef DEBUG /* ## is a gcc extension that removes trailing comma if no args */ #define DPRINTF(fmt, ...) fprintf(stderr, "%d: " fmt " \n", (int)getpid(), ##__VA_ARGS__) #else #define DPRINTF(fmt, ...) #endif #endif /* DEBUG_H */ ================================================ FILE: core-tools/src/dgsh-conc.1 ================================================ .TH DGSH-HTTPVAL 1 "14 July 2016" .\" .\" (C) Copyright 2016 Diomidis Spinellis. All rights reserved. .\" .\" Licensed under the Apache License, Version 2.0 (the "License"); .\" you may not use this file except in compliance with the License. .\" You may obtain a copy of the License at .\" .\" http://www.apache.org/licenses/LICENSE-2.0 .\" .\" Unless required by applicable law or agreed to in writing, software .\" distributed under the License is distributed on an "AS IS" BASIS, .\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .\" See the License for the specific language governing permissions and .\" limitations under the License. .\" .SH NAME dgsh-conc \- input or output pipe concentrator for dgsh negotiation .SH SYNOPSIS \fBdgsh-conc\fP \fB\-i\fP | \fB-o\fP \fInprog\fP .SH DESCRIPTION \fIdgsh-conc\fP is a helper program used in the \fIdgsh\fP negotiation phase. It is used to allow multiple output processes to negotiate with a single input process, or to allow a single output process to negotiate with multple input ones. Once the negotiation is complete, it passes around the generated pipe file descriptor and exits. The two obligatory arguments specify whether the command will act as an input or output concentrator, and the number of input or output programs to concentrate. .SH OPTIONS .IP "\fB\-i\fP Act as an input concentrator by concentrating multiple inputs to single output. .IP "\fB\-o\fP Act as an output concentrator by concentrating multiple outputs to a single input. .SH "SEE ALSO" \fIdgsh\fP(1), .SH AUTHOR Diomidis Spinellis \(em . ================================================ FILE: core-tools/src/dgsh-conc.c ================================================ /* * Copyright 2016, 2017 Diomidis Spinellis * * A passive component that aids the dgsh negotiation by passing * message blocks among participating processes. * When the negotiation is finished and the processes get connected by * pipes, it exits. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ #include #include #include #include #include #include #include #include #include /* getpid(), alarm() */ #include #include /* sig_atomic_t */ #include "negotiate.h" /* read/write_message_block(), set_negotiation_complete() */ #include "dgsh-debug.h" /* DPRINTF */ #define DGSH_TIMEOUT 5 /* Alarm mechanism and on_exit handling */ extern volatile sig_atomic_t negotiation_completed; #ifdef TIME #include static struct timespec tstart={0,0}, tend={0,0}; #endif static const char *program_name; static pid_t pid; static void usage(void) { fprintf(stderr, "Usage: %s -i|-o [-n] nprog\n" "-i" "\tInput concentrator: multiple inputs to single output\n" "-o" "\tOutput concentrator: single input to multiple outputs\n" "-n" "\tDo not consider standard input (used with -o)\n", program_name); exit(1); } /* * Information for each I/O file descriptor on which * the concentrator operates. */ static struct portinfo { pid_t pid; // The id of the process talking to this port bool seen; // True when the pid was seen bool written; // True when we wrote to pid bool run_ready; // True when the associated process can run struct dgsh_negotiation *to_write; // Block pending a write } *pi; /* * True when we're concentrating inputs, i.e. gathering 0, 3, 4, ... to 1 * Otherwise we scatter 0 to 1, 3, 4 ... */ STATIC bool multiple_inputs; /* True when input concentrator is the gather endpoint * of a scatter-first-gather-then block. * In this case using the default route of the input * concentrator results in complex cycles and ruins * the algorithms that decide when negotiation should * end both for concentrators and participating processes. * The favored scheme in this case is: * stdin -> stdout * stdout -> stdin * fd -> fd */ //STATIC bool pass_origin; STATIC bool noinput; /* * Total number of file descriptors on which the process performs I/O * (including stderr). The last fd used in nfd - 1. */ STATIC int nfd; #define FREE_FILENO (STDERR_FILENO + 1) /** * Return the next fd where a read block should be passed * Return whether we should restore the origin of the block */ STATIC int next_fd(int fd, bool *ro) { if (multiple_inputs) switch (fd) { case STDIN_FILENO: return STDOUT_FILENO; case STDOUT_FILENO: return STDIN_FILENO; default: *ro = true; return fd; } else switch (fd) { case STDIN_FILENO: if (!noinput) return STDOUT_FILENO; case STDOUT_FILENO: if (nfd > 2) { // if ==2, treat in default case if (!noinput) *ro = true; return FREE_FILENO; } default: if (fd == nfd - 1) if (!noinput) return STDIN_FILENO; else return STDOUT_FILENO; else { if (!noinput) *ro = true; return fd + 1; } } } /** * Return whether the process at port i * is ready to run. * Check whether this is the process whose * pid is set and prepare for exit. */ STATIC bool is_ready(int i, struct dgsh_negotiation *mb) { bool ready = false; if (pi[i].seen && pi[i].written) ready = true; DPRINTF(4, "pi[%d].pid: %d %s?: %d\n", i, pi[i].pid, __func__, ready); return ready; } /** * Register current concentrator to message block's * concentrator array */ STATIC int set_io_channels(struct dgsh_negotiation *mb) { if (find_conc(mb, pid)) return 0; struct dgsh_conc c; c.pid = pid; c.input_fds = -1; c.output_fds = -1; c.n_proc_pids = (nfd > 2 ? nfd - 2 : 1); c.multiple_inputs = multiple_inputs; c.proc_pids = (int *)malloc(sizeof(int) * c.n_proc_pids); int j = 0, i; DPRINTF(4, "%s: n_proc_pids: %d", __func__, c.n_proc_pids); if (multiple_inputs) { c.endpoint_pid = pi[STDOUT_FILENO].pid; if (c.endpoint_pid == 0) return 1; for (i = STDIN_FILENO; i < nfd; i == STDIN_FILENO ? i = FREE_FILENO : i++) if (pi[i].pid == 0) { free(c.proc_pids); return 1; } else c.proc_pids[j++] = pi[i].pid; } else { bool ignore; c.endpoint_pid = pi[STDIN_FILENO].pid; if (c.endpoint_pid == 0) return 1; for (i = STDOUT_FILENO; i != STDIN_FILENO; i = next_fd(i, &ignore)) if (pi[i].pid == 0) { free(c.proc_pids); return 1; } else c.proc_pids[j++] = pi[i].pid; } if (!mb->conc_array) { mb->conc_array = (struct dgsh_conc *)malloc(sizeof(struct dgsh_conc)); mb->n_concs = 1; } else { mb->n_concs++; mb->conc_array = (struct dgsh_conc *)realloc(mb->conc_array, sizeof(struct dgsh_conc) * mb->n_concs); } memcpy(&mb->conc_array[mb->n_concs - 1], &c, sizeof(struct dgsh_conc)); DPRINTF(4, "%s(): Added conc with pid: %d, now n_concs: %d", __func__, mb->conc_array[mb->n_concs - 1].pid, mb->n_concs); return 0; } STATIC void print_state(int i, int var, int pcase) { switch (pcase) { case 1: DPRINTF(4, "%s(): pi[%d].pid: %d", __func__, i, (int)pi[i].pid); DPRINTF(4, " initiator pid: %d", var); DPRINTF(4, " pi[%d].seen: %d", i, pi[i].seen); DPRINTF(4, " write: %d", pi[i].written); case 2: DPRINTF(4, "%s(): pi[%d].pid: %d", __func__, i, pi[i].pid); DPRINTF(4, " run ready?: %d, seen times: %d", (int)pi[i].run_ready, pi[i].seen); DPRINTF(4, " written: %d, nfds: %d", pi[i].written, var); } } #define max(a, b) ((a) > (b) ? (a) : (b)) /* * Pass around the message blocks so that they reach all processes * connected through the concentrator. */ STATIC int pass_message_blocks(void) { fd_set readfds, writefds; int nfds = 0; int i; int oi = -1; /* scatter/gather block's origin index */ int ofd = -1; /* ... origin fd direction */ bool ro = false; /* Whether the read block's origin should * be restored */ bool iswrite = false; if (noinput) { #ifdef TIME clock_gettime(CLOCK_MONOTONIC, &tstart); #endif construct_message_block("dgsh-conc", pid); chosen_mb->origin_fd_direction = STDOUT_FILENO; chosen_mb->is_origin_conc = true; chosen_mb->conc_pid = pid; pi[STDOUT_FILENO].to_write = chosen_mb; } for (;;) { // Create select(2) masks FD_ZERO(&readfds); FD_ZERO(&writefds); for (i = 0; i < nfd; i++) { if (noinput && i == STDIN_FILENO) continue; if (i == STDERR_FILENO) continue; if (!pi[i].seen) { FD_SET(i, &readfds); nfds = max(i + 1, nfds); } if (pi[i].to_write && !pi[i].written) { FD_SET(i, &writefds); nfds = max(i + 1, nfds); pi[i].to_write->is_origin_conc = true; pi[i].to_write->conc_pid = pid; DPRINTF(4, "Actual origin: conc with pid %d", pid); } } again: if (select(nfds, &readfds, &writefds, NULL, NULL) < 0) { if (errno == EINTR) goto again; /* All other cases are internal errors. */ err(1, "select"); } // Read/write what we can for (i = 0; i < nfd; i++) { if (FD_ISSET(i, &writefds)) { iswrite = true; assert(pi[i].to_write); chosen_mb = pi[i].to_write; DPRINTF(4, "**fd i: %d set for writing to tool with pid %d", i, pi[i].pid); write_message_block(i); // XXX check return if (pi[i].to_write->state == PS_RUN || pi[i].to_write->state == PS_DRAW_EXIT || (pi[i].to_write->state == PS_ERROR && pi[i].to_write->is_error_confirmed)) pi[i].written = true; // Write side exit if (is_ready(i, pi[i].to_write)) { pi[i].run_ready = true; DPRINTF(4, "**%s(): pi[%d] is run ready", __func__, i); } pi[i].to_write = NULL; } if (FD_ISSET(i, &readfds)) { struct dgsh_negotiation *rb; ro = false; int next = next_fd(i, &ro); assert(!pi[i].run_ready); assert(pi[next].to_write == NULL); if (read_message_block(i, &pi[next].to_write) == OP_ERROR) { chosen_mb->state = PS_ERROR; if (noinput) chosen_mb->is_error_confirmed = true; pi[next].to_write = chosen_mb; continue; } rb = pi[next].to_write; DPRINTF(4, "%s(): next write via fd %d to pid %d", __func__, next, pi[next].pid); if (oi == -1) { if ((multiple_inputs && i == 1) || (!multiple_inputs && i == 0)) { oi = rb->origin_index; ofd = rb->origin_fd_direction; DPRINTF(4, "**Store origin: %d, fd: %s", oi, ofd ? "stdout" : "stdin"); } } /* If conc talks to conc, set conc's pid * Required in order to allocate fds correctly * in the end */ if (rb->is_origin_conc) pi[i].pid = rb->conc_pid; else pi[i].pid = get_origin_pid(rb); /* If needed, re-set origin. * Don't move this block before get_origin_pid() */ if (ro) { DPRINTF(4, "**Restore origin: %d, fd: %s", oi, ofd ? "stdout" : "stdin"); pi[next].to_write->origin_index = oi; pi[next].to_write->origin_fd_direction = ofd; } else if (noinput) { pi[next].to_write->origin_index = -1; pi[next].to_write->origin_fd_direction = STDOUT_FILENO; } /* Set a conc's required/provided IO in mb */ if (!noinput) set_io_channels(pi[next].to_write); if (rb->state == PS_NEGOTIATION && noinput) { int j, seen = 0; pi[i].seen = true; for (j = 1; j < nfd; j++) if (pi[j].seen) seen++; if ((nfd > 2 && seen == nfd - 2) || seen == nfd - 1) { chosen_mb = rb; DPRINTF(1, "%s(): Gathered I/O requirements.", __func__); int state = solve_graph(); if (state == OP_ERROR) { pi[next].to_write->state = PS_ERROR; pi[next].to_write->is_error_confirmed = true; } else if (state == OP_DRAW_EXIT) pi[next].to_write->state = PS_DRAW_EXIT; else { DPRINTF(1, "%s(): Computed solution", __func__); pi[next].to_write->state = PS_RUN; } for (j = 1; j < nfd; j++) pi[j].seen = false; // Don't free chosen_mb = NULL; } } else if (rb->state == PS_RUN || rb->state == PS_DRAW_EXIT || (rb->state == PS_ERROR && rb->is_error_confirmed)) pi[i].seen = true; else if (rb->state == PS_ERROR) rb->is_error_confirmed = true; print_state(i, (int)rb->initiator_pid, 1); if (pi[i].seen && pi[i].written) { chosen_mb = pi[next].to_write; pi[i].run_ready = true; DPRINTF(4, "**%s(): pi[%d] is run ready", __func__, i); } } } // See if all processes are run-ready nfds = 0; for (i = 0; i < nfd; i++) { if (pi[i].run_ready) nfds++; print_state(i, nfds, 2); } if ((nfd > 2 && (nfds == nfd - 1 || (noinput && nfds == nfd - 2))) || (nfds == nfd || (noinput && nfds == nfd - 1))) { assert(chosen_mb != NULL); DPRINTF(4, "%s(): conc leaves negotiation", __func__); return chosen_mb->state; } else if (chosen_mb != NULL && iswrite) { // Free if we have written DPRINTF(4, "chosen_mb: %lx, i: %d, next: %d, pi[next].to_write: %lx\n", (long)chosen_mb, i, next_fd(i, &ro), (long)pi[next_fd(i, &ro)].to_write); free_mb(chosen_mb); chosen_mb = NULL; iswrite = false; } } } /* * Scatter the fds read from the input process to multiple outputs. */ STATIC void scatter_input_fds(struct dgsh_negotiation *mb) { struct dgsh_conc *this_conc = find_conc(mb, pid); if (!this_conc) { printf("%s(): Concentrator with pid %d not registered", __func__, pid); exit(1); // XXX } int n_to_read = this_conc->input_fds; int *read_fds = (int *)malloc(n_to_read * sizeof(int)); int i, j, write_index = 0; bool ignore = false; DPRINTF(4, "%s(): fds to read: %d", __func__, n_to_read); for (i = 0; i < n_to_read; i++) read_fds[i] = read_fd(STDIN_FILENO); for (i = STDOUT_FILENO; i != STDIN_FILENO; i = next_fd(i, &ignore)) { int n_to_write = get_expected_fds_n(mb, pi[i].pid); DPRINTF(4, "%s(): fds to write for p[%d].pid %d: %d", __func__, i, pi[i].pid, n_to_write); for (j = write_index; j < write_index + n_to_write; j++) { write_fd(i, read_fds[j]); DPRINTF(4, "%s(): Write fd: %d to output channel: %d", __func__, read_fds[j], i); } write_index += n_to_write; } assert(write_index == n_to_read); } /* * Gather the fds read from input processes to a single output. */ STATIC void gather_input_fds(struct dgsh_negotiation *mb) { struct dgsh_conc *this_conc = find_conc(mb, pid); if (!this_conc) { printf("%s(): Concentrator with pid %d not registered", __func__, pid); exit(1); // XXX } int n_to_write = this_conc->output_fds; int *read_fds = (int *)malloc(n_to_write * sizeof(int)); int i, j, read_index; DPRINTF(4, "%s(): fds to write: %d", __func__, n_to_write); read_index = 0; for (i = STDIN_FILENO; i < nfd; i == STDIN_FILENO ? i = FREE_FILENO : i++) { int n_to_read = get_provided_fds_n(mb, pi[i].pid); DPRINTF(4, "%s(): fds to read for p[%d].pid %d: %d", __func__, i, pi[i].pid, n_to_read); for (j = read_index; j < read_index + n_to_read; j++) { read_fds[j] = read_fd(i); DPRINTF(4, "%s(): Read fd: %d from input channel: %d", __func__, read_fds[j], i); } read_index += n_to_read; } assert(read_index == n_to_write); for (i = 0; i < n_to_write; i++) write_fd(STDOUT_FILENO, read_fds[i]); } #ifndef UNIT_TESTING int main(int argc, char *argv[]) { int ch; int exit; char *debug_level = NULL; char *timeout; program_name = argv[0]; pid = getpid(); noinput = false; while ((ch = getopt(argc, argv, "ion")) != -1) { switch (ch) { case 'i': multiple_inputs = true; break; case 'o': multiple_inputs = false; break; case 'n': // special output conc that takes no input if (!multiple_inputs) noinput = true; else usage(); break; case '?': default: usage(); } } argc -= optind; argv += optind; if (argc != 1) usage(); debug_level = getenv("DGSH_DEBUG_LEVEL"); if (debug_level != NULL) dgsh_debug_level = atoi(debug_level); signal(SIGALRM, dgsh_alarm_handler); if ((timeout = getenv("DGSH_TIMEOUT")) != NULL) alarm(atoi(timeout)); else alarm(DGSH_TIMEOUT); /* +1 for stdin when scatter/stdout when gather * +1 for stderr which is not used */ if (atoi(argv[0]) == 1) nfd = 2; else nfd = atoi(argv[0]) + 2; pi = (struct portinfo *)calloc(nfd, sizeof(struct portinfo)); chosen_mb = NULL; exit = pass_message_blocks(); if (exit == PS_RUN) { if (noinput) DPRINTF(1, "%s(): Special (no-input) conc communicated the solution", __func__); if (multiple_inputs) gather_input_fds(chosen_mb); else if (!noinput) // Output noinput conc has no job here scatter_input_fds(chosen_mb); exit = PS_COMPLETE; } free_mb(chosen_mb); free(pi); DPRINTF(3, "conc with pid %d terminates %s", pid, exit == PS_COMPLETE ? "normally" : "with error"); #ifdef DEBUG fflush(stderr); #endif #ifdef TIME if (noinput) { clock_gettime(CLOCK_MONOTONIC, &tend); fprintf(stderr, "The dgsh negotiation procedure took about %.5f seconds\n", ((double)tend.tv_sec + 1.0e-9*tend.tv_nsec) - ((double)tstart.tv_sec + 1.0e-9*tstart.tv_nsec)); fflush(stderr); } #endif set_negotiation_complete(); alarm(0); // Cancel alarm signal(SIGALRM, SIG_IGN); // Do not handle the signal return exit; } #endif ================================================ FILE: core-tools/src/dgsh-debug.h ================================================ /* * Copyright 2013-2017 Diomidis Spinellis * * Debug macros * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ #ifndef DEBUG_H #define DEBUG_H extern int dgsh_debug_level; /* ## is a gcc extension that removes trailing comma if no args */ #define DPRINTF(debug_level, fmt, ...) ((debug_level) <= dgsh_debug_level ? fprintf(stderr, "%d: " fmt "\n", (int)getpid(), ##__VA_ARGS__) : 0) #endif /* DEBUG_H */ ================================================ FILE: core-tools/src/dgsh-elf.s ================================================ /* * ELF note header to mark dgsh-compatible programs * See http://www.netbsd.org/docs/kernel/elf-notes.html * Don't use line comments as these are not portable between * different CPU architectures. * https://en.wikipedia.org/wiki/GNU_Assembler#Single-Line_comments */ .comm dgsh_force_include,4,4 .section ".note.ident", "a" .p2align 2 .long 1f - 0f /* name size (not including padding) */ .long 3f - 2f /* desc size (not including padding) */ .long 1 /* type */ 0: .asciz "DSpinellis/dgsh" /* name */ 1: .p2align 2 2: .long 0x00000001 /* desc */ .long 0x00000000 3: .p2align 2 ================================================ FILE: core-tools/src/dgsh-enumerate.1 ================================================ .TH DGSH-ENUMERATE 1 "27 January 2017" .\" .\" (C) Copyright 2017 Diomidis Spinellis. All rights reserved. .\" .\" Licensed under the Apache License, Version 2.0 (the "License"); .\" you may not use this file except in compliance with the License. .\" You may obtain a copy of the License at .\" .\" http://www.apache.org/licenses/LICENSE-2.0 .\" .\" Unless required by applicable law or agreed to in writing, software .\" distributed under the License is distributed on an "AS IS" BASIS, .\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .\" See the License for the specific language governing permissions and .\" limitations under the License. .\" .SH NAME dgsh-enumerate \- enumerate an arbitrary number of output channels .SH SYNOPSIS \fBdgsh-enumerate\fP [\fIn\fP] .SH DESCRIPTION \fIdgsh-enumerate\fP will output a single newline-terminated ascending integer on each one of its output channels. If the number of channels is not specified, the command will allow its downstream processes to specify the number and use that one. .PP The command demonstrates the \fIdgsh\fP negotiation API. It can also be used as a debug tool. .SH EXAMPLES .PP Enumerate the specified four output streams. .ft C .ps -1 .nf $ dgsh -c 'dgsh-enumerate 4 | cat' 0 1 2 3 .fi .ps +1 .ft P .PP Enumerate the two output streams required by the downstream multipipe block. .ft C .ps -1 .nf $ dgsh -c 'dgsh-enumerate | {{ sed "s/^/A /" & sed "s/^/B /" & }} | cat' A 0 B 1 .fi .ps +1 .ft P .SH "SEE ALSO" .IR dgsh (1), .IR dgsh_negotiate (3). .SH AUTHOR Diomidis Spinellis \(em ================================================ FILE: core-tools/src/dgsh-enumerate.c ================================================ /* * Copyright 2017 Diomidis Spinellis * * Enumerate an arbitrary number of output channels. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ #include #include #include #include #include #include #include "dgsh.h" int main(int argc, char *argv[]) { int n_input_fds = 0, n_output_fds; int *output_fds; int i; switch (argc) { case 1: n_output_fds = -1; break; case 2: n_output_fds = atoi(argv[1]); break; default: errx(1, "usage: %s [n]", argv[0]); } dgsh_negotiate(DGSH_HANDLE_ERROR, argv[0], &n_input_fds, &n_output_fds, NULL, &output_fds); for (i = 0; i < n_output_fds; i++) { char buff[10]; snprintf(buff, sizeof(buff), "%d\n", i); write(output_fds[i], buff, strlen(buff)); close(output_fds[i]); } return 0; } ================================================ FILE: core-tools/src/dgsh-fft-input.c ================================================ #include // assert() #include // printf #include // double complex #include // read(), write() #include // free() #include // errx() #include "dgsh.h" #include "debug.h" int main(int argc, char **argv) { char *input_file; FILE *f; int ninput = 4, nlines = 0, i; int ninputfds = 0, noutputfds; int *inputfds = NULL, *outputfds = NULL; size_t len = sizeof(long double), wsize; char line[len + 1]; long double *input = (long double *)malloc(sizeof(long double) * ninput); if (argc == 1) { noutputfds = 8; goto negotiate; } input_file = argv[1]; f = fopen(input_file, "r"); if (!f) errx(2, "Open file %s failed", input_file); DPRINTF(4, "Opened input file: %s", input_file); while (fgets(line, len, f)) { assert(len == sizeof(input[nlines - 1])); nlines++; if (nlines == ninput) { ninput *= 2; input = (long double *)realloc(input, sizeof(long double) * ninput); if (!input) errx(2, "Realloc for input numbers failed"); } input[nlines - 1] = atof(line); DPRINTF(4, "Retrieved input %.10Lf\n", input[nlines - 1]); } noutputfds = nlines; negotiate: dgsh_negotiate(DGSH_HANDLE_ERROR, "fft-input", &ninputfds, &noutputfds, &inputfds, &outputfds); DPRINTF(4, "Read %d inputs, received %d fds", nlines, noutputfds); assert(ninputfds == 0); assert(noutputfds == nlines); for (i = 0; i < noutputfds; i++) { DPRINTF(4, "Write input %.10Lf to fd %d", input[i], outputfds[i]); wsize = write(outputfds[i], &input[i], sizeof(long double)); if (wsize == -1) err(1, "write failed"); } fclose(f); free(input); return 0; } ================================================ FILE: core-tools/src/dgsh-httpval.1 ================================================ .TH DGSH-HTTPVAL 1 "14 July 2013" .\" .\" (C) Copyright 2013 Diomidis Spinellis. All rights reserved. .\" .\" Licensed under the Apache License, Version 2.0 (the "License"); .\" you may not use this file except in compliance with the License. .\" You may obtain a copy of the License at .\" .\" http://www.apache.org/licenses/LICENSE-2.0 .\" .\" Unless required by applicable law or agreed to in writing, software .\" distributed under the License is distributed on an "AS IS" BASIS, .\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .\" See the License for the specific language governing permissions and .\" limitations under the License. .\" .SH NAME dgsh-httpval \- data store HTTP server .SH SYNOPSIS \fBdgsh-httpval\fP [\fB\-a\fP] [\fB\-b\fP \fIquery:command\fP] [\fB\-m\fP \fIMIME-type\fP] [\fB\-n\fP] [\fB\-p\fP \fIport\fP] .SH DESCRIPTION \fIdgsh-httpval\fP allows other programs to access \fIdgsh\fP data stores through the HTTP protocol. This simplifies the interfacing between web-based front-ends and \fIdgsh\fP programs. When \fIdgsh-httpval\fP receives a REST request with the name of a data store whose endpoint is located in the directory where \fIdgsh-httpval\fP was launched (e.g. \fChttp://localhost:8081/mystore\fP), it will establish a connection with the store specified in the request, send a command to read the store's value, obtain the value, and respond with it as the document sent with the HTTP response. .PP Requests for files located in the directory where \fIdgsh-httpval\fP was launched will also be satisfied. The correct MIME type will be sent for files with a suffix of \fChtml\fP, \fCjs\fP, \fCjson\fP, \fCpng\fP, and \fCcss\fP. .PP A request for the resource \fC.server?quit\fP, will cause the server to terminate processing and exit. .PP \fIdgsh-httpval\fP is normally executed from within \fIdgsh\fP-generated scripts, rather than through end-user commands. This manual page serves mainly to document its operation and the flags that can be passed to \fIdgsh\fP for modifying its behavior. .SH OPTIONS .IP "\fB\-a\fP Allow any Internet host to obtain a value from the server. By default the server will only respond to requests arriving from the local host's loop-back IP address (127.0.0.1). .IP "\fB\-b\fP \fIquery:command\fP" The colon-separated pair specifies a dynamic query than can be sent to the server, so that it will execute the specified command and return its output. The query and the command can contain up to ten matching \fIscanf(3)\fP and \fIprintf(3)\fP specifications for C integer-sized arguments, which can be used to pass data from the query to the command. An unlimited number of dynamic queries can be specified through multiple .B -b options. The type of the data returned is specified using the .B -m option. .IP "\fB\-m\fP \fIMIME-type\fP" Specify the MIME-type that the server will provide on the \fCContent-type\fP HTTP header for data coming from data stores and dynamic queries. By default this value is \fCtext/plain\fP. Other reasonable types are \fCapplication/json\fP, \fCtext/CSV\fP, \fCtext/xml\fP, or \fCapplication/octet-stream\fP. .IP "\fB\-n\fP Read values from stores using a non-blocking read command. This means that the server will return an empty record, if no complete record is available. .IP "\fB\-p\fP \fIport\fP" Specify the TCP port on which the server will listen for incoming HTTP requests. If no port is specified, then the server will listen on an arbitrary, system-assigned, port, and will print that port's number on its standard output. That value can be conveniently piped into \fIdgsh-writeval\fP to be made available to other processes. .SH EXAMPLES .PP Specify that a query, such as \fChttp://localhost:63001/server-bin/pstatus?id=4892\fP, will run the \fIps(1)\fP command for the specified process-id. .ft C .nf dgsh-httpval -b 'server-bin/pstatus?id=%d:ps -p %d' .ft P .fi .SH "SEE ALSO" \fIdgsh\fP(1), \fIdgsh-writeval\fP(1), \fIdgsh-readval\fP(1) .SH BUGS The server is single-threaded and will block if a value is not available on a specified store. .PP The server only supports IPv4 and the HTTP 1.0 protocols. Some clients may require special configuration to connect to it. For instance, \fIcurl\fP(1) requires the specification of the \fC--ipv4\fP and \fC--http1.0\fP flags. .SH AUTHOR Diomidis Spinellis \(em . Jef Poskanzer \(em \(em wrote micro_httpd on which this server is based. .SH BUGS The possibilities for mallicious attacks through code injection and buffer overflows offered by the dynamic query option are too numerous to list. Use this feature only in setups where you restrict and control what is being sent to the server. ================================================ FILE: core-tools/src/dgsh-httpval.c ================================================ /*- * * Provide HTTP access to the dgsh key-value store. * * Based on micro_httpd - really small HTTP server heavily modified by * Diomidis Spinellis to use IP sockets, instead of depending on inetd, * and to serve dgsh key-value store data, instead of files. * * micro_httpd: * Copyright (c) 1999,2005 by Jef Poskanzer . * All rights reserved. * * Dgsh modifications: * Copyright 2013 Diomidis Spinellis * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "kvstore.h" #define SERVER_NAME "dgsh-httpval" #define SERVER_URL "http://www.spinellis.gr/sw/dgsh" #define PROTOCOL "HTTP/1.0" #define RFC1123FMT "%a, %d %b %Y %H:%M:%S GMT" /* Forwards. */ static void send_error(FILE *out, int status, char *title, char *extra_header, char *text); static void send_headers(FILE *out, int status, char *title, char *extra_header, const char *mime_type, off_t length, time_t mod); static char * get_mime_type(char *name); static void strdecode(char *to, char *from); static int hexit(char c); static void http_serve(FILE *in, FILE *out, const char *mime_type); #define c_isxdigit(x) isxdigit((unsigned char)(x)) static const char *program_name; static void usage(void) { fprintf(stderr, "Usage: %s [-a] [-b query:cmd] [-p port]\n" "-a\t" "\tAllow non-localhost access\n" "-b query:cmd" "\tSpecify a command for a given HTTP query\n" "-m MIME-type" "\tSpecify the store Content-type header value\n" "-n" "\tNon-blocking read from stores\n" "-p port" "\tSpecify the port to listen to\n", program_name); exit(1); } static struct query { const char *query; const char *cmd; int narg; struct query *next; } *query_list; /* Command to read from stores: blocking read current record */ static char read_cmd = 'C'; int main(int argc, char *argv[]) { int sockfd, newsockfd; struct sockaddr_in cli_addr, serv_addr; int ch, port = 0; bool localhost_access = true; const char *mime_type = "text/plain"; int so_reuseaddr = 1; struct linger so_linger; program_name = argv[0]; while ((ch = getopt(argc, argv, "ab:m:np:")) != -1) { char *p; struct query *q; switch (ch) { case 'a': localhost_access = false; break; case 'b': if ((p = strchr(optarg, ':')) == NULL) usage(); /* Save query */ *p = 0; q = malloc(sizeof(struct query)); q->query = strdup(optarg); q->cmd = strdup(p + 1); /* Count number of query arguments */ q->narg = 0; for (p = optarg; *p; p++) if (*p == '%') { if (p[1] == '%') { p++; continue; } q->narg++; } if (q->narg > 10) { fprintf(stderr, "%s: More than ten query arguments specified.\n", program_name); exit(1); } /* Insert query into the linked list */ q->next = query_list; query_list = q; break; case 'm': mime_type = optarg; break; case 'n': /* Non-blocking read */ read_cmd = 'c'; break; case 'p': port = atoi(optarg); break; case '?': default: usage(); } } argc -= optind; argv += optind; if (argc != 0) usage(); if ((sockfd = socket(AF_INET, SOCK_STREAM, 0)) < 0) err(2, "socket"); if (setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &so_reuseaddr, sizeof (so_reuseaddr)) < 0) err(2, "setsockopt SO_REUSEADDR"); so_linger.l_onoff = 1; so_linger.l_linger = 1; if (setsockopt(sockfd, SOL_SOCKET, SO_LINGER, &so_linger, sizeof (so_linger)) < 0) err(2, "setsockopt SO_LINGER"); memset((char *)&serv_addr, 0, sizeof(serv_addr)); serv_addr.sin_family = AF_INET; serv_addr.sin_addr.s_addr = htonl(INADDR_ANY); serv_addr.sin_port = htons(port); if (bind(sockfd, (struct sockaddr *)&serv_addr, sizeof(serv_addr)) < 0) err(2, "bind"); if (port == 0) { socklen_t len = sizeof(serv_addr); serv_addr.sin_port = 0; if (getsockname(sockfd, (struct sockaddr *)&serv_addr, &len) < 0) err(2, "getsockname"); printf("%d\n", ntohs(serv_addr.sin_port)); fflush(stdout); } listen(sockfd, 5); for (;;) { socklen_t cli_len = sizeof(cli_addr); FILE *in, *out; if ((newsockfd = accept(sockfd, (struct sockaddr *)&cli_addr, &cli_len)) < 0) err(2, "accept"); if (localhost_access && memcmp(inet_ntoa(cli_addr.sin_addr), "127.", 4)) { warnx("Non-localhost access: %s", inet_ntoa(cli_addr.sin_addr)); close(newsockfd); continue; } if ((in = fdopen(newsockfd, "r")) == NULL) err(2, "fdopen for input"); setvbuf(in, NULL, _IOLBF, 4096); /* Must dup(2) so that fclose will work correctly */ if ((out = fdopen(dup(newsockfd), "w")) == NULL) err(2, "fdopen for output"); http_serve(in, out, mime_type); (void)fclose(in); (void)fclose(out); } } /* Serve a single HTTP request */ static void http_serve(FILE *in, FILE *out, const char *mime_type) { char line[10000], method[10000], path[10000], protocol[10000]; char *file; size_t len; struct stat sb; struct query *q; if (fgets(line, sizeof(line), in) == (char *) 0) { send_error(out, 400, "Bad Request", (char *) 0, "No request found."); return; } if (sscanf(line, "%[^ ] %[^ ] %[^ ]", method, path, protocol) != 3) { send_error(out, 400, "Bad Request", (char *) 0, "Can't parse request."); return; } while (fgets(line, sizeof(line), in) != (char *) 0) { if (strcmp(line, "\n") == 0 || strcmp(line, "\r\n") == 0) break; } if (strcasecmp(method, "get") != 0) { send_error(out, 501, "Not Implemented", (char *) 0, "That method is not implemented."); return; } if (path[0] != '/') { send_error(out, 400, "Bad Request", (char *) 0, "Bad filename."); return; } file = &(path[1]); strdecode(file, file); if (strcmp(file, ".server?quit") == 0) { send_error(out, 200, "OK", (char *) 0, "Quitting."); exit(0); } len = strlen(file); /* Guard against attempts to move outside our directory */ if (file[0] == '/' || strcmp(file, "..") == 0 || strncmp(file, "../", 3) == 0 || strstr(file, "/../") != (char *) 0 || strcmp(&(file[len - 3]), "/..") == 0) { send_error(out, 400, "Bad Request", (char *) 0, "Illegal filename."); return; } /* User-specified query name space */ for (q = query_list; q; q = q->next) { int v0, v1, v2, v3, v4, v5, v6, v7, v8, v9; char cmd[11000]; *cmd = 0; if (q->narg == 0 && strcmp(file, q->query) == 0) strncpy(cmd, q->cmd, sizeof(cmd)); else if (q->narg && sscanf(file, q->query, &v0, &v1, &v2, &v3, &v4, &v5, &v6, &v7, &v8, &v9) == q->narg) snprintf(cmd, sizeof(cmd), q->cmd, v0, v1, v2, v3, v4, v5, v6, v7, v8, v9); if (*cmd) { int ich; FILE *fp; fp = popen(cmd, "r"); if (fp == NULL) { send_error(out, 502, "Bad Gateway", NULL, "Error in executing command."); return; } send_headers(out, 200, "Ok", NULL, mime_type, -1, time(NULL)); while ((ich = getc(fp)) != EOF) putc(ich, out); fclose(fp); return; } } /* File system name space */ if (stat(file, &sb) < 0) { send_error(out, 404, "Not Found", NULL, strerror(errno)); return; } if (S_ISSOCK(sb.st_mode)) { /* Value store */ send_headers(out, 200, "Ok", NULL, mime_type, -1, (time_t)-1); (void)fflush(out); dgsh_send_command(file, read_cmd, true, false, fileno(out)); } else if (S_ISREG(sb.st_mode)) { /* Regular file */ int ich; FILE *fp; fp = fopen(file, "r"); if (fp == NULL) { send_error(out, 403, "Forbidden", NULL, "File is protected."); return; } send_headers(out, 200, "Ok", NULL, get_mime_type(file), sb.st_size, sb.st_mtime); while ((ich = getc(fp)) != EOF) putc(ich, out); fclose(fp); } else { send_error(out, 403, "Forbidden", (char *) 0, "File is not a regular file or a Unix domain socket."); } } static void send_error(FILE *out, int status, char *title, char *extra_header, char *text) { send_headers(out, status, title, extra_header, "text/html", -1, -1); (void)fprintf(out, "%d %s\n

%d %s

\n", status, title, status, title); (void)fprintf(out, "%s\n", text); (void)fprintf(out, "
\n
%s
\n\n", SERVER_URL, SERVER_NAME); (void)fflush(out); } static void send_headers(FILE *out, int status, char *title, char *extra_header, const char *mime_type, off_t length, time_t mod) { time_t now; char timebuf[100]; (void)fprintf(out, "%s %d %s\015\012", PROTOCOL, status, title); (void)fprintf(out, "Server: %s\015\012", SERVER_NAME); now = time((time_t *) 0); (void)strftime(timebuf, sizeof(timebuf), RFC1123FMT, gmtime(&now)); (void)fprintf(out, "Date: %s\015\012", timebuf); if (extra_header != (char *) 0) (void)fprintf(out, "%s\015\012", extra_header); if (mime_type != (char *) 0) (void)fprintf(out, "Content-Type: %s\015\012", mime_type); if (length >= 0) #if __STDC_VERSION__ >= 199901L (void)fprintf(out, "Content-Length: %jd\015\012", (intmax_t) length); #else (void)fprintf(out, "Content-Length: %lld\015\012", (long long) length); #endif if (mod != (time_t) - 1) { (void)strftime(timebuf, sizeof(timebuf), RFC1123FMT, gmtime(&mod)); (void)fprintf(out, "Last-Modified: %s\015\012", timebuf); } (void)fprintf(out, "Connection: close\015\012"); (void)fprintf(out, "\015\012"); } static char * get_mime_type(char *name) { char *dot; dot = strrchr(name, '.'); if (dot == NULL) return "text/plain"; if (strcmp(dot, ".json") == 0) return "application/json"; if (strcmp(dot, ".html") == 0) return "text/html"; if (strcmp(dot, ".js") == 0) return "text/javascript"; if (strcmp(dot, ".png") == 0) return "image/png"; if (strcmp(dot, ".css") == 0) return "text/css"; return "text/plain; charset=iso-8859-1"; } static void strdecode(char *to, char *from) { for (; *from != '\0'; ++to, ++from) { if (from[0] == '%' && c_isxdigit(from[1]) && c_isxdigit(from[2])) { *to = hexit(from[1]) * 16 + hexit(from[2]); from += 2; } else *to = *from; } *to = '\0'; } static int hexit(char c) { if (c >= '0' && c <= '9') return c - '0'; if (c >= 'a' && c <= 'f') return c - 'a' + 10; if (c >= 'A' && c <= 'F') return c - 'A' + 10; /* Shouldn't happen, we're guarded by isxdigit() */ return 0; } ================================================ FILE: core-tools/src/dgsh-macho.s ================================================ # MACHO note header to mark dgsh-compatible programs .section ".note.ident", "a" .asciz "DSpinellis/dgsh" .section __TEXT,__text,regular,pure_instructions .macosx_version_min 10, 13 .globl _dgsh_force_include ## @dgsh_force_include .zerofill __DATA,__common,_dgsh_force_include,4,2 .subsections_via_symbols ================================================ FILE: core-tools/src/dgsh-merge-sum.1 ================================================ .TH DGSH-MERGE-SUM 1 "10 September 2014" .\" .\" (C) Copyright 2014 Diomidis Spinellis. All rights reserved. .\" .\" Licensed under the Apache License, Version 2.0 (the "License"); .\" you may not use this file except in compliance with the License. .\" You may obtain a copy of the License at .\" .\" http://www.apache.org/licenses/LICENSE-2.0 .\" .\" Unless required by applicable law or agreed to in writing, software .\" distributed under the License is distributed on an "AS IS" BASIS, .\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .\" See the License for the specific language governing permissions and .\" limitations under the License. .\" .SH NAME dgsh-merge-sum \- merge key value pairs, summing the values .SH SYNOPSIS \fBdgsh-merge-sum\fP \fIfile ...\fP .SH DESCRIPTION \fIdgsh-merge-sum\fP will read \fIkey\fP, \fIvalue\fP pairs from the files specified in its standard input, and print the input records merged together according to the value of the key. The input files should be sorted according to the key's value. Records with the same key will have their values summed, and a single corresponding record will be printed. Whitespace is used as the separator. Leading whitespace is not taken into account when parsing fields. Thus \fIdgsh-merge-sum\fP can process multiple files generated by \fIuniq -c\fP, and merge them into one. .SH "SEE ALSO" \fIuniq\fP(1), \fIdgsh\fP(1) .SH AUTHOR Diomidis Spinellis \(em ================================================ FILE: core-tools/src/dgsh-merge-sum.pl ================================================ #!/usr/bin/env perl # # Merge sorted (value, key) pairs, summing the values of equal keys # # Copyright 2014 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # use strict; use warnings; # Read a record from the specified file reference sub read_record { my ($fr) = @_; my $f = $fr->{file}; my $line = <$f>; if (!defined($line)) { $fr->{key} = undef; return; } ($fr->{value}, $fr->{key}) = ($line =~ m/^\s*(\d+)\s+(.*)/); } # Open input files; opening before reading prevents pipe writers from blocking my @file; # First file is always stdin binmode(STDIN, ":utf8"); $file[0]->{file} = \*STDIN; my $i = 1; for my $name (@ARGV) { open($file[$i]->{file}, '<:encoding(utf8)', $name) || die "Unable to open $name: $!\n"; $i++; } # Read first record from all files for my $f (@file) { read_record($f); } # Previous key printed my $prev; for (;;) { # Find smallest key my $smallest; for my $r (@file) { #print "Check $r->{value}, $r->{key}\n"; $smallest = $r if (!defined($smallest->{key}) || (defined($r->{key}) && $r->{key} lt $smallest->{key})); } exit 0 unless defined($smallest->{key}); #print "Smallest $smallest->{value}, $smallest->{key}\n"; # Sum up and renew all smallest keys my $sum = 0; my $key = $smallest->{key}; for my $r (@file) { if (defined($r->{key}) && ($r->{key} cmp $key) == 0) { $sum += $r->{value}; read_record($r); } } # Verify that input is sorted if (defined($prev) && ($key cmp $prev) < 0) { print STDERR "Input is not sorted: [$key] came after [$prev]\n"; exit 1; } $prev = $key; print "$sum $key\n"; } ================================================ FILE: core-tools/src/dgsh-monitor.1 ================================================ .TH DGSH-MONITOR 1 "11 December 2016" .\" .\" (C) Copyright 2013 Diomidis Spinellis. All rights reserved. .\" .\" Licensed under the Apache License, Version 2.0 (the "License"); .\" you may not use this file except in compliance with the License. .\" You may obtain a copy of the License at .\" .\" http://www.apache.org/licenses/LICENSE-2.0 .\" .\" Unless required by applicable law or agreed to in writing, software .\" distributed under the License is distributed on an "AS IS" BASIS, .\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .\" See the License for the specific language governing permissions and .\" limitations under the License. .\" .SH NAME dgsh-monitor \- monitor data on a pipe .SH SYNOPSIS \fBdgsh-monitor\fP .SH DESCRIPTION \fIdgsh-monitor\fP is a filter that reads lines from its standard input and for each line writes a JSON record containing the absolute timestamp (\fCatime\fP), the relative timestamp (\fCrtime\fP), the number of lines read (\fCnlines\fP), the number of bytes read (\fCnbytes\fP), and the actual data (\fCdata\fP). The values printed are those that were in effect before the line's first character was read (i.e. the number of bytes and lines starts at 0). The timestamps are printed as a decimal number of seconds (since epoch, January 1 1970, for the absolute one) with microsecond precision. .PP The command can be used in conjunction with \fIdgsh-writeval\fP for providing pipeline monitoring ports as a debugging aid. .SH "SEE ALSO" \fIdgsh\fP(1), \fIdgsh-writeval\fP(1) .SH BUGS When providing data regarding a single record, such as that written to a store, it can be confusing to see all the numerical values as zero. .SH AUTHOR Diomidis Spinellis \(em ================================================ FILE: core-tools/src/dgsh-monitor.c ================================================ /* * Copyright 2013 Diomidis Spinellis * * Prepend lines read with timestamp, number of lines, number of bytes. * Used for providing dgsh monitoring ports. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ #include #include #include #include #include #include #include #include #include #include #include "dgsh.h" static const char *program_name; static void usage(void) { fprintf(stderr, "Usage: %s\n", program_name); exit(1); } /* Print c with JSON escaping */ static void escape(int c) { switch (c) { case '\\': fputs("\\\\", stdout); break; case '"': fputs("\\\"", stdout); break; case '/': fputs("\\/", stdout); break; case '\b': fputs("\\b", stdout); break; case '\f': fputs("\\f", stdout); break; case '\n': fputs("\\n", stdout); break; case '\r': fputs("\\r", stdout); break; case '\t': fputs("\\t", stdout); break; default: if (c < 0x1f) printf("\\u%04x", c); else putchar(c); } } int main(int argc, char *argv[]) { int c; unsigned long nlines, nbytes; struct timeval start, t; bool write_header = true; bool wrote_header = false; program_name = argv[0]; /* Default if nothing else is specified */ if (argc != 1) usage(); nbytes = nlines = 0; gettimeofday(&start, NULL); while ((c = getchar()) != EOF) { if (write_header) { gettimeofday(&t, NULL); if (wrote_header) printf("\" }\n"); printf("{ " // Absolute time (s) "\"atime\": %lld.%06d, " // Relative time (s, from program start) "\"rtime\": %.06lf, " "\"nlines\": %lu, " "\"nbytes\": %lu, " "\"data\": \"", (long long)t.tv_sec, (int)t.tv_usec, (t.tv_sec - start.tv_sec) + (t.tv_usec - start.tv_usec) / 1e6, nlines, nbytes); wrote_header = true; write_header = false; } escape(c); nbytes++; if (c == '\n') { nlines++; write_header = true; } } if (wrote_header) printf("\" }\n"); return 0; } ================================================ FILE: core-tools/src/dgsh-parallel.1 ================================================ .TH DGSH-PARALLEL 1 "15 December 2016" .\" .\" (C) Copyright 2016 Diomidis Spinellis. All rights reserved. .\" .\" Licensed under the Apache License, Version 2.0 (the "License"); .\" you may not use this file except in compliance with the License. .\" You may obtain a copy of the License at .\" .\" http://www.apache.org/licenses/LICENSE-2.0 .\" .\" Unless required by applicable law or agreed to in writing, software .\" distributed under the License is distributed on an "AS IS" BASIS, .\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .\" See the License for the specific language governing permissions and .\" limitations under the License. .\" .SH NAME dgsh-parallel \- Create a semi-homogeneous dgsh parallel processing block .SH SYNOPSIS \fBdgsh-parallel\fP [\fB\-d\fP] \fB\-f\fP \fIfile\fP | \fB\-l\fP \fIlist\fP | \fB\-n\fP \fIn\fP \fIcommand ...\fP .SH DESCRIPTION \fIdgsh-parallel\fP creates and executes a \fIdgsh\fP block that invokes multiple times the specified command and its optional arguments. If the command or its options include the \fI{}\fP string, this is replaced by the numeric or string identifier associated with each invocation. .SH OPTIONS .IP "\fB\-d\fP Allows the debugging of the generated script, by leaving it in the temporary directory and echoing its path on the standard error. .IP "\fB\-f\fP \fIfile\fP" Obtain string arguments from the specified file: one argument per line. One command will be generated for each line in the file. Each command will have \fI{}\fP strings replaced with the contents of the corresponding line. .IP "\fB\-l\fP \fIlist\fP" Obtain string arguments from the specified comma-separated list. One command will be generated for each list element. Each command will have \fI{}\fP strings replaced with the corresponding element. .IP "\fB\-n\fP \fIn\fP" Run \fIn\fP instances of the command. Each command will have \fI{}\fP strings replaced with the command's ordinal number, starting from 1. .SH EXAMPLES .PP Count in parallel the number of times each word appears in the specified input file(s). This sequence mirrors Hadoop's WordCount example. .ft C .nf # Scatter input dgsh-tee -s | # Run four instances of the command # Emulate Java's default StringTokenizer, sort, count dgsh-parallel -n 4 "tr -s ' \\t\\n\\r\\f' '\\n' | sort | uniq -c" | # Merge the four sorted counts dgsh-merge-sum '<|' '<|' '<|' .ft P .fi .SH "SEE ALSO" \fIdgsh\fP(1), \fIdgsh-tee\fP(1), .SH BUGS The interface between the generated script and its invokers is currently (December 2016) being polished. .SH AUTHOR Diomidis Spinellis \(em . ================================================ FILE: core-tools/src/dgsh-parallel.sh ================================================ #!/usr/bin/env bash #!dgsh # # Create and execute a semi-homongeneous dgsh parallel processing block # # Remove dgsh from path, so that commands aren't wrapped here # See http://stackoverflow.com/a/2108540/20520 # PATH => /bin:.../libexec/dgsh:/sbin OPATH="$PATH" WORK=:$PATH: # WORK => :/bin:.../libexec/dgsh:/sbin: REMOVE='[^:]*/libexec/dgsh' WORK=${WORK/:$REMOVE:/:} # WORK => :/bin:/sbin: WORK=${WORK%:} WORK=${WORK#:} PATH=$WORK # PATH => /bin:/sbin # Remove DGSH_IN, OUT so that commands don't negotiate test "$DGSH_IN" && ODGSH_IN="$DGSH_IN" test "$DGSH_OUT" && ODGSH_OUT="$DGSH_OUT" unset DGSH_IN unset DGSH_OUT usage() { echo 'Usage: dgsh-parallel [-d] -n n|-f file|-l list command ...' 1>&2 exit 2 } # Process flags while getopts 'df:l:n:' o; do case "$o" in d) DEBUG=1 ;; n) n="$OPTARG" nspec=X$nspec ;; f) file="$OPTARG" nspec=X$nspec ;; -l) list=$(echo "$OPTARG" | sed 's/,/ /g') nspec=X$nspec ;; *) usage ;; esac done shift $((OPTIND-1)) # Ensure commands is specified if [ ! "$1" ] ; then usage fi # Ensure exactly one sharding target is specified if [ ! "$nspec" ] || expr $nspec : .. >/dev/null ; then usage fi # Ensure generated script is always removed SCRIPT="${TMP:-/tmp}/dgsh-parallel-$$" if [ "$DEBUG" ] ; then echo "Script is $SCRIPT" 1>&2 else trap 'rm -rf "$SCRIPT"' 0 trap 'exit 2' 1 2 15 fi cat >$SCRIPT <&2 exit 2 fi | # Escape sed(1) special characters sed 's/[&/\\]/\\&/g' | # Replace {} with the name of each node while IFS='' read -r node ; do echo " $@" | sed "s/{}/$node/" done >>$SCRIPT cat >>$SCRIPT < /* printf() */ #include /* exit() */ #include /* getpagesize() */ #include "dgsh.h" int main(int argc, char *argv[]) { int ps = getpagesize(); char buf[ps]; int n = 1; int ninputs = -1; dgsh_negotiate(DGSH_HANDLE_ERROR, "dgsh-pecho", &ninputs, NULL, NULL, NULL); if (ninputs == 1) { n = read(STDIN_FILENO, buf, ps); while (n) { printf("%s", buf); n = read(STDIN_FILENO, buf, ps); if (n < 0) exit(1); } } ++argv; while (*argv) { (void)printf("%s", *argv); if (*++argv) putchar(' '); } putchar('\n'); return 0; } ================================================ FILE: core-tools/src/dgsh-readval.1 ================================================ .TH DGSH-READVAL 1 "21 March 2013" .\" .\" (C) Copyright 2013 Diomidis Spinellis. All rights reserved. .\" .\" Licensed under the Apache License, Version 2.0 (the "License"); .\" you may not use this file except in compliance with the License. .\" You may obtain a copy of the License at .\" .\" http://www.apache.org/licenses/LICENSE-2.0 .\" .\" Unless required by applicable law or agreed to in writing, software .\" distributed under the License is distributed on an "AS IS" BASIS, .\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .\" See the License for the specific language governing permissions and .\" limitations under the License. .\" .SH NAME dgsh-readval \- data store client .SH SYNOPSIS \fBdgsh-readval\fP [\fB\-c\fP | \fB-e\fP | \fB-l\fP] [\fB\-nq\fP] [\fB\-x\fP] \fB\-s\fP \fIpath\fP .SH DESCRIPTION \fIdgsh-readval\fP is a data store client. By default it will communicate with the store specified through the path to a Unix domain socket, ask to read the last (final) record written to that store, and write the value on its standard output. .PP \fIdgsh-readval\fP is normally executed from within \fIdgsh\fP-generated scripts, rather than through end-user commands. This manual page serves mainly to document its operation and the flags that can be used in \fIdgsh\fP scripts when reading from stores. .SH OPTIONS .IP "\fB\-c\fP Read the current (rather than the last) value from the store. If no complete record has been written into the store, the operation will block until such a record is available. .IP "\fB\-e\fP Read the current or an empty value from the store. If no complete record has been written into the store, the operation will return an empty record, rather than block. .IP "\fB\-l\fP Read the last value from the store. This is the default behavior of \fIdgsh-readval\fP. The operation will block until the store's server (\fIdgsh-writeval\fP) determines that it has read the last record by detecting an end-of-file condition on its standard input. .IP "\fB\-n\fP Do not retry a failed connection to the store. By default \fIdgsh-readval\fP will try to establish a connection to the store every one second. This behavior is designed to avoid failures due to race conditions between write stores that are started asynchronously (in the background) and subsequent read operations from them. .IP "\fB\-q\fP Ask the write store (the corresponding \fIdgsh-writeval\fP process) to terminate its operation. No value is read. .IP "\fB\-x\fP Do not participate in dgsh negotiation. .IP "\fB\-s\fP \fIpath\fP" This mandatory option must be used to specify the path of the Unix-domain socket \fIdgsh-readval\fP will connect to communicate with the store. This is specified as a normal Unix file path, e.g. \fC/tmp/myvalue\fP. .SH "SEE ALSO" \fIdgsh\fP(1), \fIdgsh-writeval\fP(1) .SH AUTHOR Diomidis Spinellis \(em ================================================ FILE: core-tools/src/dgsh-readval.c ================================================ /* * Copyright 2013 Diomidis Spinellis * * Communicate with the data store specified as a Unix-domain socket. * (User interface) * By default the command will read a value. * Calling it with the -q flag will send the data store a termination * command. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ #include #include #include #include #include #include #include #include #include "dgsh.h" #include "negotiate.h" #include "kvstore.h" #include "dgsh-debug.h" static const char *program_name; static void usage(void) { fprintf(stderr, "Usage: %s [-c|e|l] [-n] [-q] [-x] -s path\n" "-c" "\tRead the current value from the store\n" "-e" "\tRead current value or empty from the store\n" "-l" "\tRead the last (before EOF) value from the store (default)\n" "-n" "\tDo not retry failed connection to write store\n" "-q" "\tAsk the write-end to quit\n" "-x" "\tDo not participate in dgsh negotiation\n" "-s path" "\tSpecify the socket to connect to\n", program_name); exit(1); } int main(int argc, char *argv[]) { int ch; bool quit = false; char cmd = 0; const char *socket_path = NULL; bool retry_connection = true; bool should_negotiate = true; int ninputs = 0; int noutputs = 1; program_name = argv[0]; /* Default if nothing else is specified */ if (argc == 3) cmd = 'L'; while ((ch = getopt(argc, argv, "celnqxs:")) != -1) { switch (ch) { case 'c': /* Read current value */ cmd = 'C'; break; case 'e': /* Read current or empty value */ cmd = 'c'; break; case 'l': /* Read last value */ cmd = 'L'; break; case 'n': retry_connection = false; break; case 'q': quit = true; break; case 's': socket_path = optarg; break; case 'x': should_negotiate = false; break; case '?': default: usage(); } } argc -= optind; argv += optind; if (argc != 0 || socket_path == NULL) usage(); if (should_negotiate) dgsh_negotiate(DGSH_HANDLE_ERROR, program_name, &ninputs, &noutputs, NULL, NULL); else set_negotiation_complete(); dgsh_send_command(socket_path, cmd, retry_connection, quit, STDOUT_FILENO); return 0; } ================================================ FILE: core-tools/src/dgsh-tee.1 ================================================ .TH DGSH-TEE 1 "13 April 2017" .\" .\" (C) Copyright 2013-2017 Diomidis Spinellis. All rights reserved. .\" .\" Licensed under the Apache License, Version 2.0 (the "License"); .\" you may not use this file except in compliance with the License. .\" You may obtain a copy of the License at .\" .\" http://www.apache.org/licenses/LICENSE-2.0 .\" .\" Unless required by applicable law or agreed to in writing, software .\" distributed under the License is distributed on an "AS IS" BASIS, .\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .\" See the License for the specific language governing permissions and .\" limitations under the License. .\" .SH NAME dgsh-tee \- buffer, copy, permute, or distribute data from input sources to output sinks .SH SYNOPSIS \fBdgsh-tee\fP [\fB\-b\fP \fIbuffer-size\fP] [\fB\-afIMs\fP] [\fB\-i\fP \fIinput-file\fP] [\fB\-o\fP \fIoutput-file\fP] [\fB\-m\fP \fImemory-size\fP] [\fB\-p\fP \fIo1,o2 ...\fP] [\fB\-T\fP \fIdirectory\fP] [\fB\-t\fP \fIcharacter\fP] .SH DESCRIPTION \fIdgsh-tee\fP will read data from the specified sources and copy or distribute it to the specified sinks. It resembles in its operation \fItee\fP(1) and \fIcat\fP(1), but offers additional capabilities required for the operation of \fIdgsh\fP(1). In contrast to these programs, \fIdgsh-tee\fP will buffer the data it handles, so it will never cause deadlock or starvation when one or more sources are unable to provide data or if sinks are unable to receive them. Furthermore, \fIdgsh-tee\fP can copy data from multiple sources to multiple sinks, permute the data between sources and sinks, and also distribute the data among the sinks. .PP When copying data from a few sources to a multiple of their number sinks, the first input tuple will appear in the first sinks, and so on. As an example, two sources \fIa, b\fP will appear in six sinks as \fIa, b, a, b, a, b\fP. When copying data from many sources to a fraction of their number sinks, a tuple of sources equal to the number of sinks is output first, followed by a tuple of the next sinks, and so on. As an example, six sources \fa, b, c, d, e, f\fP will appear in two sinks as \fIa, c, e\fP in the first sink and \fIb, d, f\fP in the second one. In effect, the group of few sources or sinks is treated as a single unit to be scattered or sequentially concatenated. .PP \fIdgsh-tee\fP is normally executed within \fIdgsh\fP through wrappers that replace the system-provided \fItee\fP and \fIcat\fP commands. This manual page serves mainly to document its operation, to how it can be used in less common use cases, and to allow the creation of plug-compatible replacements implementing different record types. .SH OPTIONS .IP "\fB\-a\fP Open files subsequently specified with the \fB-o\fP option for appending. .IP "\fB\-b\fP \fIbuffer-size\fP" Specify the size of the buffer to use. This is by default 1MB. Buffers are chained together when more space is required, so the main utility of this option is to decrease the buffer size in memory-constrained environments. The specified number can be suffixed with \fBk\fI, \fBM\fI, or \fBG\fI to specify the corresponding unit. The specified buffer size must be less than the program's maximum memory size. .IP "\fB\-f\fP When the allocated memory size reaches the maximum memory threshold, start using a temporary file for buffering the data. This extends the amount of data that can be buffered to the space available on disk. The location of the temporary file follows the \fItempnam\fP(3) rules, and can be overridden through the .B -T option. .IP "\fB\-I\fP" Implement input-side buffering. By default \fIdgsh-tee\fP will buffer only as much input data, as is needed to avoid starving one of the specified sinks. In doing so it may cause its input source to block while having data to write. When this option is enabled \fIdgsh-tee\fP will always read data if they are available on the standard input, and will write that data to any (including zero) sinks that can read it. This can be useful in cases where a command with insufficient input buffering, like \fIjoin\fP(1), \fIsort\fP(1), or \fIpaste\fP(1), is gathering input from commands executing in parallel. In such a case adding \fIdgsh-tee\fP with input side buffering enabled at the end of each data pipeline, will increase the number of processes that can operate concurrently. .IP "\fB\-i\fP \fIinput-file\fP" Read input from the specified source file, rather than the standard input. The option can be provided multiple times to specify multiple files that will be read sequentially. All input files will be opened at the beginning of the program's operation in order to unblock the execution of asynchronous shell commands that redirect their output to the corresponding named pipes. Furthermore, when input-side buffering is specified \fB-I\fP data is read asynchronously from all specified input files. .IP "\fB\-M\fP" Provide memory use statistics on termination. This is mainly used for testing, to check against leaks of buffers. .IP "\fB\-o\fP \fIoutput-file\fP" Write copies of the input data to the specified sink file, rather than the standard output. The option can be provided multiple times to specify multiple files where input data will be copied. .IP "\fB\-m\fP \fImemory-size\fP" Specify the maximum size of memory to allocate for buffers. This is by default 256MB. When \fIdgsh-tee\fP exhausts this memory, it enters a state where it waits for its output buffers drain, thus freeing allocated memory. The specified number can be suffixed with \fBk\fI, \fBM\fI, or \fBG\fI to specify the corresponding unit. The specified maximum memory size must be larger than the program's buffer size. .IP "\fB\-p\fP \fIo1,o2 ...\fP" Permute the inputs to the specified outputs. The comma-separated arguments \fIo1,o2, ...\fP specify the number of the output channel (starting from 1) where the corresponding \fIn\fPth input channel will be sent. Thus, input 1 goes to output \fIo1\fP, input 2 goes to output \fIo2\fP, and so on. As an example a cross-permutation is specified with the argument \fI-p 2,1\fP. .IP "\fB\-s\fP" Scatter the input fairly across the sinks, rather than copying it to all. When this option is in effect, the input data are divided into chunks of one or more lines, and each chunk is written only to a single sink. This is useful for dividing the work among multiple processes operating in parallel. .IP "\fB\-T\fP \fIdirectory\fP" Specify the directory to use for storing the temporary file, when the specified maximum buffer memory size is exceeded. .IP "\fB\-t\fP \fIchar\fP" Use \fIchar\fP as the record separator, By default the record separator is a newline, An empty (not missing) argument for the record separator will make the record separator be the null character. .SH "SEE ALSO" \fIdgsh\fP(1) \fItempnam\fP(3) .SH AUTHOR Diomidis Spinellis \(em ================================================ FILE: core-tools/src/dgsh-tee.c ================================================ /* * Copyright 2013 Diomidis Spinellis * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ #ifdef __linux__ #define _XOPEN_SOURCE 500 // pread pwrite #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #include "dgsh.h" #include "dgsh-debug.h" #include "minmax.h" #if defined(DEBUG_DATA) #define DATA_DUMP 1 #else #define DATA_DUMP 0 #endif /* * Data that can't be written is stored in a sequential pool of buffers, * each buffer_size long. * As more data is read the buffer pool with the pointers (buffers) * is continuously increased; there is no round-robin mechanism. * However, as data is written out to all sinks, the actual buffers are * freed, thus keeping memory consumption reasonable. */ static int buffer_size = 1024 * 1024; /* * A buffer in the memory pool. */ struct pool_buffer { void *p; /* Memory allocated for it (b_memory) */ enum { s_none, /* Stored nowhere */ s_memory, /* Stored in memory */ s_memory_backed,/* Stored in memory and backed to temporary file */ s_file /* Stored in temporary file */ } s; /* Where it is stored */ }; /* * A pool of buffers */ struct buffer_pool { struct pool_buffer *buffers; /* A dynamically adjusted vector of buffers */ int pool_size; /* Size of allocated pool_buffers vectors */ int allocated_pool_end; /* The first buffer in the above pool that has not been allocated */ /* Allocated bufffer information */ int buffers_allocated, buffers_freed, max_buffers_allocated; /* Paging information */ int buffers_paged_out, buffers_paged_in, pages_freed; int page_out_ptr; /* Pointer to first buffer to page out */ int page_file_fd; /* File descriptor of temporary file used for paging buffer pool */ int free_pool_begin; /* Start of freed area */ }; /* Construct a new buffer pool object */ static struct buffer_pool * new_buffer_pool(void) { struct buffer_pool *bp; if ((bp = (struct buffer_pool *)malloc(sizeof(struct buffer_pool))) == NULL) err(1, NULL); bp->buffers = NULL; bp->pool_size = 0; bp->page_out_ptr = 0; bp->page_file_fd = -1; bp->free_pool_begin = 0; bp->allocated_pool_end = 0; bp->buffers_allocated = bp->buffers_freed = bp->max_buffers_allocated = bp->buffers_paged_out = bp->buffers_paged_in = bp->pages_freed = 0; return bp; } /* * A buffer that is used for I/O. * It points to a part of a pool buffer. */ struct io_buffer { void *p; /* Memory pointer */ size_t size; /* Buffer size */ }; /* Maximum amount of memory to allocate. (Set through -S) */ static unsigned long max_mem = 256 * 1024 * 1204; /* Scatter the output across the files, rather than copying it. */ static bool opt_scatter = false; /* * When set, permute the inputs to the specified outputs * Ordinals and number of the destination outputs */ static int *permute_dest = NULL; static int permute_n = 0; /* Use a temporary file for overflowing buffered data */ static bool use_tmp_file = false; /* User-specified temporary directory */ static char *opt_tmp_dir = NULL; /* * Split scattered data on blocks of specified size; otherwise on line boundaries * Currently there is no support for this option; a -l option should be added. */ static bool block_len = 0; /* Set to true when we reach EOF on input */ static bool reached_eof = false; /* Record terminator */ static char rt = '\n'; /* Linked list of files we write to */ struct sink_info { struct sink_info *next; /* Next list element */ char *name; /* Output file name */ int fd; /* Output file descriptor */ off_t pos_written; /* Position up to which written */ off_t pos_to_write; /* Position up to which to write */ bool active; /* True if this sink is still active */ struct source_info *ifp;/* Input file we read from */ bool chain_last; /* True if last element in a group; Writing (copy or scatter) should not continue to next element */ }; /* Construct a new sink_info object */ static struct sink_info * new_sink_info(const char *name) { struct sink_info *ofp; if ((ofp = (struct sink_info *)malloc(sizeof(struct sink_info))) == NULL) err(1, NULL); ofp->name = name ? strdup(name) : NULL; ofp->active = true; ofp->pos_written = ofp->pos_to_write = 0; ofp->next = NULL; return ofp; } /* Linked list of files we read from */ struct source_info { struct source_info *next; /* Next list element */ char *name; /* Input file name */ int fd; /* Input file descriptor */ struct buffer_pool *bp; /* Buffers where pending input is stored */ off_t source_pos_read; /* The position up to which all sinks have read data */ bool reached_eof; /* True if we reached EOF for this source */ off_t read_min_pos; /* Minimum position read by all sinks */ bool active; /* True if this is a source that should be currently read (rather than chained later on) */ bool is_read; /* True if an active sink reads it */ bool chain_last; /* True if reading should stop at this element rather than continue to the next element */ }; /* Return the name of a source or sink */ #define fp_name(fp) ((fp)->name ? (fp)->name : fd_name((fp)->fd)) static char * fd_name(int fd) { static char buff[40]; sprintf(buff, "fd(%d)", fd); return buff; } /* Construct a new source_info object */ static struct source_info * new_source_info(const char *name) { struct source_info *ifp; if ((ifp = (struct source_info *)malloc(sizeof(struct source_info))) == NULL) err(1, NULL); ifp->name = name ? strdup(name) : NULL; ifp->bp = new_buffer_pool(); ifp->source_pos_read = 0; ifp->reached_eof = false; ifp->next = NULL; return ifp; } /* * States for the copying engine. * Two disjunct sets: * input side buffering (ib) and output side buffering (ob) * Input side buffering will always read input if it is available, * presenting an infinite output buffer to the upstream process. * The output-side buffering will read input only if at least one * active output buffer is empty. * The setting in effect is determined by the program's -I flag. * * States read_ib and read_ob have select return: * - if data is available for reading, * - if the process can write out data already read, * - not if the process can write to other fds * * States drain_ib and write_ob have select return * if the process can write to any fd. * Waiting on all output buffers (not only those with data) * is needed to avoid starvation of downstream processes * when no output is available. * If a program can accept data this process will then transition to * read_* to read more data. * * State drain_ob has select return only if the process can write out * data already read. * * See also the diagram tee-state.dot */ enum state { read_ib, /* Must read input; write if data available */ read_ob, /* As above, but don't transition to write */ drain_ib, /* Don't read input; write if possible */ drain_ob, /* Empty data buffers by writing */ write_ob, /* Write data, before reading */ }; /* * Return the total number of bytes required for storing all buffers * up to the specified memory pool */ static unsigned long memory_pool_size(struct buffer_pool *bp, int pool) { return ((bp->buffers_allocated - bp->buffers_freed) + (pool - bp->allocated_pool_end + 1)) * buffer_size; } /* Write half of the allocated buffer pool to the temporary file */ static void page_out(struct buffer_pool *bp) { if (bp->page_file_fd == -1) { char *template; /* * Create a temporary file that will be deleted on exit. * The location follows tempnam rules (argument, TMPDIR, * P_tmpdir, /tmp), while the creation through mkstemp * avoids race conditions. */ if ((template = tempnam(opt_tmp_dir, "sg-")) == NULL) err(1, "Unable to obtain temporary file name"); if ((template = realloc(template, strlen(template) + 7)) == NULL) err(1, "Error obtaining temporary file name space"); strcat(template, "XXXXXX"); if ((bp->page_file_fd = mkstemp(template)) == -1) err(1, "Unable to create temporary file %s", template); } /* * Page-out memory buffers from the pool, round-robin fashion, * starting from the oldest buffers. * This is good enough for the simple common case where one output fd is blocked. */ while (memory_pool_size(bp, bp->allocated_pool_end - 1) > max_mem / 2) { switch (bp->buffers[bp->page_out_ptr].s) { case s_memory: if (pwrite(bp->page_file_fd, bp->buffers[bp->page_out_ptr].p, buffer_size, (off_t)bp->page_out_ptr * buffer_size) != buffer_size) err(1, "Write to temporary file failed"); /* FALLTHROUGH */ case s_memory_backed: DPRINTF(4, "Page out buffer %d %p", bp->page_out_ptr, bp->buffers[bp->page_out_ptr].p); bp->buffers[bp->page_out_ptr].s = s_file; free(bp->buffers[bp->page_out_ptr].p); bp->buffers_freed++; bp->buffers_paged_out++; DPRINTF(4, "Paged out buffer %d %p", bp->page_out_ptr, bp->buffers[bp->page_out_ptr].p); break; case s_file: case s_none: break; default: assert(false); } if (++bp->page_out_ptr == bp->allocated_pool_end) bp->page_out_ptr = 0; } } /* * Allocate memory for the specified pool member. * Return false if no such memory is available. */ static bool allocate_pool_buffer(struct buffer_pool *bp, int pool) { struct pool_buffer *b = &bp->buffers[pool]; if ((b->p = malloc(buffer_size)) == NULL) { DPRINTF(4, "Unable to allocate %d bytes for buffer %ld", buffer_size, b - bp->buffers); bp->max_buffers_allocated = MAX(bp->buffers_allocated - bp->buffers_freed, bp->max_buffers_allocated); return false; } b->s = s_memory; DPRINTF(4, "Allocated buffer %ld to %p", b - bp->buffers, b->p); bp->buffers_allocated++; bp->max_buffers_allocated = MAX(bp->buffers_allocated - bp->buffers_freed, bp->max_buffers_allocated); return true; } /* * Ensure that the specified pool buffer is in memory */ static void page_in(struct buffer_pool *bp, int pool) { struct pool_buffer *b = &bp->buffers[pool]; switch (b->s) { case s_memory_backed: case s_memory: break; case s_file: /* Good time to ensure that there will be page-in memory available */ if (memory_pool_size(bp, bp->allocated_pool_end - 1) > max_mem) page_out(bp); if (!allocate_pool_buffer(bp, pool)) err(1, "Out of memory paging-in buffer"); if (pread(bp->page_file_fd, b->p, buffer_size, (off_t)pool * buffer_size) != buffer_size) err(1, "Read from temporary file failed"); bp->buffers_paged_in++; b->s = s_memory_backed; DPRINTF(4, "Page in buffer %d", pool); break; case s_none: default: DPRINTF(4, "Buffer %d has invalid storage %d", pool, b->s); assert(false); break; } } /* * Allocate memory for the specified pool * If we're out of memory by reaching the user-specified limit * or a system's hard limit return false. * If sufficient memory is available return true. * buffers[pool] will then point to a buffer_size block of available memory. */ static bool memory_allocate(struct buffer_pool *bp, int pool) { int i, orig_pool_size; struct pool_buffer *orig_buffers; if (pool < bp->allocated_pool_end) return true; DPRINTF(4, "Buffers allocated: %d Freed: %d", bp->buffers_allocated, bp->buffers_freed); /* Check soft memory limit through allocated plus requested memory. */ if (memory_pool_size(bp, pool) > max_mem) { if (use_tmp_file) page_out(bp); else return false; } /* Keep original values to undo on failure. */ orig_pool_size = bp->pool_size; orig_buffers = bp->buffers; /* Resize bank, if needed. One iteration should suffice. */ while (pool >= bp->pool_size) { if (bp->pool_size == 0) bp->pool_size = 1; else bp->pool_size *= 2; if ((bp->buffers = realloc(bp->buffers, bp->pool_size * sizeof(struct pool_buffer))) == NULL) { DPRINTF(4, "Unable to reallocate buffer pool bank"); bp->pool_size = orig_pool_size; bp->buffers = orig_buffers; return false; } } /* Allocate buffer memory [allocated_pool_end, pool]. */ for (i = bp->allocated_pool_end; i <= pool; i++) if (!allocate_pool_buffer(bp, i)) { bp->allocated_pool_end = i; return false; } bp->allocated_pool_end = pool + 1; return true; } /* * Free a file-backed buffer at the specified pool location * by punching a hole to the file. This is a best effort * operation, as it is only supported on Linux. */ static void buffer_file_free(struct buffer_pool *bp, int pool) { #ifdef FALLOC_FL_PUNCH_HOLE static bool warned = false; if (fallocate(bp->page_file_fd, FALLOC_FL_PUNCH_HOLE, pool * buffer_size, buffer_size) < 0 && !warned) { warn("Failed to free temporary buffer space"); warned = true; } #endif bp->pages_freed++; } /* * Ensure that pool buffers from [0,pos) are free. */ static void memory_free(struct buffer_pool *bp, off_t pos) { int pool_end = pos / buffer_size; int i; DPRINTF(4, "memory_free: pool=%p pos = %ld, begin=%d end=%d", bp, (long)pos, bp->free_pool_begin, pool_end); for (i = bp->free_pool_begin; i < pool_end; i++) { switch (bp->buffers[i].s) { case s_memory: free(bp->buffers[i].p); bp->buffers_freed++; break; case s_file: buffer_file_free(bp, i); break; case s_memory_backed: buffer_file_free(bp, i); free(bp->buffers[i].p); bp->buffers_freed++; break; case s_none: break; default: assert(false); break; } bp->buffers[i].s = s_none; DPRINTF(4, "Freed buffer %d %p (pos = %ld, begin=%d end=%d)", i, bp->buffers[i].p, (long)pos, bp->free_pool_begin, pool_end); #ifdef DEBUG bp->buffers[i].p = NULL; #endif } bp->free_pool_begin = pool_end; } /* * Set the buffer to write to for reading from a file from * position onward, ensuring that sufficient memory is allocated. * Return false if no memory is available. */ static bool source_buffer(struct source_info *ifp, /* OUT */ struct io_buffer *b) { int pool = ifp->source_pos_read / buffer_size; size_t pool_offset = ifp->source_pos_read % buffer_size; if (!memory_allocate(ifp->bp, pool)) return false; if (ifp->bp->buffers[pool].s != s_memory) DPRINTF(4, "ifp->bp->buffers[pool].s = 0x%x, pool=%d\n", ifp->bp->buffers[pool].s, pool); assert(ifp->bp->buffers[pool].s == s_memory); b->p = ifp->bp->buffers[pool].p + pool_offset; b->size = buffer_size - pool_offset; DPRINTF(4, "Source buffer(%ld) returns pool %d(%p) o=%ld l=%ld a=%p", (long)ifp->source_pos_read, pool, ifp->bp->buffers[pool].p, (long)pool_offset, (long)b->size, b->p); return true; } /* * Return a buffer to read from for writing to a file from a position onward * When processing lines, b.size can be 0 */ static struct io_buffer sink_buffer(struct sink_info *ofp) { struct io_buffer b; int pool = ofp->pos_written / buffer_size; size_t pool_offset = ofp->pos_written % buffer_size; size_t source_bytes = ofp->pos_to_write - ofp->pos_written; b.size = MIN(buffer_size - pool_offset, source_bytes); if (b.size == 0) b.p = NULL; else { if (ofp->ifp->bp->page_file_fd != -1) page_in(ofp->ifp->bp, pool); b.p = ofp->ifp->bp->buffers[pool].p + pool_offset; } DPRINTF(4, "Sink buffer(%ld-%ld) returns pool %d(%p) o=%ld l=%ld a=%p for input fd: %s", (long)ofp->pos_written, (long)ofp->pos_to_write, pool, b.size ? ofp->ifp->bp->buffers[pool].p : NULL, (long)pool_offset, (long)b.size, b.p, fp_name(ofp->ifp)); return b; } /* * Return a pointer to read from for writing to a file from a position onward */ static char * sink_pointer(struct buffer_pool *bp, off_t pos_written) { int pool = pos_written / buffer_size; size_t pool_offset = pos_written % buffer_size; if (bp->page_file_fd != -1) page_in(bp, pool); return bp->buffers[pool].p + pool_offset; } /* * Return the size of a buffer region that can be read for the specified endpoints */ static size_t sink_buffer_length(off_t start, off_t end) { size_t pool_offset = start % buffer_size; size_t source_bytes = end - start; DPRINTF(4, "sink_buffer_length(%ld, %ld) = %ld", (long)start, (long)end, (long)MIN(buffer_size - pool_offset, source_bytes)); return MIN(buffer_size - pool_offset, source_bytes); } /* The result of the following read operation. */ enum read_result { read_ok, /* Normal read */ read_oom, /* Out of buffer memory */ read_again, /* EAGAIN */ read_eof, /* EOF (0 bytes read) */ }; /* * Read from the source into the memory buffer * Return the number of bytes read, or -1 on end of file. */ static enum read_result source_read(struct source_info *ifp) { int n; struct io_buffer b; if (!source_buffer(ifp, &b)) { DPRINTF(4, "Memory full"); /* Provide some time for the output to drain. */ return read_oom; } if ((n = read(ifp->fd, b.p, b.size)) == -1) switch (errno) { case EAGAIN: DPRINTF(4, "EAGAIN on %s", fp_name(ifp)); return read_again; default: err(3, "Read from %s", fp_name(ifp)); } ifp->source_pos_read += n; DPRINTF(4, "Read %d out of %zu bytes from %s data=[%.*s]", n, b.size, fp_name(ifp), (int)n * DATA_DUMP, (char *)b.p); /* Return -1 on EOF */ return n ? read_ok : read_eof; } /* * Allocate available read data to empty sinks that can be written to, * by adjusting their ifp, pos_written, and pos_to_write pointers. */ static void allocate_data_to_sinks(fd_set *sink_fds, struct sink_info *files) { struct sink_info *ofp; int available_sinks = 0; off_t pos_assigned = 0; size_t available_data, data_per_sink; size_t data_to_assign = 0; bool use_reliable = false; /* Easy case: distribute to all files. */ if (!opt_scatter) { for (ofp = files; ofp; ofp = ofp->next) { /* Advance to next input file, if required */ if (ofp->pos_written == ofp->ifp->source_pos_read && ofp->ifp->reached_eof && !ofp->ifp->chain_last) { DPRINTF(4, "%s(): advance to input file %s\n", __func__, fp_name(ofp->ifp)); ofp->ifp = ofp->ifp->next; ofp->ifp->active = true; ofp->pos_written = 0; } ofp->pos_to_write = ofp->ifp->source_pos_read; } return; } /* * Difficult case: fair scattering across available sinks * Thankfully here we only have a single input file */ /* Determine amount of fresh data to write and number of available sinks. */ for (ofp = files; ofp; ofp = ofp->next) { pos_assigned = MAX(pos_assigned, ofp->pos_to_write); if (ofp->pos_written == ofp->pos_to_write && FD_ISSET(ofp->fd, sink_fds)) available_sinks++; } /* * Ensure we operate in a continuous memory region by clamping * the length of the available data to terminate at the end of * the buffer. */ available_data = sink_buffer_length(pos_assigned, files->ifp->source_pos_read); if (available_sinks == 0) return; /* Assign data to sinks. */ data_per_sink = available_data / available_sinks; for (ofp = files; ofp; ofp = ofp->next) { /* Move to next file if this has data to write, or isn't ready. */ if (ofp->pos_written != ofp->pos_to_write || !FD_ISSET(ofp->fd, sink_fds)) continue; DPRINTF(4, "pos_assigned=%ld source_pos_read=%ld available_data=%ld available_sinks=%d data_per_sink=%ld", (long)pos_assigned, (long)ofp->ifp->source_pos_read, (long)available_data, available_sinks, (long)data_per_sink); /* First file also gets the remainder bytes. */ if (data_to_assign == 0) data_to_assign = sink_buffer_length(pos_assigned, pos_assigned + data_per_sink + available_data % available_sinks); else data_to_assign = data_per_sink; /* * Assign data_to_assign to *ofp (pos_written, pos_to_write), * and advance pos_assigned. */ ofp->pos_written = pos_assigned; /* Initially nothing has been written. */ if (block_len == 0) { /* Write whole lines */ if (available_data > buffer_size / 2 && !use_reliable) { /* * Efficient algorithm: * Assume that multiple lines appear in data_per_sink. * Go to a calculated boundary and scan backward to find * a new line. */ off_t data_end = pos_assigned + data_to_assign - 1; for (;;) { if (data_end <= pos_assigned) { /* * If no newline was found with backward scanning * degenerate to the efficient algorithm. This will * scan further forward, and can defer writing the * last chunk, until more data is read. */ use_reliable = true; goto reliable; } if (*sink_pointer(ofp->ifp->bp, data_end) == rt) { pos_assigned = data_end + 1; break; } data_end--; } } else { /* * Reliable algorithm: * Scan forward for new lines until at least * data_per_sink are covered, or we reach the end of available data. * Keep a record of the last encountered newline. * This is used to backtrack when we scan past the end of the * available data. */ off_t data_end, last_nl; reliable: last_nl = -1; data_end = pos_assigned; for (;;) { if (data_end >= ofp->ifp->source_pos_read) { if (last_nl != -1) { pos_assigned = last_nl + 1; break; } else { /* No newline found in buffer; defer writing. */ ofp->pos_to_write = pos_assigned; DPRINTF(4, "scatter to file[%s] no newline from %ld to %ld", fp_name(ofp), (long)pos_assigned, (long)data_end); return; } } if (*sink_pointer(ofp->ifp->bp, data_end) == rt) { last_nl = data_end; if (data_end - pos_assigned > data_per_sink) { pos_assigned = data_end + 1; break; } } data_end++; } } } else pos_assigned += data_to_assign; ofp->pos_to_write = pos_assigned; DPRINTF(4, "scatter to file[%s] pos_written=%ld pos_to_write=%ld data=[%.*s]", fp_name(ofp), (long)ofp->pos_written, (long)ofp->pos_to_write, (int)(ofp->pos_to_write - ofp->pos_written) * DATA_DUMP, sink_pointer(ofp->ifp->bp, ofp->pos_written)); } } /* * Write out from the memory buffer to the sinks where write will not block. * Free memory no more needed even by the write pointer farthest behind. * Return the number of bytes written. */ static size_t sink_write(struct source_info *ifiles, fd_set *sink_fds, struct sink_info *ofiles) { struct sink_info *ofp; struct source_info *ifp; size_t written = 0; for (ifp = ifiles; ifp; ifp = ifp->next) { ifp->read_min_pos = ifp->source_pos_read; ifp->is_read = false; } allocate_data_to_sinks(sink_fds, ofiles); for (ofp = ofiles; ofp; ofp = ofp->next) { DPRINTF(4, "\n%s(): try write to file %s", __func__, fp_name(ofp)); if (ofp->active && FD_ISSET(ofp->fd, sink_fds)) { int n; struct io_buffer b; b = sink_buffer(ofp); DPRINTF(4, "\n%s(): sink buffer returned %d bytes to write", __func__, (int)b.size); if (b.size == 0) /* Can happen when a line spans a buffer */ n = 0; else { n = write(ofp->fd, b.p, b.size); if (n < 0) switch (errno) { /* EPIPE is acceptable, for the sink's reader can terminate early. */ case EPIPE: ofp->active = false; (void)close(ofp->fd); DPRINTF(4, "EPIPE for %s", fp_name(ofp)); break; case EAGAIN: DPRINTF(4, "EAGAIN for %s", fp_name(ofp)); n = 0; break; default: err(2, "Error writing to %s", fp_name(ofp)); } else { ofp->pos_written += n; written += n; } } DPRINTF(4, "Wrote %d out of %zu bytes for file %s pos_written=%lu data=[%.*s]", n, b.size, fp_name(ofp), (unsigned long)ofp->pos_written, (int)n * DATA_DUMP, (char *)b.p); } if (ofp->active) { ofp->ifp->read_min_pos = MIN(ofp->ifp->read_min_pos, ofp->pos_written); ofp->ifp->is_read = true; } } /* Free buffers all sinks have read */ for (ifp = ifiles; ifp; ifp = ifp->next) { memory_free(ifp->bp, ifp->read_min_pos); /* * We are reading this source, so don't even think freeing * sources after this. */ if (ifp->is_read) break; } DPRINTF(4, "Wrote %zu total bytes", written); return written; } static void usage(const char *name) { fprintf(stderr, "Usage %s [-b size] [-i file] [-IMs] [-o file] [-m size] [-t char]\n" "-a" "\tOpen output file(s) for appending\n" "-b size" "\tSpecify the size of the buffer to use (used for stress testing)\n" "-f" "\tOverflow buffered data into a temporary file\n" "-I" "\tInput-side buffering\n" "-i file" "\tGather input from specified file\n" "-m size[k|M|G]""\tSpecify the maximum buffer memory size\n" "-M" "\tProvide memory use statistics on termination\n" "-o file" "\tScatter output to specified file\n" "-p d1[,d2...]" "\tPermute inputs to specified outputs\n" "-s" "\tScatter the input across the files, rather than copying it to all\n" "-T dir" "\tSpecify directory for storing temporary file\n" "-t char" "\tProcess char-terminated records (newline default)\n", name); exit(1); } /* * Set the specified file descriptor to operate in non-blocking * mode. * It seems that even if select returns for a specified file * descriptor, performing I/O to it may block depending on the * amount of data specified. * See See http://pubs.opengroup.org/onlinepubs/009695399/functions/write.html#tag_03_866 */ static void non_block(int fd, const char *name) { int flags = fcntl(fd, F_GETFL, 0); if (flags < 0) err(2, "Error getting flags for %s", name); if (fcntl(fd, F_SETFL, flags | O_NONBLOCK) < 0) err(2, "Error setting %s to non-blocking mode", name); } /* * Show the arguments passed to select(2) in human-readable form * If check is true, abort the program if no bit is on */ static void show_select_args(const char *msg, fd_set *source_fds, struct source_info *ifiles, fd_set *sink_fds, struct sink_info *ofiles, bool check) { #ifdef DEBUG struct sink_info *ofp; struct source_info *ifp; int nbits = 0; fprintf(stderr, "%s: ", msg); for (ifp = ifiles; ifp; ifp = ifp->next) if (FD_ISSET(ifp->fd, source_fds)) { fprintf(stderr, "%s ", fp_name(ifp)); nbits++; } for (ofp = ofiles; ofp; ofp = ofp->next) if (FD_ISSET(ofp->fd, sink_fds)) { fprintf(stderr, "%s ", fp_name(ofp)); nbits++; } fputc('\n', stderr); if (check && nbits == 0) abort(); #endif } static void show_state(enum state state) { #ifdef DEBUG char *s; switch (state) { case read_ib: s = "read_ib"; break; case read_ob: s = "read_ob"; break; case drain_ib: s = "drain_ib"; break; case drain_ob: s = "drain_ob"; break; case write_ob: s = "write_ob"; break; } fprintf(stderr, "State: %s\n", s); #endif } /* Parse the specified option as a size with a suffix and return its value. */ static unsigned long parse_size(const char *progname, const char *opt) { char size; unsigned long n; size = 'b'; if (sscanf(opt, "%lu%c", &n, &size) < 1) usage(progname); switch (size) { case 'B' : case 'b': return n; case 'K' : case 'k': return n * 1024; case 'M' : case 'm': return n * 1024 * 1024; case 'G' : case 'g': return n * 1024 * 1024 * 1024; default: fprintf(stderr, "Unknown size suffix: %c\n", size); usage(progname); } /* NOTREACHED */ return 0; } /* * Parse and validate a comma-separated list of integers setting the * variables permute_dest and permute_n. */ static void parse_permute(char *s) { char *p; char *copy = strdup(s); int i; if (copy == NULL) errx(1, "Out of memory for destination string"); DPRINTF(4, "In parse_permute [%s]", s); for (p = strtok(copy, ","); p != NULL; p = strtok(NULL, ",")) permute_n++; free(copy); if ((permute_dest = (int *)malloc(sizeof(int) * permute_n)) == NULL) errx(1, "Out of memory for permutation destination"); for (p = strtok(s, ","), i= 0; p != NULL; p = strtok(NULL, ","), i++) { permute_dest[i] = atoi(p) - 1; if (permute_dest[i] < 0 || permute_dest[i] >= permute_n) errx(1, "Illegal permutation destination [%s]", s); } for (i = 0; i < permute_n; i++) DPRINTF(4, "%d = %d", i, permute_dest[i]); DPRINTF(4, "permute_n=%d", permute_n); } /* * Return the input file corresponding to the specified * permuted output file number. */ static struct source_info * output_source(struct source_info *ifiles, int output_n) { int i, input_n = -1; struct source_info *ifp; /* Find input file number */ for (i = 0; i < permute_n; i++) if (permute_dest[i] == output_n) { input_n = i; break; } if (input_n == -1) errx(1, "Unspecified output %d", output_n + 1); /* Find input file pointer */ for (ifp = ifiles, i = 0; ifp; ifp = ifp->next, i++) if (i == input_n) return ifp; assert(0); return NULL; } static void memory_stats(struct source_info *ifiles) { struct source_info *ifp; for (ifp = ifiles; ifp; ifp = ifp->next) { fprintf(stderr, "Input file: %s\n", fp_name(ifp)); fprintf(stderr, "Buffers allocated: %d Freed: %d Maximum allocated: %d\n", ifp->bp->buffers_allocated, ifp->bp->buffers_freed, ifp->bp->max_buffers_allocated); fprintf(stderr, "Page out: %d In: %d Pages freed: %d\n", ifp->bp->buffers_paged_out, ifp->bp->buffers_paged_in, ifp->bp->pages_freed); } } /* * Return true if an element with ordinal number n, * is the first element of a group in a series of groups * of group_size each. * Example: elements 0 and 3 in groups of size 3. */ static bool first_in_group(int group_size, int n) { return n % group_size == 0; } /* * Return true if an element with ordinal number n, * is the last element of a group in a series of groups * of group_size each. * Example: elements 2 and 5 in groups of size 3. */ static bool last_in_group(int group_size, int n) { return (n + 1) % group_size == 0; } struct list { struct list *next; }; /* * Transpose the elements of th specified linked list given * a notional new row length. As an example, a list of 12 elements * with a row size of 3, would be transposed as follows. * 0 -> 4 -> 8 -> * 1 -> 5 -> 9 -> * 2 -> 6 -> 10 -> * 3 -> 7 -> 11 * This function can handle arbitrary lists, as long * as the list's first element is the pointer to the * next one. */ static void list_transpose(struct list *lst, int row_length) { int i, count = 0; struct list **vector, *p; /* Create a vector of pointers to list elements */ for (p = lst; p; p = p->next) count++; vector = (struct list **)malloc(count * sizeof(struct list *)); if (vector == NULL) err(1, NULL); for (p = lst, i = 0; p; p = p->next, i++) vector[i] = p; /* Transpose notional rows into columns */ for (i = 0; i < count - row_length; i++) vector[i]->next = vector[i + row_length]; for (i = count - row_length; i < count - 1; i++) vector[i]->next = vector[(i + 1) % row_length]; free(vector); } /* * Chain input and output files into groups * by setting the chain_last of all I/O files * and the input file (ifp) field of all output * files. * * These are the possible cases and the corresponding * group chains. * * Read from many, output to one (cat) * A->B->C > a * All input files are chained together * * Read from one, output to many (tee) * A > b->c->d * All output files are chained together * * Read from many output to permuted many (perm) * A > d * B > c * C > b * D > a * No files are chained * * Read from few, output to more (multipipe tee) * A > a->d->g * B > b->e->h * C > c->f->i * Output files are chained into groups * * Read from many, output to few (multipipe cat) * A->D->G > a * B->E->H > b * C->F->I > c * Input files are chained into groups * * Note that cat, tee, and perm are special cases of the multipipe ones * are are implemented as such. */ static void chain_io_files(struct source_info *ifiles, struct sink_info *ofiles, bool permute) { int nin = 0, nout = 0; int group_size, n, i; struct source_info *ifp; struct sink_info *ofp; for (ifp = ifiles; ifp; ifp = ifp->next) nin++; for (ofp = ofiles; ofp; ofp = ofp->next) nout++; if (nin >= nout) { /* * Read from many output to few. * First input element in group is active. * Chain all but the last element of each input chain. * None of the outputs are chained. */ if (nin % nout) errx(1, "The number of inputs %d is not an exact multiple of the number of outputs %d", nin, nout); group_size = nin / nout; list_transpose((struct list *)ifiles, group_size); for (ifp = ifiles, n = 0; ifp; ifp = ifp->next, n++) { ifp->active = first_in_group(group_size, n); ifp->chain_last = last_in_group(group_size, n); } for (ofp = ofiles, ifp = ifiles, n = 0; ofp; ofp = ofp->next, n++) { ofp->chain_last = true; ofp->ifp = permute ? output_source(ifiles, n) : ifp; for (i = 0; i < group_size; i++) ifp = ifp->next; } } else { /* * Read from few output to many. * All inputs are active, none is chained. * Chain all but the last element in the output chain. */ if (nout % nin) errx(1, "The number of outputs %d is not an exact multiple of the number of inputs %d", nin, nout); group_size = nout / nin; assert(!permute); list_transpose((struct list *)ofiles, group_size); for (ifp = ifiles, n = 0; ifp; ifp = ifp->next, n++) { ifp->active = true; ifp->chain_last = true; } for (ofp = ofiles, ifp = ifiles, n = 0; ofp; ofp = ofp->next, n++) { ofp->ifp = ifp; if (last_in_group(group_size, n)) { ifp = ifp->next; ofp->chain_last = true; } else ofp->chain_last = false; } } assert(ifp == NULL); DPRINTF(3, "Input files"); for (ifp = ifiles; ifp; ifp = ifp->next) DPRINTF(3, "%p: chain_last=%d (%s)", ifp, ifp->chain_last, ifp->name); DPRINTF(3, "Output files"); for (ofp = ofiles; ofp; ofp = ofp->next) DPRINTF(3, "%p: chain_last=%d ifp=%p (%s)", ofp, ofp->chain_last, ofp->ifp, ofp->name); } int main(int argc, char *argv[]) { int max_fd = 0; struct sink_info *ofiles = NULL, *ofp; struct sink_info **oend = &ofiles; struct source_info *ifiles = NULL, *ifp; struct source_info **iend = &ifiles; struct source_info *front_ifp; /* To keep output sequential, never output past this one */ int ch; const char *progname = argv[0]; enum state state = read_ob; bool opt_memory_stats = false; bool opt_append = false; while ((ch = getopt(argc, argv, "ab:fIi:Mm:o:p:S:sTt:")) != -1) { switch (ch) { case 'a': opt_append = true; break; case 'b': buffer_size = (int)parse_size(progname, optarg); break; case 'f': use_tmp_file = true; break; case 'I': state = read_ib; break; case 'i': /* Specify input file */ ifp = new_source_info(optarg); if ((ifp->fd = open(optarg, O_RDONLY)) < 0) err(2, "Error opening %s", optarg); max_fd = MAX(ifp->fd, max_fd); non_block(ifp->fd, fp_name(ifp)); /* Add file at the end of the linked list */ *iend = ifp; iend = &ifp->next; break; case 'm': max_mem = parse_size(progname, optarg); break; case 'M': /* Provide memory use statistics on termination */ opt_memory_stats = true; break; case 'o': /* Specify output file */ ofp = new_sink_info(optarg); if ((ofp->fd = open(optarg, (opt_append ? O_APPEND : 0) | O_WRONLY | O_CREAT | O_TRUNC, 0666)) < 0) err(2, "Error opening %s", optarg); max_fd = MAX(ofp->fd, max_fd); non_block(ofp->fd, fp_name(ofp)); /* Add file at the end of the linked list */ *oend = ofp; oend = &ofp->next; break; case 'p': parse_permute(optarg); break; case 's': opt_scatter = true; break; case 'T': opt_tmp_dir = optarg; break; case 't': /* Record terminator */ /* We allow \0 as rt */ if (strlen(optarg) > 1) usage(progname); rt = *optarg; break; case '?': default: usage(progname); } } argc -= optind; argv += optind; if (argc) usage(progname); /* dgsh */ int j = 0; int noutputfds; int *outputfds; int ninputfds; int *inputfds; char *name; if (permute_n) { ninputfds = noutputfds = permute_n; name = "perm"; } else { char *in, *out; /* No stdin or stdout, if these have been specified via args */ ninputfds = ifiles ? 0 : -1; noutputfds = ofiles ? 0 : -1; /* Heuristic to determine name */ in = getenv("DGSH_IN"); if (in && *in == '0') in = NULL; out = getenv("DGSH_OUT"); if (out && *out == '0') out = NULL; if (in && !out) name = "cat"; else if (!in && out) name = "tee"; else name = "dgsh-tee"; } DPRINTF(3, "Calling negotiate in=%d out=%d", ninputfds, noutputfds); dgsh_negotiate(DGSH_HANDLE_ERROR, name, &ninputfds, &noutputfds, &inputfds, &outputfds); DPRINTF(3, "nin=%d nout=%d", ninputfds, noutputfds); assert(noutputfds >= 0); assert(ninputfds >= 0); if (permute_n && permute_n != ninputfds) errx(1, "The number of inputs %d is not equal to the specified permuted outputs %d", ninputfds, permute_n); if (permute_n && permute_n != noutputfds) errx(1, "The number of outputs %d is not equal to the specified permuted outputs %d", noutputfds, permute_n); for (j = 0; j < noutputfds; j++) { DPRINTF(3, "New ofp assigned fd %d", outputfds[j]); if (j == 0) { ofp = new_sink_info("standard output"); ofp->fd = STDOUT_FILENO; } else { ofp = new_sink_info(NULL); ofp->fd = outputfds[j]; } max_fd = MAX(ofp->fd, max_fd); non_block(ofp->fd, fp_name(ofp)); /* Add file at the end of the linked list */ *oend = ofp; oend = &ofp->next; } for (j = 0; j < ninputfds; j++) { DPRINTF(3, "New ifp assigned fd %d", inputfds[j]); if (j == 0) { ifp = new_source_info("standard input"); ifp->fd = STDIN_FILENO; } else { ifp = new_source_info(NULL); ifp->fd = inputfds[j]; } max_fd = MAX(ifp->fd, max_fd); non_block(ifp->fd, fp_name(ifp)); /* Add file at the end of the linked list */ *iend = ifp; iend = &ifp->next; } if (buffer_size > max_mem) errx(1, "Buffer size %d is larger than the program's maximum memory limit %lu", buffer_size, max_mem); if (opt_scatter && ifiles && ifiles->next) errx(1, "Scattering not supported with more than one input file"); if (opt_scatter && permute_n) errx(1, "Scattering and permutation cannot be used together"); if (ofiles == NULL) { /* Output to stdout */ ofp = new_sink_info("standard output"); ofp->fd = STDOUT_FILENO; max_fd = MAX(ofp->fd, max_fd); non_block(ofp->fd, fp_name(ofp)); ofp->next = ofiles; ofiles = ofp; } if (ifiles == NULL) { /* Input from stdin */ ifp = new_source_info("standard input"); ifp->fd = STDIN_FILENO; max_fd = MAX(ifp->fd, max_fd); non_block(ifp->fd, fp_name(ifp)); ifp->next = ifiles; ifiles = ifp; } /* We will handle SIGPIPE explicitly when calling write(2). */ signal(SIGPIPE, SIG_IGN); front_ifp = ifiles; chain_io_files(ifiles, ofiles, permute_n != 0); /* Copy source to sink without allowing any single file to block us. */ for (;;) { fd_set source_fds; fd_set sink_fds; int fd_set_count = 0; show_state(state); /* Set the fd's we're interested to read/write; close unneeded ones. */ FD_ZERO(&source_fds); FD_ZERO(&sink_fds); if (!reached_eof) switch (state) { case read_ib: for (ifp = front_ifp; ifp; ifp = ifp->next) if (!ifp->reached_eof) { FD_SET(ifp->fd, &source_fds); fd_set_count += 1; } break; case read_ob: for (ifp = front_ifp; ifp; ifp = ifp->next) if (ifp->active && !ifp->reached_eof) { FD_SET(ifp->fd, &source_fds); fd_set_count += 1; } break; default: break; } for (ofp = ofiles; ofp; ofp = ofp->next) if (ofp->active) { switch (state) { case read_ib: case read_ob: case drain_ob: DPRINTF(4, "Check active file[%s] pos_written=%ld pos_to_write=%ld", fp_name(ofp), (long)ofp->pos_written, (long)ofp->pos_to_write); if (ofp->pos_written < ofp->pos_to_write) { FD_SET(ofp->fd, &sink_fds); fd_set_count += 1; } break; case drain_ib: case write_ob: FD_SET(ofp->fd, &sink_fds); fd_set_count += 1; break; } } if (fd_set_count != 0) { /* Block until we can read or write. */ show_select_args("Entering select", &source_fds, ifiles, &sink_fds, ofiles, true); if (select(max_fd + 1, &source_fds, &sink_fds, NULL, NULL) < 0) err(3, "select"); show_select_args("Select returned", &source_fds, ifiles, &sink_fds, ofiles, false); /* Write to all file descriptors that accept writes. */ if (sink_write(ifiles, &sink_fds, ofiles) > 0) { /* * If we wrote something, we made progress on the * downstream end. Loop without reading to avoid * allocating excessive buffer memory. */ if (state == drain_ob) state = write_ob; continue; } } if (reached_eof) { int active_fds = 0; for (ofp = ofiles; ofp; ofp = ofp->next) if (ofp->active) { if (ofp->pos_written < ofp->pos_to_write) active_fds++; else { DPRINTF(3, "Retiring file %s pos_written=pos_to_write=%ld source_pos_read=%ld", fp_name(ofp), (long)ofp->pos_written, (long)ofp->ifp->source_pos_read); /* No more data to write; close fd to avoid deadlocks downstream. */ if (close(ofp->fd) == -1) err(2, "Error closing %s", fp_name(ofp)); ofp->active = false; } } if (active_fds == 0) { /* If no read possible, and no writes pending, terminate. */ if (opt_memory_stats) memory_stats(ifiles); return 0; } } /* * Note that we never reach this point after a successful write. * See the continue statement above. */ switch (state) { case read_ib: /* Read, if possible; set global reached_eof if all have reached it */ reached_eof = true; for (ifp = front_ifp; ifp; ifp = ifp->next) { if (FD_ISSET(ifp->fd, &source_fds)) switch (source_read(ifp)) { case read_eof: ifp->reached_eof = true; break; case read_oom: /* Cannot fullfill promise to never block source, so bail out. */ errx(1, "Out of memory with input-side buffering specified"); break; case read_ok: case read_again: break; } if (!ifp->reached_eof) reached_eof = false; } if (reached_eof) state = drain_ib; break; case read_ob: /* Read, from possible sources; set global reached_eof if all have reached it */ reached_eof = true; for (ifp = front_ifp; ifp; ifp = ifp->next) { if (!ifp->active) continue; if (FD_ISSET(ifp->fd, &source_fds)) switch (source_read(ifp)) { case read_eof: ifp->reached_eof = true; ifp->active = false; if (!ifp->chain_last) ifp->next->active = true; break; case read_again: break; case read_oom: /* Allow buffers to empty. */ state = drain_ob; break; case read_ok: state = write_ob; break; } if (!ifp->reached_eof) reached_eof = false; } if (reached_eof) state = drain_ib; break; case drain_ib: break; case drain_ob: if (reached_eof) state = write_ob; else state = read_ob; break; case write_ob: if (!reached_eof) state = read_ob; break; } } } ================================================ FILE: core-tools/src/dgsh-w.c ================================================ #include // assert() #include // M_PI, pow() #include // I, cexp(), cpow() #include // DPRINTF #include // atoi() #include // errx() #include // read(), write() #include // memcpy() #include "dgsh.h" #include "dgsh-debug.h" #if !defined(HAVE_CPOW) #include "../../unix-tools/cpow.c" #endif void read_number(int fd, long double *x, long double complex *xc) { char buf[sizeof(long double complex) + 5]; // \n\0 char real[sizeof(long double)]; char imag[sizeof(long double)]; int rd_size; // Read input: 2 float values rd_size = read(fd, buf, sizeof(buf)); if (rd_size == -1) err(1, "write failed"); DPRINTF(4, "Read %zu characters, long double size: %zu, long double complex size: %zu", rd_size, sizeof(long double), sizeof(long double complex)); if (rd_size == sizeof(long double)) { memcpy(x, buf, sizeof(*x)); DPRINTF(4, "Read input x: %.10Lf", *x); } else { sscanf(buf, "%s %s", real, imag); *xc = atof(real) + atof(imag)*I; DPRINTF(4, "##xc: %.10f + %.10fi (read %zu characters)\n", creal(*xc), cimag(*xc), rd_size); } } void write_number(int fd, long double complex y) { char buf[sizeof(long double complex)]; int wr_size; // Write output: 2 float values memset(buf, 0, sizeof(buf)); snprintf(buf, sizeof(buf), "%.10f %.10fi", creal(y), cimag(y)); DPRINTF(4, "##buf(y): %s, len: %d", buf, strlen(buf)); wr_size = write(fd, buf, sizeof(buf)); if (wr_size == -1) err(1, "write failed"); DPRINTF(4, "##y: %.10f + %.10fi (wrote %zu characters)\n", creal(y), cimag(y), wr_size); } int main(int argc, char** argv) { int noutputfds = 2; int *outputfds = NULL; int ninputfds = 2; int *inputfds = NULL; long double x1 = -1.0, x2 = -1.0; long double complex xc1, xc2; long double complex y1, y2, w, wmn; size_t wr_size; char negotiation_title[10]; int s; // stage (stage=1,2,3) int m; // 2^stage int n; // nth root of unity assert(argc == 3); s = atoi(argv[1]); m = pow(2, s); n = atoi(argv[2]); snprintf(negotiation_title, sizeof(negotiation_title), "%s %s %s", argv[0], argv[1], argv[2]); dgsh_negotiate(DGSH_HANDLE_ERROR, negotiation_title, &ninputfds, &noutputfds, &inputfds, &outputfds); assert(ninputfds == 2); assert(noutputfds == 2); read_number(inputfds[0], &x1, &xc1); read_number(inputfds[1], &x2, &xc2); // Calculate w = 2 * M_PI * I / m; wmn = cpow(cexp(w), n); DPRINTF(4, "w: %.10f + %.10fi", creal(w), cimag(w)); DPRINTF(4, "m: %d, n: %d, wmn: %.10f + %.10fi\n", m, n, creal(wmn), cimag(wmn)); if (x1 == -1.0 && x2 == -1.0) { y1 = xc1 + wmn * xc2; y2 = xc1 - wmn * xc2; } else { y1 = x1 + wmn * x2; y2 = x1 - wmn * x2; } write_number(outputfds[0], y1); write_number(outputfds[1], y2); return 0; } ================================================ FILE: core-tools/src/dgsh-wrap.1 ================================================ .TH DGSH-WRAP 1 "18 August 2017" .\" .\" (C) Copyright 2016-2017 Diomidis Spinellis. All rights reserved. .\" .\" Licensed under the Apache License, Version 2.0 (the "License"); .\" you may not use this file except in compliance with the License. .\" You may obtain a copy of the License at .\" .\" http://www.apache.org/licenses/LICENSE-2.0 .\" .\" Unless required by applicable law or agreed to in writing, software .\" distributed under the License is distributed on an "AS IS" BASIS, .\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .\" See the License for the specific language governing permissions and .\" limitations under the License. .\" .SH NAME dgsh-wrap \- allow any program to participate in an dgsh pipeline .SH SYNOPSIS \fBdgsh-wrap\fP [\fB-S\fP] [\fB-i\fP \fB0\fP|\fBa\fP] [\fB-o\fP \fB0\fP|\fBa\fP] [\fB-eIO\fP] \fIprogram\fP [\fIprogram-arguments\fP ...] #!/usr/libexec/dgsh/\fBdgsh-wrap\fP \fB-s\fP [\fB-i\fP \fB0\fP|\fBa\fP] [\fB-o\fP \fB0\fP|\fBa\fP] [\fB-eIO\fP] [\fIprogram-arguments\fP ...] \fBdgsh-wrap\fP [\fB-Ss\fP] \fB-x\fP \fIprogram\fP [\fIprogram-arguments\fP ...] .SH DESCRIPTION \fIdgsh-wrap\fP takes as arguments an absolute path or the name of a program to execute and its arguments. It will participate in the \fIdgsh\fP negotiation process, and then execute the specified program connected to the negotiated input and output pipes. If the program is not specified through an absolute path, it will be executed by searching the existing path, excluding from it elements ending in \fIdgsh\fP (where programs already wrapped with \fIdgsh-wrap\fP may reside). .PP Arguments specified as \fI<|\fP are presented as additional input channels and arguments specified as \fI>|\fP are presented as additional output channels. Both are suitably replaced by named file descriptor paths when the command is invoked. .PP In the context of a \fIdgsh\fP process graph, \fIdgsh(1)\fP automatically invokes \fIdgsh-wrap\fP to allow non-dgsh compatible commands to participate in the negotiation procedure. Furthermore, the \fIdgsh\fP installation process sets up many POSIX programs wrapped with \fIdgsh-wrap\fP in order to communicate their particular I/O requirements. .SH OPTIONS .IP "\fB\-e\fP Replace \fI<|\fP and \fI>|\fP strings embedded within arguments, with names of input file descriptor paths. By default, only standalone arguments are thus replaced. .IP "\fB\-i\fP \fB0\fP|\fBa\fP Specify the wrapped program's number of input channels. The \fB0\fP character specifies that the program does not read any input. The \fBa\fP character specifies that the program can read an arbitrary number of input streams; these will be automatically supplied by \fIdgsh-wrap\fP as file descriptor command line arguments. .IP "\fB\-I\fP Do not include the standard input to the command line arguments, when replacing \fI<|\fP arguments with names of input file descriptor paths. This will result in the standard input becoming available to the command as its standard input, rather than as a command line argument. Without this option, the first path with be \fI/dev/fd/0\fP. When this option is given, the program will require one input channel more than those specified by the \fI<|\fP arguments. .IP "\fB\-o\fP \fB0\fP|\fBa\fP Specify the wrapped program's number of output channels. The \fB0\fP character specifies that the program does not produce any output. The \fBa\fP character specifies that the program can write to an arbitrary number of output streams; these will be automatically supplied by \fIdgsh-wrap\fP as file descriptor command line arguments. .IP "\fB\-O\fP Do not include the standard output to the command line arguments, when replacing \fI>|\fP arguments with names of output file descriptor paths. This will result in the standard output becoming available to the command as its standard output, rather than as a command line argument. Without this option the first path with be \fI/dev/fd/1\fP. When this option is given, the program will require one output channel more than those specified by the \fI>|\fP arguments. .IP "\fB\-S\fP Process flags as a shebang-invoked (\fI#!\fP) interpreter using an invocation-supplied program name. This will change the argument processing in two ways. First, it will cause the arguments being specified on the shebang line to be properly parsed as separate arguments by splitting them on whitespace. (Most operating systems pass any of the line's arguments as a single string to the process.) Second, it will remove from the arguments the kernel-supplied path to the script that invoked \fIdgsh-wrap\fP. (The script path is not needed, because the program to wrap is specified as an argument.) If this flag is specified, it must be the first command-line argument. .IP "\fB\-s\fP Process flags as a shebang-invoked (\fI#!\fP) interpreter using an operating system-supplied program name. This will change the argument processing in two ways. First, it will split arguments in the shebang line, in the same way as the \fI-S\fP flag. Second, it will convert the kernel-supplied absolute path to the script that invoked \fIdgsh-wrap\fP, into a command name. If the name of the script is the same as the name of a command to be wrapped, this path can be used to derive the name of the program to execute, thus removing the need to supply the name of the program in the invocation line. .IP "\fB\-x\fP Execute the specified command and arguments, without performing \fIdgsh\fP negotiation on its behalf. This is useful when the specified command, \fIA\fP, is not \fIdgsh\fP-compatible, but it will in turn execute, \fIB\fP, a \fIdgsh\fP-compatible command. With this flag the \fIdgsh\fP will recognize the invocation of \fIA\fP as an invocation of a \fIdgsh\fP-compatible command (due to the wrapping), and will not attempt to autowrap it as a filter and thereby override a more sophisticated \fIdgsh\fP interface that \fIB\fP would present. .SH EXAMPLES .PP The following examples are given only to illustrate the command's functionality. Note that most of the wrappings shown here are either performed automatically or are not required, because corresponding commands with built-in \fIdgsh\fP support are already provided. .PP Wrap the \fIecho\fP command, specifying that it accepts no input. .ft C .ps -1 .nf dgsh-wrap -i 0 echo hi .fi .ps +1 .ft P .PP Wrap the \fIcp\fP command, specifying that it does not perform any I/O. .ft C .ps -1 .nf dgsh-wrap -i 0 -o 0 cp src-file dest-file .fi .ps +1 .ft P .PP Wrap the \fIpaste\fP command, supplying its two arguments. .ft C .ps -1 .nf dgsh-wrap /usr/bin/paste "<|" "<|" .fi .ps +1 .ft P .PP Wrap the \fIpaste\fP command, with an invocation that processes the standard input and uses an additional input argument. .ft C .ps -1 .nf dgsh-wrap -I /usr/bin/paste - "<|" .fi .ps +1 .ft P .PP Wrap the \fIpaste\fP command, so that it will process all input channels provided to it. .ft C .ps -1 .nf dgsh-wrap -i a /usr/bin/paste .fi .ps +1 .ft P .PP A wrapped version of the \fIuname\fP command can be created with an interpreter invocation file containing just the following line. In contrast to a shell script, no shell is ever launched. .ft C .ps -1 .nf #!/usr/libexec/dgsh/dgsh-wrap -S -d /bin/uname .fi .ps +1 .ft P .PP Even simpler, a wrapped version of the \fIuname\fP command can be created if a) the interpreter invocation file is named \fIuname\fP, b) it resides in a directory named \fIdgsh\fP (so that it will be excluded from the subsequently used search path), and c) it contains the following line. .ft C .ps -1 .nf #!/usr/libexec/dgsh/dgsh-wrap -s -d .fi .ps +1 .ft P .PP Trace the invocation of the \fItee\fP command, presenting to the shell \fIsrtace\fP as a \fIdgsh\fP-compatible command with the capabilities of \fItee\fP. .ft C .ps -1 .nf dgsh-wrap -x strace tee .fi .ps +1 .ft P .SH "SEE ALSO" \fIdgsh\fP(1), \fIdgsh-negotiate\fP(3) .SH AUTHOR Diomidis Spinellis \(em ================================================ FILE: core-tools/src/dgsh-wrap.c ================================================ /* * Copyright 2016-2017 Diomidis Spinellis * * Wrap any command to participate in the dgsh negotiation * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ /* * Examples: * dgsh-wrap -i 0 yes | fsck * tar cf - / | dgsh-wrap -o 0 dd of=/dev/st0 * ls | dgsh-wrap /usr/bin/sort -k5n | more */ #define _GNU_SOURCE /* For asprintf() */ #include #include #include #include #include #include #include #include #include "dgsh.h" #include "dgsh-debug.h" /* DPRINTF(4, ) */ /* Determine if the OS splits shebang argument or not */ #if __APPLE__ #include "TargetConditionals.h" #if TARGET_OS_MAC #define OS_SPLITS_SHEBANG_ARGS 1 #endif #endif static void usage(void) { fputs("Usage:\tdgsh-wrap [-S] [-i 0|a] [-o 0|a] [-eIO] program [program-arguments ...]\n" "\tdgsh-wrap -s [-i 0|a] [-o 0|a] [-eIO] [program-arguments ...]\n" "-e\t" "Process <| and >| embedded in arguments\n" "-i 0|a\t" "Process no (0) or arbitrary (a) input channels\n" "-I\t" "Do not provide standard input as a <| arg\n" "-o 0|a\t" "Process no (0) or arbitrary (a) output channels\n" "-O\t" "Do not provide standard output as a >| arg\n" "-S\t" "Process flags and program as a #! interpreter\n" "-s\t" "Process flags as a #! interpreter\n" "\t" "(-S or -s must be the first flag of shebang line)\n" "-x\t" "Wrap a non-dgsh command that will exec a dgsh one\n", stderr); exit(1); } static void * xmalloc(size_t size) { void *r = malloc(size); if (r == NULL) err(1, "malloc out of memory"); return r; } char * xstrdup(const char *s) { void *r = strdup(s); if (r == NULL) err(1, "stdup out of memory"); return r; } static void * xrealloc(void *ptr, size_t size) { void *r = realloc(ptr, size); if (r == NULL) err(1, "realloc out of memory"); return r; } /* * Remove from the PATH environment variable an entry with the specified string */ static void remove_from_path(const char *string) { char *start, *end, *path, *strptr; path = getenv("PATH"); if (!path) return; path = xstrdup(path); if (!path) err(1, "Error allocating path copy"); strptr = strstr(path, string); if (!strptr) return; /* Find start of this path element */ for (start = strptr; start != path && *start != ':'; start--) ; /* Find end of this path element */ for (end = strptr; *end && *end != ':'; end++) ; /* * At this point: * start can point to : or path, * end can point to : or \0. * Work through all cases. */ if (*end == '\0') *start = '\0'; else if (*start == ':') memmove(start, end, strlen(end)); else /* first element, followed by another */ memmove(start, end + 1, strlen(end + 1)); if (setenv("PATH", path, 1) != 0) err(1, "Setting path"); free(path); } static void dump_args(int argc, char *argv[]) { int i; for (i = 0; i <= argc; i++) DPRINTF(4, "argv[%d]: [%s]", i, argv[i]); } /* * On operating systems that pass unsplit all the #! line arguments * to the interpreter, process the arguments of a #! invocation to make * them equivalent to a command-line one. * This entails tokenizing argv[1], which contains all the #! line * after the name of the interpreter. * * Example: * Input arguments: * argv[0]: /usr/local/libexec/dgsh/dgsh-wrap * argv[1]: -S -d /bin/uname * argv[2]: /usr/local/libexec/dgsh/uname * argv[3]: -s * argv[4]: NULL * Output arguments: * argv[0]: /usr/local/libexec/dgsh/dgsh-wrap * argv[1]: -S * argv[2]: -d * argv[3]: /bin/uname * argv[4]: /usr/local/libexec/dgsh/uname * argv[5]: -s * argv[6]: NULL */ static void split_argv(int *argcp, char ***argvp) { int argc = *argcp; char **argv = *argvp; /* Tokenize argv[1] into multiple arguments */ argv = xmalloc((argc + 1) * sizeof(char *)); memcpy(argv, *argvp, (argc + 1) * sizeof(char *)); int i = 1; const char *delim = " \t"; char *p = strtok(xstrdup(argv[1]), delim); assert(p != NULL); /* One arg (-s or -S) is guaranteed to exist */ for (;;) { argv[i] = p; p = strtok(NULL, delim); if (p == NULL) break; /* Make room for new string */ argc += 1; argv = xrealloc(argv, (argc + 1) * sizeof(char *)); memmove(argv + i + 2, argv + i + 1, (argc - i - 1) * sizeof(char *)); i++; } *argvp = argv; *argcp = argc; DPRINTF(4, "Arguments after split_argv"); dump_args(*argcp, *argvp); } /* * -S: remove OS-supplied script path name * Input arguments: * argv[0]: /usr/local/libexec/dgsh/dgsh-wrap * argv[1]: -S -d /bin/uname * argv[2]: /usr/local/libexec/dgsh/uname * argv[3]: -s * Arguments at this point: * argv[0]: /usr/local/libexec/dgsh/dgsh-wrap * argv[1]: -S * argv[2]: -d * argv[3]: /bin/uname <- argv[optind] * argv[4]: /usr/local/libexec/dgsh/uname * argv[5]: -s * Result arguments: * argv[0]: /usr/local/libexec/dgsh/dgsh-wrap * argv[1]: -S * argv[2]: -d * argv[3]: /bin/uname * argv[4]: -s */ static void remove_os_script_path(char **argv, int *argc, int optind) { memmove(argv + optind + 1, argv + optind + 2, (*argc - optind) * sizeof(char *)); *argc -= 1; argv[*argc] = NULL; } /* Remove absolute path from specified string * Example: * -s: Remove absolute path from argv[optind] * Input arguments: * argv[0]: /usr/local/libexec/dgsh/dgsh-wrap * argv[1]: -s -d * argv[2]: /usr/local/libexec/dgsh/uname * argv[3]: -s * Arguments at this point: * argv[0]: /usr/local/libexec/dgsh/dgsh-wrap * argv[1]: -s * argv[2]: -d * argv[3]: /usr/local/libexec/dgsh/uname <- argv[optind] * argv[4]: -s * Result arguments: * argv[0]: /usr/local/libexec/dgsh/dgsh-wrap * argv[1]: -s * argv[2]: -d * argv[3]: uname * argv[4]: -s */ static void remove_absolute_path(char *s) { char *p = strrchr(s, '/'); if (p) memmove(s, p + 1, strlen(p) + 1); } /* * Replace an instance of the string special (e.g. "<|") embedded in arg * with /dev/fd/N, where N is the integer pointed * by fdptr, and increase fdptr to point to the next integer. * Return true if a replacement was made, false if not. */ static bool process_embedded_io_arg(char **arg, const char *special, int **fdptr) { char *p = strstr(*arg, special); if (p == NULL) return false; *p = 0; char *before = *arg; char *after = p + strlen(special); if (asprintf(arg, "%s/dev/fd/%d%s", before, **fdptr, after) == -1) err(1, "asprintf out of memory"); (*fdptr)++; return true; } /* * Replace an instance of the string special (e.g. "<|") matching an arg * with /dev/fd/N, where N is the integer pointed * by fdptr, and increase fdptr to point to the next integer. * If special is null, then the replacement is always made. */ static void process_standalone_io_arg(char **arg, const char *special, int **fdptr) { if (special != NULL && strcmp(*arg, special) != 0) return; if (asprintf(arg, "/dev/fd/%d", **fdptr) == -1) err(1, "asprintf out of memory"); (*fdptr)++; } /* * Increment the channels specified by the given variable. * Ensure that the corresponding variable is not * already set to an arbitrary number of channels. */ static void increment_channels(int *var) { if (*var == -1) { fputs("I/O channel arguments cannot be combined with an arbitrary I/O file specification\n", stderr); exit(1); } (*var)++; } int main(int argc, char *argv[]) { int nflags = 0; bool negotiation_flags = false; int ninputs = 1, noutputs = 1; bool xflag = false; int ch, i; char *p; char *debug_level; char *guest_program_name; /* Option-dependent flags */ bool program_from_os = false, program_supplied = false; bool embedded_args = false; /* Pass stdin/stdout as a command-line argument */ bool stdin_as_arg = true, stdout_as_arg = true; bool supply_input_args = false, supply_output_args = false; debug_level = getenv("DGSH_DEBUG_LEVEL"); if (debug_level) dgsh_debug_level = atoi(debug_level); /* Preclude recursive wrapping */ DPRINTF(4, "PATH before: [%s]", getenv("PATH")); remove_from_path("libexec/dgsh"); DPRINTF(4, "PATH after: [%s]", getenv("PATH")); DPRINTF(4, "Initial arguments"); dump_args(argc, argv); /* Check for #! (shebang) interpreter argument processing */ #if !defined(OS_SPLITS_SHEBANG_ARGS) if (argc >= 2 && argv[1][0] == '-' && tolower(argv[1][1]) == 's') split_argv(&argc, &argv); #endif /* * The + argument to getopt causes it on glibc to stop processing on * first non-flag argument. * Therefore, adjust argc, argv on entry and optind on exit. */ while ((ch = getopt(argc, argv, "+ei:Io:OSsx")) != -1) { DPRINTF(4, "getopt switch=%c", ch); switch (ch) { case 'i': nflags++; negotiation_flags = true; if (strcmp(optarg, "0") == 0) ninputs = 0; else if (strcmp(optarg, "a") == 0) { ninputs = -1; supply_input_args = true; } else usage(); break; case 'e': embedded_args = true; negotiation_flags = true; nflags++; break; case 'I': stdin_as_arg = false; negotiation_flags = true; nflags++; break; case 'o': nflags++; negotiation_flags = true; if (strcmp(optarg, "0") == 0) noutputs = 0; else if (strcmp(optarg, "a") == 0) { noutputs = -1; supply_output_args = true; } else usage(); break; case 'O': stdout_as_arg = false; negotiation_flags = true; nflags++; break; case 'S': /* Complain this is not the first flag */ if (nflags) { fputs("-S must be the first provided flag\n", stderr); usage(); } nflags++; program_supplied = true; break; case 's': /* Complain this is not the first flag */ if (nflags) { fputs("-s must be the first provided flag\n", stderr); usage(); } nflags++; program_from_os = true; break; case 'x': xflag = true; break; case '?': default: usage(); } } DPRINTF(3, "After getopt: ninputs=%d, noutputs=%d optind=%d argv[optind]=%s", ninputs, noutputs, optind, argv[optind]); DPRINTF(3, "program_supplied=%d", program_supplied); if (optind >= argc) usage(); if (xflag && negotiation_flags) { fputs("-x cannot be combined with I/O specifications\n", stderr); usage(); } /* * Process argv[2], which is the name of the script * supplied by the kernel to the interpreter, i.e. * the name of the program we are being executed as. */ if (program_supplied && argc > optind) remove_os_script_path(argv, &argc, optind); else if (program_from_os) remove_absolute_path(argv[optind]); DPRINTF(4, "Arguments after processing program name (optind=%d)", optind); dump_args(argc, argv); if (xflag) { /* * Execute (non-dgsh) command, which will execute a dgsh * command, which will negotiate on our behalf */ execvp(argv[optind], argv + optind); err(1, "Unable to execute %s", argv[optind]); return 1; } /* Obtain guest program name (without path) */ guest_program_name = xstrdup(argv[optind]); remove_absolute_path(guest_program_name); DPRINTF(4, "guest_program_name: %s", guest_program_name); /* * Adjust ninputs and noutputs by special arguments * "<|" and ">|", which mean input from or output to * /dev/fd/N */ DPRINTF(4, "embedded_args=%d", embedded_args); for (i = optind + 1; i < argc; i++) { if (embedded_args) { for (p = argv[i]; p = strstr(p, "<|"); p += 2) increment_channels(&ninputs); for (p = argv[i]; p = strstr(p, ">|"); p += 2) increment_channels(&noutputs); } else { if (strcmp(argv[i], "<|") == 0) increment_channels(&ninputs); if (strcmp(argv[i], ">|") == 0) increment_channels(&noutputs); } } /* * Adjust for the default implicit I/O channel. * E.g. if two <| are specified, ninputs will be 3 at this point, * whereas we want it to be 2. */ if (stdin_as_arg && ninputs > 1) ninputs--; if (stdout_as_arg && noutputs > 1) noutputs--; /* Participate in negotiation */ DPRINTF(3, "calling negotiate with ninputs=%d noutputs=%d", ninputs, noutputs); int *input_fds = NULL, *output_fds = NULL; dgsh_negotiate(DGSH_HANDLE_ERROR, guest_program_name, &ninputs, &noutputs, &input_fds, &output_fds); /* * Substitute special arguments "<|" and ">|" with or add file descriptor * paths /dev/fd/N using the fds received from negotiation. */ int *inptr = stdin_as_arg ? input_fds : input_fds + 1; if (supply_input_args) { if (!stdin_as_arg) ninputs--; /* Create space for arguments to add */ char **nargv = xmalloc((argc + ninputs + 1) * sizeof(char *)); memcpy(nargv, argv, argc * sizeof(char *)); memset(argv + argc, 0, (ninputs + 1) * sizeof(char *)); argv = nargv; /* Add arguments */ for (i = argc; i < argc + ninputs; i++) process_standalone_io_arg(&argv[i], NULL, &inptr); argc += ninputs; argv[argc] = NULL; } else { for (i = optind + 1; i < argc; i++) { if (embedded_args) while (process_embedded_io_arg(&argv[i], "<|", &inptr)) ; else process_standalone_io_arg(&argv[i], "<|", &inptr); } } int *outptr = stdout_as_arg ? output_fds : output_fds + 1; if (supply_output_args) { if (!stdout_as_arg) noutputs--; /* Create space for arguments to add */ char **nargv = xmalloc((argc + noutputs + 1) * sizeof(char *)); memcpy(nargv, argv, argc * sizeof(char *)); memset(argv + argc, 0, (noutputs + 1) * sizeof(char *)); argv = nargv; /* Add arguments */ for (i = argc; i < argc + noutputs; i++) process_standalone_io_arg(&argv[i], NULL, &outptr); argc += noutputs; argv[argc] = NULL; } else { for (i = optind + 1; i < argc; i++) { if (embedded_args) while (process_embedded_io_arg(&argv[i], ">|", &outptr)) ; else process_standalone_io_arg(&argv[i], ">|", &outptr); } } DPRINTF(4, "Arguments to execvp after substitung <| and >|"); dump_args(argc - optind, argv + optind); /* Execute command */ execvp(argv[optind], argv + optind); err(1, "Unable to execute %s", argv[optind]); return 1; } ================================================ FILE: core-tools/src/dgsh-writeval.1 ================================================ .TH DGSH-WRITEVAL 1 "21 March 2013" .\" .\" (C) Copyright 2013 Diomidis Spinellis. All rights reserved. .\" .\" Licensed under the Apache License, Version 2.0 (the "License"); .\" you may not use this file except in compliance with the License. .\" You may obtain a copy of the License at .\" .\" http://www.apache.org/licenses/LICENSE-2.0 .\" .\" Unless required by applicable law or agreed to in writing, software .\" distributed under the License is distributed on an "AS IS" BASIS, .\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .\" See the License for the specific language governing permissions and .\" limitations under the License. .\" .SH NAME dgsh-writeval \- write values to a data store .SH SYNOPSIS \fBdgsh-writeval\fP [\fB\-l\fP \fIlength\fP | \fB-t\fP \fIcharacter\fP ] [\fB\-b\fP \fIn\fP] [\fB\-e\fP \fIn\fP] [\fB\-u\fP \fIunit\fP] \fB\-s\fP \fIpath\fP .SH DESCRIPTION \fIdgsh-writeval\fP will read values from its standard input and make them available to other processes for reading through the specified Unix domain socket. Thus this process acts as a data store: it reads a series of values (think of them as assignments), and provides a way to read the store's current value (from the socket). By default \fIdgsh-writeval\fP will store the last value (line or data block) it reads. However, the default behavior can be modified through options so that it stores a specified window of the stream it processes. .PP \fIdgsh-writeval\fP is normally executed from within \fIdgsh\fP-generated scripts, rather than through end-user commands. This manual page serves mainly to document its operation and the flags that can be used in \fIdgsh\fP scripts when writing into stores. .SH OPTIONS .IP "\fB\-b\fP \fIn\fP" Store records beginning in a window \fIn\fP units away from the input's end. By default this value is 1. .IP "\fB\-e\fP \fIn\fP" Store records ending in a window \fIn\fP units away from the input's end. By default this value is 0. .IP "\fB\-l\fP \fIlen\fP" Process fixed-width \fIlen\fP-sized records. By default \fIdgsh-writeval\fP will process newline-terminated records. .IP "\fB\-s\fP \fIpath\fP" This mandatory option must be used to specify the path of the Unix-domain socket \fIdgsh-writeval\fP will create. This is specified as a normal Unix file path, e.g. \fC/tmp/myvalue\fP. .IP "\fB\-t\fP \fIchar\fP" Specify the record termination character to be \fIchar\fP. This is the newline by default. .IP "\fB\-u\fP \fIunit\fP" Specify the unit of the window boundaries given in the \fC-b\fP and \fC-e\fP options. The following units can be specified, using single-character identifiers. .RS .IP "\fBs\fP seconds .IP "\fBm\fP minutes .IP "\fBh\fP hours .IP "\fBd\fP days .IP "\fBr\fP records (this is the default value) .RE .SH "SEE ALSO" \fIdgsh\fP(1), \fIdgsh-readval\fP(1) .SH AUTHOR Diomidis Spinellis \(em ================================================ FILE: core-tools/src/dgsh-writeval.c ================================================ /* * Copyright 2013 Diomidis Spinellis * * Read values from its standard input and make them available to other * processes for reading through the specified Unix domain socket. * Thus, this process acts in effect as a data store: it reads a series of * values (think of them as assignements) and provides a way to read the * store's current value (from the socket). * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "dgsh.h" #include "kvstore.h" #include "dgsh-debug.h" #include "minmax.h" #ifdef DEBUG /* Small buffer size to catch errors with data spanning buffers */ #define BUFFER_SIZE 5 #else /* PIPE_BUF is a reasonable size heuristic. */ #define BUFFER_SIZE PIPE_BUF #endif /* User options start here */ /* Record terminator */ static char rt = '\n'; /* Record length; 0 if we use a record terminator */ static int rl = 0; /* True if the begin and end are specified using a time window */ static bool time_window; /* * Specified response record. * This is specified using reverse iterators (counted from the end of the stream). * The _rbegin is inclusive, _rend is exclusive * Examples: * To get the last record use the range rbegin = 0 rend = 1 * To get 5 records starting 10 records away from the end * use the range rbegin = 10 rend = 15 */ static union { struct timeval t; /* Used if time_window is true */ int r; /* Used if time_window is false */ double d; /* Used when parsing */ } record_rbegin, record_rend; /* User options end here */ /* True once we reach the end of file on standard input */ static bool reached_eof; /* True if a complete record (ending in rt) is available */ static bool have_record; /* Queue (doubly linked list) of buffers used for storing the last read record */ struct buffer { struct buffer *next; struct buffer *prev; int size; /* Actual number of bytes stored */ struct timeval timestamp; /* Time the buffer was read */ long long record_count; /* Total number of complete records read (including this buffer) (0-based ordinal of first record not in buffer) */ long long byte_count; /* Total number of bytes read (including this buffer) */ char data[BUFFER_SIZE]; }; static struct buffer *head, *tail; /* The oldest buffer whose contents are still being written to a socket. */ static struct buffer *oldest_buffer_being_written; /* A pointer to a character stored in a buffer */ struct dpointer { struct buffer *b; /* The buffer */ int pos; /* The position within the buffer */ }; /* The last complete record read */ static struct dpointer current_record_begin, current_record_end; /* The clients we're talking to */ struct client { int fd; struct dpointer write_begin; /* Start of data for next write */ struct dpointer write_end; /* End of data to write */ enum { s_inactive, /* Free (unused or closed) */ s_read_command, /* Waiting for a command (Q or R) to be read */ s_send_current, /* Waiting for the current value to be written */ s_send_current_nblk, /* Non-blocking: waiting for the current or empty value to be written */ s_send_last, /* Waiting for the last (before EOF) value to be written */ s_sending_response, /* A response is being written */ s_wait_close, /* Wait for the client to close the connection */ } state; }; #define MAX_CLIENTS 64 static struct client clients[MAX_CLIENTS]; static const char *program_name; static const char *socket_path; /* * Increment dp by one byte. * If no more bytes are available return false * leaving pos to point one byte past the last available one */ static bool dpointer_increment(struct dpointer *dp) { DPRINTF(4, "%p pos=%d", dp->b, dp->pos); dp->pos++; if (dp->pos == dp->b->size) { if (!dp->b->next) return false; dp->b = dp->b->next; dp->pos = 0; } DPRINTF(4, "return %p pos=%d", dp->b, dp->pos); return true; } /* Decrement dp by one byte. Return false if no more bytes are available */ static bool dpointer_decrement(struct dpointer *dp) { DPRINTF(4, "%p pos=%d", dp->b, dp->pos); dp->pos--; if (dp->pos == -1) { if (!dp->b->prev) { dp->pos++; return false; } dp->b = dp->b->prev; dp->pos = dp->b->size - 1; } DPRINTF(4, "return %p pos=%d", dp->b, dp->pos); return true; } /* * Add to dp the specified number of bytes. * Return true of OK. * If not enough bytes are available return false * and set dp to point to the last byte in the buffer. */ static bool dpointer_add(struct dpointer *dp, int n) { DPRINTF(4, "%p pos=%d n=%d", dp->b, dp->pos, n); while (n > 0) { int add = MIN(dp->b->size - dp->pos, n); n -= add; dp->pos += add; if (dp->pos == dp->b->size) { if (!dp->b->next) return false; dp->b = dp->b->next; dp->pos = 0; } } DPRINTF(4, "return %p pos=%d", dp->b, dp->pos); return true; } /* * Subtract from dp the specified number of bytes. * Return true of OK. * If not enough bytes are available return false * and set dp to point beyond the first available byte. */ static bool dpointer_subtract(struct dpointer *dp, int n) { DPRINTF(4, "%p pos=%d n=%d", dp->b, dp->pos, n); while (n > 0) { int subtract = MIN((dp->pos + 1) - 0, n); n -= subtract; dp->pos -= subtract; if (dp->pos == -1) { if (!dp->b->prev) return false; dp->b = dp->b->prev; dp->pos = dp->b->size - 1; } } DPRINTF(4, "return %p pos=%d", dp->b, dp->pos); return true; } /* * Move back dp the specified number of rt-terminated records. * Postcondition: dp will point at the beginning of * a record. * Return true if OK, false if not enough records are available * Example: to move back over one complete record, the function will * encounter two rts, and return with pos set immediately after the * second one. */ static bool dpointer_move_back(struct dpointer *dp, int n) { DPRINTF(4, "%p pos=%d (size=%d, prev=%p) n=%d", dp->b, dp->pos, dp->b->size, dp->b->prev, n); for (;;) { if (dpointer_decrement(dp)) { if (dp->b->data[dp->pos] == rt && --n == -1) { dpointer_increment(dp); DPRINTF(4, "return %p pos=%d", dp->b, dp->pos); return true; } } else { if (--n == -1) { DPRINTF(4, "(at begin) returns: %p pos=%d", dp->b, dp->pos); return true; } else return false; /* Not enough records available */ } } } /* * Move forward dp the specified number of rt-terminated records. * Postcondition: dp will point past the end of a record. * Return true if OK, false if not enough records are available */ static bool dpointer_move_forward(struct dpointer *dp, int n) { DPRINTF(4, "%p pos=%d (size=%d, next=%p) n=%d", dp->b, dp->pos, dp->b->size, dp->b->next, n); /* Cover the case where we are already at the beginning of the record */ if (!dpointer_decrement(dp)) { DPRINTF(4, "return %p pos=%d (at head)", dp->b, dp->pos); return true; } for (;;) { if (dp->b->data[dp->pos] == rt && --n == -1) { dpointer_increment(dp); DPRINTF(4, "return %p pos=%d", dp->b, dp->pos); return true; } if (!dpointer_increment(dp)) return false; /* Not enough records available */ } } /* Return the oldest of the two buffers (the one that comes first in the list) */ static struct buffer * oldest_buffer(struct buffer *a, struct buffer *b) { struct buffer *bp; if (a == NULL) return b; else if (b == NULL) return a; for (bp = head; bp ; bp = bp->next) if (bp == a) return a; else if (bp == b) return b; assert(0); } /* * Update oldest_buffer_being_written according to the * buffers used by all clients sending a response. */ static void update_oldest_buffer(void) { int i; oldest_buffer_being_written = NULL; for (i = 0; i < MAX_CLIENTS; i++) if (clients[i].state == s_sending_response) oldest_buffer_being_written = oldest_buffer(oldest_buffer_being_written, clients[i].write_begin.b); DPRINTF(4, "Oldest buffer beeing written is %p", oldest_buffer_being_written); } /* Free buffers preceding in position the used buffer */ static void free_unused_buffers_by_position(struct buffer *used) { struct buffer *b, *bnext; for (b = head; b; b = bnext) { if (b == used || b == oldest_buffer_being_written) { head = b; b->prev = NULL; DPRINTF(4, "After freeing buffer(s) head=%p tail=%p", head, tail); return; } bnext = b->next; DPRINTF(4, "Freeing buffer %p prev=%p next=%p", b, b->prev, b->next); free(b); } /* Should have encountered used along the way. */ assert(0); } /* Free buffers preceding in time (older than) the used buffer */ static void free_unused_buffers_by_time(struct timeval *used) { struct buffer *b; DPRINTF(4, "Free buffers older than %lld.%06d", (long long)used->tv_sec, (int)used->tv_usec); /* Find first useful record */ for (b = head; b; b = b->next) if (timercmp(&b->timestamp, used, >=) || b == oldest_buffer_being_written) break; assert(b); /* Should have encountered used along the way. */ DPRINTF(4, "First used buffer is %p", b); /* Must now leave another record in case a record extends backward */ if (rl) { int n = rl; do { b = b->prev; } while (b && (n -= b->size) > 0); } else { do { b = b->prev; } while (b && !memchr(b->data, rt, b->size)); } DPRINTF(4, "After extending back %p", b); if (b) free_unused_buffers_by_position(b); } /* Return the content length of the client's buffer */ static unsigned int content_length(struct client *c) { struct buffer *bp; unsigned int length; if (c->write_begin.b == c->write_end.b) length = c->write_end.pos - c->write_begin.pos; else { length = c->write_begin.b->size - c->write_begin.pos; for (bp = c->write_begin.b->next; bp && bp != c->write_end.b; bp = bp->next) length += bp->size; length += c->write_end.pos; } DPRINTF(4, "return %u", length); return length; } /* * Update the pointers to the current response record based on the defined * record terminator. */ static void update_current_record_by_rt_number(void) { bool ret; /* Point to the end of read data */ current_record_end.b = tail; current_record_end.pos = tail->size; /* Remove data that forms an incomplete record */ ret = dpointer_move_back(¤t_record_end, 0); assert(ret); /* Go back to the end of the specified record */ ret = dpointer_move_back(¤t_record_end, record_rbegin.r); assert(ret); /* Go further back to the begin of the specified record */ current_record_begin = current_record_end; ret = dpointer_move_back(¤t_record_begin, record_rend.r - record_rbegin.r); assert(ret); } /* * Update the pointers to the current response record based on the defined * record length. */ static void update_current_record_by_rl_number(void) { bool ret; /* Point to the end of read data */ current_record_end.b = tail; current_record_end.pos = tail->size; /* Remove data that forms an incomplete record */ ret = dpointer_subtract(¤t_record_end, tail->byte_count % rl); assert(ret); /* Go back to the end of the specified record */ ret = dpointer_subtract(¤t_record_end, record_rbegin.r * rl); assert(ret); /* Go further back to the begin of the specified record */ current_record_begin = current_record_end; ret = dpointer_subtract(¤t_record_begin, (record_rend.r - record_rbegin.r) * rl); assert(ret); } /* * Update the pointers to the current response record to include the first * record terminated record beginning in or after the begin buffer and * the last record beginning in the end buffer. * Set have_record to true if the corresponding data range exists */ static void update_current_record_by_rt_time(struct buffer *begin, struct buffer *end) { /* Point to the begin of the data window */ current_record_begin.b = begin; current_record_begin.pos = 0; /* Go to the begin of a record starting at or after the buffer */ if (!dpointer_move_forward(¤t_record_begin, 0)) return; /* Point to the end of the data window */ current_record_end.b = end; current_record_end.pos = end->size; dpointer_decrement(¤t_record_end); /* Adjust data that forms an incomplete record */ if (!dpointer_move_forward(¤t_record_end, 0)) { current_record_end.b = end; current_record_end.pos = end->size; if (!dpointer_move_back(¤t_record_end, 0)) return; if (memcmp(¤t_record_begin, ¤t_record_end, sizeof(struct dpointer)) == 0) return; } have_record = true; } /* * Update the pointers to the current response record to include the first * fixed length record beginning in or after the begin buffer and * the last record beginning in the end buffer. * Set have_record to true if the corresponding data range exists */ static void update_current_record_by_rl_time(struct buffer *begin, struct buffer *end) { int mod; DPRINTF(4, "Adjusting begin"); current_record_begin.b = begin; current_record_begin.pos = 0; if (begin->prev && (mod = begin->prev->byte_count % rl) != 0) /* * Example: rl == 10, prev->byte_count == 53 * mod = 3, dpointer_add(..., 7) */ if (!dpointer_add(¤t_record_begin, rl - mod)) return; /* Next record not there */ DPRINTF(4, "Adjusting end"); current_record_end.b = end; current_record_end.pos = end->size; if ((mod = end->byte_count % rl) != 0) { /* * Example: rl == 10, end->byte_count == 82 * mod = 2, dpointer_add(..., 8) * Decrement and increment to convert between an iterator * pointing beyond the range, and valid positions that dpointer_add * can handle correctly. */ if (!dpointer_decrement(¤t_record_end) || !dpointer_add(¤t_record_end, rl - mod)) { DPRINTF(4, "incomplete last record"); /* Try going back */ current_record_end.b = end; current_record_end.pos = end->size; if (!dpointer_subtract(¤t_record_end, mod)) return; } else (void)dpointer_increment(¤t_record_end); } if (memcmp(¤t_record_begin, ¤t_record_end, sizeof(struct dpointer)) == 0) return; have_record = true; } #ifdef DEBUG /* Dump the buffer list using relative timestamps */ static void dump_buffer_times(void) { struct buffer *bp; struct timeval now, t; gettimeofday(&now, NULL); DPRINTF(4, "update_current_record: now=%lld.%06d rend=%lld.%06d rbegin=%lld.%06d", (long long)now.tv_sec, (int)now.tv_usec, (long long)record_rend.t.tv_sec, (int)record_rend.t.tv_usec, (long long)record_rbegin.t.tv_sec, (int)record_rbegin.t.tv_usec); for (bp = head; bp != NULL; bp = bp->next) { timersub(&now, &bp->timestamp, &t); DPRINTF(4, "\t%p size=%3d byte_count=%5lld Tr=%3lld.%06d Ta=%3lld.%06d [%.*s]", bp, bp->size, bp->byte_count, (long long)t.tv_sec, (int)t.tv_usec, (long long)bp->timestamp.tv_sec, (int)bp->timestamp.tv_usec, bp->size, bp->data); } } static void timestamp(const char *msg) { struct timeval now; gettimeofday(&now, NULL); DPRINTF(4, "%lld.%06d %s", (long long)now.tv_sec, (int)now.tv_usec, msg); } #define TIMESTAMP(x) timestamp(x) #define DUMP_BUFFER_TIMES() dump_buffer_times() #else #define DUMP_BUFFER_TIMES() #define TIMESTAMP(x) #endif /* * Update the pointers to the current response record. * Set have_record if a record is available. */ static void update_current_record(void) { assert(head && tail); if (time_window) { struct timeval now, tbegin, tend; /* In absolute time units */ struct buffer *bbegin, *bend, *begin_candidate = NULL; DUMP_BUFFER_TIMES(); have_record = false; /* Records in the window come and go */ /* Convert to absolute time */ gettimeofday(&now, NULL); timersub(&now, &record_rend.t, &tbegin); DPRINTF(4, "tail->timestamp=%lld.%06d tbegin=%lld.%06d", (long long)tail->timestamp.tv_sec, (int)tail->timestamp.tv_usec, (long long)tbegin.tv_sec, (int)tbegin.tv_usec); if (timercmp(&tail->timestamp, &tbegin, <)) { free_unused_buffers_by_position(tail); return; /* No records fresh enough */ } timersub(&now, &record_rbegin.t, &tend); DPRINTF(4, "head->timestamp=%lld.%06d tend=%lld.%06d", (long long)head->timestamp.tv_sec, (int)head->timestamp.tv_usec, (long long)tend.tv_sec, (int)tend.tv_usec); if (timercmp(&head->timestamp, &tend, >)) return; /* No records old enough */ /* Find the record range */ DPRINTF(4, "Looking for record range"); for (bend = tail; timercmp(&bend->timestamp, &tend, >); bend = bend->prev) ; DPRINTF(4, "bend=%p %lld.%06d", bend, (long long)bend->timestamp.tv_sec, (int)bend->timestamp.tv_usec); for (bbegin = bend; bbegin && timercmp(&bbegin->timestamp, &tbegin, >); bbegin = bbegin->prev) begin_candidate = bbegin; if (!begin_candidate) { free_unused_buffers_by_time(&tbegin); return; /* No records within the window */ } bbegin = begin_candidate; DPRINTF(4, "bbegin=%p %lld.%06d", bbegin, (long long)bbegin->timestamp.tv_sec, (int)bbegin->timestamp.tv_usec); if (rl) update_current_record_by_rl_time(bbegin, bend); else update_current_record_by_rt_time(bbegin, bend); free_unused_buffers_by_time(&tbegin); } else { DPRINTF(4, "tail->record_count=%lld record_rend.r=%d", tail->record_count, record_rend.r); if (tail->record_count - record_rend.r < 0) /* Not enough records */ return; if (rl) update_current_record_by_rl_number(); else update_current_record_by_rt_number(); have_record = true; free_unused_buffers_by_position(current_record_begin.b); } DPRINTF(4, "have_record=%d", have_record); DPRINTF(4, "begin b=%p pos=%d", current_record_begin.b, current_record_begin.pos); DPRINTF(4, "end b=%p pos=%d", current_record_end.b, current_record_end.pos); } /* * Read a one character command from the specifid client and act on it * The following commands are supported: * R: Read value (the client wants to read our current store value) * Q: Quit (Terminate the operation of this data store) */ static void read_command(struct client *c) { char cmd; int n; switch (n = read(c->fd, &cmd, 1)) { case -1: /* Error */ switch (errno) { case EAGAIN: DPRINTF(4, "EAGAIN on client socket read"); break; default: err(3, "Read from socket"); } break; case 0: /* EOF */ close(c->fd); c->state = s_inactive; DPRINTF(4, "Done with client %p", c); update_oldest_buffer(); break; default: /* Have data. Insert buffer at the end of the queue. */ DPRINTF(4, "Read command %c from client %p", cmd, c); switch (cmd) { case 'L': c->state = s_send_last; break; case 'Q': (void)unlink(socket_path); exit(0); case 'c': c->state = s_send_current_nblk; break; case 'C': c->state = s_send_current; if (time_window && head) update_current_record(); /* Refresh have_record */ break; default: errx(5, "Unknown command [%c]", cmd); } } } /* * Write a single record to the specified client * Update the write_begin pointer * Close the connection and clean the client if done * If write_length is true, precede the record with * CONTENT_LENGTH_DIGITS digits representing the record's * length. */ static void write_record(struct client *c, bool write_length) { int n; int towrite; struct iovec iov[2], *iovptr; char length[CONTENT_LENGTH_DIGITS + 2]; DPRINTF(4, "Write %srecord for client %p", write_length ? "first " : "", c); if (c->write_begin.b == c->write_end.b) { towrite = c->write_end.pos - c->write_begin.pos; DPRINTF(4, "Single buffer %p: writing %d bytes. write_end.pos=%d write_begin.pos=%d", c->write_begin.b, towrite, c->write_end.pos, c->write_begin.pos); } else { towrite = c->write_begin.b->size - c->write_begin.pos; DPRINTF(4, "Multiple buffers %p %p: writing %d bytes. write_begin.b->size=%d write_begin.pos=%d", c->write_begin.b, c->write_end.b, towrite, c->write_begin.b->size, c->write_begin.pos); } iov[1].iov_base = c->write_begin.b->data + c->write_begin.pos; iov[1].iov_len = towrite; DPRINTF(4, "Writing [%.*s]", (int)iov[1].iov_len, (char *)iov[1].iov_base); if (write_length) { snprintf(length, sizeof(length), CONTENT_LENGTH_FORMAT, content_length(c)); iov[0].iov_base = length; iov[0].iov_len = CONTENT_LENGTH_DIGITS; iovptr = iov; } else iovptr = iov + 1; if ((n = writev(c->fd, iovptr, write_length ? 2 : 1)) == -1) switch (errno) { case EAGAIN: DPRINTF(4, "EAGAIN on client socket write"); return; default: err(3, "Write to socket"); } if (write_length) { if (n < CONTENT_LENGTH_DIGITS) errx(5, "Short content length record write: %d", n); n -= CONTENT_LENGTH_DIGITS; } c->write_begin.pos += n; DPRINTF(4, "Wrote %u data bytes. Current buffer position=%d", n, c->write_begin.pos); /* * More data to write from this buffer? * Yes, if there is still more data in the buffer * and either the end is in another buffer, or we haven't * reached it. */ if (c->write_begin.pos < c->write_begin.b->size && (c->write_begin.b != c->write_end.b || c->write_begin.pos < c->write_end.pos)) { DPRINTF(4, "Continuing with same buffer"); return; } /* More buffers to write from? */ if (c->write_begin.b != c->write_end.b) { c->write_begin.b = c->write_begin.b->next; c->write_begin.pos = 0; DPRINTF(4, "Moving to next buffer %p with size %u", c->write_begin.b, c->write_begin.b->size); return; } /* Done with this client */ DPRINTF(4, "No more data to write for client %p", c); c->state = s_wait_close; } /* Set the buffer's counters */ void set_buffer_counters(struct buffer *b) { if (time_window) gettimeofday(&b->timestamp, NULL); if (rl == 0) { /* Count records using RS */ int i; b->record_count = b->prev ? b->prev->record_count : 0; for (i = 0; i < b->size; i++) if (b->data[i] == rt) b->record_count++; } else { /* Count records using RL */ b->byte_count = b->prev ? b->prev->byte_count : 0; b->byte_count += b->size; b->record_count = b->byte_count / rl; } } #if __GNUC__ == 4 && __GNUC_MINOR__ >= 2 && __GNUC_MINOR__ < 6 #pragma GCC diagnostic ignored "-Wuninitialized" #endif /* Read data from STDIN into a new buffer */ static void buffer_read(void) { struct buffer *b; struct timeval now, abs_rend_time; if ((b = malloc(sizeof(struct buffer))) == NULL) err(1, "Unable to allocate read buffer"); DPRINTF(4, "Calling read on stdin for buffer %p", b); switch (b->size = read(STDIN_FILENO, b->data, sizeof(b->data))) { case -1: /* Error */ switch (errno) { case EAGAIN: DPRINTF(4, "EAGAIN on standard input"); free(b); break; default: err(3, "Read from standard input"); } break; case 0: /* EOF */ reached_eof = true; if (time_window) { /* Make abs_rend_time the latest absolute time that interests us */ gettimeofday(&now, NULL); timeradd(&now, &record_rend.t, &abs_rend_time); } if (have_record) { free(b); #if __GNUC__ >= 4 && __GNUC_MINOR__ >= 6 #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wmaybe-uninitialized" #endif } else if (!time_window || !tail || timercmp(&tail->timestamp, &abs_rend_time, >)) { #if __GNUC__ >= 4 && __GNUC_MINOR__ >= 6 #pragma GCC diagnostic pop #endif /* Setup an empty record, if there will never be a record to send */ b->size = 0; b->prev = b->next = NULL; head = tail = b; current_record_begin.b = current_record_end.b = b; current_record_begin.pos = current_record_end.pos = 0; have_record = true; } break; default: /* Have data. Insert buffer at the end of the queue. */ b->prev = tail; b->next = NULL; if (tail) tail->next = b; tail = b; if (!head) head = b; DPRINTF(4, "Read %d bytes into %p prev=%p next=%p head=%p tail=%p", b->size, b, b->prev, b->next, head, tail); set_buffer_counters(b); update_current_record(); break; } } /* * Set the specified file descriptor to operate in non-blocking * mode. * It seems that even if select returns for a specified file * descriptor, performing I/O to it may block depending on the * amount of data specified. * See See http://pubs.opengroup.org/onlinepubs/009695399/functions/write.html#tag_03_866 */ static void non_block(int fd) { int flags = fcntl(fd, F_GETFL, 0); if (flags < 0) err(2, "Error getting flags for socket"); if (fcntl(fd, F_SETFL, flags | O_NONBLOCK) < 0) err(2, "Error setting socket to non-blocking mode"); } /* Return an initialized free client entry, or exit with an error. */ static struct client * get_free_client(void) { int i; for (i = 0; i < MAX_CLIENTS; i++) if (clients[i].state == s_inactive) return &clients[i]; errx(5, "Maximum number of clients exceeded for socket %s", socket_path); } static void usage(void) { fprintf(stderr, "Usage: %s [-l len|-t char] [-b n] [-e n] [-u s|m|h|d|r] -s path\n" "-b n" "\tStore records beginning in a window n away from the end (default 1)\n" "-e n" "\tStore records ending in a window n away from the end (default 0)\n" "-l len" "\tProcess fixed-width len-sized records\n" "-s path" "\tSpecify the socket to create\n" "-t char" "\tProcess char-terminated records (newline default)\n" "-u unit" "\tSpecify the unit of window boundaries\n" "" "\ts: seconds\n" "" "\tm: minutes\n" "" "\th: hours\n" "" "\td: days\n" "" "\tr: records (default)\n", program_name); exit(1); } /* * Parse a number >= 0 * Exit with an error if an error occurs */ static double parse_double(const char *s) { char *endptr; double d; errno = 0; d = strtod(s, &endptr); if (endptr - s != strlen(s) || *s == 0) errx(6, "Error in parsing [%s] as a number", s); if (errno != 0) err(6, "[%s]", s); if (d < 0) errx(6, "Argument [%s] cannot be negative", s); return d; } /* Return the passed double as a timeval */ struct timeval double_to_timeval(double d) { struct timeval t; t.tv_sec = (time_t)d; t.tv_usec = (int)((d - t.tv_sec) * 1e6); return t; } /* Parse the program's arguments */ static void parse_arguments(int argc, char *argv[]) { int ch; char unit = 'r'; program_name = argv[0]; /* By default return the last record read */ record_rbegin.d = 0; record_rend.d = 1; while ((ch = getopt(argc, argv, "b:e:l:s:t:u:")) != -1) { switch (ch) { case 'b': /* Begin record, measured from the end (0) */ record_rend.d = parse_double(optarg); break; case 'e': /* End record, measured from the end (0) */ record_rbegin.d = parse_double(optarg); break; case 'l': /* Fixed record length */ rl = atoi(optarg); if (rl <= 0) usage(); break; case 's': socket_path = optarg; break; case 't': /* Record terminator */ /* We allow \0 as rt */ if (strlen(optarg) > 1) usage(); rt = *optarg; break; case 'u': /* Measurement unit */ if (strlen(optarg) != 1 || strchr("smhdr", *optarg) == NULL) usage(); unit = *optarg; break; case '?': default: usage(); } } argc -= optind; argv += optind; if (argc != 0 || socket_path == NULL) usage(); switch (unit) { case 'r': if (record_rbegin.d != (int)record_rbegin.d || record_rend.d != (int)record_rend.d) errx(6, "Record numbers must be integers"); record_rbegin.r = (int)record_rbegin.d; record_rend.r = (int)record_rend.d; time_window = false; break; case 'd': record_rbegin.d *= 24; record_rend.d *= 24; /* FALLTHROUGH */ case 'h': record_rbegin.d *= 60; record_rend.d *= 60; /* FALLTHROUGH */ case 'm': record_rbegin.d *= 60; record_rend.d *= 60; /* FALLTHROUGH */ case 's': record_rbegin.t = double_to_timeval(record_rbegin.d); record_rend.t = double_to_timeval(record_rend.d); if (!timercmp(&record_rbegin.t, &record_rend.t, <)) errx(6, "Begin time must be older than end time"); time_window = true; break; } } /* * Handle the events associated with the following elements * The passed socket * Standard input * Communicating clients * Elapsed time values * This is called in an endless loop to do the following things: * Setup select(2) arguments * Call select(2) * Process events that can be processed */ static void handle_events(int sock) { fd_set source_fds; fd_set sink_fds; struct timeval wait_time, *waitptr; bool set_waitptr; int i, max_fd, nfds; socklen_t len; struct sockaddr_un remote; /* Set the fds that interest us */ FD_ZERO(&source_fds); FD_ZERO(&sink_fds); waitptr = NULL; max_fd = -1; /* Read from standard input */ if (!reached_eof) { FD_SET(STDIN_FILENO, &source_fds); max_fd = STDIN_FILENO; } /* Accept incoming connection */ FD_SET(sock, &source_fds); max_fd = MAX(sock, max_fd); /* I/O with a client */ set_waitptr = false; for (i = 0; i < MAX_CLIENTS; i++) switch (clients[i].state) { case s_inactive: /* Free (unused or closed) */ break; case s_wait_close: /* Wait for the client to close the connection */ case s_read_command: /* Waiting for a command (Q or R) to be read */ FD_SET(clients[i].fd, &source_fds); max_fd = MAX(clients[i].fd, max_fd); break; case s_send_last: /* Waiting for the last (before EOF) value to be written */ if (reached_eof) { FD_SET(clients[i].fd, &sink_fds); max_fd = MAX(clients[i].fd, max_fd); } break; case s_send_current: /* Waiting for a response to be written */ if (have_record) { FD_SET(clients[i].fd, &sink_fds); max_fd = MAX(clients[i].fd, max_fd); } else if (time_window) set_waitptr = true; break; case s_send_current_nblk: /* Waiting for a response to be written */ FD_SET(clients[i].fd, &sink_fds); max_fd = MAX(clients[i].fd, max_fd); break; case s_sending_response: /* A response is being sent */ FD_SET(clients[i].fd, &sink_fds); max_fd = MAX(clients[i].fd, max_fd); break; } if (set_waitptr) { /* * Find the oldest buffer that hasn't yet entered the time * window and arrange for select(2) to wait for it to enter. */ struct buffer *bp, *candidate_buffer = NULL; struct timeval now, abs_rbegin_time; gettimeofday(&now, NULL); timersub(&now, &record_rbegin.t, &abs_rbegin_time); DPRINTF(4, "have to wait for a buffer to enter window %lld.%06d", (long long)abs_rbegin_time.tv_sec, (int)abs_rbegin_time.tv_usec); /* * rbegin = 10 * 13 19 20 21 23 * abs_rbegin ... ... tail now */ for (bp = tail; bp && timercmp(&bp->timestamp, &abs_rbegin_time, >); bp = bp->prev) candidate_buffer = bp; if (candidate_buffer) { /* There is a buffer worth waiting for */ waitptr = &wait_time; timersub(&candidate_buffer->timestamp, &abs_rbegin_time, waitptr); DPRINTF(4, "waiting %lld.%06d for %p %lld.%06d to enter the window", (long long)wait_time.tv_sec, (int)wait_time.tv_usec, candidate_buffer, (long long)candidate_buffer->timestamp.tv_sec, (int)candidate_buffer->timestamp.tv_usec); } else DPRINTF(4, "No candidate buffer found"); } TIMESTAMP("Calling select"); if ((nfds = select(max_fd + 1, &source_fds, &sink_fds, NULL, waitptr)) < 0) err(3, "select"); TIMESTAMP("Select returns"); if (FD_ISSET(STDIN_FILENO, &source_fds)) buffer_read(); if (waitptr && nfds == 0) /* Expired timer; records may have entered the window */ update_current_record(); for (i = 0; i < MAX_CLIENTS; i++) switch (clients[i].state) { case s_inactive: /* Free (unused or closed) */ break; case s_read_command: /* Waiting for a command (Q or R) to be read */ case s_wait_close: /* Wait for the client to close the connection */ if (FD_ISSET(clients[i].fd, &source_fds)) read_command(&clients[i]); break; case s_send_last: /* Waiting for the last (before EOF) value to be written */ /* FALLTHROUGH */ case s_send_current: /* Waiting for a response to be written */ if (FD_ISSET(clients[i].fd, &sink_fds)) { assert(have_record); /* Start writing the most fresh last record */ clients[i].write_begin = current_record_begin; clients[i].write_end = current_record_end; clients[i].state = s_sending_response; oldest_buffer_being_written = oldest_buffer(oldest_buffer_being_written, clients[i].write_begin.b); write_record(&clients[i], true); } break; case s_send_current_nblk: /* Waiting for a response (even empty) to be written */ if (FD_ISSET(clients[i].fd, &sink_fds)) { if (have_record) { /* Start writing the most fresh last record */ clients[i].write_begin = current_record_begin; clients[i].write_end = current_record_end; oldest_buffer_being_written = oldest_buffer(oldest_buffer_being_written, clients[i].write_begin.b); } else { static struct buffer empty; /* Write an empty record */ clients[i].write_begin.b = clients[i].write_end.b = ∅ clients[i].write_begin.pos = clients[i].write_end.pos = 0; } clients[i].state = s_sending_response; write_record(&clients[i], true); } break; case s_sending_response: /* A response is being written */ if (FD_ISSET(clients[i].fd, &sink_fds)) write_record(&clients[i], false); break; } if (FD_ISSET(sock, &source_fds)) { int rsock; struct client *c; len = sizeof(remote); rsock = accept(sock, (struct sockaddr *)&remote, &len); if (rsock == -1 && errno != EAGAIN) err(5, "accept"); c = get_free_client(); non_block(rsock); c->fd = rsock; c->state = s_read_command; } } int main(int argc, char *argv[]) { int sock; socklen_t len; struct sockaddr_un local; int ninputs = 1; int noutputs = 0; parse_arguments(argc, argv); dgsh_negotiate(DGSH_HANDLE_ERROR, program_name, &ninputs, &noutputs, NULL, NULL); if (strlen(socket_path) >= sizeof(local.sun_path) - 1) errx(6, "Socket name [%s] must be shorter than %lu characters", socket_path, (long unsigned)sizeof(local.sun_path)); (void)unlink(socket_path); if ((sock = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) err(2, "Error creating socket"); local.sun_family = AF_UNIX; strcpy(local.sun_path, socket_path); len = strlen(local.sun_path) + 1 + sizeof(local.sun_family); if (bind(sock, (struct sockaddr *)&local, len) == -1) err(3, "Error binding socket to Unix domain address %s", argv[1]); if (listen(sock, 5) == -1) err(4, "listen"); non_block(sock); reached_eof = false; for (;;) handle_events(sock); } ================================================ FILE: core-tools/src/dgsh.1 ================================================ .TH DGSH 1 "10 August 2017" .\" .\" (C) Copyright 2016-2017 Diomidis Spinellis. All rights reserved. .\" .\" Licensed under the Apache License, Version 2.0 (the "License"); .\" you may not use this file except in compliance with the License. .\" You may obtain a copy of the License at .\" .\" http://www.apache.org/licenses/LICENSE-2.0 .\" .\" Unless required by applicable law or agreed to in writing, software .\" distributed under the License is distributed on an "AS IS" BASIS, .\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .\" See the License for the specific language governing permissions and .\" limitations under the License. .\" .SH NAME dgsh \- directed graph shell .SH SYNOPSIS \fBdgsh\fP [\fIbash_options\fP] [command_string | file] .SH DESCRIPTION \fIdgsh\fP is a modified version of \fIbash\fP that allows the specification of pipelines with non-linear non-uniform operations. These form a directed acyclic process graph, which is typically executed by multiple processor cores, thus increasing the operation's processing throughput. The \fIdgsh\fP command is equivalent to invoking the modified version of \fIbash\fP with the \fI--dgsh\fP argument in order to enable the \fIdgsh\fP-specific inter-process communication functionality. .SH INTER-PROCESS COMMUNICATION \fIDgsh\fP provides three new ways for expressing inter-process communication. .PP \fBMultipipes\fP are expressed as usual Unix pipelines, but can connect commands with more than one output or input channel. As an example, the \fIcomm\fP command supplied with \fIdgsh\fP expects two input channels and produces on its output three output channels: the lines appearing only in first (sorted) channel, the lines appearing only in the second channel, and the lines appearing in both. Connecting the output of the \fIcomm\fP command to the \fIcat\fP command supplied with \fIdgsh\fP will make the three outputs appear in sequence, while connecting it to the \fIpaste\fP command supplied with \fIdgsh\fP will make the output appear in its customary format. \fIDgsh\fP handles the following programs as being multipipe compatible: a) those that are linked with the \fIdgsh\fP library; b) scripts that include in their first line one of the strings \fCdgsh-wrap\fP, \fCenv dgsh\fP, or \fC--dgsh\fP; c) scripts whose second line starts with \fC#!dgsh\fP. .PP \fBMultipipe blocks\fP are enclosed within double braces: {{ ... }}. These a) send the input received on their input side to the asynchronously-running processes that reside within the block, and, b) pass the output produced by the processes within the block to their output side. Multipipe blocks typically receive input from more than one channel and produce more than one output channel. For example, a multipipe block that runs \fImd5sum\fP and \fIwc -c\fP receives two inputs and produces two outputs: the MD5 hash of its input and the input's size. Data to multipipe blocks are typically provided with an \fIdgsh\fP-aware version of \fItee\fP and collected by \fIdgsh\fP-aware versions of programs such as \fIcat\fP and \fIpaste\fP. .PP \fBStored values\fP offer a convenient way for communicating computed values between arbitrary processes on the graph. They allow the storage of a data stream's last record into a named buffer. This record can be later retrieved asynchronously by one or more readers. Data in a stored value can be piped into a process or out of it, or it can be read using the shell's command output substitution syntax. Stored values are implemented internally through Unix-domain sockets, a background-running store program, \fIdgsh-writeval\fP, and a reader program, \fIdgsh-readval\fP. The behavior of a stored value's IO can be modified by adding flags to \fIdgsh-writeval\fP and \fIdgsh-readval\fP. .SH SYNTAX EXTENSIONS The syntax of \fIbash\fP is extended by \fIdgsh\fP as follows. .PP .ft C .ps -1 .nf ::= '{{' '}}' ::= '&' ::= '|' .fi .ps +1 .ft P .br .SH EXAMPLES .PP Report file type, length, and compression performance for a URL retrieved from the web. The web file never touches the disk. .ft C .ps -1 .nf #!/usr/bin/env dgsh curl -s "$1" | tee | {{ echo -n 'File type:' & file - & echo -n 'Original size:' & wc -c & echo -n 'xz:' & xz -c | wc -c & echo -n 'bzip2:' & bzip2 -c | wc -c & echo -n 'gzip:' & gzip -c | wc -c & }} | cat .fi .ps +1 .ft P .PP List the names of duplicate files in the specified directory .ft C .ps -1 .nf #!/usr/bin/env dgsh # Create list of files find "$@" -type f | # Produce lines of the form # MD5(filename)= 811bfd4b5974f39e986ddc037e1899e7 xargs openssl md5 | # Convert each line into a "filename md5sum" pair sed 's/^MD5(//;s/)= / /' | # Sort by MD5 sum sort -k2 | tee | {{ # Print an MD5 sum for each file that appears more than once awk '{print $2}' | uniq -d & # Promote the stream to gather it cat & }} | # Join the repeated MD5 sums with the corresponding file names # Join expects two inputs, second will come from scatter join -2 2 | .fi .ps +1 .ft P .PP Check if the script is running under \fIdgsh\fP or regular \fIbash\fP (for polyglot scripts) .ft C .ps -1 .nf if {{ : ; }} ; then echo dgsh else echo bash fi 2>/dev/null .fi .ps +1 .ft P .SH "SEE ALSO" .BR dgsh-tee (1), .BR dgsh-wrap (1), .BR dgsh-writeval (1), .BR dgsh-readval (1), .BR dgsh-monitor (1) .BR dgsh-conc (1), .BR dgsh-httpval (1), .BR dgsh-merge-sum (1) .SH AUTHOR \fIDgsh\fP was designed by Diomidis Spinellis \(em \(em and implemented by Marios Fragkoulis. The current design and capabilities of \fIdgsh\fP have been significantly influenced by amazing feedback generously provided by Doug McIlroy. .SH BUGS Report bugs through https://github.com/dspinellis/dgsh/issues. ================================================ FILE: core-tools/src/dgsh.h ================================================ /* * Copyright 2016-2017 Diomidis Spinellis and Marios Fragkoulis * * Dgsh public API * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ #ifndef DGSH_H #define DGSH_H #define DGSH_HANDLE_ERROR 0x100 int dgsh_negotiate(int flags, const char *tool_name, int *n_input_fds, int *n_output_fds, int **input_fds, int **output_fds); #endif ================================================ FILE: core-tools/src/dgsh_negotiate.3 ================================================ .TH DGSH_NEGOTIATE 3 "16 February 2017" .\" .\" (C) Copyright 2017 Diomidis Spinellis. All rights reserved. .\" .\" Licensed under the Apache License, Version 2.0 (the "License"); .\" you may not use this file except in compliance with the License. .\" You may obtain a copy of the License at .\" .\" http://www.apache.org/licenses/LICENSE-2.0 .\" .\" Unless required by applicable law or agreed to in writing, software .\" distributed under the License is distributed on an "AS IS" BASIS, .\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .\" See the License for the specific language governing permissions and .\" limitations under the License. .\" .SH NAME dgsh_negotiate \- specify and obtain dgsh I/O file descriptors .SH SYNOPSIS .nf .B #include .sp .BI "dgsh_negotiate(int " flags ", const char *" program_name ", .BI " int *" n_input_fds ", int *" n_output_fds , .BI " int **" input_fds ", int **" output_fds ); .fi .sp Link with \fI\-ldgsh\fP. .sp .SH DESCRIPTION The .BR dgsh_negotiate () function is called before a program participates in a .IR dgsh (1) graph to specify the number of input and output file descriptors the program can handle, and obtain the file descriptors to be used. .PP The .I flags parameter adjusts the function's behavior. The following flags are defined. .TP .B DGSH_HANDLE_ERROR . When this flag is set and the negotiation for creating the graph of communicating processes encounters an error, the function will print an error message to standard error (if required) and cause the calling program to exit with the error value .IR EX_PROTOCOL " (76)." .PP The .I program_name parameter should be specified to match the name of the program calling the function, to aid error reporting and debugging. .PP The .I n_input_fds and .I n_output_fds parameters are used to pass by reference the number of input or output file descriptors required, and obtain upon return the corresponding number of descriptors supplied. Passing a null pointer indicates that the program can handle zero or one descriptor. In this case, if a variable contains the value of 1 when the function returns, the program can use the standard input or output for the corresponding channel and no descriptors are returned through .I input_fds or .IR output_fds . Passing a value of -1 indicates that the program can handle an arbitrary number of corresponding file descriptors. In this case, upon return the variable will contain the actual number of file descriptors allocated to the program through the negotiation process. .PP The .I input_fds and .I output_fds parameters are used to return a pointer to a sequence of integers containing the file descriptors to use for input or output. The size of the integer sequence is equal to the returned corresponding value of .I n_input_fds and .IR n_output_fds . The pointers may subsequently be used as an argument to the function .IR free (3). .PP Each tool in the \fIdgsh\fP graph calls .BR dgsh_negotiate () to take part in a peer-to-peer negotiation. A message block is circulated among tools and is filled with tools' I/O requirements. When all requirements are in place, an algorithm runs to find a solution that satisfies all requirements. If a solution is found, pipes are allocated and set up according to the solution. The appropriate file descriptors are provided to each tool and the negotiation phase ends. .SH RETURN VALUE On success, the function returns 0, on failure it returns -1. .SH ENVIRONMENT The following environment variables affect the negotiation to create the communication graph. .TP .B DGSH_DEBUG_LEVEL Setting this variable to an integer (see the section \fBDEBUGGING\fP below) causes the function to output debug data regarding the negotiation on its standard error. .TP .B DGSH_DOT_DRAW Setting this variable to a file path causes the \fIdgsh\fP negotiation to save the graph of the communication processes in that file. The graph is saved in .IR dot (1) format. .TP .B DGSH_DOT_DRAW_EXIT Setting this variable in conjunction with \fBDGSH_DOT_DRAW\fP causes all processes participating in the negotiation to exit after the graph is saved to the file. .TP .B DGSH_TIMEOUT Setting this variable to an integer value specifies the number of seconds \fIdgsh\fP processes will wait for the negotiation to comlete before timing out and exiting. The default value is five seconds, but this value may need to be increased for negotiations that take a long time to complete. .SH DEBUGGING The DGSH_DEBUG_LEVEL environment variable controls debug output, which appears in \fIstderr\fP. The default level 0 produces no debug output. .TP .B DGSH_DEBUG_LEVEL=1 Level 1 outputs the phases of the negotiation process, that is gather I/O requirements, compute the solution, and communicate the solution. .TP .B DGSH_DEBUG_LEVEL=2 Level 2 outputs the progress of the negotiation process by each command. .B DGSH_DEBUG_LEVEL=3 Level 3 outputs the reading and writing of the message block by each command as it flows across the process graph. .TP .B DGSH_DEBUG_LEVEL=4 Level 4 outputs the complete debug output including the shell's initial setup of the process graph. This is very verbose. .SH ERROR MANAGEMENT Things can go wrong in two ways. First, a command might exit before reaching the call to .BR dgsh_negotiate () because of invalid command-line arguments for instance. Second, a command might not be able to complete the negotiation procedure because another command aborted during the negotiation. For commands that aborted before stepping into the negotiation procedure, a handler function is called on their exit and starts a negotiation procedure to inform the other commands on the \fIdgsh\fP graph of the error state. The exit handler is there for commands that link to the \fIdgsh\fP library. This happens by calling .BR dgsh_negotiate () and linking to the library or with the use of \fIdgsh-wrap(1)\fP. For commands that stuck in the negotiation procedure because another command aborted during it, an alarm signal triggers an exit after 5 seconds spent in negotiation to help commands exit it. .SH EXAMPLES .PP The following simple implementation of echo does not receive any input and provides one output channel. .ft C .ps -1 .nf #include #include #include #include "dgsh.h" int main(int argc, char *argv[]) { int n_input_fds = 0; int n_output_fds = 1; if (dgsh_negotiate(DGSH_HANDLE_ERROR, "echo", &n_input_fds, &n_output_fds, NULL, NULL) != 0) errx(1, "Negotiation failed"); assert(n_input_fds == 0); assert(n_output_fds == 1); ++argv; while (*argv) { (void)printf("%s", *argv); if (*++argv) putchar(' '); } putchar('\n'); return 0; } .fi .ps +1 .ft P .PP The following program will enumerate its output channels. .ft C .ps -1 .nf #include #include #include #include #include #include #include "dgsh.h" int main(int argc, char *argv[]) { int n_input_fds = 0, n_output_fds; int *output_fds; int i; switch (argc) { case 1: n_output_fds = -1; break; case 2: n_output_fds = atoi(argv[1]); break; default: errx(1, "usage: %s [n]", argv[0]); } if (dgsh_negotiate(DGSH_HANDLE_ERROR, argv[0], &n_input_fds, &n_output_fds, NULL, &output_fds) != 0) errx(1, "Negotiation failed"); for (i = 0; i < n_output_fds; i++) { char buff[10]; snprintf(buff, sizeof(buff), "%d\n", i); write(output_fds[i], buff, strlen(buff)); close(output_fds[i]); } return 0; } .fi .ps +1 .ft P .SH SEE ALSO .BR dgsh (1), .BR dgsh-wrap (1). .SH AUTHOR The .B dgsh_negotiate API and negotiation algorithm were designed by Diomidis Spinellis and extended and implemented by Marios Fragkoulis. ================================================ FILE: core-tools/src/kvstore.c ================================================ /* * Copyright 2013 Diomidis Spinellis * * Communicate with the data store specified as a Unix-domain socket. * (API) * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include "dgsh.h" #include "kvstore.h" #include "debug.h" int retry_limit = 10; /* Write a command to the specified socket, and return the socket */ static int write_command(const char *name, char cmd, bool retry_connection) { int s; socklen_t len; struct sockaddr_un remote; int counter = 0; char *env_retry_limit; if ((env_retry_limit = getenv("KVSTORE_RETRY_LIMIT")) != NULL) retry_limit = atoi(env_retry_limit); if ((s = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) err(1, "socket"); DPRINTF(3, "Connecting to %s", name); remote.sun_family = AF_UNIX; if (strlen(name) >= sizeof(remote.sun_path) - 1) errx(6, "Socket name [%s] must be shorter than %d characters", name, (int)sizeof(remote.sun_path)); strcpy(remote.sun_path, name); len = strlen(remote.sun_path) + 1 + sizeof(remote.sun_family); again: if (connect(s, (struct sockaddr *)&remote, len) == -1) { if (retry_connection && (errno == ENOENT || errno == ECONNREFUSED) && counter++ < retry_limit) { DPRINTF(3, "Retrying connection setup"); sleep(1); goto again; } err(2, "connect %s", name); } DPRINTF(3, "Connected"); if (write(s, &cmd, 1) == -1) err(3, "write"); DPRINTF(3, "Wrote command"); return s; } /* Send to the socket path the specified command */ void dgsh_send_command(const char *socket_path, char cmd, bool retry_connection, bool quit, int outfd) { int s, n; char buff[PIPE_BUF]; int content_length; char cbuff[CONTENT_LENGTH_DIGITS + 2]; struct iovec iov[2]; switch (cmd) { case 0: /* No I/O specified */ break; case 'C': /* Read current value */ case 'c': /* Read current value, non-blocking */ case 'L': /* Read last value */ s = write_command(socket_path, cmd, retry_connection); /* Read content length and some data */ iov[0].iov_base = cbuff; iov[0].iov_len = CONTENT_LENGTH_DIGITS; iov[1].iov_base = buff; iov[1].iov_len = sizeof(buff); if ((n = readv(s, iov, 2)) == -1) err(5, "readv"); DPRINTF(3, "Read %d characters", n); cbuff[CONTENT_LENGTH_DIGITS] = 0; if (sscanf(cbuff, "%u", &content_length) != 1) { fprintf(stderr, "Unable to read content length from string [%s]\n", cbuff); exit(1); } DPRINTF(3, "Content length is %u", content_length); n -= CONTENT_LENGTH_DIGITS; if (write(outfd, buff, n) == -1) err(4, "write"); content_length -= n; /* Read rest of data */ while (content_length > 0) { if ((n = read(s, buff, sizeof(buff))) == -1) err(5, "read"); DPRINTF(4, "Read %d bytes", n); if (write(outfd, buff, n) == -1) err(4, "write"); content_length -= n; } close(s); break; default: assert(0); break; } if (quit) (void)write_command(socket_path, 'Q', retry_connection); } ================================================ FILE: core-tools/src/kvstore.h ================================================ /* * Copyright 2013 Diomidis Spinellis * * Communicate with the data store specified as a Unix-domain socket. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ #ifndef KVSTORE_H #define KVSTORE_H #include /* Send to the socket path the specified command */ void dgsh_send_command(const char *socket_path, char cmd, bool retry_connection, bool quit, int outfd); /* * The read/write store communication protocol is as follows * readval -> writeval: L | Q | C * For L (read last) and C (read current) * writeval -> readval: CONTENT_LENGTH content ... * If writeval gets EOF it returns an empty (length 0) record, if no record * can ever appear. * For Q (quit) writeval exits */ #define CONTENT_LENGTH_DIGITS 10 #define CONTENT_LENGTH_FORMAT "%010u" #endif /* KVSTORE_H */ ================================================ FILE: core-tools/src/minmax.h ================================================ /* * Copyright 2013-2017 Diomidis Spinellis * * MIN and MAX macros * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ #ifndef MINMAX_H #define MINMAX_H #define MAX(a, b) ((a) > (b) ? (a) : (b)) #define MIN(a, b) ((a) < (b) ? (a) : (b)) #endif /* MINMAX_H */ ================================================ FILE: core-tools/src/negotiate.c ================================================ /* * Copyright 2016, 2017 Marios Fragkoulis * * A passive component that aids the dgsh negotiation by passing * message blocks among participating processes. * When the negotiation is finished and the processes get connected by * pipes, it exits. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ #include /* assert() */ #include /* ENOBUFS */ #include /* err() */ #include /* IOV_MAX */ #include /* bool, true, false */ #include /* fprintf() in DPRINTF() */ #include /* getenv(), errno, atexit() */ #include /* memcpy() */ #include /* EX_PROTOCOL, EX_OK */ #include /* sendmsg(), recvmsg() */ #include /* getpid(), getpagesize(), * STDIN_FILENO, STDOUT_FILENO, * STDERR_FILENO, alarm(), sysconf() */ #include /* signal(), SIGALRM */ #include /* nanosleep() */ #include /* select(), fd_set, */ #include /* printf family */ #include "negotiate.h" /* Message block and I/O */ #include "dgsh-debug.h" /* DPRINTF() */ #ifdef TIME #include static struct timespec tstart={0,0}, tend={0,0}; #endif /* Default negotiation timeout (s) */ #define DGSH_TIMEOUT 5 #ifndef UNIT_TESTING /* Models an I/O connection between tools on an dgsh graph. */ struct dgsh_edge { int from; /* Index of node on the graph where data * comes from (out). */ int to; /* Index of node on the graph that * receives the data (in). */ int instances; /* Number of instances of an edge. */ int from_instances; /* Number of instances the origin node of * an edge can provide. */ int to_instances; /* Number of instances the destination * of an edge can require. */ }; #endif /* Each tool that participates in an dgsh graph is modelled as follows. */ struct dgsh_node { pid_t pid; int index; /* Position in message block's node array. */ char name[100]; /* Tool's name */ int requires_channels; /* Input channels it can take. */ int provides_channels; /* Output channels it can provide. */ int dgsh_in; /* Takes input from other tool(s) on * dgsh graph. */ int dgsh_out; /* Provides output to other tool(s) * on dgsh graph. */ }; /* Holds a node's connections. It contains a piece of the solution. */ struct dgsh_node_connections { int node_index; /* The subject of the * connections. For * verification. */ struct dgsh_edge *edges_incoming; /* Array of edges through * which other nodes provide * input to node at node_index. */ int n_edges_incoming; /* Number of incoming edges */ int n_instances_incoming_free; /* Number of incoming edges * not yet bound to a pair * node's output channel. */ struct dgsh_edge *edges_outgoing; /* Array of edges through * which a node provides * output to other nodes. */ int n_edges_outgoing; /* Number of outgoing edges */ int n_instances_outgoing_free; /* Number of outgoing edges * not yet binded to a pair * node's outgoing edges. */ }; /* The output of the negotiation process. */ struct dgsh_node_pipe_fds { int *input_fds; /* Array of input file descriptors */ int n_input_fds; /* Number of input file descriptors */ int *output_fds; /* Array of output file descriptors */ int n_output_fds; /* Number of output file descriptors */ }; /** * Memory organisation of message block. * Message block will be passed around process address spaces. * Message block contains a number of scalar fields and two pointers * to an array of dgsh nodes and edges respectively. * To pass the message block along with nodes and edges, three writes * in this order take place. */ /* The message block implicitly used by many functions */ struct dgsh_negotiation *chosen_mb; static struct dgsh_node self_node; /* The dgsh node that models this tool. */ static char *programname; static struct node_io_side self_node_io_side; /* Dispatch info for this tool. */ static struct dgsh_node_pipe_fds self_pipe_fds; /* A tool's read and write file * descriptors to use at execution. */ static bool init_error = false; static volatile sig_atomic_t negotiation_completed = 0; int dgsh_debug_level = 0; static void get_environment_vars(); static int dgsh_exit(int state, int flags); /* Force the inclusion of the ELF note section */ extern int dgsh_force_include; void dgsh_force_include_function(void) { dgsh_force_include = 1; } #ifndef UNIT_TESTING static void dgsh_exit_handler(void) { if (negotiation_completed) return; init_error = true; /* Finish negotiation, if required */ get_environment_vars(); if (self_node.dgsh_in != 0 || self_node.dgsh_out != 0) { warnx("exiting before dgsh negotiation is complete"); DPRINTF(4, "dgsh: error state. Enter negotiation to inform the graph"); dgsh_negotiate(0, programname ? programname : "dgsh client", NULL, NULL, NULL, NULL); } } #endif void dgsh_alarm_handler(int signal) { if (signal == SIGALRM) if (negotiation_completed == 0) { char msg[100]; sprintf(msg, "%d dgsh: timeout for negotiation. Exit.\n", getpid()); negotiation_completed = 1; write(2, msg, strlen(msg)); _exit(EX_PROTOCOL); } } #ifndef UNIT_TESTING __attribute__((constructor)) static void install_exit_handler(void) { atexit(dgsh_exit_handler); } #endif static int iov_max; // Setup iov_max handling runtime, even if it is defined at runtime __attribute__((constructor)) static void setup_iov_max(void) { #if defined(IOV_MAX) iov_max = IOV_MAX; #else iov_max = (int)sysconf(_SC_IOV_MAX); #endif } /** * Remove path to command to save space in the graph plot * Find first space if any and take the name up to there * Find and remove path prepended to the name * Rejoin the name with arguments * Escape double quotes */ STATIC void process_node_name(char *name, char **processed_name) { char no_path_name[strlen(name)]; memset(no_path_name, 0, sizeof(no_path_name)); char *s = strstr(name, " "); DPRINTF(4, "Node name to process: %s", name); if (s) strncpy(no_path_name, name, s - name); else strcpy(no_path_name, name); DPRINTF(4, "no_path_name: %s, s: %s", no_path_name, s); char *p = no_path_name; char *m = strstr(no_path_name, "/"); while (m) { p = ++m; m = strstr(m, "/"); } DPRINTF(4, "no_path_name: %s, m: %s", no_path_name, m); if (s) sprintf(no_path_name, "%s%s", p, s); DPRINTF(4, "no_path_name: %s, p: %s", no_path_name, p); m = strstr(no_path_name, "\""); char *mm = NULL; while (m) { DPRINTF(4, "processed_name: %s, m: %s, mm: %s", *processed_name, m, mm); if (strlen(*processed_name) == 0) strncpy(*processed_name, no_path_name, m - no_path_name); else { strcat(*processed_name, "\\"); strncat(*processed_name, mm, m - mm); DPRINTF(4, "processed_name: %s, m - mm: %ld", *processed_name, (long)(m - mm)); } mm = m; m = strstr(++m, "\""); } if (mm) { strcat(*processed_name, "\\"); strcat(*processed_name, mm); } else strcpy(*processed_name, no_path_name); DPRINTF(4, "final processed_name: %s, m: %s, mm: %s", *processed_name, m, mm); } STATIC enum op_result output_graph(char *filename) { char ffilename[strlen(filename) + 5]; // + .dot sprintf(ffilename, "%s.dot", filename); FILE *f = fopen(ffilename, "a"); if (f == NULL) { fprintf(stderr, "Unable to open file %s", ffilename); return OP_ERROR; } char fnfilename[strlen(filename) + 9]; // + -ngt + .dot sprintf(fnfilename, "%s-ngt.dot", filename); FILE *fn = fopen(fnfilename, "a"); if (fn == NULL) { fprintf(stderr, "Unable to open file %s", fnfilename); return OP_ERROR; } int i, j; int n_nodes = chosen_mb->n_nodes; struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; DPRINTF(4, "Output graph in file %s for %d nodes and %d edges", filename, n_nodes, chosen_mb->n_edges); fprintf(f, "digraph {\n"); fprintf(fn, "digraph {\n"); for (i = 0; i < n_nodes; i++) { struct dgsh_node *node = &chosen_mb->node_array[i]; struct dgsh_node_connections *connections = &graph_solution[i]; int n_edges_outgoing = connections->n_edges_outgoing; DPRINTF(4, "Output node: %s", node->name); // Reserve space for quotes int q = 0; char *m = strstr(node->name, "\""); while (m) { q++; m = strstr(++m, "\""); } char *processed_name = (char *)malloc(sizeof(char) * (strlen(node->name) + q + 1)); DPRINTF(4, "Malloc %d bytes for processed_name", (int)strlen(node->name) + q + 1); memset(processed_name, 0, strlen(node->name) + q + 1); process_node_name(node->name, &processed_name); #ifdef DEBUG fprintf(f, " n%d [label=\"%d %s\"];\n", node->index, node->index, processed_name); fprintf(fn, " n%d [label=\"%d %s\"];\n", node->index, node->index, processed_name); #else fprintf(f, " n%d [label=\"%s\"];\n", node->index, processed_name); fprintf(fn, " n%d [label=\"%s\"];\n", node->index, processed_name); #endif DPRINTF(4, "Node: (%d) %s", node->index, processed_name); free(processed_name); for (j = 0; j < n_edges_outgoing; j++) { fprintf(fn, " n%d -> n%d;\n", node->index, chosen_mb->node_array[connections->edges_outgoing[j].to].index); if (connections->edges_outgoing[j].instances == 0) continue; fprintf(f, " n%d -> n%d;\n", node->index, chosen_mb->node_array[connections->edges_outgoing[j].to].index); DPRINTF(4, "Edge: (%d) %s -> %s (%d)", node->index, node->name, chosen_mb->node_array[connections->edges_outgoing[j].to].name, chosen_mb->node_array[connections->edges_outgoing[j].to].index); } } fprintf(f, "}\n"); fprintf(fn, "}\n"); fclose(f); fclose(fn); return OP_SUCCESS; } /** * Allocate node indexes to store a node's (at node_index) * node outgoing or incoming connections (nc_edges). */ STATIC enum op_result alloc_node_connections(struct dgsh_edge **nc_edges, int nc_n_edges, int type, int node_index) { if (!nc_edges) { DPRINTF(4, "ERROR: Double pointer to node connection edges is NULL.\n"); return OP_ERROR; } if (node_index < 0) { DPRINTF(4, "ERROR: Index of node whose connections will be allocated is negative number.\n"); return OP_ERROR; } if (type > 1 || type < 0) { DPRINTF(4, "ERROR: Type of edge is neither incoming (1) nor outgoing(0).\ntyep is: %d.\n", type); return OP_ERROR; } *nc_edges = (struct dgsh_edge *)malloc(sizeof(struct dgsh_edge) * nc_n_edges); if (!*nc_edges) { DPRINTF(4, "ERROR: Memory allocation for node's index %d %s connections \ failed.\n", node_index, (type) ? "incoming" : "outgoing"); return OP_ERROR; } return OP_SUCCESS; } /** * Copy the array of pointers to edges that go to or leave from a node * (i.e. its incoming or outgoing connections) to a self-contained compact * array of edges for easy transmission and receipt in one piece. */ STATIC enum op_result make_compact_edge_array(struct dgsh_edge **nc_edges, int nc_n_edges, struct dgsh_edge **p_edges) { int i; int array_size = sizeof(struct dgsh_edge) * nc_n_edges; if (nc_n_edges <= 0) { DPRINTF(4, "ERROR: Size identifier to be used in malloc() is non-positive number: %d.\n", nc_n_edges); return OP_ERROR; } if (nc_edges == NULL) { DPRINTF(4, "ERROR: Compact edge array to put edges (connections) is NULL.\n"); return OP_ERROR; } if (p_edges == NULL) { DPRINTF(4, "ERROR: Pointer to edge array is NULL.\n"); return OP_ERROR; } *nc_edges = (struct dgsh_edge *)malloc(array_size); if (!(*nc_edges)) { DPRINTF(4, "ERROR: Memory allocation of size %d for edge array failed.\n", array_size); return OP_ERROR; } /** * Copy the edges of interest to the node-specific edge array * that contains its connections. */ for (i = 0; i < nc_n_edges; i++) { if (p_edges[i] == NULL) { DPRINTF(4, "ERROR: Pointer to edge array contains NULL pointer.\n"); return OP_ERROR; } /** * Dereference to reach the array base, make i hops of size * sizeof(struct dgsh_edge), and point to that memory block. */ memcpy(&(*nc_edges)[i], p_edges[i], sizeof(struct dgsh_edge)); DPRINTF(4, "%s():Copied edge %d -> %d (%d) at index %d.", __func__, p_edges[i]->from, p_edges[i]->to, p_edges[i]->instances, i); } return OP_SUCCESS; } /* Reallocate array to edge pointers. */ STATIC enum op_result reallocate_edge_pointer_array(struct dgsh_edge ***edge_array, int n_elements) { void **p = NULL; if (edge_array == NULL) { DPRINTF(4, "ERROR: Edge array is NULL pointer.\n"); return OP_ERROR; } if (n_elements <= 0) { DPRINTF(4, "ERROR: Size identifier to be used in malloc() is non-positive number: %d.\n", n_elements); return OP_ERROR; } else if (n_elements == 1) p = malloc(sizeof(struct dgsh_edge *) * n_elements); else p = realloc(*edge_array, sizeof(struct dgsh_edge *) * n_elements); if (!p) { DPRINTF(4, "ERROR: Memory reallocation for edge failed.\n"); return OP_ERROR; } else *edge_array = (struct dgsh_edge **)p; return OP_SUCCESS; } /** * Gather the constraints on a node's input or output channel * and then try to find a solution that respects both the node's * channel constraint and the pair nodes' corresponding channel * constraints if edges on the channel exist. * The function is not called otherwise. * If a solution is found, allocate edge instances to each edge that * includes the node's channel (has to do with the flexible constraint). */ static enum op_result satisfy_io_constraints(int *free_instances, int this_channel_constraint, /* A node's required or * provided constraint * on this channel */ struct dgsh_edge **edges, /* Gathered pointers to edges * of this channel */ int n_edges, /* Number of edges */ bool is_edge_incoming) /* Incoming or outgoing */ { int i; int weight = -1, modulo = 0; if (this_channel_constraint > 0) { *free_instances = this_channel_constraint; weight = this_channel_constraint / n_edges; modulo = this_channel_constraint % n_edges; /* Edges that have no place in actual execution */ } else if (this_channel_constraint == 0) { *free_instances = 0; weight = 0; modulo = 0; } else /* Flexible constraint */ *free_instances = -1; /* Aggregate the constraints for the node's channel. */ for (i = 0; i < n_edges; i++) { if (this_channel_constraint > 0) *free_instances -= weight + (modulo > 0); if (is_edge_incoming) /* Outgoing for the pair node of edge */ edges[i]->to_instances = weight + (modulo > 0); else edges[i]->from_instances = weight + (modulo > 0); if (modulo > 0) modulo--; DPRINTF(4, "%s(): edge from %d to %d, is_edge_incoming: %d, free_instances: %d, weight: %d, modulo: %d, from_instances: %d, to_instances: %d.\n", __func__, edges[i]->from, edges[i]->to, is_edge_incoming, *free_instances, weight, modulo, edges[i]->from_instances, edges[i]->to_instances); } DPRINTF(4, "%s(): Number of edges: %d, this_channel_constraint: %d, free instances: %d.\n", __func__, n_edges, this_channel_constraint, *free_instances); return OP_SUCCESS; } /** * Lookup this tool's edges and store pointers to them in order * to then allow the evaluation of constraints for the current node's * input and output channels. */ static enum op_result dry_match_io_constraints(struct dgsh_node *current_node, struct dgsh_node_connections *current_connections, struct dgsh_edge ***edges_incoming, /* Uninitialised*/ struct dgsh_edge ***edges_outgoing) /* Uninitialised*/ { int n_edges = chosen_mb->n_edges; int n_free_in_channels = current_node->requires_channels; int n_free_out_channels = current_node->provides_channels; int node_index = current_node->index; int *n_edges_incoming = ¤t_connections->n_edges_incoming; int *n_edges_outgoing = ¤t_connections->n_edges_outgoing; int i; assert(node_index < chosen_mb->n_nodes); /* Gather incoming/outgoing edges for node at node_index. */ for (i = 0; i < n_edges; i++) { struct dgsh_edge *edge = &chosen_mb->edge_array[i]; DPRINTF(4, "%s(): edge at index %d from %d to %d, instances %d, from_instances %d, to_instances %d.", __func__, i, edge->from, edge->to, edge->instances, edge->from_instances, edge->to_instances); if (edge->from == node_index) { (*n_edges_outgoing)++; if (reallocate_edge_pointer_array(edges_outgoing, *n_edges_outgoing) == OP_ERROR) return OP_ERROR; (*edges_outgoing)[*n_edges_outgoing - 1] = edge; } if (edge->to == node_index) { (*n_edges_incoming)++; if (reallocate_edge_pointer_array(edges_incoming, *n_edges_incoming) == OP_ERROR) return OP_ERROR; (*edges_incoming)[*n_edges_incoming - 1] = edge; } } DPRINTF(4, "%s(): Node at index %d has %d outgoing edges and %d incoming.", __func__, node_index, *n_edges_outgoing, *n_edges_incoming); /* Record the input/output constraints at node level. */ if (*n_edges_outgoing > 0) if (satisfy_io_constraints( ¤t_connections->n_instances_outgoing_free, n_free_out_channels, *edges_outgoing, *n_edges_outgoing, 0) == OP_ERROR) return OP_ERROR; if (*n_edges_incoming > 0) if (satisfy_io_constraints( ¤t_connections->n_instances_incoming_free, n_free_in_channels, *edges_incoming, *n_edges_incoming, 1) == OP_ERROR) return OP_ERROR; return OP_SUCCESS; } /** * Free the dgsh graph's solution in face of an error. * node_index: the last node we setup conenctions before error. */ static enum op_result free_graph_solution(int node_index) { int i; struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; assert(node_index < chosen_mb->n_nodes); for (i = 0; i <= node_index; i++) { if (graph_solution[i].n_edges_incoming > 0) free(graph_solution[i].edges_incoming); if (graph_solution[i].n_edges_outgoing > 0) free(graph_solution[i].edges_outgoing); } free(graph_solution); chosen_mb->graph_solution = NULL; DPRINTF(4, "%s: freed %d nodes.", __func__, chosen_mb->n_nodes); return OP_SUCCESS; } /** * Add or subtract edge instances from an edge that meets a pair node's * flexible constraint. */ static enum op_result record_move_flexible(int *diff, int *index, int to_move_index, int *instances, int to_move) { if (*diff > 0 || (*diff < 0 && to_move > 1)) { /* In subtracting at least one edge instance should remain. */ if (*diff < 0 && *diff + (to_move - 1) <= 0) *instances = -(to_move - 1); else *instances = *diff; *diff -= *instances; *index = to_move_index; return OP_SUCCESS; } return OP_NOOP; } /** * Add or subtract edge instances from an edge that is unbalanced wrt * the pair node's constraint. */ static enum op_result record_move_unbalanced(int *diff, int *index, int to_move_index, int *instances, int to_move, int pair) { DPRINTF(4, "%s(): to_move: %d, pair: %d, diff: %d", __func__, to_move, pair, *diff); /* Can either to_move or pair be 0? I don't think so */ if ((*diff > 0 && to_move < pair) || (*diff < 0 && to_move > pair)) { *index = to_move_index; if ((*diff > 0 && *diff - (pair - to_move) >= 0) || (*diff < 0 && *diff - (pair - to_move) <= 0)) *instances = pair - to_move; else *instances = *diff; *diff -= *instances; DPRINTF(4, "%s(): move successful: to_move: %d, pair: %d, diff: %d, instances: %d, edge index: %d", __func__, to_move, pair, *diff, *instances, *index); return OP_SUCCESS; } return OP_NOOP; } /** * From the set of unbalanced constraints of a node wrt the pair node's * constraint on a specific channel, that is, input or output, * find instances to subtract or add to satisfy the constraint. * If that does not work try edges where the pair has a flexible constraint. */ static enum op_result move(struct dgsh_edge** edges, int n_edges, int diff, bool is_edge_incoming) { int i = 0, j = 0; int indexes[n_edges]; int instances[n_edges]; /* Try move unbalanced edges first. * Avoid doing the move at the same edge. */ for (i = 0; i < n_edges; i++) { struct dgsh_edge *edge = edges[i]; int *from = &edge->from_instances; int *to = &edge->to_instances; DPRINTF(4, "%s(): before move %s edge %d: from: %d, to: %d, diff %d.", __func__, is_edge_incoming ? "incoming" : "outgoing", i, *from, *to, diff); if (*from == -1 || *to == -1) continue; if (is_edge_incoming) { if (record_move_unbalanced(&diff, &indexes[j], i, &instances[j], *to, *from) == OP_SUCCESS) j++; } else { if (record_move_unbalanced(&diff, &indexes[j], i, &instances[j], *from, *to) == OP_SUCCESS) j++; } DPRINTF(4, "%s(): after move %s edge %d: from: %d, to: %d, diff %d.", __func__, is_edge_incoming ? "incoming" : "outgoing", i, *from, *to, diff); if (diff == 0) goto checkout; } /* Edges with flexible constraints are by default balanced. Try move */ for (i = 0; i < n_edges; i++) { struct dgsh_edge *edge = edges[i]; int *from = &edge->from_instances; int *to = &edge->to_instances; if (is_edge_incoming) { if (*from >= 0) continue; if (record_move_flexible(&diff, &indexes[j], i, &instances[j], *to) == OP_SUCCESS) j++; } else { if (*to >= 0) continue; if (record_move_flexible(&diff, &indexes[j], i, &instances[j], *from) == OP_SUCCESS) j++; } if (diff == 0) goto checkout; } checkout: if (diff == 0) { int k = 0; for (k = 0; k < j; k++) { if (is_edge_incoming) edges[indexes[k]]->to_instances += instances[k]; else edges[indexes[k]]->from_instances += instances[k]; DPRINTF(4, "%s(): succeeded: move %d from edge %d.", __func__, instances[k], indexes[k]); } return OP_SUCCESS; } return OP_RETRY; } /** * Try to find a solution that respects both the node's * channel constraint and the pair nodes' corresponding channel * constraints if edges on the channel exist. * The function is not called otherwise. * If a solution is found, allocate edge instances to each edge that * includes the node's channel (has to do with the flexible constraint). */ static enum op_result cross_match_io_constraints(int *free_instances, int this_channel_constraint, /* A node's required * provided constraint * on the channel */ struct dgsh_edge **edges, /* Gathered pointers * to edges */ int n_edges, /* Number of edges */ bool is_edge_incoming, /* Incoming or outgoing edges*/ bool *constraints_matched, int *edges_matched) { int i; int from_flex = 0; int to_flex = 0; for (i = 0; i < n_edges; i++) { struct dgsh_edge *e = edges[i]; int *from = &e->from_instances; int *to = &e->to_instances; int matched = *edges_matched; if (*from == -1 || *to == -1) { DPRINTF(4, "%s(): edge from %d to %d, this_channel_constraint: %d, is_incoming: %d, from_instances: %d, to_instances %d.\n", __func__, e->from, e->to, this_channel_constraint, is_edge_incoming, *from, *to); if (*from == -1 && *to == -1) { from_flex++; to_flex++; e->instances = 1; // TODO } else if (*from == -1) { from_flex++; e->instances = *to; } else if (*to == -1) { to_flex++; e->instances = *from; } (*edges_matched)++; /* fixed to more than one flexible * is not solvable in the general case */ if (this_channel_constraint > 0 && ((is_edge_incoming && from_flex > 1) || (!is_edge_incoming && to_flex > 1))) { fprintf(stderr, "ERROR: More than one edges are flexible. Cannot compute solution. Exiting.\n"); return OP_ERROR; } } else if (*from == *to) { (*edges_matched)++; e->instances = *from; } else if (*from < *to) { /* e.g. from=1, to=3; then: */ if (is_edge_incoming) { /* +2: */ if (move(edges, n_edges, (*to - *from), 1) == OP_SUCCESS) { *to -= (*to - *from); /* -2: 3 -> 1 */ (*edges_matched)++; } } else if (move(edges, n_edges, -(*to - *from), 0) == OP_SUCCESS) { *from += (*to - *from); (*edges_matched)++; } } else { if (is_edge_incoming) { /* e.g. from=3, to=1 */ if (move(edges, n_edges, -(*from - *to), 1) == OP_SUCCESS) { *to += (*from - *to); (*edges_matched)++; } } else if (move(edges, n_edges, (*from - *to), 0) == OP_SUCCESS) { *from -= (*from - *to); (*edges_matched)++; } } DPRINTF(4, "%s(): edge from %d to %d, this_channel_constraint: %d, is_incoming: %d, from_instances: %d, to_instances %d, edge instances: %d.\n", __func__, e->from, e->to, this_channel_constraint, is_edge_incoming, *from, *to, e->instances); if (matched == *edges_matched){ DPRINTF(4, "%s(): WARNING: did not manage to match this edge", __func__); return OP_SUCCESS; } } /* Is the matching for this channel in line with the (fixed) * constraint? */ if (this_channel_constraint == -1) { *constraints_matched = true; return OP_SUCCESS; } int fds = 0; for (i = 0; i < n_edges; i++) { struct dgsh_edge *e = edges[i]; fds += e->instances; } DPRINTF(4, "%s communication endpoints to setup: %d, constraint: %d", is_edge_incoming ? "Incoming" : "Outgoing", fds, this_channel_constraint); *constraints_matched = (fds == this_channel_constraint); return OP_SUCCESS; } /** * Search for conc with pid in message block mb * and return a pointer to the structure or * NULL if not found. */ struct dgsh_conc * find_conc(struct dgsh_negotiation *mb, pid_t pid) { int i; struct dgsh_conc *ca = mb->conc_array; for (i = 0; i < mb->n_concs; i++) { if (ca[i].pid == pid) return &ca[i]; } return NULL; } /** * Calculate fds for concs at the multi-pipe * endpoint. */ static enum op_result calculate_conc_fds(void) { int i, calculated = 0, retries = 0; int n_concs = chosen_mb->n_concs; DPRINTF(4, "%s for %d n_concs", __func__, n_concs); if (n_concs == 0) return OP_SUCCESS; repeat: for (i = 0; i < n_concs; i++) { struct dgsh_conc *c = &chosen_mb->conc_array[i]; DPRINTF(4, "%s() for conc %d at index %d with %d n_proc_pids", __func__, c->pid, i, c->n_proc_pids); if (c->input_fds >= 0 && c->output_fds >= 0) continue; c->input_fds = 0; c->output_fds = 0; if (c->multiple_inputs) c->output_fds = get_expected_fds_n(chosen_mb, c->endpoint_pid); else c->input_fds = get_provided_fds_n(chosen_mb, c->endpoint_pid); DPRINTF(4, "%s(): conc pid %d at index %d: %d %s fds for endpoint pid %d recovered", __func__, c->pid, i, c->multiple_inputs ? c->output_fds : c->input_fds, c->multiple_inputs ? "outgoing" : "incoming", c->endpoint_pid); int j, fds; for (j = 0; j < c->n_proc_pids; j++) { if (c->multiple_inputs) fds = get_provided_fds_n(chosen_mb, c->proc_pids[j]); else fds = get_expected_fds_n(chosen_mb, c->proc_pids[j]); if (find_conc(chosen_mb, c->proc_pids[j]) && fds == -1) { c->input_fds = c->output_fds = -1; DPRINTF(4, "%s(): conc pid %d at index %d: fds for conc with pid %d not yet available", __func__, c->pid, i, c->proc_pids[j]); break; } else if (c->multiple_inputs) c->input_fds += fds; else c->output_fds += fds; DPRINTF(4, "%s(): conc pid %d at index %d: %d %s fds for pid %d recovered", __func__, c->pid, i, fds, c->multiple_inputs ? "incoming" : "outgoing", c->proc_pids[j]); } // Use what we know for the multi-pipe end to compute the endpoint if (c->multiple_inputs && c->input_fds >= 0 && c->output_fds == -1) c->output_fds = c->input_fds; else if (!c->multiple_inputs && c->output_fds >= 0 && c->input_fds == -1) c->input_fds = c->output_fds; if (c->input_fds >= 0 && c->output_fds >= 0) { assert(c->input_fds == c->output_fds); calculated++; } DPRINTF(4, "%s(): Conc pid %d at index %d has %d %s fds and %d %s fds", __func__, c->pid, i, c->multiple_inputs ? c->input_fds : c->output_fds, c->multiple_inputs ? "incoming" : "outgoing", c->multiple_inputs ? c->output_fds : c->input_fds, c->multiple_inputs ? "outgoing" : "incoming"); DPRINTF(4, "%s(): Calculated fds for %d concs so far", __func__, calculated); } if (calculated != n_concs && retries < n_concs) { retries++; goto repeat; } if (retries == n_concs) return OP_ERROR; return OP_SUCCESS; } /** * For each node substitute pointers to edges with proper edge structures * (copies) to facilitate transmission and receipt in one piece. */ static enum op_result prepare_solution(void) { int i; int n_nodes = chosen_mb->n_nodes; struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; enum op_result exit_state = OP_SUCCESS; for (i = 0; i < n_nodes; i++) { struct dgsh_node_connections *current_connections = &graph_solution[i]; /* Hack: struct dgsh_edge* -> struct dgsh_edge** */ struct dgsh_edge **edges_incoming = (struct dgsh_edge **)current_connections->edges_incoming; current_connections->edges_incoming = NULL; /* Hack: struct dgsh_edge* -> struct dgsh_edge** */ struct dgsh_edge **edges_outgoing = (struct dgsh_edge **)current_connections->edges_outgoing; current_connections->edges_outgoing = NULL; int *n_edges_incoming = ¤t_connections->n_edges_incoming; int *n_edges_outgoing = ¤t_connections->n_edges_outgoing; DPRINTF(3, "%s(): Node %s, pid: %d, connections in: %d, connections out: %d.", __func__, chosen_mb->node_array[i].name, chosen_mb->node_array[i].pid, *n_edges_incoming, *n_edges_outgoing); if (*n_edges_incoming > 0) { if (exit_state == OP_SUCCESS) if (make_compact_edge_array( ¤t_connections->edges_incoming, *n_edges_incoming, edges_incoming) == OP_ERROR) exit_state = OP_ERROR; free(edges_incoming); } if (*n_edges_outgoing > 0) { if (exit_state == OP_SUCCESS) if (make_compact_edge_array( ¤t_connections->edges_outgoing, *n_edges_outgoing, edges_outgoing) == OP_ERROR) exit_state = OP_ERROR; free(edges_outgoing); } } return exit_state; } static void print_solution_error(int index_argc, int *index_commands_notmatched, int *side_commands_notmatched) { int i = 0, index = 0, side = 0, reqs = 0; fprintf(stderr, "dgsh: No solution was found to satisfy the I/O requirements of the following %d participating processes:\n", index_argc); for (i = 0; i < index_argc; i++) { index = index_commands_notmatched[i]; side = side_commands_notmatched[i]; if (side == STDIN_FILENO) reqs = chosen_mb->node_array[index].requires_channels; else reqs = chosen_mb->node_array[index].provides_channels; fprintf(stderr, "%s (n%s=%d)\n", chosen_mb->node_array[index].name, side == STDIN_FILENO ? "in" : "out", reqs); } free(index_commands_notmatched); free(side_commands_notmatched); } /** * Check that a node's input/output channel matched its requirements */ static void check_constraints_matched(int node_index, bool *constraints_matched, int **index_commands_notmatched, int **side_commands_notmatched, int *index_argc, int side) { if (!*constraints_matched) { (*index_argc)++; DPRINTF(4, "Constraint not matched for node at index %d. So far %d nodes not matched", node_index, *index_argc); if (*index_argc == 1) { *index_commands_notmatched = (int *)malloc( sizeof(int) * *index_argc); *side_commands_notmatched = (int *)malloc( sizeof(int) * *index_argc); } else { *index_commands_notmatched = (int *)realloc( *index_commands_notmatched, sizeof(int) * *index_argc); *side_commands_notmatched = (int *)realloc( *side_commands_notmatched, sizeof(int) * *index_argc); } (*index_commands_notmatched)[*index_argc - 1] = node_index; (*side_commands_notmatched)[*index_argc - 1] = side; } *constraints_matched = false; } /** * This function implements the algorithm that tries to satisfy reported * I/O constraints of tools on an dgsh graph. */ static enum op_result cross_match_constraints(int **index_commands_notmatched, int **side_commands_notmatched, int *index_argc) { int i; int n_nodes = chosen_mb->n_nodes; int n_edges = chosen_mb->n_edges; int edges_matched = 0; bool constraints_matched = false; struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; /* Check constraints for each node on the dgsh graph. */ for (i = 0; i < n_nodes; i++) { struct dgsh_node_connections *current_connections = &graph_solution[i]; /* Hack: struct dgsh_edge* -> struct dgsh_edge** */ struct dgsh_edge **edges_incoming = (struct dgsh_edge **)current_connections->edges_incoming; /* Hack: struct dgsh_edge* -> struct dgsh_edge** */ struct dgsh_edge **edges_outgoing = (struct dgsh_edge **)current_connections->edges_outgoing; struct dgsh_node *current_node = &chosen_mb->node_array[i]; int out_constraint = current_node->provides_channels; int in_constraint = current_node->requires_channels; int *n_edges_incoming = ¤t_connections->n_edges_incoming; int *n_edges_outgoing = ¤t_connections->n_edges_outgoing; DPRINTF(4, "%s(): node %s, index %d, channels required %d, channels_provided %d, dgsh_in %d, dgsh_out %d.", __func__, current_node->name, current_node->index, in_constraint, out_constraint, current_node->dgsh_in, current_node->dgsh_out); /* Try to satisfy the I/O channel constraints at graph level. * Assign instances to each edge. */ if (*n_edges_outgoing > 0){ if (cross_match_io_constraints( ¤t_connections->n_instances_outgoing_free, out_constraint, edges_outgoing, *n_edges_outgoing, 0, &constraints_matched, &edges_matched) == OP_ERROR) { DPRINTF(4, "ERROR: Failed to satisfy requirements for tool %s, pid %d: requires %d and gets %d, provides %d and is offered %d.\n", current_node->name, current_node->pid, current_node->requires_channels, *n_edges_incoming, current_node->provides_channels, *n_edges_outgoing); return OP_ERROR; } check_constraints_matched(i, &constraints_matched, index_commands_notmatched, side_commands_notmatched, index_argc, STDOUT_FILENO); } if (*n_edges_incoming > 0){ if (cross_match_io_constraints( ¤t_connections->n_instances_incoming_free, in_constraint, edges_incoming, *n_edges_incoming, 1, &constraints_matched, &edges_matched) == OP_ERROR) { DPRINTF(4, "ERROR: Failed to satisfy requirements for tool %s, pid %d: requires %d and gets %d, provides %d and is offered %d.\n", current_node->name, current_node->pid, current_node->requires_channels, *n_edges_incoming, current_node->provides_channels, *n_edges_outgoing); return OP_ERROR; } check_constraints_matched(i, &constraints_matched, index_commands_notmatched, side_commands_notmatched, index_argc, STDIN_FILENO); } } DPRINTF(4, "%s(): Cross matched constraints of %d out of %d nodes for %d edges out of %d edges.", __func__, n_nodes - *index_argc, n_nodes, edges_matched / 2, n_edges); if (edges_matched / 2 == n_edges && *index_argc == 0) return OP_SUCCESS; else return OP_RETRY; } /** * This function implements the algorithm that tries to satisfy reported * I/O constraints of tools on an dgsh graph. */ static enum op_result node_match_constraints(void) { int i; int n_nodes = chosen_mb->n_nodes; enum op_result exit_state = OP_SUCCESS; int graph_solution_size = sizeof(struct dgsh_node_connections) * n_nodes; /* Prealloc */ chosen_mb->graph_solution = (struct dgsh_node_connections *)malloc( graph_solution_size); struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; if (!graph_solution) { DPRINTF(4, "ERROR: Failed to allocate memory of size %d for dgsh negotiation graph solution structure.\n", graph_solution_size); return OP_ERROR; } /* Check constraints for each node on the dgsh graph. */ for (i = 0; i < n_nodes; i++) { DPRINTF(4, "%s(): node at index %d.", __func__, i); struct dgsh_node_connections *current_connections = &graph_solution[i]; memset(current_connections, 0, sizeof(struct dgsh_node_connections)); struct dgsh_edge **edges_incoming; struct dgsh_edge **edges_outgoing; struct dgsh_node *current_node = &chosen_mb->node_array[i]; current_connections->node_index = current_node->index; DPRINTF(4, "Node %s, index %d, channels required %d, channels_provided %d, dgsh_in %d, dgsh_out %d.", current_node->name, current_node->index, current_node->requires_channels, current_node->provides_channels, current_node->dgsh_in, current_node->dgsh_out); /* Find and store pointers to node's at node_index edges. * Try to satisfy the I/O channel constraints at node level. */ if (dry_match_io_constraints(current_node, current_connections, &edges_incoming, &edges_outgoing) == OP_ERROR) { DPRINTF(4, "ERROR: Failed to satisfy requirements for tool %s, pid %d: requires %d and gets %d, provides %d and is offered %d.\n", current_node->name, current_node->pid, current_node->requires_channels, current_connections->n_edges_incoming, current_node->provides_channels, current_connections->n_edges_outgoing); exit_state = OP_ERROR; } if (exit_state == OP_ERROR) { free_graph_solution(current_node->index); break; } /* Hack to retain references to edge pointer arrays. */ current_connections->edges_incoming = (struct dgsh_edge *)edges_incoming; current_connections->edges_outgoing = (struct dgsh_edge *)edges_outgoing; } return exit_state; } /** * This function implements the algorithm that tries to satisfy reported * I/O constraints of tools on an dgsh graph. */ enum op_result solve_graph(void) { char *filename; enum op_result exit_state = OP_SUCCESS; int retries = 0; int index_argc = 0; int *index_commands_notmatched; int *side_commands_notmatched; /** * The initial layout of the solution plays an important * role. We could add a scheme that allows restarting * the solution process with a different initial layout * after a failure. The key points of the scheme would be: * while * int retries = 0; * int max_retries = 10; * assign a node's available edges to adjacent nodes differently */ /** * Try to match each node's I/O resources with constraints * expressed by incoming and outgoing edges. */ if ((exit_state = node_match_constraints()) == OP_ERROR) return exit_state; /* Optimise solution using flexible constraints */ exit_state = OP_RETRY; while (exit_state == OP_RETRY) { if ((exit_state = cross_match_constraints( &index_commands_notmatched, &side_commands_notmatched, &index_argc)) == OP_ERROR || (exit_state == OP_RETRY && retries > 10)) { print_solution_error(index_argc, index_commands_notmatched, side_commands_notmatched); exit_state = OP_ERROR; goto exit; } DPRINTF(4, "%s(): exit_state: %d, retries: %d", __func__, exit_state, retries); retries++; if (index_argc > 0) { free(index_commands_notmatched); free(side_commands_notmatched); index_argc = 0; } } /** * Substitute pointers to edges with proper edge structures * (copies) to facilitate transmission and receipt in one piece. */ if ((exit_state = prepare_solution()) == OP_ERROR) goto exit; if ((exit_state = calculate_conc_fds()) == OP_ERROR) goto exit; if ((filename = getenv("DGSH_DOT_DRAW"))) if ((exit_state = output_graph(filename)) == OP_ERROR) goto exit; if (getenv("DGSH_DRAW_EXIT")) { DPRINTF(1, "Document the solution and exit\n"); exit_state = OP_DRAW_EXIT; } DPRINTF(4, "%s: exit_state: %d", __func__, exit_state); exit: if (exit_state == OP_ERROR || exit_state == OP_DRAW_EXIT) free_graph_solution(chosen_mb->n_nodes - 1); return exit_state; } /* memory deallocation when in error state? */ /** * Assign the pipes to the data structs that will carry them back to the tool. * The tool is responsible for freeing the memory allocated to these data * structures. */ static enum op_result establish_io_connections(int **input_fds, int *n_input_fds, int **output_fds, int *n_output_fds) { enum op_result re = OP_SUCCESS; DPRINTF(4, "%s(): input fds: %d, output fds: %d", __func__, self_pipe_fds.n_input_fds, self_pipe_fds.n_output_fds); if (self_pipe_fds.n_input_fds > 0) { /* Have the first returned file descriptor * take the place of stdin. */ int fd_to_dup = self_pipe_fds.input_fds[0]; if (close(STDIN_FILENO) == -1) err(1, "Close stdin failed"); if ((self_pipe_fds.input_fds[0] = dup(fd_to_dup)) == -1) err(1, "dup failed with errno %d", errno); DPRINTF(4, "%s(): closed STDIN, dup %d returned %d", __func__,fd_to_dup, self_pipe_fds.input_fds[0]); assert(self_pipe_fds.input_fds[0] == STDIN_FILENO); close(fd_to_dup); if (n_input_fds) { *n_input_fds = self_pipe_fds.n_input_fds; assert(*n_input_fds >= 0); if (input_fds) *input_fds = self_pipe_fds.input_fds; } else { self_pipe_fds.n_input_fds = 0; free(self_pipe_fds.input_fds); } } else if (n_input_fds) *n_input_fds = 0; if (self_pipe_fds.n_output_fds > 0) { /* Have the first returned file descriptor * take the place of stdin. */ int fd_to_dup = self_pipe_fds.output_fds[0]; if (close(STDOUT_FILENO) == -1) err(1, "Close stdout failed"); if ((self_pipe_fds.output_fds[0] = dup(fd_to_dup)) == -1) err(1, "dup failed with errno %d", errno); DPRINTF(4, "%s(): closed STDOUT, dup %d returned %d", __func__,fd_to_dup,self_pipe_fds.output_fds[0]); assert(self_pipe_fds.output_fds[0] == STDOUT_FILENO); close(fd_to_dup); if (n_output_fds) { *n_output_fds = self_pipe_fds.n_output_fds; assert(*n_output_fds >= 0); if (output_fds) *output_fds = self_pipe_fds.output_fds; } else { self_pipe_fds.n_output_fds = 0; free(self_pipe_fds.output_fds); } } else if (n_output_fds) *n_output_fds = 0; DPRINTF(2, "%s(): %s for node %s at index %d", __func__, (re == OP_SUCCESS ? "successful" : "failed"), self_node.name, self_node.index); return re; } /* Transmit file descriptors that will pipe this * tool's output to another tool. */ static enum op_result write_output_fds(int output_socket, int *output_fds, int flags) { /** * A node's connections are located at the same position * as the node in the node array. */ struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; struct dgsh_node_connections *this_nc = &graph_solution[self_node.index]; DPRINTF(4, "%s(): for node at index %d with %d outgoing edges.", __func__, self_node.index, this_nc->n_edges_outgoing); assert(this_nc->node_index == self_node.index); int i; int total_edge_instances = 0; enum op_result re = OP_SUCCESS; /** * Create a pipe for each instance of each outgoing edge connection. * Inject the pipe read side in the cont. * Send each pipe fd as a message to a socket descriptor, * that is write_fd, that has been * set up by the shell to support the dgsh negotiation phase. */ for (i = 0; i < this_nc->n_edges_outgoing; i++) { int k; /** * Due to channel constraint flexibility, * each edge can have more than one instances. */ for (k = 0; k < this_nc->edges_outgoing[i].instances; k++) { int fd[2]; /* Create pipe, inject the read side to the msg control * data and close the read side to let the recipient * process handle it. */ if (pipe(fd) == -1) { perror("pipe open failed"); dgsh_exit(-1, flags); } DPRINTF(4, "%s(): created pipe pair %d - %d. Transmitting fd %d through sendmsg().", __func__, fd[0], fd[1], fd[0]); write_fd(output_socket, fd[0]); close(fd[0]); output_fds[total_edge_instances] = fd[1]; total_edge_instances++; } /* XXX */ if (re == OP_ERROR) break; } if (re == OP_ERROR) { DPRINTF(4, "%s(): ERROR. Aborting.", __func__); free_graph_solution(chosen_mb->n_nodes - 1); free(self_pipe_fds.output_fds); } return re; } static int write_piece (int write_fd, void *datastruct, int struct_size) { int retries = 0, wsize; retry: DPRINTF(4, "Try write struct of size: %d", struct_size); wsize = write(write_fd, datastruct, struct_size); if (wsize == -1 && errno == ENOBUFS && retries < 3) { // sleep for 10ms nanosleep((const struct timespec[]){{0, 10000000L}}, NULL); retries++; goto retry; } return wsize; } static int get_struct_size(int struct_type) { switch (struct_type) { case 1: return sizeof(struct dgsh_node); case 2: return sizeof(struct dgsh_edge); case 3: return sizeof(struct dgsh_conc); case 4: return sizeof(struct dgsh_node_connections); } return 0; } static int do_write (int write_fd, void *datastruct, int datastruct_size, int struct_type) { int wsize = 0; if (datastruct_size > iov_max) { int all_elements, max_elements, elements = 0, pieces, prev_elements = 0, size, struct_size, i; struct_size = get_struct_size(struct_type); all_elements = datastruct_size / struct_size; max_elements = iov_max / struct_size; pieces = all_elements / max_elements; pieces += (all_elements % max_elements > 0); DPRINTF(4, "struct_type: %d, pieces: %d, all_elements: %d, max_elements: %d", struct_type, pieces, all_elements, max_elements); for (i = 0; i < pieces; i++) { if (all_elements - max_elements > 0) elements = max_elements; else if (all_elements > 0) elements = all_elements; all_elements -= elements; size = struct_size * elements; DPRINTF(4, "Round %d: elements: %d, size: %d, prev_elements: %d", i, elements, size, prev_elements); if (size > 0) { void *struct_piece = malloc(size); switch (struct_type) { case 1: memcpy(struct_piece, &((struct dgsh_node *) datastruct)[i * prev_elements], size); break; case 2: memcpy(struct_piece, &((struct dgsh_edge *) datastruct)[i * prev_elements], size); break; case 3: memcpy(struct_piece, &((struct dgsh_conc *) datastruct)[i * prev_elements], size); break; case 4: memcpy(struct_piece, &((struct dgsh_node_connections *) datastruct)[i * prev_elements], size); } wsize = write_piece(write_fd, struct_piece, size); free(struct_piece); prev_elements = elements; } } } else wsize = write_piece(write_fd, datastruct, datastruct_size); return wsize; } static enum op_result write_concs(int write_fd) { int wsize, i; int n_concs = chosen_mb->n_concs; int conc_size = sizeof(struct dgsh_conc) * n_concs; if (!chosen_mb->conc_array) return OP_SUCCESS; wsize = do_write(write_fd, chosen_mb->conc_array, conc_size, 3); if (wsize == -1) { DPRINTF(4, "ERROR: write failed: errno: %d", errno); return OP_ERROR; } DPRINTF(4, "%s(): Wrote conc structures of size %d bytes ", __func__, wsize); for (i = 0; i < n_concs; i++) { struct dgsh_conc *c = &chosen_mb->conc_array[i]; int proc_pids_size = sizeof(int) * c->n_proc_pids; wsize = do_write(write_fd, c->proc_pids, proc_pids_size, 5); if (wsize == -1) { DPRINTF(4, "ERROR: write failed: errno: %d", errno); return OP_ERROR; } DPRINTF(4, "%s(): Wrote %d proc_pids for conc %d at index %d of size %d bytes ", __func__, c->n_proc_pids, c->pid, i, wsize); } return OP_SUCCESS; } /* Transmit dgsh negotiation graph solution to the next tool on the graph. */ static enum op_result write_graph_solution(int write_fd) { int i; int n_nodes = chosen_mb->n_nodes; int graph_solution_size = sizeof(struct dgsh_node_connections) * n_nodes; struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; int wsize = -1; /* Transmit node connection structures. */ wsize = do_write(write_fd, graph_solution, graph_solution_size, 4); if (wsize == -1) { DPRINTF(4, "ERROR: write failed: errno: %d", errno); return OP_ERROR; } DPRINTF(4, "%s(): Wrote graph solution of size %d bytes ", __func__, wsize); /* We haven't invalidated pointers to arrays of node indices. */ for (i = 0; i < n_nodes; i++) { struct dgsh_node_connections *nc = &graph_solution[i]; int in_edges_size = sizeof(struct dgsh_edge) * nc->n_edges_incoming; int out_edges_size = sizeof(struct dgsh_edge) * nc->n_edges_outgoing; if (nc->n_edges_incoming) { /* Transmit a node's incoming connections. */ wsize = do_write(write_fd, nc->edges_incoming, in_edges_size, 2); if (wsize == -1) { DPRINTF(4, "ERROR: write failed: errno: %d", errno); return OP_ERROR; } DPRINTF(4, "%s(): Wrote node's %d %d incoming edges of size %d bytes ", __func__, nc->node_index, nc->n_edges_incoming, wsize); } if (nc->n_edges_outgoing) { /* Transmit a node's outgoing connections. */ wsize = do_write(write_fd, nc->edges_outgoing, out_edges_size, 2); if (wsize == -1) { DPRINTF(4, "ERROR: write failed: errno: %d", errno); return OP_ERROR; } DPRINTF(4, "%s(): Wrote node's %d %d outgoing edges of size %d bytes ", __func__, nc->node_index, nc->n_edges_outgoing, wsize); } } return OP_SUCCESS; } /** * Copy the dispatcher static object that identifies the node * in the message block node array and shows the write point of * the send operation. This is a deep copy for simplicity. */ static void set_dispatcher(void) { chosen_mb->origin_index = self_node_io_side.index; /* The process preceding the one to find the solution * will eventually set its PID. */ assert(self_node_io_side.index >= 0); /* Node is added to the graph. */ chosen_mb->origin_fd_direction = self_node_io_side.fd_direction; chosen_mb->is_origin_conc = false; chosen_mb->conc_pid = -1; DPRINTF(4, "%s(): message block origin set to %d and writing on the %s side", __func__, chosen_mb->origin_index, (chosen_mb->origin_fd_direction == 0) ? "input" : "output"); } /* * Write the chosen_mb message block to the specified file descriptor. */ enum op_result write_message_block(int write_fd) { int wsize = -1; int mb_size = sizeof(struct dgsh_negotiation); int nodes_size = chosen_mb->n_nodes * sizeof(struct dgsh_node); int edges_size = chosen_mb->n_edges * sizeof(struct dgsh_edge); struct dgsh_node *p_nodes = chosen_mb->node_array; DPRINTF(3, "%s(): %s (%d)", __func__, programname, self_node.index); if (chosen_mb->state == PS_ERROR && errno == 0) errno = EPROTO; /** * Prepare and perform message block transmission. * Formally invalidate pointers to nodes and edges * to avoid accidents on the receiver's side. */ chosen_mb->node_array = NULL; DPRINTF(4, "%s(): Write message block.", __func__); wsize = do_write(write_fd, chosen_mb, mb_size, 0); if (wsize == -1) { DPRINTF(4, "ERROR: write failed: errno: %d", errno); return OP_ERROR; } DPRINTF(4, "%s(): Wrote message block of size %d bytes ", __func__, wsize); /* Transmit nodes. */ if (chosen_mb->n_nodes > 0) { wsize = do_write(write_fd, p_nodes, nodes_size, 1); if (wsize == -1) { DPRINTF(4, "ERROR: write failed: errno: %d", errno); return OP_ERROR; } DPRINTF(4, "%s(): Wrote nodes of size %d bytes ", __func__, wsize); } chosen_mb->node_array = p_nodes; // Reinstate pointers to nodes. if (write_concs(write_fd) == OP_ERROR) return OP_ERROR; if (chosen_mb->state == PS_NEGOTIATION) { if (chosen_mb->n_edges > 0) { /* Transmit edges. */ struct dgsh_edge *p_edges = chosen_mb->edge_array; chosen_mb->edge_array = NULL; wsize = do_write(write_fd, p_edges, edges_size, 2); if (wsize == -1) { DPRINTF(4, "ERROR: write failed: errno: %d", errno); return OP_ERROR; } DPRINTF(4, "%s(): Wrote edges of size %d bytes ", __func__, wsize); chosen_mb->edge_array = p_edges; /* Reinstate edges. */ } } else if (chosen_mb->state == PS_RUN) { /* Transmit solution. */ if (write_graph_solution(write_fd) == OP_ERROR) return OP_ERROR; } DPRINTF(4, "%s(): Shipped message block or solution to next node in graph from file descriptor: %d.\n", __func__, write_fd); return OP_SUCCESS; } /* Reallocate message block to fit new node coming in. */ static enum op_result add_node(void) { int n_nodes = chosen_mb->n_nodes; void *p = realloc(chosen_mb->node_array, sizeof(struct dgsh_node) * (n_nodes + 1)); if (!p) { DPRINTF(4, "ERROR: Node array expansion for adding a new node failed.\n"); return OP_ERROR; } else { chosen_mb->node_array = (struct dgsh_node *)p; self_node.index = n_nodes; memcpy(&chosen_mb->node_array[n_nodes], &self_node, sizeof(struct dgsh_node)); self_node_io_side.index = n_nodes; DPRINTF(2, "%s(): Added node %s in position %d on dgsh graph, initiator: %d", __func__, self_node.name, self_node_io_side.index, chosen_mb->initiator_pid); chosen_mb->n_nodes++; } return OP_SUCCESS; } /* Lookup an edge in the dgsh graph. */ static enum op_result lookup_dgsh_edge(struct dgsh_edge *e) { int i; for (i = 0; i < chosen_mb->n_edges; i++) { if ((chosen_mb->edge_array[i].from == e->from && chosen_mb->edge_array[i].to == e->to) || (chosen_mb->edge_array[i].from == e->to && chosen_mb->edge_array[i].to == e->from)) { DPRINTF(4, "%s(): Edge %d to %d exists.", __func__, e->from, e->to); return OP_EXISTS; } } return OP_CREATE; } /** * Fill edge depending on input/output fd information * passed by sender and found in receiver (this tool or self). */ static enum op_result fill_dgsh_edge(struct dgsh_edge *e) { int i; int n_nodes = chosen_mb->n_nodes; for (i = 0; i < n_nodes; i++) /* Check dispatcher node exists. */ if (i == chosen_mb->origin_index) break; if (i == n_nodes) { DPRINTF(4, "ERROR: Dispatcher node with index position %d not present in graph.\n", chosen_mb->origin_index); return OP_ERROR; } if (chosen_mb->origin_fd_direction == STDIN_FILENO) { /** * MB sent from stdin, so dispatcher is the destination of the edge. * Self should be dgsh-active on output side. Self's current fd is stdin * if self is dgsh-active on input side or output side otherwise. * Self (the recipient) is the source of the edge. */ e->to = chosen_mb->origin_index; if (self_node.dgsh_in == 1) self_node_io_side.fd_direction = STDIN_FILENO; else self_node_io_side.fd_direction = STDOUT_FILENO; assert((self_node.dgsh_in && self_node_io_side.fd_direction == STDIN_FILENO) || self_node_io_side.fd_direction == STDOUT_FILENO); e->from = self_node_io_side.index; } else if (chosen_mb->origin_fd_direction == STDOUT_FILENO) { /* Similarly. */ e->from = chosen_mb->origin_index; if (self_node.dgsh_out == 1) self_node_io_side.fd_direction = STDOUT_FILENO; else self_node_io_side.fd_direction = STDIN_FILENO; assert((self_node.dgsh_out && self_node_io_side.fd_direction == STDOUT_FILENO) || self_node_io_side.fd_direction == STDIN_FILENO); e->to = self_node_io_side.index; } assert(e->from != e->to); e->instances = 0; e->from_instances = 0; e->to_instances = 0; DPRINTF(4, "New dgsh edge from %d to %d with %d instances.", e->from, e->to, e->instances); return OP_SUCCESS; } /* Add new edge coming in. */ static enum op_result add_edge(struct dgsh_edge *edge) { int n_edges = chosen_mb->n_edges; void *p = realloc(chosen_mb->edge_array, sizeof(struct dgsh_edge) * (n_edges + 1)); if (!p) { DPRINTF(4, "ERROR: Edge array expansion for adding a new edge failed.\n"); return OP_ERROR; } else { chosen_mb->edge_array = (struct dgsh_edge *)p; memcpy(&chosen_mb->edge_array[n_edges], edge, sizeof(struct dgsh_edge)); DPRINTF(4, "Added edge (%d -> %d) in dgsh graph.\n", edge->from, edge->to); chosen_mb->n_edges++; } return OP_SUCCESS; } /* Try to add a newly occured edge in the dgsh graph. */ static enum op_result try_add_dgsh_edge(void) { if (chosen_mb->origin_index >= 0) { /* If MB not created just now: */ struct dgsh_edge new_edge; fill_dgsh_edge(&new_edge); if (lookup_dgsh_edge(&new_edge) == OP_CREATE) { if (add_edge(&new_edge) == OP_ERROR) return OP_ERROR; DPRINTF(4, "Dgsh graph now has %d edges.\n", chosen_mb->n_edges); return OP_SUCCESS; } return OP_EXISTS; } return OP_NOOP; } /* A constructor-like function for struct dgsh_node. */ static void fill_node(const char *tool_name, pid_t self_pid, int *n_input_fds, int *n_output_fds) { self_node.pid = self_pid; memcpy(self_node.name, tool_name, strlen(tool_name) + 1); if (n_input_fds == NULL) if (self_node.dgsh_in) self_node.requires_channels = 1; else self_node.requires_channels = 0; else self_node.requires_channels = *n_input_fds; DPRINTF(4, "%s(): dgsh_in: %d, self_node.requires_channels: %d", __func__, self_node.dgsh_in, self_node.requires_channels); if (n_output_fds == NULL) { if (self_node.dgsh_out) self_node.provides_channels = 1; else self_node.provides_channels = 0; } else self_node.provides_channels = *n_output_fds; DPRINTF(4, "%s(): dgsh_out: %d, self_node.provides_channels: %d", __func__, self_node.dgsh_out, self_node.provides_channels); DPRINTF(4, "Dgsh node for tool %s with pid %d created.\n", tool_name, self_pid); } /** * Add node to message block. Copy the node using offset-based * calculation from the start of the array of nodes. */ static enum op_result try_add_dgsh_node(const char *tool_name, pid_t self_pid, int *n_input_fds, int *n_output_fds) { int n_nodes = chosen_mb->n_nodes; int i; for (i = 0; i < n_nodes; i++) { DPRINTF(4, "node name: %s, pid: %d", chosen_mb->node_array[i].name, chosen_mb->node_array[i].pid); if (chosen_mb->node_array[i].pid == self_pid) break; } if (i == n_nodes) { fill_node(tool_name, self_pid, n_input_fds, n_output_fds); if (add_node() == OP_ERROR) return OP_ERROR; DPRINTF(4, "Dgsh graph now has %d nodes.\n", chosen_mb->n_nodes); return OP_SUCCESS; } return OP_EXISTS; } static void free_conc_array(struct dgsh_negotiation *mb) { int i, n_concs = mb->n_concs; for (i = 0; i < n_concs; i++) if (mb->conc_array[i].proc_pids) free(mb->conc_array[i].proc_pids); free(mb->conc_array); } /* Deallocate message block together with nodes and edges. */ void free_mb(struct dgsh_negotiation *mb) { if (mb->graph_solution) free_graph_solution(mb->n_nodes - 1); if (mb->node_array) free(mb->node_array); if (mb->edge_array) free(mb->edge_array); if (mb->conc_array) free_conc_array(mb); free(mb); DPRINTF(4, "%s(): Freed message block.", __func__); } static enum op_result register_node_edge(const char *tool_name, pid_t self_pid, int *n_input_fds, int *n_output_fds) { /* Create dgsh node representation and add node, edge to the graph. */ if (try_add_dgsh_node(tool_name, self_pid, n_input_fds, n_output_fds) == OP_ERROR) return OP_ERROR; if (try_add_dgsh_edge() == OP_ERROR) return OP_ERROR; return OP_SUCCESS; } /** * Check if the arrived message block preexists our chosen one * and substitute the chosen if so. * If the arrived message block is younger discard it and don't * forward it. * If the arrived is the chosen, try to add the edge. */ static enum op_result analyse_read(struct dgsh_negotiation *fresh_mb, int *ntimes_seen_run, int *ntimes_seen_error, int *ntimes_seen_draw_exit, const char *tool_name, pid_t pid, int *n_input_fds, int *n_output_fds) { if (fresh_mb != NULL) { if (chosen_mb != NULL) free(chosen_mb); chosen_mb = fresh_mb; } else if (chosen_mb == NULL) construct_message_block(tool_name, pid); if (init_error) chosen_mb->state = PS_ERROR; if (chosen_mb->state == PS_ERROR) { if (errno == 0) errno = ECONNRESET; if (chosen_mb->is_error_confirmed) (*ntimes_seen_error)++; } else if (chosen_mb->state == PS_DRAW_EXIT) (*ntimes_seen_draw_exit)++; else if (chosen_mb->state == PS_RUN) (*ntimes_seen_run)++; else if (chosen_mb->state == PS_NEGOTIATION) if (register_node_edge(tool_name, pid, n_input_fds, n_output_fds) == OP_ERROR) chosen_mb->state = PS_ERROR; return OP_SUCCESS; } static enum op_result check_read(int bytes_read, int buf_size, int expected_read_size) { if (bytes_read != expected_read_size) { DPRINTF(4, "%s(): ERROR: Read %d bytes of message block, expected to read %d.\n", __func__, bytes_read, expected_read_size); return OP_ERROR; } if (bytes_read > buf_size) { DPRINTF(4, "%s(): ERROR: Read %d bytes of message block, but buffer can hold up to %d.", __func__, bytes_read, buf_size); return OP_ERROR; } return OP_SUCCESS; } static enum op_result alloc_copy_proc_pids(struct dgsh_conc *c, char *buf, int bytes_read, int buf_size) { int expected_read_size = sizeof(int) * c->n_proc_pids; if (check_read(bytes_read, buf_size, expected_read_size) == OP_ERROR) return OP_ERROR; c->proc_pids = (int *)malloc(bytes_read); memcpy(c->proc_pids, buf, bytes_read); return OP_SUCCESS; } static enum op_result alloc_copy_concs(struct dgsh_negotiation *mb, char *buf, int bytes_read, int buf_size) { int expected_read_size = sizeof(struct dgsh_conc) * mb->n_concs; if (check_read(bytes_read, buf_size, expected_read_size) == OP_ERROR) return OP_ERROR; mb->conc_array = (struct dgsh_conc *)malloc(bytes_read); memcpy(mb->conc_array, buf, bytes_read); return OP_SUCCESS; } /* Allocate memory for graph solution and copy from buffer. */ static enum op_result alloc_copy_graph_solution(struct dgsh_negotiation *mb, char *buf, int bytes_read, int buf_size) { int expected_read_size = sizeof(struct dgsh_node_connections) * mb->n_nodes; if (check_read(bytes_read, buf_size, expected_read_size) == OP_ERROR) return OP_ERROR; mb->graph_solution = (struct dgsh_node_connections *)malloc(bytes_read); memcpy(mb->graph_solution, buf, bytes_read); return OP_SUCCESS; } /* Allocate memory for message_block edges and copy from buffer. */ static enum op_result alloc_copy_edges(struct dgsh_negotiation *mb, char *buf, int bytes_read, int buf_size) { int expected_read_size = sizeof(struct dgsh_edge) * mb->n_edges; if (check_read(bytes_read, buf_size, expected_read_size) == OP_ERROR) return OP_ERROR; mb->edge_array = (struct dgsh_edge *)malloc(bytes_read); memcpy(mb->edge_array, buf, bytes_read); return OP_SUCCESS; } /* Allocate memory for message_block nodes and copy from buffer. */ static enum op_result alloc_copy_nodes(struct dgsh_negotiation *mb, char *buf, int bytes_read, int buf_size) { int expected_read_size = sizeof(struct dgsh_node) * mb->n_nodes; if (check_read(bytes_read, buf_size, expected_read_size) == OP_ERROR) return OP_ERROR; mb->node_array = (struct dgsh_node *)malloc(bytes_read); memcpy(mb->node_array, buf, bytes_read); DPRINTF(4, "%s(): Node array recovered.", __func__); return OP_SUCCESS; } /* Allocate memory for core message_block and copy from buffer. */ static enum op_result alloc_copy_mb(struct dgsh_negotiation **mb, char *buf, int bytes_read, int buf_size) { int expected_read_size = sizeof(struct dgsh_negotiation); if (check_read(bytes_read, buf_size, expected_read_size) == OP_ERROR) return OP_ERROR; *mb = (struct dgsh_negotiation *)malloc(bytes_read); memcpy(*mb, buf, bytes_read); (*mb)->node_array = NULL; (*mb)->edge_array = NULL; (*mb)->graph_solution = NULL; return OP_SUCCESS; } /** * The actual call to read in the message block. * If the call does not succeed or does not signal retry we have * to quit operation. */ static enum op_result call_read(int fd, char *buf, int buf_size, int *bytes_read, int *error_code) { *error_code = 0; DPRINTF(4, "Try read from fd %d.", fd); if ((*bytes_read = read(fd, buf, buf_size)) == -1) { *error_code = -errno; return OP_ERROR; } return OP_SUCCESS; } /** * Try to read a chunk of data from either side (stdin or stdout). * This function is agnostic as to what it is reading. * Its job is to manage a read operation. */ static enum op_result read_chunk(int read_fd, char *buf, int buf_size, int *bytes_read, int struct_type) { int error_code; DPRINTF(4, "%s(): buf_size: %d, IOV_MAX: %d", __func__, buf_size, iov_max); if (buf_size > iov_max) { int buf_size_piece, pieces, i, size_copied = 0, rsize = 0, struct_size, all_elements, max_elements, elements = 0; struct_size = get_struct_size(struct_type); all_elements = buf_size / struct_size; max_elements = iov_max / struct_size; pieces = all_elements / max_elements; pieces += (all_elements % max_elements > 0); DPRINTF(4, "struct_type: %d, pieces: %d, all_elements: %d, max_elements: %d", struct_type, pieces, all_elements, max_elements); for (i = 0; i < pieces; i++) { void *buf_piece; if (all_elements - max_elements > 0) elements = max_elements; else if (all_elements > 0) elements = all_elements; all_elements -= elements; buf_size_piece = struct_size * elements; DPRINTF(4, "Round %d: elements: %d, size: %d", i, elements, buf_size_piece); if (buf_size_piece > 0) { buf_piece = malloc(buf_size_piece); call_read(read_fd, buf_piece, buf_size_piece, bytes_read, &error_code); if (*bytes_read == -1) { free(buf_piece); return OP_ERROR; } rsize += *bytes_read; memcpy(buf + size_copied, buf_piece, buf_size_piece); size_copied += buf_size_piece; free(buf_piece); } } *bytes_read = rsize; } else call_read(read_fd, buf, buf_size, bytes_read, &error_code); if (*bytes_read == -1) { /* Read failed. */ DPRINTF(4, "ERROR: Reading from fd %d failed with error code %d.", read_fd, error_code); return error_code; } else /* Read succeeded. */ DPRINTF(4, "Read succeeded: %d bytes read from %d.\n", *bytes_read, read_fd); return OP_SUCCESS; } /* Allocate memory for file descriptors. */ static enum op_result alloc_fds(int **fds, int n_fds) { if (n_fds) { *fds = (int *)malloc(sizeof(int) * n_fds); if (*fds == NULL) return OP_ERROR; } return OP_SUCCESS; } /* Allocate memory for output file descriptors. */ static enum op_result alloc_io_fds() { int i = 0; struct dgsh_node_connections *this_nc = &chosen_mb->graph_solution[self_node.index]; DPRINTF(4, "%s(): self node: %d, incoming edges: %d, outgoing edges: %d", __func__, self_node.index, this_nc->n_edges_incoming, this_nc->n_edges_outgoing); self_pipe_fds.n_input_fds = 0; /* For safety. */ for (i = 0; i < this_nc->n_edges_incoming; i++) { int k; for (k = 0; k < this_nc->edges_incoming[i].instances; k++) self_pipe_fds.n_input_fds++; } if (alloc_fds(&self_pipe_fds.input_fds, self_pipe_fds.n_input_fds) == OP_ERROR) return OP_ERROR; self_pipe_fds.n_output_fds = 0; for (i = 0; i < this_nc->n_edges_outgoing; i++) { int k; for (k = 0; k < this_nc->edges_outgoing[i].instances; k++) self_pipe_fds.n_output_fds++; } if (alloc_fds(&self_pipe_fds.output_fds, self_pipe_fds.n_output_fds) == OP_ERROR) return OP_ERROR; return OP_SUCCESS; } /* Return the pid of the node that dispatched * the provided message block. */ pid_t get_origin_pid(struct dgsh_negotiation *mb) { /* Nodes may not be recorded if an error manifests early */ if (mb->node_array) { struct dgsh_node *n = &mb->node_array[mb->origin_index]; DPRINTF(4, "Logical origin: tool %s with pid %d", n->name, n->pid); return n->pid; } else return 0; } /* Return the number of input file descriptors * expected by process with pid PID. * It is applicable to concentrators too. */ int get_expected_fds_n(struct dgsh_negotiation *mb, pid_t pid) { int expected_fds_n = 0; int i = 0, j = 0; for (i = 0; i < mb->n_nodes; i++) { if (mb->node_array[i].pid == pid) { struct dgsh_node_connections *graph_solution = mb->graph_solution; for (j = 0; j < graph_solution[i].n_edges_incoming; j++) expected_fds_n += graph_solution[i].edges_incoming[j].instances; return expected_fds_n; } } /* pid may belong to another conc */ for (i = 0; i < mb->n_concs; i++) { if (mb->conc_array[i].pid == pid) return mb->conc_array[i].input_fds; } /* Invalid pid */ return -1; } /* Return the number of output file descriptors * provided by process with pid PID. * It is applicable to concentrators too. */ int get_provided_fds_n(struct dgsh_negotiation *mb, pid_t pid) { int provided_fds_n = 0; int i = 0, j = 0; for (i = 0; i < mb->n_nodes; i++) { if (mb->node_array[i].pid == pid) { struct dgsh_node_connections *graph_solution = mb->graph_solution; for (j = 0; j < graph_solution[i].n_edges_outgoing; j++) { provided_fds_n += graph_solution[i].edges_outgoing[j].instances; } return provided_fds_n; } } /* pid may belong to another conc */ for (i = 0; i < mb->n_concs; i++) if (mb->conc_array[i].pid == pid) return mb->conc_array[i].output_fds; /* Invalid pid */ return -1; } /* * Write the file descriptor fd_to_write to * the socket file descriptor output_socket. */ void write_fd(int output_socket, int fd_to_write) { struct msghdr msg; struct cmsghdr *cmsg; unsigned char buf[CMSG_SPACE(sizeof(int))]; struct iovec io = { .iov_base = " ", .iov_len = 1 }; memset(&msg, 0, sizeof(msg)); msg.msg_iov = &io; msg.msg_iovlen = 1; msg.msg_control = buf; msg.msg_controllen = sizeof(buf); cmsg = CMSG_FIRSTHDR(&msg); cmsg->cmsg_len = CMSG_LEN(sizeof(int)); cmsg->cmsg_level = SOL_SOCKET; cmsg->cmsg_type = SCM_RIGHTS; *(int *)CMSG_DATA(cmsg) = fd_to_write; if (sendmsg(output_socket, &msg, 0) == -1) err(1, "sendmsg on fd %d", output_socket); } /* * Read a file descriptor from socket input_socket and return it. */ int read_fd(int input_socket) { struct msghdr msg; struct cmsghdr *cmsg; unsigned char buf[CMSG_SPACE(sizeof(int))]; char m_buffer[2]; struct iovec io = { .iov_base = m_buffer, .iov_len = sizeof(m_buffer) }; memset(&msg, 0, sizeof(msg)); msg.msg_control = buf; msg.msg_controllen = sizeof(buf); msg.msg_iov = &io; msg.msg_iovlen = 1; again: if (recvmsg(input_socket, &msg, 0) == -1) { if (errno == EAGAIN) { sleep(1); goto again; } err(1, "recvmsg on fd %d", input_socket); } if ((msg.msg_flags & MSG_TRUNC) || (msg.msg_flags & MSG_CTRUNC)) errx(1, "control message truncated on fd %d", input_socket); for (cmsg = CMSG_FIRSTHDR(&msg); cmsg != NULL; cmsg = CMSG_NXTHDR(&msg, cmsg)) { if (cmsg->cmsg_len == CMSG_LEN(sizeof(int)) && cmsg->cmsg_level == SOL_SOCKET && cmsg->cmsg_type == SCM_RIGHTS) return *(int *)CMSG_DATA(cmsg); } errx(1, "unable to read file descriptor from fd %d", input_socket); } /* Read file descriptors piping input from another tool in the dgsh graph. */ static enum op_result read_input_fds(int input_socket, int *input_fds) { /** * A node's connections are located at the same position * as the node in the node array. */ struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; struct dgsh_node_connections *this_nc = &graph_solution[self_node.index]; assert(this_nc->node_index == self_node.index); int i; int total_edge_instances = 0; enum op_result re = OP_SUCCESS; DPRINTF(4, "%s(): %d incoming edges to inspect of node %d.", __func__, this_nc->n_edges_incoming, self_node.index); for (i = 0; i < this_nc->n_edges_incoming; i++) { int k; /** * Due to channel constraint flexibility, * each edge can have more than one instances. */ for (k = 0; k < this_nc->edges_incoming[i].instances; k++) { input_fds[total_edge_instances] = read_fd(input_socket); DPRINTF(4, "%s: Node %d received file descriptor %d.", __func__, this_nc->node_index, input_fds[total_edge_instances]); total_edge_instances++; } /* XXX */ if (re == OP_ERROR) break; } if (re == OP_ERROR) { free_graph_solution(chosen_mb->n_nodes - 1); free(input_fds); } return re; } static enum op_result read_concs(int read_fd, struct dgsh_negotiation *fresh_mb) { int bytes_read; size_t buf_size = sizeof(struct dgsh_conc) * fresh_mb->n_concs; char *buf = (char *)malloc(buf_size); enum op_result error_code = OP_SUCCESS; if (!fresh_mb->conc_array) return error_code; if ((error_code = read_chunk(read_fd, buf, buf_size, &bytes_read, 3)) != OP_SUCCESS) return error_code; error_code = alloc_copy_concs(fresh_mb, buf, bytes_read, buf_size); free(buf); int i, n_concs = fresh_mb->n_concs; for (i = 0; i < n_concs; i++) { struct dgsh_conc *c = &fresh_mb->conc_array[i]; size_t buf_size = sizeof(int) * c->n_proc_pids; buf = (char *)malloc(buf_size); if ((error_code = read_chunk(read_fd, buf, buf_size, &bytes_read, 5)) != OP_SUCCESS) return error_code; error_code = alloc_copy_proc_pids(c, buf, bytes_read, buf_size); free(buf); DPRINTF(4, "%s(): Read %d proc_pids for conc %d at index %d of size %d bytes ", __func__, c->n_proc_pids, c->pid, i, bytes_read); } return error_code; } /* Try read solution to the dgsh negotiation graph. */ static enum op_result read_graph_solution(int read_fd, struct dgsh_negotiation *fresh_mb) { int i; int bytes_read = 0; int n_nodes = fresh_mb->n_nodes; size_t buf_size = sizeof(struct dgsh_node_connections) * n_nodes; char *buf = (char *)malloc(buf_size); enum op_result error_code = OP_SUCCESS; /* Read node connection structures of the solution. */ if ((error_code = read_chunk(read_fd, buf, buf_size, &bytes_read, 4)) != OP_SUCCESS) return error_code; if ((error_code = alloc_copy_graph_solution(fresh_mb, buf, bytes_read, buf_size)) == OP_ERROR) return error_code; free(buf); struct dgsh_node_connections *graph_solution = fresh_mb->graph_solution; for (i = 0; i < n_nodes; i++) { struct dgsh_node_connections *nc = &graph_solution[i]; DPRINTF(4, "Node %d with %d incoming edges at %lx and %d outgoing edges at %lx.", nc->node_index, nc->n_edges_incoming, (long)nc->edges_incoming, nc->n_edges_outgoing, (long)nc->edges_outgoing); int in_edges_size = sizeof(struct dgsh_edge) * nc->n_edges_incoming; int out_edges_size = sizeof(struct dgsh_edge) * nc->n_edges_outgoing; /* Read a node's incoming connections. */ if (nc->n_edges_incoming > 0) { buf = (char *)malloc(in_edges_size); if ((error_code = read_chunk(read_fd, buf, in_edges_size, &bytes_read, 2)) != OP_SUCCESS) return error_code; if (in_edges_size != bytes_read) { DPRINTF(4, "%s(): ERROR: Expected %d bytes, got %d.", __func__, in_edges_size, bytes_read); return OP_ERROR; } if (alloc_node_connections(&nc->edges_incoming, nc->n_edges_incoming, 0, i) == OP_ERROR) return OP_ERROR; memcpy(nc->edges_incoming, buf, in_edges_size); free(buf); } /* Read a node's outgoing connections. */ if (nc->n_edges_outgoing) { buf = (char *)malloc(out_edges_size); if ((error_code = read_chunk(read_fd, buf, out_edges_size, &bytes_read, 2)) != OP_SUCCESS) return error_code; if (out_edges_size != bytes_read) { DPRINTF(4, "%s(): ERROR: Expected %d bytes, got %d.", __func__, out_edges_size, bytes_read); return OP_ERROR; } if (alloc_node_connections(&nc->edges_outgoing, nc->n_edges_outgoing, 1, i) == OP_ERROR) return OP_ERROR; memcpy(nc->edges_outgoing, buf, out_edges_size); free(buf); } } return OP_SUCCESS; } /** * Read a circulated message block coming in on any of the specified read_fds. * In most cases these will specify the input or output side. This capability * relies on an extension to a standard shell implementation, * e.g., bash, that allows reading and writing to both sides * for the negotiation phase. * On return: * read_fd will contain the file descriptor number that was read. * fresh_mb will contain the read message block in dynamically allocated * memory. This should be freed by calling free_mb. */ enum op_result read_message_block(int read_fd, struct dgsh_negotiation **fresh_mb) { size_t buf_size = sizeof(struct dgsh_negotiation); char *buf = (char *)malloc(buf_size); int bytes_read = 0; enum op_result error_code = 0; DPRINTF(3, "%s(): %s (%d)", __func__, programname, self_node.index); memset(buf, 0, buf_size); /* Try read core message block: struct negotiation state fields. */ if ((error_code = read_chunk(read_fd, buf, buf_size, &bytes_read, 0)) != OP_SUCCESS) return error_code; if (alloc_copy_mb(fresh_mb, buf, bytes_read, buf_size) == OP_ERROR) return OP_ERROR; free(buf); if ((*fresh_mb)->n_nodes > 0) { buf_size = sizeof(struct dgsh_node) * (*fresh_mb)->n_nodes; buf = (char *)malloc(buf_size); if ((error_code = read_chunk(read_fd, buf, buf_size, &bytes_read, 1)) != OP_SUCCESS) return error_code; if (alloc_copy_nodes(*fresh_mb, buf, bytes_read, buf_size) == OP_ERROR) return OP_ERROR; free(buf); } if (read_concs(read_fd, *fresh_mb) == OP_ERROR) return OP_ERROR; if ((*fresh_mb)->state == PS_NEGOTIATION) { if ((*fresh_mb)->n_edges > 0) { DPRINTF(4, "%s(): Read %d negotiation graph edges.", __func__, (*fresh_mb)->n_edges); buf_size = sizeof(struct dgsh_edge) * (*fresh_mb)->n_edges; buf = (char *)malloc(buf_size); if ((error_code = read_chunk(read_fd, buf, buf_size, &bytes_read, 2)) != OP_SUCCESS) return error_code; if (alloc_copy_edges(*fresh_mb, buf, bytes_read, buf_size) == OP_ERROR) return OP_ERROR; free(buf); } } else if ((*fresh_mb)->state == PS_RUN) { /** * Try read solution. * fresh_mb should be an updated version of the chosen_mb * or even the same structure because this is the phase * where we share the solution across the dgsh graph. */ if (read_graph_solution(read_fd, *fresh_mb) == OP_ERROR) return OP_ERROR; } DPRINTF(4, "%s(): Read message block or solution from node %d sent from file descriptor: %s.\n", __func__, (*fresh_mb)->origin_index, ((*fresh_mb)->origin_fd_direction) ? "stdout" : "stdin"); return OP_SUCCESS; } /* Construct a message block to use as a vehicle for the negotiation phase. */ enum op_result construct_message_block(const char *tool_name, pid_t self_pid) { chosen_mb = (struct dgsh_negotiation *)malloc( sizeof(struct dgsh_negotiation)); if (!chosen_mb) { DPRINTF(4, "ERROR: Memory allocation of message block failed."); return OP_ERROR; } chosen_mb->version = 1; chosen_mb->node_array = NULL; chosen_mb->n_nodes = 0; chosen_mb->edge_array = NULL; chosen_mb->n_edges = 0; chosen_mb->initiator_pid = self_pid; chosen_mb->state = (init_error ? PS_ERROR : PS_NEGOTIATION); chosen_mb->is_error_confirmed = false; chosen_mb->origin_index = -1; chosen_mb->origin_fd_direction = -1; chosen_mb->is_origin_conc = false; chosen_mb->conc_pid = -1; chosen_mb->graph_solution = NULL; chosen_mb->conc_array = NULL; chosen_mb->n_concs = 0; DPRINTF(3, "Message block created by process %s with pid %d.\n", tool_name, (int)self_pid); return OP_SUCCESS; } /* Get environment variable env_var. */ static void get_env_var(const char *env_var, int *value) { char *string_value = getenv(env_var); if (string_value == NULL) DPRINTF(4, "Getting environment variable %s failed.", env_var); else { DPRINTF(4, "getenv() returned string value %s.", string_value); *value = atoi(string_value); DPRINTF(4, "Integer form of value is %d.", *value); } } /** * Get environment variables DGSH_IN, DGSH_OUT set up by * the shell (through execvpe()). */ static void get_environment_vars() { DPRINTF(4, "Try to get environment variable DGSH_IN."); get_env_var("DGSH_IN", &self_node.dgsh_in); DPRINTF(4, "Try to get environment variable DGSH_OUT."); get_env_var("DGSH_OUT", &self_node.dgsh_out); } /** * Verify tool's I/O channel requirements are sane. * We might need some upper barrier for requirements too, * such as, not more than 100 or 1000. */ STATIC enum op_result validate_input(int *channels_required, int *channels_provided, const char *tool_name) { if (!tool_name) { DPRINTF(4, "ERROR: NULL pointer provided as tool name.\n"); return OP_ERROR; } if (channels_required == NULL || channels_provided == NULL) return OP_SUCCESS; if (*channels_required < -1 || *channels_provided < -1) { DPRINTF(4, "ERROR: I/O requirements entered for tool %s are less than -1. \nChannels required %d \nChannels provided: %d", tool_name, *channels_required, *channels_provided); return OP_ERROR; } return OP_SUCCESS; } static int set_fds(fd_set *read_fds, fd_set *write_fds, bool isread) { fd_set *fds; FD_ZERO(read_fds); FD_ZERO(write_fds); DPRINTF(4, "Next operation is a %s", isread ? "read" : "write"); /* The next operation is a read or a write */ if (isread) fds = read_fds; else fds = write_fds; if (self_node.dgsh_out && !self_node.dgsh_in) { self_node_io_side.fd_direction = STDOUT_FILENO; FD_SET(STDOUT_FILENO, fds); } else if (!self_node.dgsh_out && self_node.dgsh_in) { self_node_io_side.fd_direction = STDIN_FILENO; FD_SET(STDIN_FILENO, fds); } else { /* We should have all ears open for a read */ if (isread) { FD_SET(STDIN_FILENO, fds); FD_SET(STDOUT_FILENO, fds); } else { /* But for writing we should pass the message across. * If mb came from stdout channel, we got it from stdin. * So we should send it from stdout */ if (chosen_mb->origin_fd_direction == STDOUT_FILENO) { FD_SET(STDOUT_FILENO, fds); self_node_io_side.fd_direction = STDOUT_FILENO; DPRINTF(4, "STDOUT set for write"); } else { FD_SET(STDIN_FILENO, fds); self_node_io_side.fd_direction = STDIN_FILENO; DPRINTF(4, "STDIN set for write"); } } } /* so that after select() we try both 0 and 1 to see if they are set */ return 2; } void set_negotiation_complete() { negotiation_completed = 1; } static int setup_file_descriptors(int *n_input_fds, int *n_output_fds, int **input_fds, int **output_fds) { DPRINTF(4, "%s()", __func__); if (n_input_fds != NULL && (*n_input_fds == 1 || *n_input_fds == -1) && input_fds != NULL) { DPRINTF(4, "n_input_fds: %d\n", *n_input_fds); *n_input_fds = 1; *input_fds = malloc(sizeof(int)); (*input_fds)[0] = STDIN_FILENO; } if (n_output_fds != NULL && (*n_output_fds == 1 || *n_output_fds == -1) && output_fds != NULL) { DPRINTF(4, "n_output_fds: %d\n", *n_output_fds); *n_output_fds = 1; *output_fds = malloc(sizeof(int)); (*output_fds)[0] = STDOUT_FILENO; } return 0; } /* * Return function for dgsh_negotiate * Exit by printing an error (if needed) * otherwise return 0 for successful termination * or -1 for error */ static int dgsh_exit(int ret, int flags) { if (ret == PS_COMPLETE) return 0; if (ret == PS_DRAW_EXIT) exit(EX_OK); if (!(flags & DGSH_HANDLE_ERROR)) return -1; switch (errno) { case ECONNRESET: exit(EX_PROTOCOL); case 0: errx(EX_PROTOCOL, "dgsh negotiation failed"); default: err(EX_PROTOCOL, "dgsh negotiation failed"); } } /** * Return the name of the specified state */ const char * state_name(enum prot_state s) { switch (s) { case PS_COMPLETE: return "COMPLETE"; case PS_NEGOTIATION: return "NEGOTIATION"; case PS_NEGOTIATION_END: return "NEGOTIATION_END"; case PS_RUN: return "RUN"; case PS_ERROR: return "ERROR"; default: assert(0); } } /** * Each tool in the dgsh graph calls dgsh_negotiate() to take part in * peer-to-peer negotiation. A message block (MB) is circulated among tools * and is filled with tools' I/O requirements. When all requirements are in * place, an algorithm runs that tries to find a solution that satisfies * all requirements. If a solution is found, pipes are allocated and * set up according to the solution. The appropriate file descriptors * are provided to each tool and the negotiation phase ends. * The function's return value signifies success or failure of the * negotiation phase. */ int dgsh_negotiate(int flags, const char *tool_name, int *n_input_fds, int *n_output_fds, int **input_fds, int **output_fds) { int i = 0; int ntimes_seen_run = 0; int ntimes_seen_error = 0; int ntimes_seen_draw_exit = 0; pid_t self_pid = getpid(); /* Get tool's pid */ struct dgsh_negotiation *fresh_mb = NULL; /* MB just read. */ int nfds = 0, n_io_sides; bool isread = false; fd_set read_fds, write_fds; char *timeout; char *debug_level; if (negotiation_completed) { errno = EALREADY; return dgsh_exit(-1, flags); } /* Get and set user-provided debug level. * dgsh_debug_level is defined in debug.h. */ debug_level = getenv("DGSH_DEBUG_LEVEL"); if (debug_level != NULL) dgsh_debug_level = atoi(debug_level); #ifndef UNIT_TESTING self_pipe_fds.input_fds = NULL; /* Clean garbage */ self_pipe_fds.output_fds = NULL; /* Ditto */ #endif programname = strdup(tool_name); DPRINTF(2, "%s(): Tool %s with pid %d negotiating: nin=%d nout=%d.", __func__, tool_name, (int)self_pid, n_input_fds ? *n_input_fds : 1, n_output_fds ? *n_output_fds : 1); if (validate_input(n_input_fds, n_output_fds, tool_name) == OP_ERROR) { negotiation_completed = 1; return dgsh_exit(-1, flags); } self_node.dgsh_in = 0; self_node.dgsh_out = 0; get_environment_vars(); n_io_sides = self_node.dgsh_in + self_node.dgsh_out; /* Verify dgsh available on the required sides */ if ((n_input_fds != NULL && *n_input_fds > 1 && !self_node.dgsh_in) || (n_output_fds != NULL && *n_output_fds > 1 && !self_node.dgsh_out)) { errno = ENOTSOCK; negotiation_completed = 1; return dgsh_exit(-1, flags); } /* Easy case, no dgsh I/O */ if (n_io_sides == 0) { negotiation_completed = 1; return dgsh_exit(setup_file_descriptors(n_input_fds, n_output_fds, input_fds, output_fds), flags); } signal(SIGALRM, dgsh_alarm_handler); if ((timeout = getenv("DGSH_TIMEOUT")) != NULL) alarm(atoi(timeout)); else alarm(DGSH_TIMEOUT); /* Start negotiation */ if (self_node.dgsh_out && !self_node.dgsh_in) { #ifdef TIME clock_gettime(CLOCK_MONOTONIC, &tstart); #endif if (construct_message_block(tool_name, self_pid) == OP_ERROR) chosen_mb->state = PS_ERROR; if (register_node_edge(tool_name, self_pid, n_input_fds, n_output_fds) == OP_ERROR) chosen_mb->state = PS_ERROR; isread = false; } else { /* or wait to receive MB. */ isread = true; chosen_mb = NULL; } /* Perform phases and rounds. */ while (1) { again: DPRINTF(4, "%s(): perform round", __func__); nfds = set_fds(&read_fds, &write_fds, isread); if (select(nfds, &read_fds, &write_fds, NULL, NULL) < 0) { if (errno == EINTR) goto again; perror("select"); chosen_mb->state = PS_ERROR; } for (i = 0; i < nfds; i++) { if (FD_ISSET(i, &write_fds)) { DPRINTF(4, "write on fd %d is active.", i); /* Write message block et al. */ set_dispatcher(); if (write_message_block(i) == OP_ERROR) chosen_mb->state = PS_ERROR; if (n_io_sides == ntimes_seen_run || n_io_sides == ntimes_seen_error || n_io_sides == ntimes_seen_draw_exit) { if (chosen_mb->state == PS_RUN) chosen_mb->state = PS_COMPLETE; goto exit; } isread = true; } if (FD_ISSET(i, &read_fds)) { DPRINTF(4, "read on fd %d is active.", i); /* Read message block et al. */ if (read_message_block(i, &fresh_mb) == OP_ERROR && fresh_mb != NULL) fresh_mb->state = PS_ERROR; /* Check state */ analyse_read(fresh_mb, &ntimes_seen_run, &ntimes_seen_error, &ntimes_seen_draw_exit, tool_name, self_pid, n_input_fds, n_output_fds); /** * Initiator process. * It receives the block, so all IO * constraints have been stated. Now: * - solves the I/O constraint problem, * - communicates the solution, * and when it receives the block again, * it leaves negotiation. */ if (self_node.pid == chosen_mb->initiator_pid) { switch (chosen_mb->state) { case PS_NEGOTIATION: chosen_mb->state = PS_NEGOTIATION_END; DPRINTF(1, "%s(): Gathered I/O requirements.", __func__); int state = solve_graph(); if (state == OP_ERROR) { chosen_mb->state = PS_ERROR; chosen_mb->is_error_confirmed = true; } else if (state == OP_DRAW_EXIT) chosen_mb->state = PS_DRAW_EXIT; else { DPRINTF(1, "%s(): Computed solution", __func__); chosen_mb->state = PS_RUN; } break; case PS_RUN: DPRINTF(1, "%s(): Communicated the solution", __func__); chosen_mb->state = PS_COMPLETE; goto exit; case PS_ERROR: if (chosen_mb->is_error_confirmed) goto exit; else chosen_mb->is_error_confirmed = true; break; case PS_DRAW_EXIT: goto exit; default: assert(0); } } isread = false; } } } exit: DPRINTF(2, "%s(): %s (%d) leaves after %s with state %s.", __func__, programname, self_node.index, isread ? "read" : "write", state_name(chosen_mb->state)); if (chosen_mb->state == PS_COMPLETE) { if (alloc_io_fds() == OP_ERROR) chosen_mb->state = PS_ERROR; if (read_input_fds(STDIN_FILENO, self_pipe_fds.input_fds) == OP_ERROR) chosen_mb->state = PS_ERROR; if (write_output_fds(STDOUT_FILENO, self_pipe_fds.output_fds, flags) == OP_ERROR) chosen_mb->state = PS_ERROR; if (establish_io_connections(input_fds, n_input_fds, output_fds, n_output_fds) == OP_ERROR) chosen_mb->state = PS_ERROR; } else if (chosen_mb->state == PS_DRAW_EXIT) { if (n_input_fds != NULL) *n_input_fds = 0; if (n_output_fds != NULL) *n_output_fds = 0; } int state = chosen_mb->state; #ifdef TIME if (self_node.pid == chosen_mb->initiator_pid) { clock_gettime(CLOCK_MONOTONIC, &tend); fprintf(stderr, "The dgsh negotiation procedure took about %.5f seconds\n", ((double)tend.tv_sec + 1.0e-9*tend.tv_nsec) - ((double)tstart.tv_sec + 1.0e-9*tstart.tv_nsec)); fflush(stderr); } #endif free_mb(chosen_mb); negotiation_completed = 1; alarm(0); // Cancel alarm signal(SIGALRM, SIG_IGN); // Do not handle the signal return dgsh_exit(state, flags); } ================================================ FILE: core-tools/src/negotiate.h ================================================ /* * Copyright 2016-2017 Marios Fragkoulis * * Dgsh private negotiation API * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ #ifndef NEGOTIATE_H #define NEGOTIATE_H #include #include /* struct cmsghdr */ #include /* sig_atomic_t */ #include "dgsh.h" /* Negotiation protocol states */ enum prot_state { PS_COMPLETE, /* Negotiation process is complete */ PS_NEGOTIATION, /* Negotiation phase */ PS_NEGOTIATION_END, /* End of negotiation phase */ PS_RUN, /* Share solution; prepare to run */ PS_ERROR, /* Error in negotiation process */ PS_DRAW_EXIT, /* Compute and write the solution and exit */ }; union fdmsg { struct cmsghdr h; char buf[CMSG_SPACE(sizeof(int))]; }; /* * Results of operations * Also negative values signify a failed operation's errno value */ enum op_result { OP_SUCCESS, /* Successful */ OP_ERROR, /* Unresolvable error due to I/O problem * constraints provided by the processes * on the dgsh graph or memory constraints * of the systems. */ OP_EXISTS, /* Node or edge already registered with the * dgsh graph. */ OP_CREATE, /* Node ar edge registered with the dgsh * graph. */ OP_NOOP, /* No operation when trying to add an edge * on a graph with just one node at the time. */ OP_RETRY, /* Not all constraints of an I/O constraint * problem have been satisfied yet. * Retry by leveraging flexible constraints. */ OP_DRAW_EXIT, /* Compute and write the solution and exit */ }; #ifdef UNIT_TESTING #define STATIC /* Models an I/O connection between tools on an dgsh graph. */ struct dgsh_edge { int from; /* Index of node on the graph where data comes from (out). */ int to; /* Index of node on the graph that receives the data (in). */ int instances; /* Number of instances of an edge. */ int from_instances; /* Number of instances the origin node of an edge can provide. */ int to_instances; /* Number of instances the destination of an edge can require. */ }; extern bool multiple_inputs; extern int nfd; extern int next_fd(int fd, bool *ro); extern int read_fd(int input_socket); extern void write_fd(int output_socket, int fd_to_write); #else #define STATIC static #endif /* UNIT_TESTING */ /* The message block implicitly used by many functions */ extern struct dgsh_negotiation *chosen_mb; /* Identifies the node and node's fd that sent the message block. */ struct node_io_side { int index; /* Node index on message block node array */ int fd_direction; /* Message block origin node's file * descriptor */ }; /* Stores the number of a conc's IO fds */ struct dgsh_conc { pid_t pid; int input_fds; int output_fds; int n_proc_pids; int *proc_pids; /* pids at the multipipe end */ int endpoint_pid; /* pid at the other end */ bool multiple_inputs; /* true for input conc */ }; /* The message block structure that provides the vehicle for negotiation. */ struct dgsh_negotiation { int version; /* Protocol version. */ struct dgsh_node *node_array; /* Nodes, that is tools, on the dgsh * graph. */ int n_nodes; /* Number of nodes */ struct dgsh_edge *edge_array; /* Edges, that is connections between * nodes, on the dgsh graph */ int n_edges; /* Number of edges */ pid_t initiator_pid; /* pid of the tool initiating this * negotiation block. All processes that * only contribute their output * channel to the dgsh graph will * initiate the negotiation process * by constructing and sharing a * message block. The one with the * smaller pid will prevail. */ enum prot_state state; /* State of the negotiation process */ bool is_error_confirmed; /* Error state is confirmed by the initiator and propagated to the graph */ int origin_index; /* The node from which the message * block is dispatched. */ int origin_fd_direction; /* The origin's input or output channel */ bool is_origin_conc; /* True if origin is a concentrator */ pid_t conc_pid; /* Concentrator pid, otherwise -1 */ struct dgsh_node_connections *graph_solution; /* The solution to the * I/O constraint problem * at hand. */ struct dgsh_conc *conc_array; /* Array of concentrators facilitating * the negotiation. The need for this * array emerged in cases where a conc * is directly connected to another * conc. The array stores a conc's * inputs/outputs so that a conc can * retrieve another conc's * inputs/outputs. */ int n_concs; }; enum op_result solve_graph(void); enum op_result construct_message_block(const char *tool_name, pid_t pid); struct dgsh_conc *find_conc(struct dgsh_negotiation *mb, pid_t pid); pid_t get_origin_pid(struct dgsh_negotiation *mb); int get_expected_fds_n(struct dgsh_negotiation *mb, pid_t pid); int get_provided_fds_n(struct dgsh_negotiation *mb, pid_t pid); enum op_result read_message_block(int read_fd, struct dgsh_negotiation **fresh_mb); enum op_result write_message_block(int write_fd); void free_mb(struct dgsh_negotiation *mb); int read_fd(int input_socket); void write_fd(int output_socket, int fd_to_write); /* Alarm mechanism and on_exit handling */ void set_negotiation_complete(); void dgsh_alarm_handler(int); #endif /* NEGOTIATE_H */ ================================================ FILE: core-tools/src/perm.1 ================================================ .TH PERM 1 "13 December 2016" .\" .\" (C) Copyright 2016 Diomidis Spinellis. All rights reserved. .\" .\" Licensed under the Apache License, Version 2.0 (the "License"); .\" you may not use this file except in compliance with the License. .\" You may obtain a copy of the License at .\" .\" http://www.apache.org/licenses/LICENSE-2.0 .\" .\" Unless required by applicable law or agreed to in writing, software .\" distributed under the License is distributed on an "AS IS" BASIS, .\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .\" See the License for the specific language governing permissions and .\" limitations under the License. .\" .SH NAME perm \- permute inputs to outputs .SH SYNOPSIS \fBperm\fP \fIo1,o2 ...\fP .SH DESCRIPTION \fIperm\fP will read data from multiple inputs, and output each input to the specified output. \fIperm\fP is implemented by invoking \fIdgsh-tee\fP with the \fI-p\fP option. .SH OPTIONS .IP "\fIo1,o2 ...\fP" Permute the inputs to the specified outputs. The comma-separated arguments \fIo1,o2, ...\fP specify the number of the output channel (starting from 1) where the corresponding \fIn\fPth input channel will be sent. Thus, input 1 goes to output \fIo1\fP, input 2 goes to output \fIo2\fP, and so on. As an example a cross-permutation is specified with the argument \fI2,1\fP. .SH "SEE ALSO" \fIdgsh-tee\fP(1) .SH AUTHOR Diomidis Spinellis \(em ================================================ FILE: core-tools/src/perm.sh ================================================ #!/bin/sh #!dgsh # # Permute inputs to outputs by invoking dgsh-tee -p # usage() { echo 'Usage: perm n1,n2[, ...]' exit 2 } # Get first argument. Its syntax will be validated by dgsh-tee test "$1" || usage p="$1" shift # Ensure no more arguments are provided for i; do usage done exec dgsh-tee -p "$p" ================================================ FILE: core-tools/src/run-built-dgsh.sh ================================================ #!/bin/sh # # Run any dgsh script from the built (rather than the installed) # version of dgsh. # The first argument must be the directory containing the dgsh sources. # if [ -z "$1" ] ; then echo "Usage: $0 dgsh-root-path [dgsh arguments]" 1>&2 exit 1 fi TOP="$1" shift DGSH="$TOP/build/bin/dgsh" PATH="$TOP/build/bin:$PATH" export DGSHPATH="$TOP/build/libexec/dgsh" dgsh "$@" ================================================ FILE: core-tools/tests/.gitignore ================================================ Makefile.orig check_negotiate check_negotiate.log check_negotiate.trs test-suite.log unit-test-dgsh ================================================ FILE: core-tools/tests/Makefile.am ================================================ ## Process this file with automake to produce Makefile.in TESTS = check_negotiate check_PROGRAMS = check_negotiate check_negotiate_SOURCES = check_negotiate.c ../src/negotiate.h check_negotiate_CFLAGS = @CHECK_CFLAGS@ -DUNIT_TESTING -DDEBUG check_negotiate_LDADD = ../src/libdgsh.a @CHECK_LIBS@ ================================================ FILE: core-tools/tests/Makefile.patch ================================================ --- Makefile.original 2016-06-20 18:17:16.661787072 +0300 +++ Makefile 2016-06-20 18:18:39.577783863 +0300 @@ -686,17 +686,18 @@ done; \ test $$st -eq 0 || exit 1; \ fi + # MFG: Hack for identifying and aggregating test results in log. @$(am__sh_e_setup); $(am__tty_colors); $(am__set_TESTS_bases); \ ws='[ ]'; \ - results=`for b in $$bases; do echo $$b.trs; done`; \ + results=`for b in $$bases; do echo $$b.log; done`; \ test -n "$$results" || results=/dev/null; \ - all=` grep "^$$ws*:test-result:" $$results | wc -l`; \ - pass=` grep "^$$ws*:test-result:$$ws*PASS" $$results | wc -l`; \ - fail=` grep "^$$ws*:test-result:$$ws*FAIL" $$results | wc -l`; \ - skip=` grep "^$$ws*:test-result:$$ws*SKIP" $$results | wc -l`; \ - xfail=`grep "^$$ws*:test-result:$$ws*XFAIL" $$results | wc -l`; \ - xpass=`grep "^$$ws*:test-result:$$ws*XPASS" $$results | wc -l`; \ - error=`grep "^$$ws*:test-result:$$ws*ERROR" $$results | wc -l`; \ + all=` grep ":.\+:.:.\+:.\+:.\+: " $$results | wc -l`; \ + pass=` grep "^.\+\.c:.\+:.:.\+:.\+:.\+: Passed" $$results | wc -l`; \ + fail=` grep "^.\+\.c:.\+:.:.\+:.\+:.\+: Assertion .\+ failed:" $$results | wc -l`; \ + skip=` grep "^.\+\.c:.\+:.:.\+:.\+:.\+: Skipped" $$results | wc -l`; \ + xfail=`grep "^.\+\.c:.\+:.:.\+:.\+:.\+: XFailed" $$results | wc -l`; \ + xpass=`grep "^.\+\.c:.\+:.:.\+:.\+:.\+: XPassed" $$results | wc -l`; \ + error=`grep ":.\+:.:.\+:.\+:.\+: (after this point) Received" $$results | wc -l`; \ if test `expr $$fail + $$xpass + $$error` -eq 0; then \ success=true; \ else \ ================================================ FILE: core-tools/tests/check.hack ================================================ # MFG: Hack for identifying and aggregating test results in log. # Goes in test/negotiate/tests/Makefile # Grep for 'xfail' to identify the block that the current # substitutes. results=`for b in $$bases; do echo $$b.log; done`; \ test -n "$$results" || results=/dev/null; \ all=` grep ":.\+:.:.\+:.\+:.\+: " $$results | wc -l`; \ pass=` grep "^.\+\.c:.\+:.:.\+:.\+:.\+: Passed" $$results | wc -l`; \ fail=` grep "^.\+\.c:.\+:.:.\+:.\+:.\+: Assertion .\+ failed:" $$results | wc -l`; \ skip=` grep "^.\+\.c:.\+:.:.\+:.\+:.\+: Skipped" $$results | wc -l`; \ xfail=`grep "^.\+\.c:.\+:.:.\+:.\+:.\+: XFailed" $$results | wc -l`; \ xpass=`grep "^.\+\.c:.\+:.:.\+:.\+:.\+: XPassed" $$results | wc -l`; \ error=`grep ":.\+:.:.\+:.\+:.\+: (after this point) Received" $$results | wc -l`; \ ================================================ FILE: core-tools/tests/check_negotiate.c ================================================ #include /* Check unit test framework API. */ #include /* EXIT_SUCCESS, EXIT_FAILURE */ #include /* pipe() */ #include /* fcntl() */ #include /* err(), errx() */ #include /* nanosleep() */ #include /* snprintf */ #include /* pipe */ #include #include /* socket */ #include /* sockaddr_un */ #include "../src/negotiate.h" #include "../src/negotiate.c" /* struct definitions, static structures */ #include "../src/dgsh-conc.c" /* pi */ //#include "../src/dgsh-internal-api.h" /* chosen_mb */ struct dgsh_negotiation *fresh_mb; struct dgsh_edge *compact_edges; struct dgsh_edge **pointers_to_edges; int n_ptedges; int *args; /* Depending on whether a test triggers a failure or not, a different sequence * of actions may be needed to exit normally. * exit_state is the control variable for following the correct * sequence of actions. */ int exit_state = 0; void setup_concs(struct dgsh_negotiation *mb) { mb->n_concs = 2; mb->conc_array = (struct dgsh_conc *)malloc(sizeof(struct dgsh_conc) * mb->n_concs); mb->conc_array[0].pid = 2000; mb->conc_array[0].input_fds = 2; mb->conc_array[0].output_fds = 2; mb->conc_array[0].multiple_inputs = false; mb->conc_array[0].endpoint_pid = 102; mb->conc_array[0].n_proc_pids = 2; mb->conc_array[0].proc_pids = (int *)malloc(sizeof(int) * 2); mb->conc_array[0].proc_pids[0] = 100; mb->conc_array[0].proc_pids[1] = 101; mb->conc_array[1].pid = 2001; mb->conc_array[1].input_fds = 3; mb->conc_array[1].output_fds = 3; mb->conc_array[1].multiple_inputs = true; mb->conc_array[1].endpoint_pid = 103; mb->conc_array[1].n_proc_pids = 2; mb->conc_array[1].proc_pids = (int *)malloc(sizeof(int) * 2); mb->conc_array[1].proc_pids[0] = 100; mb->conc_array[1].proc_pids[1] = 101; } void setup_graph_solution(void) { chosen_mb->graph_solution = (struct dgsh_node_connections *)malloc( sizeof(struct dgsh_node_connections) * chosen_mb->n_nodes); struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; graph_solution[0].node_index = 0; graph_solution[0].n_edges_incoming = 2; graph_solution[0].edges_incoming = (struct dgsh_edge *)malloc( sizeof(struct dgsh_edge) * graph_solution[0].n_edges_incoming); memcpy(&graph_solution[0].edges_incoming[0], &chosen_mb->edge_array[0], sizeof(struct dgsh_edge)); memcpy(&graph_solution[0].edges_incoming[1], &chosen_mb->edge_array[2], sizeof(struct dgsh_edge)); graph_solution[0].n_edges_outgoing = 1; graph_solution[0].edges_outgoing = (struct dgsh_edge *)malloc( sizeof(struct dgsh_edge) * graph_solution[0].n_edges_outgoing); memcpy(&graph_solution[0].edges_outgoing[0], &chosen_mb->edge_array[4], sizeof(struct dgsh_edge)); graph_solution[1].node_index = 1; graph_solution[1].n_edges_incoming = 1; graph_solution[1].edges_incoming = (struct dgsh_edge *)malloc( sizeof(struct dgsh_edge) * graph_solution[1].n_edges_incoming); memcpy(&graph_solution[1].edges_incoming[0], &chosen_mb->edge_array[1], sizeof(struct dgsh_edge)); graph_solution[1].n_edges_outgoing = 2; graph_solution[1].edges_outgoing = (struct dgsh_edge *)malloc( sizeof(struct dgsh_edge) * graph_solution[1].n_edges_outgoing); memcpy(&graph_solution[1].edges_outgoing[0], &chosen_mb->edge_array[2], sizeof(struct dgsh_edge)); memcpy(&graph_solution[1].edges_outgoing[1], &chosen_mb->edge_array[3], sizeof(struct dgsh_edge)); graph_solution[2].node_index = 2; graph_solution[2].n_edges_incoming = 0; graph_solution[2].edges_incoming = NULL; graph_solution[2].n_edges_outgoing = 2; graph_solution[2].edges_outgoing = (struct dgsh_edge *)malloc( sizeof(struct dgsh_edge) * graph_solution[2].n_edges_outgoing); memcpy(&graph_solution[2].edges_outgoing[0], &chosen_mb->edge_array[0], sizeof(struct dgsh_edge)); memcpy(&graph_solution[2].edges_outgoing[1], &chosen_mb->edge_array[1], sizeof(struct dgsh_edge)); graph_solution[3].node_index = 3; graph_solution[3].n_edges_incoming = 2; graph_solution[3].edges_incoming = (struct dgsh_edge *)malloc( sizeof(struct dgsh_edge) * graph_solution[3].n_edges_incoming); memcpy(&graph_solution[0].edges_incoming[0], &chosen_mb->edge_array[3], sizeof(struct dgsh_edge)); memcpy(&graph_solution[0].edges_incoming[1], &chosen_mb->edge_array[4], sizeof(struct dgsh_edge)); graph_solution[3].n_edges_outgoing = 0; graph_solution[3].edges_outgoing = NULL; } void setup_chosen_mb(void) { struct dgsh_node *nodes; struct dgsh_edge *edges; int n_nodes; int n_edges; n_nodes = 4; nodes = (struct dgsh_node *)malloc(sizeof(struct dgsh_node) * n_nodes); nodes[0].pid = 100; nodes[0].index = 0; strcpy(nodes[0].name, "proc0"); nodes[0].requires_channels = 2; nodes[0].provides_channels = 1; nodes[0].dgsh_in = 1; nodes[0].dgsh_out = 1; nodes[1].pid = 101; nodes[1].index = 1; strcpy(nodes[1].name, "proc1"); nodes[1].requires_channels = 1; nodes[1].provides_channels = 2; nodes[1].dgsh_in = 1; nodes[1].dgsh_out = 1; /* dgsh OUT and not IN = initiator node. * This node could start the negotiation. * Fix. */ nodes[2].pid = 102; nodes[2].index = 2; strcpy(nodes[2].name, "proc2"); nodes[2].requires_channels = 0; nodes[2].provides_channels = 2; nodes[2].dgsh_in = 0; nodes[2].dgsh_out = 1; /* dgsh IN and not OUT = termination node. * This node couldn't start the negotiation. * Fix. */ nodes[3].pid = 103; nodes[3].index = 3; strcpy(nodes[3].name, "proc3"); nodes[3].requires_channels = 2; nodes[3].provides_channels = 0; nodes[3].dgsh_in = 1; nodes[3].dgsh_out = 0; n_edges = 5; edges = (struct dgsh_edge *)malloc(sizeof(struct dgsh_edge) *n_edges); edges[0].from = 2; edges[0].to = 0; edges[0].instances = 0; edges[0].from_instances = 0; edges[0].to_instances = 0; edges[1].from = 2; edges[1].to = 1; edges[1].instances = 0; edges[1].from_instances = 0; edges[1].to_instances = 0; edges[2].from = 1; edges[2].to = 0; edges[2].instances = 0; edges[2].from_instances = 0; edges[2].to_instances = 0; edges[3].from = 1; edges[3].to = 3; edges[3].instances = 0; edges[3].from_instances = 0; edges[3].to_instances = 0; edges[4].from = 0; edges[4].to = 3; edges[4].instances = 0; edges[4].from_instances = 0; edges[4].to_instances = 0; double dgsh_version = 0.1; chosen_mb = (struct dgsh_negotiation *)malloc(sizeof(struct dgsh_negotiation)); chosen_mb->version = dgsh_version; chosen_mb->node_array = nodes; chosen_mb->n_nodes = n_nodes; chosen_mb->edge_array = edges; chosen_mb->n_edges = n_edges; chosen_mb->graph_solution = NULL; /* check_negotiation_round() */ chosen_mb->state = PS_NEGOTIATION; chosen_mb->initiator_pid = 103; /* Node 3 */ chosen_mb->origin_fd_direction = STDOUT_FILENO; chosen_mb->n_concs = 0; chosen_mb->conc_array = NULL; } /* Identical to chosen_mb except for the initiator field. */ void setup_mb(struct dgsh_negotiation **mb) { struct dgsh_node *nodes; struct dgsh_edge *edges; int n_nodes; int n_edges; n_nodes = 4; nodes = (struct dgsh_node *)malloc(sizeof(struct dgsh_node) * n_nodes); nodes[0].pid = 100; nodes[0].index = 0; strcpy(nodes[0].name, "proc0"); nodes[0].requires_channels = 2; nodes[0].provides_channels = 1; nodes[0].dgsh_in = 1; nodes[0].dgsh_out = 1; nodes[1].pid = 101; nodes[1].index = 1; strcpy(nodes[1].name, "proc1"); nodes[1].requires_channels = 1; nodes[1].provides_channels = 2; nodes[1].dgsh_in = 1; nodes[1].dgsh_out = 1; nodes[2].pid = 102; nodes[2].index = 2; strcpy(nodes[2].name, "proc2"); nodes[2].requires_channels = 0; nodes[2].provides_channels = 2; nodes[2].dgsh_in = 0; nodes[2].dgsh_out = 1; nodes[3].pid = 103; nodes[3].index = 3; strcpy(nodes[3].name, "proc3"); nodes[3].requires_channels = 2; nodes[3].provides_channels = 0; nodes[3].dgsh_in = 1; nodes[3].dgsh_out = 0; n_edges = 5; edges = (struct dgsh_edge *)malloc(sizeof(struct dgsh_edge) *n_edges); edges[0].from = 2; edges[0].to = 0; edges[0].instances = 0; edges[0].from_instances = 0; edges[0].to_instances = 0; edges[1].from = 2; edges[1].to = 1; edges[1].instances = 0; edges[1].from_instances = 0; edges[1].to_instances = 0; edges[2].from = 1; edges[2].to = 0; edges[2].instances = 0; edges[2].from_instances = 0; edges[2].to_instances = 0; edges[3].from = 1; edges[3].to = 3; edges[3].instances = 0; edges[3].from_instances = 0; edges[3].to_instances = 0; edges[4].from = 0; edges[4].to = 3; edges[4].instances = 0; edges[4].from_instances = 0; edges[4].to_instances = 0; double dgsh_version = 0.1; struct dgsh_negotiation *temp_mb = (struct dgsh_negotiation *)malloc(sizeof(struct dgsh_negotiation)); temp_mb->version = dgsh_version; temp_mb->node_array = nodes; temp_mb->n_nodes = n_nodes; temp_mb->edge_array = edges; temp_mb->n_edges = n_edges; temp_mb->graph_solution = NULL; /* check_negotiation_round() */ temp_mb->state = PS_NEGOTIATION; temp_mb->initiator_pid = 102; /* Node 2 */ temp_mb->origin_index = 2; temp_mb->origin_fd_direction = STDOUT_FILENO; temp_mb->n_concs = 0; temp_mb->conc_array = NULL; *mb = temp_mb; } void setup_pointers_to_edges(void) { n_ptedges = 2; pointers_to_edges = (struct dgsh_edge **)malloc(sizeof(struct dgsh_edge *) *n_ptedges); int i; for (i = 0; i < n_ptedges; i++) { pointers_to_edges[i] = (struct dgsh_edge *)malloc(sizeof(struct dgsh_edge)); pointers_to_edges[i]->from = i; pointers_to_edges[i]->to = 3; // the node. pointers_to_edges[i]->instances = 0; pointers_to_edges[i]->from_instances = 0; pointers_to_edges[i]->to_instances = 0; } } void setup_self_node(void) { /* fill in self_node */ memcpy(&self_node, &chosen_mb->node_array[3], sizeof(struct dgsh_node)); } void setup_self_node_io_side(void) { self_node_io_side.index = 3; self_node_io_side.fd_direction = 0; } /* establish_io_connections() */ void setup_pipe_fds(void) { /* fill in self_pipe_fds */ self_pipe_fds.n_input_fds = 2; self_pipe_fds.input_fds = (int *)malloc(sizeof(int) * self_pipe_fds.n_input_fds); self_pipe_fds.input_fds[0] = 3; self_pipe_fds.input_fds[1] = 4; self_pipe_fds.n_output_fds = 0; } void setup_args(void) { args = (int *)malloc(sizeof(int) * 3); args[0] = -1; args[1] = -1; args[2] = -1; } void setup(void) { setup_chosen_mb(); setup_self_node(); setup_self_node_io_side(); setup_pipe_fds(); setup_graph_solution(); } void setup_test_set_fds(void) { setup_chosen_mb(); setup_self_node(); } void setup_test_add_node(void) { setup_chosen_mb(); setup_self_node(); setup_self_node_io_side(); } void setup_test_lookup_dgsh_edge(void) { setup_chosen_mb(); } void setup_test_fill_dgsh_edge(void) { setup_chosen_mb(); setup_self_node(); setup_self_node_io_side(); } void setup_test_add_edge(void) { setup_chosen_mb(); } void setup_test_try_add_dgsh_edge(void) { setup_chosen_mb(); setup_self_node(); setup_self_node_io_side(); } void setup_test_try_add_dgsh_node(void) { setup_chosen_mb(); setup_self_node(); } void setup_test_fill_node(void) { /* setup_self_node() requires setup_chosen_mb() */ setup_chosen_mb(); setup_self_node(); } void setup_test_free_mb(void) { setup_chosen_mb(); } void setup_test_analyse_read(void) { setup_chosen_mb(); setup_mb(&fresh_mb); setup_self_node(); setup_self_node_io_side(); } /*void setup_test_point_io_direction(void) { setup_chosen_mb(); setup_self_node(); setup_self_node_io_side(); }*/ void setup_test_alloc_copy_graph_solution(void) { setup_mb(&fresh_mb); } void setup_test_alloc_copy_concs(void) { setup_mb(&fresh_mb); } void setup_test_alloc_copy_edges(void) { setup_mb(&fresh_mb); } void setup_test_alloc_copy_nodes(void) { setup_mb(&fresh_mb); } void setup_test_read_chunk(void) { setup_self_node_io_side(); } void setup_test_alloc_io_fds(void) { setup_chosen_mb(); setup_graph_solution(); setup_self_node(); } void setup_test_get_provided_fds_n(void) { setup_chosen_mb(); setup_graph_solution(); } void setup_test_get_expected_fds_n(void) { setup_chosen_mb(); setup_graph_solution(); } void setup_test_get_origin_pid(void) { setup_chosen_mb(); } void setup_test_read_input_fds(void) { setup_chosen_mb(); setup_graph_solution(); setup_self_node(); } void setup_test_read_graph_solution(void) { setup_mb(&fresh_mb); setup_chosen_mb(); setup_self_node_io_side(); } void setup_test_read_concs(void) { setup_mb(&fresh_mb); setup_chosen_mb(); setup_concs(chosen_mb); setup_self_node_io_side(); } void setup_test_write_graph_solution(void) { setup_chosen_mb(); setup_graph_solution(); setup_self_node_io_side(); } void setup_test_write_concs(void) { setup_chosen_mb(); setup_concs(chosen_mb); setup_self_node_io_side(); } void setup_test_read_message_block(void) { setup_chosen_mb(); setup_self_node_io_side(); } void setup_test_write_message_block(void) { setup_chosen_mb(); setup_self_node_io_side(); } void setup_test_make_compact_edge_array(void) { setup_pointers_to_edges(); } void setup_test_reallocate_edge_pointer_array(void) { setup_pointers_to_edges(); } /*void setup_test_assign_edge_instances(void) { setup_chosen_mb(); setup_pointers_to_edges(); } void setup_test_eval_constraints(void) { setup_args(); } */ void setup_test_move(void) { setup_pointers_to_edges(); } void setup_test_satisfy_io_constraints(void) { setup_chosen_mb(); setup_pointers_to_edges(); setup_args(); } void setup_test_dry_match_io_constraints(void) { setup_chosen_mb(); setup_graph_solution(); setup_pointers_to_edges(); setup_args(); } void setup_test_node_match_constraints(void) { setup_chosen_mb(); } void setup_test_free_graph_solution(void) { setup_chosen_mb(); setup_graph_solution(); } void setup_test_solve_graph(void) { setup_chosen_mb(); setup_graph_solution(); setup_pointers_to_edges(); setup_args(); } void setup_test_calculate_conc_fds(void) { setup_chosen_mb(); setup_graph_solution(); setup_concs(chosen_mb); } void setup_test_write_output_fds(void) { setup_chosen_mb(); /* For setting up graph_solution. */ setup_graph_solution(); setup_self_node(); } void setup_test_set_dispatcher(void) { setup_chosen_mb(); setup_self_node_io_side(); } void setup_test_establish_io_connections(void) { setup_pipe_fds(); setup_chosen_mb(); setup_self_node(); } void setup_pi(void) { pi = (struct portinfo *)calloc(5, sizeof(struct portinfo)); pi[0].pid = 101; pi[0].seen = false; pi[0].written = true; pi[1].pid = 100; pi[1].seen = true; pi[1].written = false; pi[3].pid = 103; pi[3].seen = true; pi[3].written = true; } void setup_test_is_ready(void) { setup_pi(); setup_chosen_mb(); } void setup_test_set_io_channels(void) { setup_pi(); setup_chosen_mb(); } void retire_pointers_to_edges(void) { int i; for (i = 0; i < n_ptedges; i++) free(pointers_to_edges[i]); free(pointers_to_edges); } void retire_graph_solution(struct dgsh_node_connections *graph_solution, int node_index) { int i; for (i = 0; i <= node_index; i++) { if (graph_solution[i].n_edges_incoming) free(graph_solution[i].edges_incoming); if (graph_solution[i].n_edges_outgoing) free(graph_solution[i].edges_outgoing); } free(graph_solution); } void retire_concs(struct dgsh_negotiation *mb) { int i; for (i = 0; i < mb->n_concs; i++) free(mb->conc_array[i].proc_pids); free(mb->conc_array); } void retire_chosen_mb(void) { free(chosen_mb->node_array); free(chosen_mb->edge_array); free(chosen_mb); } void retire_mb(struct dgsh_negotiation *mb) { free(mb->node_array); free(mb->edge_array); free(mb); } /* establish_io_connections() */ void retire_pipe_fds(void) { if (self_pipe_fds.n_input_fds > 0) free(self_pipe_fds.input_fds); if (self_pipe_fds.n_output_fds > 0) free(self_pipe_fds.output_fds); /* What about self_pipe_fds.input_fds? */ } void retire_args(void) { free(args); } void retire(void) { retire_graph_solution(chosen_mb->graph_solution, chosen_mb->n_nodes - 1); retire_chosen_mb(); retire_pipe_fds(); } void retire_test_set_fds(void) { retire_chosen_mb(); } void retire_test_construct_message_block(void) { retire_chosen_mb(); } void retire_test_add_node(void) { retire_chosen_mb(); } void retire_test_lookup_dgsh_edge(void) { retire_chosen_mb(); } void retire_test_fill_dgsh_edge(void) { retire_chosen_mb(); } void retire_test_add_edge(void) { retire_chosen_mb(); } void retire_test_try_add_dgsh_edge(void) { retire_chosen_mb(); } void retire_test_try_add_dgsh_node(void) { retire_chosen_mb(); } void retire_test_analyse_read(void) { if (exit_state == 1) { retire_mb(fresh_mb); exit_state = 0; } else retire_chosen_mb(); } /*void retire_test_point_io_direction(void) { retire_chosen_mb(); }*/ void retire_test_alloc_copy_graph_solution(void) { retire_mb(fresh_mb); } void retire_test_alloc_copy_concs(void) { retire_mb(fresh_mb); } void retire_test_alloc_copy_edges(void) { retire_mb(fresh_mb); } void retire_test_alloc_copy_nodes(void) { retire_mb(fresh_mb); } void retire_test_alloc_io_fds(void) { retire_pipe_fds(); retire_graph_solution(chosen_mb->graph_solution, chosen_mb->n_nodes - 1); retire_chosen_mb(); } void retire_test_get_provided_fds_n(void) { retire_graph_solution(chosen_mb->graph_solution, chosen_mb->n_nodes - 1); retire_chosen_mb(); } void retire_test_get_expected_fds_n(void) { retire_graph_solution(chosen_mb->graph_solution, chosen_mb->n_nodes - 1); retire_chosen_mb(); } void retire_test_read_input_fds(void) { retire_graph_solution(chosen_mb->graph_solution, chosen_mb->n_nodes - 1); retire_chosen_mb(); } void retire_test_get_origin_pid(void) { retire_chosen_mb(); } void retire_test_read_message_block(void) { retire_chosen_mb(); } void retire_test_write_message_block(void) { retire_chosen_mb(); } void retire_test_read_graph_solution(void) { retire_chosen_mb(); retire_mb(fresh_mb); } void retire_test_read_concs(void) { retire_concs(chosen_mb); retire_chosen_mb(); retire_mb(fresh_mb); } void retire_test_write_graph_solution(void) { retire_graph_solution(chosen_mb->graph_solution, chosen_mb->n_nodes - 1); retire_chosen_mb(); } void retire_test_write_concs(void) { retire_concs(chosen_mb); retire_chosen_mb(); } void retire_test_make_compact_edge_array(void) { free(compact_edges); retire_pointers_to_edges(); } void retire_test_reallocate_edge_pointer_array(void) { retire_pointers_to_edges(); } /*void retire_test_assign_edge_instances(void) { retire_chosen_mb(); retire_pointers_to_edges(); } void retire_test_eval_constraints(void) { retire_args(); } */ void retire_test_move(void) { retire_pointers_to_edges(); } void retire_test_satisfy_io_constraints(void) { retire_chosen_mb(); retire_pointers_to_edges(); retire_args(); } void retire_test_dry_match_io_constraints(void) { retire_graph_solution(chosen_mb->graph_solution, chosen_mb->n_nodes - 1); retire_chosen_mb(); retire_pointers_to_edges(); retire_args(); } void retire_test_node_match_constraints(void) { retire_graph_solution(chosen_mb->graph_solution, chosen_mb->n_nodes - 1); retire_chosen_mb(); } void retire_test_free_graph_solution(void) { retire_chosen_mb(); } void retire_test_solve_graph(void) { /* Are the other data structures handled correctly? * They could be deallocated above our feet. */ if (!exit_state) retire_graph_solution(chosen_mb->graph_solution, chosen_mb->n_nodes - 1); else exit_state = 0; retire_chosen_mb(); retire_pointers_to_edges(); retire_args(); } void retire_test_calculate_conc_fds(void) { retire_graph_solution(chosen_mb->graph_solution, chosen_mb->n_nodes - 1); retire_concs(chosen_mb); retire_chosen_mb(); } void retire_test_write_output_fds(void) { retire_graph_solution(chosen_mb->graph_solution, chosen_mb->n_nodes - 1); } void retire_test_set_dispatcher(void) { retire_chosen_mb(); } void retire_test_establish_io_connections(void) { /* See setup_test_establish_io_connections() */ retire_chosen_mb(); retire_pipe_fds(); } void retire_pi(void) { free(pi); } void retire_test_is_ready(void) { retire_pi(); retire_chosen_mb(); } void retire_test_set_io_channels(void) { retire_concs(chosen_mb); retire_chosen_mb(); retire_pi(); } START_TEST(test_solve_graph) { DPRINTF(4, "%s", __func__); /* A normal case with fixed, tight constraints. */ ck_assert_int_eq(solve_graph(), OP_SUCCESS); struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; ck_assert_int_eq(graph_solution[3].n_edges_incoming, 2); ck_assert_int_eq(graph_solution[3].n_edges_outgoing, 0); ck_assert_int_eq(chosen_mb->edge_array[3].instances, 1); ck_assert_int_eq(chosen_mb->edge_array[4].instances, 1); ck_assert_int_eq(graph_solution[3].edges_incoming[0].instances, 1); ck_assert_int_eq(graph_solution[0].edges_outgoing[0].instances, 1); ck_assert_int_eq(graph_solution[3].edges_incoming[1].instances, 1); ck_assert_int_eq(graph_solution[1].edges_outgoing[1].instances, 1); ck_assert_int_eq((long int)graph_solution[3].edges_outgoing, 0); retire_test_solve_graph(); /* An impossible case. */ setup_test_solve_graph(); chosen_mb->node_array[3].requires_channels = 1; ck_assert_int_eq(solve_graph(), OP_ERROR); exit_state = 1; retire_test_solve_graph(); /* Relaxing our target node's constraint. */ setup_test_solve_graph(); chosen_mb->node_array[3].requires_channels = -1; ck_assert_int_eq(solve_graph(), OP_SUCCESS); graph_solution = chosen_mb->graph_solution; ck_assert_int_eq(graph_solution[3].n_edges_incoming, 2); ck_assert_int_eq(graph_solution[3].n_edges_outgoing, 0); /* Pair edges still have tight constraints. */ ck_assert_int_eq(chosen_mb->edge_array[3].instances, 1); ck_assert_int_eq(chosen_mb->edge_array[4].instances, 1); ck_assert_int_eq(graph_solution[3].edges_incoming[0].instances, 1); ck_assert_int_eq(graph_solution[0].edges_outgoing[0].instances, 1); ck_assert_int_eq(graph_solution[3].edges_incoming[1].instances, 1); ck_assert_int_eq(graph_solution[1].edges_outgoing[1].instances, 1); ck_assert_int_eq((long int)graph_solution[3].edges_outgoing, 0); retire_test_solve_graph(); /* Relaxing also pair nodes' constraints. */ setup_test_solve_graph(); chosen_mb->node_array[3].requires_channels = -1; chosen_mb->node_array[0].provides_channels = -1; chosen_mb->node_array[1].provides_channels = -1; ck_assert_int_eq(solve_graph(), OP_SUCCESS); graph_solution = chosen_mb->graph_solution; ck_assert_int_eq(graph_solution[3].n_edges_incoming, 2); ck_assert_int_eq(graph_solution[3].n_edges_outgoing, 0); /* Flexible both sides: instances previously set to 5 */ ck_assert_int_eq(chosen_mb->edge_array[3].instances, 1); ck_assert_int_eq(chosen_mb->edge_array[4].instances, 1); ck_assert_int_eq(graph_solution[3].edges_incoming[0].instances, 1); ck_assert_int_eq(graph_solution[0].edges_outgoing[0].instances, 1); ck_assert_int_eq(graph_solution[3].edges_incoming[1].instances, 1); ck_assert_int_eq(graph_solution[1].edges_outgoing[1].instances, 1); ck_assert_int_eq((long int)graph_solution[3].edges_outgoing, 0); /* Collateral impact. Node 1 (flex) -> Node 0 (tight) */ ck_assert_int_eq(chosen_mb->edge_array[2].instances, 1); ck_assert_int_eq(graph_solution[1].edges_outgoing[0].instances, 1); } END_TEST START_TEST(test_calculate_conc_fds) { DPRINTF(4, "%s()", __func__); chosen_mb->conc_array[0].input_fds = -1; chosen_mb->conc_array[0].output_fds = -1; chosen_mb->conc_array[1].input_fds = -1; chosen_mb->conc_array[1].output_fds = -1; struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; graph_solution[0].edges_incoming[0].instances = 1; graph_solution[0].edges_outgoing[0].instances = 1; graph_solution[1].edges_incoming[0].instances = 1; graph_solution[1].edges_outgoing[0].instances = 1; /* endpoint for conc with pid 2001*/ graph_solution[3].edges_incoming[0].instances = 1; graph_solution[3].edges_incoming[1].instances = 1; /* endpoint for conc with pid 2000*/ graph_solution[2].edges_outgoing[0].instances = 1; graph_solution[2].edges_outgoing[1].instances = 1; ck_assert_int_eq(calculate_conc_fds(), OP_SUCCESS); } END_TEST START_TEST(test_free_graph_solution) { ck_assert_int_eq(free_graph_solution(3), OP_SUCCESS); /* Invalid node indexes, the function's argument, are checked by assertion. */ } END_TEST START_TEST(test_establish_io_connections) { /* Should be in the solution propagation test suite. */ /* The test case contains an arrangement of 0 fds and another of >0 fds. */ int *input_fds = NULL; int n_input_fds = 2; int *output_fds = NULL; int n_output_fds = 0; int fd[2]; if (pipe(fd) == -1) { /* fd pair: 4 -- 5 */ perror("pipe open failed"); exit(1); } self_pipe_fds.input_fds[0] = fd[0]; DPRINTF(4, "%s: Opened pipe pair: input_fds[0]: %d, output: %d", __func__, fd[0], fd[1]); ck_assert_int_eq(establish_io_connections(NULL, NULL, NULL, NULL), OP_SUCCESS); /* Freed */ ck_assert_int_eq(self_pipe_fds.n_input_fds, 0); ck_assert_int_eq(self_pipe_fds.n_output_fds, 0); close(fd[1]); retire_test_establish_io_connections(); setup_test_establish_io_connections(); if (pipe(fd) == -1) { /* fd pair: 4 -- 5 */ perror("pipe open failed"); exit(1); } self_pipe_fds.input_fds[0] = fd[0]; DPRINTF(4, "%s: Opened pipe pair: input_fds[0]: %d, output: %d", __func__, fd[0], fd[1]); self_pipe_fds.input_fds[1] = 6; ck_assert_int_eq(establish_io_connections(&input_fds, &n_input_fds, &output_fds, &n_output_fds), OP_SUCCESS); ck_assert_int_eq(n_input_fds, 2); ck_assert_int_eq(input_fds[0], 0); ck_assert_int_eq(input_fds[1], 6); ck_assert_int_eq(n_output_fds, 0); close(fd[1]); } END_TEST struct dgsh_edge **edges_in; int n_edges_in; struct dgsh_edge **edges_out; int n_edges_out; void retire_dmic(void) { retire(); if (n_edges_in) free(edges_in); if (n_edges_out) free(edges_out); } START_TEST(test_node_match_constraints) { DPRINTF(4, "%s()\n", __func__); /* Default topology; take a look at setup_chosen_mb() */ chosen_mb->node_array[3].requires_channels = 2; ck_assert_int_eq(node_match_constraints(), OP_SUCCESS); struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; ck_assert_int_eq(graph_solution[0].node_index, 0); ck_assert_int_eq(graph_solution[0].n_edges_incoming, 2); ck_assert_int_eq(graph_solution[0].n_instances_incoming_free, 0); ck_assert_int_eq(graph_solution[0].n_edges_outgoing, 1); ck_assert_int_eq(graph_solution[0].n_instances_outgoing_free, 0); ck_assert_int_eq(graph_solution[1].node_index, 1); ck_assert_int_eq(graph_solution[1].n_edges_incoming, 1); ck_assert_int_eq(graph_solution[1].n_instances_incoming_free, 0); ck_assert_int_eq(graph_solution[1].n_edges_outgoing, 2); ck_assert_int_eq(graph_solution[1].n_instances_outgoing_free, 0); ck_assert_int_eq(graph_solution[2].node_index, 2); ck_assert_int_eq(graph_solution[2].n_edges_incoming, 0); ck_assert_int_eq(graph_solution[2].n_instances_incoming_free, 0); ck_assert_int_eq(graph_solution[2].n_edges_outgoing, 2); ck_assert_int_eq(graph_solution[2].n_instances_outgoing_free, 0); ck_assert_int_eq(graph_solution[3].node_index, 3); ck_assert_int_eq(graph_solution[3].n_edges_incoming, 2); ck_assert_int_eq(graph_solution[3].n_instances_incoming_free, 0); ck_assert_int_eq(graph_solution[3].n_edges_outgoing, 0); ck_assert_int_eq(graph_solution[3].n_instances_outgoing_free, 0); } END_TEST START_TEST(test_dry_match_io_constraints) { DPRINTF(4, "%s", __func__); struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; /* A normal case with fixed, tight constraints. */ struct dgsh_node_connections *current_connections = &graph_solution[3]; current_connections->n_edges_incoming = 0; current_connections->n_edges_outgoing = 0; ck_assert_int_eq(dry_match_io_constraints(&chosen_mb->node_array[3], current_connections, &edges_in, &edges_out), OP_SUCCESS); /* Hard coded. Observe the topology of the prototype solution in setup(). */ ck_assert_int_eq(current_connections->n_edges_incoming, 2); ck_assert_int_eq(current_connections->n_edges_outgoing, 0); /* A case not matching at first sight; match result will * be decided in cross_match_constraints() */ current_connections->n_edges_incoming = 0; current_connections->n_edges_outgoing = 0; chosen_mb->node_array[3].requires_channels = 3; ck_assert_int_eq(dry_match_io_constraints(&chosen_mb->node_array[3], current_connections, &edges_in, &edges_out), OP_SUCCESS); ck_assert_int_eq(current_connections->n_edges_incoming, 2); ck_assert_int_eq(current_connections->n_edges_outgoing, 0); /* Relaxing our target node's constraint. */ current_connections->n_edges_incoming = 0; current_connections->n_edges_outgoing = 0; chosen_mb->node_array[3].requires_channels = -1; ck_assert_int_eq(dry_match_io_constraints(&chosen_mb->node_array[3], current_connections, &edges_in, &edges_out), OP_SUCCESS); ck_assert_int_eq(current_connections->n_edges_incoming, 2); ck_assert_int_eq(current_connections->n_edges_outgoing, 0); ck_assert_int_eq(chosen_mb->edge_array[3].to_instances, -1); ck_assert_int_eq(chosen_mb->edge_array[4].to_instances, -1); } END_TEST START_TEST(test_satisfy_io_constraints) { /* To be concise, when changing the second argument that mirrors * the channel constraint of the node under evaluation, we should * also change it in the node array, but it does not matter since it * is the pair nodes that we are interested in. */ int free_instances = 0; /* Fixed constraint both sides, just satisfy. */ ck_assert_int_eq(satisfy_io_constraints(&free_instances, 2, pointers_to_edges, 2, true), OP_SUCCESS); ck_assert_int_eq(free_instances, 0); /* Fixed constraint both sides, not matching at * first sight, but will leave it to cross_match_constraints() * to decide */ ck_assert_int_eq(satisfy_io_constraints(&free_instances, 1, pointers_to_edges, 2, true), OP_SUCCESS); ck_assert_int_eq(free_instances, 0); /* Fixed constraint bith sides, plenty. */ ck_assert_int_eq(satisfy_io_constraints(&free_instances, 5, pointers_to_edges, 2, true), OP_SUCCESS); ck_assert_int_eq(free_instances, 0); /* Fixed constraint node, flexible pair, just one. */ chosen_mb->node_array[0].provides_channels = -1; ck_assert_int_eq(satisfy_io_constraints(&free_instances, 2, pointers_to_edges, 2, true), OP_SUCCESS); ck_assert_int_eq(free_instances, 0); /* Fixed constraint node, flexible pair, * cross_match_constraints() will decide */ chosen_mb->node_array[0].provides_channels = -1; ck_assert_int_eq(satisfy_io_constraints(&free_instances, 1, pointers_to_edges, 2, true), OP_SUCCESS); ck_assert_int_eq(free_instances, 0); retire_test_satisfy_io_constraints(); /* Expand the semantics of remaining_free_channels to fixed constraints as in this case. */ /* Fixed constraint node, flexible pair, plenty. */ setup_test_satisfy_io_constraints(); chosen_mb->node_array[0].provides_channels = -1; ck_assert_int_eq(satisfy_io_constraints(&free_instances, 5, pointers_to_edges, 2, true), OP_SUCCESS); ck_assert_int_eq(free_instances, 0); free_instances = 0; retire_test_satisfy_io_constraints(); /* Flexible constraint both sides */ setup_test_satisfy_io_constraints(); chosen_mb->node_array[0].provides_channels = -1; ck_assert_int_eq(satisfy_io_constraints(&free_instances, -1, pointers_to_edges, 2, 1), OP_SUCCESS); ck_assert_int_eq(free_instances, -1); free_instances = 0; } END_TEST START_TEST(test_move) { int diff = 1; bool is_edge_incoming = true; DPRINTF(4, "%s()", __func__); pointers_to_edges[0]->from_instances = 1; pointers_to_edges[0]->to_instances = 2; pointers_to_edges[1]->from_instances = 2; pointers_to_edges[1]->to_instances = 1; ck_assert_int_eq(move(pointers_to_edges, n_ptedges, diff, is_edge_incoming), OP_SUCCESS); ck_assert_int_eq(pointers_to_edges[0]->from_instances, 1); ck_assert_int_eq(pointers_to_edges[0]->to_instances, 2); ck_assert_int_eq(pointers_to_edges[1]->from_instances, 2); ck_assert_int_eq(pointers_to_edges[1]->to_instances, 2); } END_TEST START_TEST(test_record_move_flexible) { /* Successful increase move */ int diff = 1; int index = -1; int to_move_index = 2; int instances = 0; int to_move = 2; record_move_flexible(&diff, &index, to_move_index, &instances, to_move); ck_assert_int_eq(diff, 0); ck_assert_int_eq(index, to_move_index); ck_assert_int_eq(instances, 1); /* Can't subtract instances from size 1 */ diff = -1; index = -1; to_move_index = 2; instances = 0; to_move = 1; record_move_flexible(&diff, &index, to_move_index, &instances, to_move); ck_assert_int_eq(diff, -1); ck_assert_int_eq(index, -1); ck_assert_int_eq(instances, 0); /* Successful decrease. diff greater than to_move */ diff = -3; index = -1; to_move_index = 2; instances = 0; to_move = 2; record_move_flexible(&diff, &index, to_move_index, &instances, to_move); ck_assert_int_eq(diff, -2); ck_assert_int_eq(index, to_move_index); ck_assert_int_eq(instances, -1); /* Successful decrease. diff smaller than to_move */ diff = -2; index = -1; to_move_index = 2; instances = 0; to_move = 4; record_move_flexible(&diff, &index, to_move_index, &instances, to_move); ck_assert_int_eq(diff, 0); ck_assert_int_eq(index, to_move_index); ck_assert_int_eq(instances, -2); } END_TEST START_TEST(test_record_move_unbalanced) { /* Successful increase move */ int diff = 1; int index = -1; int to_move_index = 2; int instances = 0; int to_move = 2; int pair = 3; record_move_unbalanced(&diff, &index, to_move_index, &instances, to_move, pair); ck_assert_int_eq(diff, 0); ck_assert_int_eq(index, to_move_index); ck_assert_int_eq(instances, 1); /* Successful decrease. diff greater than to_move - pair */ diff = -3; index = -1; to_move_index = 2; instances = 0; to_move = 2; pair = 1; record_move_unbalanced(&diff, &index, to_move_index, &instances, to_move, pair); ck_assert_int_eq(diff, -2); ck_assert_int_eq(index, to_move_index); ck_assert_int_eq(instances, -1); /* Successful decrease. diff smaller than to_move - pair */ diff = -2; index = -1; to_move_index = 2; instances = 0; to_move = 4; pair = 1; record_move_unbalanced(&diff, &index, to_move_index, &instances, to_move, pair); ck_assert_int_eq(diff, 0); ck_assert_int_eq(index, to_move_index); ck_assert_int_eq(instances, -2); } END_TEST START_TEST(test_reallocate_edge_pointer_array) { ck_assert_int_eq(reallocate_edge_pointer_array(NULL, 1), OP_ERROR); ck_assert_int_eq(reallocate_edge_pointer_array(&pointers_to_edges, -2), OP_ERROR); ck_assert_int_eq(reallocate_edge_pointer_array(&pointers_to_edges, 0), OP_ERROR); /* Not incresing the value of n_ptedges to not perplex freeing * pointers_to_edges because reallocation only accounts for * struct dgsh_edge *. */ ck_assert_int_eq(reallocate_edge_pointer_array(&pointers_to_edges, n_ptedges + 1), OP_SUCCESS); } END_TEST START_TEST(test_make_compact_edge_array) { ck_assert_int_eq(make_compact_edge_array(NULL, 2, pointers_to_edges), OP_ERROR); ck_assert_int_eq(make_compact_edge_array(&compact_edges, -2, pointers_to_edges), OP_ERROR); ck_assert_int_eq(make_compact_edge_array(&compact_edges, 0, pointers_to_edges), OP_ERROR); ck_assert_int_eq(make_compact_edge_array(&compact_edges, n_ptedges, NULL), OP_ERROR); struct dgsh_edge *p = pointers_to_edges[0]; pointers_to_edges[0] = NULL; ck_assert_int_eq(make_compact_edge_array(&compact_edges, n_ptedges, pointers_to_edges), OP_ERROR); pointers_to_edges[0] = p; ck_assert_int_eq(make_compact_edge_array(&compact_edges, n_ptedges, pointers_to_edges), OP_SUCCESS); } END_TEST START_TEST(test_write_output_fds) { int write_fd = 1; int *output_fds; /* 0 outgoing edges for node 3, so no action really. */ ck_assert_int_eq(write_output_fds(write_fd, output_fds, 0), OP_SUCCESS); /* Switch to node 2 that has 2 outgoing edges. */ memcpy(&self_node, &chosen_mb->node_array[2], sizeof(struct dgsh_node)); output_fds = (int *)malloc(sizeof(int) * 2); ck_assert_int_eq(write_output_fds(write_fd, output_fds, DGSH_HANDLE_ERROR), OP_SUCCESS); free(output_fds); /* Incomplete testing since socket descriptors have not yet been setup. * This will hapeen through the shell. */ } END_TEST START_TEST(test_set_dispatcher) { set_dispatcher(); ck_assert_int_eq(chosen_mb->origin_index, 3); ck_assert_int_eq(chosen_mb->origin_fd_direction, 0); /* The input side */ } END_TEST START_TEST(test_alloc_node_connections) { struct dgsh_edge *test; /* It is assumed that negative number of edges have already * been checked. See e.g. read_graph_solution(). */ ck_assert_int_eq(alloc_node_connections(NULL, 2, 1, 2), OP_ERROR); ck_assert_int_eq(alloc_node_connections(&test, 1, 2, 2), OP_ERROR); ck_assert_int_eq(alloc_node_connections(&test, 1, -1, 2), OP_ERROR); ck_assert_int_eq(alloc_node_connections(&test, 1, 1, -2), OP_ERROR); ck_assert_int_eq(alloc_node_connections(&test, 1, 1, 2), OP_SUCCESS); free(test); } END_TEST START_TEST(test_write_concs) { int fd[2]; int buf_size = getpagesize(); int pid; int i; int n_concs = chosen_mb->n_concs; int concs_size = sizeof(struct dgsh_conc) * n_concs; struct dgsh_conc *conc_array = (struct dgsh_conc *)malloc(concs_size); if (pipe(fd) == -1) { perror("pipe open failed"); exit(1); } DPRINTF(4, "%s()...", __func__); DPRINTF(4, "Opened pipe pair %d - %d.", fd[0], fd[1]); pid = fork(); if (pid <= 0) { int rsize = -1; DPRINTF(4, "Child speaking with pid %d.", (int)getpid()); close(fd[1]); DPRINTF(4, "Child reads concs of size %d.", concs_size); rsize = read(fd[0], conc_array, concs_size); if (rsize == -1) { DPRINTF(4, "Write concs failed."); exit(1); } DPRINTF(4, "Child: closes fd %d.", fd[0]); close(fd[0]); DPRINTF(4, "Child with pid %d exits.", (int)getpid()); } else { DPRINTF(4, "Parent speaking with pid %d.", (int)getpid()); ck_assert_int_eq(write_concs(fd[1]), OP_SUCCESS); } } END_TEST /* Incomplete? */ START_TEST(test_write_graph_solution) { int fd[2]; int buf_size = getpagesize(); int pid; int i; int n_nodes = chosen_mb->n_nodes; int graph_solution_size = sizeof(struct dgsh_node_connections) * n_nodes; struct dgsh_node_connections *graph_solution = (struct dgsh_node_connections *)malloc(graph_solution_size); if (pipe(fd) == -1) { perror("pipe open failed"); exit(1); } DPRINTF(4, "%s()...", __func__); DPRINTF(4, "Opened pipe pair %d - %d.", fd[0], fd[1]); pid = fork(); if (pid <= 0) { int rsize = -1; DPRINTF(4, "Child speaking with pid %d.", (int)getpid()); close(fd[1]); DPRINTF(4, "Child reads graph solution of size %d.", graph_solution_size); rsize = read(fd[0], graph_solution, graph_solution_size); if (rsize == -1) { DPRINTF(4, "Write graph solution failed."); exit(1); } for (i = 0; i < chosen_mb->n_nodes; i++) { struct dgsh_node_connections *nc = &graph_solution[i]; int in_edges_size = sizeof(struct dgsh_edge) * nc->n_edges_incoming; int out_edges_size = sizeof(struct dgsh_edge) * nc->n_edges_outgoing; if ((in_edges_size > buf_size) || (out_edges_size > buf_size)) { DPRINTF(4, "Dgsh negotiation graph solution for node at index %d: incoming connections of size %d or outgoing connections of size %d do not fit to buffer of size %d.\n", nc->node_index, in_edges_size, out_edges_size, buf_size); exit(1); } DPRINTF(4, "Child reads incoming edges of node %d in fd %d. Total size: %d", i, fd[1], in_edges_size); /* Transmit a node's incoming connections. */ rsize = read(fd[0], nc->edges_incoming, in_edges_size); if (rsize == -1) { DPRINTF(4, "Read edges incoming failed."); exit(1); } DPRINTF(4, "Child reads outgoing edges of node %d in fd %d. Total size: %d", i, fd[1], out_edges_size); /* Transmit a node's outgoing connections. */ rsize = read(fd[0], nc->edges_outgoing, out_edges_size); if (rsize == -1) { DPRINTF(4, "Read edges outgoing failed."); exit(1); } } DPRINTF(4, "Child: closes fd %d.", fd[0]); close(fd[0]); DPRINTF(4, "Child with pid %d exits.", (int)getpid()); } else { DPRINTF(4, "Parent speaking with pid %d.", (int)getpid()); ck_assert_int_eq(write_graph_solution(fd[1]), OP_SUCCESS); } } END_TEST /* Incomplete? */ START_TEST(test_write_message_block) { int fd[2]; int pid; DPRINTF(4, "%s()", __func__); if(pipe(fd) == -1){ perror("pipe open failed"); exit(1); } DPRINTF(4, "Opened pipe pair %d - %d.", fd[0], fd[1]); pid = fork(); if (pid <= 0) { DPRINTF(4, "Child speaking with pid %d.", (int)getpid()); struct dgsh_negotiation *test_mb = (struct dgsh_negotiation *) malloc(sizeof(struct dgsh_negotiation)); int mb_struct_size = sizeof(struct dgsh_negotiation); int i = 0; int rsize = -1; close(fd[1]); DPRINTF(4, "Child reads message block structure of size %d.", mb_struct_size); rsize = read(fd[0], test_mb, mb_struct_size); if (rsize == -1) { DPRINTF(4, "Read message block failed."); exit(1); } int n_nodes = test_mb->n_nodes; int n_edges = test_mb->n_edges; int mb_nodes_size = sizeof(struct dgsh_node) * n_nodes; test_mb->node_array = (struct dgsh_node *)malloc(mb_nodes_size); int mb_edges_size = sizeof(struct dgsh_edge) * n_edges; test_mb->edge_array = (struct dgsh_edge *)malloc(mb_edges_size); DPRINTF(4, "Child reads message block node array of size %d.", mb_nodes_size); rsize = read(fd[0], test_mb->node_array, mb_nodes_size); if (rsize == -1) { DPRINTF(4, "Read node array failed."); exit(1); } DPRINTF(4, "Child reads message block edge array of size %d.", mb_edges_size); rsize = read(fd[0], test_mb->edge_array, mb_edges_size); if (rsize == -1) { DPRINTF(4, "Read edge array failed."); exit(1); } for (i = 0; i < test_mb->n_edges; i++) { struct dgsh_edge *e = &test_mb->edge_array[i]; DPRINTF(4, "Edge from: %d, to: %d", e->from, e->to); } DPRINTF(4, "Child: closes fd %d.", fd[0]); close(fd[0]); DPRINTF(4, "Child with pid %d exits.", (int)getpid()); retire_mb(test_mb); } else { DPRINTF(4, "Parent speaking with pid %d.", (int)getpid()); ck_assert_int_eq(write_message_block(fd[1]), OP_SUCCESS); } } END_TEST /* Incomplete? */ START_TEST(test_read_message_block) { int fd[2]; int pid; DPRINTF(4, "%s()", __func__); if(pipe(fd) == -1){ perror("pipe open failed"); exit(1); } DPRINTF(4, "Opened pipe pair %d - %d.", fd[0], fd[1]); pid = fork(); if (pid <= 0) { DPRINTF(4, "Child speaking with pid %d.", (int)getpid()); struct dgsh_negotiation *test_mb; setup_mb(&test_mb); int n_nodes = test_mb->n_nodes; int n_edges = test_mb->n_edges; int mb_struct_size = sizeof(struct dgsh_negotiation); int mb_nodes_size = sizeof(struct dgsh_node) * n_nodes; int mb_edges_size = sizeof(struct dgsh_edge) * n_edges; int i = 0; int wsize = -1; struct timespec tm; tm.tv_sec = 0; tm.tv_nsec = 1000000; close(fd[0]); DPRINTF(4, "Child writes message block structure of size %d.", mb_struct_size); wsize = write(fd[1], test_mb, mb_struct_size); if (wsize == -1) { DPRINTF(4, "Write message block structure failed."); exit(1); } /* Sleep for 1 millisecond to write-read orderly. * Why do we need this? * Shouldn't the write block by deafult? */ nanosleep(&tm, NULL); DPRINTF(4, "Child writes message block node array of size %d.", mb_nodes_size); wsize = write(fd[1], test_mb->node_array, mb_nodes_size); if (wsize == -1) { DPRINTF(4, "Write message block node array failed."); exit(1); } /* Sleep for 1 millisecond before the next operation. */ nanosleep(&tm, NULL); DPRINTF(4, "Child writes message block edge array of size %d.", mb_edges_size); for (i = 0; i < test_mb->n_edges; i++) { struct dgsh_edge *e = &test_mb->edge_array[i]; DPRINTF(4, "Edge from: %d, to: %d", e->from, e->to); } wsize = write(fd[1], test_mb->edge_array, mb_edges_size); if (wsize == -1) { DPRINTF(4, "Write message block edge array failed."); exit(1); } DPRINTF(4, "Child: closes fd %d.", fd[1]); close(fd[1]); DPRINTF(4, "Child with pid %d exits.", (int)getpid()); retire_mb(test_mb); } else { DPRINTF(4, "Parent speaking with pid %d.", (int)getpid()); ck_assert_int_eq(read_message_block(fd[0], &fresh_mb), OP_SUCCESS); } } END_TEST /* Incomplete? */ START_TEST(test_read_graph_solution) { int fd[2]; int pid; int i; int n_nodes = fresh_mb->n_nodes; int buf_size = getpagesize(); int graph_solution_size = sizeof(struct dgsh_node_connections) * n_nodes; struct timespec tm; tm.tv_sec = 0; tm.tv_nsec = 1000000; DPRINTF(4, "%s()", __func__); if(pipe(fd) == -1){ perror("pipe open failed"); exit(1); } DPRINTF(4, "Opened pipe pair %d - %d.", fd[0], fd[1]); pid = fork(); if (pid <= 0) { int wsize = -1; DPRINTF(4, "Child speaking with pid %d.", (int)getpid()); setup_graph_solution(); struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; close(fd[0]); DPRINTF(4, "Child writes graph solution of size %d.", graph_solution_size); wsize = write(fd[1], graph_solution, graph_solution_size); if (wsize == -1) { DPRINTF(4, "Write graph solution failed."); exit(1); } /* Sleep for 1 millisecond before the next operation. */ nanosleep(&tm, NULL); for (i = 0; i < chosen_mb->n_nodes; i++) { struct dgsh_node_connections *nc = &graph_solution[i]; int in_edges_size = sizeof(struct dgsh_edge) * nc->n_edges_incoming; int out_edges_size = sizeof(struct dgsh_edge) * nc->n_edges_outgoing; int wsize = -1; if ((in_edges_size > buf_size) || (out_edges_size > buf_size)) { DPRINTF(4, "Dgsh negotiation graph solution for node at index %d: incoming connections of size %d or outgoing connections of size %d do not fit to buffer of size %d.\n", nc->node_index, in_edges_size, out_edges_size, buf_size); exit(1); } DPRINTF(4, "Child writes incoming edges of node %d in fd %d. Total size: %d", i, fd[1], in_edges_size); /* Transmit a node's incoming connections. */ wsize = write(fd[1], nc->edges_incoming, in_edges_size); if (wsize == -1) { DPRINTF(4, "Write edges incoming failed."); exit(1); } /* Sleep for 1 millisecond before the next operation. */ nanosleep(&tm, NULL); DPRINTF(4, "Child writes outgoing edges of node %d in fd %d. Total size: %d", i, fd[1], out_edges_size); /* Transmit a node's outgoing connections. */ wsize = write(fd[1], nc->edges_outgoing, out_edges_size); if (wsize == -1) { DPRINTF(4, "Write edges outgoing failed."); exit(1); } /* Sleep for 1 millisecond before the next operation. */ nanosleep(&tm, NULL); } DPRINTF(4, "Child: closes fd %d.", fd[1]); close(fd[1]); DPRINTF(4, "Child with pid %d exits.", (int)getpid()); retire_graph_solution(graph_solution, chosen_mb->n_nodes - 1); } else { DPRINTF(4, "Parent speaking with pid %d.", (int)getpid()); ck_assert_int_eq(read_graph_solution(fd[0], fresh_mb), OP_SUCCESS); } } END_TEST START_TEST(test_read_concs) { int fd[2]; int pid; int i; int n_concs = fresh_mb->n_concs; int buf_size = getpagesize(); int concs_size = sizeof(struct dgsh_conc) * n_concs; DPRINTF(4, "%s()", __func__); if(pipe(fd) == -1){ perror("pipe open failed"); exit(1); } DPRINTF(4, "Opened pipe pair %d - %d.", fd[0], fd[1]); pid = fork(); if (pid <= 0) { int wsize = -1; DPRINTF(4, "Child speaking with pid %d.", (int)getpid()); setup_graph_solution(); struct dgsh_conc *concs = chosen_mb->conc_array; close(fd[0]); DPRINTF(4, "Child writes concs of size %d.", concs_size); wsize = write(fd[1], concs, concs_size); if (wsize == -1) { DPRINTF(4, "Write concs failed."); exit(1); } } else { DPRINTF(4, "Parent speaking with pid %d.", (int)getpid()); ck_assert_int_eq(read_concs(fd[0], fresh_mb), OP_SUCCESS); } } END_TEST START_TEST(test_alloc_fds) { int *fds = NULL; int n_fds = 0; ck_assert_int_eq(alloc_fds(&fds, n_fds), OP_SUCCESS); ck_assert_int_eq((int)(long)fds, 0); n_fds = 2; ck_assert_int_eq(alloc_fds(&fds, n_fds), OP_SUCCESS); ck_assert_int_ne((int)(long)fds, 0); free(fds); } END_TEST START_TEST(test_alloc_io_fds) { /* By default initialised to 0. See setup_chosen_mb() */ chosen_mb->graph_solution[3].edges_incoming[0].instances = 1; chosen_mb->graph_solution[3].edges_incoming[1].instances = 1; ck_assert_int_eq(alloc_io_fds(), OP_SUCCESS); ck_assert_int_eq(self_pipe_fds.n_input_fds, 2); ck_assert_int_eq(self_pipe_fds.n_output_fds, 0); } END_TEST START_TEST(test_get_origin_pid) { chosen_mb->origin_index = 3; ck_assert_int_eq(get_origin_pid(chosen_mb), 103); chosen_mb->origin_index = 1; ck_assert_int_eq(get_origin_pid(chosen_mb), 101); } END_TEST START_TEST(test_get_expected_fds_n) { struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; graph_solution[0].edges_incoming[0].instances = 1; graph_solution[1].edges_incoming[0].instances = 1; graph_solution[3].edges_incoming[0].instances = 1; graph_solution[3].edges_incoming[1].instances = 1; ck_assert_int_eq(get_expected_fds_n(chosen_mb, 103), 2); ck_assert_int_eq(get_expected_fds_n(chosen_mb, 100), 1); ck_assert_int_eq(get_expected_fds_n(chosen_mb, 101), 1); ck_assert_int_eq(get_expected_fds_n(chosen_mb, 102), 0); } END_TEST START_TEST(test_get_provided_fds_n) { struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; graph_solution[0].edges_outgoing[0].instances = 1; graph_solution[1].edges_outgoing[0].instances = 1; graph_solution[1].edges_outgoing[1].instances = 1; graph_solution[2].edges_outgoing[0].instances = 1; graph_solution[2].edges_outgoing[1].instances = 1; ck_assert_int_eq(get_provided_fds_n(chosen_mb, 103), 0); ck_assert_int_eq(get_provided_fds_n(chosen_mb, 100), 1); ck_assert_int_eq(get_provided_fds_n(chosen_mb, 101), 2); ck_assert_int_eq(get_provided_fds_n(chosen_mb, 102), 2); } END_TEST START_TEST (test_read_write_fd) { int pipefd[2]; int sock, rsock; int readfd; char msg[] = "hello"; char buff[20]; int n; pid_t pid; socklen_t len; struct sockaddr_un local, remote; if (pipe(pipefd) == -1) err(1, "pipe"); local.sun_family = AF_UNIX; snprintf(local.sun_path, sizeof(local.sun_path), "/tmp/conc-%d", getpid()); len = strlen(local.sun_path) + 1 + sizeof(local.sun_family); switch ((pid = fork())) { case 0: /* Child: connect, pass fd and write test message */ if ((sock = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) err(1, "socket"); sleep(1); if (connect(sock, (struct sockaddr *)&local, len) == -1) err(1, "connect %s", local.sun_path); write_fd(sock, pipefd[STDIN_FILENO]); close(sock); // Should wait for data to be transmitted close(pipefd[STDIN_FILENO]); sleep(1); if (write(pipefd[STDOUT_FILENO], msg, sizeof(msg)) <= 0) err(1, "write"); exit(0); default: /* Parent: accept connection, read fd and read test message */ close(pipefd[STDIN_FILENO]); close(pipefd[STDOUT_FILENO]); if ((sock = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) err(1, "socket"); if (bind(sock, (struct sockaddr *)&local, len) == -1) err(1, "bind %s", local.sun_path); if (listen(sock, 5) == -1) err(1, "listen"); rsock = accept(sock, (struct sockaddr *)&remote, &len); readfd = read_fd(rsock); if ((n = read(readfd, buff, sizeof(buff))) == -1) err(1, "read"); (void)unlink(local.sun_path); ck_assert_int_eq(n, sizeof(msg)); ck_assert_str_eq(msg, buff); break; case -1: /* Error */ err(1, "fork"); } } END_TEST /* Incomplete? */ START_TEST(test_read_input_fds) { int sockets[2]; int fd, ping; struct msghdr msg; struct iovec vec[1]; union fdmsg cmsg; struct cmsghdr *h; int wsize = -1; memset(&msg, 0, sizeof(struct msghdr)); DPRINTF(4, "%s()...pid %d", __func__, (int)getpid()); if (socketpair(AF_UNIX, SOCK_DGRAM, 0, sockets) < 0) { perror("Error opening stream socket pair. Exiting now."); exit(1); } DPRINTF(4, "Opened socket pair %d - %d.", sockets[0], sockets[1]); fd = open("unit-test-dgsh", O_CREAT | O_RDWR, 0660); wsize = write(fd, "Unit testing dgsh...", 21); if (wsize == -1) { DPRINTF(4, "Write to 'unit-test-dgsh' failed."); exit(1); } close(fd); fd = open("unit-test-dgsh", O_RDONLY); if (fd < 0) { perror("Failed to open file test-dgsh for reading."); exit(1); } int pid = fork(); if (pid <= 0) { DPRINTF(4, "Child speaking with pid %d.", (int)getpid()); DPRINTF(4, "Child closes socket %d.", sockets[1]); close(sockets[1]); vec[0].iov_base = &ping; vec[0].iov_len = 1; msg.msg_iov = vec; msg.msg_iovlen = 1; msg.msg_name = 0; msg.msg_namelen = 0; msg.msg_control = cmsg.buf; msg.msg_controllen = sizeof(union fdmsg); msg.msg_flags = 0; h = CMSG_FIRSTHDR(&msg); h->cmsg_level = SOL_SOCKET; h->cmsg_type = SCM_RIGHTS; h->cmsg_len = CMSG_LEN(sizeof(int)); *((int*)CMSG_DATA(h)) = fd; DPRINTF(4, "Child goes sendmsg()"); if((sendmsg(sockets[0], &msg, 0)) < 0){ perror("sendmsg()"); exit(EXIT_FAILURE); } DPRINTF(4, "Child: closes fd %d.", fd); close(fd); DPRINTF(4, "Child with pid %d exits.", (int)getpid()); } else { struct dgsh_node_connections *graph_solution = chosen_mb->graph_solution; memcpy(&self_node, &chosen_mb->node_array[1], sizeof(struct dgsh_node)); /* Edges have been setup with 0 instances. * See setup_chosen_mb(). * Edges then copied to graph_solution. * See setup_graph_solution(). */ graph_solution[1].edges_incoming[0].instances = 1; int *input_fds = (int *)malloc(sizeof(int)); input_fds[0] = -1; DPRINTF(4, "Parent speaking with pid %d.", (int)getpid()); ck_assert_int_eq(read_input_fds(sockets[1], input_fds), OP_SUCCESS); ck_assert_int_ge(input_fds[0], 3); /* Hard-coded ceiling to check whether some weird * error has caused some random number to slip in. */ ck_assert_int_le(input_fds[0], 20); free(input_fds); DPRINTF(4, "Parent with pid %d exits.", (int)getpid()); } close(sockets[0]); close(sockets[1]); } END_TEST /* Incomplete? */ START_TEST(test_read_chunk) { /* Requires setting up of I/O multiplexing (a bash shell extension) * in order to be able to support bidirectional negotiation. * Without it we cannot test this function because it tries to read * repeatedly from stdin and then stdout until it manages. * We cannot feed it; writing to stdin or stdout * ends up in the unit test file output). * The good news is that the core of this function is the call to * call read, which has been successfully tested. * Then there is variable checking to determine the exit code. */ int fd[2]; int wsize = -1; DPRINTF(4, "%s()...", __func__); if(pipe(fd) == -1){ perror("pipe open failed"); exit(1); } /* Broken pipe error if we close the other side. */ //close(fd[0]); wsize = write(fd[1], "test-in", 9); if (wsize == -1) { DPRINTF(4, "Write to 'test-in' failed."); exit(1); } char buf[32]; int read_fd = -1; int bytes_read = -1; ck_assert_int_eq(read_chunk(fd[0], buf, 32, &bytes_read, 5), OP_SUCCESS); ck_assert_int_eq(bytes_read, 9); close(fd[0]); close(fd[1]); } END_TEST START_TEST(test_call_read) { int fd[2]; int wsize = -1; DPRINTF(4, "%s()...", __func__); if(pipe(fd) == -1){ perror("pipe open failed"); exit(1); } if ((wsize = write(fd[1], "test", 5)) == -1) { DPRINTF(4, "Write to 'test' failed.\n"); exit(1); } char buf[32]; int bytes_read = -1; int error_code = -1; ck_assert_int_eq(call_read(fd[0], buf, 32, &bytes_read, &error_code), OP_SUCCESS); ck_assert_int_eq(bytes_read, 5); ck_assert_int_eq(error_code, 0); close(fd[0]); close(fd[1]); } END_TEST START_TEST(test_alloc_copy_mb) { const int size = sizeof(struct dgsh_negotiation); char buf[512]; struct dgsh_negotiation *mb; ck_assert_int_eq(alloc_copy_mb(&mb, buf, 86, 512), OP_ERROR); char buf2[32]; ck_assert_int_eq(alloc_copy_mb(&mb, buf2, size, 32), OP_ERROR); ck_assert_int_eq(alloc_copy_mb(&mb, buf, size, 512), OP_SUCCESS); free(mb); } END_TEST START_TEST(test_alloc_copy_proc_pids) { struct dgsh_conc c; c.n_proc_pids = 2; const int size = sizeof(int) * c.n_proc_pids; int pids[2] = {101, 103}; char buf[512]; memcpy(buf, pids, size); ck_assert_int_eq(alloc_copy_proc_pids(&c, buf, 86, 512), OP_ERROR); char buf2[8]; ck_assert_int_eq(alloc_copy_proc_pids(&c, buf2, size, 4), OP_ERROR); ck_assert_int_eq(alloc_copy_proc_pids(&c, buf, size, 512), OP_SUCCESS); } END_TEST START_TEST(test_alloc_copy_concs) { const int n_concs = fresh_mb->n_concs = 1; const int size = sizeof(struct dgsh_conc) * n_concs; struct dgsh_conc c; char buf[512]; memcpy(buf, &c, size); ck_assert_int_eq(alloc_copy_concs(fresh_mb, buf, 86, 512), OP_ERROR); char buf2[8]; ck_assert_int_eq(alloc_copy_concs(fresh_mb, buf2, size, 8), OP_ERROR); ck_assert_int_eq(alloc_copy_concs(fresh_mb, buf, size, 512), OP_SUCCESS); } END_TEST START_TEST(test_alloc_copy_nodes) { const int size = sizeof(struct dgsh_node) * fresh_mb->n_nodes; char buf[512]; ck_assert_int_eq(alloc_copy_nodes(fresh_mb, buf, 86, 512), OP_ERROR); char buf2[32]; ck_assert_int_eq(alloc_copy_nodes(fresh_mb, buf2, size, 32), OP_ERROR); free(fresh_mb->node_array); /* to avoid memory leak */ ck_assert_int_eq(alloc_copy_nodes(fresh_mb, buf, size, 512), OP_SUCCESS); } END_TEST START_TEST(test_alloc_copy_edges) { const int size = sizeof(struct dgsh_edge) * fresh_mb->n_edges; char buf[256]; ck_assert_int_eq(alloc_copy_edges(fresh_mb, buf, 86, 256), OP_ERROR); char buf2[32]; ck_assert_int_eq(alloc_copy_edges(fresh_mb, buf2, size, 32), OP_ERROR); free(fresh_mb->edge_array); /* to avoid memory leak */ ck_assert_int_eq(alloc_copy_edges(fresh_mb, buf, size, 256), OP_SUCCESS); } END_TEST START_TEST(test_alloc_copy_graph_solution) { const int size = sizeof(struct dgsh_node_connections) * fresh_mb->n_nodes; char buf[256]; ck_assert_int_eq(alloc_copy_graph_solution(fresh_mb, buf, 86, 256), OP_ERROR); char buf2[32]; ck_assert_int_eq(alloc_copy_edges(fresh_mb, buf2, size, 32), OP_ERROR); /* Free to avoid memory leak. * About the free() see how graph_solution is malloc'ed * at alloc_copy_graph_solution. */ free(fresh_mb->graph_solution); ck_assert_int_eq(alloc_copy_graph_solution(fresh_mb, buf, size, 256), OP_SUCCESS); free(fresh_mb->graph_solution); /* Easier to handle here. */ } END_TEST START_TEST(test_check_read) { ck_assert_int_eq(check_read(512, 1024, 256), OP_ERROR); ck_assert_int_eq(check_read(512, 256, 512), OP_ERROR); ck_assert_int_eq(check_read(512, 1024, 512), OP_SUCCESS); } END_TEST /* *START_TEST(test_point_io_direction) { ck_assert_int_eq(point_io_direction(STDOUT_FILENO), STDIN_FILENO); memcpy(&self_node, &chosen_mb->node_array[2], sizeof(struct dgsh_node)); ck_assert_int_eq(point_io_direction(STDIN_FILENO), STDOUT_FILENO); } END_TEST */ START_TEST(test_analyse_read) { DPRINTF(4, "%s()", __func__); /* error state flag seen; terminal process such as node 3 * which is the current node leave. */ bool should_transmit_mb = false; int serialno_ntimes_same = 0; int run_ntimes_same = 0; int error_ntimes_same = 0; int draw_exit_ntimes_same = 0; fresh_mb->state = PS_ERROR; fresh_mb->is_error_confirmed = true; ck_assert_int_eq(analyse_read(fresh_mb, &run_ntimes_same, &error_ntimes_same, &draw_exit_ntimes_same, self_node.name, self_node.pid, &self_node.requires_channels, &self_node.provides_channels), OP_SUCCESS); ck_assert_int_eq(chosen_mb->state, PS_ERROR); ck_assert_int_eq(error_ntimes_same, 1); ck_assert_int_eq((long int)chosen_mb, (long int)fresh_mb); retire_test_analyse_read(); setup_test_analyse_read(); /* error state flag seen; non-terminal process such as node 1 * which is the current node leave when they see it the second time. * This is the first. */ memcpy(&self_node, &chosen_mb->node_array[1], sizeof(struct dgsh_node)); error_ntimes_same = 0; fresh_mb->state = PS_ERROR; fresh_mb->is_error_confirmed = true; ck_assert_int_eq(analyse_read(fresh_mb, &run_ntimes_same, &error_ntimes_same, &draw_exit_ntimes_same, self_node.name, self_node.pid, &self_node.requires_channels, &self_node.provides_channels), OP_SUCCESS); ck_assert_int_eq(chosen_mb->state, PS_ERROR); ck_assert_int_eq(error_ntimes_same, 1); ck_assert_int_eq((long int)chosen_mb, (long int)fresh_mb); retire_test_analyse_read(); setup_test_analyse_read(); /* error state flag seen; non-terminal process such as node 1 * which is the current node have to leave when they see it * the second time. This is the second. * Before leaving they have to pass the block. */ memcpy(&self_node, &chosen_mb->node_array[1], sizeof(struct dgsh_node)); error_ntimes_same = 1; fresh_mb->state = PS_ERROR; fresh_mb->is_error_confirmed = true; ck_assert_int_eq(analyse_read(fresh_mb, &run_ntimes_same, &error_ntimes_same, &draw_exit_ntimes_same, self_node.name, self_node.pid, &self_node.requires_channels, &self_node.provides_channels), OP_SUCCESS); ck_assert_int_eq(chosen_mb->state, PS_ERROR); ck_assert_int_eq(error_ntimes_same, 2); ck_assert_int_eq((long int)chosen_mb, (long int)fresh_mb); retire_test_analyse_read(); setup_test_analyse_read(); /* All processes have to pass the block first except * for the ones who passed the block the last time * before finding a solution. */ error_ntimes_same = 1; memcpy(&self_node, &chosen_mb->node_array[1], sizeof(struct dgsh_node)); fresh_mb->state = PS_ERROR; fresh_mb->is_error_confirmed = true; ck_assert_int_eq(analyse_read(fresh_mb, &run_ntimes_same, &error_ntimes_same, &draw_exit_ntimes_same, self_node.name, self_node.pid, &self_node.requires_channels, &self_node.provides_channels), OP_SUCCESS); ck_assert_int_eq(chosen_mb->state, PS_ERROR); ck_assert_int_eq(error_ntimes_same, 2); ck_assert_int_eq((long int)chosen_mb, (long int)fresh_mb); retire_test_analyse_read(); setup_test_analyse_read(); /* run state flag seen; terminal process such as node 3 * which is the current node leave. */ run_ntimes_same = 0; error_ntimes_same = 0; fresh_mb->state = PS_RUN; ck_assert_int_eq(analyse_read(fresh_mb, &run_ntimes_same, &error_ntimes_same, &draw_exit_ntimes_same, self_node.name, self_node.pid, &self_node.requires_channels, &self_node.provides_channels), OP_SUCCESS); ck_assert_int_eq(chosen_mb->state, PS_RUN); ck_assert_int_eq(run_ntimes_same, 1); ck_assert_int_eq((long int)chosen_mb, (long int)fresh_mb); retire_test_analyse_read(); setup_test_analyse_read(); /* run state flag seen; non-terminal process such as node 1 * which is the current node leave when they see it the second time. * This is the first. */ memcpy(&self_node, &chosen_mb->node_array[1], sizeof(struct dgsh_node)); run_ntimes_same = 0; fresh_mb->state = PS_RUN; ck_assert_int_eq(analyse_read(fresh_mb, &run_ntimes_same, &error_ntimes_same, &draw_exit_ntimes_same, self_node.name, self_node.pid, &self_node.requires_channels, &self_node.provides_channels), OP_SUCCESS); ck_assert_int_eq(chosen_mb->state, PS_RUN); ck_assert_int_eq(run_ntimes_same, 1); ck_assert_int_eq((long int)chosen_mb, (long int)fresh_mb); retire_test_analyse_read(); setup_test_analyse_read(); /* run state flag seen; non-terminal process such as node 1 * which is the current node have to leave when they see it * the second time. This is the second. * Before leaving they have to pass the block. */ memcpy(&self_node, &chosen_mb->node_array[1], sizeof(struct dgsh_node)); run_ntimes_same = 1; fresh_mb->state = PS_RUN; ck_assert_int_eq(analyse_read(fresh_mb, &run_ntimes_same, &error_ntimes_same, &draw_exit_ntimes_same, self_node.name, self_node.pid, &self_node.requires_channels, &self_node.provides_channels), OP_SUCCESS); ck_assert_int_eq(chosen_mb->state, PS_RUN); ck_assert_int_eq(run_ntimes_same, 2); ck_assert_int_eq((long int)chosen_mb, (long int)fresh_mb); retire_test_analyse_read(); setup_test_analyse_read(); /* When they are to leave they pass the block first except * if they are the ones who passed the block the last time * before finding a solution. */ memcpy(&self_node, &chosen_mb->node_array[1], sizeof(struct dgsh_node)); should_transmit_mb = false; fresh_mb->state = PS_RUN; run_ntimes_same = 1; ck_assert_int_eq(analyse_read(fresh_mb, &run_ntimes_same, &error_ntimes_same, &draw_exit_ntimes_same, self_node.name, self_node.pid, &self_node.requires_channels, &self_node.provides_channels), OP_SUCCESS); ck_assert_int_eq(chosen_mb->state, PS_RUN); ck_assert_int_eq(run_ntimes_same, 2); ck_assert_int_eq((long int)chosen_mb, (long int)fresh_mb); retire_test_analyse_read(); /* Negotiation state */ setup_test_analyse_read(); run_ntimes_same = 0; error_ntimes_same = 0; /* set_dispatcher() */ self_node_io_side.index = 3; self_node_io_side.fd_direction = STDIN_FILENO; fresh_mb->initiator_pid = 110; /* Younger than chosen_mb. */ ck_assert_int_eq(analyse_read(fresh_mb, &run_ntimes_same, &error_ntimes_same, &draw_exit_ntimes_same, self_node.name, self_node.pid, &self_node.requires_channels, &self_node.provides_channels), OP_SUCCESS); retire_test_analyse_read(); setup_test_analyse_read(); should_transmit_mb = false; memcpy(&self_node, &chosen_mb->node_array[3], sizeof(struct dgsh_node)); /* set_dispatcher() */ self_node_io_side.index = 3; self_node_io_side.fd_direction = STDIN_FILENO; fresh_mb->initiator_pid = 103; /* Same initiator */ ck_assert_int_eq(analyse_read(fresh_mb, &run_ntimes_same, &error_ntimes_same, &draw_exit_ntimes_same, self_node.name, self_node.pid, &self_node.requires_channels, &self_node.provides_channels), OP_SUCCESS); ck_assert_int_eq((long int)chosen_mb, (long int)fresh_mb); retire_test_analyse_read(); setup_test_analyse_read(); memcpy(&self_node, &chosen_mb->node_array[0], sizeof(struct dgsh_node)); /* set_dispatcher() */ self_node_io_side.index = 0; self_node_io_side.fd_direction = STDOUT_FILENO; chosen_mb->origin_index = 3; chosen_mb->origin_fd_direction = STDIN_FILENO; fresh_mb->initiator_pid = 103; /* Same initiator */ ck_assert_int_eq(analyse_read(fresh_mb, &run_ntimes_same, &error_ntimes_same, &draw_exit_ntimes_same, self_node.name, self_node.pid, &self_node.requires_channels, &self_node.provides_channels), OP_SUCCESS); } END_TEST START_TEST(test_free_mb) { free_mb(chosen_mb); } END_TEST START_TEST(test_fill_node) { /* self node is node at index 3 of chosen_mb */ fill_node("test", 1003, NULL, NULL); ck_assert_int_eq(strcmp(self_node.name, "test"), 0); ck_assert_int_eq(self_node.pid, 1003); ck_assert_int_eq(self_node.requires_channels, 1); ck_assert_int_eq(self_node.provides_channels, 0); int n_input_fds = 1; int n_output_fds = 1; fill_node("test", 1003, &n_input_fds, &n_output_fds); ck_assert_int_eq(self_node.requires_channels, 1); ck_assert_int_eq(self_node.provides_channels, 1); } END_TEST START_TEST(test_try_add_dgsh_node) { int n_input_fds = 1; int n_output_fds = 1; ck_assert_int_eq(try_add_dgsh_node("proc3", 103, &n_input_fds, &n_output_fds), OP_EXISTS); ck_assert_int_eq(chosen_mb->n_nodes, 4); ck_assert_int_eq(self_node_io_side.index, 0); ck_assert_int_eq(self_node.index, 3); ck_assert_int_eq(try_add_dgsh_node("proc4", 104, &n_input_fds, &n_output_fds), OP_SUCCESS); ck_assert_int_eq(chosen_mb->n_nodes, 5); ck_assert_int_eq(self_node_io_side.index, 4); ck_assert_int_eq(self_node.index, 4); } END_TEST START_TEST(test_try_add_dgsh_edge) { /* Better in a setup function. */ chosen_mb->origin_fd_direction = STDOUT_FILENO; chosen_mb->origin_index = 0; /* self_node_io_side should also be set; it is set in setup */ ck_assert_int_eq(try_add_dgsh_edge(), OP_EXISTS); /* New edge: from new node to existing */ struct dgsh_node new; new.index = 4; new.pid = 104; memcpy(new.name, "proc4", 6); new.requires_channels = 1; new.provides_channels = 1; new.dgsh_in = 1; new.dgsh_out = 1; /* Better in a setup function. */ chosen_mb->origin_fd_direction = STDOUT_FILENO; chosen_mb->origin_index = new.index; /* self_node_io_side should also be set; it is set in setup */ memcpy(&self_node, &new, sizeof(struct dgsh_node)); chosen_mb->n_nodes++; chosen_mb->node_array = realloc(chosen_mb->node_array, sizeof(struct dgsh_node) * chosen_mb->n_nodes); memcpy(&chosen_mb->node_array[chosen_mb->n_nodes - 1], &new, sizeof(struct dgsh_node)); ck_assert_int_eq(try_add_dgsh_edge(), OP_SUCCESS); /* New edge: from existing to new node */ /* Better in a setup function. */ chosen_mb->origin_fd_direction = STDOUT_FILENO; chosen_mb->origin_index = 0; /* self_node_io_side should also be set; it is set in setup */ self_node_io_side.index = new.index; self_node_io_side.fd_direction = STDIN_FILENO; ck_assert_int_eq(try_add_dgsh_edge(), OP_SUCCESS); /* NOOP: message block created just now */ chosen_mb->origin_index = -1; ck_assert_int_eq(try_add_dgsh_edge(), OP_NOOP); } END_TEST START_TEST(test_add_edge) { struct dgsh_edge new; new.from = 2; new.to = 3; new.instances = 0; ck_assert_int_eq(add_edge(&new), OP_SUCCESS); ck_assert_int_eq(chosen_mb->n_edges, 6); } END_TEST START_TEST(test_fill_dgsh_edge) { struct dgsh_edge new; /* STDIN -> STDOUT */ /* Better in a setup function. */ chosen_mb->origin_fd_direction = STDOUT_FILENO; chosen_mb->origin_index = 0; /* self_node_io_side should also be set; it is set in setup */ ck_assert_int_eq(fill_dgsh_edge(&new), OP_SUCCESS); /* Impossible case. No such origin. */ chosen_mb->origin_index = 7; ck_assert_int_eq(fill_dgsh_edge(&new), OP_ERROR); /* STDOUT -> STDIN */ chosen_mb->origin_fd_direction = STDIN_FILENO; chosen_mb->origin_index = 3; memcpy(&self_node, &chosen_mb->node_array[0], sizeof(struct dgsh_node)); self_node_io_side.fd_direction = STDOUT_FILENO; self_node_io_side.index = 0; /* self_node_io_side should also be set; it is set in setup */ ck_assert_int_eq(fill_dgsh_edge(&new), OP_SUCCESS); } END_TEST START_TEST(test_lookup_dgsh_edge) { struct dgsh_edge new; new.from = 2; new.to = 3; ck_assert_int_eq(lookup_dgsh_edge(&new), OP_CREATE); ck_assert_int_eq(lookup_dgsh_edge(&chosen_mb->edge_array[4]), OP_EXISTS); } END_TEST START_TEST(test_add_node) { struct dgsh_node new; new.pid = 104; memcpy(new.name, "proc4", 6); new.requires_channels = 1; new.provides_channels = 1; memcpy(&self_node, &new, sizeof(struct dgsh_node)); ck_assert_int_eq(add_node(), OP_SUCCESS); ck_assert_int_eq(chosen_mb->n_nodes, 5); ck_assert_int_eq(self_node_io_side.index, 4); ck_assert_int_eq(self_node.index, 4); } END_TEST START_TEST(test_construct_message_block) { DPRINTF(4, "%s()", __func__); int pid = 7; const char tool_name[10] = "test"; ck_assert_int_eq(construct_message_block(tool_name, pid), OP_SUCCESS); ck_assert_int_eq(chosen_mb->version, 1); ck_assert_int_eq((long)chosen_mb->node_array, 0); ck_assert_int_eq(chosen_mb->n_nodes, 0); ck_assert_int_eq(chosen_mb->initiator_pid, pid); ck_assert_int_eq(chosen_mb->state, PS_NEGOTIATION); ck_assert_int_eq(chosen_mb->origin_index, -1); ck_assert_int_eq(chosen_mb->origin_fd_direction, -1); free(chosen_mb); } END_TEST START_TEST(test_get_env_var) { DPRINTF(4, "%s()...", __func__); int value = -1; putenv("DGSH_IN=0"); get_env_var("DGSH_IN", &value); ck_assert_int_eq(value, 0); value = -1; putenv("DGSH_OUT=1"); get_env_var("DGSH_OUT", &value); ck_assert_int_eq(value, 1); } END_TEST START_TEST(test_get_environment_vars) { DPRINTF(4, "%s()...", __func__); putenv("DGSH_IN=0"); putenv("DGSH_OUT=1"); get_environment_vars(); ck_assert_int_eq(self_node.dgsh_in, 0); ck_assert_int_eq(self_node.dgsh_out, 1); } END_TEST START_TEST(test_validate_input) { int i = 0; int o = 0; ck_assert_int_eq(validate_input(&i, &o, NULL), OP_ERROR); ck_assert_int_eq(validate_input(NULL, &o, "test"), OP_SUCCESS); ck_assert_int_eq(validate_input(&i, NULL, "test"), OP_SUCCESS); ck_assert_int_eq(validate_input(NULL, NULL, "test"), OP_SUCCESS); ck_assert_int_eq(validate_input(&i, &o, "test"), OP_SUCCESS); i = 0; o = 1; ck_assert_int_eq(validate_input(&i, &o, "test"), OP_SUCCESS); i = 1; o = 0; ck_assert_int_eq(validate_input(&i, &o, "test"), OP_SUCCESS); i = -1; o = -1; ck_assert_int_eq(validate_input(&i, &o, "test"), OP_SUCCESS); i = -2; o = -1; ck_assert_int_eq(validate_input(&i, &o, "test"), OP_ERROR); i = -1; o = -2; ck_assert_int_eq(validate_input(&i, &o, "test"), OP_ERROR); i = 1000; o = 1000; ck_assert_int_eq(validate_input(&i, &o, "test"), OP_SUCCESS); } END_TEST START_TEST(test_set_fds) { /* For node 3 which is a terminal node */ fd_set read_fds, write_fds; ck_assert_int_eq(set_fds(&read_fds, &write_fds, 0), 2); ck_assert_int_eq(self_node_io_side.fd_direction, STDIN_FILENO); ck_assert_int_eq(set_fds(&read_fds, &write_fds, 1), 2); ck_assert_int_eq(self_node_io_side.fd_direction, STDIN_FILENO); /* Make node 1 self node, which is a non terminal node */ memcpy(&self_node, &chosen_mb->node_array[1], sizeof(struct dgsh_node)); ck_assert_int_eq(set_fds(&read_fds, &write_fds, 0), 2); ck_assert_int_eq(self_node_io_side.fd_direction, STDOUT_FILENO); ck_assert_int_eq(set_fds(&read_fds, &write_fds, 1), 2); } END_TEST START_TEST(test_dgsh_negotiate) { int *input_fds; int n_input_fds = 0; int *output_fds; int n_output_fds = 0; ck_assert_int_eq(dgsh_negotiate(0, "test", &n_input_fds, &n_output_fds, &input_fds, &output_fds), 0); } END_TEST /* Suite conc */ START_TEST(test_is_ready) { chosen_mb->state = PS_RUN; ck_assert_int_eq(is_ready(3, chosen_mb), true); ck_assert_int_eq(is_ready(1, chosen_mb), false); ck_assert_int_eq(is_ready(0, chosen_mb), false); ck_assert_int_eq(pi[0].seen, false); ck_assert_int_eq(pi[1].written, false); ck_assert_int_eq(pi[1].run_ready, false); } END_TEST START_TEST (test_next_fd) { multiple_inputs = true; nfd = 5; bool ro = false; /* restore origin */ ck_assert_int_eq(next_fd(0, &ro), 1); ck_assert_int_eq(ro, false); ck_assert_int_eq(next_fd(1, &ro), 0); ck_assert_int_eq(ro, false); ck_assert_int_eq(next_fd(4, &ro), 4); ck_assert_int_eq(ro, true); ro = false; ck_assert_int_eq(next_fd(3, &ro), 3); ck_assert_int_eq(ro, true); multiple_inputs = false; noinput = false; ro = false; ck_assert_int_eq(next_fd(0, &ro), 1); ck_assert_int_eq(ro, false); ck_assert_int_eq(next_fd(1, &ro), 3); ck_assert_int_eq(ro, true); ro = false; ck_assert_int_eq(next_fd(3, &ro), 4); ck_assert_int_eq(ro, true); ro = false; ck_assert_int_eq(next_fd(4, &ro), 0); ck_assert_int_eq(ro, false); noinput = true; ro = false; ck_assert_int_eq(next_fd(1, &ro), 3); ck_assert_int_eq(ro, false); ck_assert_int_eq(next_fd(3, &ro), 4); ck_assert_int_eq(ro, false); ck_assert_int_eq(next_fd(4, &ro), 1); ck_assert_int_eq(ro, false); } END_TEST START_TEST(test_set_io_channels) { pid = 2000; /* static in dgsh-conc.c */ nfd = 4; /* ditto */ multiple_inputs = false; /* ditto */ ck_assert_int_eq(set_io_channels(chosen_mb), 0); ck_assert_int_eq(chosen_mb->n_concs, 1); ck_assert_int_eq(chosen_mb->conc_array[0].pid, 2000); ck_assert_int_eq(chosen_mb->conc_array[0].input_fds, -1); ck_assert_int_eq(chosen_mb->conc_array[0].output_fds, -1); ck_assert_int_eq(chosen_mb->conc_array[0].multiple_inputs, false); ck_assert_int_eq(chosen_mb->conc_array[0].n_proc_pids, 2); ck_assert_int_eq(chosen_mb->conc_array[0].proc_pids[0], 100); ck_assert_int_eq(chosen_mb->conc_array[0].proc_pids[1], 103); /* Exists with channels set: keep as it is */ ck_assert_int_eq(set_io_channels(chosen_mb), 0); ck_assert_int_eq(chosen_mb->n_concs, 1); ck_assert_int_eq(chosen_mb->conc_array[0].pid, 2000); ck_assert_int_eq(chosen_mb->conc_array[0].input_fds, -1); ck_assert_int_eq(chosen_mb->conc_array[0].output_fds, -1); /* Not exists: set channels (same pi, same channels as before */ pid = 2001; /* static in dgsh-conc.c */ multiple_inputs = true; ck_assert_int_eq(set_io_channels(chosen_mb), 0); ck_assert_int_eq(chosen_mb->n_concs, 2); DPRINTF(4, "%d", chosen_mb->conc_array[0].pid); ck_assert_int_eq(chosen_mb->conc_array[1].pid, 2001); ck_assert_int_eq(chosen_mb->conc_array[1].input_fds, -1); ck_assert_int_eq(chosen_mb->conc_array[1].output_fds, -1); ck_assert_int_eq(chosen_mb->conc_array[1].multiple_inputs, true); ck_assert_int_eq(chosen_mb->conc_array[1].n_proc_pids, 2); ck_assert_int_eq(chosen_mb->conc_array[1].proc_pids[0], 101); ck_assert_int_eq(chosen_mb->conc_array[1].proc_pids[1], 103); } END_TEST Suite * suite_connect(void) { Suite *s = suite_create("Connect"); TCase *tc_aiof = tcase_create("alloc io fds"); tcase_add_checked_fixture(tc_aiof, setup_test_alloc_io_fds, retire_test_alloc_io_fds); tcase_add_test(tc_aiof, test_alloc_io_fds); suite_add_tcase(s, tc_aiof); TCase *tc_af = tcase_create("alloc fds"); tcase_add_checked_fixture(tc_af, NULL, NULL); tcase_add_test(tc_af, test_alloc_fds); suite_add_tcase(s, tc_af); TCase *tc_trw = tcase_create("test read/write fd"); tcase_add_checked_fixture(tc_trw, NULL, NULL); tcase_add_test(tc_trw, test_read_write_fd); suite_add_tcase(s, tc_trw); TCase *tc_rif = tcase_create("read input fds"); tcase_add_checked_fixture(tc_rif, setup_test_read_input_fds, retire_test_read_input_fds); tcase_add_test(tc_rif, test_read_input_fds); suite_add_tcase(s, tc_rif); TCase *tc_gop = tcase_create("get origin pid"); tcase_add_checked_fixture(tc_gop, setup_test_get_origin_pid, retire_test_get_origin_pid); tcase_add_test(tc_gop, test_get_origin_pid); suite_add_tcase(s, tc_gop); TCase *tc_gefn = tcase_create("get expected fds number"); tcase_add_checked_fixture(tc_gefn, setup_test_get_expected_fds_n, retire_test_get_expected_fds_n); tcase_add_test(tc_gefn, test_get_expected_fds_n); suite_add_tcase(s, tc_gefn); TCase *tc_gpfn = tcase_create("get provided fds number"); tcase_add_checked_fixture(tc_gpfn, setup_test_get_provided_fds_n, retire_test_get_provided_fds_n); tcase_add_test(tc_gpfn, test_get_provided_fds_n); suite_add_tcase(s, tc_gpfn); TCase *tc_eic = tcase_create("establish io connections"); tcase_add_checked_fixture(tc_eic, setup_test_establish_io_connections, retire_test_establish_io_connections); tcase_add_test(tc_eic, test_establish_io_connections); suite_add_tcase(s, tc_eic); TCase *tc_anc = tcase_create("alloc node connections"); tcase_add_checked_fixture(tc_anc, NULL, NULL); tcase_add_test(tc_anc, test_alloc_node_connections); suite_add_tcase(s, tc_anc); TCase *tc_sd = tcase_create("set dispatcher"); tcase_add_checked_fixture(tc_sd, setup_test_set_dispatcher, retire_test_set_dispatcher); tcase_add_test(tc_sd, test_set_dispatcher); suite_add_tcase(s, tc_sd); /* Need to also simulate sendmsg; make sure it works. */ TCase *tc_awof = tcase_create("write output fds"); tcase_add_checked_fixture(tc_awof, setup_test_write_output_fds, retire_test_write_output_fds); tcase_add_test(tc_awof, test_write_output_fds); suite_add_tcase(s, tc_awof); return s; } Suite * suite_solve(void) { Suite *s = suite_create("Solve"); TCase *tc_wc = tcase_create("write concs"); tcase_add_checked_fixture(tc_wc, setup_test_write_concs, retire_test_write_concs); tcase_add_test(tc_wc, test_write_concs); suite_add_tcase(s, tc_wc); TCase *tc_rc = tcase_create("read concs"); tcase_add_checked_fixture(tc_rc, setup_test_read_concs, retire_test_read_concs); tcase_add_test(tc_rc, test_read_concs); suite_add_tcase(s, tc_rc); TCase *tc_rgs = tcase_create("read graph solution"); tcase_add_checked_fixture(tc_rgs, setup_test_read_graph_solution, retire_test_read_graph_solution); tcase_add_test(tc_rgs, test_read_graph_solution); suite_add_tcase(s, tc_rgs); TCase *tc_wgs = tcase_create("write graph solution"); tcase_add_checked_fixture(tc_wgs, setup_test_write_graph_solution, retire_test_write_graph_solution); tcase_add_test(tc_wgs, test_write_graph_solution); suite_add_tcase(s, tc_wgs); TCase *tc_ssg = tcase_create("solve dgsh graph"); tcase_add_checked_fixture(tc_ssg, setup_test_solve_graph, retire_test_solve_graph); tcase_add_test(tc_ssg, test_solve_graph); suite_add_tcase(s, tc_ssg); TCase *tc_ccf = tcase_create("calculate conc fds"); tcase_add_checked_fixture(tc_ccf, setup_test_calculate_conc_fds, retire_test_calculate_conc_fds); tcase_add_test(tc_ccf, test_calculate_conc_fds); suite_add_tcase(s, tc_ccf); TCase *tc_fgs = tcase_create("free graph solution"); tcase_add_checked_fixture(tc_fgs, setup_test_free_graph_solution, retire_test_free_graph_solution); tcase_add_test(tc_fgs, test_free_graph_solution); suite_add_tcase(s, tc_fgs); TCase *tc_nmc = tcase_create("node match constraints"); tcase_add_checked_fixture(tc_nmc, setup_test_node_match_constraints, retire_test_node_match_constraints); tcase_add_test(tc_nmc, test_node_match_constraints); suite_add_tcase(s, tc_nmc); TCase *tc_dmic = tcase_create("dry match io constraints"); tcase_add_checked_fixture(tc_dmic, setup_test_dry_match_io_constraints, retire_test_dry_match_io_constraints); tcase_add_test(tc_dmic, test_dry_match_io_constraints); suite_add_tcase(s, tc_dmic); TCase *tc_sic = tcase_create("satisfy io constraints"); tcase_add_checked_fixture(tc_sic, setup_test_satisfy_io_constraints, retire_test_satisfy_io_constraints); tcase_add_test(tc_sic, test_satisfy_io_constraints); suite_add_tcase(s, tc_sic); TCase *tc_mov = tcase_create("move"); tcase_add_checked_fixture(tc_mov, setup_test_move, retire_test_move); tcase_add_test(tc_mov, test_move); suite_add_tcase(s, tc_mov); TCase *tc_rmf = tcase_create("record move flexible"); tcase_add_checked_fixture(tc_rmf, NULL, NULL); tcase_add_test(tc_rmf, test_record_move_flexible); suite_add_tcase(s, tc_rmf); TCase *tc_rmu = tcase_create("record move unbalanced"); tcase_add_checked_fixture(tc_rmu, NULL, NULL); tcase_add_test(tc_rmu, test_record_move_unbalanced); suite_add_tcase(s, tc_rmu); /*TCase *tc_ec = tcase_create("evaluate constraints"); tcase_add_checked_fixture(tc_ec, setup_test_eval_constraints, retire_test_eval_constraints); tcase_add_test(tc_ec, test_eval_constraints); suite_add_tcase(s, tc_ec); *TCase *tc_aei = tcase_create("assign edge instances"); tcase_add_checked_fixture(tc_aei, setup_test_assign_edge_instances, retire_test_assign_edge_instances); tcase_add_test(tc_aei, test_assign_edge_instances); suite_add_tcase(s, tc_aei); */ TCase *tc_repa = tcase_create("reallocate edge pointer array"); tcase_add_checked_fixture(tc_repa, setup_test_reallocate_edge_pointer_array, retire_test_reallocate_edge_pointer_array); tcase_add_test(tc_repa, test_reallocate_edge_pointer_array); suite_add_tcase(s, tc_repa); TCase *tc_mcea = tcase_create("make compact edge array"); tcase_add_checked_fixture(tc_mcea, setup_test_make_compact_edge_array, retire_test_make_compact_edge_array); tcase_add_test(tc_mcea, test_make_compact_edge_array); suite_add_tcase(s, tc_mcea); return s; } Suite * suite_broadcast(void) { Suite *s = suite_create("Broadcast"); TCase *tc_wmb = tcase_create("write message block"); tcase_add_checked_fixture(tc_wmb, setup_test_write_message_block, retire_test_write_message_block); tcase_add_test(tc_wmb, test_write_message_block); suite_add_tcase(s, tc_wmb); TCase *tc_trm = tcase_create("read message block"); tcase_add_checked_fixture(tc_trm, setup_test_read_message_block, retire_test_read_message_block); tcase_add_test(tc_trm, test_read_message_block); suite_add_tcase(s, tc_trm); TCase *tc_trc = tcase_create("read chunk"); tcase_add_checked_fixture(tc_trc, setup_test_read_chunk, NULL); tcase_add_test(tc_trc, test_read_chunk); suite_add_tcase(s, tc_trc); TCase *tc_clr = tcase_create("call read"); tcase_add_checked_fixture(tc_clr, NULL, NULL); tcase_add_test(tc_clr, test_call_read); suite_add_tcase(s, tc_clr); TCase *tc_acm = tcase_create("alloc copy message block"); tcase_add_checked_fixture(tc_acm, NULL, NULL); tcase_add_test(tc_acm, test_alloc_copy_mb); suite_add_tcase(s, tc_acm); TCase *tc_acn = tcase_create("alloc copy nodes"); tcase_add_checked_fixture(tc_acn, setup_test_alloc_copy_nodes, retire_test_alloc_copy_nodes); tcase_add_test(tc_acn, test_alloc_copy_nodes); suite_add_tcase(s, tc_acn); TCase *tc_ace = tcase_create("alloc copy edges"); tcase_add_checked_fixture(tc_ace, setup_test_alloc_copy_edges, retire_test_alloc_copy_edges); tcase_add_test(tc_ace, test_alloc_copy_edges); suite_add_tcase(s, tc_ace); TCase *tc_acg = tcase_create("alloc copy graph solution"); tcase_add_checked_fixture(tc_acg, setup_test_alloc_copy_graph_solution, retire_test_alloc_copy_graph_solution); tcase_add_test(tc_acg, test_alloc_copy_graph_solution); suite_add_tcase(s, tc_acg); TCase *tc_acc = tcase_create("alloc copy concs"); tcase_add_checked_fixture(tc_acc, setup_test_alloc_copy_concs, retire_test_alloc_copy_concs); tcase_add_test(tc_acc, test_alloc_copy_concs); suite_add_tcase(s, tc_acc); TCase *tc_acp = tcase_create("alloc copy proc pids"); tcase_add_test(tc_acp, test_alloc_copy_proc_pids); suite_add_tcase(s, tc_acp); TCase *tc_cr = tcase_create("check read"); tcase_add_checked_fixture(tc_cr, NULL, NULL); tcase_add_test(tc_cr, test_check_read); suite_add_tcase(s, tc_cr); /* *TCase *tc_pid = tcase_create("point io direction"); tcase_add_checked_fixture(tc_pid, setup_test_point_io_direction, retire_test_point_io_direction); tcase_add_test(tc_pid, test_point_io_direction); suite_add_tcase(s, tc_pid); */ TCase *tc_ar = tcase_create("analyse read"); tcase_add_checked_fixture(tc_ar, setup_test_analyse_read, retire_test_analyse_read); tcase_add_test(tc_ar, test_analyse_read); suite_add_tcase(s, tc_ar); TCase *tc_fm = tcase_create("free message block"); tcase_add_checked_fixture(tc_fm, setup_test_free_mb, NULL); tcase_add_test(tc_fm, test_free_mb); suite_add_tcase(s, tc_fm); TCase *tc_fsn = tcase_create("fill dgsh node"); tcase_add_checked_fixture(tc_fsn, setup_test_fill_node, NULL); tcase_add_test(tc_fsn, test_fill_node); suite_add_tcase(s, tc_fsn); TCase *tc_tasn = tcase_create("try add dgsh node"); tcase_add_checked_fixture(tc_tasn, setup_test_try_add_dgsh_node, retire_test_try_add_dgsh_node); tcase_add_test(tc_tasn, test_try_add_dgsh_node); suite_add_tcase(s, tc_tasn); TCase *tc_tase = tcase_create("try add dgsh edge"); tcase_add_checked_fixture(tc_tase, setup_test_try_add_dgsh_edge, retire_test_try_add_dgsh_edge); tcase_add_test(tc_tase, test_try_add_dgsh_edge); suite_add_tcase(s, tc_tase); TCase *tc_ae = tcase_create("add edge"); tcase_add_checked_fixture(tc_ae, setup_test_add_edge, retire_test_add_edge); tcase_add_test(tc_ae, test_add_edge); suite_add_tcase(s, tc_ae); TCase *tc_fse = tcase_create("fill dgsh edge"); tcase_add_checked_fixture(tc_fse, setup_test_fill_dgsh_edge, retire_test_fill_dgsh_edge); tcase_add_test(tc_fse, test_fill_dgsh_edge); suite_add_tcase(s, tc_fse); TCase *tc_lse = tcase_create("lookup dgsh edge"); tcase_add_checked_fixture(tc_lse, setup_test_lookup_dgsh_edge, retire_test_lookup_dgsh_edge); tcase_add_test(tc_lse, test_lookup_dgsh_edge); suite_add_tcase(s, tc_lse); TCase *tc_an = tcase_create("add node"); tcase_add_checked_fixture(tc_an, setup_test_add_node, retire_test_add_node); tcase_add_test(tc_an, test_add_node); suite_add_tcase(s, tc_an); TCase *tc_cnmb = tcase_create("construct message block"); tcase_add_checked_fixture(tc_cnmb, NULL, NULL); tcase_add_test(tc_cnmb, test_construct_message_block); suite_add_tcase(s, tc_cnmb); TCase *tc_gev = tcase_create("get environment variable"); tcase_add_checked_fixture(tc_gev, NULL, NULL); tcase_add_test(tc_gev, test_get_env_var); suite_add_tcase(s, tc_gev); TCase *tc_gevs = tcase_create("get environment variables"); tcase_add_checked_fixture(tc_gevs, NULL, NULL); tcase_add_test(tc_gevs, test_get_environment_vars); suite_add_tcase(s, tc_gevs); TCase *tc_vi = tcase_create("validate input"); tcase_add_checked_fixture(tc_vi, NULL, NULL); tcase_add_test(tc_vi, test_validate_input); suite_add_tcase(s, tc_vi); TCase *tc_sf = tcase_create("set fds"); tcase_add_checked_fixture(tc_sf, setup_test_set_fds, retire_test_set_fds); tcase_add_test(tc_sf, test_set_fds); suite_add_tcase(s, tc_sf); TCase *tc_sn = tcase_create("dgsh negotiate"); tcase_add_checked_fixture(tc_sn, setup, retire); tcase_add_test(tc_sn, test_dgsh_negotiate); suite_add_tcase(s, tc_sn); return s; } Suite * suite_conc(void) { Suite *s = suite_create("Concentrator"); TCase *tc_tn = tcase_create("test next_fd"); TCase *tc_ir = tcase_create("test is_ready"); TCase *tc_si = tcase_create("set io"); TCase *tc_sich = tcase_create("set io channels"); tcase_add_checked_fixture(tc_tn, NULL, NULL); tcase_add_test(tc_tn, test_next_fd); suite_add_tcase(s, tc_tn); tcase_add_checked_fixture(tc_ir, setup_test_is_ready, retire_test_is_ready); tcase_add_test(tc_ir, test_is_ready); suite_add_tcase(s, tc_ir); tcase_add_checked_fixture(tc_sich, setup_test_set_io_channels, retire_test_set_io_channels); tcase_add_test(tc_sich, test_set_io_channels); suite_add_tcase(s, tc_sich); return s; } int run_suite(Suite *s) { int number_failed; SRunner *sr = srunner_create(s); srunner_run_all(sr, CK_VERBOSE); number_failed = srunner_ntests_failed(sr); srunner_free(sr); return (number_failed == 0) ? 1 : 0; } int run_suite_connect(void) { Suite *s = suite_connect(); return run_suite(s); } int run_suite_solve(void) { Suite *s = suite_solve(); return run_suite(s); } int run_suite_broadcast(void) { Suite *s = suite_broadcast(); return run_suite(s); } int run_suite_conc(void) { Suite *s = suite_conc(); return run_suite(s); } /* Output is not appropriate; only pass fail. */ int main() { int failed_neg, failed_sol, failed_conn, failed_conc; failed_neg = run_suite_broadcast(); failed_sol = run_suite_solve(); failed_conn = run_suite_connect(); failed_conc = run_suite_conc(); return (failed_neg && failed_sol && failed_conn && failed_conc) ? EXIT_SUCCESS : EXIT_FAILURE; } ================================================ FILE: core-tools/tests-regression/.gitignore ================================================ test/ ================================================ FILE: core-tools/tests-regression/author-compare/out.ok ================================================ conf/icse/ papers: 219 journals/software/ papers: 182 Authors only in conf/icse/: 547 Authors only in journals/software/: 299 Authors common in both venues: 22 ================================================ FILE: core-tools/tests-regression/bin/gdate ================================================ #!/bin/sh # # Poor man's GNU date # This implements the GNU date -f - option for converting the date as # represented in the web log into the week day using the FreeBSD # date command # if [ x"$*" != x'-f - +%a' ] then echo "(poor man's) gdate usage: gdate -f - +%a" 1>&2 exit 1 fi while read d do date -j -f '%d-%b-%Y' "$d" +%a done ================================================ FILE: core-tools/tests-regression/code-metrics/in/date/date.c ================================================ /*- * Copyright (c) 1985, 1987, 1988, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #ifndef lint static char const copyright[] = "@(#) Copyright (c) 1985, 1987, 1988, 1993\n\ The Regents of the University of California. All rights reserved.\n"; #endif /* not lint */ #if 0 #ifndef lint static char sccsid[] = "@(#)date.c 8.2 (Berkeley) 4/28/95"; #endif /* not lint */ #endif #include __FBSDID("$FreeBSD: src/bin/date/date.c,v 1.48.2.1.8.2 2012/11/17 08:24:31 svnexp Exp $"); #include #include #include #include #include #include #include #include #include #include #include #include "extern.h" #include "vary.h" #ifndef TM_YEAR_BASE #define TM_YEAR_BASE 1900 #endif static time_t tval; int retval; static void setthetime(const char *, const char *, int, int); static void badformat(void); static void usage(void); int main(int argc, char *argv[]) { struct timezone tz; int ch, rflag; int jflag, nflag; const char *format; char buf[1024]; char *endptr, *fmt; char *tmp; int set_timezone; struct vary *v; const struct vary *badv; struct tm lt; v = NULL; fmt = NULL; (void) setlocale(LC_TIME, ""); tz.tz_dsttime = tz.tz_minuteswest = 0; rflag = 0; jflag = nflag = 0; set_timezone = 0; while ((ch = getopt(argc, argv, "d:f:jnr:t:uv:")) != -1) switch((char)ch) { case 'd': /* daylight savings time */ tz.tz_dsttime = strtol(optarg, &endptr, 10) ? 1 : 0; if (endptr == optarg || *endptr != '\0') usage(); set_timezone = 1; break; case 'f': fmt = optarg; break; case 'j': jflag = 1; /* don't set time */ break; case 'n': /* don't set network */ nflag = 1; break; case 'r': /* user specified seconds */ rflag = 1; tval = strtoq(optarg, &tmp, 0); if (*tmp != 0) usage(); break; case 't': /* minutes west of UTC */ /* error check; don't allow "PST" */ tz.tz_minuteswest = strtol(optarg, &endptr, 10); if (endptr == optarg || *endptr != '\0') usage(); set_timezone = 1; break; case 'u': /* do everything in UTC */ (void)setenv("TZ", "UTC0", 1); break; case 'v': v = vary_append(v, optarg); break; default: usage(); } argc -= optind; argv += optind; /* * If -d or -t, set the timezone or daylight savings time; this * doesn't belong here; the kernel should not know about either. */ if (set_timezone && settimeofday((struct timeval *)NULL, &tz)) err(1, "settimeofday (timezone)"); if (!rflag && time(&tval) == -1) err(1, "time"); format = "%+"; /* allow the operands in any order */ if (*argv && **argv == '+') { format = *argv + 1; ++argv; } if (*argv) { setthetime(fmt, *argv, jflag, nflag); ++argv; } else if (fmt != NULL) usage(); if (*argv && **argv == '+') format = *argv + 1; lt = *localtime(&tval); badv = vary_apply(v, <); if (badv) { fprintf(stderr, "%s: Cannot apply date adjustment\n", badv->arg); vary_destroy(v); usage(); } vary_destroy(v); (void)strftime(buf, sizeof(buf), format, <); (void)printf("%s\n", buf); if (fflush(stdout)) err(1, "stdout"); exit(retval); } #define ATOI2(s) ((s) += 2, ((s)[-2] - '0') * 10 + ((s)[-1] - '0')) static void setthetime(const char *fmt, const char *p, int jflag, int nflag) { struct tm *lt; struct timeval tv; const char *dot, *t; int century; lt = localtime(&tval); lt->tm_isdst = -1; /* divine correct DST */ if (fmt != NULL) { t = strptime(p, fmt, lt); if (t == NULL) { fprintf(stderr, "Failed conversion of ``%s''" " using format ``%s''\n", p, fmt); badformat(); } else if (*t != '\0') fprintf(stderr, "Warning: Ignoring %ld extraneous" " characters in date string (%s)\n", (long) strlen(t), t); } else { for (t = p, dot = NULL; *t; ++t) { if (isdigit(*t)) continue; if (*t == '.' && dot == NULL) { dot = t; continue; } badformat(); } if (dot != NULL) { /* .ss */ dot++; /* *dot++ = '\0'; */ if (strlen(dot) != 2) badformat(); lt->tm_sec = ATOI2(dot); if (lt->tm_sec > 61) badformat(); } else lt->tm_sec = 0; century = 0; /* if p has a ".ss" field then let's pretend it's not there */ switch (strlen(p) - ((dot != NULL) ? 3 : 0)) { case 12: /* cc */ lt->tm_year = ATOI2(p) * 100 - TM_YEAR_BASE; century = 1; /* FALLTHROUGH */ case 10: /* yy */ if (century) lt->tm_year += ATOI2(p); else { lt->tm_year = ATOI2(p); if (lt->tm_year < 69) /* hack for 2000 ;-} */ lt->tm_year += 2000 - TM_YEAR_BASE; else lt->tm_year += 1900 - TM_YEAR_BASE; } /* FALLTHROUGH */ case 8: /* mm */ lt->tm_mon = ATOI2(p); if (lt->tm_mon > 12) badformat(); --lt->tm_mon; /* time struct is 0 - 11 */ /* FALLTHROUGH */ case 6: /* dd */ lt->tm_mday = ATOI2(p); if (lt->tm_mday > 31) badformat(); /* FALLTHROUGH */ case 4: /* HH */ lt->tm_hour = ATOI2(p); if (lt->tm_hour > 23) badformat(); /* FALLTHROUGH */ case 2: /* MM */ lt->tm_min = ATOI2(p); if (lt->tm_min > 59) badformat(); break; default: badformat(); } } /* convert broken-down time to GMT clock time */ if ((tval = mktime(lt)) == -1) errx(1, "nonexistent time"); if (!jflag) { /* set the time */ if (nflag || netsettime(tval)) { logwtmp("|", "date", ""); tv.tv_sec = tval; tv.tv_usec = 0; if (settimeofday(&tv, (struct timezone *)NULL)) err(1, "settimeofday (timeval)"); logwtmp("{", "date", ""); } if ((p = getlogin()) == NULL) p = "???"; syslog(LOG_AUTH | LOG_NOTICE, "date set by %s", p); } } static void badformat(void) { warnx("illegal time format"); usage(); } static void usage(void) { (void)fprintf(stderr, "%s\n%s\n", "usage: date [-jnu] [-d dst] [-r seconds] [-t west] " "[-v[+|-]val[ymwdHMS]] ... ", " " "[-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]"); exit(1); } ================================================ FILE: core-tools/tests-regression/code-metrics/in/date/extern.h ================================================ /*- * Copyright (c) 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)extern.h 8.1 (Berkeley) 5/31/93 * $FreeBSD: src/bin/date/extern.h,v 1.7.30.1.8.2 2012/11/17 08:24:31 svnexp Exp $ */ int netsettime(time_t); ================================================ FILE: core-tools/tests-regression/code-metrics/in/date/netdate.c ================================================ /*- * Copyright (c) 1990, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #if 0 #ifndef lint static char sccsid[] = "@(#)netdate.c 8.1 (Berkeley) 5/31/93"; #endif /* not lint */ #endif #include __FBSDID("$FreeBSD: src/bin/date/netdate.c,v 1.19.10.1.8.2 2012/11/17 08:24:31 svnexp Exp $"); #include #include #include #include #include #define TSPTYPES #include #include #include #include #include #include "extern.h" #define WAITACK 2 /* seconds */ #define WAITDATEACK 5 /* seconds */ extern int retval; /* * Set the date in the machines controlled by timedaemons by communicating the * new date to the local timedaemon. If the timedaemon is in the master state, * it performs the correction on all slaves. If it is in the slave state, it * notifies the master that a correction is needed. * Returns 0 on success. Returns > 0 on failure, setting retval to 2; */ int netsettime(time_t tval) { struct timeval tout; struct servent *sp; struct tsp msg; struct sockaddr_in lsin, dest, from; fd_set ready; long waittime; int s, port, timed_ack, found, lerr; socklen_t length; char hostname[MAXHOSTNAMELEN]; if ((sp = getservbyname("timed", "udp")) == NULL) { warnx("timed/udp: unknown service"); return (retval = 2); } dest.sin_port = sp->s_port; dest.sin_family = AF_INET; dest.sin_addr.s_addr = htonl((u_long)INADDR_ANY); s = socket(AF_INET, SOCK_DGRAM, 0); if (s < 0) { if (errno != EPROTONOSUPPORT) warn("timed"); return (retval = 2); } memset(&lsin, 0, sizeof(lsin)); lsin.sin_family = AF_INET; for (port = IPPORT_RESERVED - 1; port > IPPORT_RESERVED / 2; port--) { lsin.sin_port = htons((u_short)port); if (bind(s, (struct sockaddr *)&lsin, sizeof(lsin)) >= 0) break; if (errno == EADDRINUSE) continue; if (errno != EADDRNOTAVAIL) warn("bind"); goto bad; } if (port == IPPORT_RESERVED / 2) { warnx("all ports in use"); goto bad; } memset(&msg, 0, sizeof(msg)); msg.tsp_type = TSP_SETDATE; msg.tsp_vers = TSPVERSION; if (gethostname(hostname, sizeof(hostname))) { warn("gethostname"); goto bad; } (void)strlcpy(msg.tsp_name, hostname, sizeof(msg.tsp_name)); msg.tsp_seq = htons((u_short)0); msg.tsp_time.tv_sec = htonl((u_long)tval); msg.tsp_time.tv_usec = htonl((u_long)0); length = sizeof(struct sockaddr_in); if (connect(s, (struct sockaddr *)&dest, length) < 0) { warn("connect"); goto bad; } if (send(s, (char *)&msg, sizeof(struct tsp), 0) < 0) { if (errno != ECONNREFUSED) warn("send"); goto bad; } timed_ack = -1; waittime = WAITACK; loop: tout.tv_sec = waittime; tout.tv_usec = 0; FD_ZERO(&ready); FD_SET(s, &ready); found = select(FD_SETSIZE, &ready, (fd_set *)0, (fd_set *)0, &tout); length = sizeof(lerr); if (!getsockopt(s, SOL_SOCKET, SO_ERROR, (char *)&lerr, &length) && lerr) { if (lerr != ECONNREFUSED) warnc(lerr, "send (delayed error)"); goto bad; } if (found > 0 && FD_ISSET(s, &ready)) { length = sizeof(struct sockaddr_in); if (recvfrom(s, &msg, sizeof(struct tsp), 0, (struct sockaddr *)&from, &length) < 0) { if (errno != ECONNREFUSED) warn("recvfrom"); goto bad; } msg.tsp_seq = ntohs(msg.tsp_seq); msg.tsp_time.tv_sec = ntohl(msg.tsp_time.tv_sec); msg.tsp_time.tv_usec = ntohl(msg.tsp_time.tv_usec); switch (msg.tsp_type) { case TSP_ACK: timed_ack = TSP_ACK; waittime = WAITDATEACK; goto loop; case TSP_DATEACK: (void)close(s); return (0); default: warnx("wrong ack received from timed: %s", tsptype[msg.tsp_type]); timed_ack = -1; break; } } if (timed_ack == -1) warnx("can't reach time daemon, time set locally"); bad: (void)close(s); return (retval = 2); } ================================================ FILE: core-tools/tests-regression/code-metrics/in/date/vary.c ================================================ /*- * Copyright (c) 1997 Brian Somers * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD: src/bin/date/vary.c,v 1.16.30.1.8.2 2012/11/17 08:24:31 svnexp Exp $"); #include #include #include #include #include "vary.h" struct trans { int val; const char *str; }; static struct trans trans_mon[] = { { 1, "january" }, { 2, "february" }, { 3, "march" }, { 4, "april" }, { 5, "may"}, { 6, "june" }, { 7, "july" }, { 8, "august" }, { 9, "september" }, { 10, "october" }, { 11, "november" }, { 12, "december" }, { -1, NULL } }; static struct trans trans_wday[] = { { 0, "sunday" }, { 1, "monday" }, { 2, "tuesday" }, { 3, "wednesday" }, { 4, "thursday" }, { 5, "friday" }, { 6, "saturday" }, { -1, NULL } }; static char digits[] = "0123456789"; static int adjhour(struct tm *, char, int, int); static int domktime(struct tm *t, char type) { time_t ret; while ((ret = mktime(t)) == -1 && t->tm_year > 68 && t->tm_year < 138) /* While mktime() fails, adjust by an hour */ adjhour(t, type == '-' ? type : '+', 1, 0); return ret; } static int trans(const struct trans t[], const char *arg) { int f; for (f = 0; t[f].val != -1; f++) if (!strncasecmp(t[f].str, arg, 3) || !strncasecmp(t[f].str, arg, strlen(t[f].str))) return t[f].val; return -1; } struct vary * vary_append(struct vary *v, char *arg) { struct vary *result, **nextp; if (v) { result = v; while (v->next) v = v->next; nextp = &v->next; } else nextp = &result; if ((*nextp = (struct vary *)malloc(sizeof(struct vary))) == NULL) err(1, "malloc"); (*nextp)->arg = arg; (*nextp)->next = NULL; return result; } static int mdays[12] = { 31, 0, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 }; static int daysinmonth(const struct tm *t) { int year; year = t->tm_year + 1900; if (t->tm_mon == 1) if (!(year % 400)) return 29; else if (!(year % 100)) return 28; else if (!(year % 4)) return 29; else return 28; else if (t->tm_mon >= 0 && t->tm_mon < 12) return mdays[t->tm_mon]; return 0; } static int adjyear(struct tm *t, char type, int val, int mk) { switch (type) { case '+': t->tm_year += val; break; case '-': t->tm_year -= val; break; default: t->tm_year = val; if (t->tm_year < 69) t->tm_year += 100; /* as per date.c */ else if (t->tm_year > 1900) t->tm_year -= 1900; /* struct tm holds years since 1900 */ break; } return !mk || domktime(t, type) != -1; } static int adjmon(struct tm *t, char type, int val, int istext, int mk) { int lmdays; if (val < 0) return 0; switch (type) { case '+': if (istext) { if (val <= t->tm_mon) val += 11 - t->tm_mon; /* early next year */ else val -= t->tm_mon + 1; /* later this year */ } if (val) { if (!adjyear(t, '+', (t->tm_mon + val) / 12, 0)) return 0; val %= 12; t->tm_mon += val; if (t->tm_mon > 11) t->tm_mon -= 12; } break; case '-': if (istext) { if (val-1 > t->tm_mon) val = 13 - val + t->tm_mon; /* later last year */ else val = t->tm_mon - val + 1; /* early this year */ } if (val) { if (!adjyear(t, '-', val / 12, 0)) return 0; val %= 12; if (val > t->tm_mon) { if (!adjyear(t, '-', 1, 0)) return 0; val -= 12; } t->tm_mon -= val; } break; default: if (val > 12 || val < 1) return 0; t->tm_mon = --val; } /* e.g., -v-1m on March, 31 is the last day of February in common sense */ lmdays = daysinmonth(t); if (t->tm_mday > lmdays) t->tm_mday = lmdays; return !mk || domktime(t, type) != -1; } static int adjday(struct tm *t, char type, int val, int mk) { int lmdays; switch (type) { case '+': while (val) { lmdays = daysinmonth(t); if (val > lmdays - t->tm_mday) { val -= lmdays - t->tm_mday + 1; t->tm_mday = 1; if (!adjmon(t, '+', 1, 0, 0)) return 0; } else { t->tm_mday += val; val = 0; } } break; case '-': while (val) if (val >= t->tm_mday) { val -= t->tm_mday; t->tm_mday = 1; if (!adjmon(t, '-', 1, 0, 0)) return 0; t->tm_mday = daysinmonth(t); } else { t->tm_mday -= val; val = 0; } break; default: if (val > 0 && val <= daysinmonth(t)) t->tm_mday = val; else return 0; break; } return !mk || domktime(t, type) != -1; } static int adjwday(struct tm *t, char type, int val, int istext, int mk) { if (val < 0) return 0; switch (type) { case '+': if (istext) if (val < t->tm_wday) val = 7 - t->tm_wday + val; /* early next week */ else val -= t->tm_wday; /* later this week */ else val *= 7; /* "-v+5w" == "5 weeks in the future" */ return !val || adjday(t, '+', val, mk); case '-': if (istext) { if (val > t->tm_wday) val = 7 - val + t->tm_wday; /* later last week */ else val = t->tm_wday - val; /* early this week */ } else val *= 7; /* "-v-5w" == "5 weeks ago" */ return !val || adjday(t, '-', val, mk); default: if (val < t->tm_wday) return adjday(t, '-', t->tm_wday - val, mk); else if (val > 6) return 0; else if (val > t->tm_wday) return adjday(t, '+', val - t->tm_wday, mk); } return 1; } static int adjhour(struct tm *t, char type, int val, int mk) { if (val < 0) return 0; switch (type) { case '+': if (val) { int days; days = (t->tm_hour + val) / 24; val %= 24; t->tm_hour += val; t->tm_hour %= 24; if (!adjday(t, '+', days, 0)) return 0; } break; case '-': if (val) { int days; days = val / 24; val %= 24; if (val > t->tm_hour) { days++; val -= 24; } t->tm_hour -= val; if (!adjday(t, '-', days, 0)) return 0; } break; default: if (val > 23) return 0; t->tm_hour = val; } return !mk || domktime(t, type) != -1; } static int adjmin(struct tm *t, char type, int val, int mk) { if (val < 0) return 0; switch (type) { case '+': if (val) { if (!adjhour(t, '+', (t->tm_min + val) / 60, 0)) return 0; val %= 60; t->tm_min += val; if (t->tm_min > 59) t->tm_min -= 60; } break; case '-': if (val) { if (!adjhour(t, '-', val / 60, 0)) return 0; val %= 60; if (val > t->tm_min) { if (!adjhour(t, '-', 1, 0)) return 0; val -= 60; } t->tm_min -= val; } break; default: if (val > 59) return 0; t->tm_min = val; } return !mk || domktime(t, type) != -1; } static int adjsec(struct tm *t, char type, int val, int mk) { if (val < 0) return 0; switch (type) { case '+': if (val) { if (!adjmin(t, '+', (t->tm_sec + val) / 60, 0)) return 0; val %= 60; t->tm_sec += val; if (t->tm_sec > 59) t->tm_sec -= 60; } break; case '-': if (val) { if (!adjmin(t, '-', val / 60, 0)) return 0; val %= 60; if (val > t->tm_sec) { if (!adjmin(t, '-', 1, 0)) return 0; val -= 60; } t->tm_sec -= val; } break; default: if (val > 59) return 0; t->tm_sec = val; } return !mk || domktime(t, type) != -1; } const struct vary * vary_apply(const struct vary *v, struct tm *t) { char type; char which; char *arg; size_t len; int val; for (; v; v = v->next) { type = *v->arg; arg = v->arg; if (type == '+' || type == '-') arg++; else type = '\0'; len = strlen(arg); if (len < 2) return v; if (type == '\0') t->tm_isdst = -1; if (strspn(arg, digits) != len-1) { val = trans(trans_wday, arg); if (val != -1) { if (!adjwday(t, type, val, 1, 1)) return v; } else { val = trans(trans_mon, arg); if (val != -1) { if (!adjmon(t, type, val, 1, 1)) return v; } else return v; } } else { val = atoi(arg); which = arg[len-1]; switch (which) { case 'S': if (!adjsec(t, type, val, 1)) return v; break; case 'M': if (!adjmin(t, type, val, 1)) return v; break; case 'H': if (!adjhour(t, type, val, 1)) return v; break; case 'd': t->tm_isdst = -1; if (!adjday(t, type, val, 1)) return v; break; case 'w': t->tm_isdst = -1; if (!adjwday(t, type, val, 0, 1)) return v; break; case 'm': t->tm_isdst = -1; if (!adjmon(t, type, val, 0, 1)) return v; break; case 'y': t->tm_isdst = -1; if (!adjyear(t, type, val, 1)) return v; break; default: return v; } } } return 0; } void vary_destroy(struct vary *v) { struct vary *n; while (v) { n = v->next; free(v); v = n; } } ================================================ FILE: core-tools/tests-regression/code-metrics/in/date/vary.h ================================================ /*- * Copyright (c) 1997 Brian Somers * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD: src/bin/date/vary.h,v 1.4.56.1.8.2 2012/11/17 08:24:31 svnexp Exp $ */ struct vary { char *arg; struct vary *next; }; extern struct vary *vary_append(struct vary *v, char *arg); extern const struct vary *vary_apply(const struct vary *v, struct tm *t); extern void vary_destroy(struct vary *v); ================================================ FILE: core-tools/tests-regression/code-metrics/out.ok ================================================ FNAMELEN: 7 NSTRUCT: 2 NTYPEDEF: 0 NVOID: 18 NGETS: 0 IDLEN: 6.2008 CHLINESCHAR: 1062:27586 NCCHAR: 18 NCOMMENT: 53 NCOPYRIGHT: 17 NCFILE: 3 NCDIR: 1 CLINESCHAR: 993:24159 NFUNCTION: 18 NGOTO: 8 NREGISTER: 0 NMACRO: 0 NINCLUDE: 32 NCONST: 82 NHFILE: 2 ================================================ FILE: core-tools/tests-regression/commit-stats/out.ok ================================================ Authors ordered by number of commits 557 Jim Meyering 252 Paul Eggert 146 Paolo Bonzini 79 Norihiro Tanaka 9 Bruno Haible 8 Eric Blake 7 Stefano Lattarini 6 Aharon Robbins 5 Eli Zaretskii 5 Arnold D. Robbins 4 Reuben Thomas 4 Norihirio Tanaka 3 Arnold Robbins 2 behoffski 2 Stephane Chazelas 2 Santiago Ruano Rincón 2 Mike Frysinger 2 Benno Schulenberg 1 Zev Weiss 1 Yuliy Pisetsky 1 Tony Abou-Assaleh 1 Pádraig Brady 1 Petr Písař 1 Petr Pisar 1 Patrick Boyd 1 Nicolas Vigier 1 Mike Haertel 1 Matthew Burgess 1 Karl Berry 1 Javier Villavicencio 1 Jaroslav Škarvada 1 Dmitry V. Levin 1 Corinna Vinschen 1 Allan McRae 1 Adam Katz Days ordered by number of commits 186 Thu 167 Fri 161 Mon 152 Wed 149 Sun 149 Sat 148 Tue ================================================ FILE: core-tools/tests-regression/compress-compare/out.ok ================================================ File type: ASCII Original size:40735 xz:16728 bzip2:15556 gzip:17730 ================================================ FILE: core-tools/tests-regression/dgsh-wrap/.gitignore ================================================ echo echo-S ================================================ FILE: core-tools/tests-regression/dgsh-wrap/dd-args.ok ================================================ 0 ================================================ FILE: core-tools/tests-regression/dgsh-wrap/echo-deaf.ok ================================================ hi 0 ================================================ FILE: core-tools/tests-regression/dgsh-wrap/echo-s.ok ================================================ hi 0 ================================================ FILE: core-tools/tests-regression/dgsh-wrap/echo-s_caps.ok ================================================ hi 0 ================================================ FILE: core-tools/tests-regression/dgsh-wrap/paste1.ok ================================================ 0 1 ================================================ FILE: core-tools/tests-regression/dgsh-wrap/paste2.ok ================================================ 0 1 ================================================ FILE: core-tools/tests-regression/dgsh-wrap/paste3.ok ================================================ 0 1 ================================================ FILE: core-tools/tests-regression/dgsh-wrap/paste4.ok ================================================ 0 1 ================================================ FILE: core-tools/tests-regression/dgsh-wrap/tee1.ok ================================================ ahi bhi ================================================ FILE: core-tools/tests-regression/dgsh-wrap/tee2.ok ================================================ ahi bhi ================================================ FILE: core-tools/tests-regression/duplicate-files/in/another-same-1 ================================================ Another same ================================================ FILE: core-tools/tests-regression/duplicate-files/in/another-same-2 ================================================ Another same ================================================ FILE: core-tools/tests-regression/duplicate-files/in/different-file-1 ================================================ hi there ================================================ FILE: core-tools/tests-regression/duplicate-files/in/different-file-2 ================================================ Some more text ================================================ FILE: core-tools/tests-regression/duplicate-files/in/different-file-3 ================================================ More text ================================================ FILE: core-tools/tests-regression/duplicate-files/in/same-file-1 ================================================ hello, world ================================================ FILE: core-tools/tests-regression/duplicate-files/in/same-file-2 ================================================ hello, world ================================================ FILE: core-tools/tests-regression/duplicate-files/in/same-file-3 ================================================ hello, world ================================================ FILE: core-tools/tests-regression/duplicate-files/out.ok ================================================ duplicate-files/in/same-file-1 duplicate-files/in/same-file-2 duplicate-files/in/same-file-3 duplicate-files/in/another-same-1 duplicate-files/in/another-same-2 ================================================ FILE: core-tools/tests-regression/map-hierarchy/in/a/date/date.c ================================================ /*- * Copyright (c) 1985, 1987, 1988, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #ifndef lint static char const copyright[] = "@(#) Copyright (c) 1985, 1987, 1988, 1993\n\ The Regents of the University of California. All rights reserved.\n"; #endif /* not lint */ #include #include #include #include #include #include #include #include #include #include #include #include #include "vary.h" #ifndef TM_YEAR_BASE #define TM_YEAR_BASE 1900 #endif static time_t tval; int retval; static void setthetime(const char *, const char *, int, int); static void badformat(void); static void usage(void); int main(int argc, char *argv[]) { struct timezone tz; int ch, rflag; int jflag; const char *format; char buf[1024]; char *endptr, *fmt; char *tmp; int set_timezone; struct vary *v; const struct vary *badv; struct tm lt; v = NULL; fmt = NULL; (void) setlocale(LC_TIME, ""); tz.tz_dsttime = tz.tz_minuteswest = 0; rflag = 0; jflag = 0; set_timezone = 0; while ((ch = getopt(argc, argv, "d:f:jnr:t:uv:")) != -1) switch((char)ch) { case 'd': /* daylight savings time */ tz.tz_dsttime = strtol(optarg, &endptr, 10) ? 1 : 0; if (endptr == optarg || *endptr != '\0') usage(); set_timezone = 1; break; case 'f': fmt = optarg; break; case 'j': jflag = 1; /* don't set time */ break; case 'r': /* user specified seconds */ rflag = 1; tval = strtoq(optarg, &tmp, 0); if (*tmp != 0) usage(); break; case 't': /* minutes west of UTC */ /* error check; don't allow "PST" */ tz.tz_minuteswest = strtol(optarg, &endptr, 10); if (endptr == optarg || *endptr != '\0') usage(); set_timezone = 1; break; case 'u': /* do everything in UTC */ (void)setenv("TZ", "UTC0", 1); break; case 'v': v = vary_append(v, optarg); break; default: usage(); } argc -= optind; argv += optind; /* * If -d or -t, set the timezone or daylight savings time; this * doesn't belong here; the kernel should not know about either. */ if (set_timezone && settimeofday((struct timeval *)NULL, &tz)) err(1, "settimeofday (timezone)"); if (!rflag && time(&tval) == -1) err(1, "time"); format = "%+"; /* allow the operands in any order */ if (*argv && **argv == '+') { format = *argv + 1; ++argv; } if (*argv) { setthetime(fmt, *argv, jflag); ++argv; } else if (fmt != NULL) usage(); if (*argv && **argv == '+') format = *argv + 1; lt = *localtime(&tval); badv = vary_apply(v, <); if (badv) { fprintf(stderr, "%s: Cannot apply date adjustment\n", badv->arg); vary_destroy(v); usage(); } vary_destroy(v); (void)strftime(buf, sizeof(buf), format, <); (void)printf("%s\n", buf); if (fflush(stdout)) err(1, "stdout"); exit(retval); } #define ATOI2(s) ((s) += 2, ((s)[-2] - '0') * 10 + ((s)[-1] - '0')) static void setthetime(const char *fmt, const char *p, int jflag) { struct tm *lt; struct timeval tv; const char *dot, *t; int century; lt = localtime(&tval); lt->tm_isdst = -1; /* divine correct DST */ if (fmt != NULL) { t = strptime(p, fmt, lt); if (t == NULL) { fprintf(stderr, "Failed conversion of ``%s''" " using format ``%s''\n", p, fmt); badformat(); } else if (*t != '\0') fprintf(stderr, "Warning: Ignoring %ld extraneous" " characters in date string (%s)\n", (long) strlen(t), t); } else { for (t = p, dot = NULL; *t; ++t) { if (isdigit(*t)) continue; if (*t == '.' && dot == NULL) { dot = t; continue; } badformat(); } if (dot != NULL) { /* .ss */ dot++; /* *dot++ = '\0'; */ if (strlen(dot) != 2) badformat(); lt->tm_sec = ATOI2(dot); if (lt->tm_sec > 61) badformat(); } else lt->tm_sec = 0; century = 0; /* if p has a ".ss" field then let's pretend it's not there */ switch (strlen(p) - ((dot != NULL) ? 3 : 0)) { case 12: /* cc */ lt->tm_year = ATOI2(p) * 100 - TM_YEAR_BASE; century = 1; /* FALLTHROUGH */ case 10: /* yy */ if (century) lt->tm_year += ATOI2(p); else { lt->tm_year = ATOI2(p); if (lt->tm_year < 69) /* hack for 2000 ;-} */ lt->tm_year += 2000 - TM_YEAR_BASE; else lt->tm_year += 1900 - TM_YEAR_BASE; } /* FALLTHROUGH */ case 8: /* mm */ lt->tm_mon = ATOI2(p); if (lt->tm_mon > 12) badformat(); --lt->tm_mon; /* time struct is 0 - 11 */ /* FALLTHROUGH */ case 6: /* dd */ lt->tm_mday = ATOI2(p); if (lt->tm_mday > 31) badformat(); /* FALLTHROUGH */ case 4: /* HH */ lt->tm_hour = ATOI2(p); if (lt->tm_hour > 23) badformat(); /* FALLTHROUGH */ case 2: /* MM */ lt->tm_min = ATOI2(p); if (lt->tm_min > 59) badformat(); break; default: badformat(); } } /* convert broken-down time to GMT clock time */ if ((tval = mktime(lt)) == -1) errx(1, "nonexistent time"); if (!jflag) { /* set the time */ if ((p = getlogin()) == NULL) p = "???"; syslog(LOG_AUTH | LOG_NOTICE, "date set by %s", p); } } static void badformat(void) { warnx("illegal time format"); usage(); } static void usage(void) { (void)fprintf(stderr, "%s\n%s\n", "usage: date [-jnu] [-d dst] [-r seconds] [-t west] " "[-v[+|-]val[ymwdHMS]] ... ", " " "[-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]"); exit(1); } ================================================ FILE: core-tools/tests-regression/map-hierarchy/in/a/date/vary.c ================================================ /*- * Copyright (c) 1997 Brian Somers * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include #include #include #include #include #include "vary.h" struct trans { int val; const char *str; }; static struct trans trans_mon[] = { { 1, "january" }, { 2, "february" }, { 3, "march" }, { 4, "april" }, { 5, "may"}, { 6, "june" }, { 7, "july" }, { 8, "august" }, { 9, "september" }, { 10, "october" }, { 11, "november" }, { 12, "december" }, { -1, NULL } }; static struct trans trans_wday[] = { { 0, "sunday" }, { 1, "monday" }, { 2, "tuesday" }, { 3, "wednesday" }, { 4, "thursday" }, { 5, "friday" }, { 6, "saturday" }, { -1, NULL } }; static char digits[] = "0123456789"; static int adjhour(struct tm *, char, int, int); static int domktime(struct tm *t, char type) { time_t ret; while ((ret = mktime(t)) == -1 && t->tm_year > 68 && t->tm_year < 138) /* While mktime() fails, adjust by an hour */ adjhour(t, type == '-' ? type : '+', 1, 0); return ret; } static int trans(const struct trans t[], const char *arg) { int f; for (f = 0; t[f].val != -1; f++) if (!strncasecmp(t[f].str, arg, 3) || !strncasecmp(t[f].str, arg, strlen(t[f].str))) return t[f].val; return -1; } struct vary * vary_append(struct vary *v, char *arg) { struct vary *result, **nextp; if (v) { result = v; while (v->next) v = v->next; nextp = &v->next; } else nextp = &result; if ((*nextp = (struct vary *)malloc(sizeof(struct vary))) == NULL) err(1, "malloc"); (*nextp)->arg = arg; (*nextp)->next = NULL; return result; } static int mdays[12] = { 31, 0, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 }; static int daysinmonth(const struct tm *t) { int year; year = t->tm_year + 1900; if (t->tm_mon == 1) if (!(year % 400)) return 29; else if (!(year % 100)) return 28; else if (!(year % 4)) return 29; else return 28; else if (t->tm_mon >= 0 && t->tm_mon < 12) return mdays[t->tm_mon]; return 0; } static int adjyear(struct tm *t, char type, int val, int mk) { switch (type) { case '+': t->tm_year += val; break; case '-': t->tm_year -= val; break; default: t->tm_year = val; if (t->tm_year < 69) t->tm_year += 100; /* as per date.c */ else if (t->tm_year > 1900) t->tm_year -= 1900; /* struct tm holds years since 1900 */ break; } return !mk || domktime(t, type) != -1; } static int adjmon(struct tm *t, char type, int val, int istext, int mk) { int lmdays; if (val < 0) return 0; switch (type) { case '+': if (istext) { if (val <= t->tm_mon) val += 11 - t->tm_mon; /* early next year */ else val -= t->tm_mon + 1; /* later this year */ } if (val) { if (!adjyear(t, '+', (t->tm_mon + val) / 12, 0)) return 0; val %= 12; t->tm_mon += val; if (t->tm_mon > 11) t->tm_mon -= 12; } break; case '-': if (istext) { if (val-1 > t->tm_mon) val = 13 - val + t->tm_mon; /* later last year */ else val = t->tm_mon - val + 1; /* early this year */ } if (val) { if (!adjyear(t, '-', val / 12, 0)) return 0; val %= 12; if (val > t->tm_mon) { if (!adjyear(t, '-', 1, 0)) return 0; val -= 12; } t->tm_mon -= val; } break; default: if (val > 12 || val < 1) return 0; t->tm_mon = --val; } /* e.g., -v-1m on March, 31 is the last day of February in common sense */ lmdays = daysinmonth(t); if (t->tm_mday > lmdays) t->tm_mday = lmdays; return !mk || domktime(t, type) != -1; } static int adjday(struct tm *t, char type, int val, int mk) { int lmdays; switch (type) { case '+': while (val) { lmdays = daysinmonth(t); if (val > lmdays - t->tm_mday) { val -= lmdays - t->tm_mday + 1; t->tm_mday = 1; if (!adjmon(t, '+', 1, 0, 0)) return 0; } else { t->tm_mday += val; val = 0; } } break; case '-': while (val) if (val >= t->tm_mday) { val -= t->tm_mday; t->tm_mday = 1; if (!adjmon(t, '-', 1, 0, 0)) return 0; t->tm_mday = daysinmonth(t); } else { t->tm_mday -= val; val = 0; } break; default: if (val > 0 && val <= daysinmonth(t)) t->tm_mday = val; else return 0; break; } return !mk || domktime(t, type) != -1; } static int adjwday(struct tm *t, char type, int val, int istext, int mk) { if (val < 0) return 0; switch (type) { case '+': if (istext) if (val < t->tm_wday) val = 7 - t->tm_wday + val; /* early next week */ else val -= t->tm_wday; /* later this week */ else val *= 7; /* "-v+5w" == "5 weeks in the future" */ return !val || adjday(t, '+', val, mk); case '-': if (istext) { if (val > t->tm_wday) val = 7 - val + t->tm_wday; /* later last week */ else val = t->tm_wday - val; /* early this week */ } else val *= 7; /* "-v-5w" == "5 weeks ago" */ return !val || adjday(t, '-', val, mk); default: if (val < t->tm_wday) return adjday(t, '-', t->tm_wday - val, mk); else if (val > 6) return 0; else if (val > t->tm_wday) return adjday(t, '+', val - t->tm_wday, mk); } return 1; } static int adjhour(struct tm *t, char type, int val, int mk) { if (val < 0) return 0; switch (type) { case '+': if (val) { int days; days = (t->tm_hour + val) / 24; val %= 24; t->tm_hour += val; t->tm_hour %= 24; if (!adjday(t, '+', days, 0)) return 0; } break; case '-': if (val) { int days; days = val / 24; val %= 24; if (val > t->tm_hour) { days++; val -= 24; } t->tm_hour -= val; if (!adjday(t, '-', days, 0)) return 0; } break; default: if (val > 23) return 0; t->tm_hour = val; } return !mk || domktime(t, type) != -1; } static int adjmin(struct tm *t, char type, int val, int mk) { if (val < 0) return 0; switch (type) { case '+': if (val) { if (!adjhour(t, '+', (t->tm_min + val) / 60, 0)) return 0; val %= 60; t->tm_min += val; if (t->tm_min > 59) t->tm_min -= 60; } break; case '-': if (val) { if (!adjhour(t, '-', val / 60, 0)) return 0; val %= 60; if (val > t->tm_min) { if (!adjhour(t, '-', 1, 0)) return 0; val -= 60; } t->tm_min -= val; } break; default: if (val > 59) return 0; t->tm_min = val; } return !mk || domktime(t, type) != -1; } static int adjsec(struct tm *t, char type, int val, int mk) { if (val < 0) return 0; switch (type) { case '+': if (val) { if (!adjmin(t, '+', (t->tm_sec + val) / 60, 0)) return 0; val %= 60; t->tm_sec += val; if (t->tm_sec > 59) t->tm_sec -= 60; } break; case '-': if (val) { if (!adjmin(t, '-', val / 60, 0)) return 0; val %= 60; if (val > t->tm_sec) { if (!adjmin(t, '-', 1, 0)) return 0; val -= 60; } t->tm_sec -= val; } break; default: if (val > 59) return 0; t->tm_sec = val; } return !mk || domktime(t, type) != -1; } const struct vary * vary_apply(const struct vary *v, struct tm *t) { char type; char which; char *arg; size_t len; int val; for (; v; v = v->next) { type = *v->arg; arg = v->arg; if (type == '+' || type == '-') arg++; else type = '\0'; len = strlen(arg); if (len < 2) return v; if (type == '\0') t->tm_isdst = -1; if (strspn(arg, digits) != len-1) { val = trans(trans_wday, arg); if (val != -1) { if (!adjwday(t, type, val, 1, 1)) return v; } else { val = trans(trans_mon, arg); if (val != -1) { if (!adjmon(t, type, val, 1, 1)) return v; } else return v; } } else { val = atoi(arg); which = arg[len-1]; switch (which) { case 'S': if (!adjsec(t, type, val, 1)) return v; break; case 'M': if (!adjmin(t, type, val, 1)) return v; break; case 'H': if (!adjhour(t, type, val, 1)) return v; break; case 'd': t->tm_isdst = -1; if (!adjday(t, type, val, 1)) return v; break; case 'w': t->tm_isdst = -1; if (!adjwday(t, type, val, 0, 1)) return v; break; case 'm': t->tm_isdst = -1; if (!adjmon(t, type, val, 0, 1)) return v; break; case 'y': t->tm_isdst = -1; if (!adjyear(t, type, val, 1)) return v; break; default: return v; } } } return 0; } void vary_destroy(struct vary *v) { struct vary *n; while (v) { n = v->next; free(v); v = n; } } ================================================ FILE: core-tools/tests-regression/map-hierarchy/in/a/date/vary.h ================================================ /*- * Copyright (c) 1997 Brian Somers * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD: src/bin/date/vary.h,v 1.4.56.1.8.2 2012/11/17 08:24:31 svnexp Exp $ */ struct vary { char *arg; struct vary *next; }; extern struct vary *vary_append(struct vary *v, char *arg); extern const struct vary *vary_apply(const struct vary *v, struct tm *t); extern void vary_destroy(struct vary *v); ================================================ FILE: core-tools/tests-regression/map-hierarchy/in/b/bin/date/date.c ================================================ /*- * Copyright (c) 1985, 1987, 1988, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #ifndef lint static char const copyright[] = "@(#) Copyright (c) 1985, 1987, 1988, 1993\n\ The Regents of the University of California. All rights reserved.\n"; #endif /* not lint */ #if 0 #ifndef lint static char sccsid[] = "@(#)date.c 8.2 (Berkeley) 4/28/95"; #endif /* not lint */ #endif #include __FBSDID("$FreeBSD: src/bin/date/date.c,v 1.48.2.1.8.2 2012/11/17 08:24:31 svnexp Exp $"); #include #include #include #include #include #include #include #include #include #include #include #include "extern.h" #include "vary.h" #ifndef TM_YEAR_BASE #define TM_YEAR_BASE 1900 #endif static time_t tval; int retval; static void setthetime(const char *, const char *, int, int); static void badformat(void); static void usage(void); int main(int argc, char *argv[]) { struct timezone tz; int ch, rflag; int jflag, nflag; const char *format; char buf[1024]; char *endptr, *fmt; char *tmp; int set_timezone; struct vary *v; const struct vary *badv; struct tm lt; v = NULL; fmt = NULL; (void) setlocale(LC_TIME, ""); tz.tz_dsttime = tz.tz_minuteswest = 0; rflag = 0; jflag = nflag = 0; set_timezone = 0; while ((ch = getopt(argc, argv, "d:f:jnr:t:uv:")) != -1) switch((char)ch) { case 'd': /* daylight savings time */ tz.tz_dsttime = strtol(optarg, &endptr, 10) ? 1 : 0; if (endptr == optarg || *endptr != '\0') usage(); set_timezone = 1; break; case 'f': fmt = optarg; break; case 'j': jflag = 1; /* don't set time */ break; case 'n': /* don't set network */ nflag = 1; break; case 'r': /* user specified seconds */ rflag = 1; tval = strtoq(optarg, &tmp, 0); if (*tmp != 0) usage(); break; case 't': /* minutes west of UTC */ /* error check; don't allow "PST" */ tz.tz_minuteswest = strtol(optarg, &endptr, 10); if (endptr == optarg || *endptr != '\0') usage(); set_timezone = 1; break; case 'u': /* do everything in UTC */ (void)setenv("TZ", "UTC0", 1); break; case 'v': v = vary_append(v, optarg); break; default: usage(); } argc -= optind; argv += optind; /* * If -d or -t, set the timezone or daylight savings time; this * doesn't belong here; the kernel should not know about either. */ if (set_timezone && settimeofday((struct timeval *)NULL, &tz)) err(1, "settimeofday (timezone)"); if (!rflag && time(&tval) == -1) err(1, "time"); format = "%+"; /* allow the operands in any order */ if (*argv && **argv == '+') { format = *argv + 1; ++argv; } if (*argv) { setthetime(fmt, *argv, jflag, nflag); ++argv; } else if (fmt != NULL) usage(); if (*argv && **argv == '+') format = *argv + 1; lt = *localtime(&tval); badv = vary_apply(v, <); if (badv) { fprintf(stderr, "%s: Cannot apply date adjustment\n", badv->arg); vary_destroy(v); usage(); } vary_destroy(v); (void)strftime(buf, sizeof(buf), format, <); (void)printf("%s\n", buf); if (fflush(stdout)) err(1, "stdout"); exit(retval); } #define ATOI2(s) ((s) += 2, ((s)[-2] - '0') * 10 + ((s)[-1] - '0')) static void setthetime(const char *fmt, const char *p, int jflag, int nflag) { struct tm *lt; struct timeval tv; const char *dot, *t; int century; lt = localtime(&tval); lt->tm_isdst = -1; /* divine correct DST */ if (fmt != NULL) { t = strptime(p, fmt, lt); if (t == NULL) { fprintf(stderr, "Failed conversion of ``%s''" " using format ``%s''\n", p, fmt); badformat(); } else if (*t != '\0') fprintf(stderr, "Warning: Ignoring %ld extraneous" " characters in date string (%s)\n", (long) strlen(t), t); } else { for (t = p, dot = NULL; *t; ++t) { if (isdigit(*t)) continue; if (*t == '.' && dot == NULL) { dot = t; continue; } badformat(); } if (dot != NULL) { /* .ss */ dot++; /* *dot++ = '\0'; */ if (strlen(dot) != 2) badformat(); lt->tm_sec = ATOI2(dot); if (lt->tm_sec > 61) badformat(); } else lt->tm_sec = 0; century = 0; /* if p has a ".ss" field then let's pretend it's not there */ switch (strlen(p) - ((dot != NULL) ? 3 : 0)) { case 12: /* cc */ lt->tm_year = ATOI2(p) * 100 - TM_YEAR_BASE; century = 1; /* FALLTHROUGH */ case 10: /* yy */ if (century) lt->tm_year += ATOI2(p); else { lt->tm_year = ATOI2(p); if (lt->tm_year < 69) /* hack for 2000 ;-} */ lt->tm_year += 2000 - TM_YEAR_BASE; else lt->tm_year += 1900 - TM_YEAR_BASE; } /* FALLTHROUGH */ case 8: /* mm */ lt->tm_mon = ATOI2(p); if (lt->tm_mon > 12) badformat(); --lt->tm_mon; /* time struct is 0 - 11 */ /* FALLTHROUGH */ case 6: /* dd */ lt->tm_mday = ATOI2(p); if (lt->tm_mday > 31) badformat(); /* FALLTHROUGH */ case 4: /* HH */ lt->tm_hour = ATOI2(p); if (lt->tm_hour > 23) badformat(); /* FALLTHROUGH */ case 2: /* MM */ lt->tm_min = ATOI2(p); if (lt->tm_min > 59) badformat(); break; default: badformat(); } } /* convert broken-down time to GMT clock time */ if ((tval = mktime(lt)) == -1) errx(1, "nonexistent time"); if (!jflag) { /* set the time */ if (nflag || netsettime(tval)) { logwtmp("|", "date", ""); tv.tv_sec = tval; tv.tv_usec = 0; if (settimeofday(&tv, (struct timezone *)NULL)) err(1, "settimeofday (timeval)"); logwtmp("{", "date", ""); } if ((p = getlogin()) == NULL) p = "???"; syslog(LOG_AUTH | LOG_NOTICE, "date set by %s", p); } } static void badformat(void) { warnx("illegal time format"); usage(); } static void usage(void) { (void)fprintf(stderr, "%s\n%s\n", "usage: date [-jnu] [-d dst] [-r seconds] [-t west] " "[-v[+|-]val[ymwdHMS]] ... ", " " "[-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]"); exit(1); } ================================================ FILE: core-tools/tests-regression/map-hierarchy/in/b/bin/date/headers/extern.h ================================================ /*- * Copyright (c) 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)extern.h 8.1 (Berkeley) 5/31/93 * $FreeBSD: src/bin/date/extern.h,v 1.7.30.1.8.2 2012/11/17 08:24:31 svnexp Exp $ */ int netsettime(time_t); ================================================ FILE: core-tools/tests-regression/map-hierarchy/in/b/bin/date/headers/vary.h ================================================ /*- * Copyright (c) 1997 Brian Somers * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD: src/bin/date/vary.h,v 1.4.56.1.8.2 2012/11/17 08:24:31 svnexp Exp $ */ struct vary { char *arg; struct vary *next; }; extern struct vary *vary_append(struct vary *v, char *arg); extern const struct vary *vary_apply(const struct vary *v, struct tm *t); extern void vary_destroy(struct vary *v); ================================================ FILE: core-tools/tests-regression/map-hierarchy/in/b/bin/date/netdate.c ================================================ /*- * Copyright (c) 1990, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #if 0 #ifndef lint static char sccsid[] = "@(#)netdate.c 8.1 (Berkeley) 5/31/93"; #endif /* not lint */ #endif #include __FBSDID("$FreeBSD: src/bin/date/netdate.c,v 1.19.10.1.8.2 2012/11/17 08:24:31 svnexp Exp $"); #include #include #include #include #include #define TSPTYPES #include #include #include #include #include #include "extern.h" #define WAITACK 2 /* seconds */ #define WAITDATEACK 5 /* seconds */ extern int retval; /* * Set the date in the machines controlled by timedaemons by communicating the * new date to the local timedaemon. If the timedaemon is in the master state, * it performs the correction on all slaves. If it is in the slave state, it * notifies the master that a correction is needed. * Returns 0 on success. Returns > 0 on failure, setting retval to 2; */ int netsettime(time_t tval) { struct timeval tout; struct servent *sp; struct tsp msg; struct sockaddr_in lsin, dest, from; fd_set ready; long waittime; int s, port, timed_ack, found, lerr; socklen_t length; char hostname[MAXHOSTNAMELEN]; if ((sp = getservbyname("timed", "udp")) == NULL) { warnx("timed/udp: unknown service"); return (retval = 2); } dest.sin_port = sp->s_port; dest.sin_family = AF_INET; dest.sin_addr.s_addr = htonl((u_long)INADDR_ANY); s = socket(AF_INET, SOCK_DGRAM, 0); if (s < 0) { if (errno != EPROTONOSUPPORT) warn("timed"); return (retval = 2); } memset(&lsin, 0, sizeof(lsin)); lsin.sin_family = AF_INET; for (port = IPPORT_RESERVED - 1; port > IPPORT_RESERVED / 2; port--) { lsin.sin_port = htons((u_short)port); if (bind(s, (struct sockaddr *)&lsin, sizeof(lsin)) >= 0) break; if (errno == EADDRINUSE) continue; if (errno != EADDRNOTAVAIL) warn("bind"); goto bad; } if (port == IPPORT_RESERVED / 2) { warnx("all ports in use"); goto bad; } memset(&msg, 0, sizeof(msg)); msg.tsp_type = TSP_SETDATE; msg.tsp_vers = TSPVERSION; if (gethostname(hostname, sizeof(hostname))) { warn("gethostname"); goto bad; } (void)strlcpy(msg.tsp_name, hostname, sizeof(msg.tsp_name)); msg.tsp_seq = htons((u_short)0); msg.tsp_time.tv_sec = htonl((u_long)tval); msg.tsp_time.tv_usec = htonl((u_long)0); length = sizeof(struct sockaddr_in); if (connect(s, (struct sockaddr *)&dest, length) < 0) { warn("connect"); goto bad; } if (send(s, (char *)&msg, sizeof(struct tsp), 0) < 0) { if (errno != ECONNREFUSED) warn("send"); goto bad; } timed_ack = -1; waittime = WAITACK; loop: tout.tv_sec = waittime; tout.tv_usec = 0; FD_ZERO(&ready); FD_SET(s, &ready); found = select(FD_SETSIZE, &ready, (fd_set *)0, (fd_set *)0, &tout); length = sizeof(lerr); if (!getsockopt(s, SOL_SOCKET, SO_ERROR, (char *)&lerr, &length) && lerr) { if (lerr != ECONNREFUSED) warnc(lerr, "send (delayed error)"); goto bad; } if (found > 0 && FD_ISSET(s, &ready)) { length = sizeof(struct sockaddr_in); if (recvfrom(s, &msg, sizeof(struct tsp), 0, (struct sockaddr *)&from, &length) < 0) { if (errno != ECONNREFUSED) warn("recvfrom"); goto bad; } msg.tsp_seq = ntohs(msg.tsp_seq); msg.tsp_time.tv_sec = ntohl(msg.tsp_time.tv_sec); msg.tsp_time.tv_usec = ntohl(msg.tsp_time.tv_usec); switch (msg.tsp_type) { case TSP_ACK: timed_ack = TSP_ACK; waittime = WAITDATEACK; goto loop; case TSP_DATEACK: (void)close(s); return (0); default: warnx("wrong ack received from timed: %s", tsptype[msg.tsp_type]); timed_ack = -1; break; } } if (timed_ack == -1) warnx("can't reach time daemon, time set locally"); bad: (void)close(s); return (retval = 2); } ================================================ FILE: core-tools/tests-regression/map-hierarchy/in/b/bin/date/vary.c ================================================ /*- * Copyright (c) 1997 Brian Somers * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD: src/bin/date/vary.c,v 1.16.30.1.8.2 2012/11/17 08:24:31 svnexp Exp $"); #include #include #include #include #include "vary.h" struct trans { int val; const char *str; }; static struct trans trans_mon[] = { { 1, "january" }, { 2, "february" }, { 3, "march" }, { 4, "april" }, { 5, "may"}, { 6, "june" }, { 7, "july" }, { 8, "august" }, { 9, "september" }, { 10, "october" }, { 11, "november" }, { 12, "december" }, { -1, NULL } }; static struct trans trans_wday[] = { { 0, "sunday" }, { 1, "monday" }, { 2, "tuesday" }, { 3, "wednesday" }, { 4, "thursday" }, { 5, "friday" }, { 6, "saturday" }, { -1, NULL } }; static char digits[] = "0123456789"; static int adjhour(struct tm *, char, int, int); static int domktime(struct tm *t, char type) { time_t ret; while ((ret = mktime(t)) == -1 && t->tm_year > 68 && t->tm_year < 138) /* While mktime() fails, adjust by an hour */ adjhour(t, type == '-' ? type : '+', 1, 0); return ret; } static int trans(const struct trans t[], const char *arg) { int f; for (f = 0; t[f].val != -1; f++) if (!strncasecmp(t[f].str, arg, 3) || !strncasecmp(t[f].str, arg, strlen(t[f].str))) return t[f].val; return -1; } struct vary * vary_append(struct vary *v, char *arg) { struct vary *result, **nextp; if (v) { result = v; while (v->next) v = v->next; nextp = &v->next; } else nextp = &result; if ((*nextp = (struct vary *)malloc(sizeof(struct vary))) == NULL) err(1, "malloc"); (*nextp)->arg = arg; (*nextp)->next = NULL; return result; } static int mdays[12] = { 31, 0, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 }; static int daysinmonth(const struct tm *t) { int year; year = t->tm_year + 1900; if (t->tm_mon == 1) if (!(year % 400)) return 29; else if (!(year % 100)) return 28; else if (!(year % 4)) return 29; else return 28; else if (t->tm_mon >= 0 && t->tm_mon < 12) return mdays[t->tm_mon]; return 0; } static int adjyear(struct tm *t, char type, int val, int mk) { switch (type) { case '+': t->tm_year += val; break; case '-': t->tm_year -= val; break; default: t->tm_year = val; if (t->tm_year < 69) t->tm_year += 100; /* as per date.c */ else if (t->tm_year > 1900) t->tm_year -= 1900; /* struct tm holds years since 1900 */ break; } return !mk || domktime(t, type) != -1; } static int adjmon(struct tm *t, char type, int val, int istext, int mk) { int lmdays; if (val < 0) return 0; switch (type) { case '+': if (istext) { if (val <= t->tm_mon) val += 11 - t->tm_mon; /* early next year */ else val -= t->tm_mon + 1; /* later this year */ } if (val) { if (!adjyear(t, '+', (t->tm_mon + val) / 12, 0)) return 0; val %= 12; t->tm_mon += val; if (t->tm_mon > 11) t->tm_mon -= 12; } break; case '-': if (istext) { if (val-1 > t->tm_mon) val = 13 - val + t->tm_mon; /* later last year */ else val = t->tm_mon - val + 1; /* early this year */ } if (val) { if (!adjyear(t, '-', val / 12, 0)) return 0; val %= 12; if (val > t->tm_mon) { if (!adjyear(t, '-', 1, 0)) return 0; val -= 12; } t->tm_mon -= val; } break; default: if (val > 12 || val < 1) return 0; t->tm_mon = --val; } /* e.g., -v-1m on March, 31 is the last day of February in common sense */ lmdays = daysinmonth(t); if (t->tm_mday > lmdays) t->tm_mday = lmdays; return !mk || domktime(t, type) != -1; } static int adjday(struct tm *t, char type, int val, int mk) { int lmdays; switch (type) { case '+': while (val) { lmdays = daysinmonth(t); if (val > lmdays - t->tm_mday) { val -= lmdays - t->tm_mday + 1; t->tm_mday = 1; if (!adjmon(t, '+', 1, 0, 0)) return 0; } else { t->tm_mday += val; val = 0; } } break; case '-': while (val) if (val >= t->tm_mday) { val -= t->tm_mday; t->tm_mday = 1; if (!adjmon(t, '-', 1, 0, 0)) return 0; t->tm_mday = daysinmonth(t); } else { t->tm_mday -= val; val = 0; } break; default: if (val > 0 && val <= daysinmonth(t)) t->tm_mday = val; else return 0; break; } return !mk || domktime(t, type) != -1; } static int adjwday(struct tm *t, char type, int val, int istext, int mk) { if (val < 0) return 0; switch (type) { case '+': if (istext) if (val < t->tm_wday) val = 7 - t->tm_wday + val; /* early next week */ else val -= t->tm_wday; /* later this week */ else val *= 7; /* "-v+5w" == "5 weeks in the future" */ return !val || adjday(t, '+', val, mk); case '-': if (istext) { if (val > t->tm_wday) val = 7 - val + t->tm_wday; /* later last week */ else val = t->tm_wday - val; /* early this week */ } else val *= 7; /* "-v-5w" == "5 weeks ago" */ return !val || adjday(t, '-', val, mk); default: if (val < t->tm_wday) return adjday(t, '-', t->tm_wday - val, mk); else if (val > 6) return 0; else if (val > t->tm_wday) return adjday(t, '+', val - t->tm_wday, mk); } return 1; } static int adjhour(struct tm *t, char type, int val, int mk) { if (val < 0) return 0; switch (type) { case '+': if (val) { int days; days = (t->tm_hour + val) / 24; val %= 24; t->tm_hour += val; t->tm_hour %= 24; if (!adjday(t, '+', days, 0)) return 0; } break; case '-': if (val) { int days; days = val / 24; val %= 24; if (val > t->tm_hour) { days++; val -= 24; } t->tm_hour -= val; if (!adjday(t, '-', days, 0)) return 0; } break; default: if (val > 23) return 0; t->tm_hour = val; } return !mk || domktime(t, type) != -1; } static int adjmin(struct tm *t, char type, int val, int mk) { if (val < 0) return 0; switch (type) { case '+': if (val) { if (!adjhour(t, '+', (t->tm_min + val) / 60, 0)) return 0; val %= 60; t->tm_min += val; if (t->tm_min > 59) t->tm_min -= 60; } break; case '-': if (val) { if (!adjhour(t, '-', val / 60, 0)) return 0; val %= 60; if (val > t->tm_min) { if (!adjhour(t, '-', 1, 0)) return 0; val -= 60; } t->tm_min -= val; } break; default: if (val > 59) return 0; t->tm_min = val; } return !mk || domktime(t, type) != -1; } static int adjsec(struct tm *t, char type, int val, int mk) { if (val < 0) return 0; switch (type) { case '+': if (val) { if (!adjmin(t, '+', (t->tm_sec + val) / 60, 0)) return 0; val %= 60; t->tm_sec += val; if (t->tm_sec > 59) t->tm_sec -= 60; } break; case '-': if (val) { if (!adjmin(t, '-', val / 60, 0)) return 0; val %= 60; if (val > t->tm_sec) { if (!adjmin(t, '-', 1, 0)) return 0; val -= 60; } t->tm_sec -= val; } break; default: if (val > 59) return 0; t->tm_sec = val; } return !mk || domktime(t, type) != -1; } const struct vary * vary_apply(const struct vary *v, struct tm *t) { char type; char which; char *arg; size_t len; int val; for (; v; v = v->next) { type = *v->arg; arg = v->arg; if (type == '+' || type == '-') arg++; else type = '\0'; len = strlen(arg); if (len < 2) return v; if (type == '\0') t->tm_isdst = -1; if (strspn(arg, digits) != len-1) { val = trans(trans_wday, arg); if (val != -1) { if (!adjwday(t, type, val, 1, 1)) return v; } else { val = trans(trans_mon, arg); if (val != -1) { if (!adjmon(t, type, val, 1, 1)) return v; } else return v; } } else { val = atoi(arg); which = arg[len-1]; switch (which) { case 'S': if (!adjsec(t, type, val, 1)) return v; break; case 'M': if (!adjmin(t, type, val, 1)) return v; break; case 'H': if (!adjhour(t, type, val, 1)) return v; break; case 'd': t->tm_isdst = -1; if (!adjday(t, type, val, 1)) return v; break; case 'w': t->tm_isdst = -1; if (!adjwday(t, type, val, 0, 1)) return v; break; case 'm': t->tm_isdst = -1; if (!adjmon(t, type, val, 0, 1)) return v; break; case 'y': t->tm_isdst = -1; if (!adjyear(t, type, val, 1)) return v; break; default: return v; } } } return 0; } void vary_destroy(struct vary *v) { struct vary *n; while (v) { n = v->next; free(v); v = n; } } ================================================ FILE: core-tools/tests-regression/map-hierarchy/out.ok/map-hierarchy/in/b/bin/date/date.c ================================================ /*- * Copyright (c) 1985, 1987, 1988, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #ifndef lint static char const copyright[] = "@(#) Copyright (c) 1985, 1987, 1988, 1993\n\ The Regents of the University of California. All rights reserved.\n"; #endif /* not lint */ #include #include #include #include #include #include #include #include #include #include #include #include #include "vary.h" #ifndef TM_YEAR_BASE #define TM_YEAR_BASE 1900 #endif static time_t tval; int retval; static void setthetime(const char *, const char *, int, int); static void badformat(void); static void usage(void); int main(int argc, char *argv[]) { struct timezone tz; int ch, rflag; int jflag; const char *format; char buf[1024]; char *endptr, *fmt; char *tmp; int set_timezone; struct vary *v; const struct vary *badv; struct tm lt; v = NULL; fmt = NULL; (void) setlocale(LC_TIME, ""); tz.tz_dsttime = tz.tz_minuteswest = 0; rflag = 0; jflag = 0; set_timezone = 0; while ((ch = getopt(argc, argv, "d:f:jnr:t:uv:")) != -1) switch((char)ch) { case 'd': /* daylight savings time */ tz.tz_dsttime = strtol(optarg, &endptr, 10) ? 1 : 0; if (endptr == optarg || *endptr != '\0') usage(); set_timezone = 1; break; case 'f': fmt = optarg; break; case 'j': jflag = 1; /* don't set time */ break; case 'r': /* user specified seconds */ rflag = 1; tval = strtoq(optarg, &tmp, 0); if (*tmp != 0) usage(); break; case 't': /* minutes west of UTC */ /* error check; don't allow "PST" */ tz.tz_minuteswest = strtol(optarg, &endptr, 10); if (endptr == optarg || *endptr != '\0') usage(); set_timezone = 1; break; case 'u': /* do everything in UTC */ (void)setenv("TZ", "UTC0", 1); break; case 'v': v = vary_append(v, optarg); break; default: usage(); } argc -= optind; argv += optind; /* * If -d or -t, set the timezone or daylight savings time; this * doesn't belong here; the kernel should not know about either. */ if (set_timezone && settimeofday((struct timeval *)NULL, &tz)) err(1, "settimeofday (timezone)"); if (!rflag && time(&tval) == -1) err(1, "time"); format = "%+"; /* allow the operands in any order */ if (*argv && **argv == '+') { format = *argv + 1; ++argv; } if (*argv) { setthetime(fmt, *argv, jflag); ++argv; } else if (fmt != NULL) usage(); if (*argv && **argv == '+') format = *argv + 1; lt = *localtime(&tval); badv = vary_apply(v, <); if (badv) { fprintf(stderr, "%s: Cannot apply date adjustment\n", badv->arg); vary_destroy(v); usage(); } vary_destroy(v); (void)strftime(buf, sizeof(buf), format, <); (void)printf("%s\n", buf); if (fflush(stdout)) err(1, "stdout"); exit(retval); } #define ATOI2(s) ((s) += 2, ((s)[-2] - '0') * 10 + ((s)[-1] - '0')) static void setthetime(const char *fmt, const char *p, int jflag) { struct tm *lt; struct timeval tv; const char *dot, *t; int century; lt = localtime(&tval); lt->tm_isdst = -1; /* divine correct DST */ if (fmt != NULL) { t = strptime(p, fmt, lt); if (t == NULL) { fprintf(stderr, "Failed conversion of ``%s''" " using format ``%s''\n", p, fmt); badformat(); } else if (*t != '\0') fprintf(stderr, "Warning: Ignoring %ld extraneous" " characters in date string (%s)\n", (long) strlen(t), t); } else { for (t = p, dot = NULL; *t; ++t) { if (isdigit(*t)) continue; if (*t == '.' && dot == NULL) { dot = t; continue; } badformat(); } if (dot != NULL) { /* .ss */ dot++; /* *dot++ = '\0'; */ if (strlen(dot) != 2) badformat(); lt->tm_sec = ATOI2(dot); if (lt->tm_sec > 61) badformat(); } else lt->tm_sec = 0; century = 0; /* if p has a ".ss" field then let's pretend it's not there */ switch (strlen(p) - ((dot != NULL) ? 3 : 0)) { case 12: /* cc */ lt->tm_year = ATOI2(p) * 100 - TM_YEAR_BASE; century = 1; /* FALLTHROUGH */ case 10: /* yy */ if (century) lt->tm_year += ATOI2(p); else { lt->tm_year = ATOI2(p); if (lt->tm_year < 69) /* hack for 2000 ;-} */ lt->tm_year += 2000 - TM_YEAR_BASE; else lt->tm_year += 1900 - TM_YEAR_BASE; } /* FALLTHROUGH */ case 8: /* mm */ lt->tm_mon = ATOI2(p); if (lt->tm_mon > 12) badformat(); --lt->tm_mon; /* time struct is 0 - 11 */ /* FALLTHROUGH */ case 6: /* dd */ lt->tm_mday = ATOI2(p); if (lt->tm_mday > 31) badformat(); /* FALLTHROUGH */ case 4: /* HH */ lt->tm_hour = ATOI2(p); if (lt->tm_hour > 23) badformat(); /* FALLTHROUGH */ case 2: /* MM */ lt->tm_min = ATOI2(p); if (lt->tm_min > 59) badformat(); break; default: badformat(); } } /* convert broken-down time to GMT clock time */ if ((tval = mktime(lt)) == -1) errx(1, "nonexistent time"); if (!jflag) { /* set the time */ if ((p = getlogin()) == NULL) p = "???"; syslog(LOG_AUTH | LOG_NOTICE, "date set by %s", p); } } static void badformat(void) { warnx("illegal time format"); usage(); } static void usage(void) { (void)fprintf(stderr, "%s\n%s\n", "usage: date [-jnu] [-d dst] [-r seconds] [-t west] " "[-v[+|-]val[ymwdHMS]] ... ", " " "[-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]"); exit(1); } ================================================ FILE: core-tools/tests-regression/map-hierarchy/out.ok/map-hierarchy/in/b/bin/date/headers/vary.h ================================================ /*- * Copyright (c) 1997 Brian Somers * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD: src/bin/date/vary.h,v 1.4.56.1.8.2 2012/11/17 08:24:31 svnexp Exp $ */ struct vary { char *arg; struct vary *next; }; extern struct vary *vary_append(struct vary *v, char *arg); extern const struct vary *vary_apply(const struct vary *v, struct tm *t); extern void vary_destroy(struct vary *v); ================================================ FILE: core-tools/tests-regression/map-hierarchy/out.ok/map-hierarchy/in/b/bin/date/vary.c ================================================ /*- * Copyright (c) 1997 Brian Somers * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include #include #include #include #include #include "vary.h" struct trans { int val; const char *str; }; static struct trans trans_mon[] = { { 1, "january" }, { 2, "february" }, { 3, "march" }, { 4, "april" }, { 5, "may"}, { 6, "june" }, { 7, "july" }, { 8, "august" }, { 9, "september" }, { 10, "october" }, { 11, "november" }, { 12, "december" }, { -1, NULL } }; static struct trans trans_wday[] = { { 0, "sunday" }, { 1, "monday" }, { 2, "tuesday" }, { 3, "wednesday" }, { 4, "thursday" }, { 5, "friday" }, { 6, "saturday" }, { -1, NULL } }; static char digits[] = "0123456789"; static int adjhour(struct tm *, char, int, int); static int domktime(struct tm *t, char type) { time_t ret; while ((ret = mktime(t)) == -1 && t->tm_year > 68 && t->tm_year < 138) /* While mktime() fails, adjust by an hour */ adjhour(t, type == '-' ? type : '+', 1, 0); return ret; } static int trans(const struct trans t[], const char *arg) { int f; for (f = 0; t[f].val != -1; f++) if (!strncasecmp(t[f].str, arg, 3) || !strncasecmp(t[f].str, arg, strlen(t[f].str))) return t[f].val; return -1; } struct vary * vary_append(struct vary *v, char *arg) { struct vary *result, **nextp; if (v) { result = v; while (v->next) v = v->next; nextp = &v->next; } else nextp = &result; if ((*nextp = (struct vary *)malloc(sizeof(struct vary))) == NULL) err(1, "malloc"); (*nextp)->arg = arg; (*nextp)->next = NULL; return result; } static int mdays[12] = { 31, 0, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 }; static int daysinmonth(const struct tm *t) { int year; year = t->tm_year + 1900; if (t->tm_mon == 1) if (!(year % 400)) return 29; else if (!(year % 100)) return 28; else if (!(year % 4)) return 29; else return 28; else if (t->tm_mon >= 0 && t->tm_mon < 12) return mdays[t->tm_mon]; return 0; } static int adjyear(struct tm *t, char type, int val, int mk) { switch (type) { case '+': t->tm_year += val; break; case '-': t->tm_year -= val; break; default: t->tm_year = val; if (t->tm_year < 69) t->tm_year += 100; /* as per date.c */ else if (t->tm_year > 1900) t->tm_year -= 1900; /* struct tm holds years since 1900 */ break; } return !mk || domktime(t, type) != -1; } static int adjmon(struct tm *t, char type, int val, int istext, int mk) { int lmdays; if (val < 0) return 0; switch (type) { case '+': if (istext) { if (val <= t->tm_mon) val += 11 - t->tm_mon; /* early next year */ else val -= t->tm_mon + 1; /* later this year */ } if (val) { if (!adjyear(t, '+', (t->tm_mon + val) / 12, 0)) return 0; val %= 12; t->tm_mon += val; if (t->tm_mon > 11) t->tm_mon -= 12; } break; case '-': if (istext) { if (val-1 > t->tm_mon) val = 13 - val + t->tm_mon; /* later last year */ else val = t->tm_mon - val + 1; /* early this year */ } if (val) { if (!adjyear(t, '-', val / 12, 0)) return 0; val %= 12; if (val > t->tm_mon) { if (!adjyear(t, '-', 1, 0)) return 0; val -= 12; } t->tm_mon -= val; } break; default: if (val > 12 || val < 1) return 0; t->tm_mon = --val; } /* e.g., -v-1m on March, 31 is the last day of February in common sense */ lmdays = daysinmonth(t); if (t->tm_mday > lmdays) t->tm_mday = lmdays; return !mk || domktime(t, type) != -1; } static int adjday(struct tm *t, char type, int val, int mk) { int lmdays; switch (type) { case '+': while (val) { lmdays = daysinmonth(t); if (val > lmdays - t->tm_mday) { val -= lmdays - t->tm_mday + 1; t->tm_mday = 1; if (!adjmon(t, '+', 1, 0, 0)) return 0; } else { t->tm_mday += val; val = 0; } } break; case '-': while (val) if (val >= t->tm_mday) { val -= t->tm_mday; t->tm_mday = 1; if (!adjmon(t, '-', 1, 0, 0)) return 0; t->tm_mday = daysinmonth(t); } else { t->tm_mday -= val; val = 0; } break; default: if (val > 0 && val <= daysinmonth(t)) t->tm_mday = val; else return 0; break; } return !mk || domktime(t, type) != -1; } static int adjwday(struct tm *t, char type, int val, int istext, int mk) { if (val < 0) return 0; switch (type) { case '+': if (istext) if (val < t->tm_wday) val = 7 - t->tm_wday + val; /* early next week */ else val -= t->tm_wday; /* later this week */ else val *= 7; /* "-v+5w" == "5 weeks in the future" */ return !val || adjday(t, '+', val, mk); case '-': if (istext) { if (val > t->tm_wday) val = 7 - val + t->tm_wday; /* later last week */ else val = t->tm_wday - val; /* early this week */ } else val *= 7; /* "-v-5w" == "5 weeks ago" */ return !val || adjday(t, '-', val, mk); default: if (val < t->tm_wday) return adjday(t, '-', t->tm_wday - val, mk); else if (val > 6) return 0; else if (val > t->tm_wday) return adjday(t, '+', val - t->tm_wday, mk); } return 1; } static int adjhour(struct tm *t, char type, int val, int mk) { if (val < 0) return 0; switch (type) { case '+': if (val) { int days; days = (t->tm_hour + val) / 24; val %= 24; t->tm_hour += val; t->tm_hour %= 24; if (!adjday(t, '+', days, 0)) return 0; } break; case '-': if (val) { int days; days = val / 24; val %= 24; if (val > t->tm_hour) { days++; val -= 24; } t->tm_hour -= val; if (!adjday(t, '-', days, 0)) return 0; } break; default: if (val > 23) return 0; t->tm_hour = val; } return !mk || domktime(t, type) != -1; } static int adjmin(struct tm *t, char type, int val, int mk) { if (val < 0) return 0; switch (type) { case '+': if (val) { if (!adjhour(t, '+', (t->tm_min + val) / 60, 0)) return 0; val %= 60; t->tm_min += val; if (t->tm_min > 59) t->tm_min -= 60; } break; case '-': if (val) { if (!adjhour(t, '-', val / 60, 0)) return 0; val %= 60; if (val > t->tm_min) { if (!adjhour(t, '-', 1, 0)) return 0; val -= 60; } t->tm_min -= val; } break; default: if (val > 59) return 0; t->tm_min = val; } return !mk || domktime(t, type) != -1; } static int adjsec(struct tm *t, char type, int val, int mk) { if (val < 0) return 0; switch (type) { case '+': if (val) { if (!adjmin(t, '+', (t->tm_sec + val) / 60, 0)) return 0; val %= 60; t->tm_sec += val; if (t->tm_sec > 59) t->tm_sec -= 60; } break; case '-': if (val) { if (!adjmin(t, '-', val / 60, 0)) return 0; val %= 60; if (val > t->tm_sec) { if (!adjmin(t, '-', 1, 0)) return 0; val -= 60; } t->tm_sec -= val; } break; default: if (val > 59) return 0; t->tm_sec = val; } return !mk || domktime(t, type) != -1; } const struct vary * vary_apply(const struct vary *v, struct tm *t) { char type; char which; char *arg; size_t len; int val; for (; v; v = v->next) { type = *v->arg; arg = v->arg; if (type == '+' || type == '-') arg++; else type = '\0'; len = strlen(arg); if (len < 2) return v; if (type == '\0') t->tm_isdst = -1; if (strspn(arg, digits) != len-1) { val = trans(trans_wday, arg); if (val != -1) { if (!adjwday(t, type, val, 1, 1)) return v; } else { val = trans(trans_mon, arg); if (val != -1) { if (!adjmon(t, type, val, 1, 1)) return v; } else return v; } } else { val = atoi(arg); which = arg[len-1]; switch (which) { case 'S': if (!adjsec(t, type, val, 1)) return v; break; case 'M': if (!adjmin(t, type, val, 1)) return v; break; case 'H': if (!adjhour(t, type, val, 1)) return v; break; case 'd': t->tm_isdst = -1; if (!adjday(t, type, val, 1)) return v; break; case 'w': t->tm_isdst = -1; if (!adjwday(t, type, val, 0, 1)) return v; break; case 'm': t->tm_isdst = -1; if (!adjmon(t, type, val, 0, 1)) return v; break; case 'y': t->tm_isdst = -1; if (!adjyear(t, type, val, 1)) return v; break; default: return v; } } } return 0; } void vary_destroy(struct vary *v) { struct vary *n; while (v) { n = v->next; free(v); v = n; } } ================================================ FILE: core-tools/tests-regression/parallel-word-count/out.ok ================================================ 1 "'Challenger, 2 "A 1 "Ah, 1 "An 3 "And 1 "Anything 1 "Anything--anywhere--I 1 "As 6 "But 1 "Can't 1 "Challenger 1 "Challenger!" 1 "Challenger?" 1 "Come 1 "Come, 1 "DEAR 4 "Dear 1 "Did 2 "Do 3 "Don't 1 "ENMORE 1 "Entirely, 1 "Exactly. 1 "Expected?" 1 "FOR 1 "GEORGE 1 "General 1 "Get 1 "Gibberish!" 1 "Good 1 "Got 1 "HE 1 "Had 3 "He 1 "Here 1 "How 1 "How's 1 "How?" 21 "I 1 "I'd 1 "I'll 1 "I'm 1 "I've 1 "IT 1 "IT'S 1 "If 6 "It 5 "It's 1 "Look 1 "My 1 "Naturally," 1 "No 2 "No, 1 "No," 2 "Not 1 "Nothing 1 "Now 2 "Now, 1 "OUR 3 "Oh, 3 "One 1 "Outlines 1 "Pray 1 "Professor 1 "Protests," 1 "QUESTION!" 1 "Right!" 1 "SIR,--I 2 "Shall 1 "So 1 "Some 1 "Spirited 1 "Suppose," 2 "THE 1 "THERE 1 "THOSE 1 "TO-MORROW 1 "TRY 1 "Talking 1 "Tell 2 "That 1 "That's 2 "The 2 "Then 2 "There 1 "There's 1 "There, 1 "They 1 "This 1 "Tickled, 3 "To 1 "Try 1 "Tut, 1 "Undoubtedly." 1 "Uproar," 1 "WHO 1 "We 1 "We'll 1 "Weissmann 1 "Well 11 "Well, 1 "Well," 1 "Well?" 1 "Went 9 "What 2 "What's 2 "Why 1 "Why, 1 "Yes, 13 "You 1 "You're 1 "You've 1 "Yours 1 "besides, 1 "hardly 1 "only 3 "that 1 "what 1 "you 1 #139] 1 'I 1 'In 1 'Professor 1 'The 1 'em--a 1 'journalists.' 1 'special 1 'speculation' 1 (Wednesday) 2 *** 1 --"by 1 1863. 1 1892. 1 1893. 1 19, 1 1912 1 2008 6 A 1 ALL 1 ARE 1 ARTHUR 1 ASCII 1 Academy 1 Academy; 1 Act, 1 Address: 1 Adelphi 1 Afterwards 1 Ah, 1 Al 2 All 1 Alpine 2 America 1 America, 1 America." 3 American 5 And 1 And, 1 Anthropology 2 Anyhow, 1 Anyway, 1 Are 2 Arthur 3 As 1 Assistant, 1 Assistant-Keeper 1 Association'--so 1 Assyrian 2 At 1 Austin 3 Austin, 1 Austin. 1 Author: 1 B., 1 BIGGEST 1 Balkans 1 Beaumont 1 Began 1 Behold 1 Belge, 2 Besides, 1 Blundell 1 Blundell, 1 Blundell." 1 Born: 1 Boss. 3 British 1 Burton! 1 Burtons," 6 But 2 But, 1 But--oh, 2 By 2 C. 1 CHALLENGER" 1 CHALLENGER," 1 CHALLENGER." 4 CHAPTER 1 CONAN 1 CONQUESTS" 1 COPYRIGHT, 1 COULD 1 Cabinet. 1 Camberwell 1 Came 1 Catharine-wheel 1 Chain 1 Chairman" 1 Challenger 1 Challenger" 1 Challenger's 5 Challenger, 1 Challenger. 2 Challenger?" 1 Character 1 Chestnuts 1 Chinese 1 Clive--just 1 Club, 1 Comparative 2 Conan 1 Congress 1 Contents 1 Continental 1 Crayston 1 Creeping 2 D. 1 DEFINITE 1 DISAPPEAR 1 DOYLE 1 DREADFUL 1 Daily 1 Darwin 1 Darwin," 1 Darwinism, 1 Date: 1 Death 1 Department, 2 Did 1 Didn't 4 Do 3 Don't 2 Doyle 4 E. 1 EBOOK 1 EBook 2 EDWARD 1 EYES 1 Edinburgh 1 Educ.: 1 Edward. 1 Eh, 1 Employers' 4 English 1 English, 3 Enmore 3 Even 1 Evolution"; 1 Evolution. 1 Ex-President 1 Expensive, 1 FLAIL 1 FORESEEN 1 FOREST" 1 FORGET" 1 Fate 1 Following 1 For 1 Foreign 1 Foreword 1 Forfeit, 1 French 1 Frenchman 3 G. 1 GREAT 1 GUTENBERG 1 Gazette 1 Gazette, 1 Gazette,--perfectly 1 George 1 George! 1 Give 1 Gladys 1 Gladys! 5 Gladys, 1 Gladys. 1 Gladys? 1 Got 1 Gulf. 2 Gutenberg 1 H, 1 HAPPENED" 4 HAVE 1 HERO" 1 HEROISMS 1 HTML 2 Had 1 Haines. 1 Half 1 Has 1 Have 25 He 2 He's 2 Henry, 1 Henry. 1 Her 1 Hercules 1 Heroisms 4 His 3 How 1 However, 1 Hungerton, 199 I 2 I'd 7 I'll 7 I'm 4 I've 4 I, 5 I. 1 I; 1 II 1 II. 1 III 1 III. 1 IMPOSSIBLE 2 IN 1 INTO 1 IS 1 IT?" 1 IV. 1 IX. 9 If 1 Impossible 1 In 1 India! 1 Institute 1 Institute, 1 Institute. 2 Irish 1 Irish. 1 Is 19 It 4 It's 1 JUST 1 Judith 1 June 1 Just 1 Kalmuck 1 Kensington 1 Kensington, 1 Knowable, 1 LORD" 3 LOST 1 LUCK 1 La 1 Lady 1 Language: 1 Largs 1 Largs, 1 Liability 1 License 1 Lively 1 London 1 London--a 2 Look 1 Lord 1 Lord!" 4 Lost 1 Luck 1 MAKE 1 MALONE." 1 MOST 1 Macs 1 Malone 3 Malone, 2 Malone. 1 Malone?" 1 Masonic 1 May 2 McArdle 1 McArdle, 1 Medal 1 Member 1 Most 1 Move 8 Mr. 1 Munchausen--and 1 Museum 5 My 1 N. 1 NEVER 1 NEW 2 Nature, 1 Ned, 1 Ned," 1 Ned. 1 No, 1 November 3 Now, 3 OF 1 ONCE 1 OUTLYING 1 Observations 2 Of 2 Oh, 1 Olympian 1 Once 1 Or 1 PARK, 1 PERFECTLY 1 PERSON" 1 PICKETS 1 PROCESSION! 1 PROCESSION!" 2 PROFESSOR 1 PROJECT 1 Palaeontological 1 Park, 1 Park. 1 Park?" 1 Perfectly 1 Perhaps 1 Persian 1 Person" 1 Plata, 2 President 1 Pressman, 1 Pretend 1 Proceedings." 1 Produced 11 Professor 1 Professor!" 3 Professor. 1 Professor." 2 Project 1 Protest 1 Put 1 REAL 1 ROUND 1 Recreations: 1 Release 1 Remember 1 Research. 1 Resigned 1 Reuter's, 1 Richard 1 Round 1 Russia. 1 SEEN 1 SHALL 1 SIGHT 1 SIR 1 START 1 Saturday 1 Savage 1 Sciences, 1 Scotch 1 Section 1 Series 1 Several 13 She 1 Sir! 3 Sir, 1 Sir." 1 Skulls"; 2 So 1 Society. 3 Some 1 Somehow 1 Something 1 Sometimes 5 South 1 Southwark 1 Stanley! 1 Stanleys 1 Step 1 Such 1 Swollen 12 THE 1 THING 1 THINGS 1 THIS 2 Tarp 1 Telegraph?" 1 Terrace 6 That 2 That's 26 The 3 Then 6 There 2 There's 1 These 2 Think 6 This 1 Three 1 Time 1 Title: 1 To 1 UNKNOWN" 1 US" 1 University. 1 Upon 1 Us" 1 V. 1 VERY 1 VI. 1 VII. 1 VIII. 1 Vertebrate 2 Vienna, 3 Vienna. 1 W. 1 W.' 3 WAS 1 WE 1 WERE 1 WILL 1 WITH 1 WONDERFUL 1 WONDERS" 3 WORLD 2 WORLD" 2 Wadley 1 Wadley, 1 Wait 1 Walking, 2 Was 1 Wasn't 5 We 2 Wednesday 2 Weissmann 1 Weissmann. 1 Weissmannism," 4 Well, 1 West 7 What 2 When 1 Where 1 Why 1 Wigan 1 Winner 3 With 3 World 1 World, 1 Would 1 X. 1 XI. 1 XII. 1 XIII. 1 XIV. 1 XV. 1 XVI. 1 YOUR 1 Yes, 19 You 1 You'll 1 You're 3 You've 2 Your 5 Zoological 1 [EBook 170 a 1 able 1 abominably," 13 about 1 about." 1 about?" 3 above 1 absolutely 1 abusive 1 accent. 1 accepted 1 accessible? 1 account 1 acrimonious 3 across 1 act, 1 acting 1 action 1 adapt 1 added, 2 address 1 admirable 1 advance, 1 advance. 1 advanced 1 advancing 1 adventure 3 adventures 1 aeronaut, 1 afraid 7 after 1 after?" 1 afterwards 1 again. 1 again." 3 against 1 age 1 age, 1 age; 1 aggressive 1 ago. 1 ago; 1 agreeable 1 air. 1 alarmed, 1 alive 25 all 2 all, 1 all. 1 all." 1 allowances." 1 alluded 4 almost 1 alone 1 alone, 1 aloof! 2 aloud 1 alternating 1 although 6 always 13 am 1 am, 1 amazement 1 amiss 1 amplification 30 an 139 and 1 and, 1 and--oh, 1 anger 1 angular 1 animal 1 animals 1 announced 1 annoyed 2 another 4 answer 1 answered. 1 anxious 8 any 3 anyone 3 anything 1 anywhere 1 anywhere. 1 apart 1 apologize 2 appeal 1 appear 2 appearance 1 appearance, 1 appeared 1 appears 1 appointment, 1 appointment. 1 appointment." 1 approve, 1 architecture 1 ardent 23 are 2 are." 1 argue 1 argument, 1 argument. 1 arm-chair 2 arms 1 arnica." 1 around 1 article 35 as 1 ashamed" 4 ask 2 asked 2 asked, 1 asked. 1 asks 1 assault 1 assaults 1 assertion 1 associate 1 assurances 2 assure 1 assured. 1 asthmatic. 47 at 1 atmosphere 1 atmosphere." 1 attack 1 attacked 1 attempt 1 attention 1 attitude 1 audacity. 1 authority. 1 averaged. 1 averted 1 awaits 2 aware 1 aware," 2 away 1 away--his 1 awfully 6 back 1 back. 1 backing 1 bacteriologist 1 bacteriologist, 2 bad 2 bald 1 balloon. 1 barbed-wire 1 barrel 1 battery 1 battles, 34 be 1 be,--envied 1 beard 1 beard, 1 beast. 3 beautiful 1 beauty, 4 because 16 been 5 before 1 before, 1 before?" 1 began: 1 begins, 1 behaved 1 behavior. 2 behind 4 being 1 being. 3 believe 1 believe, 1 believe." 1 believed 1 believes 1 bell 1 bellow 1 bellowing, 1 bent 1 beside 2 best 1 best. 1 bestial 7 better 1 between 2 beyond 4 big 1 bimetallism, 1 biography, 1 bit, 5 black 1 blackened 2 blame 2 blank 1 blast 1 blew 1 blowing 1 blue 1 blue, 1 blue-gray 1 bodies 1 bolted 2 book 1 books, 1 boss; 3 both 1 bottom, 1 bounced 1 bounded 1 bow 1 bowdlerized 2 boy 1 boy, 1 boy. 1 boyish 1 brace 1 brackets 2 brain. 1 brain? 1 brave 1 breadth, 2 break 1 breaking 1 breaks 1 breast, 1 breath 1 breathed 1 brethren? 1 briefly:-- 1 bright, 1 bring 1 broad 2 broke 1 bronze 1 bronzed 1 brother. 1 brought 1 brown 1 brown, 1 brows 1 brutal 1 bucking 1 bull's 1 bull; 1 bully!" 1 business 1 business, 1 business. 1 business." 31 but 1 butcher's 1 butlers. 1 butt 16 by 1 by. 4 call 1 call, 2 called 1 calling 1 calm. 3 came 21 can 6 can't 2 cannot 2 cantankerous 2 care 1 carried 1 case 1 caught 1 caused 1 censor 1 center 1 centered 1 certain 1 certainly 7 chair 2 chair, 1 chair. 1 champion 1 chance, 1 chance,--at 1 chance. 1 chances 1 chances, 1 chances. 2 chapter 1 character, 1 character. 1 character?" 1 charge, 1 chauffeur, 1 checking 1 chest 1 chest. 1 chirrup 1 choke-damp?" 1 choked 1 chosen--Tarp 1 circulation. 3 claim 1 clear 1 clear, 1 clearly 1 clerk, 1 clever 1 climbing. 1 clipped. 1 closed 1 club. 1 clumped 1 coal 1 cock-and-bull 1 cockatoo 1 cold 1 colleagues. 1 collected. 1 colliery 1 coloring, 6 come 1 come! 1 come!" 1 come, 1 come. 1 come?" 2 comes 1 comes." 1 command 1 comment 1 comments 1 companions, 1 company, 1 compliment 2 compliments 1 composition!" 1 comradeship 1 conceited 1 conclusions 1 conditions 1 conducted 1 confession. 1 confidence 1 confidence. 2 connection 1 conquered 1 conscience 2 conscious 1 consent 1 consent, 1 considerate 1 constant 1 content 1 contentious, 1 contents 1 context 2 control 1 conversation. 1 conversaziones 1 convey 1 conveyed 1 convinced 1 convinces 1 cooed. 2 copy 1 copy, 1 copy." 1 correspondence 1 cost 27 could 4 course, 2 covered 1 crabbed, 1 cranial 1 crawling 1 creature, 1 creatures. 1 cried 3 cried, 3 cried. 1 crisis 1 critical 2 critical, 1 criticism 1 crying 1 cunning 1 curtail 1 curtain. 1 curving 1 cuticura 1 damaged 1 damnedest 1 dancing 1 danger 2 dangerous 1 dangerous, 1 dangerous--really 1 dangerous--ring 3 dare 2 dark 1 dark-eyed 1 date 3 day 1 day, 2 days 2 dear 1 debts 1 decency 1 deception 2 deed 1 deeds 1 deeds." 1 deep 1 deeper." 2 definite 1 degree. 1 deigns 1 deliberate 1 delicate 1 delicately 1 demonstrate 1 dependent 1 depreciation 1 depth, 1 descreeptive 1 described 1 desire 1 desire, 1 desires 1 desk 1 desk. 1 destined 1 detached 2 determination 1 devil.'" 1 diagrams. 5 did 1 did, 1 did." 2 didn't 1 differences 1 different 1 different. 1 difficult 1 difficult. 1 difficulty 1 dignity 1 dining-room 1 directness 1 disapproval. 1 discovered. 1 discovery 1 discreditable 1 discretion 1 discuss 1 discussion 1 distance. 1 distasteful 1 distinguish 2 distrust 19 do 1 do, 1 do----" 2 do. 1 do." 1 do? 2 do?" 7 does 1 dogmatic 3 doing 1 doing, 1 don 9 don't 3 done 1 done. 1 done?" 3 door 3 door, 2 door. 1 doubly 14 down 1 down! 1 down!" 1 down. 1 drawer. 1 drawing 1 dress 1 dressing 2 drew 1 dried-up 1 driven 2 driving 1 dry, 1 duly 1 during 1 duty 2 eBook 1 each 1 each--that 1 eager 1 early 1 earnest 1 earnestness. 1 ears 1 earth,--a 1 ease 1 edge 1 editor 1 editor, 1 education 1 education, 1 effervescence. 1 effervescing 1 effort 1 egg?" 1 eh?" 1 either 1 either," 1 elaborate 1 elaborated 1 elapse 1 elasticity 1 electric 1 eleven 1 eleven, 1 else 1 else," 1 else. 1 emphatic. 1 encoding: 1 encouraging 2 end 1 end. 1 endorse 1 endorsement 1 energy. 1 enormous 1 enormous, 4 enough 1 enough?" 2 entered 1 entered, 1 enthusiasm 3 entirely 2 envelope 2 envelope. 1 envied 1 especially 1 established 1 establishing 1 etc. 1 etc., 1 evasion 1 even 4 evening 6 ever 5 every 1 everyone 1 everything 1 everything, 1 evidence 1 evidence. 1 evidence?" 1 evident 1 evidently 1 exactly 1 exalted. 1 exceeding 1 excellent. 1 exchange. 1 exists. 1 expect 1 expedeetion 1 expense 1 experience 1 experienced 1 experiences. 1 explained, 1 exploits, 2 explosion 1 exposed 1 exposing 1 express 2 expression 1 expression. 1 exquisite 1 extraordinary 1 extreme 1 eye 1 eye, 4 eye. 4 eyes 1 eyes, 3 eyes. 13 face 1 face, 1 face. 1 face: 1 fact 1 factor?" 1 fad?" 1 faddist, 1 faddist. 2 failed 1 fair 1 fairly 1 faithfully, 1 fakes. 1 faking 1 fallacy 1 faltering 2 famous 1 fanatic 1 fancies. 1 fang 3 far 1 fashion, 1 fashion. 1 father, 1 father-in-law. 1 fatuous 3 favor 1 favor." 1 favorable 3 fear 1 fearful 1 feathery, 1 feeble 3 feel 2 feeling 1 feet 1 feet, 1 fell 2 fellow 1 fellow, 1 fellow," 1 fellow-reporters 3 felt 1 fever 2 few 3 fifteen 1 fifth. 1 fifty 1 fight 1 fight, 1 figure--these, 1 filed 2 filled 4 find 1 fine. 1 fingers 1 fingers, 1 finished. 2 fire. 6 first 1 fists 1 flashed 1 flatter 1 flock 1 florid, 1 fluff; 1 fluffy, 1 foggy 1 folk 3 follow 1 followed 1 follows:-- 1 fool, 1 foolish 1 football 47 for 1 for? 1 force 2 forehead. 1 forewarned 1 forfeit! 1 forlorn 1 former 1 formidable 1 forth 1 forth. 1 fortunately 2 forward 1 forward, 4 found 1 fourth 1 frame 2 frank 1 frank, 1 fraud--a 1 friend's 1 friendly--or 1 friends, 1 friends; 1 friendship 1 fringed 20 from 3 front 1 frontiersman 1 fronting 1 fugitive 3 full 2 full, 1 full-charged 1 furniture." 1 fury, 1 gaiters. 1 gale 1 game, 1 gaps 1 gas-bags! 1 gasp. 3 gathered 5 gave 1 gaze 1 gazed 3 general 1 generations.' 1 genius 1 gentle, 1 gentleman. 1 genuine 1 germ 10 get 1 gibberish! 1 gingery 2 girl's 8 give 1 given. 1 gives 1 glad 1 glad--if 1 glad--so 1 glared 1 gloried 1 glories 1 glorification? 1 glow 1 glowing 4 go 1 goes 1 goes." 2 going 1 goings-on," 3 gone 9 good 3 good, 1 good-bye. 1 good-natured, 3 got 1 gracious, 1 grasp 5 great 1 great, 1 greater, 1 gregarious 1 grimly. 1 grinning 1 gripped 1 grunt 1 guaranteed 1 guidance, 1 gutter. 1 habit 1 habitual 39 had 1 had, 1 hadn't 1 hair 1 hair, 1 hair. 3 half 1 half-past 2 hall 1 halls, 2 hand 1 hand--such 2 hand. 1 hand; 1 handed 1 handiwork. 1 handled 1 handling 3 hands 1 handwriting 1 happen 1 happened--or 1 happiest 2 hard 1 hard, 1 hard; 1 harder, 3 hardly 1 hardness, 10 has 1 hated 63 have 1 have,--youth, 1 haven't 1 having 69 he 1 he's 2 he, 2 he. 5 head 2 head, 2 head. 1 heading, 1 health, 2 hear 1 hear, 3 heard 4 heart 1 heated 1 heaves 1 heavily 1 heavily-curtained 1 height 4 help 1 help-mate. 1 helped 13 her 1 her! 1 herd 3 here 3 here, 1 heritage 1 heroic 2 heroisms 1 hers 1 higher 19 him 7 him, 1 him," 12 him. 1 him." 1 him? 2 him?" 1 himself 1 himself. 63 his 1 historical 1 hog." 2 hold 1 holes, 1 homicidal 1 honest 2 honor 1 honor." 1 honored 4 hope 1 hope; 1 hoped 1 hopelessly 5 hour 1 hours, 1 house 2 house. 1 hovering 6 how 2 however, 1 however--namely: 1 howl 2 huge 1 hulking 2 human 1 humanity. 1 humble 1 humbly, 1 humor. 1 humorous 1 hundred 1 hurting 1 husband 1 husband? 1 id 1 idea 1 idea, 1 idea. 1 ideal 1 ideal. 16 if 1 ignominious. 1 ignorance 1 ignore. 1 ill-conditioned 1 image, 1 imagine 1 imagined 1 immediate 1 impediment 2 imposing 3 impossible 1 impossible. 1 imposter 1 impression 94 in 1 in! 1 in!" 1 in, 1 in. 1 inches 2 inclined 1 included 1 including 1 incredible 1 indeed! 1 indeed?" 1 index 1 indication 1 inexpressibly 4 infernal 1 inherited 1 injunction 1 injured 1 inner 1 inquirer. 1 insignificant 2 insisted 1 insisted, 1 insolent 1 inspiration. 1 inspirer 1 instantly 1 instead 1 instinct. 1 instincts 2 insufferable 1 intelligence 1 intercourse----' 1 interest 1 interest. 1 international 1 interrupted 1 intertwined, 4 interview 2 interview, 2 interview. 13 into 1 intruded 1 intrusive 1 invent 1 irksome 50 is 1 is! 1 is, 1 isn't 2 isolated 83 it 2 it! 1 it's 8 it, 1 it," 1 it----" 7 it. 6 it." 1 it; 2 it? 1 it?" 4 its 2 jacket 1 job 1 journalist, 1 joy 1 judged 1 judice?" 1 jump 1 jumped 4 just 1 justifies 1 justify 3 keep 5 kind 3 kindly 1 kindly, 1 kiss 1 knew 1 knew. 6 know 1 know, 1 know. 2 know." 1 know? 1 knows, 1 lady 1 lady, 2 lady. 1 laid 1 lamp. 1 land 1 large 1 largest 7 last 2 last. 1 late." 2 later 1 latest 1 latter 2 laughed 1 laughing 1 laughing. 1 layman." 1 lead 2 leaned 1 leaning 1 learned 3 least 1 least, 1 least. 1 leather 1 leathery 1 leave 1 lecture, 2 led 1 left 1 leg 1 legs 2 length 1 less 1 less, 2 let 5 letter 1 letter--nothing 1 letter?" 1 level 1 levity, 1 liar 1 liar!" 1 liar, 1 libel 1 liberty 2 lie 1 lieutenant, 4 life 1 life, 1 life." 1 lift. 1 lifted 1 light 14 like 1 like. 1 like?" 2 liked 1 likely 1 limit. 1 line 1 link 1 lips,--all 1 lips. 1 liquid 1 list 1 listened 7 little 1 little. 1 live 1 lived 1 lived. 1 loafers 1 locked, 1 lonely 3 long 1 long, 1 longer 1 longer, 4 look 10 looked 1 looks 1 lose 2 lost 1 lot 4 love 2 love! 1 love!" 3 love, 1 love. 1 love." 1 loved, 1 lover 1 lucid 2 luck 1 luck, 1 lucky 1 mad 1 madam, 1 madam." 5 made 1 magnetism, 1 maid, 1 mail, 1 majesty 7 make 2 makes 2 making 1 malice, 24 man 1 man!" 3 man's 8 man, 1 man----" 1 man-servant, 1 man. 1 man." 2 map 1 maps, 1 margin 1 mark 1 marry 1 marry, 1 massive 2 master. 1 masterful. 1 masterly 1 match 1 matchwood 2 matter 2 matter. 1 matter; 1 matters 1 mature 11 may 1 maybe, 33 me 2 me! 1 me!" 12 me, 2 me," 1 me--you 10 me. 2 me." 1 me?" 3 mean 1 mean. 1 mean." 1 meaning 1 meant 1 meaty 1 meesion 1 meesion' 1 meeting, 1 meeting. 1 meeting.' 1 megalomaniac 2 memory 5 men 1 men. 1 menaces 2 menacing 1 mend 1 mentioned 1 mere 3 merely 1 message, 1 message: 3 met 1 methods 1 microcosm 1 microscope. 2 middle 7 might 1 might, 1 miles 2 mind 1 mind, 1 mind. 1 mind? 1 mine. 1 mine." 1 minutes. 1 mission 2 modern 2 modify 3 moment 3 moment, 1 moment," 1 money 1 monotonous 1 month 1 month, 16 more 1 more, 1 morning--if 1 morning. 8 most 1 motive! 1 motive, 1 moustache 1 mouth 1 move 5 much 1 much, 1 murmured 1 murmured. 10 must 1 must--you, 1 must." 70 my 4 myself 2 myself. 1 myself." 1 mystic 1 nails, 1 naked 3 name 1 named, 1 narrative 1 narrative; 2 natural 1 nature, 1 nature. 1 nearer. 2 nearly 1 necessary. 2 need 1 needed 1 needs 1 neglected, 13 never 1 new 2 news 1 newspaper 1 next 1 nicer 1 night, 1 nine-hundred-diameter 11 no 1 no; 1 noble 2 nobody 2 nodded 1 none 1 nonsense 33 not 1 not." 1 not? 3 not?" 1 note, 1 notebook 1 notebook. 1 notes 5 nothing 1 nothing. 1 nothing." 1 notice 1 noticed 1 notions 1 notorious 3 now 2 now, 1 now?" 1 numerous 1 o'clock 1 occasion 1 odd, 1 odious 155 of 1 of'--well, 4 off 2 off, 1 off." 3 offensive 3 office 1 office. 1 official 1 often 1 oily 5 old 1 old, 1 omnipotent, 17 on 1 on!" 1 on!--'Publications: 2 on, 1 once 8 one 1 one's 1 oneself. 1 online 14 only 3 open 3 opened 2 opening 2 opening. 1 opinion 1 opinion, 22 or 1 order. 1 oriental 1 original." 2 other 1 others 1 otherwise 2 ought 6 our 13 out 1 out, 1 out. 1 outlined 1 oval, 9 over 1 over-accentuated? 1 over. 1 overpowering 7 own 1 oyster. 1 panted. 2 paper 1 paper. 1 paper? 1 papers, 2 part 1 parthenogenetic 1 particular 1 particular," 1 parts 1 pass 1 passage 2 passage. 1 passing 1 passion 1 passion. 1 past. 1 pathetically, 1 payment 1 peculiar 1 peculiar, 1 peculiarly 2 people 1 people, 1 people--seempathy, 4 perfectly 2 perhaps 1 perhaps, 1 permission, 4 person 1 person--absolutely 3 personal 1 personality 1 persuasive 1 photographs 1 photographs, 1 pick 1 pilot 2 pink 1 pity 1 pity! 3 place 1 place, 1 place. 1 plain 1 plan 1 plasm 1 plastered 2 play 1 played 1 pleaded. 1 pleasant! 3 please 1 pleasure 2 plunged 1 pocketed 1 point 1 point, 1 point?" 1 pointing 1 points 1 police-court 2 policeman 1 policeman, 3 policeman. 1 poor 1 porticoed 1 position 1 position?" 1 possessed 1 possible 1 possible, 1 possibly 1 postmark 1 pound 1 practise 1 praise 1 precaution 1 prepared 1 presence, 1 presence. 2 present 1 presentiment 2 presents 1 pressed 1 pride 1 primitive 1 probable 1 proceedings 1 proceedings, 1 produced 1 producing 1 profile 2 profound 1 projecting. 1 proper 1 proportion. 1 propose 1 propose, 1 propose?" 1 proposed 1 proposition 1 prosaic 1 protest 1 proud 1 proud, 2 prove?" 1 proved 1 proves," 2 public 1 publication 1 pursuing 1 pushed 4 put 1 putting 1 quality. 1 quarrelsome, 1 queer 1 quest 1 questions, 1 quickly 5 quite 1 quote 1 race 1 radiated 1 rage 1 railing. 1 railings 1 rank 1 rarefied 1 rascals 8 rather 1 rational 1 raven 1 re-reading 1 re-reading----" 1 re-use 5 read 1 reader 2 ready 3 real 1 realized 1 realized. 6 really 1 really, 1 reasonable 2 received 1 recently 1 record. 1 recriminations, 3 red 1 red-headed 1 reflected 1 reflects 1 refresh 1 refuse 1 refused 2 regard 1 relations 1 relented. 1 remain, 2 remark 2 remember 1 remind 1 reply, 1 reporters 1 representative 1 reproof. 1 reproved 1 repulse 1 repulsed 1 rescued 1 research, 1 reserve 1 respect, 2 rested 1 restraint 1 restrictions 1 result 1 result, 1 resumed 1 retracted 1 reward 1 rewards. 1 rideeculous? 2 right. 1 righteous 1 rippling 1 risk 1 risk. 1 river. 1 roared, 1 roaring, 1 rolled 2 romance 5 room 1 room, 1 rotating 1 rough, 9 round 1 round-backed, 2 row 1 rudeness 1 ruined 1 rumbled. 1 rumbling 1 run 1 run, 1 run: 1 rupee, 1 rush 1 rushed 1 sadly 1 safe. 1 safely 23 said 4 said, 4 said. 4 same 1 sanctum, 1 sanely 3 sat 1 satisfied 1 satisfy 1 save 3 saw 7 say 2 say, 1 say?" 1 says--I'm 1 scandal 1 scandal, 1 science 2 science. 5 scientific 1 scrawled 1 scribblers, 1 searching 1 seat 2 seated 1 secret 8 see 1 see, 7 seem 4 seemed 3 seems 2 seen 1 self, 1 self-evident 1 self. 1 selfishness, 2 send 1 sense 1 sent 3 sentence 1 separate 1 series 1 serious 1 seriously, 1 serve 3 set 1 settled 1 several 1 severe 1 severely, 1 severely. 1 sex 1 shaken 1 shaking 6 shall 1 shape 1 sharp, 11 she 2 she. 1 shield 4 short 20 should 1 shoulder--a 1 shoulders 1 shoulders. 1 shouldn't 2 show 1 shunned 1 shut 1 side-pockets 1 sight 1 signal 1 signals 1 signs 1 silence, 2 silly 1 silver, 2 simple 1 simultaneously, 1 since 1 since. 1 single 1 sinister 1 sinned 4 sir, 4 sir," 1 sir--entirely!" 1 sir--scientific 2 sir. 1 sir." 1 sir?" 2 sit 1 situation. 2 size 1 skeleton 1 skin, 1 skull 1 slip 1 slipped 2 slowly 2 small 1 smaller 1 smile 1 smile, 1 smile. 1 smiled 1 smiling 1 snapped 1 sneer. 22 so 1 so, 1 so." 1 soldier 1 solitary 20 some 2 somebody 1 somersault 7 something 1 something, 1 something. 1 somewhat 1 soon 1 sorely? 1 sorry 6 sort 1 soul! 1 soul, 1 sound, 2 spaces 1 spade-shaped 1 speak 1 spectacles 1 speculations 1 speech 1 spirit, 1 spite 1 splendid 1 split 1 spoil 1 spoiled 1 spoke 1 spoke. 1 sportsman 1 sprang 1 spread 1 springing 2 sprung 1 spun 2 staff 1 stairs. 3 stand 1 standards 1 stare. 1 staring 1 started 1 starting. 1 startled. 1 state 2 statement 1 statement? 1 station-master." 1 station. 1 statue 1 stealthy 1 steely 1 stepped 1 steps 1 steps. 1 sterner 1 stigmata 3 still 1 still, 1 stone, 1 stoop 1 stooping 1 stopped 1 stopped, 1 story 1 story?" 2 strange 1 strange, 1 street, 1 street. 1 strength, 1 strengthen 1 strengthens 1 strikes 1 striking 2 strongly 1 student 1 student," 1 students, 1 study 1 stuff, 1 stunted 2 sub 1 sub-human 4 subject 1 subject, 2 subject. 1 succession 13 such 3 sudden 1 suddenly 1 suggestion 1 suggestions 1 suit 1 suit--that's 1 summary 1 superman. 1 support 4 suppose 1 suppose, 1 supposeetion. 1 suppressed 3 sure 1 sure, 1 surely!" 1 suspect. 1 suspense 1 suspicion 1 swarthy, 1 sweetness 1 swine 1 sympathy. 1 table 2 table, 1 taciturn 1 tact 1 tactless 1 tactlessness 11 take 1 take, 2 taken 5 talk 1 talked?" 2 talking 1 tall, 1 tap 1 taxicab 1 telegony 7 tell 1 ten 1 tended 1 tension 2 terms 1 terrible 11 than 1 thanked 82 that 1 that! 2 that's 2 that," 1 that--or 4 that. 2 that?" 275 the 3 their 2 them 1 them, 1 themselves 1 then! 1 then, 1 then. 2 then?" 1 theosophist, 11 there 1 there's 2 there, 3 there. 3 these 1 thesis." 7 they 2 thin, 3 thing 1 thing. 2 things 1 things, 6 think 1 think, 1 thinking 19 this 1 this. 1 this? 1 thoroughly 3 those 2 though 1 though!" 6 thought 1 thoughtfully 3 thoughts 1 thousand, 2 three 1 three-and-twenty 1 three-quarter 7 through 2 thrown 1 tickled." 1 tickling. 1 till 4 time 2 time, 1 time." 1 timidity 161 to 1 to--well, 1 to-morrow 1 to-night. 1 to-night." 1 to. 1 toes 1 together 1 together. 1 token 1 told 1 tome 4 too 6 took 1 top-hat, 2 touch 1 touch. 1 touchy 1 towards 1 tram 1 transfixed 1 translate 1 translation 1 translation." 1 treason. 1 tremendous 1 trend 1 tried 1 triumphantly. 1 trotted, 2 trouble 3 true 1 truly, 1 trust 5 try 1 try. 1 tufts, 2 turn 1 turn. 2 turned 2 turning 1 tut! 1 twenty-four 1 twice 1 twilight 1 twinkled 8 two 1 type--'Societe 1 type. 1 unable 1 unawares? 1 unbend 1 uncertain 4 under 1 underlying 3 understand 1 understand?" 1 understanding 1 undoubtedly 1 uneasy 1 unit, 1 unpleasant 1 unprintable." 1 unreservedly 1 unscrupulous 1 unsexual. 1 unshrinking 1 untidy 3 until 14 up 1 up, 1 up. 1 up." 24 upon 1 upon,--what 1 upturned 4 us 1 us, 4 us. 3 use 1 usually 2 vague 1 vaguely 1 valuable. 1 value 1 vegetarian, 1 velvet 1 venture. 2 ventured 1 ventures 1 vermin, 2 version 1 versus 19 very 1 very--very 1 victory 1 view 1 view, 1 views 1 views, 1 views. 1 vile, 1 violence 1 violence, 1 violent 1 violent, 2 violent. 1 visit, 1 visitors 1 visits 1 vitality 2 vitality, 1 vivacious, 1 voice 1 voice, 1 voice. 1 wail 5 wait 1 wait, 1 waited 1 waiting 1 wakens 1 walked 1 walked, 1 wall. 1 walnut 4 want 1 want----" 2 want. 1 wanted 1 wants. 1 warm 1 warned 1 warning." 83 was 1 was! 2 was, 1 was--and 2 wasn't 1 watchful 1 waving 3 way 3 way, 12 we 1 we? 1 wealth 2 week 1 well," 1 well. 6 went 16 were 17 what 1 what's 1 what?" 1 whatever 1 whatsoever. 1 wheezing 11 when 2 where 1 where. 1 whether 36 which 1 whim. 1 white 1 white!--and 23 who 2 who's 1 who, 1 who--who 2 whole 1 whole-hearted 2 whom 2 whose 4 why 1 wicked 1 wide 1 wife's 10 will 1 wincing 1 wind 1 wind; 1 windows 2 wish 1 wisp 1 wistful 73 with 1 with--with 1 withdrawn 3 within 1 within, 2 without 1 wits, 6 woman 1 woman, 1 womanly 4 women 1 won 3 won't 1 won; 1 wonder 2 wonder. 3 wonderful 1 wondering 1 wooden 4 word 2 word, 1 word,--teetotal, 1 words 1 words. 1 wore. 1 work 6 world 1 world, 1 worship 1 worst 1 worth 2 worthy 28 would 1 wouldn't 1 wouldn't; 2 write 1 written 3 wrong 1 wrought 1 www.gutenberg.net 2 year. 2 years 1 yes, 1 yes; 8 yet 1 yet!" 2 yet." 100 you 1 you! 1 you!" 3 you'll 6 you, 4 you. 5 you." 2 you? 2 you?" 6 young 34 your 1 yours 1 yourself, 1 yourself. 1 yourself?" 1 youthful 1 zoologist! ================================================ FILE: core-tools/tests-regression/regression/errors/stream-scatter-cycle.ok ================================================ The following dependencies across streams form a cycle: head /stream/d tail /stream/b ================================================ FILE: core-tools/tests-regression/regression/errors/stream-scatter-cycle.sh ================================================ #!/usr/bin/env sgsh # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # scatter |{ .| head /stream/d |>/stream/b .| tail /stream/b |>/stream/d |} gather |{ |} ================================================ FILE: core-tools/tests-regression/regression/errors/unsafe-gather.ok ================================================ test/regression/errors/unsafe-gather.sh(29): warning: Unsafe use of pass-through /stream/b in the gather section test/regression/errors/unsafe-gather.sh(29): error: Consult the DEADLOCK section of the manual page ================================================ FILE: core-tools/tests-regression/regression/errors/unsafe-gather.sh ================================================ #!/usr/bin/env sgsh # # SYNOPSIS Highlight misspelled words # DESCRIPTION # Highlight the words that are misspelled in the command's standard # input # Demonstrates stream buffering. # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # scatter |{ -| cat |>/stream/a -||>/stream/b |} gather |{ cat /stream/a cat /stream/b |} ================================================ FILE: core-tools/tests-regression/regression/errors/unsafe-gather2.ok ================================================ test/regression/errors/unsafe-gather2.sh(24): warning: Unsafe use of pass-through /stream/b in the gather section test/regression/errors/unsafe-gather2.sh(24): error: Consult the DEADLOCK section of the manual page ================================================ FILE: core-tools/tests-regression/regression/errors/unsafe-gather2.sh ================================================ #!/usr/bin/env sgsh # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # scatter |{ -| cat |>/stream/a -||>/stream/b |} gather |{ # Join is unsafe in this context join /stream/a /dev/null join /stream/b /dev/null |} ================================================ FILE: core-tools/tests-regression/regression/graphs/NMRPipe.ok ================================================ digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; node_tee_0 [id="node_tee_0", label="sgsh-tee ", shape="ellipse"]; node_cmd_0_0_0 [id="node_cmd_0_0_0", label="nmrPipe |\lnmrPipe -fn SOL |\lnmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -c 0.5 |\lnmrPipe -fn ZF -auto |\lnmrPipe -fn FT |\lnmrPipe -fn PS -p0 177 -p1 0.0 -di |\lnmrPipe -fn EXT -left -sw -verb |\lnmrPipe -fn TP |\lnmrPipe -fn COADD -cList 1 0 -time |\lnmrPipe -fn SP -off 0.5 -end 0.98 -pow 1 -c 0.5 |\lnmrPipe -fn ZF -auto |\lnmrPipe -fn FT |\lnmrPipe -fn PS -p0 0 -p1 0 -di |\lnmrPipe -fn TP |\lnmrPipe -fn POLY -auto -verb >A\l", shape="ellipse"]; node_cmd_0_1_0 [id="node_cmd_0_1_0", label="nmrPipe |\lnmrPipe -fn SOL |\lnmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -c 0.5 |\lnmrPipe -fn ZF -auto |\lnmrPipe -fn FT |\lnmrPipe -fn PS -p0 177 -p1 0.0 -di |\lnmrPipe -fn EXT -left -sw -verb |\lnmrPipe -fn TP |\lnmrPipe -fn COADD -cList 0 1 -time |\lnmrPipe -fn SP -off 0.5 -end 0.98 -pow 1 -c 0.5 |\lnmrPipe -fn ZF -auto |\lnmrPipe -fn FT |\lnmrPipe -fn PS -p0 -90 -p1 0 -di |\lnmrPipe -fn TP |\lnmrPipe -fn POLY -auto -verb >B\l", shape="ellipse"]; node_tee_0 -> node_cmd_0_1_0 [id="npi-0.1.0"]; node_tee_0 -> node_cmd_0_0_0 [id="npi-0.0.0"]; } ================================================ FILE: core-tools/tests-regression/regression/graphs/code-metrics.ok ================================================ digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; "FNAMELEN" [id="store:FNAMELEN", shape="box"]; node_cmd_1_0_0 -> "FNAMELEN" [id="nps-FNAMELEN"]; "NSTRUCT" [id="store:NSTRUCT", shape="box"]; node_cmd_3_0_0 -> "NSTRUCT" [id="nps-NSTRUCT"]; "NTYPEDEF" [id="store:NTYPEDEF", shape="box"]; node_cmd_3_1_0 -> "NTYPEDEF" [id="nps-NTYPEDEF"]; "NVOID" [id="store:NVOID", shape="box"]; node_cmd_3_2_0 -> "NVOID" [id="nps-NVOID"]; "NGETS" [id="store:NGETS", shape="box"]; node_cmd_3_3_0 -> "NGETS" [id="nps-NGETS"]; "IDLEN" [id="store:IDLEN", shape="box"]; node_cmd_3_4_0 -> "IDLEN" [id="nps-IDLEN"]; "CHLINESCHAR" [id="store:CHLINESCHAR", shape="box"]; node_cmd_2_1_0 -> "CHLINESCHAR" [id="nps-CHLINESCHAR"]; "NCCHAR" [id="store:NCCHAR", shape="box"]; node_cmd_2_2_0 -> "NCCHAR" [id="nps-NCCHAR"]; "NCOMMENT" [id="store:NCOMMENT", shape="box"]; node_cmd_2_3_0 -> "NCOMMENT" [id="nps-NCOMMENT"]; "NCOPYRIGHT" [id="store:NCOPYRIGHT", shape="box"]; node_cmd_2_4_0 -> "NCOPYRIGHT" [id="nps-NCOPYRIGHT"]; "NCFILE" [id="store:NCFILE", shape="box"]; node_cmd_5_0_0 -> "NCFILE" [id="nps-NCFILE"]; "NCDIR" [id="store:NCDIR", shape="box"]; node_cmd_5_1_0 -> "NCDIR" [id="nps-NCDIR"]; "CLINESCHAR" [id="store:CLINESCHAR", shape="box"]; node_cmd_6_0_0 -> "CLINESCHAR" [id="nps-CLINESCHAR"]; "NFUNCTION" [id="store:NFUNCTION", shape="box"]; node_cmd_7_0_0 -> "NFUNCTION" [id="nps-NFUNCTION"]; "NGOTO" [id="store:NGOTO", shape="box"]; node_cmd_7_1_0 -> "NGOTO" [id="nps-NGOTO"]; "NREGISTER" [id="store:NREGISTER", shape="box"]; node_cmd_7_2_0 -> "NREGISTER" [id="nps-NREGISTER"]; "NMACRO" [id="store:NMACRO", shape="box"]; node_cmd_7_3_0 -> "NMACRO" [id="nps-NMACRO"]; "NINCLUDE" [id="store:NINCLUDE", shape="box"]; node_cmd_7_4_0 -> "NINCLUDE" [id="nps-NINCLUDE"]; "NCONST" [id="store:NCONST", shape="box"]; node_cmd_7_5_0 -> "NCONST" [id="nps-NCONST"]; "NHFILE" [id="store:NHFILE", shape="box"]; node_cmd_0_2_0 -> "NHFILE" [id="nps-NHFILE"]; node_cmd_0_0_0 [id="node_cmd_0_0_0", label="find \"$@\" \\( -name \\*.c -or -name \\*.h \\) -type f -print0\l", shape="ellipse"]; node_cmd_1_0_0 [id="node_cmd_1_0_0", label="tr \\\\0 \\\\n |\l# Remove path\lsed 's|^.*/||' |\l# Maintain average\lawk '{s += length($1); n++} END {print s / n}'\l", shape="ellipse"]; node_cmd_1_1_0 [id="node_cmd_1_1_0", label="xargs -0 cat\l", shape="ellipse"]; node_cmd_2_0_0 [id="node_cmd_2_0_0", label="sed 's/#/@/g;s/\\\\[\\\\\"'\\'']/@/g;s/\"[^\"]*\"/\"\"/g;'\"s/'[^']*'/''/g\" |\lcpp -P 2>/dev/null\l", shape="ellipse"]; node_cmd_3_0_0 [id="node_cmd_3_0_0", label="egrep -c 'struct[ ]*{|struct[ ]*[a-zA-Z_][a-zA-Z0-9_]*[ ]*{'\l", shape="ellipse"]; node_cmd_3_1_0 [id="node_cmd_3_1_0", label="grep -cw typedef\l", shape="ellipse"]; node_cmd_3_2_0 [id="node_cmd_3_2_0", label="grep -cw void\l", shape="ellipse"]; node_cmd_3_3_0 [id="node_cmd_3_3_0", label="grep -cw gets\l", shape="ellipse"]; node_cmd_3_4_0 [id="node_cmd_3_4_0", label="tr -cs 'A-Za-z0-9_' '\\n' |\lsort -u |\lawk '/^[A-Za-z]/ { len += length($1); n++ } END {print len / n}'\l", shape="ellipse"]; node_cmd_2_1_0 [id="node_cmd_2_1_0", label="wc -lc | awk '{OFS=\":\"; print $1, $2}'\l", shape="ellipse"]; node_cmd_2_2_0 [id="node_cmd_2_2_0", label="sed 's/#/@/g' |\lcpp -traditional -P 2>/dev/null |\lwc -c |\lawk '{OFMT = \"%.0f\"; print $1/1000}'\l", shape="ellipse"]; node_cmd_2_3_0 [id="node_cmd_2_3_0", label="egrep -c '/\\*|//'\l", shape="ellipse"]; node_cmd_2_4_0 [id="node_cmd_2_4_0", label="grep -ci copyright\l", shape="ellipse"]; node_cmd_0_1_0 [id="node_cmd_0_1_0", label="find \"$@\" -name \\*.c -type f -print0\l", shape="ellipse"]; node_cmd_4_0_0 [id="node_cmd_4_0_0", label="tr \\\\0 \\\\n\l", shape="ellipse"]; node_cmd_5_0_0 [id="node_cmd_5_0_0", label="wc -l\l", shape="ellipse"]; node_cmd_5_1_0 [id="node_cmd_5_1_0", label="sed 's,/[^/]*$,,;s,^.*/,,' | sort -u | wc -l\l", shape="ellipse"]; node_cmd_4_1_0 [id="node_cmd_4_1_0", label="xargs -0 cat\l", shape="ellipse"]; node_cmd_6_0_0 [id="node_cmd_6_0_0", label="wc -lc | awk '{OFS=\":\"; print $1, $2}'\l", shape="ellipse"]; node_cmd_6_1_0 [id="node_cmd_6_1_0", label="sed 's/#/@/g;s/\\\\[\\\\\"'\\'']/@/g;s/\"[^\"]*\"/\"\"/g;'\"s/'[^']*'/''/g\" |\lcpp -P 2>/dev/null\l", shape="ellipse"]; node_cmd_7_0_0 [id="node_cmd_7_0_0", label="grep -c '^{'\l", shape="ellipse"]; node_cmd_7_1_0 [id="node_cmd_7_1_0", label="grep -cw goto\l", shape="ellipse"]; node_cmd_7_2_0 [id="node_cmd_7_2_0", label="grep -cw register\l", shape="ellipse"]; node_cmd_7_3_0 [id="node_cmd_7_3_0", label="grep -c '@[ ]*define[ ][ ]*[a-zA-Z_][a-zA-Z0-9_]*('\l", shape="ellipse"]; node_cmd_7_4_0 [id="node_cmd_7_4_0", label="grep -c '@[ ]*include'\l", shape="ellipse"]; node_cmd_7_5_0 [id="node_cmd_7_5_0", label="grep -o -h '[0-9][x0-9][0-9a-f]*' | wc -l\l", shape="ellipse"]; node_cmd_0_2_0 [id="node_cmd_0_2_0", label="find \"$@\" -name \\*.h -type f | wc -l\l", shape="ellipse"]; FNAMELEN -> gather_node_0 [id="store-read"]; NSTRUCT -> gather_node_0 [id="store-read"]; NTYPEDEF -> gather_node_0 [id="store-read"]; IDLEN -> gather_node_0 [id="store-read"]; CHLINESCHAR -> gather_node_0 [id="store-read"]; NCCHAR -> gather_node_0 [id="store-read"]; NCOMMENT -> gather_node_0 [id="store-read"]; NCOPYRIGHT -> gather_node_0 [id="store-read"]; NCFILE -> gather_node_0 [id="store-read"]; NCDIR -> gather_node_0 [id="store-read"]; CLINESCHAR -> gather_node_0 [id="store-read"]; NFUNCTION -> gather_node_0 [id="store-read"]; NGOTO -> gather_node_0 [id="store-read"]; NREGISTER -> gather_node_0 [id="store-read"]; NMACRO -> gather_node_0 [id="store-read"]; NINCLUDE -> gather_node_0 [id="store-read"]; NCONST -> gather_node_0 [id="store-read"]; NVOID -> gather_node_0 [id="store-read"]; NHFILE -> gather_node_0 [id="store-read"]; NGETS -> gather_node_0 [id="store-read"]; gather_node_0 [id="gather_node_0", label="cat < node_cmd_7_5_0 [id="npi-7.5.0"]; node_cmd_4_0_0 -> node_cmd_5_1_0 [id="npi-5.1.0"]; node_cmd_2_0_0 -> node_cmd_3_1_0 [id="npi-3.1.0"]; node_cmd_0_0_0 -> node_cmd_1_1_0 [id="npi-1.1.0"]; node_cmd_2_0_0 -> node_cmd_3_3_0 [id="npi-3.3.0"]; node_cmd_1_1_0 -> node_cmd_2_4_0 [id="npi-2.4.0"]; node_cmd_0_0_0 -> node_cmd_1_0_0 [id="npi-1.0.0"]; node_cmd_2_0_0 -> node_cmd_3_0_0 [id="npi-3.0.0"]; node_cmd_4_1_0 -> node_cmd_6_1_0 [id="npi-6.1.0"]; node_cmd_4_0_0 -> node_cmd_5_0_0 [id="npi-5.0.0"]; node_cmd_6_1_0 -> node_cmd_7_2_0 [id="npi-7.2.0"]; node_cmd_1_1_0 -> node_cmd_2_2_0 [id="npi-2.2.0"]; node_cmd_2_0_0 -> node_cmd_3_2_0 [id="npi-3.2.0"]; node_cmd_0_1_0 -> node_cmd_4_0_0 [id="npi-4.0.0"]; node_cmd_6_1_0 -> node_cmd_7_1_0 [id="npi-7.1.0"]; node_cmd_1_1_0 -> node_cmd_2_3_0 [id="npi-2.3.0"]; node_cmd_4_1_0 -> node_cmd_6_0_0 [id="npi-6.0.0"]; node_cmd_6_1_0 -> node_cmd_7_4_0 [id="npi-7.4.0"]; node_cmd_0_1_0 -> node_cmd_4_1_0 [id="npi-4.1.0"]; node_cmd_1_1_0 -> node_cmd_2_0_0 [id="npi-2.0.0"]; node_cmd_6_1_0 -> node_cmd_7_3_0 [id="npi-7.3.0"]; node_cmd_1_1_0 -> node_cmd_2_1_0 [id="npi-2.1.0"]; node_cmd_2_0_0 -> node_cmd_3_4_0 [id="npi-3.4.0"]; node_cmd_6_1_0 -> node_cmd_7_0_0 [id="npi-7.0.0"]; } ================================================ FILE: core-tools/tests-regression/regression/graphs/commit-stats.ok ================================================ digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; node_tee_0 [id="node_tee_0", label="sgsh-tee ", shape="ellipse"]; node_cmd_0_0_0 [id="node_cmd_0_0_0", label="awk -F: '{print $1}' | forder\l", shape="ellipse"]; node_cmd_0_1_0 [id="node_cmd_0_1_0", label="awk -F: '{print substr($2, 1, 3)}' | forder\l", shape="ellipse"]; gather_node_1 [id="gather_node_1", label="cat /stream/authors\l", shape="ellipse"]; gather_node_3 [id="gather_node_3", label="cat /stream/days\l", shape="ellipse"]; node_tee_0 -> node_cmd_0_1_0 [id="npi-0.1.0"]; node_cmd_0_0_0 -> gather_node_1 [id="npfo-authors.0"]; node_cmd_0_1_0 -> gather_node_3 [id="npfo-days.0"]; node_tee_0 -> node_cmd_0_0_0 [id="npi-0.0.0"]; } ================================================ FILE: core-tools/tests-regression/regression/graphs/committer-plot.ok ================================================ digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; node_tee_0 [id="node_tee_0", label="sgsh-tee ", shape="ellipse"]; "NCOMMITTERS" [id="store:NCOMMITTERS", shape="box"]; node_cmd_0_0_0 -> "NCOMMITTERS" [id="nps-NCOMMITTERS"]; "LAST" [id="store:LAST", shape="box"]; node_cmd_0_1_0 -> "LAST" [id="nps-LAST"]; "FIRST" [id="store:FIRST", shape="box"]; node_cmd_0_2_0 -> "FIRST" [id="nps-FIRST"]; "NDAYS" [id="store:NDAYS", shape="box"]; node_cmd_0_3_0 -> "NDAYS" [id="nps-NDAYS"]; node_cmd_0_0_0 [id="node_cmd_0_0_0", label="awk '{print $2}' | sort -u | wc -l\l", shape="ellipse"]; node_cmd_0_1_0 [id="node_cmd_0_1_0", label="tail -1 | awk '{print $1}'\l", shape="ellipse"]; node_cmd_0_2_0 [id="node_cmd_0_2_0", label="head -1 | awk '{print $1}'\l", shape="ellipse"]; LAST -> node_cmd_0_3_0 [id="store-read"]; FIRST -> node_cmd_0_3_0 [id="store-read"]; node_cmd_0_3_0 [id="node_cmd_0_3_0", label="expr \\( `store:LAST` - `store:FIRST` \\) / 60 / 60 / 24\l", shape="ellipse"]; NCOMMITTERS -> node_cmd_0_4_0 [id="store-read"]; node_cmd_0_4_0 [id="node_cmd_0_4_0", label="awk '{print $2}' |\lsort |\luniq -c |\lsort -n |\lawk 'BEGIN {l = 0; r = '`store:NCOMMITTERS`';}\l{print NR % 2 ? l++ : --r, $2}' |\lsort -k2\l", shape="ellipse"]; NCOMMITTERS -> node_cmd_0_5_0 [id="store-read"]; NDAYS -> node_cmd_0_5_0 [id="store-read"]; NCOMMITTERS -> node_cmd_0_5_0 [id="store-read"]; node_cmd_0_5_0 [id="node_cmd_0_5_0", label="sort -k2 |\ljoin -j 2 - /stream/committerpos |\l# Order by time\lsort -k 2n |\l{\l# Create portable bitmap\lecho 'P1'\lecho \"`store:NCOMMITTERS` `store:NDAYS`\"\lperl -na -e '\lBEGIN { @empty['`store:NCOMMITTERS`' - 1] = 0; @committers = @empty; }\lsub out { print join(\"\", map($_ ? \"1\" : \"0\", @committers)), \"\\n\"; }\l$day = int($F[1] / 60 / 60 / 24);\l$pday = $day if (!defined($pday));\lwhile ($day != $pday) {\lout();\l@committers = @empty;\l$pday++;\l}\l$committers[$F[2]] = 1;\lEND { out(); }\l'\l} |\l# Enlarge points into discs through morphological convolution\lpgmmorphconv -erode <(\lcat < node_cmd_0_1_0 [id="npi-0.1.0"]; node_cmd_0_5_0 -> node_cmd_1_0_0 [id="npi-1.0.0"]; node_cmd_0_4_0 -> node_cmd_0_5_0 [id="npfo-committerpos.0"]; node_tee_0 -> node_cmd_0_5_0 [id="npi-0.5.0"]; node_tee_0 -> node_cmd_0_4_0 [id="npi-0.4.0"]; node_tee_0 -> node_cmd_0_2_0 [id="npi-0.2.0"]; node_tee_0 -> node_cmd_0_0_0 [id="npi-0.0.0"]; node_cmd_0_5_0 -> node_cmd_1_1_0 [id="npi-1.1.0"]; } ================================================ FILE: core-tools/tests-regression/regression/graphs/compress-compare.ok ================================================ digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; node_tee_0 [id="node_tee_0", label="sgsh-tee ", shape="ellipse"]; "NBYTES" [id="store:NBYTES", shape="box"]; node_cmd_0_0_0 -> "NBYTES" [id="nps-NBYTES"]; "FILETYPE" [id="store:FILETYPE", shape="box"]; node_cmd_0_1_0 -> "FILETYPE" [id="nps-FILETYPE"]; "XZ" [id="store:XZ", shape="box"]; node_cmd_0_2_0 -> "XZ" [id="nps-XZ"]; "BZIP2" [id="store:BZIP2", shape="box"]; node_cmd_0_3_0 -> "BZIP2" [id="nps-BZIP2"]; "GZIP" [id="store:GZIP", shape="box"]; node_cmd_0_4_0 -> "GZIP" [id="nps-GZIP"]; node_cmd_0_0_0 [id="node_cmd_0_0_0", label="wc -c\l", shape="ellipse"]; node_cmd_0_1_0 [id="node_cmd_0_1_0", label="file -\l", shape="ellipse"]; node_cmd_0_2_0 [id="node_cmd_0_2_0", label="xz -c | wc -c\l", shape="ellipse"]; node_cmd_0_3_0 [id="node_cmd_0_3_0", label="bzip2 -c | wc -c\l", shape="ellipse"]; node_cmd_0_4_0 [id="node_cmd_0_4_0", label="gzip -c | wc -c\l", shape="ellipse"]; FILETYPE -> gather_node_0 [id="store-read"]; NBYTES -> gather_node_0 [id="store-read"]; GZIP -> gather_node_0 [id="store-read"]; BZIP2 -> gather_node_0 [id="store-read"]; XZ -> gather_node_0 [id="store-read"]; gather_node_0 [id="gather_node_0", label="cat < node_cmd_0_1_0 [id="npi-0.1.0"]; node_tee_0 -> node_cmd_0_3_0 [id="npi-0.3.0"]; node_tee_0 -> node_cmd_0_4_0 [id="npi-0.4.0"]; node_tee_0 -> node_cmd_0_2_0 [id="npi-0.2.0"]; node_tee_0 -> node_cmd_0_0_0 [id="npi-0.0.0"]; } ================================================ FILE: core-tools/tests-regression/regression/graphs/dir.ok ================================================ digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; node_tee_0 [id="node_tee_0", label="sgsh-tee ", shape="ellipse"]; "NFILES" [id="store:NFILES", shape="box"]; node_cmd_0_1_0 -> "NFILES" [id="nps-NFILES"]; "NDIRS" [id="store:NDIRS", shape="box"]; node_cmd_0_2_0 -> "NDIRS" [id="nps-NDIRS"]; "NBYTES" [id="store:NBYTES", shape="box"]; node_cmd_0_3_0 -> "NBYTES" [id="nps-NBYTES"]; node_cmd_0_0_0 [id="node_cmd_0_0_0", label="awk '!/^total/ {print $6, $7, $8, $1, sprintf(\"%8d\", $5), $9}'\l", shape="ellipse"]; node_cmd_0_1_0 [id="node_cmd_0_1_0", label="wc -l\l", shape="ellipse"]; node_cmd_0_2_0 [id="node_cmd_0_2_0", label="grep -c '^d'\l", shape="ellipse"]; node_cmd_0_3_0 [id="node_cmd_0_3_0", label="awk '{s += $5} END {print s}'\l", shape="ellipse"]; gather_node_0 [id="gather_node_0", label="cat /stream/files\l", shape="ellipse"]; NFILES -> gather_node_1 [id="store-read"]; NBYTES -> gather_node_1 [id="store-read"]; gather_node_1 [id="gather_node_1", label="echo \" `store:NFILES` File(s) `store:NBYTES` bytes\"\l", shape="ellipse"]; NDIRS -> gather_node_2 [id="store-read"]; gather_node_2 [id="gather_node_2", label="echo \" `store:NDIRS` Dir(s) $FREE bytes free\"\l", shape="ellipse"]; node_tee_0 -> node_cmd_0_1_0 [id="npi-0.1.0"]; node_cmd_0_0_0 -> gather_node_0 [id="npfo-files.0"]; node_tee_0 -> node_cmd_0_3_0 [id="npi-0.3.0"]; node_tee_0 -> node_cmd_0_2_0 [id="npi-0.2.0"]; node_tee_0 -> node_cmd_0_0_0 [id="npi-0.0.0"]; } ================================================ FILE: core-tools/tests-regression/regression/graphs/duplicate-files.ok ================================================ digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; node_tee_0 [id="node_tee_0", label="sgsh-tee ", shape="ellipse"]; node_cmd_0_0_0 [id="node_cmd_0_0_0", label="awk '{print $2}' | uniq -d\l", shape="ellipse"]; gather_node_1 [id="gather_node_1", label="join -2 2 /stream/dupes /stream/names |\l# Output same files on a single line\lawk '\l", shape="ellipse"]; node_cmd_0_0_0 -> gather_node_1 [id="npfo-dupes.0"]; node_tee_0 -> gather_node_1 [id="npfo-names.0"]; node_tee_0 -> node_cmd_0_0_0 [id="npi-0.0.0"]; } ================================================ FILE: core-tools/tests-regression/regression/graphs/ft2d.ok ================================================ digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; node_cmd_0_0_0 [id="node_cmd_0_0_0", label="sfspike n1=64 n2=64 d1=1 d2=1 nsp=2 k1=16,17 k2=5,5 mag=16,16 \\\llabel1='time' label2='space' unit1= unit2= |\lsfsmooth rect2=2 |\lsfsmooth rect2=2\l", shape="ellipse"]; node_cmd_1_0_0 [id="node_cmd_1_0_0", label="sfgrey pclip=100 wanttitle=n\l", shape="ellipse"]; node_cmd_1_1_0 [id="node_cmd_1_1_0", label="sffft1 | sffft3 axis=2 pad=1 | sfreal\l", shape="ellipse"]; node_cmd_2_0_0 [id="node_cmd_2_0_0", label="sgsh-tee -I\l", shape="ellipse"]; node_cmd_2_1_0 [id="node_cmd_2_1_0", label="sfwindow f1=1 |\lsfreverse which=3 |\lsfcat axis=1 /stream/ft2d |\lsfgrey pclip=100 wanttitle=n \\\llabel1=\"1/time\" label2=\"1/space\"\l", shape="ellipse"]; node_cmd_0_1_0 [id="node_cmd_0_1_0", label="side_by_side_iso /stream/pulse.vpl /stream/ft2d.vpl \\\lyscale=1.25 >Fig/ft2dofpulse.vpl\l", shape="ellipse"]; node_cmd_2_0_0 -> node_cmd_2_1_0 [id="npfo-ft2d.0"]; node_cmd_0_0_0 -> node_cmd_1_0_0 [id="npi-1.0.0"]; node_cmd_1_1_0 -> node_cmd_2_0_0 [id="npi-2.0.0"]; node_cmd_0_0_0 -> node_cmd_1_1_0 [id="npi-1.1.0"]; node_cmd_1_1_0 -> node_cmd_2_1_0 [id="npi-2.1.0"]; node_cmd_1_0_0 -> node_cmd_0_1_0 [id="npfo-pulse.vpl.0"]; node_cmd_2_1_0 -> node_cmd_0_1_0 [id="npfo-ft2d.vpl.0"]; } digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; node_cmd_0_0_0 [id="node_cmd_0_0_0", label="sfspike n1=64 d1=1 o1=32 nsp=4 k1=1,2,3,4 mag=1,3,3,1 \\\llabel1='time' unit1= |\lsfspray n=32 d=1 o=0 |\lsfput label2=space |\lsflmostretch delay=0 v0=-1\l", shape="ellipse"]; node_cmd_1_0_0 [id="node_cmd_1_0_0", label="sgsh-tee -I\l", shape="ellipse"]; node_cmd_1_1_0 [id="node_cmd_1_1_0", label="sfwindow f2=1 |\lsfreverse which=2 |\lsfcat axis=2 /stream/air\l", shape="ellipse"]; node_cmd_2_0_0 [id="node_cmd_2_0_0", label="sfgrey pclip=100 wanttitle=n\l", shape="ellipse"]; node_cmd_2_1_0 [id="node_cmd_2_1_0", label="sffft1 |\lsffft3 sign=1\l", shape="ellipse"]; node_cmd_3_0_0 [id="node_cmd_3_0_0", label="sfreal\l", shape="ellipse"]; node_cmd_3_1_0 [id="node_cmd_3_1_0", label="sfimag\l", shape="ellipse"]; node_cmd_0_1_0 [id="node_cmd_0_1_0", label="sfmath re=/stream/airftr im=/stream/airfti output=\"sqrt(re*re+im*im)\"\l", shape="ellipse"]; node_cmd_4_0_0 [id="node_cmd_4_0_0", label="sgsh-tee -I\l", shape="ellipse"]; node_cmd_4_1_0 [id="node_cmd_4_1_0", label="sfwindow f1=1 |\lsfreverse which=3 |\lsfcat axis=1 /stream/airft1 |\lsfgrey pclip=100 wanttitle=n label1=\"1/time\" \\\llabel2=\"1/space\"\l", shape="ellipse"]; node_cmd_0_2_0 [id="node_cmd_0_2_0", label="side_by_side_iso /stream/airtx.vpl /stream/airfk.vpl \\\lyscale=1.25 >Fig/airwave.vpl\l", shape="ellipse"]; node_cmd_2_1_0 -> node_cmd_3_1_0 [id="npi-3.1.0"]; node_cmd_0_0_0 -> node_cmd_1_1_0 [id="npi-1.1.0"]; node_cmd_1_0_0 -> node_cmd_0_1_0 [id="npfo-pulse.vpl.0"]; node_cmd_2_1_0 -> node_cmd_0_1_0 [id="npfo-ft2d.vpl.0"]; node_cmd_0_0_0 -> node_cmd_1_0_0 [id="npi-1.0.0"]; node_cmd_2_1_0 -> node_cmd_3_0_0 [id="npi-3.0.0"]; node_cmd_2_0_0 -> node_cmd_0_2_0 [id="npfo-airtx.vpl.0"]; node_cmd_4_0_0 -> node_cmd_4_1_0 [id="npfo-airft1.0"]; node_cmd_3_1_0 -> node_cmd_0_1_0 [id="npfo-airfti.0"]; node_cmd_0_1_0 -> node_cmd_4_0_0 [id="npi-4.0.0"]; node_cmd_4_1_0 -> node_cmd_0_2_0 [id="npfo-airfk.vpl.0"]; node_cmd_3_0_0 -> node_cmd_0_1_0 [id="npfo-airftr.0"]; node_cmd_0_1_0 -> node_cmd_4_1_0 [id="npi-4.1.0"]; node_cmd_2_0_0 -> node_cmd_2_1_0 [id="npfo-ft2d.0"]; node_cmd_1_1_0 -> node_cmd_2_0_0 [id="npi-2.0.0"]; node_cmd_1_1_0 -> node_cmd_2_1_0 [id="npi-2.1.0"]; node_cmd_1_0_0 -> node_cmd_1_1_0 [id="npfo-air.0"]; } ================================================ FILE: core-tools/tests-regression/regression/graphs/map-hierarchy.ok ================================================ digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; node_cmd_0_0_0 [id="node_cmd_0_0_0", label="line_signatures $1\l", shape="ellipse"]; node_cmd_0_1_0 [id="node_cmd_0_1_0", label="line_signatures $2\l", shape="ellipse"]; node_cmd_0_2_0 [id="node_cmd_0_2_0", label="join -t -1 2 -2 2 /stream/a /stream/b |\l# Print filename dir1 dir2\lsed 's///g' |\lawk -F 'BEGIN{OFS=\" \"}{print $1, $3, $4}' |\l# Unique occurrences\lsort -u\l", shape="ellipse"]; node_cmd_1_0_0 [id="node_cmd_1_0_0", label="awk '{print \"cp \\\"\" $2 \"/\" $1 \"\\\" \\\"'$NEWDIR'/\" $3 \"/\" $1 \"\\\"\"}'\l", shape="ellipse"]; node_cmd_1_1_0 [id="node_cmd_1_1_0", label="awk '{print \"mkdir -p \\\"'$NEWDIR'/\" $3 \"\\\"\"}' | sort -u\l", shape="ellipse"]; gather_node_1 [id="gather_node_1", label="cat /stream/mkdir /stream/cp |\lsh\l", shape="ellipse"]; node_cmd_1_1_0 -> gather_node_1 [id="npfo-mkdir.0"]; node_cmd_0_2_0 -> node_cmd_1_0_0 [id="npi-1.0.0"]; node_cmd_0_0_0 -> node_cmd_0_2_0 [id="npfo-a.0"]; node_cmd_1_0_0 -> gather_node_1 [id="npfo-cp.0"]; node_cmd_0_2_0 -> node_cmd_1_1_0 [id="npi-1.1.0"]; node_cmd_0_1_0 -> node_cmd_0_2_0 [id="npfo-b.0"]; } ================================================ FILE: core-tools/tests-regression/regression/graphs/parallel-logresolve.ok ================================================ digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; node_tee_0 [id="node_tee_0", label="sgsh-tee -s", shape="ellipse"]; node_cmd_0_0_0 [id="node_cmd_0_0_0", label="logresolve\l", shape="ellipse"]; node_cmd_0_0_1 [id="node_cmd_0_0_1", label="logresolve\l", shape="ellipse"]; node_cmd_0_0_2 [id="node_cmd_0_0_2", label="logresolve\l", shape="ellipse"]; node_cmd_0_0_3 [id="node_cmd_0_0_3", label="logresolve\l", shape="ellipse"]; node_cmd_0_0_4 [id="node_cmd_0_0_4", label="logresolve\l", shape="ellipse"]; node_cmd_0_0_5 [id="node_cmd_0_0_5", label="logresolve\l", shape="ellipse"]; node_cmd_0_0_6 [id="node_cmd_0_0_6", label="logresolve\l", shape="ellipse"]; node_cmd_0_0_7 [id="node_cmd_0_0_7", label="logresolve\l", shape="ellipse"]; node_cmd_0_0_8 [id="node_cmd_0_0_8", label="logresolve\l", shape="ellipse"]; node_cmd_0_0_9 [id="node_cmd_0_0_9", label="logresolve\l", shape="ellipse"]; gather_node_1 [id="gather_node_1", label="sort -m -k2n /stream/resolved |\l# Remove second field\lcut -d ' ' -f 1,3-\l", shape="ellipse"]; node_cmd_0_0_0 -> gather_node_1 [id="npfo-resolved.0"]; node_tee_0 -> node_cmd_0_0_2 [id="npi-0.0.2"]; node_tee_0 -> node_cmd_0_0_7 [id="npi-0.0.7"]; node_tee_0 -> node_cmd_0_0_3 [id="npi-0.0.3"]; node_cmd_0_0_7 -> gather_node_1 [id="npfo-resolved.7"]; node_tee_0 -> node_cmd_0_0_9 [id="npi-0.0.9"]; node_cmd_0_0_8 -> gather_node_1 [id="npfo-resolved.8"]; node_cmd_0_0_6 -> gather_node_1 [id="npfo-resolved.6"]; node_cmd_0_0_2 -> gather_node_1 [id="npfo-resolved.2"]; node_cmd_0_0_9 -> gather_node_1 [id="npfo-resolved.9"]; node_tee_0 -> node_cmd_0_0_1 [id="npi-0.0.1"]; node_tee_0 -> node_cmd_0_0_8 [id="npi-0.0.8"]; node_cmd_0_0_3 -> gather_node_1 [id="npfo-resolved.3"]; node_cmd_0_0_4 -> gather_node_1 [id="npfo-resolved.4"]; node_cmd_0_0_5 -> gather_node_1 [id="npfo-resolved.5"]; node_tee_0 -> node_cmd_0_0_0 [id="npi-0.0.0"]; node_tee_0 -> node_cmd_0_0_6 [id="npi-0.0.6"]; node_tee_0 -> node_cmd_0_0_5 [id="npi-0.0.5"]; node_tee_0 -> node_cmd_0_0_4 [id="npi-0.0.4"]; node_cmd_0_0_1 -> gather_node_1 [id="npfo-resolved.1"]; } ================================================ FILE: core-tools/tests-regression/regression/graphs/spell-highlight.ok ================================================ digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; node_tee_0 [id="node_tee_0", label="sgsh-tee ", shape="ellipse"]; node_cmd_0_0_0 [id="node_cmd_0_0_0", label="sort /usr/share/dict/words\l", shape="ellipse"]; node_cmd_0_1_0 [id="node_cmd_0_1_0", label="tr -cs A-Za-z \\\\n |\ltr A-Z a-z |\lsort -u |\lcomm -23 - /stream/dict\l", shape="ellipse"]; node_cmd_0_2_0 [id="node_cmd_0_2_0", label="cat\l", shape="ellipse"]; gather_node_0 [id="gather_node_0", label="fgrep -f /stream/errors -i --color -w -C 2 /stream/text\l", shape="ellipse"]; node_tee_0 -> node_cmd_0_1_0 [id="npi-0.1.0"]; node_tee_0 -> node_cmd_0_2_0 [id="npi-0.2.0"]; node_cmd_0_2_0 -> gather_node_0 [id="npfo-text.0"]; node_cmd_0_0_0 -> node_cmd_0_1_0 [id="npfo-dict.0"]; node_cmd_0_1_0 -> gather_node_0 [id="npfo-errors.0"]; } ================================================ FILE: core-tools/tests-regression/regression/graphs/text-properties.ok ================================================ digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; node_tee_0 [id="node_tee_0", label="sgsh-tee ", shape="ellipse"]; "NCHARS" [id="store:NCHARS", shape="box"]; node_cmd_0_1_0 -> "NCHARS" [id="nps-NCHARS"]; node_cmd_0_0_0 [id="node_cmd_0_0_0", label="tr -cs a-zA-Z \\\\n\l", shape="ellipse"]; node_cmd_1_0_0 [id="node_cmd_1_0_0", label="ngram 2 >digram.txt\l", shape="ellipse"]; node_cmd_1_1_0 [id="node_cmd_1_1_0", label="ngram 3 >trigram.txt\l", shape="ellipse"]; node_cmd_1_2_0 [id="node_cmd_1_2_0", label="ranked_frequency >words.txt\l", shape="ellipse"]; node_cmd_0_1_0 [id="node_cmd_0_1_0", label="wc -c\l", shape="ellipse"]; NCHARS -> node_cmd_0_2_0 [id="store-read"]; node_cmd_0_2_0 [id="node_cmd_0_2_0", label="sed 's/./&\\\l/g' |\l# Print absolute and percentage\lranked_frequency |\lawk 'BEGIN {OFMT = \"%.2g%%\"}\l{print $1, $2, $1 / '\"`store:NCHARS`\"' * 100}' >character.txt\l", shape="ellipse"]; node_tee_0 -> node_cmd_0_1_0 [id="npi-0.1.0"]; node_cmd_0_0_0 -> node_cmd_1_2_0 [id="npi-1.2.0"]; node_cmd_0_0_0 -> node_cmd_1_0_0 [id="npi-1.0.0"]; node_tee_0 -> node_cmd_0_2_0 [id="npi-0.2.0"]; node_tee_0 -> node_cmd_0_0_0 [id="npi-0.0.0"]; node_cmd_0_0_0 -> node_cmd_1_1_0 [id="npi-1.1.0"]; } ================================================ FILE: core-tools/tests-regression/regression/graphs/web-log-report.ok ================================================ digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; node_tee_0 [id="node_tee_0", label="sgsh-tee ", shape="ellipse"]; "nXBytes" [id="store:nXBytes", shape="box"]; node_cmd_0_0_0 -> "nXBytes" [id="nps-nXBytes"]; "nLogBytes" [id="store:nLogBytes", shape="box"]; node_cmd_0_1_0 -> "nLogBytes" [id="nps-nLogBytes"]; "nAccess" [id="store:nAccess", shape="box"]; node_cmd_1_0_0 -> "nAccess" [id="nps-nAccess"]; "nHosts" [id="store:nHosts", shape="box"]; node_cmd_3_0_0 -> "nHosts" [id="nps-nHosts"]; "nTLD" [id="store:nTLD", shape="box"]; node_cmd_3_1_0 -> "nTLD" [id="nps-nTLD"]; "nDomain" [id="store:nDomain", shape="box"]; node_cmd_4_0_0 -> "nDomain" [id="nps-nDomain"]; "nPages" [id="store:nPages", shape="box"]; node_cmd_5_1_0 -> "nPages" [id="nps-nPages"]; "nDays" [id="store:nDays", shape="box"]; node_cmd_7_0_0 -> "nDays" [id="nps-nDays"]; node_cmd_0_0_0 [id="node_cmd_0_0_0", label="awk '{s += $NF} END {print s}'\l", shape="ellipse"]; node_cmd_0_1_0 [id="node_cmd_0_1_0", label="wc -c\l", shape="ellipse"]; node_cmd_0_2_0 [id="node_cmd_0_2_0", label="awk '{print $1}'\l", shape="ellipse"]; node_cmd_1_0_0 [id="node_cmd_1_0_0", label="wc -l\l", shape="ellipse"]; node_cmd_1_1_0 [id="node_cmd_1_1_0", label="sort\l", shape="ellipse"]; node_cmd_2_0_0 [id="node_cmd_2_0_0", label="uniq\l", shape="ellipse"]; node_cmd_3_0_0 [id="node_cmd_3_0_0", label="wc -l\l", shape="ellipse"]; node_cmd_3_1_0 [id="node_cmd_3_1_0", label="awk -F. '$NF !~ /[0-9]/ {print $NF}' |\lsort -u | wc -l\l", shape="ellipse"]; node_cmd_2_1_0 [id="node_cmd_2_1_0", label="{\lheader 'Top 10 Hosts'\ltoplist 10\l}\l", shape="ellipse"]; node_cmd_1_2_0 [id="node_cmd_1_2_0", label="{\lheader 'Top 20 Level Domain Accesses'\lawk -F. '$NF !~ /^[0-9]/ {print $NF}' |\lsort |\ltoplist 20\l}\l", shape="ellipse"]; node_cmd_1_3_0 [id="node_cmd_1_3_0", label="awk -F. 'BEGIN {OFS = \".\"}\l$NF !~ /^[0-9]/ {$1 = \"\"; print}' | sort\l", shape="ellipse"]; node_cmd_4_0_0 [id="node_cmd_4_0_0", label="uniq | wc -l\l", shape="ellipse"]; node_cmd_4_1_0 [id="node_cmd_4_1_0", label="{\lheader 'Top 10 Domains'\ltoplist 10\l}\l", shape="ellipse"]; node_cmd_0_3_0 [id="node_cmd_0_3_0", label="{\lheader 'Top 10 Hosts by Transfer'\lawk ' {bytes[$1] += $NF}\lEND {for (h in bytes) print bytes[h], h}' |\lsort -rn |\lhead -10\l}\l", shape="ellipse"]; node_cmd_0_4_0 [id="node_cmd_0_4_0", label="awk '{print $7}' | sort\l", shape="ellipse"]; node_cmd_5_0_0 [id="node_cmd_5_0_0", label="{\lheader 'Top 20 Area Requests'\lawk -F/ '{print $2}' |\ltoplist 20\l}\l", shape="ellipse"]; node_cmd_5_1_0 [id="node_cmd_5_1_0", label="uniq | wc -l\l", shape="ellipse"]; node_cmd_5_2_0 [id="node_cmd_5_2_0", label="{\lheader 'Top 20 Requests'\ltoplist 20\l}\l", shape="ellipse"]; node_cmd_0_5_0 [id="node_cmd_0_5_0", label="awk '{print substr($4, 2)}'\l", shape="ellipse"]; node_cmd_6_0_0 [id="node_cmd_6_0_0", label="awk -F: '{print $1}'\l", shape="ellipse"]; node_cmd_7_0_0 [id="node_cmd_7_0_0", label="uniq | wc -l\l", shape="ellipse"]; node_cmd_7_1_0 [id="node_cmd_7_1_0", label="{\lheader 'Accesses by Date'\luniq -c\l}\l", shape="ellipse"]; node_cmd_7_2_0 [id="node_cmd_7_2_0", label="{\lheader 'Accesses by Day of Week'\lsed 's|/|-|g' |\l(date -f - +%a 2>/dev/null || gdate -f - +%a) |\lsort |\luniq -c |\lsort -rn\l}\l", shape="ellipse"]; node_cmd_6_1_0 [id="node_cmd_6_1_0", label="{\lheader 'Accesses by Local Hour'\lawk -F: '{print $2}' |\lsort |\luniq -c\l}\l", shape="ellipse"]; nAccess -> node_cmd_0_6_0 [id="store-read"]; nXBytes -> node_cmd_0_6_0 [id="store-read"]; nHosts -> node_cmd_0_6_0 [id="store-read"]; nDomain -> node_cmd_0_6_0 [id="store-read"]; nTLD -> node_cmd_0_6_0 [id="store-read"]; nPages -> node_cmd_0_6_0 [id="store-read"]; nAccess -> node_cmd_0_6_0 [id="store-read"]; nDays -> node_cmd_0_6_0 [id="store-read"]; nXBytes -> node_cmd_0_6_0 [id="store-read"]; nDays -> node_cmd_0_6_0 [id="store-read"]; nLogBytes -> node_cmd_0_6_0 [id="store-read"]; node_cmd_0_6_0 [id="node_cmd_0_6_0", label="cat < node_cmd_0_4_0 [id="npi-0.4.0"]; node_cmd_0_2_0 -> node_cmd_1_1_0 [id="npi-1.1.0"]; node_cmd_0_6_0 -> gather_node_2 [id="npfo-summary.0"]; node_cmd_0_5_0 -> node_cmd_6_1_0 [id="npi-6.1.0"]; node_cmd_0_4_0 -> node_cmd_5_0_0 [id="npi-5.0.0"]; node_tee_0 -> node_cmd_0_1_0 [id="npi-0.1.0"]; node_cmd_6_0_0 -> node_cmd_7_1_0 [id="npi-7.1.0"]; node_tee_0 -> node_cmd_0_2_0 [id="npi-0.2.0"]; node_cmd_0_5_0 -> node_cmd_6_0_0 [id="npi-6.0.0"]; node_cmd_0_3_0 -> gather_node_2 [id="npfo-top10HostsByVol.0"]; node_cmd_0_4_0 -> node_cmd_5_2_0 [id="npi-5.2.0"]; node_cmd_2_1_0 -> gather_node_2 [id="npfo-top10HostsByN.0"]; node_cmd_7_2_0 -> gather_node_2 [id="npfo-accessByDoW.0"]; node_cmd_1_3_0 -> node_cmd_4_1_0 [id="npi-4.1.0"]; node_tee_0 -> node_cmd_0_3_0 [id="npi-0.3.0"]; node_cmd_5_2_0 -> gather_node_2 [id="npfo-top20Request.0"]; node_cmd_5_0_0 -> gather_node_2 [id="npfo-top20Area.0"]; node_cmd_2_0_0 -> node_cmd_3_1_0 [id="npi-3.1.0"]; node_cmd_0_4_0 -> node_cmd_5_1_0 [id="npi-5.1.0"]; node_cmd_0_2_0 -> node_cmd_1_0_0 [id="npi-1.0.0"]; node_cmd_2_0_0 -> node_cmd_3_0_0 [id="npi-3.0.0"]; node_cmd_6_0_0 -> node_cmd_7_2_0 [id="npi-7.2.0"]; node_cmd_1_3_0 -> node_cmd_4_0_0 [id="npi-4.0.0"]; node_cmd_0_2_0 -> node_cmd_1_2_0 [id="npi-1.2.0"]; node_tee_0 -> node_cmd_0_0_0 [id="npi-0.0.0"]; node_cmd_6_1_0 -> gather_node_2 [id="npfo-accessByHour.0"]; node_cmd_0_2_0 -> node_cmd_1_3_0 [id="npi-1.3.0"]; node_tee_0 -> node_cmd_0_5_0 [id="npi-0.5.0"]; node_cmd_1_2_0 -> gather_node_2 [id="npfo-top20TLD.0"]; node_cmd_1_1_0 -> node_cmd_2_0_0 [id="npi-2.0.0"]; node_cmd_4_1_0 -> gather_node_2 [id="npfo-top10Domain.0"]; node_cmd_1_1_0 -> node_cmd_2_1_0 [id="npi-2.1.0"]; node_cmd_6_0_0 -> node_cmd_7_0_0 [id="npi-7.0.0"]; node_cmd_7_1_0 -> gather_node_2 [id="npfo-accessByDate.0"]; } ================================================ FILE: core-tools/tests-regression/regression/graphs/web-log-stats.ok ================================================ digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; node_tee_0 [id="node_tee_0", label="sgsh-tee ", shape="ellipse"]; "page" [id="store:page", shape="box"]; node_cmd_0_0_0 -> "page" [id="nps-page"]; "total_bytes" [id="store:total_bytes", shape="box"]; node_cmd_1_0_0 -> "total_bytes" [id="nps-total_bytes"]; "total_pages" [id="store:total_pages", shape="box"]; node_cmd_1_1_0 -> "total_pages" [id="nps-total_pages"]; "bytes" [id="store:bytes", shape="box"]; node_cmd_1_2_0 -> "bytes" [id="nps-bytes"]; "bytes_old" [id="store:bytes_old", shape="box"]; node_cmd_1_3_0 -> "bytes_old" [id="nps-bytes_old"]; node_cmd_0_0_0 [id="node_cmd_0_0_0", label="awk -Winteractive '{print $7}'\l", shape="ellipse"]; node_cmd_0_1_0 [id="node_cmd_0_1_0", label="awk -Winteractive '{print $10}'\l", shape="ellipse"]; node_cmd_1_0_0 [id="node_cmd_1_0_0", label="awk -Winteractive '{ s += $1; print s}'\l", shape="ellipse"]; node_cmd_1_1_0 [id="node_cmd_1_1_0", label="awk -Winteractive '{print ++n}'\l", shape="ellipse"]; node_cmd_1_2_0 [id="node_cmd_1_2_0", label="\l", shape="ellipse"]; node_cmd_1_3_0 [id="node_cmd_1_3_0", label="\l", shape="ellipse"]; bytes -> gather_node_3 [id="store-read"]; gather_node_3 [id="gather_node_3", label="WINDOW_PAGES=$(store:bytes -c | wc -l)\l", shape="ellipse"]; bytes -> gather_node_4 [id="store-read"]; gather_node_4 [id="gather_node_4", label="WINDOW_BYTES=$(store:bytes -c | sum )\l", shape="ellipse"]; bytes_old -> gather_node_5 [id="store-read"]; gather_node_5 [id="gather_node_5", label="WINDOW_PAGES_OLD=$(store:bytes_old -c | wc -l)\l", shape="ellipse"]; bytes_old -> gather_node_6 [id="store-read"]; gather_node_6 [id="gather_node_6", label="WINDOW_BYTES_OLD=$(store:bytes_old -c | sum)\l", shape="ellipse"]; total_pages -> gather_node_8 [id="store-read"]; total_bytes -> gather_node_8 [id="store-read"]; page -> gather_node_8 [id="store-read"]; gather_node_8 [id="gather_node_8", label="cat < node_cmd_0_1_0 [id="npi-0.1.0"]; node_cmd_0_1_0 -> node_cmd_1_2_0 [id="npi-1.2.0"]; node_cmd_0_1_0 -> node_cmd_1_0_0 [id="npi-1.0.0"]; node_tee_0 -> node_cmd_0_0_0 [id="npi-0.0.0"]; node_cmd_0_1_0 -> node_cmd_1_1_0 [id="npi-1.1.0"]; node_cmd_0_1_0 -> node_cmd_1_3_0 [id="npi-1.3.0"]; } ================================================ FILE: core-tools/tests-regression/regression/graphs/word-properties.ok ================================================ digraph "" { rankdir = LR; node [fontname="Courier"]; edge []; node_tee_0 [id="node_tee_0", label="sgsh-tee ", shape="ellipse"]; node_cmd_0_1_0 [id="node_cmd_0_1_0", label="sed 's/.*\\(.\\)\\(.\\)\\2\\1.*/p: \\1\\2-\\2\\1/;t\lg'\l", shape="ellipse"]; node_cmd_0_2_0 [id="node_cmd_0_2_0", label="sed -E 's/.*([^aeiouyAEIOUY]{4}).*/c: \\1/;t\lg'\l", shape="ellipse"]; node_cmd_0_3_0 [id="node_cmd_0_3_0", label="awk '{if (length($1) > 12) print \"l:\", length($1);\lelse print \"\"}'\l", shape="ellipse"]; gather_node_1 [id="gather_node_1", label="paste /stream/words /stream/palindromes /stream/consonants /stream/long |\l# List only words satisfying one or more properties\lgrep :\l", shape="ellipse"]; node_tee_0 -> node_cmd_0_1_0 [id="npi-0.1.0"]; node_cmd_0_1_0 -> gather_node_1 [id="npfo-palindromes.0"]; node_tee_0 -> node_cmd_0_3_0 [id="npi-0.3.0"]; node_cmd_0_3_0 -> gather_node_1 [id="npfo-long.0"]; node_cmd_0_2_0 -> gather_node_1 [id="npfo-consonants.0"]; node_tee_0 -> gather_node_1 [id="npfo-words.0"]; node_tee_0 -> node_cmd_0_2_0 [id="npi-0.2.0"]; } ================================================ FILE: core-tools/tests-regression/regression/scripts/NMRPipe.ok ================================================ #!/bin/sh # Automatically generated file # Source file example/NMRPipe.sh #!/usr/bin/env sgsh # # SYNOPSIS Nuclear magnetic resonance processing # DESCRIPTION # Nuclear magnetic resonance in-phase/anti-phase channel conversion and # processing in heteronuclear single quantum coherence spectroscopy. # Demonstrate processing of NMR data using the NMRPipe family of programs. # # See also F. Delaglio, S. Grzesiek, G. W. Vuister, G. Zhu, J. Pfeifer # and A. Bax: NMRPipe: a multidimensional spectral processing system based # on UNIX pipes. J. Biomol. NMR. 6, 277-293 (1995). # http://spin.niddk.nih.gov/NMRPipe/ # # The conversion is configured for the following file: # http://www.bmrb.wisc.edu/ftp/pub/bmrb/timedomain/bmr6443/timedomain_data/c13-hsqc/june11-se-6426-CA.fid/fid var2pipe -in $1 \ -xN 1280 -yN 256 \ -xT 640 -yT 128 \ -xMODE Complex -yMODE Complex \ -xSW 8000 -ySW 6000 \ -xOBS 599.4489584 -yOBS 60.7485301 \ -xCAR 4.73 -yCAR 118.000 \ -xLAB 1H -yLAB 15N \ -ndim 2 -aq2D States \ -verb | ( export SGDIR=/tmp/sg-$$.0 rm -rf $SGDIR # Cleanup on exit or interrupt cleanup() { SIGNAL=$1 [ $SIGNAL = EXIT ] || echo sgsh interrupted. Cleaning up... 1>&2 # Stop key-value stores # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npfo-none-0.0.0 \ $SGDIR/npfo-none-0.1.0 \ $SGDIR/npi-0.0.0 \ $SGDIR/npi-0.1.0 sgsh-tee -o $SGDIR/npi-0.0.0 -o $SGDIR/npi-0.1.0 <&3 3<&- & SGPID="$! $SGPID" { nmrPipe | nmrPipe -fn SOL | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -c 0.5 | nmrPipe -fn ZF -auto | nmrPipe -fn FT | nmrPipe -fn PS -p0 177 -p1 0.0 -di | nmrPipe -fn EXT -left -sw -verb | nmrPipe -fn TP | nmrPipe -fn COADD -cList 1 0 -time | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 1 -c 0.5 | nmrPipe -fn ZF -auto | nmrPipe -fn FT | nmrPipe -fn PS -p0 0 -p1 0 -di | nmrPipe -fn TP | nmrPipe -fn POLY -auto -verb >A : ; } <$SGDIR/npi-0.0.0 >$SGDIR/npfo-none-0.0.0 & SGPID="$! $SGPID" { nmrPipe | nmrPipe -fn SOL | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -c 0.5 | nmrPipe -fn ZF -auto | nmrPipe -fn FT | nmrPipe -fn PS -p0 177 -p1 0.0 -di | nmrPipe -fn EXT -left -sw -verb | nmrPipe -fn TP | nmrPipe -fn COADD -cList 0 1 -time | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 1 -c 0.5 | nmrPipe -fn ZF -auto | nmrPipe -fn FT | nmrPipe -fn PS -p0 -90 -p1 0 -di | nmrPipe -fn TP | nmrPipe -fn POLY -auto -verb >B : ; } <$SGDIR/npi-0.1.0 >$SGDIR/npfo-none-0.1.0 & SGPID="$! $SGPID" # Gather the results sgsh-tee -i $SGDIR/npfo-none-0.0.0 -i $SGDIR/npfo-none-0.1.0 >/dev/null ) 3<&0 # We use temporary files rather than streams, because # addNMR mmaps its input files. The diagram displayed in the # example shows the notional data flow. addNMR -in1 A -in2 B -out A+B.sgsh.ft2 -c1 1.0 -c2 1.25 -add addNMR -in1 A -in2 B -out A-B.sgsh.ft2 -c1 1.0 -c2 1.25 -sub ================================================ FILE: core-tools/tests-regression/regression/scripts/code-metrics.ok ================================================ #!/bin/sh # Automatically generated file # Source file example/code-metrics.sh #!/usr/bin/env sgsh # # SYNOPSIS C code metrics # DESCRIPTION # Process a directory containing C source code, and produce a summary # of various metrics. # Demonstrates nesting, commands without input. # # Copyright 2012-2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ( export SGDIR=/tmp/sg-$$.0 rm -rf $SGDIR # Cleanup on exit or interrupt cleanup() { SIGNAL=$1 [ $SIGNAL = EXIT ] || echo sgsh interrupted. Cleaning up... 1>&2 # Stop key-value stores sgsh-readval -q -s "$SGDIR/FNAMELEN" 2>/dev/null sgsh-readval -q -s "$SGDIR/NSTRUCT" 2>/dev/null sgsh-readval -q -s "$SGDIR/NTYPEDEF" 2>/dev/null sgsh-readval -q -s "$SGDIR/NVOID" 2>/dev/null sgsh-readval -q -s "$SGDIR/NGETS" 2>/dev/null sgsh-readval -q -s "$SGDIR/IDLEN" 2>/dev/null sgsh-readval -q -s "$SGDIR/CHLINESCHAR" 2>/dev/null sgsh-readval -q -s "$SGDIR/NCCHAR" 2>/dev/null sgsh-readval -q -s "$SGDIR/NCOMMENT" 2>/dev/null sgsh-readval -q -s "$SGDIR/NCOPYRIGHT" 2>/dev/null sgsh-readval -q -s "$SGDIR/NCFILE" 2>/dev/null sgsh-readval -q -s "$SGDIR/NCDIR" 2>/dev/null sgsh-readval -q -s "$SGDIR/CLINESCHAR" 2>/dev/null sgsh-readval -q -s "$SGDIR/NFUNCTION" 2>/dev/null sgsh-readval -q -s "$SGDIR/NGOTO" 2>/dev/null sgsh-readval -q -s "$SGDIR/NREGISTER" 2>/dev/null sgsh-readval -q -s "$SGDIR/NMACRO" 2>/dev/null sgsh-readval -q -s "$SGDIR/NINCLUDE" 2>/dev/null sgsh-readval -q -s "$SGDIR/NCONST" 2>/dev/null sgsh-readval -q -s "$SGDIR/NHFILE" 2>/dev/null # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npi-1.0.0 \ $SGDIR/npi-1.1.0 \ $SGDIR/npi-2.0.0 \ $SGDIR/npi-2.1.0 \ $SGDIR/npi-2.2.0 \ $SGDIR/npi-2.3.0 \ $SGDIR/npi-2.4.0 \ $SGDIR/npi-3.0.0 \ $SGDIR/npi-3.1.0 \ $SGDIR/npi-3.2.0 \ $SGDIR/npi-3.3.0 \ $SGDIR/npi-3.4.0 \ $SGDIR/npi-4.0.0 \ $SGDIR/npi-4.1.0 \ $SGDIR/npi-5.0.0 \ $SGDIR/npi-5.1.0 \ $SGDIR/npi-6.0.0 \ $SGDIR/npi-6.1.0 \ $SGDIR/npi-7.0.0 \ $SGDIR/npi-7.1.0 \ $SGDIR/npi-7.2.0 \ $SGDIR/npi-7.3.0 \ $SGDIR/npi-7.4.0 \ $SGDIR/npi-7.5.0 { find "$@" \( -name \*.c -or -name \*.h \) -type f -print0 }/dev/null } <$SGDIR/npi-2.0.0 | sgsh-tee -o $SGDIR/npi-3.0.0 -o $SGDIR/npi-3.1.0 -o $SGDIR/npi-3.2.0 -o $SGDIR/npi-3.3.0 -o $SGDIR/npi-3.4.0 & SGPID="$! $SGPID" { egrep -c 'struct[ ]*{|struct[ ]*[a-zA-Z_][a-zA-Z0-9_]*[ ]*{' } <$SGDIR/npi-3.0.0 | sgsh-writeval -s $SGDIR/NSTRUCT & SGPID="$! $SGPID" { grep -cw typedef } <$SGDIR/npi-3.1.0 | sgsh-writeval -s $SGDIR/NTYPEDEF & SGPID="$! $SGPID" { grep -cw void } <$SGDIR/npi-3.2.0 | sgsh-writeval -s $SGDIR/NVOID & SGPID="$! $SGPID" { grep -cw gets } <$SGDIR/npi-3.3.0 | sgsh-writeval -s $SGDIR/NGETS & SGPID="$! $SGPID" { tr -cs 'A-Za-z0-9_' '\n' | sort -u | awk '/^[A-Za-z]/ { len += length($1); n++ } END {print len / n}' } <$SGDIR/npi-3.4.0 | sgsh-writeval -s $SGDIR/IDLEN & SGPID="$! $SGPID" { wc -lc | awk '{OFS=":"; print $1, $2}' } <$SGDIR/npi-2.1.0 | sgsh-writeval -s $SGDIR/CHLINESCHAR & SGPID="$! $SGPID" { sed 's/#/@/g' | cpp -traditional -P 2>/dev/null | wc -c | awk '{OFMT = "%.0f"; print $1/1000}' } <$SGDIR/npi-2.2.0 | sgsh-writeval -s $SGDIR/NCCHAR & SGPID="$! $SGPID" { egrep -c '/\*|//' } <$SGDIR/npi-2.3.0 | sgsh-writeval -s $SGDIR/NCOMMENT & SGPID="$! $SGPID" { grep -ci copyright } <$SGDIR/npi-2.4.0 | sgsh-writeval -s $SGDIR/NCOPYRIGHT & SGPID="$! $SGPID" { find "$@" -name \*.c -type f -print0 }/dev/null } <$SGDIR/npi-6.1.0 | sgsh-tee -o $SGDIR/npi-7.0.0 -o $SGDIR/npi-7.1.0 -o $SGDIR/npi-7.2.0 -o $SGDIR/npi-7.3.0 -o $SGDIR/npi-7.4.0 -o $SGDIR/npi-7.5.0 & SGPID="$! $SGPID" { grep -c '^{' } <$SGDIR/npi-7.0.0 | sgsh-writeval -s $SGDIR/NFUNCTION & SGPID="$! $SGPID" { grep -cw goto } <$SGDIR/npi-7.1.0 | sgsh-writeval -s $SGDIR/NGOTO & SGPID="$! $SGPID" { grep -cw register } <$SGDIR/npi-7.2.0 | sgsh-writeval -s $SGDIR/NREGISTER & SGPID="$! $SGPID" { grep -c '@[ ]*define[ ][ ]*[a-zA-Z_][a-zA-Z0-9_]*(' } <$SGDIR/npi-7.3.0 | sgsh-writeval -s $SGDIR/NMACRO & SGPID="$! $SGPID" { grep -c '@[ ]*include' } <$SGDIR/npi-7.4.0 | sgsh-writeval -s $SGDIR/NINCLUDE & SGPID="$! $SGPID" { grep -o -h '[0-9][x0-9][0-9a-f]*' | wc -l } <$SGDIR/npi-7.5.0 | sgsh-writeval -s $SGDIR/NCONST & SGPID="$! $SGPID" { find "$@" -name \*.h -type f | wc -l }&2 # Stop key-value stores # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npfo-authors.0 \ $SGDIR/npfo-days.0 \ $SGDIR/npi-0.0.0 \ $SGDIR/npi-0.1.0 sgsh-tee -o $SGDIR/npi-0.0.0 -o $SGDIR/npi-0.1.0 <&3 3<&- & SGPID="$! $SGPID" { awk -F: '{print $1}' | forder } <$SGDIR/npi-0.0.0 >$SGDIR/npfo-authors.0 & SGPID="$! $SGPID" { awk -F: '{print substr($2, 1, 3)}' | forder } <$SGDIR/npi-0.1.0 >$SGDIR/npfo-days.0 & SGPID="$! $SGPID" # Gather the results echo "Authors ordered by number of commits" cat $SGDIR/npfo-authors.0 echo "Days ordered by number of commits" cat $SGDIR/npfo-days.0 ) 3<&0 ================================================ FILE: core-tools/tests-regression/regression/scripts/committer-plot.ok ================================================ #!/bin/sh # Automatically generated file # Source file example/committer-plot.sh #!/usr/bin/env sgsh -s /bin/bash # # SYNOPSIS Plot git committer activity over time # DESCRIPTION # Process the git history, and create two PNG diagrams depicting # committer activity over time. The most active committers appear # at the center vertical of the diagram. # Demonstrates image processing and no-output scatter blocks. # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Commit history in the form of ascending Unix timestamps, emails git log --pretty=tformat:'%at %ae' | awk 'NF == 2 && $1 > 100000 && $1 < '`date +%s` | sort -n | ( export SGDIR=/tmp/sg-$$.0 rm -rf $SGDIR # Cleanup on exit or interrupt cleanup() { SIGNAL=$1 [ $SIGNAL = EXIT ] || echo sgsh interrupted. Cleaning up... 1>&2 # Stop key-value stores sgsh-readval -q -s "$SGDIR/NCOMMITTERS" 2>/dev/null sgsh-readval -q -s "$SGDIR/LAST" 2>/dev/null sgsh-readval -q -s "$SGDIR/FIRST" 2>/dev/null sgsh-readval -q -s "$SGDIR/NDAYS" 2>/dev/null # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npfo-committerpos.0 \ $SGDIR/npfo-none-1.0.0 \ $SGDIR/npfo-none-1.1.0 \ $SGDIR/npi-0.0.0 \ $SGDIR/npi-0.1.0 \ $SGDIR/npi-0.2.0 \ $SGDIR/npi-0.4.0 \ $SGDIR/npi-0.5.0 \ $SGDIR/npi-1.0.0 \ $SGDIR/npi-1.1.0 sgsh-tee -o $SGDIR/npi-0.0.0 -o $SGDIR/npi-0.1.0 -o $SGDIR/npi-0.2.0 -o $SGDIR/npi-0.4.0 -o $SGDIR/npi-0.5.0 <&3 3<&- & SGPID="$! $SGPID" { awk '{print $2}' | sort -u | wc -l } <$SGDIR/npi-0.0.0 | sgsh-writeval -s $SGDIR/NCOMMITTERS & SGPID="$! $SGPID" { tail -1 | awk '{print $1}' } <$SGDIR/npi-0.1.0 | sgsh-writeval -s $SGDIR/LAST & SGPID="$! $SGPID" { head -1 | awk '{print $1}' } <$SGDIR/npi-0.2.0 | sgsh-writeval -s $SGDIR/FIRST & SGPID="$! $SGPID" { expr \( `sgsh-readval -s $SGDIR/LAST` - `sgsh-readval -s $SGDIR/FIRST` \) / 60 / 60 / 24 }$SGDIR/npfo-committerpos.0 & SGPID="$! $SGPID" { sort -k2 | join -j 2 - $SGDIR/npfo-committerpos.0 | # Order by time sort -k 2n | { # Create portable bitmap echo 'P1' echo "`sgsh-readval -s $SGDIR/NCOMMITTERS` `sgsh-readval -s $SGDIR/NDAYS`" perl -na -e ' BEGIN { @empty['`sgsh-readval -s $SGDIR/NCOMMITTERS`' - 1] = 0; @committers = @empty; } sub out { print join("", map($_ ? "1" : "0", @committers)), "\n"; } $day = int($F[1] / 60 / 60 / 24); $pday = $day if (!defined($pday)); while ($day != $pday) { out(); @committers = @empty; $pday++; } $committers[$F[2]] = 1; END { out(); } ' } | # Enlarge points into discs through morphological convolution pgmmorphconv -erode <( cat <large.png : ; } <$SGDIR/npi-1.0.0 >$SGDIR/npfo-none-1.0.0 & SGPID="$! $SGPID" { pamscale -width 640 | pnmtopng >small.png : ; } <$SGDIR/npi-1.1.0 >$SGDIR/npfo-none-1.1.0 & SGPID="$! $SGPID" # Gather the results sgsh-tee -i $SGDIR/npfo-none-1.0.0 -i $SGDIR/npfo-none-1.1.0 >/dev/null ) 3<&0 ================================================ FILE: core-tools/tests-regression/regression/scripts/compress-compare.ok ================================================ #!/bin/sh # Automatically generated file # Source file example/compress-compare.sh #!/usr/bin/env sgsh # # SYNOPSIS Compression benchmark # DESCRIPTION # Report file type, length, and compression performance for # data received from the standard input. The data never touches the # disk. # Demonstrates the use of stores. # # Copyright 2012-2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ( export SGDIR=/tmp/sg-$$.0 rm -rf $SGDIR # Cleanup on exit or interrupt cleanup() { SIGNAL=$1 [ $SIGNAL = EXIT ] || echo sgsh interrupted. Cleaning up... 1>&2 # Stop key-value stores sgsh-readval -q -s "$SGDIR/NBYTES" 2>/dev/null sgsh-readval -q -s "$SGDIR/FILETYPE" 2>/dev/null sgsh-readval -q -s "$SGDIR/XZ" 2>/dev/null sgsh-readval -q -s "$SGDIR/BZIP2" 2>/dev/null sgsh-readval -q -s "$SGDIR/GZIP" 2>/dev/null # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npi-0.0.0 \ $SGDIR/npi-0.1.0 \ $SGDIR/npi-0.2.0 \ $SGDIR/npi-0.3.0 \ $SGDIR/npi-0.4.0 sgsh-tee -o $SGDIR/npi-0.0.0 -o $SGDIR/npi-0.1.0 -o $SGDIR/npi-0.2.0 -o $SGDIR/npi-0.3.0 -o $SGDIR/npi-0.4.0 <&3 3<&- & SGPID="$! $SGPID" { wc -c } <$SGDIR/npi-0.0.0 | sgsh-writeval -s $SGDIR/NBYTES & SGPID="$! $SGPID" { file - } <$SGDIR/npi-0.1.0 | sgsh-writeval -s $SGDIR/FILETYPE & SGPID="$! $SGPID" { xz -c | wc -c } <$SGDIR/npi-0.2.0 | sgsh-writeval -s $SGDIR/XZ & SGPID="$! $SGPID" { bzip2 -c | wc -c } <$SGDIR/npi-0.3.0 | sgsh-writeval -s $SGDIR/BZIP2 & SGPID="$! $SGPID" { gzip -c | wc -c } <$SGDIR/npi-0.4.0 | sgsh-writeval -s $SGDIR/GZIP & SGPID="$! $SGPID" # Gather the results cat <ls -l | awk. # Demonstrates combined use of stores and streams. # # Copyright 2012-2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # FREE=`df -h . | awk '!/Use%/{print $4}'` ls -n | ( export SGDIR=/tmp/sg-$$.0 rm -rf $SGDIR # Cleanup on exit or interrupt cleanup() { SIGNAL=$1 [ $SIGNAL = EXIT ] || echo sgsh interrupted. Cleaning up... 1>&2 # Stop key-value stores sgsh-readval -q -s "$SGDIR/NFILES" 2>/dev/null sgsh-readval -q -s "$SGDIR/NDIRS" 2>/dev/null sgsh-readval -q -s "$SGDIR/NBYTES" 2>/dev/null # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npfo-files.0 \ $SGDIR/npi-0.0.0 \ $SGDIR/npi-0.1.0 \ $SGDIR/npi-0.2.0 \ $SGDIR/npi-0.3.0 sgsh-tee -o $SGDIR/npi-0.0.0 -o $SGDIR/npi-0.1.0 -o $SGDIR/npi-0.2.0 -o $SGDIR/npi-0.3.0 <&3 3<&- & SGPID="$! $SGPID" { awk '!/^total/ {print $6, $7, $8, $1, sprintf("%8d", $5), $9}' } <$SGDIR/npi-0.0.0 >$SGDIR/npfo-files.0 & SGPID="$! $SGPID" { wc -l } <$SGDIR/npi-0.1.0 | sgsh-writeval -s $SGDIR/NFILES & SGPID="$! $SGPID" { grep -c '^d' } <$SGDIR/npi-0.2.0 | sgsh-writeval -s $SGDIR/NDIRS & SGPID="$! $SGPID" { awk '{s += $5} END {print s}' } <$SGDIR/npi-0.3.0 | sgsh-writeval -s $SGDIR/NBYTES & SGPID="$! $SGPID" # Gather the results cat $SGDIR/npfo-files.0 echo " `sgsh-readval -s $SGDIR/NFILES` File(s) `sgsh-readval -s $SGDIR/NBYTES` bytes" echo " `sgsh-readval -s $SGDIR/NDIRS` Dir(s) $FREE bytes free" ) 3<&0 ================================================ FILE: core-tools/tests-regression/regression/scripts/duplicate-files.ok ================================================ #!/bin/sh # Automatically generated file # Source file example/duplicate-files.sh #!/usr/bin/env sgsh # # SYNOPSIS Find duplicate files # DESCRIPTION # List the names of duplicate files in the specified directory. # Demonstrates the combination of streams with a relational join. # # Copyright 2012-2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Create list of files find "$@" -type f | # Produce lines of the form # MD5(filename)= 811bfd4b5974f39e986ddc037e1899e7 xargs openssl md5 | # Convert each line into a "filename md5sum" pair sed 's/^MD5(//;s/)= / /' | # Sort by MD5 sum sort -k2 | ( export SGDIR=/tmp/sg-$$.0 rm -rf $SGDIR # Cleanup on exit or interrupt cleanup() { SIGNAL=$1 [ $SIGNAL = EXIT ] || echo sgsh interrupted. Cleaning up... 1>&2 # Stop key-value stores # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npfo-dupes.0 \ $SGDIR/npi-0.0.0 \ $SGDIR/npi-0.1.0 sgsh-tee -o $SGDIR/npi-0.0.0 -o $SGDIR/npi-0.1.0 <&3 3<&- & SGPID="$! $SGPID" { awk '{print $2}' | uniq -d } <$SGDIR/npi-0.0.0 >$SGDIR/npfo-dupes.0 & SGPID="$! $SGPID" ln -s $SGDIR/npi-0.1.0 $SGDIR/npfo-names.0 # Gather the results # Join the repeated MD5 sums with the corresponding file names join -2 2 $SGDIR/npfo-dupes.0 $SGDIR/npfo-names.0 | # Output same files on a single line awk ' BEGIN {ORS=""} $1 != prev && prev {print "\n"} END {if (prev) print "\n"} {if (prev) print " "; prev = $1; print $2}' ) 3<&0 ================================================ FILE: core-tools/tests-regression/regression/scripts/ft2d.ok ================================================ #!/bin/sh # Automatically generated file # Source file example/ft2d.sh #!/usr/bin/env sgsh # # SYNOPSIS Waves: 2D Fourier transforms # DESCRIPTION # Create two graphs: # 1) a broadened pulse and the real part of its 2D Fourier transform, and # 2) a simulated air wave and the amplitude of its 2D Fourier transform. # Demonstrates using the tools of the Madagascar shared research environment # for computational data analysis in geophysics and related fields. # Also demonstrates the use of two scatter blocks in the same script. # # Adapted from: http://www.reproducibility.org/RSF/book/bei/ft1/ft2d.html # Description: http://www.reproducibility.org/RSF/book/bei/ft1/paper_html/node14.html # Madagascar project: http://www.reproducibility.org # mkdir -p Fig # The SConstruct SideBySideIso "Result" method side_by_side_iso() { vppen size=r vpstyle=n gridnum=2,1 $* } # A broadened pulse and the real part of its 2D Fourier transform ( export SGDIR=/tmp/sg-$$.0 rm -rf $SGDIR # Cleanup on exit or interrupt cleanup() { SIGNAL=$1 [ $SIGNAL = EXIT ] || echo sgsh interrupted. Cleaning up... 1>&2 # Stop key-value stores # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npfo-ft2d.0 \ $SGDIR/npfo-ft2d.vpl.0 \ $SGDIR/npfo-none-0.1.0 \ $SGDIR/npfo-pulse.vpl.0 \ $SGDIR/npi-1.0.0 \ $SGDIR/npi-1.1.0 \ $SGDIR/npi-2.0.0 \ $SGDIR/npi-2.1.0 { sfspike n1=64 n2=64 d1=1 d2=1 nsp=2 k1=16,17 k2=5,5 mag=16,16 \ label1='time' label2='space' unit1= unit2= | sfsmooth rect2=2 | sfsmooth rect2=2 }$SGDIR/npfo-pulse.vpl.0 & SGPID="$! $SGPID" { sffft1 | sffft3 axis=2 pad=1 | sfreal } <$SGDIR/npi-1.1.0 | sgsh-tee -o $SGDIR/npi-2.0.0 -o $SGDIR/npi-2.1.0 & SGPID="$! $SGPID" { sgsh-tee -I } <$SGDIR/npi-2.0.0 >$SGDIR/npfo-ft2d.0 & SGPID="$! $SGPID" { sfwindow f1=1 | sfreverse which=3 | sfcat axis=1 $SGDIR/npfo-ft2d.0 | sfgrey pclip=100 wanttitle=n \ label1="1/time" label2="1/space" } <$SGDIR/npi-2.1.0 >$SGDIR/npfo-ft2d.vpl.0 & SGPID="$! $SGPID" { side_by_side_iso $SGDIR/npfo-pulse.vpl.0 $SGDIR/npfo-ft2d.vpl.0 \ yscale=1.25 >Fig/ft2dofpulse.vpl : ; }$SGDIR/npfo-none-0.1.0 & SGPID="$! $SGPID" # Gather the results sgsh-tee -i $SGDIR/npfo-none-0.1.0 >/dev/null ) 3<&0 # A simulated air wave and the amplitude of its 2D Fourier transform ( export SGDIR=/tmp/sg-$$.1 rm -rf $SGDIR # Cleanup on exit or interrupt cleanup() { SIGNAL=$1 [ $SIGNAL = EXIT ] || echo sgsh interrupted. Cleaning up... 1>&2 # Stop key-value stores # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npfo-air.0 \ $SGDIR/npfo-airfk.vpl.0 \ $SGDIR/npfo-airft1.0 \ $SGDIR/npfo-airfti.0 \ $SGDIR/npfo-airftr.0 \ $SGDIR/npfo-airtx.vpl.0 \ $SGDIR/npfo-none-0.2.0 \ $SGDIR/npi-1.0.0 \ $SGDIR/npi-1.1.0 \ $SGDIR/npi-2.0.0 \ $SGDIR/npi-2.1.0 \ $SGDIR/npi-3.0.0 \ $SGDIR/npi-3.1.0 \ $SGDIR/npi-4.0.0 \ $SGDIR/npi-4.1.0 { sfspike n1=64 d1=1 o1=32 nsp=4 k1=1,2,3,4 mag=1,3,3,1 \ label1='time' unit1= | sfspray n=32 d=1 o=0 | sfput label2=space | sflmostretch delay=0 v0=-1 }$SGDIR/npfo-air.0 & SGPID="$! $SGPID" { sfwindow f2=1 | sfreverse which=2 | sfcat axis=2 $SGDIR/npfo-air.0 } <$SGDIR/npi-1.1.0 | sgsh-tee -o $SGDIR/npi-2.0.0 -o $SGDIR/npi-2.1.0 & SGPID="$! $SGPID" { sfgrey pclip=100 wanttitle=n } <$SGDIR/npi-2.0.0 >$SGDIR/npfo-airtx.vpl.0 & SGPID="$! $SGPID" { sffft1 | sffft3 sign=1 } <$SGDIR/npi-2.1.0 | sgsh-tee -o $SGDIR/npi-3.0.0 -o $SGDIR/npi-3.1.0 & SGPID="$! $SGPID" { sfreal } <$SGDIR/npi-3.0.0 >$SGDIR/npfo-airftr.0 & SGPID="$! $SGPID" { sfimag } <$SGDIR/npi-3.1.0 >$SGDIR/npfo-airfti.0 & SGPID="$! $SGPID" { sfmath re=$SGDIR/npfo-airftr.0 im=$SGDIR/npfo-airfti.0 output="sqrt(re*re+im*im)" }$SGDIR/npfo-airft1.0 & SGPID="$! $SGPID" { sfwindow f1=1 | sfreverse which=3 | sfcat axis=1 $SGDIR/npfo-airft1.0 | sfgrey pclip=100 wanttitle=n label1="1/time" \ label2="1/space" } <$SGDIR/npi-4.1.0 >$SGDIR/npfo-airfk.vpl.0 & SGPID="$! $SGPID" { side_by_side_iso $SGDIR/npfo-airtx.vpl.0 $SGDIR/npfo-airfk.vpl.0 \ yscale=1.25 >Fig/airwave.vpl : ; }$SGDIR/npfo-none-0.2.0 & SGPID="$! $SGPID" # Gather the results sgsh-tee -i $SGDIR/npfo-none-0.2.0 >/dev/null ) 3<&0 ================================================ FILE: core-tools/tests-regression/regression/scripts/map-hierarchy.ok ================================================ #!/bin/sh # Automatically generated file # Source file example/map-hierarchy.sh #!/usr/bin/env sgsh # # SYNOPSIS Hierarchy map # DESCRIPTION # Given two directory hierarchies A and B passed as input arguments # (where these represent a project at different parts of its lifetime) # copy the files of hierarchy A to a new directory, passed as a third # argument, corresponding to the structure of directories in B. # Demonstrates the use of join to gather results from streams # in the scatter part, # and the use of cat to order asynchronous results in the gather part. # # Copyright 2012-2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # if [ ! -d "$1" -o ! -d "$2" -o -z "$3" ] then echo "Usage: $0 dir-1 dir-2 new-dir-name" 1>&2 exit 1 fi NEWDIR="$3" export LC_ALL=C line_signatures() { find $1 -type f -name '*.[chly]' -print | # Split path name into directory and file sed 's|\(.*\)/\([^/]*\)|\1 \2|' | while read dir file do # Print "directory filename content" of lines with # at least one alphabetic character # The fields are separated by ^A and ^B sed -n "/[a-z]/s|^|$dir$file|p" "$dir/$file" done | sort -t -k 2 } ( export SGDIR=/tmp/sg-$$.0 rm -rf $SGDIR # Cleanup on exit or interrupt cleanup() { SIGNAL=$1 [ $SIGNAL = EXIT ] || echo sgsh interrupted. Cleaning up... 1>&2 # Stop key-value stores # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npfo-a.0 \ $SGDIR/npfo-b.0 \ $SGDIR/npfo-cp.0 \ $SGDIR/npfo-mkdir.0 \ $SGDIR/npi-1.0.0 \ $SGDIR/npi-1.1.0 { line_signatures $1 }$SGDIR/npfo-a.0 & SGPID="$! $SGPID" { line_signatures $2 }$SGDIR/npfo-b.0 & SGPID="$! $SGPID" { join -t -1 2 -2 2 $SGDIR/npfo-a.0 $SGDIR/npfo-b.0 | # Print filename dir1 dir2 sed 's///g' | awk -F 'BEGIN{OFS=" "}{print $1, $3, $4}' | # Unique occurrences sort -u }$SGDIR/npfo-cp.0 & SGPID="$! $SGPID" { awk '{print "mkdir -p \"'$NEWDIR'/" $3 "\""}' | sort -u } <$SGDIR/npi-1.1.0 >$SGDIR/npfo-mkdir.0 & SGPID="$! $SGPID" # Gather the results # Order: first make directories, then copy files sgsh-tee -f -I -i $SGDIR/npfo-mkdir.0 -i $SGDIR/npfo-cp.0 | sh ) 3<&0 ================================================ FILE: core-tools/tests-regression/regression/scripts/parallel-logresolve.ok ================================================ #!/bin/sh # Automatically generated file # Source file example/parallel-logresolve.sh #!/usr/bin/env sgsh # # SYNOPSIS Parallel logresolve # DESCRIPTION # Resolve IP addresses of web logs in parallel. # Demonstrates parallel execution. # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Add record number as the second field awk '{$2 = ++n " " $2; print}' "$@" | # Two parallel line scatter invocations ( export SGDIR=/tmp/sg-$$.0 rm -rf $SGDIR # Cleanup on exit or interrupt cleanup() { SIGNAL=$1 [ $SIGNAL = EXIT ] || echo sgsh interrupted. Cleaning up... 1>&2 # Stop key-value stores # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npfo-resolved.0 \ $SGDIR/npfo-resolved.1 \ $SGDIR/npfo-resolved.2 \ $SGDIR/npfo-resolved.3 \ $SGDIR/npfo-resolved.4 \ $SGDIR/npfo-resolved.5 \ $SGDIR/npfo-resolved.6 \ $SGDIR/npfo-resolved.7 \ $SGDIR/npfo-resolved.8 \ $SGDIR/npfo-resolved.9 \ $SGDIR/npi-0.0.0 \ $SGDIR/npi-0.0.1 \ $SGDIR/npi-0.0.2 \ $SGDIR/npi-0.0.3 \ $SGDIR/npi-0.0.4 \ $SGDIR/npi-0.0.5 \ $SGDIR/npi-0.0.6 \ $SGDIR/npi-0.0.7 \ $SGDIR/npi-0.0.8 \ $SGDIR/npi-0.0.9 sgsh-tee -s -o $SGDIR/npi-0.0.0 -o $SGDIR/npi-0.0.1 -o $SGDIR/npi-0.0.2 -o $SGDIR/npi-0.0.3 -o $SGDIR/npi-0.0.4 -o $SGDIR/npi-0.0.5 -o $SGDIR/npi-0.0.6 -o $SGDIR/npi-0.0.7 -o $SGDIR/npi-0.0.8 -o $SGDIR/npi-0.0.9 <&3 3<&- & SGPID="$! $SGPID" { logresolve | sgsh-tee -I } <$SGDIR/npi-0.0.0 >$SGDIR/npfo-resolved.0 & SGPID="$! $SGPID" { logresolve | sgsh-tee -I } <$SGDIR/npi-0.0.1 >$SGDIR/npfo-resolved.1 & SGPID="$! $SGPID" { logresolve | sgsh-tee -I } <$SGDIR/npi-0.0.2 >$SGDIR/npfo-resolved.2 & SGPID="$! $SGPID" { logresolve | sgsh-tee -I } <$SGDIR/npi-0.0.3 >$SGDIR/npfo-resolved.3 & SGPID="$! $SGPID" { logresolve | sgsh-tee -I } <$SGDIR/npi-0.0.4 >$SGDIR/npfo-resolved.4 & SGPID="$! $SGPID" { logresolve | sgsh-tee -I } <$SGDIR/npi-0.0.5 >$SGDIR/npfo-resolved.5 & SGPID="$! $SGPID" { logresolve | sgsh-tee -I } <$SGDIR/npi-0.0.6 >$SGDIR/npfo-resolved.6 & SGPID="$! $SGPID" { logresolve | sgsh-tee -I } <$SGDIR/npi-0.0.7 >$SGDIR/npfo-resolved.7 & SGPID="$! $SGPID" { logresolve | sgsh-tee -I } <$SGDIR/npi-0.0.8 >$SGDIR/npfo-resolved.8 & SGPID="$! $SGPID" { logresolve | sgsh-tee -I } <$SGDIR/npi-0.0.9 >$SGDIR/npfo-resolved.9 & SGPID="$! $SGPID" # Gather the results # Merge the files on the second numerical field sort -m -k2n $SGDIR/npfo-resolved.0 $SGDIR/npfo-resolved.1 $SGDIR/npfo-resolved.2 $SGDIR/npfo-resolved.3 $SGDIR/npfo-resolved.4 $SGDIR/npfo-resolved.5 $SGDIR/npfo-resolved.6 $SGDIR/npfo-resolved.7 $SGDIR/npfo-resolved.8 $SGDIR/npfo-resolved.9 | # Remove second field cut -d ' ' -f 1,3- ) 3<&0 ================================================ FILE: core-tools/tests-regression/regression/scripts/spell-highlight.ok ================================================ #!/bin/sh # Automatically generated file # Source file example/spell-highlight.sh #!/usr/bin/env sgsh # # SYNOPSIS Highlight misspelled words # DESCRIPTION # Highlight the words that are misspelled in the command's standard # input. # Demonstrates stream buffering and the avoidance of pass-through # constructs to avoid deadlocks. # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # export LC_ALL=C ( export SGDIR=/tmp/sg-$$.0 rm -rf $SGDIR # Cleanup on exit or interrupt cleanup() { SIGNAL=$1 [ $SIGNAL = EXIT ] || echo sgsh interrupted. Cleaning up... 1>&2 # Stop key-value stores # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npfo-dict.0 \ $SGDIR/npfo-errors.0 \ $SGDIR/npfo-text.0 \ $SGDIR/npi-0.1.0 \ $SGDIR/npi-0.2.0 sgsh-tee -o $SGDIR/npi-0.1.0 -o $SGDIR/npi-0.2.0 <&3 3<&- & SGPID="$! $SGPID" { sort /usr/share/dict/words }$SGDIR/npfo-dict.0 & SGPID="$! $SGPID" { tr -cs A-Za-z \\n | tr A-Z a-z | sort -u | comm -23 - $SGDIR/npfo-dict.0 } <$SGDIR/npi-0.1.0 >$SGDIR/npfo-errors.0 & SGPID="$! $SGPID" { cat } <$SGDIR/npi-0.2.0 >$SGDIR/npfo-text.0 & SGPID="$! $SGPID" # Gather the results fgrep -f $SGDIR/npfo-errors.0 -i --color -w -C 2 $SGDIR/npfo-text.0 ) 3<&0 ================================================ FILE: core-tools/tests-regression/regression/scripts/text-properties.ok ================================================ #!/bin/sh # Automatically generated file # Source file example/text-properties.sh #!/usr/bin/env sgsh # # SYNOPSIS Text properties # DESCRIPTION # Read text from the standard input and create files # containing word, character, digram, and trigram frequencies. # # Demonstrates the use of scatter blocks without output and the use # of stores within the scatter block. # # Example: # curl ftp://sunsite.informatik.rwth-aachen.de/pub/mirror/ibiblio/gutenberg/1/3/139/139.txt | text-properties # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Consitent sorting across machines export LC_ALL=C # Convert input into a ranked frequency list ranked_frequency() { sort | uniq -c | sort -rn } # Convert standard input to a ranked frequency list of specified n-grams ngram() { local N=$1 perl -ne 'for ($i = 0; $i < length($_) - '$N'; $i++) { print substr($_, $i, '$N'), "\n"; }' | ranked_frequency } ( export SGDIR=/tmp/sg-$$.0 rm -rf $SGDIR # Cleanup on exit or interrupt cleanup() { SIGNAL=$1 [ $SIGNAL = EXIT ] || echo sgsh interrupted. Cleaning up... 1>&2 # Stop key-value stores sgsh-readval -q -s "$SGDIR/NCHARS" 2>/dev/null # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npfo-none-0.2.0 \ $SGDIR/npfo-none-1.0.0 \ $SGDIR/npfo-none-1.1.0 \ $SGDIR/npfo-none-1.2.0 \ $SGDIR/npi-0.0.0 \ $SGDIR/npi-0.1.0 \ $SGDIR/npi-0.2.0 \ $SGDIR/npi-1.0.0 \ $SGDIR/npi-1.1.0 \ $SGDIR/npi-1.2.0 sgsh-tee -o $SGDIR/npi-0.0.0 -o $SGDIR/npi-0.1.0 -o $SGDIR/npi-0.2.0 <&3 3<&- & SGPID="$! $SGPID" { tr -cs a-zA-Z \\n } <$SGDIR/npi-0.0.0 | sgsh-tee -o $SGDIR/npi-1.0.0 -o $SGDIR/npi-1.1.0 -o $SGDIR/npi-1.2.0 & SGPID="$! $SGPID" { ngram 2 >digram.txt : ; } <$SGDIR/npi-1.0.0 >$SGDIR/npfo-none-1.0.0 & SGPID="$! $SGPID" { ngram 3 >trigram.txt : ; } <$SGDIR/npi-1.1.0 >$SGDIR/npfo-none-1.1.0 & SGPID="$! $SGPID" { ranked_frequency >words.txt : ; } <$SGDIR/npi-1.2.0 >$SGDIR/npfo-none-1.2.0 & SGPID="$! $SGPID" { wc -c } <$SGDIR/npi-0.1.0 | sgsh-writeval -s $SGDIR/NCHARS & SGPID="$! $SGPID" { sed 's/./&\ /g' | # Print absolute and percentage ranked_frequency | awk 'BEGIN {OFMT = "%.2g%%"} {print $1, $2, $1 / '"`sgsh-readval -s $SGDIR/NCHARS`"' * 100}' >character.txt : ; } <$SGDIR/npi-0.2.0 >$SGDIR/npfo-none-0.2.0 & SGPID="$! $SGPID" # Gather the results sgsh-tee -i $SGDIR/npfo-none-1.0.0 -i $SGDIR/npfo-none-1.1.0 -i $SGDIR/npfo-none-1.2.0 -i $SGDIR/npfo-none-0.2.0 >/dev/null ) 3<&0 ================================================ FILE: core-tools/tests-regression/regression/scripts/web-log-report.ok ================================================ #!/bin/sh # Automatically generated file # Source file example/web-log-report.sh #!/usr/bin/env sgsh -s /bin/bash # # SYNOPSIS Web log reporting # DESCRIPTION # Creates a report for a fixed-size web log file read from the standard input. # Demonstrates the combined use of stores and streams, the use of shell group # commands and functions in the scatter block, and the use of cat(1) as # a way to sequentially combine multiple streams. # Used to measure throughput increase achieved through parallelism. # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Output the top X elements of the input by the number of their occurrences # X is the first argument toplist() { uniq -c | sort -rn | head -$1 } # Output the argument as a section header header() { echo echo "$1" echo "$1" | sed 's/./-/g' } # Consistent sorting export LC_ALL=C ( export SGDIR=/tmp/sg-$$.0 rm -rf $SGDIR # Cleanup on exit or interrupt cleanup() { SIGNAL=$1 [ $SIGNAL = EXIT ] || echo sgsh interrupted. Cleaning up... 1>&2 # Stop key-value stores sgsh-readval -q -s "$SGDIR/nXBytes" 2>/dev/null sgsh-readval -q -s "$SGDIR/nLogBytes" 2>/dev/null sgsh-readval -q -s "$SGDIR/nAccess" 2>/dev/null sgsh-readval -q -s "$SGDIR/nHosts" 2>/dev/null sgsh-readval -q -s "$SGDIR/nTLD" 2>/dev/null sgsh-readval -q -s "$SGDIR/nDomain" 2>/dev/null sgsh-readval -q -s "$SGDIR/nPages" 2>/dev/null sgsh-readval -q -s "$SGDIR/nDays" 2>/dev/null # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npfo-accessByDate.0 \ $SGDIR/npfo-accessByDoW.0 \ $SGDIR/npfo-accessByHour.0 \ $SGDIR/npfo-summary.0 \ $SGDIR/npfo-top10Domain.0 \ $SGDIR/npfo-top10HostsByN.0 \ $SGDIR/npfo-top10HostsByVol.0 \ $SGDIR/npfo-top20Area.0 \ $SGDIR/npfo-top20Request.0 \ $SGDIR/npfo-top20TLD.0 \ $SGDIR/npi-0.0.0 \ $SGDIR/npi-0.1.0 \ $SGDIR/npi-0.2.0 \ $SGDIR/npi-0.3.0 \ $SGDIR/npi-0.4.0 \ $SGDIR/npi-0.5.0 \ $SGDIR/npi-1.0.0 \ $SGDIR/npi-1.1.0 \ $SGDIR/npi-1.2.0 \ $SGDIR/npi-1.3.0 \ $SGDIR/npi-2.0.0 \ $SGDIR/npi-2.1.0 \ $SGDIR/npi-3.0.0 \ $SGDIR/npi-3.1.0 \ $SGDIR/npi-4.0.0 \ $SGDIR/npi-4.1.0 \ $SGDIR/npi-5.0.0 \ $SGDIR/npi-5.1.0 \ $SGDIR/npi-5.2.0 \ $SGDIR/npi-6.0.0 \ $SGDIR/npi-6.1.0 \ $SGDIR/npi-7.0.0 \ $SGDIR/npi-7.1.0 \ $SGDIR/npi-7.2.0 sgsh-tee -o $SGDIR/npi-0.0.0 -o $SGDIR/npi-0.1.0 -o $SGDIR/npi-0.2.0 -o $SGDIR/npi-0.3.0 -o $SGDIR/npi-0.4.0 -o $SGDIR/npi-0.5.0 <&3 3<&- & SGPID="$! $SGPID" { awk '{s += $NF} END {print s}' } <$SGDIR/npi-0.0.0 | sgsh-writeval -s $SGDIR/nXBytes & SGPID="$! $SGPID" { wc -c } <$SGDIR/npi-0.1.0 | sgsh-writeval -s $SGDIR/nLogBytes & SGPID="$! $SGPID" { awk '{print $1}' } <$SGDIR/npi-0.2.0 | sgsh-tee -o $SGDIR/npi-1.0.0 -o $SGDIR/npi-1.1.0 -o $SGDIR/npi-1.2.0 -o $SGDIR/npi-1.3.0 & SGPID="$! $SGPID" { wc -l } <$SGDIR/npi-1.0.0 | sgsh-writeval -s $SGDIR/nAccess & SGPID="$! $SGPID" { sort } <$SGDIR/npi-1.1.0 | sgsh-tee -o $SGDIR/npi-2.0.0 -o $SGDIR/npi-2.1.0 & SGPID="$! $SGPID" { uniq } <$SGDIR/npi-2.0.0 | sgsh-tee -o $SGDIR/npi-3.0.0 -o $SGDIR/npi-3.1.0 & SGPID="$! $SGPID" { wc -l } <$SGDIR/npi-3.0.0 | sgsh-writeval -s $SGDIR/nHosts & SGPID="$! $SGPID" { awk -F. '$NF !~ /[0-9]/ {print $NF}' | sort -u | wc -l } <$SGDIR/npi-3.1.0 | sgsh-writeval -s $SGDIR/nTLD & SGPID="$! $SGPID" { { header 'Top 10 Hosts' toplist 10 } } <$SGDIR/npi-2.1.0 >$SGDIR/npfo-top10HostsByN.0 & SGPID="$! $SGPID" { { header 'Top 20 Level Domain Accesses' awk -F. '$NF !~ /^[0-9]/ {print $NF}' | sort | toplist 20 } } <$SGDIR/npi-1.2.0 >$SGDIR/npfo-top20TLD.0 & SGPID="$! $SGPID" { awk -F. 'BEGIN {OFS = "."} $NF !~ /^[0-9]/ {$1 = ""; print}' | sort } <$SGDIR/npi-1.3.0 | sgsh-tee -o $SGDIR/npi-4.0.0 -o $SGDIR/npi-4.1.0 & SGPID="$! $SGPID" { uniq | wc -l } <$SGDIR/npi-4.0.0 | sgsh-writeval -s $SGDIR/nDomain & SGPID="$! $SGPID" { { header 'Top 10 Domains' toplist 10 } } <$SGDIR/npi-4.1.0 >$SGDIR/npfo-top10Domain.0 & SGPID="$! $SGPID" { { header 'Top 10 Hosts by Transfer' awk ' {bytes[$1] += $NF} END {for (h in bytes) print bytes[h], h}' | sort -rn | head -10 } } <$SGDIR/npi-0.3.0 >$SGDIR/npfo-top10HostsByVol.0 & SGPID="$! $SGPID" { awk '{print $7}' | sort } <$SGDIR/npi-0.4.0 | sgsh-tee -o $SGDIR/npi-5.0.0 -o $SGDIR/npi-5.1.0 -o $SGDIR/npi-5.2.0 & SGPID="$! $SGPID" { { header 'Top 20 Area Requests' awk -F/ '{print $2}' | toplist 20 } } <$SGDIR/npi-5.0.0 >$SGDIR/npfo-top20Area.0 & SGPID="$! $SGPID" { uniq | wc -l } <$SGDIR/npi-5.1.0 | sgsh-writeval -s $SGDIR/nPages & SGPID="$! $SGPID" { { header 'Top 20 Requests' toplist 20 } } <$SGDIR/npi-5.2.0 >$SGDIR/npfo-top20Request.0 & SGPID="$! $SGPID" { awk '{print substr($4, 2)}' } <$SGDIR/npi-0.5.0 | sgsh-tee -o $SGDIR/npi-6.0.0 -o $SGDIR/npi-6.1.0 & SGPID="$! $SGPID" { awk -F: '{print $1}' } <$SGDIR/npi-6.0.0 | sgsh-tee -o $SGDIR/npi-7.0.0 -o $SGDIR/npi-7.1.0 -o $SGDIR/npi-7.2.0 & SGPID="$! $SGPID" { uniq | wc -l } <$SGDIR/npi-7.0.0 | sgsh-writeval -s $SGDIR/nDays & SGPID="$! $SGPID" { { header 'Accesses by Date' uniq -c } } <$SGDIR/npi-7.1.0 >$SGDIR/npfo-accessByDate.0 & SGPID="$! $SGPID" { { header 'Accesses by Day of Week' sed 's|/|-|g' | (date -f - +%a 2>/dev/null || gdate -f - +%a) | sort | uniq -c | sort -rn } } <$SGDIR/npi-7.2.0 >$SGDIR/npfo-accessByDoW.0 & SGPID="$! $SGPID" { { header 'Accesses by Local Hour' awk -F: '{print $2}' | sort | uniq -c } } <$SGDIR/npi-6.1.0 >$SGDIR/npfo-accessByHour.0 & SGPID="$! $SGPID" { cat <$SGDIR/npfo-summary.0 & SGPID="$! $SGPID" # Gather the results # Gather all results together sequentially into a single report sgsh-tee -f -I -i $SGDIR/npfo-summary.0 -i $SGDIR/npfo-top20Request.0 -i $SGDIR/npfo-top20Area.0 -i $SGDIR/npfo-top10HostsByN.0 -i $SGDIR/npfo-top10HostsByVol.0 -i $SGDIR/npfo-top10Domain.0 -i $SGDIR/npfo-top20TLD.0 -i $SGDIR/npfo-accessByDoW.0 -i $SGDIR/npfo-accessByHour.0 -i $SGDIR/npfo-accessByDate.0 ) 3<&0 ================================================ FILE: core-tools/tests-regression/regression/scripts/web-log-stats.ok ================================================ #!/bin/sh # Automatically generated file # Source file example/web-log-stats.sh #!/usr/bin/env sgsh -s /bin/bash # # SYNOPSIS Web log statistics # DESCRIPTION # Provides continuous statistics over web log stream data. # Demonstrates stream processing. # Provide as an argument either the name of a growing web log file # or -s and a static web log file, which will be processed at a rate # of about 10 lines per second. # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Size of the window to report in seconds WINDOW=10 WINDOW_OLD=$(expr $WINDOW \* 2) # Update interval in seconds UPDATE=2 # Print the sum of the numbers read from the standard input sum() { awk '{ sum += $1 } END {print sum}' } # Print the rate of change as a percentage # between the first (old) and second (new) value change() { # Can't use bc, because we have numbers in scientific notation awk "END {OFMT=\"%.2f%%\"; print ($2 - $1) * 100 / $1}" &2 # Stop key-value stores sgsh-readval -q -s "$SGDIR/page" 2>/dev/null sgsh-readval -q -s "$SGDIR/total_bytes" 2>/dev/null sgsh-readval -q -s "$SGDIR/total_pages" 2>/dev/null sgsh-readval -q -s "$SGDIR/bytes" 2>/dev/null sgsh-readval -q -s "$SGDIR/bytes_old" 2>/dev/null # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npi-0.0.0 \ $SGDIR/npi-0.1.0 \ $SGDIR/npi-1.0.0 \ $SGDIR/npi-1.1.0 \ $SGDIR/npi-1.2.0 \ $SGDIR/npi-1.3.0 sgsh-tee -o $SGDIR/npi-0.0.0 -o $SGDIR/npi-0.1.0 <&3 3<&- & SGPID="$! $SGPID" { awk -Winteractive '{print $7}' } <$SGDIR/npi-0.0.0 | sgsh-writeval -b $WINDOW -u s -s $SGDIR/page & SGPID="$! $SGPID" { awk -Winteractive '{print $10}' } <$SGDIR/npi-0.1.0 | sgsh-tee -o $SGDIR/npi-1.0.0 -o $SGDIR/npi-1.1.0 -o $SGDIR/npi-1.2.0 -o $SGDIR/npi-1.3.0 & SGPID="$! $SGPID" { awk -Winteractive '{ s += $1; print s}' } <$SGDIR/npi-1.0.0 | sgsh-writeval -s $SGDIR/total_bytes & SGPID="$! $SGPID" { awk -Winteractive '{print ++n}' } <$SGDIR/npi-1.1.0 | sgsh-writeval -s $SGDIR/total_pages & SGPID="$! $SGPID" { } <$SGDIR/npi-1.2.0 sgsh-writeval -b $WINDOW -u s -s $SGDIR/bytes & SGPID="$! $SGPID" { } <$SGDIR/npi-1.3.0 sgsh-writeval -b $WINDOW_OLD -e $WINDOW -u s -s $SGDIR/bytes_old & SGPID="$! $SGPID" # Gather the results # Produce periodic reports while : do WINDOW_PAGES=$(sgsh-readval -s $SGDIR/bytes -c | wc -l) WINDOW_BYTES=$(sgsh-readval -s $SGDIR/bytes -c | sum ) WINDOW_PAGES_OLD=$(sgsh-readval -s $SGDIR/bytes_old -c | wc -l) WINDOW_BYTES_OLD=$(sgsh-readval -s $SGDIR/bytes_old -c | sum) clear cat <&2 # Stop key-value stores # Kill processes we have launched in the background kill $SGPID 2>/dev/null # Remove temporary directory rm -rf "$SGDIR" # Propagate real signals and exit with non-0 if [ $SIGNAL != EXIT ] then trap - $SIGNAL EXIT kill -s $SIGNAL $$ fi # Exit with the original exit value exit } for sig in HUP INT QUIT TERM EXIT do trap "cleanup $sig" $sig done mkdir $SGDIR mkfifo $SGDIR/npfo-consonants.0 \ $SGDIR/npfo-long.0 \ $SGDIR/npfo-palindromes.0 \ $SGDIR/npi-0.0.0 \ $SGDIR/npi-0.1.0 \ $SGDIR/npi-0.2.0 \ $SGDIR/npi-0.3.0 sgsh-tee -o $SGDIR/npi-0.0.0 -o $SGDIR/npi-0.1.0 -o $SGDIR/npi-0.2.0 -o $SGDIR/npi-0.3.0 <&3 3<&- & SGPID="$! $SGPID" ln -s $SGDIR/npi-0.0.0 $SGDIR/npfo-words.0 { sed 's/.*\(.\)\(.\)\2\1.*/p: \1\2-\2\1/;t g' } <$SGDIR/npi-0.1.0 >$SGDIR/npfo-palindromes.0 & SGPID="$! $SGPID" { sed -E 's/.*([^aeiouyAEIOUY]{4}).*/c: \1/;t g' } <$SGDIR/npi-0.2.0 >$SGDIR/npfo-consonants.0 & SGPID="$! $SGPID" { awk '{if (length($1) > 12) print "l:", length($1); else print ""}' } <$SGDIR/npi-0.3.0 >$SGDIR/npfo-long.0 & SGPID="$! $SGPID" # Gather the results # Paste the four streams side-by-side paste $SGDIR/npfo-words.0 $SGDIR/npfo-palindromes.0 $SGDIR/npfo-consonants.0 $SGDIR/npfo-long.0 | # List only words satisfying one or more properties grep : ) 3<&0 ================================================ FILE: core-tools/tests-regression/regression/warnings/single-target.ok ================================================ test/regression/warnings/single-target.sh(26): warning: Scatter to a single target ================================================ FILE: core-tools/tests-regression/regression/warnings/single-target.sh ================================================ #!/usr/bin/env sgsh # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # scatter |{ #-| scatter1 |store:S -| scatter2 |{ -| toa |>/stream/a -| tob |>/stream/b |} |} gather |{ cat /stream/a /stream/b |} ================================================ FILE: core-tools/tests-regression/regression/warnings/unsafe-scatter.ok ================================================ test/regression/warnings/unsafe-scatter.sh(21): warning: Unsafe use of pass-through /stream/b in the scatter section test/regression/warnings/unsafe-scatter.sh(21): warning: Consult the DEADLOCK section of the manual page ================================================ FILE: core-tools/tests-regression/regression/warnings/unsafe-scatter.sh ================================================ #!/usr/bin/env sgsh # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # scatter |{ -| cat |>/stream/a -||>/stream/b .| cat /stream/a /stream/b |>/stream/c |} gather |{ cat /stream/c |} ================================================ FILE: core-tools/tests-regression/spell-highlight/out.ok ================================================ hello cruwl world ================================================ FILE: core-tools/tests-regression/tee/oom.err ================================================ dgsh-tee: Out of memory with input-side buffering specified ================================================ FILE: core-tools/tests-regression/tee/perm.ok ================================================ 3 1 2 0 ================================================ FILE: core-tools/tests-regression/tee/tee-fastout-I-l.ok ================================================ Buffers allocated: 1025 Freed: 1024 Maximum allocated: 2 ================================================ FILE: core-tools/tests-regression/tee/tee-fastout-I-m 2k -f.ok ================================================ Buffers allocated: 1025 Freed: 1024 Maximum allocated: 2 Page out: 0 In: 0 Pages freed: 0 ================================================ FILE: core-tools/tests-regression/tee/tee-fastout-I-m 2k.ok ================================================ Buffers allocated: 1025 Freed: 1024 Maximum allocated: 2 Page out: 0 In: 0 Pages freed: 0 ================================================ FILE: core-tools/tests-regression/tee/tee-fastout-I.ok ================================================ Buffers allocated: 1025 Freed: 1024 Maximum allocated: 2 Page out: 0 In: 0 Pages freed: 0 ================================================ FILE: core-tools/tests-regression/tee/tee-fastout-l.ok ================================================ Buffers allocated: 1025 Freed: 1024 Maximum allocated: 1 ================================================ FILE: core-tools/tests-regression/tee/tee-fastout-m 2k -f.ok ================================================ Buffers allocated: 1025 Freed: 1024 Maximum allocated: 1 Page out: 0 In: 0 Pages freed: 0 ================================================ FILE: core-tools/tests-regression/tee/tee-fastout-m 2k.ok ================================================ Buffers allocated: 1025 Freed: 1024 Maximum allocated: 1 Page out: 0 In: 0 Pages freed: 0 ================================================ FILE: core-tools/tests-regression/tee/tee-fastout.ok ================================================ Buffers allocated: 1025 Freed: 1024 Maximum allocated: 1 Page out: 0 In: 0 Pages freed: 0 ================================================ FILE: core-tools/tests-regression/tee/tee-lagout-I-l.ok ================================================ Buffers allocated: 1025 Freed: 1024 Maximum allocated: 960 ================================================ FILE: core-tools/tests-regression/tee/tee-lagout-I-m 2k -f.ok ================================================ Buffers allocated: 1987 Freed: 1986 Maximum allocated: 3 Page out: 962 In: 962 Pages freed: 962 ================================================ FILE: core-tools/tests-regression/tee/tee-lagout-I-m 2k.ok ================================================ sgsh-tee: Out of memory with input-side buffering specified ================================================ FILE: core-tools/tests-regression/tee/tee-lagout-I.ok ================================================ Buffers allocated: 1025 Freed: 1024 Maximum allocated: 964 Page out: 0 In: 0 Pages freed: 0 ================================================ FILE: core-tools/tests-regression/tee/tee-lagout-l.ok ================================================ Buffers allocated: 1025 Freed: 1024 Maximum allocated: 1 ================================================ FILE: core-tools/tests-regression/tee/tee-lagout-m 2k -f.ok ================================================ Buffers allocated: 1025 Freed: 1024 Maximum allocated: 1 Page out: 0 In: 0 Pages freed: 0 ================================================ FILE: core-tools/tests-regression/tee/tee-lagout-m 2k.ok ================================================ Buffers allocated: 1025 Freed: 1024 Maximum allocated: 1 Page out: 0 In: 0 Pages freed: 0 ================================================ FILE: core-tools/tests-regression/tee/tee-lagout.ok ================================================ Buffers allocated: 1025 Freed: 1024 Maximum allocated: 1 Page out: 0 In: 0 Pages freed: 0 ================================================ FILE: core-tools/tests-regression/tee/tee-lahout.ok ================================================ Buffers allocated: 1025 Freed: 1024 Maximum allocated: 1 ================================================ FILE: core-tools/tests-regression/test-dgsh.sh ================================================ #!/bin/sh # # Regression testing of the provided examples # TOP=$(cd ../.. ; pwd) DGSH="$TOP/build/bin/dgsh" PATH="$TOP/build/bin:$PATH" export DGSHPATH="$TOP/build/libexec/dgsh" EXAMPLE="$TOP/example" # Ensure that the generated test file matches the reference one # File names are by conventions $base/out.{ok,test} ensure_same() { local base=$1 echo -n "$base.sh " if diff -rw $base/out.ok $base/out.test then echo OK else echo "$base.sh: Files differ: $base/out.ok $base/out.test" 1>&2 exit 1 fi } # Include fallback commands in our executable path export PATH="$PATH:bin" LOGFILE=web-log-report/logfile # Generate a small log file if it is not there # See http://ita.ee.lbl.gov/html/contrib/ClarkNet-HTTP.html CLARKNET=ftp://ita.ee.lbl.gov/traces/clarknet_access_log_Aug28.gz if ! [ -f $LOGFILE ] then mkdir -p $(dirname $LOGFILE) echo "Fetching web test data" { curl $CLARKNET 2>/dev/null || wget -O - $CLARKNET 2>/dev/null } | gzip -dc | perl -ne 'print if ($i++ % 1000 == 0)' >$LOGFILE fi rm -rf */out.test mkdir -p spell-highlight echo hello cruwl world | $DGSH $EXAMPLE/spell-highlight.sh >spell-highlight/out.test ensure_same spell-highlight $DGSH $EXAMPLE/map-hierarchy.sh map-hierarchy/in/a map-hierarchy/in/b map-hierarchy/out.test ensure_same map-hierarchy if [ -f "$TOP/unix-tools/grep/.git" ] && [ -d "$TOP/.git" ]; then ( cd $TOP/unix-tools/grep LC_ALL=C $DGSH $EXAMPLE/commit-stats.sh --since=2010-01-01Z00:00 \ --until=2015-12-31Z23:59 \ >$TOP/core-tools/tests-regression/commit-stats/out.test ) ensure_same commit-stats else echo "Skip commit-stats test because input is missing (grep's git repo)" fi # This example outputs different result for NMACRO than the template # due to a variation in the behavior of grep or a variation in the # implementation of the sgsh example in regression/scripts/code-metrics.ok. # The test succeeds in Mac OS. #DGSH_TIMEOUT=20 $DGSH $EXAMPLE/code-metrics.sh code-metrics/in/ >code-metrics/out.test 2>/dev/null #ensure_same code-metrics $DGSH $EXAMPLE/duplicate-files.sh duplicate-files >duplicate-files/out.test ensure_same duplicate-files $DGSH $EXAMPLE/word-properties.sh word-properties/out.test ensure_same word-properties $DGSH $EXAMPLE/compress-compare.sh compress-compare/out.test ensure_same compress-compare KVSTORE_RETRY_LIMIT=100 DGSH_TIMEOUT=60 $DGSH $EXAMPLE/web-log-report.sh web-log-report/out.test ensure_same web-log-report ( cd text-properties rm -rf out.test mkdir out.test cd out.test $DGSH $EXAMPLE/text-properties.sh <../../word-properties/LostWorldChap1-3 ) ensure_same text-properties # The correct file was generated using # tr -s ' \t\n\r\f' \\n parallel-word-count/out.test ensure_same parallel-word-count $DGSH $EXAMPLE/author-compare.sh conf/icse/ journals/software/ \ author-compare/out.test ensure_same author-compare exit 0 ================================================ FILE: core-tools/tests-regression/test-kvstore.sh ================================================ #!/bin/sh DGSH_READVAL=../src/dgsh-readval DGSH_WRITEVAL=../src/dgsh-writeval DGSH_HTTPVAL=../src/dgsh-httpval # Helper functions {{{1 # Repeat x times sequence() { jot $1 2>/dev/null || seq $1 2>/dev/null } fail() { echo "FAIL" echo "$1" $DGSH_READVAL -q -s testsocket 2>/dev/null if [ "$SERVER_PID" ] then stop_server fi echo echo Server errors: test server.err && cat server.err echo Client errors: test client.err && cat client.err exit 1 } # Verify a test result # -n disables the termination of writeval check() { if [ "$1" != '-n' ] then $DGSH_READVAL -q -s testsocket 2>/dev/null fi if [ "$TRY" != "$EXPECT" ] then fail "Expected [$EXPECT] got [$TRY]" else echo "OK" fi } # A section of the test section() { SECTION="$1" echo "$1" rm -f client.err server.err } # A particular test case testcase() { TESTCASE="$1" echo -n " $1: " } # Start the HTTP server start_server() { $DGSH_HTTPVAL -p $PORT "$@" & SERVER_PID=$! } # Stop the HTTP server stop_server() { curl -s40 http://localhost:$PORT/.server?quit >/dev/null sleep 1 kill $SERVER_PID 2>/dev/null sleep 2 SERVER_PID='' } # Simple tests {{{1 section 'Simple tests' # {{{2 testcase "Single record" # {{{3 echo single record | $DGSH_WRITEVAL -s testsocket 2>server.err & sleep 1 TRY="`$DGSH_READVAL -l -s testsocket 2>client.err `" EXPECT='single record' check testcase "Single record fixed width" # {{{3 printf 0123456789 | $DGSH_WRITEVAL -l 10 -s testsocket 2>server.err & sleep 1 EXPECT='0123456789' TRY="`$DGSH_READVAL -l -s testsocket 2>client.err `" check testcase "Record separator" # {{{3 printf record1:record2 | $DGSH_WRITEVAL -t : -s testsocket 2>server.err & sleep 1 TRY="`$DGSH_READVAL -l -s testsocket 2>client.err `" EXPECT='record1:' check sleep 1 testcase "HTTP interface - text data" # {{{3 PORT=53843 echo single record | $DGSH_WRITEVAL -s testsocket 2>server.err & start_server # -s40: silent, IPv4 HTTP 1.0 TRY="`curl -s40 http://localhost:$PORT/testsocket`" EXPECT='single record' check stop_server testcase "HTTP interface - non-blocking empty record" # {{{3 PORT=53843 { sleep 2 ; echo hi ; } | $DGSH_WRITEVAL -s testsocket 2>server.err & start_server -n # -s40: silent, IPv4 HTTP 1.0 TRY="`curl -s40 http://localhost:$PORT/testsocket`" EXPECT='' check stop_server testcase "HTTP interface - non-blocking first record" # {{{3 PORT=53843 echo 'first record' | $DGSH_WRITEVAL -s testsocket 2>server.err & start_server -n # -s40: silent, IPv4 HTTP 1.0 sleep 1 TRY="`curl -s40 http://localhost:$PORT/testsocket`" EXPECT='first record' check stop_server testcase "HTTP interface - non-blocking second record" # {{{3 PORT=53843 ( echo 'first record' ; sleep 1 ; echo 'second record' ) | $DGSH_WRITEVAL -s testsocket 2>server.err & start_server -n # -s40: silent, IPv4 HTTP 1.0 sleep 2 TRY="`curl -s40 http://localhost:$PORT/testsocket`" EXPECT='second record' check stop_server testcase "HTTP interface - binary data" # {{{3 PORT=53843 perl -e 'BEGIN { binmode STDOUT; } for ($i = 0; $i < 256; $i++) { print chr($i); }' | $DGSH_WRITEVAL -l 256 -s testsocket 2>server.err & start_server -m application/octet-stream # -s40: silent, IPv4 HTTP 1.0 curl -s40 http://localhost:$PORT/testsocket | perl -e 'BEGIN { binmode STDIN; } for ($i = 0; $i < 256; $i++) { $expect .= chr($i); } read STDIN, $try, 256; if ($try ne $expect) { print "FAIL\n"; print "Expected [$expect] got [$try]\n"; } else { print "OK\n"; } exit ($try ne $expect); ' EXITCODE=$? stop_server $DGSH_READVAL -q -s testsocket 2>client.err 1>/dev/null test $EXITCODE = 0 || exit 1 testcase "HTTP interface - large data" # {{{3 PORT=53843 dd if=/dev/zero bs=1000000 count=1 2>/dev/null | $DGSH_WRITEVAL -l 1000000 -s testsocket 2>server.err & start_server -m application/octet-stream # -s40: silent, IPv4 HTTP 1.0 BYTES=$(curl -s40 http://localhost:$PORT/testsocket | wc -c | sed 's/^[^0-9]*//') stop_server $DGSH_READVAL -q -s testsocket 2>client.err 1>/dev/null if [ "$BYTES" != 1000000 ] then echo FAIL echo "Expected [1000000 bytes] got [$BYTES bytes]" exit 1 else echo "OK" fi # Last record tests {{{1 section 'Reading of fixed-length records in stream' # {{{2 (printf A12345A7AB; sleep 4; printf 12345B7BC; sleep 4; printf 12345C7CD) | $DGSH_WRITEVAL -l 9 -s testsocket 2>server.err & testcase "Record one" # {{{3 sleep 2 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err`" EXPECT=A12345A7A check -n testcase "Record two" # {{{3 sleep 4 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT=B12345B7B check -n testcase "Record three" # {{{3 sleep 4 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err`" EXPECT=C12345C7C check # The socket is no longer there # Verify that readval doesn't block retrying $DGSH_READVAL -nq -s testsocket 2>client.err section 'Reading of newline-separated records in stream' # {{{2 (echo record one; sleep 4; echo record two; sleep 4; echo record three) | $DGSH_WRITEVAL -s testsocket 2>server.err & testcase "Record one" # {{{3 sleep 2 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='record one' check -n testcase "Record two" # {{{3 sleep 4 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='record two' check -n testcase "Record three" # {{{3 sleep 4 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='record three' check section 'Non-blocking reading of newline-separated records in stream' # {{{2 (sleep 2 ; echo record one; sleep 4; echo record two; sleep 6; echo record three) | $DGSH_WRITEVAL -s testsocket 2>server.err & testcase "Empty record" # {{{3 TRY="`$DGSH_READVAL -e -s testsocket 2>client.err `" EXPECT='' check -n testcase "Record one" # {{{3 sleep 3 TRY="`$DGSH_READVAL -e -s testsocket 2>client.err `" EXPECT='record one' check -n testcase "Record two" # {{{3 sleep 5 TRY="`$DGSH_READVAL -e -s testsocket 2>client.err `" EXPECT='record two' check section 'Reading last record' # {{{2 testcase "Last record" # {{{3 (echo record one; sleep 1; echo last record) | $DGSH_WRITEVAL -s testsocket 2>server.err & TRY="`$DGSH_READVAL -l -s testsocket 2>client.err `" EXPECT='last record' check testcase "Last record as default" # {{{3 (echo record one; sleep 1; echo last record) | $DGSH_WRITEVAL -s testsocket 2>server.err & TRY="`$DGSH_READVAL -s testsocket 2>client.err `" EXPECT='last record' check testcase "Empty record" # {{{3 $DGSH_WRITEVAL -s testsocket 2>server.err server.err & TRY="`$DGSH_READVAL -l -s testsocket 2>client.err `" EXPECT='' check # Window tests {{{1 section 'Window from stream' # {{{2 testcase "Second record" # {{{3 (echo first record ; echo second record ; echo third record; sleep 2) | $DGSH_WRITEVAL -b 2 -e 1 -s testsocket 2>server.err & sleep 1 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='second record' check testcase "First record" # {{{3 (echo first record ; echo second record ; echo third record; sleep 2) | $DGSH_WRITEVAL -b 3 -e 2 -s testsocket 2>server.err & sleep 1 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='first record' check testcase "Second record" # {{{3 (echo first record ; echo second record ; echo third record; sleep 2) | $DGSH_WRITEVAL -b 3 -e 1 -s testsocket 2>server.err & sleep 1 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='first record second record' check testcase "All records" # {{{3 (echo first record ; echo second record ; echo third record; sleep 2) | $DGSH_WRITEVAL -b 3 -e 0 -s testsocket 2>server.err & sleep 1 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='first record second record third record' check section 'Window from fixed record stream' # {{{2 testcase "Middle record" # {{{3 (printf 000 ; printf 011112 ; printf 222; sleep 2) | $DGSH_WRITEVAL -l 4 -b 2 -e 1 -s testsocket 2>server.err & sleep 1 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='1111' check testcase "First record" # {{{3 (printf 000 ; printf 011112 ; printf 222; sleep 2) | $DGSH_WRITEVAL -l 4 -b 3 -e 2 -s testsocket 2>server.err & sleep 1 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='0000' check testcase "First two records" # {{{3 (printf 000 ; printf 011112 ; printf 222; sleep 2) | $DGSH_WRITEVAL -l 4 -b 3 -e 1 -s testsocket 2>server.err & sleep 1 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='00001111' check testcase "All records" # {{{3 (printf 000 ; printf 011112 ; printf 222; sleep 2) | $DGSH_WRITEVAL -l 4 -b 3 -e 0 -s testsocket 2>server.err & sleep 1 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='000011112222' check # Time window tests {{{1 section 'Time window from terminated record stream' # {{{2 testcase "First record" # {{{3 (echo First record ; sleep 1 ; echo Second--record ; sleep 1 ; echo The third record; sleep 3) | $DGSH_WRITEVAL -u s -b 3.5 -e 2.5 -s testsocket 2>server.err & #Rel 3 2 1 sleep 3 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='First record' check testcase "Middle record" # {{{3 (echo First record ; sleep 1 ; echo Second record ; sleep 1 ; echo The third record; sleep 3) | $DGSH_WRITEVAL -u s -b 2.5 -e 1.5 -s testsocket 2>server.err & #Rel 3 2 1 sleep 3 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='Second record' check testcase "First two records (full buffer)" # {{{3 (echo First record ; sleep 1 ; echo Second--record ; sleep 1 ; echo The third record; sleep 3) | $DGSH_WRITEVAL -u s -b 3.5 -e 1.5 -s testsocket 2>server.err & #Rel 3 2 1 sleep 3 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='First record Second--record' check testcase "First two records (incomplete buffer)" # {{{3 (echo First record ; sleep 1 ; echo Second record ; sleep 1 ; echo The third record; sleep 3) | $DGSH_WRITEVAL -u s -b 3.5 -e 1.5 -s testsocket 2>server.err & #Rel 3 2 1 sleep 3 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='First record Second record' check testcase "Last record" # {{{3 (echo First record ; sleep 1 ; echo Second--record ; sleep 1 ; echo The third record; sleep 3) | $DGSH_WRITEVAL -u s -b 1.5 -e 0.5 -s testsocket 2>server.err & #Rel 3 2 1 sleep 3 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='The third record' check testcase "All records" # {{{3 (echo First record ; sleep 1 ; echo Second--record ; sleep 1 ; echo The third record; sleep 3) | $DGSH_WRITEVAL -u s -b 3.5 -e 0.5 -s testsocket 2>server.err & #Rel 3 2 1 sleep 3 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='First record Second--record The third record' check testcase "First record after wait" # {{{3 (echo First record ; sleep 1 ; echo Second--record ; sleep 1 ; echo The third record; sleep 3) | $DGSH_WRITEVAL -u s -b 4.5 -e 3.5 -s testsocket 2>server.err & #Rel 3 2 1 sleep 3 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='First record' check section 'Time window from fixed record stream' # {{{2 testcase "Fixed record stream, first record" # {{{3 (printf 000 ; sleep 1 ; printf 011112 ; sleep 1 ; printf 222; sleep 3) | $DGSH_WRITEVAL -l 4 -u s -b 3.5 -e 2.5 -s testsocket 2>server.err & #Rel 3 2 1 sleep 3 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='0000' check testcase "First two records" # {{{3 (printf 000 ; sleep 1 ; printf 011112 ; sleep 1 ; printf 222; sleep 3) | $DGSH_WRITEVAL -l 4 -u s -b 3.5 -e 1.5 -s testsocket 2>server.err & #Rel 3 2 1 sleep 3 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='000011112222' check testcase "Middle record" # {{{3 (printf 000 ; sleep 1 ; printf 011112 ; sleep 1 ; printf 222; sleep 3) | $DGSH_WRITEVAL -l 4 -u s -b 2.5 -e 1.5 -s testsocket 2>server.err & #Rel 3 2 1 sleep 3 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='11112222' check testcase "Last record" # {{{3 (printf 000 ; sleep 1 ; printf 01111 ; sleep 1 ; printf 222; printf 2 ; sleep 3) | $DGSH_WRITEVAL -l 4 -u s -b 1.5 -e 0.5 -s testsocket 2>server.err & #Rel 3 2 1 sleep 3 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='2222' check testcase "All records" # {{{3 (printf 000 ; sleep 1 ; printf 01111 ; sleep 1 ; printf 222; printf 2 ; sleep 3) | $DGSH_WRITEVAL -l 4 -u s -b 3.5 -e 0.5 -s testsocket 2>server.err & #Rel 3 2 1 sleep 3 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='000011112222' check testcase "First record after wait" # {{{3 (printf 000 ; sleep 1 ; printf 01111 ; sleep 1 ; printf 222; printf 2 ; sleep 3) | $DGSH_WRITEVAL -l 4 -u s -b 4.5 -e 3.5 -s testsocket 2>server.err & #Rel 3 2 1 sleep 3 TRY="`$DGSH_READVAL -c -s testsocket 2>client.err `" EXPECT='0000' check section 'Multi-client stress test' # {{{1 echo -n " Running" if [ ! -r test/sorted-words ] then mkdir -p test { cat /usr/share/dict/words 2>/dev/null || curl -s https://raw.githubusercontent.com/freebsd/freebsd/master/share/dict/web2 } | head -50000 | sort >test/sorted-words fi $DGSH_WRITEVAL -s testsocket 2>server.err client.err done >read-words-$i & done wait echo echo " All clients finished" $DGSH_READVAL -lq -s testsocket 2>client.err 1>/dev/null echo " Server finished" ) echo -n " Compare: " # Wait for the store to terminate wait for i in `sequence $NUMCLIENTS` do if sort -u read-words-$i | comm -13 test/sorted-words - | grep . then fail "Stress test client $i has trash output" fi if [ `wc -l < read-words-$i` -ne $NUMREADS ] then fail "Stress test client is missing output" fi done rm -f read-words-* echo "OK" section 'Time window stress test' # {{{1 echo -n " Running" # Feed words at a slower pace perl -pe 'select(undef, undef, undef, 0.001);' test/sorted-words | $DGSH_WRITEVAL -u s -b 3 -e 2 -s testsocket 2>server.err & echo echo " Server started" # Number of parallel clients to run NUMCLIENTS=5 # Number of read commands to issue NUMREADS=300 echo -n " Running clients" ( for i in `sequence $NUMCLIENTS` do for j in `sequence $NUMREADS` do $DGSH_READVAL -c -s testsocket 2>client.err done >read-words-$i & done wait echo echo " All clients finished" $DGSH_READVAL -q -s testsocket 2>client.err 1>/dev/null echo " Server finished" ) echo -n " Compare: " # Wait for the store to terminate wait for i in `sequence $NUMCLIENTS` do if sort -u read-words-$i | comm -13 test/sorted-words - | grep . then fail "Stress test client $i has trash output" fi if [ `wc -l < read-words-$i` -lt $NUMREADS ] then fail "Stress test client may be missing output" fi done rm -f read-words-* client.err server.err echo "OK" ================================================ FILE: core-tools/tests-regression/test-merge-sum.sh ================================================ #!/usr/bin/env bash # # Tests for dgsh-merge-sum # MERGE_SUM=../src/dgsh-merge-sum.pl # Shortcut testcase() { local name="$1" local expect="$2" local in="$3" shift 3 if ! diff <(perl $MERGE_SUM <"$in" "$@") $expect then echo 1>&2 "Test $name failed" exit 1 else echo 1>&2 "Test $name OK" fi } testcase merge <(cat </dev/null then echo "$1: $2 and $3 differ" 1>&2 exit 1 fi echo OK } # Produce statistics on the character count of the data received from # standard input charcount() { sed 's/./&\ /g' | sort | uniq -c } # Ensure that the numbers in the files passed as 2nd and 3rd arguments # about are the same # Line format: # Buffers allocated: 1025 Freed: 1024 Maximum allocated: 960 ensure_similar_buffers() { echo -n "$1 " for nr in 2 3 do for nf in 3 5 8 do if ! awk " function abs(v) { return (v < 0 ? -v : v) } NR == $nr {ref = \$$nf} NR == $nr + 3 {test = \$$nf} END { exit (ref + 0 != 0 && abs((test - ref) / ref * 100) > 10) ? 1 : 0 }" "$2" "$3" then echo "$1: Line $nr, fields $nf of $2 and $3 differ by more than 10%" 1>&2 exit 1 fi done done echo OK } for flags in '' -I do # Test two input files cat $DGSH_TEE_C $DGSH_TEE_C $WORDS >expected $DGSH_TEE $flags -b 64 -i $DGSH_TEE_C -i $DGSH_TEE_C -i $WORDS >a ensure_same "Three input $flags" expected a rm expected a # Test line scatter reliable algorithm (stdin) cat -n $WORDS >words $DGSH_TEE $flags -s -b 1000000 words2 ensure_same "Line scatter reliable stdin $flags" words words2 # Test line scatter reliable algorithm (pipe) cat -n $WORDS | $DGSH_TEE $flags -s -b 1000000 -o a -o b -o c -o d cat a b c d | sort -n >words2 ensure_same "Line scatter reliable pipe $flags" words words2 # Test line scatter reliable algorithm (file argument) cat -n $WORDS >words $DGSH_TEE $flags -s -b 1000000 -i words -o a -o b -o c -o d cat a b c d | sort -n >words2 ensure_same "Line scatter reliable file $flags" words words2 # Test scatter to blocking sinks cat -n $WORDS >words for buffer in 128 1000000 do $DGSH -c " $DGSH_TEE -b $buffer -s -i words | dgsh-parallel -n 8 cat | sort -mn >a" ensure_same "Scatter to blocking sinks $flags -b $buffer" words a done rm a # Test line scatter efficient algorithm $DGSH_TEE $flags -s -b 128 words2 ensure_same "Line scatter efficient $flags" words words2 # Test with a buffer smaller than line size $DGSH_TEE $flags -s -b 5 words2 ensure_same "Small buffer $flags" words words2 # Test with data less than the buffer size head -50 $WORDS | cat -n >words $DGSH_TEE $flags -s -b 1000000 words2 ensure_same "Large buffer $flags" words words2 rm words words2 # Test block scatter $DGSH_TEE $flags -s -b 64 <$DGSH_TEE_C -o a -o b -o c -o d charcount <$DGSH_TEE_C >orig cat a b c d | charcount >new ensure_same "Block scatter $flags" orig new rm a b c d orig new # Test plain distribution $DGSH_TEE $flags -b 64 <$DGSH_TEE_C -o a -o b ensure_same "Plain distribution $flags" $DGSH_TEE_C a ensure_same "Plain distribution $flags" $DGSH_TEE_C b rm a b # Test 2->4 distribution $DGSH_TEE $flags -b 64 -i $WORDS -i $DGSH_TEE_C -o a -o b -o c -o d ensure_same "2->4 distribution $flags" $WORDS a ensure_same "2->4 distribution $flags" $WORDS c ensure_same "2->4 distribution $flags" $DGSH_TEE_C b ensure_same "2->4 distribution $flags" $DGSH_TEE_C d rm a b c d # Test 4->2 distribution cat $WORDS /etc/services >result1 cat $DGSH_TEE_C /etc/hosts >result2 $DGSH_TEE $flags -b 64 -i $WORDS -i $DGSH_TEE_C -i /etc/services -i /etc/hosts -o a -o b ensure_same "4->2 distribution $flags" result1 a ensure_same "4->2 distribution $flags" result2 b rm a b result1 result2 # Test permutation $DGSH -c "$DGSH_ENUMERATE 4 | $DGSH_TEE -p 4,2,3,1 | $DGSH_TEE" >a ensure_same "Permutation $flags" a tee/perm.ok rm a # Test output to stdout $DGSH_TEE $flags -b 64 <$DGSH_TEE_C >a ensure_same "Stdout $flags" $DGSH_TEE_C a rm a # Test buffering # When -l is supported add, say, -l 16 for flags2 in '' '-m 2k' '-m 2k -f' do test="tee-fastout$flags$flags2" dd bs=1k count=1024 if=/dev/zero 2>/dev/null | $DGSH_TEE -M $flags $flags2 -b 1024 >/dev/null 2>"tee/$test.test" ensure_similar_buffers "$test" "tee/$test.ok" "tee/$test.test" test="tee-lagout$flags$flags2" dd bs=1k count=1024 if=/dev/zero 2>/dev/null | $DGSH_TEE -M $flags $flags2 -b 1024 2>"tee/$test.test" | (sleep 1 ; cat >/dev/null) ensure_similar_buffers "$test" "tee/$test.ok" "tee/$test.test" done # Test low-memory behavior (memory) rm -f try try2 mkfifo try try2 perl -e 'for ($i = 0; $i < 500; $i++) { print "x" x 500, "\n"}' | tee lines | $DGSH_TEE $flags -b 512 -m 2k -o try -o try2 2>err & cat try2 >try2.out & { read x ; echo $x ; sleep 1 ; cat ; } < try > try.out & wait if [ "$flags" = '-I' ] then ensure_same "Low-memory $flags error" err tee/oom.err else ensure_same "Low-memory (try) $flags" lines try.out ensure_same "Low-memory (try2) $flags" lines try2.out fi rm -f lines try try2 try.out try2.out err # Test low-memory behavior (file) rm -f try try2 mkfifo try try2 perl -e 'for ($i = 0; $i < 500; $i++) { print "x" x 500, "\n"}' | tee lines | $DGSH_TEE -f $flags -b 512 -m 2k -o try -o try2 2>err & cat try2 >try2.out & { read x ; echo $x ; sleep 1 ; cat ; } < try > try.out & wait cat err ensure_same "Low-memory temporary file (try) $flags" lines try.out ensure_same "Low-memory temporary file (try2) $flags" lines try2.out rm -f lines try try2 try.out try2.out err done # Test asynchronous reading from multiple input files # Without it the following blocks # Also test multiple temporary files for flags in '' '-m 65536 -f' do rm -f a b c d mkfifo a b c $DGSH_TEE -b 4096 -m 65536 -i /usr/share/dict/words -o a -o b -o c & $DGSH_TEE -b 4096 -I $flags -i a -i b -i c -o d & wait for i in . . . do cat /usr/share/dict/words done >expect ensure_same "Asynchronous gather $flags" expect d rm -f a b c d expect done exit 0 ================================================ FILE: core-tools/tests-regression/test-wrap.sh ================================================ #!/bin/sh # # Regression testing of the provided examples # TOP=$(cd ../.. ; pwd) DGSH="$TOP/build/bin/dgsh" PATH="$TOP/build/bin:$PATH" export DGSHPATH="$TOP/build/libexec/dgsh" # Ensure that the generated test file matches the reference one # File names are by conventions dgsh-wrap/$case.{ok,test} ensure_same() { local case=$1 echo -n "$case " if diff dgsh-wrap/$case.ok dgsh-wrap/$case.test then echo OK else echo "$case: Files differ: dgsh-wrap/$case.ok dgsh-wrap/$case.test" 1>&2 exit 1 fi } # Include fallback commands in our executable path export PATH="$PATH:bin" # Test that echo is wrapped as deaf $DGSH -c 'dgsh-enumerate 1 | {{ dgsh-wrap -i 0 echo hi ; dgsh-wrap dd 2>/dev/null ; }} | cat' >dgsh-wrap/echo-deaf.test ensure_same echo-deaf # Test that echo is wrapped as deaf when wrapped as script with supplied exec echo "#!$TOP/build/libexec/dgsh/dgsh-wrap -S -i 0 `which echo`" >dgsh-wrap/echo-s_caps chmod +x dgsh-wrap/echo-s_caps $DGSH -c 'dgsh-enumerate 1 | {{ dgsh-wrap/echo-s_caps hi ; dgsh-wrap dd 2>/dev/null ; }} | cat' >dgsh-wrap/echo-s_caps.test ensure_same echo-s_caps # Test that echo is wrapped as deaf when wrapped as script with implied exec echo "#!$TOP/build/libexec/dgsh/dgsh-wrap -s -i 0" >dgsh-wrap/echo chmod +x dgsh-wrap/echo $DGSH -c 'dgsh-enumerate 1 | {{ dgsh-wrap/echo hi ; dgsh-wrap dd 2>/dev/null ; }} | cat' >dgsh-wrap/echo-s.test ensure_same echo-s # Verify the fdescfs functionality (required by dgsh-wrap <| and >| # arguments) is available if [ $(ls /dev/fd | wc -l) -lt 4 ] ; then cat <<\EOF 1>&2 The full functionality of dgsh-wrap requires full /dev/fd functionality; i.e. that *all* file descriptors of a process are available under /dev/fd. It appears that on this system only the first three file descriptors are available under /dev/fd. Consequently, dgsh-wrap will not be able support <| and >| arguments and dependent programs (e.g. dgsh-parallel) will fail. In FreeBSD full /dev/fd functionality is provided by fdescfs; the functionality that FreeBSD devfs provides by default only includes the first three file descriptors under /dev/fd. In FreeBSD systems consider mounting fdescfs(5) on /dev/fd to avoid this problem. EOF exit 1 fi # Test stand-alone input path substitution (with stdin) $DGSH -c 'dgsh-enumerate 2 | dgsh-wrap paste "<|" "<|" ' >dgsh-wrap/paste1.test ensure_same paste1 # Test stand-alone input path substitution (without stdin) $DGSH -c 'dgsh-enumerate 2 | dgsh-wrap -I /usr/bin/paste - "<|" ' >dgsh-wrap/paste2.test ensure_same paste2 # Test arbitrary input argument provision (with stdin) $DGSH -c 'dgsh-enumerate 2 | dgsh-wrap -i a paste' >dgsh-wrap/paste3.test ensure_same paste3 # Test arbitrary input argument provision (without stdin) $DGSH -c 'dgsh-enumerate 2 | dgsh-wrap -i a -I /usr/bin/paste -' >dgsh-wrap/paste4.test ensure_same paste4 # Test stand-alone output path substitution (without stdout) $DGSH -c 'echo hi | dgsh-wrap -O /usr/bin/tee ">|" | {{ sed "s/^/a/" ; sed "s/^/b/" ; }} | cat' >dgsh-wrap/tee1.test ensure_same tee1 # Test arbitrary input argument provision (without stdout) $DGSH -c 'echo hi | dgsh-wrap -o a -O /usr/bin/tee | {{ sed "s/^/a/" ; sed "s/^/b/" ; }} | cat' >dgsh-wrap/tee2.test ensure_same tee2 # Test substitution of embedded arguments $DGSH -c 'dgsh-enumerate 1 | {{ dgsh-wrap -e dd "if=<|" "of=>|" 2>/dev/null ; }} | cat' >dgsh-wrap/dd-args.test ensure_same dd-args exit 0 ================================================ FILE: core-tools/tests-regression/text-properties/out.ok/character.txt ================================================ 8380 21% 3546 e 8.7% 2535 t 6.2% 2368 a 5.8% 2345 o 5.8% 1942 n 4.8% 1850 i 4.5% 1746 s 4.3% 1700 h 4.2% 1610 r 4% 1273 l 3.1% 1208 d 3% 1011 u 2.5% 801 m 2% 697 c 1.7% 684 y 1.7% 675 w 1.7% 619 f 1.5% 528 , 1.3% 517 g 1.3% 479 p 1.2% 476 . 1.2% 422 " 1% 389 b 0.95% 370 I 0.91% 355 v 0.87% 239 k 0.59% 183 - 0.45% 138 T 0.34% 138 ' 0.34% 117 E 0.29% 100 W 0.25% 99 A 0.24% 98 H 0.24% 97 S 0.24% 79 ? 0.19% 70 O 0.17% 68 R 0.17% 60 P 0.15% 58 C 0.14% 57 ! 0.14% 54 N 0.13% 54 L 0.13% 52 Y 0.13% 52 D 0.13% 43 G 0.11% 43 B 0.11% 41 M 0.1% 38 x 0.093% 30 j 0.074% 22 F 0.054% 21 ; 0.052% 20 V 0.049% 20 U 0.049% 17 z 0.042% 17 q 0.042% 17 : 0.042% 11 K 0.027% 8 X 0.02% 7 1 0.017% 6 * 0.015% 5 Z 0.012% 5 J 0.012% 5 9 0.012% 4 8 0.0098% 3 3 0.0074% 3 2 0.0074% 2 Q 0.0049% 2 0 0.0049% 1 ] 0.0025% 1 [ 0.0025% 1 6 0.0025% 1 ) 0.0025% 1 ( 0.0025% 1 # 0.0025% ================================================ FILE: core-tools/tests-regression/text-properties/out.ok/digram.txt ================================================ 689 th 602 he 473 er 469 ou 468 in 454 an 392 ha 371 re 346 at 331 en 310 on 309 it 287 nd 286 ed 281 ve 248 or 244 hi 240 me 237 is 231 ng 229 to 228 te 218 st 218 as 215 es 213 al 201 ar 197 nt 195 of 193 ll 188 le 179 yo 167 ea 162 se 159 ti 157 ro 152 wa 152 ma 150 ne 141 el 138 ul 135 ho 130 li 128 ut 128 ur 127 ic 127 de 127 ce 126 om 126 be 125 ad 122 wh 119 wi 119 si 117 ly 116 so 115 no 114 co 114 ch 110 us 108 ri 108 ld 107 io 106 lo 105 ee 103 ss 103 pe 103 ow 102 et 101 ra 98 ai 97 ta 92 ot 91 sh 91 la 90 wo 90 po 90 ca 90 av 89 ke 89 im 88 ge 88 fo 88 do 88 ac 87 oo 84 id 80 my 79 un 77 ie 77 fe 76 sa 75 nc 74 ns 71 os 70 ir 69 Th 68 rs 68 mo 67 di 66 ol 65 su 64 ev 62 rt 62 bl 61 gh 60 ry 59 we 58 up 58 pr 58 ec 58 ct 58 ay 55 fi 54 na 54 if 53 em 52 il 51 tu 50 ab 49 tr 49 rd 48 vi 48 ni 48 fa 48 am 47 tt 46 op 46 bo 45 ov 45 bu 44 ap 43 iv 43 ig 42 ug 42 mi 42 Yo 41 uc 41 ck 40 pl 40 ok 39 ci 38 sp 38 fr 38 da 37 ye 37 ht 37 He 36 ts 36 pa 36 ak 35 od 35 cr 35 ag 35 We 34 It 33 pp 33 mp 33 ey 32 rn 32 go 32 gi 32 ff 31 tl 31 ex 30 sc 30 br 30 Pr 29 ys 29 wn 29 ew 28 ty 28 lu 28 ga 28 ep 27 sm 27 ki 27 ia 26 ru 26 oi 26 fu 25 ba 25 au 25 Wh 24 ny 24 nk 24 mu 24 gr 24 dl 23 ud 23 lf 23 ei 23 cu 22 um 22 rl 22 nn 22 dy 21 ub 21 nl 21 ik 21 hu 21 ef 21 cl 21 TH 20 rm 20 pi 20 og 20 lt 20 ft 20 Ch 19 xp 19 ib 19 by 18 ue 18 gl 18 aw 18 So 18 HE 18 ER 17 sk 17 rg 17 qu 17 hr 17 ds 17 dr 17 Bu 16 va 16 ui 16 rv 16 oc 16 hy 16 An 15 tw 15 rk 15 lk 15 gu 15 eg 15 dd 15 af 15 Sh 15 OR 15 No 14 ze 14 rr 14 pu 14 ms 14 kn 14 Do 13 rf 13 oe 13 nv 13 ju 13 II 13 HA 12 ua 12 ph 12 ls 12 gs 12 gn 12 bi 11 ps 11 oy 11 mb 11 je 11 ip 11 dn 11 Lo 10 ws 10 vo 10 sl 10 rw 10 pt 10 eo 10 du 10 ST 10 ON 10 Ma 10 If 10 ES 9 fl 9 eh 9 RO 9 RE 9 Gl 9 En 8 wr 8 sw 8 rc 8 lp 8 dv 8 Wa 8 Mr 8 EN 8 Co 8 As 8 AR 7 yt 7 uf 7 tm 7 tc 7 sn 7 nf 7 gt 7 eb 7 cc 7 bj 7 az 7 WO 7 IN 7 Ho 7 Ha 7 GE 7 CH 7 Am 7 AL 6 oa 6 lw 6 lm 6 iz 6 VE 6 TE 6 SI 6 Re 6 OS 6 NG 6 My 6 LL 6 LD 6 In 6 De 6 Bo 6 Be 6 Au 6 Ar 6 AP 5 wl 5 rp 5 ob 5 nu 5 nb 5 mm 5 ks 5 kl 5 gg 5 fy 5 Zo 5 Wo 5 Wi 5 WA 5 Vi 5 Si 5 SS 5 RL 5 PR 5 Oh 5 OU 5 OF 5 La 5 LO 5 LE 5 IT 5 Fo 5 Ev 5 EA 5 Da 5 Ca 4 yw 4 ym 4 xt 4 tn 4 rh 4 py 4 nr 4 jo 4 eu 4 bb 4 XI 4 VI 4 To 4 Ta 4 Pe 4 Pa 4 PT 4 PE 4 On 4 NE 4 IS 4 Hi 4 Ge 4 FO 4 Ex 4 Di 4 DE 4 CO 4 Al 4 AV 4 AS 3 yi 3 xi 3 xc 3 xa 3 sy 3 sf 3 sd 3 oj 3 nm 3 lv 3 hf 3 gy 3 cA 3 bt 3 bs 3 UR 3 UL 3 Ti 3 Te 3 St 3 Se 3 SO 3 SE 3 RG 3 RD 3 Ne 3 Na 3 ND 3 Mc 3 MO 3 Li 3 Ke 3 Ju 3 Ir 3 IO 3 IG 3 HT 3 HI 3 Gu 3 Go 3 Ga 3 FE 3 Ed 3 ED 3 EB 3 CE 3 Br 3 Bl 3 Ac 2 zi 2 yp 2 yl 2 yh 2 ww 2 uo 2 tf 2 sb 2 oh 2 nz 2 nq 2 np 2 nj 2 mn 2 ku 2 ka 2 ja 2 iu 2 hw 2 hs 2 hm 2 gm 2 ek 2 eB 2 dw 2 dg 2 cy 2 ae 2 Ye 2 XV 2 WI 2 WE 2 Up 2 Un 2 UT 2 US 2 UN 2 UE 2 TS 2 TO 2 TL 2 Su 2 Sc 2 Sa 2 Ri 2 RY 2 RT 2 RS 2 RF 2 QU 2 Pu 2 PP 2 Of 2 OW 2 OC 2 Mu 2 Mo 2 Me 2 MA 2 LY 2 KE 2 IV 2 IR 2 IL 2 HO 2 Gi 2 GH 2 Fr 2 FU 2 ET 2 EE 2 EC 2 DW 2 Cr 2 Cl 2 CT 2 CK 2 By 2 At 2 Ah 2 Ad 1 zo 1 yr 1 yb 1 xu 1 xq 1 xe 1 wf 1 wd 1 tb 1 rb 1 nx 1 nh 1 mr 1 ml 1 ln 1 lg 1 ky 1 ix 1 iw 1 iq 1 hn 1 hl 1 hd 1 ez 1 dt 1 dm 1 cs 1 ax 1 ao 1 aj 1 YR 1 YO 1 YL 1 YI 1 YE 1 WN 1 WH 1 Ve 1 Us 1 UC 1 Tu 1 Tr 1 TR 1 TM 1 TI 1 TA 1 Sw 1 Sp 1 Sk 1 SM 1 SH 1 SC 1 SA 1 Ru 1 Ro 1 RR 1 RK 1 RI 1 Pl 1 PY 1 PO 1 PI 1 PA 1 Ou 1 Or 1 Ol 1 Ob 1 OY 1 OP 1 OO 1 OK 1 OJ 1 OI 1 NT 1 NQ 1 NO 1 NM 1 NK 1 NI 1 NC 1 NB 1 NA 1 MS 1 MP 1 ML 1 Lu 1 LU 1 LA 1 Kn 1 Ka 1 KN 1 JU 1 JE 1 Is 1 Im 1 IX 1 IM 1 IC 1 IB 1 Hu 1 HU 1 GU 1 GS 1 GR 1 GG 1 Fa 1 FL 1 FI 1 Em 1 Eh 1 EY 1 EW 1 EV 1 EO 1 EF 1 DR 1 DO 1 DI 1 DF 1 CI 1 Ba 1 BO 1 BL 1 BI 1 BE 1 Af 1 AT 1 AN 1 AK 1 AI 1 AD ================================================ FILE: core-tools/tests-regression/text-properties/out.ok/trigram.txt ================================================ 350 the 176 and 173 you 168 ing 155 hat 112 tha 107 ent 95 her 94 his 90 ver 89 was 89 ith 87 ter 87 all 86 ere 84 oul 84 ion 83 wit 82 uld 79 ave 75 man 70 ome 70 for 68 thi 68 hav 64 our 59 not 57 ess 56 tio 56 hin 55 eve 54 nce 50 res 50 hou 48 ear 48 The 46 him 45 ell 44 rea 43 ove 42 out 42 men 42 You 41 had 41 are 40 ted 40 ive 40 int 40 ers 40 ati 39 whi 39 one 38 ous 37 und 37 ugh 37 ich 37 ght 37 can 36 ore 36 hic 35 who 35 som 35 rou 35 nge 35 ery 35 cou 35 ble 35 ate 34 han 34 but 34 aid 33 sta 33 red 33 ook 33 nte 32 ure 32 tur 32 sai 32 per 32 oug 32 est 32 ect 31 tte 30 wou 30 sho 30 pos 30 oun 30 ine 30 hal 30 ger 30 enc 30 con 29 ust 29 pro 29 ned 29 len 29 hea 29 cha 29 ard 28 ssi 28 own 28 nde 28 ind 28 ica 28 ead 28 ast 27 sor 27 see 27 pon 27 mor 27 ked 27 ake 27 ace 26 kin 26 hen 26 der 25 wor 25 upo 25 uch 25 sti 25 lle 25 lea 25 ist 25 ien 25 ide 25 fro 25 eri 24 wha 24 tin 24 tho 24 rie 24 ons 24 ely 24 com 24 ant 24 Pro 23 wer 23 ven 23 ste 23 ort 23 ope 23 nti 23 igh 23 den 23 ath 23 anc 22 rom 22 ost 22 nes 22 ish 22 een 21 tic 21 sio 21 rof 21 rat 21 ice 21 ful 21 eat 21 cal 21 abo 20 whe 20 ten 20 sso 20 sen 20 pre 20 ose 20 led 20 ill 20 ike 20 eng 20 end 20 eal 20 art 20 ack 20 abl 19 war 19 tan 19 ran 19 oth 19 now 19 nin 19 low 19 lik 19 ite 19 har 19 fes 19 ese 19 ame 19 air 19 ain 18 she 18 sel 18 ofe 18 nly 18 lly 18 lin 18 fac 18 ett 18 eas 18 dow 18 any 18 act 18 Wha 18 Wel 18 Cha 17 tle 17 tal 17 str 17 sir 17 ppe 17 ood 17 ont 17 oke 17 met 17 loo 17 iti 17 ins 17 fic 17 exp 17 elf 17 don 17 bou 17 app 16 way 16 use 16 sin 16 rit 16 rin 16 rel 16 pen 16 onl 16 lon 16 lit 16 lat 16 lad 16 ity 16 hai 16 eye 16 cri 16 bee 16 ass 16 age 16 ady 15 win 15 vie 15 ull 15 thr 15 suc 15 ord 15 ond 15 oin 15 off 15 nal 15 min 15 llo 15 ink 15 ime 15 ied 15 how 15 fte 15 eth 15 erv 15 ens 15 eem 15 eed 15 ces 15 att 15 ang 15 THE 15 But 14 tak 14 ple 14 ped 14 oss 14 nto 14 mos 14 mer 14 lov 14 lic 14 las 14 ire 14 goo 14 esi 14 ene 14 eme 14 dis 14 cie 14 atu 13 too 13 tel 13 son 13 sci 13 ron 13 pla 13 nev 13 mat 13 lis 13 iou 13 inc 13 iew 13 get 13 ero 13 dly 13 din 13 des 13 cul 13 ber 13 alo 13 alk 13 She 12 won 12 wom 12 urs 12 urn 12 tra 12 tly 12 sit 12 sha 12 rst 12 pec 12 orl 12 ong 12 olo 12 old 12 nst 12 nsi 12 nat 12 mus 12 may 12 lac 12 kno 12 ile 12 ibl 12 gre 12 fer 12 era 12 eli 12 ded 12 cte 12 ait 12 ade 11 yet 11 wil 11 uth 11 ute 11 ung 11 tif 11 sma 11 say 11 rth 11 rld 11 ree 11 rac 11 ppo 11 par 11 oma 11 oll 11 ole 11 mak 11 log 11 lie 11 imp 11 ign 11 hap 11 ffe 11 eet 11 bla 11 bel 11 bac 11 ark 11 ali 11 Tha 10 yes 10 wan 10 wai 10 ved 10 ttl 10 tim 10 sur 10 sub 10 spe 10 sib 10 rso 10 rse 10 ric 10 ren 10 poi 10 ori 10 ona 10 ogi 10 oes 10 nts 10 let 10 les 10 jec 10 ifi 10 hro 10 hes 10 has 10 giv 10 gen 10 fin 10 fel 10 eti 10 erf 10 ema 10 edi 10 ean 10 dle 10 did 10 dee 10 day 10 ctl 10 cke 10 bro 10 ays 10 ask 10 arg 10 ara 10 ans 9 upp 9 ult 9 tru 9 tre 9 tor 9 ton 9 tiv 9 tat 9 tac 9 sup 9 sto 9 siv 9 sid 9 ser 9 sed 9 rvi 9 rne 9 rep 9 ral 9 qui 9 pol 9 pea 9 pas 9 ote 9 osi 9 oor 9 oli 9 nve 9 ntu 9 nse 9 ner 9 mpo 9 med 9 lif 9 itt 9 iss 9 ini 9 hed 9 ert 9 ern 9 eni 9 ele 9 dys 9 dre 9 cti 9 cen 9 cat 9 bre 9 ann 9 ank 9 ail 9 aft 9 Thi 9 Gla 9 And 8 xpe 8 wen 8 usi 8 urt 8 ude 8 two 8 til 8 ssa 8 rti 8 row 8 rop 8 roo 8 rna 8 rly 8 ris 8 rfe 8 rem 8 rec 8 rai 8 ool 8 nor 8 ngs 8 ngl 8 ndo 8 mys 8 mig 8 mea 8 mar 8 mad 8 itu 8 its 8 ism 8 iol 8 hum 8 hey 8 gra 8 gin 8 fou 8 fir 8 ffi 8 fee 8 evi 8 erm 8 eno 8 eep 8 doo 8 dif 8 dan 8 cre 8 ced 8 bet 8 awa 8 ape 8 ani 8 adv 7 yth 7 yse 7 vio 7 vin 7 ves 7 vel 7 uti 7 unt 7 uit 7 uck 7 ubl 7 ubj 7 tro 7 tis 7 tar 7 tai 7 tab 7 sol 7 sis 7 shi 7 scr 7 rte 7 rov 7 rni 7 rds 7 rdl 7 pin 7 opo 7 oom 7 omp 7 ock 7 nou 7 ngt 7 ndl 7 ndi 7 muc 7 mpl 7 mon 7 mom 7 lar 7 jus 7 irs 7 ift 7 ife 7 iev 7 ies 7 hor 7 hop 7 hol 7 ged 7 elp 7 efo 7 eel 7 duc 7 doe 7 dit 7 del 7 dea 7 dde 7 dar 7 cle 7 cin 7 che 7 cau 7 bra 7 bli 7 bje 7 bef 7 bea 7 aut 7 arr 7 arn 7 aps 7 alt 7 Som 7 Mal 7 How 7 CHA 7 Ame 6 yon 6 vid 6 ura 6 ula 6 uff 6 try 6 tri 6 thy 6 tes 6 tee 6 swe 6 stu 6 ssm 6 spo 6 ske 6 sig 6 sea 6 sat 6 rus 6 run 6 rta 6 rot 6 rio 6 rge 6 rew 6 ref 6 por 6 pli 6 pap 6 owe 6 onv 6 ono 6 ntl 6 nta 6 nne 6 nit 6 new 6 nds 6 mou 6 mil 6 mes 6 mbe 6 mas 6 mal 6 lwa 6 los 6 lor 6 lia 6 lec 6 lai 6 ken 6 ize 6 ita 6 isc 6 ina 6 iff 6 icu 6 ick 6 iar 6 hre 6 hos 6 hel 6 gic 6 gai 6 fri 6 fra 6 fol 6 fec 6 ews 6 ete 6 env 6 ega 6 ecu 6 ece 6 cia 6 cem 6 bes 6 aze 6 aus 6 ase 6 alw 6 alm 6 aci 6 Sou 6 Now 6 Don 5 zed 5 xpr 5 wis 5 wel 5 vor 5 van 5 unc 5 tud 5 tir 5 tch 5 sse 5 spr 5 spi 5 smi 5 san 5 sag 5 rwa 5 rve 5 ros 5 rmi 5 rig 5 rfu 5 ret 5 raw 5 rab 5 put 5 oub 5 oti 5 ory 5 ors 5 odi 5 nyt 5 nsw 5 nna 5 nio 5 nfe 5 nee 5 ncl 5 nam 5 mpa 5 mem 5 mee 5 mag 5 lun 5 luc 5 lse 5 lki 5 liv 5 lip 5 lim 5 ley 5 lev 5 ler 5 ket 5 isi 5 ima 5 ili 5 ici 5 iat 5 ial 5 hon 5 gue 5 gth 5 glo 5 gli 5 gav 5 fif 5 fea 5 fav 5 fai 5 esc 5 enn 5 emb 5 ela 5 eit 5 eis 5 ein 5 eca 5 eau 5 eak 5 dia 5 dam 5 cur 5 cro 5 cop 5 col 5 clu 5 cio 5 cce 5 bus 5 boy 5 boo 5 bod 5 bly 5 bei 5 avo 5 avi 5 aug 5 arm 5 arl 5 ari 5 alf 5 aki 5 aga 5 add 5 acc 5 Zoo 5 Wei 5 WOR 5 Vie 5 Sir 5 RLD 5 Pre 5 PRO 5 ORL 5 HAP 5 Eng 5 Dea 5 Con 5 Aus 5 Any 5 ALL 4 yed 4 yea 4 xpl 4 wro 4 wri 4 why 4 wev 4 wee 4 wal 4 vit 4 vil 4 val 4 vag 4 ump 4 ume 4 umb 4 uma 4 udd 4 ual 4 twi 4 tut 4 tou 4 tog 4 tme 4 tit 4 tep 4 tem 4 sus 4 sud 4 sts 4 ssu 4 spa 4 sil 4 sev 4 set 4 sam 4 rwi 4 rup 4 rsi 4 roi 4 roc 4 roa 4 rms 4 riv 4 rim 4 rha 4 rch 4 rap 4 que 4 pte 4 pri 4 plo 4 pir 4 peo 4 ows 4 owi 4 oud 4 ouc 4 oto 4 orn 4 orm 4 ora 4 opy 4 opl 4 olu 4 oic 4 ody 4 oci 4 nyw 4 nvi 4 ntr 4 nth 4 nsc 4 nno 4 nis 4 net 4 nea 4 nct 4 nci 4 nch 4 nbe 4 mur 4 mod 4 mid 4 mbl 4 lut 4 lue 4 lth 4 lte 4 lou 4 lop 4 lmo 4 lli 4 leg 4 lay 4 lau 4 lan 4 kes 4 ipp 4 inn 4 inf 4 imi 4 ily 4 ibe 4 ian 4 hem 4 hei 4 gon 4 gla 4 fil 4 ext 4 erw 4 erh 4 erg 4 ept 4 epr 4 eop 4 ena 4 ems 4 els 4 elo 4 ebo 4 eav 4 dve 4 dva 4 dri 4 dou 4 doi 4 dne 4 dal 4 cra 4 cov 4 cli 4 ckl 4 cer 4 cee 4 bul 4 bot 4 big 4 beh 4 bec 4 bal 4 ash 4 ary 4 ars 4 arp 4 arc 4 aph 4 ane 4 amp 4 agu 4 adl 4 adi 4 acr 4 ach 4 abi 4 Wor 4 Why 4 TER 4 Per 4 PTE 4 OST 4 Los 4 III 4 His 4 Her 4 HAV 4 HAL 4 For 4 FOR 4 EST 4 ESS 4 Did 4 Boo 4 Ass 4 AVE 4 APT 3 zet 3 ywh 3 yst 3 voi 3 vis 3 utt 3 utl 3 uss 3 usp 3 ush 3 uns 3 una 3 umo 3 uli 3 uin 3 ues 3 udi 3 uar 3 twe 3 tua 3 top 3 tne 3 thu 3 teb 3 sui 3 sou 3 sms 3 sly 3 sla 3 sim 3 sia 3 sda 3 scu 3 sco 3 sca 3 saw 3 sau 3 rue 3 rto 3 rsu 3 rsa 3 rry 3 rre 3 rra 3 rol 3 roj 3 rod 3 rid 3 rib 3 rgu 3 rer 3 reg 3 rde 3 ray 3 ram 3 rag 3 qua 3 pul 3 pub 3 pra 3 ppr 3 pit 3 pat 3 pan 3 owl 3 owa 3 orw 3 orr 3 opi 3 onf 3 onc 3 omm 3 omi 3 oje 3 ois 3 oil 3 ogr 3 oge 3 odu 3 ode 3 odd 3 oce 3 oar 3 nyo 3 nut 3 ntm 3 nsl 3 nry 3 nob 3 nni 3 nmo 3 nke 3 nig 3 nic 3 ngu 3 nec 3 nda 3 nar 3 nan 3 nac 3 nab 3 mse 3 mpe 3 mme 3 mis 3 mic 3 map 3 mai 3 lve 3 lud 3 loc 3 lke 3 lig 3 lib 3 ldn 3 lde 3 lam 3 kle 3 kee 3 ito 3 ise 3 irl 3 ify 3 ifu 3 ier 3 idn 3 ida 3 ibb 3 hus 3 hur 3 hun 3 hts 3 hom 3 hit 3 hip 3 hfu 3 gri 3 got 3 goi 3 gno 3 gis 3 ghe 3 ges 3 gat 3 gar 3 gan 3 foo 3 fig 3 fie 3 fen 3 fat 3 far 3 fan 3 fad 3 exc 3 ewa 3 eta 3 esu 3 esp 3 esd 3 err 3 epe 3 epa 3 enr 3 enb 3 emp 3 emo 3 elt 3 eir 3 eig 3 efu 3 efl 3 efi 3 eff 3 eds 3 edn 3 edl 3 ede 3 ecr 3 eci 3 dra 3 det 3 dep 3 dem 3 ddr 3 ctu 3 cla 3 cki 3 cit 3 cho 3 chi 3 cei 3 car 3 cam 3 cAr 3 bri 3 bov 3 blu 3 bit 3 bed 3 bbe 3 aul 3 atc 3 aso 3 asn 3 asi 3 arw 3 apt 3 anl 3 als 3 aim 3 agi 3 afe 3 ada 3 Wit 3 Whe 3 Wed 3 Was 3 Wad 3 WAS 3 Tar 3 THI 3 SSI 3 REA 3 Par 3 One 3 ORE 3 Not 3 Ned 3 Nat 3 NGE 3 McA 3 Loo 3 LOS 3 LLE 3 LEN 3 Iri 3 Ins 3 ION 3 ING 3 Hen 3 Had 3 HER 3 Gaz 3 GER 3 Eve 3 Enm 3 ENG 3 Dar 3 Com 3 Bri 3 Blu 3 Ard 2 ype 2 ymp 2 yle 2 yin 2 yho 2 xtr 2 xpo 2 xce 2 xac 2 woo 2 wly 2 wic 2 vol 2 vey 2 uty 2 usl 2 usb 2 urm 2 upt 2 upe 2 unp 2 unn 2 une 2 uly 2 uls 2 ulo 2 uis 2 uid 2 ugg 2 uge 2 uci 2 uce 2 uca 2 ubt 2 uat 2 typ 2 tun 2 tuf 2 tti 2 tta 2 tom 2 tol 2 tmo 2 tli 2 thf 2 tfu 2 tea 2 taf 2 sum 2 sul 2 sug 2 suf 2 sua 2 stn 2 spl 2 sph 2 soc 2 sne 2 slo 2 sli 2 siz 2 shu 2 sfi 2 sex 2 sba 2 saf 2 ryt 2 rva 2 rum 2 rud 2 rts 2 rru 2 rri 2 rok 2 rmu 2 rmo 2 rme 2 rma 2 rip 2 rik 2 ria 2 rgs 2 reh 2 rcu 2 rav 2 ras 2 rad 2 pti 2 pru 2 pok 2 poc 2 plu 2 pic 2 phs 2 pho 2 phi 2 phe 2 pha 2 pee 2 pac 2 oyl 2 oye 2 ova 2 ott 2 ota 2 osp 2 oro 2 ork 2 org 2 orf 2 opp 2 oop 2 oon 2 oof 2 onz 2 onn 2 oni 2 ola 2 oks 2 ofo 2 oed 2 obo 2 nze 2 nyh 2 nty 2 nsu 2 nsp 2 nso 2 nqu 2 non 2 nod 2 nle 2 nki 2 nju 2 nim 2 nia 2 nfi 2 nex 2 nel 2 ndr 2 nco 2 mpi 2 mph 2 mot 2 mma 2 mit 2 meb 2 luf 2 lub 2 lot 2 lls 2 liz 2 lab 2 kul 2 kne 2 ker 2 kel 2 jum 2 jud 2 jou 2 jac 2 ivi 2 iso 2 isk 2 isf 2 iri 2 ira 2 ips 2 inj 2 ims 2 iet 2 idd 2 ict 2 icr 2 hug 2 hot 2 hok 2 hma 2 hio 2 het 2 hee 2 hau 2 ham 2 hak 2 hab 2 gut 2 gum 2 gui 2 gua 2 gto 2 goe 2 gns 2 gni 2 gne 2 gna 2 gma 2 gly 2 gle 2 gir 2 ghi 2 gge 2 gaz 2 gas 2 gal 2 fus 2 fur 2 flu 2 flo 2 fle 2 fla 2 fis 2 fid 2 few 2 fei 2 fas 2 fam 2 fal 2 fak 2 eyo 2 exa 2 eut 2 ets 2 eto 2 etc 2 esk 2 esh 2 erl 2 erc 2 epu 2 eps 2 epo 2 eou 2 eor 2 emy 2 elv 2 eiv 2 eho 2 ehi 2 ehe 2 eha 2 egr 2 ees 2 eer 2 eek 2 eec 2 edu 2 eag 2 eac 2 eBo 2 dor 2 die 2 dic 2 dge 2 def 2 dec 2 ddl 2 ddi 2 cus 2 cts 2 cto 2 cos 2 cor 2 coc 2 clo 2 cid 2 cep 2 cas 2 cad 2 bte 2 bso 2 bow 2 bor 2 bin 2 bey 2 ben 2 beg 2 bat 2 bar 2 ban 2 bad 2 aym 2 awl 2 aud 2 atm 2 ata 2 asp 2 asm 2 asa 2 ano 2 amo 2 ama 2 alu 2 ald 2 ala 2 agr 2 ago 2 aff 2 abs 2 Yes 2 XII 2 WON 2 WAR 2 VII 2 VER 2 UES 2 Tel 2 TLY 2 Sta 2 Soc 2 Sha 2 SSO 2 SOR 2 SIR 2 SIO 2 SEE 2 Res 2 ROF 2 ROC 2 RGE 2 RES 2 QUE 2 PPE 2 PER 2 OUR 2 ORG 2 OND 2 OFE 2 OCE 2 NDE 2 MOR 2 Lor 2 Lon 2 Lar 2 Ken 2 IGH 2 HIN 2 Gut 2 Got 2 Geo 2 GHT 2 Fre 2 FUL 2 FES 2 Exp 2 Evo 2 ERS 2 ERO 2 ERF 2 ERE 2 EEN 2 EDW 2 ECT 2 EBo 2 EAR 2 Doy 2 DWA 2 DER 2 Cam 2 CON 2 CES 2 Bur 2 Bes 2 Art 2 All 2 Aca 2 ART 2 ARD 2 APP 1 zoo 1 zio 1 zin 1 zem 1 ywa 1 yri 1 yme 1 yma 1 yis 1 yer 1 ybe 1 xua 1 xqu 1 xis 1 xio 1 xic 1 xed 1 xch 1 xal 1 www 1 wsp 1 wol 1 wli 1 wle 1 wir 1 wif 1 wid 1 wfu 1 wed 1 wea 1 wdl 1 wav 1 wat 1 wak 1 wab 1 viv 1 vic 1 vet 1 vem 1 veg 1 vat 1 vas 1 vac 1 uts 1 utc 1 uta 1 usu 1 usa 1 ury 1 urv 1 uri 1 urg 1 urd 1 upu 1 uou 1 uot 1 unr 1 uni 1 unb 1 umm 1 ulk 1 ulf 1 ule 1 uir 1 uic 1 ugi 1 uft 1 uer 1 uel 1 uee 1 ued 1 udy 1 udg 1 uda 1 uct 1 ucc 1 uas 1 uag 1 uab 1 tuo 1 tum 1 tue 1 tto 1 tso 1 tsm 1 tow 1 tot 1 tok 1 toe 1 tnu 1 tma 1 tig 1 tie 1 tid 1 tia 1 thw 1 thm 1 thd 1 tex 1 tev 1 teo 1 tef 1 tec 1 tba 1 tax 1 tas 1 tap 1 tag 1 syr 1 sym 1 swi 1 swa 1 sty 1 stm 1 sth 1 stf 1 ssy 1 ssn 1 ssl 1 spu 1 sop 1 soo 1 soe 1 sna 1 sku 1 sks 1 ski 1 shr 1 shn 1 sfy 1 seu 1 ses 1 sep 1 sem 1 sec 1 sce 1 saz 1 sav 1 sar 1 sap 1 sad 1 ryo 1 ryi 1 rwe 1 rut 1 rul 1 rui 1 rty 1 rtw 1 rtu 1 rtm 1 rtl 1 rsm 1 rsh 1 rro 1 rpo 1 rob 1 rlo 1 rks 1 riz 1 riu 1 rif 1 rgy 1 rgi 1 rgh 1 rei 1 rdn 1 rdi 1 rda 1 rco 1 rce 1 rbe 1 rau 1 rar 1 rao 1 quo 1 pus 1 pur 1 pun 1 ptu 1 pth 1 pse 1 ppl 1 ppi 1 pow 1 pou 1 pot 1 poo 1 ply 1 pio 1 pil 1 pie 1 pia 1 phy 1 pel 1 pay 1 pad 1 oys 1 oyi 1 owd 1 otc 1 otb 1 oso 1 osm 1 osc 1 osa 1 orc 1 oph 1 oot 1 ooe 1 ony 1 onq 1 omr 1 omn 1 olt 1 olk 1 oit 1 ogy 1 ogm 1 ogg 1 oft 1 ofi 1 oev 1 ods 1 oco 1 occ 1 obl 1 oba 1 oal 1 oaf 1 oad 1 nxi 1 num 1 nui 1 nsh 1 nsf 1 nre 1 npr 1 npl 1 noy 1 nom 1 nog 1 nli 1 nkl 1 niv 1 niu 1 nip 1 nif 1 nhe 1 ngr 1 ngi 1 ney 1 neo 1 nen 1 neg 1 ndw 1 ndu 1 ncy 1 ncr 1 nbu 1 naw 1 nau 1 nap 1 nak 1 nai 1 mul 1 mra 1 mpt 1 mpr 1 mov 1 mni 1 mne 1 mly 1 mir 1 mew 1 mel 1 meh 1 meg 1 mbi 1 maz 1 maj 1 lym 1 lyi 1 lus 1 lum 1 lua 1 lty 1 lts 1 lta 1 lso 1 lpi 1 lph 1 lpe 1 loy 1 lom 1 loi 1 loa 1 lnu 1 lmu 1 llu 1 lla 1 lka 1 liq 1 lge 1 lfi 1 lew 1 lem 1 lef 1 ldi 1 law 1 lap 1 lae 1 kso 1 kly 1 kli 1 kis 1 kat 1 kan 1 jur 1 jun 1 joy 1 job 1 jes 1 ixe 1 iwo 1 iva 1 ius 1 ium 1 itl 1 isp 1 isn 1 isa 1 irr 1 irm 1 irk 1 irc 1 iqu 1 ipo 1 ior 1 iog 1 inv 1 inu 1 inq 1 inl 1 inh 1 inb 1 imu 1 imo 1 imm 1 iml 1 imb 1 ilv 1 ils 1 ilo 1 iki 1 igu 1 igm 1 igi 1 ige 1 iga 1 ieu 1 iel 1 ief 1 idy 1 idi 1 ico 1 icl 1 ias 1 iam 1 iag 1 iac 1 iab 1 hwo 1 hwa 1 hut 1 hul 1 htf 1 hte 1 hri 1 hov 1 hog 1 hod 1 hne 1 hly 1 hir 1 hig 1 hie 1 heo 1 hec 1 hdr 1 gur 1 gul 1 gru 1 giz 1 git 1 gib 1 ghl 1 ggy 1 ggr 1 gap 1 gam 1 fug 1 fty 1 fts 1 fth 1 fre 1 fog 1 fly 1 fix 1 ffy 1 ffo 1 fev 1 feu 1 ezi 1 eys 1 exu 1 exq 1 exi 1 ewo 1 ewh 1 eva 1 eur 1 eum 1 ety 1 etw 1 etr 1 etn 1 erp 1 erk 1 erd 1 epp 1 epl 1 epi 1 eos 1 eon 1 enu 1 enl 1 emi 1 elg 1 eld 1 egs 1 ego 1 egl 1 egi 1 egg 1 ege 1 eft 1 efr 1 eez 1 eeb 1 eea 1 edg 1 eda 1 eco 1 eck 1 ech 1 ebt 1 ebr 1 ebl 1 eab 1 dwr 1 dwa 1 dut 1 dur 1 dul 1 dth 1 dsh 1 dry 1 dog 1 dmi 1 dli 1 diw 1 dir 1 dio 1 dim 1 dig 1 dib 1 dex 1 dev 1 dei 1 deg 1 deb 1 dat 1 dap 1 dac 1 dab 1 cut 1 cun 1 cue 1 ctr 1 ctn 1 cta 1 cry 1 cru 1 cot 1 coo 1 coe 1 cod 1 coa 1 cky 1 cka 1 cis 1 cir 1 chy 1 chw 1 chm 1 chf 1 cel 1 cco 1 cca 1 cab 1 bye 1 bur 1 buc 1 bts 1 bse 1 bru 1 bos 1 bom 1 bol 1 blo 1 bio 1 bim 1 bil 1 bbl 1 bag 1 bab 1 azi 1 aye 1 ayb 1 axi 1 awn 1 awi 1 awf 1 awe 1 ava 1 aum 1 auf 1 aty 1 ats 1 ato 1 asy 1 asu 1 asc 1 aro 1 arf 1 arb 1 apo 1 apa 1 aor 1 anx 1 ana 1 ams 1 amn 1 ami 1 amb 1 aln 1 ale 1 aks 1 aje 1 ais 1 aic 1 ags 1 agn 1 agg 1 afr 1 aer 1 aeo 1 adt 1 adn 1 adm 1 acy 1 acs 1 acl 1 abu 1 abb 1 YRI 1 YOU 1 YLE 1 YIN 1 YES 1 XVI 1 XIV 1 Wou 1 Win 1 Wig 1 Wes 1 Wen 1 Wal 1 Wai 1 WIT 1 WIL 1 WHO 1 WER 1 Ver 1 Upr 1 Upo 1 Uni 1 Und 1 UTL 1 UTE 1 UST 1 UNK 1 UND 1 ULD 1 UCK 1 Tut 1 Try 1 Tit 1 Tim 1 Tic 1 Thr 1 Ter 1 Tal 1 TRY 1 TML 1 TIO 1 THU 1 THO 1 TEN 1 TAR 1 Swo 1 Sup 1 Suc 1 Ste 1 Spi 1 Sku 1 Sev 1 Ser 1 Sec 1 Sco 1 Sci 1 Sav 1 Sat 1 STS 1 STI 1 STA 1 SON 1 SMS 1 SIG 1 SIB 1 SHA 1 SCI 1 SAP 1 Rus 1 Rou 1 Rig 1 Ric 1 Reu 1 Rem 1 Rel 1 Rec 1 RTH 1 RSO 1 RRO 1 ROW 1 ROU 1 ROJ 1 ROI 1 RIG 1 RFU 1 RFE 1 Put 1 Pub 1 Pra 1 Pla 1 Pal 1 PYR 1 POS 1 PIC 1 PEN 1 PEA 1 PAR 1 Out 1 Onc 1 Oly 1 Obs 1 OYL 1 OWN 1 OUT 1 OUN 1 OUL 1 OSS 1 OSE 1 ORR 1 ORD 1 OPY 1 OOK 1 ONQ 1 ONE 1 ONC 1 ONA 1 OJE 1 OIS 1 Nov 1 NTO 1 NQU 1 NOW 1 NMO 1 NKN 1 NIT 1 NGS 1 NEW 1 NEV 1 NED 1 NCE 1 NBE 1 NAN 1 Mus 1 Mun 1 Mov 1 Mos 1 Mem 1 Med 1 May 1 Mas 1 Mac 1 MPO 1 MOS 1 MAL 1 MAK 1 Luc 1 Liv 1 Lic 1 Lia 1 Lan 1 Lad 1 LYI 1 LUC 1 LOR 1 LON 1 LAI 1 Kno 1 Kee 1 Kal 1 KNO 1 KET 1 Jus 1 Jun 1 Jud 1 JUS 1 JEC 1 Ind 1 Imp 1 ITH 1 ITE 1 ISM 1 ISA 1 INT 1 INI 1 IMP 1 ILL 1 IGG 1 ICK 1 IBL 1 Hun 1 Hav 1 Has 1 Hal 1 Hai 1 HUR 1 HTM 1 HOS 1 HIS 1 Gul 1 Goo 1 Giv 1 Gib 1 Get 1 Gen 1 GUT 1 GRE 1 GGE 1 GET 1 GES 1 GEO 1 Fol 1 Fat 1 FLA 1 FIN 1 FEC 1 Exa 1 Ent 1 Emp 1 Edw 1 Edu 1 Edi 1 EYE 1 EVE 1 ETS 1 ESE 1 ERY 1 ERG 1 EOR 1 ENM 1 ENE 1 ENB 1 EFI 1 EBO 1 EAT 1 EAL 1 EAD 1 Dep 1 Dat 1 Dai 1 DRE 1 DOY 1 DIS 1 DFU 1 DEF 1 DEA 1 Cre 1 Cra 1 Clu 1 Cli 1 Chi 1 Che 1 Cat 1 Can 1 Cab 1 CTL 1 COU 1 COP 1 CKE 1 CII 1 Bos 1 Bor 1 Bel 1 Beh 1 Beg 1 Bea 1 Bal 1 BOO 1 BLE 1 BIG 1 BER 1 Aut 1 Are 1 Ant 1 Alp 1 Aft 1 Ade 1 Add 1 Act 1 ASC 1 ARK 1 ARE 1 ALO 1 AKE 1 AIL 1 ADF ================================================ FILE: core-tools/tests-regression/text-properties/out.ok/words.txt ================================================ 275 the 258 I 175 a 166 to 156 of 146 and 127 you 113 it 99 that 99 in 87 was 75 with 74 he 70 my 64 me 64 his 64 have 52 is 48 for 48 at 43 him 41 s 40 man 40 had 39 You 38 not 37 t 36 which 35 be 35 as 34 your 34 It 31 said 31 but 30 an 30 all 30 He 29 The 28 would 28 who 27 do 27 could 27 can 26 so 25 upon 25 or 25 are 22 on 21 what 21 very 21 this 20 some 20 should 20 from 18 up 18 if 18 by 18 What 18 Well 18 Professor 17 there 17 more 17 down 16 were 16 like 16 face 16 been 15 out 15 only 15 about 15 Challenger 15 But 14 such 14 sir 14 her 14 good 14 am 14 THE 13 we 13 she 13 never 13 ll 13 into 13 She 12 take 12 no 12 must 12 love 12 There 11 yet 11 when 11 than 11 over 11 may 11 know 11 come 11 chair 11 That 10 will 10 say 10 round 10 looked 10 has 10 get 10 don 10 If 9 ve 9 us 9 something 9 see 9 one 9 m 9 last 9 head 9 Gladys 9 And 8 two 8 really 8 rather 8 nothing 8 most 8 might 8 little 8 interview 8 give 8 eyes 8 door 8 any 8 after 8 Mr 8 A 7 world 7 word 7 woman 7 want 7 time 7 through 7 think 7 they 7 tell 7 subject 7 seem 7 room 7 own 7 off 7 myself 7 moment 7 make 7 letter 7 does 7 did 7 cried 7 better 7 before 7 back 7 We 7 This 7 Malone 6 young 6 went 6 way 6 wait 6 try 6 took 6 thought 6 sort 6 shall 6 scientific 6 policeman 6 our 6 old 6 now 6 much 6 men 6 life 6 how 6 here 6 hand 6 great 6 full 6 first 6 eye 6 ever 6 always 6 Now 6 My 6 How 6 Don 6 Do 5 won 5 then 5 talk 5 suppose 5 read 5 re 5 quite 5 place 5 person 5 perfectly 5 mind 5 mean 5 matter 5 made 5 least 5 kind 5 just 5 hour 5 hope 5 help 5 gave 5 fellow 5 every 5 enough 5 done 5 dangerous 5 call 5 black 5 believe 5 being 5 asked 5 Zoological 5 WORLD 5 Vienna 5 Then 5 South 5 Sir 5 Oh 5 No 5 English 5 Austin 4 women 4 within 4 why 4 well 4 violent 4 use 4 understand 4 under 4 too 4 three 4 thing 4 these 4 sure 4 still 4 short 4 seemed 4 same 4 red 4 put 4 people 4 paper 4 opening 4 office 4 look 4 long 4 lady 4 kindly 4 its 4 infernal 4 impossible 4 heart 4 hardly 4 hard 4 half 4 go 4 found 4 find 4 favor 4 evening 4 envelope 4 entirely 4 doing 4 day 4 course 4 copy 4 business 4 boy 4 big 4 because 4 ask 4 answer 4 almost 4 World 4 Why 4 Weissmann 4 To 4 Some 4 Lost 4 His 4 HAVE 4 E 4 Dear 4 CHAPTER 4 As 4 America 3 yourself 3 wrong 3 wonderful 3 wonder 3 whole 3 where 3 voice 3 vitality 3 views 3 until 3 turn 3 true 3 touch 3 thoughts 3 though 3 those 3 things 3 them 3 their 3 table 3 sudden 3 sub 3 strange 3 statement 3 stand 3 smile 3 set 3 sentence 3 self 3 seems 3 science 3 saw 3 sat 3 run 3 real 3 propose 3 point 3 please 3 personal 3 perhaps 3 passage 3 opened 3 open 3 offensive 3 night 3 name 3 met 3 merely 3 meeting 3 master 3 luck 3 liar 3 keep 3 insisted 3 idea 3 human 3 however 3 house 3 honor 3 heard 3 hear 3 hands 3 hair 3 got 3 gone 3 glad 3 general 3 gathered 3 front 3 frank 3 forward 3 follow 3 fifteen 3 felt 3 feel 3 fear 3 far 3 expression 3 evidence 3 entered 3 end 3 else 3 dark 3 dare 3 d 3 critical 3 comes 3 claim 3 character 3 chances 3 chance 3 came 3 bull 3 brain 3 both 3 blue 3 best 3 beautiful 3 away 3 aware 3 appointment 3 appearance 3 anywhere 3 anything 3 anyone 3 age 3 against 3 adventures 3 across 3 above 3 With 3 Wednesday 3 Wadley 3 WAS 3 Tarp 3 So 3 President 3 Park 3 One 3 OF 3 Ned 3 McArdle 3 Look 3 LOST 3 Irish 3 Institute 3 IT 3 Henry 3 Had 3 Gazette 3 G 3 Even 3 Enmore 3 Did 3 CHALLENGER 3 British 3 Blundell 3 American 2 yes 2 years 2 year 2 write 2 wouldn 2 worthy 2 words 2 without 2 wish 2 wind 2 whose 2 whom 2 white 2 week 2 wasn 2 walked 2 violence 2 view 2 version 2 ventured 2 vague 2 type 2 twenty 2 turning 2 turned 2 trouble 2 translation 2 together 2 thrown 2 thin 2 terms 2 talking 2 taken 2 suit 2 student 2 strongly 2 street 2 story 2 stopped 2 steps 2 station 2 staff 2 sprung 2 spoke 2 spaces 2 soul 2 somebody 2 small 2 slowly 2 size 2 sit 2 since 2 simple 2 silly 2 show 2 shoulders 2 severely 2 send 2 seen 2 seated 2 scandal 2 row 2 romance 2 risk 2 right 2 result 2 rested 2 reporters 2 remember 2 remark 2 regard 2 received 2 realized 2 ready 2 reading 2 public 2 prove 2 proud 2 profound 2 proceedings 2 presents 2 present 2 presence 2 possible 2 position 2 plunged 2 play 2 pity 2 pink 2 photographs 2 peculiar 2 past 2 passion 2 particular 2 part 2 ought 2 other 2 opinion 2 oh 2 notebook 2 nodded 2 nobody 2 news 2 need 2 nearly 2 nature 2 natural 2 narrative 2 murmured 2 motive 2 morning 2 month 2 modify 2 modern 2 mine 2 middle 2 message 2 menacing 2 memory 2 meesion 2 marry 2 map 2 making 2 makes 2 madam 2 lost 2 longer 2 lived 2 lips 2 liked 2 lie 2 let 2 less 2 length 2 led 2 leaned 2 laughing 2 laughed 2 later 2 knew 2 jacket 2 isolated 2 interest 2 insufferable 2 indeed 2 inclined 2 imposing 2 ideal 2 husband 2 hundred 2 huge 2 hold 2 himself 2 heroisms 2 heavily 2 hall 2 going 2 goes 2 girl 2 friends 2 forth 2 forehead 2 fire 2 fingers 2 filled 2 fight 2 few 2 feet 2 feeling 2 father 2 fashion 2 famous 2 failed 2 faddist 2 explosion 2 evident 2 everything 2 etc 2 envied 2 enormous 2 eleven 2 either 2 education 2 editor 2 each 2 eBook 2 driving 2 drew 2 distrust 2 difficult 2 different 2 didn 2 determination 2 desk 2 desire 2 definite 2 deeds 2 deed 2 dear 2 days 2 covered 2 control 2 consent 2 conscious 2 connection 2 confidence 2 compliments 2 clear 2 chest 2 chapter 2 care 2 cantankerous 2 cannot 2 called 2 brown 2 broke 2 break 2 book 2 blank 2 blame 2 beyond 2 behind 2 beard 2 bald 2 bad 2 bacteriologist 2 atmosphere 2 assure 2 arms 2 argument 2 appeal 2 another 2 aloud 2 alone 2 ago 2 again 2 advance 2 address 2 absolutely 2 Your 2 Yes 2 When 2 Was 2 W 2 Think 2 Shall 2 SIR 2 Project 2 PROFESSOR 2 PROCESSION 2 Of 2 Not 2 Nature 2 Lord 2 London 2 Largs 2 Kensington 2 In 2 IN 2 III 2 II 2 Gutenberg 2 Got 2 George 2 Evolution 2 EDWARD 2 EBook 2 Doyle 2 Darwin 2 D 2 Conan 2 Come 2 C 2 By 2 Besides 2 At 2 Assistant 2 Arthur 2 Anything 2 Anyhow 2 All 2 Ah 2 Academy 1 zoologist 1 youthful 1 youth 1 yours 1 www 1 wrought 1 written 1 worth 1 worst 1 worship 1 work 1 wore 1 wooden 1 wondering 1 womanly 1 wits 1 withdrawn 1 wistful 1 wisp 1 wire 1 windows 1 wincing 1 wife 1 wide 1 wicked 1 whim 1 whether 1 wheezing 1 wheel 1 whatsoever 1 whatever 1 wealth 1 waving 1 watchful 1 warning 1 warned 1 warm 1 wants 1 wanted 1 walnut 1 wall 1 wakens 1 waiting 1 waited 1 wail 1 vivacious 1 visits 1 visitors 1 visit 1 vile 1 victory 1 versus 1 vermin 1 ventures 1 venture 1 velvet 1 vegetarian 1 value 1 valuable 1 vaguely 1 usually 1 upturned 1 untidy 1 unshrinking 1 unsexual 1 unscrupulous 1 unreservedly 1 unprintable 1 unpleasant 1 unit 1 uneasy 1 undoubtedly 1 understanding 1 underlying 1 uncertain 1 unbend 1 unawares 1 unable 1 twinkled 1 twilight 1 twice 1 tut 1 tufts 1 trust 1 truly 1 trotted 1 triumphantly 1 tried 1 trend 1 tremendous 1 treason 1 translate 1 transfixed 1 tram 1 towards 1 touchy 1 top 1 tome 1 told 1 token 1 toes 1 timidity 1 till 1 tickling 1 tickled 1 thousand 1 thoughtfully 1 thoroughly 1 thinking 1 thesis 1 theosophist 1 themselves 1 thanked 1 terrible 1 tension 1 tended 1 ten 1 telegony 1 teetotal 1 taxicab 1 tap 1 tall 1 talked 1 tactlessness 1 tactless 1 tact 1 taciturn 1 sympathy 1 swine 1 sweetness 1 swarthy 1 suspicion 1 suspense 1 suspect 1 surely 1 suppressed 1 supposeetion 1 support 1 superman 1 summary 1 suggestions 1 suggestion 1 suddenly 1 succession 1 stunted 1 stuff 1 study 1 students 1 striking 1 strikes 1 strengthens 1 strengthen 1 strength 1 stooping 1 stoop 1 stone 1 stigmata 1 sterner 1 stepped 1 steely 1 stealthy 1 statue 1 state 1 startled 1 starting 1 started 1 staring 1 stare 1 standards 1 stairs 1 spun 1 springing 1 spread 1 sprang 1 sportsman 1 spoiled 1 spoil 1 split 1 splendid 1 spite 1 spirit 1 speech 1 speculations 1 speculation 1 spectacles 1 special 1 speak 1 spade 1 sound 1 sorry 1 sorely 1 soon 1 somewhat 1 somersault 1 solitary 1 soldier 1 sneer 1 snapped 1 smiling 1 smiled 1 smaller 1 slipped 1 slip 1 skull 1 skin 1 skeleton 1 situation 1 sinned 1 sinister 1 single 1 simultaneously 1 silver 1 silence 1 signs 1 signals 1 signal 1 sight 1 side 1 shut 1 shunned 1 shouldn 1 shoulder 1 shield 1 sharp 1 shaped 1 shape 1 shaking 1 shaken 1 sex 1 severe 1 several 1 settled 1 serve 1 servant 1 seriously 1 serious 1 series 1 separate 1 sent 1 sense 1 selfishness 1 seempathy 1 secret 1 seat 1 searching 1 scribblers 1 scrawled 1 says 1 save 1 satisfy 1 satisfied 1 sanely 1 sanctum 1 safely 1 safe 1 sadly 1 rushed 1 rush 1 rupee 1 rumbling 1 rumbled 1 ruined 1 rudeness 1 rough 1 rotating 1 rolled 1 roaring 1 roared 1 river 1 rippling 1 ring 1 righteous 1 rideeculous 1 rewards 1 reward 1 retracted 1 resumed 1 restrictions 1 restraint 1 respect 1 reserve 1 research 1 rescued 1 repulsed 1 repulse 1 reproved 1 reproof 1 representative 1 reply 1 remind 1 remain 1 relented 1 relations 1 refused 1 refuse 1 refresh 1 reflects 1 reflected 1 recriminations 1 record 1 recently 1 reasonable 1 reader 1 raven 1 rational 1 rascals 1 rarefied 1 rank 1 railings 1 railing 1 rage 1 radiated 1 race 1 quote 1 quickly 1 questions 1 quest 1 queer 1 quarter 1 quarrelsome 1 quality 1 putting 1 pushed 1 pursuing 1 publication 1 proves 1 proved 1 protest 1 prosaic 1 proposition 1 proposed 1 proportion 1 proper 1 projecting 1 profile 1 producing 1 produced 1 probable 1 primitive 1 pride 1 pressed 1 presentiment 1 prepared 1 precaution 1 praise 1 practise 1 pound 1 postmark 1 possibly 1 possessed 1 porticoed 1 poor 1 police 1 points 1 pointing 1 pockets 1 pocketed 1 pleasure 1 pleasant 1 pleaded 1 played 1 plastered 1 plasm 1 plan 1 plain 1 pilot 1 pick 1 persuasive 1 personality 1 permission 1 peculiarly 1 payment 1 pathetically 1 passing 1 pass 1 parts 1 parthenogenetic 1 papers 1 panted 1 oyster 1 overpowering 1 oval 1 outlined 1 otherwise 1 others 1 original 1 oriental 1 order 1 online 1 oneself 1 once 1 omnipotent 1 oily 1 often 1 official 1 odious 1 odd 1 occasion 1 o 1 numerous 1 notorious 1 notions 1 noticed 1 notice 1 notes 1 note 1 nonsense 1 none 1 noble 1 nine 1 nicer 1 next 1 newspaper 1 new 1 net 1 neglected 1 needs 1 needed 1 necessary 1 nearer 1 natured 1 namely 1 named 1 naked 1 nails 1 mystic 1 move 1 mouth 1 moustache 1 morrow 1 monotonous 1 money 1 mission 1 minutes 1 miles 1 microscope 1 microcosm 1 methods 1 mere 1 mentioned 1 mend 1 menaces 1 megalomaniac 1 meaty 1 meant 1 meaning 1 maybe 1 mature 1 matters 1 mate 1 matchwood 1 match 1 masterly 1 masterful 1 massive 1 mark 1 margin 1 maps 1 malice 1 majesty 1 mail 1 maid 1 magnetism 1 mad 1 lucky 1 lucid 1 lover 1 loved 1 lot 1 lose 1 looks 1 lonely 1 locked 1 loafers 1 live 1 listened 1 list 1 liquid 1 link 1 line 1 limit 1 likely 1 light 1 lifted 1 lift 1 lieutenant 1 liberty 1 libel 1 levity 1 level 1 legs 1 leg 1 left 1 lecture 1 leave 1 leathery 1 leather 1 learned 1 leaning 1 lead 1 layman 1 law 1 latter 1 latest 1 late 1 largest 1 large 1 land 1 lamp 1 laid 1 knows 1 kiss 1 justify 1 justifies 1 jumped 1 jump 1 judice 1 judged 1 joy 1 journalists 1 journalist 1 job 1 isn 1 irksome 1 invent 1 intrusive 1 intruded 1 intertwined 1 interrupted 1 international 1 intercourse 1 intelligence 1 instincts 1 instinct 1 instead 1 instantly 1 inspirer 1 inspiration 1 insolent 1 insignificant 1 inquirer 1 inner 1 injured 1 injunction 1 inherited 1 inexpressibly 1 indication 1 index 1 incredible 1 including 1 included 1 inches 1 impression 1 imposter 1 impediment 1 immediate 1 imagined 1 imagine 1 image 1 ill 1 ignore 1 ignorance 1 ignominious 1 id 1 hurting 1 humorous 1 humor 1 humbly 1 humble 1 humanity 1 hulking 1 howl 1 hovering 1 hours 1 hopelessly 1 hoped 1 honored 1 honest 1 homicidal 1 holes 1 hog 1 historical 1 higher 1 hers 1 heroic 1 heritage 1 herd 1 helped 1 height 1 heaves 1 heated 1 hearted 1 health 1 heading 1 headed 1 having 1 haven 1 hated 1 hat 1 hardness 1 harder 1 happiest 1 happened 1 happen 1 handwriting 1 handling 1 handled 1 handiwork 1 handed 1 halls 1 hadn 1 habitual 1 habit 1 gutter 1 gutenberg 1 guidance 1 guaranteed 1 grunt 1 gripped 1 grinning 1 grimly 1 gregarious 1 greater 1 gray 1 grasp 1 gracious 1 goings 1 glowing 1 glow 1 glorification 1 glories 1 gloried 1 glared 1 gives 1 given 1 gingery 1 gibberish 1 germ 1 genuine 1 gentleman 1 gentle 1 genius 1 generations 1 gazed 1 gaze 1 gasp 1 gas 1 gaps 1 game 1 gale 1 gaiters 1 fury 1 furniture 1 fugitive 1 fronting 1 frontiersman 1 fringed 1 friendship 1 friendly 1 friend 1 fraud 1 frame 1 fourth 1 four 1 fortunately 1 formidable 1 former 1 forlorn 1 forfeit 1 forewarned 1 force 1 football 1 foolish 1 fool 1 follows 1 followed 1 folk 1 foggy 1 fluffy 1 fluff 1 florid 1 flock 1 flatter 1 flashed 1 fists 1 finished 1 fine 1 filed 1 figure 1 fifty 1 fifth 1 fever 1 fell 1 feeble 1 feathery 1 fearful 1 favorable 1 fatuous 1 fang 1 fancies 1 fanatic 1 faltering 1 fallacy 1 faking 1 fakes 1 faithfully 1 fairly 1 fair 1 fad 1 factor 1 fact 1 eyed 1 extreme 1 extraordinary 1 exquisite 1 express 1 exposing 1 exposed 1 exploits 1 explained 1 experiences 1 experienced 1 experience 1 expense 1 expedeetion 1 expect 1 exists 1 exchange 1 excellent 1 exceeding 1 exalted 1 exactly 1 evidently 1 everyone 1 even 1 evasion 1 establishing 1 established 1 especially 1 enthusiasm 1 energy 1 endorsement 1 endorse 1 encouraging 1 encoding 1 emphatic 1 em 1 electric 1 elasticity 1 elapse 1 elaborated 1 elaborate 1 eh 1 egg 1 effort 1 effervescing 1 effervescence 1 edge 1 ease 1 earth 1 ears 1 earnestness 1 earnest 1 early 1 eager 1 duty 1 during 1 duly 1 dry 1 driven 1 dried 1 dressing 1 dress 1 drawing 1 drawer 1 doubly 1 dogmatic 1 distinguish 1 distasteful 1 distance 1 discussion 1 discuss 1 discretion 1 discreditable 1 discovery 1 discovered 1 disapproval 1 directness 1 dining 1 dignity 1 difficulty 1 differences 1 diameter 1 diagrams 1 devil 1 detached 1 destined 1 desires 1 described 1 descreeptive 1 depth 1 depreciation 1 dependent 1 demonstrate 1 delicately 1 delicate 1 deliberate 1 deigns 1 degree 1 deeper 1 deep 1 deception 1 decency 1 debts 1 date 1 danger 1 dancing 1 damp 1 damnedest 1 damaged 1 cuticura 1 curving 1 curtained 1 curtain 1 curtail 1 cunning 1 crying 1 criticism 1 crisis 1 creatures 1 creature 1 crawling 1 cranial 1 crabbed 1 court 1 cost 1 correspondence 1 cooed 1 convinces 1 convinced 1 conveyed 1 convey 1 conversaziones 1 conversation 1 context 1 contents 1 contentious 1 content 1 constant 1 considerate 1 conscience 1 conquered 1 confession 1 conducted 1 conditions 1 conditioned 1 conclusions 1 conceited 1 comradeship 1 composition 1 compliment 1 company 1 companions 1 comments 1 comment 1 command 1 coloring 1 colliery 1 collected 1 colleagues 1 cold 1 cockatoo 1 cock 1 coal 1 clumped 1 club 1 closed 1 clock 1 clipped 1 climbing 1 clever 1 clerk 1 clearly 1 circulation 1 chosen 1 choked 1 choke 1 chirrup 1 checking 1 chauffeur 1 charged 1 charge 1 champion 1 certainly 1 certain 1 centered 1 center 1 censor 1 caused 1 caught 1 case 1 carried 1 calm 1 calling 1 bye 1 butt 1 butlers 1 butcher 1 bully 1 bucking 1 brutal 1 brows 1 brought 1 brother 1 bronzed 1 bronze 1 broad 1 bring 1 bright 1 briefly 1 brethren 1 breathed 1 breath 1 breast 1 breaks 1 breaking 1 breadth 1 brave 1 brackets 1 brace 1 boyish 1 bowdlerized 1 bow 1 bounded 1 bounced 1 bottom 1 boss 1 books 1 bolted 1 bodies 1 blowing 1 blew 1 blast 1 blackened 1 bit 1 biography 1 bimetallism 1 between 1 bestial 1 besides 1 beside 1 bent 1 bellowing 1 bellow 1 bell 1 believes 1 believed 1 behavior 1 behaved 1 begins 1 began 1 beauty 1 beast 1 battles 1 battery 1 barrel 1 barbed 1 balloon 1 bags 1 backing 1 backed 1 awfully 1 awaits 1 averted 1 averaged 1 authority 1 audacity 1 attitude 1 attention 1 attempt 1 attacked 1 attack 1 asthmatic 1 assured 1 assurances 1 associate 1 assertion 1 assaults 1 assault 1 asks 1 ashamed 1 article 1 around 1 arnica 1 arm 1 argue 1 ardent 1 architecture 1 approve 1 appears 1 appeared 1 appear 1 apologize 1 apart 1 anxious 1 answered 1 annoyed 1 announced 1 animals 1 animal 1 angular 1 anger 1 amplification 1 amiss 1 amazement 1 although 1 alternating 1 aloof 1 alluded 1 allowances 1 alive 1 alarmed 1 air 1 agreeable 1 aggressive 1 afterwards 1 afraid 1 aeronaut 1 adventure 1 advancing 1 advanced 1 admirable 1 added 1 adapt 1 action 1 acting 1 act 1 acrimonious 1 account 1 accessible 1 accepted 1 accentuated 1 accent 1 abusive 1 abominably 1 able 1 Yours 1 YOUR 1 XVI 1 XV 1 XIV 1 XIII 1 XII 1 XI 1 X 1 Would 1 Winner 1 Wigan 1 Where 1 West 1 Went 1 Weissmannism 1 Wasn 1 Walking 1 Wait 1 WONDERS 1 WONDERFUL 1 WITH 1 WILL 1 WHO 1 WERE 1 WE 1 Vertebrate 1 VIII 1 VII 1 VI 1 VERY 1 V 1 Us 1 Uproar 1 Upon 1 University 1 Undoubtedly 1 US 1 UNKNOWN 1 Tut 1 Try 1 Title 1 Time 1 Tickled 1 Three 1 They 1 These 1 Terrace 1 Tell 1 Telegraph 1 Talking 1 TRY 1 TO 1 THOSE 1 THIS 1 THINGS 1 THING 1 THERE 1 Swollen 1 Suppose 1 Such 1 Step 1 Stanleys 1 Stanley 1 Spirited 1 Southwark 1 Sometimes 1 Something 1 Somehow 1 Society 1 Societe 1 Skulls 1 Several 1 Series 1 Section 1 Scotch 1 Sciences 1 Savage 1 Saturday 1 START 1 SIGHT 1 SHALL 1 SEEN 1 S 1 Russia 1 Round 1 Right 1 Richard 1 Reuter 1 Resigned 1 Research 1 Remember 1 Release 1 Recreations 1 ROUND 1 REAL 1 QUESTION 1 Put 1 Publications 1 Protests 1 Protest 1 Produced 1 Proceedings 1 Pretend 1 Pressman 1 Pray 1 Plata 1 Person 1 Persian 1 Perhaps 1 Perfectly 1 Palaeontological 1 PROJECT 1 PICKETS 1 PERSON 1 PERFECTLY 1 PARK 1 Outlines 1 Or 1 Once 1 Olympian 1 Observations 1 OUTLYING 1 OUR 1 ONCE 1 November 1 Nothing 1 Naturally 1 NEW 1 NEVER 1 N 1 Museum 1 Munchausen 1 Move 1 Most 1 Member 1 Medal 1 May 1 Masonic 1 Macs 1 MOST 1 MORROW 1 MALONE 1 MAKE 1 Luck 1 Lively 1 License 1 Liability 1 Language 1 Lady 1 La 1 LUCK 1 LORD 1 Knowable 1 Keeper 1 Kalmuck 1 Just 1 June 1 Judith 1 JUST 1 Is 1 India 1 Impossible 1 IX 1 IV 1 IS 1 INTO 1 IMPOSSIBLE 1 Hungerton 1 However 1 Heroisms 1 Here 1 Hercules 1 Her 1 Have 1 Has 1 Half 1 Haines 1 HTML 1 HEROISMS 1 HERO 1 HE 1 HAPPENED 1 H 1 Gulf 1 Good 1 Give 1 Gibberish 1 Get 1 General 1 GUTENBERG 1 GREAT 1 GEORGE 1 Frenchman 1 French 1 Forfeit 1 Foreword 1 Foreign 1 For 1 Following 1 Fate 1 FORGET 1 FOREST 1 FORESEEN 1 FOR 1 FLAIL 1 Expensive 1 Expected 1 Exactly 1 Ex 1 Entirely 1 Employers 1 Eh 1 Edward 1 Educ 1 Edinburgh 1 EYES 1 ENMORE 1 EBOOK 1 Didn 1 Department 1 Death 1 Date 1 Darwinism 1 Daily 1 DREADFUL 1 DOYLE 1 DISAPPEAR 1 DEFINITE 1 DEAR 1 Creeping 1 Crayston 1 Continental 1 Contents 1 Congress 1 Comparative 1 Club 1 Clive 1 Chinese 1 Chestnuts 1 Character 1 Chairman 1 Chain 1 Catharine 1 Can 1 Came 1 Camberwell 1 Cabinet 1 COULD 1 COPYRIGHT 1 CONQUESTS 1 CONAN 1 Burtons 1 Burton 1 Boss 1 Born 1 Belge 1 Behold 1 Began 1 Beaumont 1 Balkans 1 BIGGEST 1 B 1 Author 1 Assyrian 1 Association 1 Are 1 Anyway 1 Anthropology 1 An 1 Alpine 1 Al 1 Afterwards 1 Adelphi 1 Address 1 Act 1 ASCII 1 ARTHUR 1 ARE 1 ALL ================================================ FILE: core-tools/tests-regression/web-log-report/logfile ================================================ 204.249.225.59 - - [28/Aug/1995:00:00:34 -0400] "GET /pub/rmharris/catalogs/dawsocat/intro.html HTTP/1.0" 200 3542 pc23-slip.ccs-stug.deakin.edu.au - - [28/Aug/1995:00:06:54 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 sl06.chrysalis.org - - [28/Aug/1995:00:13:01 -0400] "GET /pub/sshay/images/crthumb5.jpg HTTP/1.0" 200 4495 uaf-du-02-14.alaska.edu - - [28/Aug/1995:00:19:51 -0400] "GET /atomicbk/images/shemale1.gif HTTP/1.0" 200 43665 minerva.athenet.net - - [28/Aug/1995:00:27:13 -0400] "GET /pub/job/vk/view17.jpg" 200 5944 pc25.acrs-helm.wku.edu - - [28/Aug/1995:00:36:53 -0400] "GET /pub/sshay/images/chrsbutn.gif HTTP/1.0" 200 1070 Huwawa.async.smsu.edu - - [28/Aug/1995:00:43:18 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 304 - 204.101.181.14 - - [28/Aug/1995:00:50:09 -0400] "GET /pub/jeffd/grayball.gif HTTP/1.0" 200 995 ix-sfe-nm1-05.ix.netcom.com - - [28/Aug/1995:00:56:50 -0400] "GET /pub/sshay/images/gallery.gif HTTP/1.0" 200 8010 dialin33448.slip.nts.uci.edu - - [28/Aug/1995:01:05:15 -0400] "GET /atomicbk/new/catalog.gif HTTP/1.0" 200 693 S143A-quadra.Stanford.EDU - - [28/Aug/1995:01:15:20 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 ix-ftl2-09.ix.netcom.com - - [28/Aug/1995:01:24:21 -0400] "GET /pub/job/vk/flowers1.gif HTTP/1.0" 200 4288 mpdpc01448.nneco.nu.com - - [28/Aug/1995:01:34:21 -0400] "GET /pub/job/vk/view07.jpg HTTP/1.0" 200 7057 async110.zrz.TU-Berlin.DE - - [28/Aug/1995:01:42:10 -0400] "GET /pub/cardman/1curious.gif HTTP/1.0" 200 1281 133.225.internal.radian.com - - [28/Aug/1995:01:52:52 -0400] "GET /pub/job/vk/view17.jpg HTTP/1.0" 200 5944 sundial.sundial.net - - [28/Aug/1995:02:01:52 -0400] "GET /pub/mbrophy/pix/thing.jpg HTTP/1.0" 200 3606 tuna.hooked.net - - [28/Aug/1995:02:09:55 -0400] "GET /pub/journalism/awesome.html/ HTTP/1.0" 404 207 Cust15.Max1.San-Diego.CA.MS.UU.NET - - [28/Aug/1995:02:19:38 -0400] "GET /pub/sinkers/introsp2.gif HTTP/1.0" 200 22960 ip-pdx11-20.teleport.com - - [28/Aug/1995:02:29:14 -0400] "GET /pub/rmharris/alldlrs/dlrimage/books.gif HTTP/1.0" 304 - clark.net - - [28/Aug/1995:02:38:50 -0400] "GET /pub/kolesar/turtle.gif HTTP/1.0" 304 - humfact2.mac.ccad.uiowa.edu - - [28/Aug/1995:02:48:38 -0400] "GET /pub/job/vk/view21.jpg HTTP/1.0" 200 5228 ppp178.scott.net - - [28/Aug/1995:02:57:14 -0400] "GET /pub/aztec/shesails/ HTTP/1.0" 200 1533 mac27.phlab.missouri.edu - - [28/Aug/1995:03:07:50 -0400] "GET /pub/mbrophy/pix/about.jpg HTTP/1.0" 200 5674 gate.tsh.cae.ntt.jp - - [28/Aug/1995:03:18:12 -0400] "GET /pub/job/vk/view30.jpg HTTP/1.0" 200 5991 inter3.ytol.fi - - [28/Aug/1995:03:29:29 -0400] "GET /pub/job/vk/view33.jpg HTTP/1.0" 200 5903 lks.covd.se - - [28/Aug/1995:03:39:18 -0400] "GET /pub/job/b-back.jpg HTTP/1.0" 200 4195 endBb.laurel.us.net - - [28/Aug/1995:03:49:46 -0400] "GET /pub/craver/surreal/anim7.htm HTTP/1.0" 200 391 pc1078.psi.ch - - [28/Aug/1995:03:59:01 -0400] "GET /pub/job/vk/view18.jpg HTTP/1.0" 200 6578 cc7sun-d6.cc.au.ac.th - - [28/Aug/1995:04:10:43 -0400] "GET /pub/mbrophy/pix/arling.jpg HTTP/1.0" 200 7905 sunlab9.essex.ac.uk - - [28/Aug/1995:04:25:16 -0400] "GET /pub/job/vk/view34.jpg HTTP/1.0" 200 5156 dum28.interserve.com.hk - - [28/Aug/1995:04:37:17 -0400] "GET /pub/job/vk/flowers1.gif HTTP/1.0" 200 4288 interlock.turner.com - - [28/Aug/1995:04:51:11 -0400] "GET /pub/atomicbk/catalog/new.gif HTTP/1.0" 200 744 elara.horizon.bc.ca - - [28/Aug/1995:05:04:26 -0400] "GET /pub/sshay/about.html HTTP/1.0" 200 18751 general.sbceo.k12.ca.us - - [28/Aug/1995:05:19:36 -0400] "GET /pub/cosmic/joyce1.gif HTTP/1.0" 200 29751 dal01-10.ppp.iadfw.net - - [28/Aug/1995:05:34:25 -0400] "GET /pub/ecn/Newton.gif HTTP/1.0" 200 2724 relay02.jpmorgan.com - - [28/Aug/1995:05:45:35 -0400] "GET /pub/job/swim/v-54.jpg HTTP/1.0" 200 2817 pm013.bby.wis.net - - [28/Aug/1995:05:54:30 -0400] "GET /atomicbk/images/tart.gif HTTP/1.0" 200 38555 extra.ucc.su.OZ.AU - - [28/Aug/1995:06:05:28 -0400] "GET /pub/menswear/redball.gif HTTP/1.0" 200 326 gaudi.ac.upc.es - - [28/Aug/1995:06:18:36 -0400] "GET /pub/rjgula/os/linux/lpr1.txt HTTP/1.0" 200 2229 Lei56.pi.net - - [28/Aug/1995:06:29:36 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 www-e1b.gnn.com - - [28/Aug/1995:06:42:25 -0400] "GET /pub/mmuller/uel_email.html HTTP/1.0" 200 7205 gk-east.usps.gov - - [28/Aug/1995:06:55:36 -0400] "GET /pub/job/vk/view07.jpg HTTP/1.0" 200 7057 166.42.216.37 - - [28/Aug/1995:07:05:59 -0400] "GET /pub/job/vk/v-sig.gif HTTP/1.0" 200 4864 ppp-dc-1-4.ios.com - - [28/Aug/1995:07:16:51 -0400] "GET /pub/gen/fas/ HTTP/1.0" 200 7893 icpmac16.epfl.ch - - [28/Aug/1995:07:28:43 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 zephyr.aub.auc.dk - - [28/Aug/1995:07:37:43 -0400] "GET /graphics/bar.gif HTTP/1.0" 200 2857 152.123.194.3 - - [28/Aug/1995:07:46:39 -0400] "GET /pub/usenet-b/www/home.html HTTP/1.0" 200 1710 gk-east.usps.gov - - [28/Aug/1995:07:56:44 -0400] "GET /pub/job/vk/view03.jpg HTTP/1.0" 200 8506 indy32.sfc.keio.ac.jp - - [28/Aug/1995:08:05:42 -0400] "GET /pub/jeffd/speaker.gif HTTP/1.0" 200 2646 spectrum.xerox.com - - [28/Aug/1995:08:13:59 -0400] "GET /pub/ibos/iback.gif HTTP/1.0" 304 - poppy.hensa.ac.uk - - [28/Aug/1995:08:22:48 -0400] "GET /pub/madsox/ma-links.html HTTP/1.0" 200 3765 199.128.20.35 - - [28/Aug/1995:08:32:44 -0400] "GET /pub/jeffd/sm_eaghd.gif HTTP/1.0" 200 2866 128.178.83.41 - - [28/Aug/1995:08:38:23 -0400] "GET /atomicbk/seanc.gif HTTP/1.0" 200 734 graal.cril-ing.fr - - [28/Aug/1995:08:46:24 -0400] "GET /pub/pribut/email2.gif HTTP/1.0" 304 - hel.fi - - [28/Aug/1995:08:54:12 -0400] "GET /pub/heroes/wavs/sheen.wav HTTP/1.0" 200 173690 akash.CV.COM - - [28/Aug/1995:09:01:27 -0400] "GET /pub/howie/OO/H.GIF HTTP/1.0" 200 989 ad07-024.compuserve.com - - [28/Aug/1995:09:07:50 -0400] "GET /ntp/ecit/ecitlog1.gif HTTP/1.0" 200 4952 intgate.raleigh.ibm.com - - [28/Aug/1995:09:14:51 -0400] "GET /pub/job/vk/view36.jpg HTTP/1.0" 200 3522 pppdroth.ari.net - - [28/Aug/1995:09:22:02 -0400] "GET /mpt/ttc/female.gif HTTP/1.0" 200 493 piweba2y.prodigy.com - - [28/Aug/1995:09:28:53 -0400] "GET /pub/jumpsam/line_yel.gif HTTP/1.0" 200 1243 pinta.kolumbus.fi - - [28/Aug/1995:09:35:28 -0400] "GET /pub/job/vk/view05.jpg HTTP/1.0" 200 5012 yin.nj.nec.com - - [28/Aug/1995:09:41:58 -0400] "GET /ccentral/npbutton.gif HTTP/1.0" 200 2062 pc105104.shef.ac.uk - - [28/Aug/1995:09:46:44 -0400] "GET /pub/sshay/galybutn.map?210,15 HTTP/1.0" 302 227 eehpx26.cen.uiuc.edu - - [28/Aug/1995:09:54:14 -0400] "GET /atomicbk/catalog/new.gif HTTP/1.0" 200 744 grndserver.library.gatech.edu - - [28/Aug/1995:09:59:44 -0400] "GET /pub/lschank/web/line.gif HTTP/1.0" 200 1131 beta.Xerox.COM - - [28/Aug/1995:10:05:29 -0400] "GET /pub/linda/cody.html HTTP/1.0" 200 664 130.160.248.23 - - [28/Aug/1995:10:11:36 -0400] "GET /pub/jeffd/rush&et.gif HTTP/1.0" 200 15412 155.229.34.2 - - [28/Aug/1995:10:17:08 -0400] "GET /pub/jgbustam/heritage/heritage.html HTTP/1.0" 200 2324 stortek1.stortek.com - - [28/Aug/1995:10:21:14 -0400] "GET /pub/k2/am4x44u/whats_new/4wa2.htm HTTP/1.0" 200 1262 ftp.delmarva.com - - [28/Aug/1995:10:25:20 -0400] "GET /pub/job/vk/bam.gif HTTP/1.0" 200 9076 131.115.140.66 - - [28/Aug/1995:10:30:38 -0400] "GET /pub/job/vk/kathy.jpg HTTP/1.0" 200 7450 128.206.15.192 - - [28/Aug/1995:10:35:15 -0400] "GET /pub/jrinker/redball.gif HTTP/1.0" 200 326 ad22-011.compuserve.com - - [28/Aug/1995:10:40:05 -0400] "GET /pub/jumpsam/logogosp.jpg HTTP/1.0" 200 15227 pipe3.nyc.pipeline.com - - [28/Aug/1995:10:44:22 -0400] "GET /pub/journalism/awesome.html HTTP/1.0" 200 11610 ggb3334-mac1.engin.umich.edu - - [28/Aug/1995:10:49:35 -0400] "GET /pub/job/vk/view03.jpg HTTP/1.0" 200 8506 mic5.hudk.hokudai.ac.jp - - [28/Aug/1995:10:53:44 -0400] "GET /pub/atomicbk/images/bpqueen.gif HTTP/1.0" 200 24160 194.51.81.248 - - [28/Aug/1995:10:59:13 -0400] "GET /pub/lupine/images/tlk/timon.gif HTTP/1.0" 304 - 128.255.196.188 - - [28/Aug/1995:11:04:39 -0400] "GET /pub/job/vk/bam.gif HTTP/1.0" 200 9076 147.74.10.26 - - [28/Aug/1995:11:08:21 -0400] "GET /pub/global/index1.html HTTP/1.0" 200 1065 payday.cc.vt.edu - - [28/Aug/1995:11:11:41 -0400] "GET /pub/mbrophy/pix/washlee.jpg HTTP/1.0" 200 1203 vagrant.vf.mmc.com - - [28/Aug/1995:11:16:04 -0400] "GET /pub/job/vk/view14.jpg HTTP/1.0" 200 4733 153.74.191.27 - - [28/Aug/1995:11:20:17 -0400] "GET /pub/job/vk/view07.jpg HTTP/1.0" 200 7057 clark.net - - [28/Aug/1995:11:23:58 -0400] "GET /pub/job/vk/view26.jpg HTTP/1.0" 304 - www-relay.pa-x.dec.com - - [28/Aug/1995:11:28:49 -0400] "GET /pub/k2/am4x44u/gifs/BGPRCHM2.JPG HTTP/1.0" 404 318 amf99-0.Amersfoort.NL.net - - [28/Aug/1995:11:33:53 -0400] "GET /pub/networx/autopage/exotic/ex002_1s.gif HTTP/1.0" 200 17130 155.225.12.25 - - [28/Aug/1995:11:37:31 -0400] "GET /theme/demockracy/logo3.gif HTTP/1.0" 200 9565 mcdowell-12.mcdowell.udel.edu - - [28/Aug/1995:11:42:26 -0400] "GET /pub/jgbustam/gifs/coloico2.gif HTTP/1.0" 200 4298 remws01.who.ch - - [28/Aug/1995:11:46:58 -0400] "GET /atomicbk/new/newaug25.html HTTP/1.0" 200 6407 bouchon-ppp.clark.net - - [28/Aug/1995:11:51:34 -0400] "GET /pub/bouchon/meter.gif HTTP/1.0" 304 - gatekeeper.bridge.com - - [28/Aug/1995:11:56:03 -0400] "GET /pub/job/vk/view24.jpg HTTP/1.0" 200 6209 x242-16.cla.umn.edu - - [28/Aug/1995:11:59:57 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 200 813 COMPC1.COMP.PANAM.EDU - - [28/Aug/1995:12:05:14 -0400] "GET /pub/job/vk/view08.jpg HTTP/1.0" 200 6476 ix-orl1-08.ix.netcom.com - - [28/Aug/1995:12:09:27 -0400] "GET /pub/job/vk/view30.jpg HTTP/1.0" 200 5991 isr0973.urh.uiuc.edu - - [28/Aug/1995:12:12:45 -0400] "GET /pub/job/vk/view10.jpg HTTP/1.0" 200 5303 144.30.1.220 - - [28/Aug/1995:12:16:31 -0400] "GET /pub/nih/fig_5A.jpg HTTP/1.0" 200 23156 ZEBRA.ENGR.UTK.EDU - - [28/Aug/1995:12:21:04 -0400] "GET /pub/alweiner/w3-bg-1.gif HTTP/1.0" 200 3495 tazman.mindspring.com - - [28/Aug/1995:12:26:05 -0400] "GET /atomicbk/orders.gif HTTP/1.0" 200 800 user32.lightside.com - - [28/Aug/1995:12:29:41 -0400] "GET /pub/ibos/buslinks.html HTTP/1.0" 304 - pshelton.northstar.k12.ak.us - - [28/Aug/1995:12:34:07 -0400] "GET /pub/networx/autopage/dealers/de001_3b.gif HTTP/1.0" 200 70257 rm503-3.inn.usu.edu - - [28/Aug/1995:12:38:13 -0400] "GET /pub/job/vk/view27.jpg HTTP/1.0" 200 6611 eurocent.clark.net - - [28/Aug/1995:12:42:00 -0400] "GET /pub/eurocent/inquiry.htm HTTP/1.0" 200 1988 heather.eee.strath.ac.uk - - [28/Aug/1995:12:46:06 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 www-b6.proxy.aol.com - - [28/Aug/1995:12:50:44 -0400] "GET /pub/heroes/gifs/heston.gif HTTP/1.0" 304 - enterprise.nettrek.com.au - - [28/Aug/1995:12:53:56 -0400] "GET /atomicbk/catalog/sleazmag.html HTTP/1.0" 200 6688 161.106.1.2 - - [28/Aug/1995:12:57:54 -0400] "GET /atomicbk/new/new.html HTTP/1.0" 200 2914 magicall.dacom.co.kr - - [28/Aug/1995:13:02:24 -0400] "GET /pub/job/vk/view02.jpg HTTP/1.0" 200 7290 venus.aacc.cc.md.us - - [28/Aug/1995:13:06:03 -0400] "GET /pub/tblake/www/home.gif HTTP/1.0" 304 - www-a1.proxy.aol.com - - [28/Aug/1995:13:10:20 -0400] "GET /pub/atomicbk/new/new.html HTTP/1.0" 200 2914 Cust20.Max7.San-Francisco.CA.MS.UU.NET - - [28/Aug/1995:13:13:46 -0400] "GET /atomicbk/new/logo2.gif HTTP/1.0" 200 12871 jschultz-pc.chronicle.com - - [28/Aug/1995:13:17:05 -0400] "GET /pub/lschank/web/line2.gif HTTP/1.0" 200 267 slip01.hrz.uni-giessen.de - - [28/Aug/1995:13:21:44 -0400] "GET /atomicbk/images/hbslut.gif HTTP/1.0" 200 43138 www-b4.proxy.aol.com - - [28/Aug/1995:13:25:26 -0400] "GET /pub/networx/autopage/dealers.html HTTP/1.0" 200 1201 ad12-021.compuserve.com - - [28/Aug/1995:13:29:03 -0400] "GET /pub/sshay/interact.html HTTP/1.0" 200 2694 192.72.107.107 - - [28/Aug/1995:13:35:13 -0400] "GET /pub/atomicbk/images/nudistbk.gif HTTP/1.0" 404 335 141.161.119.116 - - [28/Aug/1995:13:39:36 -0400] "GET /pub/jgbustam/gifs/at-work.gif HTTP/1.0" 200 373 io.is.co.za - - [28/Aug/1995:13:44:37 -0400] "GET /pub/murple/home.html HTTP/1.0" 200 3676 192.239.92.145 - - [28/Aug/1995:13:48:44 -0400] "GET /pub/fbc/www/image/md.gif HTTP/1.0" 200 1373 webgate1.mot.com - - [28/Aug/1995:13:52:57 -0400] "GET /atomicbk/seanc.gif HTTP/1.0" 200 734 pc1-195.health.org - - [28/Aug/1995:13:56:55 -0400] "GET /pub/michele/backgrds/zag-zig.gif HTTP/1.0" 200 10016 www-a1.proxy.aol.com - - [28/Aug/1995:14:00:47 -0400] "GET /pub/job/vk/view36.jpg HTTP/1.0" 304 - buckman.com - - [28/Aug/1995:14:04:01 -0400] "GET /pub/sgs/usdotsaf.htm HTTP/1.0" 200 615030 165.134.64.150 - - [28/Aug/1995:14:07:53 -0400] "GET /pub/sshay/images/nrcthumb.jpg HTTP/1.0" 200 4215 www-c3.proxy.aol.com - - [28/Aug/1995:14:11:37 -0400] "GET /pub/jeffd/grad_lin.gif HTTP/1.0" 200 3346 130.163.112.144 - - [28/Aug/1995:14:16:00 -0400] "GET /pub/job/vk/view17.jpg HTTP/1.0" 200 5944 ph20324.cpmc.columbia.edu - - [28/Aug/1995:14:20:26 -0400] "GET /pub/ibos/fishform.html HTTP/1.0" 200 1750 asp.erinet.com - - [28/Aug/1995:14:25:00 -0400] "GET /pub/journalism/awesome.html HTTP/1.0" 304 - pk21010.bwi.wec.com - - [28/Aug/1995:14:28:16 -0400] "GET /graphics/clark5.gif HTTP/1.0" 200 39568 gort.byu.edu - - [28/Aug/1995:14:31:45 -0400] "GET /pub/job/vk/view06.jpg HTTP/1.0" 200 6553 a42.mor.itesm.mx - - [28/Aug/1995:14:35:24 -0400] "GET /pub/jgbustam/toros/bullarts.html HTTP/1.0" 200 507 woobie.UU.NET - - [28/Aug/1995:14:39:16 -0400] "GET /pub/jgbustam/paises/peruic.gif HTTP/1.0" 304 - sgs.clark.net - - [28/Aug/1995:14:43:16 -0400] "GET /pub/sgs/spr/l_arrow.gif HTTP/1.0" 200 1003 interlock.turner.com - - [28/Aug/1995:14:46:19 -0400] "GET /pub/sshay/images/linkbutn.gif HTTP/1.0" 200 1080 colt.mais.hydro.qc.ca - - [28/Aug/1995:14:50:59 -0400] "GET /pub/k2/am4x44u/events/clubs/jcoa.htm HTTP/1.0" 200 1878 indy4.ecf.toronto.edu - - [28/Aug/1995:14:57:50 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 14969 204.70.239.154 - - [28/Aug/1995:15:01:23 -0400] "GET /pub/robert/rain_l~1.gif HTTP/1.0" 200 2243 140.150.91.197 - - [28/Aug/1995:15:04:53 -0400] "GET /pub/job/vk/view23.jpg HTTP/1.0" 200 5165 avarice.obsolete.com - - [28/Aug/1995:15:08:23 -0400] "GET /pub/murple/home.html HTTP/1.0" 304 - labg2.comer.wvu.edu - - [28/Aug/1995:15:11:29 -0400] "GET /pub/job/vk/view02.jpg HTTP/1.0" 200 7290 gwa.ericsson.com - - [28/Aug/1995:15:14:37 -0400] "GET /pub/job/canyon.jpg HTTP/1.0" 200 9125 exinos.ms.com - - [28/Aug/1995:15:18:28 -0400] "GET /pub/jridgely/dc/wash.gif HTTP/1.0" 200 1034 ppru-lj-w3.ucsd.edu - - [28/Aug/1995:15:23:13 -0400] "GET /pub/ashp/grafx\acrobatg.gif HTTP/1.0" 404 323 www-b3.proxy.aol.com - - [28/Aug/1995:15:26:43 -0400] "GET /pub/job/vk/view14.jpg HTTP/1.0" 200 4733 199.212.26.36 - - [28/Aug/1995:15:30:19 -0400] "GET /pub/lupine/www/images/support/mickey.gif HTTP/1.0" 200 2383 204.160.213.159 - - [28/Aug/1995:15:34:24 -0400] "GET /pub/k2/am4x44u/gifs/jyj.gif HTTP/1.0" 200 2241 slip145-44.ut.nl.ibm.net - - [28/Aug/1995:15:37:45 -0400] "GET /pub/rsjdfg/ HTTP/1.0" 200 2109 csc005.cas.usf.edu - - [28/Aug/1995:15:40:26 -0400] "GET /pub/jeffd/grad_lin.gif HTTP/1.0" 200 3346 gateway.rsinc.com - - [28/Aug/1995:15:44:14 -0400] "GET /mc-icons/back.gif HTTP/1.0" 200 174 sku-pc17.esu3.k12.ne.us - - [28/Aug/1995:15:47:39 -0400] "GET /pub/k2/am4x44u/gifs/netscape.gif HTTP/1.0" 200 1050 fnugget.intel.com - - [28/Aug/1995:15:50:58 -0400] "GET /pub/peace/grball.gif HTTP/1.0" 304 - venus.ivic.ve - - [28/Aug/1995:15:54:27 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 flynnpc.sunydutchess.edu - - [28/Aug/1995:15:58:17 -0400] "GET /pub/eclipse/home/telegraph-key.xbm HTTP/1.0" 200 7011 dd19-004.compuserve.com - - [28/Aug/1995:16:02:07 -0400] "GET /pub/sshay/images/rainlin2.gif HTTP/1.0" 200 1229 shelvik_ibmpc.slc.mke.ab.com - - [28/Aug/1995:16:06:55 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 14969 cumac.truevision.com - - [28/Aug/1995:16:11:08 -0400] "GET /pub/shelby/imaglogo.gif HTTP/1.0" 200 3662 cs1-13.eph.ptd.net - - [28/Aug/1995:16:13:31 -0400] "GET /pub/job/swim/v-47.jpg HTTP/1.0" 200 2964 136.205.143.61 - - [28/Aug/1995:16:16:22 -0400] "GET /pub/job/vk/view07.jpg HTTP/1.0" 200 7057 ervinper.natinst.com - - [28/Aug/1995:16:19:33 -0400] "GET /pub/job/vk/view12.jpg HTTP/1.0" 200 5966 194.170.2.41 - - [28/Aug/1995:16:22:29 -0400] "GET /pub/listserv/listserv.html HTTP/1.0" 200 1138 128.204.73.68 - - [28/Aug/1995:16:25:36 -0400] "GET /pub/job/swim/swim.jpg HTTP/1.0" 200 15234 news.stucen.gatech.edu - - [28/Aug/1995:16:29:39 -0400] "GET /pub/job/vk/view08.jpg HTTP/1.0" 200 6476 gesg51.geo.census.gov - - [28/Aug/1995:16:33:54 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 orange.ge.com - - [28/Aug/1995:16:38:03 -0400] "GET /pub/job/vk/cindy.jpg HTTP/1.0" 200 4267 zeke.gov.yk.ca - - [28/Aug/1995:16:41:41 -0400] "GET / HTTP/1.0" 200 1834 143.166.148.44 - - [28/Aug/1995:16:46:15 -0400] "GET /pub/cargui/ctpltmst.gif HTTP/1.0" 200 13757 Laplace.CL.UH.EDU - - [28/Aug/1995:16:49:43 -0400] "GET /pub/fan/albumpics/overdose.jpg HTTP/1.0" 200 3152 emeldir.dd.chalmers.se - - [28/Aug/1995:16:55:17 -0400] "GET /pub/alweiner/cgi-bin/homepage.html?wall-kidfun HTTP/1.0" 200 959 p2151_01.ifmt.nf.ca - - [28/Aug/1995:16:58:27 -0400] "GET /atomicbk/new/new.html HTTP/1.0" 200 2914 155.234.241.41 - - [28/Aug/1995:17:01:52 -0400] "GET /mpt/motorwk.gif HTTP/1.0" 200 6008 wagnerjh.norand.com - - [28/Aug/1995:17:05:28 -0400] "GET /pub/jeffd/rushpage.html HTTP/1.0" 304 - 137.53.22.115 - - [28/Aug/1995:17:09:27 -0400] "GET /pub/job/vk/cindy.jpg HTTP/1.0" 200 4267 osiris.gsfc.nasa.gov - - [28/Aug/1995:17:13:42 -0400] "GET /pub/sandy/chromatics/chrom3.gif HTTP/1.0" 200 15422 Avauca.usae.bah.com - - [28/Aug/1995:17:17:30 -0400] "GET /pub/job/vk/vk-bk.jpg HTTP/1.0" 200 4569 198.232.176.77 - - [28/Aug/1995:17:22:02 -0400] "GET /pub/job/vk/view17.jpg HTTP/1.0" 200 5944 p827229.lanl.gov - - [28/Aug/1995:17:25:57 -0400] "GET /pub/k2/am4x44u/gifs/spacer.gif HTTP/1.0" 200 833 user61.ef.com - - [28/Aug/1995:17:30:45 -0400] "GET /pub/atomicbk/catalog/new.gif HTTP/1.0" 200 744 136.183.44.40 - - [28/Aug/1995:17:34:15 -0400] "GET /pub/job/vk/view09.jpg HTTP/1.0" 200 6011 jeremy.leitess.com - - [28/Aug/1995:17:37:48 -0400] "GET /pub/ HTTP/1.0" 200 419 156.80.138.125 - - [28/Aug/1995:17:41:25 -0400] "GET /pub/alweiner/work/_0153T.gif HTTP/1.0" 200 813 ichthus.stsci.edu - - [28/Aug/1995:17:46:03 -0400] "GET /pub/job/vk/view14.jpg HTTP/1.0" 200 4733 advising.life.uiuc.edu - - [28/Aug/1995:17:50:24 -0400] "GET /pub/atomicbk/catalog/mini.gif HTTP/1.0" 200 360 192.139.10.109 - - [28/Aug/1995:17:54:38 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 schwesig.lib.uchicago.edu - - [28/Aug/1995:17:59:54 -0400] "GET /pub/lschank/web/gpo.gif HTTP/1.0" 200 3866 beach.sctc.com - - [28/Aug/1995:18:04:09 -0400] "GET /pub/rjgula/network.htm HTTP/1.0" 200 2017 user.8.30.dcccd.edu - - [28/Aug/1995:18:08:14 -0400] "GET /atomicbk/promo/promo.html HTTP/1.0" 200 1657 stout.Stanford.EDU - - [28/Aug/1995:18:13:09 -0400] "GET /khem/cgi-bin/odometer.xbm HTTP/1.0" 500 305 sopines210.nando.net - - [28/Aug/1995:18:18:00 -0400] "GET /pub/sshay/about.html HTTP/1.0" 200 18751 nwchi-d153.net.interaccess.com - - [28/Aug/1995:18:21:33 -0400] "GET /pub/k2/am4x44u/gifs/jzj.gif HTTP/1.0" 200 2351 fciris9.NCIFCRF.GOV - - [28/Aug/1995:18:24:55 -0400] "GET /pub/job/vk/view02.jpg HTTP/1.0" 200 7290 hamelwp1.nu.com - - [28/Aug/1995:18:29:14 -0400] "GET /pub/sshay/images/jaseeddy.gif HTTP/1.0" 200 9455 elizabeth.lighthouse.com - - [28/Aug/1995:18:33:27 -0400] "GET /pub/ahasuer/heinlein/stinkeroos.html HTTP/1.0" 200 1096 prpayne-ppp.clark.net - - [28/Aug/1995:18:37:21 -0400] "GET /pub/shows/starback.gif HTTP/1.0" 200 2053 ngriffin.itc.gu.edu.au - - [28/Aug/1995:18:42:37 -0400] "GET /pub/amacks/Pages/misc/hot.gif HTTP/1.0" 200 1007 lagrange.pnl.gov - - [28/Aug/1995:18:46:51 -0400] "GET /pub/job/vk/view27.jpg HTTP/1.0" 200 6611 fs-jupiter.fvh.csc.com - - [28/Aug/1995:18:51:46 -0400] "GET /pub/job/vk/v-sig.gif HTTP/1.0" 200 4864 pilot4.clark.net - - [28/Aug/1995:18:56:58 -0400] "GET /pub/journalism/awesome.html HTTP/1.0" 304 - peg1-ts1.peganet.com - - [28/Aug/1995:19:02:40 -0400] "GET /pub/job/vk/v-sig.gif HTTP/1.0" 200 4864 succeed.engnet.ufl.edu - - [28/Aug/1995:19:07:02 -0400] "GET /pub/jeffd/cgi-bin/dodger.html HTTP/1.0" 200 76656 bfas02.fas.okstate.edu - - [28/Aug/1995:19:12:59 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 cla-jvirts.unl.edu - - [28/Aug/1995:19:17:18 -0400] "GET /pub/lupine/images/tlk/pride-rock.gif HTTP/1.0" 200 51600 line106.worldweb.net - - [28/Aug/1995:19:21:22 -0400] "GET /pub/robert/rain_l~1.gif HTTP/1.0" 200 2243 doenx-452-2.Lib.Berkeley.EDU - - [28/Aug/1995:19:25:09 -0400] "GET /pub/jgbustam/toros/toroglos.html HTTP/1.0" 200 20302 info1.lancs.ac.uk - - [28/Aug/1995:19:29:02 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 www-a1.proxy.aol.com - - [28/Aug/1995:19:33:45 -0400] "GET /pub/job/vk/v-sig.gif HTTP/1.0" 200 4864 uranus40.terraport.net - - [28/Aug/1995:19:37:56 -0400] "GET /pub/atomicbk/images/tease.gif HTTP/1.0" 200 43516 trusty.lmsc.lockheed.com - - [28/Aug/1995:19:43:40 -0400] "GET /pub/howie/OO/construc.gif HTTP/1.0" 200 252 hawaii-5.u.aloha.net - - [28/Aug/1995:19:48:28 -0400] "GET /pub/jeffd/cgi-bin/ngquote.html HTTP/1.0" 200 1564 beauty.fnet.cs.mci.com - - [28/Aug/1995:19:54:37 -0400] "GET /pub/atomicbk/catalog/pinup.html HTTP/1.0" 200 8316 slip-14-9.ots.utexas.edu - - [28/Aug/1995:19:59:17 -0400] "GET /pub/job/vk/view06.jpg HTTP/1.0" 200 6553 orpheus.amdahl.com - - [28/Aug/1995:20:05:03 -0400] "GET /pub/job/swim/v-44.jpg HTTP/1.0" 200 2483 yukon.econ.ucsb.edu - - [28/Aug/1995:20:10:58 -0400] "GET /pub/job/vk/view07.jpg HTTP/1.0" 200 7057 128.102.143.208 - - [28/Aug/1995:20:15:37 -0400] "GET /pub/job/swim/v-45.jpg HTTP/1.0" 200 2531 205.133.11.40 - - [28/Aug/1995:20:22:01 -0400] "GET /network/graphics/sps.logo.gif HTTP/1.0" 200 3134 panopticon.vnet.net - - [28/Aug/1995:20:27:09 -0400] "GET /pub/rasputin/www/buttons/DocsRightArrow.gif HTTP/1.0" 200 1273 lh023-776102.ucs.indiana.edu - - [28/Aug/1995:20:32:02 -0400] "GET /pub/jeffd/grayball.gif HTTP/1.0" 200 995 chrivb01.cch.com - - [28/Aug/1995:20:36:12 -0400] "GET /pub/jumpsam/index.htm HTTP/1.0" 200 3107 www-b1.proxy.aol.com - - [28/Aug/1995:20:42:42 -0400] "GET /pub/k2/am4x44u/gifs/redball.gif HTTP/1.0" 200 206 www-c4.proxy.aol.com - - [28/Aug/1995:20:49:01 -0400] "GET /pub/networx/autopage/autopage.html HTTP/1.0" 304 - saturn202.terraport.net - - [28/Aug/1995:20:54:21 -0400] "GET /ntp/ecit/ecitlog1.gif HTTP/1.0" 200 4952 in27.inetnebr.com - - [28/Aug/1995:20:59:48 -0400] "GET /pub/jeffd/feedback.html HTTP/1.0" 200 1911 204.119.181.214 - - [28/Aug/1995:21:05:23 -0400] "GET /pub/job/vk/vk-ad-32.jpg HTTP/1.0" 200 59411 204.119.181.214 - - [28/Aug/1995:21:11:15 -0400] "GET /pub/job/swim/v-57.jpg HTTP/1.0" 200 2717 204.255.177.3 - - [28/Aug/1995:21:16:08 -0400] "GET /pub/russadam/cgi-bin/textbase.cgi?COMMAND=DOSEARCH&FIELDS=Company%2CAddress%2CCity%2CState%2CZip%2CPhone%2CFax%0A&FILENAME=idname2.txt&TITLE=BarCode+1+Company+Name+Search&HTMLFILE=%2Fhomea%2Fftp%2Fpub%2Frussadam%2Fcobase.html&HTMLURL=http%3A%2F%2Fwww.adams1.com%2Fpub%2Frussadam%2Fcobase.html&PAGENAME=test.html&DISPLAYKEY=0%2C1%2C2%2C3%2C4%2C5%2C6%2C&ENDINGKEY=BR%2CBR%2CCS%2CS%2CBR%2CS%2CBR%2C&SEARCHKEY=0%2C3%2C&PROMPTKEY=Company+Name%2CState%2C&DPROMPTKEY=%2C%2C%2C%2C%2CPhone%2CFax%2C&TYPEKEY=T%2CL2%2C&CHKVALUEKEY=YES%2CYES%2C&L1VALUEKEY=%2C%2C&ANDOR=OR&FS0=Y&SV0=&FS3=Y&SV3=NY HTTP/1.0" 200 450 www-b5.proxy.aol.com - - [28/Aug/1995:21:21:05 -0400] "GET /pub/rj/INDEX.HTM HTTP/1.0" 200 146936 STLGATE.ALMADEN.IBM.COM - - [28/Aug/1995:21:25:12 -0400] "GET /pub/sshay/partner.html HTTP/1.0" 200 7065 RSC535.smumn.edu - - [28/Aug/1995:21:29:12 -0400] "GET /pub/job/vk/v-line.gif HTTP/1.0" 200 1254 pc2571.Princeton.EDU - - [28/Aug/1995:21:35:26 -0400] "GET /pub/job/swim/v-51.jpg HTTP/1.0" 200 2574 202.24.142.160 - - [28/Aug/1995:21:41:06 -0400] "GET /pub/job/vk/view29.jpg HTTP/1.0" 200 6695 slip34-211.il.us.ibm.net - - [28/Aug/1995:21:46:15 -0400] "GET /pub/atomicbk/email.gif HTTP/1.0" 200 756 dip-24.mbay.net - - [28/Aug/1995:21:51:17 -0400] "GET /printing/ HTTP/1.0" 200 669 msp7-8.nas.MR.Net - - [28/Aug/1995:21:57:07 -0400] "GET /pub/lupine/www/other-disney.html HTTP/1.0" 200 17386 hlm03.itc.uiowa.edu - - [28/Aug/1995:22:02:10 -0400] "GET /pub/aztec/hans/hc43-03z.gif HTTP/1.0" 200 5602 199.199.168.44 - - [28/Aug/1995:22:07:52 -0400] "GET /pub/ashp/grafx/guest.gif HTTP/1.0" 200 1171 dal17.onramp.net - - [28/Aug/1995:22:13:26 -0400] "GET /graphics/lh7.gif HTTP/1.0" 200 23642 athena.veritas.com - - [28/Aug/1995:22:18:03 -0400] "GET /pub/ibos/dollar.gif HTTP/1.0" 200 1014 itg.itg.net - - [28/Aug/1995:22:22:36 -0400] "GET /pub/alweiner/cgi-bin/homepage.html?viewsrc2 HTTP/1.0" 200 2565 www-a1.proxy.aol.com - - [28/Aug/1995:22:28:04 -0400] "GET /atomicbk/orderform.html HTTP/1.0" 200 4540 ix-lou-ky1-03.ix.netcom.com - - [28/Aug/1995:22:35:03 -0400] "GET /pub/jeffd/header.gif HTTP/1.0" 200 24376 164.110.101.235 - - [28/Aug/1995:22:41:03 -0400] "GET /pub/listserv/lsedu1.html HTTP/1.0" 200 49944 131.115.140.171 - - [28/Aug/1995:22:45:40 -0400] "GET /pub/job/vk/view36.jpg HTTP/1.0" 200 3522 tgif.bradley.edu - - [28/Aug/1995:22:51:54 -0400] "GET /pub/job/vk/view31.jpg HTTP/1.0" 200 5611 piweba4y.prodigy.com - - [28/Aug/1995:22:57:54 -0400] "GET /pub/mbrophy/pix/address.jpg HTTP/1.0" 304 - Cust11.Max1.San-Francisco.CA.MS.UU.NET - - [28/Aug/1995:23:03:12 -0400] "GET /pub/atomicbk/catalog/mini.gif HTTP/1.0" 200 360 pppa095.compuserve.com - - [28/Aug/1995:23:09:46 -0400] "GET /pub/mbrophy/pix/holly.jpg HTTP/1.0" 200 4840 ad03-023.compuserve.com - - [28/Aug/1995:23:15:01 -0400] "GET /pub/lwallach/me.html HTTP/1.0" 200 719 limnos.wes.army.mil - - [28/Aug/1995:23:21:37 -0400] "GET /pub/job/vk/view02.jpg HTTP/1.0" 200 7290 ict56.southwind.net - - [28/Aug/1995:23:25:47 -0400] "GET /pub/rsjdfg/Images/MissSaigon.gif HTTP/1.0" 200 21560 indy1.indy.net - - [28/Aug/1995:23:31:18 -0400] "GET /pub/job/swim/v-51.jpg HTTP/1.0" 200 2574 cs125-11.u.washington.edu - - [28/Aug/1995:23:35:45 -0400] "GET /pub/k2/cgi-bin/imagemap/events?251,93 HTTP/1.0" 302 102 PC11134.GSIA.CMU.EDU - - [28/Aug/1995:23:40:27 -0400] "GET /atomicbk/logo2.gif HTTP/1.0" 200 12871 130.127.26.67 - - [28/Aug/1995:23:46:23 -0400] "GET /pub/job/vendela.jpg HTTP/1.0" 200 36746 DIALIP128.GOV.BC.CA - - [28/Aug/1995:23:51:33 -0400] "GET /atomicbk/catalog/emboss.jpg HTTP/1.0" 200 4741 ppp20.tcs.tulane.edu - - [28/Aug/1995:23:57:47 -0400] "HEAD /pub/robert/ HTTP/1.0" 200 1992 forth.microserve.com - - [29/Aug/1995:00:05:26 -0400] "GET /pub/rarnold/firesign/passport.wav HTTP/1.0" 200 53390 amf3.res.Lehigh.EDU - - [29/Aug/1995:00:11:19 -0400] "GET /pub/job/swim/swim.jpg HTTP/1.0" 200 15234 miriworld.its.unimelb.EDU.AU - - [29/Aug/1995:00:17:29 -0400] "GET /pub/job/vk/view01.jpg HTTP/1.0" 304 - www-d3.proxy.aol.com - - [29/Aug/1995:00:25:31 -0400] "GET /pub/mikebilt/feldie.jpg HTTP/1.0" 200 6808 piweba3y.prodigy.com - - [29/Aug/1995:00:32:08 -0400] "GET /pub/downin/cgi-bin/arlonet.html HTTP/1.0" 200 4339 137.112.203.144 - - [29/Aug/1995:00:39:34 -0400] "GET /pub/job/vk/view16.jpg HTTP/1.0" 200 6808 www-b2.proxy.aol.com - - [29/Aug/1995:00:45:33 -0400] "GET /pub/edseiler/WWW/asimov_FAQ.html HTTP/1.0" 304 - garlic.cchem.berkeley.edu - - [29/Aug/1995:00:52:05 -0400] "GET /pub/sshay/images/chris1.jpg HTTP/1.0" 200 13306 dialup8.woodtech.com - - [29/Aug/1995:00:59:19 -0400] "GET /pub/job/swim/v-50.jpg HTTP/1.0" 200 3369 s330250.slip.cc.uq.oz.au - - [29/Aug/1995:01:05:37 -0400] "GET /pub/global/search.html HTTP/1.0" 304 - host29.cyberg8t.com - - [29/Aug/1995:01:11:47 -0400] "GET /pub/rsjdfg/Links.html HTTP/1.0" 200 1632 cs13port.netvoyage.net - - [29/Aug/1995:01:19:23 -0400] "GET /pub/wilsey/cat31.html HTTP/1.0" 200 62906 F182-077.net.wisc.edu - - [29/Aug/1995:01:27:07 -0400] "GET /pub/rsjdfg/Images/Saigon.gif HTTP/1.0" 200 16721 crosslnk.cts.com - - [29/Aug/1995:01:34:35 -0400] "GET /pub/alweiner/color_sw.gif HTTP/1.0" 200 1645 pm59.ilhawaii.net - - [29/Aug/1995:01:42:31 -0400] "GET /pub/sshay/images/rainlin2.gif HTTP/1.0" 200 1229 165.113.187.61 - - [29/Aug/1995:01:50:15 -0400] "GET /pub/job/swim/s-back3.jpg HTTP/1.0" 200 5118 200.23.36.61 - - [29/Aug/1995:01:57:44 -0400] "GET /atomicbk/catalog/sleazmag.html HTTP/1.0" 200 6688 TS9-41.UPENN.EDU - - [29/Aug/1995:02:07:03 -0400] "POST /pub/alweiner/cgi-bin/w3magic.cgi?-f|color HTTP/1.0" 200 121 204.112.94.55 - - [29/Aug/1995:02:15:30 -0400] "GET /atomicbk/images/angie.gif HTTP/1.0" 200 33638 pm82.sonic.net - - [29/Aug/1995:02:24:12 -0400] "GET /pub/atomicbk/catalog/adultcom.html HTTP/1.0" 200 8705 z06125.iie.org.mx - - [29/Aug/1995:02:29:45 -0400] "GET /atomicbk/catalog/orders.gif HTTP/1.0" 200 800 www-b3.proxy.aol.com - - [29/Aug/1995:02:36:54 -0400] "GET /pub/job/vk/view22.jpg HTTP/1.0" 304 - klothos.crl.research.digital.com - - [29/Aug/1995:02:46:13 -0400] "GET /pub/job/vk/view15.jpg HTTP/1.0" 200 6036 n115.solano.community.net - - [29/Aug/1995:02:54:26 -0400] "GET /pub/quaker/web/archive.html HTTP/1.0" 200 8865 Thomas6-12.FandM.edu - - [29/Aug/1995:03:04:57 -0400] "GET /atomicbk/catalog/mini.gif HTTP/1.0" 200 360 hopi2.ResComp.Arizona.EDU - - [29/Aug/1995:03:18:46 -0400] "GET /atomicbk/new/logo2.gif HTTP/1.0" 200 12871 www-b2.proxy.aol.com - - [29/Aug/1995:03:29:52 -0400] "GET /pub/sshay/gallery.html HTTP/1.0" 200 2101 kb1165.konbib.nl - - [29/Aug/1995:03:41:32 -0400] "GET /pub/lschank/web/fwd.gif HTTP/1.0" 200 275 tuz.isf.ru - - [29/Aug/1995:03:53:10 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 14969 client23.sct.fr - - [29/Aug/1995:04:06:23 -0400] "GET /pub/murple/gifs/button.www.gif HTTP/1.0" 200 358 slip-48-8.ots.utexas.edu - - [29/Aug/1995:04:18:38 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 mpngate1.ny.us.ibm.net - - [29/Aug/1995:04:33:40 -0400] "GET /pub/job/swim/swim.jpg HTTP/1.0" 200 15234 nttcsa.tas.NTT.JP - - [29/Aug/1995:04:48:25 -0400] "GET /pub/fan/pics/ManyMakiS.jpg HTTP/1.0" 200 6792 alweiner.clark.net - - [29/Aug/1995:05:01:37 -0400] "GET /pub/alweiner/home.gif HTTP/1.0" 200 372 194.170.2.47 - - [29/Aug/1995:05:16:47 -0400] "GET /atomicbk/catalog/zines.html HTTP/1.0" 200 2968 194.170.2.47 - - [29/Aug/1995:05:26:39 -0400] "GET /pub/atomicbk/userlink.gif HTTP/1.0" 200 816 194.20.202.222 - - [29/Aug/1995:05:39:28 -0400] "GET /atomicbk/catalog/new.gif HTTP/1.0" 200 744 novi.auc.dk - - [29/Aug/1995:05:51:29 -0400] "GET /pub/job/vk/vk-ad-10.jpg HTTP/1.0" 200 53676 pearl.cs.umu.se - - [29/Aug/1995:06:03:24 -0400] "GET /pub/howie/OO/oo_home.html HTTP/1.0" 200 989 199.100.193.14 - - [29/Aug/1995:06:17:37 -0400] "GET /pub/peace/PC1.map?116,53 HTTP/1.0" 302 226 ix-gra2-09.ix.netcom.com - - [29/Aug/1995:06:30:32 -0400] "GET /pub/jeffd/question.html HTTP/1.0" 200 56585 chiak.kaist.ac.kr - - [29/Aug/1995:06:43:00 -0400] "GET /pub/sshay/images/btthumb.jpg HTTP/1.0" 200 4624 hardy.ocs.mq.edu.au - - [29/Aug/1995:06:52:04 -0400] "GET /pub/rsjdfg/ HTTP/1.0" 200 2109 130.237.148.200 - - [29/Aug/1995:07:00:37 -0400] "GET /pub/job/vk/view27.jpg HTTP/1.0" 200 6611 lotte.lmaster.u-bordeaux.fr - - [29/Aug/1995:07:09:41 -0400] "GET /pub/aztec/shesails/sslogo2.gif HTTP/1.0" 200 7214 igate.nrc.gov - - [29/Aug/1995:07:21:11 -0400] "GET /pub/job/vk/view24.jpg HTTP/1.0" 200 6209 isr1101.urh.uiuc.edu - - [29/Aug/1995:07:29:58 -0400] "GET /atomicbk/shocked/shocked.jpg HTTP/1.0" 200 43819 east.ge.com - - [29/Aug/1995:07:38:31 -0400] "GET /pub/networx/autopage/dealers/de002_ba.gif HTTP/1.0" 200 62639 kaz2.chem.s.u-tokyo.ac.jp - - [29/Aug/1995:07:47:45 -0400] "GET /pub/atomicbk/images/axfords.gif" 200 47332 annap2.jsc.mil - - [29/Aug/1995:07:56:55 -0400] "GET /home.html HTTP/1.0" 304 - 131.85.81.10 - - [29/Aug/1995:08:05:21 -0400] "GET /pub/mmathai/mls/dcmls.html HTTP/1.0" 200 3831 164.9.127.22 - - [29/Aug/1995:08:13:16 -0400] "GET /pub/job/vk/view11.jpg HTTP/1.0" 200 5001 133.50.75.49 - - [29/Aug/1995:08:20:27 -0400] "GET /atomicbk/catalog/sleazmag.html HTTP/1.0" 200 6688 140.112.52.180 - - [29/Aug/1995:08:27:25 -0400] "GET /pub/phil/WinonaRyder/articles/EntertainmentWeekly.27Jan95.html HTTP/1.0" 200 1205 gatekeeper.mitre.org - - [29/Aug/1995:08:32:54 -0400] "GET /pub/gen/fas/irp/offdocs.htm HTTP/1.0" 200 2654 edslink.eds.com - - [29/Aug/1995:08:39:51 -0400] "GET /pub/job/vk/view18.jpg HTTP/1.0" 200 6578 roland.univ-rennes1.fr - - [29/Aug/1995:08:44:16 -0400] "GET /pub/atomicbk/catalog/mini.gif HTTP/1.0" 304 - ns.sis.steria.fr - - [29/Aug/1995:08:50:50 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 14969 osk0155.bekkoame.or.jp - - [29/Aug/1995:08:56:45 -0400] "GET /pub/sshay/images/home6.gif HTTP/1.0" 200 58652 194.17.131.6 - - [29/Aug/1995:09:01:56 -0400] "GET /pub/job/vk/flowers1.gif HTTP/1.0" 304 - 165.13.35.100 - - [29/Aug/1995:09:07:23 -0400] "GET /pub/job/vk/view33.jpg HTTP/1.0" 200 5903 macsim1.sim.ucm.es - - [29/Aug/1995:09:13:48 -0400] "GET /pub/jgbustam/coloquio/jb.html HTTP/1.0" 200 338 maine.sdi.agate.net - - [29/Aug/1995:09:19:30 -0400] "GET /pub/robert/past99.gif HTTP/1.0" 200 4993 gandalf2.ccm.itesm.mx - - [29/Aug/1995:09:26:35 -0400] "GET /pub/gorbachev/gorbglobe.jpg HTTP/1.0" 200 4646 entek1.entek.chalmers.se - - [29/Aug/1995:09:32:17 -0400] "GET /pub/job/vk/view20.jpg HTTP/1.0" 200 5070 www.its.it - - [29/Aug/1995:09:36:45 -0400] "GET /pub/job/vk/view02.jpg HTTP/1.0" 200 7290 jwolff.clark.net - - [29/Aug/1995:09:41:21 -0400] "GET /pub/jwolff/cki/secret/secret.html HTTP/1.0" 403 142 rosamond.tacsim.com - - [29/Aug/1995:09:45:45 -0400] "GET /pub/wsg/html/jan/ HTTP/1.0" 200 476 fourier.civil.gla.ac.uk - - [29/Aug/1995:09:51:20 -0400] "GET /pub/alweiner/snip.gif HTTP/1.0" 200 369 165.173.39.252 - - [29/Aug/1995:09:57:57 -0400] "GET /pub/wmcbrine/html/Madonna.html HTTP/1.0" 200 13440 crockett1d.its.rpi.edu - - [29/Aug/1995:10:03:41 -0400] "GET /pub/atomicbk/images/sorayama.gif HTTP/1.0" 404 335 h_adept.roanoke.infi.net - - [29/Aug/1995:10:08:15 -0400] "GET /html/usage/usage.graph.small.gif HTTP/1.0" 200 145 yarrow.wt.com.au - - [29/Aug/1995:10:13:29 -0400] "GET /pub/job/vk/view32.jpg HTTP/1.0" 200 4852 ix-sj28-02.ix.netcom.com - - [29/Aug/1995:10:17:37 -0400] "POST /theme/cgi-bin/serchstr.html HTTP/1.0" 200 999 kalv99.ubt.unit.no - - [29/Aug/1995:10:22:24 -0400] "GET /pub/job/vk/vk-ad-28.jpg HTTP/1.0" 200 44081 sun20.ccd.bnl.gov - - [29/Aug/1995:10:26:34 -0400] "GET /pub/job/vk/view15.jpg HTTP/1.0" 200 6036 mtrsppp49.epix.net - - [29/Aug/1995:10:31:51 -0400] "GET /pub/alweiner/blue.gif HTTP/1.0" 200 148 bettong.client.uq.oz.au - - [29/Aug/1995:10:37:24 -0400] "GET /pub/wick/other_queer.html HTTP/1.0" 200 2498 gsv.gu.se - - [29/Aug/1995:10:42:22 -0400] "GET /pub/job/vk/view34.jpg HTTP/1.0" 200 5156 harold.nrl.navy.mil - - [29/Aug/1995:10:46:22 -0400] "GET /pub/wick/misc/clarknet.gif HTTP/1.0" 200 3937 HEPHP02.PHYS.UTK.EDU - - [29/Aug/1995:10:49:35 -0400] "GET /pub/job/vk/view24.jpg HTTP/1.0" 200 6209 193.12.239.220 - - [29/Aug/1995:10:53:19 -0400] "GET /pub/phil/images/jh21.gif HTTP/1.0" 200 3021 stanbk.pr.mcs.net - - [29/Aug/1995:10:57:02 -0400] "GET /pub/lschank/web/redball.gif HTTP/1.0" 200 326 webb.pc.uamont.edu - - [29/Aug/1995:11:01:34 -0400] "GET /pub/linda/fcr/mouthful.jpg HTTP/1.0" 200 31963 piweba3y.prodigy.com - - [29/Aug/1995:11:06:36 -0400] "GET /pub/jgbustam/gifs/coloico2.gif HTTP/1.0" 304 - mencel.healthspan.edu - - [29/Aug/1995:11:11:48 -0400] "GET /pub/mjoneill/links.html HTTP/1.0" 404 346 walters.aiss.uiuc.edu - - [29/Aug/1995:11:15:53 -0400] "GET /pub/pribut/spsport.html HTTP/1.0" 200 3589 in116.brann.co.uk - - [29/Aug/1995:11:19:59 -0400] "GET /pub/k2/am4x44u/gifs/jjbg2.jpg HTTP/1.0" 200 169367 macomb.lib.mi.us - - [29/Aug/1995:11:24:55 -0400] "GET /pub/cargui/search.html HTTP/1.0" 200 2848 tongkey.kol.co.kr - - [29/Aug/1995:11:30:20 -0400] "GET /pub/job/vk/vk-bk.jpg HTTP/1.0" 200 4569 149.157.246.44 - - [29/Aug/1995:11:35:31 -0400] "GET /pub/k2/am4x44u/gifs/jjbg2.jpg HTTP/1.0" 200 169367 194.170.2.64 - - [29/Aug/1995:11:39:28 -0400] "GET /pub/job/vk/claudia.jpg HTTP/1.0" 200 7225 turbodog.edoc.com - - [29/Aug/1995:11:44:30 -0400] "GET /pub/craver/surreal/surreal.html HTTP/1.0" 200 1479 www.npd.com - - [29/Aug/1995:11:49:08 -0400] "GET /pub/k2/am4x44u/gifs/scut.gif HTTP/1.0" 200 1404 datan.sk.uppsala.se - - [29/Aug/1995:11:52:50 -0400] "GET /pub/job/vk/view20.jpg HTTP/1.0" 200 5070 sndaniel.reston.ingr.com - - [29/Aug/1995:11:56:58 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 14969 leithaus.leitess.com - - [29/Aug/1995:12:02:04 -0400] "GET /graphics/winerack.gif HTTP/1.0" 200 21475 136.181.50.23 - - [29/Aug/1995:12:06:10 -0400] "GET /pub/job/vk/view28.jpg HTTP/1.0" 200 4594 dresden.bmc.com - - [29/Aug/1995:12:11:18 -0400] "GET /pub/alweiner/backgrounds/brushedy.gif HTTP/1.0" 404 335 140.218.3.63 - - [29/Aug/1995:12:15:28 -0400] "GET /pub/job/swim/v-44.jpg HTTP/1.0" 200 2483 G7257.258.InterLink.NET - - [29/Aug/1995:12:19:14 -0400] "GET /pub/peace/note.gif HTTP/1.0" 200 565 l2-24.en.net - - [29/Aug/1995:12:22:44 -0400] "GET /pub/sshay/images/home6.gif HTTP/1.0" 200 58652 128.192.114.66 - - [29/Aug/1995:12:26:36 -0400] "GET /pub/jeffd/header.gif HTTP/1.0" 200 24376 www-b4.proxy.aol.com - - [29/Aug/1995:12:31:18 -0400] "GET /atomicbk/images/girlvsgir.gif HTTP/1.0" 200 28352 ix-sac5-02.ix.netcom.com - - [29/Aug/1995:12:35:18 -0400] "GET /pub/jeffd/smreagan.gif HTTP/1.0" 200 18018 rheo.matse.fukui-u.ac.jp - - [29/Aug/1995:12:38:45 -0400] "GET /atomicbk/catalog/adultcom.html HTTP/1.0" 200 8705 204.217.208.177 - - [29/Aug/1995:12:42:31 -0400] "GET /pub/howie/OO/home.gif HTTP/1.0" 200 1040 newsgw.mentorg.com - - [29/Aug/1995:12:46:12 -0400] "GET /pub/job/vk/view24.jpg HTTP/1.0" 200 6209 arajkuma.worldbank.org - - [29/Aug/1995:12:49:37 -0400] "GET /pub/alweiner/cgi-bin/homepage.html?game+0 HTTP/1.0" 200 2159 aria.softlab.is.tsukuba.ac.jp - - [29/Aug/1995:12:53:18 -0400] "GET /pub/sshay/images/galybutn.gif HTTP/1.0" 200 1080 charlotte.anu.edu.au - - [29/Aug/1995:12:56:28 -0400] "GET /pub/temonk/heavwww.htm HTTP/1.0" 200 2005 na1.dow.com - - [29/Aug/1995:13:00:18 -0400] "GET /pub/jeffd/95-61.gif HTTP/1.0" 200 31982 130.134.32.30 - - [29/Aug/1995:13:03:33 -0400] "GET /pub/job/theman.jpg HTTP/1.0" 200 8905 204.156.15.7 - - [29/Aug/1995:13:07:21 -0400] "GET /pub/lemat/i/excom3.gif HTTP/1.0" 200 76371 206.25.44.45 - - [29/Aug/1995:13:10:55 -0400] "GET /pub/atomicbk/artgal.gif HTTP/1.0" 200 823 rheo.matse.fukui-u.ac.jp - - [29/Aug/1995:13:14:13 -0400] "GET /pub/atomicbk/images/forcedw.gif HTTP/1.0" 200 77512 cluster-61.cluster.brown.edu - - [29/Aug/1995:13:18:30 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 ad15-045.compuserve.com - - [29/Aug/1995:13:21:50 -0400] "GET /pub/rmharris/alldlrs/mw/60657bea.html HTTP/1.0" 200 2074 GATEKEEPER.HCC.COM - - [29/Aug/1995:13:25:53 -0400] "GET /pub/jeffd/grad_lin.gif HTTP/1.0" 200 3346 vccnw11.its.rpi.edu - - [29/Aug/1995:13:29:14 -0400] "GET /pub/job/swim/v-50.jpg HTTP/1.0" 200 3369 gatekeeper.mitre.org - - [29/Aug/1995:13:33:04 -0400] "GET /atomicbk/catalog/catalog.html HTTP/1.0" 200 4654 cmgraham.nexus.olemiss.edu - - [29/Aug/1995:13:37:14 -0400] "GET /pub/cardman/1curious.gif HTTP/1.0" 200 1281 voliver.adp.iastate.edu - - [29/Aug/1995:13:41:01 -0400] "GET /pub/job/vk/v-sig.gif HTTP/1.0" 200 4864 nd015181.global.medtronic.COM - - [29/Aug/1995:13:44:47 -0400] "GET /pub/job/vk/vk-bk.jpg HTTP/1.0" 200 4569 gateway.amoco.com - - [29/Aug/1995:13:48:08 -0400] "GET /pub/atomicbk/catalog/mini.gif HTTP/1.0" 200 360 hpws92.hctg.SAIC.COM - - [29/Aug/1995:13:52:08 -0400] "GET /pub/jeffd/dole.gif HTTP/1.0" 200 4364 ts-h08-15-17.ucc.su.OZ.AU - - [29/Aug/1995:13:55:37 -0400] "GET /pub/jeffd/paper.gif HTTP/1.0" 200 20354 oirp-pc3.admin.umass.edu - - [29/Aug/1995:13:59:09 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 baloo.dc.luth.se - - [29/Aug/1995:14:03:21 -0400] "GET /pub/bell/review/bullet/grn_bullet_half.gif HTTP/1.0" 200 546 kmims.mindspring.com - - [29/Aug/1995:14:06:56 -0400] "GET /pub/alweiner/castle_w.gif HTTP/1.0" 200 1082 wwwproxy2.ca.sandia.gov - - [29/Aug/1995:14:10:46 -0400] "GET /pub/rjgula/os/windows/nasaccml.txt HTTP/1.0" 200 766 ba_bowers.pnl.gov - - [29/Aug/1995:14:13:59 -0400] "GET /pub/k2/am4x44u/maps/truck_stop.gif HTTP/1.0" 200 37573 euro.sunbee.co.kr - - [29/Aug/1995:14:17:06 -0400] "GET /pub/atomicbk/promo.gif HTTP/1.0" 200 849 theoris.rz.uni-konstanz.de - - [29/Aug/1995:14:20:22 -0400] "GET /atomicbk/seanc.gif HTTP/1.0" 200 734 webgate1.mot.com - - [29/Aug/1995:14:23:33 -0400] "GET /pub/k2/am4x44u/gifs/yellbut.gif HTTP/1.0" 200 1160 webgate1.mot.com - - [29/Aug/1995:14:27:44 -0400] "GET /pub/lschank/web/up.gif HTTP/1.0" 200 264 r-wilson.estates.gla.ac.uk - - [29/Aug/1995:14:31:28 -0400] "GET /pub/atomicbk/catalog/pinup.html HTTP/1.0" 304 - piweba1y.prodigy.com - - [29/Aug/1995:14:34:46 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 eblaese.stx.com - - [29/Aug/1995:14:37:53 -0400] "GET /pub/blaese/image/small_kids.gif HTTP/1.0" 304 - pc242.dev.esoc.esa.de - - [29/Aug/1995:14:41:07 -0400] "GET /pub/lupine/www/images/lines/tan.gif HTTP/1.0" 304 - netcom9.netcom.com - - [29/Aug/1995:14:43:55 -0400] "GET /pub/alweiner/work/_0231C.gif HTTP/1.0" 200 6232 ts-03.marin.k12.ca.us - - [29/Aug/1995:14:47:13 -0400] "GET /pub/sshay/images/blaksped.gif HTTP/1.0" 200 3681 155.229.78.26 - - [29/Aug/1995:14:49:35 -0400] "GET /graphics/b.gif HTTP/1.0" 200 1145 ts-03.marin.k12.ca.us - - [29/Aug/1995:14:54:00 -0400] "GET /pub/sshay/images/tom1.jpg HTTP/1.0" 200 21336 network.clark.net - - [29/Aug/1995:14:58:24 -0400] "GET /network/graphics/ypaper.gif HTTP/1.0" 200 10782 141.211.73.171 - - [29/Aug/1995:15:01:41 -0400] "GET /atomicbk/images/buffy.gif HTTP/1.0" 200 59956 modem014.offcampus.calpoly.edu - - [29/Aug/1995:15:05:34 -0400] "GET /pub/k2/am4x44u/gifs/scut.gif HTTP/1.0" 200 1404 131.218.10.40 - - [29/Aug/1995:15:09:05 -0400] "GET /pub/downin/html/Graphics/mac.gif HTTP/1.0" 200 143 agh008h-1.forestry.okstate.edu - - [29/Aug/1995:15:12:23 -0400] "GET /pub/peace/PeaceCorps.html HTTP/1.0" 304 - trout.os.varian.com - - [29/Aug/1995:15:16:57 -0400] "GET /pub/lschank/web/myguides.html HTTP/1.0" 200 9435 st0857.infonet.tufts.edu - - [29/Aug/1995:15:20:21 -0400] "GET /pub/atomicbk/catalog/orders.gif HTTP/1.0" 200 800 annex-ariel.libraries.psu.edu - - [29/Aug/1995:15:24:04 -0400] "GET /pub/wmcbrine/html/Madonna.html HTTP/1.0" 200 13440 pc7069.corp.dialog.com - - [29/Aug/1995:15:28:25 -0400] "GET /pub/alweiner/reddot.gif HTTP/1.0" 200 886 nagy.cs.Colorado.EDU - - [29/Aug/1995:15:32:09 -0400] "GET /pub/k2/jeep/jxj.htm HTTP/1.0" 304 - 134.243.230.103 - - [29/Aug/1995:15:36:42 -0400] "GET /pub/job/vk/view05.jpg HTTP/1.0" 200 5012 gatekeeper.mitre.org - - [29/Aug/1995:15:39:21 -0400] "GET /atomicbk/catalog/wimmens.html HTTP/1.0" 200 2612 Mudd169-PC2.ACNS.Carleton.edu - - [29/Aug/1995:15:42:18 -0400] "GET /pub/job/vk/view14.jpg HTTP/1.0" 200 4733 pfizergate.pfizer.com - - [29/Aug/1995:15:45:56 -0400] "GET /pub/job/vk/view23.jpg HTTP/1.0" 200 5165 gatekeeper.us.oracle.com - - [29/Aug/1995:15:49:23 -0400] "GET /atomicbk/contest.gif HTTP/1.0" 200 669 mty0108.infosel.com.mx - - [29/Aug/1995:15:52:15 -0400] "GET /pub/k2/am4x44u/maps/truck_stop.gif HTTP/1.0" 200 37573 cs500-9.sl013.cns.vt.edu - - [29/Aug/1995:15:55:26 -0400] "GET /pub/job/vk/flowers1.gif HTTP/1.0" 200 4288 147.234.11.8 - - [29/Aug/1995:15:58:39 -0400] "GET /atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 mustang.pica.army.mil - - [29/Aug/1995:16:01:52 -0400] "GET /pub/job/vk/view15.jpg HTTP/1.0" 200 6036 na3.dow.com - - [29/Aug/1995:16:05:27 -0400] "GET /pub/job/swim/v-59.jpg HTTP/1.0" 200 2947 wavopt431.tbe.com - - [29/Aug/1995:16:09:15 -0400] "GET /pub/job/vk/view30.jpg HTTP/1.0" 200 5991 mcga03.med.nyu.edu - - [29/Aug/1995:16:12:02 -0400] "GET /pub/sshay/images/partbutn.gif HTTP/1.0" 200 1085 137.57.10.239 - - [29/Aug/1995:16:14:48 -0400] "GET /theme/vrml/media/hr.gif HTTP/1.0" 404 313 136.204.3.33 - - [29/Aug/1995:16:17:51 -0400] "GET /atomicbk/bizlink.gif HTTP/1.0" 200 726 langmuir.engin.umich.edu - - [29/Aug/1995:16:21:05 -0400] "GET /pub/job/vk/view16.jpg HTTP/1.0" 200 6808 198.77.113.40 - - [29/Aug/1995:16:24:30 -0400] "GET /pub/mglamb/ranch/gungifs/paraord.gif HTTP/1.0" 200 4478 fas.clark.net - - [29/Aug/1995:16:27:30 -0400] "GET /pub/gen/mswg/stealth/sky.jpg HTTP/1.0" 200 1044 proxy.bellatlantic.com - - [29/Aug/1995:16:30:55 -0400] "GET /pub/downin/html/arlonet/icons/rbr.GIF HTTP/1.0" 200 1634 SOL08.AMS.ORG - - [29/Aug/1995:16:33:07 -0400] "GET /pub/keating/ri-hs.html HTTP/1.0" 200 23932 sndaniel.reston.ingr.com - - [29/Aug/1995:16:36:18 -0400] "GET /pub/job/vk/view07.jpg HTTP/1.0" 200 7057 rorke.com - - [29/Aug/1995:16:40:27 -0400] "GET /pub/rjgula/rongula.gif HTTP/1.0" 200 22150 keyhole.es.dupont.com - - [29/Aug/1995:16:44:29 -0400] "GET /pub/jeffd/grayball.gif HTTP/1.0" 304 - halsun.umsl.edu - - [29/Aug/1995:16:47:38 -0400] "GET /pub/listserv/new.gif HTTP/1.0" 200 911 l9.cslab.unf.edu - - [29/Aug/1995:16:50:59 -0400] "GET /pub/sshay/images/tywetsut.jpg HTTP/1.0" 200 21652 ddd.tel.kutztown.edu - - [29/Aug/1995:16:54:33 -0400] "GET /pub/atomicbk/email.gif HTTP/1.0" 200 756 ra.isy.liu.se - - [29/Aug/1995:16:57:51 -0400] "GET /pub/job/vk/babe.jpg HTTP/1.0" 200 8235 192.127.151.13 - - [29/Aug/1995:17:01:25 -0400] "GET /html/clarknet.html HTTP/1.0" 304 - 20.4.92.52 - - [29/Aug/1995:17:05:12 -0400] "GET /pub/reichera/rcp.html HTTP/1.0" 200 5522 hyh207.phys.andrews.edu - - [29/Aug/1995:17:08:33 -0400] "GET /pub/job/vk/view18.jpg HTTP/1.0" 200 6578 nbdclab-pc02.unomaha.edu - - [29/Aug/1995:17:12:18 -0400] "GET /pub/jeffd/grayball.gif HTTP/1.0" 304 - nagik.cs.colorado.edu - - [29/Aug/1995:17:15:41 -0400] "GET /pub/k2/am4x44u/gifs/jxj.gif HTTP/1.0" 304 - Atrium-PC15.INRE.ASU.EDU - - [29/Aug/1995:17:18:50 -0400] "GET /pub/k2/am4x44u/gifs/greenball.gif HTTP/1.0" 200 326 fuller.lance.colostate.edu - - [29/Aug/1995:17:21:56 -0400] "GET /pub/k2/am4x44u/gifs/netscape.gif HTTP/1.0" 200 1050 ppp077-stdkn2.ulaval.ca - - [29/Aug/1995:17:24:56 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 www-b5.proxy.aol.com - - [29/Aug/1995:17:28:56 -0400] "GET /pub/wick/photos/chair_mini.gif HTTP/1.0" 200 12589 xyplex3-5-2.ucs.indiana.edu - - [29/Aug/1995:17:32:12 -0400] "GET /pub/sshay/interact.html HTTP/1.0" 200 2732 e012.unitedway.org - - [29/Aug/1995:17:36:12 -0400] "GET /pub/pwalker/Internet_in_General/Catalogues_of_Resources/ HTTP/1.0" 200 6616 kbt27.library.yale.edu - - [29/Aug/1995:17:40:06 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 14969 155.82.105.107 - - [29/Aug/1995:17:44:06 -0400] "GET /pub/atomicbk/new/emboss.jpg HTTP/1.0" 200 4741 164.106.121.146 - - [29/Aug/1995:17:48:23 -0400] "GET /pub/atomicbk/iamges/erotuniv.gif HTTP/1.0" 404 335 ath-gw7.hol.gr - - [29/Aug/1995:17:52:10 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 U104.ucs.indiana.edu - - [29/Aug/1995:17:56:21 -0400] "GET /pub/jeffd/bg2.gif HTTP/1.0" 200 1343 gorbadoc.leeds.ac.uk - - [29/Aug/1995:17:59:55 -0400] "GET /pub/job/vk/view01.jpg HTTP/1.0" 200 5733 www-ea.proxy.aol.com - - [29/Aug/1995:18:03:08 -0400] "GET /pub/wsg/html/jan/ HTTP/1.0" 200 476 ac079.du.pipex.com - - [29/Aug/1995:18:07:28 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 dialup10.Bloomington.mci.net - - [29/Aug/1995:18:11:39 -0400] "GET /pub/jeffd/smreagan.gif HTTP/1.0" 200 18018 asd01-21.dial.xs4all.nl - - [29/Aug/1995:18:15:35 -0400] "GET /pub/networx/autopage/dealers/de001_3b.gif HTTP/1.0" 200 70257 dal09-25.ppp.iadfw.net - - [29/Aug/1995:18:20:00 -0400] "GET /pub/wick/misc/newsign.gif HTTP/1.0" 200 20279 webgate1.mot.com - - [29/Aug/1995:18:23:29 -0400] "GET /pub/job/vk/view03.jpg HTTP/1.0" 200 8506 smartwall.acuson.com - - [29/Aug/1995:18:26:48 -0400] "GET /pub/job/swim/v-56.jpg HTTP/1.0" 304 - 142.36.134.52 - - [29/Aug/1995:18:31:44 -0400] "GET /pub/abaa-booknet/images/at_work.gif HTTP/1.0" 200 244 n73h003.nlnet.nf.ca - - [29/Aug/1995:18:35:57 -0400] "GET /pub/atomicbk/direct.gif HTTP/1.0" 200 833 128.196.127.24 - - [29/Aug/1995:18:40:34 -0400] "GET /pub/jeffd/header.gif HTTP/1.0" 200 24376 138.92.11.26 - - [29/Aug/1995:18:44:59 -0400] "GET /pub/jgbustam/gifs/coloico2.gif HTTP/1.0" 200 4298 reapc.b20.ingr.com - - [29/Aug/1995:18:49:20 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 14969 martini-1.ics.Hawaii.Edu - - [29/Aug/1995:18:53:42 -0400] "GET /pub/BlackMagic/ses/snake.gif HTTP/1.0" 200 51324 smcvax.smcvt.edu - - [29/Aug/1995:18:58:09 -0400] "GET / HTTP/1.0" 200 1834 noir.kpnw.org - - [29/Aug/1995:19:02:17 -0400] "GET /home/oncrhome.gif HTTP/1.0" 200 12126 MCNPPP157.MCN.NET - - [29/Aug/1995:19:05:48 -0400] "GET /pub/job/vk/view36.jpg HTTP/1.0" 200 3522 156.39.124.71 - - [29/Aug/1995:19:09:06 -0400] "GET /pub/bell/review/image/5percsmt.gif HTTP/1.0" 200 2321 128.180.113.1 - - [29/Aug/1995:19:12:46 -0400] "GET /pub/atomicbk/kathman/nra5.gif HTTP/1.0" 200 10127 frogfoot.dsto.defence.gov.au - - [29/Aug/1995:19:17:06 -0400] "GET /pub/rmharris/catalogs.html HTTP/1.0" 200 6164 www-tdo.lanl.gov - - [29/Aug/1995:19:21:19 -0400] "GET /pub/wine/spinbt.gif HTTP/1.0" 200 1050 tlaloc.nb.com - - [29/Aug/1995:19:26:11 -0400] "GET /pub/job/swim/v-52.jpg HTTP/1.0" 200 2886 h_adherent.roanoke.infi.net - - [29/Aug/1995:19:30:48 -0400] "GET /pub/k2/am4x44u/gifs/bggrey.jpg HTTP/1.0" 304 - mn00170w-726419.mcnutt.indiana.edu - - [29/Aug/1995:19:34:51 -0400] "GET /pub/jeffd/paper.gif HTTP/1.0" 200 20354 alweiner.clark.net - - [29/Aug/1995:19:39:18 -0400] "GET /pub/alweiner/cgi-bin/squiggle.cgi?w3magic1 HTTP/1.0" 200 4467 kebab.infomatch.com - - [29/Aug/1995:19:45:09 -0400] "GET /pub/sshay/images/tywetsut.jpg HTTP/1.0" 200 21652 dd19-045.compuserve.com - - [29/Aug/1995:19:50:27 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 gate.tv2.no - - [29/Aug/1995:19:55:17 -0400] "GET /pub/job/vk/view02.jpg HTTP/1.0" 200 7290 slip4.pixel.com.mx - - [29/Aug/1995:20:00:18 -0400] "GET /pub/fervor/bob3.gif HTTP/1.0" 200 20679 cragateway.cra.com.au - - [29/Aug/1995:20:05:53 -0400] "GET /pub/job/vk/view32.jpg HTTP/1.0" 200 4852 200.0.156.15 - - [29/Aug/1995:20:11:43 -0400] "GET /pub/atomicbk/catalog/sleazmag.html HTTP/1.0" 200 6688 128.187.42.21 - - [29/Aug/1995:20:16:24 -0400] "GET /pub/mbrophy/pix/weh.jpg HTTP/1.0" 200 3045 pinole.os.varian.com - - [29/Aug/1995:20:20:40 -0400] "GET /pub/job/vk/view18.jpg HTTP/1.0" 200 6578 204.73.81.71 - - [29/Aug/1995:20:27:11 -0400] "GET /pub/k2/am4x44u/gifs/superwin.gif HTTP/1.0" 200 2263 159.142.95.163 - - [29/Aug/1995:20:32:06 -0400] "GET /pub/lupine/www/images/icons/news.gif HTTP/1.0" 200 1202 pdxgp1.intel.com - - [29/Aug/1995:20:37:58 -0400] "GET /pub/job/vk/view03.jpg HTTP/1.0" 200 8506 proxy.austin.ibm.com - - [29/Aug/1995:20:42:20 -0400] "GET /pub/job/swim/v-53.jpg HTTP/1.0" 200 2421 slip2.cafe.net - - [29/Aug/1995:20:48:42 -0400] "GET /pub/job/vk/vk-bk.jpg HTTP/1.0" 200 4569 192.84.112.145 - - [29/Aug/1995:20:53:29 -0400] "GET /pub/atomicbk/catalog/new.gif HTTP/1.0" 200 744 www-d3.proxy.aol.com - - [29/Aug/1995:20:59:29 -0400] "GET /pub/journalism/awesome.html HTTP/1.0" 304 - 160.227.101.63 - - [29/Aug/1995:21:03:32 -0400] "GET /pub/epchurch/bottom_2.gif HTTP/1.0" 200 7224 net181.metronet.com - - [29/Aug/1995:21:08:33 -0400] "GET /pub/lupine/www/images/support/sig.jpg HTTP/1.0" 200 17480 GATEWAY-2.WTI.COM - - [29/Aug/1995:21:14:41 -0400] "GET /pub/jeffd/header.gif HTTP/1.0" 200 24376 piweba5y.prodigy.com - - [29/Aug/1995:21:20:44 -0400] "GET /pub/atomicbk/images/nataliec.gif HTTP/1.0" 404 207 poppy.hensa.ac.uk - - [29/Aug/1995:21:26:14 -0400] "GET /atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 www-a1.proxy.aol.com - - [29/Aug/1995:21:30:40 -0400] "GET /pub/wick/misc/queer.gif HTTP/1.0" 200 1015 152.97.1.118 - - [29/Aug/1995:21:36:34 -0400] "GET /pub/job/vk/view15.jpg HTTP/1.0" 200 6036 NATHALIE.remote.ualberta.ca - - [29/Aug/1995:21:42:51 -0400] "GET /pub/mbrophy/pix/address.jpg HTTP/1.0" 200 2956 marion3.midwest.net - - [29/Aug/1995:21:47:28 -0400] "GET /pub/theme/demockracy/ HTTP/1.0" 200 1378 pelican.ucsd.edu - - [29/Aug/1995:21:53:11 -0400] "GET /pub/job/vk/claudia.jpg HTTP/1.0" 200 7225 lcc_g50.lanecc.edu - - [29/Aug/1995:21:59:23 -0400] "GET /pub/chivato/littleprog.gif HTTP/1.0" 200 973 onet2.cup.hp.com - - [29/Aug/1995:22:04:58 -0400] "GET /pub/job/vk/view06.jpg HTTP/1.0" 200 6553 csscdrom.css.beckman.com - - [29/Aug/1995:22:10:11 -0400] "GET /pub/job/vk/2-page-2.gif HTTP/1.0" 200 2527 ix-den11-27.ix.netcom.com - - [29/Aug/1995:22:17:01 -0400] "GET /pub/micros/rmi/listen.gif HTTP/1.0" 200 353 ix-bal1-23.ix.netcom.com - - [29/Aug/1995:22:23:07 -0400] "GET /pub/jeffd/grayball.gif HTTP/1.0" 200 995 cclamp.Colorado.EDU - - [29/Aug/1995:22:29:57 -0400] "GET /pub/mpowers/j4j/web/Tablegifs/LibraryIcon.gif HTTP/1.0" 200 6914 sanantonio-1-14.i-link.net - - [29/Aug/1995:22:34:22 -0400] "GET /pub/networx/autopage/autopage.gif HTTP/1.0" 200 57900 128.18.44.62 - - [29/Aug/1995:22:40:02 -0400] "GET /pub/job/swim/v-46.jpg HTTP/1.0" 200 2439 ix-den6-28.ix.netcom.com - - [29/Aug/1995:22:46:26 -0400] "GET /atomicbk/catalog/catalog.html HTTP/1.0" 200 4654 159.57.25.49 - - [29/Aug/1995:22:51:43 -0400] "GET /atomicbk/catalog/mini.gif HTTP/1.0" 200 360 163.226.7.130 - - [29/Aug/1995:22:57:20 -0400] "GET /pub/job/vk/view03.jpg HTTP/1.0" 200 8506 phyun5.ucr.edu - - [29/Aug/1995:23:02:45 -0400] "GET /pub/job/vk/view03.jpg HTTP/1.0" 200 8506 link103.txdirect.net - - [29/Aug/1995:23:08:27 -0400] "GET /pub/lupine/www/werewolf.html HTTP/1.0" 200 1041 elsun156.cc.purdue.edu - - [29/Aug/1995:23:13:00 -0400] "GET /pub/job/vk/arrow-r.gif HTTP/1.0" 200 3042 129.130.211.102 - - [29/Aug/1995:23:18:08 -0400] "GET /pub/sshay/images/links.gif HTTP/1.0" 304 - www-c3.proxy.aol.com - - [29/Aug/1995:23:23:04 -0400] "GET /pub/journalism/awesome.html HTTP/1.0" 304 - petrides-ppp.clark.net - - [29/Aug/1995:23:28:19 -0400] "GET /pub/TMC/image/globe.gif HTTP/1.0" 304 - openlab99.mty.itesm.mx - - [29/Aug/1995:23:34:55 -0400] "GET /pub/job/vk/view07.jpg HTTP/1.0" 200 7057 snoopy.vlsi.com - - [29/Aug/1995:23:40:38 -0400] "GET /pub/job/vk/view33.jpg HTTP/1.0" 200 5903 h-langoliers.norfolk.infi.net - - [29/Aug/1995:23:47:10 -0400] "GET /pub/bdalzell/borzoiinfo.html HTTP/1.0" 200 1659 mon2-07.planete.net - - [29/Aug/1995:23:54:21 -0400] "GET /pub/alweiner/cgi-bin/homepage.html?w3magic12 HTTP/1.0" 200 1396 belize.lrcc.vichosp.london.on.ca - - [30/Aug/1995:00:02:10 -0400] "GET /atomicbk/catalog/sleazmag.html HTTP/1.0" 200 6688 ppp7.cm.worldlinx.com - - [30/Aug/1995:00:10:06 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 200 813 usr16-dialup51.Atlanta.mci.net - - [30/Aug/1995:00:15:02 -0400] "GET /pub/atomicbk/images/tease.gif HTTP/1.0" 200 43516 ts9-11.westwood.ts.ucla.edu - - [30/Aug/1995:00:21:40 -0400] "GET /pub/terra/pics/purpleball.gif HTTP/1.0" 200 322 amanda.dorsai.org - - [30/Aug/1995:00:27:47 -0400] "GET /pub/ceallach/bill.gif HTTP/1.0" 200 7886 www-b3.proxy.aol.com - - [30/Aug/1995:00:33:47 -0400] "GET /pub/networx/autopage/autoicon.gif HTTP/1.0" 304 - ppp0e-05.rns.tamu.edu - - [30/Aug/1995:00:39:40 -0400] "GET /pub/job/vk/view09.jpg HTTP/1.0" 200 6011 bugfix.ikos2.iao.fhg.de - - [30/Aug/1995:00:46:04 -0400] "GET /pub/mbrophy/pix/yahoo.jpg HTTP/1.0" 200 7570 www-a1.proxy.aol.com - - [30/Aug/1995:00:52:39 -0400] "GET /pub/atomicbk/catalog/orders.gif HTTP/1.0" 200 800 ppp31.cac.psu.edu - - [30/Aug/1995:01:00:32 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 304 - www-c9.proxy.aol.com - - [30/Aug/1995:01:08:23 -0400] "GET /pub/sknightly/babash.html HTTP/1.0" 404 207 ppp020.st.rim.or.jp - - [30/Aug/1995:01:14:07 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 n00026-103sul.unity.ncsu.edu - - [30/Aug/1995:01:21:16 -0400] "GET /pub/job/vk/view04.jpg HTTP/1.0" 200 4847 192.121.54.163 - - [30/Aug/1995:01:27:42 -0400] "GET /pub/job/swim/swim.jpg HTTP/1.0" 304 - romulus.ultranet.com - - [30/Aug/1995:01:34:58 -0400] "GET /pub/job/vk/page1.gif HTTP/1.0" 200 1960 dana.lw.com - - [30/Aug/1995:01:44:52 -0400] "GET /pub/job/vk/view22.jpg HTTP/1.0" 200 6281 quandong.itd.adelaide.edu.au - - [30/Aug/1995:01:51:12 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 9084 www-c9.proxy.aol.com - - [30/Aug/1995:02:00:52 -0400] "GET /pub/networx/autopage/classic/cl003_1s.gif HTTP/1.0" 200 16250 www-c8.proxy.aol.com - - [30/Aug/1995:02:08:27 -0400] "GET /pub/sshay/images/peley.gif HTTP/1.0" 200 31745 montana-max.inoc.dl.nec.com - - [30/Aug/1995:02:17:57 -0400] "GET /pub/job/vk/view05.jpg HTTP/1.0" 304 - ip061.lax.primenet.com - - [30/Aug/1995:02:26:45 -0400] "GET /pub/heroes/ HTTP/1.0" 200 12396 as2511-1.sl004.cns.vt.edu - - [30/Aug/1995:02:33:51 -0400] "GET /pub/lori/friends.html HTTP/1.0" 200 1049 204.97.215.103 - - [30/Aug/1995:02:44:33 -0400] "GET /pub/job/vk/view23.jpg HTTP/1.0" 304 - netcom16.netcom.com - - [30/Aug/1995:02:54:12 -0400] "GET /pub/job/vk/view08.jpg HTTP/1.0" 200 6476 keyhole.es.dupont.com - - [30/Aug/1995:03:03:11 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 200 813 ucxy08_03.slip.uc.edu - - [30/Aug/1995:03:15:23 -0400] "GET /pub/atomicbk/bobk.gif HTTP/1.0" 200 816 jentzsch-slip.usa.net - - [30/Aug/1995:03:23:34 -0400] "GET /pub/jeffd/header.gif HTTP/1.0" 200 24376 proxy0.research.att.com - - [30/Aug/1995:03:33:42 -0400] "GET /pub/ecn/fastfruit.gif HTTP/1.0" 200 115 Cust37.Max3.Orlando.FL.MS.UU.NET - - [30/Aug/1995:03:46:26 -0400] "GET /atomicbk/logo2.gif HTTP/1.0" 200 12871 puppydog.clark.net - - [30/Aug/1995:04:00:39 -0400] "GET /pub/rant/rant/text1b.gif HTTP/1.0" 304 - mailhost.cdn.fr - - [30/Aug/1995:04:11:22 -0400] "GET /pub/atomicbk/images/earlyero.gif HTTP/1.0" 404 335 161.98.20.106 - - [30/Aug/1995:04:23:18 -0400] "GET /pub/sshay/images/crthumb6.jpg HTTP/1.0" 200 5733 itasempara.mes.co.jp - - [30/Aug/1995:04:36:29 -0400] "GET /mc-icons/blank.gif HTTP/1.0" 200 60 hwins.uia.ac.be - - [30/Aug/1995:04:45:59 -0400] "GET /pub/atomicbk/catalog/new.gif HTTP/1.0" 200 744 beta.Xerox.COM - - [30/Aug/1995:04:58:45 -0400] "GET /pub/job/vk/view02.jpg HTTP/1.0" 200 7290 z008.euronet.nl - - [30/Aug/1995:05:08:35 -0400] "GET /atomicbk/orders.gif HTTP/1.0" 200 800 merc23.calon.com - - [30/Aug/1995:05:19:43 -0400] "GET /pub/sshay/images/scotname.gif HTTP/1.0" 200 27064 bam.nuri.net - - [30/Aug/1995:05:30:13 -0400] "GET /pub/sshay/images/scotface.gif HTTP/1.0" 200 12100 heather.eee.strath.ac.uk - - [30/Aug/1995:05:44:41 -0400] "GET /atomicbk/catalog/orders.gif HTTP/1.0" 200 800 pmekisic.mikom.csir.co.za - - [30/Aug/1995:05:59:01 -0400] "GET /pub/k2/am4x44u/gifs/ambut.gif HTTP/1.0" 304 - hagi-76.kuentos.guam.net - - [30/Aug/1995:06:14:17 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 stoossz.ethz.ch - - [30/Aug/1995:06:28:16 -0400] "GET /pub/job/vk/view22.jpg HTTP/1.0" 200 6281 ecker.vir.univie.ac.at - - [30/Aug/1995:06:39:41 -0400] "GET /pub/job/vk/view05.jpg HTTP/1.0" 200 5012 van06171.direct.ca - - [30/Aug/1995:06:51:26 -0400] "GET /pub/wick/misc/redbg.gif HTTP/1.0" 200 878 RJEPC.MSFC.NASA.GOV - - [30/Aug/1995:07:02:14 -0400] "GET /pub/howie/OO/E.GIF HTTP/1.0" 200 977 bobbmac.base.bellcore.com - - [30/Aug/1995:07:14:00 -0400] "GET /pub/job/vk/view03.jpg HTTP/1.0" 200 8506 158.39.79.122 - - [30/Aug/1995:07:24:04 -0400] "GET /pub/networx/autopage/automisc.gif HTTP/1.0" 200 47927 lbb22.HUB.ofthe.NET - - [30/Aug/1995:07:33:10 -0400] "GET /atomicbk/catalog/home.gif HTTP/1.0" 200 813 192.176.149.18 - - [30/Aug/1995:07:40:49 -0400] "GET /pub/atomicbk/images/blondes.gif HTTP/1.0" 200 21173 setup-1.math.ethz.ch - - [30/Aug/1995:07:47:31 -0400] "GET /pub/job/swim/v-42.jpg HTTP/1.0" 304 - 202.44.216.20 - - [30/Aug/1995:07:55:16 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 sgigate.SGI.COM - - [30/Aug/1995:08:03:04 -0400] "GET /pub/conquest/one/contents.gif HTTP/1.0" 200 39330 165.90.141.20 - - [30/Aug/1995:08:12:05 -0400] "GET /atomicbk/new/email.gif HTTP/1.0" 200 756 pc0578.iapc.bbsrc.ac.uk - - [30/Aug/1995:08:18:05 -0400] "GET /atomicbk/seanc.gif HTTP/1.0" 200 734 t345.chem.tue.nl - - [30/Aug/1995:08:26:34 -0400] "GET /pub/jumpsam/minilogo.gif HTTP/1.0" 200 1368 res.WPI.EDU - - [30/Aug/1995:08:33:57 -0400] "GET /pub/kevina/sl/ HTTP/1.0" 200 10536 ip5.dialup.paltech.com - - [30/Aug/1995:08:41:43 -0400] "GET /pub/cargui/catapul2.gif HTTP/1.0" 200 28411 ix-akr-oh2-05.ix.netcom.com - - [30/Aug/1995:08:47:59 -0400] "GET /pub/aztec/mariner/ HTTP/1.0" 200 1993 agora.carleton.ca - - [30/Aug/1995:08:53:18 -0400] "GET /mpt/mdstate.gif HTTP/1.0" 200 7505 203.2.83.121 - - [30/Aug/1995:08:58:34 -0400] "GET /pub/job/vk/view10.jpg HTTP/1.0" 200 5303 158.22.69.109 - - [30/Aug/1995:09:03:51 -0400] "GET /pub/job/vk/v-sig.gif HTTP/1.0" 200 4864 datw235.seinf.abb.se - - [30/Aug/1995:09:09:37 -0400] "GET /pub/job/vk/view14.jpg HTTP/1.0" 200 4733 198.53.235.136 - - [30/Aug/1995:09:14:51 -0400] "GET /pub/wine/purch.html HTTP/1.0" 200 1643 ekm116.jsc.nasa.gov - - [30/Aug/1995:09:21:53 -0400] "GET /pub/job/vendela.jpg HTTP/1.0" 200 36746 www-d4.proxy.aol.com - - [30/Aug/1995:09:27:10 -0400] "GET /pub/networks/l.gif HTTP/1.0" 200 1097 keyring.aecom.yu.edu - - [30/Aug/1995:09:32:31 -0400] "GET /pub/job/vk/kathy.jpg HTTP/1.0" 200 7450 131.218.10.40 - - [30/Aug/1995:09:37:06 -0400] "GET /pub/jwolff/home/bluedot.gif HTTP/1.0" 200 153 128.253.75.157 - - [30/Aug/1995:09:41:27 -0400] "GET /pub/job/vk/arrow-r.gif HTTP/1.0" 200 3042 gate.bmgmusic.com - - [30/Aug/1995:09:47:25 -0400] "GET /pub/lschank/web/note.gif HTTP/1.0" 200 264 www-d1.proxy.aol.com - - [30/Aug/1995:09:54:58 -0400] "GET /pub/journalism/awesome.html HTTP/1.0" 200 11610 nephthys.egr.uri.edu - - [30/Aug/1995:10:00:13 -0400] "GET /pub/job/vk/view32.jpg HTTP/1.0" 200 4852 199.79.206.35 - - [30/Aug/1995:10:04:11 -0400] "GET /atomicbk/logo2.gif HTTP/1.0" 200 12871 147.138.100.25 - - [30/Aug/1995:10:08:34 -0400] "GET /pub/jeffd/wall-6.html HTTP/1.0" 404 323 165.127.98.68 - - [30/Aug/1995:10:13:56 -0400] "GET /pub/TMC/image/euromideast.gif HTTP/1.0" 200 2221 196.3.72.64 - - [30/Aug/1995:10:18:03 -0400] "GET /pub/pribut/idea.gif HTTP/1.0" 200 909 dialup-3-241.gw.umn.edu - - [30/Aug/1995:10:22:35 -0400] "GET /pub/atomicbk/catalog/new.gif HTTP/1.0" 200 744 p40.euronet.nl - - [30/Aug/1995:10:28:06 -0400] "GET /pub/russadam/share.html HTTP/1.0" 200 8154 130.207.65.13 - - [30/Aug/1995:10:32:25 -0400] "GET /pub/k2/cgi-bin/imagemap/events?252,171 HTTP/1.0" 302 107 192.190.9.56 - - [30/Aug/1995:10:36:48 -0400] "GET /pub/global/search.html HTTP/1.0" 200 6602 www-c9.proxy.aol.com - - [30/Aug/1995:10:41:58 -0400] "GET /pub/atomicbk/images/tattoow.gif HTTP/1.0" 200 29519 bulwark.switch.com - - [30/Aug/1995:10:46:14 -0400] "GET /pub/jeffd/grad_lin.gif HTTP/1.0" 200 3346 novix.ncaur.gov - - [30/Aug/1995:10:51:14 -0400] "GET /pub/rmharris/images/note.gif HTTP/1.0" 200 565 gail.dialup.inch.com - - [30/Aug/1995:10:55:30 -0400] "GET /graphics/BUTTON1.GIF HTTP/1.0" 200 1434 nonpro.clark.net - - [30/Aug/1995:11:00:07 -0400] "GET /pub/stimson/stimson/sites.htm HTTP/1.0" 200 1370 WS63.CC.YSU.EDU - - [30/Aug/1995:11:05:35 -0400] "GET /pub/sshay/images/rainback.gif HTTP/1.0" 200 2541 npt1.netspace.or.jp - - [30/Aug/1995:11:10:33 -0400] "GET /pub/atomicbk/catalog/erotica.html HTTP/1.0" 200 11362 ppp121.moscow.com - - [30/Aug/1995:11:14:27 -0400] "GET /pub/alweiner/work/spacer10.gif HTTP/1.0" 304 - murray.cnwl.igs.net - - [30/Aug/1995:11:18:23 -0400] "GET /pub/lschank/web/exclamw.gif HTTP/1.0" 200 187 plc.clark.net - - [30/Aug/1995:11:22:38 -0400] "GET /pub/plc/return.gif HTTP/1.0" 304 - cary113.its.rpi.edu - - [30/Aug/1995:11:25:50 -0400] "GET /atomicbk/new/logo2.gif HTTP/1.0" 200 12871 www-c5.proxy.aol.com - - [30/Aug/1995:11:29:28 -0400] "GET /atomicbk/artgal.gif HTTP/1.0" 304 - www-b6.proxy.aol.com - - [30/Aug/1995:11:33:31 -0400] "GET /pub/jbl/apple_small.gif HTTP/1.0" 200 173 www-d1.proxy.aol.com - - [30/Aug/1995:11:37:58 -0400] "GET /pub/robert/current.html HTTP/1.0" 200 30337 crc2-fddi.cris.com - - [30/Aug/1995:11:42:23 -0400] "GET /pub/jrinker/news.jpg HTTP/1.0" 200 4471 204.245.159.8 - - [30/Aug/1995:11:46:46 -0400] "GET /pub/clw/clwblue2.gif HTTP/1.0" 304 - 131.116.19.243 - - [30/Aug/1995:11:51:25 -0400] "GET /pub/job/vk/view19.jpg HTTP/1.0" 200 3915 nyssa.swt.edu - - [30/Aug/1995:11:55:06 -0400] "GET /pub/job/mpegs/util.html HTTP/1.0" 200 8531 user11x48.lacoe.edu - - [30/Aug/1995:11:58:13 -0400] "GET /pub/atomicbk/catalog/mini.gif HTTP/1.0" 200 360 lo-pc2015.HITC.COM - - [30/Aug/1995:12:02:17 -0400] "GET /pub/atomicbk/images/fetdream.gif HTTP/1.0" 200 30850 dyna-09.bart.nl - - [30/Aug/1995:12:06:11 -0400] "GET /pub/lupine/www/images/support/mickey.gif HTTP/1.0" 304 - 129.219.49.55 - - [30/Aug/1995:12:10:23 -0400] "GET /pub/job/vk/view14.jpg HTTP/1.0" 200 4733 osorno.dim.uchile.cl - - [30/Aug/1995:12:13:34 -0400] "GET /pub/job/vk/page2.gif HTTP/1.0" 200 2056 box705.labs.cis.pitt.edu - - [30/Aug/1995:12:17:27 -0400] "GET /pub/ajc-dc/regions.html HTTP/1.0" 200 865 fapc68.far.intg.telia.se - - [30/Aug/1995:12:20:56 -0400] "GET /pub/job/swim/v-53.jpg HTTP/1.0" 200 2421 FPM-Eros.Stanford.EDU - - [30/Aug/1995:12:24:45 -0400] "GET /pub/network/jobs/01063095CPB2.html HTTP/1.0" 200 1057 Toucan.CS.UCLA.EDU - - [30/Aug/1995:12:28:23 -0400] "GET /pub/alweiner/jet_2.gif HTTP/1.0" 200 616 199.20.17.192 - - [30/Aug/1995:12:32:27 -0400] "GET /pub/job/vk/flowers1.gif HTTP/1.0" 200 4288 cirt_153.unm.edu - - [30/Aug/1995:12:35:53 -0400] "GET /pub/lupine/images/tlk/mufasa-simba.gif HTTP/1.0" 200 26330 claude.ifi.unizh.ch - - [30/Aug/1995:12:39:31 -0400] "GET /pub/alweiner/beta.gif HTTP/1.0" 200 294 lfsgate.lfs.loral.com - - [30/Aug/1995:12:43:25 -0400] "GET /pub/pgarrett/trash.gif HTTP/1.0" 200 248 dialup96-110.swipnet.se - - [30/Aug/1995:12:47:32 -0400] "GET /pub/job/vk/note.gif HTTP/1.0" 200 9413 161.187.201.138 - - [30/Aug/1995:12:50:55 -0400] "GET /pub/sshay/images/brwthumb.jpg HTTP/1.0" 200 3306 learn129.gestalt-sys.com - - [30/Aug/1995:12:54:51 -0400] "GET /pub/cargui/net_apps.html HTTP/1.0" 200 7735 halon.sybase.com - - [30/Aug/1995:12:58:46 -0400] "GET /pub/atomicbk/catalog/orders.gif HTTP/1.0" 200 800 pc116.imt.hist.no - - [30/Aug/1995:13:02:39 -0400] "GET /pub/job/swim/v-42.jpg HTTP/1.0" 200 2506 j11.kl2.jaring.my - - [30/Aug/1995:13:06:10 -0400] "GET /atomicbk/catalog/orders.gif HTTP/1.0" 200 800 halon.sybase.com - - [30/Aug/1995:13:10:40 -0400] "GET /pub/wick/misc/clarknet.gif HTTP/1.0" 200 3937 eagle.uis.edu - - [30/Aug/1995:13:14:21 -0400] "GET /pub/gen/ada/ HTTP/1.0" 200 3754 sundial.cs.cuhk.hk - - [30/Aug/1995:13:18:36 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 200 813 ad19-037.compuserve.com - - [30/Aug/1995:13:22:56 -0400] "GET /pub/sshay/links.html HTTP/1.0" 200 2992 mab-146.umd.edu - - [30/Aug/1995:13:26:31 -0400] "GET /pub/TMC/image/raiisaleading.gif HTTP/1.0" 200 2209 gasparrofp.med.yale.edu - - [30/Aug/1995:13:29:53 -0400] "GET /pub/sshay/images/brwthumb.jpg HTTP/1.0" 200 3306 ws16.csblab.uncwil.edu - - [30/Aug/1995:13:33:02 -0400] "GET /pub/lschank/web/up.gif HTTP/1.0" 200 264 inter014.internet.com.mx - - [30/Aug/1995:13:36:53 -0400] "GET /pub/job/swim/v-41.jpg HTTP/1.0" 200 2721 relay02.jpmorgan.com - - [30/Aug/1995:13:40:59 -0400] "GET /pub/job/swim/v-57.jpg HTTP/1.0" 200 2717 piweba3y.prodigy.com - - [30/Aug/1995:13:45:02 -0400] "GET /pub/job/vk/view18.jpg HTTP/1.0" 200 6578 comserv-b-24.usc.edu - - [30/Aug/1995:13:48:31 -0400] "GET /pub/wine/arrowrt.gif HTTP/1.0" 200 216 ssmtpva1.slma.com - - [30/Aug/1995:13:52:08 -0400] "GET /pub/artsci/boat.gif HTTP/1.0" 304 - 129.193.164.81 - - [30/Aug/1995:13:55:08 -0400] "GET /pub/govimag/overview.html HTTP/1.0" 200 1478 198.243.61.152 - - [30/Aug/1995:13:58:02 -0400] "GET /pub/k2/am4x44u/gifs/camtrol2.gif HTTP/1.0" 200 1865 synerget.demon.co.uk - - [30/Aug/1995:14:00:42 -0400] "GET /pub/howie/OO/ooheader.gif HTTP/1.0" 200 2631 ad05-009.compuserve.com - - [30/Aug/1995:14:03:14 -0400] "GET /pub/atomicbk/catalog.gif HTTP/1.0" 200 693 198.83.140.62 - - [30/Aug/1995:14:07:06 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 200 813 ix-al10-26.ix.netcom.com - - [30/Aug/1995:14:10:34 -0400] "GET /infouser/endidc.htm HTTP/1.0" 200 2146 bruny.cc.utas.edu.au - - [30/Aug/1995:14:13:50 -0400] "GET /pub/job/vk/view10.jpg HTTP/1.0" 200 5303 ix-wc1-21.ix.netcom.com - - [30/Aug/1995:14:16:58 -0400] "GET /pub/sshay/images/btthumb.jpg HTTP/1.0" 200 4624 146.186.69.35 - - [30/Aug/1995:14:20:42 -0400] "GET /pub/k2/am4x44u/maps/trails.gif HTTP/1.0" 200 41962 CPCD.cam.muskingum.edu - - [30/Aug/1995:14:24:07 -0400] "GET /pub/job/vk/v-line.gif HTTP/1.0" 200 1254 134.20.235.250 - - [30/Aug/1995:14:26:58 -0400] "GET /pub/job/vk/view02.jpg HTTP/1.0" 200 7290 bigtexan.clark.net - - [30/Aug/1995:14:30:10 -0400] "GET /pub/batman/hline.gif HTTP/1.0" 200 2424 slip1137.rmii.com - - [30/Aug/1995:14:33:16 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 200 813 rkln9.quiknet.com - - [30/Aug/1995:14:36:38 -0400] "GET /pub/k2/am4x44u/maps/trails.gif HTTP/1.0" 200 41962 lab44.fas.yorku.ca - - [30/Aug/1995:14:39:53 -0400] "GET /atomicbk/new/logo.gif HTTP/1.0" 200 1942 204.191.33.129 - - [30/Aug/1995:14:42:24 -0400] "GET /pub/cargui/autos.html HTTP/1.0" 200 7748 beta.Xerox.COM - - [30/Aug/1995:14:45:18 -0400] "GET /pub/bell/review/religion/miller_complete_gospels.shtml HTTP/1.0" 200 4309 E04GUSER.MNSFLD.EDU - - [30/Aug/1995:14:49:11 -0400] "GET /pub/lschank/web/line.gif HTTP/1.0" 200 1131 pizza.INnet.net - - [30/Aug/1995:14:52:45 -0400] "GET /pub/job/vk/view01.jpg HTTP/1.0" 200 5733 mpdgw2.hmpd.com - - [30/Aug/1995:14:56:25 -0400] "GET /graphics/cgi.gif HTTP/1.0" 200 11499 144.18.8.44 - - [30/Aug/1995:15:00:51 -0400] "GET /pub/k2/am4x44u/gifs/scut.gif HTTP/1.0" 200 1404 bootstrap.agcs.com - - [30/Aug/1995:15:04:32 -0400] "GET /pub/jeffd/spcornr.gif HTTP/1.0" 200 7584 sf-124.sfo.com - - [30/Aug/1995:15:07:11 -0400] "GET /pub/ibos/home.html HTTP/1.0" 200 2755 petey.bwi.wec.com - - [30/Aug/1995:15:10:26 -0400] "GET /pub/gen/fas/spp/ HTTP/1.0" 304 - 168.18.135.109 - - [30/Aug/1995:15:14:02 -0400] "GET /pub/atomicbk/catalog/orders.gif HTTP/1.0" 200 800 learn129.gestalt-sys.com - - [30/Aug/1995:15:17:43 -0400] "GET /pub/globe1.gif HTTP/1.0" 404 306 lkolker-ppp.clark.net - - [30/Aug/1995:15:21:07 -0400] "GET /html/usage/usage.graph.small.gif HTTP/1.0" 200 144 198.88.162.54 - - [30/Aug/1995:15:24:37 -0400] "GET /pub/job/vk/view16.jpg HTTP/1.0" 200 6808 grda.csi.uottawa.ca - - [30/Aug/1995:15:27:45 -0400] "GET /pub/job/vk/view10.jpg HTTP/1.0" 200 5303 MTM1IP.usia.gov - - [30/Aug/1995:15:31:03 -0400] "GET /pub/lschank/web/fwd.gif HTTP/1.0" 200 275 msabouri.gtis.gc.ca - - [30/Aug/1995:15:34:59 -0400] "GET /pub/job/swim/swimline.jpg HTTP/1.0" 200 2641 victoria.phar.cam.ac.uk - - [30/Aug/1995:15:39:04 -0400] "GET /pub/job/vk/view11.jpg HTTP/1.0" 200 5001 www-d4.proxy.aol.com - - [30/Aug/1995:15:41:29 -0400] "GET /atomicbk/images/cherry.gif HTTP/1.0" 304 - isr-m28-18.urh.uiuc.edu - - [30/Aug/1995:15:44:53 -0400] "GET /pub/lupine/www/pixs/backgrounds/beige.gif HTTP/1.0" 404 318 vaxb.isc.rit.edu - - [30/Aug/1995:15:48:45 -0400] "GET /pub/phil/JulianaHatfield/stories/jh16.txt HTTP/1.0" 200 5194 samc04.dne.bnl.gov - - [30/Aug/1995:15:52:06 -0400] "GET /pub/jeffd/cicgbtn.gif HTTP/1.0" 304 - yacht.ee.fit.edu - - [30/Aug/1995:15:55:19 -0400] "GET /pub/jcase/fspscan.tar.gz HTTP/1.0" 200 5337 204.180.189.28 - - [30/Aug/1995:15:58:48 -0400] "GET /pub/rjamesd/images/foxpro.gif HTTP/1.0" 304 - dd15-012.compuserve.com - - [30/Aug/1995:16:02:33 -0400] "GET /pub/jeffd/culture.html HTTP/1.0" 200 3934 mann.miracle.net - - [30/Aug/1995:16:06:05 -0400] "GET /atomicbk/catalog/emboss.jpg HTTP/1.0" 200 4741 a038.dayt.tasc.com - - [30/Aug/1995:16:10:11 -0400] "GET /pub/jeffd/cicgbtn.gif HTTP/1.0" 304 - ts22p14.NetVision.net.il - - [30/Aug/1995:16:13:27 -0400] "GET /pub/job/vk/vk-bk.jpg HTTP/1.0" 200 4569 rothplan.demon.co.uk - - [30/Aug/1995:16:16:35 -0400] "GET /atomicbk/images/atomgirl.jpg HTTP/1.0" 200 34164 access.rrd.com - - [30/Aug/1995:16:19:36 -0400] "GET /pub/atomicbk/catalog/orders.gif HTTP/1.0" 200 800 webgate1.mot.com - - [30/Aug/1995:16:22:45 -0400] "GET /pub/mpowers/j4j/web/Tablegifs/TorahIcon.gif HTTP/1.0" 200 2664 dh1.tt.umist.ac.uk - - [30/Aug/1995:16:26:09 -0400] "GET /pub/atomicbk/catalog/pinup.html HTTP/1.0" 200 8316 dialup97-106.swipnet.se - - [30/Aug/1995:16:29:14 -0400] "GET /atomicbk/catalog/orders.gif HTTP/1.0" 200 800 204.62.25.144 - - [30/Aug/1995:16:32:53 -0400] "GET /pub/rmharris/catalogs/bookwcat/282-5.html HTTP/1.0" 200 26018 zen.com - - [30/Aug/1995:16:36:13 -0400] "GET /pub/mmathai/images/halfmls.gif HTTP/1.0" 304 - inferno.dbna.com - - [30/Aug/1995:16:40:08 -0400] "GET /pub/jgbustam/toros/garrochi.gif HTTP/1.0" 200 5675 london.Informatik.Uni-Oldenburg.DE - - [30/Aug/1995:16:43:48 -0400] "GET /pub/pgarrett/cycle.gif HTTP/1.0" 200 108564 pth7.ioo.bpmf.ac.uk - - [30/Aug/1995:16:47:25 -0400] "GET /pub/atomicbk/catalog/orders.gif HTTP/1.0" 200 800 ip201-179.wiu.bgu.edu - - [30/Aug/1995:16:50:54 -0400] "GET /pub/job/vk/view15.jpg HTTP/1.0" 200 6036 142.139.235.178 - - [30/Aug/1995:16:54:37 -0400] "GET /pub/job/swim/v-57.jpg HTTP/1.0" 200 2717 rmiller.vip.best.com - - [30/Aug/1995:16:57:55 -0400] "GET /pub/atomicbk/catalog/new.gif HTTP/1.0" 200 744 kg.msc.edu - - [30/Aug/1995:17:02:14 -0400] "GET /pub/job/swim/v-46.jpg HTTP/1.0" 200 2439 astralb.clark.net - - [30/Aug/1995:17:05:47 -0400] "GET /pub/paulgame HTTP/1.0" 302 220 stonewall.sra.com - - [30/Aug/1995:17:09:45 -0400] "GET /atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 204.96.226.84 - - [30/Aug/1995:17:13:21 -0400] "GET /pub/job/vk/page2.gif HTTP/1.0" 200 2056 192.112.102.166 - - [30/Aug/1995:17:17:24 -0400] "GET /pub/rsjdfg/Pictures/Finale2.jpg HTTP/1.0" 200 36744 www-e1b.gnn.com - - [30/Aug/1995:17:20:55 -0400] "GET /pub/ibos/iback.gif HTTP/1.0" 200 9326 jd_jensen.pnl.gov - - [30/Aug/1995:17:24:29 -0400] "GET /pub/heroes/ HTTP/1.0" 200 12396 spectrum.xerox.com - - [30/Aug/1995:17:28:04 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 304 - hplabs.hpl.hp.com - - [30/Aug/1995:17:32:16 -0400] "GET /pub/job/vk/view01.jpg HTTP/1.0" 200 5733 128.97.52.52 - - [30/Aug/1995:17:37:11 -0400] "GET /pub/arthur/gamebank.html HTTP/1.0" 200 12910 onet2.cup.hp.com - - [30/Aug/1995:17:41:27 -0400] "GET /pub/job/vk/view17.jpg HTTP/1.0" 304 - pv041f.vincent.iastate.edu - - [30/Aug/1995:17:45:13 -0400] "GET /pub/job/vk/vk-ad-09.jpg HTTP/1.0" 200 70678 elp.div-cc.firn.edu - - [30/Aug/1995:17:49:02 -0400] "GET /pub/paulgame/paulface.gif HTTP/1.0" 200 13774 atomicbk-ppp.clark.net - - [30/Aug/1995:17:53:25 -0400] "GET /atomicbk/new/new.gif HTTP/1.0" 200 744 tec207ws30.st.usm.edu - - [30/Aug/1995:17:57:17 -0400] "GET /pub/sshay/images/crthumb5.jpg HTTP/1.0" 200 4495 lv-050.internext.com - - [30/Aug/1995:18:01:03 -0400] "GET /pub/atomicbk/catalog/erotica.html HTTP/1.0" 200 11362 interlock.lexmark.com - - [30/Aug/1995:18:03:48 -0400] "GET /pub/job/vk/2-page-2.gif HTTP/1.0" 200 2527 h-hanuman.norfolk.infi.net - - [30/Aug/1995:18:07:11 -0400] "GET /pub/jeffd/martin.gif HTTP/1.0" 200 4551 205.219.29.21 - - [30/Aug/1995:18:10:51 -0400] "GET /pub/atomicbk/images/reguide.gif HTTP/1.0" 200 39861 rain.ccg.uoknor.edu - - [30/Aug/1995:18:15:09 -0400] "GET /pub/job/vk/view22.jpg HTTP/1.0" 200 6281 cassini.gsfc.nasa.gov - - [30/Aug/1995:18:18:25 -0400] "GET /pub/job/vk/claudia.jpg HTTP/1.0" 200 7225 MARK.BTS.EARLHAM.EDU - - [30/Aug/1995:18:23:35 -0400] "GET /pub/wick/misc/speaker.gif HTTP/1.0" 200 956 129.81.26.24 - - [30/Aug/1995:18:27:00 -0400] "GET /pub/cosmic/garcia2a.gif HTTP/1.0" 200 17578 130.17.6.131 - - [30/Aug/1995:18:30:22 -0400] "GET /pub/bdalzell/Borzoi/zoipics.html HTTP/1.0" 200 1961 gbol13.dct.com - - [30/Aug/1995:18:35:15 -0400] "GET /pub/sshay/images/rainback.gif HTTP/1.0" 200 2541 163.236.251.33 - - [30/Aug/1995:18:39:10 -0400] "GET /pub/atomicbk/catalog/orders.gif HTTP/1.0" 200 800 192.152.149.230 - - [30/Aug/1995:18:43:27 -0400] "GET /pub/job/vk/vk-bk.jpg HTTP/1.0" 200 4569 clark.net - - [30/Aug/1995:18:46:42 -0400] "GET /pub/job/utils.jpg HTTP/1.0" 304 - ipt002.rdyne.rockwell.com - - [30/Aug/1995:18:51:13 -0400] "GET /pub/sshay/defbutn.map?42,42?99,13 HTTP/1.0" 302 227 www-d3.proxy.aol.com - - [30/Aug/1995:18:54:41 -0400] "GET /atomicbk/images/alfetish.gif HTTP/1.0" 200 33094 van04098.direct.ca - - [30/Aug/1995:18:58:15 -0400] "GET /pub/sshay/images/hrdthumb.jpg HTTP/1.0" 200 5094 www-d2.proxy.aol.com - - [30/Aug/1995:19:02:59 -0400] "GET /pub/mglamb/ranch/gifs/ranchbnr.gif HTTP/1.0" 200 36257 130.204.251.96 - - [30/Aug/1995:19:08:24 -0400] "GET /pub/job/vk/view15.jpg HTTP/1.0" 200 6036 142.139.203.6 - - [30/Aug/1995:19:12:58 -0400] "GET /pub/job/vk/view04.jpg HTTP/1.0" 200 4847 www-b2.proxy.aol.com - - [30/Aug/1995:19:17:43 -0400] "GET /pub/networx/autopage/autodeal.gif HTTP/1.0" 200 48029 hpbs100.boi.hp.com - - [30/Aug/1995:19:21:23 -0400] "GET /pub/journalism/5percsmt.gif HTTP/1.0" 200 2321 ppp050.inreach.com - - [30/Aug/1995:19:24:47 -0400] "GET /pub/jeffd/grayball.gif HTTP/1.0" 200 995 node1.heritage.com - - [30/Aug/1995:19:29:09 -0400] "GET /pub/sshay/images/chrsbutn.gif HTTP/1.0" 200 1070 TRFPC07.okladot.state.ok.us - - [30/Aug/1995:19:33:31 -0400] "GET /pub/job/vk/view02.jpg HTTP/1.0" 200 7290 north-110-10.dorm.duke.edu - - [30/Aug/1995:19:38:05 -0400] "GET /pub/job/vk/vendela2.html HTTP/1.0" 200 8267 mpngate1.ny.us.ibm.net - - [30/Aug/1995:19:42:13 -0400] "GET /pub/job/vk/view16.jpg HTTP/1.0" 200 6808 159.242.115.53 - - [30/Aug/1995:19:46:44 -0400] "GET /pub/wick/photos/grass_mini.gif HTTP/1.0" 200 11378 oak.citicorp.com - - [30/Aug/1995:19:50:52 -0400] "GET /pub/job/vk/view20.jpg HTTP/1.0" 200 5070 miriworld.its.unimelb.EDU.AU - - [30/Aug/1995:19:55:32 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 304 - SUNPHY1.PHY.UIC.EDU - - [30/Aug/1995:20:00:02 -0400] "GET /pub/job/vk/view14.jpg HTTP/1.0" 200 4733 picasso.ccad.uiowa.edu - - [30/Aug/1995:20:05:52 -0400] "GET /pub/jeffd/sm_eaghd.gif HTTP/1.0" 304 - calator.adp.wisc.edu - - [30/Aug/1995:20:10:58 -0400] "GET /pub/job/vk/view14.jpg HTTP/1.0" 200 4733 worf.qntm.com - - [30/Aug/1995:20:15:24 -0400] "GET /pub/job/vk/view11.jpg HTTP/1.0" 200 5001 www.neocom.ca - - [30/Aug/1995:20:19:06 -0400] "GET /pub/global/russia.html HTTP/1.0" 200 1599 164.154.28.4 - - [30/Aug/1995:20:23:06 -0400] "GET /pub/job/swim/v-58.jpg HTTP/1.0" 200 3577 solomon.syrres.com - - [30/Aug/1995:20:27:33 -0400] "GET /pub/job/vk/view09.jpg HTTP/1.0" 200 6011 webmaker.clark.net - - [30/Aug/1995:20:32:02 -0400] "GET /pub/webmaker/glimpse/glimpsehttp/wwwlib/ HTTP/1.0" 403 142 clark.net - - [30/Aug/1995:20:37:24 -0400] "GET /pub/job/swim/v-02.jpg HTTP/1.0" 200 2928 client31.sct.fr - - [30/Aug/1995:20:43:17 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 9084 acs4.acs.ucalgary.ca - - [30/Aug/1995:20:48:15 -0400] "GET /atomicbk/new/emboss.jpg HTTP/1.0" 200 4741 nawlns.monsanto.com - - [30/Aug/1995:20:53:13 -0400] "GET /pub/networx/autopage/classic/cl003_1s.gif HTTP/1.0" 200 16250 www-c9.proxy.aol.com - - [30/Aug/1995:20:57:54 -0400] "GET /pub/sshay/partner.html HTTP/1.0" 200 7065 ts1-and-4.iquest.net - - [30/Aug/1995:21:02:59 -0400] "GET /pub/downin/html/Graphics/contact-infot.gif HTTP/1.0" 200 7060 m06.leba.net - - [30/Aug/1995:21:08:56 -0400] "GET /pub/russadam/wnew.html HTTP/1.0" 200 4653 ZEBRA.ENGR.UTK.EDU - - [30/Aug/1995:21:15:11 -0400] "GET /pub/alweiner/work/spacer25.gif HTTP/1.0" 200 45 line179.worldweb.net - - [30/Aug/1995:21:19:29 -0400] "GET /winfield/gifs/navy_mar.jpg HTTP/1.0" 200 2228 slip37-254-99.ibm.net - - [30/Aug/1995:21:23:46 -0400] "GET /atomicbk/images/knockers.gif HTTP/1.0" 200 41213 pool73.maple.net - - [30/Aug/1995:21:28:11 -0400] "GET /pub/sshay/gallery.html HTTP/1.0" 200 2101 neptune.pharm.hiroshima-u.ac.jp - - [30/Aug/1995:21:33:17 -0400] "GET /pub/job/swim/v-18.jpg HTTP/1.0" 200 2838 alweiner.clark.net - - [30/Aug/1995:21:38:43 -0400] "POST /pub/alweiner/cgi-bin/w3magic.cgi?-f|beta-register@__00 HTTP/1.0" 200 234 gin.obspm.fr - - [30/Aug/1995:21:43:46 -0400] "GET /pub/rjgula/attacks/mail/sml3128a.txt HTTP/1.0" 200 2615 bdmgate.bdm.com - - [30/Aug/1995:21:47:59 -0400] "GET /pub/sshay/images/rainback.gif HTTP/1.0" 200 2541 nvcon26.pcsub9.db.erau.edu - - [30/Aug/1995:21:52:10 -0400] "GET /pub/gen/fas/irp/arrowlf.gif HTTP/1.0" 200 496 nx20.mik.uky.edu - - [30/Aug/1995:21:56:18 -0400] "GET /pub/job/vk/note.gif HTTP/1.0" 200 9413 h-abyss.annap.infi.net - - [30/Aug/1995:22:00:43 -0400] "GET /graphics/cgi.gif HTTP/1.0" 200 11499 gateway.Kwantlen.BC.CA - - [30/Aug/1995:22:06:11 -0400] "GET /pub/alweiner/owl.gif HTTP/1.0" 200 1285 205.133.22.104 - - [30/Aug/1995:22:10:50 -0400] "GET /pub/job/vk/note.gif HTTP/1.0" 304 - crc-selq10.unl.edu - - [30/Aug/1995:22:16:08 -0400] "GET /pub/atomicbk/catalog/new.gif HTTP/1.0" 200 744 131.90.144.58 - - [30/Aug/1995:22:21:28 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 brian.cs.moc.govt.nz - - [30/Aug/1995:22:26:57 -0400] "HEAD /pub/aztec/mariner/ HTTP/1.0" 200 1993 scopen.sc.hp.com - - [30/Aug/1995:22:33:02 -0400] "GET /pub/wmcbrine/Madonna/punk-bw.jpg HTTP/1.0" 200 63843 ix-war-mi4-06.ix.netcom.com - - [30/Aug/1995:22:38:45 -0400] "GET /atomicbk/catalog/mini.gif HTTP/1.0" 200 360 www-d4.proxy.aol.com - - [30/Aug/1995:22:43:45 -0400] "GET /pub/mbrophy/pix/interv.jpg HTTP/1.0" 200 5047 ip190.boi.primenet.com - - [30/Aug/1995:22:48:05 -0400] "GET /pub/jeffd/bbs.gif HTTP/1.0" 200 6715 wt10.library.luc.edu - - [30/Aug/1995:22:53:46 -0400] "GET /pub/phil/images/jh63.gif HTTP/1.0" 200 3371 205.199.120.184 - - [30/Aug/1995:22:59:56 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 dial3.ecua.net.ec - - [30/Aug/1995:23:07:19 -0400] "GET /pub/networx/autopage/autopage.html HTTP/1.0" 200 1413 linc.dialup.access.net - - [30/Aug/1995:23:13:24 -0400] "GET /pub/wsg/html/web/home.html HTTP/1.0" 200 950 Gatordyn15.ac.hillsdale.edu - - [30/Aug/1995:23:19:59 -0400] "GET /pub/jeffd/speaker.gif HTTP/1.0" 200 2646 cw8.ppp.ornl.gov - - [30/Aug/1995:23:26:02 -0400] "GET /pub/job/vk/view05.jpg HTTP/1.0" 200 5012 isr1104.urh.uiuc.edu - - [30/Aug/1995:23:30:46 -0400] "GET /atomicbk/promo.gif HTTP/1.0" 200 849 piweba3y.prodigy.com - - [30/Aug/1995:23:35:57 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 304 - slip-156.california.com - - [30/Aug/1995:23:41:49 -0400] "GET /pub/cardman/tinyidrm.gif HTTP/1.0" 200 4284 192.206.65.126 - - [30/Aug/1995:23:46:45 -0400] "GET /atomicbk/email.gif HTTP/1.0" 200 756 cs29-75.usafa.af.mil - - [30/Aug/1995:23:52:57 -0400] "GET /pub/wsg/html/jan/vespers.html HTTP/1.0" 200 476 ix-pas11-26.ix.netcom.com - - [30/Aug/1995:23:59:12 -0400] "GET /pub/moncomm/html/ft.jpg HTTP/1.0" 200 62501 pukatea.its.vuw.ac.nz - - [31/Aug/1995:00:04:13 -0400] "GET /pub/job/vk/view14.jpg HTTP/1.0" 200 4733 cs1-09.haz.ptd.net - - [31/Aug/1995:00:10:12 -0400] "GET /pub/epchurch/push/push3.html HTTP/1.0" 200 363 ts2-08.InfoRamp.Net - - [31/Aug/1995:00:15:35 -0400] "GET /pub/sshay/images/davthumb.jpg HTTP/1.0" 200 3914 cs08-30.usafa.af.mil - - [31/Aug/1995:00:22:51 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 9084 ts4-11.westwood.ts.ucla.edu - - [31/Aug/1995:00:27:37 -0400] "GET /pub/lupine/images/fantasia/fantasia.gif HTTP/1.0" 200 28795 www-a1.proxy.aol.com - - [31/Aug/1995:00:34:15 -0400] "GET /pub/atomicbk/catalog/erotica.html HTTP/1.0" 200 11362 port26.lnd.com - - [31/Aug/1995:00:40:16 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 304 - p00.slip.pad.MKS.COM - - [31/Aug/1995:00:46:10 -0400] "GET /pub/job/swim/v-09.jpg HTTP/1.0" 200 3755 SLIPROCK-01.DIALIN.UIC.EDU - - [31/Aug/1995:00:59:53 -0400] "GET /pub/aztec/mariner/images/mnlognew.gif HTTP/1.0" 200 12193 edgar.cs.washington.edu - - [31/Aug/1995:01:11:01 -0400] "GET /pub/rothman/telhome.html HTTP/1.0" 200 35887 star65.hkstar.com - - [31/Aug/1995:01:22:16 -0400] "GET /pub/sshay/images/greysped.gif HTTP/1.0" 200 4542 163.135.205.237 - - [31/Aug/1995:01:34:27 -0400] "GET /pub/job/vk/vk-ad-19.jpg HTTP/1.0" 200 44323 ppp076-stdkn2.ulaval.ca - - [31/Aug/1995:01:43:01 -0400] "GET /pub/lupine/www/werewolf.html HTTP/1.0" 200 1041 158.122.1.191 - - [31/Aug/1995:01:51:48 -0400] "GET /pub/job/vk/view16.jpg HTTP/1.0" 200 6808 asm_19.unm.edu - - [31/Aug/1995:02:00:17 -0400] "GET /pub/peace/line1.gif HTTP/1.0" 304 - van08240.direct.ca - - [31/Aug/1995:02:08:51 -0400] "GET /pub/lazarus/citadel/karenpic.gif HTTP/1.0" 200 3709 labpc33.acs.uci.edu - - [31/Aug/1995:02:18:47 -0400] "GET /pub/job/vk/view11.jpg HTTP/1.0" 200 5001 mac_yfr.chem.ucla.edu - - [31/Aug/1995:02:26:53 -0400] "GET /atomicbk/catalog/home.gif HTTP/1.0" 200 813 ip220.pom.primenet.com - - [31/Aug/1995:02:35:13 -0400] "GET /pub/job/vk/view02.jpg HTTP/1.0" 200 7290 slip23.island.net - - [31/Aug/1995:02:44:13 -0400] "GET /atomicbk/email.gif HTTP/1.0" 200 756 brother.cc.monash.edu.au - - [31/Aug/1995:02:53:40 -0400] "GET /pub/job/vk/babe.jpg HTTP/1.0" 200 8235 sfsp73.slip.net - - [31/Aug/1995:03:04:30 -0400] "GET /pub/wine/home.html HTTP/1.0" 200 1526 ztivax.zfe.siemens.de - - [31/Aug/1995:03:14:47 -0400] "GET /pub/job/vk/vk-bk.jpg HTTP/1.0" 200 4569 madonna.dep.no - - [31/Aug/1995:03:24:37 -0400] "GET /pub/job/vk/cindy.jpg HTTP/1.0" 200 4267 169.140.209.207 - - [31/Aug/1995:03:36:27 -0400] "GET /pub/k2/am4x44u/maps/4x4_home.gif HTTP/1.0" 200 43750 tlrouter.motctl.gov.tw - - [31/Aug/1995:03:46:28 -0400] "GET /pub/k2/am4x44u/gifs/netscape.gif HTTP/1.0" 200 1050 laurel.yorku.ca - - [31/Aug/1995:03:54:39 -0400] "GET /pub/job/vk/vk-ad-10.jpg HTTP/1.0" 200 53676 194.14.81.162 - - [31/Aug/1995:04:05:02 -0400] "GET /pub/pgarrett/rad.gif HTTP/1.0" 200 216 128.39.174.54 - - [31/Aug/1995:04:14:05 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 205.208.30.107 - - [31/Aug/1995:04:22:58 -0400] "GET /pub/atomicbk/catalog/mini.gif HTTP/1.0" 200 360 192.65.228.97 - - [31/Aug/1995:04:30:55 -0400] "GET /pub/networx/autopage/exotic/ex001s.gif HTTP/1.0" 200 15418 r221pc16.vtyh.fi - - [31/Aug/1995:04:45:09 -0400] "GET /pub/job/vk/flowers1.gif HTTP/1.0" 200 4288 vigour2.univ-lyon1.fr - - [31/Aug/1995:04:56:25 -0400] "GET /pub/atomicbk/images/centur.gif HTTP/1.0" 200 52508 poppy.hensa.ac.uk - - [31/Aug/1995:05:07:56 -0400] "GET /pub/lupine/www/images/balls/blue.gif HTTP/1.0" 200 326 macb4.sj.sg.cnrs-dir.fr - - [31/Aug/1995:05:20:00 -0400] "GET /pub/job/vk/view24.jpg HTTP/1.0" 200 6209 131.11.32.164 - - [31/Aug/1995:05:31:46 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 quandong.itd.adelaide.edu.au - - [31/Aug/1995:05:43:00 -0400] "GET /pub/job/vk/flowers1.gif HTTP/1.0" 304 - slx.cc.kumamoto-u.ac.jp - - [31/Aug/1995:05:54:47 -0400] "GET /pub/job/vk/view13.jpg HTTP/1.0" 200 6219 bonanno.pisoft.it - - [31/Aug/1995:06:06:29 -0400] "GET /atomicbk/seanc.gif HTTP/1.0" 200 734 194.89.170.103 - - [31/Aug/1995:06:17:30 -0400] "GET /pub/job/vk/view22.jpg HTTP/1.0" 200 6281 uryu.elsip.hokudai.ac.jp - - [31/Aug/1995:06:31:23 -0400] "GET /pub/job/vk/view22.jpg HTTP/1.0" 200 6281 193.38.83.152 - - [31/Aug/1995:06:43:42 -0400] "GET /pub/atomicbk/news.gif HTTP/1.0" 200 912 ball.uunet.ca - - [31/Aug/1995:06:54:58 -0400] "GET /pub/mjoneill/links.html HTTP/1.0" 404 207 nhhs-1.nhh.no - - [31/Aug/1995:07:07:08 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 9084 130.41.37.138 - - [31/Aug/1995:07:19:48 -0400] "GET /pub/badger/schedule.html HTTP/1.0" 304 - 194.18.98.78 - - [31/Aug/1995:07:29:34 -0400] "GET /pub/phil/images/jh6.gif HTTP/1.0" 200 9534 151.104.20.81 - - [31/Aug/1995:07:39:19 -0400] "GET /pub/job/vk/2-page-2.gif HTTP/1.0" 200 2527 dialup7-hugin.oden.se - - [31/Aug/1995:07:50:57 -0400] "GET /pub/job/vk/view23.jpg HTTP/1.0" 200 5165 intgate.raleigh.ibm.com - - [31/Aug/1995:07:58:05 -0400] "GET /pub/sshay/images/brwthumb.jpg HTTP/1.0" 200 3306 194.72.160.77 - - [31/Aug/1995:08:05:15 -0400] "GET /pub/wmcbrine/html/encyc1s.gif HTTP/1.0" 200 4701 143.217.50.14 - - [31/Aug/1995:08:12:55 -0400] "GET /pub/lupine/images/batb/batb.gif HTTP/1.0" 200 7913 inetg1.Arco.COM - - [31/Aug/1995:08:20:45 -0400] "GET /pub/sshay/gallery.html HTTP/1.0" 200 2101 151.208.31.66 - - [31/Aug/1995:08:27:48 -0400] "GET /pub/ibos/ibos2.gif HTTP/1.0" 200 7889 term19.vol.net - - [31/Aug/1995:08:34:46 -0400] "GET /pub/job/vk/v-sig.gif HTTP/1.0" 200 4864 annex2000-01.open.ac.uk - - [31/Aug/1995:08:42:42 -0400] "GET /pub/atomicbk/catalog/orders.gif HTTP/1.0" 304 - rscibm01.sagus.com - - [31/Aug/1995:08:47:05 -0400] "GET /theme/rule.gif HTTP/1.0" 200 8728 scott.hris.msu.edu - - [31/Aug/1995:08:53:03 -0400] "GET /pub/jeffd/question.html HTTP/1.0" 200 60823 gw3.att.com - - [31/Aug/1995:09:00:14 -0400] "GET /pub/job/swim/v-03.jpg HTTP/1.0" 200 5882 slmel4p33.ozemail.com.au - - [31/Aug/1995:09:06:13 -0400] "GET /atomicbk/images/atomgirl.jpg HTTP/1.0" 200 34164 smr144.montrouge.smr.slb.com - - [31/Aug/1995:09:12:14 -0400] "GET /atomicbk/direct.gif HTTP/1.0" 200 833 piweba2y.prodigy.com - - [31/Aug/1995:09:17:04 -0400] "GET /atomicbk/catalog/mini.gif HTTP/1.0" 200 360 fin002.urmc.rochester.edu - - [31/Aug/1995:09:23:15 -0400] "GET /pub/rsjdfg/ HTTP/1.0" 200 2109 alweiner.clark.net - - [31/Aug/1995:09:29:13 -0400] "GET /pub/alweiner/cgi-bin/homepage.html?test HTTP/1.0" 200 744 131.115.183.104 - - [31/Aug/1995:09:34:44 -0400] "GET /pub/atomicbk/images/spicy2.gif HTTP/1.0" 200 29018 130.207.65.13 - - [31/Aug/1995:09:40:16 -0400] "GET /pub/k2/am4x44u/gifs/bgrivit2.gif HTTP/1.0" 304 - 199.34.140.200 - - [31/Aug/1995:09:45:46 -0400] "GET /atomicbk/new/logo.gif HTTP/1.0" 200 1942 web.kyoto-inet.or.jp - - [31/Aug/1995:09:51:07 -0400] "GET /pub/job/vaca-02.jpg HTTP/1.0" 200 8589 suske.nikhef.nl - - [31/Aug/1995:09:55:55 -0400] "GET /pub/peace/VRSFoot.GIF HTTP/1.0" 200 16052 grail616.nando.net - - [31/Aug/1995:10:01:29 -0400] "GET /pub/alweiner/string.gif HTTP/1.0" 200 503 obara.gw.tohoku.ac.jp - - [31/Aug/1995:10:07:21 -0400] "GET /pub/sshay/home.html HTTP/1.0" 200 3149 dutnsi0.tn.tudelft.nl - - [31/Aug/1995:10:12:31 -0400] "GET /pub/job/swim/v-01.jpg HTTP/1.0" 200 2839 andromeda.cselt.stet.it - - [31/Aug/1995:10:17:42 -0400] "GET /pub/mairhart/ HTTP/1.0" 200 3257 piweba1y.prodigy.com - - [31/Aug/1995:10:22:15 -0400] "GET /pub/sgs/gifs/bann01.gif HTTP/1.0" 200 2203 www-c3.proxy.aol.com - - [31/Aug/1995:10:26:50 -0400] "GET /pub/job/vk/babe.jpg HTTP/1.0" 200 8235 kwoods.clark.net - - [31/Aug/1995:10:31:29 -0400] "GET /pub/kwoods/kens.html HTTP/1.0" 200 2772 204.213.224.109 - - [31/Aug/1995:10:36:20 -0400] "GET /pub/listserv/diamond.gif HTTP/1.0" 404 328 webgate1.mot.com - - [31/Aug/1995:10:41:07 -0400] "GET /pub/job/vk/view14.jpg HTTP/1.0" 304 - as2511-3.sl013.cns.vt.edu - - [31/Aug/1995:10:45:23 -0400] "GET /pub/sshay/images/justin.jpg HTTP/1.0" 200 3489 www-b2.proxy.aol.com - - [31/Aug/1995:10:49:52 -0400] "GET /pub/sshay/images/gallery.gif HTTP/1.0" 200 8010 mpngate4.ny.us.ibm.net - - [31/Aug/1995:10:54:59 -0400] "GET /pub/job/vk/vk-bk.jpg HTTP/1.0" 200 4569 dns.uni-trier.de - - [31/Aug/1995:10:58:36 -0400] "GET /pub/sinkers/whatsnew.gif HTTP/1.0" 200 7708 connex.ip.holonet.net - - [31/Aug/1995:11:02:16 -0400] "GET /pub/russadam/share.html HTTP/1.0" 304 - helser57.res.iastate.edu - - [31/Aug/1995:11:07:10 -0400] "GET /pub/job/vk/view16.jpg HTTP/1.0" 200 6808 130.127.212.44 - - [31/Aug/1995:11:12:04 -0400] "GET /pub/job/vk/note.gif HTTP/1.0" 200 9413 204.252.201.173 - - [31/Aug/1995:11:16:33 -0400] "GET /pub/sshay/images/homebutn.gif HTTP/1.0" 200 1071 dlp49.erinet.com - - [31/Aug/1995:11:21:27 -0400] "GET /pub/listserv/diamond.gif HTTP/1.0" 404 328 port1.one.se - - [31/Aug/1995:11:25:41 -0400] "GET /pub/jeffd/sm_eaghd.gif HTTP/1.0" 200 2866 148.136.113.211 - - [31/Aug/1995:11:29:46 -0400] "GET /pub/job/vk/view14.jpg HTTP/1.0" 200 4733 nile.finance.tisc.titan.com - - [31/Aug/1995:11:33:07 -0400] "GET /pub/job/vk/flowers1.gif HTTP/1.0" 200 4288 gate.mmd.com - - [31/Aug/1995:11:37:10 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 pc41.cen.bris.ac.uk - - [31/Aug/1995:11:40:32 -0400] "GET /pub/job/vk/v-sig.gif HTTP/1.0" 200 4864 slhp22.ug.eds.com - - [31/Aug/1995:11:43:37 -0400] "GET /pub/jeffd/sm_eaghd.gif HTTP/1.0" 200 2866 129.120.107.197 - - [31/Aug/1995:11:47:43 -0400] "GET /pub/job/vk/view16.jpg HTTP/1.0" 200 6808 130.226.35.189 - - [31/Aug/1995:11:51:25 -0400] "GET /pub/job/vk/view04.jpg HTTP/1.0" 200 4847 publ-2.Lib.Berkeley.EDU - - [31/Aug/1995:11:54:24 -0400] "GET /pub/sshay/images/tywetsut.jpg HTTP/1.0" 200 21652 199.1.111.4 - - [31/Aug/1995:11:57:33 -0400] "GET /pub/menswear/colorbar.gif HTTP/1.0" 200 1087 128.158.129.169 - - [31/Aug/1995:12:01:33 -0400] "GET /pub/jumpsam/logogosp.jpg HTTP/1.0" 200 15227 pc-3948.safb.af.mil - - [31/Aug/1995:12:05:12 -0400] "GET /pub/rmharris/images/tpointbt.gif HTTP/1.0" 200 1743 Michael-ISDN.Stanford.EDU - - [31/Aug/1995:12:09:39 -0400] "GET /pub/iz/Books/Top100/top100.html HTTP/1.0" 200 3096 thompson.urel.wsu.edu - - [31/Aug/1995:12:14:10 -0400] "GET /pub/jeffd/index.html HTTP/1.0" 200 11428 gopher.ic.ac.uk - - [31/Aug/1995:12:17:44 -0400] "GET /pub/job/vk/view06.jpg HTTP/1.0" 200 6553 london.scsn.net - - [31/Aug/1995:12:22:29 -0400] "GET /pub/job/vk/note.gif HTTP/1.0" 200 9413 www-e4.proxy.aol.com - - [31/Aug/1995:12:26:43 -0400] "GET /theme/cgi-bin/serchrnd.html HTTP/1.0" 200 6052 ad01-021.compuserve.com - - [31/Aug/1995:12:31:00 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 198.17.29.244 - - [31/Aug/1995:12:34:31 -0400] "GET /pub/gregd/dude3c.jpg HTTP/1.0" 200 15305 132.156.33.197 - - [31/Aug/1995:12:37:31 -0400] "GET /pub/job/vk/view09.jpg HTTP/1.0" 200 6011 student37.cba.ua.edu - - [31/Aug/1995:12:40:54 -0400] "GET /pub/jeffd/95-60.gif HTTP/1.0" 200 23583 198.209.101.76 - - [31/Aug/1995:12:45:06 -0400] "GET /pub/k2/am4x44u/gifs/redball.gif HTTP/1.0" 200 206 ryp92.ryp.umu.se - - [31/Aug/1995:12:48:22 -0400] "GET /pub/job/vk/note.gif HTTP/1.0" 200 9413 157.99.120.122 - - [31/Aug/1995:12:51:53 -0400] "GET /atomicbk/images/atomgirl.jpg HTTP/1.0" 200 34164 cindy.nocom.se - - [31/Aug/1995:12:55:25 -0400] "GET /pub/job/vk/cindy.jpg HTTP/1.0" 200 4267 rcwusr.bp.com - - [31/Aug/1995:12:59:37 -0400] "GET /pub/mpowers/j4j/web/Tablegifs/MailIcon.gif HTTP/1.0" 200 2129 crystal.kues.kyoto-u.ac.jp - - [31/Aug/1995:13:04:31 -0400] "GET /pub/job/swim/swim.jpg HTTP/1.0" 200 17401 204.101.3.32 - - [31/Aug/1995:13:08:10 -0400] "GET /pub/micros/mr/microall.html HTTP/1.0" 200 3645 rab.clark.net - - [31/Aug/1995:13:12:05 -0400] "GET /pub/gorbachev/home.html HTTP/1.0" 200 2228 annex3-60.dial.umd.edu - - [31/Aug/1995:13:16:31 -0400] "GET /pub/downin/html/arlonet/arlo.html HTTP/1.0" 200 730 RELABB8.RE.UOKHSC.EDU - - [31/Aug/1995:13:20:40 -0400] "GET /pub/atomicbk/news.gif HTTP/1.0" 200 912 whitey.near.net - - [31/Aug/1995:13:24:01 -0400] "GET /pub/atomicbk/catalog/orders.gif HTTP/1.0" 200 800 ix-smx-ca1-07.ix.netcom.com - - [31/Aug/1995:13:27:49 -0400] "GET /pub/job/vk/note.gif HTTP/1.0" 200 9413 world236.worldaccess.COM - - [31/Aug/1995:13:31:27 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 200 813 CSACL30.SALSEM.AC.AT - - [31/Aug/1995:13:35:05 -0400] "GET /pub/job/vk/view07.jpg HTTP/1.0" 200 7057 ws25-07.dev.oclc.org - - [31/Aug/1995:13:39:12 -0400] "GET /pub/global/toc.html HTTP/1.0" 200 1696 149.123.63.80 - - [31/Aug/1995:13:42:41 -0400] "GET /pub/journalism/awesome.html HTTP/1.0" 200 11610 webgate1.mot.com - - [31/Aug/1995:13:46:54 -0400] "GET /atomicbk/contest.gif HTTP/1.0" 200 669 unknown.atms.be - - [31/Aug/1995:13:50:39 -0400] "GET /pub/murple/html/propaganda.html HTTP/1.0" 200 1795 zappa.ilx.com - - [31/Aug/1995:13:54:34 -0400] "GET /pub/job/vk/babe.jpg HTTP/1.0" 200 8235 peacock.tcinc.com - - [31/Aug/1995:13:58:05 -0400] "GET /atomicbk/new.gif HTTP/1.0" 200 744 Alstonk.er.doe.gov - - [31/Aug/1995:14:01:35 -0400] "GET /pub/cybersoft/home.html HTTP/1.0" 200 918 www.westpub.com - - [31/Aug/1995:14:05:35 -0400] "GET /pub/kevina/sl/7signals/apology.gif HTTP/1.0" 200 1222 fw1.torolab.ibm.com - - [31/Aug/1995:14:10:07 -0400] "GET /pub/job/swim/v-20.jpg HTTP/1.0" 200 2510 gate.chips.ibm.com - - [31/Aug/1995:14:14:22 -0400] "GET /pub/job/vk/arrow-r.gif HTTP/1.0" 200 3042 169.147.155.104 - - [31/Aug/1995:14:17:38 -0400] "GET /pub/mikebilt/dchome.gif HTTP/1.0" 200 7041 dummy241.library.pitt.edu - - [31/Aug/1995:14:21:01 -0400] "GET /pub/job/vk/view12.jpg HTTP/1.0" 200 5966 gk-east.usps.gov - - [31/Aug/1995:14:24:19 -0400] "GET /pub/cmp/netguide/free.gif HTTP/1.0" 304 - orion.lasierra.edu - - [31/Aug/1995:14:28:23 -0400] "GET /pub/rsjdfg/ HTTP/1.0" 200 2109 OOPS.NCSL.NIST.GOV - - [31/Aug/1995:14:32:54 -0400] "GET /html/usage/index.html HTTP/1.0" 200 994 kellyn.nwscc.sea06.navy.mil - - [31/Aug/1995:14:36:42 -0400] "GET /pub/job/vk/view21.jpg HTTP/1.0" 200 5228 piweba2y.prodigy.com - - [31/Aug/1995:14:40:08 -0400] "GET /pub/lupine/images/batb/beast-bw.gif HTTP/1.0" 200 37584 vagrant.vf.mmc.com - - [31/Aug/1995:14:44:16 -0400] "GET /pub/lupine/www/lionking.html HTTP/1.0" 200 1962 unogate.unocal.com - - [31/Aug/1995:14:48:08 -0400] "GET /pub/job/vk/kathy.jpg HTTP/1.0" 200 7450 pc2.mrree.cl - - [31/Aug/1995:14:51:43 -0400] "GET /pub/diplonet/los-schd.gif HTTP/1.0" 200 7504 WEDMONDS.econ.ag.gov - - [31/Aug/1995:14:55:57 -0400] "GET /pub/atomicbk/catalog/emboss.jpg HTTP/1.0" 200 4741 mail.rogers.com - - [31/Aug/1995:14:59:01 -0400] "GET /pub/networx/autopage/exotic.html HTTP/1.0" 200 1274 ottgate2.bnr.ca - - [31/Aug/1995:15:03:16 -0400] "GET /pub/lschank/web/up.gif HTTP/1.0" 200 264 proximity.cs.purdue.edu - - [31/Aug/1995:15:07:13 -0400] "GET /pub/alweiner/cgi-bin/homepage.html?beta-login HTTP/1.0" 200 1319 204.120.230.198 - - [31/Aug/1995:15:10:32 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 rose_ip178.efn.org - - [31/Aug/1995:15:15:00 -0400] "GET /pub/sshay/images/galybutn.gif HTTP/1.0" 200 1080 metrix.metrobbs.com - - [31/Aug/1995:15:18:20 -0400] "GET /atomicbk/catalog/new.gif HTTP/1.0" 200 744 gate.chips.ibm.com - - [31/Aug/1995:15:21:49 -0400] "GET /pub/atomicbk/images/tease.gif HTTP/1.0" 200 43516 combsds.image.uky.edu - - [31/Aug/1995:15:25:16 -0400] "GET /pub/sshay/images/partner.gif HTTP/1.0" 200 14363 G-PC-009.TAMU.EDU - - [31/Aug/1995:15:28:32 -0400] "GET /pub/lupine/www/images/icons/note.gif HTTP/1.0" 200 565 129.108.32.28 - - [31/Aug/1995:15:32:04 -0400] "GET /pub/atomicbk/catalog/new.gif HTTP/1.0" 200 744 east-9-36.dorm.duke.edu - - [31/Aug/1995:15:35:21 -0400] "GET /atomicbk/emboss.jpg HTTP/1.0" 200 4741 160.205.101.14 - - [31/Aug/1995:15:38:37 -0400] "GET /pub/k2/am4x44u/gifs/alum1.gif HTTP/1.0" 200 457 ad16-019.compuserve.com - - [31/Aug/1995:15:42:31 -0400] "GET /pub/sshay/images/tyusvest.jpg HTTP/1.0" 200 26791 nebula7.clark.net - - [31/Aug/1995:15:46:49 -0400] "GET /pub/k2/am4x44u/gifs/ametal.gif HTTP/1.0" 200 10335 mglamb.clark.net - - [31/Aug/1995:15:51:24 -0400] "GET /pub/mglamb/homebar.map?74,48 HTTP/1.0" 302 227 165.95.48.240 - - [31/Aug/1995:15:55:04 -0400] "GET /pub/atomicbk/catalog/new.gif HTTP/1.0" 200 744 faculty8.mps.org - - [31/Aug/1995:15:58:44 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 thalie.imag.fr - - [31/Aug/1995:16:01:22 -0400] "GET /pub/job/vk/view07.jpg HTTP/1.0" 200 7057 par-m104a-2.urh.uiuc.edu - - [31/Aug/1995:16:04:30 -0400] "GET /pub/k2/am4x44u/gifs/clubline.gif HTTP/1.0" 200 1254 gtx.com - - [31/Aug/1995:16:08:27 -0400] "GET /pub/howie/OO/methods.html HTTP/1.0" 200 558 tdb.datatel.com - - [31/Aug/1995:16:12:51 -0400] "GET /pub/sshay/images/chris3.jpg HTTP/1.0" 200 50739 gorgon.pwgsc.gc.ca - - [31/Aug/1995:16:16:15 -0400] "GET /pub/k2/am4x44u/gifs/slgmc.gif HTTP/1.0" 200 106074 198.3.50.158 - - [31/Aug/1995:16:19:28 -0400] "GET /pub/job/vk/vk-ad-14.jpg HTTP/1.0" 200 67441 www00.Btx.DTAG.DE - - [31/Aug/1995:16:23:00 -0400] "GET /pub/micros/note01.gif HTTP/1.0" 200 1719 www-e7.proxy.aol.com - - [31/Aug/1995:16:28:15 -0400] "GET /pub/fan/albumpics/sweetsoul.jpg HTTP/1.0" 200 2552 bgumail.bgu.ac.il - - [31/Aug/1995:16:31:50 -0400] "GET /pub/rant/rant/ HTTP/1.0" 200 5049 server1-2.durham.net - - [31/Aug/1995:16:35:48 -0400] "GET /atomicbk/catalog/emboss.jpg HTTP/1.0" 200 4741 foglernt06.ursus.maine.edu - - [31/Aug/1995:16:39:05 -0400] "GET /pub/listserv/lsmus1.html HTTP/1.0" 200 2713 mro2577.pclan.ets.org - - [31/Aug/1995:16:42:50 -0400] "GET /pub/job/vk/v-sig.gif HTTP/1.0" 200 4864 edelman.his.com - - [31/Aug/1995:16:46:38 -0400] "GET /pub/robert/current.html HTTP/1.0" 200 30337 HRSB741.RESNET.UPENN.EDU - - [31/Aug/1995:16:49:30 -0400] "GET /pub/job/vk/vk-ad-03.jpg HTTP/1.0" 200 97751 reach.com - - [31/Aug/1995:16:53:08 -0400] "GET /pub/atomicbk/catalog/sleazmag.html HTTP/1.0" 200 6688 pc51ms108-f.cislabs.okstate.edu - - [31/Aug/1995:16:56:40 -0400] "GET /pub/rsjdfg/Reprise.html HTTP/1.0" 200 234 PC_joemu.compu.com - - [31/Aug/1995:17:00:27 -0400] "GET /pub/job/vendela.jpg HTTP/1.0" 200 36746 129.219.40.121 - - [31/Aug/1995:17:04:10 -0400] "GET /pub/atomicbk/catalog/new.gif HTTP/1.0" 200 744 h-nain.nr.infi.net - - [31/Aug/1995:17:09:02 -0400] "GET /pub/k2/suv/suv1.htm HTTP/1.0" 200 4859 gateway.senate.gov - - [31/Aug/1995:17:13:36 -0400] "GET /pub/jeffd/jfk.au HTTP/1.0" 200 105716 MATHSUN15.MATH.UTK.EDU - - [31/Aug/1995:17:16:53 -0400] "GET /pub/job/vk/view09.jpg HTTP/1.0" 200 6011 box708.labs.cis.pitt.edu - - [31/Aug/1995:17:20:27 -0400] "GET /pub/rsjdfg/Images/Sean.McDermott.gif HTTP/1.0" 200 18090 203.4.167.3 - - [31/Aug/1995:17:24:12 -0400] "GET /atomicbk/catalog/erotica.html HTTP/1.0" 200 11362 www-c3.proxy.aol.com - - [31/Aug/1995:17:27:55 -0400] "GET /pub/robert/current.html HTTP/1.0" 200 30337 ix-sf9-21.ix.netcom.com - - [31/Aug/1995:17:32:12 -0400] "GET /pub/rmharris/images/books.gif HTTP/1.0" 200 2253 bouchon-ppp.clark.net - - [31/Aug/1995:17:36:39 -0400] "GET /pub/bouchon/dragon.gif HTTP/1.0" 304 - H306A-Performa636.Stanford.EDU - - [31/Aug/1995:17:40:53 -0400] "GET /pub/fervor/bob6.gif HTTP/1.0" 200 12142 pc06ben.stflabs.okstate.edu - - [31/Aug/1995:17:44:11 -0400] "GET /pub/job/swim/v-09.jpg HTTP/1.0" 200 3755 edslink3.eds.com - - [31/Aug/1995:17:48:20 -0400] "GET /pub/pribut/5percmdt.gif HTTP/1.0" 200 3678 sshirk.orem.novell.com - - [31/Aug/1995:17:52:34 -0400] "GET /pub/k2/am4x44u/gifs/superwin.gif HTTP/1.0" 200 2263 162.38.183.250 - - [31/Aug/1995:17:56:07 -0400] "GET /pub/job/mail.jpg HTTP/1.0" 200 4282 Port30.TS1.MsState.Edu - - [31/Aug/1995:18:00:16 -0400] "GET /atomicbk/catalog/pinup.html HTTP/1.0" 200 8316 trvl.magnet.ca - - [31/Aug/1995:18:04:44 -0400] "GET /pub/sshay/images/chris3.jpg HTTP/1.0" 200 50739 132.68.21.60 - - [31/Aug/1995:18:09:39 -0400] "GET /pub/job/swim/v-02.jpg HTTP/1.0" 200 2928 www-b2.proxy.aol.com - - [31/Aug/1995:18:14:25 -0400] "GET /pub/job/swim/ss-05.jpg HTTP/1.0" 200 47937 nameless.house.gov - - [31/Aug/1995:18:18:11 -0400] "GET /pub/wick/photos.html HTTP/1.0" 200 1482 192.190.252.20 - - [31/Aug/1995:18:23:32 -0400] "GET /pub/job/vk/vk-bk.jpg HTTP/1.0" 200 4569 198.188.250.175 - - [31/Aug/1995:18:27:36 -0400] "GET /pub/job/vk/view03.jpg HTTP/1.0" 200 8506 lapphp.in2p3.fr - - [31/Aug/1995:18:32:33 -0400] "GET /pub/atomicbk/catalog/new.gif HTTP/1.0" 200 744 131.100.160.24 - - [31/Aug/1995:18:36:37 -0400] "GET /pub/league/sweats1.jpg HTTP/1.0" 200 32523 206.27.236.3 - - [31/Aug/1995:18:39:54 -0400] "GET /pub/jeffd/grad_lin.gif HTTP/1.0" 200 3346 morti.eng2.uconn.edu - - [31/Aug/1995:18:43:01 -0400] "GET /pub/job/swim/swim.jpg HTTP/1.0" 200 17401 www-c5.proxy.aol.com - - [31/Aug/1995:18:46:07 -0400] "GET /pub/brendan/suspectm.gif HTTP/1.0" 200 30603 gk-west.usps.gov - - [31/Aug/1995:18:50:34 -0400] "GET /pub/job/vk/vendela2.html HTTP/1.0" 200 8267 newsgw.mentorg.com - - [31/Aug/1995:18:54:26 -0400] "GET /pub/theme/ HTTP/1.0" 200 799 130.68.26.52 - - [31/Aug/1995:18:58:35 -0400] "GET /pub/phil/images/jh61.gif HTTP/1.0" 200 2456 fnugget.intel.com - - [31/Aug/1995:19:01:51 -0400] "GET /pub/job/swim/v-05.jpg HTTP/1.0" 200 2957 ppp04.vestnett.no - - [31/Aug/1995:19:07:06 -0400] "GET /pub/alweiner/new.gif HTTP/1.0" 200 313 128.171.46.72 - - [31/Aug/1995:19:11:55 -0400] "GET /pub/pribut/sb-1-2.gif HTTP/1.0" 200 1253 ad01-026.compuserve.com - - [31/Aug/1995:19:16:09 -0400] "GET /pub/networx/autopage/exotic/ex002_1.gif HTTP/1.0" 200 71366 mall24.tiac.net - - [31/Aug/1995:19:20:26 -0400] "GET /pub/lupine/www/tlk/cast.html HTTP/1.0" 200 3150 129.219.51.118 - - [31/Aug/1995:19:25:07 -0400] "GET /pub/wmcbrine/html/Madonna.html HTTP/1.0" 200 13440 158.68.225.182 - - [31/Aug/1995:19:30:36 -0400] "GET /pub/jellybn/back.gif HTTP/1.0" 200 883 192.43.248.56 - - [31/Aug/1995:19:35:44 -0400] "GET /pub/job/vk/view13.jpg HTTP/1.0" 200 6219 www-e5.proxy.aol.com - - [31/Aug/1995:19:40:26 -0400] "GET /mc-icons/back.gif HTTP/1.0" 200 174 cepc4.swan.ac.uk - - [31/Aug/1995:19:46:43 -0400] "GET /pub/job/vk/vk-page2.jpg HTTP/1.0" 200 10302 dunn.nmt.edu - - [31/Aug/1995:19:51:29 -0400] "GET /pub/peace/VRSBUS.html HTTP/1.0" 200 2093 stimpy.clark.net - - [31/Aug/1995:19:56:53 -0400] "GET /pub/phil/images/jh22.gif HTTP/1.0" 304 - Darcy.islandnet.com - - [31/Aug/1995:20:00:39 -0400] "GET /pub/rmharris/images/clark.gif HTTP/1.0" 200 5823 205.237.236.36 - - [31/Aug/1995:20:05:21 -0400] "GET /atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 med-bcmm-kstar-node.med.yale.edu - - [31/Aug/1995:20:10:55 -0400] "GET /pub/job/vk/view31.jpg HTTP/1.0" 200 5611 cs0.dasd.honeywell.com - - [31/Aug/1995:20:16:16 -0400] "GET /pub/job/vk/view07.jpg HTTP/1.0" 200 7057 ad22-022.compuserve.com - - [31/Aug/1995:20:22:22 -0400] "GET /pub/sshay/images/brwthumb.jpg HTTP/1.0" 200 3306 bway-slip7.dynamic.usit.net - - [31/Aug/1995:20:26:55 -0400] "GET /pub/k2/am4x44u/gifs/yellwpin.gif HTTP/1.0" 304 - 192.100.197.125 - - [31/Aug/1995:20:33:08 -0400] "GET /pub/atomicbk/catalog/orders.gif HTTP/1.0" 200 800 gw3.att.com - - [31/Aug/1995:20:38:08 -0400] "GET /pub/job/vk/vk-bk.jpg HTTP/1.0" 200 4569 204.157.204.210 - - [31/Aug/1995:20:42:22 -0400] "GET /pub/abaa-booknet/images/clark.gif HTTP/1.0" 200 5823 dal04-05.ppp.iadfw.net - - [31/Aug/1995:20:47:31 -0400] "GET /pub/alweiner/a27.gif HTTP/1.0" 200 4594 dialup10.wdbg.va.qnet.com - - [31/Aug/1995:20:54:47 -0400] "GET /printing HTTP/1.0" 302 216 165.152.162.22 - - [31/Aug/1995:21:01:10 -0400] "GET /pub/job/vk/view12.jpg HTTP/1.0" 200 5966 pppcib119.asahi-net.or.jp - - [31/Aug/1995:21:08:08 -0400] "GET /pub/journalism/awesome.htmli HTTP/1.0" 404 318 kuts10p08.cc.ukans.edu - - [31/Aug/1995:21:13:34 -0400] "GET /pub/job/vk/view01.jpg HTTP/1.0" 304 - 199.4.102.43 - - [31/Aug/1995:21:19:30 -0400] "GET /pub/alweiner/spacer.gif HTTP/1.0" 200 810 xenon.chem.ucla.edu - - [31/Aug/1995:21:26:13 -0400] "GET /pub/atomicbk/images/busty.gif HTTP/1.0" 200 73609 205.144.62.141 - - [31/Aug/1995:21:31:36 -0400] "GET /pub/job/swim/v-12.jpg HTTP/1.0" 200 2625 noyce.caere.com - - [31/Aug/1995:21:35:55 -0400] "GET /pub/sshay/images/partner.gif HTTP/1.0" 200 14363 aha401.ccs.itd.umich.edu - - [31/Aug/1995:21:42:03 -0400] "GET /pub/downin/html/backgrounds/iceblue.gif HTTP/1.0" 200 2301 ts1-9.net.netbistro.com - - [31/Aug/1995:21:49:09 -0400] "GET /atomicbk/shocked/shocked.jpg HTTP/1.0" 200 43819 barb.wchat.on.ca - - [31/Aug/1995:21:55:18 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 200 813 cl196042.callutheran.edu - - [31/Aug/1995:22:00:57 -0400] "GET /pub/networx/autopage/dealers/de001_cs.gif HTTP/1.0" 200 1995 vern-a2.ip.realtime.net - - [31/Aug/1995:22:07:27 -0400] "GET /pub/hutchens/chicksquawk.aiff HTTP/1.0" 200 25980 camille.amgen.com - - [31/Aug/1995:22:13:17 -0400] "GET /pub/job/vk/view10.jpg HTTP/1.0" 200 5303 piweba3y.prodigy.com - - [31/Aug/1995:22:18:31 -0400] "GET /pub/k2/am4x44u/gifs/ppcat800.gif HTTP/1.0" 200 2883 dd01-021.compuserve.com - - [31/Aug/1995:22:23:50 -0400] "GET /pub/mrosen/usta.html HTTP/1.0" 200 5192 yorick.umd.edu - - [31/Aug/1995:22:29:20 -0400] "GET /larouche/nigeria.html HTTP/1.0" 200 10303 central17.onramp.net - - [31/Aug/1995:22:35:45 -0400] "GET /pub/robert/home.html HTTP/1.0" 200 1992 BIO.BU.EDU - - [31/Aug/1995:22:42:03 -0400] "GET /pub/job/vk/view33.jpg HTTP/1.0" 200 5903 dpi-gw.ind.dpi.qld.gov.au - - [31/Aug/1995:22:47:56 -0400] "GET /pub/atomicbk/images/taste.gif HTTP/1.0" 200 32694 198.248.98.86 - - [31/Aug/1995:22:53:36 -0400] "GET /pub/howieb/howie2.gif HTTP/1.0" 200 31464 www-a2.proxy.aol.com - - [31/Aug/1995:22:57:57 -0400] "GET /pub/k2/am4x44u/events/adventure/sfnc2.htm HTTP/1.0" 200 3677 dial66.phoenix.net - - [31/Aug/1995:23:02:43 -0400] "GET /pub/sshay/images/narcisis.jpg HTTP/1.0" 200 39523 XR01-A9.citenet.net - - [31/Aug/1995:23:07:43 -0400] "GET /pub/atomicbk/images/tease.gif HTTP/1.0" 200 43516 slip104-85.mn.us.ibm.net - - [31/Aug/1995:23:14:15 -0400] "GET /pub/job/vk/view01.jpg HTTP/1.0" 200 5733 161.69.133.57 - - [31/Aug/1995:23:20:48 -0400] "GET /pub/job/vk/view33.jpg HTTP/1.0" 200 5903 lan104.rocktel.com - - [31/Aug/1995:23:26:05 -0400] "GET /pub/k2/am4x44u/gifs/redball.gif HTTP/1.0" 200 206 www.ottawa.net - - [31/Aug/1995:23:32:20 -0400] "GET /pub/pribut/idea.gif HTTP/1.0" 200 909 156.50.118.36 - - [31/Aug/1995:23:38:54 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 comserv-i-26.usc.edu - - [31/Aug/1995:23:44:47 -0400] "GET /pub/mbrophy/pix/imgsite.jpg HTTP/1.0" 200 11636 dd22-021.compuserve.com - - [31/Aug/1995:23:50:19 -0400] "GET /pub/jeffd/final.html HTTP/1.0" 200 66219 piweba4y.prodigy.com - - [31/Aug/1995:23:56:50 -0400] "GET /pub/k2/am4x44u/gifs/vidlib.gif HTTP/1.0" 200 2138 150.216.65.55 - - [01/Sep/1995:00:02:08 -0400] "GET /pub/job/vk/v-sig.gif HTTP/1.0" 200 4864 sukiyaki.nri.com - - [01/Sep/1995:00:07:25 -0400] "GET /pub/howie/OO/construc.gif HTTP/1.0" 200 252 JH220-D327.Student-Lab.Butler.EDU - - [01/Sep/1995:00:13:06 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 199.74.248.32 - - [01/Sep/1995:00:20:21 -0400] "GET /pub/k2/am4x44u/maps/events.gif HTTP/1.0" 304 - SEC007.spu.edu - - [01/Sep/1995:00:26:37 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 bfld-21.rust.net - - [01/Sep/1995:00:31:27 -0400] "GET /pub/sshay/images/jaseeddy.gif HTTP/1.0" 304 - du139-222.cc.iastate.edu - - [01/Sep/1995:00:36:28 -0400] "GET /pub/job/vk/vk-ad-02.jpg HTTP/1.0" 200 82972 kali.me.ksu.edu - - [01/Sep/1995:00:42:44 -0400] "GET /pub/job/vk/flowers1.gif HTTP/1.0" 200 4288 rvlink.rvlink.com - - [01/Sep/1995:00:48:57 -0400] "GET /pub/k2/am4x44u/gifs/alum1.gif HTTP/1.0" 304 - ad02-032.compuserve.com - - [01/Sep/1995:00:54:44 -0400] "GET /pub/rsjdfg/ProTic.html HTTP/1.0" 200 2453 bitnaut.wat.hookup.net - - [01/Sep/1995:01:02:30 -0400] "GET /pub/k2/am4x44u/maps/truck_tech.gif HTTP/1.0" 200 41160 gondola.ipc.otaru-uc.ac.jp - - [01/Sep/1995:01:08:52 -0400] "GET /pub/lupine/images/tlk/pride-rock.gif HTTP/1.0" 200 51600 labpc40.acs.uci.edu - - [01/Sep/1995:01:15:57 -0400] "GET /pub/kboahene/photoa.gif HTTP/1.0" 200 16963 alfred.uib.no - - [01/Sep/1995:01:21:23 -0400] "GET /pub/job/vk/note.gif HTTP/1.0" 200 9413 massif.helix.net - - [01/Sep/1995:01:26:51 -0400] "GET /pub/alweiner/spacer.gif HTTP/1.0" 304 - dmouton.amgen.com - - [01/Sep/1995:01:34:44 -0400] "GET /pub/atomicbk/catalog/mini.gif HTTP/1.0" 200 360 ad01-027.compuserve.com - - [01/Sep/1995:01:43:18 -0400] "GET /pub/pgarrett/sober.wav HTTP/1.0" 200 60176 PELLEW.NTU.EDU.AU - - [01/Sep/1995:01:52:18 -0400] "GET /pub/networx/cris2.gif HTTP/1.0" 200 8473 pipe3.h1.usa.pipeline.com - - [01/Sep/1995:02:01:11 -0400] "GET /pub/job/vk/view15.jpg HTTP/1.0" 200 6036 firefly.prairienet.org - - [01/Sep/1995:02:10:59 -0400] "GET /pub/sshay/links.html HTTP/1.0" 200 2992 mozart.forest.dnj.ynu.ac.jp - - [01/Sep/1995:02:19:20 -0400] "GET /pub/fan/discog.html HTTP/1.0" 200 3266 cbriggs.interlog.com - - [01/Sep/1995:02:27:04 -0400] "GET /pub/alweiner/eye.gif HTTP/1.0" 304 - 193.13.222.40 - - [01/Sep/1995:02:37:18 -0400] "GET /pub/job/vk/view02.jpg HTTP/1.0" 200 7290 dialup1.qnis.net - - [01/Sep/1995:02:43:32 -0400] "GET /pub/atomicbk/catalog/orders.gif HTTP/1.0" 200 800 nikhefh.nikhef.nl - - [01/Sep/1995:02:52:23 -0400] "GET /pub/wick/photos/chair_mini.gif HTTP/1.0" 200 12589 kema.vxo.telub.se - - [01/Sep/1995:03:00:51 -0400] "GET /pub/atomicbk/catalog/pinup.html HTTP/1.0" 200 8316 net-5.pix.za - - [01/Sep/1995:03:09:36 -0400] "GET /atomicbk/catalog/sleazmag.html HTTP/1.0" 200 6688 ad10-005.compuserve.com - - [01/Sep/1995:03:19:17 -0400] "GET /pub/wsg/html/web/portfoli.html HTTP/1.0" 200 2341 ix-la24-28.ix.netcom.com - - [01/Sep/1995:03:28:58 -0400] "GET /pub/job/vendela.jpg HTTP/1.0" 200 36746 oume-gate.meisei-u.ac.jp - - [01/Sep/1995:03:36:54 -0400] "GET /pub/job/vk/arrow-r.gif HTTP/1.0" 200 3042 Cust45.Max2.San-Francisco.CA.MS.UU.NET - - [01/Sep/1995:03:46:27 -0400] "GET /pub/atomicbk/catalog/orders.gif HTTP/1.0" 200 800 dd05-020.compuserve.com - - [01/Sep/1995:03:56:56 -0400] "GET /pub/networx/autopage/exotic.html HTTP/1.0" 200 1274 proxy.umu.se - - [01/Sep/1995:04:08:54 -0400] "GET /pub/job/mail.jpg HTTP/1.0" 200 4282 199.183.254.10 - - [01/Sep/1995:04:22:41 -0400] "GET /pub/WWIV/PLANETB.GIF HTTP/1.0" 200 5008 gregal.uji.es - - [01/Sep/1995:04:35:08 -0400] "GET /pub/rjgula/ HTTP/1.0" 200 - 192.101.124.43 - - [01/Sep/1995:04:46:33 -0400] "GET /pub/ashp/grafx/phone6.jpg HTTP/1.0" 200 10607 ts900-1021.singnet.com.sg - - [01/Sep/1995:05:00:01 -0400] "GET /atomicbk/images/emboss.jpg HTTP/1.0" 200 4741 161.142.15.180 - - [01/Sep/1995:05:14:15 -0400] "GET /pub/ashp/grafx/globe.gif HTTP/1.0" 403 142 dialup42.geko.com.au - - [01/Sep/1995:05:27:27 -0400] "GET /pub/atomicbk/new/email.gif HTTP/1.0" 200 756 130.99.86.92 - - [01/Sep/1995:05:42:12 -0400] "GET /pub/job/vk/view29.jpg HTTP/1.0" 200 6695 rzsuna.rrze.uni-erlangen.de - - [01/Sep/1995:05:54:46 -0400] "GET /pub/mbrophy/pix/arling.jpg HTTP/1.0" 200 7905 faith.waikato.ac.nz - - [01/Sep/1995:06:07:42 -0400] "GET /pub/zilla/sick.html HTTP/1.0" 200 2359 bombasto.Informatik.RWTH-Aachen.DE - - [01/Sep/1995:06:23:12 -0400] "GET /pub/mbrophy/interviews.html HTTP/1.0" 200 1279 slip36-78.il.us.ibm.net - - [01/Sep/1995:06:39:27 -0400] "GET /pub/wick/otherpages.html HTTP/1.0" 200 1024 miller.cad.cea.fr - - [01/Sep/1995:06:50:33 -0400] "GET /pub/job/vk/note.gif HTTP/1.0" 200 9413 ts900-1201.singnet.com.sg - - [01/Sep/1995:07:03:03 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 145.2.134.83 - - [01/Sep/1995:07:14:31 -0400] "GET /pub/job/vk/view18.jpg HTTP/1.0" 200 6578 164.117.161.206 - - [01/Sep/1995:07:25:36 -0400] "GET /pub/peace/VRSAA.GIF HTTP/1.0" 200 30837 204.101.53.170 - - [01/Sep/1995:07:34:03 -0400] "GET /pub/alweiner/thin_gre.gif HTTP/1.0" 200 78 umav115.verwaltung.uni-mannheim.de - - [01/Sep/1995:07:40:30 -0400] "GET /pub/k2/am4x44u/maps/4x4_home.gif HTTP/1.0" 200 43750 202.252.76.14 - - [01/Sep/1995:07:47:30 -0400] "GET /pub/job/vk/2-page-1.gif HTTP/1.0" 200 2733 aspen.plexus.com - - [01/Sep/1995:07:53:41 -0400] "GET /pub/k2/jeep/jeep.htm HTTP/1.0" 200 1047 lab11.civil.dundee.ac.uk - - [01/Sep/1995:08:02:59 -0400] "GET /pub/job/vk/view23.jpg HTTP/1.0" 200 5165 mersey.hursley.ibm.com - - [01/Sep/1995:08:10:22 -0400] "GET /pub/atomicbk/images/venusinf.gif HTTP/1.0" 200 53308 www-b4.proxy.aol.com - - [01/Sep/1995:08:18:41 -0400] "GET /pub/job/vk/view11.jpg HTTP/1.0" 200 5001 sunsite.nus.sg - - [01/Sep/1995:08:25:03 -0400] "GET /pub/sshay/interact.html HTTP/1.0" 304 - news.ti.com - - [01/Sep/1995:08:30:53 -0400] "GET /pub/job/vk/view27.jpg HTTP/1.0" 200 6611 gatekeeper.ray.com - - [01/Sep/1995:08:37:22 -0400] "GET /pub/stc/www/gifs/home_sm.gif HTTP/1.0" 200 273 csgps1.leeds.ac.uk - - [01/Sep/1995:08:43:29 -0400] "GET /pub/job/vk/view26.jpg HTTP/1.0" 200 6186 kaarna.cc.jyu.fi - - [01/Sep/1995:08:50:15 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 as2.cs.gmr.com - - [01/Sep/1995:08:58:05 -0400] "GET /pub/global/globe.gif HTTP/1.0" 200 703 h-dracula.nr.infi.net - - [01/Sep/1995:09:04:12 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 128.206.21.213 - - [01/Sep/1995:09:09:48 -0400] "GET /pub/abaa-booknet/whatsnew.html HTTP/1.0" 200 13586 info.telenor.no - - [01/Sep/1995:09:16:28 -0400] "GET /pub/job/vk/arrow-r.gif HTTP/1.0" 200 3042 213123.nhn.navair.navy.mil - - [01/Sep/1995:09:21:45 -0400] "GET /pub/peace/PeaceCorps.html HTTP/1.0" 200 510 scrimal.camp.org - - [01/Sep/1995:09:26:51 -0400] "GET /pub/job/vk/view04.jpg HTTP/1.0" 200 4847 cinnamon.spice.com - - [01/Sep/1995:09:30:59 -0400] "GET /pub/alweiner/work/spacer10.gif HTTP/1.0" 200 44 170.142.8.139 - - [01/Sep/1995:09:34:55 -0400] "GET /pub/job/vk/view03.jpg HTTP/1.0" 200 8506 fad4f7fe.fad.okstate.edu - - [01/Sep/1995:09:38:31 -0400] "GET /pub/russadam/bbsbut.gif HTTP/1.0" 304 - ix-stl3-21.ix.netcom.com - - [01/Sep/1995:09:43:01 -0400] "GET /pub/cardman/ask.htm HTTP/1.0" 200 1269 relay.tqs.iunet.it - - [01/Sep/1995:09:47:42 -0400] "GET /pub/kiaman/sound03.gif HTTP/1.0" 200 1201 www-b6.proxy.aol.com - - [01/Sep/1995:09:52:42 -0400] "GET /pub/job/vk/arrow-l.gif HTTP/1.0" 200 3031 193.131.192.35 - - [01/Sep/1995:09:57:45 -0400] "GET /atomicbk/new/logo2.gif HTTP/1.0" 200 12871 box255.labs.cis.pitt.edu - - [01/Sep/1995:10:03:21 -0400] "GET /pub/job/vk/view03.jpg HTTP/1.0" 200 8506 ppp-hck-1-17.ios.com - - [01/Sep/1995:10:08:15 -0400] "GET /pub/wine/portbtn.gif HTTP/1.0" 200 764 sabre21.sasknet.sk.ca - - [01/Sep/1995:10:12:47 -0400] "GET /pub/rsjdfg HTTP/1.0" 302 218 gatekeeper.tv2.no - - [01/Sep/1995:10:17:57 -0400] "GET /pub/atomicbk/catalog/mini.gif HTTP/1.0" 200 360 147.134.225.168 - - [01/Sep/1995:10:22:49 -0400] "GET /pub/mbrophy/pix/interv.jpg HTTP/1.0" 200 5047 156.40.174.153 - - [01/Sep/1995:10:26:00 -0400] "GET /pub/cybersoft/graphics/colorbar.gif HTTP/1.0" 200 479 mhoolboo.worldbank.org - - [01/Sep/1995:10:29:24 -0400] "GET /infouser/logsnap.txt HTTP/1.0" 304 - Gayle-Gaston.tenet.edu - - [01/Sep/1995:10:33:11 -0400] "GET /pub/robert/home.html HTTP/1.0" 200 1992 usr12-dialup34.Atlanta.mci.net - - [01/Sep/1995:10:37:42 -0400] "GET /pub/journalism/awesome.html HTTP/1.0" 200 11610 exlsun.epm.ornl.gov - - [01/Sep/1995:10:41:38 -0400] "GET /pub/job/mail.jpg HTTP/1.0" 200 4282 gator01.iit.edu - - [01/Sep/1995:10:45:19 -0400] "GET /pub/networx/autopage/dealers/de001_3.html HTTP/1.0" 200 541 146.186.65.26 - - [01/Sep/1995:10:49:42 -0400] "GET /pub/lupine/www/images/icons/note.gif HTTP/1.0" 200 565 alweiner.clark.net - - [01/Sep/1995:10:53:44 -0400] "GET /pub/alweiner/reddot.gif HTTP/1.0" 200 886 dd24-003.compuserve.com - - [01/Sep/1995:10:58:08 -0400] "GET /atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 146.13.185.67 - - [01/Sep/1995:11:01:55 -0400] "GET /pub/mbrophy/pix/sbpromo.jpg HTTP/1.0" 200 14185 p2.ts1.walrus.com - - [01/Sep/1995:11:06:37 -0400] "GET /pub/mpowers/j4j/web/ HTTP/1.0" 200 1735 NUBS35.ccs.itd.umich.edu - - [01/Sep/1995:11:10:42 -0400] "GET /pub/lupine/www/images/icons/roundbtn.gif HTTP/1.0" 200 1077 fad4f7fe.fad.okstate.edu - - [01/Sep/1995:11:14:46 -0400] "GET /pub/russadam/tblite.gif HTTP/1.0" 200 8066 watt.oedison.com - - [01/Sep/1995:11:18:23 -0400] "GET /pub/pribut/backgrou.gif HTTP/1.0" 304 - rm003a63.brookes.ac.uk - - [01/Sep/1995:11:22:08 -0400] "GET /pub/job/theman.jpg HTTP/1.0" 200 8905 slip101.gamemaster.qc.ca - - [01/Sep/1995:11:26:32 -0400] "GET /pub/mbrophy/pix/louie.jpg HTTP/1.0" 200 4544 gsi32.gestalt-sys.com - - [01/Sep/1995:11:30:04 -0400] "GET /pub/cargui/btn_prev.gif HTTP/1.0" 304 - www-c3.proxy.aol.com - - [01/Sep/1995:11:33:50 -0400] "GET /pub/job/swim/v-13.jpg HTTP/1.0" 200 2985 bus2_2.bus.misu.NoDak.edu - - [01/Sep/1995:11:38:16 -0400] "GET /pub/cargui/links.html HTTP/1.0" 200 6417 PCEAS4.SAS.UPENN.EDU - - [01/Sep/1995:11:41:47 -0400] "GET /atomicbk/catalog/home.gif HTTP/1.0" 200 813 jstone.clark.net - - [01/Sep/1995:11:45:44 -0400] "GET /pub/gen/mswg/seawolf/ HTTP/1.0" 304 - iglou2.iglou.com - - [01/Sep/1995:11:49:35 -0400] "GET /pub/macgyver/VidGames/Vidgames.html HTTP/1.0" 200 29810 199.78.230.18 - - [01/Sep/1995:11:53:41 -0400] "GET /pub/theme/cgi-bin/ign.html/l/Frankie Ybanez/n/citationx/ HTTP/1.0" 200 588 mempc103.unibe.ch - - [01/Sep/1995:11:57:47 -0400] "GET /pub/pribut/email2.gif HTTP/1.0" 200 1209 wph-pc11.usc.edu - - [01/Sep/1995:12:02:56 -0400] "GET /pub/rsjdfg HTTP/1.0" 302 218 dial-30.win.net - - [01/Sep/1995:12:07:11 -0400] "GET /pub/ibos/watkins/hands4.gif HTTP/1.0" 200 22770 mersey.hursley.ibm.com - - [01/Sep/1995:12:10:33 -0400] "GET /atomicbk/scotth.gif HTTP/1.0" 200 790 hamlet.fokus.gmd.de - - [01/Sep/1995:12:13:28 -0400] "GET /pub/job/vk/view07.jpg HTTP/1.0" 200 7057 hucomgw.hucom.co.jp - - [01/Sep/1995:12:17:03 -0400] "GET /pub/job/vk/view15.jpg HTTP/1.0" 200 6036 macbeth.umd.edu - - [01/Sep/1995:12:21:03 -0400] "GET /html/phones.html HTTP/1.0" 200 8840 165.247.243.4 - - [01/Sep/1995:12:25:32 -0400] "GET /pub/job/swim/ss-10.jpg HTTP/1.0" 200 74315 cherubim.rp.org - - [01/Sep/1995:12:29:47 -0400] "GET /pub/atomicbk/contest.gif HTTP/1.0" 200 669 ppptky132.asahi-net.or.jp - - [01/Sep/1995:12:33:38 -0400] "GET /pub/sinkers/signicon.gif HTTP/1.0" 200 4911 butchp2.si.univ-compiegne.fr - - [01/Sep/1995:12:37:32 -0400] "GET /pub/atomicbk/home.html HTTP/1.0" 200 4051 DTNET43-170.dt.navy.mil - - [01/Sep/1995:12:41:24 -0400] "GET /pub/job/vk/2-page-2.gif HTTP/1.0" 200 2527 galileo.thp.univie.ac.at - - [01/Sep/1995:12:45:09 -0400] "GET /pub/jeffd/header.gif HTTP/1.0" 200 24376 arh_gate-pm.winternet.com - - [01/Sep/1995:12:48:47 -0400] "GET /atomicbk/images/atomgirl.jpg HTTP/1.0" 200 34164 ts5-03.InfoRamp.Net - - [01/Sep/1995:12:52:06 -0400] "GET /pub/nractive/images/wbtic.gif HTTP/1.0" 200 15873 141.233.134.22 - - [01/Sep/1995:12:54:57 -0400] "GET /pub/listserv/yellowli.gif HTTP/1.0" 200 905 hamlet.fokus.gmd.de - - [01/Sep/1995:12:58:54 -0400] "GET /pub/job/vk/view26.jpg HTTP/1.0" 200 6186 128.95.51.215 - - [01/Sep/1995:13:02:42 -0400] "GET /pub/lupine/images/tlk/tlk-card.gif HTTP/1.0" 200 15142 www-b3.proxy.aol.com - - [01/Sep/1995:13:07:03 -0400] "GET /pub/journalism/awesome.html HTTP/1.0" 200 11610 192.100.197.102 - - [01/Sep/1995:13:10:19 -0400] "GET /pub/k2/am4x44u/truck_stop/truck_stop.htm HTTP/1.0" 200 1467 pc.chem.sc.edu - - [01/Sep/1995:13:14:24 -0400] "GET /pub/theme/demockracy/thanks.html HTTP/1.0" 200 497 corrigan-ppp.clark.net - - [01/Sep/1995:13:17:18 -0400] "GET /pub/corrigan/czech.gif HTTP/1.0" 304 - unknown-181.soe.umich.edu - - [01/Sep/1995:13:20:36 -0400] "GET /tara/sacpl10k.htm HTTP/1.0" 200 800 proxy.austin.ibm.com - - [01/Sep/1995:13:24:44 -0400] "GET /pub/heroes/wldtrade.html HTTP/1.0" 200 1342 ts900-1408.singnet.com.sg - - [01/Sep/1995:13:28:36 -0400] "GET /pub/sshay/interact.html HTTP/1.0" 200 2541 129.174.32.36 - - [01/Sep/1995:13:31:38 -0400] "GET /pub/jrinker/cbutun1.gif HTTP/1.0" 200 2638 ftp-relay.mv.us.adobe.com - - [01/Sep/1995:13:35:11 -0400] "GET /pub/job/vk/view34.jpg HTTP/1.0" 200 5156 rajeesh.WPI.EDU - - [01/Sep/1995:13:38:44 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 9084 loon.house.leg.state.mn.us - - [01/Sep/1995:13:42:51 -0400] "GET /pub/job/vendela.jpg HTTP/1.0" 200 36746 mac3141.affymax.com - - [01/Sep/1995:13:47:34 -0400] "GET /pub/k2/am4x44u/gifs/redball.gif HTTP/1.0" 200 206 r14s2.nswc.navy.mil - - [01/Sep/1995:13:51:51 -0400] "GET /pub/job/vk/view12.jpg HTTP/1.0" 200 5966 van11313.direct.ca - - [01/Sep/1995:13:55:09 -0400] "GET /pub/rsjdfg/Pictures/Heat.jpg HTTP/1.0" 304 - imperial.rp1.sdl.usu.edu - - [01/Sep/1995:13:58:36 -0400] "GET /pub/k2/am4x44u/gifs/ttiplin2.gif HTTP/1.0" 200 2958 192.5.63.93 - - [01/Sep/1995:14:01:22 -0400] "GET /pub/atomicbk/images/adbaby.gif HTTP/1.0" 200 72807 pc1419.utdallas.edu - - [01/Sep/1995:14:04:31 -0400] "GET /pub/kiaman/or_ball.gif HTTP/1.0" 200 967 asm4-3.sl115.cns.vt.edu - - [01/Sep/1995:14:08:25 -0400] "GET /pub/sshay/images/brwthumb.jpg HTTP/1.0" 200 3306 198.29.27.10 - - [01/Sep/1995:14:13:10 -0400] "GET /pub/mackall/home.html HTTP/1.0" 200 331 152.120.105.199 - - [01/Sep/1995:14:16:41 -0400] "GET /pub/tmphoto/rmd.homicide.pages/homicide.3.rmd.gif HTTP/1.0" 200 28603 hamlet.fokus.gmd.de - - [01/Sep/1995:14:19:15 -0400] "GET /pub/job/vk/view16.jpg HTTP/1.0" 200 6808 piweba5y.prodigy.com - - [01/Sep/1995:14:22:02 -0400] "GET /pub/lschank/web/census.html HTTP/1.0" 200 13912 mercury.image.ctt.com - - [01/Sep/1995:14:25:04 -0400] "GET /nrlc/btn1.gif HTTP/1.0" 200 407 europa.sunlab.ece.usu.edu - - [01/Sep/1995:14:27:25 -0400] "GET /pub/job/vk/view15.jpg HTTP/1.0" 200 6036 ecn1521925.gsfc.nasa.gov - - [01/Sep/1995:14:30:33 -0400] "GET /pub/jeffd/back.gif HTTP/1.0" 404 331 gateway.senate.gov - - [01/Sep/1995:14:33:32 -0400] "GET /pub/jeffd/unchkbox.gif HTTP/1.0" 200 855 stsim.interax.net - - [01/Sep/1995:14:38:09 -0400] "GET /pub/listserv/listserv.html HTTP/1.0" 200 1138 dts.rrddts.donnelley.com - - [01/Sep/1995:14:41:44 -0400] "GET /pub/shelby/yellowli.gif HTTP/1.0" 200 905 AL207H1.MNSFLD.EDU - - [01/Sep/1995:14:46:47 -0400] "GET /pub/lschank/web/new.gif HTTP/1.0" 304 - 199.234.151.14.du.nauticom.net - - [01/Sep/1995:14:51:02 -0400] "GET /pub/cardman/1vote.gif HTTP/1.0" 200 1284 store.lutz.com - - [01/Sep/1995:14:54:05 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 200 813 bader.moc.kw - - [01/Sep/1995:14:57:35 -0400] "GET /pub/jeffd/header.gif HTTP/1.0" 200 24376 pentium22.ee.umr.edu - - [01/Sep/1995:15:01:11 -0400] "GET /pub/k2/am4x44u/gifs/clwild.gif HTTP/1.0" 200 1227 hickory.cad.clarkson.edu - - [01/Sep/1995:15:04:26 -0400] "GET /pub/atomicbk/catalog/orders.gif HTTP/1.0" 200 800 nmpch.nokia.com - - [01/Sep/1995:15:07:47 -0400] "GET /pub/job/vk/view02.jpg HTTP/1.0" 200 7290 dobrien.clark.net - - [01/Sep/1995:15:10:43 -0400] "GET /pub/sshay/images/rgblack.jpg HTTP/1.0" 304 - e659229.boeing.com - - [01/Sep/1995:15:14:02 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 9084 hpux.nis.za - - [01/Sep/1995:15:17:38 -0400] "GET /pub/k2/cgi-bin/imagemap/4x4_home?77,85 HTTP/1.0" 200 1298 law-pc28.law.utexas.edu - - [01/Sep/1995:15:21:11 -0400] "GET /pub/peace/casset.gif HTTP/1.0" 200 1071 eels11.ee.surrey.ac.uk - - [01/Sep/1995:15:24:26 -0400] "GET /pub/job/vk/2-page-1.gif HTTP/1.0" 200 2733 151.186.72.43 - - [01/Sep/1995:15:26:54 -0400] "GET /pub/job/theman.jpg HTTP/1.0" 200 8905 gateway.TPP.com - - [01/Sep/1995:15:29:25 -0400] "GET /pub/job/swim/v-11.jpg HTTP/1.0" 200 2419 server03.zrz.TU-Berlin.DE - - [01/Sep/1995:15:32:50 -0400] "GET /pub/heroes/gifs/language.gif HTTP/1.0" 200 29097 pme000.awinc.com - - [01/Sep/1995:15:35:34 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 r1001326.mayo.edu - - [01/Sep/1995:15:39:02 -0400] "GET /pub/atomicbk/artgal.gif HTTP/1.0" 200 823 apci.com - - [01/Sep/1995:15:42:19 -0400] "GET /pub/watc/gifs/LINE.GIF HTTP/1.0" 404 308 128.206.98.205 - - [01/Sep/1995:15:45:28 -0400] "GET /pub/lupine/www/images/icons/paint.gif HTTP/1.0" 200 516 157.136.21.194 - - [01/Sep/1995:15:48:18 -0400] "GET /pub/job/vk/view04.jpg HTTP/1.0" 200 4847 166.41.232.99 - - [01/Sep/1995:15:51:48 -0400] "GET /pub/mmathai/mls/images/halfmls.gif HTTP/1.0" 200 5483 jlehett.iag.net - - [01/Sep/1995:15:55:29 -0400] "GET /pub/pwalker/ball.gif HTTP/1.0" 200 953 engn-6.olivet.edu - - [01/Sep/1995:15:59:35 -0400] "GET /atomicbk/logo2.gif HTTP/1.0" 200 12871 ppp08-17.rns.tamu.edu - - [01/Sep/1995:16:03:42 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 200 813 128.231.232.101 - - [01/Sep/1995:16:06:24 -0400] "GET /pub/wine/sauterne.gif HTTP/1.0" 200 16992 132.235.105.46 - - [01/Sep/1995:16:09:31 -0400] "GET /pub/sshay/images/tyusvest.jpg HTTP/1.0" 200 26791 atom.terranet.com - - [01/Sep/1995:16:13:53 -0400] "GET /pub/jgbustam/gifs/at-work.gif HTTP/1.0" 200 373 128.160.55.8 - - [01/Sep/1995:16:17:45 -0400] "GET /pub/job/vk/view11.jpg HTTP/1.0" 200 5001 132.236.113.193 - - [01/Sep/1995:16:21:32 -0400] "GET /pub/rsjdfg/Pictures/AmericanMan.jpg HTTP/1.0" 200 44118 comp.uark.edu - - [01/Sep/1995:16:24:23 -0400] "GET /pub/job/vk/view04.jpg HTTP/1.0" 200 4847 ix-rea-pa1-22.ix.netcom.com - - [01/Sep/1995:16:27:15 -0400] "GET /atomicbk/catalog/catalog.html HTTP/1.0" 200 4654 137.91.85.202 - - [01/Sep/1995:16:31:18 -0400] "GET /pub/fervor/meet.htm HTTP/1.0" 200 787 ptpm002.olympus.net - - [01/Sep/1995:16:34:55 -0400] "GET /pub/mharding/Ragedoc23.GIF HTTP/1.0" 200 32467 139.87.20.145 - - [01/Sep/1995:16:39:06 -0400] "GET /pub/alweiner/home.gif HTTP/1.0" 200 372 directweb.com - - [01/Sep/1995:16:42:01 -0400] "GET /pub/pgarrett/redball.gif HTTP/1.0" 200 925 csmac1.ucsf.EDU - - [01/Sep/1995:16:45:38 -0400] "GET /atomicbk/new/orders.gif HTTP/1.0" 200 800 tnygreen.remote.Princeton.EDU - - [01/Sep/1995:16:48:51 -0400] "GET /pub/jeffd/back.gif HTTP/1.0" 404 207 www-b5.proxy.aol.com - - [01/Sep/1995:16:52:00 -0400] "GET /pub/k2/am4x44u/gifs/ambut.gif HTTP/1.0" 200 213 140.175.22.107 - - [01/Sep/1995:16:55:25 -0400] "GET /pub/jeffd/spkrnewt.gif HTTP/1.0" 200 45247 163.185.21.182 - - [01/Sep/1995:16:58:58 -0400] "GET /pub/job/vk/2-page-1.gif HTTP/1.0" 200 2733 sabre20.sasknet.sk.ca - - [01/Sep/1995:17:02:47 -0400] "GET /pub/job/vk/view07.jpg HTTP/1.0" 200 7057 137.240.150.204 - - [01/Sep/1995:17:06:31 -0400] "GET /pub/k2/am4x44u/trails/books/books.htm HTTP/1.0" 200 298 chrivb01.cch.com - - [01/Sep/1995:17:10:00 -0400] "GET /pub/murple/drugs.html HTTP/1.0" 404 329 prpayne-ppp.clark.net - - [01/Sep/1995:17:13:59 -0400] "GET /pub/show/usboat2s.gif HTTP/1.0" 304 - 134.121.164.107 - - [01/Sep/1995:17:17:21 -0400] "GET /pub/job/vk/2-page-2.gif HTTP/1.0" 200 2527 167.83.197.20 - - [01/Sep/1995:17:21:13 -0400] "GET /pub/macgyver/VidGames/gifs/supernes.gif HTTP/1.0" 200 15329 161.31.66.37 - - [01/Sep/1995:17:26:04 -0400] "GET /pub/rmharris/catalogs/detercat/17w-2.html HTTP/1.0" 200 22941 alfa.ist.utl.pt - - [01/Sep/1995:17:30:03 -0400] "GET /pub/micros/mnet.html HTTP/1.0" 200 1309 gjc.delphi.com - - [01/Sep/1995:17:33:21 -0400] "GET /pub/alweiner/work/_01711T.gif HTTP/1.0" 200 77 www-d2.proxy.aol.com - - [01/Sep/1995:17:36:29 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 304 - homer.SAIC.COM - - [01/Sep/1995:17:39:49 -0400] "GET /pub/job/vk/view04.jpg HTTP/1.0" 200 4847 atomicbk-ppp.clark.net - - [01/Sep/1995:17:43:22 -0400] "GET /graphics/wsglogo.gif HTTP/1.0" 200 4586 www-e6.proxy.aol.com - - [01/Sep/1995:17:48:42 -0400] "GET /pub/jgbustam/famosos/famei.gif HTTP/1.0" 304 - nana.eunet.no - - [01/Sep/1995:17:52:52 -0400] "GET /pub/sshay/images/swet.gif HTTP/1.0" 200 43341 204.249.224.107 - - [01/Sep/1995:17:56:11 -0400] "GET /pub/atomicbk/catalog/adultcom.html HTTP/1.0" 200 16578 128.149.51.44 - - [01/Sep/1995:18:00:21 -0400] "GET /pub/lupine/www/images/lines/leaves.gif HTTP/1.0" 200 4044 lrl0.lis.uiuc.edu - - [01/Sep/1995:18:05:05 -0400] "GET /larouche/larouche_welcome.gif HTTP/1.0" 200 16362 tmrotek.bevc.blacksburg.va.us - - [01/Sep/1995:18:08:38 -0400] "GET /pub/k2/am4x44u/4x4.htm HTTP/1.0" 200 2788 disarray.demon.co.uk - - [01/Sep/1995:18:13:12 -0400] "GET /pub/hutchens/budgies.html HTTP/1.0" 200 1617 sahp315.sandia.gov - - [01/Sep/1995:18:17:42 -0400] "GET /pub/rsjdfg/Images/5.gif HTTP/1.0" 200 2321 198.109.136.6 - - [01/Sep/1995:18:22:26 -0400] "GET /pub/sshay/images/scotswet.jpg HTTP/1.0" 200 58027 gamma.cleaf.com - - [01/Sep/1995:18:27:09 -0400] "GET /atomicbk/catalog/drugs.html HTTP/1.0" 200 23006 199.77.68.5 - - [01/Sep/1995:18:32:03 -0400] "GET /pub/sshay/images/nrcthumb.jpg HTTP/1.0" 200 4215 piweba3y.prodigy.com - - [01/Sep/1995:18:36:36 -0400] "GET /pub/jeffd/cyberrt.gif HTTP/1.0" 304 - poster.ptc.com - - [01/Sep/1995:18:41:03 -0400] "GET /atomicbk/promo.gif HTTP/1.0" 200 849 GSB-MLonergan.Stanford.EDU - - [01/Sep/1995:18:44:42 -0400] "GET /pub/aztec/mariner/marnet.html HTTP/1.0" 200 9787 Cal046061.student.utwente.nl - - [01/Sep/1995:18:49:24 -0400] "GET /pub/job/vk/view19.jpg HTTP/1.0" 200 3915 georgepc.tsb-intl.ca - - [01/Sep/1995:18:53:09 -0400] "GET /pub/atomicbk/catalog/catalog.html HTTP/1.0" 200 4654 atomicbk-ppp.clark.net - - [01/Sep/1995:18:56:42 -0400] "GET /pub/atomicbk/home.html HTTP/1.0" 304 - piweba5y.prodigy.com - - [01/Sep/1995:19:00:51 -0400] "GET /pub/sshay/images/tywetsut.jpg HTTP/1.0" 200 21652 agric-bo.srv.gov.ab.ca - - [01/Sep/1995:19:05:24 -0400] "GET /pub/atomicbk/direct.gif HTTP/1.0" 304 - richardr.nursing.arizona.edu - - [01/Sep/1995:19:09:53 -0400] "GET /atomicbk/images/atomgirl.jpg HTTP/1.0" 200 34164 davidt_pc.orem.novell.com - - [01/Sep/1995:19:14:14 -0400] "GET /pub/job/vk/view16.jpg HTTP/1.0" 200 6808 van13367.direct.ca - - [01/Sep/1995:19:18:17 -0400] "GET /atomicbk/catalog/home.gif HTTP/1.0" 200 813 msiu.res.WPI.EDU - - [01/Sep/1995:19:22:55 -0400] "GET /pub/job/vk/v-line.gif HTTP/1.0" 200 1254 visitor1.Berkeley.EDU - - [01/Sep/1995:19:28:08 -0400] "GET /pub/journalism/awesome.html HTTP/1.0" 200 11610 www-b1.proxy.aol.com - - [01/Sep/1995:19:33:03 -0400] "GET /pub/lupine/www/images/icons/news.gif HTTP/1.0" 200 1202 giaeb.cc.monash.edu.au - - [01/Sep/1995:19:37:43 -0400] "GET /pub/sshay/images/hardtodd.jpg HTTP/1.0" 200 39303 www-b2.proxy.aol.com - - [01/Sep/1995:19:42:32 -0400] "GET /atomicbk/home.gif HTTP/1.0" 200 813 204.243.29.199 - - [01/Sep/1995:19:46:59 -0400] "GET /pub/murple/drugs.html HTTP/1.0" 404 313 pasyn-46.rice.edu - - [01/Sep/1995:19:51:52 -0400] "GET /pub/job/vk/view16.jpg HTTP/1.0" 200 6808 ad22-020.compuserve.com - - [01/Sep/1995:19:58:11 -0400] "GET /pub/mbrophy/the_scoop.html HTTP/1.0" 200 390 sco.clark.net - - [01/Sep/1995:20:02:34 -0400] "GET /pub/sco/blue.gif HTTP/1.0" 304 - bigred.ncsa.uiuc.edu - - [01/Sep/1995:20:07:41 -0400] "GET /pub/howie/new.gif HTTP/1.0" 200 116 cmp8.ucr.edu - - [01/Sep/1995:20:13:43 -0400] "GET /atomicbk/catalog/catalog.html HTTP/1.0" 200 4654 internal003144.NeXT.COM - - [01/Sep/1995:20:18:39 -0400] "GET /atomicbk/catalog/mini.gif HTTP/1.0" 200 360 inet1.nsc.com - - [01/Sep/1995:20:24:44 -0400] "GET /pub/cargui/btn_home.gif HTTP/1.0" 304 - www-c1.proxy.aol.com - - [01/Sep/1995:20:30:53 -0400] "GET /pub/sshay/images/chris1.jpg HTTP/1.0" 200 13306 benjy.lvision.com - - [01/Sep/1995:20:36:14 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 200 813 ns1.faseb.org - - [01/Sep/1995:20:41:22 -0400] "GET /pub/journalism/awesome.html HTTP/1.0" 304 - ppp1.itw.com - - [01/Sep/1995:20:49:16 -0400] "GET /atomicbk/direct.gif HTTP/1.0" 200 833 larmor.newton.cam.ac.uk - - [01/Sep/1995:20:54:47 -0400] "GET /pub/job/swim/v-12.jpg HTTP/1.0" 200 2625 www-b4.proxy.aol.com - - [01/Sep/1995:21:00:22 -0400] "GET /pub/sshay/images/justin.jpg HTTP/1.0" 304 - 198.49.169.79 - - [01/Sep/1995:21:06:09 -0400] "GET /pub/atomicbk/catalog/mini.gif HTTP/1.0" 200 360 204.245.156.1 - - [01/Sep/1995:21:11:47 -0400] "GET /pub/evins/Icons/Nameplates/chrome-people.gif HTTP/1.0" 200 2120 acops.cat.csiro.au - - [01/Sep/1995:21:17:01 -0400] "GET /pub/sshay/images/chrsbutn.gif HTTP/1.0" 200 1070 krlee.demon.co.uk - - [01/Sep/1995:21:21:50 -0400] "GET /pub/job/vk/view01.jpg HTTP/1.0" 200 5733 138.112.195.30 - - [01/Sep/1995:21:28:09 -0400] "GET /pub/gen/fas/asmp/asm30.html HTTP/1.0" 200 44652 asm4-3.sl114.cns.vt.edu - - [01/Sep/1995:21:32:56 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 line107.nwm.mindlink.net - - [01/Sep/1995:21:39:05 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 philosophers.execpc.com - - [01/Sep/1995:21:44:36 -0400] "GET /pub/heroes/gifs/sheen.gif HTTP/1.0" 200 10161 cmlds5.ME.Berkeley.EDU - - [01/Sep/1995:21:50:51 -0400] "GET /pub/sshay/images/rainlin2.gif HTTP/1.0" 200 1229 Cust21.Max5.New-York.NY.MS.UU.NET - - [01/Sep/1995:21:56:34 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 ndlc.occ.uky.edu - - [01/Sep/1995:22:02:04 -0400] "GET /pub/jeffd/smreagan.gif HTTP/1.0" 200 18018 sbpc8.rdg.ac.uk - - [01/Sep/1995:22:07:14 -0400] "GET /pub/sshay/partbutn.map?261,20 HTTP/1.0" 302 229 pm2e1-30.valleynet.com - - [01/Sep/1995:22:12:19 -0400] "GET /atomicbk/images/buffy.gif HTTP/1.0" 200 59956 MUA44.MARSHALL.EDU - - [01/Sep/1995:22:17:10 -0400] "GET /pub/sshay/images/links.gif HTTP/1.0" 200 6171 dialup44.Washington.mci.net - - [01/Sep/1995:22:24:03 -0400] "GET /pub/job/vk/flowers1.gif HTTP/1.0" 200 4288 www-b4.proxy.aol.com - - [01/Sep/1995:22:29:48 -0400] "GET /atomicbk/logo2.gif HTTP/1.0" 200 12871 www-e9.proxy.aol.com - - [01/Sep/1995:22:35:55 -0400] "GET /pub/rebel/stuff.gif HTTP/1.0" 200 1589 custer.umt.edu - - [01/Sep/1995:22:41:48 -0400] "GET /pub/jeffd/liealot.html HTTP/1.0" 200 3742 mmfw.macromedia.com - - [01/Sep/1995:22:48:06 -0400] "GET /pub/sshay/images/bksh.gif HTTP/1.0" 200 13112 140.107.22.15 - - [01/Sep/1995:22:55:03 -0400] "GET /pub/sshay/images/rainlin2.gif HTTP/1.0" 200 1229 ix-aug-ga1-01.ix.netcom.com - - [01/Sep/1995:23:00:26 -0400] "GET /pub/job/swim/v-05.jpg HTTP/1.0" 200 2957 128.220.62.147 - - [01/Sep/1995:23:08:06 -0400] "GET /pub/job/theman.jpg HTTP/1.0" 200 8905 sabre3.sasknet.sk.ca - - [01/Sep/1995:23:13:52 -0400] "GET /atomicbk/scotth.gif HTTP/1.0" 200 790 kfps-Old-Union-A-dynamic-37.Stanford.EDU - - [01/Sep/1995:23:19:37 -0400] "GET /pub/sshay/images/greysped.gif HTTP/1.0" 200 4542 drjo002A089.embratel.net.br - - [01/Sep/1995:23:26:35 -0400] "GET /atomicbk/catalog/emboss.jpg HTTP/1.0" 200 4741 piweba3y.prodigy.com - - [01/Sep/1995:23:33:55 -0400] "GET /aztec/iso9000/mant.gif HTTP/1.0" 304 - ip150.vivanet.com - - [01/Sep/1995:23:40:34 -0400] "GET /pub/jeffd/plnlhedr.gif HTTP/1.0" 200 9445 learn102.gestalt-sys.com - - [01/Sep/1995:23:48:14 -0400] "GET /pub/cargui/cool_til.gif HTTP/1.0" 304 - LIBRARY4.HOSP.UTK.EDU - - [01/Sep/1995:23:56:47 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 dialin-192.wustl.edu - - [02/Sep/1995:00:04:47 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 202.21.16.120 - - [02/Sep/1995:00:11:41 -0400] "GET /pub/sshay/images/chris3.jpg HTTP/1.0" 200 50739 205.177.33.4 - - [02/Sep/1995:00:17:52 -0400] "GET /pub/alweiner/cgi-bin/homepage.html?alive HTTP/1.0" 200 469 Fleming-John.collins.indiana.edu - - [02/Sep/1995:00:24:57 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 www-b6.proxy.aol.com - - [02/Sep/1995:00:33:31 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 ix-tam4-25.ix.netcom.com - - [02/Sep/1995:00:41:29 -0400] "GET /pub/job/vk/view04.jpg HTTP/1.0" 200 4847 mandyk.pr.mcs.net - - [02/Sep/1995:00:46:36 -0400] "GET /pub/abaa-booknet/research/librar.html HTTP/1.0" 200 5339 ad05-027.compuserve.com - - [02/Sep/1995:00:55:37 -0400] "GET /pub/listserv/lsrel1b.html HTTP/1.0" 200 21120 Cust45.Max1.New-Orleans.LA.MS.UU.NET - - [02/Sep/1995:01:04:14 -0400] "GET /pub/atomicbk/iamges/erotuniv.gif HTTP/1.0" 404 335 kierkegaard.helios.nd.edu - - [02/Sep/1995:01:11:10 -0400] "GET /pub/job/vk/view11.jpg HTTP/1.0" 200 5001 port4.den1-annex.usa.net - - [02/Sep/1995:01:18:21 -0400] "GET /atomicbk/new/new.html HTTP/1.0" 200 3025 bootp-232-84.bootp.Virginia.EDU - - [02/Sep/1995:01:26:33 -0400] "GET /pub/job/vk/view09.jpg HTTP/1.0" 200 6011 Clam.Stanford.EDU - - [02/Sep/1995:01:37:51 -0400] "GET /devtools/gonew.gif HTTP/1.0" 200 1624 ppp52-148.gol.com - - [02/Sep/1995:01:47:32 -0400] "GET /atomicbk/catalog/home.gif HTTP/1.0" 200 813 peterd.midcoast.com.au - - [02/Sep/1995:01:57:41 -0400] "GET /pub/journalism/awesome.html HTTP/1.0" 200 11610 rmassoc.seanet.com - - [02/Sep/1995:02:05:57 -0400] "GET /pub/phil/images/WinonaSheets-little.gif HTTP/1.0" 200 33454 abba.ccs.neu.edu - - [02/Sep/1995:02:13:27 -0400] "GET /pub/sshay/images/hardtodd.jpg HTTP/1.0" 200 39303 net-1-224.eden.com - - [02/Sep/1995:02:21:59 -0400] "GET /pub/job/vendela.jpg HTTP/1.0" 200 36746 167.216.5.63 - - [02/Sep/1995:02:29:44 -0400] "GET /pub/alweiner/work/_0271T.gif HTTP/1.0" 200 813 daalia.vtkk.fi - - [02/Sep/1995:02:36:07 -0400] "GET /pub/job/swim/v-19.jpg HTTP/1.0" 200 2912 204.166.233.151 - - [02/Sep/1995:02:43:44 -0400] "GET /pub/cargui/img2002.gif HTTP/1.0" 200 18827 buchanan31.res.iastate.edu - - [02/Sep/1995:02:54:38 -0400] "GET /pub/atomicbk/images/earlyero.gif HTTP/1.0" 404 335 flotsam.ee.pdx.edu - - [02/Sep/1995:03:06:59 -0400] "GET /pub/atomicbk/images/nudistbk.gif HTTP/1.0" 404 335 netcom14.netcom.com - - [02/Sep/1995:03:18:42 -0400] "GET /pub/robert/home.html HTTP/1.0" 200 1992 155.142.200.1 - - [02/Sep/1995:03:32:55 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 briant.reslife.okstate.edu - - [02/Sep/1995:03:45:13 -0400] "GET /pub/job/swim/v-14.jpg HTTP/1.0" 200 2615 gn2.getnet.com - - [02/Sep/1995:03:57:04 -0400] "GET /pub/job/vk/view03.jpg HTTP/1.0" 200 8506 ccs.sogang.ac.kr - - [02/Sep/1995:04:12:46 -0400] "GET /pub/wmcbrine/html/wmcbrine.html HTTP/1.0" 200 538 emf2-208.emf.net - - [02/Sep/1995:04:26:17 -0400] "GET /pub/atomicbk/new.gif HTTP/1.0" 200 744 n00122-104afc.unity.ncsu.edu - - [02/Sep/1995:04:42:13 -0400] "GET /pub/sshay/images/scott1.jpg HTTP/1.0" 200 18102 www-b5.proxy.aol.com - - [02/Sep/1995:04:56:18 -0400] "GET /pub/k2/am4x44u/gifs/cj2.gif HTTP/1.0" 200 36798 ebrady.clark.net - - [02/Sep/1995:05:15:01 -0400] "GET /www.clark.net/pub/green/friends.htm HTTP/1.0" 404 312 lanesplc.MT.net - - [02/Sep/1995:05:31:41 -0400] "GET /pub/sshay/chris.html HTTP/1.0" 200 7172 Cust21.Max1.Seattle.WA.MS.UU.NET - - [02/Sep/1995:05:50:56 -0400] "GET /pub/peace/PC1.map?49,55 HTTP/1.0" 302 226 198.165.3.2 - - [02/Sep/1995:06:18:57 -0400] "GET /pub/wick/photos/lay.jpg HTTP/1.0" 200 20923 ma4ppp.tinet.ch - - [02/Sep/1995:06:39:18 -0400] "GET /pub/atomicbk/contest.gif HTTP/1.0" 200 669 203.15.32.1 - - [02/Sep/1995:06:53:01 -0400] "GET /pub/gen/fas/irp/nsa/ HTTP/1.0" 200 91372 133.28.55.143 - - [02/Sep/1995:07:08:45 -0400] "GET /pub/job/vk/view16.jpg HTTP/1.0" 200 6808 dyna-ge-2.dial.eunet.ch - - [02/Sep/1995:07:26:40 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 200 813 e2b202.bri.enternet.com.au - - [02/Sep/1995:07:41:13 -0400] "GET /pub/eurocent/home.htm HTTP/1.0" 200 2432 ppp062.yo.rim.or.jp - - [02/Sep/1995:07:56:02 -0400] "GET /atomicbk/catalog/erotica.html HTTP/1.0" 200 11362 vccnw09.its.rpi.edu - - [02/Sep/1995:08:10:31 -0400] "GET /atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 ppp228.aix.or.jp - - [02/Sep/1995:08:23:27 -0400] "GET /pub/fan/albumpics/expo.jpg HTTP/1.0" 200 2744 ad14-029.compuserve.com - - [02/Sep/1995:08:36:15 -0400] "GET /pub/sshay/images/scott1.jpg HTTP/1.0" 200 18102 maggie.mame.mu.OZ.AU - - [02/Sep/1995:08:50:10 -0400] "GET /pub/job/swim/v-09.jpg HTTP/1.0" 200 3755 196.3.132.15 - - [02/Sep/1995:09:01:27 -0400] "GET /pub/atomicbk/catalog/new.gif HTTP/1.0" 200 744 p102.infolink.co.za - - [02/Sep/1995:09:12:19 -0400] "GET /pub/job/swim/ss-01.jpg HTTP/1.0" 200 51524 139.230.2.39 - - [02/Sep/1995:09:21:07 -0400] "GET /pub/phil/VerucaSalt.html HTTP/1.0" 200 3954 sophocles.algonet.se - - [02/Sep/1995:09:31:18 -0400] "GET /pub/job/vk/view05.jpg HTTP/1.0" 200 5012 www-e5.proxy.aol.com - - [02/Sep/1995:09:40:22 -0400] "GET /pub/jeffd/smrushnw.gif HTTP/1.0" 200 23351 dd06-036.compuserve.com - - [02/Sep/1995:09:51:30 -0400] "GET /pub/wine/btnchamp.gif HTTP/1.0" 200 845 prpayne-ppp.clark.net - - [02/Sep/1995:10:00:05 -0400] "GET /pub/shows/images/pblogrt1.gif HTTP/1.0" 304 - continental.com - - [02/Sep/1995:10:12:35 -0400] "GET /pub/k2/am4x44u/gifs/4x4_small.gif HTTP/1.0" 200 9401 www-c2.proxy.aol.com - - [02/Sep/1995:10:21:51 -0400] "GET /pub/jeffd/castww.html HTTP/1.0" 200 37184 jazz.iinet.com.au - - [02/Sep/1995:10:31:28 -0400] "GET /pub/csamsi/ HTTP/1.0" 304 - 147.56.61.5 - - [02/Sep/1995:10:38:05 -0400] "GET /atomicbk/images/buffy.gif HTTP/1.0" 200 59956 matt.qmi.mei.com - - [02/Sep/1995:10:47:01 -0400] "GET /pub/mmathai/mls HTTP/1.0" 302 223 bfwspc33.forwiss.uni-erlangen.de - - [02/Sep/1995:10:51:59 -0400] "GET /pub/job/vk/view07.jpg HTTP/1.0" 200 7057 genmac14.gen.gu.se - - [02/Sep/1995:11:00:29 -0400] "GET /pub/macgyver/VidGames/fo/gifs/fo7small.jpg HTTP/1.0" 404 334 www-c3.proxy.aol.com - - [02/Sep/1995:11:07:14 -0400] "GET /pub/wick/photos/beavers_mini.gif HTTP/1.0" 304 - wpbfl2-26.gate.net - - [02/Sep/1995:11:15:18 -0400] "GET /pub/job/vk/v-line.gif HTTP/1.0" 200 1254 www.rrz.Uni-Koeln.DE - - [02/Sep/1995:11:23:06 -0400] "GET /pub/job/swim/s-back2.jpg HTTP/1.0" 200 4653 wsmac02.acs.nmu.edu - - [02/Sep/1995:11:29:42 -0400] "GET /pub/job/vendela.jpg HTTP/1.0" 200 36746 husserl.sbu.edu - - [02/Sep/1995:11:37:35 -0400] "GET /pub/sshay/images/davthumb.jpg HTTP/1.0" 200 3914 206.42.112.218 - - [02/Sep/1995:11:45:53 -0400] "GET /atomicbk/images/youngw.gif HTTP/1.0" 404 338 141.42.1.145 - - [02/Sep/1995:11:55:06 -0400] "GET /pub/sshay/images/updated.gif HTTP/1.0" 200 168 tplawton.is.marist.edu - - [02/Sep/1995:12:05:24 -0400] "GET /pub/sshay/images/sglt.gif HTTP/1.0" 200 19929 www-c3.proxy.aol.com - - [02/Sep/1995:12:12:54 -0400] "GET /pub/jeffd/spkrnewt.gif HTTP/1.0" 304 - gate0.ftech.net - - [02/Sep/1995:12:19:07 -0400] "GET /pub/mikebilt/mikeb.jpg HTTP/1.0" 200 10311 lumts-1-02.luminet.net - - [02/Sep/1995:12:25:44 -0400] "GET /pub/atomicbk/catalog/erotica.html HTTP/1.0" 200 11362 dcs-79.umd.edu - - [02/Sep/1995:12:33:00 -0400] "GET /pub/atomicbk/images/pagepix.gif HTTP/1.0" 200 35635 ppp221.gol.com - - [02/Sep/1995:12:40:13 -0400] "GET /pub/wine/favbtn.gif HTTP/1.0" 200 955 volt.elab.ece.ntua.gr - - [02/Sep/1995:12:46:26 -0400] "GET /pub/iz/cgi-bin/counter?doc=Books/books.html HTTP/1.0" 200 654 ppp29.sbbs.se - - [02/Sep/1995:12:53:29 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 ENV-HP5970.TAMU.EDU - - [02/Sep/1995:13:00:29 -0400] "GET /pub/phil/images/jh11.gif HTTP/1.0" 200 9407 nswenson-mac.qualcomm.com - - [02/Sep/1995:13:07:18 -0400] "GET /atomicbk/shocked/shocked.jpg HTTP/1.0" 200 43819 psm101-1.agsc.ttu.edu - - [02/Sep/1995:13:13:27 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 9084 W20-575-61.MIT.EDU - - [02/Sep/1995:13:19:41 -0400] "GET /pub/job/vk/view29.jpg HTTP/1.0" 200 6695 W20-575-61.MIT.EDU - - [02/Sep/1995:13:24:11 -0400] "GET /pub/job/mail.jpg HTTP/1.0" 200 4282 dial6.netrix.net - - [02/Sep/1995:13:31:36 -0400] "GET /pub/shelby/scroll.gif HTTP/1.0" 200 999 camil44.music.uiuc.edu - - [02/Sep/1995:13:40:20 -0400] "GET /pub/rmharris/catalogs/blksncat/intro.html HTTP/1.0" 200 2821 130.206.110.6 - - [02/Sep/1995:13:47:14 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 W20-575-61.MIT.EDU - - [02/Sep/1995:13:52:27 -0400] "GET /pub/job/vk/view03.jpg HTTP/1.0" 200 8506 wintermute.synergy.net - - [02/Sep/1995:13:59:04 -0400] "GET /theme/factinfo.html HTTP/1.0" 200 2420 theatre104.scales.wfu.edu - - [02/Sep/1995:14:05:05 -0400] "GET /pub/sshay/images/galybutn.gif HTTP/1.0" 200 1080 mariana.rsvs.ulaval.ca - - [02/Sep/1995:14:13:04 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 304 - studpc04.uio.no - - [02/Sep/1995:14:20:36 -0400] "GET /pub/pribut/new.gif HTTP/1.0" 200 147 192.228.240.51 - - [02/Sep/1995:14:27:13 -0400] "GET /pub/atomicbk/catalog/mini.gif HTTP/1.0" 200 360 www-e6.proxy.aol.com - - [02/Sep/1995:14:34:05 -0400] "GET /pub/pribut/new.gif HTTP/1.0" 304 - sylviepaul.cnwl.igs.net - - [02/Sep/1995:14:40:48 -0400] "GET /mc-icons/blank.gif HTTP/1.0" 200 60 129.30.50.37 - - [02/Sep/1995:14:46:12 -0400] "GET /atomicbk/catalog/sleazmag.html HTTP/1.0" 200 6786 pae1.monmouth.army.mil - - [02/Sep/1995:14:52:15 -0400] "GET /pub/job/vk/view09.jpg HTTP/1.0" 200 6011 smtp.inet.fi - - [02/Sep/1995:14:59:28 -0400] "GET /atomicbk/seanc.gif HTTP/1.0" 200 734 Findlay-IBMPC45.cac-labs.psu.edu - - [02/Sep/1995:15:06:50 -0400] "GET /pub/job/vk/page1.gif HTTP/1.0" 200 1960 www-e4.proxy.aol.com - - [02/Sep/1995:15:14:08 -0400] "GET /atomicbk/images/hbslut.gif HTTP/1.0" 304 - piweba3y.prodigy.com - - [02/Sep/1995:15:20:50 -0400] "GET /pub/networx/autopage/classic/cl001s.gif HTTP/1.0" 200 19347 box775.labs.cis.pitt.edu - - [02/Sep/1995:15:28:57 -0400] "GET /pub/mbrophy/pix/interv.jpg HTTP/1.0" 200 5047 B3.cyberspy.com - - [02/Sep/1995:15:35:04 -0400] "GET /pub/job/vk/flowers1.gif HTTP/1.0" 200 4288 ts1-s1.adeptcom.com - - [02/Sep/1995:15:40:34 -0400] "GET /pub/job/vk/view04.jpg HTTP/1.0" 200 4847 gemsgw.med.ge.com - - [02/Sep/1995:15:47:32 -0400] "GET /pub/jeffd/pjb.gif HTTP/1.0" 200 4849 chaos.idirect.com - - [02/Sep/1995:15:54:09 -0400] "GET /pub/shelby/creambox.gif HTTP/1.0" 200 856 198.105.193.95 - - [02/Sep/1995:15:59:05 -0400] "GET /pub/job/swim/v-07.jpg HTTP/1.0" 200 5703 acme.clark.net - - [02/Sep/1995:16:04:42 -0400] "GET /pub/mirko/web/images/Buttons/dir_W.gif HTTP/1.0" 200 276 ad03-033.compuserve.com - - [02/Sep/1995:16:09:53 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 200 813 jimw.macon.tec.ga.us - - [02/Sep/1995:16:15:15 -0400] "GET /pub/alweiner/a27.gif HTTP/1.0" 200 4594 slc91.xmission.com - - [02/Sep/1995:16:20:51 -0400] "GET /pub/mikebilt/ruffbear.jpg HTTP/1.0" 200 13188 directweb.com - - [02/Sep/1995:16:26:21 -0400] "GET /pub/evins/Icons/Misc/ HTTP/1.0" 200 - ip215.phx.primenet.com - - [02/Sep/1995:16:32:56 -0400] "GET /pub/sshay/images/scotbksh.jpg HTTP/1.0" 200 19141 pm2-48.TVS.NET - - [02/Sep/1995:16:38:55 -0400] "GET /pub/job/vk/view02.jpg HTTP/1.0" 200 7290 panel-pc.cba.uh.edu - - [02/Sep/1995:16:47:50 -0400] "GET /pub/job/vk/vk-ad-09.jpg HTTP/1.0" 200 70678 205.208.58.46 - - [02/Sep/1995:16:52:59 -0400] "GET /pub/job/vk/view07.jpg HTTP/1.0" 200 7057 137.99.19.5 - - [02/Sep/1995:16:59:47 -0400] "GET /pub/networx/fusion.html HTTP/1.0" 200 1806 yellow54.nada.kth.se - - [02/Sep/1995:17:06:10 -0400] "GET /pub/job/vendela.jpg HTTP/1.0" 200 36746 oahu-54.u.aloha.net - - [02/Sep/1995:17:12:58 -0400] "GET /pub/alweiner/up_red.gif HTTP/1.0" 200 218 relay02.jpmorgan.com - - [02/Sep/1995:17:20:08 -0400] "GET /pub/job/vk/view10.jpg HTTP/1.0" 200 5303 204.245.163.3 - - [02/Sep/1995:17:25:47 -0400] "GET /pub/cheebie/graphics/pen.gif HTTP/1.0" 200 541 enm324a-17.engl.ttu.edu - - [02/Sep/1995:17:29:44 -0400] "GET /pub/listserv/purpleba.gif HTTP/1.0" 200 900 Psychstat-12.Psych.UH.EDU - - [02/Sep/1995:17:36:22 -0400] "GET /larouche/strategic.html HTTP/1.0" 200 756 sdt.com - - [02/Sep/1995:17:42:51 -0400] "GET /pub/job/vk/view34.jpg HTTP/1.0" 200 5156 ad06-027.compuserve.com - - [02/Sep/1995:17:48:44 -0400] "GET /pub/sshay/images/resume.gif HTTP/1.0" 200 8760 128.84.38.10 - - [02/Sep/1995:17:54:25 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 200 813 mars.ark.com - - [02/Sep/1995:17:59:32 -0400] "GET /pub/k2/am4x44u/gifs/bgpaper.gif HTTP/1.0" 200 5960 204.251.2.200 - - [02/Sep/1995:18:04:39 -0400] "GET /pub/RAI/links.html HTTP/1.0" 200 4847 osf1.gmu.edu - - [02/Sep/1995:18:11:30 -0400] "GET /pub/jeffd/sm_eaghd.gif HTTP/1.0" 200 2866 slc76.xmission.com - - [02/Sep/1995:18:18:52 -0400] "GET /pub/mjoneill/links.html HTTP/1.0" 404 336 www-b5.proxy.aol.com - - [02/Sep/1995:18:26:20 -0400] "GET /pub/atomicbk/catalog/erotica.html HTTP/1.0" 200 11362 pc180010.dit.co.fairfax.va.us - - [02/Sep/1995:18:33:42 -0400] "GET /pub/sshay/images/crthumb5.jpg HTTP/1.0" 200 4495 hacknet.demon.co.uk - - [02/Sep/1995:18:41:18 -0400] "GET /pub/rjgula/hack/ HTTP/1.0" 403 142 lovely.tiac.net - - [02/Sep/1995:18:47:27 -0400] "GET /pub/sknight/t_whitem.gif HTTP/1.0" 200 5708 JEDI.TAMU.EDU - - [02/Sep/1995:18:55:03 -0400] "GET /pub/sshay/interact.html HTTP/1.0" 200 2541 pm192.smartlink.net - - [02/Sep/1995:19:02:16 -0400] "GET /pub/job/vk/view02.jpg HTTP/1.0" 200 7290 204.137.204.229 - - [02/Sep/1995:19:09:23 -0400] "GET /pub/journalism/hate.html HTTP/1.0" 200 4602 nswenson-mac.qualcomm.com - - [02/Sep/1995:19:17:35 -0400] "GET /pub/job/vk/view01.jpg HTTP/1.0" 200 5733 jeffery.clark.net - - [02/Sep/1995:19:25:47 -0400] "GET /pub/pribut/email2.gif HTTP/1.0" 200 1209 jbddup-a-5.rmt.net.pitt.edu - - [02/Sep/1995:19:31:53 -0400] "GET /pub/phil/images/jh26.gif HTTP/1.0" 304 - studpc04.uio.no - - [02/Sep/1995:19:39:54 -0400] "GET /pub/pribut/upbtn.gif HTTP/1.0" 200 145 stealth.FOUR.net - - [02/Sep/1995:19:48:34 -0400] "GET /pub/alweiner/work/_0291A.gif HTTP/1.0" 200 450 Trestle.AI.SRI.COM - - [02/Sep/1995:19:55:28 -0400] "GET /pub/job/swim/v-18.jpg HTTP/1.0" 200 2838 please.re.open.de - - [02/Sep/1995:20:04:53 -0400] "GET /pub/job/vk/view15.jpg HTTP/1.0" 200 6036 cd-37.continuum.net - - [02/Sep/1995:20:12:12 -0400] "GET /pub/rebel/stuff.gif HTTP/1.0" 200 1589 153.104.13.90 - - [02/Sep/1995:20:18:22 -0400] "GET /pub/atomicbk/catalog/orders.gif HTTP/1.0" 200 800 sn22e025.resnet.drexel.edu - - [02/Sep/1995:20:25:54 -0400] "GET /pub/job/vk/view12.jpg HTTP/1.0" 200 5966 dialup1.logical.net - - [02/Sep/1995:20:31:41 -0400] "GET /pub/atomicbk/emboss.jpg HTTP/1.0" 200 4741 204.255.215.47 - - [02/Sep/1995:20:38:09 -0400] "GET /pub/aztec/shesails/butover.gif HTTP/1.0" 200 1761 mpngate6.ny.us.ibm.net - - [02/Sep/1995:20:45:08 -0400] "GET /pub/job/vendela.jpg HTTP/1.0" 200 36746 cardio-mac1.ucsf.EDU - - [02/Sep/1995:20:52:46 -0400] "GET /atomicbk/images/comic75.gif HTTP/1.0" 200 444 ix-ir5-13.ix.netcom.com - - [02/Sep/1995:21:01:21 -0400] "GET /pub/job/vk/flowers1.gif HTTP/1.0" 200 4288 204.191.195.105 - - [02/Sep/1995:21:09:52 -0400] "GET /pub/networx/autopage/autobar.gif HTTP/1.0" 200 2751 138.26.196.178 - - [02/Sep/1995:21:16:41 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 9084 port56.annex4.net.ubc.ca - - [02/Sep/1995:21:24:53 -0400] "GET /pub/ibos/moto/star.gif HTTP/1.0" 200 1901 ebrady.clark.net - - [02/Sep/1995:21:33:47 -0400] "GET /pub/green/more.gif HTTP/1.0" 200 6030 ephedra.psych.indiana.edu - - [02/Sep/1995:21:40:53 -0400] "GET /pub/atomicbk/orders.gif HTTP/1.0" 200 800 www-d1.proxy.aol.com - - [02/Sep/1995:21:48:28 -0400] "GET /pub/jrinker/editors.htm HTTP/1.0" 200 467 ana1056.deltanet.com - - [02/Sep/1995:21:57:33 -0400] "GET /pub/atomicbk/iamges/vargas.gif HTTP/1.0" 404 335 zeus.towson.edu - - [02/Sep/1995:22:04:56 -0400] "GET /pub/atomicbk/direct.gif HTTP/1.0" 200 833 dialup03.wdbg.va.qnet.com - - [02/Sep/1995:22:12:38 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 148.61.231.32 - - [02/Sep/1995:22:18:19 -0400] "GET /pub/peace/VRSFoot.GIF HTTP/1.0" 200 16052 kuts7p04.cc.ukans.edu - - [02/Sep/1995:22:25:40 -0400] "GET /pub/k2/cgi-bin/imagemap/4x4_home?242,52 HTTP/1.0" 200 1892 www-c4.proxy.aol.com - - [02/Sep/1995:22:33:04 -0400] "GET /pub/job/vk/view12.jpg HTTP/1.0" 200 5966 giaec.cc.monash.edu.au - - [02/Sep/1995:22:39:47 -0400] "GET /atomicbk/images/angie.gif HTTP/1.0" 200 33638 slip-9-10.ots.utexas.edu - - [02/Sep/1995:22:46:04 -0400] "GET /pub/job/vk/view16.jpg HTTP/1.0" 200 6808 mn00170w-726421.mcnutt.indiana.edu - - [02/Sep/1995:22:53:27 -0400] "GET /pub/atomicbk/catalog/home.gif HTTP/1.0" 200 813 lindsay.midland.co.nz - - [02/Sep/1995:23:01:11 -0400] "GET /pub/atomicbk/catalog/tattoo.html HTTP/1.0" 200 3073 star39.hkstar.com - - [02/Sep/1995:23:08:11 -0400] "GET /pub/job/vendela.jpg HTTP/1.0" 200 36746 queen.ua.pt - - [02/Sep/1995:23:14:50 -0400] "GET /pub/alweiner/blue_sha.gif HTTP/1.0" 200 991 Alice-Thurman.tenet.edu - - [02/Sep/1995:23:20:36 -0400] "GET /pub/journalism/kid.html HTTP/1.0" 200 5343 acme.clark.net - - [02/Sep/1995:23:28:43 -0400] "GET /graphics/bar.gif HTTP/1.0" 304 - acme.clark.net - - [02/Sep/1995:23:35:16 -0400] "GET /pub/mirko/web/images/Buttons/gopher_page.gif HTTP/1.0" 200 369 205.218.190.33 - - [02/Sep/1995:23:42:12 -0400] "GET /aztec/ccyc/stern.gif HTTP/1.0" 200 14241 buzzr.earthlink.net - - [02/Sep/1995:23:47:33 -0400] "GET /pub/ibos/link.gif HTTP/1.0" 304 - ip239.tokyo.jp.interramp.com - - [02/Sep/1995:23:53:23 -0400] "GET /pub/abaa-booknet/images/abaabutt.gif HTTP/1.0" 200 1041 129.237.115.30 - - [03/Sep/1995:00:00:01 -0400] "GET /pub/theme/ HTTP/1.0" 200 799 piweba5y.prodigy.com - - [03/Sep/1995:00:06:40 -0400] "GET /pub/shelby/yellowli.gif HTTP/1.0" 200 905 Cust10.Max1.Houston.TX.MS.UU.NET - - [03/Sep/1995:00:15:06 -0400] "GET /pub/sshay/images/brnthumb.jpg HTTP/1.0" 200 2775 ablake.bevb.blacksburg.va.us - - [03/Sep/1995:00:22:25 -0400] "GET /pub/cybersoft/ HTTP/1.0" 200 918 suefew.sc.scruznet.com - - [03/Sep/1995:00:29:38 -0400] "GET /pub/listserv/lsmus1.html HTTP/1.0" 200 2713 dialupS31.ici.net - - [03/Sep/1995:00:35:35 -0400] "GET /pub/job/vk/arrow-r.gif HTTP/1.0" 200 3042 port01.gateway2.ic.net - - [03/Sep/1995:00:43:29 -0400] "GET /theme/cgi-bin/ig.wrl HTTP/1.0" 200 4280 huey.met.fsu.edu - - [03/Sep/1995:00:48:41 -0400] "GET /pub/job/vendela.jpg HTTP/1.0" 200 36746 mzmo1.ResComp.Arizona.EDU - - [03/Sep/1995:00:53:35 -0400] "GET /pub/mikebilt/gaydc.html HTTP/1.0" 200 6485 204.101.163.101 - - [03/Sep/1995:00:59:59 -0400] "GET /atomicbk/catalog/sleazmag.html HTTP/1.0" 200 6786 www-d2.proxy.aol.com - - [03/Sep/1995:01:05:07 -0400] "GET /pub/mpowers/j4j/web/Balls/greengem.gif HTTP/1.0" 200 110 piweba4y.prodigy.com - - [03/Sep/1995:01:11:28 -0400] "GET /pub/jeffd/cicgbtn.gif HTTP/1.0" 304 - ppp4.cowan.edu.au - - [03/Sep/1995:01:17:14 -0400] "GET /pub/job/swim/swim.html HTTP/1.0" 200 4429 146.155.89.20 - - [03/Sep/1995:01:24:13 -0400] "GET /atomicbk/emboss.jpg HTTP/1.0" 200 4741 paradox.ncsa.uiuc.edu - - [03/Sep/1995:01:30:12 -0400] "GET /atomicbk/direct.gif HTTP/1.0" 200 833 slip8-205.beta.delphi.com - - [03/Sep/1995:01:38:00 -0400] "GET /pub/phil/winonaryder.html HTTP/1.0" 200 19870 pma27.loop.com - - [03/Sep/1995:01:47:22 -0400] "GET /pub/jeffd/95-63.gif HTTP/1.0" 200 31640 piweba3y.prodigy.com - - [03/Sep/1995:01:54:26 -0400] "GET /pub/job/vk/vk-page2.jpg HTTP/1.0" 200 10302 piweba1y.prodigy.com - - [03/Sep/1995:02:01:21 -0400] "GET /pub/job/vk/kathy.jpg HTTP/1.0" 200 7450 pbd-m35-15.urh.uiuc.edu - - [03/Sep/1995:02:09:41 -0400] "GET /atomicbk/news.gif HTTP/1.0" 200 912 dynip28.efn.org - - [03/Sep/1995:02:18:31 -0400] "GET /pub/sshay/images/rainback.gif HTTP/1.0" 200 2541 stockyard16.onramp.net - - [03/Sep/1995:02:26:50 -0400] "GET /pub/k2/am4x44u/maps/usamap.map?242,230 HTTP/1.0" 302 245 yak-ts1-p21.wolfe.net - - [03/Sep/1995:02:35:03 -0400] "GET /pub/abaa-booknet/images/search.gif HTTP/1.0" 200 938 slip017.loa.com - - [03/Sep/1995:02:43:17 -0400] "GET /atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 203.251.191.9 - - [03/Sep/1995:02:51:19 -0400] "GET /pub/k2/am4x44u/trails/locations/locations.htm HTTP/1.0" 200 904 flores.msm.com - - [03/Sep/1995:03:00:39 -0400] "GET /pub/wine/flgausta.gif HTTP/1.0" 200 203 pp7.westdat.com - - [03/Sep/1995:03:12:39 -0400] "GET /pub/atomicbk/userlink.gif HTTP/1.0" 200 816 ad07-008.compuserve.com - - [03/Sep/1995:03:20:51 -0400] "GET /pub/winfield/brian.html HTTP/1.0" 200 1514 e1s110.syd.enternet.com.au - - [03/Sep/1995:03:33:15 -0400] "GET /pub/networx/autopage/dealers/de001_3b.gif HTTP/1.0" 200 70257 d12.itsnet.com - - [03/Sep/1995:03:43:24 -0400] "GET /pub/sshay/images/scott1.jpg HTTP/1.0" 200 18102 Cust10.Max1.Seattle.WA.MS.UU.NET - - [03/Sep/1995:03:55:27 -0400] "GET /pub/job/vk/page1.gif HTTP/1.0" 200 1960 www-e9.proxy.aol.com - - [03/Sep/1995:04:10:58 -0400] "GET /pub/sshay/images/rainlin2.gif HTTP/1.0" 304 - slip-5-11.shore.net - - [03/Sep/1995:04:28:17 -0400] "GET /pub/pgarrett/award.gif HTTP/1.0" 200 5385 netcom15.netcom.com - - [03/Sep/1995:04:41:25 -0400] "GET /pub/macgyver HTTP/1.0" 302 220 Cust47.Max6.Los-Angeles.CA.MS.UU.NET - - [03/Sep/1995:04:58:16 -0400] "GET /pub/job/theman.jpg HTTP/1.0" 200 8905 wipro.wipsys.soft.net - - [03/Sep/1995:05:15:21 -0400] "GET /pub/mbrophy/sandras_home.html HTTP/1.0" 200 1260 dust3.dur.ac.uk - - [03/Sep/1995:05:32:33 -0400] "GET /pub/job/vk/vk-bk.jpg HTTP/1.0" 200 4569 www-c5.proxy.aol.com - - [03/Sep/1995:05:48:54 -0400] "GET /pub/wick/misc/redbg.gif HTTP/1.0" 200 878 mat12.cip.uni-regensburg.de - - [03/Sep/1995:06:12:36 -0400] "GET /pub/atomicbk/catalog/mini.gif HTTP/1.0" 200 360 sw25-158.iol.it - - [03/Sep/1995:06:33:55 -0400] "GET /atomicbk/catalog/home.gif HTTP/1.0" 200 813 dialup98-102.swipnet.se - - [03/Sep/1995:06:51:13 -0400] "GET /pub/job/vk/view31.jpg HTTP/1.0" 200 5611 vicsd209.DIAL.GOV.BC.CA - - [03/Sep/1995:07:10:54 -0400] "GET /pub/conquest/one/books/noiz.html HTTP/1.0" 200 6369 ix-sf18-07.ix.netcom.com - - [03/Sep/1995:07:28:58 -0400] "GET /pub/sshay/images/bigtoed.jpg HTTP/1.0" 200 31158 ad11-011.compuserve.com - - [03/Sep/1995:07:44:20 -0400] "GET /pub/job/vk/view08.jpg HTTP/1.0" 200 6476 A113030.sna1.as.crl.com - - [03/Sep/1995:07:59:34 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 techp755.naruto-u.ac.jp - - [03/Sep/1995:08:14:21 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 9084 gluon.phys.hokudai.ac.jp - - [03/Sep/1995:08:27:58 -0400] "GET /pub/fan/discog.html HTTP/1.0" 304 - 138.103.112.242 - - [03/Sep/1995:08:42:04 -0400] "GET /pub/job/vk/view16.jpg HTTP/1.0" 200 6808 ztivax.zfe.siemens.de - - [03/Sep/1995:09:00:01 -0400] "GET /atomicbk/new/home.gif HTTP/1.0" 200 813 iteck2.infoteck.qc.ca - - [03/Sep/1995:09:13:07 -0400] "GET /atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 gd.neuro.aau.dk - - [03/Sep/1995:09:23:08 -0400] "GET /pub/sshay/images/pthumb.jpg HTTP/1.0" 200 1302 dial-1.t1.ncshlt.sunbelt.net - - [03/Sep/1995:09:34:15 -0400] "GET /pub/pwalker/new.gif HTTP/1.0" 200 153 l2-29.en.net - - [03/Sep/1995:09:46:29 -0400] "GET /pub/ibos/company/delaware.html HTTP/1.0" 200 3362 sida.ifi.uio.no - - [03/Sep/1995:09:55:48 -0400] "GET /pub/job/vk/page1.gif HTTP/1.0" 200 1960 mjens.Cadence.COM - - [03/Sep/1995:10:08:20 -0400] "GET /pub/sshay/home.html HTTP/1.0" 200 3141 isgate.is - - [03/Sep/1995:10:17:41 -0400] "GET /pub/job/vk/bam.gif HTTP/1.0" 304 - ad15-025.compuserve.com - - [03/Sep/1995:10:30:13 -0400] "GET /pub/wick/photos/lay_mini.gif HTTP/1.0" 200 13272 skpc1.rdg.ac.uk - - [03/Sep/1995:10:40:44 -0400] "GET /pub/job/vendela.jpg HTTP/1.0" 200 36746 ssomad.pp.fi - - [03/Sep/1995:10:49:27 -0400] "GET /pub/atomicbk/catalog/orders.gif HTTP/1.0" 200 800 QUERY1.LYCOS.CS.CMU.EDU - - [03/Sep/1995:10:59:02 -0400] "GET /pub/luthardt/ HTTP/1.0" 200 4328 piweba3y.prodigy.com - - [03/Sep/1995:11:07:35 -0400] "GET /pub/robert/robert7.gif HTTP/1.0" 304 - fall.psyber.com - - [03/Sep/1995:11:18:52 -0400] "GET /pub/rmharris/images/search.gif HTTP/1.0" 200 938 clark.net - - [03/Sep/1995:11:27:22 -0400] "GET /pub/bars.gif HTTP/1.0" 404 207 ix-dc9-11.ix.netcom.com - - [03/Sep/1995:11:36:46 -0400] "GET /pub/mikebilt/dchome.gif HTTP/1.0" 304 - arhppp13.uni-c.dk - - [03/Sep/1995:11:44:01 -0400] "GET /pub/russadam/thund03.gif HTTP/1.0" 200 6906 J301855432.RESNET.CORNELL.EDU - - [03/Sep/1995:11:53:15 -0400] "GET /pub/jeffd/grad_lin.gif HTTP/1.0" 200 3346 duxbury30.pcix.com - - [03/Sep/1995:12:04:29 -0400] "GET /pub/atomicbk/home.html HTTP/1.0" 200 4051 aristotle.algonet.se - - [03/Sep/1995:12:14:37 -0400] "GET /pub/phil/WinonaRyder.html HTTP/1.0" 200 19870 199.78.224.27 - - [03/Sep/1995:12:23:22 -0400] "GET /pub/job/vk/view24.jpg HTTP/1.0" 200 6209 dyn51-140.biosci.wayne.edu - - [03/Sep/1995:12:32:24 -0400] "GET /atomicbk/images/tart.gif HTTP/1.0" 200 38555 xkplace.monsanto.com - - [03/Sep/1995:12:40:14 -0400] "GET /pub/sshay/gallery.html HTTP/1.0" 304 - dolphin-29.netrunner.net - - [03/Sep/1995:12:47:34 -0400] "GET /atomicbk/catalog/home.gif HTTP/1.0" 200 813 dial004.future.net - - [03/Sep/1995:12:55:55 -0400] "GET /pub/conquest/one/bfr.gif HTTP/1.0" 200 74981 www-c4.proxy.aol.com - - [03/Sep/1995:13:03:07 -0400] "GET /pub/robert/current.html HTTP/1.0" 200 30337 spinne.web.net - - [03/Sep/1995:13:10:38 -0400] "GET /pub/listserv/top40.html HTTP/1.0" 200 3009 ix-pat1-09.ix.netcom.com - - [03/Sep/1995:13:18:36 -0400] "GET /pub/k2/am4x44u/gifs/4xflogo.gif HTTP/1.0" 200 4719 volkswagon.europa.com - - [03/Sep/1995:13:26:44 -0400] "GET /pub/bell/review/newage/image/strieber_breakthrough.gif HTTP/1.0" 200 11471 130.14.25.216 - - [03/Sep/1995:13:35:22 -0400] "GET /pub/mikebilt/feldie.jpg HTTP/1.0" 200 6808 smf-c14.facsmf.utexas.edu - - [03/Sep/1995:13:43:17 -0400] "GET /pub/atomicbk/promo/people.html HTTP/1.0" 200 2328 bootp-123-53.bootp.Virginia.EDU - - [03/Sep/1995:13:52:45 -0400] "GET /atomicbk/new/newsep01.html HTTP/1.0" 200 11324 ufcsb1.health.ufl.edu - - [03/Sep/1995:14:00:50 -0400] "GET /pub/job/vk/view08.jpg HTTP/1.0" 200 6476 govannon.cee.hw.ac.uk - - [03/Sep/1995:14:10:28 -0400] "GET /pub/job/vk/arrow-r.gif HTTP/1.0" 304 - ad11-031.compuserve.com - - [03/Sep/1995:14:17:54 -0400] "GET /pub/terra/nba.html HTTP/1.0" 200 2402 box785.labs.cis.pitt.edu - - [03/Sep/1995:14:25:15 -0400] "GET /pub/mbrophy/pix/pbquote2.jpg HTTP/1.0" 200 9507 enigma.kbbsnet.com - - [03/Sep/1995:14:32:43 -0400] "GET /atomicbk/images/buffy.gif HTTP/1.0" 200 59956 cola74.scsn.net - - [03/Sep/1995:14:41:30 -0400] "GET /pub/listserv/lsvl1.html HTTP/1.0" 304 - SOLSTADP.hqpacaf.af.mil - - [03/Sep/1995:14:49:26 -0400] "GET /pub/jeffd/mr_newt.html HTTP/1.0" 304 - dal08-15.ppp.iadfw.net - - [03/Sep/1995:14:56:27 -0400] "GET /atomicbk/new/logo.gif HTTP/1.0" 200 1942 dcc05227.slip.digex.net - - [03/Sep/1995:15:01:35 -0400] "GET /html/usage/usage.graph.gif HTTP/1.0" 200 1727 ts32p14.NetVision.net.il - - [03/Sep/1995:15:09:15 -0400] "GET /pub/rmd/success2.html HTTP/1.0" 200 4977 www-b5.proxy.aol.com - - [03/Sep/1995:15:16:01 -0400] "GET /atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 theoris.rz.uni-konstanz.de - - [03/Sep/1995:15:22:25 -0400] "GET /atomicbk/catalog/sleazmag.html HTTP/1.0" 200 6786 crl10.crl.com - - [03/Sep/1995:15:28:05 -0400] "GET /pub/sshay/images/galybutn.gif HTTP/1.0" 200 1080 poppy.hensa.ac.uk - - [03/Sep/1995:15:34:20 -0400] "GET /pub/murple/local/head-explode.html HTTP/1.0" 200 4465 slip4059.sirius.com - - [03/Sep/1995:15:41:38 -0400] "GET /pub/job/swim/v-08.jpg HTTP/1.0" 200 2155 3dweb.vip.best.com - - [03/Sep/1995:15:49:40 -0400] "GET /theme/cgi-bin/veewf.wrl HTTP/1.0" 200 919 199.190.77.158 - - [03/Sep/1995:15:57:21 -0400] "GET /pub/downin/html/arlonet/icons/video.GIF HTTP/1.0" 200 1644 137.104.136.201 - - [03/Sep/1995:16:05:22 -0400] "GET /pub/job/theman.jpg HTTP/1.0" 200 8905 walker.amcc.com - - [03/Sep/1995:16:12:46 -0400] "GET /pub/job/vk/view04.jpg HTTP/1.0" 200 4847 dd14-071.compuserve.com - - [03/Sep/1995:16:19:26 -0400] "GET /pub/mbrophy/other_sites.html HTTP/1.0" 200 2175 ix-nyc11-21.ix.netcom.com - - [03/Sep/1995:16:26:26 -0400] "GET /pub/rsjdfg/Guilt.html HTTP/1.0" 200 724 euler.dcc.unicamp.br - - [03/Sep/1995:16:34:09 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 ppp-66-61.dialup.winternet.com - - [03/Sep/1995:16:41:28 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 rover7.microsoft.com - - [03/Sep/1995:16:47:57 -0400] "GET /pub/wick/misc/newgray.gif HTTP/1.0" 200 17085 137.219.80.52 - - [03/Sep/1995:16:56:21 -0400] "GET /pub/atomicbk/images/library.gif HTTP/1.0" 200 10696 balu.sch.bme.hu - - [03/Sep/1995:17:04:00 -0400] "GET /pub/job/vk/view02.jpg HTTP/1.0" 200 7290 isglcr.sherritt.CA - - [03/Sep/1995:17:10:35 -0400] "GET /pub/atomicbk/catalog/logo2.gif HTTP/1.0" 200 12871 rathlee.internet-eireann.ie - - [03/Sep/1995:17:17:24 -0400] "GET /atomicbk/images/alfetish.gif HTTP/1.0" 200 33094 Basque.Stanford.EDU - - [03/Sep/1995:17:23:35 -0400] "GET /pub/royfc/graphics/red_rt.gif HTTP/1.0" 200 846 131.115.147.204 - - [03/Sep/1995:17:31:13 -0400] "GET /pub/job/vk/page1.gif HTTP/1.0" 200 1960 bluestem.prairienet.org - - [03/Sep/1995:17:38:41 -0400] "GET /pub/lschank/explore/aboutme.html HTTP/1.0" 200 4244 schuller-pc1.ucsd.edu - - [03/Sep/1995:17:46:01 -0400] "GET /pub/atomicbk/catalog/emboss.jpg HTTP/1.0" 200 4741 hasle.oslonett.no - - [03/Sep/1995:17:53:06 -0400] "GET /pub/networx/autopage/autoadv.gif HTTP/1.0" 200 33457 who.AFTAC.GOV - - [03/Sep/1995:18:00:16 -0400] "GET /pub/job/vk/vendela.html HTTP/1.0" 200 9084 153.11.210.163 - - [03/Sep/1995:18:08:28 -0400] "GET /pub/sshay/images/justin.jpg HTTP/1.0" 200 3489 piweba3y.prodigy.com - - [03/Sep/1995:18:14:14 -0400] "GET /pub/robert/current.html HTTP/1.0" 200 30337 204.174.112.53 - - [03/Sep/1995:18:20:37 -0400] "GET /pub/micros/forbes/bo-17-4.html HTTP/1.0" 200 6115 pm1-42.inx.net - - [03/Sep/1995:18:26:49 -0400] "GET /pub/pribut/idea.gif HTTP/1.0" 200 909 dialup-pkr-7-13.network.umr.edu - - [03/Sep/1995:18:33:49 -0400] "GET /pub/wsg/html/jan/ HTTP/1.0" 200 476 chaos.idirect.com - - [03/Sep/1995:18:41:37 -0400] "GET /pub/pribut/back.gif HTTP/1.0" 200 713 svin09.win.tue.nl - - [03/Sep/1995:18:48:01 -0400] "GET /pub/job/vk/view21.jpg HTTP/1.0" 200 5228 129.96.207.154 - - [03/Sep/1995:18:57:50 -0400] "GET /pub/job/vk/view09.jpg HTTP/1.0" 200 6011 www-d2.proxy.aol.com - - [03/Sep/1995:19:04:39 -0400] "GET /pub/robert/past99.gif HTTP/1.0" 200 4993 modem023.offcampus.calpoly.edu - - [03/Sep/1995:19:13:48 -0400] "GET /pub/jumpsam/new_icon.gif HTTP/1.0" 200 99 holli-ko-017.holli.com - - [03/Sep/1995:19:21:43 -0400] "GET /pub/journalism/busncard.html HTTP/1.0" 200 1432 dnet033.sat.texas.net - - [03/Sep/1995:19:29:51 -0400] "GET /pub/grneyedl/in2.htm HTTP/1.0" 200 4642 chapman.ICSI.Net - - [03/Sep/1995:19:37:27 -0400] "GET /pub/job/vk/view12.jpg HTTP/1.0" 200 5966 dial5-8.midwest.net - - [03/Sep/1995:19:44:23 -0400] "GET /pub/peace/NEW1.html HTTP/1.0" 200 1411 131.144.82.190 - - [03/Sep/1995:19:52:33 -0400] "GET /pub/jeffd/spkrnewt.gif HTTP/1.0" 200 45247 scihum07.uqtr.uquebec.ca - - [03/Sep/1995:20:02:13 -0400] "GET /atomicbk/catalog/catalog.html HTTP/1.0" 200 4654 ice.on.ca.ibm.net - - [03/Sep/1995:20:10:02 -0400] "GET /pub/job/vk/view36.jpg HTTP/1.0" 200 3522 cornflower.cord.edu - - [03/Sep/1995:20:17:24 -0400] "GET /atomicbk/catalog/adultcom.html HTTP/1.0" 200 18673 141.209.3.228 - - [03/Sep/1995:20:23:45 -0400] "GET /pub/job/vk/view31.jpg HTTP/1.0" 200 5611 ix-dgr-il1-20.ix.netcom.com - - [03/Sep/1995:20:32:28 -0400] "GET /atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 freenet2.carleton.ca - - [03/Sep/1995:20:40:11 -0400] "GET /pub/macgyver/ HTTP/1.0" 200 11337 snug.caltech.edu - - [03/Sep/1995:20:48:50 -0400] "GET /ccentral/thinline.gif HTTP/1.0" 200 1142 hella.stm.it - - [03/Sep/1995:20:56:00 -0400] "GET /pub/atomicbk/catalog/sleazbk.html HTTP/1.0" 200 18338 chegd40.che.utexas.edu - - [03/Sep/1995:21:03:02 -0400] "GET /pub/atomicbk/catalog/catalog.gif HTTP/1.0" 200 693 ppp3.blackhills.com - - [03/Sep/1995:21:09:12 -0400] "GET /atomicbk/new/emboss.jpg HTTP/1.0" 200 4741 dcn68.dcn.davis.ca.us - - [03/Sep/1995:21:15:23 -0400] "GET /pub/peace/Country1.GIF HTTP/1.0" 200 55856 S16.IC.Owatonna.MN.US - - [03/Sep/1995:21:23:44 -0400] "GET /atomicbk/new/emboss.jpg HTTP/1.0" 304 - ix-tam1-13.ix.netcom.com - - [03/Sep/1995:21:30:17 -0400] "GET /pub/lupine/www/disney.html HTTP/1.0" 200 410 134.124.43.218 - - [03/Sep/1995:21:37:14 -0400] "GET /rtside/smreagan.gif HTTP/1.0" 200 18018 ad15-014.compuserve.com - - [03/Sep/1995:21:45:02 -0400] "GET /pub/lschank/web/foreign.html HTTP/1.0" 200 8765 brianb.FASTech.Com - - [03/Sep/1995:21:53:06 -0400] "GET /atomicbk/images/diarydom.gif HTTP/1.0" 200 26849 bernice23.pixi.com - - [03/Sep/1995:22:00:36 -0400] "GET /pub/sshay/images/sglt.gif HTTP/1.0" 200 19929 netcom5.netcom.com - - [03/Sep/1995:22:06:06 -0400] "GET /pub/sshay/images/bigtoed.jpg HTTP/1.0" 200 31158 zuul.lcp.com - - [03/Sep/1995:22:12:48 -0400] "POST /pub/theme/cgi-bin/virtwf.pl/l/mxd/n/mkultra/ HTTP/1.0" 200 1315 netblazer1-s24.telalink.net - - [03/Sep/1995:22:19:00 -0400] "GET /pub/sshay/images/sglt.gif HTTP/1.0" 200 19929 149.171.166.13 - - [03/Sep/1995:22:27:17 -0400] "GET /pub/rsjdfg/Finale.html HTTP/1.0" 200 564 sud106.vianet.on.ca - - [03/Sep/1995:22:33:43 -0400] "GET /pub/sshay/images/tyusvest.jpg HTTP/1.0" 200 26791 dial30.ppp.iastate.edu - - [03/Sep/1995:22:39:22 -0400] "GET /pub/job/job-bk2.jpg HTTP/1.0" 200 6947 smf-e9.facsmf.utexas.edu - - [03/Sep/1995:22:45:33 -0400] "GET /atomicbk/catalog/orders.gif HTTP/1.0" 200 800 pasyn-43.rice.edu - - [03/Sep/1995:22:52:35 -0400] "GET /pub/networx/autopage/autopage.gif HTTP/1.0" 200 57900 dragon.com - - [03/Sep/1995:22:59:33 -0400] "GET /pub/jeffd/gramm.gif HTTP/1.0" 200 4364 Findlay-Mac16.cac-labs.psu.edu - - [03/Sep/1995:23:06:13 -0400] "GET /pub/atomicbk/images/bpann.gif HTTP/1.0" 200 52405 dal1318.computek.net - - [03/Sep/1995:23:12:55 -0400] "GET /pub/sshay/images/scotface.gif HTTP/1.0" 200 12100 www-c1.proxy.aol.com - - [03/Sep/1995:23:20:19 -0400] "GET /pub/atomicbk/new/newjul15.html HTTP/1.0" 304 - rose_ip162.efn.org - - [03/Sep/1995:23:27:48 -0400] "GET /pub/atomicbk/email.gif HTTP/1.0" 200 756 pslip005.mia-fl.ids.net - - [03/Sep/1995:23:34:05 -0400] "GET /pub/alweiner/cgi-bin/homepage.html?pass HTTP/1.0" 200 1753 www-b5.proxy.aol.com - - [03/Sep/1995:23:40:00 -0400] "GET /pub/journalism/awesome.html HTTP/1.0" 304 - ebrady.clark.net - - [03/Sep/1995:23:47:00 -0400] "GET /pub/green/ftp/cbm/games HTTP/1.0" 302 231 badguy.clark.net - - [03/Sep/1995:23:53:40 -0400] "GET /html/usage/usage.graph.small.gif HTTP/1.0" 200 144 ================================================ FILE: core-tools/tests-regression/web-log-report/out.ok ================================================ WWW server statistics ===================== Summary ------- Number of accesses: 1655 Number of Gbytes transferred: 0.0142005 Number of hosts: 1507 Number of domains: 922 Number of top level domains: 58 Number of different pages: 948 Accesses per day: 236.429 MBytes per day: 2.07733 MBytes log file size: 0.163606 Top 10 Hosts ------------ 11 piweba3y.prodigy.com 7 www-c3.proxy.aol.com 7 www-b5.proxy.aol.com 7 www-a1.proxy.aol.com 7 webgate1.mot.com 6 www-b2.proxy.aol.com 5 www-b4.proxy.aol.com 5 clark.net 5 alweiner.clark.net 4 www-d2.proxy.aol.com Top 20 Level Domain Accesses ---------------------------- 452 com 224 edu 181 net 59 EDU 38 ca 35 uk 28 gov 27 jp 25 au 24 se 21 org 18 COM 16 de 16 NET 14 us 14 no 14 fr 13 mil 11 nl 9 ch Top 10 Domains -------------- 80 .proxy.aol.com 44 .clark.net 34 .compuserve.com 29 .ix.netcom.com 24 .prodigy.com 11 .com 9 .Stanford.EDU 7 .mot.com 6 .urh.uiuc.edu 5 .netcom.com Top 10 Hosts by Transfer ------------------------ 615030 buckman.com 208591 www-b5.proxy.aol.com 173690 hel.fi 169367 in116.brann.co.uk 169367 149.157.246.44 108564 london.Informatik.Uni-Oldenburg.DE 106890 www-b2.proxy.aol.com 106571 gateway.senate.gov 106074 gorgon.pwgsc.gc.ca 97751 HRSB741.RESNET.UPENN.EDU Top 20 Area Requests -------------------- 1461 pub 136 atomicbk 10 graphics 8 theme 7 html 4 mc-icons 3 mpt 3 larouche 2 printing 2 ntp 2 network 2 infouser 2 ccentral 2 aztec 2 1 www.clark.net 1 winfield 1 tara 1 rtside 1 nrlc Top 20 Requests --------------- 24 /pub/atomicbk/catalog/catalog.gif 18 /pub/atomicbk/catalog/logo2.gif 17 /pub/job/vk/vendela.html 17 /pub/atomicbk/catalog/sleazbk.html 17 /pub/atomicbk/catalog/home.gif 15 /pub/job/vk/view02.jpg 15 /pub/atomicbk/catalog/orders.gif 14 /pub/job/vk/view07.jpg 13 /pub/journalism/awesome.html 13 /pub/job/vk/view16.jpg 12 /pub/job/vk/view03.jpg 12 /pub/job/vk/flowers1.gif 12 /pub/job/vendela.jpg 12 /pub/atomicbk/catalog/new.gif 11 /pub/job/vk/vk-bk.jpg 11 /pub/job/vk/view14.jpg 11 /pub/atomicbk/catalog/mini.gif 10 /pub/job/vk/view15.jpg 10 /pub/job/vk/view04.jpg 10 /pub/job/vk/v-sig.gif Accesses by Date ---------------- 252 28/Aug/1995 267 29/Aug/1995 274 30/Aug/1995 259 31/Aug/1995 269 01/Sep/1995 171 02/Sep/1995 163 03/Sep/1995 Accesses by Day of Week ----------------------- 274 Wed 269 Fri 267 Tue 259 Thu 252 Mon 171 Sat 163 Sun Accesses by Local Hour ---------------------- 64 00 50 01 49 02 40 03 32 04 31 05 29 06 41 07 49 08 66 09 78 10 86 11 92 12 95 13 100 14 105 15 102 16 95 17 89 18 79 19 73 20 71 21 72 22 67 23 ================================================ FILE: core-tools/tests-regression/word-properties/LostWorldChap1-3 ================================================ The Project Gutenberg EBook of The Lost World, by Arthur Conan Doyle This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.net Title: The Lost World Author: Arthur Conan Doyle Release Date: June 19, 2008 [EBook #139] Language: English Character set encoding: ASCII *** START OF THIS PROJECT GUTENBERG EBOOK THE LOST WORLD *** Produced by Judith Boss. HTML version by Al Haines. THE LOST WORLD I have wrought my simple plan If I give one hour of joy To the boy who's half a man, Or the man who's half a boy. The Lost World By SIR ARTHUR CONAN DOYLE COPYRIGHT, 1912 Foreword Mr. E. D. Malone desires to state that both the injunction for restraint and the libel action have been withdrawn unreservedly by Professor G. E. Challenger, who, being satisfied that no criticism or comment in this book is meant in an offensive spirit, has guaranteed that he will place no impediment to its publication and circulation. Contents CHAPTER I. "THERE ARE HEROISMS ALL ROUND US" II. "TRY YOUR LUCK WITH PROFESSOR CHALLENGER" III. "HE IS A PERFECTLY IMPOSSIBLE PERSON" IV. "IT'S JUST THE VERY BIGGEST THING IN THE WORLD" V. "QUESTION!" VI. "I WAS THE FLAIL OF THE LORD" VII. "TO-MORROW WE DISAPPEAR INTO THE UNKNOWN" VIII. "THE OUTLYING PICKETS OF THE NEW WORLD" IX. "WHO COULD HAVE FORESEEN IT?" X. "THE MOST WONDERFUL THINGS HAVE HAPPENED" XI. "FOR ONCE I WAS THE HERO" XII. "IT WAS DREADFUL IN THE FOREST" XIII. "A SIGHT I SHALL NEVER FORGET" XIV. "THOSE WERE THE REAL CONQUESTS" XV. "OUR EYES HAVE SEEN GREAT WONDERS" XVI. "A PROCESSION! A PROCESSION!" THE LOST WORLD The Lost World CHAPTER I "There Are Heroisms All Round Us" Mr. Hungerton, her father, really was the most tactless person upon earth,--a fluffy, feathery, untidy cockatoo of a man, perfectly good-natured, but absolutely centered upon his own silly self. If anything could have driven me from Gladys, it would have been the thought of such a father-in-law. I am convinced that he really believed in his heart that I came round to the Chestnuts three days a week for the pleasure of his company, and very especially to hear his views upon bimetallism, a subject upon which he was by way of being an authority. For an hour or more that evening I listened to his monotonous chirrup about bad money driving out good, the token value of silver, the depreciation of the rupee, and the true standards of exchange. "Suppose," he cried with feeble violence, "that all the debts in the world were called up simultaneously, and immediate payment insisted upon,--what under our present conditions would happen then?" I gave the self-evident answer that I should be a ruined man, upon which he jumped from his chair, reproved me for my habitual levity, which made it impossible for him to discuss any reasonable subject in my presence, and bounced off out of the room to dress for a Masonic meeting. At last I was alone with Gladys, and the moment of Fate had come! All that evening I had felt like the soldier who awaits the signal which will send him on a forlorn hope; hope of victory and fear of repulse alternating in his mind. She sat with that proud, delicate profile of hers outlined against the red curtain. How beautiful she was! And yet how aloof! We had been friends, quite good friends; but never could I get beyond the same comradeship which I might have established with one of my fellow-reporters upon the Gazette,--perfectly frank, perfectly kindly, and perfectly unsexual. My instincts are all against a woman being too frank and at her ease with me. It is no compliment to a man. Where the real sex feeling begins, timidity and distrust are its companions, heritage from old wicked days when love and violence went often hand in hand. The bent head, the averted eye, the faltering voice, the wincing figure--these, and not the unshrinking gaze and frank reply, are the true signals of passion. Even in my short life I had learned as much as that--or had inherited it in that race memory which we call instinct. Gladys was full of every womanly quality. Some judged her to be cold and hard; but such a thought was treason. That delicately bronzed skin, almost oriental in its coloring, that raven hair, the large liquid eyes, the full but exquisite lips,--all the stigmata of passion were there. But I was sadly conscious that up to now I had never found the secret of drawing it forth. However, come what might, I should have done with suspense and bring matters to a head to-night. She could but refuse me, and better be a repulsed lover than an accepted brother. So far my thoughts had carried me, and I was about to break the long and uneasy silence, when two critical, dark eyes looked round at me, and the proud head was shaken in smiling reproof. "I have a presentiment that you are going to propose, Ned. I do wish you wouldn't; for things are so much nicer as they are." I drew my chair a little nearer. "Now, how did you know that I was going to propose?" I asked in genuine wonder. "Don't women always know? Do you suppose any woman in the world was ever taken unawares? But--oh, Ned, our friendship has been so good and so pleasant! What a pity to spoil it! Don't you feel how splendid it is that a young man and a young woman should be able to talk face to face as we have talked?" "I don't know, Gladys. You see, I can talk face to face with--with the station-master." I can't imagine how that official came into the matter; but in he trotted, and set us both laughing. "That does not satisfy me in the least. I want my arms round you, and your head on my breast, and--oh, Gladys, I want----" She had sprung from her chair, as she saw signs that I proposed to demonstrate some of my wants. "You've spoiled everything, Ned," she said. "It's all so beautiful and natural until this kind of thing comes in! It is such a pity! Why can't you control yourself?" "I didn't invent it," I pleaded. "It's nature. It's love." "Well, perhaps if both love, it may be different. I have never felt it." "But you must--you, with your beauty, with your soul! Oh, Gladys, you were made for love! You must love!" "One must wait till it comes." "But why can't you love me, Gladys? Is it my appearance, or what?" She did unbend a little. She put forward a hand--such a gracious, stooping attitude it was--and she pressed back my head. Then she looked into my upturned face with a very wistful smile. "No it isn't that," she said at last. "You're not a conceited boy by nature, and so I can safely tell you it is not that. It's deeper." "My character?" She nodded severely. "What can I do to mend it? Do sit down and talk it over. No, really, I won't if you'll only sit down!" She looked at me with a wondering distrust which was much more to my mind than her whole-hearted confidence. How primitive and bestial it looks when you put it down in black and white!--and perhaps after all it is only a feeling peculiar to myself. Anyhow, she sat down. "Now tell me what's amiss with me?" "I'm in love with somebody else," said she. It was my turn to jump out of my chair. "It's nobody in particular," she explained, laughing at the expression of my face: "only an ideal. I've never met the kind of man I mean." "Tell me about him. What does he look like?" "Oh, he might look very much like you." "How dear of you to say that! Well, what is it that he does that I don't do? Just say the word,--teetotal, vegetarian, aeronaut, theosophist, superman. I'll have a try at it, Gladys, if you will only give me an idea what would please you." She laughed at the elasticity of my character. "Well, in the first place, I don't think my ideal would speak like that," said she. "He would be a harder, sterner man, not so ready to adapt himself to a silly girl's whim. But, above all, he must be a man who could do, who could act, who could look Death in the face and have no fear of him, a man of great deeds and strange experiences. It is never a man that I should love, but always the glories he had won; for they would be reflected upon me. Think of Richard Burton! When I read his wife's life of him I could so understand her love! And Lady Stanley! Did you ever read the wonderful last chapter of that book about her husband? These are the sort of men that a woman could worship with all her soul, and yet be the greater, not the less, on account of her love, honored by all the world as the inspirer of noble deeds." She looked so beautiful in her enthusiasm that I nearly brought down the whole level of the interview. I gripped myself hard, and went on with the argument. "We can't all be Stanleys and Burtons," said I; "besides, we don't get the chance,--at least, I never had the chance. If I did, I should try to take it." "But chances are all around you. It is the mark of the kind of man I mean that he makes his own chances. You can't hold him back. I've never met him, and yet I seem to know him so well. There are heroisms all round us waiting to be done. It's for men to do them, and for women to reserve their love as a reward for such men. Look at that young Frenchman who went up last week in a balloon. It was blowing a gale of wind; but because he was announced to go he insisted on starting. The wind blew him fifteen hundred miles in twenty-four hours, and he fell in the middle of Russia. That was the kind of man I mean. Think of the woman he loved, and how other women must have envied her! That's what I should like to be,--envied for my man." "I'd have done it to please you." "But you shouldn't do it merely to please me. You should do it because you can't help yourself, because it's natural to you, because the man in you is crying out for heroic expression. Now, when you described the Wigan coal explosion last month, could you not have gone down and helped those people, in spite of the choke-damp?" "I did." "You never said so." "There was nothing worth bucking about." "I didn't know." She looked at me with rather more interest. "That was brave of you." "I had to. If you want to write good copy, you must be where the things are." "What a prosaic motive! It seems to take all the romance out of it. But, still, whatever your motive, I am glad that you went down that mine." She gave me her hand; but with such sweetness and dignity that I could only stoop and kiss it. "I dare say I am merely a foolish woman with a young girl's fancies. And yet it is so real with me, so entirely part of my very self, that I cannot help acting upon it. If I marry, I do want to marry a famous man!" "Why should you not?" I cried. "It is women like you who brace men up. Give me a chance, and see if I will take it! Besides, as you say, men ought to MAKE their own chances, and not wait until they are given. Look at Clive--just a clerk, and he conquered India! By George! I'll do something in the world yet!" She laughed at my sudden Irish effervescence. "Why not?" she said. "You have everything a man could have,--youth, health, strength, education, energy. I was sorry you spoke. And now I am glad--so glad--if it wakens these thoughts in you!" "And if I do----" Her dear hand rested like warm velvet upon my lips. "Not another word, Sir! You should have been at the office for evening duty half an hour ago; only I hadn't the heart to remind you. Some day, perhaps, when you have won your place in the world, we shall talk it over again." And so it was that I found myself that foggy November evening pursuing the Camberwell tram with my heart glowing within me, and with the eager determination that not another day should elapse before I should find some deed which was worthy of my lady. But who--who in all this wide world could ever have imagined the incredible shape which that deed was to take, or the strange steps by which I was led to the doing of it? And, after all, this opening chapter will seem to the reader to have nothing to do with my narrative; and yet there would have been no narrative without it, for it is only when a man goes out into the world with the thought that there are heroisms all round him, and with the desire all alive in his heart to follow any which may come within sight of him, that he breaks away as I did from the life he knows, and ventures forth into the wonderful mystic twilight land where lie the great adventures and the great rewards. Behold me, then, at the office of the Daily Gazette, on the staff of which I was a most insignificant unit, with the settled determination that very night, if possible, to find the quest which should be worthy of my Gladys! Was it hardness, was it selfishness, that she should ask me to risk my life for her own glorification? Such thoughts may come to middle age; but never to ardent three-and-twenty in the fever of his first love. CHAPTER II "Try Your Luck with Professor Challenger" I always liked McArdle, the crabbed, old, round-backed, red-headed news editor, and I rather hoped that he liked me. Of course, Beaumont was the real boss; but he lived in the rarefied atmosphere of some Olympian height from which he could distinguish nothing smaller than an international crisis or a split in the Cabinet. Sometimes we saw him passing in lonely majesty to his inner sanctum, with his eyes staring vaguely and his mind hovering over the Balkans or the Persian Gulf. He was above and beyond us. But McArdle was his first lieutenant, and it was he that we knew. The old man nodded as I entered the room, and he pushed his spectacles far up on his bald forehead. "Well, Mr. Malone, from all I hear, you seem to be doing very well," said he in his kindly Scotch accent. I thanked him. "The colliery explosion was excellent. So was the Southwark fire. You have the true descreeptive touch. What did you want to see me about?" "To ask a favor." He looked alarmed, and his eyes shunned mine. "Tut, tut! What is it?" "Do you think, Sir, that you could possibly send me on some mission for the paper? I would do my best to put it through and get you some good copy." "What sort of meesion had you in your mind, Mr. Malone?" "Well, Sir, anything that had adventure and danger in it. I really would do my very best. The more difficult it was, the better it would suit me." "You seem very anxious to lose your life." "To justify my life, Sir." "Dear me, Mr. Malone, this is very--very exalted. I'm afraid the day for this sort of thing is rather past. The expense of the 'special meesion' business hardly justifies the result, and, of course, in any case it would only be an experienced man with a name that would command public confidence who would get such an order. The big blank spaces in the map are all being filled in, and there's no room for romance anywhere. Wait a bit, though!" he added, with a sudden smile upon his face. "Talking of the blank spaces of the map gives me an idea. What about exposing a fraud--a modern Munchausen--and making him rideeculous? You could show him up as the liar that he is! Eh, man, it would be fine. How does it appeal to you?" "Anything--anywhere--I care nothing." McArdle was plunged in thought for some minutes. "I wonder whether you could get on friendly--or at least on talking terms with the fellow," he said, at last. "You seem to have a sort of genius for establishing relations with people--seempathy, I suppose, or animal magnetism, or youthful vitality, or something. I am conscious of it myself." "You are very good, sir." "So why should you not try your luck with Professor Challenger, of Enmore Park?" I dare say I looked a little startled. "Challenger!" I cried. "Professor Challenger, the famous zoologist! Wasn't he the man who broke the skull of Blundell, of the Telegraph?" The news editor smiled grimly. "Do you mind? Didn't you say it was adventures you were after?" "It is all in the way of business, sir," I answered. "Exactly. I don't suppose he can always be so violent as that. I'm thinking that Blundell got him at the wrong moment, maybe, or in the wrong fashion. You may have better luck, or more tact in handling him. There's something in your line there, I am sure, and the Gazette should work it." "I really know nothing about him," said I. "I only remember his name in connection with the police-court proceedings, for striking Blundell." "I have a few notes for your guidance, Mr. Malone. I've had my eye on the Professor for some little time." He took a paper from a drawer. "Here is a summary of his record. I give it you briefly:-- "'Challenger, George Edward. Born: Largs, N. B., 1863. Educ.: Largs Academy; Edinburgh University. British Museum Assistant, 1892. Assistant-Keeper of Comparative Anthropology Department, 1893. Resigned after acrimonious correspondence same year. Winner of Crayston Medal for Zoological Research. Foreign Member of'--well, quite a lot of things, about two inches of small type--'Societe Belge, American Academy of Sciences, La Plata, etc., etc. Ex-President Palaeontological Society. Section H, British Association'--so on, so on!--'Publications: "Some Observations Upon a Series of Kalmuck Skulls"; "Outlines of Vertebrate Evolution"; and numerous papers, including "The underlying fallacy of Weissmannism," which caused heated discussion at the Zoological Congress of Vienna. Recreations: Walking, Alpine climbing. Address: Enmore Park, Kensington, W.' "There, take it with you. I've nothing more for you to-night." I pocketed the slip of paper. "One moment, sir," I said, as I realized that it was a pink bald head, and not a red face, which was fronting me. "I am not very clear yet why I am to interview this gentleman. What has he done?" The face flashed back again. "Went to South America on a solitary expedeetion two years ago. Came back last year. Had undoubtedly been to South America, but refused to say exactly where. Began to tell his adventures in a vague way, but somebody started to pick holes, and he just shut up like an oyster. Something wonderful happened--or the man's a champion liar, which is the more probable supposeetion. Had some damaged photographs, said to be fakes. Got so touchy that he assaults anyone who asks questions, and heaves reporters down the stairs. In my opinion he's just a homicidal megalomaniac with a turn for science. That's your man, Mr. Malone. Now, off you run, and see what you can make of him. You're big enough to look after yourself. Anyway, you are all safe. Employers' Liability Act, you know." A grinning red face turned once more into a pink oval, fringed with gingery fluff; the interview was at an end. I walked across to the Savage Club, but instead of turning into it I leaned upon the railings of Adelphi Terrace and gazed thoughtfully for a long time at the brown, oily river. I can always think most sanely and clearly in the open air. I took out the list of Professor Challenger's exploits, and I read it over under the electric lamp. Then I had what I can only regard as an inspiration. As a Pressman, I felt sure from what I had been told that I could never hope to get into touch with this cantankerous Professor. But these recriminations, twice mentioned in his skeleton biography, could only mean that he was a fanatic in science. Was there not an exposed margin there upon which he might be accessible? I would try. I entered the club. It was just after eleven, and the big room was fairly full, though the rush had not yet set in. I noticed a tall, thin, angular man seated in an arm-chair by the fire. He turned as I drew my chair up to him. It was the man of all others whom I should have chosen--Tarp Henry, of the staff of Nature, a thin, dry, leathery creature, who was full, to those who knew him, of kindly humanity. I plunged instantly into my subject. "What do you know of Professor Challenger?" "Challenger?" He gathered his brows in scientific disapproval. "Challenger was the man who came with some cock-and-bull story from South America." "What story?" "Oh, it was rank nonsense about some queer animals he had discovered. I believe he has retracted since. Anyhow, he has suppressed it all. He gave an interview to Reuter's, and there was such a howl that he saw it wouldn't do. It was a discreditable business. There were one or two folk who were inclined to take him seriously, but he soon choked them off." "How?" "Well, by his insufferable rudeness and impossible behavior. There was poor old Wadley, of the Zoological Institute. Wadley sent a message: 'The President of the Zoological Institute presents his compliments to Professor Challenger, and would take it as a personal favor if he would do them the honor to come to their next meeting.' The answer was unprintable." "You don't say?" "Well, a bowdlerized version of it would run: 'Professor Challenger presents his compliments to the President of the Zoological Institute, and would take it as a personal favor if he would go to the devil.'" "Good Lord!" "Yes, I expect that's what old Wadley said. I remember his wail at the meeting, which began: 'In fifty years experience of scientific intercourse----' It quite broke the old man up." "Anything more about Challenger?" "Well, I'm a bacteriologist, you know. I live in a nine-hundred-diameter microscope. I can hardly claim to take serious notice of anything that I can see with my naked eye. I'm a frontiersman from the extreme edge of the Knowable, and I feel quite out of place when I leave my study and come into touch with all you great, rough, hulking creatures. I'm too detached to talk scandal, and yet at scientific conversaziones I HAVE heard something of Challenger, for he is one of those men whom nobody can ignore. He's as clever as they make 'em--a full-charged battery of force and vitality, but a quarrelsome, ill-conditioned faddist, and unscrupulous at that. He had gone the length of faking some photographs over the South American business." "You say he is a faddist. What is his particular fad?" "He has a thousand, but the latest is something about Weissmann and Evolution. He had a fearful row about it in Vienna, I believe." "Can't you tell me the point?" "Not at the moment, but a translation of the proceedings exists. We have it filed at the office. Would you care to come?" "It's just what I want. I have to interview the fellow, and I need some lead up to him. It's really awfully good of you to give me a lift. I'll go with you now, if it is not too late." Half an hour later I was seated in the newspaper office with a huge tome in front of me, which had been opened at the article "Weissmann versus Darwin," with the sub heading, "Spirited Protest at Vienna. Lively Proceedings." My scientific education having been somewhat neglected, I was unable to follow the whole argument, but it was evident that the English Professor had handled his subject in a very aggressive fashion, and had thoroughly annoyed his Continental colleagues. "Protests," "Uproar," and "General appeal to the Chairman" were three of the first brackets which caught my eye. Most of the matter might have been written in Chinese for any definite meaning that it conveyed to my brain. "I wish you could translate it into English for me," I said, pathetically, to my help-mate. "Well, it is a translation." "Then I'd better try my luck with the original." "It is certainly rather deep for a layman." "If I could only get a single good, meaty sentence which seemed to convey some sort of definite human idea, it would serve my turn. Ah, yes, this one will do. I seem in a vague way almost to understand it. I'll copy it out. This shall be my link with the terrible Professor." "Nothing else I can do?" "Well, yes; I propose to write to him. If I could frame the letter here, and use your address it would give atmosphere." "We'll have the fellow round here making a row and breaking the furniture." "No, no; you'll see the letter--nothing contentious, I assure you." "Well, that's my chair and desk. You'll find paper there. I'd like to censor it before it goes." It took some doing, but I flatter myself that it wasn't such a bad job when it was finished. I read it aloud to the critical bacteriologist with some pride in my handiwork. "DEAR PROFESSOR CHALLENGER," it said, "As a humble student of Nature, I have always taken the most profound interest in your speculations as to the differences between Darwin and Weissmann. I have recently had occasion to refresh my memory by re-reading----" "You infernal liar!" murmured Tarp Henry. --"by re-reading your masterly address at Vienna. That lucid and admirable statement seems to be the last word in the matter. There is one sentence in it, however--namely: 'I protest strongly against the insufferable and entirely dogmatic assertion that each separate id is a microcosm possessed of an historical architecture elaborated slowly through the series of generations.' Have you no desire, in view of later research, to modify this statement? Do you not think that it is over-accentuated? With your permission, I would ask the favor of an interview, as I feel strongly upon the subject, and have certain suggestions which I could only elaborate in a personal conversation. With your consent, I trust to have the honor of calling at eleven o'clock the day after to-morrow (Wednesday) morning. "I remain, Sir, with assurances of profound respect, yours very truly, EDWARD D. MALONE." "How's that?" I asked, triumphantly. "Well if your conscience can stand it----" "It has never failed me yet." "But what do you mean to do?" "To get there. Once I am in his room I may see some opening. I may even go the length of open confession. If he is a sportsman he will be tickled." "Tickled, indeed! He's much more likely to do the tickling. Chain mail, or an American football suit--that's what you'll want. Well, good-bye. I'll have the answer for you here on Wednesday morning--if he ever deigns to answer you. He is a violent, dangerous, cantankerous character, hated by everyone who comes across him, and the butt of the students, so far as they dare take a liberty with him. Perhaps it would be best for you if you never heard from the fellow at all." CHAPTER III "He is a Perfectly Impossible Person" My friend's fear or hope was not destined to be realized. When I called on Wednesday there was a letter with the West Kensington postmark upon it, and my name scrawled across the envelope in a handwriting which looked like a barbed-wire railing. The contents were as follows:-- "ENMORE PARK, W. "SIR,--I have duly received your note, in which you claim to endorse my views, although I am not aware that they are dependent upon endorsement either from you or anyone else. You have ventured to use the word 'speculation' with regard to my statement upon the subject of Darwinism, and I would call your attention to the fact that such a word in such a connection is offensive to a degree. The context convinces me, however, that you have sinned rather through ignorance and tactlessness than through malice, so I am content to pass the matter by. You quote an isolated sentence from my lecture, and appear to have some difficulty in understanding it. I should have thought that only a sub-human intelligence could have failed to grasp the point, but if it really needs amplification I shall consent to see you at the hour named, though visits and visitors of every sort are exceeding distasteful to me. As to your suggestion that I may modify my opinion, I would have you know that it is not my habit to do so after a deliberate expression of my mature views. You will kindly show the envelope of this letter to my man, Austin, when you call, as he has to take every precaution to shield me from the intrusive rascals who call themselves 'journalists.' "Yours faithfully, "GEORGE EDWARD CHALLENGER." This was the letter that I read aloud to Tarp Henry, who had come down early to hear the result of my venture. His only remark was, "There's some new stuff, cuticura or something, which is better than arnica." Some people have such extraordinary notions of humor. It was nearly half-past ten before I had received my message, but a taxicab took me round in good time for my appointment. It was an imposing porticoed house at which we stopped, and the heavily-curtained windows gave every indication of wealth upon the part of this formidable Professor. The door was opened by an odd, swarthy, dried-up person of uncertain age, with a dark pilot jacket and brown leather gaiters. I found afterwards that he was the chauffeur, who filled the gaps left by a succession of fugitive butlers. He looked me up and down with a searching light blue eye. "Expected?" he asked. "An appointment." "Got your letter?" I produced the envelope. "Right!" He seemed to be a person of few words. Following him down the passage I was suddenly interrupted by a small woman, who stepped out from what proved to be the dining-room door. She was a bright, vivacious, dark-eyed lady, more French than English in her type. "One moment," she said. "You can wait, Austin. Step in here, sir. May I ask if you have met my husband before?" "No, madam, I have not had the honor." "Then I apologize to you in advance. I must tell you that he is a perfectly impossible person--absolutely impossible. If you are forewarned you will be the more ready to make allowances." "It is most considerate of you, madam." "Get quickly out of the room if he seems inclined to be violent. Don't wait to argue with him. Several people have been injured through doing that. Afterwards there is a public scandal and it reflects upon me and all of us. I suppose it wasn't about South America you wanted to see him?" I could not lie to a lady. "Dear me! That is his most dangerous subject. You won't believe a word he says--I'm sure I don't wonder. But don't tell him so, for it makes him very violent. Pretend to believe him, and you may get through all right. Remember he believes it himself. Of that you may be assured. A more honest man never lived. Don't wait any longer or he may suspect. If you find him dangerous--really dangerous--ring the bell and hold him off until I come. Even at his worst I can usually control him." With these encouraging words the lady handed me over to the taciturn Austin, who had waited like a bronze statue of discretion during our short interview, and I was conducted to the end of the passage. There was a tap at a door, a bull's bellow from within, and I was face to face with the Professor. He sat in a rotating chair behind a broad table, which was covered with books, maps, and diagrams. As I entered, his seat spun round to face me. His appearance made me gasp. I was prepared for something strange, but not for so overpowering a personality as this. It was his size which took one's breath away--his size and his imposing presence. His head was enormous, the largest I have ever seen upon a human being. I am sure that his top-hat, had I ever ventured to don it, would have slipped over me entirely and rested on my shoulders. He had the face and beard which I associate with an Assyrian bull; the former florid, the latter so black as almost to have a suspicion of blue, spade-shaped and rippling down over his chest. The hair was peculiar, plastered down in front in a long, curving wisp over his massive forehead. The eyes were blue-gray under great black tufts, very clear, very critical, and very masterful. A huge spread of shoulders and a chest like a barrel were the other parts of him which appeared above the table, save for two enormous hands covered with long black hair. This and a bellowing, roaring, rumbling voice made up my first impression of the notorious Professor Challenger. "Well?" said he, with a most insolent stare. "What now?" I must keep up my deception for at least a little time longer, otherwise here was evidently an end of the interview. "You were good enough to give me an appointment, sir," said I, humbly, producing his envelope. He took my letter from his desk and laid it out before him. "Oh, you are the young person who cannot understand plain English, are you? My general conclusions you are good enough to approve, as I understand?" "Entirely, sir--entirely!" I was very emphatic. "Dear me! That strengthens my position very much, does it not? Your age and appearance make your support doubly valuable. Well, at least you are better than that herd of swine in Vienna, whose gregarious grunt is, however, not more offensive than the isolated effort of the British hog." He glared at me as the present representative of the beast. "They seem to have behaved abominably," said I. "I assure you that I can fight my own battles, and that I have no possible need of your sympathy. Put me alone, sir, and with my back to the wall. G. E. C. is happiest then. Well, sir, let us do what we can to curtail this visit, which can hardly be agreeable to you, and is inexpressibly irksome to me. You had, as I have been led to believe, some comments to make upon the proposition which I advanced in my thesis." There was a brutal directness about his methods which made evasion difficult. I must still make play and wait for a better opening. It had seemed simple enough at a distance. Oh, my Irish wits, could they not help me now, when I needed help so sorely? He transfixed me with two sharp, steely eyes. "Come, come!" he rumbled. "I am, of course, a mere student," said I, with a fatuous smile, "hardly more, I might say, than an earnest inquirer. At the same time, it seemed to me that you were a little severe upon Weissmann in this matter. Has not the general evidence since that date tended to--well, to strengthen his position?" "What evidence?" He spoke with a menacing calm. "Well, of course, I am aware that there is not any what you might call DEFINITE evidence. I alluded merely to the trend of modern thought and the general scientific point of view, if I might so express it." He leaned forward with great earnestness. "I suppose you are aware," said he, checking off points upon his fingers, "that the cranial index is a constant factor?" "Naturally," said I. "And that telegony is still sub judice?" "Undoubtedly." "And that the germ plasm is different from the parthenogenetic egg?" "Why, surely!" I cried, and gloried in my own audacity. "But what does that prove?" he asked, in a gentle, persuasive voice. "Ah, what indeed?" I murmured. "What does it prove?" "Shall I tell you?" he cooed. "Pray do." "It proves," he roared, with a sudden blast of fury, "that you are the damnedest imposter in London--a vile, crawling journalist, who has no more science than he has decency in his composition!" He had sprung to his feet with a mad rage in his eyes. Even at that moment of tension I found time for amazement at the discovery that he was quite a short man, his head not higher than my shoulder--a stunted Hercules whose tremendous vitality had all run to depth, breadth, and brain. "Gibberish!" he cried, leaning forward, with his fingers on the table and his face projecting. "That's what I have been talking to you, sir--scientific gibberish! Did you think you could match cunning with me--you with your walnut of a brain? You think you are omnipotent, you infernal scribblers, don't you? That your praise can make a man and your blame can break him? We must all bow to you, and try to get a favorable word, must we? This man shall have a leg up, and this man shall have a dressing down! Creeping vermin, I know you! You've got out of your station. Time was when your ears were clipped. You've lost your sense of proportion. Swollen gas-bags! I'll keep you in your proper place. Yes, sir, you haven't got over G. E. C. There's one man who is still your master. He warned you off, but if you WILL come, by the Lord you do it at your own risk. Forfeit, my good Mr. Malone, I claim forfeit! You have played a rather dangerous game, and it strikes me that you have lost it." "Look here, sir," said I, backing to the door and opening it; "you can be as abusive as you like. But there is a limit. You shall not assault me." "Shall I not?" He was slowly advancing in a peculiarly menacing way, but he stopped now and put his big hands into the side-pockets of a rather boyish short jacket which he wore. "I have thrown several of you out of the house. You will be the fourth or fifth. Three pound fifteen each--that is how it averaged. Expensive, but very necessary. Now, sir, why should you not follow your brethren? I rather think you must." He resumed his unpleasant and stealthy advance, pointing his toes as he walked, like a dancing master. I could have bolted for the hall door, but it would have been too ignominious. Besides, a little glow of righteous anger was springing up within me. I had been hopelessly in the wrong before, but this man's menaces were putting me in the right. "I'll trouble you to keep your hands off, sir. I'll not stand it." "Dear me!" His black moustache lifted and a white fang twinkled in a sneer. "You won't stand it, eh?" "Don't be such a fool, Professor!" I cried. "What can you hope for? I'm fifteen stone, as hard as nails, and play center three-quarter every Saturday for the London Irish. I'm not the man----" It was at that moment that he rushed me. It was lucky that I had opened the door, or we should have gone through it. We did a Catharine-wheel together down the passage. Somehow we gathered up a chair upon our way, and bounded on with it towards the street. My mouth was full of his beard, our arms were locked, our bodies intertwined, and that infernal chair radiated its legs all round us. The watchful Austin had thrown open the hall door. We went with a back somersault down the front steps. I have seen the two Macs attempt something of the kind at the halls, but it appears to take some practise to do it without hurting oneself. The chair went to matchwood at the bottom, and we rolled apart into the gutter. He sprang to his feet, waving his fists and wheezing like an asthmatic. "Had enough?" he panted. "You infernal bully!" I cried, as I gathered myself together. Then and there we should have tried the thing out, for he was effervescing with fight, but fortunately I was rescued from an odious situation. A policeman was beside us, his notebook in his hand. "What's all this? You ought to be ashamed" said the policeman. It was the most rational remark which I had heard in Enmore Park. "Well," he insisted, turning to me, "what is it, then?" "This man attacked me," said I. "Did you attack him?" asked the policeman. The Professor breathed hard and said nothing. "It's not the first time, either," said the policeman, severely, shaking his head. "You were in trouble last month for the same thing. You've blackened this young man's eye. Do you give him in charge, sir?" I relented. "No," said I, "I do not." "What's that?" said the policeman. "I was to blame myself. I intruded upon him. He gave me fair warning." The policeman snapped up his notebook. "Don't let us have any more such goings-on," said he. "Now, then! Move on, there, move on!" This to a butcher's boy, a maid, and one or two loafers who had collected. He clumped heavily down the street, driving this little flock before him. The Professor looked at me, and there was something humorous at the back of his eyes. "Come in!" said he. "I've not done with you yet." The speech had a sinister sound, but I followed him none the less into the house. The man-servant, Austin, like a wooden image, closed the door behind us. ================================================ FILE: core-tools/tests-regression/word-properties/out.ok ================================================ Anthropology c: nthr Following p: ol-lo Frenchman c: nchm Gazette p: et-te HTML c: HTML MORROW p: OR-RO Palaeontological l: 16 amplification l: 13 assault p: as-sa assaults p: as-sa asthmatic c: sthm attack p: at-ta attacked p: at-ta bacteriologist l: 14 better p: et-te bottom p: ot-to conversaziones l: 14 correspondence l: 14 deed p: de-ed deeds p: de-ed demonstrate c: nstr determination l: 13 difficult p: if-fi difficulty p: if-fi discreditable l: 13 effervescence p: ef-fe l: 13 effervescing p: ef-fe excellent p: el-le extraordinary l: 13 fallacy p: al-la follow p: ol-lo followed p: ol-lo follows p: ol-lo friendship c: ndsh glorification l: 13 grinning p: in-ni handwriting c: ndwr indeed p: de-ed inexpressibly l: 13 insignificant l: 13 instincts c: ncts international l: 13 length c: ngth letter p: et-te matchwood c: tchw mission p: is-si morrow p: or-ro narrative p: ar-ra parthenogenetic l: 15 passage p: as-sa permission p: is-si possessed p: es-se pressed p: es-se recriminations l: 14 representative l: 14 simultaneously l: 14 sportsman c: rtsm stepped p: ep-pe strength c: ngth strengthen c: ngth strengthens c: ngth suppressed p: es-se teetotal p: te-et thoughtfully c: ghtf thoughts c: ghts understanding l: 13 unscrupulous c: nscr unshrinking c: nshr watchful c: tchf withdrawn c: thdr ================================================ FILE: eval/.gitignore ================================================ *.class access-small.log access.log books.txt books1.txt character.txt emptydir emptygit hier linux.new linux.old linux out time ================================================ FILE: eval/Makefile ================================================ include ../.config all: data WebStats.class TextProperties.class sh perf-eval.sh data: books.txt linux access.log linux.new linux.old emptydir access-small.log # To ensure repeatability use specific hash for an old and a new Linux commit LOLD=1da177e4c3f41524e886b7f1b8a0c1fc7321cac2 LNEW=899552d6e84babd24611fd36ac7051068cb1eb2d ELOLD:=$(LOLD) export ELOLD ELNEW:=$(LNEW) export ELNEW # History of the United States by Charles A. Beard and Mary Ritter Beard # The Adventures of Sherlock Holmes by Arthur Conan Doyle # Les Misérables by Victor Hugo books1.txt: curl http://www.gutenberg.org/files/16960/16960.txt \ http://www.gutenberg.org/files/1661/1661.txt \ http://www.gutenberg.org/files/135/135.txt >books1.txt books.txt: books1.txt for i in `seq 1 10` ; do cat $? ; done >$@ access.log: curl ftp://ita.ee.lbl.gov/traces/clarknet_access_log_Aug28.gz |\ gzip -dc >$@ access-small.log: access.log head -10 $? >$@ linux: git clone https://github.com/torvalds/linux.git linux.new: linux (cd linux && git checkout $(LNEW)) mkdir $@ tar -C linux --exclude .git -cf - . | tar -C $@ -xf - (cd linux && git checkout master) touch $@ linux.old: linux (cd linux && git checkout $(LOLD)) mkdir $@ tar -C linux --exclude .git -cf - . | tar -C $@ -xf - (cd linux && git checkout master) touch $@ emptydir: linux mkdir $@ cp linux/kernel/up.c $@/ emptygit: mkdir $@ cd $@ ; git init ; echo hello >hello ; git add hello ; git commit -am 'Add hello' WebStats.class: WebStats.java javac $? TextProperties.class: TextProperties.java javac $? clean: rm -rf `cat .gitignore` ================================================ FILE: eval/README.md ================================================ The evaluation method for the classic shell performance is based on an earlier version of *dgsh*, *sgsh*, which can translate its input into plain shell scripts. To measure the classic shell performance, checkout a release containing *sgsh*, such as `46b1aec`, run `make`, and then run the baseline measurements. ================================================ FILE: eval/SConstruct ================================================ # ft2d example souped up to compare with dgsh from rsf.proj import * Flow('pulse',None, ''' spike n1=10000 n2=10000 d1=1 d2=1 nsp=2 k1=16,17 k2=5,5 mag=16,16 label1='time' label2='space' unit1= unit2= | smooth rect2=2 | smooth rect2=2 ''') Plot('pulse','grey pclip=100 wanttitle=n') Flow('ft2d','pulse','fft1 | fft3 axis=2 pad=1 | real') Flow('ft2d2','ft2d','window f1=1 | reverse which=3 | cat axis=1 $SOURCE') Plot('ft2d','ft2d2', 'grey pclip=100 wanttitle=n label1="1/time" label2="1/space" ') Result('ft2dofpulse','pulse ft2d','SideBySideIso',vppen='yscale=1.25') Flow('air',None, ''' spike n1=10000 d1=1 o1=5000 nsp=4 k1=1,2,3,4 mag=1,3,3,1 label1='time' unit1= | spray n=32 d=1 o=0 | put label2=space | lmostretch delay=0 v0=-1 ''') Flow('air2','air','window f2=1 | reverse which=2 | cat axis=2 $SOURCE') Plot('airtx','air2','grey pclip=100 wanttitle=n') Flow('airft','air2','fft1 | fft3 sign=1') Flow('airft1 airftr airfti','airft', ''' real > ${TARGETS[1]} && imag < $SOURCE > ${TARGETS[2]} && math re=${TARGETS[1]} im=${TARGETS[2]} output="sqrt(re*re+im*im)" ''') Flow('airft2','airft1','window f1=1 | reverse which=3 | cat axis=1 $SOURCE') Plot('airfk','airft2', 'grey pclip=100 wanttitle=n label1="1/time" label2="1/space" ') Result('airwave','airtx airfk','SideBySideIso',vppen='yscale=1.25') End() ================================================ FILE: eval/TextProperties.java ================================================ /*- * * Collect and print text frequency statistics from the text * read from the standard input. * * Copyright 2014 Diomidis Spinellis * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ import java.io.BufferedReader; import java.io.BufferedWriter; import java.io.FileOutputStream; import java.io.InputStreamReader; import java.io.OutputStreamWriter; import java.io.PrintWriter; import java.util.Comparator; import java.util.HashMap; import java.util.Map; import java.util.TreeMap; import java.util.TreeSet; class TextProperties { /** Increment the map's member by one */ static void increment(Map map, String member) { Integer i = map.get(member); if (i == null) i = new Integer(1); else i = new Integer(i.intValue() + 1); map.put(member, i); } /** Increment in the map all n-grams in the specified word */ static void ngramIncrement(Map map, StringBuffer word, int n) { for (int i = 0; i <= word.length() - n; i++) increment(map, word.substring(i, i + n)); } /** Save the contents of the given map ordered by their values. */ static void sortedList(Map map, String fileName) { // Sort by value TreeSet > valueOrder = new TreeSet>(new Comparator>() { public int compare(Map.Entry a, Map.Entry b) { int c = -a.getValue().compareTo(b.getValue()); if (c != 0) return (c); else return -a.getKey().compareTo(b.getKey()); } } ); valueOrder.addAll(map.entrySet()); try { PrintWriter out = new PrintWriter(new BufferedWriter(new OutputStreamWriter(new FileOutputStream(fileName)))); // Print entries for (Map.Entry e : valueOrder) out.println(e.getValue() + " " + e.getKey()); out.close(); } catch (Exception e) { System.err.println("Error writing to file: " + e); System.exit(1); } } public static void main(String args[]) { // Data structures for gathering the results HashMap countCharacter = new HashMap(); HashMap countDigram = new HashMap(); HashMap countTrigram = new HashMap(); HashMap countWord = new HashMap(); BufferedReader in = new BufferedReader(new InputStreamReader(System.in)); try { int c; StringBuffer word = new StringBuffer(); while ((c = System.in.read()) != -1) { if (Character.isLetter(c)) word.append((char)c); else { if (word.length() > 0) { increment(countWord, word.toString()); ngramIncrement(countCharacter, word, 1); ngramIncrement(countDigram, word, 2); ngramIncrement(countTrigram, word, 3); word.setLength(0); } } } } catch (Exception e) { System.err.println("Error reading character: " + e); System.exit(1); } sortedList(countWord, "words.txt"); sortedList(countCharacter, "character.txt"); sortedList(countDigram, "digram.txt"); sortedList(countTrigram, "trigram.txt"); } } ================================================ FILE: eval/WebStats.java ================================================ /*- * * Collect and print Web statistics from common log format data * read from the standard input. * * Copyright 2004-2013 Diomidis Spinellis * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */ import java.io.BufferedReader; import java.io.InputStreamReader; import java.util.Calendar; import java.util.Comparator; import java.util.GregorianCalendar; import java.util.HashMap; import java.util.Locale; import java.util.Map; import java.util.TreeMap; import java.util.TreeSet; import java.util.regex.Matcher; import java.util.regex.Pattern; import java.util.regex.PatternSyntaxException; import javax.management.openmbean.InvalidKeyException; class WebStats { /** Print header with an underline of the same length */ static void header(String s) { System.out.println(); System.out.println(s); for (int i = 0; i < s.length(); i++) System.out.print('-'); System.out.println(); } /** Add to the map's member the specified value */ static void add(Map map, String member, int value) { Integer i = map.get(member); if (i == null) i = new Integer(value); else i = new Integer(i.intValue() + value); map.put(member, i); } /** List the contents of the given map */ static void list(String title, Map map) { header(title); for (Map.Entry e : map.entrySet()) System.out.println(e.getValue() + " " + e.getKey()); } /** List the n-top contents of the given map ordered by their values. */ static void sortedList(String title, Map map, int n) { header(title); // Sort by value TreeSet > valueOrder = new TreeSet>(new Comparator>() { public int compare(Map.Entry a, Map.Entry b) { int c = -a.getValue().compareTo(b.getValue()); if (c != 0) return (c); else return -a.getKey().compareTo(b.getKey()); } } ); valueOrder.addAll(map.entrySet()); // Print top n entries int i = 0; for (Map.Entry e : valueOrder) { System.out.println(e.getValue() + " " + e.getKey()); if (++i == n) break; } } public static void main(String args[]) { // Compile regular expressions Pattern logLinePattern = null; Pattern areaPattern = null; Pattern numericIPPattern = null; Pattern domainPattern = null; Pattern topDomainPattern = null; try { // A standard log line is a line like: // 192.168.136.16 - - [26/Jan/2004:19:45:48 +0200] "GET /c136.html HTTP/1.1" 200 1674 logLinePattern = Pattern.compile( "(?[-\\w.]+)\\s+" + // Host "([-\\w]+)\\s+" + // Logname "([-\\w]+)\\s+" + // User "\\[(?\\d+)/" + // Day of month "(?\\w+)/" + // Month "(?\\d+):" + // Year "(?\\d+):" + // Hour "(?\\d+)" + // Minute "([^]]+?)\\]\\s+" + // Rest of time "\"([-\\w]+)\\s*" + // Request verb "(?[^\\s]*)" + // Request URL "([^\"]*?)\"\\s+" + // Request protocol etc. "(\\d+)\\s+" + // Status "(?[-\\d]+)" // Bytes ); // The first part of a request areaPattern = Pattern.compile("^/?([^/]+)"); // Host names numericIPPattern = Pattern.compile("\\.\\d+$"); domainPattern = Pattern.compile("[^.]\\.(.*)"); topDomainPattern = Pattern.compile(".*\\.(.*)"); } catch (PatternSyntaxException e) { System.err.println("Invalid RE syntax: " + e); System.exit(1); } // A Gregorian calendar, used for converting dates to day of week GregorianCalendar cal = new GregorianCalendar(); // Map from month name to month value Locale english = new Locale("EN"); Map monthValue = cal.getDisplayNames(Calendar.MONTH, Calendar.SHORT, english); // Data structures for gathering the results HashMap hostCount = new HashMap(); HashMap hourCount = new HashMap(); HashMap requestCount = new HashMap(); HashMap areaCount = new HashMap(); HashMap transferVolume = new HashMap(); HashMap topDomainCount = new HashMap(); HashMap domainCount = new HashMap(); TreeMap dateCount = new TreeMap(); HashMap dowCount = new HashMap(); long accessCount = 0; long byteCount = 0; long logByteCount = 0; BufferedReader in = new BufferedReader(new InputStreamReader(System.in)); try { String line; while ((line = in.readLine()) != null) { logByteCount += line.length() + 1; accessCount++; Matcher logLineMatched = logLinePattern.matcher(line); if (!logLineMatched.matches()) continue; add(hourCount, logLineMatched.group("hour"), 1); // Remote host String host = logLineMatched.group("host"); add(hostCount, host, 1); // Transfer bytes by remote host try { int bytes = Integer.parseInt(logLineMatched.group("bytes")); byteCount += bytes; add(transferVolume, host, bytes); } catch (NumberFormatException e) { } // Remote domains and top domains Matcher numericIPMatched = numericIPPattern.matcher(host); if (!numericIPMatched.find()) { Matcher topDomainMatched = topDomainPattern.matcher(host); if (topDomainMatched.find()) add(topDomainCount, topDomainMatched.group(1), 1); Matcher domainMatched = domainPattern.matcher(host); if (domainMatched.find()) add(domainCount, domainMatched.group(1), 1); } // Request String url = logLineMatched.group("url"); add(requestCount, url, 1); // Area part of the URL Matcher areaMatched = areaPattern.matcher(url); if (areaMatched.find()) add(areaCount, areaMatched.group(1), 1); // Date String date = logLineMatched.group("day") + "/" + logLineMatched.group("month") + "/" + logLineMatched.group("year"); add(dateCount, date, 1); // Day of week try { Integer month = monthValue.get(logLineMatched.group("month")); if (month == null) throw new InvalidKeyException(); int day = Integer.parseInt(logLineMatched.group("day")); int year = Integer.parseInt(logLineMatched.group("year")); cal.set(year, month, day); cal.get(Calendar.DAY_OF_WEEK); String dayOfWeek = cal.getDisplayName(Calendar.DAY_OF_WEEK, Calendar.SHORT, english); add(dowCount, dayOfWeek, 1); } catch (NumberFormatException | InvalidKeyException e) { ; } } } catch (Exception e) { System.err.println("Error reading line: " + e); System.exit(1); } System.out.println(" WWW server statistics"); System.out.println(" ====================="); header("Summary"); System.out.println("Number of accesses: " + accessCount); System.out.println("Number of Gbytes transferred: " + byteCount / 1024 / 1024 / 1024); System.out.println("Number of hosts: " + hostCount.size()); System.out.println("Number of domains: " + domainCount.size()); System.out.println("Number of top level domains: " + topDomainCount.size()); System.out.println("Number of different pages: " + requestCount.size()); System.out.println("Accesses per day: " + accessCount / dateCount.size()); System.out.println("Mbytes per day: " + byteCount / dateCount.size() / 1024 / 1024); System.out.println("Mbytes log file size: " + logByteCount / 1024 / 1024); sortedList("Top 20 Requests", requestCount, 20); sortedList("Top 20 Area Requests", areaCount, 20); sortedList("Top 10 Hosts", hostCount, 10); sortedList("Top 10 Hosts by Transfer", transferVolume, 10); sortedList("Top 10 Domains", domainCount, 10); sortedList("Top 20 Top Level Domain Accesses", topDomainCount, 20); sortedList("Accesses by Day of Week", dowCount, -1); sortedList("Accesses by Local Hour", hourCount, -1); list("Accesses by Date", dateCount); } } ================================================ FILE: eval/eval-lib.sh ================================================ #!/bin/sh # # Portable time function # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Run the command with the specified description # logging the output and time to {out,err,time} directories timerun() { local -r DESC="$1" shift case `uname` in FreeBSD|Darwin) /usr/bin/time -l sh -c "$@ >out/$DESC 2>err/$DESC" 2>time/$DESC ;; Linux|CYGWIN*) /usr/bin/time -v -o time/$DESC "$@" >out/$DESC 2>err/$DESC ;; esac } # Download the specified URL, if needed, to a file of its last component download() { local -r URL="$1" FILENAME="`basename $URL`" if ! [ -f "$FILENAME" ] then wget $URL 2>/dev/null || curl $URL >$FILENAME fi } # Output the specified URL to standard output download_stdout() { local -r URL="$1" curl $URL 2>/dev/null || wget -O - $URL } ================================================ FILE: eval/ft2d.sh ================================================ #!/usr/local/bin/dgsh # # Modified version of the example with the same name, to test performance # on larger data sets mkdir -p Fig # The SConstruct SideBySideIso "Result" method side_by_side_iso() { vppen size=r vpstyle=n gridnum=2,1 $* } # A broadened pulse and the real part of its 2D Fourier transform scatter |{ .| sfspike n1=10000 n2=10000 d1=1 d2=1 nsp=2 k1=16,17 k2=5,5 mag=16,16 \ label1='time' label2='space' unit1= unit2= | sfsmooth rect2=2 | sfsmooth rect2=2 |{ -| sfgrey pclip=100 wanttitle=n |>/stream/pulse.vpl -| sffft1 | sffft3 axis=2 pad=1 | sfreal |{ -| dgsh-tee -I |>/stream/ft2d -| sfwindow f1=1 | sfreverse which=3 | sfcat axis=1 /stream/ft2d | sfgrey pclip=100 wanttitle=n \ label1="1/time" label2="1/space" |>/stream/ft2d.vpl |} |} .| side_by_side_iso /stream/pulse.vpl /stream/ft2d.vpl \ yscale=1.25 >Fig/ft2dofpulse.vpl |. |} # A simulated air wave and the amplitude of its 2D Fourier transform scatter |{ .| sfspike n1=10000 d1=1 o1=5000 nsp=4 k1=1,2,3,4 mag=1,3,3,1 \ label1='time' unit1= | sfspray n=32 d=1 o=0 | sfput label2=space | sflmostretch delay=0 v0=-1 |{ -| dgsh-tee -I |>/stream/air -| sfwindow f2=1 | sfreverse which=2 | sfcat axis=2 /stream/air |{ -| sfgrey pclip=100 wanttitle=n |>/stream/airtx.vpl -| sffft1 | sffft3 sign=1 |{ -| sfreal |>/stream/airftr -| sfimag |>/stream/airfti |} |} |} .| sfmath re=/stream/airftr im=/stream/airfti output="sqrt(re*re+im*im)" |{ -| dgsh-tee -I |>/stream/airft1 -| sfwindow f1=1 | sfreverse which=3 | sfcat axis=1 /stream/airft1 | sfgrey pclip=100 wanttitle=n label1="1/time" \ label2="1/space" |>/stream/airfk.vpl |} .| side_by_side_iso /stream/airtx.vpl /stream/airfk.vpl \ yscale=1.25 >Fig/airwave.vpl |. |} ================================================ FILE: eval/log-grow.pl ================================================ #!/usr/bin/perl # # Grow the size of a web server log file by a factor of N, # which is specified as the first argument, by repeating each # line N times. The host name and the request are changed by # substituting them with a random pick form a list of 1000 entries. # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # use strict; use warnings; # Reproducable results srand(0); my $x = $ARGV[0]; shift; my @host; my @request; while (<>) { print; my ($host, $time, $request, $rest) = ($_ =~ m/^([^\s]+)(\s+[^"]+)(\"[^"]*\")(.*)$/); $host[int(rand() * 1000)] = $host; $request[int(rand() * 1000)] = $request; for (my $i = 0; $i < $x - 1; $i++) { my $hpos = int(rand() * 1000); my $nhost; $host = $nhost if ($nhost = $host[$hpos]); my $ppos = int(rand() * 1000); my $nrequest; $request = $nrequest if ($nrequest = $request[$ppos]); print $host, $time, $request, $rest, "\n"; } } ================================================ FILE: eval/perf-eval.sh ================================================ #!/bin/sh # # Benchamrk the performance of alternative implementations of the # provided example files # # Copyright 2014 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # mkdir -p time out rm -f time/* RESULTS='*.rsf *.vpl Fig *.rsf\@ .sconsign.dbhash hier words.txt trigram.txt digram.txt character.txt' PERFDIR=`pwd` # Run the specified command twice, in order to minimize caching effects run_twice() { local input=$1 shift rm -rf $RESULTS "$@" <$input rm -rf $RESULTS "$@" <$input rm -rf $RESULTS } measure() { local flags=$1 local script=$2 local size=$3 local input=$4 local name="$PERFDIR/time/$script:$size:$flags" shift 4 echo 1>&2 "Running for $name" run_twice $input /usr/bin/time -p -o $name $PERFDIR/../dgsh -p $PERFDIR/.. $flags \ $PERFDIR/../example/$script "$@" } run_examples() { local flags="$1" local size=$2 local text=$3 local old=$4 local new=$5 local weblog=$6 local git=$7 local range=$8 ( cd $git ; measure "$flags" commit-stats.sh $size /dev/null ) >out/commit-stats.out measure "$flags" spell-highlight.sh $size $text >out/spell.out measure "$flags" map-hierarchy.sh $size /dev/null $old $new hier measure "$flags" code-metrics.sh $size /dev/null $new >out/metrics.out measure "$flags" duplicate-files.sh $size /dev/null $new >out/dup.out measure "$flags" word-properties.sh $size $text >out/word-properties.out measure "$flags" compress-compare.sh $size $text >out/compress.out measure "$flags" web-log-report.sh $size $weblog >out/weblog.out measure "$flags" text-properties.sh $size $text } # Alternative implementations of (modified) ft2d /usr/bin/time -p -o time/ft2d.sh:large: ../dgsh -p `pwd`/.. ft2d.sh /usr/bin/time -p -o time/ft2d.sh:large:-S ../dgsh -p `pwd`/.. -S ft2d.sh # Compare with native tool run_twice /dev/null /usr/bin/time -p -o time/ft2d.scons:large: scons # Alternative implementation of text statistics run_twice books.txt /usr/bin/time -p -o time/TextProperties:large: java TextProperties # Alternative implementations of web log statistics run_twice access.log /usr/bin/time -p -o time/WebStats:large: java WebStats >out/webstats-java.out run_twice access.log /usr/bin/time -p -o time/web-log-report.pl:large: perl web-log-report.pl >out/webstats-pl.out # sh and dgsh implementations of other examples for flags in '' -S do run_examples "$flags" small /dev/null emptydir emptydir access-small.log emptygit HEAD run_examples "$flags" large books.txt linux.old linux.new access.log linux $ELOLD..$ELNEW done ================================================ FILE: eval/web-log-report.pl ================================================ #!/usr/bin/perl # # Copyright 1995-2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # use Time::Local; # LogFormat "%h %l %u %t \"%r\" %s %b # %...h: remote host # %...l: remote logname (from identd, if supplied) # %...u: remote user (from auth; may be bogus if return status (%s) is 401) # %...t: time, in common log format time format # %...r: first line of request # %...s: status. For requests that got internally redirected, this # is status of the *original* request --- %...>s for the last. # %...b: bytes sent. # %...{Foobar}i: The contents of Foobar: header line(s) in the request # sent to the client. # %...{Foobar}o: The contents of Foobar: header line(s) in the reply. # $monthcard{'Jan'} = '01'; $monthcard{'Feb'} = '02'; $monthcard{'Mar'} = '03'; $monthcard{'Apr'} = '04'; $monthcard{'May'} = '05'; $monthcard{'Jun'} = '06'; $monthcard{'Jul'} = '07'; $monthcard{'Aug'} = '08'; $monthcard{'Sep'} = '09'; $monthcard{'Oct'} = '10'; $monthcard{'Nov'} = '11'; $monthcard{'Dec'} = '12'; $downame{1} = 'Mon'; $downame{2} = 'Tue'; $downame{3} = 'Wed'; $downame{4} = 'Thu'; $downame{5} = 'Fri'; $downame{6} = 'Sat'; $downame{0} = 'Sun'; while (<>) { $logbytecount += length($_); chop; if (!( # ifweb.dimi.uniud.it - - [11/Mar/2002:22:31:30 +0200] "GET /pubs/conf/1999-ESREL-SoftRel/html/chal.html HTTP/1.1" 404 322 ($host, $logname, $user, $day, $month, $year, $hour, $minute, $verb, $url, $status, $bytes) = / ([-\w.]+)\s+ (?# Host) ([-\w]+)\s+ (?# Logname) ([-\w]+)\s+ (?# User) \[(\d+)\/ (?# Date) (\w+)\/ (?# Month) (\d+)\: (?# Year) (\d+)\: (?# Hour) (\d+) (?# Minute) [^]]+?\]\s+ (?# Rest of time) \"([-\w]+)\s* (?# Request verb) ([^\s]*) (?# Request URL) [^"]*?\"\s+ (?# Request protocol etc.) (\d+)\s+ (?# Status) ([-\d]+) (?# Bytes) /x)) { print STDERR "$ARGV($.): Unable to process: $_\n"; next; } if ($host !~ m/\.\d+$/) { ($topdomain) = ($host =~ m/.*\.(.*)/); $topdomaincount{$topdomain}++; ($domain) = ($host =~ m/[^.]\.(.*)/); $domaincount{$domain}++; } ($area) = ($url =~ m/^\/?([^\/]+)/); $area = '/' if ($area eq ''); $month = $monthcard{$month}; $date = $year . '-' . $month . '-' . $day; $accesscount++; $hostcount{$host}++; $urlcount{$url}++; $datecount{$date}++; $ltime = timelocal(0, 0, 0, $day, $month - 1, $year - 1900); $daynum = (localtime $ltime)[6]; $dowcount{$daynum}++; $hourcount{$hour}++; $areacount{$area}++; $bytecount += $bytes; $hostbytecount{$host} += $bytes; } print " WWW server statistics ===================== Summary ------- "; printf("Number of accesses: %d\n", $accesscount); printf("Number of Gbytes transferred: %d\n", $bytecount / 1024 / 1024 / 1024); printf("Number of hosts: %d\n", ($hostcount = grep 1, values %hostcount)); printf("Number of domains: %d\n", ($domaincount = grep 1, values %domaincount)); printf("Number of top level domains: %d\n", ($topdomaincount = grep 1, values %topdomaincount)); printf("Number of different pages: %d\n", ($urlcount = grep 1, keys %urlcount)); printf("Accesses per day: %d\n", $accesscount / ($datecount = grep 1, values %datecount)); printf("Mbytes per day: %d\n", $bytecount / $datecount / 1024 / 1024); printf("Mbytes log file size: %d\n", $logbytecount / 1024 / 1024); print ' Top 20 Requests --------------- '; $count = 0; foreach (sort {$urlcount{$b} <=> $urlcount{$a}} keys %urlcount) { last if ($count++ == 20); printf("%10d %s\n", $urlcount{$_}, $_); } print ' Top 20 Area Requests --------------- '; $count = 0; foreach (sort {$areacount{$b} <=> $areacount{$a}} keys %areacount) { last if ($count++ == 20); printf("%10d %s\n", $areacount{$_}, $_); } print ' Top 10 Hosts ------------ '; $count = 0; foreach (sort {$hostcount{$b} <=> $hostcount{$a}} keys %hostcount) { last if ($count++ == 10); printf("%10d %s\n", $hostcount{$_}, $_); } print ' Top 10 Hosts by Transfer ------------------------ '; $count = 0; foreach (sort {$hostbytecount{$b} <=> $hostbytecount{$a}} keys %hostbytecount) { last if ($count++ == 10); printf("%10d %s\n", $hostbytecount{$_}, $_); } print ' Top 10 Domains -------------- '; $count = 0; foreach (sort {$domaincount{$b} <=> $domaincount{$a}} keys %domaincount) { last if ($count++ == 10); printf("%10d %s\n", $domaincount{$_}, $_); } print ' Top 20 Level Domain Accesses ------------------------- '; $count = 0; foreach (sort {$topdomaincount{$b} <=> $topdomaincount{$a}} keys %topdomaincount) { printf("%10d %s\n", $topdomaincount{$_}, $_); last if ($count++ == 20); } print ' Accesses by Day of Week ----------------------- '; foreach (sort {$dowcount{$b} <=> $dowcount{$a}} keys %dowcount) { printf("%10d %s\n", $dowcount{$_}, $downame{$_}); } print ' Accesses by Local Hour ---------------------- '; foreach (sort keys %hourcount) { printf("%10d %s\n", $hourcount{$_}, $_); } print ' Accesses by Date ---------------- '; foreach (sort keys %datecount) { printf("%10d %s\n", $datecount{$_}, $_); } ================================================ FILE: eval/webeval.sh ================================================ #!/bin/sh # # Run performance evaluations # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # . 'eval-lib.sh' # See http://ita.ee.lbl.gov/html/contrib/ClarkNet-HTTP.html download ftp://ita.ee.lbl.gov/traces/clarknet_access_log_Aug28.gz download ftp://ita.ee.lbl.gov/traces/clarknet_access_log_Sep4.gz # Compile Java code if needed if [ ! -r WebStats.class -o WebStats.java -nt WebStats.java ] then if ! javac WebStats.java then echo "Unable to compile WebStats.java" 1>&2 exit 1 fi fi # Set machine type if ! [ -d /opt/aws ] || ! IID=`curl -s http://169.254.169.254/2011-01-01/meta-data/instance-id` || ! TYPE=`ec2-describe-instance-attribute $IID --instance-type | awk '{print $3}'` then TYPE=`hostname` fi mkdir -p time out err # Log grow factor GROW=1 while : do for PROG in dgsh perl java do DESC=web-$TYPE-$PROG-$GROW if [ -r err/$DESC ] then continue fi for i in clarknet*.gz do gzip -dc $i done | /usr/bin/perl log-grow.pl $GROW | case $PROG in perl) timerun $DESC /usr/bin/perl web-log-report.pl ;; dgsh) timerun $DESC ../dgsh -p .. ../example/web-log-report.sh ;; java) timerun $DESC java WebStats ;; esac done GROW=`expr $GROW \* 2` if [ $GROW -gt 128 ] then break fi done ================================================ FILE: example/NMRPipe.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS Nuclear magnetic resonance processing # DESCRIPTION # Nuclear magnetic resonance in-phase/anti-phase channel conversion and # processing in heteronuclear single quantum coherence spectroscopy. # Demonstrate processing of NMR data using the NMRPipe family of programs. # # See also F. Delaglio, S. Grzesiek, G. W. Vuister, G. Zhu, J. Pfeifer # and A. Bax: NMRPipe: a multidimensional spectral processing system based # on UNIX pipes. J. Biomol. NMR. 6, 277-293 (1995). # http://spin.niddk.nih.gov/NMRPipe/ # # The conversion is configured for the following file: # http://www.bmrb.wisc.edu/ftp/pub/bmrb/timedomain/bmr6443/timedomain_data/c13-hsqc/june11-se-6426-CA.fid/fid var2pipe -in $1 \ -xN 1280 -yN 256 \ -xT 640 -yT 128 \ -xMODE Complex -yMODE Complex \ -xSW 8000 -ySW 6000 \ -xOBS 599.4489584 -yOBS 60.7485301 \ -xCAR 4.73 -yCAR 118.000 \ -xLAB 1H -yLAB 15N \ -ndim 2 -aq2D States \ -verb | tee | {{ # IP/AP channel conversion # See http://tech.groups.yahoo.com/group/nmrpipe/message/389 nmrPipe | nmrPipe -fn SOL | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -c 0.5 | nmrPipe -fn ZF -auto | nmrPipe -fn FT | nmrPipe -fn PS -p0 177 -p1 0.0 -di | nmrPipe -fn EXT -left -sw -verb | nmrPipe -fn TP | nmrPipe -fn COADD -cList 1 0 -time | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 1 -c 0.5 | nmrPipe -fn ZF -auto | nmrPipe -fn FT | nmrPipe -fn PS -p0 0 -p1 0 -di | nmrPipe -fn TP | nmrPipe -fn POLY -auto -verb >A nmrPipe | nmrPipe -fn SOL | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -c 0.5 | nmrPipe -fn ZF -auto | nmrPipe -fn FT | nmrPipe -fn PS -p0 177 -p1 0.0 -di | nmrPipe -fn EXT -left -sw -verb | nmrPipe -fn TP | nmrPipe -fn COADD -cList 0 1 -time | nmrPipe -fn SP -off 0.5 -end 0.98 -pow 1 -c 0.5 | nmrPipe -fn ZF -auto | nmrPipe -fn FT | nmrPipe -fn PS -p0 -90 -p1 0 -di | nmrPipe -fn TP | nmrPipe -fn POLY -auto -verb >B }} # We use temporary files rather than streams, because # addNMR mmaps its input files. The diagram displayed in the # example shows the notional data flow. if [ -z "${DGSH_DRAW_EXIT}" ] then addNMR -in1 A -in2 B -out A+B.dgsh.ft2 -c1 1.0 -c2 1.25 -add addNMR -in1 A -in2 B -out A-B.dgsh.ft2 -c1 1.0 -c2 1.25 -sub fi ================================================ FILE: example/author-compare.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS Venue author compare # DESCRIPTION # Given the specification of two publication venues, read a compressed # DBLP computer science bibliography from the standard input (e.g. piped # from curl -s http://dblp.uni-trier.de/xml/dblp.xml.gz or from a locally # cached copy) and output the number of papers published in each of the # two venues as well as the number of authors who have published only in # the first venue, the number who have published only in the second one, # and authors who have published in both. The venues are specified through # the script's first two command-line arguments as a DBLP key prefix, e.g. # journals/acta/, conf/icse/, journals/software/, conf/iwpc/, or conf/msr/. # Demonstrates the use of dgsh-wrap -e to have sed(1) create two output # streams and the use of tee to copy a pair of streams into four ones. # # Copyright 2017 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Extract and sort author names sorted_authors() { sed -n 's/\([^<]*\)<\/author>/\1/p' | sort } # Escape a string to make it a valid sed(1) pattern escape() { echo "$1" | sed 's/\([/\\]\)/\\\1/g' } export -f sorted_authors if [ ! "$2" -a ! "$DGSH_DOT_DRAW"] ; then echo "Usage: $0 key1 key2" 1>&2 echo "Example: $0 conf/icse/ journals/software/" 1>&2 exit 1 fi gzip -dc | # Output the two venue authors as two output streams dgsh-wrap -e sed -n " /^<.*key=\"$(escape $1)/,// w >| /^<.*key=\"$(escape $2)/,/<title>/ w >|" | # 2 streams in 4 streams out: venue1, venue2, venue1, venue2 tee | {{ {{ echo -n "$1 papers: " grep -c '^<.* mdate=.* key=' echo -n "$2 papers: " grep -c '^<.* mdate=.* key=' }} {{ call sorted_authors call sorted_authors }} | comm | {{ echo -n "Authors only in $1: " wc -l echo -n "Authors only in $2: " wc -l echo -n 'Authors common in both venues: ' wc -l }} }} | cat ================================================ FILE: example/code-metrics.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS C code metrics # DESCRIPTION # Process a directory containing C source code, and produce a summary # of various metrics. # Demonstrates nesting, commands without input. # # Copyright 2012-2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # {{ # C and header code find "$@" \( -name \*.c -or -name \*.h \) -type f -print0 | tee | {{ # Average file name length # Convert to newline separation for counting echo -n 'FNAMELEN: ' tr \\0 \\n | # Remove path sed 's|^.*/||' | # Maintain average awk '{s += length($1); n++} END { if (n>0) print s / n; else print 0; }' xargs -0 /bin/cat | tee | {{ # Remove strings and comments sed 's/#/@/g;s/\\[\\"'\'']/@/g;s/"[^"]*"/""/g;'"s/'[^']*'/''/g" | cpp -P | tee | {{ # Structure definitions echo -n 'NSTRUCT: ' egrep -c 'struct[ ]*{|struct[ ]*[a-zA-Z_][a-zA-Z0-9_]*[ ]*{' #}} (match preceding openings) # Type definitions echo -n 'NTYPEDEF: ' grep -cw typedef # Use of void echo -n 'NVOID: ' grep -cw void # Use of gets echo -n 'NGETS: ' grep -cw gets # Average identifier length echo -n 'IDLEN: ' tr -cs 'A-Za-z0-9_' '\n' | sort -u | awk '/^[A-Za-z]/ { len += length($1); n++ } END { if (n>0) print len / n; else print 0; }' }} # Lines and characters echo -n 'CHLINESCHAR: ' wc -lc | awk '{OFS=":"; print $1, $2}' # Non-comment characters (rounded thousands) # -traditional avoids expansion of tabs # We round it to avoid failing due to minor # differences between preprocessors in regression # testing echo -n 'NCCHAR: ' sed 's/#/@/g' | cpp -traditional -P | wc -c | awk '{OFMT = "%.0f"; print $1/1000}' # Number of comments echo -n 'NCOMMENT: ' egrep -c '/\*|//' # Occurences of the word Copyright echo -n 'NCOPYRIGHT: ' grep -ci copyright }} }} # C files find "$@" -name \*.c -type f -print0 | tee | {{ # Convert to newline separation for counting tr \\0 \\n | tee | {{ # Number of C files echo -n 'NCFILE: ' wc -l # Number of directories containing C files echo -n 'NCDIR: ' sed 's,/[^/]*$,,;s,^.*/,,' | sort -u | wc -l }} # C code xargs -0 /bin/cat | tee | {{ # Lines and characters echo -n 'CLINESCHAR: ' wc -lc | awk '{OFS=":"; print $1, $2}' # C code without comments and strings sed 's/#/@/g;s/\\[\\"'\'']/@/g;s/"[^"]*"/""/g;'"s/'[^']*'/''/g" | cpp -P | tee | {{ # Number of functions echo -n 'NFUNCTION: ' grep -c '^{' # Number of gotos echo -n 'NGOTO: ' grep -cw goto # Occurrences of the register keyword echo -n 'NREGISTER: ' grep -cw register # Number of macro definitions echo -n 'NMACRO: ' grep -c '@[ ]*define[ ][ ]*[a-zA-Z_][a-zA-Z0-9_]*(' # Number of include directives echo -n 'NINCLUDE: ' grep -c '@[ ]*include' # Number of constants echo -n 'NCONST: ' grep -ohw '[0-9][x0-9][0-9a-f]*' | wc -l }} }} }} # Header files echo -n 'NHFILE: ' find "$@" -name \*.h -type f | wc -l }} | # Gather and print the results cat ================================================ FILE: example/commit-stats.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS Git commit statistics # DESCRIPTION # Process the Git history, and list the authors and days of the week # ordered by the number of their commits. # Demonstrates streams and piping through a function. # # Copyright 2012-2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # forder() { sort | uniq -c | sort -rn } git log --format="%an:%ad" --date=default "$@" | tee | {{ echo "Authors ordered by number of commits" # Order by frequency awk -F: '{print $1}' | forder echo "Days ordered by number of commits" # Order by frequency awk -F: '{print substr($2, 1, 3)}' | forder }} | cat ================================================ FILE: example/committer-plot.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS Plot Git committer activity over time # DESCRIPTION # Process the Git history, and create two PNG diagrams depicting # committer activity over time. The most active committers appear # at the center vertical of the diagram. # Demonstrates image processing, mixining of synchronous and # asynchronous processing in a scatter block, and the use of an # dgsh-compliant join command. # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Commit history in the form of ascending Unix timestamps, emails git log --pretty=tformat:'%at %ae' | # Filter records according to timestamp: keep (100000, now) seconds awk 'NF == 2& $1 > 100000& $1 < '`date +%s` | sort -n | tee | {{ {{ # Calculate number of committers awk '{print $2}' | sort -u | wc -l | tee | {{ dgsh-writeval -s committers1 dgsh-writeval -s committers2 dgsh-writeval -s committers3 }} # Calculate last commit timestamp in seconds tail -1 | awk '{print $1}' # Calculate first commit timestamp in seconds head -1 | awk '{print $1}' }} | # Gather last and first commit timestamp cat | # Make one space-delimeted record tr '\n' ' ' | # Compute the difference in days awk '{print int(($1 - $2) / 60 / 60 / 24)}' | # Store number of days dgsh-writeval -s days sort -k2 # <timestamp, email> # Place committers left/right of the median # according to the number of their commits awk '{print $2}' | sort | uniq -c | sort -n | awk ' BEGIN { "dgsh-readval -l -x -q -s committers1" | getline NCOMMITTERS l = 0; r = NCOMMITTERS;} {print NR % 2 ? l++ : --r, $2}' | sort -k2 # <left/right, email> }} | # Join committer positions with commit time stamps # based on committer email join -j 2 | # <email, timestamp, left/right> # Order by timestamp sort -k 2n | tee | {{ # Create portable bitmap echo 'P1' {{ dgsh-readval -l -q -s committers2 dgsh-readval -l -q -s days }} | cat | tr '\n' ' ' | awk '{print $1, $2}' perl -na -e ' BEGIN { open(my $ncf, "-|", "dgsh-readval -l -x -q -s committers3"); $ncommitters = <$ncf>; @empty[$ncommitters - 1] = 0; @committers = @empty; } sub out { print join("", map($_ ? "1" : "0", @committers)), "\n"; } $day = int($F[1] / 60 / 60 / 24); $pday = $day if (!defined($pday)); while ($day != $pday) { out(); @committers = @empty; $pday++; } $committers[$F[2]] = 1; END { out(); } ' }} | cat | # Enlarge points into discs through morphological convolution pgmmorphconv -erode <( cat <<EOF P1 7 7 1 1 1 0 1 1 1 1 1 0 0 0 1 1 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 1 1 0 0 0 1 1 1 1 1 0 1 1 1 EOF ) | tee | {{ # Full-scale image pnmtopng >large.png # A smaller image pamscale -width 640 | pnmtopng >small.png }} ================================================ FILE: example/compress-compare.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS Compression benchmark # DESCRIPTION # Report file type, length, and compression performance for # data received from the standard input. The data never touches the # disk. # Demonstrates the use of an output multipipe to source many commands # from one followed by an input multipipe to sink to one command # the output of many and the use of dgsh-tee that is used both to # propagate the same input to many commands and collect output from # many commands orderly in a way that is transparent to users. # # # Copyright 2012-2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http:/www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # tee | {{ printf 'File type:\t' file - printf 'Original size:\t' wc -c printf 'xz:\t\t' xz -c | wc -c printf 'bzip2:\t\t' bzip2 -c | wc -c printf 'gzip:\t\t' gzip -c | wc -c }} | cat ================================================ FILE: example/dir.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS Directory listing # DESCRIPTION # Windows-like DIR command for the current directory. # Nothing that couldn't be done with <code>ls -l | awk</code>. # Demonstrates use of wrapped commands with no input (df, echo). # # Copyright 2012-2017 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ls -n | tee | {{ # Reorder fields in DIR-like way awk '!/^total/ {print $6, $7, $8, $1, sprintf("%8d", $5), $9}' # Count number of files wc -l | tr -d \\n # Print label for number of files echo -n ' File(s) ' # Tally number of bytes awk '{s += $5} END {printf("%d bytes\n", s)}' # Count number of directories grep -c '^d' | tr -d \\n # Print label for number of dirs and calculate free bytes df -h . | awk '!/Use%/{print " Dir(s) " $4 " bytes free"}' }} | cat ================================================ FILE: example/duplicate-files.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS Find duplicate files # DESCRIPTION # List the names of duplicate files in the specified directory. # Demonstrates the combination of streams with a relational join. # # Copyright 2012-2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Create list of files find "$@" -type f | # Produce lines of the form # MD5(filename)= 811bfd4b5974f39e986ddc037e1899e7 xargs openssl md5 | # Convert each line into a "filename md5sum" pair sed 's/^MD5(//;s/)= / /' | # Sort by MD5 sum sort -k2 | tee | {{ # Print an MD5 sum for each file that appears more than once awk '{print $2}' | uniq -d # Promote the stream to gather it cat }} | # Join the repeated MD5 sums with the corresponding file names # Join expects two inputs, second will come from scatter # XXX make streaming input identifiers transparent to users join -2 2 | # Output same files on a single line awk ' BEGIN {ORS=""} $1 != prev && prev {print "\n"} END {if (prev) print "\n"} {if (prev) print " "; prev = $1; print $2}' ================================================ FILE: example/fft-block8.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS FFT calculation # DESCRIPTION # Calculate the iterative FFT for n = 8 in parallel. # Demonstrates combined use of permute and multipipe blocks. # # Copyright 2016 Marios Fragkoulis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # dgsh-fft-input $1 | perm 1,5,3,7,2,6,4,8 | {{ {{ dgsh-w 1 0 dgsh-w 1 0 }} | perm 1,3,2,4 | {{ dgsh-w 2 0 dgsh-w 2 1 }} {{ dgsh-w 1 0 dgsh-w 1 0 }} | perm 1,3,2,4 | {{ dgsh-w 2 0 dgsh-w 2 1 }} }} | perm 1,5,3,7,2,6,4,8 | {{ dgsh-w 3 0 dgsh-w 3 1 dgsh-w 3 2 dgsh-w 3 3 }} | perm 1,5,2,6,3,7,4,8 | cat ================================================ FILE: example/ft2d.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS Waves: 2D Fourier transforms # DESCRIPTION # Create two graphs: # 1) a broadened pulse and the real part of its 2D Fourier transform, and # 2) a simulated air wave and the amplitude of its 2D Fourier transform. # Demonstrates using the tools of the Madagascar shared research environment # for computational data analysis in geophysics and related fields. # Also demonstrates the use of two scatter blocks in the same script, # and the used of named streams. # # Adapted from: http://www.reproducibility.org/RSF/book/bei/ft1/ft2d.html # Description: http://www.reproducibility.org/RSF/book/bei/ft1/paper_html/node14.html # Madagascar project: http://www.reproducibility.org # mkdir -p Fig # The SConstruct SideBySideIso "Result" method side_by_side_iso() { vppen size=r vpstyle=n gridnum=2,1 /dev/stdin $* } export -f side_by_side_iso # A broadened pulse and the real part of its 2D Fourier transform sfspike n1=64 n2=64 d1=1 d2=1 nsp=2 k1=16,17 k2=5,5 mag=16,16 \ label1='time' label2='space' unit1= unit2= | sfsmooth rect2=2 | sfsmooth rect2=2 | tee | {{ sfgrey pclip=100 wanttitle=n sffft1 | sffft3 axis=2 pad=1 | sfreal | tee | {{ sfwindow f1=1 | sfreverse which=3 cat }} | sfcat axis=1 "<|" | sfgrey pclip=100 wanttitle=n label1="1/time" label2="1/space" }} | call_with_stdin side_by_side_iso '<|' yscale=1.25 >Fig/ft2dofpulse.vpl # A simulated air wave and the amplitude of its 2D Fourier transform sfspike n1=64 d1=1 o1=32 nsp=4 k1=1,2,3,4 mag=1,3,3,1 \ label1='time' unit1= | sfspray n=32 d=1 o=0 | sfput label2=space | sflmostretch delay=0 v0=-1 | tee | {{ sfwindow f2=1 | sfreverse which=2 cat }} | sfcat axis=2 "<|" | tee | {{ sfgrey pclip=100 wanttitle=n sffft1 | sffft3 sign=1 | tee | {{ sfreal sfimag }} | dgsh-wrap -e sfmath nostdin=y re="<|" im="<|" \ output="sqrt(re*re+im*im)" | tee | {{ sfwindow f1=1 | sfreverse which=3 cat }} | sfcat axis=1 "<|" | sfgrey pclip=100 wanttitle=n label1="1/time" label2="1/space" }} | call_with_stdin side_by_side_iso '<|' yscale=1.25 >Fig/airwave.vpl wait ================================================ FILE: example/map-hierarchy.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS Hierarchy map # DESCRIPTION # Given two directory hierarchies A and B passed as input arguments # (where these represent a project at different parts of its lifetime) # copy the files of hierarchy A to a new directory, passed as a third # argument, corresponding to the structure of directories in B. # Demonstrates the use of <em>join</em> to process results from two # inputs and the use of <em>gather</em> to order asynchronously # produced results. # # Copyright 2012-2014 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # if [ -z "${DGSH_DRAW_EXIT}" -a \( ! -d "$1" -o ! -d "$2" -o -z "$3" \) ] then echo "Usage: $0 dir-1 dir-2 new-dir-name" 1>&2 exit 1 fi NEWDIR="$3" export LC_ALL=C line_signatures() { find $1 -type f -name '*.[chly]' -print | # Split path name into directory and file sed 's|\(.*\)/\([^/]*\)|\1 \2|' | while read dir file do # Print "directory filename content" of lines with # at least one alphabetic character # The fields are separated by  and  sed -n "/[a-z]/s|^|$dir$file|p" "$dir/$file" done | # Error: multi-character tab '\001\001' sort -T `pwd` -t -k 2 } export -f line_signatures {{ # Generate the signatures for the two hierarchies call 'line_signatures "$1"' -- "$1" call 'line_signatures "$1"' -- "$2" }} | # Join signatures on file name and content join -t -1 2 -2 2 | # Print filename dir1 dir2 sed 's///g' | awk -F 'BEGIN{OFS=" "}{print $1, $3, $4}' | # Unique occurrences sort -u | tee | {{ # Commands to copy awk '{print "mkdir -p '$NEWDIR'/" $3 ""}' | sort -u awk '{print "cp " $2 "/" $1 " '$NEWDIR'/" $3 "/" $1 ""}' }} | # Order: first make directories, then copy files # TODO: dgsh-tee does not pass along first incoming stream cat | sh ================================================ FILE: example/parallel-word-count.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS Parallel word count # DESCRIPTION # Count number of times each word appears in the specified input file(s) # Demonstrates parallel execution mirroring the Hadoop WordCount example # via the dgsh-parallel command. # In contrast to GNU parallel, the block generated by dgsh-parallel # has N input and output streams, which can be combined by any # dgsh-compatible tool, such as dgsh-merge-sum or sort -m. # # Copyright 2014-2016 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Number of processes N=8 # Collation order for sorting export LC_ALL=C # Scatter input dgsh-tee -s | # Emulate Java's default StringTokenizer, sort, count dgsh-parallel -n $N "tr -s ' \t\n\r\f' '\n' | sort -S 512M | uniq -c" | # Merge sorted counts by providing N input channels dgsh-merge-sum $(for i in $(seq $N) ; do printf '<| ' ; done) ================================================ FILE: example/reorder-columns.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS Reorder columns # DESCRIPTION # Reorder columns in a CSV document. # Demonstrates the combined use of tee, cut, and paste. # # Copyright 2016 Marios Fragkoulis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # tee | {{ cut -d , -f 5-6 - cut -d , -f 2-4 - }} | paste -d , ================================================ FILE: example/spell-highlight.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS Highlight misspelled words # DESCRIPTION # Highlight the words that are misspelled in the command's first # argument. # Demonstrates stream processing with multipipes and # the avoidance of pass-through constructs to avoid deadlocks. # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # export LC_ALL=C tee | {{ # Find errors {{ # Obtain list of words in text tr -cs A-Za-z \\n | tr A-Z a-z | sort -u # Ensure dictionary is compatibly sorted sort /usr/share/dict/words }} | # List errors as a set difference comm -23 # Pass through text cat }} | grep --fixed-strings --file=- --ignore-case --color --word-regex --context=2 ================================================ FILE: example/static-functions.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS C/C++ symbols that should be static # DESCRIPTION # Given as an argument a directory containing object files, show which # symbols are declared with global visibility, but should have been # declared with file-local (static) visibility instead. # Demonstrates the use of dgsh-capable comm (1) to combine data from # two sources. # # Copyright 2014 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Find object files find "$1" -name \*.o | # Print defined symbols xargs nm | tee | {{ # List all defined (exported) symbols awk 'NF == 3 && $2 ~ /[A-Z]/ {print $3}' | sort # List all undefined (imported) symbols awk '$1 == "U" {print $2}' | sort }} | # Print exports that are not imported comm -23 ================================================ FILE: example/text-properties.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS Text properties # DESCRIPTION # Read text from the standard input and create files # containing word, character, digram, and trigram frequencies. # # Demonstrates the use of scatter blocks without output and the use # of stores within the scatter block. # # Example: # curl ftp://sunsite.informatik.rwth-aachen.de/pub/mirror/ibiblio/gutenberg/1/3/139/139.txt | text-properties # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Consistent sorting across machines export LC_ALL=C # Convert input into a ranked frequency list ranked_frequency() { awk '{count[$1]++} END {for (i in count) print count[i], i}' | # We want the standard sort here sort -rn } # Convert standard input to a ranked frequency list of specified n-grams ngram() { local N=$1 perl -ne 'for ($i = 0; $i < length($_) - '$N'; $i++) { print substr($_, $i, '$N'), "\n"; }' | ranked_frequency } export -f ranked_frequency export -f ngram tee | {{ # Split input one word per line tr -cs a-zA-Z \\n | tee | {{ # Digram frequency call 'ngram 2 >digram.txt' # Trigram frequency call 'ngram 3 >trigram.txt' # Word frequency call 'ranked_frequency >words.txt' }} # Store number of characters to use in awk below wc -c | dgsh-writeval -s nchars # Character frequency sed 's/./&\ /g' | # Print absolute call 'ranked_frequency' | awk 'BEGIN { "dgsh-readval -l -x -q -s nchars" | getline NCHARS OFMT = "%.2g%%"} {print $1, $2, $1 / NCHARS * 100}' > character.txt }} ================================================ FILE: example/uniform-5x5.sh ================================================ #!/usr/bin/env dgsh row() { dgsh-parallel -n 5 'echo C{}' | paste } matrix() { dgsh-parallel -n 5 row } export -f row matrix | cat ================================================ FILE: example/web-log-report.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS Web log reporting # DESCRIPTION # Creates a report for a fixed-size web log file read from the standard input. # Demonstrates the combined use of multipipe blocks, writeval and readval # to store and retrieve values, and functions in the scatter block. # Used to measure throughput increase achieved through parallelism. # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Output the top X elements of the input by the number of their occurrences # X is the first argument toplist() { uniq -c | sort -rn | head -$1 echo } # Output the argument as a section header header() { echo echo "$1" echo "$1" | sed 's/./-/g' } # Consistent sorting export LC_ALL=C export -f toplist export -f header if [ -z "${DGSH_DRAW_EXIT}" ] then cat <<EOF WWW server statistics ===================== Summary ------- EOF fi tee | {{ # Number of accesses echo -n 'Number of accesses: ' dgsh-readval -l -s nAccess # Number of transferred bytes awk '{s += $NF} END {print s}' | tee | {{ echo -n 'Number of Gbytes transferred: ' awk '{print $1 / 1024 / 1024 / 1024}' dgsh-writeval -s nXBytes }} echo -n 'Number of hosts: ' dgsh-readval -l -q -s nHosts echo -n 'Number of domains: ' dgsh-readval -l -q -s nDomains echo -n 'Number of top level domains: ' dgsh-readval -l -q -s nTLDs echo -n 'Number of different pages: ' dgsh-readval -l -q -s nUniqPages echo -n 'Accesses per day: ' dgsh-readval -l -q -s nDayAccess echo -n 'MBytes per day: ' dgsh-readval -l -q -s nDayMB # Number of log file bytes echo -n 'MBytes log file size: ' wc -c | awk '{print $1 / 1024 / 1024}' # Host names awk '{print $1}' | tee | {{ # Number of accesses wc -l | dgsh-writeval -s nAccess # Sorted hosts sort | tee | {{ # Unique hosts uniq | tee | {{ # Number of hosts wc -l | dgsh-writeval -s nHosts # Number of TLDs awk -F. '$NF !~ /[0-9]/ {print $NF}' | sort -u | wc -l | dgsh-writeval -s nTLDs }} # Top 10 hosts {{ call 'header "Top 10 Hosts"' call 'toplist 10' }} }} # Top 20 TLDs {{ call 'header "Top 20 Level Domain Accesses"' awk -F. '$NF !~ /^[0-9]/ {print $NF}' | sort | call 'toplist 20' }} # Domains awk -F. 'BEGIN {OFS = "."} $NF !~ /^[0-9]/ {$1 = ""; print}' | sort | tee | {{ # Number of domains uniq | wc -l | dgsh-writeval -s nDomains # Top 10 domains {{ call 'header "Top 10 Domains"' call 'toplist 10' }} }} }} # Hosts by volume {{ call 'header "Top 10 Hosts by Transfer"' awk ' {bytes[$1] += $NF} END {for (h in bytes) print bytes[h], h}' | sort -rn | head -10 }} # Sorted page name requests awk '{print $7}' | sort | tee | {{ # Top 20 area requests (input is already sorted) {{ call 'header "Top 20 Area Requests"' awk -F/ '{print $2}' | call 'toplist 20' }} # Number of different pages uniq | wc -l | dgsh-writeval -s nUniqPages # Top 20 requests {{ call 'header "Top 20 Requests"' call 'toplist 20' }} }} # Access time: dd/mmm/yyyy:hh:mm:ss awk '{print substr($4, 2)}' | tee | {{ # Just dates awk -F: '{print $1}' | tee | {{ # Number of days uniq | wc -l | tee | {{ awk ' BEGIN { "dgsh-readval -l -x -s nAccess" | getline NACCESS;} {print NACCESS / $1}' | dgsh-writeval -s nDayAccess awk ' BEGIN { "dgsh-readval -l -x -q -s nXBytes" | getline NXBYTES;} {print NXBYTES / $1 / 1024 / 1024}' | dgsh-writeval -s nDayMB }} {{ call 'header "Accesses by Date"' uniq -c }} # Accesses by day of week {{ call 'header "Accesses by Day of Week"' sed 's|/|-|g' | call '(date -f - +%a 2>/dev/null || gdate -f - +%a)' | sort | uniq -c | sort -rn }} }} # Hour {{ call 'header "Accesses by Local Hour"' awk -F: '{print $2}' | sort | uniq -c }} }} dgsh-readval -q -s nAccess }} | cat ================================================ FILE: example/word-properties.sh ================================================ #!/usr/bin/env dgsh # # SYNOPSIS Word properties # DESCRIPTION # Read text from the standard input and list words # containing a two-letter palindrome, words containing # four consonants, and words longer than 12 characters. # # Demonstrates the use of dgsh-compatible paste as a gather function # # Example: # curl ftp://sunsite.informatik.rwth-aachen.de/pub/mirror/ibiblio/gutenberg/1/3/139/139.txt | word-properties # # Copyright 2013 Diomidis Spinellis # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Consistent sorting across machines export LC_ALL=C # Stream input from file cat $1 | # Split input one word per line tr -cs a-zA-Z \\n | # Create list of unique words sort -u | tee | {{ # Pass through the original words cat # List two-letter palindromes sed 's/.*\(.\)\(.\)\2\1.*/p: \1\2-\2\1/;t g' # List four consecutive consonants sed -E 's/.*([^aeiouyAEIOUY]{4}).*/c: \1/;t g' # List length of words longer than 12 characters awk '{if (length($1) > 12) print "l:", length($1); else print ""}' }} | # Paste the four streams side-by-side paste | # List only words satisfying one or more properties fgrep : ================================================ FILE: png/README ================================================ This is the destination for auto-generated PNG files ================================================ FILE: simple-shell/comm_paste.dgsh ================================================ 1 comm ../test-data/f1s ../test-data/f2s 2 paste % socketpipe 1 2 ================================================ FILE: simple-shell/comm_paste.success ================================================ 1 3 2 4 3 6 4 5 ================================================ FILE: simple-shell/comm_paste_join_diff.dgsh ================================================ 1 comm ../test-data/f4ss ../test-data/f5ss 2 dgsh-conc -o 3 3 paste ../test-data/p1 4 join ../test-data/j2 5 diff ../test-data/d3 % socketpipe 1 2 socketpipe 2 3 socketpipe 2 4 socketpipe 2 5 ================================================ FILE: simple-shell/comm_paste_join_diff.success ================================================ 3 10 2 2 2 2 10 3 12 match 4 match ================================================ FILE: simple-shell/comm_sort.dgsh ================================================ 1 comm ../test-data/f1s ../test-data/f2s 2 sort 3 wc -l 4 tr -d " " % socketpipe 1 2 socketpipe 2 3 socketpipe 3 4 ================================================ FILE: simple-shell/comm_sort.success ================================================ 9 ================================================ FILE: simple-shell/dir-plain.dgsh ================================================ 1 dgsh-wrap ls -n 2 dgsh-tee 3 dgsh-conc -o 2 4 dgsh-wrap awk '!/^total/ {print $6, $7, $8, $1, sprintf("%8d", $5), $9}' 5 dgsh-wrap awk '{s += $5} END {printf("%d bytes", s)}' % socketpipe 1 2 socketpipe 2 3 socketpipe 3 4 socketpipe 3 5 ================================================ FILE: simple-shell/grep_comm.dgsh ================================================ 1 grep -l -L match ../test-data/ff ../test-data/F 2 comm % socketpipe 1 2 ================================================ FILE: simple-shell/grep_comm.success ================================================ ../test-data/ff ../test-data/F ================================================ FILE: simple-shell/grep_comm.success-bash ================================================ ../test-data/F ../test-data/ff ================================================ FILE: simple-shell/grep_diff.dgsh ================================================ 1 grep -v -w match ../test-data/F ../test-data/ff 2 diff % socketpipe 1 2 ================================================ FILE: simple-shell/grep_diff.success ================================================ 1c1 < F:not --- > ff:match ================================================ FILE: simple-shell/grep_diff.success-bash ================================================ 1c1 < ../test-data/F:not --- > ../test-data/ff:match ================================================ FILE: simple-shell/grep_diff_comm.dgsh ================================================ 1 grep -l -L -w -v match ../test-data/ff ../test-data/F 2 dgsh-conc -o 2 3 diff 4 comm % socketpipe 1 2 socketpipe 2 3 socketpipe 2 4 ================================================ FILE: simple-shell/grep_diff_comm.success1 ================================================ 1c1 < ff --- > F F:not ff:match ================================================ FILE: simple-shell/grep_diff_comm.success1-bash ================================================ ../test-data/F:not ../test-data/ff:match 1c1 < ../test-data/ff --- > ../test-data/F ================================================ FILE: simple-shell/grep_diff_comm.success2 ================================================ F:not ff:match 1c1 < ff --- > F ================================================ FILE: simple-shell/grep_diff_comm.success2-bash ================================================ 1c1 < ../test-data/ff --- > ../test-data/F ../test-data/F:not ../test-data/ff:match ================================================ FILE: simple-shell/grep_diff_comm.success3 ================================================ 1c1 < ff --- > F ff:match F:not ================================================ FILE: simple-shell/grep_diff_comm.success3-bash ================================================ 1c1 < ../test-data/ff --- > ../test-data/F ../test-data/ff:match ../test-data/F:not ================================================ FILE: simple-shell/grep_diff_comm.success4 ================================================ ff:match F:not 1c1 < ff --- > F ================================================ FILE: simple-shell/grep_diff_comm.success4-bash ================================================ ../test-data/ff:match ../test-data/F:not 1c1 < ../test-data/ff --- > ../test-data/F ================================================ FILE: simple-shell/join_sort.dgsh ================================================ 1 join ../test-data/f1s ../test-data/f2s 2 sort % socketpipe 1 2 ================================================ FILE: simple-shell/join_sort.success ================================================ 3 3 4 4 5 ================================================ FILE: simple-shell/join_sort_diff.dgsh ================================================ 1 join ../test-data/f1s ../test-data/f2s 2 sort 3 diff ../test-data/f3s % socketpipe 1 2 socketpipe 2 3 ================================================ FILE: simple-shell/join_sort_diff.success ================================================ 1,7c1,6 < < 1 3 < 2 3 < 3 4 < 4 4 < 5 5 < 6 --- > > 3 > 3 > 4 > 4 > 5 ================================================ FILE: simple-shell/ls_wc.dgsh ================================================ 1 ls 2 wc -l % socketpipe 1 2 ================================================ FILE: simple-shell/paste_diff.dgsh ================================================ 1 paste ../test-data/f1s ../test-data/f2s 2 diff ../test-data/f1s % socketpipe 1 2 ================================================ FILE: simple-shell/paste_diff.success ================================================ 1,7c1,7 < < 1 < 2 < 3 < 4 < 5 < 6 --- > > 1 3 > 2 3 > 3 4 > 4 4 > 5 5 > 6 ================================================ FILE: simple-shell/secho_paste.dgsh ================================================ 1 secho ../test-data/hello 2 paste ../test-data/world % socketpipe 1 2 ================================================ FILE: simple-shell/secho_paste.success ================================================ hello world ================================================ FILE: simple-shell/secho_secho_fgrep.dgsh ================================================ 1 dgsh-conc -o -n 2 2 secho match 3 secho not 4 dgsh-conc -i 2 5 grep -F -h match % socketpipe 1 2 socketpipe 1 3 socketpipe 2 4 socketpipe 3 4 socketpipe 4 5 ================================================ FILE: simple-shell/secho_secho_fgrep.success ================================================ match ================================================ FILE: simple-shell/simple-shell.py ================================================ from subprocess import Popen, PIPE, STDOUT import sys from socket import socketpair, AF_UNIX, SOCK_DGRAM from os import pipe, fork, close, execlp, dup, dup2, \ open as osopen, O_WRONLY, O_CREAT, environ from collections import OrderedDict import re prefix = '/usr/local/dgsh/bin' outFile = None def debug(s): if DEBUG: sys.stderr.write(s) class Process: processes = {} def __init__(self, command): self.command = command self.inputConnectors = [] self.outputConnectors = [] self.fileDescriptorsInUse = [] def selectInputFileDescriptor(self): fd = -1 if 0 not in self.fileDescriptorsInUse: fd = 0 elif 0 in self.fileDescriptorsInUse and \ 3 not in self.fileDescriptorsInUse: fd = 3 else: fd = self.fileDescriptorsInUse[-1] + 1 self.fileDescriptorsInUse.append(fd) debug("input file descriptor return: %d\n" % fd) return fd def selectOutputFileDescriptor(self): fd = -1 if 1 not in self.fileDescriptorsInUse: fd = 1 elif 1 in self.fileDescriptorsInUse and \ 3 not in self.fileDescriptorsInUse: fd = 3 else: fd = self.fileDescriptorsInUse[-1] + 1 self.fileDescriptorsInUse.append(fd) debug("output file descriptor return: %d\n" % fd) return fd def setupProcess(index, channel, connector): debug("index: %d, toolDict[index]: %s, channel: %s\n" % \ (index, toolDict[index], channel)) try: if channel == 'output': # index - 1: offset because index = 1, 2,... Process.processes[index].outputConnectors.append(connector) elif channel == 'input': Process.processes[index].inputConnectors.append(connector) except KeyError: Process.processes[index] = Process(toolDict[index]) setupProcess(index, channel, connector) def parse(command): if '\'' and '"' not in command: return command.split() else: splits = [] splitS = True splitD = True for pos, letter in enumerate(command): if letter == "'": if splitS: splitS = False else: splitS = True elif letter == '"': if splitD: splitD = False else: splitD = True elif letter is ' ' and splitS and splitD: splits.append(pos) print splits args = [] prev = 0 for pos in splits: args.append(command[prev:pos].replace("'", "")) prev = pos+1 args.append(command[prev:].replace("'", "")) return args # Debug configuration DEBUG = False for arg in sys.argv[2:]: if arg == "DEBUG": DEBUG = True elif arg.startswith("PREFIX="): match = re.match(r'PREFIX=(.+)$', arg) prefix = match.group(1) # Get output file name elif arg.startswith("OUT="): match = re.match(r'OUT=(.+)$', arg) outFile = match.group(1) # Read specification of processes and their interconnections try: dgshGraph = sys.argv[1] except IndexError: print "Input error: please specify an input file with tool and pipe specifications." exit(1) with open(dgshGraph, 'r') as f: lines = f.readlines() toolDefsEnd = 0 for index, line in enumerate(lines): if line == "%\n": toolDefsEnd = index break if toolDefsEnd == 0: print "Failed to find tool definition end line (\n%\n)" exit(1) debug("toolDefsEnd: %s\n" % toolDefsEnd) toolDict = {} for line in lines[:toolDefsEnd]: if line == '\n': continue match = re.match(r'^(\d+) (.+)\n$', line) if not match: print "Did not match command index and description: %s\n" % line exit(1) toolIndex = match.group(1) toolCommand = match.group(2) if not toolCommand.startswith('/') and not toolCommand.startswith('.'): toolCommand = prefix + '/' + toolCommand toolDict[int(toolIndex)] = toolCommand debug("index: %s, command: %s\n" % \ (toolIndex, toolCommand)) connectorDict = OrderedDict() for line in lines[toolDefsEnd+1:]: match = re.match(r'^(\w+) (\d+) (\d+)\n$', line) if not match: print "Did not match connector description and endpoints: %s\n" % line exit(1) connector = match.group(1) fromIndex = match.group(2) toIndex = match.group(3) connectorDict[int('%s%s' % (fromIndex, toIndex))] = connector debug("connector: %s, from index: %s, to_index: %s\n" % \ (connector, fromIndex, toIndex)) # Setup objects that represent objects along with their interconnections for processPair, connector in connectorDict.iteritems(): # convention: connector[0]: output, connector[1]: input connectorPair = [None, None] if connector == 'socketpipe': connectorPair[0], connectorPair[1] = socketpair(AF_UNIX, SOCK_DGRAM) elif connector == 'pipe': connectorPair[0], connectorPair[1] = pipe() else: print 'Do not understand connector %s' % connector exit(1) node_index_out = processPair / 10 node_index_inp = processPair % 10 debug("out: %d, inp: %d, connector[0]: %d, connector[1]: %d\n" % (node_index_out, node_index_inp, connectorPair[0].fileno(), connectorPair[1].fileno())) setupProcess(node_index_out, 'output', connectorPair) setupProcess(node_index_inp, 'input', connectorPair) # Open output file if outFile: outfile_fd = osopen(outFile, O_WRONLY | O_CREAT) # Activate interconnections and execute processes for index, process in Process.processes.iteritems(): debug('process %s, input channels: %d, output channels: %d\n' \ % (process.command, len(process.inputConnectors), \ len(process.outputConnectors))) pid = fork() if pid: debug("%s: inputConnectors: %d\n" % (process.command, len(process.inputConnectors))) if process.inputConnectors: environ["DGSH_IN"] = "1" else: environ["DGSH_IN"] = "0" for ic in process.inputConnectors: fd = process.selectInputFileDescriptor() try: close(fd) except OSError: debug("FAIL: close input fd %d for process %s. Discard and move on" \ % (fd, process.command)) fd = dup(ic[1].fileno()) debug("%s: dup %d, gives %d\n" % (process.command, ic[1].fileno(), fd)) ic[1].close() ic[0].close() debug("%s: outputConnectors: %d\n" % (process.command, len(process.outputConnectors))) if process.outputConnectors: environ["DGSH_OUT"] = "1" else: environ["DGSH_OUT"] = "0" for oc in process.outputConnectors: fd = process.selectOutputFileDescriptor() debug("%s: fd selected: %d, fd brought: %d\n" % (process.command, process.fileDescriptorsInUse[-1], oc[0].fileno())) try: close(fd) except OSError: print "FAIL: close output fd %d for process %s. Discard and move on" \ % (fd, process.command) fd = dup(oc[0].fileno()) debug("%s: dup %d, gives %d\n" % (process.command, oc[0].fileno(), fd)) oc[0].close() oc[1].close() if not process.outputConnectors and outFile: close(1) dup(outfile_fd) close(outfile_fd) args = parse(process.command) debug("\n\nARGS[0]: %s\n\n" % args[0]) execlp(args[0], args[0], *args[1:]) ================================================ FILE: simple-shell/sort_sort_comm.dgsh ================================================ 1 dgsh-conc -o -n 2 2 sort ../test-data/f4s 3 sort ../test-data/f5s 4 dgsh-conc -i 2 5 comm % socketpipe 1 2 socketpipe 1 3 socketpipe 2 4 socketpipe 3 4 socketpipe 4 5 ================================================ FILE: simple-shell/sort_sort_comm.success ================================================ 10 2 2 3 1 12 4 5 6 4 8 9 ================================================ FILE: simple-shell/sort_sort_comm_paste_join_diff.dgsh ================================================ 1 dgsh-conc -o -n 2 2 sort ../test-data/f4s 3 sort ../test-data/f5s 4 dgsh-conc -i 2 5 comm 6 dgsh-conc -o 3 7 paste ../test-data/p1 8 join ../test-data/j2 9 diff ../test-data/d3 % socketpipe 1 2 socketpipe 1 3 socketpipe 2 4 socketpipe 3 4 socketpipe 4 5 socketpipe 5 6 socketpipe 6 7 socketpipe 6 8 socketpipe 6 9 ================================================ FILE: simple-shell/sort_sort_comm_paste_join_diff.success ================================================ 3 10 2 2 2 2 10 3 12 match 4 match ================================================ FILE: simple-shell/tee-copy_diff_comm.dgsh ================================================ 1 dgsh-tee -i hello 2 dgsh-conc -o 2 3 diff ../test-data/world 4 comm ../test-data/hello % socketpipe 1 2 socketpipe 2 3 socketpipe 2 4 ================================================ FILE: simple-shell/tee-copy_diff_comm.success ================================================ 1c1 < world --- > hello hello ================================================ FILE: simple-shell/tee-scatter_diff_comm.dgsh ================================================ 1 tee -i ../test-data/hello -s 2 dgsh-conc -o 2 3 diff ../test-data/world 4 comm ../test-data/hello % socketpipe 1 2 socketpipe 2 3 socketpipe 2 4 ================================================ FILE: simple-shell/tee-scatter_diff_comm.success1 ================================================ 1d0 < world hello ================================================ FILE: simple-shell/tee-scatter_diff_comm.success2 ================================================ hello 1d0 < world ================================================ FILE: simple-shell/wrap-cat_comm_sort.dgsh ================================================ 1 dgsh-wrap /bin/cat ../test-data/f1s 2 comm ../test-data/f2s 3 sort 4 wc -l 5 tr -d " " % socketpipe 1 2 socketpipe 2 3 socketpipe 3 4 socketpipe 4 5 ================================================ FILE: simple-shell/wrap-cat_comm_sort.success ================================================ 9 ================================================ FILE: test-data/.gitignore ================================================ *.err *.errb fid res ================================================ FILE: test-data/F ================================================ not ================================================ FILE: test-data/access.log ================================================ ::1 - - [30/Sep/2016:15:04:15 +0300] "GET / HTTP/1.1" 200 5974 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0" ::1 - - [30/Sep/2016:15:04:16 +0300] "GET /js/bootstrap.min.js HTTP/1.1" 200 37192 "http://okanime.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0" 127.0.0.1 - - [30/Sep/2016:15:05:24 +0300] "GET / HTTP/1.1" 200 5974 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0" 127.0.0.1 - - [30/Sep/2016:15:05:25 +0300] "GET /js/bootstrap.min.js HTTP/1.1" 200 37192 "http://9281.net/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0" 127.0.0.1 - - [30/Sep/2016:15:07:16 +0300] "GET / HTTP/1.1" 200 5974 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0" 127.0.0.1 - - [30/Sep/2016:15:07:16 +0300] "GET /js/bootstrap.min.js HTTP/1.1" 200 37192 "http://adsupplyads.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0" ::1 - - [30/Sep/2016:15:37:10 +0300] "GET / HTTP/1.1" 200 5974 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0" ::1 - - [30/Sep/2016:15:37:11 +0300] "GET /js/bootstrap.min.js HTTP/1.1" 200 37192 "http://okanime.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0" 127.0.0.1 - - [30/Sep/2016:15:44:04 +0300] "GET / HTTP/1.1" 200 5974 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0" 127.0.0.1 - - [30/Sep/2016:15:44:05 +0300] "GET /js/bootstrap.min.js HTTP/1.1" 200 37192 "http://9281.net/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0" 127.0.0.1 - - [30/Sep/2016:15:48:16 +0300] "GET / HTTP/1.1" 200 5974 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0" 127.0.0.1 - - [30/Sep/2016:15:48:16 +0300] "GET /js/bootstrap.min.js HTTP/1.1" 200 37192 "http://127.0.0.1/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0" 127.0.0.1 - - [30/Sep/2016:15:54:43 +0300] "GET / HTTP/1.1" 200 5974 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0" 127.0.0.1 - - [30/Sep/2016:15:54:43 +0300] "GET /js/bootstrap.min.js HTTP/1.1" 200 37192 "http://adsupplyads.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0" 46.166.190.184 - - [30/Sep/2016:16:47:43 +0300] "GET /.git/config HTTP/1.1" 404 451 "-" "fasthttp, Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.63 Safari/537.36" 23.20.60.205 - - [30/Sep/2016:17:43:38 +0300] "HEAD / HTTP/1.1" 200 254 "-" "Cloud mapping experiment. Contact research@pdrlabs.net" 141.212.122.16 - - [30/Sep/2016:17:43:54 +0300] "GET / HTTP/1.1" 200 2335 "-" "Mozilla/5.0 zgrab/0.x" 185.25.148.240 - - [30/Sep/2016:19:38:46 +0300] "GET http://testp1.piwo.pila.pl/testproxy.php HTTP/1.1" 404 457 "-" "Mozilla/5.0 (Windows NT 5.1; rv:32.0) Gecko/20100101 Firefox/31.0" 182.118.55.246 - - [30/Sep/2016:20:34:47 +0300] "GET / HTTP/1.1" 200 2335 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2251.0 Safari/537.36" 109.201.138.240 - - [30/Sep/2016:21:49:26 +0300] "GET / HTTP/1.1" 200 2391 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko" 213.57.114.155 - - [30/Sep/2016:22:37:45 +0300] "GET / HTTP/1.0" 200 5937 "-" "masscan/1.0 (https://github.com/robertdavidgraham/masscan)" 179.210.140.22 - - [01/Oct/2016:00:10:07 +0300] "GET /cgi/common.cgi HTTP/1.0" 404 473 "-" "Wget(linux)" 179.210.140.22 - - [01/Oct/2016:00:10:07 +0300] "GET /stssys.htm HTTP/1.0" 404 469 "-" "Wget(linux)" 179.210.140.22 - - [01/Oct/2016:00:10:08 +0300] "GET / HTTP/1.0" 200 5937 "-" "Wget(linux)" 179.210.140.22 - - [01/Oct/2016:00:10:09 +0300] "POST /command.php HTTP/1.0" 404 470 "-" "Wget(linux)" 193.92.3.66 - - [01/Oct/2016:00:23:28 +0300] "GET / HTTP/1.0" 200 5937 "-" "-" 159.203.168.255 - - [01/Oct/2016:02:30:55 +0300] "GET /muieblackcat HTTP/1.1" 404 471 "-" "-" 159.203.168.255 - - [01/Oct/2016:02:30:55 +0300] "GET //phpMyAdmin/scripts/setup.php HTTP/1.1" 404 487 "-" "-" 159.203.168.255 - - [01/Oct/2016:02:30:56 +0300] "GET //phpmyadmin/scripts/setup.php HTTP/1.1" 404 487 "-" "-" 159.203.168.255 - - [01/Oct/2016:02:30:56 +0300] "GET //pma/scripts/setup.php HTTP/1.1" 404 480 "-" "-" 159.203.168.255 - - [01/Oct/2016:02:30:56 +0300] "GET //myadmin/scripts/setup.php HTTP/1.1" 404 484 "-" "-" 159.203.168.255 - - [01/Oct/2016:02:30:57 +0300] "GET //MyAdmin/scripts/setup.php HTTP/1.1" 404 484 "-" "-" 164.132.51.91 - - [01/Oct/2016:04:30:22 +0300] "GET / HTTP/1.1" 200 2391 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko" 170.233.142.124 - - [01/Oct/2016:07:21:40 +0300] "GET / HTTP/1.1" 200 5918 "-" "curl/7.17.1 (mips-unknown-linux-gnu) libcurl/7.17.1 OpenSSL/0.9.8i zlib/1.2.3" 173.208.213.195 - - [01/Oct/2016:07:22:02 +0300] "GET / HTTP/1.1" 200 5918 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)" 183.129.160.229 - - [01/Oct/2016:08:10:39 +0300] "test" 501 287 "-" "-" 183.129.160.229 - - [01/Oct/2016:08:10:39 +0300] "GET / HTTP/1.1" 200 5937 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:47.0) Gecko/20100101 Firefox/47.0" ================================================ FILE: test-data/cmp0.success ================================================ ================================================ FILE: test-data/cmp1-same1.success ================================================ ================================================ FILE: test-data/cmp1-same2.success ================================================ ================================================ FILE: test-data/cmp2-diff.success ================================================ ================================================ FILE: test-data/cmp2-same.success ================================================ ================================================ FILE: test-data/comm_paste.success ================================================ 1 3 2 4 3 6 4 5 ================================================ FILE: test-data/comm_paste_join_diff.success ================================================ 3 10 2 2 2 2 10 3 12 match 4 match ================================================ FILE: test-data/comm_sort.success ================================================ 9 ================================================ FILE: test-data/d3 ================================================ 4 8 9 ================================================ FILE: test-data/data.csv ================================================ -1,\N,-1,"noproject","Fake entry to indicate a previously existing but currently absent repo",\N,"2016-04-18 11:45:02",\N,0,"0000-00-00 00:00:00" 1,"https://api.github.com/repos/tosch/ruote-kit",1,"ruote-kit","RESTful wrapper for ruote workflow engine","Ruby","2009-12-08 11:17:27",2,0,"2016-03-05 12:23:34" 2,"https://api.github.com/repos/kennethkalmer/ruote-kit",4,"ruote-kit","RESTish wrapper for ruote workflow engine","Ruby","2009-06-10 20:32:21",\N,0,"2016-02-24 07:33:16" 3,"https://api.github.com/repos/matplotlib/basemap",23,"basemap","","C++","2011-02-19 02:58:42",\N,0,"2016-02-27 10:44:13" 4,"https://api.github.com/repos/jswhit/basemap",24,"basemap","","C++","2012-06-14 14:14:56",3,1,"0000-00-00 00:00:00" 5,"https://api.github.com/repos/funkaster/cocos2d-x",28,"cocos2d-x","Port of cocos2d-iphone in C++","C","2012-03-12 17:48:19",6,0,"2016-02-16 23:56:14" 6,"https://api.github.com/repos/cocos2d/cocos2d-x",31,"cocos2d-x","cocos2d-x for C++","C++","2010-11-18 23:17:00",\N,0,"2016-02-11 05:16:59" 7,"https://api.github.com/repos/pixonic/cocos2d-x",42,"cocos2d-x","Port of cocos2d-iphone in C++","C","2012-04-23 12:20:29",6,0,"2016-03-02 01:32:24" 8,"https://api.github.com/repos/NUBIC/ncs_navigator_core",65,"ncs_navigator_core","Case management and instrument administration for NCS Navigator","Ruby","2012-04-19 21:45:52",\N,0,"2016-03-01 07:42:14" 9,"https://api.github.com/repos/sgonyea/rake-compiler",66,"rake-compiler","Provide a standard and simplified way to build and package Ruby C and Java extensions using Rake as glue.","Ruby","2012-08-01 20:33:51",14556189,0,"2016-03-03 18:16:33" 10,"https://api.github.com/repos/chapuni/llvm",67,"llvm","Chapuni's branch based on http://llvm.org/git/llvm.git","C++","2011-02-01 09:11:48",\N,0,"2016-02-10 07:17:58" 11,"https://api.github.com/repos/heroku/heroku-buildpack-scala",69,"heroku-buildpack-scala","Heroku buildpack: Scala","Shell","2011-05-16 21:41:09",\N,0,"2016-02-18 23:07:41" 12,"https://api.github.com/repos/rafacm/heroku-buildpack-scala",70,"heroku-buildpack-scala","","Shell","2012-08-02 14:50:18",11,0,"2016-03-02 12:16:22" 13,"https://api.github.com/repos/Fluttershy/locria",72,"locria","Project Locria - The Beauty of Elegance","JavaScript","2012-07-02 04:29:21",\N,1,"0000-00-00 00:00:00" 14,"https://api.github.com/repos/edvorg/drash",73,"drash","Free box2d based platformer with complete level destruction","C++","2012-07-24 17:17:49",\N,1,"0000-00-00 00:00:00" 15,"https://api.github.com/repos/abarocio80/clide",74,"clide","A CLI project manager (not an IDE)","Ruby","2012-08-01 17:54:15",\N,1,"0000-00-00 00:00:00" 16,"https://api.github.com/repos/thorlax402/thor-cms",75,"thor-cms","PHP-Based content management system developed off the CodeIgniter Framework.",\N,"2012-07-18 17:54:08",\N,1,"0000-00-00 00:00:00" 17,"https://api.github.com/repos/Offsite/TaskCodes",76,"TaskCodes","Kamtech","JavaScript","2012-06-11 17:21:34",\N,0,"2016-03-01 09:43:27" 18,"https://api.github.com/repos/markpasc/gameshake",71,"gameshake","Post MLKSHK game trailers posts to the gameshake tumblr","Python","2012-08-01 20:25:36",\N,0,"2016-02-27 05:44:18" 19,"https://api.github.com/repos/samuelclay/NewsBlur",78,"NewsBlur","NewsBlur is a personal news reader that brings people together to talk about the world. A new sound of an old instrument.","Objective-C","2009-01-05 14:00:43",\N,0,"2016-03-03 09:40:25" 20,"https://api.github.com/repos/chrisjaure/git-lava",80,"git-lava","Branching metaphor for git","Shell","2012-02-27 03:45:44",\N,0,"2016-02-10 16:49:01" 21,"https://api.github.com/repos/ES-DOC/django-cim-forms",82,"django-cim-forms","","Python","2012-03-23 14:20:28",\N,1,"0000-00-00 00:00:00" 23,"https://api.github.com/repos/adammark/Markup.js",86,"Markup.js","Powerful JavaScript templates","JavaScript","2011-08-23 02:30:04",\N,0,"2016-02-04 02:32:47" 24,"https://api.github.com/repos/leoamigood/1stdibs_V2.1",88,"1stdibs_V2.1","Initial setup with mule-pull project","Java","2012-05-03 11:43:31",\N,1,"0000-00-00 00:00:00" 25,"https://api.github.com/repos/pyrovski/Large-Scale-Forward-Regression-using-a-partitioned-linear-model",87,"Large-Scale-Forward-Regression-using-a-partitioned-linear-model","","C++","2012-01-17 20:58:01",\N,0,"2016-03-02 09:21:28" 26,"https://api.github.com/repos/podarsmarty/cobertura-plugin",90,"cobertura-plugin","Jenkins cobertura plugin","Java","2012-07-26 20:46:47",193522,0,"2016-03-02 03:15:20" 27,"https://api.github.com/repos/fbettag/scala-vs-erlang",89,"scala-vs-erlang","A performance test comparing Scala verses Erlang with simple agents to determine messaging performance","Erlang","2011-12-25 14:51:08",1262879,0,"2016-02-16 05:32:13" 28,"https://api.github.com/repos/luislavena/rake-compiler",102,"rake-compiler","Provide a standard and simplified way to build and package Ruby C and Java extensions using Rake as glue.","Ruby","2008-11-03 11:19:43",\N,1,"0000-00-00 00:00:00" 29,"https://api.github.com/repos/Itseez/opencv",111,"opencv","Open Source Computer Vision Library","C++","2012-07-19 09:40:17",\N,0,"2016-02-20 20:12:54" 30,"https://api.github.com/repos/jkammerl/opencv",112,"opencv","OpenCV GitHub Mirror","C++","2012-08-02 14:50:30",29,0,"2016-02-22 14:29:16" 31,"https://api.github.com/repos/gpjt/webgl-lessons",136,"webgl-lessons","https://github.com/tparisi/webgl-lessons is now the officially maintained fork for this project","Python","2009-10-06 12:30:19",\N,0,"2016-02-17 23:53:36" 32,"https://api.github.com/repos/kerolasa/lelux-utiliteetit",139,"lelux-utiliteetit","A contributor clone of the util-linux, see link bellow for official repo","C","2011-01-19 11:34:14",\N,0,"2016-02-24 08:38:28" 33,"https://api.github.com/repos/SnowblindFatal/Glomes",140,"Glomes","The war of balls","Java","2012-06-30 21:05:08",\N,0,"2016-03-04 05:28:23" 34,"https://api.github.com/repos/pockethub/PocketHub",142,"android","PocketHub Android App","JavaScript","2011-09-08 16:52:50",\N,0,"2016-02-17 15:46:56" 35,"https://api.github.com/repos/Mirocow/yii-EasyAPNs",145,"yii-EasyAPNs","","PHP","2012-06-24 19:08:50",\N,0,"2016-02-28 10:21:21" 37,"https://api.github.com/repos/angular/angular.js",159,"angular.js","HTML enhanced for web apps","JavaScript","2010-01-06 00:34:37",\N,0,"2016-02-05 21:29:36" 38,"https://api.github.com/repos/wallysalami/yii-EasyAPNs",154,"yii-EasyAPNs","","PHP","2012-08-02 14:55:50",35,0,"2016-03-06 05:33:28" 39,"https://api.github.com/repos/macmade/OpenCV-iOS",161,"OpenCV-iOS","OpenCV (Open Source Computer Vision) is a library of programming functions for real time computer vision. This project is a port of the OpenCV library for Apple iOS. It includes two XCode projects: one for iPhone, the other one for iPad.","Makefile","2011-04-02 16:38:32",\N,0,"2016-02-26 18:23:18" 40,"https://api.github.com/repos/powmedia/buildify",166,"buildify","Builder for creating distributable JavaScript files from source. Concatenate, wrap, uglify.","JavaScript","2012-05-31 11:09:08",\N,0,"2016-03-02 04:47:25" 41,"https://api.github.com/repos/Liberty-Concepts/redmine_git_hosting",169,"redmine_git_hosting","A ChiliProject/Redmine plugin which makes configuring your own git hosting easy. ","Ruby","2012-07-30 14:53:09",42,0,"2016-02-25 21:01:12" 42,"https://api.github.com/repos/kubitron/redmine_git_hosting",178,"redmine_git_hosting","A ChiliProject/Redmine plugin which makes configuring your own git hosting easy. ","Ruby","2011-10-27 01:17:37",207450,0,"2016-02-25 03:48:29" 43,"https://api.github.com/repos/hpc/iptablesbuild",181,"iptablesbuild","iptablesbuild is effectively a configuration manager for iptables. It is intended to manage iptables configurations in a centralized location for multiple systems.","Perl","2012-08-01 15:19:25",\N,0,"2016-02-19 10:22:47" 44,"https://api.github.com/repos/chenniaoc/OpenCV-iOS",180,"OpenCV-iOS","This project is a port of the OpenCV library for Apple iOS. It includes two XCode projects: one for iPhone, the other one for iPad.","Objective-C","2012-08-02 12:55:52",39,1,"0000-00-00 00:00:00" 45,"https://api.github.com/repos/tijsverkoyen/dotfiles",173,"dotfiles","","Shell","2012-05-05 14:01:59",\N,0,"2016-03-05 07:18:26" 46,"https://api.github.com/repos/6a68/browserid",191,"browserid","A secure, distributed, and easy to use identification system.","JavaScript","2012-07-01 00:35:40",6419421,0,"2016-02-03 18:49:12" 47,"https://api.github.com/repos/samtubbax/dotfiles",194,"dotfiles","","Shell","2012-08-02 14:56:32",45,0,"2016-03-03 09:37:33" 48,"https://api.github.com/repos/jman01/customizations",200,"customizations","Various configuration scripts, templates etc used for customizations","Shell","2012-08-01 19:43:55",\N,0,"2016-02-22 16:19:15" 49,"https://api.github.com/repos/alexgorbatchev/syntaxhighlighter",208,"SyntaxHighlighter","The project has moved to https://github.com/syntaxhighlighter",\N,"2014-10-07 13:48:58",12363088,0,"2016-02-04 15:47:27" 52,"https://api.github.com/repos/fredwu/jquery-endless-scroll",216,"jquery-endless-scroll","Endless/infinite scrolling/pagination.","CoffeeScript","2009-12-02 06:46:03",\N,0,"2016-02-16 20:43:27" 53,"https://api.github.com/repos/kanishkaganguly/Zero-Requiem",240,"Zero-Requiem",\N,"PHP","2012-08-01 20:03:04",\N,0,"2016-02-23 23:07:29" 54,"https://api.github.com/repos/Bronsa/brochure",186,"brochure","[ABANDONED]Experimental implementation of clojure in clojure, WIP","Java","2012-04-14 20:08:58",\N,0,"2016-02-09 03:01:52" 55,"https://api.github.com/repos/yui/yui3",272,"yui3","A library for building richly interactive web applications.","JavaScript","2008-12-05 19:12:24",\N,0,"2016-03-07 01:30:39" 61,"https://api.github.com/repos/jesperes/protobuf-cmake",320,"protobuf-cmake","CMake build support for Google Protobufs","CMake","2010-06-21 18:54:50",\N,0,"2016-02-22 04:13:54" 63,"https://api.github.com/repos/pculture/unisubs",394,"unisubs","Amara","Python","2011-08-19 22:04:08",\N,0,"2016-03-01 20:09:29" 64,"https://api.github.com/repos/imtapps/django-request-signer",359,"django-request-signer","","Python","2012-01-25 14:52:20",\N,0,"2016-02-20 08:17:54" 65,"https://api.github.com/repos/nadafigment/protobuf-cmake",496,"protobuf-cmake","CMake build support for Google Protobufs",\N,"2012-08-02 16:35:08",61,0,"2016-02-29 09:09:12" 66,"https://api.github.com/repos/libraM/django-request-signer",504,"django-request-signer","","Python","2012-06-08 03:13:55",64,1,"0000-00-00 00:00:00" 67,"https://api.github.com/repos/victorlin/loso",537,"loso","Chinese segmentation library","Python","2011-04-15 14:17:20",\N,0,"2016-03-06 00:08:26" 68,"https://api.github.com/repos/lucas0412/loso",616,"loso","Chinese segmentation library","Python","2012-08-02 12:57:02",67,1,"0000-00-00 00:00:00" 69,"https://api.github.com/repos/apipkin/yui3",635,"yui3","YUI 3.x Source Tree","JavaScript","2012-07-13 14:48:22",55,1,"0000-00-00 00:00:00" 70,"https://api.github.com/repos/docTag/doctag_java",650,"doctag_java","","Java","2012-08-02 13:31:59",\N,0,"2016-02-14 03:59:00" 71,"https://api.github.com/repos/llvm-mirror/llvm",672,"llvm","Mirror of official llvm git repository located at http://llvm.org/git/llvm. Updated every five minutes.","C++","2012-01-27 23:49:56",\N,0,"2016-02-26 04:41:20" 72,"https://api.github.com/repos/smarchive/doctag_java",687,"doctag_java","Java library for processing docTag meta data","Java","2012-08-02 12:57:15",70,1,"0000-00-00 00:00:00" 74,"https://api.github.com/repos/joyent/libuv",705,"libuv","Go to ","C","2011-03-29 21:10:45",\N,0,"2016-02-23 07:03:20" 75,"https://api.github.com/repos/schatten/schatten.github.com",704,"schatten.github.com","","CSS","2011-01-18 09:06:12",\N,0,"2016-03-03 13:04:20" 79,"https://api.github.com/repos/gosquared/nvm-cookbook",735,"nvm-cookbook","Chef cookbook for installing and managing nvm on Ubuntu","Ruby","2012-07-03 09:06:42",\N,0,"2016-02-17 23:13:35" 80,"https://api.github.com/repos/daveWid/legacy-php-talk",754,"legacy-php-talk","A slide deck about updating legacy PHP code, for the August 2012 meeting of NWO PUG.","JavaScript","2012-07-30 20:04:59",\N,0,"2016-02-12 20:25:15" 82,"https://api.github.com/repos/mshk/Data-Journalism-Handbook-Ja",820,"Data-Journalism-Handbook-Ja","",\N,"2012-07-11 08:39:41",\N,0,"2016-02-29 02:00:14" 83,"https://api.github.com/repos/RussellSpitzer/sample_app",831,"sample_app","Sample App from Rails Tutorial","Ruby","2012-08-01 17:20:12",\N,0,"2016-03-03 05:12:28" 84,"https://api.github.com/repos/willdurand/willdurand.github.com",858,"willdurand.github.com","My very own blog.","CSS","2012-01-15 22:41:31",\N,0,"2016-03-06 10:38:35" 85,"https://api.github.com/repos/stof/willdurand.github.com",859,"willdurand.github.com","My new blog!","JavaScript","2012-08-02 14:06:43",84,0,"2016-03-04 13:31:33" 86,"https://api.github.com/repos/rxgx/dotfiles",863,"dotfiles",":computer: My user configuration files for OS X Mavericks","Shell","2012-07-25 20:59:33",\N,0,"2016-03-03 05:43:19" 87,"https://api.github.com/repos/Ablu/manaserv",861,"manaserv","A flexible 2D MMORPG server","C++","2011-08-01 19:05:36",90,0,"2016-02-03 23:40:50" 88,"https://api.github.com/repos/garyrussell/spring-integration",866,"spring-integration","","Java","2011-08-12 15:45:00",4952338,0,"2016-02-17 05:46:00" 89,"https://api.github.com/repos/yomoyomo/Data-Journalism-Handbook-Ja",864,"Data-Journalism-Handbook-Ja","",\N,"2012-08-02 14:57:51",82,0,"2016-03-06 23:54:35" 90,"https://api.github.com/repos/mana/manaserv",876,"manaserv","A flexible 2D MMORPG server","C++","2011-03-23 22:15:29",\N,0,"2016-02-26 23:18:10" 91,"https://api.github.com/repos/bjorn/manaserv",875,"manaserv","A flexible 2D MMORPG server","C++","2011-03-24 18:38:40",90,0,"2016-02-08 10:33:30" 92,"https://api.github.com/repos/fnando/i18n-js",877,"i18n-js","It's a small library to provide the I18n translations on the Javascript. It comes with Rails support.","JavaScript","2009-09-02 03:35:45",\N,0,"2016-02-16 15:44:58" 93,"https://api.github.com/repos/olegz/spring-integration",884,"spring-integration","","Java","2011-08-11 16:57:01",4952338,0,"2016-03-01 10:38:26" 94,"https://api.github.com/repos/chapuni/llvm-project",67,"llvm-project","Chapuni's branches for llvm-project. (See also https://github.com/llvm-project/llvm-project )","C++","2011-11-11 11:08:58",\N,0,"2016-02-10 07:17:58" 95,"https://api.github.com/repos/neverabc/libuv",879,"libuv","platform layer for node.js","C","2012-08-02 14:57:13",74,0,"2016-02-29 18:31:17" 96,"https://api.github.com/repos/blinkbox/cucumber-js",882,"cucumber-js","Pure Javascript Cucumber","JavaScript","2012-05-28 15:47:12",10457,1,"0000-00-00 00:00:00" 97,"https://api.github.com/repos/elaird/supy",888,"supy","","Python","2011-09-28 16:51:36",\N,0,"2016-02-15 06:20:02" 98,"https://api.github.com/repos/janrain/jump.ios",891,"jump.ios","created for Nathan Ramsey on 7/13/12","Objective-C","2012-07-13 17:10:38",\N,0,"2016-02-21 10:21:45" 99,"https://api.github.com/repos/timblinkbox/cucumber-js",889,"cucumber-js","Pure Javascript Cucumber","JavaScript","2012-06-21 13:47:25",96,0,"2016-03-05 07:29:30" 100,"https://api.github.com/repos/angular/angular-seed",159,"angular-seed","Seed project for angular apps. ","HTML","2010-12-24 06:07:50",\N,0,"2016-02-05 21:29:36" 101,"https://api.github.com/repos/mashiro/i18n-js",915,"i18n-js","It's a small library to provide the I18n translations on the Javascript. It comes with Rails support.","JavaScript","2012-08-02 14:58:14",92,0,"2016-02-27 08:41:16" 102,"https://api.github.com/repos/JakeWharton/Android-ViewPagerIndicator",896,"Android-ViewPagerIndicator","Paging indicator widgets compatible with the ViewPager from the Android Support Library and ActionBarSherlock. Originally based on Patrik Åkerfeldt's ViewFlow.","Java","2011-08-04 15:13:44",\N,1,"0000-00-00 00:00:00" 103,"https://api.github.com/repos/Evh27/angular-seed",900,"angular-seed","Seed project for angular apps. ","JavaScript","2012-08-02 12:58:13",100,1,"0000-00-00 00:00:00" 104,"https://api.github.com/repos/leon/play-salat",947,"play-salat","MongoDB / Salat plugin for Play 2 [MOVED]","Scala","2012-04-10 11:01:51",\N,0,"2016-02-25 17:27:09" 105,"https://api.github.com/repos/bnoordhuis/libuv",963,"libuv","Cross-platform asychronous I/O","C","2015-01-23 22:47:25",6690806,0,"2016-02-08 15:54:01" 106,"https://api.github.com/repos/oftc/libuv",971,"libuv","platform layer for node.js","C","2012-07-09 14:56:04",74,0,"2016-03-01 09:45:24" 107,"https://api.github.com/repos/shepheb/jotto",981,"jotto","Webapp for the game Jotto using AngularJs, Node.js, Socket.io and PhoneGap.","JavaScript","2012-07-13 23:07:31",\N,0,"2016-03-03 20:42:29" 108,"https://api.github.com/repos/racker/virgo",982,"virgo","","Lua","2011-07-12 04:25:25",\N,1,"0000-00-00 00:00:00" 109,"https://api.github.com/repos/incuna/django-extensible-profiles",985,"django-extensible-profiles","Extensible profiles","Python","2012-02-03 08:58:27",\N,1,"0000-00-00 00:00:00" 110,"https://api.github.com/repos/redaemn/angular-seed",993,"angular-seed","Seed project for angular apps. ","JavaScript","2012-05-19 08:30:13",100,0,"2016-03-02 17:20:26" 112,"https://api.github.com/repos/ZorGleH/try_git",1013,"try_git",\N,\N,"2012-08-01 20:21:56",\N,0,"2016-03-07 08:15:41" 114,"https://api.github.com/repos/madrobby/zepto",1019,"zepto","Zepto.js is a minimalist JavaScript library for modern browsers, with a jQuery-compatible API","HTML","2010-09-20 07:57:57",\N,0,"2016-02-26 19:21:16" 115,"https://api.github.com/repos/ochameau/addon-sdk",1023,"addon-sdk","The Add-on SDK repository.","JavaScript","2010-11-29 14:17:32",127,0,"2016-03-01 09:13:22" 116,"https://api.github.com/repos/brandonwamboldt/utilphp",1024,"utilphp","util.php is a collection of useful functions and snippets that you need or could use every day, designed to avoid conflicts with existing projects","PHP","2012-07-25 00:11:26",\N,0,"2016-02-08 22:44:02" ================================================ FILE: test-data/diff0-noin.success ================================================ ================================================ FILE: test-data/diff0-stdin1.success ================================================ 1d0 < 0 ================================================ FILE: test-data/diff0-stdin2.success ================================================ 0a1 > 0 ================================================ FILE: test-data/diff0.success ================================================ ================================================ FILE: test-data/diff1-stdin.success ================================================ 1c1 < 0 --- > 1 ================================================ FILE: test-data/diff1.success ================================================ 0a1 > 0 ================================================ FILE: test-data/diff2.success ================================================ 1c1 < 0 --- > 1 ================================================ FILE: test-data/diff3-0.success ================================================ ================================================ FILE: test-data/diff3-1.success ================================================ ====3 1:0a 2:0a 3:1c a ================================================ FILE: test-data/diff3-2-stdin1.success ================================================ ==== 1:1,2c a b 2:0a 3:1c b ================================================ FILE: test-data/diff3-2-stdin2.success ================================================ ====2 1:1c 3:1c b 2:1c a ================================================ FILE: test-data/diff3-2.success ================================================ ====2 1:1c 3:1c b 2:1c a ================================================ FILE: test-data/diff3-3.success ================================================ ==== 1:1,2c a c 2:1c b 3:1c c ================================================ FILE: test-data/diff4.success ================================================ 1d0 < 0 1d0 < 1 1d0 < 2 1d0 < 3 ================================================ FILE: test-data/dir-plain.sh ================================================ ls -n | dgsh-tee | {{ awk '!/^total/ {print $6, $7, $8, $1, sprintf("%8d", $5), $9}' & awk '{s += $5} END {printf("%d bytes", s)}' & }} | dgsh-tee ================================================ FILE: test-data/f1s ================================================ 1 2 3 4 5 6 ================================================ FILE: test-data/f2s ================================================ 3 3 4 4 5 ================================================ FILE: test-data/f3s ================================================ 1 3 2 3 3 4 4 4 5 5 6 ================================================ FILE: test-data/f4s ================================================ 4 2 8 10 3 2 9 ================================================ FILE: test-data/f4ss ================================================ 10 2 2 3 4 8 9 ================================================ FILE: test-data/f5s ================================================ 5 1 6 12 4 8 4 9 ================================================ FILE: test-data/f5ss ================================================ 1 12 4 4 5 6 8 9 ================================================ FILE: test-data/ff ================================================ match ================================================ FILE: test-data/function_bash_tools.success ================================================ ================================================ FILE: test-data/function_dgsh_tools.success ================================================ ================================================ FILE: test-data/grep-Lcap-c-l-matching-lines-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-Lcap-c-l-matching-lines.success ================================================ 1 (standard input) hi ================================================ FILE: test-data/grep-Lcap-c-l-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-Lcap-c-l.success ================================================ 1 (standard input) ================================================ FILE: test-data/grep-Lcap-c-matching-lines-l-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-Lcap-c-matching-lines-l.success ================================================ 1 hi (standard input) ================================================ FILE: test-data/grep-Lcap-c-matching-lines-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-Lcap-c-matching-lines.success ================================================ 1 hi ================================================ FILE: test-data/grep-Lcap-c-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-Lcap-c.success ================================================ 1 ================================================ FILE: test-data/grep-Lcap-cat-nomatch.success ================================================ (standard input) ================================================ FILE: test-data/grep-Lcap-cat.success ================================================ ================================================ FILE: test-data/grep-Lcap-l-c-matching-lines-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-Lcap-l-c-matching-lines.success ================================================ (standard input) 1 hi ================================================ FILE: test-data/grep-Lcap-l-c-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-Lcap-l-c.success ================================================ (standard input) 1 ================================================ FILE: test-data/grep-Lcap-l-matching-lines-c-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-Lcap-l-matching-lines-c.success ================================================ (standard input) hi 1 ================================================ FILE: test-data/grep-Lcap-l-matching-lines-nomatch.success ================================================ (standard input) ================================================ FILE: test-data/grep-Lcap-l-matching-lines.success ================================================ (standard input) hi ================================================ FILE: test-data/grep-Lcap-l-nomatch.success ================================================ (standard input) ================================================ FILE: test-data/grep-Lcap-l.success ================================================ (standard input) ================================================ FILE: test-data/grep-Lcap-matching-lines-c-l-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-Lcap-matching-lines-c-l.success ================================================ hi 1 (standard input) ================================================ FILE: test-data/grep-Lcap-matching-lines-c-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-Lcap-matching-lines-c.success ================================================ hi 1 ================================================ FILE: test-data/grep-Lcap-matching-lines-l-c-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-Lcap-matching-lines-l-c.success ================================================ hi (standard input) 1 ================================================ FILE: test-data/grep-Lcap-matching-lines-l-nomatch.success ================================================ (standard input) ================================================ FILE: test-data/grep-Lcap-matching-lines-l.success ================================================ hi (standard input) ================================================ FILE: test-data/grep-Lcap-matching-lines-nomatch.success ================================================ (standard input) ================================================ FILE: test-data/grep-Lcap-matching-lines.success ================================================ hi ================================================ FILE: test-data/grep-Lcap-nomatch.success ================================================ (standard input) ================================================ FILE: test-data/grep-Lcap.success ================================================ ================================================ FILE: test-data/grep-c-Lcap-l-matching-lines-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-c-Lcap-l-matching-lines.success ================================================ 1 (standard input) hi ================================================ FILE: test-data/grep-c-Lcap-l-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-c-Lcap-l.success ================================================ 1 (standard input) ================================================ FILE: test-data/grep-c-Lcap-matching-lines-l-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-c-Lcap-matching-lines-l.success ================================================ 1 hi (standard input) ================================================ FILE: test-data/grep-c-Lcap-matching-lines-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-c-Lcap-matching-lines.success ================================================ 1 hi ================================================ FILE: test-data/grep-c-Lcap-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-c-Lcap.success ================================================ 1 ================================================ FILE: test-data/grep-c-cat.success ================================================ 1 ================================================ FILE: test-data/grep-c-l-Lcap-matching-lines-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-c-l-Lcap-matching-lines.success ================================================ 1 (standard input) hi ================================================ FILE: test-data/grep-c-l-Lcap-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-c-l-Lcap.success ================================================ 1 (standard input) ================================================ FILE: test-data/grep-c-l-matching-lines-Lcap-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-c-l-matching-lines-Lcap.success ================================================ 1 (standard input) hi ================================================ FILE: test-data/grep-c-l-matching-lines.success ================================================ 1 (standard input) hi ================================================ FILE: test-data/grep-c-l.success ================================================ 1 (standard input) ================================================ FILE: test-data/grep-c-matching-lines-Lcap-l-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-c-matching-lines-Lcap-l.success ================================================ 1 hi (standard input) ================================================ FILE: test-data/grep-c-matching-lines-Lcap-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-c-matching-lines-Lcap.success ================================================ 1 hi ================================================ FILE: test-data/grep-c-matching-lines-l-Lcap-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-c-matching-lines-l-Lcap.success ================================================ 1 hi (standard input) ================================================ FILE: test-data/grep-c-matching-lines-l.success ================================================ 1 hi (standard input) ================================================ FILE: test-data/grep-c-matching-lines.success ================================================ 1 hi ================================================ FILE: test-data/grep-c.success ================================================ 1 ================================================ FILE: test-data/grep-f-cat.success ================================================ hi ================================================ FILE: test-data/grep-f.success ================================================ hi ================================================ FILE: test-data/grep-l-Lcap-c-matching-lines-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-l-Lcap-c-matching-lines.success ================================================ (standard input) 1 hi ================================================ FILE: test-data/grep-l-Lcap-c-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-l-Lcap-c.success ================================================ (standard input) 1 ================================================ FILE: test-data/grep-l-Lcap-matching-lines-c-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-l-Lcap-matching-lines-c.success ================================================ (standard input) hi 1 ================================================ FILE: test-data/grep-l-Lcap-matching-lines-nomatch.success ================================================ (standard input) ================================================ FILE: test-data/grep-l-Lcap-matching-lines.success ================================================ (standard input) hi ================================================ FILE: test-data/grep-l-Lcap-nomatch.success ================================================ (standard input) ================================================ FILE: test-data/grep-l-Lcap.success ================================================ (standard input) ================================================ FILE: test-data/grep-l-c-Lcap-matching-lines-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-l-c-Lcap-matching-lines.success ================================================ (standard input) 1 hi ================================================ FILE: test-data/grep-l-c-Lcap-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-l-c-Lcap.success ================================================ (standard input) 1 ================================================ FILE: test-data/grep-l-c-matching-lines-Lcap-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-l-c-matching-lines-Lcap.success ================================================ (standard input) 1 hi ================================================ FILE: test-data/grep-l-c-matching-lines.success ================================================ (standard input) 1 hi ================================================ FILE: test-data/grep-l-c.success ================================================ (standard input) 1 ================================================ FILE: test-data/grep-l-cat.success ================================================ (standard input) ================================================ FILE: test-data/grep-l-matching-lines-Lcap-c-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-l-matching-lines-Lcap-c.success ================================================ (standard input) hi 1 ================================================ FILE: test-data/grep-l-matching-lines-Lcap-nomatch.success ================================================ (standard input) ================================================ FILE: test-data/grep-l-matching-lines-Lcap.success ================================================ (standard input) hi ================================================ FILE: test-data/grep-l-matching-lines-c-Lcap-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-l-matching-lines-c-Lcap.success ================================================ (standard input) hi 1 ================================================ FILE: test-data/grep-l-matching-lines-c.success ================================================ (standard input) hi 1 ================================================ FILE: test-data/grep-l-matching-lines.success ================================================ (standard input) hi ================================================ FILE: test-data/grep-l.success ================================================ (standard input) ================================================ FILE: test-data/grep-matching-lines-Lcap-c-l-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-matching-lines-Lcap-c-l.success ================================================ hi 1 (standard input) ================================================ FILE: test-data/grep-matching-lines-Lcap-c-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-matching-lines-Lcap-c.success ================================================ hi 1 ================================================ FILE: test-data/grep-matching-lines-Lcap-l-c-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-matching-lines-Lcap-l-c.success ================================================ hi (standard input) 1 ================================================ FILE: test-data/grep-matching-lines-Lcap-l-nomatch.success ================================================ (standard input) ================================================ FILE: test-data/grep-matching-lines-Lcap-l.success ================================================ hi (standard input) ================================================ FILE: test-data/grep-matching-lines-Lcap-nomatch.success ================================================ (standard input) ================================================ FILE: test-data/grep-matching-lines-Lcap.success ================================================ hi ================================================ FILE: test-data/grep-matching-lines-c-Lcap-l-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-matching-lines-c-Lcap-l.success ================================================ hi 1 (standard input) ================================================ FILE: test-data/grep-matching-lines-c-Lcap-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-matching-lines-c-Lcap.success ================================================ hi 1 ================================================ FILE: test-data/grep-matching-lines-c-l-Lcap-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-matching-lines-c-l-Lcap.success ================================================ hi 1 (standard input) ================================================ FILE: test-data/grep-matching-lines-c-l.success ================================================ hi 1 (standard input) ================================================ FILE: test-data/grep-matching-lines-c.success ================================================ hi 1 ================================================ FILE: test-data/grep-matching-lines-cat.success ================================================ hi ================================================ FILE: test-data/grep-matching-lines-l-Lcap-c-nomatch.success ================================================ (standard input) 0 ================================================ FILE: test-data/grep-matching-lines-l-Lcap-c.success ================================================ hi (standard input) 1 ================================================ FILE: test-data/grep-matching-lines-l-Lcap-nomatch.success ================================================ (standard input) ================================================ FILE: test-data/grep-matching-lines-l-Lcap.success ================================================ hi (standard input) ================================================ FILE: test-data/grep-matching-lines-l-c-Lcap-nomatch.success ================================================ 0 (standard input) ================================================ FILE: test-data/grep-matching-lines-l-c-Lcap.success ================================================ hi (standard input) 1 ================================================ FILE: test-data/grep-matching-lines-l-c.success ================================================ hi (standard input) 1 ================================================ FILE: test-data/grep-matching-lines-l.success ================================================ hi (standard input) ================================================ FILE: test-data/grep-matching-lines.success ================================================ hi ================================================ FILE: test-data/grep-noargs-cat.success ================================================ hi ================================================ FILE: test-data/grep-noargs.success ================================================ hi ================================================ FILE: test-data/grep-o-cat.success ================================================ hi ================================================ FILE: test-data/grep-o.success ================================================ hi ================================================ FILE: test-data/grep-v-cat.success ================================================ hi ================================================ FILE: test-data/grep-v.success ================================================ hi ================================================ FILE: test-data/grep_comm.success ================================================ ../test-data/ff ../test-data/F ================================================ FILE: test-data/group.success ================================================ a ================================================ FILE: test-data/hello ================================================ hello ================================================ FILE: test-data/j2 ================================================ 12 match 4 match 9 don't match ================================================ FILE: test-data/join_sort.success ================================================ 3 3 4 4 5 ================================================ FILE: test-data/join_sort_diff.success ================================================ 1,7c1,6 < < 1 3 < 2 3 < 3 4 < 4 4 < 5 5 < 6 --- > > 3 > 3 > 4 > 4 > 5 ================================================ FILE: test-data/last ================================================ 4 8 9 ================================================ FILE: test-data/multipipe_one_last.success ================================================ ================================================ FILE: test-data/multipipe_one_start.success ================================================ ================================================ FILE: test-data/nondgsh.success ================================================ 0 1 ================================================ FILE: test-data/p1 ================================================ 3 2 2 10 ================================================ FILE: test-data/paste_diff.success ================================================ 1,7c1,7 < < 1 < 2 < 3 < 4 < 5 < 6 --- > > 1 3 > 2 3 > 3 4 > 4 4 > 5 5 > 6 ================================================ FILE: test-data/read_while.success ================================================ hi there ================================================ FILE: test-data/recursive_multipipe_oneline_end.success ================================================ world hello ================================================ FILE: test-data/recursive_multipipe_oneline_start.success ================================================ hello world ================================================ FILE: test-data/results ================================================ 3 2 2 10 ================================================ FILE: test-data/secho_paste.success ================================================ hello world ================================================ FILE: test-data/secho_secho_fgrep.success ================================================ match ================================================ FILE: test-data/sort_sort_comm.success ================================================ 10 2 2 3 1 12 4 5 6 4 8 9 ================================================ FILE: test-data/sort_sort_comm_paste_join_diff.success ================================================ 3 10 2 2 2 2 10 3 12 match 4 match ================================================ FILE: test-data/subshell.success ================================================ a ================================================ FILE: test-data/tee-copy_diff_comm.success ================================================ 1c1 < world --- > hello hello ================================================ FILE: test-data/top ================================================ 12 match 4 match 9 don't match ================================================ FILE: test-data/world ================================================ world ================================================ FILE: test-data/wrap-cat_comm_sort.success ================================================ 9 ================================================ FILE: unix-tools/.gitignore ================================================ bin/* cat cmp coreutils/* diff diff3 etc/* gawk/* gnulib/* grep/* include/* lib/* libexec/* parallel/* patch/* secho sed/* share/* tar/* tee w # Test files A A+B.dgsh.ft2 A-B.dgsh.ft2 B Fig/ character.txt diff_last digram.txt fft-input grep-p100/ grep2/ grepp/ pecho results top_results trigram.txt words.txt dgsh-compat ================================================ FILE: unix-tools/Makefile ================================================ include ../.config # All but ginstall comm join paste sort # ginstall, renamed from install, is used to install the programs DISABLE_COREUTILS_PROGRAMS=\ who users uptime stty stdbuf pinky nice install hostid df chroot chcon \ cat basename base64 base32 [ expr expand env echo du dirname dircolors \ dir dd date csplit cp cksum chown chmod chgrp nproc nl mv \ mktemp mknod mkfifo mkdir md5sum ls logname ln link kill id head \ groups fold fmt false factor sha512sum sha384sum sha256sum sha224sum \ sha1sum seq runcon rmdir rm realpath readlink pwd ptx printf printenv \ pr pathchk od numfmt nohup unexpand uname tty tsort truncate true \ tr touch timeout test tee tail tac sync sum stat split sleep shuf \ shred yes whoami wc vdir unlink uniq FID=http://www.bmrb.wisc.edu/ftp/pub/bmrb/timedomain/bmr6443/timedomain_data/c13-hsqc/june11-se-6426-CA.fid/fid PREFIX?=/usr/local/dgsh DGSHPATH=$(PREFIX)/libexec/dgsh DGSHDIR=../core-tools/src PSDIR=../test-data EGDIR=../example MANDUMPDIR=/var/tmp/dgsh-share-duplicates # TOOLDIR TD=unix-tools STD=/usr/bin SCRIPTDIR=tool_scripts INCPATHS= -I$(DGSHDIR) # Color GR=\033[0;32m # Green R=\033[0;31m # Red B=\033[0;34m # Blue EC=\033[0m # End color S=${GR}successful${EC} F=${R}failed${EC} ifdef DEBUG CFLAGS+=-DDEBUG endif OS = $(shell uname -s) ifeq ($(OS), Linux) BZIP2LOCATION=/bin/bzip2 GZIPLOCATION=/bin/gzip SEDLOCATION=/bin/sed XZLOCATION=/usr/bin/xz else ifeq ($(OS), Darwin) BZIP2LOCATION=/usr/bin/bzip2 GZIPLOCATION=/usr/bin/gzip SEDLOCATION=/usr/bin/sed XZLOCATION=/usr/local/bin/xz endif endif # If not cloned repo with --recursive use this target to # clone the submodule repos get-submodules: cd .. && git submodule update --init --recursive --depth=5000 \ $(TD)/bash \ $(TD)/coreutils \ $(TD)/grep configure: cd bash && ./configure --prefix=$(PREFIX) --bindir=$(DGSHPATH) \ --docdir=$(MANDUMPDIR) \ --mandir=$(MANDUMPDIR) \ --infodir=$(MANDUMPDIR) \ --localedir=$(MANDUMPDIR) cd coreutils && ./bootstrap && ../cygwin-sys-select-patch.sh && \ ./configure --prefix=$(PREFIX) \ --bindir=$(DGSHPATH) \ --enable-no-install-program="$(DISABLE_COREUTILS_PROGRAMS)" \ --docdir=$(MANDUMPDIR) \ --mandir=$(MANDUMPDIR) \ --infodir=$(MANDUMPDIR) \ --localedir=$(MANDUMPDIR) cd grep && ./bootstrap && ./configure --prefix=$(PREFIX) \ --bindir=$(DGSHPATH) \ --docdir=$(MANDUMPDIR) \ --mandir=$(MANDUMPDIR) \ --infodir=$(MANDUMPDIR) \ --localedir=$(MANDUMPDIR) make: $(MAKE) -C bash CFLAGS="$(CFLAGS)" $(MAKE) -C coreutils $(MAKE) -C grep build-install: cat cmp diff diff3 tee make mkdir -p ../build/libexec/dgsh ../build/bin cp bash/bash ../build/libexec/dgsh/ rm -f ../build/bin/dgsh ln bash/bash ../build/bin/dgsh cp coreutils/src/comm ../build/libexec/dgsh/ cp coreutils/src/cut ../build/libexec/dgsh/ cp coreutils/src/join ../build/libexec/dgsh/ cp coreutils/src/paste ../build/libexec/dgsh/ cp coreutils/src/sort ../build/libexec/dgsh/ cp grep/src/grep ../build/libexec/dgsh/ cp grep/src/egrep ../build/libexec/dgsh/ cp grep/src/fgrep ../build/libexec/dgsh/ chmod 755 cat cmp diff diff3 tee cp -p cat cmp diff diff3 tee ../build/libexec/dgsh/ ./install-wrapped.sh $$(cd .. ; pwd)/build install: cat cmp diff diff3 tee make $(MAKE) -C bash install rm -f $(DESTDIR)$(PREFIX)/bin/dgsh ln $(DESTDIR)$(DGSHPATH)/bash $(DESTDIR)$(PREFIX)/bin/dgsh || install $(DESTDIR)$(DGSHPATH)/bash $(DESTDIR)$(PREFIX)/bin/dgsh $(MAKE) -C coreutils install $(MAKE) -C grep install # Install last to overwrite standard tools of coreutils install cat $(DESTDIR)$(DGSHPATH) install cmp $(DESTDIR)$(DGSHPATH) install diff $(DESTDIR)$(DGSHPATH) install diff3 $(DESTDIR)$(DGSHPATH) install tee $(DESTDIR)$(DGSHPATH) ./install-wrapped.sh rm -rf $(DESTDIR)$(MANDUMPDIR) cat: cat.sh install $? $@ cmp: cmp.sh install $? $@ diff: diff.sh install $? $@ diff3: diff3.sh install $? $@ tee: tee.sh install $? $@ test: test-bash test-compat path: export PATH echo $(PATH) ../test-data/fid: wget -O $@ $(FID) || curl $(FID) >$@ test-compat: dgsh-compat ./test-compat.sh dgsh-compat: bash/dgsh_util.c cc -o $@ $(INCPATHS) -DDGSH_COMPAT $? test-bash: ../test-data/fid rm -f $(PSDIR)/*.out $(PSDIR)/*.outb printf "$B\nBash tests:${EC}\n" ./run_all_simple_tests.sh $(PSDIR) printf " $BPaper examples under ../example:${EC}\n" # No diff, just check execution exit status ./run_test.sh $(PSDIR) $(EGDIR)/dir.sh 0 ./run_test.sh $(PSDIR) $(EGDIR)/reorder-columns.sh pipe $(PSDIR)/data.csv # TODO reinstate after fixin it # ./run_test.sh $(PSDIR) $(EGDIR)/set-operations.sh file $(PSDIR) ./run_test.sh $(PSDIR) $(EGDIR)/compress-compare.sh pipe Readme.md # TODO check that .git repo exists ./run_test.sh $(PSDIR) $(EGDIR)/commit-stats.sh DGSH_TIMEOUT=20 ./run_test.sh $(PSDIR) $(EGDIR)/uniform-5x5.sh ./run_test.sh $(PSDIR) $(EGDIR)/duplicate-files.sh file coreutils ./run_test.sh $(PSDIR) $(EGDIR)/spell-highlight.sh pipe Readme.md ./run_test.sh $(PSDIR) $(EGDIR)/static-functions.sh file coreutils ./run_test.sh $(PSDIR) $(EGDIR)/word-properties.sh file Readme.md ./run_test.sh $(PSDIR) $(EGDIR)/map-hierarchy.sh file ../core-tools/tests-regression/map-hierarchy/in/a ../core-tools/tests-regression/map-hierarchy/in/b c ./run_test.sh $(PSDIR) $(EGDIR)/text-properties.sh pipe Readme.md cd coreutils && \ ../run_test.sh ../$(PSDIR) ../$(EGDIR)/committer-plot.sh -- pnmtopng pamscale pgmmorphconv DGSH_TIMEOUT=60 ./run_test.sh $(PSDIR) $(EGDIR)/code-metrics.sh file coreutils/src KVSTORE_RETRY_LIMIT=100 DGSH_TIMEOUT=100 ./run_test.sh $(PSDIR) $(EGDIR)/web-log-report.sh pipe $(PSDIR)/access.log ./run_test.sh $(PSDIR) $(EGDIR)/fft-block8.sh file $(PSDIR)/fft-input.dat ./run_test.sh $(PSDIR) $(EGDIR)/ft2d.sh -- sfimag sfgrey sfspike sfput sfmath \ sfspray sffft1 sffft3 sfcat sflmostretch \ sfreverse sfwindow vppen sfsmooth # Requires file fid in $(PSDIR). See $(EGDIR)/NMRPipe.sh ./run_test.sh $(PSDIR) $(EGDIR)/NMRPipe.sh file $(PSDIR)/fid -- nmrPipe addNMR clean: $(MAKE) -C bash clean $(MAKE) -C coreutils clean $(MAKE) -C grep clean ================================================ FILE: unix-tools/Readme.md ================================================ ## Installation of unix tools adapted for dgsh While under the dgsh root directory, type the following sequence of commands: ```bash make libdgsh.a # compile the dgsh negotiation library cd unix-tools make get-submodules # download repos of Unix tools if not already cloned # with the superproject through --recursive make configure # configure tools make make # compile make install # install in ./bin make -s test # run tests ``` The current collection of Unix tools include *coreutils, gawk, grep, parallel, sed, and tar*. Of those currently *comm*, *join*, *paste*, and *sort* have been adapted for use with dgsh. ## Adaptation workflow Head to the tool's repo, e.g., unix-tools/coreutils and locate its source code, e.g., src/comm.c The workflow includes the following steps: - adapt the tool for use with dgsh - invoke the dgsh negotiation API similarly to the code that follows - use the provided file descriptors in place of the tool's original input and output interface. You can take a look at the already adapted tools source. ```C #include <stdio.h> /* printf() */ #include <stdlib.h> /* exit() */ #include "dgsh.h" int main(int argc, char *argv[]) { if (dgsh_negotiate("secho", NULL, NULL, NULL, NULL) != 0) exit(1); ++argv; while (*argv) { (void)printf("%s", *argv); if (*++argv) putchar(' '); } putchar('\n'); return 0; } ``` - contribute tests - *cd simple_shell* - express a graph of dgsh adapted processes with interconnected inputs and outputs, such as the example that follows. The example has the secho and paste tools connect the former's output to the latter's input. Because they are both dgsh-compatible they use a socketpipe to carry out the dgsh negotiation phase. You can find this file as *secho_paste.dgsh*. - run the test with *python simple_shell.py secho_paste.dgsh secho_paste.out* to produce and store the output in file *secho_paste.out*. ```bash 1 /home/mfg/dds/dgsh/unix-tools/bin/secho hello 2 /home/mfg/dds/dgsh/unix-tools/bin/paste - world % socketpipe 1 2 ``` - add tests to Makefile's *make test* recipe ```Makefile PSEUDOSHELLDIR=../simple-shell test: cd $PSEUDOSHELLDIR && \ python simple-shell.py secho_paste.dgsh secho_paste.out && \ diff secho_paste.out secho_paste.success && \ echo "Test secho | paste successful." || \ echo "Test secho | paste failed." ``` - run tests ```bash make -s test ``` ================================================ FILE: unix-tools/cat.sh ================================================ #!/usr/bin/env bash #!dgsh # # Implementation of POSIX tee through dgsh-tee # usage() { echo 'Usage: cat [-u] [file ...]' 1>&2 exit 2 } while getopts 'u' o; do case "$o" in u) ;; *) usage ;; esac done shift $((OPTIND-1)) declare -a opts # Process file arguments for i; do opts+=('-i' "$i") shift done exec dgsh-tee "${opts[@]}" ================================================ FILE: unix-tools/cmp.sh ================================================ #!/usr/bin/env bash #!dgsh # # Dgsh-compatible wrapping of the GNU cmp command # Depending on the arguments, the command can accept 0-2 inputs # oopt='' # For the parsing of long options see # https://stackoverflow.com/questions/402377/using-getopts-in-bash-shell-script-to-get-long-and-short-command-line-options/7680682#7680682 optspec=":bi:ln:sv-:" while getopts "$optspec" optchar; do case "${optchar}" in -) case "${OPTARG}" in quiet) oopt='-o 0' ;; silent) oopt='-o 0' ;; # Long options with mandatory arguments. In these the argument # can appear separated from the option name, and must be removed. ignore-initial|bytes) OPTIND=$(($OPTIND + 1)) ;; esac ;; s) oopt='-o 0' ;; esac done # Examine the number of remaining (non-option) arguments case $(($# - $OPTIND + 1)) in 0) # Exactly two inputs are required exec dgsh-wrap $oopt /usr/bin/cmp "$@" '<|' '<|' ;; 1) if [[ "${!OPTIND}" == '-' ]] ; then # One (non-stdin) input is required exec dgsh-wrap -I $oopt /usr/bin/cmp "$@" '<|' else # One (stdin) input is required exec dgsh-wrap $oopt /usr/bin/cmp "$@" - fi ;; *) ARG2=$((OPTIND + 1)) if [[ "${!OPTIND}" == '-' ]] || [[ "${!ARG2}" == '-' ]] ; then # One (stdin) input is required exec dgsh-wrap /usr/bin/cmp $oopt "$@" else # No input is required exec dgsh-wrap -i 0 /usr/bin/cmp $oopt "$@" fi ;; esac ================================================ FILE: unix-tools/cpow.c ================================================ /* $NetBSD: cpow.c,v 1.1 2007/08/20 16:01:35 drochner Exp $ */ /*- * Copyright (c) 2007 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software written by Stephen L. Moshier. * It is redistributed by the NetBSD Foundation by permission of the author. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include <complex.h> #include <math.h> double complex cpow(double complex a, double complex z) { double complex w; double x, y, r, theta, absa, arga; x = creal(z); y = cimag(z); absa = cabs(a); if (absa == 0.0) { return (0.0 + 0.0 * I); } arga = carg(a); r = pow(absa, x); theta = x * arga; if (y != 0.0) { r = r * exp(-y * arga); theta = theta + y * log(absa); } w = r * cos(theta) + (r * sin(theta)) * I; return w; } ================================================ FILE: unix-tools/cygwin-sys-select-patch.sh ================================================ #!/bin/sh # # Patch gnulib to compile under Cygwin # Must be executed after bootstrapping and before configuring gnulib # # See https://github.com/stilor/crosstool-ng/blob/f6ea9a68b26830f72f8f5242aba9b950f2e4da78/patches/gettext/0.19.7/140-Fix-Cygwin-sys-select.patch # Exit unless running under Cygwin uname -o | grep -q Cygwin || exit 0 # Patch the bootstrapped gnulib patch lib/sys_select.in.h <<\EOF diff --git a/gettext-tools/gnulib-lib/sys_select.in.h b/gettext-tools/gnulib-lib/sys_select.in.h index d6d3f9f..7281144 100644 --- a/gettext-tools/gnulib-lib/sys_select.in.h +++ b/gettext-tools/gnulib-lib/sys_select.in.h @@ -81,8 +81,9 @@ of 'struct timeval', and no definition of this type. Also, Mac OS X, AIX, HP-UX, IRIX, Solaris, Interix declare select() in <sys/time.h>. - But avoid namespace pollution on glibc systems. */ -# ifndef __GLIBC__ + But avoid namespace pollution on glibc systems and "unknown type + name" problems on Cygwin. */ +# if !(defined __GLIBC__ || defined __CYGWIN__) # include <sys/time.h> # endif @@ -100,10 +101,11 @@ #endif /* Get definition of 'sigset_t'. - But avoid namespace pollution on glibc systems. + But avoid namespace pollution on glibc systems and "unknown type + name" problems on Cygwin. Do this after the include_next (for the sake of OpenBSD 5.0) but before the split double-inclusion guard (for the sake of Solaris). */ -#if !(defined __GLIBC__ && !defined __UCLIBC__) +#if !((defined __GLIBC__ || defined __CYGWIN__) && !defined __UCLIBC__) # include <signal.h> #endif EOF ================================================ FILE: unix-tools/diff.sh ================================================ #!/usr/bin/env bash #!dgsh # # Dgsh-compatible wrapping of the GNU diff command # Depending on the arguments, the command can accept 0-N inputs # # For the parsing of long options see # https://stackoverflow.com/questions/402377/using-getopts-in-bash-shell-script-to-get-long-and-short-command-line-options/7680682#7680682 optspec=":qscC:uU:enyW:pF:tTlrNx:X:S:iEZbwBIaD:dv-:" while getopts "$optspec" optchar; do case "${optchar}" in -) case "${OPTARG}" in from-file) from_file="${!OPTIND}" OPTIND=$(($OPTIND + 1)) ;; from-file=*) from_file=${OPTARG#*=} ;; to-file) to_file="${!OPTIND}" OPTIND=$(($OPTIND + 1)) ;; to-file=*) to_file=${OPTARG#*=} ;; # Long options with mandatory arguments. In these the argument # can appear separated from the option name, and must be removed. width|show-function-line|label|tabsize|exclude|exclude-from|starting-file|ignore-matching-lines|ifdef|GTYPE-group-format|line-format|LTYPE-line-format|horizon-lines|palette) OPTIND=$(($OPTIND + 1)) ;; esac ;; esac done # Examine the number of remaining (non-option) arguments case $(($# - $OPTIND + 1)) in 0) if [[ -n "$from_file" ]] || [[ -n "$to_file" ]] ; then # An arbitrary number of inputs can be handled exec dgsh-wrap -i a /usr/bin/diff "$@" else # Exactly two inputs are required exec dgsh-wrap /usr/bin/diff "$@" '<|' '<|' fi ;; 1) if [[ -n "$from_file" ]] || [[ -n "$to_file" ]] ; then # No input is required exec dgsh-wrap -i 0 /usr/bin/diff "$@" elif [[ "${!OPTIND}" == '-' ]] ; then # One (non-stdin) input is required exec dgsh-wrap -I /usr/bin/diff "$@" '<|' else # One (stdin) input is required exec dgsh-wrap /usr/bin/diff "$@" - fi ;; *) ARG2=$((OPTIND + 1)) if [[ "${!OPTIND}" == '-' ]] || [[ "${!ARG2}" == '-' ]] ; then # One (stdin) input is required exec dgsh-wrap /usr/bin/diff "$@" else # No input is required exec dgsh-wrap -i 0 /usr/bin/diff "$@" fi ;; esac ================================================ FILE: unix-tools/diff3.sh ================================================ #!/usr/bin/env bash #!dgsh # # Dgsh-compatible wrapping of the GNU diff3 command # Depending on the arguments, the command can accept 0-3 inputs # # For the parsing of long options see # https://stackoverflow.com/questions/402377/using-getopts-in-bash-shell-script-to-get-long-and-short-command-line-options/7680682#7680682 optspec=":AeE3xXimaTL:v-:" while getopts "$optspec" optchar; do case "${optchar}" in -) case "${OPTARG}" in # Long options with mandatory arguments. In these the argument # can appear separated from the option name, and must be removed. diff-program|label) OPTIND=$(($OPTIND + 1)) ;; esac ;; esac done # Examine the number of remaining (non-option) arguments case $(($# - $OPTIND + 1)) in 0) # Exactly three inputs are required exec dgsh-wrap /usr/bin/diff3 "$@" '<|' '<|' '<|' ;; 1) if [[ "${!OPTIND}" = '-' ]] ; then # Two inputs (excluding stdin) are required exec dgsh-wrap -I /usr/bin/diff3 "$@" '<|' '<|' else # Two inputs (including stdin) are required exec dgsh-wrap /usr/bin/diff3 "$@" '<|' '<|' fi ;; 2) ARG2=$((OPTIND + 1)) if [[ "${!OPTIND}" = '-' ]] || [[ "${!ARG2}" = '-' ]] ; then # One (non-stdin) input is required exec dgsh-wrap -I /usr/bin/diff3 "$@" '<|' else # One (stdin) input is required exec dgsh-wrap /usr/bin/diff3 "$@" - fi ;; *) ARG2=$((OPTIND + 1)) ARG3=$((OPTIND + 2)) if [[ "${!OPTIND}" = '-' ]] || [[ "${!ARG2}" = '-' ]] || [[ "${!ARG3}" = '-' ]] ; then # One (stdin) input is required exec dgsh-wrap /usr/bin/diff3 "$@" else # No input is required exec dgsh-wrap -i 0 /usr/bin/diff3 "$@" fi ;; esac ================================================ FILE: unix-tools/echo_echo_dgsh-tee.sh ================================================ {{ echo hello & echo world & }} | dgsh-tee ================================================ FILE: unix-tools/install-wrapped.sh ================================================ #!/bin/sh # # Install the commands wrapped by dgsh-wrap # if [ "$1" = "" ] ; then DGPATH=$DESTDIR$PREFIX/libexec/dgsh mkdir -p $DGPATH else DGPATH=$1/libexec/dgsh mkdir -p $DGPATH fi # Remove comments and blank lines sed 's/[ \t]*#.*//;/^$/d' wrapped-commands-posix wrapped-commands-tests | while read mode name ; do # Continue if command is not available if ! which $name 2>/dev/null >/dev/null ; then continue fi # Continue if command is custom-implemented or both deaf and mute if [ $mode = c -o $mode = dm ] ; then continue fi # Iterate over the mode's characters creating $opt opt=' -s' for m in $(echo "$mode" | sed 's/./& /g') ; do case $m in m) # Mute opt="$opt -o 0" ;; M) # Mute unless - is specified (TODO) ;; d) # Deaf opt="$opt -i 0" ;; I) # Count stdin in channel assignments opt="$opt -I" ;; D) # Deaf unless - is specified or no arguments are provided (TODO) ;; f) # Filter ;; *) echo "Unknown I/O mode character $m for $name" 1>&2 exit 1 esac done target=$DGPATH/$name echo "#!$DGPATH/dgsh-wrap$opt" >$target chmod 755 $target done ================================================ FILE: unix-tools/run_all_simple_tests.sh ================================================ #!/bin/sh PSDIR=$1 set -e ./run_simple_test.sh $PSDIR multipipe_one_last \ 'cat /dev/null | {{ cat & }}' ./run_simple_test.sh $PSDIR multipipe_one_start \ '{{ cat /dev/null & }} | cat' CAT=`which cat` ./run_simple_test.sh $PSDIR function_bash_tools \ "function h { $CAT | $CAT } function g { $CAT | h | $CAT } $CAT /dev/null | g" ./run_simple_test.sh $PSDIR function_dgsh_tools \ 'function h { cat | cat } function g { cat | h | cat } cat /dev/null | g' ./run_simple_test.sh $PSDIR read_while \ 'echo "hi there" | while read X; do echo $X; done' ./run_simple_test.sh $PSDIR subshell \ "(echo a) | cat" ./run_simple_test.sh $PSDIR group \ "{ echo a ;} | cat" ./run_simple_test.sh $PSDIR recursive_multipipe_oneline_start \ "{{ {{ echo hello ; }} | cat ; echo world ; }} | cat" ./run_simple_test.sh $PSDIR recursive_multipipe_oneline_end \ "{{ echo world ; {{ echo hello ; }} | cat ; }} | cat" ./run_simple_test.sh $PSDIR nondgsh \ "true || false dgsh-enumerate 2 | cat" ./run_simple_test.sh $PSDIR secho_paste \ "dgsh-pecho hello | paste $PSDIR/world" ./run_simple_test.sh $PSDIR wrap-cat_comm_sort \ "cat $PSDIR/f1s | comm $PSDIR/f2s | sort | wc -l | tr -d \" \"" ./run_simple_test.sh $PSDIR comm_sort \ "comm $PSDIR/f1s $PSDIR/f2s | sort | wc -l | tr -d \" \"" ./run_simple_test.sh $PSDIR comm_paste \ "comm $PSDIR/f1s $PSDIR/f2s | paste" ./run_simple_test.sh $PSDIR join_sort \ "join $PSDIR/f1s $PSDIR/f2s | sort" ./run_simple_test.sh $PSDIR paste_diff \ "paste $PSDIR/f1s $PSDIR/f2s | diff $PSDIR/f1s | cat" #./run_simple_test.sh $(DGSHPATH) $PSDIR grep_diff \ # "grep -v -w match $PSDIR/F $PSDIR/ff \ # | diff" ./run_simple_test.sh $PSDIR grep_comm \ "grep -l -L match $PSDIR/ff $PSDIR/F | comm | paste" ./run_simple_test.sh $PSDIR join_sort_diff \ "join $PSDIR/f1s $PSDIR/f2s | sort | diff $PSDIR/f3s | cat" # `date`: Check that command substitution # does not mess pipe substitution ./run_simple_test.sh $PSDIR secho_secho_fgrep \ "{{ dgsh-pecho match dgsh-pecho \"not `date`\" }} | grep -F -h match" ./run_simple_test.sh $PSDIR tee-copy_diff_comm \ "tee <$PSDIR/hello | {{ diff $PSDIR/world comm $PSDIR/hello }} | cat" # ditto #./run_simple_test.sh $(DGSHPATH) $PSDIR grep_diff_comm \ # "grep -l -L -w -v match $PSDIR/ff $PSDIR/F \ # | {{ \ # diff & \ # comm & \ # }}" ./run_simple_test.sh $PSDIR comm_paste_join_diff \ "comm $PSDIR/f4ss $PSDIR/f5ss | {{ paste $PSDIR/p1 join $PSDIR/j2 diff $PSDIR/d3 }} | cat" ./run_simple_test.sh $PSDIR sort_sort_comm \ "{{ sort $PSDIR/f4s 2>$PSDIR/f4s.errb sort $PSDIR/f5s 2>$PSDIR/f5s.errb }} | comm | cat" ./run_simple_test.sh $PSDIR sort_sort_comm_paste_join_diff \ "{{ sort $PSDIR/f4s sort $PSDIR/f5s }} | comm | {{ paste $PSDIR/p1 join $PSDIR/j2 diff $PSDIR/d3 }} | cat" # Wrapped diff ./run_simple_test.sh $PSDIR diff4 '! dgsh-enumerate 4 | diff -w --label foo --to-file=/dev/null' ./run_simple_test.sh $PSDIR diff2 '! dgsh-enumerate 2 | diff -w --label foo' ./run_simple_test.sh $PSDIR diff1 '! dgsh-enumerate 1 | diff -w --label foo /dev/null' ./run_simple_test.sh $PSDIR diff1-stdin '! dgsh-enumerate 2 | diff -w --label foo -' ./run_simple_test.sh $PSDIR diff0 'diff -w --label foo /dev/null /dev/null' ./run_simple_test.sh $PSDIR diff0-stdin1 '! dgsh-enumerate 1 | diff -w --label foo - /dev/null' ./run_simple_test.sh $PSDIR diff0-stdin2 '! dgsh-enumerate 1 | diff -w --label foo /dev/null -' ./run_simple_test.sh $PSDIR diff0-noin '! dgsh-enumerate 1 | diff -w --label foo /dev/null /dev/null' # Wrapped cmp ./run_simple_test.sh $PSDIR cmp2-same '{{ echo a ; echo a ; }} | cmp' ./run_simple_test.sh $PSDIR cmp2-diff '! {{ echo a ; echo b ; }} | cmp -s' ./run_simple_test.sh $PSDIR cmp1-same1 'echo -n | cmp - /dev/null' ./run_simple_test.sh $PSDIR cmp1-same2 'echo -n | cmp /dev/null -' ./run_simple_test.sh $PSDIR cmp0 'cmp /dev/null /dev/null' # Wrapped diff3 ./run_simple_test.sh $PSDIR diff3-3 '{{ echo a ; echo b ; echo c; }} | diff3' ./run_simple_test.sh $PSDIR diff3-2 '{{ echo a ; echo b ; }} | diff3 /dev/null' ./run_simple_test.sh $PSDIR diff3-2-stdin1 '{{ echo a ; echo b ; }} | diff3 - /dev/null' ./run_simple_test.sh $PSDIR diff3-2-stdin2 '{{ echo a ; echo b ; }} | diff3 /dev/null -' ./run_simple_test.sh $PSDIR diff3-1 '{{ echo a ; }} | diff3 /dev/null /dev/null' ./run_simple_test.sh $PSDIR diff3-0 'diff3 /dev/null /dev/null /dev/null' # grep # Single arg ./run_simple_test.sh $PSDIR grep-noargs 'echo hi | grep .' ./run_simple_test.sh $PSDIR grep-noargs-cat 'echo hi | grep . | cat' ./run_simple_test.sh $PSDIR grep-matching-lines 'echo hi | grep --matching-lines .' ./run_simple_test.sh $PSDIR grep-matching-lines-cat 'echo hi | grep --matching-lines . | cat' ./run_simple_test.sh $PSDIR grep-f 'echo hi | grep -f <(echo .)' ./run_simple_test.sh $PSDIR grep-f-cat 'echo hi | grep -f <(echo .) | cat' ./run_simple_test.sh $PSDIR grep-c 'echo hi | grep -c . ' ./run_simple_test.sh $PSDIR grep-c-cat 'echo hi | grep -c . | cat' ./run_simple_test.sh $PSDIR grep-l 'echo hi | grep -l . ' ./run_simple_test.sh $PSDIR grep-l-cat 'echo hi | grep -l . | cat' ./run_simple_test.sh $PSDIR grep-Lcap 'echo hi | grep -L hi' ./run_simple_test.sh $PSDIR grep-Lcap-nomatch 'echo hi | grep -L hello | cat' ./run_simple_test.sh $PSDIR grep-Lcap-cat 'echo hi | grep -L . | cat' ./run_simple_test.sh $PSDIR grep-Lcap-cat-nomatch 'echo hi | grep -L hello | cat' ./run_simple_test.sh $PSDIR grep-o 'echo hi | grep -o hi' ./run_simple_test.sh $PSDIR grep-o-cat 'echo hi | grep -o hi | cat' ./run_simple_test.sh $PSDIR grep-v 'echo hi | grep -v hello | cat' ./run_simple_test.sh $PSDIR grep-v-cat 'echo hi | grep -v hello | cat' # Double args ./run_simple_test.sh $PSDIR grep-matching-lines-c 'echo hi | grep --matching-lines -c . | cat' ./run_simple_test.sh $PSDIR grep-matching-lines-l 'echo hi | grep --matching-lines -l . | cat' ./run_simple_test.sh $PSDIR grep-matching-lines-Lcap 'echo hi | grep --matching-lines -L . | cat' ./run_simple_test.sh $PSDIR grep-matching-lines-Lcap-nomatch 'echo hi | grep --matching-lines -L hello | paste' ./run_simple_test.sh $PSDIR grep-c-matching-lines 'echo hi | grep -c --matching-lines . | cat' ./run_simple_test.sh $PSDIR grep-c-l 'echo hi | grep -c -l . | cat' ./run_simple_test.sh $PSDIR grep-c-Lcap 'echo hi | grep -c -L . | paste' ./run_simple_test.sh $PSDIR grep-c-Lcap-nomatch 'echo hi | grep -c -L hello | paste' ./run_simple_test.sh $PSDIR grep-l-matching-lines 'echo hi | grep -l --matching-lines . | cat' ./run_simple_test.sh $PSDIR grep-l-c 'echo hi | grep -l -c . | cat' ./run_simple_test.sh $PSDIR grep-l-Lcap 'echo hi | grep -l -L . | paste' ./run_simple_test.sh $PSDIR grep-l-Lcap-nomatch 'echo hi | grep -l -L hello | paste' ./run_simple_test.sh $PSDIR grep-Lcap-matching-lines 'echo hi | grep -L --matching-lines . | paste' ./run_simple_test.sh $PSDIR grep-Lcap-matching-lines-nomatch 'echo hi | grep -L --matching-lines hello | paste' ./run_simple_test.sh $PSDIR grep-Lcap-c 'echo hi | grep -L -c . | cat' ./run_simple_test.sh $PSDIR grep-Lcap-c-nomatch 'echo hi | grep -L -c hello | cat' ./run_simple_test.sh $PSDIR grep-Lcap-l 'echo hi | grep -L -l . | paste' ./run_simple_test.sh $PSDIR grep-Lcap-l-nomatch 'echo hi | grep -L -l hello | paste' # Triple args ./run_simple_test.sh $PSDIR grep-matching-lines-c-l 'echo hi | grep --matching-lines -c -l hi | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-c-Lcap 'echo hi | grep --matching-lines -c -L hi | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-c-Lcap-nomatch 'echo hi | grep --matching-lines -c -L hello | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-l-c 'echo hi | grep --matching-lines -l -c hi | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-l-Lcap 'echo hi | grep --matching-lines -l -L hi | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-l-Lcap-nomatch 'echo hi | grep --matching-lines -l -L hello | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-Lcap-c 'echo hi | grep --matching-lines -L -c hi | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-Lcap-c-nomatch 'echo hi | grep --matching-lines -L -c hello | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-Lcap-l 'echo hi | grep --matching-lines -L -l hi | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-Lcap-l-nomatch 'echo hi | grep --matching-lines -L -l hello | paste' ./run_simple_test.sh $PSDIR grep-c-matching-lines-l 'echo hi | grep -c --matching-lines -l hi | paste' ./run_simple_test.sh $PSDIR grep-c-matching-lines-Lcap 'echo hi | grep -c --matching-lines -L hi | paste' ./run_simple_test.sh $PSDIR grep-c-matching-lines-Lcap-nomatch 'echo hi | grep -c --matching-lines -L hello | paste' ./run_simple_test.sh $PSDIR grep-c-l-matching-lines 'echo hi | grep -c -l --matching-lines hi | paste' ./run_simple_test.sh $PSDIR grep-c-l-Lcap 'echo hi | grep -c -l -L hi | paste' ./run_simple_test.sh $PSDIR grep-c-l-Lcap-nomatch 'echo hi | grep -c -l -L hello | paste' ./run_simple_test.sh $PSDIR grep-c-Lcap-matching-lines 'echo hi | grep -c -L --matching-lines hi | paste' ./run_simple_test.sh $PSDIR grep-c-Lcap-matching-lines-nomatch 'echo hi | grep -c -L --matching-lines hello | paste' ./run_simple_test.sh $PSDIR grep-c-Lcap-l 'echo hi | grep -c -L -l hi | paste' ./run_simple_test.sh $PSDIR grep-c-Lcap-l-nomatch 'echo hi | grep -c -L -l hello | paste' ./run_simple_test.sh $PSDIR grep-l-matching-lines-c 'echo hi | grep -l --matching-lines -c hi | paste' ./run_simple_test.sh $PSDIR grep-l-matching-lines-Lcap 'echo hi | grep -l --matching-lines -L hi | paste' ./run_simple_test.sh $PSDIR grep-l-matching-lines-Lcap-nomatch 'echo hi | grep -l --matching-lines -L hello | paste' ./run_simple_test.sh $PSDIR grep-l-c-matching-lines 'echo hi | grep -l -c --matching-lines hi | paste' ./run_simple_test.sh $PSDIR grep-l-c-Lcap 'echo hi | grep -l -c -L hi | paste' ./run_simple_test.sh $PSDIR grep-l-c-Lcap-nomatch 'echo hi | grep -l -c -L hello | paste' ./run_simple_test.sh $PSDIR grep-l-Lcap-matching-lines 'echo hi | grep -l -L --matching-lines hi | paste' ./run_simple_test.sh $PSDIR grep-l-Lcap-matching-lines-nomatch 'echo hi | grep -l -L --matching-lines hello | paste' ./run_simple_test.sh $PSDIR grep-l-Lcap-c 'echo hi | grep -l -L -c hi | paste' ./run_simple_test.sh $PSDIR grep-l-Lcap-c-nomatch 'echo hi | grep -l -L -c hello | paste' ./run_simple_test.sh $PSDIR grep-Lcap-matching-lines-c 'echo hi | grep -L --matching-lines -c hi | paste' ./run_simple_test.sh $PSDIR grep-Lcap-matching-lines-c-nomatch 'echo hi | grep -L --matching-lines -c hello | paste' ./run_simple_test.sh $PSDIR grep-Lcap-matching-lines-l 'echo hi | grep -L --matching-lines -l hi | paste' ./run_simple_test.sh $PSDIR grep-Lcap-matching-lines-l-nomatch 'echo hi | grep -L --matching-lines -l hello | paste' ./run_simple_test.sh $PSDIR grep-Lcap-c-matching-lines 'echo hi | grep -L -c --matching-lines hi | paste' ./run_simple_test.sh $PSDIR grep-Lcap-c-matching-lines-nomatch 'echo hi | grep -L -c --matching-lines hello | paste' ./run_simple_test.sh $PSDIR grep-Lcap-c-l 'echo hi | grep -L -c -l hi | paste' ./run_simple_test.sh $PSDIR grep-Lcap-c-l-nomatch 'echo hi | grep -L -c -l hello | paste' ./run_simple_test.sh $PSDIR grep-Lcap-l-matching-lines 'echo hi | grep -L -l --matching-lines hi | paste' ./run_simple_test.sh $PSDIR grep-Lcap-l-matching-lines-nomatch 'echo hi | grep -L -l --matching-lines hello | paste' ./run_simple_test.sh $PSDIR grep-Lcap-l-c 'echo hi | grep -L -l -c hi | paste' ./run_simple_test.sh $PSDIR grep-Lcap-l-c-nomatch 'echo hi | grep -L -l -c hello | paste' # Quadraple args ./run_simple_test.sh $PSDIR grep-matching-lines-c-l-Lcap 'echo hi | grep --matching-lines -c -l -L hi | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-c-Lcap-l 'echo hi | grep --matching-lines -c -L -l hi | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-l-c-Lcap 'echo hi | grep --matching-lines -l -c -L hi | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-l-Lcap-c 'echo hi | grep --matching-lines -l -L -c hi | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-Lcap-c-l 'echo hi | grep --matching-lines -L -c -l hi | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-Lcap-l-c 'echo hi | grep --matching-lines -L -l -c hi | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-c-l-Lcap-nomatch 'echo hi | grep --matching-lines -c -l -L hello | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-c-Lcap-l-nomatch 'echo hi | grep --matching-lines -c -L -l hello | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-l-c-Lcap-nomatch 'echo hi | grep --matching-lines -l -c -L hello | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-l-Lcap-c-nomatch 'echo hi | grep --matching-lines -l -L -c hello | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-Lcap-c-l-nomatch 'echo hi | grep --matching-lines -L -c -l hello | paste' ./run_simple_test.sh $PSDIR grep-matching-lines-Lcap-l-c-nomatch 'echo hi | grep --matching-lines -L -l -c hello | paste' ./run_simple_test.sh $PSDIR grep-c-matching-lines-l-Lcap 'echo hi | grep -c --matching-lines -l -L hi | paste' ./run_simple_test.sh $PSDIR grep-c-matching-lines-Lcap-l 'echo hi | grep -c --matching-lines -L -l hi | paste' ./run_simple_test.sh $PSDIR grep-c-l-matching-lines-Lcap 'echo hi | grep -c -l --matching-lines -L hi | paste' ./run_simple_test.sh $PSDIR grep-c-l-Lcap-matching-lines 'echo hi | grep -c -l -L --matching-lines hi | paste' ./run_simple_test.sh $PSDIR grep-c-Lcap-matching-lines-l 'echo hi | grep -c -L --matching-lines -l hi | paste' ./run_simple_test.sh $PSDIR grep-c-Lcap-l-matching-lines 'echo hi | grep -c -L -l --matching-lines hi | paste' ./run_simple_test.sh $PSDIR grep-c-matching-lines-l-Lcap-nomatch 'echo hi | grep -c --matching-lines -l -L hello | paste' ./run_simple_test.sh $PSDIR grep-c-matching-lines-Lcap-l-nomatch 'echo hi | grep -c --matching-lines -L -l hello | paste' ./run_simple_test.sh $PSDIR grep-c-l-matching-lines-Lcap-nomatch 'echo hi | grep -c -l --matching-lines -L hello | paste' ./run_simple_test.sh $PSDIR grep-c-l-Lcap-matching-lines-nomatch 'echo hi | grep -c -l -L --matching-lines hello | paste' ./run_simple_test.sh $PSDIR grep-c-Lcap-matching-lines-l-nomatch 'echo hi | grep -c -L --matching-lines -l hello | paste' ./run_simple_test.sh $PSDIR grep-c-Lcap-l-matching-lines-nomatch 'echo hi | grep -c -L -l --matching-lines hello | paste' ./run_simple_test.sh $PSDIR grep-l-matching-lines-c-Lcap 'echo hi | grep -l --matching-lines -c -L hi | paste' ./run_simple_test.sh $PSDIR grep-l-matching-lines-Lcap-c 'echo hi | grep -l --matching-lines -L -c hi | paste' ./run_simple_test.sh $PSDIR grep-l-c-matching-lines-Lcap 'echo hi | grep -l -c --matching-lines -L hi | paste' ./run_simple_test.sh $PSDIR grep-l-c-Lcap-matching-lines 'echo hi | grep -l -c -L --matching-lines hi | paste' ./run_simple_test.sh $PSDIR grep-l-Lcap-matching-lines-c 'echo hi | grep -l -L --matching-lines -c hi | paste' ./run_simple_test.sh $PSDIR grep-l-Lcap-c-matching-lines 'echo hi | grep -l -L -c --matching-lines hi | paste' ./run_simple_test.sh $PSDIR grep-l-matching-lines-c-Lcap-nomatch 'echo hi | grep -l --matching-lines -c -L hello | paste' ./run_simple_test.sh $PSDIR grep-l-matching-lines-Lcap-c-nomatch 'echo hi | grep -l --matching-lines -L -c hello | paste' ./run_simple_test.sh $PSDIR grep-l-c-matching-lines-Lcap-nomatch 'echo hi | grep -l -c --matching-lines -L hello | paste' ./run_simple_test.sh $PSDIR grep-l-c-Lcap-matching-lines-nomatch 'echo hi | grep -l -c -L --matching-lines hello | paste' ./run_simple_test.sh $PSDIR grep-l-Lcap-matching-lines-c-nomatch 'echo hi | grep -l -L --matching-lines -c hello | paste' ./run_simple_test.sh $PSDIR grep-l-Lcap-c-matching-lines-nomatch 'echo hi | grep -l -L -c --matching-lines hello | paste' ./run_simple_test.sh $PSDIR grep-Lcap-matching-lines-c-l 'echo hi | grep -L --matching-lines -c -l hi | paste' ./run_simple_test.sh $PSDIR grep-Lcap-matching-lines-l-c 'echo hi | grep -L --matching-lines -l -c hi | paste' ./run_simple_test.sh $PSDIR grep-Lcap-c-matching-lines-l 'echo hi | grep -L -c --matching-lines -l hi | paste' ./run_simple_test.sh $PSDIR grep-Lcap-c-l-matching-lines 'echo hi | grep -L -c -l --matching-lines hi | paste' ./run_simple_test.sh $PSDIR grep-Lcap-l-matching-lines-c 'echo hi | grep -L -l --matching-lines -c hi | paste' ./run_simple_test.sh $PSDIR grep-Lcap-l-c-matching-lines 'echo hi | grep -L -l -c --matching-lines hi | paste' ./run_simple_test.sh $PSDIR grep-Lcap-matching-lines-c-l-nomatch 'echo hi | grep -L --matching-lines -c -l hello | paste' ./run_simple_test.sh $PSDIR grep-Lcap-matching-lines-l-c-nomatch 'echo hi | grep -L --matching-lines -l -c hello | paste' ./run_simple_test.sh $PSDIR grep-Lcap-c-matching-lines-l-nomatch 'echo hi | grep -L -c --matching-lines -l hello | paste' ./run_simple_test.sh $PSDIR grep-Lcap-c-l-matching-lines-nomatch 'echo hi | grep -L -c -l --matching-lines hello | paste' ./run_simple_test.sh $PSDIR grep-Lcap-l-matching-lines-c-nomatch 'echo hi | grep -L -l --matching-lines -c hello | paste' ./run_simple_test.sh $PSDIR grep-Lcap-l-c-matching-lines-nomatch 'echo hi | grep -L -l -c --matching-lines hello | paste' # The remaining (double args) combinations don't work # does not work: --matching-lines #echo hi | grep --matching-lines -v hi | paste #echo hi | grep --matching-lines -v hello | paste # does not work: -o #echo hi | grep --matching-lines -o hi | paste # does not work: -o #echo hi | grep -c -o hi | cat #echo hi | grep -c -v hi | paste # does not work: -c #echo hi | grep -c -v hello | paste # does not work: -o #echo hi | grep -l -o hi | cat # does not work: -l #echo hi | grep -l -v hi | cat #echo hi | grep -l -v hello | cat # does not work: -o #echo hi | grep -L -o hi | cat # does not work: -L #echo hi | grep -L -v hi | paste #echo hi | grep -L -v hello | paste # does not work: --matching-lines #echo hi | grep -o --matching-lines hi | paste # does not work: -o #echo hi | grep -o -c hi | cat # does not work: -o #echo hi | grep -o -l hi | paste # does not work: -o #echo hi | grep -o -L hi | cat # does not make sense: -o -v #echo hi | grep -o -v hi | paste # does not work: -v #echo hi | grep -o -v hello | paste # does not work: --matching-lines #echo hi | grep -v --matching-lines hi | paste #echo hi | grep -v --matching-lines hello | paste #echo hi | grep -v -c hi | paste # does not work -c #echo hi | grep -v -c hello | paste # does not work: -l #echo hi | grep -v -l hi | paste #echo hi | grep -v -l hello | paste # does not work: -L #echo hi | grep -v -L hi | paste #echo hi | grep -v -L hello | paste # does not make sense: -v -o #echo hi | grep -v -o hi | paste # does not make sense: -v -o #echo hi | grep -v -o hello | paste ================================================ FILE: unix-tools/run_simple_test.sh ================================================ #!/usr/bin/env bash # Color GR="\033[0;32m" # Green R="\033[0;31m" # Red B="\033[0;34m" # Blue EC="\033[0m" # End color S=${GR}successful${EC} F=${R}failed${EC} DGSH='bash/bash --dgsh' PSDIR=$1 FILENAME=$2 SCRIPT=$3 export DGSHPATH="$(pwd)/../build/libexec/dgsh" PATH="$(pwd)/../build/bin:$PATH" $DGSH -c "$SCRIPT > $PSDIR/$FILENAME.outb" \ 2>$PSDIR/$FILENAME.errb \ && diff $PSDIR/$FILENAME.outb $PSDIR/$FILENAME.success \ && printf "$FILENAME $S\n" \ || (printf "$FILENAME $F\n" \ && exit 1) ================================================ FILE: unix-tools/run_test.sh ================================================ #!/bin/sh PSDIR=$1 FSCRIPT=$2 BSCRIPT=$(basename $2 .sh) INPUT_TYPE=$3 INPUT1=$4 INPUT2=$5 INPUT3=$6 TOP=$(cd $(dirname $0)/.. ; pwd) DGSH="$TOP/build/bin/dgsh" PATH="$TOP/build/bin:$PATH" export DGSHPATH="$TOP/build/libexec/dgsh" GR="\033[0;32m" # Green R="\033[0;31m" # Red B="\033[0;34m" # Blue EC="\033[0m" # End color S=${GR}successful${EC} F=${R}failed${EC} # Skip test if required custom commands are missing ( iscommand=0 for arg in "$@"; do if [ "$arg" = "--" ]; then iscommand=1 continue fi if [ $iscommand -eq 0 ]; then continue fi if ! which $arg >/dev/null; then exit 1 fi done exit 0 ) if [ $? -ne 0 ]; then echo "Skip test $BSCRIPT.sh" exit 0 fi if [ "$INPUT_TYPE" = pipe ]; then if $DGSH $FSCRIPT <$INPUT1 >$PSDIR/$BSCRIPT.outb 2>$PSDIR/$BSCRIPT.err ; then printf "$BSCRIPT.sh $S\n" else printf "$BSCRIPT.sh $F\n" cat $PSDIR/$BSCRIPT.err exit 1 fi else if $DGSH $FSCRIPT $INPUT1 $INPUT2 $INPUT3 >$PSDIR/$BSCRIPT.outb \ 2>$PSDIR/$BSCRIPT.err ; then printf "$BSCRIPT.sh $S\n" else printf "$BSCRIPT.sh $F\n" cat $PSDIR/$BSCRIPT.err exit 1 fi fi ================================================ FILE: unix-tools/tee.sh ================================================ #!/usr/bin/env bash #!dgsh # # Implementation of POSIX tee through dgsh-tee # usage() { echo 'Usage: tee [-ai] [file ...]' 1>&2 exit 2 } declare -a opts # Process flags while getopts 'ai' o; do case "$o" in a) opts=(-a) ;; i) ;; *) usage ;; esac done shift $((OPTIND-1)) # Process file arguments for i; do opts+=('-o' "$i") shift done exec dgsh-tee "${opts[@]}" ================================================ FILE: unix-tools/test-compat.sh ================================================ #!/bin/sh # # Test the dgsh-compatibility checking echo 'Testing dgsh compatibility checker' check() { local title="$1" local prog="$2" local is_dgsh="$3" echo -n "$title: " case $is_dgsh in F) if ./dgsh-compat "$prog" ; then echo FAIL exit 1 else echo PASS fi ;; T) if ./dgsh-compat "$prog" ; then echo PASS else echo FAIL exit 1 fi ;; esac } check 'Non-dgsh binary file' /bin/ls F check 'Dgsh binary file' ../build/libexec/dgsh/dgsh-tee T check 'Dgsh wrap script' ../build/libexec/dgsh/date T check 'Dgsh magic script dgsh-parallel' ../build/libexec/dgsh/dgsh-parallel T check 'Dgsh magic script tee' ../build/libexec/dgsh/tee T check 'Dgsh magic script cat' ../build/libexec/dgsh/cat T check 'Non-dgsh script' ./test-compat.sh F exit 0 ================================================ FILE: unix-tools/wrapped-commands-posix ================================================ # Commands specified by POSIX d alias d ar d basename dm bg d c99 d cal c cat dm cd d cflow dm chgrp dm chmod dm chown D cksum c cmp c comm d command f compress # TODO custom dm cp f crontab # TODO custom f csplit # TODO custom f ctags # TODO custom d date d df c diff d dirname d du d echo dm env d expr dm false dm fg d find d getopts c grep d ipcrm d jobs c join dm kill dm link dm ln d ls d make d man dm mesg dm mkdir dm mkfifo dm mv dm newgrp dm nice dm nohup c paste d printf d ps d pwd m read dm renice dm rm D rmdel # SCCS dm rmdir dm sleep c sort f strings # TODO custom dm strip c tee dm test f time # Depends on the command dm touch dm true f tsort d tty d type d ulimit d umask dm unalias d uname f uncompress # TODO custom Dm unget dm unlink D val # SCCS dm wait d what # SCCS d who m write dm yacc ================================================ FILE: unix-tools/wrapped-commands-tests ================================================ # Commands required for running the negotiation and performance tests d git I sfcat ================================================ FILE: web/format-eg.sh ================================================ #!/bin/sh # # Format the examples into Bootstrap HTML # SECTION='compress-compare commit-stats code-metrics duplicate-files spell-highlight word-properties web-log-report text-properties static-functions map-hierarchy committer-plot parallel-word-count author-compare ft2d NMRPipe fft-block8 reorder-columns dir' for NAME in ${SECTION} do TITLE="`sed -n 's/^# SYNOPSIS //p;s/^# TITLE //p' example/$NAME.sh | head -1`" if [ "$1" = '-c' ] then echo "<li> <a href='#$NAME'>$TITLE</a></li>" else cat <<EOF <section id="$NAME"> <!-- {{{2 --> <h2>$TITLE</h2> </section> <img src="$NAME-pretty.png" class="img-polaroid" alt="$TITLE" /> <!-- Extracted description --> <p> `sed -n '/^# DESCRIPTION/,/^#$/{;/^# DESCRIPTION/d;s/^# //p;}' example/$NAME.sh` </p> <!-- Extracted code --> <pre class="prettyprint lang-bash"> `sed '1a\ 2,/^$/d' example/$NAME.sh | sed 's/&/\&/g;s/</\</g;s/>/\>/g'` </pre> EOF fi done ================================================ FILE: web/format-syntax.sh ================================================ #!/bin/sh # # Extract the dgsh syntax from the manual page and format it into HTML # sed -n '/^<dgsh_block/,/^\.fi/ { s/&/\&/g s/</\</g s/>/\>/g /^\.fi/d p }' core-tools/src/dgsh.1 ================================================ FILE: web/index.html ================================================ <!DOCTYPE html> <html> <head> <meta http-equiv="Content-type" content="text/html; charset=utf-8" /> <title>dgsh — directed graph shell
dgsh dds

dgsh — directed graph shell

The directed graph shell, dgsh (pronounced /dæɡʃ/ — dagsh), provides an expressive way to construct sophisticated and efficient big data set and stream processing pipelines using existing Unix tools as well as custom-built components. It is a Unix-style shell (based on bash) allowing the specification of pipelines with non-linear non-uniform operations. These form a directed acyclic process graph, which is typically executed by multiple processor cores, thus increasing the operation's processing throughput.

If you want to get a feeling on how dgsh works in practice, skip right down to the examples section.

For a more formal introduction to dgsh or to cite it in your work, see:
Diomidis Spinellis and Marios Fragkoulis. Extending Unix Pipelines to DAGs. IEEE Transactions on Computers, 2017. doi: 10.1109/TC.2017.2695447

Inter-process communication

Dgsh provides two new ways for expressing inter-process communication.

Multipipes
are expressed as usual Unix pipelines, but can connect commands with more than one output or input channel. As an example, the comm command supplied with dgsh expects two input channels and produces on its output three output channels: the lines appearing only in first (sorted) channel, the lines appearing only in the second channel, and the lines appearing in both. Connecting the output of the comm command to the cat command supplied with dgsh will make the three outputs appear in sequence, while connecting it to the paste command supplied with dgsh will make the output appear in its customary format.
Multipipe blocks {{ ... }}
a) send (multiple) input streams received on their input side to the asynchronously-running processes that reside within the block, and, b) pass the output produced by the processes within the block as (multiple) streams on their output side. Multipipe blocks typically receive input from more than one channel and produce more than one output channel. For example, a multipipe block that runs md5sum and wc -c receives two inputs and produces two outputs: the MD5 hash of its input and the input's size. Data to multipipe blocks are typically provided with a dgsh-aware version of tee and collected by dgsh-aware versions of programs such as cat and paste.
Stored values
offer a convenient way for communicating computed values between arbitrary processes on the graph. They allow the storage of a data stream's last record into a named buffer. This record can be later retrieved asynchronously by one or more readers. Data in a stored value can be piped into a process or out of it, or it can be read using the shell's command output substitution syntax. Stored values are implemented internally through Unix-domain sockets, a background-running store program, dgsh-writeval, and a reader program, dgsh-readval. The behavior of a stored value's IO can be modified by adding flags to dgsh-writeval and dgsh-readval.

Syntax

A dgsh script follows the syntax of a bash(1) shell script with the addition of multipipe blocks. A multipipe block contains one or more dgsh simple commands, other multipipe blocks, or pipelines of the previous two types of commands. The commands in a multipipe block are executed asynchronously (in parallel, in the background). Data may be redirected or piped into and out of a multipipe block. With multipipe blocks dgsh scripts form directed acyclic process graphs. It follows from the above description that multipipe blocks can be recursively composed.

As a simple example consider running the following command directly within dgsh 

{{ echo hello & echo world & }} | paste

or by invoking dgsh with the command as an argument. 

dgsh -c '{{ echo hello & echo world & }} | paste'

The command will run paste with input from the two echo processes to output hello world. This is equivalent to running the following bash command, but with the flow of data appearing in the natural left-to-right order. 

paste <(echo hello) <(echo world)

In the following larger example, which compares the performance of different compression utilities, the script's standard input is distributed to three compression utilities (xz, bzip2, and gzip), to assess their performance, and also to file and wc to report the input data's type and size. The printf commands label the data of each processing type. All eight commands pass their output to the cat command, which gathers their outputs in order. 

tee |
{{
	printf 'File type:\t'
	file -

	printf 'Original size:\t'
	wc -c

	printf 'xz:\t\t'
	xz -c | wc -c

	printf 'bzip2:\t\t'
	bzip2 -c | wc -c

	printf 'gzip:\t\t'
	gzip -c | wc -c
}} |
cat

Formally, dgsh extends the syntax of the (modified) Unix Bourne-shell when bash provided with the --dgsh argument as follows.

Adapted tools

A number of Unix tools have been adapted to support multiple inputs and outputs to match their natural capabilities. This echoes a similar adaptation that was performed in the early 1970s when Unix and the shell got pipes and the pipeline syntax. Many programs that worked with files were adjusted to work as filters. The number of input and output channels of dgsh-compatible commands are as follows, based on the supplied command-line arguments.

Tool Inputs Outputs Notes
cat (dgsh-tee) 0—N 0—M No options are supported
cmp 0—2 0—1
comm 0—2 0—3 Output streams in order: lines only in first file, lines only in second one, and lines in both files
cut 0—1 1—N With --multistream output each range into a different stream
diff 0—N 1 Typically two inputs. Compare an arbitrary number of input streams with the --from-file or --to-file options
diff3 0—3 1
grep 0—2 0—4 Available output streams (via arguments): matching files, non-matching files, matching lines, and non-matching lines
join 0—2 1
paste 0—N 1 Paste N input streams
perm 1—N 1—N Rearrange the order of N input streams
sort 0—N 0—1 With the -m option, merge sort N input streams
tee (dgsh-tee) 0—N 0—M Only the -a option is supported
dgsh-readval 0 1 Read a value from a socket
dgsh-wrap 0—N 0—1 Wrap non-dgsh commands and negotiate on their behalf
dgsh-writeval 1 0 Write a value to a socket

In addition, POSIX user commands that receive no input or only generate no output, when executed in a dgsh context are wrapped to specify the corresponding input or output capability. For example, an echo command in a multipipe block will appear to receive no input, but will provide one output stream. By default dgsh automatically wraps all other commands as filters.

Input-only
Output-only
No input and output

Finally, note that any dgsh script will accept and generate the number of inputs and outputs associated with the commands or multipipe blocks at its two endpoints.

Downloading and installation

The dgsh suite has been tested under Debian and Ubuntu Linux, FreeBSD, and Mac OS X. A Cygwin port is underway.

An installation of GraphViz will allow you to visualize the dgsh graphs that you specify in your programs.

Debian and Ubuntu GNU/Linux

Prerequisites

To compile and run dgsh you will need to have the following commands installed on your system: 


git xz-utils gettext
To test dgsh you will need to have the following commands installed in your system: 

curl bzip2

Installation steps

Go through the following steps.

  1. Recursively clone the project's source code through its GitHub page. 
    git clone --recursive https://github.com/dspinellis/dgsh.git
  2. Configure bash and the Unix tools adapted for dgsh. 
    make config
  3. Compile all programs. 
    make
  4. Install. 
    sudo make install

By default, the program and its documentation are installed under /usr/local. You can modify this by setting the PREFIX variable in the `config` step, for example: 

make PREFIX=$HOME config
make
make install

Testing

Issue the following command. 

make test

FreeBSD

Prerequisites

To compile and run dgsh you will need to have the following packages installed in your system: 

devel/automake
devel/bison
devel/check
devel/git
devel/gmake
devel/gperf
misc/help2man
print/texinfo
shells/bash
To test dgsh you will need to have the following ports installed on your system: 
archivers/bzip2
ftp/curl

Installation steps

Go through the following steps.

  1. Recursively clone the project's source code through its GitHub page. 
    git clone --recursive https://github.com/dspinellis/dgsh.git
  2. Configure bash and the Unix tools adapted for dgsh. 
    gmake config
  3. Compile all programs. 
    gmake
  4. Install. 
    sudo gmake install

By default, the program and its documentation are installed under /usr/local. You can modify this by setting the PREFIX variable in the `config` step, for example: 

gmake PREFIX=$HOME config
gmake
gmake install

Testing

Issue the following command. 

gmake test

Reference documentation

These are the manual pages for dgsh, the associated helper programs and the API in formats suitable for browsing and printing. The commands are listed in the order of usefulness in everyday scenarios.

dgsh
directed graph shell HTML, PDF
dgsh-tee
buffer and copy or scatter standard input to one or more sinks HTML, PDF
dgsh-wrap
allow any filter program to participate in an dgsh pipeline HTML, PDF
dgsh-writeval
write values to a data store HTML, PDF
dgsh-readval
data store client HTML, PDF
dgsh-monitor
monitor data on a pipe HTML, PDF
dgsh-parallel
create a semi-homogeneous dgsh parallel processing block HTML, PDF
perm
permute inputs to outputs HTML, PDF
dgsh-httpval
provide data store values through HTTP HTML, PDF
dgsh-merge-sum
merge key value pairs, summing the values HTML, PDF
dgsh-conc
input or output pipe concentrator for dgsh negotiation (used internally) HTML, PDF
dgsh-enumerate
enumerate an arbitrary number of output channels (demonstration and debugging tool) HTML, PDF
dgsh_negotiate
API for dgsh-compatible programs to specify and obtain dgsh I/O file descriptors HTML, PDF

Examples

Creative Commons License Unless otherwise expressly stated, all original material on this page created by Diomidis Spinellis is licensed under a Creative Commons Attribution-Share Alike 3.0 Greece License.