[
{
"path": ".gitignore",
"content": "tmp_libcurl\ncurl.zip\nthird-party\ntmp_url"
},
{
"path": "README.md",
"content": "Auto download & compile libcurl\n-----------\nThis batch script will automatically download the latest libcurl source code and build it using Visual Studio compiler.\n\nSupported Visual Studio are:\n* Visual C++ 6 (require Windows Server 2003 Platform SDK released in February 2003)\n* Visual Studio 2005\n* Visual Studio 2008\n* Visual Studio 2010\n* Visual Studio 2012\n* Visual Studio 2013 [](https://ci.appveyor.com/project/blackrosezy/build-libcurl-windows)\n* Visual Studio 2015 [](https://ci.appveyor.com/project/blackrosezy/build-libcurl-windows-unln0)\n\n\n*Note-1*: All versions of **Visual Studio Express are unsupported**.\n\n*Note-2*: This script is using third-party open source software\n* `bin/7-zip` http://www.7-zip.org/download.html\n* `bin/unxutils` http://sourceforge.net/projects/unxutils/\n* `bin/xidel` http://sourceforge.net/projects/videlibri/files/Xidel/\n\nUsage :\n\n $ build.bat\n\nTo build using /MT rather than /MD:\n\n $ build.bat -static \n\nOutput :\n\n```\nthird-party\n└───libcurl\n ├───include\n │ └───curl\n │ curl.h\n │ curlbuild.h\n │ curlrules.h\n │ curlver.h\n │ easy.h\n │ mprintf.h\n │ multi.h\n │ stdcheaders.h\n │ typecheck-gcc.h\n │\n └───lib\n ├───dll-debug-x64\n │ libcurl_debug.dll\n │ libcurl_debug.lib\n │ libcurl_debug.pdb\n │\n ├───dll-debug-x86\n │ libcurl_debug.dll\n │ libcurl_debug.lib\n │ libcurl_debug.pdb\n │\n ├───dll-release-x64\n │ libcurl.dll\n │ libcurl.lib\n │ libcurl.pdb\n │\n ├───dll-release-x86\n │ libcurl.dll\n │ libcurl.lib\n │ libcurl.pdb\n │\n ├───static-debug-x64\n │ libcurl_a_debug.lib\n │\n ├───static-debug-x86\n │ libcurl_a_debug.lib\n │\n ├───static-release-x64\n │ libcurl_a.lib\n │\n └───static-release-x86\n libcurl_a.lib\n```\n\n## FAQ\nIf you get message something like below, please re-run build.bat again.\n\n **** Retrieving:http://curl.haxx.se/download.html ****\n Downloading latest curl...\n http://curl.haxx.seAn unhandled exception occurred at $004C7D39 :: Bad port number.\n\nLicense (build.bat)\n-----------\n\n The MIT License (MIT)\n \n Copyright (c) 2014 Mohd Rozi\n \n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n \n The above copyright notice and this permission notice shall be included in\n all copies or substantial portions of the Software.\n \n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\n THE SOFTWARE.\n"
},
{
"path": "bin/7-zip/license.txt",
"content": " 7-Zip Command line version\n ~~~~~~~~~~~~~~~~~~~~~~~~~~\n License for use and distribution\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n 7-Zip Copyright (C) 1999-2010 Igor Pavlov.\n\n 7za.exe is distributed under the GNU LGPL license\n\n Notes: \n You can use 7-Zip on any computer, including a computer in a commercial \n organization. You don't need to register or pay for 7-Zip.\n\n\n GNU LGPL information\n --------------------\n\n This library is free software; you can redistribute it and/or\n modify it under the terms of the GNU Lesser General Public\n License as published by the Free Software Foundation; either\n version 2.1 of the License, or (at your option) any later version.\n\n This library is distributed in the hope that it will be useful,\n but WITHOUT ANY WARRANTY; without even the implied warranty of\n MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU\n Lesser General Public License for more details.\n\n You can receive a copy of the GNU Lesser General Public License from \n http://www.gnu.org/\n"
},
{
"path": "bin/unxutils/StdDisclaimer.html",
"content": "\n\n\n\nDisclaimer\n\n\n\n\n
Disclaimer
\n\n
THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESSED OR IMPLIED\nWARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND\nFITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE CONTRIBUTORS\nBE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\nDAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS\nOF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) \nHOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR\nTORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\nSOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
\n\n\n"
},
{
"path": "bin/unxutils/UnxUtilsDist.html",
"content": "\n\n\n\n\n\nNative Win32 ports of some GNU utilities\n\n\n\n\n
\n
\n
\n
\n
\n
GNU utilities for Win32
\n
\n
\n
\n This are some ports of common GNU utilities to native\n Win32. In this context, native means the executables do only depend on the Microsoft\n C-runtime (msvcrt.dll) and not an emulation layer like that provided by Cygwin tools. \n
I have started an Open source project at http://unxutils.sourceforge.net. There is a\n CVS repository of all sources, even if the changes to the\n original GNU code are trivial. The repository can be accessed via anonymous CVS with the\n command \n cvs\n -d:pserver:anonymous@cvs.UnxUtils.sourceforge.net:/cvsroot/unxutils login
for (de)compressing .Z files; this is actually a program\n called ncompress and was found on one of the Linux mirrors
\n
\n
\n
cp.exe
\n
works only on NT, does real hardlinks on NTFS
\n
\n
\n
csplit.exe
\n
\n
\n
\n
cut.exe
\n
\n
\n
\n
date.exe
\n
\n
\n
\n
dd.exe
\n
\n
\n
\n
df.exe
\n
dummy
\n
\n
\n
diff.exe
\n
\n
\n
\n
diff3.exe
\n
\n
\n
\n
dirname.exe
\n
\n
\n
\n
du.exe
\n
\n
\n
\n
echo.exe
\n
\n
\n
\n
egrep.exe
\n
\n
\n
\n
env.exe
\n
dummy
\n
\n
\n
expand.exe
\n
\n
\n
\n
expr.exe
\n
\n
\n
\n
factor.exe
\n
\n
\n
\n
fgrep.exe
\n
\n
\n
\n
find.exe
\n
\n
Example of command line under Cmd/Command shell: \n find -name *.txt -exec cat {} ;
\n
\n
\n
\n
\n
flex.exe
\n
\n
\n
\n
fmt.exe
\n
\n
\n
\n
fold.exe
\n
\n
\n
\n
gawk.exe
\n
input files are opened in text mode
\n
\n
\n
make.exe
\n
From v3.77 upwards, make searches for a sh.exe on the path. If\n it does not find one, it switches to win32 make mode that is it uses intermediate batch\n files for command processing. \n This is fine until your makefile tries to execute something like mkdir, which will invoke\n the internal mkdir from cmd.exe or command.com. As the results may not be to your liking,\n you may prefer to use the sh.exe provide here.
\n
\n
\n
grep.exe
\n
\n
\n
\n
gsar.exe
\n
\n
\n
\n
gunzip.exe
\n
\n
\n
\n
gzip.exe
\n
\n
\n
\n
head.exe
\n
\n
\n
\n
id.exe
\n
dummy
\n
\n
\n
install.exe
\n
\n
\n
\n
join.exe
\n
\n
\n
\n
less.exe
\n
\n
\n
\n
ln.exe
\n
works only on NT, does real hardlinks on NTFS
\n
\n
\n
logname.exe
\n
\n
\n
\n
ls.exe
\n
\n
\n
\n
m4.exe
\n
\n
\n
\n
md5sum.exe
\n
\n
\n
\n
mkdir.exe
\n
\n
\n
\n
mkfifo.exe
\n
dummy
\n
\n
\n
mknod.exe
\n
dummy
\n
\n
\n
mv.exe
\n
\n
\n
\n
mvdir.exe
\n
\n
\n
\n
nl.exe
\n
\n
\n
\n
od.exe
\n
\n
\n
\n
paste.exe
\n
\n
\n
\n
patch.exe
\n
\n
\n
\n
pathchk.exe
\n
\n
\n
\n
pr.exe
\n
\n
\n
\n
printenv.exe
\n
\n
\n
\n
printf.exe
\n
\n
\n
\n
ptx.exe
\n
\n
\n
\n
recode.exe
\n
\n
\n
\n
rm.exe
\n
\n
\n
\n
rman.exe
\n
PolyglotMan by Thomas A. Phelps ( source)
\n
\n
\n
rmdir.exe
\n
\n
\n
\n
sdiff.exe
\n
\n
\n
\n
sed.exe
\n
\n
\n
\n
seq.exe
\n
\n
\n
\n
sleep.exe
\n
\n
\n
\n
sort.exe
\n
\n
\n
\n
sh.exe
\n
renamed zsh - this is no replacement for the Cygwin bash
\n
\n
\n
shar.exe
\n
only works with -T (text) option
\n
\n
\n
split.exe
\n
\n
\n
\n
stego.exe
\n
stego -E encode.me -T words.txt \n Encodes file encode.me as gibberish text using words from the words.txt file to\n stdout. If the -T option is omitted, stego looks for a file called "words" in\n the current directory. Of course, the -D option will decode the input file; remember to\n redistribute words.txt together with the encoded file. \n The purpose of this program is to disguise binary files as a kind of text file or to drive\n your coworkers mad.
\n
\n
\n
su.exe
\n
dummy
\n
\n
\n
sum.exe
\n
\n
\n
\n
sync.exe
\n
\n
\n
\n
tac.exe
\n
\n
\n
\n
tail.exe
\n
\n
\n
\n
tar.exe
\n
\n
only forward slashes are accepted
\n
the -z option (compression) does not work
\n
no remote archives
\n
works only on NT, does real hardlinks on NTFS
\n
\n
\n
\n
\n
tee.exe
\n
\n
\n
\n
test.exe
\n
\n
\n
\n
touch.exe
\n
\n
\n
\n
tr.exe
\n
\n
\n
\n
uname.exe
\n
\n
\n
\n
unexpand.exe
\n
\n
\n
\n
uniq.exe
\n
\n
\n
\n
unrar.exe
\n
This is unrar 3.00 beta 7, which seems to have been\n "free". Works good enough.
\n
\n
\n
unshar.exe
\n
\n
\n
\n
uudecode.exe
\n
\n
\n
\n
uuencode.exe
\n
\n
\n
\n
wc.exe
\n
\n
\n
\n
wget.exe
\n
wget 1.8.2 builds out of the box with MS Visual C
\n
\n
\n
which.exe
\n
does not search the current directory
\n
\n
\n
whoami.exe
\n
\n
\n
\n
xargs.exe
\n
\n
\n
\n
yes.exe
\n
\n
\n
\n
zcat.exe
\n
\n
\n
\n
\n
\n
\n
Additional\n programs
\n
\n
\n
pclip.exe
\n
put the Windows clipboard text to stdout
\n
\n
\n
gclip.exe
\n
get the Windows clipboard text from stdin \n Example: run the text from the clipboard through sed and put the result back \n \n pclip | sed "s/string1/string2/g" | gclip
\n
\n
\n
gplay.exe
\n
My minimalist console multimedia player using DirectShow. With \n gplay filename | URL \n you should be able to play just everything, as long as Microsoft supports it.
The gcc compiler for Win32: \n http://www.mingw.org \nDigital Mars compiler: \nhttp://www.digitalmars.com \nA zsh shell for Win32: \n ftp://ftp.blarg.net/users/amol/zsh \nThe home of vim, has a Win32 version: \n http://www.vim.org \nAn emacs for NT: \n http://www.gnu.org/software/emacs/windows/ntemacs.html \nThe CVS version control system: \n http://www.cvshome.org \n http://www.cvsnt.org \nA free IDE for the Win32 gcc: \n VIDE
Useful free NT system utilities: \n http://www.sysinternals.com \nSome hints on cloning your system with dd: \n http://www.rajeevnet.com/hacks_hints/os_clone/os_cloning.html \nPrograms missing from this distribution, especially a command line registry editor. \n http://www.dmst.aueb.gr/dds/sw/outwit \n \nThe GNU Privacy Guard: \n http://www.gnupg.org \nzip and unzip from the Infozip distribution: \n http://www.info-zip.org/pub/infozip
\n\n
An Un*x emulation on Win32: \n http://www.cygwin.com \nAnother Un*x emulation on Win32: \n http://www.research.att.com/sw/tools/uwin/ \nWindows Services for UNIX, integrates the Interix subsystem: \n http://www.microsoft.com/windows/sfu/productinfo/overview/default.asp
\n\n\n\n
Read my standard disclaimer: (really?) \nLast change on 15-04-03 by Karl M. Syring (bug reports\nare welcome) \nOriginal location of this page: http://www.weihenstephan.de/~syring/win32/UnxUtils.html
\n\n\n\n\n\n\n\n"
},
{
"path": "bin/xidel/readme.txt",
"content": "\n================================================ Basics ================================================\n\n\nThe trivial usage is to extract an expression from a webpage like:\n \n xidel http://www.example.org --extract //title\n\nInstead of one or more urls, you can also pass file names or the xml data itself (xidel \"..\" ...). \nThe --extract option can be abbreviated as -e, and there are five different kind of extract expressions:\n \n 1 ) XPath 2 expressions, with some changes and additional functions.\n \n 2 ) XQuery 1 expressions\n \n 3 ) CSS 3 selectors. \n \n 4 ) Templates, a simplified version of the page which is pattern matched against the input\n \n 5 ) Multipage templates, i.e. a file that contains templates for several pages\n\nThe different kinds except multipage templates are usually automatically detected, but \na certain type can be forced with the extract-kind option.\nOr by using the shorter --xpath \"..\", --xquery \"..\", --css \"..\" options.\nEspecially XQuery and template expressions are easily confused by the auto detector. \n(Xidel assumes templates, if the expression starts with a \"<\" )\n\nSee the sections below for a more detailed description of each expression kind.\n\n\n\nThe next important option is --follow (abbreviated as -f) to follow links on a page. E.g: \n\n xidel http://www.example.org --follow //a --extract //title \n\nThis will print the titles of all pages that are linked from http://www.example.org.\n\n--follow supports the same expressions as --extract, and it will follow the href or src attributes of the \nusual elements, or the contained text if there are no such attributes.\n\n\n\n============================== Recursion / Argument order and grouping ===============================\n\n\nYou can specify multiple --extract (-e) and --follow (-f) arguments to extract values from one page, \nfollow the links to the next pages and extract values from there as well ...\nThen it becomes important in which order the arguments are given, so it extracts before following, \nor the other way around. \nYou can usually read it left-to-right like an English sentence, extracting from the current page,\nor following to a new one, which will then become the next current page.\nFor example:\n\na) xidel http://site1 -e \"select content 1\" http://site2 -e \"select content 2\" \n \n This will extract content 1 from site 1 and content 2 from site 2\n\nb) xidel http://site1 http://site2 -e \"select content 1\" -e \"select content 2\"\n\n This will extract content 1 and 2 from site 1 as well as from site 2\n\nc) xidel http://site1 -e \"select content 1\" -f \"//a (:select links:)\" -e \"select content 2\"\n\n This will extract the \"content 1\" from site1, and \"content 2\" from all sites the first site has links to. \n \nd) xidel http://site1 -f \"//a (:select links:)\" -e \"select content 1\" -e \"select content 2\"\n \n This will extract \"content 1\" and \"content 2\" from all sites the first site links to, and will not \n extract anything from site1. \n\ne) xidel http://site1 -e \"select content 1\" -e \"select content 2\" -f \"//a (:select links:)\"\n\n This is some kind of special case. Since -f is the last option, it will repeat the previous operation, i.e.\n it will extract content 1 and 2 from site1 and ALL sites that can be reached by an selected link on site1 \n or any other of the processed sites. \n Only if there were another -e after -f, it would extract that from the first set of followed links and stop.\n \nIn some kinds of extract expression you can create new variables, if you assign values to a variable called\n\"_follow\", that value will be included in the next follow expression. \nIf you assign an object to _follow, its properties will override the command line parameters with the same\nvalue. \n\nGenerally an option modifier (like --extract-kind) affects all succeeding options, unless there are none, \nthen it affects the immediate preceding option.\n\n\n\n\nYou can always override the argument order by using [ and ] to group the options.\nFor example:\n\nf) xidel http://site1 [ -f \"//a (:select links:)\" -e \"select content 1\" ] -e \"select content 2\"\n \n This will extract content 1 from all sites linked by site1 and content 2 from site1 itself.\n I.e. the extract of content 2 is not affected by the follow-option within the [..] brackets.\n\ng) xidel http://site1 [ -f //a[@type1] --download type1/ ] \n [ -f //a[@type2] --download type2/ ] \n [ -f //a[@type3] --download type3/ ] \n \n This will download all links of type 1 in a directory type1, all links of type2 in directory type2...\n (if written on one line)\n \n[ and ] must be surrounded by a space.\n\n\n\n\n\n\n========================================== XPath 2.0 / XQuery ===========================================\n\nXPath expressions provide an easy way to extract calculated values from x/html. \nSee http://en.wikipedia.org/wiki/XPath_2.0 for details.\n\nXidel also supports JSONiq and some custom extensions, so it deviates in a few ways from the standard. \nHowever, you can disable this differences with the respective options (see link below or the\ncommand line parameter listing printed by --help).\nSwitched to full standard compatible mode, its implementation passes 99.3% of the XPath 2 only tests and \n97.8% of the XQuery 1 tests in the XQuery Testsuite (skipping tests for invalid input queries)\n\nHowever, in the default mode, there are the following important extensions:\n\n Syntax:\n \n Variable assignment: $var := value\n \n adds $var to a set of global variables, which can be created and accessed \n everywhere\n\n JSONiq literals true, false, null\n \n true and false are evaluated as true(), false(), null becomes jn:null()\n \n JSONiq arrays: [a,b,c]\n\n Arrays store a list of values and can be nested with each other and \n within sequences.\n jn:members converts an array to a sequence.\n\n JSONiq objects: {\"name\": value, ...}\n \n Object stores a set of values as associative map. The values can be \n accessed similar to a function call, e.g.: {\"name\": value, ...}(\"name\").\n Xidel also has {\"name\": value, ..}.name as an additional syntax to \n access properties.\n jn:keys returns a sequence of all property names, libjn:values a sequence\n of values.\n Used with global variables, you can copy an object with obj2 := obj \n (objects are immutable, but properties can be changed with \n obj2.foo := 12, which will create a new object with the changed property)\n \n Extended strings: x\"..{..}..\"\n \n If a string is prefixed by an \"x\", all expressions inside {}-parentheses \n are evaluated, like in the value of a direct attribute constructor.\n (Warning: This was changed in Xidel 0.7. Xidel <= 0.6 used \n \"foo$var;bar\" without prefix for this)\n \n \n Semantic:\n \n All string comparisons are case insensitive, and \"clever\", e.g.: \n '9xy' = '9XY' < '10XY' < 'xy'\n This is more useful for html (think of @class = 'foobar'), but can be \n disabled by passing collation urls to the string functions. \n \n Everything is weakly typed, e.g 'false' = false() is true, and 1+\"2\" is 3. \n\n Unknown namespace prefixes are resolved with the namespace bindings of the \n input data. \n Therefore //a always finds all links, independent of any xmlns-attributes.\n (however, if you explicitly declare a namespace like \n 'declare default element namespace \"...\"' in XQuery, it will only find \n elements in that namespace)\n\n XML Schemas, error codes and static type checking are not supported.\n\n Certain additional functions:\n \n jn:*, libjn:* The standard JSONiq and JSONlib functions\n json(\"str.\") Parses a string as json, or downloads json from an url.(only use with trusted input)\n serialize-json(value) \n Converts a value to JSON\n extract(\"string\",\"regex\"[,,[]])\n This applies the regex \"regex\" to \"string\" and returns only the matching part. \n If the argument is used, only the -th submatch will be returned.\n (this function used to be called \"filter\")\n css(\"sel\") This returns the nodes below the context node matched by the specified css 3 selector.\n You can use this to combine css and XPath, like in 'css(\"a.aclass\")/@href'.\n eval(\"xpath\") This will evaluate the string \"xpath\" as an XPath expression\n system(\"..\") Runs a certain program and returns its stdout result as string\n deep-text() This is the concatenated plain text of the every tag inside the current text. \n You can also pass a separator like deep-text(' ') to separate text of different nodes.\n inner-html() This is the html content of node ., like innerHTML in javascript. \n outer-html() This is the same as inner-html, but includes the node itself\n inner-xml() This is the xml content of node, similar to inner-html()\n outer-xml() Like outer-html(), but xml-serialized\n split-equal(\"list\", \"string\"[, \"sep\" = \" \"])\n Treats the string \"list\" as a list of strings separated by \"sep\" and tests if \n \"string\" is contained in that list. (just like css class matching)\n form(form, [overridden parameters = ()])\n Converts a html form in a http request, by url encoding all inputs descendants\n of the given form node. You can give a sequence of parameters to override.\n e.g. form(//form[1], \"foo=bar&xyz=123\") returns a request for the first form,\n with the foo and xyz parameters overriden by bar and 123.\n You can also use a JSON object to set the override parameters, e.g.\n {\"foo\": \"bar\", \"xyz\": 123}, in that case they are url encoded.\n It returns an object with .url, .method and .post properties.\n match(, )\n Performs pattern matching between the template (see below for template documentation) \n and the nodes, and returns a list or an object of matched values.\n For exmple match({{.}}, FOOBAR) returns FOO, and\n match(*{{.}}, FOOBAR) returns (FOO, BAR).\n It is also possible to use named variables in the template, in which case an object \n is returned, e.g:\n match({{first:=.}}{{second:=.}}, FOOBAR)\n returns an object with two properties \"first\" and \"bar\", containing respectively\n FOO and BAR.\n The template can be a node or a string. Written as string the above example would be\n match(\"{.}\", FOOBAR).\n \n All additional functions except the jn/libjn functions are in the pxp: namespace, which is also set\n as default namespace.\n\nThe pasdoc documentation of my XPath 2 / XQuery library explains more details and lists more functions:\nhttp://www.benibela.de/documentation/internettools/xquery.TXQueryEngine.html\n\n\n\n========================================== CSS 3.0 Selectors ==========================================\n\n\nCSS 3 Selectors are fully supported, except some pseudoclasses like :hover and ::before that do not \nmake sense in a gui less, reading-only application.\n(It is however not much tested, since I personally only use XPath)\n\nThe easiest way to use CSS selectors with the command line is to write it like --extract \"css('selector')\"\n(the \"-quotes are necessary to escape the '-quotes.) \n\nAlternatively you can use --extract-kind=css --extract=\"your selector\", or --css=\"your selector\"\n\n\n\n============================================== Templates ==============================================\n\nTemplates are a very easy way to extract complex data from a webpage. \nEach template is basically a stripped-down excerpt of the webpage, in which the relevant parts have \nbeen annotated.\n\nThe best way to describe templates is with a real world example:\n\nThe following is the html of an entry of one of the recommended videos you can always see at the right\nside of an youtube video: (skipped the image part for clarity)\n\n
\n\nAs you see there are actual interesting values like the url/title/username/view texts, and irrelevant, \nchanging values like the session url.\nIf you now remove the irrelevant parts, and annotate the interesting values as {name:=value}, \nyou get the following:\n\n
+\n\nThis template can directly passed as an extract-expression, applied to the page of an youtube video,\nand will return all recommended/related videos.\nMore precisely, it will return four (interleaved) arrays \"title\", \"username\", \"views\", \"url\" each \ncontaining the relevant values.\n\nA basic template as above consists of three different kind of expressions:\n\n A normal html element will be matched to the processed html page.\n This means it will search the first element on the page, that has the same node name,\n all the attributes with the same values, and whose children match the children of the \n template element. \n \n {..} A {} marker will execute the contained XPath expression, once the corresponding \n place in the html page has been found. \n The context node . will refer to the surrounding element, and you can use my extended \n XPath syntax (var := value) to create a variable. (see XPath above)\n Often you want to read the entire matched element in a variable with $name, which\n can be written as {$name := .} or further abbreviated as {$name} . \n It can also be used within attributes, like to read the attribute value.\n (the parentheses can be also replaced by .. or ..)\n \n \n + Finally the loop marker will repeat the matching of the previous element as long as \n possible (an similar syntax is .. or ..).\n \n \n\nThis is sufficient for most basic scraping operations, but you can also use the following things in a \ntemplate:\n \n textnodes Textnodes are matched like html element nodes.\n A textnode in the webpage is considered a valid match, if it starts\n with the same text as the text node in the template.\n (but you can change this behavior to ends-with/exact/regex-comparisons with \n the \n command)\n \n All children of a template:if-tag are ignored if the test-XPath-expressions \n evaluates to false()\n \n Only one of the child elements will be used for matching\n\n Same a t:switch, but it will choose the earliest template child that has a match.\n \n t:optional=\"true\" Html nodes can be marked as optional, and they will be ignored, if no possible\n match can be found\n \n t:condition=\"??\" An XPath expression that c be The context node (.) refers to a potential match.\n \n * Like +, but it can also match none\n \n {min,max} or {count} Matches between [min,max] or {count}-many occurrences of the previous element\n \n \n The same as above. However, t:loop will repeat all its children, while a marker \n like + can only repeat the single, previous element.\n \n ? Short notation for t:optional. \n \n(see http://www.benibela.de/documentation/internettools/extendedhtmlparser.THtmlTemplateParser.html \n for more detailed explanations)\n \n\nThere is also a Greasemonkey script to create templates directly by just selecting the text on the \ncorresponding webpage.\n\n\n\n========================================= Multipage templates ==========================================\n\nMultipage templates collect several single page templates in a xml file.\nThey are basically just a list of nodes with data and associated s\nE.g.\n\n \n \n unescaped post data\n your=escaped&post=data&...\n \n {alink:=.}* \n \n \n \n ... \n \n ...\n \n\nAll pages are downloaded with GET or respectively POST requests, and processed with the given template.\nThe page-node also accepts a \"test\" attribute, which gives an XPath expression that needs to be true, \nif the page element should be used.\nIn the attributes and the text of post-nodes, everything enclosed in {..} parentheses is evaluated\nas xpath expression. (like in an extended x\"..\" string, see above)\n\nSince this would be cumbersome to pass directly to --extract, you can also specify the containing file\nwith the --template-file argument.\n\nYou can also have multiple s in a multipage template (surrounded by a parent element with \nname ), and call the later actions with from another action.\nIf a template with multiple actions is passed to Xidel it will always perform the first action,\nunless the --template-action parameter specifies another action to run. (in Xidel > 0.5)\n\nThere are also -elements to declare variables and -elements to repeat other elements, \nsee http://www.benibela.de/documentation/internettools/multipagetemplate.TMultiPageTemplate.html\nfor more details.\n\n=========================================== Input formats =============================================\n\nXidel supports html and xml input, and the option input-format can be used to set the parsing behaviour:\n \nauto: Automatically switch between html and xml\n \nhtml: The input will be parsed as html. \n Missing tags like head, body, tbody are automatically created.\n (beware that this means table/tr is never valid, and either table//tr or table/tbody/tr\n has to be used)\n \nxml: The input will be parsed as xml. \n However, it still uses the html parser, so it will correct missing end tags and not\n support DTDs.\n \nxml-strict: The input will be parsed as strict xml. \n This uses the standard fpc, validating xml parser.\n \nYou can also use json files, by loading them explicitly with pxp:json() or jn:json-doc() within a\nXPath/XQuery expression.\n\n=========================================== Output formats =============================================\n\nXidel has several different output formats, which can be chosen with the output-format option:\n\nadhoc: A very simple format, it will just print all values (default)\n\nxml: The output will be serialized as xml\n\nhtml: The output will be serialized as html\n\nxml-wrapped: It will print a xml-based machine readable output.\n Sequences will become value 1value 2...\n Objects will become \n (so in contrast to xml, it will keep variable names and type information intact)\n\njson-wrapped: It will print a json-based machine readable output.\n Sequences become arrays [ ... ].\n Objects become objects. {\"prop-1\": \"value 1\", \"prop-2\": \"value 2\", ... } \n (this was called json before Xidel 0.7)\n\nbash: Prints a bash script that sets the internal variables as bash variables.\n E.g.\n eval $(xidel http://data -e 'title:=//title' -e 'links:=//a')\n can be used to set the bash variable $title to the title of a page and the\n variable $links to a bash array of all links on the page.\n\ncmd: Like bash, but for Windows cmd.exe\n\nGenerally it prints a sequence of all processed pages (i.e. each page a single sequence element), \nand the variables defined as global variables or read by a template become variables or \nobject properties.\nThere is a special rule for json-wrapped output, if the template assigns multiple values to the same\nvariable: Xidel will collect all these values in an array. I.e. (a:=1, b:=2, a:=3, c:=4)\nbecomes \"a\": [1, 3], \"b\": 2. \"c\": 4\n\n\n\n\n\n"
},
{
"path": "build.bat",
"content": "@echo off\nsetlocal EnableDelayedExpansion \n\nset PROGFILES=%ProgramFiles%\nif not \"%ProgramFiles(x86)%\" == \"\" set PROGFILES=%ProgramFiles(x86)%\n\nREM Check if Visual Studio 2017 is installed\nset MSVCDIR=\"%PROGFILES%\\Microsoft Visual Studio\\2017\"\nset VCVARSALLPATH=\"%PROGFILES%\\Microsoft Visual Studio\\2017\\Community\\VC\\Auxiliary\\Build\\vcvarsall.bat\"\nif exist %MSVCDIR% (\n if exist %VCVARSALLPATH% (\n \tset COMPILER_VER=\"2017\"\n echo Using Visual Studio 2017 Community\n\tgoto setup_env\n )\n)\nREM Check if Visual Studio 2015 is installed\nset MSVCDIR=\"%PROGFILES%\\Microsoft Visual Studio 14.0\"\nset VCVARSALLPATH=\"%PROGFILES%\\Microsoft Visual Studio 14.0\\VC\\vcvarsall.bat\"\nif exist %MSVCDIR% (\n if exist %VCVARSALLPATH% (\n \tset COMPILER_VER=\"2015\"\n echo Using Visual Studio 2015\n\tgoto setup_env\n )\n)\nREM Check if Visual Studio 2013 is installed\nset MSVCDIR=\"%PROGFILES%\\Microsoft Visual Studio 12.0\"\nset VCVARSALLPATH=\"%PROGFILES%\\Microsoft Visual Studio 12.0\\VC\\vcvarsall.bat\"\nif exist %MSVCDIR% (\n if exist %VCVARSALLPATH% (\n set COMPILER_VER=\"2013\"\n echo Using Visual Studio 2013\n\tgoto setup_env\n )\n)\n\nREM Check if Visual Studio 2012 is installed\nset MSVCDIR=\"%PROGFILES%\\Microsoft Visual Studio 11.0\"\nset VCVARSALLPATH=\"%PROGFILES%\\Microsoft Visual Studio 11.0\\VC\\vcvarsall.bat\"\nif exist %MSVCDIR% (\n if exist %VCVARSALLPATH% (\n set COMPILER_VER=\"2012\"\n echo Using Visual Studio 2012\n\tgoto setup_env\n )\n)\n\nREM Check if Visual Studio 2010 is installed\nset MSVCDIR=\"%PROGFILES%\\Microsoft Visual Studio 10.0\"\nset VCVARSALLPATH=\"%PROGFILES%\\Microsoft Visual Studio 10.0\\VC\\vcvarsall.bat\"\nif exist %MSVCDIR% (\n if exist %VCVARSALLPATH% (\n set COMPILER_VER=\"2010\"\n echo Using Visual Studio 2010\n\tgoto setup_env\n )\n)\n\nREM Check if Visual Studio 2008 is installed\nset MSVCDIR=\"%PROGFILES%\\Microsoft Visual Studio 9.0\"\nset VCVARSALLPATH=\"%PROGFILES%\\Microsoft Visual Studio 9.0\\VC\\vcvarsall.bat\"\nif exist %MSVCDIR% (\n if exist %VCVARSALLPATH% (\n set COMPILER_VER=\"2008\"\n echo Using Visual Studio 2008\n\tgoto setup_env\n )\n)\n\nREM Check if Visual Studio 2005 is installed\nset MSVCDIR=\"%PROGFILES%\\Microsoft Visual Studio 8\"\nset VCVARSALLPATH=\"%PROGFILES%\\Microsoft Visual Studio 8\\VC\\vcvarsall.bat\"\nif exist %MSVCDIR% (\n if exist %VCVARSALLPATH% (\n\tset COMPILER_VER=\"2005\"\n echo Using Visual Studio 2005\n\tgoto setup_env\n )\n) \n\nREM Check if Visual Studio 6 is installed\nset MSVCDIR=\"%PROGFILES%\\Microsoft Visual Studio\\VC98\"\nset VCVARSALLPATH=\"%PROGFILES%\\Microsoft Visual Studio\\VC98\\vcvarsall.bat\"\nif exist %MSVCDIR% (\n if exist %VCVARSALLPATH% (\n\tset COMPILER_VER=\"6\"\n echo Using Visual Studio 6\n\tgoto setup_env\n )\n) \n\necho No compiler : Microsoft Visual Studio (6, 2005, 2008, 2010, 2012, 2013 or 2015) is not installed.\ngoto end\n\n:setup_env\n\necho Setting up environment\nif %COMPILER_VER% == \"6\" (\n\tcall %MSVCDIR%\\Bin\\VCVARS32.BAT\n\tgoto begin\n)\n\n:begin\n\nREM Setup path to helper bin\nset ROOT_DIR=\"%CD%\"\nset RM=\"%CD%\\bin\\unxutils\\rm.exe\"\nset CP=\"%CD%\\bin\\unxutils\\cp.exe\"\nset MKDIR=\"%CD%\\bin\\unxutils\\mkdir.exe\"\nset SEVEN_ZIP=\"%CD%\\bin\\7-zip\\7za.exe\"\nset XIDEL=\"%CD%\\bin\\xidel\\xidel.exe\"\n\nREM Housekeeping\n%RM% -rf tmp_*\n%RM% -rf third-party\n%RM% -rf curl.zip\n%RM% -rf build_*.txt\n\nREM Get download url .Look under