[ { "path": "LICENSE", "content": "MIT License\n\nCopyright (c) 2020 Willem Hengeveld \n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "README.md", "content": "IDBTOOL\n=======\n\nA tool for extracting information from IDA databases.\n`idbtool` knows how to handle databases from all IDA versions since v2.0, both `i64` and `idb` files.\nYou can also use `idbtool` to recover information from unclosed databases.\n\n`idbtool` works without change with IDA v7.0.\n\n\nMuch faster than loading a file in IDA\n--------------------------------------\n\nWith idbtool you can search thousands of .idb files in seconds.\n\nMore precisely: on my laptop it takes:\n\n * 1.5 seconds to extract 143 idc scripts from 119 idb and i64 files.\n * 3.8 seconds to print idb info for 441 files.\n * 5.6 seconds to extract 281 enums containing 4726 members from 35 files.\n * 67.8 seconds to extract 5942 structs containing 33672 members from 265 files.\n\nLoading a approximately 5 Gbyte idb file in IDA, takes about 45 minutes.\nWhile idb3.h takes basically no time at all, no more than a few milliseconds.\n\n\n\nDownload\n========\n\nTwo versions of this tool exist:\n\nOne written in python\n * https://github.com/nlitsme/pyidbutil\n\nOne written in C++\n * https://github.com/nlitsme/idbutil\n\nBoth repositories contain a library which can be used for reading `.idb` or `.i64` files.\n\n\nUsage\n=====\n\nUsage: \n\n idbtool [options] [database file(s)]\n\n * `-n` or `--names` will list all named values in the database.\n * `-s` or `--scripts` will list all scripts stored in the database.\n * `-u` or `--structs` will list all structs stored in the database.\n * `-e` or `--enums` will list all enums stored in the database.\n * `--imports` will list all imported symbols from the database.\n * `--funcdirs` will list function folders stored in the database.\n * `-i` or `--info` will print some general info about the database. \n * `-d` or `--pagedump` dump btree page tree contents.\n * `--inc`, `--dec` list all records in ascending / descending order.\n * `-q` or `--query` search specific records in the database.\n * `-m` or `--limit` limit the number of results returned by `-q`.\n * `-id0`, `-id1` dump only one specific section.\n * `--i64`, `--i32` tell idbtool that the specified file is from a 64 or 32 bit database.\n * `--recover` group files from an unpacked database.\n * `--classify` summarizes node usage in the database\n * `--dump` hexdump the original binary data\n\nquery\n-----\n\nQueries need to be specified last on the commandline.\n\nexample:\n\n idbtool [database file(s)] --query \"Root Node;V\"\n\nWill list the source binary for all the databases specified on the commandline.\n\nA query is a string with the following format:\n\n * [==,<=,>=,<,>] - optional relation, default: ==\n * a base node key:\n * a DOT followed by the numeric value of the nodeid.\n * a HASH followed by the numeric value of the system-nodeid.\n * a QUESTION followed by the name of the node. -> a 'N'ame node\n * the name of the node. -> the name is resolved, results in a '.'Dot node\n * an optional tag ( A for Alt, S for Supval, etc )\n * an optional index value\n\nexample queries:\n * `Root Node;V` -> prints record containing the source binary name\n * `?Root Node` -> prints the Name record pointing to the root\n * `>Root Node` -> prints the first 10 records starting with the root node id.\n * ` prints the 10 records startng with the recordsbefore the rootnode.\n * `.0xff000001;N` -> prints the rootnode name entry.\n * `#1;N` -> prints the rootnode name entry.\n\nList the highest node and following record in the database in two different ways,\nthe first: starting at the first record below `ffc00000`, and listing the next.\nThe second: starting at the first record after `ffc00000`, and listing the previous:\n * `--query \"<#0xc00000\" --limit 2 --inc -v`\n * `--query \">#0xc00000\" --limit 2 --dec -v`\n\nNote that this should be the nodeid in the `$ MAX NODE` record.\n\nList the last two records:\n * `--limit 2 --dec -v`\n\nList the first two records, the `$ MAX LINK` and `$ MAX NODE` records:\n * `--limit 2 --inc -v`\n\n\nA full database dump\n--------------------\n\nSeveral methods exist for printing all records in the database. This may be useful if\nyou want to investigate more of IDA''s internals. But can also be useful in recovering\ndata from corrupted databases.\n\n * `--inc`, `--dec` can be used to enumerate all b-tree records in either forward, or backward direction.\n * add `-v` to get a prettier key/value output\n * `--id0` walks the page tree, instead of the record tree, printing the contents of each page\n * `--pagedump` linearly skip through the file, this will also reveal information in deleted pages.\n\nnaked files\n===========\n\nWhen IDA or your computer crashed while working on a disassembly, and you did not yet save the database,\nyou are left with a couple of files with extensions like `.id0`, `.id1`, `.nam`, etc.\n\nThese files are the unpacked database, i call them `naked` files.\n\nUsing the `--filetype` and `--i64` or `--i32` options you can inspect these `naked` files individually.\nor use the `--recover` option to view them as a complete database together.\n`idbtool` will figure out automatically which files would belong together.\n\n`idbtool` can figure out the bitsize of the database from an `.id0` file, but not(yet) from the others.\n\n\nLIBRARY\n=======\n\nThe file `idblib.py` contains a library.\n\n\nTODO\n====\n\n * add option to list all comments stored in the database\n * add option to list flags for a list of addresses.\n\nAuthor\n======\n\nWillem Hengeveld \n\n" }, { "path": "idaunpack.py", "content": "\"\"\"\n`idaunpack` is a tool to aid in decoding packed data structures from an\nIDA idb or i64 database.\n\"\"\"\nfrom __future__ import print_function, division\nimport struct\nimport re\nimport sys\nfrom binascii import a2b_hex, b2a_hex\nfrom idblib import IdaUnpacker\n\ndef dump_packed(data, wordsize, pattern):\n p = IdaUnpacker(wordsize, data)\n if pattern:\n for c in pattern:\n if p.eof():\n print(\"EOF\")\n break\n if c == 'H':\n val = p.next16()\n fmt = \"%04x\"\n elif c == 'L':\n val = p.next32()\n fmt = \"%08x\"\n elif c == 'Q':\n val = p.next64()\n fmt = \"%016x\"\n elif c == 'W':\n val = p.nextword()\n if wordsize==4:\n fmt = \"[%08x]\"\n else:\n fmt = \"[%016x]\"\n else:\n raise Exception(\"unknown pattern: %s\" % c)\n print(fmt % val, end=\" \")\n\n while not p.eof():\n val = p.next32()\n print(\"%08x\" % val, end=\" \")\n\n print()\n\ndef unhex(hextxt):\n return a2b_hex(re.sub(r'\\W+', '', hextxt, flags=re.DOTALL))\n\ndef main():\n import argparse\n parser = argparse.ArgumentParser(description='idaunpack')\n parser.add_argument('--verbose', '-v', action='store_true')\n parser.add_argument('--debug', action='store_true', help='abort on exceptions.')\n parser.add_argument('--pattern', '-p', type=str, help='unpack pattern: sequence of H, L, Q, W')\n parser.add_argument('-4', '-3', '-32', const=4, dest='wordsize', action='store_const', help='use 32 bit words')\n parser.add_argument('-8', '-6', '-64', const=8, dest='wordsize', action='store_const', help='use 64 bit words')\n parser.add_argument('--wordsize', '-w', type=int, help='specify wordsize')\n parser.add_argument('hexconsts', nargs='*', type=str)\n\n args = parser.parse_args()\n if args.wordsize is None:\n args.wordsize = 4\n\n for x in args.hexconsts:\n dump_packed(unhex(x), args.wordsize, args.pattern)\n\nif __name__ == '__main__':\n main()\n" }, { "path": "idblib.py", "content": "\"\"\"\nidblib - a module for reading hex-rays Interactive DisAssembler databases\n\nSupports database versions starting with IDA v2.0\n\nIDA v1.x is not supported, that was an entirely different file format.\nIDA v2.x databases are organised as several files, in a directory\nIDA v3.x databases are bundled into .idb files\nIDA v4 .. v6 various improvements, like databases larger than 4Gig, and 64 bit support.\n\nCopyright (c) 2016 Willem Hengeveld \n\n\nAn IDB file can contain up to 6 sections:\n id0 the main database\n id1 contains flags for each byte - what is returned by idc.GetFlags(ea)\n nam contains a list of addresses of named items\n seg .. only in older databases\n til type info\n id2 ?\n\nThe id0 database is a simple key/value database, much like leveldb\n\ntypes of records:\n\nSome bookkeeping:\n\n \"$ MAX NODE\" -> the highest numbered node value in use.\n\nA list of names:\n\n \"N\" + name -> the node id for that name.\n\nnames are both user/disassembler symbols assigned to addresses\nin the disassembled code, and IDA internals, like lists of items,\nFor example: '$ structs', or 'Root Node'.\n\nThe main part:\n\n \".\" + nodeid + tag + index\n\nThis maps directly onto the idasdk netnode interface.\nThe size of the nodeid and index is 32bits for .idb files and 64 bits for .i64 files.\nThe nodeid and index are encoded as bigendian numbers in the key, and as little endian\nnumbers in (most of) the values.\n\n\n\"\"\"\nfrom __future__ import division, print_function, absolute_import, unicode_literals\nimport struct\nimport binascii\nimport re\nimport os\n\n#############################################################################\n# some code to make this library run with both python2 and python3\n#############################################################################\n\nimport sys\nif sys.version_info[0] == 3:\n long = int\nelse:\n bytes = bytearray\n\ntry:\n cmp(1, 2)\nexcept:\n # python3 does not have cmp\n def cmp(a, b): return (a > b) - (a < b)\n\n\nclass cachedproperty(object):\n ## .. only works with python3 somehow. -- todo: figure out why not with python2\n def __init__(self, method):\n self.method = method\n self.name = '_' + method.__name__\n def __get__(self, obj, cls):\n if not hasattr(obj, self.name):\n value = self.method(obj)\n setattr(obj, self.name, value)\n else:\n value = getattr(obj, self.name)\n return value\n\n\ndef strz(b, o):\n return b[o:b.find(b'\\x00', o)].decode('utf-8', 'ignore')\n\ndef makeStringIO(data):\n if sys.version_info[0] == 2:\n from StringIO import StringIO\n return StringIO(data)\n else:\n from io import BytesIO\n return BytesIO(data)\n\n\n#############################################################################\n# some utility functions\n#############################################################################\n\n\ndef nonefmt(fmt, item):\n # helper for outputting None without raising an error\n if item is None:\n return \"-\"\n return fmt % item\n\n\ndef hexdump(data):\n if data is None:\n return\n return binascii.b2a_hex(data).decode('utf-8')\n\n\n#############################################################################\n\n\nclass FileSection(object):\n \"\"\"\n Presents a file like object which is a section of a larger file.\n\n `fh` is expected to have a seek and read method.\n\n\n This class is used to access a section (e.g. the .id0 file) of a larger file (e.g. the .idb file)\n and make read/seek behave as if it were a separate file.\n \"\"\"\n def __init__(self, fh, start, end):\n self.fh = fh\n self.start = start\n self.end = end\n\n self.curpos = 0\n self.fh.seek(self.start)\n\n def read(self, size=None):\n want = self.end - self.start - self.curpos\n if size is not None and want > size:\n want = size\n\n if want <= 0:\n return b\"\"\n\n # make sure filepointer is at correct position since we are sharing the fh object with others.\n self.fh.seek(self.curpos + self.start)\n data = self.fh.read(want)\n self.curpos += len(data)\n return data\n\n def seek(self, offset, *args):\n def isvalidpos(offset):\n return 0 <= offset <= self.end - self.start\n\n if len(args) == 0:\n whence = 0\n else:\n whence = args[0]\n if whence == 0:\n if not isvalidpos(offset):\n print(\"invalid seek: from %x to SET:%x\" % (self.curpos, offset))\n raise Exception(\"illegal offset\")\n self.curpos = offset\n elif whence == 1:\n if not isvalidpos(self.curpos + offset):\n raise Exception(\"illegal offset\")\n self.curpos += offset\n elif whence == 2:\n if not isvalidpos(self.end - self.start + offset):\n raise Exception(\"illegal offset\")\n self.curpos = self.end - self.start + offset\n self.fh.seek(self.curpos + self.start)\n\n def tell(self):\n return self.curpos\n\n\nclass IdaUnpacker:\n \"\"\"\n Decodes packed ida structures.\n This is used o.a. in struct definitions, and .id2 files\n\n Related sdk functions: pack_dd, unpack_dd, etc.\n \"\"\"\n def __init__(self, wordsize, data):\n self.wordsize = wordsize\n self.data = data\n self.o = 0\n\n def eof(self):\n return self.o >= len(self.data)\n def have(self, n):\n return self.o+n <= len(self.data)\n\n def nextword(self):\n \"\"\"\n Return an unsigned word-sized integer from the buffer\n \"\"\"\n if self.wordsize == 4:\n return self.next32()\n elif self.wordsize == 8:\n return self.next64()\n else:\n raise Exception(\"unsupported wordsize\")\n\n def nextwordsigned(self):\n \"\"\"\n Return a signed word-sized integer from the buffer\n \"\"\"\n if self.wordsize == 4:\n val = self.next32()\n if val < 0x80000000:\n return val\n return val - 0x100000000\n elif self.wordsize == 8:\n val = self.next64()\n if val < 0x8000000000000000:\n return val\n return val - 0x10000000000000000\n else:\n raise Exception(\"unsupported wordsize\")\n\n\n def next64(self):\n if self.eof():\n return None\n lo = self.next32()\n hi = self.next32()\n return (hi<<32) | lo\n\n def next16(self):\n \"\"\"\n Return a packed 16 bit integer from the buffer\n \"\"\"\n if self.eof():\n return None\n byte = self.data[self.o:self.o+1]\n if byte == b'\\xff':\n # a 16 bit value:\n # 1111 1111 xxxx xxxx xxxx xxxx \n if self.o+3 > len(self.data):\n return None\n val, = struct.unpack_from(\">H\", self.data, self.o+1)\n self.o += 3\n return val\n elif byte < b'\\x80':\n # a 7 bit value:\n # 0xxx xxxx\n self.o += 1\n val, = struct.unpack(\"B\", byte)\n return val\n elif byte < b'\\xc0':\n # a 14 bit value:\n # 10xx xxxx xxxx xxxx\n if self.o+2 > len(self.data):\n return None\n val, = struct.unpack_from(\">H\", self.data, self.o)\n self.o += 2\n return val&0x3FFF\n else:\n return None\n\n def next8(self):\n if self.eof():\n return None\n byte = self.data[self.o:self.o+1]\n self.o += 1\n val, = struct.unpack(\"B\", byte)\n\n return val\n\n def next32(self):\n \"\"\"\n Return a packed integer from the buffer\n \"\"\"\n if self.eof():\n return None\n byte = self.data[self.o:self.o+1]\n if byte == b'\\xff':\n # a 32 bit value:\n # 1111 1111 xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx\n if self.o+5 > len(self.data):\n return None\n val, = struct.unpack_from(\">L\", self.data, self.o+1)\n self.o += 5\n return val\n elif byte < b'\\x80':\n # a 7 bit value:\n # 0xxx xxxx\n self.o += 1\n val, = struct.unpack(\"B\", byte)\n return val\n elif byte < b'\\xc0':\n # a 14 bit value:\n # 10xx xxxx xxxx xxxx\n if self.o+2 > len(self.data):\n return None\n val, = struct.unpack_from(\">H\", self.data, self.o)\n self.o += 2\n return val&0x3FFF\n elif byte < b'\\xe0':\n # a 29 bit value:\n # 110x xxxx xxxx xxxx xxxx xxxx xxxx xxxx\n if self.o+4 > len(self.data):\n return None\n val, = struct.unpack_from(\">L\", self.data, self.o)\n self.o += 4\n return val&0x1FFFFFFF\n else:\n return None\n\n def bytes(self, n):\n \"\"\"\n Return fixed length string from buffer\n \"\"\"\n if not self.have(n):\n return None\n data = self.data[self.o : self.o+n]\n self.o += n\n return data\n\n\nclass IDBFile(object):\n \"\"\"\n Provide access to the various sections in an .idb file.\n\n Usage:\n\n idb = IDBFile(fhandle)\n id0 = idb.getsection(ID0File)\n\n ID0File is expected to have a class property 'INDEX'\n\n# v1..v5 id1 and nam files start with 'Va0' .. 'Va4'\n# v6 id1 and nam files start with 'VA*'\n# til files start with 'IDATIL'\n# id2 files start with 'IDAS\\x1d\\xa5\\x55\\x55'\n\n \"\"\"\n def __init__(self, fh):\n \"\"\" constructor takes a filehandle \"\"\"\n self.fh = fh\n self.fh.seek(0)\n hdrdata = self.fh.read(0x100)\n\n self.magic = hdrdata[0:4].decode('utf-8', 'ignore')\n if self.magic not in ('IDA0', 'IDA1', 'IDA2'):\n raise Exception(\"invalid file magic\")\n\n values = struct.unpack_from(\"<6LH6L\", hdrdata, 6)\n if values[5] != 0xaabbccdd:\n fileversion = 0\n offsets = list(values[0:5])\n offsets.append(0)\n checksums = [0 for _ in range(6)]\n else:\n fileversion = values[6]\n\n if fileversion < 5:\n offsets = list(values[0:5])\n checksums = list(values[8:13])\n idsofs, idscheck = struct.unpack_from(\"> 1\n if k < a[mid].key:\n last = mid\n else:\n first = mid + 1\n return first - 1\n\n\n\"\"\"\n################################################################################\n\nI would have liked to make these classes a nested class of BTree, but\nthe problem is than there is no way for a nested-nested class\nof BTree to refer back to a toplevel nested class of BTree.\nSo moving these outside of BTree so i can use them as baseclasses\nin the various page implementations\n\nclass BTree:\n class BaseEntry(object): pass\n class BasePage(object): pass\n class Page15(BasePage):\n class Entry(BTree.BaseEntry):\n pass\n\n>>> NameError: name 'BTree' is not defined\n\n\"\"\"\n\n\nclass BaseIndexEntry(object):\n \"\"\"\n Baseclass for Index Entries.\n\n Index entries have a key + value, and a page containing keys larger than that key\n in this index entry.\n\n \"\"\"\n def __init__(self, data):\n ofs = self.recofs\n if self.recofs < 6:\n # reading an invalid page...\n self.val = self.key = None\n return\n\n keylen, = struct.unpack_from(\", <, ==, >=, <= )\n \"\"\"\n class BasePage(object):\n \"\"\"\n Baseclass for Pages. for the various btree versions ( 1.5, 1.6 and 2.0 )\n there are subclasses which specify the exact layout of the page header,\n and index / leaf entries.\n\n Leaf pages don't have a 'preceeding' page pointer.\n\n \"\"\"\n def __init__(self, data, entsize, entfmt):\n self.preceeding, self.count = struct.unpack_from(entfmt, data)\n if self.preceeding:\n entrytype = self.IndexEntry\n else:\n entrytype = self.LeafEntry\n\n self.index = []\n key = b\"\"\n for i in range(self.count):\n ent = entrytype(key, data, entsize * (1 + i))\n self.index.append(ent)\n key = ent.key\n self.unknown, self.freeptr = struct.unpack_from(entfmt, data, entsize * (1 + self.count))\n\n def find(self, key):\n \"\"\"\n Searches pages for key, returns relation to key:\n\n recurse -> found a next level index page to search for key.\n also returns the next level page nr\n gt -> found a value with a key greater than the one searched for.\n lt -> found a value with a key less than the one searched for.\n eq -> found a value with a key equal to the one searched for.\n gt, lt and eq return the index for the key found.\n\n # for an index entry: the key is 'less' than anything in the page pointed to.\n \"\"\"\n i = binary_search(self.index, key)\n if i < 0:\n if self.isindex():\n return ('recurse', -1)\n return ('gt', 0)\n if self.index[i].key == key:\n return ('eq', i)\n if self.isindex():\n return ('recurse', i)\n return ('lt', i)\n\n def getpage(self, ix):\n \"\"\" For Indexpages, returns the page ptr for the specified entry \"\"\"\n return self.preceeding if ix < 0 else self.index[ix].page\n\n def getkey(self, ix):\n \"\"\" For all page types, returns the key for the specified entry \"\"\"\n return self.index[ix].key\n\n def getval(self, ix):\n \"\"\" For all page types, returns the value for the specified entry \"\"\"\n return self.index[ix].val\n\n def isleaf(self):\n \"\"\" True when this is a Leaf Page \"\"\"\n return self.preceeding == 0\n\n def isindex(self):\n \"\"\" True when this is an Index Page \"\"\"\n return self.preceeding != 0\n\n def __repr__(self):\n return (\"leaf\" if self.isleaf() else (\"index<%d>\" % self.preceeding)) + repr(self.index)\n\n ######################################################\n # Page objects for the various versions of the database\n ######################################################\n class Page15(BasePage):\n \"\"\" v1.5 b-tree page \"\"\"\n class IndexEntry(BaseIndexEntry):\n def __init__(self, key, data, ofs):\n self.page, self.recofs = struct.unpack_from(\"= 0:\n self.stack.append((page, ix))\n else:\n # move towards leaf\n self.stack.append((page, ix))\n while page.isindex():\n page = self.db.readpage(page.getpage(ix))\n ix = len(page.index) - 1\n self.stack.append((page, ix))\n\n def eof(self):\n return len(self.stack) == 0\n\n def getkey(self):\n \"\"\" return the key value pointed to by the cursor \"\"\"\n page, ix = self.stack[-1]\n return page.getkey(ix)\n\n def getval(self):\n \"\"\" return the data value pointed to by the cursor \"\"\"\n page, ix = self.stack[-1]\n return page.getval(ix)\n\n def __repr__(self):\n return \"cursor:\" + repr(self.stack)\n\n def __init__(self, fh):\n \"\"\" BTree constructor - takes a filehandle \"\"\"\n self.fh = fh\n\n self.fh.seek(0)\n data = self.fh.read(64)\n\n if data[13:].startswith(b\"B-tree v 1.5 (C) Pol 1990\"):\n self.parseheader15(data)\n self.page = self.Page15\n self.version = 15\n elif data[19:].startswith(b\"B-tree v 1.6 (C) Pol 1990\"):\n self.parseheader16(data)\n self.page = self.Page16\n self.version = 16\n elif data[19:].startswith(b\"B-tree v2\"):\n self.parseheader16(data)\n self.page = self.Page20\n self.version = 20\n else:\n print(\"unknown btree: %s\" % hexdump(data))\n raise Exception(\"unknown b-tree\")\n\n def parseheader15(self, data):\n self.firstfree, self.pagesize, self.firstindex, self.reccount, self.pagecount = struct.unpack_from(\" record equal to the key, None when not found\n 'le' -> last record with key <= to key\n 'ge' -> first record with key >= to key\n 'lt' -> last record with key < to key\n 'gt' -> first record with key > to key\n \"\"\"\n\n # descend tree to leaf nearest to the `key`\n page = self.readpage(self.firstindex)\n stack = []\n while len(stack) < 256:\n act, ix = page.find(key)\n stack.append((page, ix))\n if act != 'recurse':\n break\n page = self.readpage(page.getpage(ix))\n\n if len(stack) == 256:\n raise Exception(\"b-tree corrupted\")\n cursor = BTree.Cursor(self, stack)\n\n # now correct for what was actually asked.\n if act == rel:\n pass\n elif rel == 'eq' and act != 'eq':\n return None\n elif rel in ('ge', 'le') and act == 'eq':\n pass\n elif rel in ('gt', 'ge') and act == 'lt':\n cursor.next()\n elif rel == 'gt' and act == 'eq':\n cursor.next()\n elif rel in ('lt', 'le') and act == 'gt':\n cursor.prev()\n elif rel == 'lt' and act == 'eq':\n cursor.prev()\n\n return cursor\n\n def dump(self):\n \"\"\" raw dump of all records in the b-tree \"\"\"\n print(\"pagesize=%08x, reccount=%08x, pagecount=%08x\" % (self.pagesize, self.reccount, self.pagecount))\n self.dumpfree()\n self.dumptree(self.firstindex)\n\n def dumpfree(self):\n \"\"\" list all free pages \"\"\"\n fmt = \"L\" if self.version > 15 else \"H\"\n hdrsize = 8 if self.version > 15 else 4\n pn = self.firstfree\n if pn == 0:\n print(\"no free pages\")\n return\n while pn:\n self.fh.seek(pn * self.pagesize)\n data = self.fh.read(self.pagesize)\n if len(data) == 0:\n print(\"could not read FREE data at page %06x\" % pn)\n break\n count, nextfree = struct.unpack_from(\"<\" + (fmt * 2), data)\n freepages = list(struct.unpack_from(\"<\" + (fmt * count), data, hdrsize))\n freepages.insert(0, pn)\n for pn in freepages:\n self.fh.seek(pn * self.pagesize)\n data = self.fh.read(self.pagesize)\n print(\"%06x: free: %s\" % (pn, hexdump(data[:64])))\n pn = nextfree\n\n def dumpindented(self, pn, indent=0):\n \"\"\"\n Dump all nodes of the current page with keys indented, showing how the `indent`\n feature works\n \"\"\"\n page = self.readpage(pn)\n print(\" \" * indent, page)\n if page.isindex():\n print(\" \" * indent, end=\"\")\n self.dumpindented(page.preceeding, indent + 1)\n for p in range(len(page.index)):\n print(\" \" * indent, end=\"\")\n self.dumpindented(page.getpage(p), indent + 1)\n\n def dumptree(self, pn):\n \"\"\"\n Walks entire tree, dumping all records on each page\n in sequential order\n \"\"\"\n page = self.readpage(pn)\n print(\"%06x: preceeding = %06x, reccount = %04x\" % (pn, page.preceeding, page.count))\n for ent in page.index:\n print(\" %s\" % ent)\n if page.preceeding:\n self.dumptree(page.preceeding)\n for ent in page.index:\n self.dumptree(ent.page)\n\n def pagedump(self):\n \"\"\"\n dump the contents of all pages, ignoring links between pages,\n this will enable you to view contents of pages which have become\n lost due to datacorruption.\n \"\"\"\n self.fh.seek(self.pagesize)\n pn = 1\n while True:\n try:\n pagedata = self.fh.read(self.pagesize)\n if len(pagedata) == 0:\n break\n elif len(pagedata) != self.pagesize:\n print(\"%06x: incomplete - %d bytes ( pagesize = %d )\" % (pn, len(pagedata), self.pagesize))\n break\n elif pagedata == b'\\x00' * self.pagesize:\n print(\"%06x: empty\" % (pn))\n else:\n page = self.page(pagedata)\n\n print(\"%06x: preceeding = %06x, reccount = %04x\" % (pn, page.preceeding, page.count))\n for ent in page.index:\n print(\" %s\" % ent)\n except Exception as e:\n print(\"%06x: ERROR decoding as B-tree page: %s\" % (pn, e))\n pn += 1\n\n\nclass ID0File(object):\n \"\"\"\n Reads .id0 or 0.ida files, containing a v1.5, v1.6 or v2.0 b-tree database.\n\n This is basically the low level netnode interface from the idasdk.\n\n There are two major groups of nodes in the database:\n\n key = \"N\"+name -> value = littleendian(nodeid)\n key = \".\"+bigendian(nodeid)+char(tag)+bigendian(value)\n key = \".\"+bigendian(nodeid)+char(tag)+string\n\n key = \".\"+bigendian(nodeid)+char(tag)\n\n and some special nodes for bookkeeping:\n \"$ MAX LINK\"\n \"$ MAX NODE\"\n \"$ NET DESC\"\n\n Very old databases also have name entries with a lowercase 'n',\n and corresponding '-'+value nodes.\n I am not sure what those are for.\n\n several items have specially named nodes, like \"$ structs\", \"$ enums\", \"Root Node\"\n\n nodeByName(name) returns the nodeid for a name\n bytes(nodeid, tag, val) returns the value for a specific node.\n\n \"\"\"\n INDEX = 0\n\n def __init__(self, idb, fh):\n self.btree = BTree(fh)\n\n self.wordsize = None\n self.maxnode = None\n\n if idb.magic == 'IDA2':\n # .i64 files use 64 bit values for some things.\n self.wordsize = 8\n elif idb.magic in ('IDA0', 'IDA1'):\n self.wordsize = 4\n else:\n # determine wordsize from value of '$ MAX NODE'\n c = self.btree.find('eq', b'$ MAX NODE')\n if c and not c.eof():\n self.maxnode = c.getval()\n self.wordsize = len(c.getval())\n\n if self.wordsize not in (4, 8):\n print(\"Can not determine wordsize for database - assuming 32 bit\")\n self.wordsize = 4\n\n if self.wordsize == 4:\n self.nodebase = 0xFF000000\n if not self.maxnode:\n self.maxnode = self.nodebase + 0x0FFFFF\n self.fmt = \"L\"\n else:\n self.nodebase = 0xFF00000000000000\n if not self.maxnode:\n self.maxnode = self.nodebase + 0x0FFFFFFF\n\n self.fmt = \"Q\"\n\n # set the keyformat for this database\n self.keyfmt = \">s\" + self.fmt + \"s\" + self.fmt\n\n @cachedproperty\n def root(self): return self.nodeByName(\"Root Node\")\n\n # note: versions before 4.7 used a short instead of a long\n # and stored the versions with one minor digit ( 43 ) , instead of two ( 480 )\n @cachedproperty\n def idaver(self): return self.int(self.root, 'A', -1)\n\n @cachedproperty\n def idbparams(self): return self.bytes(self.root, 'S', 0x41b994)\n @cachedproperty\n def idaverstr(self): return self.string(self.root, 'S', 1303)\n @cachedproperty\n def nropens(self): return self.int(self.root, 'A', -4)\n @cachedproperty\n def creationtime(self): return self.int(self.root, 'A', -2)\n @cachedproperty\n def originmd5(self): return self.bytes(self.root, 'S', 1302)\n @cachedproperty\n def somecrc(self): return self.int(self.root, 'A', -5)\n\n def prettykey(self, key):\n \"\"\"\n returns the key in a readable format.\n \"\"\"\n f = list(self.decodekey(key))\n f[0] = f[0].decode('utf-8')\n if len(f) > 2 and type(f[2]) == bytes:\n f[2] = f[2].decode('utf-8')\n\n if f[0] == '.':\n if len(f) == 2:\n return \"%s%16x\" % tuple(f)\n elif len(f) == 3:\n return \"%s%16x %s\" % tuple(f)\n elif len(f) == 4:\n if f[2] == 'H' and type(f[3]) in (str, bytes):\n f[3] = f[3].decode('utf-8')\n return \"%s%16x %s '%s'\" % tuple(f)\n elif type(f[3]) in (int, long):\n return \"%s%16x %s %x\" % tuple(f)\n else:\n f[3] = hexdump(f[3])\n return \"%s%16x %s %s\" % tuple(f)\n elif f[0] in ('N', 'n', '$'):\n if type(f[1]) in (int, long):\n return \"%s %x %16x\" % tuple(f)\n else:\n return \"%s'%s'\" % tuple(f)\n elif f[0] == '-':\n return \"%s %x\" % tuple(f)\n\n return hexdump(key)\n\n def prettyval(self, val):\n \"\"\"\n returns the value in a readable format.\n \"\"\"\n if len(val) == self.wordsize and val[-1:] in (b'\\x00', b'\\xff'):\n return \"%x\" % struct.unpack(\"<\" + self.fmt, val)\n if len(val) == self.wordsize and re.search(b'[\\x00-\\x08\\x0b\\x0c\\x0e-\\x1f]', val, re.DOTALL):\n return \"%x\" % struct.unpack(\"<\" + self.fmt, val)\n if len(val) < 2 or not re.match(b'^[\\x09\\x0a\\x0d\\x20-\\xff]+.$', val, re.DOTALL):\n return hexdump(val)\n val = val.replace(b\"\\n\", b\"\\\\n\")\n return \"'%s'\" % val.decode('utf-8', 'ignore')\n\n def nodeByName(self, name):\n \"\"\" Return a nodeid by name \"\"\"\n # note: really long names are encoded differently:\n # 'N'+'\\x00'+pack('Q', nameid) => ofs\n # and (ofs, 'N') -> nameid\n\n # at nodebase ( 0xFF000000, 'S', 0x100*nameid ) there is a series of blobs for max 0x80000 sized names.\n cur = self.btree.find('eq', self.namekey(name))\n if cur:\n return struct.unpack('<' + self.fmt, cur.getval())[0]\n\n def namekey(self, name):\n if type(name) in (int, long):\n return struct.pack(\" 1:\n # utf-8 encode the tag\n args = args[:1] + (args[1].encode('utf-8'),) + args[2:]\n\n if len(args) == 3 and type(args[-1]) == str:\n # node.tag.string type keys\n return struct.pack(self.keyfmt[:1 + len(args)], b'.', *args[:-1]) + args[-1].encode('utf-8')\n elif len(args) == 3 and type(args[-1]) == type(-1) and args[-1] < 0:\n # negative values -> need lowercase fmt char\n return struct.pack(self.keyfmt[:1 + len(args)] + self.fmt.lower(), b'.', *args)\n else:\n # node.tag.value type keys\n return struct.pack(self.keyfmt[:2 + len(args)], b'.', *args)\n\n def decodekey(self, key):\n \"\"\"\n splits a key in a tuple, one of:\n ( [ 'N', 'n', '$' ], 0, bignameid )\n ( [ 'N', 'n', '$' ], name )\n ( '-', id )\n ( '.', id )\n ( '.', id, tag )\n ( '.', id, tag, value )\n ( '.', id, 'H', name )\n \"\"\"\n if key[:1] in (b'n', b'N', b'$'):\n if key[1:2] == b\"\\x00\" and len(key) == 2 + self.wordsize:\n return struct.unpack(\">sB\" + self.fmt, key)\n else:\n return key[:1], key[1:].decode('utf-8', 'ignore')\n if key[:1] == b'-':\n return struct.unpack(\">s\" + self.fmt, key)\n if len(key) == 1 + self.wordsize:\n return struct.unpack(self.keyfmt[:3], key)\n if len(key) == 1 + self.wordsize + 1:\n return struct.unpack(self.keyfmt[:4], key)\n if len(key) == 1 + 2 * self.wordsize + 1:\n return struct.unpack(self.keyfmt[:5], key)\n if len(key) > 1 + self.wordsize + 1:\n f = struct.unpack_from(self.keyfmt[:4], key)\n return f + (key[2 + self.wordsize:], )\n raise Exception(\"unknown key format\")\n\n def bytes(self, *args):\n \"\"\" return a raw value for the given arguments \"\"\"\n if len(args) == 1 and isinstance(args[0], BTree.Cursor):\n cur = args[0]\n else:\n cur = self.btree.find('eq', self.makekey(*args))\n\n if cur:\n return cur.getval()\n\n def int(self, *args):\n \"\"\"\n Return the integer stored in the specified node.\n\n Any type of integer will be decoded: byte, short, long, long long\n\n \"\"\"\n data = self.bytes(*args)\n if data is not None:\n if len(data) == 1:\n return struct.unpack(\"\" + self.fmt, data, 1)\n nameblob = self.blob(self.nodebase, 'S', nameid * 256, nameid * 256 + 32)\n return nameblob.rstrip(b\"\\x00\").decode('utf-8')\n return data.rstrip(b\"\\x00\").decode('utf-8')\n\n def blob(self, nodeid, tag, start=0, end=0xFFFFFFFF):\n \"\"\"\n Blobs are stored in sequential nodes\n with increasing index values.\n\n most blobs, like scripts start at index\n 0, long names start at a specified\n offset.\n\n \"\"\"\n startkey = self.makekey(nodeid, tag, start)\n endkey = self.makekey(nodeid, tag, end)\n cur = self.btree.find('ge', startkey)\n data = b''\n while cur.getkey() <= endkey:\n data += cur.getval()\n cur.next()\n return data\n\n\nclass ID1File(object):\n \"\"\"\n Reads .id1 or 1.IDA files, containing byte flags\n\n This is basically the information for the .idc GetFlags(ea),\n FirstSeg(), NextSeg(ea), SegStart(ea), SegEnd(ea) functions\n \"\"\"\n INDEX = 1\n\n class SegInfo:\n def __init__(self, startea, endea, offset):\n self.startea = startea\n self.endea = endea\n self.offset = offset\n\n def __init__(self, idb, fh):\n if idb.magic == 'IDA2':\n wordsize, fmt = 8, \"Q\"\n else:\n wordsize, fmt = 4, \"L\"\n # todo: verify wordsize using the following heuristic:\n # L -> starting at: seglistofs + nsegs*seginfosize are all zero\n # L -> starting at seglistofs .. nsegs*seginfosize every even word must be unique\n\n self.fh = fh\n fh.seek(0)\n hdrdata = fh.read(32)\n magic = hdrdata[:4]\n if magic in (b'Va4\\x00', b'Va3\\x00', b'Va2\\x00', b'Va1\\x00', b'Va0\\x00'):\n nsegments, npages = struct.unpack_from(\" starting at: seglistofs + nsegs*seginfosize are all zero\n # L -> starting at seglistofs .. nsegs*seginfosize every even word must be unique\n\n def dump(self):\n \"\"\" print first and last bits for each segment \"\"\"\n for seg in self.seglist:\n print(\"==== %08x-%08x\" % (seg.startea, seg.endea))\n if seg.endea - seg.startea < 30:\n for ea in range(seg.startea, seg.endea):\n print(\" %08x: %08x\" % (ea, self.getFlags(ea)))\n else:\n for ea in range(seg.startea, seg.startea + 10):\n print(\" %08x: %08x\" % (ea, self.getFlags(ea)))\n print(\"...\")\n for ea in range(seg.endea - 10, seg.endea):\n print(\" %08x: %08x\" % (ea, self.getFlags(ea)))\n\n def find_segment(self, ea):\n \"\"\" do a linear search for the given address in the segment list \"\"\"\n for seg in self.seglist:\n if seg.startea <= ea < seg.endea:\n return seg\n\n def getFlags(self, ea):\n seg = self.find_segment(ea)\n if not seg:\n return 0\n self.fh.seek(seg.offset + 4 * (ea - seg.startea))\n return struct.unpack(\">= 1\n self.wordsize = wordsize\n self.wordfmt = fmt\n self.nnames = nnames\n self.pagesize = pagesize\n\n def dump(self):\n print(\"nam: nnames=%d, npages=%d, pagesize=%08x\" % (self.nnames, self.npages, self.pagesize))\n\n def allnames(self):\n self.fh.seek(self.pagesize)\n n = 0\n while n < self.nnames:\n data = self.fh.read(self.pagesize)\n want = min(self.nnames - n, int(self.pagesize / self.wordsize))\n ofslist = struct.unpack_from(\"<%d%s\" % (want, self.wordfmt), data, 0)\n for ea in ofslist:\n yield ea\n n += want\n\n\nclass SEGFile(object):\n \"\"\" reads .seg or $SEGS.IDA files. \"\"\"\n INDEX = 3\n\n def __init__(self, idb, fh):\n pass\n\n\nclass TILFile(object):\n \"\"\" reads .til files \"\"\"\n INDEX = 4\n\n def __init__(self, idb, fh):\n pass\n# note: v3 databases had a .reg instead of .til\n\n\nclass ID2File(object):\n \"\"\"\n Reads .id2 files\n\n ID2 sections contain packed data, resulting in tripples\n of unknown use.\n \"\"\"\n INDEX = 5\n\n def __init__(self, idb, fh):\n pass\n\n\nclass Struct:\n \"\"\"\n Decodes info for structures\n\n (structnode, N) = structname\n (structnode, D, address) = xref-type\n (structnode, M, 0) = packed struct info\n (structnode, S, 27) = packed value(addr, byte)\n \"\"\"\n class Member:\n \"\"\"\n (membernode, N) = struct.member-name\n (membernode, A, 3) = structid+1\n (membernode, A, 8) = \n (membernode, A, 11) = enumid+1\n (membernode, A, 16) = flag? -- 4:variable length flag?\n (membernode, S, 0x3000) = type (set with 'Y')\n (membernode, S, 0x3001) = names used in 'type'\n (membernode, S, 5) = array type?\n (membernode, S, 9) = offset-type\n (membernode, D, address) = xref-type\n (membernode, d, structid) = xref-type -- for sub-structs\n \"\"\"\n def __init__(self, id0, spec):\n self._id0 = id0\n self._nodeid = spec.nextword() + self._id0.nodebase\n self.skip = spec.nextword()\n self.size = spec.nextword()\n self.flags = spec.next32()\n self.props = spec.next32()\n self.ofs = None\n @cachedproperty\n def name(self): return self._id0.name(self._nodeid)\n @cachedproperty\n def enumid(self): return self._id0.int(self._nodeid, 'A', 11)\n @cachedproperty\n def stringtype(self): return self._id0.int(self._nodeid, 'A', 16)\n @cachedproperty\n def structid(self): return self._id0.int(self._nodeid, 'A', 3)\n @cachedproperty\n def comment(self, repeatable): return self._id0.string(self._nodeid, 'S', 1 if repeatable else 0)\n @cachedproperty\n def ptrinfo(self): return self._id0.bytes(self._nodeid, 'S', 9)\n @cachedproperty\n def typeinfo(self): return self._id0.bytes(self._nodeid, 'S', 0x3000)\n\n def __init__(self, id0, nodeid):\n self._id0 = id0\n self._nodeid = nodeid\n\n spec = self._id0.blob(self._nodeid, 'M')\n p = IdaUnpacker(self._id0.wordsize, spec)\n if self._id0.idaver >= 40:\n # 1 = SF_VAR, 2 = SF_UNION, 4 = SF_HASHUNI, 8 = SF_NOLIST, 0x10 = SF_TYPLIB, 0x20 = SF_HIDDEN, 0x40 = SF_FRAME, 0xF80 = SF_ALIGN, 0x1000 = SF_GHOST\n self.flags = p.next32()\n else:\n self.flags = 0\n\n nmembers = p.next32()\n\n self.members = []\n o = 0\n for i in range(nmembers):\n m = Struct.Member(self._id0, p)\n m.ofs = o\n o += m.size\n\n self.members.append(m)\n\n self.extra = []\n while not p.eof():\n self.extra.append(p.next32())\n\n @cachedproperty\n def comment(self, repeatable): return self._id0.string(self._nodeid, 'S', 1 if repeatable else 0)\n @cachedproperty\n def name(self): return self._id0.name(self._nodeid)\n\n def __iter__(self):\n for m in self.members:\n yield m\n\n\nclass Enum:\n \"\"\"\n (enumnode, N) = enum-name\n (enumnode, A, -1) = nr of values\n (enumnode, A, -3) = representation\n (enumnode, A, -5) = flags: bitfield, hidden, ...\n (enumnode, A, -8) = \n (enumnode, E, value) = valuenode + 1\n \n \"\"\"\n class Member:\n \"\"\"\n (membernode, N) = membername\n (membernode, A, -2) = enumnode + 1\n (membernode, A, -3) = member value\n \"\"\"\n def __init__(self, id0, nodeid):\n self._id0 = id0\n self._nodeid = nodeid\n\n @cachedproperty\n def value(self): return self._id0.int(self._nodeid, 'A', -3)\n @cachedproperty\n def comment(self, repeatable): return self._id0.string(self._nodeid, 'S', 1 if repeatable else 0)\n @cachedproperty\n def name(self): return self._id0.name(self._nodeid)\n\n def __init__(self, id0, nodeid):\n self._id0 = id0\n self._nodeid = nodeid\n\n @cachedproperty\n def count(self): return self._id0.int(self._nodeid, 'A', -1)\n @cachedproperty\n def representation(self): return self._id0.int(self._nodeid, 'A', -3)\n\n # flags>>3 -> width\n # flags&1 -> bitfield\n @cachedproperty\n def flags(self): return self._id0.int(self._nodeid, 'A', -5)\n\n @cachedproperty\n def comment(self, repeatable): return self._id0.string(self._nodeid, 'S', 1 if repeatable else 0)\n @cachedproperty\n def name(self): return self._id0.name(self._nodeid)\n\n def __iter__(self):\n startkey = self._id0.makekey(self._nodeid, 'E')\n endkey = self._id0.makekey(self._nodeid, 'F')\n cur = self._id0.btree.find('ge', startkey)\n while cur.getkey() < endkey:\n yield Enum.Member(self._id0, self._id0.int(cur) - 1)\n cur.next()\n\n\nclass Bitfield:\n class Member:\n def __init__(self, id0, nodeid):\n self._id0 = id0\n self._nodeid = nodeid\n\n @cachedproperty\n def value(self): return self._id0.int(self._nodeid, 'A', -3)\n @cachedproperty\n def mask(self): return self._id0.int(self._nodeid, 'A', -6) - 1\n @cachedproperty\n def comment(self, repeatable): return self._id0.string(self._nodeid, 'S', 1 if repeatable else 0)\n @cachedproperty\n def name(self): return self._id0.name(self._nodeid)\n\n class Mask:\n def __init__(self, id0, nodeid, mask):\n self._id0 = id0\n self._nodeid = nodeid\n self.mask = mask\n\n @cachedproperty\n def comment(self, repeatable): return self._id0.string(self._nodeid, 'S', 1 if repeatable else 0)\n @cachedproperty\n def name(self): return self._id0.name(self._nodeid)\n\n def __iter__(self):\n \"\"\"\n Enumerates all Masks\n \"\"\"\n startkey = self._id0.makekey(self._nodeid, 'E')\n endkey = self._id0.makekey(self._nodeid, 'F')\n cur = self._id0.btree.find('ge', startkey)\n while cur.getkey() < endkey:\n yield Bitfield.Member(self._id0, self._id0.int(cur) - 1)\n cur.next()\n\n\n def __init__(self, id0, nodeid):\n self._id0 = id0\n self._nodeid = nodeid\n\n @cachedproperty\n def count(self): return self._id0.int(self._nodeid, 'A', -1)\n @cachedproperty\n def representation(self): return self._id0.int(self._nodeid, 'A', -3)\n @cachedproperty\n def flags(self): return self._id0.int(self._nodeid, 'A', -5)\n\n @cachedproperty\n def comment(self, repeatable): return self._id0.string(self._nodeid, 'S', 1 if repeatable else 0)\n @cachedproperty\n def name(self): return self._id0.name(self._nodeid)\n\n def __iter__(self):\n \"\"\"\n Enumerates all Masks\n \"\"\"\n startkey = self._id0.makekey(self._nodeid, 'm')\n endkey = self._id0.makekey(self._nodeid, 'n')\n cur = self._id0.btree.find('ge', startkey)\n while cur.getkey() < endkey:\n key = self._id0.decodekey(cur.getkey())\n yield Bitfield.Mask(self._id0, self._id0.int(cur) - 1, key[-1])\n cur.next()\n\nclass IDBParams:\n def __init__(self, id0, data):\n self._id0 = id0\n magic, self.version, = struct.unpack_from(\"<3sH\", data, 0)\n if self.version<700:\n cpu, self.idpflags, self.demnames, self.filetype, self.coresize, self.corestart, self.ostype, self.apptype = struct.unpack_from(\"<8sBBH\" + (id0.fmt * 2) + \"HH\", data, 5)\n self.cpu = strz(cpu, 0)\n else:\n p = IdaUnpacker(id0.wordsize, data[5:])\n cpulen = p.next32()\n self.cpu = p.bytes(cpulen)\n genflags = p.next32()\n self.idpflags = p.next32()\n self.demnames = 0\n changecount = p.next32()\n self.filetype = p.next32()\n self.ostype = p.next32()\n self.apptype = p.next32()\n asmtype = p.next32()\n specsegs = p.next32()\n specsegs = p.next32()\n aflags = p.next32()\n aflags2 = p.next32()\n base = p.nextword()\n startss = p.nextword()\n startcs = p.nextword()\n startip = p.nextword()\n startea = p.nextword()\n startsp = p.nextword()\n main = p.nextword()\n minea = p.nextword()\n maxea = p.nextword()\n\n self.coresize = 0\n self.corestart = 0\n\nclass Script:\n def __init__(self, id0, nodeid):\n self._id0 = id0\n self._nodeid = nodeid\n\n @cachedproperty\n def name(self): return self._id0.string(self._nodeid, 'S', 0)\n @cachedproperty\n def language(self): return self._id0.string(self._nodeid, 'S', 1)\n @cachedproperty\n def body(self): return strz(self._id0.blob(self._nodeid, 'X'), 0)\n\nclass Segment:\n \"\"\"\n Decodes a value from \"$ segs\", see segment_t in segment.hpp for details.\n \"\"\"\n def __init__(self, id0, spec):\n self._id0 = id0\n p = IdaUnpacker(id0.wordsize, spec)\n self.startea = p.nextword()\n self.size = p.nextword()\n self.name_id = p.nextword()\n self.class_id = p.nextword()\n self.orgbase = p.nextword()\n self.unknown = p.next16()\n self.align = p.next8()\n self.comb = p.next8()\n self.perm = p.next8()\n self.bitness = p.next8()\n self.flags = p.next8()\n self.selector = p.nextword()\n self.defsr = [p.nextword() for _ in range(16)]\n self.color = p.next32()\n\n" }, { "path": "idbtool.py", "content": "#!/usr/bin/python3\n\"\"\"\nTool for querying information from Hexrays .idb and .i64 files\nwithout launching IDA.\n\nCopyright (c) 2016 Willem Hengeveld \n\"\"\"\n\n# todo:\n# '$ segs'\n# S = packed(startea, size, ....)\n# '$ srareas'\n# a = packed(startea, size, flag, flag) -- includes functions\n# b = packed(startea, size, flag, flag) -- segment\n# c = packed(startea, size, flag, flag) -- same as 'b'\n# \nfrom __future__ import division, print_function, absolute_import, unicode_literals\nimport sys\nimport os\nif sys.version_info[0] == 2:\n import scandir\n os.scandir = scandir.scandir\nif sys.version_info[0] == 2:\n reload(sys)\n sys.setdefaultencoding('utf-8')\n\nif sys.version_info[0] == 2:\n stdout = sys.stdout\nelse:\n stdout = sys.stdout.buffer\n\nimport struct\nimport binascii\nimport argparse\nimport itertools\nfrom collections import defaultdict\n\nimport re\n\nfrom datetime import datetime\n\nimport idblib\nfrom idblib import hexdump\n\n\ndef timestring(t):\n if t == 0:\n return \"....-..-.. ..:..:..\"\n return datetime.strftime(datetime.fromtimestamp(t), \"%Y-%m-%d %H:%M:%S\")\n\n\ndef strz(b, o):\n return b[o:b.find(b'\\x00', o)].decode('utf-8', 'ignore')\n\ndef nonefmt(fmt, num):\n if num is None:\n return \"-\"\n return fmt % num\n\n######### license encoding ################\n\n\ndef decryptuser(data):\n \"\"\"\n The '$ original user' node is encrypted with hexray's private key.\n Hence we can easily decrypt it, but not change it to something else.\n We can however copy the entry from another database, or just replace it with garbage.\n\n The node contains 128 bytes encrypted license, followed by 32 bytes zero.\n\n Note: i found several ida55 databases online where this does not work.\n possible these were created using a cracked version of IDA.\n \"\"\"\n data = int(binascii.b2a_hex(data[127::-1]), 16)\n user = pow(data, 0x13, 0x93AF7A8E3A6EB93D1B4D1FB7EC29299D2BC8F3CE5F84BFE88E47DDBDD5550C3CE3D2B16A2E2FBD0FBD919E8038BB05752EC92DD1498CB283AA087A93184F1DD9DD5D5DF7857322DFCD70890F814B58448071BBABB0FC8A7868B62EB29CC2664C8FE61DFBC5DB0EE8BF6ECF0B65250514576C4384582211896E5478F95C42FDED)\n user = binascii.a2b_hex(\"%0256x\" % user)\n return user[1:]\n\n\ndef licensestring(lic):\n \"\"\" decode a license blob \"\"\"\n if not lic:\n return\n if len(lic) < 127:\n print(\"too short license format: %s\" % binascii.b2a_hex(lic))\n return\n elif len(lic) > 127 and sum(lic[127:]) != 0:\n print(\"too long license format: %s\" % binascii.b2a_hex(lic))\n return\n\n if struct.unpack_from(\"= 128:\n user0 = decryptuser(user0)\n else:\n user0 = user0[:127]\n # user0 has 128 bytes rsa encrypted license, followed by 32 bytes zero\n print(\"orig: %s\" % licensestring(user0))\n # ida9 has S10+S11 == license json\n user10 = id0.blob(orignode, 'S', 16)\n if user10:\n import json\n user10 = json.loads(user10)\n print(\"orig: %s\" % user10)\n curnode = id0.nodeByName('$ user1')\n if curnode:\n user1 = id0.bytes(curnode, 'S', 0)\n print(\"user: %s\" % licensestring(user1))\n\n\n######### idb summary #########\n\n\nfiletypelist = [\n \"MS DOS EXE File\",\n \"MS DOS COM File\",\n \"Binary File\",\n \"MS DOS Driver\",\n \"New Executable (NE)\",\n \"Intel Hex Object File\",\n \"MOS Technology Hex Object File\",\n \"Linear Executable (LX)\",\n \"Linear Executable (LE)\",\n \"Netware Loadable Module (NLM)\",\n \"Common Object File Format (COFF)\",\n \"Portable Executable (PE)\",\n \"Object Module Format\",\n \"R-records\",\n \"ZIP file (this file is never loaded to IDA database)\",\n \"Library of OMF Modules\",\n \"ar library\",\n \"file is loaded using LOADER DLL\",\n \"Executable and Linkable Format (ELF)\",\n \"Watcom DOS32 Extender (W32RUN)\",\n \"Linux a.out (AOUT)\",\n \"PalmPilot program file\",\n \"MS DOS EXE File\",\n \"MS DOS COM File\",\n \"AIX ar library\",\n \"Mac OS X Mach-O file\",\n]\n\n\ndef dumpinfo(id0):\n \"\"\" print various infos on the idb file \"\"\"\n def ftstring(ft):\n if 0 < ft < len(filetypelist):\n return \"%02x:%s\" % (ft, filetypelist[ft])\n return \"%02x:unknown\" % ft\n\n def decodebitmask(fl, bitnames):\n l = []\n knownbits = 0\n for bit, name in enumerate(bitnames):\n if fl & (1 << bit) and name is not None:\n l.append(name)\n knownbits |= 1 << bit\n if fl & ~knownbits:\n l.append(\"unknown_%x\" % (fl & ~knownbits))\n return \",\".join(l)\n\n def osstring(fl):\n return decodebitmask(fl, ['msdos', 'win', 'os2', 'netw', 'unix', 'other'])\n\n def appstring(fl):\n return decodebitmask(fl, ['console', 'graphics', 'exe', 'dll', 'driver', '1thread', 'mthread', '16bit', '32bit', '64bit'])\n\n ldr = id0.nodeByName(\"$ loader name\")\n if ldr:\n print(\"loader: %s %s\" % (id0.string(ldr, 'S', 0), id0.string(ldr, 'S', 1)))\n\n if not id0.root:\n print(\"database has no RootNode\")\n return\n\n if id0.idbparams:\n params = idblib.IDBParams(id0, id0.idbparams)\n print(\"cpu: %s, version=%d, filetype=%s, ostype=%s, apptype=%s, core:%x, size:%x\" % (params.cpu, params.version, ftstring(params.filetype), osstring(params.ostype), appstring(params.apptype), params.corestart, params.coresize))\n\n print(\"idaver=%s: %s\" % (nonefmt(\"%04d\", id0.idaver), id0.idaverstr))\n\n srcmd5 = id0.originmd5\n print(\"nopens=%s, ctime=%s, crc=%s, md5=%s\" % (nonefmt(\"%d\", id0.nropens), nonefmt(\"%08x\", id0.creationtime), nonefmt(\"%08x\", id0.somecrc), hexdump(srcmd5) if srcmd5 else \"-\"))\n\n dumpuser(id0)\n\n\ndef dumpnames(args, id0, nam):\n for ea in nam.allnames():\n print(\"%08x: %s\" % (ea, id0.name(ea)))\n\n\ndef dumpscript(id0, node):\n \"\"\" dump all stored scripts \"\"\"\n s = idblib.Script(id0, node)\n\n print(\"======= %s %s =======\" % (s.language, s.name))\n print(s.body)\n\n\ndef dumpstructmember(m):\n \"\"\"\n Dump info for a struct member.\n \"\"\"\n print(\" %02x %02x %08x %02x: %-40s\" % (m.skip, m.size, m.flags, m.props, m.name), end=\"\")\n if m.enumid:\n print(\" enum %08x\" % m.enumid, end=\"\")\n if m.structid:\n print(\" struct %08x\" % m.structid, end=\"\")\n if m.ptrinfo:\n # packed\n # note: 64bit nrs are stored low32, high32\n # flags1, target, base, delta, flags2\n\n # flags1:\n # 0=off8 1=off16 2=off32 3=low8 4=low16 5=high8 6=high16 9=off64\n # 0x10 = targetaddr, 0x20 = baseaddr, 0x40 = delta, 0x80 = base is plainnum\n # flags2:\n # 1=image is off, 0x10 = subtract, 0x20 = signed operand\n print(\" ptr %s\" % m.ptrinfo, end=\"\")\n if m.typeinfo:\n print(\" type %s\" % m.typeinfo, end=\"\")\n print()\n\n\ndef dumpstruct(id0, node):\n \"\"\"\n dump all info for the struct defined by `node`\n \"\"\"\n s = idblib.Struct(id0, node)\n\n\n print(\"struct %s, 0x%x\" % (s.name, s.flags))\n for m in s:\n dumpstructmember(m)\n\ndef dumpbitmember(m):\n print(\" %08x %s\" % (m.value or 0, m.name))\ndef dumpmask(m):\n print(\" mask %08x %s\" % (m.mask, m.name))\n for m in m:\n dumpbitmember(m)\ndef dumpbitfield(id0, node):\n b = idblib.Bitfield(id0, node)\n print(\"bitfield %s, %s, %s, %s\" % (b.name, nonefmt(\"0x%x\", b.count), nonefmt(\"0x%x\", b.representation), nonefmt(\"0x%x\", b.flags)))\n for m in b:\n dumpmask(m)\n\ndef dumpenummember(m):\n \"\"\"\n Print information on a single enum member\n \"\"\"\n print(\" %08x %s\" % (m.value or 0, m.name))\n\ndef dumpenum(id0, node):\n \"\"\"\n Dump all info for the enum defined by `node`\n \"\"\"\n e = idblib.Enum(id0, node)\n if e.flags and e.flags&1:\n dumpbitfield(id0, node)\n return\n print(\"enum %s, %s, %s, %s\" % (e.name, nonefmt(\"0x%x\", e.count), nonefmt(\"0x%x\", e.representation), nonefmt(\"0x%x\", e.flags)))\n\n for m in e:\n dumpenummember(m)\n\n\ndef dumpimport(id0, node):\n # Note that '$ imports' is a list where the actual nodes\n # are stored in the list, therefore we add '1' to the node here.\n\n # first the named imports\n startkey = id0.makekey(node+1, 'S')\n endkey = id0.makekey(node+1, 'T')\n cur = id0.btree.find('ge', startkey)\n while cur.getkey() < endkey:\n txt = id0.string(cur)\n key = cur.getkey()\n ea = id0.decodekey(key)[3]\n print(\"%08x: %s\" % (ea, txt))\n cur.next()\n\n # then list the imports by ordinal\n startkey = id0.makekey(node+1, 'A')\n endkey = id0.makekey(node+1, 'B')\n cur = id0.btree.find('ge', startkey)\n while cur.getkey() < endkey:\n ordinal = id0.decodekey(cur.getkey())[3]\n ea = id0.int(cur)\n print(\"%08x: (ord%04d) %s\" % (ea, ordinal, id0.name(ea)))\n cur.next()\n\n\ndef enumlist(id0, listname, callback):\n \"\"\"\n Lists are all stored in a similar way.\n\n (listnode, 'N') = listname\n (listnode, 'A', -1) = list size <-- not for '$ scriptsnippets'\n (listnode, 'A', seqnr) = itemnode+1\n\n (listnode, 'Y', itemnode) = seqnr <-- only with '$ enums'\n\n (listnode, 'Y', 0) = list size <-- only '$ scriptsnippets'\n (listnode, 'Y', 1) = ? <-- only '$ scriptsnippets'\n\n (listnode, 'S', seqnr) = dllname <-- only '$ imports'\n\n \"\"\"\n listnode = id0.nodeByName(listname)\n if not listnode:\n return\n\n startkey = id0.makekey(listnode, 'A')\n endkey = id0.makekey(listnode, 'A', 0xFFFFFFFF)\n cur = id0.btree.find('ge', startkey)\n while cur.getkey() < endkey:\n item = id0.int(cur)\n callback(id0, item - 1)\n cur.next()\n\n\ndef listfuncdirs(id0):\n listnode = id0.nodeByName('$ dirtree/funcs')\n if not listnode:\n return\n\n dir_id = 0\n while True:\n start = dir_id * 0x10000\n end = start + 0xFFFF\n data = id0.blob(listnode, 'S', start, end)\n if data == b'':\n break\n dumpfuncdir(id0, dir_id, data)\n dir_id += 1\n\n\ndef dumpfuncdir(id0, dir_index, data):\n terminate = data.find(b'\\0', 1)\n name = data[1:terminate].decode('utf-8')\n\n p = idblib.IdaUnpacker(id0.wordsize, data[terminate+1:])\n parent = p.nextword()\n unk = p.next32()\n \n if data[0] == 0: # IDA 7.5\n subdir_count = p.next32()\n subdirs = []\n while subdir_count:\n subdir_id = p.nextwordsigned()\n if subdirs:\n subdir_id = subdirs[-1] + subdir_id\n subdirs.append(subdir_id)\n subdir_count -= 1\n\n func_count = p.next32()\n funcs = []\n while func_count:\n func_id = p.nextwordsigned()\n if funcs:\n func_id = funcs[-1] + func_id\n funcs.append(func_id)\n func_count -= 1\n\n elif data[0] == 1: # IDA 7.6\n children_count = p.next32()\n children = []\n for i in range(children_count):\n next_child = p.nextwordsigned()\n if children:\n next_child += children[-1]\n children.append(next_child)\n\n subdir_count = p.next32()\n children_count -= subdir_count\n childtype_counts = [subdir_count]\n while children_count:\n childtype_count = p.next32()\n children_count -= childtype_count\n childtype_counts.append(childtype_count)\n\n subdirs = []\n funcs = []\n i = 0\n parsing_subdirs = True # switch back and forth\n for childtype_count in childtype_counts:\n for _ in range(childtype_count):\n if parsing_subdirs:\n subdirs.append(children[i])\n else:\n funcs.append(children[i])\n i += 1\n parsing_subdirs = not parsing_subdirs\n else:\n raise NotImplementedError('unsupported funcdir schema')\n\n if not p.eof():\n raise Exception('not EOF after dir parsed')\n\n print(\"dir %d = %s\" % (dir_index, name))\n print(\" parent = %d\" % parent)\n print(\" subdirs:\")\n for subdir in subdirs:\n print(\" %d\" % subdir)\n print(\" functions:\")\n for func in funcs:\n print(\" 0x%x\" % func)\n\n\ndef printent(args, id0, c):\n if args.verbose:\n print(\"%s = %s\" % (id0.prettykey(c.getkey()), id0.prettyval(c.getval())))\n else:\n print(\"%s = %s\" % (hexdump(c.getkey()), hexdump(c.getval())))\n\n\ndef createkey(args, id0, base, tag, ix):\n \"\"\"\n\n parse base node specification:\n\n '?' -> explicit N key\n '#' -> relative to nodebase\n '.' -> absolute nodeid\n\n '' -> lookup by name.\n\n \"\"\"\n if base[:1] == '?':\n return id0.namekey(base[1:])\n\n if re.match(r'^#(?:0[xX][0-9a-fA-F]+|\\d+)$', base):\n nodeid = int(base[1:], 0) + id0.nodebase\n elif re.match(r'^\\.(?:0[xX][0-9a-fA-F]+|\\d+)$', base):\n nodeid = int(base[1:], 0)\n else:\n nodeid = id0.nodeByName(base)\n if nodeid and args.verbose > 1:\n print(\"found node %x for %s\" % (nodeid, base))\n if nodeid is None:\n print(\"Could not find '%s'\" % base)\n return\n\n s = [nodeid]\n if tag is not None:\n s.append(tag)\n if ix is not None:\n try:\n ix = int(ix, 0)\n except:\n pass\n s.append(ix)\n\n return id0.makekey(*s)\n\n\ndef enumeratecursor(args, c, onerec, callback):\n \"\"\"\n Enumerate cursor in direction specified by `--dec` or `--inc`,\n taking into account the optional limit set by `--limit`\n\n Output according to verbosity level set by `--verbose`.\n \"\"\"\n limit = args.limit\n while c and not c.eof() and (limit is None or limit > 0):\n callback(c)\n if args.dec:\n c.prev()\n else:\n c.next()\n if limit is not None:\n limit -= 1\n elif onerec:\n break\n\n\ndef id0query(args, id0, query):\n \"\"\"\n queries start with an optional operator: <,<=,>,>=,==\n\n followed by either a name or address or nodeid\n\n Addresses are specified as a sequence of hexadecimal charaters.\n Nodeid's may be specified either as the full node id, starting with ff00,\n or starting with a '_'\n Names are anything which can be found under the name tree in the database.\n\n after the name/addr/node there is optionally a slash, followed by a node tag,\n and another slash, followed by a index or hash string.\n\n \"\"\"\n\n xlatop = {'=': 'eq', '==': 'eq', '>': 'gt', '<': 'lt', '>=': 'ge', '<=': 'le'}\n\n SEP = r\";\"\n m = re.match(r'^([=<>]=?)?(.+?)(?:' + SEP + r'(\\w+)(?:' + SEP + r'(.+))?)?$', query)\n op = m.group(1) or \"==\"\n base = m.group(2)\n tag = m.group(3) # optional ;tag\n ix = m.group(4) # optional ;ix\n\n op = xlatop[op]\n\n c = id0.btree.find(op, createkey(args, id0, base, tag, ix))\n\n enumeratecursor(args, c, op=='eq', lambda c:printent(args, id0, c))\n\n\ndef getsegs(id0):\n \"\"\"\n Returns a list of all segments.\n \"\"\"\n seglist = []\n node = id0.nodeByName('$ segs')\n if not node:\n return\n startkey = id0.makekey(node, 'S')\n endkey = id0.makekey(node, 'T')\n cur = id0.btree.find('ge', startkey)\n while cur.getkey() < endkey:\n s = idblib.Segment(id0, cur.getval())\n seglist.append(s)\n cur.next()\n\n return seglist\n\n\ndef listsegments(id0):\n \"\"\"\n Print a summary of all segments found in the IDB.\n \"\"\"\n ssnode = id0.nodeByName('$ segstrings')\n if not ssnode:\n print(\"can't find '$ segstrings' node\")\n return\n segstrings = id0.blob(ssnode, 'S')\n p = idblib.IdaUnpacker(id0.wordsize, segstrings)\n unk = p.next32()\n nextid = p.next32()\n slist = []\n while not p.eof():\n slen = p.next32()\n if slen is None:\n break\n name = p.bytes(slen)\n if name is None:\n break\n slist.append(name.decode('utf-8', 'ignore'))\n\n segs = getsegs(id0)\n for s in segs:\n print(\"%08x - %08x %s\" % (s.startea, s.startea+s.size, slist[s.name_id-1]))\n\ndef classifynodes(args, id0):\n \"\"\"\n Attempt to classify all nodes in the IDA database.\n\n Note: this does not work for very old dbs\n \"\"\"\n nodetype = {}\n tagstats = defaultdict(lambda : defaultdict(int))\n\n segs = getsegs(id0)\n\n print(\"node: %x .. %x\" % (id0.nodebase, id0.maxnode))\n\n def addstat(nodetype, k):\n if len(k)<3:\n print(\"??? strange, expected longer key - %s\" % k)\n return\n tag = k[2].decode('utf-8')\n if len(k)==3:\n tagstats[nodetype][(tag, )] += 1\n elif len(k)==4:\n value = k[3]\n if type(value)==int:\n if isaddress(value):\n tagstats[nodetype][(tag, 'addr')] += 1\n elif isnode(value):\n tagstats[nodetype][(tag, 'node')] += 1\n else:\n if value >= id0.maxnode:\n value -= pow(0x100, id0.wordsize)\n tagstats[nodetype][(tag, value)] += 1\n else:\n tagstats[nodetype][(tag, 'string')] += 1\n else:\n print(\"??? strange, expected shorter key - %s\" % k)\n return\n\n def isaddress(addr):\n for s in segs:\n if s.startea <= addr < s.startea+s.size:\n return True\n\n def isnode(addr):\n return id0.nodebase <= addr <= id0.maxnode\n\n def processbitfieldvalue(v):\n nodetype[v._nodeid] = 'bitfieldvalue'\n\n def processbitfieldmask(m):\n nodetype[m._nodeid] = 'bitfieldmask'\n\n for m in m:\n processbitfieldvalue(m)\n\n def processbitfield(id0, node):\n nodetype[node] = 'bitfield'\n\n b = idblib.Bitfield(id0, node)\n for m in b:\n processbitfieldmask(m)\n\n\n def processenummember(m):\n nodetype[m._nodeid] = 'enummember'\n\n def processenums(id0, node):\n nodetype[node] = 'enum'\n\n e = idblib.Enum(id0, node)\n if e.flags&1:\n processbitfield(id0, node)\n return\n\n for m in e:\n processenummember(m)\n\n def processstructmember(m, typename):\n nodetype[m._nodeid] = typename\n\n def processstructs(id0, node, typename):\n nodetype[node] = typename\n s = idblib.Struct(id0, node)\n\n for m in s:\n processstructmember(m, typename+\"member\")\n\n def processscripts(id0, node):\n nodetype[node] = 'script'\n\n def processaddr(id0, cur):\n k = id0.decodekey(cur.getkey())\n if len(k)==4 and k[2:4] == (b'A', 2):\n nodetype[id0.int(cur)-1] = 'hexrays'\n\n addstat('addr', k)\n\n def processfunc(id0, funcspec):\n p = idblib.IdaUnpacker(id0.wordsize, funcspec)\n\n funcstart = p.nextword()\n funcsize = p.nextword()\n flags = p.next16()\n if flags is None:\n return\n if flags&0x8000: # is tail\n return\n\n node = p.nextword()\n\n if node<0xFFFFFF and node!=0:\n processstructs(id0, node + id0.nodebase, \"frame\")\n\n def processimport(id0, node):\n print(\"imp %08x\" % node)\n startkey = id0.makekey(node+1, 'A')\n endkey = id0.makekey(node+1, 'B')\n cur = id0.btree.find('ge', startkey)\n while cur.getkey() < endkey:\n dllnode = id0.int(cur)\n nodetype[dllnode] = 'import'\n cur.next()\n\n\n # mark enums, structs, scripts.\n enumlist(id0, '$ enums', processenums)\n enumlist(id0, '$ structs', lambda id0, node : processstructs(id0, node, \"struct\"))\n enumlist(id0, '$ scriptsnippets', processscripts)\n enumlist(id0, '$ imports', processimport)\n\n # enum functions, scan for stackframes\n funcsnode = id0.nodeByName('$ funcs')\n startkey = id0.makekey(funcsnode, 'S')\n endkey = id0.makekey(funcsnode, 'T')\n cur = id0.btree.find('ge', startkey)\n while cur.getkey() < endkey:\n processfunc(id0, cur.getval())\n cur.next()\n\n clinode = id0.nodeByName('$ cli')\n if clinode:\n for letter in \"ABCDEFGHIJKMcio\":\n startkey = id0.makekey(clinode, letter)\n endkey = id0.makekey(clinode, chr(ord(letter)+1))\n cur = id0.btree.find('ge', startkey)\n while cur.getkey() < endkey:\n nodetype[id0.int(cur)] = 'cli.'+letter\n cur.next()\n\n\n # enum addresses, scan for hex-rays nodes\n startkey = b'.'\n endkey = id0.makekey(id0.nodebase)\n cur = id0.btree.find('ge', startkey)\n while cur.getkey() < endkey:\n processaddr(id0, cur)\n cur.next()\n\n # addresses above node list\n startkey = id0.makekey(id0.maxnode+1)\n endkey = b'/'\n cur = id0.btree.find('ge', startkey)\n while cur.getkey() < endkey:\n processaddr(id0, cur)\n cur.next()\n\n # scan for unmarked nodes\n # $ fr[0-9a-f]+\\.\\w+\n # $ fr[0-9a-f]+\\. [rs]\n # $ F[0-9A-F]+\\.\\w+\n # $ Stack of \\w+\n # Stack[0000007C]\n # xrefs to \\w+\n\n startkey = id0.makekey(id0.nodebase)\n endkey = id0.makekey(id0.maxnode+1)\n cur = id0.btree.find('ge', startkey)\n while cur.getkey() < endkey:\n k = id0.decodekey(cur.getkey())\n node = k[1]\n if node not in nodetype:\n nodetype[node] = \"unknown\"\n if nodetype[node] == \"unknown\" and k[2] == b'N':\n name = cur.getval().rstrip(b'\\x00')\n if re.match(br'\\$ fr[0-9a-f]+\\.\\w+$', name):\n name = 'fr-type-functionframe'\n elif re.match(br'\\$ fr[0-9a-f]+\\. [rs]$', name):\n name = 'fr-type-functionframe'\n elif re.match(br'\\$ F[0-9A-F]+\\.\\w+$', name):\n name = 'F-type-functionframe'\n elif name.startswith(b'Stack of '):\n name = 'stack-type-functionframe'\n elif name.startswith(b'Stack['):\n name = 'old-stack-type-functionframe'\n elif name.startswith(b'xrefs to '):\n name = 'old-xrefs'\n else:\n name = name.decode('utf-8', 'ignore')\n nodetype[node] = name\n\n cur.next()\n\n # output node classification\n if args.verbose:\n for k, v in sorted(nodetype.items(), key=lambda kv:kv[0]):\n print(\"%08x: %s\" % (k, v))\n\n # summarize tags per nodetype\n startkey = id0.makekey(id0.nodebase)\n endkey = id0.makekey(id0.maxnode+1)\n cur = id0.btree.find('ge', startkey)\n while cur.getkey() < endkey:\n k = id0.decodekey(cur.getkey())\n node = k[1]\n nt = nodetype[node]\n\n addstat(nt, k)\n\n cur.next()\n\n # output tag statistics\n for nt, ntstats in sorted(tagstats.items(), key=lambda kv:kv[0]):\n print(\"====== %s =====\" % nt)\n for k, v in ntstats.items():\n if len(k)==1:\n print(\"%5d - %s\" % (v, k[0]))\n elif len(k)==2 and type(k[1])==type(1):\n print(\"%5d - %s %8x\" % (v, k[0], k[1]))\n elif type(k[1])==type(1):\n print(\"%5d - %s %8x %s\" % (v, k[0], k[1], k[2:]))\n else:\n print(\"%5d - %s %s %s\" % (v, k[0], k[1], k[2:]))\n\n\ndef processid0(args, id0):\n if args.info:\n dumpinfo(id0)\n\n if args.pagedump:\n id0.btree.pagedump()\n\n if args.query:\n for query in args.query:\n id0query(args, id0, query)\n elif args.id0:\n id0.btree.dump()\n elif args.inc:\n c = id0.btree.find('ge', b'')\n enumeratecursor(args, c, False, lambda c:printent(args, id0, c))\n elif args.dec:\n c = id0.btree.find('le', b'\\x80')\n enumeratecursor(args, c, False, lambda c:printent(args, id0, c))\n\n\ndef hexascdumprange(id1, a, b):\n line = asc = \"\"\n for ea in range(a, b):\n if len(line)==0:\n line = \"%08x:\" % ea\n byte = id1.getFlags(ea)&0xFF\n line += \" %02x\" % byte\n asc += chr(byte) if 32 1:\n print(\"magic=%s, filever=%d\" % (idb.magic, idb.fileversion))\n for i in range(6):\n comp, ofs, size, checksum = idb.getsectioninfo(i)\n if ofs:\n part = idb.getpart(i)\n print(\"%2d: %02x, %08x %8x [%08x]: %s\" % (i, comp, ofs, size, checksum, hexdump(part.read(256))))\n\n nam = idb.getsection(idblib.NAMFile)\n id0 = idb.getsection(idblib.ID0File)\n id1 = idb.getsection(idblib.ID1File)\n processid0(args, id0)\n processid1(args, id1)\n processid2(args, idb.getsection(idblib.ID2File))\n processnam(args, nam)\n processtil(args, idb.getsection(idblib.TILFile))\n processseg(args, idb.getsection(idblib.SEGFile))\n\n if args.names:\n dumpnames(args, id0, nam)\n if args.classify:\n classifynodes(args, id0)\n\n if args.scripts:\n enumlist(id0, '$ scriptsnippets', dumpscript)\n if args.structs:\n enumlist(id0, '$ structs', dumpstruct)\n if args.enums:\n enumlist(id0, '$ enums', dumpenum)\n if args.funcdirs:\n listfuncdirs(id0)\n if args.imports:\n enumlist(id0, '$ imports', dumpimport)\n if args.segs:\n listsegments(id0)\n\n\ndef processfile(args, filetypehint, fh):\n class DummyIDB:\n def __init__(idb, args):\n if args.i64:\n idb.magic = 'IDA2'\n elif args.i32:\n idb.magic = 'IDA1'\n else:\n idb.magic = None\n\n try:\n magic = fh.read(64)\n fh.seek(-64, 1)\n if magic.startswith(b\"Va\") or magic.startswith(b\"VA\"):\n idb = DummyIDB(args)\n if filetypehint == 'id1':\n processid1(args, idblib.ID1File(idb, fh))\n elif filetypehint == 'nam':\n processnam(args, idblib.NAMFile(idb, fh))\n elif filetypehint == 'seg':\n processseg(args, idblib.SEGFile(idb, fh))\n else:\n print(\"unknown VA type file: %s\" % hexdump(magic))\n elif magic.startswith(b\"IDAS\"):\n processid2(args, idblib.ID2File(DummyIDB(args), fh))\n elif magic.startswith(b\"IDATIL\"):\n processtil(args, idblib.ID2File(DummyIDB(args), fh))\n elif magic.startswith(b\"IDA\"):\n processidb(args, idblib.IDBFile(fh))\n elif magic.find(b'B-tree v') > 0:\n processid0(args, idblib.ID0File(DummyIDB(args), fh))\n\n except Exception as e:\n print(\"ERROR %s\" % e)\n if args.debug:\n raise\n\n\ndef recover_database(args, basepath, dbfiles):\n processidb(args, idblib.RecoverIDBFile(args, basepath, dbfiles))\n\n\ndef DirEnumerator(args, path):\n \"\"\"\n Enumerate all files / links in a directory,\n optionally recursing into subdirectories,\n or ignoring links.\n \"\"\"\n for d in os.scandir(path):\n try:\n if d.name == '.' or d.name == '..':\n pass\n elif d.is_symlink() and args.skiplinks:\n pass\n elif d.is_file():\n yield d.path\n elif d.is_dir() and args.recurse:\n for f in DirEnumerator(args, d.path):\n yield f\n except Exception as e:\n print(\"EXCEPTION %s accessing %s/%s\" % (e, path, d.name))\n\n\ndef EnumeratePaths(args, paths):\n \"\"\"\n Enumerate all paths, files from the commandline\n optionally recursing into subdirectories.\n \"\"\"\n for fn in paths:\n try:\n # 3 - for ftp://, 4 for http://, 5 for https://\n if fn.find(\"://\") in (3, 4, 5):\n yield fn\n if os.path.islink(fn) and args.skiplinks:\n pass\n elif os.path.isdir(fn) and args.recurse:\n for f in DirEnumerator(args, fn):\n yield f\n elif os.path.isfile(fn):\n yield fn\n except Exception as e:\n print(\"EXCEPTION %s accessing %s\" % (e, fn))\n\n\ndef filetype_from_name(fn):\n i = max(fn.rfind('.'), fn.rfind('/'))\n return fn[i + 1:].lower()\n\n\ndef isv2name(name):\n return name.lower() in ('$segregs.ida', '$segs.ida', '0.ida', '1.ida', 'ida.idl', 'names.ida')\n\n\ndef isv3ext(ext):\n return ext.lower() in ('.id0', '.id1', '.id2', '.nam', '.til')\n\n\ndef xlatv2name(name):\n oldnames = {\n '$segregs.ida': 'reg',\n '$segs.ida': 'seg',\n '0.ida': 'id0',\n '1.ida': 'id1',\n 'ida.idl': 'idl',\n 'names.ida': 'nam',\n }\n\n return oldnames.get(name.lower())\n\n\ndef main():\n parser = argparse.ArgumentParser(description='idbtool - print info from hex-rays IDA .idb and .i64 files',\n formatter_class=argparse.RawDescriptionHelpFormatter,\n epilog=\"\"\"\nidbtool can process complete .idb and .i64 files, but also naked .id0, .id1, .nam, .til files.\nAll versions since IDA v2.0 are supported.\n\nQueries start with an optional operator: <,<=,>,>=,==.\nFollowed by either a name or address or nodeid.\nAddresses are specified as a sequence of hexadecimal charaters.\nNodeid's may be specified either as the full node id, starting with ff00,\nor starting with a '_'.\nNames are anything which can be found under the name tree in the database.\n\nAfter the name/addr/node there is optionally a slash, followed by a node tag,\nand another slash, followed by a index or hash string.\n\nMultiple queries can be specified, terminated by another option, or `--`.\nAdd `-v` for pretty printed keys and values.\n\nExamples:\n\n idbtool -v --query \"$ user1;S;0\" -- x.idb\n idbtool -v --limit 4 --query \">#0xa\" -- x.idb\n idbtool -v --limit 5 --query \">Root Node;S;0\" -- x.idb\n idbtool -v --limit 10 --query \">Root Node;S\" -- x.idb\n idbtool -v --query \".0xff000001;N\" -- x.idb\n\"\"\")\n parser.add_argument('--verbose', '-v', action='count', default=0)\n parser.add_argument('--recurse', '-r', action='store_true', help='recurse into directories')\n parser.add_argument('--skiplinks', '-L', action='store_true', help='skip symbolic links')\n parser.add_argument('--filetype', '-t', type=str, help='specify filetype when loading `naked` id1,nam or seg files')\n parser.add_argument('--i64', '-i64', action='store_true', help='specify that `naked` file is from a 64 bit database')\n parser.add_argument('--i32', '-i32', action='store_true', help='specify that `naked` file is from a 32 bit database')\n\n parser.add_argument('--names', '-n', action='store_true', help='print names')\n parser.add_argument('--scripts', '-s', action='store_true', help='print scripts')\n parser.add_argument('--structs', '-u', action='store_true', help='print structs')\n # parser.add_argument('--comments', '-c', action='store_true', help='print comments')\n parser.add_argument('--enums', '-e', action='store_true', help='print enums and bitfields')\n parser.add_argument('--imports', action='store_true', help='print imports')\n parser.add_argument('--segs', action='store_true', help='print segments')\n parser.add_argument('--funcdirs', action='store_true', help='print function dirs (folders)')\n parser.add_argument('--info', '-i', action='store_true', help='database info')\n parser.add_argument('--inc', action='store_true', help='dump id0 records by cursor increment')\n parser.add_argument('--dec', action='store_true', help='dump id0 records by cursor decrement')\n parser.add_argument('--id0', \"-id0\", action='store_true', help='dump id0 records, by walking the page tree')\n parser.add_argument('--id1', \"-id1\", action='store_true', help='dump id1 records')\n parser.add_argument('--dump', type=str, help='hexdump id1 bytes', metavar='FROM-UNTIL')\n parser.add_argument('--dumpraw', type=str, help='output id1 bytes', metavar='FROM-UNTIL')\n parser.add_argument('--pagedump', \"-d\", action='store_true', help='dump all btree pages, including any that might have become inaccessible due to datacorruption.')\n parser.add_argument('--classify', action='store_true', help='Classify nodes found in the database.')\n\n parser.add_argument('--query', \"-q\", type=str, nargs='*', help='search the id0 file for a specific record.')\n parser.add_argument('--limit', '-m', type=int, help='Max nr of records to return for a query.')\n\n parser.add_argument('--recover', action='store_true', help='recover idb from unpacked files, of v2 database')\n parser.add_argument('--debug', action='store_true')\n\n parser.add_argument('FILES', type=str, nargs='*', help='Files')\n\n args = parser.parse_args()\n\n if args.FILES:\n dbs = dict()\n\n for fn in EnumeratePaths(args, args.FILES):\n basepath, filename = os.path.split(fn)\n if isv2name(filename):\n d = dbs.setdefault(basepath, dict())\n d[xlatv2name(filename)] = fn\n print(\"%s -> %s : %s\" % (xlatv2name(filename), basepath, filename))\n else:\n basepath, ext = os.path.splitext(fn)\n if isv3ext(ext):\n d = dbs.setdefault(basepath, dict())\n d[ext.lower()] = fn\n\n if not args.dumpraw:\n print(\"\\n==> \" + fn + \" <==\\n\")\n\n try:\n filetype = args.filetype or filetype_from_name(fn)\n with open(fn, \"rb\") as fh:\n processfile(args, filetype, fh)\n except Exception as e:\n print(\"ERROR: %s\" % e)\n if args.debug:\n raise\n\n if args.recover:\n for basepath, dbfiles in dbs.items():\n if len(dbfiles) > 1:\n try:\n print(\"\\n==> \" + basepath + \" <==\\n\")\n recover_database(args, basepath, dbfiles)\n except Exception as e:\n print(\"ERROR: %s\" % e)\n else:\n print(\"==> STDIN <==\")\n processfile(args, args.filetype, sys.stdin.buffer)\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "setup.cfg", "content": "[flake8]\nignore = E402,E501,E731\n\n" }, { "path": "test_idblib.py", "content": "import unittest\nfrom idblib import FileSection, binary_search, makeStringIO\n\n\nclass TestFileSection(unittest.TestCase):\n \"\"\" unittest for FileSection object \"\"\"\n def test_file(self):\n s = makeStringIO(b\"0123456789abcdef\")\n fh = FileSection(s, 3, 11)\n self.assertEqual(fh.read(3), b\"345\")\n self.assertEqual(fh.read(8), b\"6789a\")\n self.assertEqual(fh.read(8), b\"\")\n\n fh.seek(-1, 2)\n self.assertEqual(fh.read(8), b\"a\")\n fh.seek(3)\n self.assertEqual(fh.read(2), b\"67\")\n fh.seek(-2, 1)\n self.assertEqual(fh.read(2), b\"67\")\n fh.seek(2, 1)\n self.assertEqual(fh.read(2), b\"a\")\n\n fh.seek(8)\n self.assertEqual(fh.read(1), b\"\")\n with self.assertRaises(Exception):\n fh.seek(9)\n\n\nclass TestBinarySearch(unittest.TestCase):\n \"\"\" unittests for binary_search \"\"\"\n class Object:\n def __init__(self, num):\n self.key = num\n\n def __repr__(self):\n return \"o(%d)\" % self.num\n\n def test_bs(self):\n obj = self.Object\n lst = [obj(_) for _ in (2, 3, 5, 6)]\n self.assertEqual(binary_search(lst, 1), -1)\n self.assertEqual(binary_search(lst, 2), 0)\n self.assertEqual(binary_search(lst, 3), 1)\n self.assertEqual(binary_search(lst, 4), 1)\n self.assertEqual(binary_search(lst, 5), 2)\n self.assertEqual(binary_search(lst, 6), 3)\n self.assertEqual(binary_search(lst, 7), 3)\n\n def test_emptylist(self):\n obj = self.Object\n lst = []\n self.assertEqual(binary_search(lst, 1), -1)\n\n def test_oneelem(self):\n obj = self.Object\n lst = [obj(1)]\n self.assertEqual(binary_search(lst, 0), -1)\n self.assertEqual(binary_search(lst, 1), 0)\n self.assertEqual(binary_search(lst, 2), 0)\n\n def test_twoelem(self):\n obj = self.Object\n lst = [obj(1), obj(3)]\n self.assertEqual(binary_search(lst, 0), -1)\n self.assertEqual(binary_search(lst, 1), 0)\n self.assertEqual(binary_search(lst, 2), 0)\n self.assertEqual(binary_search(lst, 3), 1)\n self.assertEqual(binary_search(lst, 4), 1)\n\n def test_listsize(self):\n obj = self.Object\n for l in range(3, 32):\n lst = [obj(_ + 1) for _ in range(l)]\n lst = lst[:1] + lst[2:]\n self.assertEqual(binary_search(lst, 0), -1)\n self.assertEqual(binary_search(lst, 1), 0)\n self.assertEqual(binary_search(lst, 2), 0)\n self.assertEqual(binary_search(lst, 3), 1)\n self.assertEqual(binary_search(lst, l - 1), l - 3)\n self.assertEqual(binary_search(lst, l), l - 2)\n self.assertEqual(binary_search(lst, l + 1), l - 2)\n self.assertEqual(binary_search(lst, l + 2), l - 2)\n" }, { "path": "tree-walking.py", "content": "\"\"\"\nCopyright (c) 2016 Willem Hengeveld \n\nExperiment in btree walking\n\n\n *-------->[00]\n *------>[02]---+ [01]\nroot ->[08]---+ [05]-+ |\n [17]-+ | | +--->[03]\n | | | [04]\n | | |\n | | +----->[06]\n | | [07]\n | |\n | | *-------->[09]\n | +->[11]---+ [10]\n | [14]-+ |\n | | +--->[12]\n | | [13]\n | |\n | +----->[15]\n | [16]\n |\n | *-------->[18]\n +--->[20]---+ [19]\n [23]-+ |\n | +--->[21]\n | [22]\n |\n +----->[24]\n [25]\n\n\ndecrement from 08 : ix-- -> getpage, ix=len-1 -> getpage -> ix=len-1\ndecrement from 17 : ix-- -> getpage, ix=len-1 -> getpage -> ix=len-1\ndecrement from 02 : ix-- -> getpage, ix=len-1\ndecrement from 05 : ix-- -> getpage, ix=len-1\n\ndecrement from 01 : ix-- -> ix>=0 -> use key at ix\ndecrement from 03 : ix-- -> <0 -> pop -> ix>=0 -> use key at ix\ndecrement from 09 : ix-- -> <0 -> pop -> ix<0 -> pop -> ix>=0 -> use key at ix\n\nincrement from 09 : ix++\nincrement from 10 : ix++ -> ix==len(index) -> pop: ix==-1 -> ix++ -> ix==0 -> use\nincrement from 11 : recurse, ix=0 -> use\nincrement from 08 : recurse, ix=-1 -> recurse, ix=0 -> use\nincrement from 07 : ix++ -> ix==len(index) -> pop, ix++ -> ix==len -> pop -> ix++ -> ix==0 -> use\n\"\"\"\nfrom __future__ import division, print_function, absolute_import, unicode_literals\n\n# shape of the tree\n# a <2,2> tree is basically like the tree pictured in the ascii art above.\nTREEDEPTH = 2\nNODEWIDTH = 2\n\n\ndef binary_search(a, k):\n # c++: a.upperbound(k)--\n first, last = 0, len(a)\n while first < last:\n mid = (first + last) >> 1\n if k < a[mid].key:\n last = mid\n else:\n first = mid + 1\n return first - 1\n\n\nclass Entry(object):\n \"\"\"\n a key/value entry from a b-tree page\n \"\"\"\n def __init__(self, key, val):\n self.key = key\n self.val = val\n\n def __repr__(self):\n return \"%s=%d\" % (self.key, self.val)\n\n\nclass BasePage(object):\n \"\"\"\n BasePage has methods common to both leaf and index pages\n \"\"\"\n def __init__(self, kv):\n self.index = []\n for k, v in kv:\n self.index.append(Entry(k, v))\n\n def find(self, key):\n i = binary_search(self.index, key)\n if i < 0:\n if self.isindex():\n return ('recurse', -1)\n return ('gt', 0)\n if self.index[i].key == key:\n return ('eq', i)\n if self.isindex():\n return ('recurse', i)\n return ('lt', i)\n\n def getkey(self, ix):\n return self.index[ix].key\n\n def getval(self, ix):\n return self.index[ix].val\n\n def isleaf(self):\n return self.preceeding is None\n\n def isindex(self):\n return self.preceeding is not None\n\n def __repr__(self):\n return (\"leaf\" if self.isleaf() else (\"index<%d>\" % self.preceeding)) + repr(self.index)\n\n\nclass LeafPage(BasePage):\n \"\"\" a leaf page in the b-tree \"\"\"\n def __init__(self, kv):\n super(self.__class__, self).__init__(kv)\n self.preceeding = None\n\n\nclass IndexPage(BasePage):\n \"\"\"\n An index page in the b-tree.\n This page has a preceeding page plus several key+subpage pairs.\n For each key+subpage: all keys in the subpage are greater than the key\n \"\"\"\n def __init__(self, preceeding, kv):\n super(self.__class__, self).__init__(kv)\n self.preceeding = preceeding\n\n def getpage(self, ix):\n return self.preceeding if ix < 0 else self.index[ix].val\n\n\nclass Cursor:\n \"\"\"\n A Cursor object represents a position in the b-tree.\n\n It has methods for moving to the next or previous item.\n And methods for retrieving the key and value of the current position\n \"\"\"\n def __init__(self, db, stack):\n self.db = db\n self.stack = stack\n\n def next(self):\n page, ix = self.stack.pop()\n if page.isleaf():\n # from leaf move towards root\n ix += 1\n while self.stack and ix == len(page.index):\n page, ix = self.stack.pop()\n ix += 1\n if ix < len(page.index):\n self.stack.append((page, ix))\n else:\n # from node move towards leaf\n self.stack.append((page, ix))\n page = self.db.readpage(page.getpage(ix))\n while page.isindex():\n ix = -1\n self.stack.append((page, ix))\n page = self.db.readpage(page.getpage(ix))\n ix = 0\n self.stack.append((page, ix))\n\n self.verify()\n\n def prev(self):\n page, ix = self.stack.pop()\n ix -= 1\n if page.isleaf():\n # move towards root, until non 'prec' item found\n while self.stack and ix < 0:\n page, ix = self.stack.pop()\n if ix >= 0:\n self.stack.append((page, ix))\n else:\n # move towards leaf\n self.stack.append((page, ix))\n while page.isindex():\n page = self.db.readpage(page.getpage(ix))\n ix = len(page.index) - 1\n self.stack.append((page, ix))\n\n self.verify()\n\n def verify(self):\n \"\"\" verify cursor state consistency \"\"\"\n if len(self.stack) == 3:\n if not self.stack[-1][0].isleaf():\n print(\"WARN no leaf\")\n elif len(self.stack) > 3:\n print(\"WARN: stack too large\")\n\n if len(self.stack) >= 2:\n if self.stack[0][0] == self.stack[1][0]:\n print(\"WARN: identical index pages on stack\")\n if not self.stack[0][0].isindex():\n print(\"WARN: expected root=index\")\n if not self.stack[1][0].isindex():\n print(\"WARN: expected 2nd=index\")\n\n def eof(self):\n return len(self.stack) == 0\n\n def getkey(self):\n page, ix = self.stack[-1]\n return page.getkey(ix)\n\n def getval(self):\n page, ix = self.stack[-1]\n return page.getval(ix)\n\n def __repr__(self):\n return \"cursor:\" + repr(self.stack)\n\n\nclass Btree:\n \"\"\"\n A B-tree implementation\n \"\"\"\n def __init__(self):\n self.pages = []\n self.generate(TREEDEPTH, NODEWIDTH)\n\n def manual(self):\n \"\"\" manually construct the ascii art tree \"\"\"\n for i in range(9):\n self.pages.append(LeafPage(((\"%02d\" % (3 * i), 0), (\"%02d\" % (3 * i + 1), 0))))\n for i in range(3):\n self.pages.append(IndexPage(3 * i, ((\"%02d\" % (9 * i + 2), 3 * i + 1), (\"%02d\" % (9 * i + 5), 3 * i + 2))))\n self.pages.append(IndexPage(9, ((\"08\", 10), (\"17\", 11))))\n self.rootindex = len(self.pages) - 1\n\n def generate(self, depth, nodesize):\n \"\"\" automatically generate the try in the ascii art above \"\"\"\n\n def namegen():\n i = 0\n while True:\n yield \"%03d\" % i\n i += 1\n\n self.rootindex = self.construct(namegen(), depth, nodesize)\n print(\"%d pages\" % (len(self.pages)))\n\n def construct(self, namegen, depth, nodesize):\n if depth:\n return self.createindex(namegen, depth, nodesize)\n else:\n return self.createleaf(namegen, nodesize)\n\n def createindex(self, namegen, depth, nodesize):\n page = IndexPage(self.construct(namegen, depth - 1, nodesize),\n [(next(namegen), self.construct(namegen, depth - 1, nodesize)) for _ in range(nodesize)])\n self.pages.append(page)\n return len(self.pages) - 1\n\n def createleaf(self, namegen, nodesize):\n page = LeafPage([(next(namegen), 0) for _ in range(nodesize)])\n self.pages.append(page)\n return len(self.pages) - 1\n\n def readpage(self, pn):\n return self.pages[pn]\n\n def find(self, key):\n \"\"\"\n Find a node in the tree, returns the cursor plus the reletion to the wanted key:\n 'eq' for equal, 'lt' when the found key is less than the wanted key,\n or 'gt' when the found key is greater than the wanted key.\n \"\"\"\n page = self.readpage(self.rootindex)\n stack = []\n while True:\n act, ix = page.find(key)\n stack.append((page, ix))\n if act != 'recurse':\n break\n page = self.readpage(page.getpage(ix))\n return act, Cursor(self, stack)\n\n def dumptree(self, pn, indent=0):\n \"\"\" dump all nodes of the current b-tree \"\"\"\n page = self.readpage(pn)\n print(\" \" * indent, page)\n if page.isindex():\n print(\" \" * indent, end=\"\")\n self.dumptree(page.preceeding, indent + 1)\n for p in range(len(page.index)):\n print(\" \" * indent, end=\"\")\n self.dumptree(page.getpage(p), indent + 1)\n\n\ndb = Btree()\nprint(\"<<\")\ndb.dumptree(db.rootindex)\nprint(\">>\")\n\n\nfor i in range(NODEWIDTH * len(db.pages)):\n print(\"--------- %03d\" % i)\n act, cursor = db.find(\"%03d\" % i)\n print(\"found\", act, cursor.getkey(), cursor)\n cursor.prev()\n if not cursor.eof():\n print(\"prev:\", \"..\", cursor.getkey(), cursor)\n else:\n print(\"prev: EOF\", cursor)\n\nfor i in range(NODEWIDTH * len(db.pages)):\n print(\"--------- %03d\" % i)\n act, cursor = db.find(\"%03d\" % i)\n print(\"found\", act, cursor.getkey(), cursor)\n cursor.next()\n if not cursor.eof():\n print(\"next:\", \"..\", cursor.getkey(), cursor)\n else:\n print(\"next: EOF\", cursor)\n\nfor k in ('', '0', '1', '2', '3', '000', '010', '020', '100'):\n print(\"--------- %s\" % k)\n act, cursor = db.find(k)\n print(cursor)\n print(act, cursor.getkey(), end=\" next=\")\n cursor.next()\n if cursor.eof():\n print(\"EOF\")\n else:\n print(cursor.getkey())\n\nact, cursor = db.find(\"000\")\nprint(\"get000\", end=\" \")\nfor i in range(NODEWIDTH * len(db.pages)):\n cursor.next()\n if cursor.eof():\n print(\"EOF\")\n else:\n print(\"-> %s\" % cursor.getkey(), end=\" \")\nprint()\n\nact, cursor = db.find(\"025\")\nprint(\"get025\", end=\" \")\nfor i in range(NODEWIDTH * len(db.pages)):\n cursor.prev()\n if cursor.eof():\n print(\"EOF\")\n else:\n print(\"-> %s\" % cursor.getkey(), end=\" \")\nprint()\n" }, { "path": "tstbs.py", "content": "def binary_search(a, k):\n # c++: a.upperbound(k)--\n first, last = 0, len(a)\n while first>1\n if k < a[mid]:\n last = mid\n else:\n first = mid+1\n return first-1\nfor x in range(8):\n print(x, binary_search([2,3,5,6], x))\n" } ]