[ { "path": "README.md", "content": "\n\n# Block Oriented Programming Compiler (BOPC)\n\n___\n\n\n## What is BOPC\n\n**NEW:** The talk from CCS'18 presentation is available\n[here](https://www.youtube.com/watch?v=iK7jhrK5uyg).\n\n\n\nBOPC (stands for _BOP Compiler_) is a tool for automatically synthesizing arbitrary,\nTuring-complete, _Data-Only_ payloads. BOPC finds execution traces in the binary that\nexecute the desired payload while adhering to the binary's Control Flow Graph (CFG).\nThis implies that the existing control flow hijacking defenses are not sufficient to\ndetect this style of execution, as execution does never violates the Control Flow\nIntegrity (CFI).\n\nEssentially, we can say that Block Oriented Programming is _code reuse under CFI_. \n\nBOPC works with basic blocks (hence the name \"block-oriented\"). What it does is to find\na set of _functional_ blocks (i.e., blocks that perform useful computations). This step\nis somewhat similar with finding Return Oriented Programming (ROP) gadgets.\nHaving the functional blocks, BOPC looks for _dispatcher_ blocks to that are used to\nstitch functional blocks together. Compared to ROP (that we can move from one gadget\nto the next without any limitation), here we can't do that as it would violate the CFI.\nInstead, BOPC finds a proper sequence for dispatcher blocks that naturally lead the\nexecution from one functional block to the next one.\nUnfortunately the problem of building _Data-Only_ payloads is NP-hard. \nHowever it turns out that in practice BOPC finds solution in a reasonable amount\nof time.\n\n\nFor more details on how BOPC works, please refer to our [paper](./ccs18_paper.pdf),\nand our [slides](./ccs18_slides.pdf) from CCS'18.\n\n\nTo operate, BOPC requires 3 inputs:\n* A target binary that has an _Arbitrary Memory Write_ (AWP) vulnerability (**hard requirement**)\n* The desired payload, expressed in a high level language called SPL (stands for _SPloit Language_)\n* The so-called \"_entry point_\", which is the first instruction in the binary that the\npayload execution should start. There can be more than one entry points and determining it is\npart of the vulnerability discovery process.\n\n\nThe output of BOPC is a set of \"what-where\" memory writes that indicate how the memory \nshould be initialized (i.e., what values to write at which memory addresses). \nWhen the execution reaches the entry point and the memory is initialized according to\nthe output of BOPC, the target binary execute the desired payload instead of continuing\nthe original execution.\n\n\n**Disclaimer:** This is a research project coded by a single guy. It's not a product,\nso do **not** expect it to work perfectly under all scenarios. It works nicely for the\n provided test cases, but beyond that we cannot guarantee that will work as expected.\n\n___\n\n\n## Installation\nJust run `setup.sh` :)\n\n___\n\n\n## How to use BOPC\n\nBOPC started as a hacky project, so several changes made to adapt it into an scientific\ncontext. That is, the implementation in the [paper](./ccs18_paper.pdf) is slightly\ndifferent from the actual implementation, as we omitted several implementation details\nfrom the paper. The actual implementation overview is shown below:\n![alt text](./source/images/BOPC_overview.png)\n\n\n\n### Command line arguments explained\n\nA good place to start are the command line arguments:\n\n```\nusage: BOPC.py [-h] [-b BINARY] [-a {save,load,saveonly}] [--emit-IR] [-d]\n [-dd] [-ddd] [-dddd] [-V] [-s SOURCE] [-e ENTRY]\n [-O {none,ooo,rewrite,full}] [-f {raw,idc,gdb}] [--find-all]\n [--mapping-id ID] [--mapping MAP [MAP ...]] [--enum-mappings]\n [--abstract-blk BLKADDR] [-c OPTIONS [OPTIONS ...]]\n\noptional arguments:\n -h, --help show this help message and exit\n\nGeneral Arguments:\n -b BINARY, --binary BINARY\n Binary file of the target application\n -a {save,load,saveonly}, --abstractions {save,load,saveonly}\n Work with abstraction file\n --emit-IR Dump SPL IR to a file and exit\n -d Set debugging level to minimum\n -dd Set debugging level to basic (recommended)\n -ddd Set debugging level to verbose (DEBUG ONLY)\n -dddd Set debugging level to print-everything (DEBUG ONLY)\n -V, --version show program's version number and exit\n\nSearch Options:\n -s SOURCE, --source SOURCE\n Source file with SPL payload\n -e ENTRY, --entry ENTRY\n The entry point in the binary that payload starts\n -O {none,ooo,rewrite,full}, --optimizer {none,ooo,rewrite,full}\n Use the SPL optimizer (Default: none)\n -f {raw,idc,gdb}, --format {raw,idc,gdb}\n The format of the solution (Default: raw)\n --find-all Find all the solutions\n\nApplication Capability:\n -c OPTIONS [OPTIONS ...], --capability OPTIONS [OPTIONS ...]\n Measure application's capability. Options (can be many)\n \n all\tSearch for all Statements\n regset\tSearch for Register Assignments\n regmod\tSearch for Register Modifications\n memrd\tSearch for Memory Reads\n memwr\tSearch for Memory Writes\n call\tSearch for Function/System Calls\n cond\tSearch for Conditional Jumps\n load\tLoad capabilities from file\n save\tSave capabilities to file\n noedge\tDump statements and exit (don't calculate edges)\n\nDebugging Options:\n --mapping-id ID Run the Trace Searching algorithm on a given mapping ID\n --mapping MAP [MAP ...]\n Run the Trace Searching algorithm on a given register mapping\n --enum-mappings Enumerate all possible mappings and exit\n --abstract-blk BLKADDR\n Abstract a specific basic block and exit\n```\n\nOk, there are a lot of options here (divided into 4 categories) as BOPC can do several things.\n\nLet's start with the **General Arguments**. To avoid working directly with assembly, BOPC,\n\"abstracts\" each basic block into a set of \"actions\". For more details, please check\n[absblk.py](./source/absblk.py). Abstraction process symbolically executes each basic block\nin the binary and carefully monitors its actions. The abstraction process can take from a few\nminutes (for small binaries) to several hours (for the larger ones). Waiting that much every\ntime that you want to run BOPC does not sound a good idea, so BOPC uses an old trick: _caching_.\n\nThe abstraction process depends on the binary and not on the SPL payload nor the entry point,\nso we only need to calculate them *once* per binary. Therefore, we have to calculate the\nabstractions only one time, then save them into a file and each time loading them. \nThe `save` and `saveonly` options save the abstractions into a file. The only difference is that\n`saveonly` halts execution after it saves the abstractions, while `save` continues to search\nfor a solution. As you can guess, the `load` option loads the abstractions from a file.\n\nThe `--emit-IR` option is used to \"dump\" the IR representation of the SPL payload (this is\nanother intermediate result that you should not worry about it).\n\nBOPC provides 5 verbosity levels: no option, `-d`, `-dd`, `-ddd` and `-dddd`. I recommend you\nto use either the `-dd` or the `-ddd` to get a detailed progress status.\n\nLet's get into the **Search Options** options. The most important arguments here are the\n`--source` (which is a file that contains the SPL payload) and the `--entry` which is an\naddress inside the binary that indicates the entry point. Trace searching starts from the\nentry point, so this is quite important.\n\n\nThe optimizer (`-O` option) is double edge knife. On the one hand, it optimizes the SPL\npayload to make it more flexible. This means that it increases the likelihood to find a\nsolution. On the other hand, the search space (along with the execution time) is increased.\nThe decision is up to the user, hence the use of optimizer is optional. The 2 possible\noptimizations are the _out of order execution_ (`ooo` option) and the _statement rewriting_\n(`rewrite` option). \n\n\nThe out-of-order optimization reorders payload statements.\nConsider for example the following SPL payload:\n```\n\t__r0 = 13;\n\t__r1 = 37;\n```\n\nTo find a solution here, BOPC must find a functional block for the first statement (`__r0 = 13`)\nthen a functional block for the second statement (`__r1 = 37`) and a set of dispatcher blocks\nto connect these two statements. However these functional blocks may be far apart so a dispatcher\nmay not exist. However there's no difference if you execute the `__r0 = 13` statement first\nor second as it does not have any dependencies with the other statement. Thus if we rewrite\nthe payload as follows:\n```\n\t__r1 = 37;\n\t__r0 = 13;\n```\n\nIt may be possible to find another set dispatcher blocks, hopefully much smaller \n(path `A -> B` may be much longer than path `B -> A`) and find a solution.\n\nInternally, this is a **two-step** process. First the optimizer **groups** independent\nstatements together (for more details take a look [here](./source/optimize.py)) and\ngenerated and augmented SPL IR. Then, the trace search module, permutes statements\nwithin each group, each time resulting in a different SPL payload. However all these\npayloads are equivalent. As you can guess there are can be an exponential number of \npermutations, so this can take forever. To alleviate that, you can adjust\n`N_OUT_OF_ORDER_ATTEMPTS` configuration parameter and tell BOPC to stop after trying \n**N** iterations, instead of trying all of them.\n\n\n\nThe statement rewriting is an under development optimization that rewrites\nsome statements that do not exist in the binary. For instance if the SPL payload\nspawns a shell through 'execve()' but the target binary does not invoke\n`execve()` at all, then BOPC fails as there are no functional blocks for that statement.\nHowever, if the target binary invokes `execv()`, it may be possible to find a solution\nby replacing `execve()` with `execv()`. The optimizer contains a list of possible replacements,\nand adjust payload accordingly.\n\n\nAs we already explained, the output of BOPC is a set of \"what-where\" memory writes. There\nare several ways to express the output. For instance they can be raw lines containing the\naddress, the value and the size of the data that should be written in memory. Or they can\nbe a gdb/IDA script that can run directly on the debugger and modify the memory accordingly.\nThe last option is the best one as it you only need to run the BOPC output into the debugger.\nCurrently only the `gdb` format is implemented.\n\n\n\nThe **Application Capability** options used to measure _Application's capabilities_, that\ngives us upper bounds on **what** payloads the target binary is capable of executing.\n\n\nFinally the **Debugging Options** assist the audit/debugging/development process. They are used\nto bypass parts of the BOP work-flow. Do not use them unless you're doing changes in the code.\nRecall that BOPC finds a mapping between virtual and host registers along with a mapping\nbetween SPL variables and underlying memory addresses. If that mapping does not lead to\na solution it goes back and tries another one. If you want to focus on a specific mapping\n(e.g., let's say that code crashes at mapping 458), you don't have to wait for BOPC to try\nthe first 457 mappings first. By supplying the `--mapping-id=458` option you can skip\nall mappings and focus on that one. In case that you don't know the mapping number but you\nknow the actual mapping you can instead you the `--mapping` option: `--mapping=`__r0=rax __r1=rbx`\n\n\n\nFinally, BOPC has a lot of configuration options. You see all of them in \n[config.py](./source/config.py) and adjust them according to our needs. The default\nvalues are a nice trade off between accuracy and performance that I found during\nthen evaluation.\n\n\n## Example\n\nLet's see now how to actually use BOPC. The first thing to do is to get the basic block\nabstractions. This step is optional, but I expect that you are going to run BOPC several times,\nso it's a good idea to get the abstractions first:\n```\n./source/BOPC.py -dd --binary $BINARY --abstractions saveonly\n```\n\nThis calculates the abstractions and saves them into a file named `$BINARY.abs`. Don't forget\nto enable debugging to see the status on the screen.\n\n\nWriting an SPL payload is pretty much like writing C:\n```C\nvoid payload() \n{ \n string prog = \"/bin/sh\\0\";\n int argv = {&prog, 0x0};\n\n __r0 = &prog;\n __r1 = &argv;\n __r2 = 0;\n \n execve(__r0, __r1, __r2);\n}\n```\n\n\nPlease take a look at the available [payloads](./payloads) to see all features of SPL.\nDon't expect to write crazy program with SPL; Yes, in theory you can write any program.\nIn practice the more complicated is the SPL payload, the more the complexity increases\nand the harder it gets to find a solution.\n\n\nRunning BOPC is as simple as the following:\n```\n./source/BOPC.py -dd --binary $BINARY --source $PAYLOAD --abstractions load \\\n--entry $ENTRY --format gdb\n```\n\nIf everything goes well an `*.gdb` file will be created that contains the set of memory writes\nto execute the desired payload.\n\n\n### Pruning search space\n\nA common problem is that there can be thousands of mappings (it's exponential based on the \nnumber of registers and variables that are used). Each mapping can take up to a minute to test\n(assuming out of order execution and other optimizations), so BOPC may run for days.\n\nHowever, if you know approximately where a solution could be, you can ask BOPC to quickly find\n(and verify) it, without trying all mappings. Let's assume that you want to execute the following\nSPL payload:\n```C\nvoid payload() \n{ \n string msg = \"This is my random message! :)\\0\";\n\n __r0 = 0;\n __r1 = &msg;\n __r2 = 32;\n\n write( __r0, __r1, __r2 );\n}\n```\n\nBecause we have a system call, we know the register mapping: \n`__r0 <-> rdi, __r1 <-> rsi, __r2 <-> rdx`.\n\nLet's assume that we're on `proftpd` binary which contains the following \"all-in-one\"\nfunctional block:\n```Assembly\n.text:000000000041D0B5 loc_41D0B5:\n.text:000000000041D0B5 mov edi, cs:scoreboard_fd ; fd\n.text:000000000041D0BB mov edx, 20h ; n\n.text:000000000041D0C0 mov esi, offset header ; buf\n.text:000000000041D0C5 call _write\n```\n\nThe abstractions for this basic block, will be the following (recall that to get the\nabstractions for a single basic block, you need to pass the `--abstract-blk 0x41D0B5`\nin the command line).\n```\n[22:02:07,822] [+] Abstractions for basic block 0x41d0b5:\n[22:02:07,823] [+] regwr :\n[22:02:07,823] [+] \t\trsp = {'writable': True, 'const': 576460752303359992L, 'type': 'concrete'}\n[22:02:07,823] [+] \t\trdi = {'sym': {}, 'memrd': None, 'type': 'deref', 'addr': , 'deps': []}\n[22:02:07,823] [+] \t\trsi = {'writable': True, 'const': 6787008L, 'type': 'concrete'}\n[22:02:07,823] [+] \t\trdx = {'writable': False, 'const': 32L, 'type': 'concrete'}\n[22:02:07,823] [+] memrd : set([(>, 32)])\n[22:02:07,823] [+] memwr : set([(>, >)])\n[22:02:07,823] [+] conwr : set([(576460752303359992L, 64)])\n[22:02:07,823] [+] splmemwr : []\n[22:02:07,823] [+] call : {}\n[22:02:07,823] [+] cond : {}\n[22:02:07,823] [+] symvars : {}\n[22:02:07,823] [*] \n```\n\nHere, `__r0 <-> rdi` is loaded indirectly and the value of `__r1 <-> rsi` (which holds the `msg` \nvariable) is `6787008` or `0x678fc0` in hex. Then we enumerate all possible mappings with the\n`--enum-mappings` option. Here, there are *287* possible mappinges, but there are instances that\nwe have thousands of mappings:\n\n\nIf we look at the output we can quickly search for the appropriate mapping, which in our case\nis mapping *#89*:\n```\n[.... TRUNCATED FOR BREVITY ....]\n[21:59:28,471] [*] Trying mapping #88:\n[21:59:28,471] [*] \tRegisters: __r0 <-> rdi | __r1 <-> rsi | __r2 <-> rdx\n[21:59:28,471] [*] \tVariables: msg <-> *\n[21:59:28,614] [*] Trying mapping #89:\n[21:59:28,614] [*] \tRegisters: __r0 <-> rdi | __r1 <-> rsi | __r2 <-> rdx\n[21:59:28,614] [*] \tVariables: msg <-> 0x678fc0L\n[21:59:28,762] [*] Trying mapping #90:\n[21:59:28,762] [*] \tRegisters: __r0 <-> rdi | __r1 <-> rsi | __r2 <-> rdx\n[21:59:28,762] [*] \tVariables: msg <-> *\n[.... TRUNCATED FOR BREVITY ....]\n[22:00:04,709] [*] Trying mapping #287:\n[22:00:04,709] [*] \tRegisters: __r0 <-> rdi | __r1 <-> rsi | __r2 <-> rdx\n[22:00:04,709] [*] \tVariables: msg <-> *\n[22:00:04,979] [+] Trace searching algorithm finished with exit code 0\n```\n\nNow that we know the actual mapping, we can tell BOPC to focus on this one. All we have to\ndo is to pass the `--mapping-id 89` option.\n\n\nWe run this and after 1 minute and 51 seconds later, we get the solution:\n```\n#\n# This file has been created by BOPC at: 29/03/2018 22:04\n# \n# Solution #1\n# Mapping #89\n# Registers: __r0 <-> rdi | __r1 <-> rsi | __r2 <-> rdx\n# Variables: msg <-> 0x678fc0L\n# \n# Simulated Trace: [(0, '41d0b5', '41d0b5'), (4, '41d0b5', '41d0b5'), (6, '41d0b5', '41d0b5'), (8, '41d0b5', '41d0b5'), (10, '41d0b5', '41d0b5')]\n# \n\nbreak *0x403740\nbreak *0x41d0b5\n\n# Entry point\nset $pc = 0x41d0b5 \n\n# Allocation size is always bigger (it may not needed at all)\nset $pool = malloc(20480)\n\n# In case that rbp is not initialized\nset $rbp = $rsp + 0x800 \n\n# Stack and frame pointers aliases\nset $stack = $rsp \nset $frame = $rbp \n\nset {char[30]} (0x678fc0) = {0x54, 0x68, 0x69, 0x73, 0x20, 0x69, 0x73, 0x20, 0x6d, 0x79, 0x20, 0x72, 0x61, 0x6e, 0x64, 0x6f, 0x6d, 0x20, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x21, 0x20, 0x3a, 0x29, 0x00}\n\nset {char[8]} (0x66e9e0) = {0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00}\n```\n\nLet's take a closer look here. The _Simulated Trace_ comment shows the path that BOPC followed.\nThis is a list of `($pc, $src, $dst)` tuples. `$pc` is the program counter of the SPL statement.\n`$src` is the address of the functional block for the current SPL statement and `$dst` is the\naddress of the next functional block.\n\n\nBefore it runs, script adjusts `$rip` to point to the entry point, and makes sure that\nstack pointers (`$rsp`, `$rbp`) are valid. It also allocates a \"variable pool\" (for\nmore details please look at [simulate.py](./source/simulate.py)) which in our case is not\nused.\n\nThen we have the two actual memory writes at `0x678fc0` and at `0x66e9e0`. If you load\nthe binary in gdb and run this script you will see your payload being executed:\n\n```\n(gdb) break main\nBreakpoint 5 at 0x4041a0\n(gdb) run\nStarting program: /home/ispo/BOPC/evaluation/proftpd \n\nBreakpoint 1, 0x00000000004041a0 in main ()\n(gdb) continue\nContinuing.\n\nBreakpoint 3, 0x000000000041d0b5 in pr_open_scoreboard ()\n(gdb) continue\nContinuing.\n\nBreakpoint 2, 0x0000000000403740 in write@plt ()\n(gdb) continue\nContinuing.\nThis is my random message! :)\nProgram received signal SIGSEGV, Segmentation fault.\n0x00007fffffffde60 in ?? ()\n```\n\nNote that BOPC stops after executing the desired payload (hence the crash). If you\nwant to avoid this situation you can use the `returnto` SPL statement to naturally\ntransfer execution to a safe location.\n\n\n\n### Measuring application capabilities\n\n**NOTE:** This is a new concept, which is not mentioned in the paper. \n\nBeyond finding Data-Only payloads, BOPC provides some basic capability measurements.\nAlthough it is not related to the Block Oriented Programming, it can provide upper\nbounds and strong \"indications\" on what types of payloads can be executed and what\nare not. This is very useful as we can quickly find types of payloads that **cannot**\nbe executed in the target binary. \nTo get the all application capabilities run the following code:\n```\n./source/BOPC.py -dd --binary $BINARY --abstractions load --capability all save\n```\n\nIf you want to simply dump all functional gadgets for a specific statement, you can do\nit as follows:\n```\n./source/BOPC.py -dd --binary $BINARY --abstractions load --capability $STMT noedge\n```\n\nWhere `$STMT` can be one ore more from `{all, regset, regmod, memrd, memwr, call, cond}`.\nThe `noedge` option is to boost things up (essentially it does not calculate edges in the\ncapability graph; Each node in the capability graph represents a functional block from\nthe binary while and edge represents the context-sensitive shortest path distance\nbetween two functional blocks).\n\n\n___\n\n\n## Final Notes (please read them carefully!)\n\n* When the symbolic execution engine deals with filesystem (i.e., it has to `open` a file),\nwe have to provide it a valid file. Filename is defined in `SYMBOLIC_FILENAME` in \n[coreutils.py](./source/coreutils.py).\n\n* If you want to visualize things, just uncomment the code in search.py. I'm too lazy to add\nCLI flags to trigger it :P\n\n* In case that addresses used by concolic execution do not work, adjust them from \n[simulate.py](./source/simulate.py)\n\n* Make sure that `$rsp` is consistent in `dump()` in [simulate.py](./source/simulate.py)\n\n* For any questions/concerns regarding the code, you can contact [ispo](https://github.com/ispoleet)\n\n___\n\n" }, { "path": "evaluation/README.md", "content": "\n\n# Block Oriented Programming Compiler (BOPC)\n___\n\n\n### Vulnerable Application Overview\n\n\n| Application | CVE |\n|----------------------------|---------------|\n|[ProFTPd](./proftpd) | CVE-2006-5815 |\n|[nginx](./nginx1) | CVE-2013-2028 |\n|[sudo](./sudo) | CVE-2012-0809 |\n|[orzhttpd](./orzhttpd) | BugtraqID 41956 |\n|[wuftdp](./wuftpd) | CVE-2000-0573 |\n|[nullhttpd](./nullhttpd) | CVE-2002-1496 |\n|[opensshd](./opensshd) | CVE-2001-0144 |\n|[wireshark](./lt-wireshark) | CVE-2014-2299 |\n|[apache](./httpd) | CVE-2006-3747 |\n|[smbclient](./smbclient) | CVE-2009-1886 |\n\n___\n" }, { "path": "payloads/README.md", "content": "\n\n# Block Oriented Programming Compiler (BOPC)\n___\n\n\n### SPL Payload Overview\n\n\n| Payload | Description |\n|--------------------------|---------------------------------------------|\n| [regset4](./regset4.spl) | Initialize 4 registers with arbitrary values |\n| [regref4](./regref4.spl) | Initialize 4 registers with pointers to arbitrary memory |\n| [regset5](./regset5.spl) | Initialize 5 registers with arbitrary values |\n| [regref5](./regref5.spl) | Initialize 5 registers with pointers to arbitrary memory |\n| [regmod](./regmod.spl) | Initialize a register with an arbitrary value and modify it |\n| [memrd](./memrd.spl) | Read from arbitrary memory |\n| [memwr](./memwr.spl) | Write to arbitrary memory |\n| [print](./print.spl) | Display a message to stdout using write |\n| [execve](./execve.spl) | Spawn a shell through execve |\n| [abloop](./abloop.spl) | Perform an arbitrarily long bounded loop utilizing regmod |\n| [infloop](./infloop.spl) | Perform an infinite loop that sets a register in its body |\n| [ifelse](./ifelse.spl) | An if-else condition based on a register comparison |\n| [loop](./loop.spl) | Conditional loop with register modification |\n\n\n___\n" }, { "path": "payloads/abloop.spl", "content": "// \n// BOPC Evaluation\n//\n// Arbitrary bounded loop with register modification\n//\nvoid payload() \n{ \n __r0 = 0;\n\nLOOP:\n\n __r0 += 1;\n\n goto LOOP;\n}\n\n" }, { "path": "payloads/execve.spl", "content": "// \n// BOPC Evaluation\n//\n// execve('/bin/sh') payload\n//\nvoid payload() \n{ \n string prog = \"/bin/sh\\0\";\n int argv = {&prog, 0x0};\n\n __r0 = &prog;\n __r1 = &argv;\n __r2 = 0;\n \n execve(__r0, __r1, __r2);\n\n // return ?\n}\n" }, { "path": "payloads/ifelse.spl", "content": "// \n// BOPC Evaluation\n//\n// Simple if-else condition\n//\nvoid payload() \n{ \n __r0 = 0;\n\n if (__r0 != 0 ) goto SKIP;\n\n __r1 = 1;\n\n goto END;\n\nSKIP:\n __r1 = 0;\n\n\nEND:\n // return 0x400ca7;\n __r0 = 0;\n\n}\n" }, { "path": "payloads/infloop.spl", "content": "// \n// BOPC Evaluation\n//\n// Infinity loop with constant assignment\n//\nvoid payload() \n{ \n\n// __r0 = 1;\n\nLOOP:\n __r1 = 0;\n\n goto LOOP;\n}\n" }, { "path": "payloads/loop.spl", "content": "// \n// BOPC Evaluation\n//\n// Simple loop\n//\nvoid payload() \n{ \n __r0 = 0;\n\nLOOP:\n __r0 += 1;\n\n if (__r0 != 3 ) goto LOOP;\n}\n" }, { "path": "payloads/memrd.spl", "content": "// \n// BOPC Evaluation\n//\n// Memory Load\n//\nvoid payload() \n{ \n int array = {1, 2, 3, 4, 5};\n \n __r1 = &array;\n\n __r0 = *__r1;\n\n}\n" }, { "path": "payloads/memwr.spl", "content": "// \n// BOPC Evaluation\n//\n// Memory Store\n//\nvoid payload() \n{ \n int array = {1, 2, 3, 4, 5};\n \n __r0 = &array;\n __r1 = 0;\n\n *__r0 = __r1;\n\n}\n" }, { "path": "payloads/print.spl", "content": "// \n// BOPC Evaluation\n//\n// Print an arbitrary message to stdout using write()\n//\nvoid payload() \n{ \n string msg = \"This is my random message! :)\\0\";\n\n __r0 = 0;\n __r1 = &msg;\n __r2 = 1;\n\n\n write( __r0, __r1, __r2 );\n\n // return 0x?? ;s\n}\n" }, { "path": "payloads/regmod.spl", "content": "// \n// BOPC Evaluation\n//\n// Register modification\n//\nvoid payload() \n{ \n __r0 = 0;\n\n __r0 += 1;\n}\n" }, { "path": "payloads/regref4.spl", "content": "// \n// BOPC Evaluation\n//\n// Initialize 4 registers with references\n//\nvoid payload() \n{ \n int var_a = 0x100;\n string var_b = \"this is a random string\";\n int var_c = {1, 2, 3, &var_a, 4, &var_b};\n int var_d = &var_c;\n\n __r0 = &var_a;\n __r1 = &var_b;\n __r2 = &var_c;\n __r3 = &var_d;\n}\n" }, { "path": "payloads/regref5.spl", "content": "// \n// BOPC Evaluation\n//\n// Initialize 5 registers with references\n//\nvoid payload() \n{ \n\tlong var_a = 0x100;\n\tstring var_b = \"this is a random string\\x00\";\n\n\tlong *var_c = {1, 2, 3, 4, &var_a, &var_b};\n\tlong var_d = &var_c;\n\tlong *var_e = {&var_d, &var_d, &var_d};\n\n\t__r0 = &var_a;\n\t__r1 = &var_b;\n\t__r2 = &var_c;\n\t__r3 = &var_d;\n\t__r4 = &var_e;\n\n\t// return ??\n}\n" }, { "path": "payloads/regset4.spl", "content": "// \n// BOPC Evaluation\n//\n// Initialize 4 registers\n//\nvoid payload() \n{ \n __r0 = 0;\n __r1 = 1;\n __r2 = 2;\n __r3 = 3;\n}\n" }, { "path": "payloads/regset5.spl", "content": "// \n// BOPC Evaluation\n//\n// Initialize 5 registers\n//\nvoid payload() \n{ \n __r0 = 0;\n __r1 = 1;\n __r2 = 2;\n __r3 = 3;\n __r4 = 4;\n}\n" }, { "path": "setup.sh", "content": "#!/bin/bash\n# -------------------------------------------------------------------------------------------------\n#\n# ,ggggggggggg, _,gggggg,_ ,ggggggggggg, ,gggg, \n# dP\"\"\"88\"\"\"\"\"\"Y8, ,d8P\"\"d8P\"Y8b, dP\"\"\"88\"\"\"\"\"\"Y8, ,88\"\"\"Y8b,\n# Yb, 88 `8b,d8' Y8 \"8b,dPYb, 88 `8b d8\" `Y8\n# `\" 88 ,8Pd8' `Ybaaad88P' `\" 88 ,8Pd8' 8b d8\n# 88aaaad8P\" 8P `\"\"\"\"Y8 88aaaad8P\",8I \"Y88P'\n# 88\"\"\"\"Y8ba 8b d8 88\"\"\"\"\" I8' \n# 88 `8bY8, ,8P 88 d8 \n# 88 ,8P`Y8, ,8P' 88 Y8, \n# 88_____,d8' `Y8b,,__,,d8P' 88 `Yba,,_____, \n# 88888888P\" `\"Y8888P\"' 88 `\"Y8888888 \n#\n# The Block Oriented Programming (BOP) Compiler - v2.1\n#\n#\n# Kyriakos Ispoglou (ispo) - ispo@purdue.edu\n# PURDUE University, Fall 2016-18\n# -------------------------------------------------------------------------------------------------\nmsg() {\n GREEN='\\033[01;32m' # bold green\n NC='\\033[0m' # no color\n echo -e \"${GREEN}[INFO]${NC} $1\"\n}\n\nerror() {\n RED='\\033[01;31m' # bold red\n NC='\\033[0m' # no color\n echo -e \"${RED}[ERROR]${NC} $1\"\n}\n\n\n# display fancy foo\nclear\necho\necho -e '\\t%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%'\necho -e '\\t% %'\necho -e '\\t% ::::::::: :::::::: ::::::::: :::::::: %'\necho -e '\\t% :+: :+: :+: :+: :+: :+: :+: :+: %'\necho -e '\\t% +:+ +:+ +:+ +:+ +:+ +:+ +:+ %'\necho -e '\\t% +#++:++#+ +#+ +:+ +#++:++#+ +#+ %'\necho -e '\\t% +#+ +#+ +#+ +#+ +#+ +#+ %'\necho -e '\\t% #+# #+# #+# #+# #+# #+# #+# %'\necho -e '\\t% ######### ######## ### ######## %'\necho -e '\\t% %'\necho -e '\\t% Block Oriented Programming Compiler %'\necho -e '\\t% %'\necho -e '\\t%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%'\necho \nmsg \"BOPC Installation Guide has been started ...\"\n\n\n# base check (we need root)\nif [ \"$EUID\" -ne 0 ]; then\n error \"Script needs root permissions to install the required packages.\"\n msg \"Please run as 'sudo $0' (you can have a look at the source, if you don't trust me)\"\n echo\n\n exit\nfi\n\n# install prerequisites first\napt-get install --yes python-pip\napt-get install --yes graphviz libgraphviz-dev\napt-get install --yes pkg-config python-tk \n\n\n# install pip packages\npip install angr==7.8.9.26\npip install claripy==7.8.9.26\npip install matplotlib\npip install simuvex\n# networkx must be installed after simuvex and angr, since they depend\n# on networkx 2.1\npip install networkx==1.11\npip install graphviz==0.8.1\npip install pygraphviz==1.3.1\n\n\nmsg \"BOPC Installation completed ...\"\nmsg \"Have a nice day :)\"\necho\n\n# -------------------------------------------------------------------------------------------------\n" }, { "path": "source/BOPC.py", "content": "#!/usr/bin/env python2\n# -------------------------------------------------------------------------------------------------\n#\n# ,ggggggggggg, _,gggggg,_ ,ggggggggggg, ,gggg, \n# dP\"\"\"88\"\"\"\"\"\"Y8, ,d8P\"\"d8P\"Y8b, dP\"\"\"88\"\"\"\"\"\"Y8, ,88\"\"\"Y8b,\n# Yb, 88 `8b,d8' Y8 \"8b,dPYb, 88 `8b d8\" `Y8\n# `\" 88 ,8Pd8' `Ybaaad88P' `\" 88 ,8Pd8' 8b d8\n# 88aaaad8P\" 8P `\"\"\"\"Y8 88aaaad8P\",8I \"Y88P'\n# 88\"\"\"\"Y8ba 8b d8 88\"\"\"\"\" I8' \n# 88 `8bY8, ,8P 88 d8 \n# 88 ,8P`Y8, ,8P' 88 Y8, \n# 88_____,d8' `Y8b,,__,,d8P' 88 `Yba,,_____, \n# 88888888P\" `\"Y8888P\"' 88 `\"Y8888888 \n#\n# The Block Oriented Programming (BOP) Compiler - v2.1\n#\n#\n# Kyriakos Ispoglou (ispo) - ispo@purdue.edu\n# PURDUE University, Fall 2016-18\n# -------------------------------------------------------------------------------------------------\n#\n# BOPC.py:\n#\n#\n# This is the main module of BOPC. It configures the environment and launches the other modules.\n#\n# -------------------------------------------------------------------------------------------------\nfrom coreutils import *\nimport absblk as A\nimport compile as C\nimport optimize as O\nimport mark as M\nimport search as S\nimport capability as P\n\nimport argparse\nimport textwrap\nimport ntpath\nimport angr\nimport os\nimport sys\n\n\n\n# ------------------------------------------------------------------------------------------------\n# Constant Definitions\n# ------------------------------------------------------------------------------------------------\nVERSION = 'v2.1' # current version\ncomments = '' # Additional comments to display on startup\n\n\n\n# -------------------------------------------------------------------------------------------------\n# parse_args(): This function processes the command line arguments.\n#\n# :Ret: None.\n#\ndef parse_args():\n # create the parser object and the groups\n parser = argparse.ArgumentParser(formatter_class=argparse.RawTextHelpFormatter)\n\n group_g = parser.add_argument_group('General Arguments')\n group_s = parser.add_argument_group('Search Options')\n group_c = parser.add_argument_group('Application Capability')\n group_d = parser.add_argument_group('Debugging Options')\n\n\n # -------------------------------------------------------------------------\n # Group for general arguments\n # -------------------------------------------------------------------------\n group_g.add_argument(\n '-b', \"--binary\",\n help = \"Binary file of the target application\",\n action = 'store',\n dest = 'binary',\n required = False, # True\n )\n\n group_g.add_argument(\n '-a', \"--abstractions\",\n help = \"Work with abstraction file\",\n choices = ['save', 'load', 'saveonly'],\n default = 'none',\n action = 'store',\n dest = 'abstractions',\n required = False\n )\n\n group_g.add_argument(\n \"--emit-IR\",\n help = \"Dump SPL IR to a file and exit\",\n action = 'store_const',\n const = True,\n dest = 'emit_IR',\n required = False\n )\n\n # action='count'\n group_g.add_argument(\n '-d',\n help = \"Set debugging level to minimum\",\n action = 'store_const',\n const = DBG_LVL_1,\n dest = 'dbg_lvl',\n required = False\n )\n\n group_g.add_argument(\n '-dd',\n help = \"Set debugging level to basic (recommended)\",\n action = 'store_const',\n const = DBG_LVL_2,\n dest = 'dbg_lvl',\n required = False\n )\n\n group_g.add_argument(\n '-ddd',\n help = \"Set debugging level to verbose (DEBUG ONLY)\",\n action = 'store_const',\n const = DBG_LVL_3,\n dest = 'dbg_lvl',\n required = False\n )\n\n group_g.add_argument(\n '-dddd',\n help = \"Set debugging level to print-everything (DEBUG ONLY)\",\n action = 'store_const',\n const = DBG_LVL_4,\n dest = 'dbg_lvl',\n required = False\n )\n\n group_g.add_argument(\n '-V', \"--version\",\n action = 'version',\n version = 'BOPC %s' % VERSION\n )\n\n\n # -------------------------------------------------------------------------\n # Group for searching arguments\n # -------------------------------------------------------------------------\n group_s.add_argument(\n '-s', \"--source\",\n help = \"Source file with SPL payload\",\n action = 'store',\n dest = 'source',\n required = False\n )\n\n group_s.add_argument(\n '-e', \"--entry\",\n help = \"The entry point in the binary that payload starts\",\n action = 'store',\n dest = 'entry',\n required = False\n )\n\n group_s.add_argument(\n '-O', \"--optimizer\",\n help = \"Use the SPL optimizer (Default: none)\",\n choices = ['none', 'ooo', 'rewrite', 'full'],\n action = 'store',\n default = 'none',\n dest = 'optimizer',\n required = False\n )\n\n group_s.add_argument(\n '-f', \"--format\",\n help = \"The format of the solution (Default: raw)\",\n choices = ['raw', 'idc', 'gdb'],\n action = 'store',\n default = 'raw',\n dest = 'format',\n required = False,\n )\n\n group_s.add_argument(\n \"--find-all\",\n help = \"Find all the solutions\",\n action = 'store_const',\n default = 'one',\n const = 'all',\n dest = 'findall',\n required = False\n )\n\n\n # -------------------------------------------------------------------------\n # Group for debugging arguments\n # -------------------------------------------------------------------------\n group_d.add_argument(\n \"--mapping-id\",\n help = \"Run the Trace Searching algorithm on a given mapping ID\",\n metavar = 'ID',\n action = 'store',\n default = -1,\n dest = 'mapping_id',\n required = False\n )\n\n group_d.add_argument(\n \"--mapping\",\n help = \"Run the Trace Searching algorithm on a given register mapping\",\n metavar = 'MAP',\n nargs = '+',\n action = 'store',\n default = [],\n dest = 'mapping',\n required = False\n )\n\n group_d.add_argument(\n \"--enum-mappings\",\n help = \"Enumerate all possible mappings and exit\",\n action = 'store_const',\n default = False,\n const = True,\n dest = 'enum_mappings',\n required = False\n )\n\n group_d.add_argument(\n \"--abstract-blk\",\n help = \"Abstract a specific basic block and exit\",\n metavar = 'BLKADDR',\n action = 'store',\n dest = 'absblk',\n required = False\n )\n\n\n # -------------------------------------------------------------------------\n # Group for application capabilities\n # -------------------------------------------------------------------------\n group_c.add_argument(\n '-c', \"--capability\",\n help = textwrap.dedent('''\\\n Measure application's capability. Options (can be many)\n\n all\\tSearch for all Statements\n regset\\tSearch for Register Assignments\n regmod\\tSearch for Register Modifications\n memrd\\tSearch for Memory Reads\n memwr\\tSearch for Memory Writes\n call\\tSearch for Function/System Calls\n cond\\tSearch for Conditional Jumps\n load\\tLoad capabilities from file\n save\\tSave capabilities to file\n noedge\\tDump statements and exit (don't calculate edges)'''),\n choices = ['all', 'regset', 'regmod', 'memrd', 'memwr', 'call', 'cond',\n 'save', 'load', 'noedge'],\n metavar = 'OPTIONS',\n nargs = '+', # consume >=1 arguments (multiple options)\n action = 'store',\n dest = 'capabilities',\n required = False\n )\n\n\n if len(sys.argv) == 1:\n parser.print_help(sys.stderr)\n sys.exit(1)\n\n return parser.parse_args() # do the parsing (+ error handling)\n\n\n\n# ---------------------------------------------------------------------------------------------\n# load(): Load the target binary and generate its CFG.\n#\n# :Arg filename: Binary's file name\n# :Ret: Function returns\n#\ndef load( filename ):\n # load the binary (exception is thrown if name is invalid)\n project = angr.Project(filename, load_options={'auto_load_libs': False})\n\n\n\n # generate CFG\n dbg_prnt(DBG_LVL_0, \"Generating CFG. It might take a while...\")\n CFG = project.analyses.CFGFast()\n dbg_prnt(DBG_LVL_0, \"CFG generated.\")\n\n\n # normalize CFG (i.e. make sure that there are no overlapping basic blocks)\n dbg_prnt(DBG_LVL_0, \"Normalizing CFG...\")\n CFG.normalize()\n\n # normalize every function object as well\n for _, func in project.kb.functions.iteritems():\n if not func.normalized:\n dbg_prnt(DBG_LVL_4, \"Normalizing function '%s' ...\" % func.name)\n func.normalize()\n\n dbg_prnt(DBG_LVL_0, \"Done.\")\n\n\n emph(\"CFG has %s nodes and %s edges\" %\n (bold(len(CFG.graph.nodes())), bold(len(CFG.graph.edges()))))\n\n\n # create a quick mapping between addresses and nodes (basic blocks)\n for node in CFG.graph.nodes():\n ADDR2NODE[ node.addr ] = node\n\n\n # create a quick mapping between basic block addresses and their corresponding functions\n for _, func in CFG.functions.iteritems(): # for each function\n for addr in func.block_addrs: # for each basic block in that function\n ADDR2FUNC[ addr ] = func\n\n\n return project, CFG\n\n\n\n# ---------------------------------------------------------------------------------------------\n# abstract(): Abstract the CFG and apply any further abstraction-related operations.\n#\n# :Arg mark: A valid graph marking object.\n# :Arg mode: Abstraction mode (load, save, saveonly, none)\n# :Arg filename: Abstraction's file name (if applicable)\n# :Ret: None.\n#\ndef abstract( mark, mode, filename ):\n if mode == 'none':\n mark.abstract_cfg() # calculate the abstractions\n\n if mode == 'load':\n mark.load_abstractions(filename) # simply load the abstractions\n\n elif mode == 'save':\n mark.abstract_cfg() # calculate the abstractions\n mark.save_abstractions(filename) # and save them\n\n elif mode == 'saveonly':\n mark.abstract_cfg()\n mark.save_abstractions(filename)\n return -1\n\n return 0\n\n\n\n# ---------------------------------------------------------------------------------------------\n# capability_analyses(): Apply any (custom) analyses to the capabilities.\n#\n# :Arg cap: The capability object\n# :Ret: None.\n#\ndef capability_analyses( cap ):\n dbg_prnt(DBG_LVL_0, 'Applying additional Capability analyses...')\n return\n\n '''\n # analyze all islands\n # cap.analyze(P.CAP_LOOPS, P.CAP_STMT_MIN_DIST)\n\n # analyze a specific island\n # cap.analyze_island(0x400885, P.CAP_STMT_COMB_CTR)\n\n i = 0\n def foo( graph ):\n global i\n print 'Visualing island %d' % i\n cap.visualize(graph, 'island_%d' % i, show_labels=True)\n\n i += 1\n\n for _, d in graph.nodes_iter(data=True):\n print d['type'] # check capability.__add() for all keys\n\n\n # apply the callback to every island\n cap.callback( foo )\n '''\n\n\n# -------------------------------------------------------------------------------------------------\n# main(): This is the main function of BOPC.\n#\n# Ret: None.\n#\nif __name__ == '__main__':\n args = parse_args() # process arguments\n set_dbg_lvl( args.dbg_lvl ) # set debug level in coreutils\n\n now = datetime.datetime.now() # get current time\n\n\n # -------------------------------------------------------------------------\n # Display banner\n # -------------------------------------------------------------------------\n print rainbow(textwrap.dedent('''\n %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n % %\n % ::::::::: :::::::: ::::::::: :::::::: %\n % :+: :+: :+: :+: :+: :+: :+: :+: %\n % +:+ +:+ +:+ +:+ +:+ +:+ +:+ %\n % +#++:++#+ +#+ +:+ +#++:++#+ +#+ %\n % +#+ +#+ +#+ +#+ +#+ +#+ %\n % #+# #+# #+# #+# #+# #+# #+# %\n % ######### ######## ### ######## %\n % %\n % Block Oriented Programming Compiler %\n % %\n %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n '''))\n\n print comments\n print \"[*] Starting BOPC %s at %s\" % (VERSION, bolds(now.strftime(\"%d/%m/%Y %H:%M\")))\n\n\n # -------------------------------------------------------------------------\n # BOPC operation: Emit SPL IR\n # -------------------------------------------------------------------------\n if args.emit_IR and args.source:\n IR = C.compile(args.source)\n IR.compile() # compile the SPL payload\n\n IR = O.optimize(IR.get_ir())\n IR.optimize(mode=args.optimizer) # optimize IR (if needed)\n\n IR.emit(args.source)\n\n\n # -------------------------------------------------------------------------\n # BOPC operation: Trace Search\n # -------------------------------------------------------------------------\n elif args.source and args.entry:\n IR = C.compile(args.source)\n IR.compile() # compile the SPL payload\n\n IR = O.optimize(IR.get_ir())\n IR.optimize(mode=args.optimizer) # optimize IR (if needed)\n\n\n project, CFG = load(args.binary)\n mark = M.mark(project, CFG, IR, 'puts')\n\n if abstract(mark, args.abstractions, args.binary) > -1:\n entry = int(args.entry, 0) # get entry point\n\n X = mark.mark_candidate(sorted(map(lambda s : tuple(s.split('=')), args.mapping)))\n\n if not X:\n print 'abort';\n exit()\n\n\n # visualize('cfg_cand', entry=entry, options=VO_DRAW_CFG|VO_DRAW_CANDIDATE)\n\n # extract payload name (without the extenstion)\n payload_name = ntpath.basename(args.source)\n payload_name = os.path.splitext(payload_name)[0]\n\n\n try:\n options = {\n 'format' : args.format,\n 'solutions' : args.findall,\n 'mapping-id' : int(args.mapping_id),\n 'mapping' : sorted(map(lambda s : tuple(s.split('=')), args.mapping)),\n 'filename' : '%s-%s' % (args.binary, payload_name),\n 'enum' : args.enum_mappings,\n\n 'simulate' : False,\n '#mappings' : 0,\n '#solutions' : 0\n }\n\n except ValueError:\n fatal(\"'mapping' argument must be an integer\")\n\n\n tsearch = S.search(project, CFG, IR, entry, options)\n tsearch.trace_searching(mark)\n\n # -----------------------------------------------------------------\n # Show some statistics\n # -----------------------------------------------------------------\n emph(\"Trace Searching Statistics:\" )\n emph(\"\\tUsed Simulation? %s\" % bolds(options['simulate']))\n emph(\"\\t%s Mapping(s) tried\" % bold(options['#mappings']))\n emph(\"\\t%s Solution(s) found\" % bold(options['#solutions']))\n\n\n # -------------------------------------------------------------------------\n # BOPC operation: Dump abstractions\n # -------------------------------------------------------------------------\n elif args.abstractions == 'saveonly':\n # IR is useless; we're only dumping abstractions\n project, CFG = load(args.binary)\n mark = M.mark(project, CFG, None, 'puts')\n\n abstract(mark, args.abstractions, args.binary)\n\n\n # -------------------------------------------------------------------------\n # BOPC operation: Application Capability\n # -------------------------------------------------------------------------\n elif args.capabilities:\n # IR is useless; we're measuring capability\n project, CFG = load(args.binary)\n mark = M.mark(project, CFG, None, 'puts')\n\n abstract(mark, args.abstractions, args.binary)\n\n # cfg is loaded with abstractions\n cap = P.capability(CFG, args.binary)\n\n options = 0\n\n for stmt in args.capabilities:\n options = options | {\n 'all' : P.CAP_ALL,\n 'regset' : P.CAP_REGSET,\n 'regmod' : P.CAP_REGMOD,\n 'memrd' : P.CAP_MEMRD,\n 'memwr' : P.CAP_MEMWR,\n 'call' : P.CAP_CALL,\n 'cond' : P.CAP_COND,\n 'load' : P.CAP_LOAD,\n 'save' : P.CAP_SAVE,\n 'noedge' : P.CAP_NO_EDGE\n }[stmt] # argparse ensures no KeyError\n\n cap.build(options=options) # build the Capability Graph\n cap.save() # save nodes to a file\n cap.explore() # explore Islands\n\n capability_analyses( cap )\n\n\n # -------------------------------------------------------------------------\n # BOPC operation: Single block abstraction\n # -------------------------------------------------------------------------\n elif args.binary and args.absblk:\n project = angr.Project(args.binary, load_options={'auto_load_libs': False})\n\n load(args.binary)\n\n abstr = A.abstract_ng(project, int(args.absblk, 0))\n\n dbg_prnt(DBG_LVL_0, 'Abstractions for basic block 0x%x:' % int(args.absblk, 0))\n for a, b in abstr:\n if a == 'regwr':\n dbg_prnt(DBG_LVL_0, '%14s :' % a)\n for c, d in b.iteritems():\n dbg_prnt(DBG_LVL_0, '\\t\\t%s = %s' % (c, str(d)))\n\n else:\n dbg_prnt(DBG_LVL_0, '%14s : %s' % (a, str(b)))\n\n\n # -------------------------------------------------------------------------\n # invalid BOPC operation\n # -------------------------------------------------------------------------\n else:\n fatal('Invalid configuration argument')\n\n\n emph('')\n emph('BOPC has finished.', DBG_LVL_0)\n emph('Have a nice day!', DBG_LVL_0)\n emph('Bye bye :)', DBG_LVL_0)\n\n warn('A segmentation fault may occur now, due to an internal angr issue')\n\n\n\n# ---------------------------------------------------------------------------------------\n" }, { "path": "source/README.md", "content": "\n\n# Block Oriented Programming Compiler (BOPC)\n\n\n___\n\n### BOPC Implementation Overview\n\n![alt text](./images/BOPC_overview.png)\n\n\n### Source Code Overview\n\n\n| File | Description |\n| ---------------------------------|---------------------------------------------|\n| [BOPC.py](./BOPC.py) | Main file |\n| [absblk.py](./absblk.py) | Basic block abstraction |\n| [calls.py](./calls.py) | Supported library and system calls |\n| [capability.py](./capability.py) | Application Capability |\n| [compile.py](./compile.py) | SPL compiler |\n| [config.py](./config.py) | Configuration file |\n| [coreutils.py](./coreutils.py) | Shared utils across modules |\n| [delta.py](./delta.py) | Delta graph |\n| [map.py](./map.py) | Mapping across registers and variables |\n| [mark.py](./mark.py) | Marking and re-Marking CFG |\n| [optimize.py](./optimize.py) | SPL optimizer |\n| [output.py](./output.py) | Write solutions to a file |\n| [path.py](./path.py) | CFG shortest paths |\n| [search.py](./search.py) | Trace Searching algorithm |\n| [simulate.py](./simulate.py) | Concolic execution |\n\n\n___" }, { "path": "source/absblk.py", "content": "#!/#!/usr/bin/env python2\n# -------------------------------------------------------------------------------------------------\n#\n# ,ggggggggggg, _,gggggg,_ ,ggggggggggg, ,gggg, \n# dP\"\"\"88\"\"\"\"\"\"Y8, ,d8P\"\"d8P\"Y8b, dP\"\"\"88\"\"\"\"\"\"Y8, ,88\"\"\"Y8b,\n# Yb, 88 `8b,d8' Y8 \"8b,dPYb, 88 `8b d8\" `Y8\n# `\" 88 ,8Pd8' `Ybaaad88P' `\" 88 ,8Pd8' 8b d8\n# 88aaaad8P\" 8P `\"\"\"\"Y8 88aaaad8P\",8I \"Y88P'\n# 88\"\"\"\"Y8ba 8b d8 88\"\"\"\"\" I8' \n# 88 `8bY8, ,8P 88 d8 \n# 88 ,8P`Y8, ,8P' 88 Y8, \n# 88_____,d8' `Y8b,,__,,d8P' 88 `Yba,,_____, \n# 88888888P\" `\"Y8888P\"' 88 `\"Y8888888 \n#\n# The Block Oriented Programming (BOP) Compiler - v2.1\n#\n#\n# Kyriakos Ispoglou (ispo) - ispo@purdue.edu\n# PURDUE University, Fall 2016-18\n# -------------------------------------------------------------------------------------------------\n#\n#\n# absblk.py:\n#\n# This module implements the basic block \"abstractions\". Abstraction is a process that summarizes\n# a basic block into the \"impact\" on program's state.\n#\n# -------------------------------------------------------------------------------------------------\nfrom coreutils import *\nimport signal\nimport simuvex\nimport claripy\nimport archinfo\nimport angr\n\n\n\n# ------------------------------------------------------------------------------------------------\n# Constant Definitions\n# ------------------------------------------------------------------------------------------------\n_STACK_SZ = 0x1000 # size of symbolic stack\n\n\n\n# -------------------------------------------------------------------------------------------------\n# abstract_ng: This class implements the next generation of the basic block \"abstraction\". So\n# far, the following abstractions are supported:\n# \n# * * Register Writes * *\n# A dictionary that contains all registers that are being written. The \"write\" information is\n# another dictionary with the following fields:\n#\n# * type : Can be 'concrete', 'deref', 'mod' or 'clob'. A register is of type 'clob'\n# when, it does not fall to any of the other types\n# * const : ('concrete' and 'mod' types). The constant value that is written to the\n# register\n# * writable : ('concrete' types). If the constant value is a valid and writable memory\n# address, then this field is set to True\n# * op : ('mod' types). The modification operator\n# * addr : ('deref' types). The address that register value is loaded from\n# * deps : ('deref' types). Any registers that participate in addr field\n# * sym : ('deref' types). A mapping between registers and their symbolic variables\n# * memrd : ('deref' types). When the register write can be used as a memory read, this\n# field contains the size of the memory read in bytes (1,2,4,8). Otherwise it\n# is set to None\n#\n# Example:\n# regwr = {\n# rsp : {'type': 'concrete', 'const': 576460752303357888L, 'writable': True },\n# rcx : {'type': 'deref', 'addr': , 'deps': ['rsi']},\n# r9 : {'type': 'mod', 'op': '+', 'const': 1337L}\n# }\n#\n#\n# * * Memory Reads * *\n# A list of tuples (address, size) for every memory read.\n#\n# Example:\n# memrd = set([(>, 64), (>, 64)])\n#\n#\n# * * Memory Writes * *\n# A list of tuples (address, data) for every memory write (len(data) indicates the size)\n#\n# Example:\n# memwr = set([(>, >), \n# (>, >)])\n#\n#\n# * * Concrete Writes * *\n# A list of tuples (address, size) for every concrete memory write.\n#\n# Example:\n# conwr = set([(576460752303359992L, 64), (576460752303359968L, 64)])\n#\n#\n# * * SPL Memory Writes * *\n# A list of dictionaries for every SPL memory write (memory writes that are in the form:\n# \"mov [rax], rbx\"). Each dictionary contains the following fields:\n#\n# * mem : The register that holds the address to write (string)\n# * val : The register that holds the value to be written (string)\n# * size : The number of bytes to write (e.g., mov [rax], cl, mov [rbx], dx)\n# * sym : A mapping between registers and their symbolic variables\n#\n# Example:\n# splmemwr = [{\n# 'mem' : 'rbx', \n# 'val' : 'rax', \n# 'size' : 4,\n# 'sym' : {'rax': , 'rbx': }\n# }]\n#\n#\n# * * Calls * *\n# A dictionary with the following fields:\n#\n# * type : Can be 'syscall', or 'libcall'\n# * name : The name of the call\n#\n# Example:\n# call = {'type': 'libcall', 'name': u'puts'}\n#\n#\n# * * Conditional Jumps * *\n# A dictionary with the following fields:\n#\n# * form : The form of the conditional jump ('simple' / 'extended')\n# * reg : The register that participates in the conditional jump\n# * const : The constant value that register is compared against\n# * op : The comparison operator\n# * mod_op : ('extended' types). The operator of the register modification\n# * mod_const : ('extended' types). The constant of the register modification\n#\n# Example:\n# cond = {'reg': 'r11', 'op': '==', 'const': 11L}\n# cond = {'mod_op': '^', 'const': 0L, 'form': 'extended', 'op': '=='}\n#\n#\n# * * Symbolic Variables * *\n# A dictionary that maps the symbolic variables to their actual addresses that they correspond\n#\n# Example:\n# symvar = {' : 0x7fffffffffef1e8}\n#\n#\n# * * * ---===== TODO list =====--- *\n#\n# [1]. Make absblk more precise i.e., check the order of memory writes\n# [2]. Move this list at the beginning of the file.\n#\nclass abstract_ng( object ):\n ''' ======================================================================================= '''\n ''' AUXILIARY FUNCTIONS '''\n ''' ======================================================================================= '''\n \n # ---------------------------------------------------------------------------------------------\n # __reg_w(): Analyze the register writes of the symbolic execution.\n #\n # :Arg state: Program's state after symbolic execution\n # :Ret: None.\n #\n def __reg_w( self, state ): \n visited = set() # visited registers\n\n for action in reversed(state.actions): # for every action (start backwards) \n if not (action.type == 'reg' and action.action == 'write'):\n continue # we care about register writes only \n\n try:\n # we only care about the most recent register write only \n reg = self.__proj.arch.register_names[action.offset]\n except KeyError:\n continue\n\n # get the last write only\n if reg not in HARDWARE_REGISTERS or reg in visited:\n continue\n\n data = { } # various data related to the write\n visited.add(reg) # make sure that you won't visit this again\n\n\n # ---------------------------------------------------------------------------\n # If some address (initialized or not) is used as a dereference, the regwr\n # entry for that register must be preserved (we should not overwrite register\n # with the actual value in that address)\n # ---------------------------------------------------------------------------\n if reg in self.regwr and self.regwr[ reg ]['type'] == 'deref':\n continue\n\n # The register is being modified, so we start by marking it as clobbering\n if reg not in self.regwr:\n self.regwr[ reg ] = {'type' : 'clob'}\n\n \n # -----------------------------------------------------------------\n if action.data.concrete: # if register gets a concrete value,\n value = state.se.eval(action.data) # concretize it\n\n data['type'] = 'concrete' # set data\n data['const'] = value\n data['writable'] = True # initialize this first\n in_section = False\n\n # now, check whether this value is a writable address \n try: \n # The problem: There are some weird sections (.e.g., \".comment\") whose VA\n # starts from 0. Therefore, we may have register writes with constants like\n # 1, 2 and so on, which are marked as +W. This means that at the end we can \n # have memory reservations (writes) at those addresses. Our old approach with \n # \"state.memory.permissions(value)\" doesn't work here.\n #\n # So iterate over ELF sections looking for it\n for _, sec in self.__proj.loader.main_object.sections_map.iteritems(): \n # it's possible for the value to be part of >1 sections (usually when\n # section's VA is 0; sec.vaddr != 0). We mark value as +W only when *all*\n # sections are writable\n if sec.contains_addr(value):\n data['writable'] &= sec.is_writable\n in_section = True\n\n\n # if can't find section (b/c it's generated at runtime, like .stack)\n if not in_section:\n # TODO: check if value+1, value+2, etc. are writable as well\n rwx = state.memory.permissions(value)\n\n if state.se.eval(rwx) & 2 == 2: # is +W (2nd bit) set?\n data['writable'] = True\n else:\n data['writable'] = False\n \n except Exception, e: # page does not exist at given address\n data['writable'] = False # not writable at all\n\n try:\n # special case when a stack address is in the next page (-W)\n if value & 0x07ffffffffff0000 == 0x07ffffffffff0000:\n rwx = state.memory.permissions(value-0x4000)\n\n # give it a second change\n if state.se.eval(rwx) & 2 == 2:\n data['writable'] = True\n\n except Exception, e: # or angr.errors.SimMemoryError\n pass\n\n # -----------------------------------------------------------------\n else: # register doesn't get a concrete value\n\n # register gets an expression. Check for simple register modifications: \n # \" = \" (we can easily scale this to = )\n # Note that modified register should be the same with action.offset\n node = [leaf for leaf in action.data.recursive_leaf_asts]\n \n # we need an AST with depth 2, 2 leaves and 1 variable (i.e., register)\n if action.data.depth == 2 and len(action.data.variables) == 1 and len(node) == 2:\n try:\n data['op'] = { # cast operator\n '__add__' : '+',\n '__sub__' : '-',\n '__mul__' : '*',\n '__div__' : '/',\n '__and__' : '&',\n '__or__' : '|',\n '__xor__' : '^',\n '__invert__' : '~',\n '__lshift__' : '<<',\n '__rshift__' : '>>'\n }[ action.data.op ]\n \n # if constant is on the left, swap sides\n if node[0].op == 'BVV' and node[0].concrete:\n node[0], node[1] = node[1], node[0]\n\n\n # check if we're in the form: \n if node[0].op == 'BVS' and self.__symreg[node[0]] == reg and \\\n node[1].op == 'BVV' and node[1].concrete:\n data['type'] = 'mod'\n data['const'] = state.se.eval(node[1])\n else: # not in the right form\n continue\n\n except KeyError: # __symreg() threw an exception\n continue\n\n \n # -----------------------------------------------------------------------\n # Consider the following case:\n # .text:000000000040BA49 mov eax, [rbp+tfd]\n # .text:000000000040BA52 mov edi, eax ; fd\n #\n # Here, edi gets exactly the same value with eax, but edi is marked as\n # 'clob', while eax as 'deref'. The root cause is that edi does not\n # participate in any memory reads and the assigned value is not constant\n # (i.e., it doesn't come directly from a register).\n #\n # To fix that we check whether a 'clob' register has *exactly* the same \n # symbolic value with another one (eax in our example), and if so we \n # assign the same regwr entry to it.\n # -----------------------------------------------------------------------\n else:\n # iterate over previous writes\n for reg2, val in self.__reg_rawval.iteritems():\n try:\n\n # check if raw values match\n if reg != reg2 and val.shallow_repr() == action.data.shallow_repr():\n\n self.regwr[ reg ] = self.regwr[ reg2 ]\n pass\n\n except KeyError:\n pass\n\n\n # -----------------------------------------------------------------\n if data:\n self.regwr[ reg ] = data # set data to this register\n \n\n\n # ---------------------------------------------------------------------------------------------\n # __mem_r(): Analyze the memory reads of the symbolic execution.\n #\n # :Arg state: Program's state after symbolic execution\n # :Ret: None.\n #\n def __mem_r( self, state ):\n for action in state.actions: # for every action \n if not (action.type == 'mem' and action.action == 'read'):\n continue # we care about memory reads only\n\n # simply add address (can be an expression) and size to the list\n self.memrd.add( (action.addr, len(action.data)) )\n\n\n\n # ---------------------------------------------------------------------------------------------\n # __mem_w(): Analyze the memory writes of the symbolic execution.\n #\n # :Arg state: Program's state after symbolic execution\n # :Ret: None.\n #\n def __mem_w( self, state ):\n for action in state.actions: # for every action \n if not (action.type == 'mem' and action.action == 'write'):\n continue # we care about memory writes only\n\n # simply add address (can be an expression) and data to the list\n self.memwr.add( (action.addr, action.data) ) \n \n if action.addr.concrete: # if address is concrete\n # concretize it as well\n self.conwr.add( (state.se.eval(action.addr), len(action.data)) )\n\n\n deps = [ ]\n symtab = { }\n\n # -----------------------------------------------------------------\n # Check for memory register writes (mov [rax], rbx)\n #\n # In this case, both action.addr and action.data will consist of a\n # single leaf in their ast which is a register\n # -----------------------------------------------------------------\n mem_reg = [leaf for leaf in action.addr.recursive_leaf_asts]\n val_reg = [leaf for leaf in action.data.recursive_leaf_asts]\n\n\n # print 'ADDR', mem_reg, action.addr\n # print 'ADDR', val_reg, action.addr\n \n # check AST have a single leaf\n if len(mem_reg) == 1 and len(val_reg) == 1:\n mem, val = None, None\n\n # check whether the leaf is a register\n for sym, nam in self.__symreg.iteritems():\n # skip registers that are not symbolic (e.g., rbp)\n if isinstance(sym.args[0], str) and sym.args[0] in mem_reg[0].shallow_repr(): \n symtab[nam] = sym\n mem = nam\n\n elif isinstance(sym.args[0], str) and sym.args[0] in val_reg[0].shallow_repr(): \n symtab[nam] = sym\n val = nam\n\n # if both leaves are registers we have a memory register write!\n if mem and val: \n self.splmemwr.append({\n 'mem' : mem,\n 'val' : val,\n 'size' : int(action.size) >> 3,\n 'sym' : symtab, \n })\n\n\n\n # ---------------------------------------------------------------------------------------------\n # __call(): Analyze the (sys|lib)calls of the symbolic execution. Because we're analyzing a\n # single basic block, we can have up to one such (sys|lib)call (the last instruction).\n #\n # :Arg state: Program's state after symbolic execution\n # :Ret: None.\n #\n def __call( self, state ):\n blk = self.__proj.factory.block(self.__entry)\n\n # check if symbolic execution stopped on a syscall\n # (don't use \"if self.__proj._simos.is_syscall_addr(state.addr)\"; it throws exceptions)\n if blk.vex.jumpkind == \"Ijk_Sys_syscall\":\n # a system call was invoked\n # we assume that simproc.cc == SimCCAMD64LinuxSyscall \n simproc = self.__proj._simos.syscall(state)\n\n self.call['type'] = 'syscall'\n self.call['name'] = simproc.display_name\n # self.call['nargs'] = simproc.num_args\n\n else: \n if blk.vex.jumpkind != \"Ijk_Call\": # skip block when it doesn't end with a call\n return\n\n\n # check if symbolic execution stopped on a library call\n for action in reversed(state.actions): # for every action \n if action.type != 'exit':\n continue # we care about branches only\n\n\n # concretize function's entry point\n target = state.se.eval(action.target)\n\n # Note: Before you use kb.functions, calculate CFG (e.g., analyses.CFGFast())\n try:\n self.call['type'] = 'libcall'\n self.call['name'] = self.__proj.kb.functions[target].name\n except Exception: # no function name at that address\n self.call = { }\n\n\n\n # ---------------------------------------------------------------------------------------------\n # __cond(): Analyze the conditional jump of the symbolic execution. Because we're analyzing a\n # single basic block, we can have up to one conditional jump.\n #\n # :Arg state: Program's state after symbolic execution\n # :Ret: None.\n #\n def __cond( self, state ): \n for action in reversed(state.actions): # for every action \n if not (action.type == 'exit' and action.exit_type == 'conditional'):\n continue # we care about conditional jumps only\n \n\n # as in __reg_w(), we only care about simple conditional jumps: \" \"\n if len(action.condition.variables) == 1: \n try:\n self.cond['op'] = { # cast operator\n '__eq__' : '==',\n '__ne__' : '!=',\n '__le__' : '<=',\n '__lt__' : '<',\n '__ge__' : '>=',\n '__gt__' : '>',\n\n 'SGT' : '>', \n 'SGE' : '>=',\n 'SLT' : '<',\n 'SLE' : '<=', \n 'UGT' : '>', # do not distinguish signed/unsigned operators\n 'UGE' : '>=',\n 'ULT' : '<',\n 'ULE' : '<=',\n }[ action.condition.op ]\n except KeyError: \n warn('Unknown conditional jump operator \"%s\"' % action.condition.op)\n self.cond = { }\n return\n\n \n node = [leaf for leaf in action.condition.recursive_leaf_asts]\n\n\n # -----------------------------------------------------------------------\n # Check if we're in the simple form: \n # -----------------------------------------------------------------------\n if len(node) == 2: # we need 2 leaves + 1 operator\n self.cond['form'] = 'simple' # we're in the simple form\n\n try:\n # swap register and constant if needed\n if node[1].op == 'BVS' and node[0].op == 'BVV' and node[0].concrete:\n node[0], node[1] = node[1], node[0]\n\n\n # if we're in the right form (reg and const), we have our condition\n if node[0].op == 'BVS' and node[1].op == 'BVV' and node[1].concrete:\n self.cond['reg'] = self.__symreg[node[0]]\n self.cond['const'] = state.se.eval(node[1])\n else:\n self.cond = { } # not in the right form\n return\n\n except KeyError: \n # if not in the right form, __symreg() will throw a KeyError exception\n self.cond = { }\n return\n\n\n # -----------------------------------------------------------------------\n # Check if we're in the extended form: ( ) \n # (example: \">\")\n # \n # This is when the iterator (register) gets modified and compared at the\n # same basic block.\n # -----------------------------------------------------------------------\n elif len(node) == 3: # we need 3 leaves and 2 operators\n self.cond['form'] = 'extended' # we're in the extended form\n\n try:\n # get left and right side of the comparison\n left, right = action.condition.split( action.condition.op )\n\n # if the constant is on the left side, swap sides\n if left.op == 'BVV' and left.concrete:\n left, right = right, left\n\n\n mod_ops = { # register modification operations\n '__add__' : '+',\n '__sub__' : '-',\n '__mul__' : '*',\n '__div__' : '/',\n '__and__' : '&',\n '__or__' : '|',\n '__xor__' : '^',\n '__invert__' : '~',\n '__lshift__' : '<<',\n '__rshift__' : '>>'\n }\n\n \n # if the left side is a modification and the right side a constant\n if left.op in mod_ops and right.op == 'BVV' and right.concrete:\n self.cond['const'] = state.se.eval(right)\n self.cond['mod_op'] = mod_ops[ left.op ]\n\n reg, const = left.split( left.op )\n\n # if the constant is on the left side, swap sides\n if reg.op == 'BVV' and reg.concrete:\n reg, const = const, reg\n\n # if the modification uses a constant and a register\n if reg.op == 'BVS' and reg in self.__symreg and \\\n const.op == 'BVV' and const.concrete:\n self.cond['reg'] = self.__symreg[reg]\n self.cond['mod_const'] = state.se.eval(const)\n else:\n self.cond = { } # something is not in the right form\n return \n else:\n self.cond = { }\n return \n \n except ValueError: # != 2 values to split()\n self.cond = { }\n return\n\n\n # -----------------------------------------------------------------------\n # Otherwise we're not in the right form\n # -----------------------------------------------------------------------\n else:\n self.cond = { }\n continue\n\n\n # The problem here, is that simgr sometimes \"inverts\" the condition, so the \n # \"target\" basic block is the block immediately after the current block. To \n # be consistent, we have to \"invert\" the operator, so the target basic block\n # is executed when the jump is taken.\n blk = self.__proj.factory.block(self.__entry) \n\n # check if the target is the next block (assume action.target is concrete)\n if state.se.eval(action.target) == blk.addr + blk.size:\n self.cond['op'] = { # invert the condition\n '==' : '!=',\n '!=' : '==',\n '>' : '<=',\n '>=' : '<',\n '<' : '>=',\n '<=' : '>'\n }[ self.cond['op'] ] \n\n break # there's up to 1 conditional jump\n\n\n\n # ---------------------------------------------------------------------------------------------\n # __add_sym_vars(): This function extracts all (memory) symbolic variables from an expression.\n # For instance, given the expression: , we want to\n # map the variable 'mem_7fffffffffef1e8_82_64' to its actual address: 0x7fffffffffef1e8.\n #\n # :Arg addr_expr: The address expression to get variables from\n # :Ret: None.\n #\n def __add_sym_vars( self, addr_expr ):\n # A memory symbolic variable is in the form: mem_ADDRESS_RANDOM_SIZE. The AST leaf\n # will be like this: \"\"\n #\n # We want to extract the ADDRESS and SIZE fields\n for leaf in addr_expr.recursive_leaf_asts: # for each leaf in the AST\n leafstr = leaf.shallow_repr() # cast it to sting\n\n # if leaf is a memory variable, extract its address and its size\n if re.search(r'mem_[0-9a-f]+_[0-9]+_[0-9]+', leafstr):\n _, addr, rand, size = leafstr.split('_')\n\n # size might be followed by the \"{UNINITIALIZED}\" keyword, so it must be dropped\n # if not the \">\" must also be dropped\n size = size.replace(\"{UNINITIALIZED}>\", \"\").replace(\">\", \"\")\n\n # add the symbolic variable to the map\n self.symvars[ leaf ] = (int(addr, 16), int(size, 10) >> 3)\n\n\n\n # ---------------------------------------------------------------------------------------------\n # __memread_callback(): This function is invoked every time that a memory read operation is \n # performed.\n #\n # :Arg state: Current state to read memory from\n # :Ret: None.\n #\n def __memread_callback( self, state ):\n if self.__callback_mutex == 1: # if mutex is taken, return\n return\n \n self.__callback_mutex = 1 # get lock\n\n # ---------------------------------------------------------------------\n # If address is part of the .bss/.data, it will be initialized with a\n # default value of 0. However, it can get any value (due to AWP) so it\n # should get a symbolic value.\n # ---------------------------------------------------------------------\n # get ELF sections that give default values to their uninitialized variables\n bss = self.__proj.loader.main_object.sections_map[\".bss\"]\n data = self.__proj.loader.main_object.sections_map[\".data\"]\n\n addr = state.se.eval(state.inspect.mem_read_address)\n # print '=== READ', hex(state.inspect.instruction), hex(addr)\n\n # check if address is inside .bss or .data sections\n if bss.min_addr <= addr and addr <= bss.max_addr or \\\n data.min_addr <= addr and addr <= data.max_addr:\n # This is also works, but is for Big Endian:\n # state.memory.make_symbolic('mem', state.inspect.mem_read_address, length)\n\n # make address symbolic\n symv = state.se.BVS(\"mem_%x\" % addr, state.inspect.mem_read_length << 3)\n \n state.memory.store(state.inspect.mem_read_address, symv, \n state.inspect.mem_read_length, endness=archinfo.Endness.LE)\n\n # we should read it to update state.inspect.mem_read_expr\n state.memory.load(state.inspect.mem_read_address,\n state.inspect.mem_read_length, endness=archinfo.Endness.LE)\n\n\n # -------------------------------------------------------------------------------\n # Identifying dereferences is a two stage process. Here (1st step) we capture all\n # memory load information (which happens before the register write) that happen \n # at this instruction (x64 has 1 distinct memory read per insruction; however \n # instructions like popad do multiple register writes, but this is not an issue \n # here).\n # -------------------------------------------------------------------------------\n self.__load[ state.inspect.instruction ] = (\n state.inspect.mem_read_address, \n state.inspect.mem_read_length, \n state.inspect.mem_read_expr # this will be updated\n )\n\n # associate memory expression with memory address (needed for later on)\n self.__mem2addr[ state.inspect.mem_read_expr.shallow_repr() ] = \\\n (state.inspect.mem_read_address, state.inspect.mem_read_length)\n \n # extract memory symbolic variables\n self.__add_sym_vars( state.inspect.mem_read_address ) \n\n self.__callback_mutex = 0 # release lock\n\n \n\n # ---------------------------------------------------------------------------------------------\n # __regwrite_callback(): This function is invoked every time that a register write operation\n # is performed.\n #\n # :Arg state: Current state to write register to\n # :Ret: None.\n #\n def __regwrite_callback( self, state ):\n if self.__callback_mutex == 1: # if mutex is taken, return\n return\n\n self.__callback_mutex = 1 # get lock\n \n try:\n # get register that is being written\n reg = self.__proj.arch.register_names[state.inspect.reg_write_offset]\n except KeyError: # just in case\n return\n\n\n # TODO: Regwrite only checks writes, but it doesn't check if the previous value perists after\n # .text:000000000040BCEA mov eax, [rbp+ac]\n # .text:000000000040BCF0 cdqe\n # .text:000000000040BCF2 shl rax, 3\n # .text:000000000040BCF6 mov rcx, rax\n # .text:000000000040BCF9 add rcx, [rbp+nargv]\n # \n # ('sudo' example)\n #\n # We should add some checks to test whether the regwrite is \"mov\" or something else\n\n\n # print '--------------- ', hex(state.addr), hex(state.inspect.instruction), reg, \n # state.inspect.reg_write_expr\n\n\n # remember the \"raw\" value that is being written to the register\n self.__reg_rawval[ reg ] = state.inspect.reg_write_expr\n\n if reg not in HARDWARE_REGISTERS: # we only care about specific registers\n self.__callback_mutex = 0 # release lock\n return \n\n\n # -------------------------------------------------------------------------------\n # This is the 2nd step of the dereference identification process. At this point \n # we match the instruction that writes a register with the instruction that read\n # from memory. This is because we want to match the memory read expression with\n # the register write.\n # -------------------------------------------------------------------------------\n elif state.inspect.instruction in self.__load:\n addr, length, _ = self.__load[ state.inspect.instruction ]\n\n\n # ok we have a dereference!\n deps = [ ] # dependent registers\n symtab = { }\n\n # find register dependencies on the address (e.g., rsi on )\n for sym, nam in self.__symreg.iteritems():\n # skip registers that are not symbolic (e.g., rbp)\n if isinstance(sym.args[0], str) and sym.args[0] in addr.shallow_repr():\n deps.append(nam)\n symtab[nam] = sym\n\n\n # there might be dependencies with constant memory addresses as well (i.e., reading\n # from global variables). Such dependencies are handled during trace searching, so \n # we ignore them for now. However the register dependencies are needed to check\n # whether a register mapping is valid or not.\n\n\n # if \"deps\" has a single element, we know that a register is containted in \"addr\"\n # expression. If also that expression has a single node, we know that this will be\n # that register.\n if len(deps) == 1 and len([leaf for leaf in addr.recursive_leaf_asts]) == 1:\n memrd = length\n else:\n memrd = None\n\n \n # (if basic block has >1 dereferences on the same register, use the most recent one)\n self.regwr[ reg ] = { # set data\n 'type' : 'deref',\n 'addr' : addr,\n 'deps' : deps,\n 'sym' : symtab,\n 'memrd' : memrd\n }\n\n\n # -------------------------------------------------------------------------------\n # The current approach for detecting dereferences is not transitive. Consider the\n # following example:\n # mov rcx, [rsi + 0x10]\n # mov rdi, rcx\n #\n # In the 2nd register write, rdi gets an unconstrained symbolic variable (e.g., \n # >) and therefore it's of\n # type 'clob'. However, we want rdi to be treated in the same way with rcx, as\n # they both have the exact same value. Because SE engine gives a unique symbolic\n # variable on every memory cell, we can associate them with their addresses. \n # Thus, when a register gets a random symbolic value, we can figure out whether\n # it is actually a dereference.\n # -------------------------------------------------------------------------------\n elif state.inspect.reg_write_expr.shallow_repr() in self.__mem2addr:\n addr, length = self.__mem2addr[ state.inspect.reg_write_expr.shallow_repr() ]\n\n # this code is copy-pasta from above\n deps = [ ]\n symtab = { }\n\n for sym, nam in self.__symreg.iteritems():\n if isinstance(sym.args[0], str) and sym.args[0] in addr.shallow_repr():\n deps.append(nam)\n symtab[nam] = sym\n\n\n if len(deps) == 1 and len([leaf for leaf in addr.recursive_leaf_asts]) == 1:\n memrd = length\n else:\n memrd = None\n\n\n self.regwr[ reg ] = {\n 'type' : 'deref',\n 'addr' : addr,\n 'deps' : deps,\n 'sym' : symtab,\n 'memrd' : memrd\n }\n \n\n # -------------------------------------------------------------------------------\n\n self.__callback_mutex = 0 # release lock\n\n\n\n # ---------------------------------------------------------------------------------------------\n # __sig_handler(): Symbolic execution may take forever to complete. To deal with it, we set\n # an alarm. When the alarm is triggered, this singal handler is invoked and throws an\n # exception that causes the symbolic execution to halt.\n #\n # :Arg signum: Signal number\n # :Arg frame: Current stack frame\n # :Ret: None.\n #\n def __sig_handler( self, signum, frame ): \n if signum == signal.SIGALRM: # we only care about SIGALRM\n\n # angr may ignore the exception, so let's throw many of them :P\n raise Exception(\"Alarm triggered after %d seconds\" % ABSBLK_TIMEOUT)\n raise Exception(\"Alarm triggered after %d seconds\" % ABSBLK_TIMEOUT)\n raise Exception(\"Alarm triggered after %d seconds\" % ABSBLK_TIMEOUT)\n raise Exception(\"Alarm triggered after %d seconds\" % ABSBLK_TIMEOUT)\n\n\n\n # ---------------------------------------------------------------------------------------------\n\n ''' ======================================================================================= '''\n ''' CLASS INTERFACE '''\n ''' ======================================================================================= '''\n\n # ---------------------------------------------------------------------------------------------\n # __init__(): Class constructor. This function initializes the environment for the symbolic\n # execution, it executes the basic block, and performs the abstraction.\n #\n # :Arg project: Instance of angr project\n # :Arg addr: Entry point of the basic block\n # :Ret: None.\n #\n def __init__( self, project, addr ):\n self.__proj = project # we'll need these\n self.__entry = addr\n\n \n # ---------------------------------------------------------------------\n # initialize abstraction variables\n # ---------------------------------------------------------------------\n self.regwr = { } # all register writes for that block\n self.memrd = set() # all memory reads for that block\n self.memwr = set() # all memory writes for that block\n self.conwr = set() # all concrete memory writes for that block\n self.splmemwr = [ ] # all memory register writes for that block\n self.call = { } # function/system call (if any) for that block\n self.cond = { } # conditional jumps (if any) for that block\n self.symvars = { } # symbolic variables for memory\n self.__load = { } # memory loads (for internal use)\n self.__mem2addr = { } # map between memory expressions and addresses\n\n self.__mem = { }\n self.__reg_rawval = { }\n\n # ---------------------------------------------------------------------\n # Create a blank state and prepare it for symbolic execution.\n #\n # TODO: Check options again\n # ---------------------------------------------------------------------\n inist = self.__proj.factory.blank_state( # create a blank state\n addr=addr, # set address\n #mode='symbolic', \n add_options={ # configure options\n simuvex.o.AVOID_MULTIVALUED_READS,\n simuvex.o.AVOID_MULTIVALUED_WRITES,\n simuvex.o.NO_SYMBOLIC_JUMP_RESOLUTION,\n simuvex.o.CGC_NO_SYMBOLIC_RECEIVE_LENGTH,\n simuvex.o.NO_SYMBOLIC_SYSCALL_RESOLUTION,\n simuvex.o.TRACK_ACTION_HISTORY,\n \n # newly added option\n simuvex.o.SYMBOLIC_INITIAL_VALUES\n },\n remove_options=simuvex.o.resilience_options | simuvex.o.simplification \n )\n\n # configure more options (add/remove)\n inist.options.discard(simuvex.o.CGC_ZERO_FILL_UNCONSTRAINED_MEMORY)\n inist.options.update( {\n simuvex.o.TRACK_REGISTER_ACTIONS,\n simuvex.o.TRACK_MEMORY_ACTIONS,\n simuvex.o.TRACK_JMP_ACTIONS,\n simuvex.o.TRACK_CONSTRAINT_ACTIONS }\n )\n\n \n # ---------------------------------------------------------------------\n # initialize all registers with a symbolic variable\n # ---------------------------------------------------------------------\n inist.regs.rax = inist.se.BVS(\"rax\", 64) # give convenient names\n inist.regs.rbx = inist.se.BVS(\"rbx\", 64)\n inist.regs.rcx = inist.se.BVS(\"rcx\", 64)\n inist.regs.rdx = inist.se.BVS(\"rdx\", 64)\n inist.regs.rsi = inist.se.BVS(\"rsi\", 64)\n inist.regs.rdi = inist.se.BVS(\"rdi\", 64)\n\n\n # rbp may also needed as it's mostly used to access local variables (e.g., \n # rax = [rbp-0x40]) but some binaries don't use rbp and all references are\n # rsp related. In these cases it may worth to use rbp as well.\n if MAKE_RBP_SYMBOLIC:\n inist.regs.rbp = inist.se.BVS(\"rbp\",64) # keep rbp symbolic\n else:\n inist.registers.store('rbp', FRAMEPTR_BASE_ADDR, size=8, endness=archinfo.Endness.LE)\n \n # rsp must be concrete and properly initialized\n inist.registers.store('rsp', RSP_BASE_ADDR, size=8, endness=archinfo.Endness.LE)\n\n inist.regs.r8 = inist.se.BVS(\"r08\", 64)\n inist.regs.r9 = inist.se.BVS(\"r09\", 64)\n inist.regs.r10 = inist.se.BVS(\"r10\", 64)\n inist.regs.r11 = inist.se.BVS(\"r11\", 64)\n inist.regs.r12 = inist.se.BVS(\"r12\", 64)\n inist.regs.r13 = inist.se.BVS(\"r13\", 64)\n inist.regs.r14 = inist.se.BVS(\"r14\", 64)\n inist.regs.r15 = inist.se.BVS(\"r15\", 64)\n\n\n # ---------------------------------------------------------------------\n # Other initializations\n # --------------------------------------------------------------------- \n # map symbolic names to registers\n\n # self.__symreg = { self.__getreg(inist, r):r for r in HARDWARE_REGISTERS }\n self.__symreg = { \n inist.regs.rax : 'rax',\n inist.regs.rbx : 'rbx',\n inist.regs.rcx : 'rcx',\n inist.regs.rdx : 'rdx',\n inist.regs.rsi : 'rsi',\n inist.regs.rdi : 'rdi',\n inist.regs.rbp : 'rbp',\n inist.regs.rsp : 'rsp',\n inist.regs.r8 : 'r8',\n inist.regs.r9 : 'r9',\n inist.regs.r10 : 'r10',\n inist.regs.r11 : 'r11',\n inist.regs.r12 : 'r12',\n inist.regs.r13 : 'r13',\n inist.regs.r14 : 'r14',\n inist.regs.r15 : 'r15'\n }\n\n\n # UPDATE: Don't create a symbolic stack, as this consumes all the Virtual Memory and\n # may crash the machine. By carefully configuring rsp and rbp within the limit of virtual\n # page limit, we can achieve the same effect, so we don't need a symbolic stack.\n #\n # The main issue here are the permissions (stack may not appear as R+W), but as long as\n # both rsp and rbp point in the same page, there is no problem.\n #\n #\n # # create a symbolic stack (required to have writable pages)\n # stack = inist.se.BVS(\"stack\", self.__proj.arch.bits * _STACK_SZ) \n #\n # # write symbolic stack to memory \n # # inist.memory.store(inist.regs.sp, stack, endness=archinfo.Endness.LE) \n # inist.memory.store(STACK_BASE_ADDR, stack, endness=archinfo.Endness.LE)\n\n # when solver gives up (in milliseconds)\n inist.se._solver.timeout = ABSBLK_TIMEOUT*1000\n\n\n # ---------------------------------------------------------------------\n # Hooks for identifying dereferences\n # ---------------------------------------------------------------------\n self.__callback_mutex = 0 # hooks are enabled\n\n inist.inspect.b('reg_write', when=angr.BP_BEFORE, action=self.__regwrite_callback)\n inist.inspect.b('mem_read', when=angr.BP_AFTER, action=self.__memread_callback)\n \n \n # -------------------------------------------------------------------------\n # Do the symbolic execution (using simulation managers)\n # ------------------------------------------------------------------------- \n simgr = self.__proj.factory.simulation_manager(thing=inist)\n simgr.save_unconstrained = True # do not discard unconstrained stashes\n\n\n signal.signal(signal.SIGALRM, self.__sig_handler)\n signal.alarm(ABSBLK_TIMEOUT) \n\n\n # make sure that you execute the normalized block\n # TODO: cleanup\n node = ADDR2NODE[self.__entry]\n num_inst = len(node.instruction_addrs) if node is not None else None\n if num_inst:\n simgr.step(num_inst=num_inst)\n \n else:\n simgr.step() # execute 1 basic block\n \n signal.alarm(0) # disable alarm\n\n\n if simgr.active: # check if execution was successful\n newst = simgr.active[0] # get the new state (after execution)\n\n elif simgr.unconstrained:\n # because we execute a single basic block, it's possible to end up in an state that\n # instruction pointer depends on symbolic data and hence to not know how to proceed\n # (i.e., unconstrained stash)\n newst = simgr.unconstrained[0]\n\n elif simgr.deadended: # check if execution can't continue (retq)\n newst = simgr.deadended[0] # work with what you have\n \n else: # everything else should generate an error\n print simgr.stashes\n raise Exception('There are no usable stashes!')\n\n\n # -------------------------------------------------------------------------\n # Analyze results and generate the abstractions\n # ------------------------------------------------------------------------- \n self.__reg_w(newst) # analyze register writes\n self.__mem_r(newst) # analyze memory reads\n self.__mem_w(newst) # analyze memory writes\n self.__call(newst) # analyze function/system calls\n self.__cond(newst) # analyze conditional jumps\n\n\n # -------------------------------------------------------------------------\n # Apply (any) patches\n #\n # Instructions like 'rep movsq' incorrectly classify rsi and rdi in 'deref'\n # types. This is because angr assigns a basic block with a single rep* \n # instruction (as VEX IR contains loops). To fix that, we simply mark the\n # used registers as clobbering.\n # ------------------------------------------------------------------------- \n blk_insns = node.block.capstone.insns # get block instructions\n\n if len(blk_insns) == 1 and 'rep' in blk_insns[0].insn.mnemonic:\n # name = blk_insns[0].insn.insn_name() # get instruction name (w/o the rep*)\n \n # make 'rsi', 'rdi' and 'rcx' clobbering (all of them are modified)\n self.regwr['rdi'] = {'type' : 'clob'} \n self.regwr['rsi'] = {'type' : 'clob'}\n self.regwr['rcx'] = {'type' : 'clob'} \n\n\n '''\n print\n print '-------------------- Register Writes --------------------' \n for a, b in self.regwr.iteritems():\n print a, b\n\n print '-------------------- Memory Reads --------------------' \n for a, b in self.memrd:\n print a, b\n\n print '-------------------- Memory Writes --------------------' \n for a, b in self.memwr:\n print a, b\n\n print '-------------------- Concrete Writes --------------------' \n for a, b in self.conwr:\n print a, b\n\n print '-------------------- SPL Memory Writes --------------------' \n for a in self.splmemwr:\n print a\n\n print '-------------------- Calls --------------------' \n print self.call\n\n print '-------------------- Conditional Jumps --------------------' \n print self.cond\n '''\n\n\n\n # ---------------------------------------------------------------------------------------------\n # __getitem__(): An alternative way to get block \"abstractions\". \n #\n # :Arg what: The name of the abstraction that you want to get\n # :Ret: The requested abstraction.\n # \n def __getitem__( self, what ):\n try:\n return {\n 'regwr' : self.regwr,\n 'memrd' : self.memrd,\n 'memwr' : self.memwr,\n 'conwr' : self.conwr,\n 'splmemwr' : self.splmemwr,\n 'call' : self.call,\n 'cond' : self.cond,\n 'symvars' : self.symvars\n }[ what ]\n except KeyError:\n return None # abstraction not found\n\n\n\n # ---------------------------------------------------------------------------------------------\n # __iter__(): Iterate over all abstractions. This function is a generator over all possible\n # abstractions.\n #\n # :Ret: Each time function returns a different tuple (name, abstraction).\n # \n def __iter__( self ): \n yield 'regwr', self.regwr\n yield 'memrd', self.memrd\n yield 'memwr', self.memwr\n yield 'conwr', self.conwr\n yield 'splmemwr', self.splmemwr\n yield 'call', self.call\n yield 'cond', self.cond\n yield 'symvars', self.symvars \n\n\n\n# -------------------------------------------------------------------------------------------------\n'''\nif __name__ == '__main__': # DEBUG ONLY\n import angr\n\n project = angr.Project('eval/opensshd/sshd', load_options={'auto_load_libs': False}) \n # project.analyses.CFGFast() # to prepare project.kb.functions\n\n # Problem: Inidirect pointers in .bss:\n # .text:00000000004050B1 mov rax, cs:public_key\n # .text:00000000004050B8 mov rdi, [rax+20h] ; value\n #\n # abstr = abstract_ng(project, 0x4050B1)\n\n # abstr = abstract_ng(project, 0x416610)\n abstr = abstract_ng(project, 0x416631)\n\n # TODO: check me again!\n abstr = abstract_ng(project, 0x0x40c01f)\n\n for a, b in abstr:\n print '\\t', a, b\n\n print 'done!'\n'''\n# -------------------------------------------------------------------------------------------------\n" }, { "path": "source/calls.py", "content": "#!/usr/bin/env python2\n# -------------------------------------------------------------------------------------------------\n#\n# ,ggggggggggg, _,gggggg,_ ,ggggggggggg, ,gggg, \n# dP\"\"\"88\"\"\"\"\"\"Y8, ,d8P\"\"d8P\"Y8b, dP\"\"\"88\"\"\"\"\"\"Y8, ,88\"\"\"Y8b,\n# Yb, 88 `8b,d8' Y8 \"8b,dPYb, 88 `8b d8\" `Y8\n# `\" 88 ,8Pd8' `Ybaaad88P' `\" 88 ,8Pd8' 8b d8\n# 88aaaad8P\" 8P `\"\"\"\"Y8 88aaaad8P\",8I \"Y88P'\n# 88\"\"\"\"Y8ba 8b d8 88\"\"\"\"\" I8' \n# 88 `8bY8, ,8P 88 d8 \n# 88 ,8P`Y8, ,8P' 88 Y8, \n# 88_____,d8' `Y8b,,__,,d8P' 88 `Yba,,_____, \n# 88888888P\" `\"Y8888P\"' 88 `\"Y8888888 \n#\n# The Block Oriented Programming (BOP) Compiler - v2.1\n#\n#\n# Kyriakos Ispoglou (ispo) - ispo@purdue.edu\n# PURDUE University, Fall 2016-18\n# -------------------------------------------------------------------------------------------------\n#\n#\n# calls.py\n#\n# This module contains all declarations for system and library calls that SPL supports. A call is\n# declared as a tuple (name, nargs, modregs):\n#\n# name : The library/system call name\n# nargs : The number of its arguments. Set to INFINITY for variadic functions.\n# modregs : A list of all registers that are modified when the call returns. Note that rax \n# is always modified as it has the return value.\n#\n# To keep the implementation simple, We do not support library calls that take arguments on the\n# stack.\n#\n# Also, it is possible to declare any custom calls that reside in the binary.\n# -------------------------------------------------------------------------------------------------\nfrom coreutils import *\n\n\n\n# -------------------------------------------------------------------------------------------------\n# Calling Conventions\n# -------------------------------------------------------------------------------------------------\nSYSCALL_CC = ['rdi', 'rsi', 'rdx', 'rcx', 'r8', 'r9']\nLIBCALL_CC = ['rdi', 'rsi', 'rdx', 'r10', 'r8', 'r9']\n\n\n\n# -------------------------------------------------------------------------------------------------\n# Supported system calls\n# -------------------------------------------------------------------------------------------------\nsyscalls__ = [\n # ssize_t read(int fd, void *buf, size_t count)\n ('read', 3, ['rax', 'rcx', 'r10', 'r11']),\n\n # ssize_t write(int fd, const void *buf, size_t count)\n ('write', 3, ['rax', 'rcx', 'r10', 'r11']),\n\n # void *sbrk(intptr_t increment)\n ('sbrk', 1, ['rax', 'rcx', 'rdx', 'r10', 'r11']),\n\n # int brk(void *addr)\n ('brk', 1, ['rax', 'rcx', 'rdx', 'r10', 'r11']),\n\n # int dup(int oldfd)\n ('dup', 1, ['rax', 'rcx', 'r11']),\n\n # int dup2(int oldfd, int newfd)\n ('dup2', 2, ['rax', 'rcx', 'r10', 'r11']),\n\n # unsigned int alarm(unsigned int seconds)\n ('alarm', 1, ['rax', 'rcx', 'r10', 'r11']),\n\n\n '''\n Feel free to append more syscalls...\n '''\n]\n\n\n\n# -------------------------------------------------------------------------------------------------\n# Supported library calls\n# -------------------------------------------------------------------------------------------------\nlibcalls__ = [\n # int system(const char *command)\n ('system', 1, ['rax', 'rcx', 'rdx', 'rdi', 'rsi', 'r8', 'r9', 'r10', 'r11']),\n\n # int puts(const char *s)\n ('puts', 1, ['rax', 'rcx', 'rdx', 'rdi', 'rsi', 'r8', 'r9', 'r10', 'r11']),\n\n # int execve(const char *filename, char *const argv[], char *const envp[])\n ('execve', 3, ['rax', 'rcx', 'rdx', 'r10', 'r11']),\n\n # int execv(const char *filename, char *const argv[])\n ('execv', 2, ['rax', 'rcx', 'rdx', 'r10', 'r11']),\n \n # int execl(const char *path, const char *arg, ...);\n ('execl', 2, ['rax', 'rcx', 'rdx', 'r10', 'r11']),\n\n # int printf(const char *format, ...)\n ('printf', INFINITY, ['rax', 'rcx', 'rdx', 'rsi', 'rdi', 'r8', 'r10', 'r11']),\n\n # ssize_t send(int sockfd, const void *buf, size_t len, int flags);\n # (we can ignore the 4th parameter for now)\n ('send', 3, []),\n\n # void exit(int status)\n ('exit', 1, []),\n\n\n '''\n Feel free to append more libcalls...\n '''\n]\n\n\n\n# -------------------------------------------------------------------------------------------------\n# In case that you don't want to distinguish them\n# -------------------------------------------------------------------------------------------------\ncalls__ = syscalls__ + libcalls__\n\n\n\n# -------------------------------------------------------------------------------------------------\n# Groups of function calls that have similar effects\n# -------------------------------------------------------------------------------------------------\ncall_groups__ = [\n ['puts', 'printf'],\n ['execve', 'execv', 'execl' ],\n]\n\n\n\n# -------------------------------------------------------------------------------------------------\n# find_syscall(): Search for a specific system call.\n#\n# :Arg name: Name of the syscall\n# :Ret: If system call exists, function returns the associated entry in syscalls__. Otherwise None\n# is returned.\n#\ndef find_syscall( name ):\n call = filter(lambda call: call[0] == name, syscalls__)\n\n if len(call) == 0:\n return None\n\n elif len(call) == 1:\n return call[0]\n\n else:\n raise Exception(\"System call '%s' has >1 entries in syscalls__ table.\" % name)\n\n\n\n# -------------------------------------------------------------------------------------------------\n# find_libcall(): Search for a specific library call.\n#\n# :Arg name: Name of the library call\n# :Ret: If library call exists, function returns the associated entry in libcalls__. Otherwise None\n# is returned.\n#\ndef find_libcall( name ):\n call = filter(lambda call: call[0] == name, libcalls__)\n\n if len(call) == 0:\n return None\n\n elif len(call) == 1:\n return call[0]\n\n else:\n raise Exception(\"Library call '%s' has >1 entries in libcalls__ table.\" % name)\n\n\n\n# -------------------------------------------------------------------------------------------------\n# find_call(): Search for a specific call (either library or system)\n#\n# :Arg name: Name of the call\n# :Ret: If call exists, function returns the associated entry in calls__. Otherwise None is\n# returned.\n#\ndef find_call( name ):\n sys = find_syscall(name)\n lib = find_libcall(name)\n\n return sys if sys else lib # logic OR\n\n\n\n# -------------------------------------------------------------------------------------------------\n" }, { "path": "source/capability.py", "content": "#!/usr/bin/env python2\n# -------------------------------------------------------------------------------------------------\n#\n# ,ggggggggggg, _,gggggg,_ ,ggggggggggg, ,gggg, \n# dP\"\"\"88\"\"\"\"\"\"Y8, ,d8P\"\"d8P\"Y8b, dP\"\"\"88\"\"\"\"\"\"Y8, ,88\"\"\"Y8b,\n# Yb, 88 `8b,d8' Y8 \"8b,dPYb, 88 `8b d8\" `Y8\n# `\" 88 ,8Pd8' `Ybaaad88P' `\" 88 ,8Pd8' 8b d8\n# 88aaaad8P\" 8P `\"\"\"\"Y8 88aaaad8P\",8I \"Y88P'\n# 88\"\"\"\"Y8ba 8b d8 88\"\"\"\"\" I8' \n# 88 `8bY8, ,8P 88 d8 \n# 88 ,8P`Y8, ,8P' 88 Y8, \n# 88_____,d8' `Y8b,,__,,d8P' 88 `Yba,,_____, \n# 88888888P\" `\"Y8888P\"' 88 `\"Y8888888 \n#\n# The Block Oriented Programming (BOP) Compiler - v2.1\n#\n#\n# Kyriakos Ispoglou (ispo) - ispo@purdue.edu\n# PURDUE University, Fall 2016-18\n# -------------------------------------------------------------------------------------------------\n#\n#\n# capability.py\n#\n# This module measures the capability of the program. That is, program's capability gives a good\n# indication, on \"what the program is capable of executing\" in terms of SPL payloads. However, all\n# these metrics, aim to identify *upper bounds*; that is, they overestimate the set of SPL programs\n# that can be truly executed on this binary.\n# -------------------------------------------------------------------------------------------------\nfrom coreutils import *\nfrom calls import *\nimport path as P\n\nimport networkx as nx\nimport textwrap\nimport datetime\nimport cPickle as pickle\nimport math\nimport numpy\n\n\n\n# -----------------------------------------------------------------------------\n# Capability Options\n# -----------------------------------------------------------------------------\nCAP_ALL = 0x00FF # all types of statements\nCAP_REGSET = 0x0001 # register assignments \nCAP_REGMOD = 0x0002 # register modifications\nCAP_MEMRD = 0x0004 # memory reads\nCAP_MEMWR = 0x0008 # memory writes\nCAP_CALL = 0x0010 # system and library calls\nCAP_COND = 0x0020 # conditional statements\nCAP_LOAD = 0x0100 # load the capability graph from a file\nCAP_SAVE = 0x0200 # save the capability graph to a file\nCAP_NO_EDGE = 0x0400 # don't calculate edges in capability graph\n\n# types of analyses\nCAP_STMT_COMB_CTR = 'STMT_COMB_CTR' # Count combinations of statements\nCAP_STMT_MIN_DIST = 'STMT_MIN_DIST' # Count min distance between statements\nCAP_LOOPS = 'LOOPS' # Analyze loops\n\n\n\n# -------------------------------------------------------------------------------------------------\n# capability: This class is responsible for performing several measurements in the target binary.\n#\nclass capability( object ):\n ''' ======================================================================================= '''\n ''' INTERNAL VARIABLES '''\n ''' ======================================================================================= '''\n __cap = nx.DiGraph() # the capability graph (CAP)\n __uid = 0 # a unique ID\n \n\n\n ''' ======================================================================================= '''\n ''' INTERNAL FUNCTIONS '''\n ''' ======================================================================================= '''\n\n # ---------------------------------------------------------------------------------------------\n # __add(): Add a node to the capability graph.\n #\n # :Arg addr: Address of the basic block tha contains the statement\n # :Arg ty: Statement type: regset / regmod / call / cond\n # :Arg reg: Register name (for regset/regmod/cond)\n # :Arg val: Statement's value (for regset/regmod/cond)\n # :Arg mode: Statement mode (const/deref for regset and syscall/libcall for call)\n # :Arg isW: A flag indicating whether \"val\" points to a writable address (for regset)\n # :Arg op: Statement operator (for regmod/cond)\n # :Arg mem: Memory address (for memrd/memwr)\n # :Arg name: Function name (for call)\n # :Ret: None.\n #\n def __add( self, addr, ty, reg=None, val=None, mode=None, isW=None, op=None, name=None, mem=None, size=None ):\n # NOTE: We assume that arguments are not malformed, so we don't do any checks\n cap = {\n 'regset' : {'addr':int(addr), 'type':ty, 'reg':reg, 'val':val, '+W':isW, 'mode':mode},\n 'regmod' : {'addr':int(addr), 'type':ty, 'reg':reg, 'op':op, 'val':val},\n 'memrd' : {'addr':int(addr), 'type':ty, 'reg':reg, 'mem':mem, 'size':size},\n 'memwr' : {'addr':int(addr), 'type':ty, 'mem':mem, 'val':val, 'size':size},\n 'call' : {'addr':int(addr), 'type':ty, 'name':name, 'mode':mode},\n 'cond' : {'addr':int(addr), 'type':ty, 'reg':reg, 'op':op, 'val':val}\n }[ ty ] # nicely \"switch\" the appropriate statement\n \n self.__cap.add_node(self.__uid, **cap) # add statement to the graph\n self.__uid += 1 # update UID counter\n\n\n\n # ---------------------------------------------------------------------------------------------\n\n ''' ======================================================================================= '''\n ''' CLASS INTERFACE '''\n ''' ======================================================================================= '''\n\n # ---------------------------------------------------------------------------------------------\n # __init__(): Class constructor. Simply initialize private variables.\n #\n # :Arg cfg: Program's CFG.\n # :Arg name: Program's filename\n #\n def __init__( self, cfg, name ): \n self.__cfg = cfg # save cfg to internal variables\n self.__name = name # program's filename\n\n\n\n # ---------------------------------------------------------------------------------------------\n # build(): Build the Capability Graph. This is a very slow process, so it's possible to save\n # the graph once its generated, thus without having to re-calculate it the next time.\n # \n # :Arg options: An integer that describes how the capability graph should be built. It can be\n # the logical OR of one or more of the following:\n #\n # CAP_ALL | Include all types of statements in the graph\n # CAP_REGSET | Include register assignments in the graph\n # CAP_REGMOD | Include register modifications in the graph\n # CAP_CALL | Include system and library calls in the graph\n # CAP_COND | Include conditional statements in the graph\n # CAP_LOAD | Load the capability graph from a file\n # CAP_SAVE | Save the capability graph to a file\n #\n # :Ret: None.\n #\n def build( self, options=CAP_ALL ):\n dbg_prnt(DBG_LVL_1, \"Exploring program's capability...\")\n\n # ---------------------------------------------------------------------\n # Load Capability Graph from file ?\n # --------------------------------------------------------------------- \n if options & CAP_LOAD:\n dbg_prnt(DBG_LVL_1, \"Loading the Capability Graph from file...\")\n\n try:\n self.__cap = nx.read_gpickle(self.__name + '.cap')\n\n dbg_prnt(DBG_LVL_1, \"Done.\") \n\n return # your job is done here\n\n except IOError, err:\n # if you can't load it, simply re-calculate it ;)\n\n error(\"Cannot load Capability Graph: %s\" % str(err))\n\n\n # ---------------------------------------------------------------------\n # Iterate over abstracted basic blocks\n # --------------------------------------------------------------------- \n dbg_prnt(DBG_LVL_1, \"Searching CFG for 'interesting' statements...\")\n\n nnodes = len(nx.get_node_attributes(self.__cfg.graph, 'abstr').items())\n counter = 1\n \n p = P._cfg_shortest_path(self.__cfg)\n\n\n for node, abstr in nx.get_node_attributes(self.__cfg.graph,'abstr').iteritems():\n addr = node.addr\n\n dbg_prnt(DBG_LVL_3, \"Analyzing block at 0x%x (%d/%d)...\" % (addr, counter, nnodes))\n \n\n if options & CAP_REGSET:\n for reg, data in abstr['regwr'].iteritems():\n\n if data['type'] == 'concrete':\n self.__add(addr, ty='regset', reg=reg, val=data['const'], mode='const',\n isW=data['writable'])\n\n elif data['type'] == 'deref':\n self.__add(addr, ty='regset', reg=reg, val=data['addr'], mode='deref')\n \n\n if options & CAP_REGMOD:\n for reg, data in abstr['regwr'].iteritems():\n if data['type'] == 'mod': \n self.__add(addr, ty='regmod', reg=reg, op=data['op'], val=data['const'])\n\n\n if options & CAP_MEMRD:\n for reg, data in abstr['regwr'].iteritems():\n if data['type'] == 'deref' and data['memrd']:\n loadreg = data['deps'][0]\n\n self.__add(addr, ty='memrd', reg=reg, mem=loadreg, size=data['memrd'])\n \n \n if options & CAP_MEMWR:\n for memwr in abstr['splmemwr']:\n self.__add(addr, ty='memwr', mem=memwr['mem'], val=memwr['val'], size=memwr['size'])\n\n\n\n if options & CAP_CALL and abstr['call'] and find_call(abstr['call']['name']):\n self.__add(addr, ty='call', name=abstr['call']['name'], mode=abstr['call']['type'])\n\n\n elif options & CAP_COND and abstr['cond']:\n \n # elif because we can't have call and cond at the same basic block\n self.__add(addr, ty='cond', reg=abstr['cond']['reg'], op=abstr['cond']['op'],\n val=abstr['cond']['const'])\n\n\n '''\n # -----------------------------------------------------------------------\n # hacky way to quickly find a loop\n # -----------------------------------------------------------------------\n for length, loop in p.k_shortest_loops(addr, 0, 10):\n length, loop = p.shortest_loop(addr)\n\n R = abstr['cond']['reg']\n\n regmod = 0\n regset = 0\n step = 0\n\n if length < INFINITY:\n\n for l in loop[:-1]:\n try:\n X = self.__cfg.graph.node[ADDR2NODE[l]]['abstr']\n except KeyError:\n continue\n \n for reg, data in X['regwr'].iteritems():\n if data['type'] == 'mod' and reg == R:\n regmod += 1\n step = data['const']\n\n elif reg == R:\n regset += 1\n\n\n if regmod == 1 and regset == 0:\n emph(bolds('GOOD LOOP (%d - %d - %s) %s' % \n (abstr['cond']['const'], step, abstr['cond']['op'], \n pretty_list(loop))))\n\n # else:\n # print 'BAD LOOP (mod: %d, set: %d) (%d - %d - %s) %s' % \\\n # (regmod, regset, abstr['cond']['const'], step, abstr['cond']['op'],\n # pretty_list(loop))\n '''\n\n counter += 1 # update counter\n\n dbg_prnt(DBG_LVL_1, \"Done.\")\n\n\n # ---------------------------------------------------------------------\n # Show some statistics\n # --------------------------------------------------------------------- \n emph(\"Binary has %s interesting statements:\" % bold(self.__cap.order()))\n\n stmt_ctr = { 'regset' : 0, 'regmod' : 0, 'memrd' : 0, 'memwr' : 0, 'call' : 0, 'cond' : 0 }\n \n for _, data in self.__cap.nodes(data=True):\n stmt_ctr[ data['type'] ] += 1 # count statements\n\n\n emph(\"\\t%s register assignments\" % bold(stmt_ctr['regset'], pad=5))\n emph(\"\\t%s register modifications\" % bold(stmt_ctr['regmod'], pad=5))\n emph(\"\\t%s memory reads \" % bold(stmt_ctr['memrd'], pad=5))\n emph(\"\\t%s memory writes \" % bold(stmt_ctr['memwr'], pad=5))\n emph(\"\\t%s system/library calls\" % bold(stmt_ctr['call'], pad=5))\n emph(\"\\t%s conditional jumps\" % bold(stmt_ctr['cond'], pad=5))\n\n\n # ---------------------------------------------------------------------\n # Add edges to the Capability Graph\n # ---------------------------------------------------------------------\n\n # don't calculate edges if asked (it's time consuming)\n if options & CAP_NO_EDGE:\n dbg_prnt(DBG_LVL_1, \"Skipping edge calculation of capability graph.\")\n return\n\n\n dbg_prnt(DBG_LVL_1, \"Building the Capability Graph...\")\n\n\n # list of node addresses\n node_list = [ d['addr'] for _, d in self.__cap.nodes_iter(data=True) ] \n SPT = nx.DiGraph() # create the Shortest Path Tree\n completed = 0 # % completed\n\n csp = P._cfg_shortest_path(self.__cfg) # create the CFG Shortest Path object\n\n\n warn(\"This can be a very slow process ('-dd' and '-ddd' options show a progress bar)\")\n\n # for each node u_ in Capability Graph\n for u_, du in self.__cap.nodes_iter(data=True): \n v_ = -1 # v_ is the uid of the target node (u_ -> v_) \n\n SPT.clear() # clear Shortest Path Tree\n\n # Find the shortest paths (in CFG) to every other statement. Unfortunately, shortest\n # paths in CFG are not like regular shortest paths, as we explain in path.py. Thus we\n # have to re-calculate all shortest paths for every node in the capability graph.\n for length, path in csp.shortest_path(du['addr'], node_list):\n v_ += 1 # the uid of the current node (it's linear)\n\n if length == INFINITY:\n continue # skip nodes with non-existing paths\n\n # ---------------------------------------------------------------------------------\n # Now, if we directly add the edges with shortest path lengths to the capability\n # graph, we'll have an interesting problem: Consider the path A - x - x - B - x - C\n # in CFG. The Capability Graph should contain the edges (A, B, 3) and (B, C, 2). \n # However the naive approach, will also add the edge (A, C, 5) to the graph. The\n # problem here is that we cannot accurately measure chains of statements due to the\n # direct edges.\n #\n # To fix this issue we build the Shortest Path Tree (SPT). That is, we merge all\n # shortest paths, into a single graph. The resulting graph will be tree as it\n # consists only of single source shortest paths (without loops), with all edges\n # having weight = 1. SPT has two types of nodes: Black and White. Black nodes \n # contain statements (should appear on capability graph) while White nodes are used\n # for transitions. The first and the last nodes of each shortest path are Black\n # while every other node between is White. Our goal is to remove all White nodes\n # and merge the resulting SPT with the capability graph.\n #\n # We remove the White nodes one by one. When we remove a White node, we also update\n # the weights in SPT.\n # ---------------------------------------------------------------------------------\n \n # add first and last nodes (Black) to the SPT (if already exists, make them Black)\n SPT.add_nodes_from([path[0], path[-1]], color='Black')\n\n # keep track of the statement uids that use this node (map address to UID)\n SPT.node[path[0] ].setdefault('uid', set()).add(u_)\n SPT.node[path[-1]].setdefault('uid', set()).add(v_)\n\n # convert nodes [1,2,3,4], into edges [(1,2),(2,3),(3,4)] and add them to SPT\n SPT.add_edges_from(zip(path, path[1:]), weight=1)\n\n # color the intermediate nodes White (if they're not Black)\n for p in path[1:-1]:\n if 'color' not in SPT.node[p] or SPT.node[p]['color'] != 'Black':\n SPT.node[p]['color'] = 'White'\n\n\n # iteratively delete the White nodes\n for n in [node for node, data in SPT.nodes(data=True) if data['color'] == 'White']:\n\n # for each pair of (incoming, outgoing) edges\n for src, _, d1 in SPT.in_edges(n, data=True):\n for _, dst, d2 in SPT.out_edges(n, data=True):\n # add a new edge that bypasses the White node\n SPT.add_edge(src, dst, weight=d1['weight']+d2['weight'])\n\n\n SPT.remove_node(n) # delete White node (along with its edges)\n\n\n ''' at this point, SPT will only contain Black nodes '''\n\n # merge SPT to the capability graph\n for e1, e2, data in SPT.edges_iter(data=True):\n # copy it edge-by-edge\n for u in SPT.node[e1]['uid']: # move from addresses back to UIDs\n for v in SPT.node[e2]['uid']: \n if u != v: # that's to avoid self-loops\n self.__cap.add_edge(u, v, weight=data['weight'])\n \n\n # show current progress (%)\n percent = math.floor(100. / len(self.__cap) * u_)\n if completed < percent:\n completed = percent \n dbg_prnt(DBG_LVL_2, \"%d%% completed\" % completed)\n\n del SPT # we don't need the SPT anymore\n\n dbg_prnt(DBG_LVL_1, \"Done. Capability Graph generated successfully.\")\n \n visualize(self.__cap)\n\n \n\n # ---------------------------------------------------------------------\n # Save Capability Graph to a file ?\n # --------------------------------------------------------------------- \n if options & CAP_SAVE:\n dbg_prnt(DBG_LVL_1, \"Saving Capability Graph...\")\n\n try:\n nx.write_gpickle(self.__cap, self.__name + '.cap')\n dbg_prnt(DBG_LVL_1, \"Done. Capability Graph saved as %s\" % self.__name + '.cap')\n\n except IOError, err:\n error(\"Cannot save Capability Graph: %s\" % str(err))\n\n\n\n # ---------------------------------------------------------------------------------------------\n # get(): Return the Capability Graph. Just in case ;)\n #\n # :Ret: The Capability Graph\n #\n def get( self ):\n return self.__cap\n\n\n\n # ---------------------------------------------------------------------------------------------\n # save(): Save the nodes of the Capability Graph (i.e., the interesting statements) to a file.\n #\n # :Ret: None.\n #\n def save( self ):\n now = datetime.datetime.now() # get current timestamp\n banner = textwrap.dedent(\"\"\"\\\n #\n # This file has been created by BOPC at %s\n # '%s' has %d interesting statements. Each line shows a statement.\n #\n # The columns are: address | type | register | value | mode | +W | operator | name\n # When an attribute is not available, a dot '.' is presented.\n #\n #\n # Attribute list:\n #\n # address : Address of the basic block tha contains the statement\n # type : Statement type: regset / regmod / call / cond\n # register : Register name (for regset / regmod / cond)\n # memory : Memory address (for memrd / memwr)\n # value : Statement's value (for regset / regmod / cond)\n # mode : Statement mode (const / deref for regset and syscall / libcall for call)\n # +W : A flag indicating whether \"val\" points to a writable address (for regset)\n # operator : Statement operator (for regmod / cond)\n # name : Function name (for call)\n #\n \"\"\" % (now.strftime(\"%d/%m/%Y %H:%M\"), self.__name, self.__cap.order()))\n\n\n dbg_prnt(DBG_LVL_1, \"Dumping interesting statments to a file...\") \n \n try: \n cap = open(self.__name + '.stmt', 'w')\n\n cap.write(banner) # write banner first\n\n # write statements one by one\n for _, d in self.__cap.nodes_iter(data=True): \n opt = '%10s' % (d['reg'] if 'reg' in d else '.')\n opt += '%10s' % (d['mem'] if 'mem' in d else '.')\n opt += ' %32s ' % (d['val'] if 'val' in d else '.')\n opt += '%10s' % (d['mode'] if 'mode' in d else '.')\n opt += '%10s' % (d['+W'] if '+W' in d else '.')\n opt += '%10s' % (d['op'] if 'op' in d else '.')\n opt += '%16s' % (d['name'] if 'name' in d else '.')\n opt += '%10s' % (d['size'] if 'size' in d else '.')\n\n cap.write( \"0x%08x %10s %s\\n\" % (d['addr'], d['type'], opt) )\n \n cap.close()\n \n dbg_prnt(DBG_LVL_1, \"Done. Capability Graph saved as %s\" % self.__name + '.stmt')\n\n except IOError, err:\n error(\"Cannot create statements file: %s\" % str(err))\n\n\n\n # ---------------------------------------------------------------------------------------------\n # explore(): Explore the Capability Graph and look for \"islands\".\n # \n # :Ret: None.\n #\n def explore( self ): \n dbg_prnt(DBG_LVL_1, \"Exploring the Capability Graph...\")\n\n self.__islands = [] # store islands here\n n_inslands = 0 # number of islands\n size, diam = [], [] # size and diameter lists\n \n\n # ---------------------------------------------------------------------\n # The first step is to extract the \"islands\" from the Capability Graph,\n # which are essentially the Strong Connected Components (SCC) of the\n # undirected version of the graph.\n # ---------------------------------------------------------------------\n capU = self.__cap.to_undirected() # make Capability Graph undirected\n unvisited = set(capU.nodes()) # initially, no node is visited\n\n while len(unvisited): # while there are unvisited nodes\n root = unvisited.pop() # pick a random node\n unvisited.add( root ) # and remove it from set\n \n nodeset = [] # nodes in the current island\n\n # explore the island using DFS and obtain the node set\n for u in nx.dfs_preorder_nodes(capU, root): \n unvisited.remove(u) # mark u as visited\n nodeset.append(u) # and add it to node set\n\n self.__cap.node[ u ]['island'] = n_inslands\n \n\n # get island as induced (directed) subgraph and relabel nodes in [0, order(G)-1] range\n graph = self.__cap.subgraph(nodeset) \n relabel = dict(zip(graph.nodes(), range(graph.order())))\n graph = nx.relabel_nodes(graph, relabel)\n \n\n # ---------------------------------------------------------------------\n # Calculate island's diameter. Although the island is fully connected\n # in the undirected version, it's not in the directed version. Thus,\n # nx.diameter(graph) throws an exception. The diameter of the island,\n # is the longest shortest path between any two nodes.\n # ---------------------------------------------------------------------\n D = 0 # island's diameter\n\n for n in graph.nodes_iter():\n # caclulate all shortest paths from the given node\n length = nx.single_source_shortest_path_length(graph, n)\n maxlen = max(length.values()) # get the longest shortest path\n\n if D < maxlen: D = maxlen # keep track of the longest among all nodes\n\n\n size.append(len(nodeset)) # island size\n diam.append( D) # island's diameter\n\n self.__islands.append( { # store island's information\n 'root' : root,\n 'size' : graph.order(),\n 'diameter' : D,\n 'graph' : graph\n } )\n \n n_inslands += 1 # total # islands\n\n dbg_prnt(DBG_LVL_1, \"Done.\")\n\n\n # ---------------------------------------------------------------------\n # Show some statistics\n # --------------------------------------------------------------------- \n warn(\"'-dd' and '-ddd' options show the 'size' and 'diameter' lists\")\n\n emph(\"Capability Graph has %s islands\" % bold(n_inslands))\n\n emph(\"Island sizes: max = %s, min = %s, avg = %s\" % \n (bold(max(size)), bold(min(size)), bold(1.*sum(size)/n_inslands, 'float')))\n\n dbg_arb(DBG_LVL_2, \"Island size list\", size)\n\n emph(\"Island diameters: max = %s, min = %s, avg = %s\" % \n (bold(max(diam)), bold(min(diam)), bold(1.*sum(diam)/n_inslands, 'float')))\n\n dbg_arb(DBG_LVL_2, \"Island diameter list\", diam)\n\n\n\n # ---------------------------------------------------------------------------------------------\n # analyze(): Perform various analyses to the islands of the Capability Graph.\n #\n # :Arg analyses: The analyses to perform (can be many)\n # :Ret: None.\n #\n def analyze( self, *analyses ):\n dbg_prnt(DBG_LVL_1, \"Analyzing the Capability Graph...\")\n\n for analysis in analyses: # for every different analysis\n try:\n # based on the analysis, select the appropriate function and invoke it\n func = {\n CAP_STMT_COMB_CTR : self.__analyze_stmt_comb_ctr,\n CAP_STMT_MIN_DIST : self.__analyze_stmt_min_dist,\n CAP_LOOPS : self.__analyze_loops\n }[ analysis ]\n\n\n for island in self.__islands: # perform the analysis to every island\n func( island['graph'] )\n\n except KeyError, err:\n fatal('Unknow analysis %s' % str(err))\n\n\n\n # ---------------------------------------------------------------------------------------------\n # analyze_island(): Analyze a specific island.\n #\n # :Arg addr: An address of any node of the island\n # :Arg analyses: The analyses to perform (can be many)\n # :Ret: None.\n #\n def analyze_island( self, addr, *analyses ):\n # ---------------------------------------------------------------------\n # Search for the island to analyze\n # ---------------------------------------------------------------------\n island_id = -1\n\n for _, d in self.__cap.nodes_iter(data=True):\n if d['addr'] == addr:\n island_id = d['island']\n break\n\n if island_id < 0:\n fatal(\"Node '0x%x' does not contained in any island\" % addr)\n\n dbg_prnt(DBG_LVL_1, \"Analyzing the Island %d...\" % island_id)\n\n\n # ---------------------------------------------------------------------\n # Perform the analyses\n # ---------------------------------------------------------------------\n for analysis in analyses: # for every different analysis\n try:\n # based on the analysis, select the appropriate function and invoke it\n func = {\n CAP_STMT_COMB_CTR : self.__analyze_stmt_comb_ctr,\n CAP_STMT_MIN_DIST : self.__analyze_stmt_min_dist,\n CAP_LOOPS : self.__analyze_loops\n }[ analysis ]\n\n func( self.__islands[ island_id ]['graph'] )\n\n except KeyError, err:\n fatal('Unknow analysis %s' % str(err))\n\n\n\n # ---------------------------------------------------------------------------------------------\n # callback(): Invoke a callback function for every island.\n #\n # :Arg cbfunc: The callback function to invoke\n # :Ret: None.\n #\n def callback( self, cbfunc ):\n for island in self.__islands:\n cbfunc( island['graph'] )\n\n \n # TODO: Move these to private function sections\n\n\n # ---------------------------------------------------------------------------------------------\n # __analyze_stmt_comb_ctr(): Count the total number of combinations that K SPL statements can\n # be chained together (repetitions of statements are allowed) on a given island.\n # \n # :Arg island: The island graph to work on\n # :Ret: None.\n #\n def __analyze_stmt_comb_ctr( self, island ):\n dbg_prnt(DBG_LVL_1, \"Starting Analysis: Statement Combinations...\")\n\n\n # TODO: Check this again. Too many combinations :\\\n K = 20\n\n\n # ---------------------------------------------------------------------\n # Find the total number of paths between any 2 nodes that use exactly\n # K edges. We calculate that using Dynamic Programming. Let C^k_{ij} be\n # the total number of paths from i to j with exactly k edges. Then we\n # have:\n #\n # C^0_{ii} = 1, forall i in V\n # C^k_{ij} = C^1_{ij} = 1, iff (i,j) in E\n # C^k_{ij} = SUM(C^{k-1}_[xj]), for all x adjacent to i\n #\n # We build this table in a bottom-up fashion. Time/Space Complexity is \n # O(|V|^2 * K). We can improve space complexity by storing only the\n # last 2 K's (K and K-1).\n # ---------------------------------------------------------------------\n C = numpy.zeros((K, island.order(), island.order()), dtype=numpy.int64)\n \n for i in range(island.order()): # initialize for K = 0\n C[0][i][i] = 1\n \n for i,j, d in island.edges_iter(data=True): # initialize for K = 1\n C[1][i][j] = 1\n \n for k in range(2, K): # main loop\n for i in island.nodes():\n for j in island.nodes():\n for x in island.neighbors(i):\n C[k][i][j] += C[k-1][x][j]\n\n # ---------------------------------------------------------------------\n for k in range(K):\n dbg_arb(DBG_LVL_1, \"Combinations with up to %d statements:\", sum(sum(C[k][:][:])))\n\n\n\n # ---------------------------------------------------------------------------------------------\n # __analyze_stmt_min_dist(): Calculate the minimum distance with between any two statements\n # that have exactly K edges between on a given island.\n #\n # :Arg island: The island graph to work on\n # :Ret: None.\n #\n def __analyze_stmt_min_dist( self, island ):\n '''\n B = { }\n\n # enumerate all simple paths from i to j \n # WARNING: O(n!) complexity !!!\n for i in island.nodes_iter():\n for j in island.nodes_iter():\n if i == j: continue\n\n for x in nx.all_simple_paths(island, i, j):\n \n A = [island[a][b]['weight'] for a,b in zip(x, x[1:])]\n\n B.setdefault(len(x), []).append(sum(A))\n '''\n\n\n dbg_prnt(DBG_LVL_1, \"Starting Analysis: Statement Minimum Distances...\")\n\n\n K = 20\n\n # ---------------------------------------------------------------------\n # Find the minimum distance between any 2 nodes that use exactly K edges.\n # This is very similar with the algorithm in __analyze_stmt_comb_ctr(),\n # but with different Dynamic Programming equations:\n #\n # M^0_{ii} = 0, forall i in V\n # M^k_{ij} = M^1_{ij} = weight[i][j], iff (i,j) in E\n # M^k_{ij} = MIN(M^k_[ij], weight[i][x] + M^{k-1}_{xj}), \n # for all x adjacent to i\n # ---------------------------------------------------------------------\n M = numpy.full((K, island.order(), island.order()), dtype=numpy.int32, fill_value=INFINITY)\n \n\n for i in range(island.order()): # initialize for K = 0\n M[0][i][i] = 0\n \n for i,j, d in island.edges_iter(data=True): # initialize for K = 1\n M[1][i][j] = d['weight']\n \n for k in range(2, K): # main loop\n for i in island.nodes():\n for j in island.nodes():\n for x in island.neighbors(i): \n\n M[k][i][j] = min(M[k][i][j], island[i][x]['weight'] + M[k-1][x][j])\n\n # ---------------------------------------------------------------------\n for k in range(K):\n m = numpy.min(M[k][:][:]) \n if m == INFINITY: break\n\n dbg_prnt(DBG_LVL_1, \"Min shortest path with up to %d statements: %d\" % (k, m))\n\n\n # ---------------------------------------------------------------------------------------------\n # __analyze_loops(): Analyze the loops on an a given island.\n # \n # :Arg island: The island graph to work on\n # :Ret: None.\n #\n def __analyze_loops( self, island ):\n warn('Loop analysis is not supported yet')\n \n\n# -------------------------------------------------------------------------------------------------\n" }, { "path": "source/compile.py", "content": "#!/usr/bin/env python2\n# -------------------------------------------------------------------------------------------------\n#\n# ,ggggggggggg, _,gggggg,_ ,ggggggggggg, ,gggg, \n# dP\"\"\"88\"\"\"\"\"\"Y8, ,d8P\"\"d8P\"Y8b, dP\"\"\"88\"\"\"\"\"\"Y8, ,88\"\"\"Y8b,\n# Yb, 88 `8b,d8' Y8 \"8b,dPYb, 88 `8b d8\" `Y8\n# `\" 88 ,8Pd8' `Ybaaad88P' `\" 88 ,8Pd8' 8b d8\n# 88aaaad8P\" 8P `\"\"\"\"Y8 88aaaad8P\",8I \"Y88P'\n# 88\"\"\"\"Y8ba 8b d8 88\"\"\"\"\" I8' \n# 88 `8bY8, ,8P 88 d8 \n# 88 ,8P`Y8, ,8P' 88 Y8, \n# 88_____,d8' `Y8b,,__,,d8P' 88 `Yba,,_____, \n# 88888888P\" `\"Y8888P\"' 88 `\"Y8888888 \n#\n# The Block Oriented Programming (BOP) Compiler - v2.1\n#\n#\n# Kyriakos Ispoglou (ispo) - ispo@purdue.edu\n# PURDUE University, Fall 2016-18\n# -------------------------------------------------------------------------------------------------\n#\n#\n# compile.py:\n#\n# This module compiles an program written in SPL into an equivalent Intermediate Representation\n# (IR) suitable for processing by subsequent modules. Please do not confuse it with the VEX IR.\n#\n# SPL is actually a subset of C, so it has the same syntax. Comments are denoted with '//'. Multi\n# line comments are not supported.The specs of the language (expressed in EBNF) are shown below:\n#\n# := 'void' 'payload' '(' ')' '{' '}'\n# := ( |