[
  {
    "path": "LICENSE",
    "content": "This is free and unencumbered software released into the public domain.\n\nAnyone is free to copy, modify, publish, use, compile, sell, or\ndistribute this software, either in source code form or as a compiled\nbinary, for any purpose, commercial or non-commercial, and by any\nmeans.\n\nIn jurisdictions that recognize copyright laws, the author or authors\nof this software dedicate any and all copyright interest in the\nsoftware to the public domain. We make this dedication for the benefit\nof the public at large and to the detriment of our heirs and\nsuccessors. We intend this dedication to be an overt act of\nrelinquishment in perpetuity of all present and future rights to this\nsoftware under copyright law.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND,\nEXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF\nMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.\nIN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR\nOTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,\nARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR\nOTHER DEALINGS IN THE SOFTWARE.\n\nFor more information, please refer to <http://unlicense.org>\n"
  },
  {
    "path": "README.md",
    "content": "# Apple Notes Export Tools\n\nThis repository includes a few python export tools for Apple's \"Notes.app\".  The scripts require python3, but I tried to make the scripts self-contained and concise, with no additional dependencies. I may revisit this and break the common code into a library in the future.\n\nThis repository is in the public domain.\n\n## `notes.md`\n\nDescription of how notes data is stored.\n\n## `notes2bear`\n\nWrites a `notes.bearbk` file in Bear's backup format (a zip file with their own markup flavor).  It doesn't handle tables because Bear doesn't do tables yet. This shaves about 100 lines from the script.\n\n## `notes2html` \n\n`notes2html` takes an destination directory and writes a tree of html files and images.  One html file per note and any associated media in the media directory. If you specify `--title` the files with be named with the title guessed by Apple (with / replaced by _).  If you specify `--svg`, any drawings will be rendered as inline SVG; otherwise, the fallback jpg files provided by Apple will be used.\n\nUsage is:\n\n```\nnotes2html [--svg] [--title] dest\n```\n\n\n\n\n\n\n\n\n\n\n"
  },
  {
    "path": "notes.md",
    "content": "# Notes on Notes.app\nFor future reference and to aid anyone else who might want to extract data from the \"Notes\" app. I compiled this out of curiosity and a desire to backup my notes content. \n\nThis document only covers the current format of notes, as synced with iCloud. The application also supports IMAP synced notes with reduced functionality. They are stored in a separate database (and on the IMAP server). You're on your own there, but it's pretty much just MIME and HTML.\n\n## OSX Files\n\nThe notes are stored in `~/Library/Group Containers/group.com.apple.notes` in a sqlite database named `NoteStore.sqlite`. The database contains sync state from iCloud. It's a CoreData store, but I approached it from plain sqlite.\n\nWe're interested in the `ZICCLOUDSYNCINGOBJECT` table and the `ZICNOTEDATA` table. The former contains the sync state of each iCloud object - notes, attachments, and folders, and the latter contains the note data. Everything has a UUID identifier `ZIDENTIFIER`. The `ZICCLOUDSYNCINGOBJECT` table column `ZNOTEDATA` points to the `Z_PK` column in `ZICNOTEDATA`.\nThe meat of the data is in zlib compressed protobuf blobs. For documents, it's `ZDATA` in `ZNOTEDATA` and for tables/drawings, it's `ZMERGEABLEDATA` in `ZICCLOUDSYNCINGOBJECT`.\n\nApple uses a [CvRDT](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) for syncing (evidenced by various strings appearing in the data and analysis of what happens as you edit). I suspect the one used for documents, which they call \"topotext\" is not actually a CvRDT (they seem to order conflicts on a first to sync basis rather than vector clock) but I could be mistaken. It doesn't matter since everything goes through iCloud, which serializes the merges.\n\n## Protobuf Data\n\n**Document Wrapper**\nEverything is wrapped with a versioned document object:\n\n    message Document {\n        repeated Version version = 2;\n    }\n    message Version {\n        optional bytes data = 3;\n    }\n\nThere are additional fields that aren't relevant to us, and I've never seen more than one `version`. The content of the `data` field is also protobuf but varies depending on whether we're looking at a note, table, or drawing.\n\n**Notes**\nThe protobuf data for a note is a `String` as described below. Assume everything is optional and I’ve elided the CRDT stuff.  (Repeated field 3 of String is a sequence clock, length, attribute clock, tombstone, and children.  It forms a DAG. For chunks of length >1 the clock is implicitly incremented for each character.)\n\n\n    message String {\n        string string = 2;\n        // these are in order, disjoint, and their length sums to the length of string\n        repeated AttributeRun attributeRun = 5;\n    }\n    message AttributeRun {\n        uint32 length = 1;\n        ParagraphStyle paragraphStyle = 2;\n        Font font = 3;    \n        uint32 fontHints = 5; // 1:bold, 2:italic, 3:bold italic\n        uint32 underline = 6;\n        uint32 strikethrough = 7;\n        int32 superscript = 8; // sign indicates super/subscript\n        string link = 9;\n        Color color = 10;\n        AttachmentInfo attachmentInfo = 12;\n    }\n    message ParagraphStyle {\n        // 0:title, 1:heading, 4:monospace, 100:dotitem, 101:dashitem, 102:numitem, \n        // 103:todoitem\n        uint32 style = 1;\n        uint32 alignment = 2; // 0:left, 1:center, 2:right, 3:justified\n        int32 indent = 4;\n        Todo todo = 5;\n    }\n    message Font {\n        string name = 1;\n        float pointSize = 2;\n        uint32 fontHints = 3;\n    }\n    message AttachmentInfo {\n        string attachmentIdentifier = 1;\n        string typeUTI = 2;\n    }\n    message Todo {\n        bytes todoUUID = 1;\n        bool done = 2;\n    }\n    message Color {\n        float red = 1;\n        float green = 2;\n        float blue = 3;\n        float alpha = 4;\n    }\n\n**Drawings**\nThis info isn’t strictly necessary.  For each drawing, you’ll find a rendering in `FallbackImages/UUID.jpg`. I was curious whether I could recover / backup the original vector data, so I came up with the following. (The root object here is `Drawing`.)\n\n\n    message Drawing {\n        int64 serializationVersion = 1;\n        repeated bytes replicaUUIDs = 2;\n        repeated StrokeID versionVector = 3;\n        repeated Ink inks = 4;\n        repeated Stroke strokes = 5;\n        int64 orientation = 6;\n        StrokeID orientationVersion = 7;\n        Rectangle bounds = 8;\n        bytes uuid = 9;\n    }\n    message Color {\n        float red = 1;\n        float green = 2;\n        float blue = 3;\n        float alpha = 4;\n    }\n    message Rectangle {\n        float height = 4;\n        float originX = 1;\n        float originY = 2;\n        float width = 3;\n    }\n    message Transform {\n        float a = 1;\n        float b = 2;\n        float c = 3;\n        float d = 4;\n        float tx = 5;\n        float ty = 6;\n    }\n    message Ink {\n        Color color = 1;\n        string identifier = 2;\n        int64 version = 3;\n    }\n    message Stroke {\n        int64 inkIndex = 3;\n        int64 pointsCount = 4;\n        bytes points = 5;\n        Rectangle bounds = 6;\n        bool hidden = 9;\n        double timestamp = 11;\n        bool createdWithFinger = 12;\n        Transform transform = 10;\n    }\n\nThe byte array `points` is compactly encoded list of points. It is a sequence of this struct:\n\n\n    struct PKCompressedStrokePoint {\n        float timestamp;            // timestamp (delta from somethin, probably previous)\n        float xpos;\n        float ypos;\n        unsigned short radius;      // radius*10\n        unsigned short aspectRatio; // aspectRatio*1000\n        unsigned short edgeWidth;   // edgeWidth*10\n        unsigned short force;       // force*1000\n        unsigned short azimuth;     // azimuth*10430.2191955274\n        unsigned char altitude;     // altitude*162.338041953733\n        unsigned char opacity;      // opacity*255\n    };\n\nThe `inkIndex` field points into the array `inks`. The `identifier` in an `Ink` includes stuff like `com.apple.ink.marker`. I kinda fudged this in my svg generation - apple’s rendering code is much more sophisticated, taking azimuth/altitude into account.  My code works well enough for pen, but falls short on marker.  You may be better off with the jpeg.\n\n**Tables**\nThis one is complicated, and I think a bit of explanation is in order to explain why it’s structured this way. They are trying to model a table with multiple people editing at the same time. Editing the contents of a cell is essentially a solved problem (it’s complicated, but it’s solved above - each cell is its own \"document” - a `String` object from above).\n\nBut in addition to this, people are adding, removing, and reordering columns.  You want to ensure that if two people move a column or one person adds a column and another adds a row, things end up in a sane state, no matter which order you see the operations.\n\nTo do this, we consider the rows to be an ordered set of uuids. (And the same for the columns.) Then you have a map of column guid → row guid → String object. \n\nThe data itself a pile of CRDTs encoded with something like NSKeyedArchiver, but built on top of protobuf for these CRDTs. The root object contains a few tables, and a list of objects (like NSKeyedArchiver), and reverenced by index via a variant time that apple calls `ObjectID`. (I managed to figure this out by generically decoding the protobuf data and looking at it, but later found they ship an older revision of the proto files to their web app.)\n\nThis is the variant type used below:\n\n\n    message ObjectID {\n        uint64 unsignedIntegerValue = 2;\n        string stringValue = 4;\n        uint32 objectIndex = 6;\n    }\n\nThe `objectIndex` is a index into the list of `object` in `Document`. \n\nThe root `Document` is:\n\n\n    message Document {   \n        repeated DocObject object = 3;\n        repeated string keyItem = 4;\n        repeated string typeItem = 5;\n        repeated bytes uuidItem = 6;\n    }\n    \n    message DocObject {\n        RegisterLatest registerLatest = 1;\n        Dictionary dictionary = 6;\n        String string = 10;  // this is our fancy String above\n        CustomObject custom = 13;\n        OrderedSet orderedSet = 16;    \n    }\n    \n\nThe first object in the `object` field is the root object. A `CustomObject` is essentially a key/value map with a type.  The keys are indexed from `keyItem` and type is from `typeItem`.\n\n\n    message CustomObject {\n        int32 type = 1; // index into \"typeItem\" below\n        message MapEntry {\n            required int32 key = 1; // index into keyItem below\n            required ObjectID value = 2;\n        }\n        repeated MapEntry mapEntry = 3;\n    }\n\nFor a UUID, the type is `com.apple.CRDT.NSUUID` and there is a `UUIDIndex` field whose value is the index of the UUID in `uuidItem` of the `Document`.\n\nA `RegisterLatest` is just a CRDT for a value.  There is a clock, not shown here, which helps with merging conflicts.  It’s last write wins. It only appears to be used to point at an NSString custom object holding “CRTableColumnDirectionLeftToRight”.  I’m ignoring this at the moment.\n\n\n    message RegisterLatest {\n        ObjectID contents = 2;\n    }\n\nA `Dictionary` object holds object id for both key and value. In practice the keys points to a UUID `CustomObject` and the value is either a UUID or another dictionary. (This is a last-write-wins CRDT, I’m leaving out the clock values.) \n\n\n    message Dictionary {\n        message Element {\n           ObjectID key = 1;\n           ObjectID value = 2;\n        }\n        repeated Element element = 1;\n    }\n\nWhich leaves us with `OrderedSet`.  An ordered set leverages `String` to provide an vector of UUIDs via `TTArray`.  Pairs of string position to UUID are stored in `attachments` of `TTArray`.  Surrounding that is `Array` which has a CRDT Dictionary to map the uuid of the TTArray to the content in that position of the array (which happens to also be a UUID in this case).  And surrounding that is `OrderedSet` which contains another `Dictionary` of UUID to UUID, but the keys and values are the same (uuids from the TTArray space). This seems to be used to filter out deleted items in the case where you simultaneously move and delete a column. (The move does delete + add and the delete does a delete, so a copy remains in the Array.)\n\nTwo conflicting moves will also create duplicates in the array. Apple appears to handle this by ignoring all but the first instance. (And cleaning it up on the next move of that column.)\n\n\n    message OrderedSet {\n        Array ordering = 1;\n        Dictionary elements = 2; // set of elements that haven't been deleted\n    }\n    message Array {\n        TTArray array = 1; // TTArray\n        Dictionary contents = 2; // map of TTArray uuid to content uuid\n    }\n    message TTArray {\n        String contents = 1; // we don't actually reference this.\n        ArrayAttachment attachments = 2; // list of (position -> uuid\n    }\n    message ArrayAttachment {\n        int64 index = 1;\n        bytes uuid = 2;\n    }\n\n\n**Decoding Tables**\n\nOk, so to decode a table, the root object will be a CustomObject with the fields:\n\n| Field       | Value                                                       |\n| ----------- | ----------------------------------------------------------- |\n| crRows      | OrderedSet for row uuids                                    |\n| crColumns   | OrderedSet for column uuids                                 |\n| cellColumns | Dictionary of column uuid → Dictionary of row uuid → String |\n\nThe `value` of these fields are referenced by `objectIndex`.  Both `crRows` and `crColumns` are a `CustomObject` of type `OrderedSet`. \n\nTo get the list of uuids for these ordered sets, take each `ordering.array.attachments.uuid`, filter out values that don’t appear as keys in `elements` , and look up each of the resulting values in the dictionary `ordering.contents`. \n\nThen iterate through the column uuids and look them up in cellColumns. The result, for each column, will be a dictionary.  In this dictionary, look up each row uuid.  The result, if present, will be a String object (or rather an ObjectId pointing to a String object).  This is the content of the table cell.\n\n\n\n## TODO\n\nI still need to write up the attachment, folder, and encryption stuff.\n\n\n"
  },
  {
    "path": "notes2bear",
    "content": "#!/usr/bin/env python3\nimport zlib, os, sqlite3, re, zipfile, json\nfrom struct import unpack_from\nfrom datetime import datetime\n\n# Simple script to export Notes.app to Bear.app backup format\n# I have code to decode the tables and drawings (to svg), but not necessary for Bear\n\ndef uvarint(data,pos):\n    x = s = 0\n    while True:\n        b = data[pos]\n        pos += 1\n        x = x | ((b&0x7f)<<s)\n        if b < 0x80: return x,pos\n        s += 7\n\ndef readbytes(data,pos):\n    l,pos = uvarint(data,pos)\n    return data[pos:pos+l], pos+l\n\ndef readstruct(fmt,l):\n    return lambda data,pos: (unpack_from(fmt,data,pos)[0],pos+l)\n\nreaders = [ uvarint, readstruct('<d',8), readbytes, None, None, readstruct('<f',4) ]\n\ndef parse(data, schema):\n    \"parses a protobuf\"\n    obj = {}\n    pos = 0\n    while pos < len(data):\n        val,pos = uvarint(data,pos)\n        typ = val & 7\n        key = val >> 3\n        val, pos = readers[typ](data,pos)\n        if key not in schema: \n            continue\n        name, repeated, typ = schema[key]\n        if isinstance(typ, dict):\n            val = parse(val, typ)\n        if typ == 'string':\n            val = val.decode('utf8')\n        if repeated:\n            val = obj.get(name,[]) + [val]\n        obj[name] = val\n    return obj\n\ndef translate(data, media):\n    styles = {0: '# ', 1: '## ',100: '* ', 101: '* ', 102: '1. ', 103: '- '}\n    rval = []\n    refs = []\n    txt = data['string']\n    pos = 0\n    acc = None\n    pre = False\n    for run in data['attributeRun']:\n        l = run['length']\n        for frag in re.findall(r'\\n|[^\\n]+',txt[pos:pos+l]):\n            if acc is None: # start paragraph\n                pstyle = run.get('paragraphStyle',{}).get('style')\n                indent = run.get('paragraphStyle',{}).get('indent',0)\n                acc = \"  \"*indent+styles.get(pstyle,\"\")\n                if pstyle == 103 and run['paragraphStyle']['todo']['done']:\n                    acc = \"  \"*indent+\"+ \"\n                if pstyle == 4:\n                    if not pre: \n                        rval.append(\"```\")\n                elif pre:\n                    rval.append(\"```\")\n                pre = pstyle == 4\n            if frag == '\\n': # end paragraph\n                rval.append(acc)\n                acc = None\n            else: # accumulate and handle inline styles - although bear doesn't seem to support nested ones. \n                link = run.get('link')\n                info = run.get('attachmentInfo')\n                style = run.get('fontHints',0) + 4*run.get('underline',0) + 8*run.get('strikethrough',0)\n                if style & 1: frag = f'*{frag}*'\n                if style & 2: frag = f'/{frag}/'\n                if style & 4: frag = f'_{frag}_'\n                if style & 8: frag = f'~{frag}~'\n                if link: frag = f'[{frag}]({link})'\n                if info:\n                    id = info.get('attachmentIdentifier')\n                    fn = media.get(id)\n                    if fn:\n                        _,e = os.path.splitext(fn)\n                        acc += f'[assets/{id}{e}]'\n                        refs.append(id)\n                    else:\n                        acc += f\"ATTACH {info}\"\n                else:\n                    acc += frag\n        pos += l\n    if acc: rval.append(acc)\n    rval = '\\n'.join(rval)+\"\\n\"\n    return rval,refs\n\n# The schema subset needed for bear export\ndocschema = { \n    2: [ \"version\", 1, { \n        3: [ \"data\", 0, {\n            2: [ \"string\", 0, \"string\"],\n            5: [ \"attributeRun\", 1, {\n                1: [\"length\",0,0],\n                2: [\"paragraphStyle\", 0, {\n                    1: [\"style\", 0,0],\n                    4: [\"indent\",0,0],\n                    5: [\"todo\",0,{ \n                        1: [\"todoUUID\", 0, \"bytes\"],\n                        2: [\"done\",0,0]\n                    }]\n                }],\n                5: [\"fontHints\",0,0],\n                6: [\"underline\",0,0],\n                7: [\"strikethrough\",0,0],\n                9: [ \"link\", 0, \"string\" ],\n                12: [ \"attachmentInfo\", 0, {\n                    1: [ \"attachmentIdentifier\", 0, \"string\"],\n                    2: [ \"typeUTI\", 0, \"string\"]\n                }]\n            }]\n        }]\n    }]\n}\n\nif __name__ == '__main__':\n    root = os.path.expanduser(\"~/Library/Group Containers/group.com.apple.notes\")\n    db = sqlite3.Connection(os.path.join(root,'NoteStore.sqlite'))\n    media = {} # there is some indirection for attachments\n    for a,b,fn in db.execute('select a.zidentifier, b.zidentifier, b.zfilename from ziccloudsyncingobject a left join ziccloudsyncingobject b on a.zmedia = b.z_pk'):\n        if fn:\n            full = os.path.join(root,'Media',b,fn)\n        else:\n            full = os.path.join(root,'FallbackImages',a+\".jpg\")\n        if os.path.exists(full):\n            media[a] = full\n\n    count = 0\n    with zipfile.ZipFile('notes.bearbk','w') as zip:\n        for id, title, data, cdate, mdate in db.execute('select o.zidentifier, ztitle1, zdata, o.zcreationdate1, o.zmodificationdate1 from zicnotedata join ziccloudsyncingobject o on znotedata = zicnotedata.z_pk where zicnotedata.zcryptotag is null'):\n            ze = f'notes.bearbk/{id}.textbundle'\n            if data:\n                pb = zlib.decompress(data, 47)\n                doc = parse(pb, docschema)['version'][0]['data']\n                if not doc['string']: continue # some are blank\n                text,refs = translate(doc, media)\n                info = {\n                    \"type\":\"public.plain-text\", \n                    \"creatorIdentifier\": \"net.shinyfrog.bear\",\n                    \"net.shinyfrog.bear\": { \n                        \"modificationDate\": datetime.fromtimestamp(int(mdate)+978307200).isoformat(),\n                        \"creationDate\": datetime.fromtimestamp(int(cdate)+978307200).isoformat()\n                    },\n                    \"version\":2\n                }\n                zip.writestr(f'{ze}/info.json',json.dumps(info))\n                zip.writestr(f'{ze}/text.txt',text)\n                for ref in refs:\n                    _,e = os.path.splitext(media[ref])\n                    fn = f'{ze}/assets/{ref}{e}'\n                    zip.write(media[ref],fn)\n                count += 1\n    print(f\"wrote {count} notes to notes.bearbk\")\n\n"
  },
  {
    "path": "notes2html",
    "content": "#!/usr/bin/env python3\nimport os, sqlite3, json, struct, re, zipfile, sys\nfrom zlib import decompress\nimport xml.etree.ElementTree as ET\n\n# HTML construction utils\n\ndef append(rval,a):\n    \"append a to rval and return a\"\n    if isinstance(a,str):\n        i = len(rval)-1\n        if i<0:\n            rval.text = (rval.text or \"\")+a\n        else:\n            rval[i].tail = (rval[i].tail or \"\")+a\n    elif isinstance(a,ET.Element):\n        rval.append(a)\n    elif isinstance(a,dict):\n        rval.attrib.update(a)\n    else:\n        raise Exception(f\"unhandled type {type(a)}\")\n    return a\n\ndef E(tag,*args):\n    tag,*cc = tag.split('.')\n    rval = ET.Element(tag)\n    tail = None\n    if cc: rval.set('class',' '.join(cc))\n    for a in args:\n        append(rval,a)\n    return rval\n\n# protobuf parser\n\ndef uvarint(data,pos):\n    x = s = 0\n    while True:\n        b = data[pos]\n        pos += 1\n        x = x | ((b&0x7f)<<s)\n        if b < 0x80: return x,pos\n        s += 7\n\ndef readbytes(data,pos):\n    l,pos = uvarint(data,pos)\n    return data[pos:pos+l], pos+l\n\ndef readstruct(fmt,l):\n    return lambda data,pos: (struct.unpack_from(fmt,data,pos)[0],pos+l)\n\nreaders = [ uvarint, readstruct('<d',8), readbytes, None, None, readstruct('<f',4) ]\n\ndef parse(data, schema):\n    \"parses a protobuf\"\n    obj = {}\n    pos = 0\n    while pos < len(data):\n        val,pos = uvarint(data,pos)\n        typ = val & 7\n        key = val >> 3\n        val, pos = readers[typ](data,pos)\n        if key not in schema: \n            continue\n        name, repeated, typ = schema[key]\n        if isinstance(typ, dict):\n            val = parse(val, typ)\n        if typ == 'string':\n            val = val.decode('utf8')\n        if repeated:\n            val = obj.get(name,[]) + [val]\n        obj[name] = val\n    return obj\n\ndef svg(drawing):\n    \"Convert note drawing to SVG\"\n    width = drawing['bounds']['width']\n    height = drawing['bounds']['height']\n    rval = E('svg',{'width':str(width),'height':str(height)})\n    inks = drawing.get('inks')\n    for stroke in drawing.get('strokes',[]):\n        if stroke.get('hidden'):\n            continue\n        if 'points' in stroke:\n            swidth=1\n            ink = inks[stroke['inkIndex']]\n            c = ink['color']\n            red = int(c['red']*255)\n            green = int(c['green']*255)\n            blue = int(c['blue']*255)\n            alpha = c['alpha']\n            if ink['identifier'] == 'com.apple.ink.marker':\n                swidth = 15\n                alpha = 0.5\n\n            color = f'rgba({red},{green},{blue},{alpha})'\n            path = ''\n            for _,x,y,*rest in struct.iter_unpack('<3f5H2B',stroke['points']):\n                path += f\"L{x:.2f} {y:.2f}\"\n            path = \"M\"+path[1:]\n            \n            rval.append(E('path',{'d':\"M\"+path[1:],'stroke':color,'stroke-width':str(swidth),'stroke-cap':'round','fill':'none'}))\n            if 'transform' in stroke:\n                rval[-1].set('transform',\"matrix({a} {b} {c} {d} {tx:.2f} {ty:.2f})\".format(**stroke['transform']))\n    return rval\n\ndef render_html(note,attachments={}):\n    if note is None:\n        return \"\"\n    \"Convert note attributed string to HTML\"\n    # TODO\n    # - attachments\n    styles = {0:'h1',1:'h2',4:'pre',100:'li',101:'li',102:'li',103:'li'}\n    rval = E('div')\n    txt = note['string']\n    pos = 0\n    par = None\n    for run in note.get('attributeRun',[]):\n        l = run['length']\n        for frag in re.findall(r'\\n|[^\\n]+',txt[pos:pos+l]):\n            if par is None: # start paragraph\n                pstyle = run.get('paragraphStyle',{}).get('style',-1)\n                indent = run.get('paragraphStyle',{}).get('indent',0)\n                if pstyle > 100: # this mess handles merging bulleted lists\n                    tag = ['ul','ul','ol','ul'][pstyle - 100]\n                    par = rval\n                    while indent > 0:\n                        last = par[-1]\n                        if last.tag != tag:\n                            break\n                        par = last\n                        indent -= 1\n                    while indent >= 0:\n                        par = append(par,E(tag))\n                        indent -= 1\n                    par = append(par,E('li'))\n                elif pstyle == 4 and rval[-1].tag == 'pre':\n                    par = rval[-1]\n                    append(par,\"\\n\")\n                else:\n                    par = append(rval,E(styles.get(pstyle,'p')))\n                if pstyle == 103:\n                    par.append(E('input',{\"type\":\"checkbox\"}))\n                    if run.get('paragraphStyle',{}).get('todo',{}).get('done'):\n                        par[0].set('checked','')\n            if frag == '\\n':\n                par = None\n            else:\n                link = run.get('link')\n                info = run.get('attachmentInfo')\n                style = run.get('fontHints',0) + 4*run.get('underline',0) + 8*run.get('strikethrough',0)\n                if style & 1: frag = E('b',frag)\n                if style & 2: frag = E('em',frag)\n                if style & 4: frag = E('u',frag)\n                if style & 8: frag = E('strike',frag)\n                if info:\n                    attach = attachments.get(info.get('attachmentIdentifier'))\n                    if attach and attach.get('html'):\n                        frag = attach.get('html')\n                append(par,frag)\n        pos += l\n    return rval\n\ndef process_archive(table):\n    \"Decode a 'CRArchive'\"\n    objects = []\n\n    def dodict(v):\n        rval = {}\n        for e in v.get('element',[]):\n            rval[coerce(e['key'])] = coerce(e['value'])\n        return rval\n\n    def coerce(o):\n        [(k,v)]= o.items()\n        if 'custom' == k:\n            rval = dict((table['keyItem'][e['key']],coerce(e['value'])) for e in v['mapEntry'])\n            typ = table['typeItem'][v['type']]\n            if typ == 'com.apple.CRDT.NSUUID':\n                return table['uuidItem'][rval['UUIDIndex']]\n            if typ == 'com.apple.CRDT.NSString':\n                return rval['self']\n            return rval\n        if k == 'objectIndex':\n            return coerce(table['object'][v])\n        if k == 'registerLatest':\n            return coerce(v['contents'])\n        if k == 'orderedSet':\n            elements = dodict(v['elements'])\n            contents = dodict(v['ordering']['contents'])\n            rval = []\n            for a in v['ordering']['array']['attachments']:\n                value = contents[a['uuid']]\n                if value not in rval and a['uuid'] in elements:\n                    rval.append(value)\n            return rval\n        if k == 'dictionary':\n            return dodict(v)\n        if k in ('stringValue','unsignedIntegerValue','string'):\n            return v\n        raise Exception(f\"unhandled type {k}\")\n\n    return coerce(table['object'][0])\n\ndef render_table(table):\n    \"Render a table to html\"\n    table = process_archive(table)\n    rval = E('table')\n    for row in table['crRows']:\n        tr = E('tr')\n        rval.append(tr)\n        for col in table['crColumns']:\n            cell = table.get('cellColumns').get(col,{}).get(row)\n            td = E('td',render_html(cell))\n            rval.append(td)\n    return rval\n\ns_string = {\n    2: [ \"string\", 0, \"string\"],\n    5: [ \"attributeRun\", 1, {\n        1: [\"length\",0,0],\n        2: [\"paragraphStyle\", 0, {\n            1: [\"style\", 0,0],\n            4: [\"indent\",0,0],\n            5: [\"todo\",0,{ \n                1: [\"todoUUID\", 0, \"bytes\"],\n                2: [\"done\",0,0]\n            }]\n        }],\n        5: [\"fontHints\",0,0],\n        6: [\"underline\",0,0],\n        7: [\"strikethrough\",0,0],\n        9: [\"link\",0,\"string\"],\n        12: [ \"attachmentInfo\", 0, {\n            1: [ \"attachmentIdentifier\", 0, \"string\"],\n            2: [ \"typeUTI\", 0, \"string\"]\n        }]\n    }]\n}\n\ns_doc = { 2: [\"version\", 1, { 3: [\"data\", 0, s_string ]}]}\n\ns_drawing = { 2: [\"version\", 1, { 3: [\"data\", 0, {\n            4: [\"inks\",1, {\n                1:[\"color\",0,{1:[\"red\",0,0],2:[\"green\",0,0],3:[\"blue\",0,0],4:[\"alpha\",0,0]}],\n                2:[\"identifier\",0,\"string\"]\n            }],\n            5: [\"strokes\",1, {\n                3:[\"inkIndex\",0,0],\n                5:[\"points\",0,\"bytes\"],\n                9:[\"hidden\",0,0],\n                10: [\"transform\",0,{1:[\"a\",0,0],2:[\"b\",0,0],3:[\"c\",0,0],4:[\"d\",0,0],5:[\"tx\",0,0],6:[\"ty\",0,0]}]\n            }],\n            8: [\"bounds\", 0, {1:[\"originX\",0,0],2:[\"originY\",0,0],3:[\"width\",0,0],4:[\"height\",0,0]}]\n        }]\n    }\n]}\n\n# this essentially is a variant type\ns_oid = { 2:[\"unsignedIntegerValue\",0,0], 4:[\"stringValue\",0,'string'], 6:[\"objectIndex\",0,0] }\ns_dictionary = {1:[\"element\",1,{ 1:[\"key\",0,s_oid], 2:[\"value\",0,s_oid]}]}\ns_table = { 2: [\"version\", 1, { 3: [\"data\", 0, {\n    3: [\"object\",1,{\n        1:[\"registerLatest\",0,{2:[\"contents\",0,s_oid]}],\n        6:[\"dictionary\",0,s_dictionary],\n        10:[\"string\",0,s_string],\n        13:[\"custom\",0,{\n            1:[\"type\",0,0],\n            3:[\"mapEntry\",1,{\n                1:[\"key\",0,0],\n                2:[\"value\",0,s_oid]\n            }]\n        }],\n        16:[\"orderedSet\",0,{\n            1: [\"ordering\",0, {\n                1:[\"array\",0,{\n                    1:[\"contents\",0,s_string],\n                    2:[\"attachments\",1,{1:[\"index\",0,0],2:[\"uuid\",0,0]}]\n                }],\n                2:[\"contents\",0,s_dictionary]\n            }],\n            2: [\"elements\",0,s_dictionary]\n        }]\n    }],\n    4:[\"keyItem\",1,\"string\"],\n    5:[\"typeItem\",1,\"string\"],\n    6:[\"uuidItem\",1,\"bytes\"]\n}]}]}\n\ndef write(data,*path):\n    path = os.path.join(*path)\n    os.makedirs(os.path.dirname(path),exist_ok=True)\n    open(path,'wb').write(data)\n\nif __name__ == '__main__':\n    css = '''\n.underline { text-decoration: underline; }\n.strikethrough { text-decoration: line-through; }\n.todo { list-style-type: none; margin-left: -20px; }\n.dashitem { list-style-type: none; }\n.dashitem:before { content: \"-\"; text-indent: -5px }\n'''\n\n    def help():\n        print(f'Usage:\\n')\n        print(f'   {sys.argv[0]} [--svg] [--title] dest')\n        print(f'   --svg    Use svg for drawings')\n        print(f'   --title  Use title for filenames')\n        print(f'   dest     destination directory')\n        print()\n        exit(-1)\n\n    dest = None\n    use_svg = False    \n    use_title = False\n    for x in sys.argv[1:]:\n        if x == '--svg': use_svg = True\n        elif x == '--title': use_title = True\n        elif x.startswith('--'):\n            help()\n        else:\n            dest = x\n\n    if not dest:\n        help()\n\n\n    root = os.path.expanduser(\"~/Library/Group Containers/group.com.apple.notes\")\n    dbpath = os.path.join(root,'NoteStore.sqlite')\n    write(open(dbpath,'rb').read(),dest,'NoteStore.sqlite')\n    db = sqlite3.Connection(dbpath)\n\n    # process attachments first\n    attachments = {}\n    mquery = '''select a.zidentifier, a.zmergeabledata, a.ztypeuti, b.zidentifier, b.zfilename, a.zurlstring,a.ztitle\n        from ziccloudsyncingobject a left join ziccloudsyncingobject b on a.zmedia = b.z_pk\n        where a.zcryptotag is null and a.ztypeuti is not null'''\n    for id, data, typ, id2, fname, url,title in db.execute(mquery):\n        if typ == 'com.apple.drawing' and data and use_svg:\n            doc = parse(decompress(data,47),s_drawing)\n            attachments[id] = {'html': svg(doc['version'][0]['data'])}\n        elif typ == 'com.apple.notes.table' and data:\n            doc = parse(decompress(data,47),s_table)\n            attachments[id] = {'html': render_table(doc['version'][0]['data']) }\n        elif typ == 'public.url':\n            # there is a preview image somewhere too, but not sure I care\n            attachments[id] = {'html': E('a',{'href':url},title or url)}\n        elif fname:\n            fn = os.path.join('Media',id2,fname)\n            if typ in ['public.tiff','public.jpeg','public.png']:\n                attachments[id] = {'html': E('img',{'src':fn})}\n            else:\n                attachments[id] = {'html': E('a',{'href':fn},fname)}\n            src = os.path.join(root,fn)\n            if os.path.exists(src):\n                write(open(src,'rb').read(),dest,fn)\n        else:\n            fn = os.path.join('FallbackImages',id+'.jpg')\n            src = os.path.join(root,fn)\n            if os.path.exists(src):\n                attachments[id] = {'html': E('img',{'src':fn})}\n                write(open(src,'rb').read(),dest,fn)\n            \n    nquery = '''select a.zidentifier, a.ztitle1, n.zdata from zicnotedata n join ziccloudsyncingobject a on a.znotedata = n.z_pk \n        where n.zcryptotag is null and zdata is not null'''\n\n    seen = set()\n    count = 0\n    for id,title,data in db.execute(nquery):\n        pb = decompress(data,47)\n        doc = parse(pb,s_doc)['version'][0]['data']\n        section = render_html(doc,attachments)\n        section.tag = 'section'\n        hdoc = E('html',E('head',E('style',css)),E('body',section))\n        fn = id\n        if use_title and title:\n            tmp = title.replace(':','_').replace('/','_')[:64]\n            fn = tmp\n            ix = 1\n            while fn in seen:\n                fn = tmp + '_' + str(ix)\n                ix += 1\n            seen.add(fn)\n        html = ET.tostring(hdoc,method='html')\n        try:\n            write(html,dest,f'{fn}.html')\n        except:\n            print(f'write to {fn}.html failed, trying {id}.html')\n            write(html,dest,f'{fn}.html')\n        count += 1\n\n    print(f\"wrote {count} documents to {dest}\")\n"
  },
  {
    "path": "notes2quiver",
    "content": "#!/usr/bin/env python3\nimport os, sqlite3, json, struct, re, zipfile, sys\nfrom zlib import decompress\nimport xml.etree.ElementTree as ET\n\n# protobuf parser\n\ndef uvarint(data,pos):\n    x = s = 0\n    while True:\n        b = data[pos]\n        pos += 1\n        x = x | ((b&0x7f)<<s)\n        if b < 0x80: return x,pos\n        s += 7\n\ndef readbytes(data,pos):\n    l,pos = uvarint(data,pos)\n    return data[pos:pos+l], pos+l\n\ndef readstruct(fmt,l):\n    return lambda data,pos: (struct.unpack_from(fmt,data,pos)[0],pos+l)\n\nreaders = [ uvarint, readstruct('<d',8), readbytes, None, None, readstruct('<f',4) ]\n\ndef parse(data, schema):\n    \"parses a protobuf\"\n    obj = {}\n    pos = 0\n    while pos < len(data):\n        val,pos = uvarint(data,pos)\n        typ = val & 7\n        key = val >> 3\n        val, pos = readers[typ](data,pos)\n        if key not in schema: \n            continue\n        name, repeated, typ = schema[key]\n        if isinstance(typ, dict):\n            val = parse(val, typ)\n        if typ == 'string':\n            val = val.decode('utf8')\n        if repeated:\n            val = obj.get(name,[]) + [val]\n        obj[name] = val\n    return obj\n\n# HTML construction utils\n\ndef append(rval,a):\n    \"append a to rval and return a\"\n    if isinstance(a,str):\n        i = len(rval)-1\n        if i<0:\n            rval.text = (rval.text or \"\")+a\n        else:\n            rval[i].tail = (rval[i].tail or \"\")+a\n    elif isinstance(a,ET.Element):\n        rval.append(a)\n    elif isinstance(a,dict):\n        rval.attrib.update(a)\n    else:\n        raise Exception(f\"unhandled type {type(a)}\")\n    return a\n\ndef E(tag,*args,**attrs):\n    tag,*cc = tag.split('.')\n    rval = ET.Element(tag)\n    tail = None\n    if cc: rval.set('class',' '.join(cc))\n    if attrs:\n        append(rval,attrs)\n    for a in args:\n        append(rval,a)\n    return rval\n\n# Util for processing CRArchive\n\ndef process_archive(table):\n    \"Decode a 'CRArchive' (for tables)\"\n    objects = []\n\n    def dodict(v):\n        return {coerce(e['key']):coerce(e['value']) for e in v.get('element',[])}\n\n    def coerce(o):\n        [(k,v)] = o.items()\n        if 'custom' == k:\n            rval = dict((table['keyItem'][e['key']],coerce(e['value'])) for e in v['mapEntry'])\n            typ = table['typeItem'][v['type']]\n            if typ == 'com.apple.CRDT.NSUUID':\n                return table['uuidItem'][rval['UUIDIndex']]\n            if typ == 'com.apple.CRDT.NSString':\n                return rval['self']\n            return rval\n        if k == 'objectIndex':\n            return coerce(table['object'][v])\n        if k == 'registerLatest':\n            return coerce(v['contents'])\n        if k == 'orderedSet':\n            elements = dodict(v['elements'])\n            contents = dodict(v['ordering']['contents'])\n            rval = []\n            for a in v['ordering']['array']['attachments']:\n                value = contents[a['uuid']]\n                if value not in rval and a['uuid'] in elements:\n                    rval.append(value)\n            return rval\n        if k == 'dictionary':\n            return dodict(v)\n        if k in ('stringValue','unsignedIntegerValue','string'):\n            return v\n        raise Exception(f\"unhandled type {k}\")\n\n    return coerce(table['object'][0])\n\n# HTML\n\ndef render_html(note,get_attach=lambda x:None):\n    if note is None:\n        return \"\"\n    \"Convert note attributed string to HTML\"\n    # TODO\n    # - attachments\n    styles = {0:'h1',1:'h2',4:'pre',100:'li',101:'li',102:'li',103:'li'}\n    rval = E('div')\n    txt = note['string']\n    pos = 0\n    par = None\n    for run in note.get('attributeRun',[]):\n        l = run['length']\n        for frag in re.findall(r'\\n|[^\\n]+',txt[pos:pos+l]):\n            if par is None: # start paragraph\n                pstyle = run.get('paragraphStyle',{}).get('style',-1)\n                indent = run.get('paragraphStyle',{}).get('indent',0)\n                if pstyle >= 100: # this mess handles merging todo lists\n                    tag = ['ul','ul','ol','ul'][pstyle - 100]\n                    par = rval\n                    while indent > 0:\n                        last = par[-1]\n                        if last.tag != tag:\n                            break\n                        par = last\n                        indent -= 1\n                    while indent >= 0:\n                        par = append(par,E(tag))\n                        indent -= 1\n                    par = append(par,E('li'))\n                elif pstyle == 4 and rval[-1].tag == 'pre':\n                    par = rval[-1]\n                    append(par,\"\\n\")\n                else:\n                    par = append(rval,E(styles.get(pstyle,'p')))\n                if pstyle == 103:\n                    par.append(E('input',{\"type\":\"checkbox\"}))\n                    if run.get('paragraphStyle',{}).get('todo',{}).get('done'):\n                        par[0].set('checked','')\n            if frag == '\\n':\n                par = None\n            else:\n                link = run.get('link')\n                info = run.get('attachmentInfo')\n                style = run.get('fontHints',0) + 4*run.get('underline',0) + 8*run.get('strikethrough',0)\n                if style & 1: frag = E('b',frag)\n                if style & 2: frag = E('em',frag)\n                if style & 4: frag = E('u',frag)\n                if style & 8: frag = E('strike',frag)\n                if info:\n                    attach = get_attach(info.get('attachmentIdentifier'))\n                    if attach is not None:\n                        frag = attach\n                if link:\n                    frag = E('a',frag,href=link)\n\n                append(par,frag)\n        pos += l\n    return rval\n\ndef render_table_html(table):\n    \"Render a table to html\"\n    table = process_archive(table)\n    rval = E('table')\n    for row in table['crRows']:\n        tr = E('tr')\n        rval.append(tr)\n        for col in table['crColumns']:\n            cell = table.get('cellColumns').get(col,{}).get(row)\n            td = E('td',render_html(cell))\n            rval.append(td)\n    return rval\n\n\n# protobuf schema\n\ns_string = {\n    2: [ \"string\", 0, \"string\"],\n    5: [ \"attributeRun\", 1, {\n        1: [\"length\",0,0],\n        2: [\"paragraphStyle\", 0, {\n            1: [\"style\", 0,0],\n            4: [\"indent\",0,0],\n            5: [\"todo\",0,{ \n                1: [\"todoUUID\", 0, \"bytes\"],\n                2: [\"done\",0,0]\n            }]\n        }],\n        5: [\"fontHints\",0,0],\n        6: [\"underline\",0,0],\n        7: [\"strikethrough\",0,0],\n        9: [\"link\",0,\"string\"],\n        12: [ \"attachmentInfo\", 0, {\n            1: [ \"attachmentIdentifier\", 0, \"string\"],\n            2: [ \"typeUTI\", 0, \"string\"]\n        }]\n    }]\n}\n\ns_doc = { 2: [\"version\", 1, { 3: [\"data\", 0, s_string ]}]}\n\n# this essentially is a variant type\ns_oid = { 2:[\"unsignedIntegerValue\",0,0], 4:[\"stringValue\",0,'string'], 6:[\"objectIndex\",0,0] }\ns_dictionary = {1:[\"element\",1,{ 1:[\"key\",0,s_oid], 2:[\"value\",0,s_oid]}]}\ns_table = { 2: [\"version\", 1, { 3: [\"data\", 0, {\n    3: [\"object\",1,{\n        1:[\"registerLatest\",0,{2:[\"contents\",0,s_oid]}],\n        6:[\"dictionary\",0,s_dictionary],\n        10:[\"string\",0,s_string],\n        13:[\"custom\",0,{\n            1:[\"type\",0,0],\n            3:[\"mapEntry\",1,{\n                1:[\"key\",0,0],\n                2:[\"value\",0,s_oid]\n            }]\n        }],\n        16:[\"orderedSet\",0,{\n            1: [\"ordering\",0, {\n                1:[\"array\",0,{\n                    1:[\"contents\",0,s_string],\n                    2:[\"attachments\",1,{1:[\"index\",0,0],2:[\"uuid\",0,0]}]\n                }],\n                2:[\"contents\",0,s_dictionary]\n            }],\n            2: [\"elements\",0,s_dictionary]\n        }]\n    }],\n    4:[\"keyItem\",1,\"string\"],\n    5:[\"typeItem\",1,\"string\"],\n    6:[\"uuidItem\",1,\"bytes\"]\n}]}]}\n\n\nif __name__ == '__main__':\n    import uuid\n    from hashlib import md5\n\n    def write(data,*path):\n        path = os.path.join(*path)\n        os.makedirs(os.path.dirname(path),exist_ok=True)\n        open(path,'wb').write(data)\n\n    def writej(data,*path):\n        write(json.dumps(data,indent=True).encode('utf8'),*path)\n\n    dest = 'notes.qvnotebook'\n    if len(sys.argv)>1:\n        dest = sys.argv[1]\n\n    root = os.path.expanduser(\"~/Library/Group Containers/group.com.apple.notes\")\n    dbpath = os.path.join(root,'NoteStore.sqlite')\n    db = sqlite3.Connection(dbpath)\n\n    fn = os.path.join(dest,'meta.json')\n    if not os.path.exists(fn):\n        import uuid\n        writej({'name': 'Notes', 'uuid': str(uuid.uuid4())},fn)\n\n    # process attachments first\n    attachments = {}\n    mquery = '''select a.zmergeabledata, a.ztypeuti, b.zidentifier, b.zfilename, a.zurlstring,a.ztitle\n                  from ziccloudsyncingobject a left join ziccloudsyncingobject b on a.zmedia = b.z_pk\n                 where a.zcryptotag is null and a.ztypeuti is not null and a.zidentifier = ?'''\n\n    nquery = '''select a.zidentifier, a.ztitle1, a.zcreationdate1, a.zmodificationdate1, n.zdata \n                  from zicnotedata n \n                  join ziccloudsyncingobject a on a.znotedata = n.z_pk \n                 where n.zcryptotag is null and zdata is not null and zmarkedfordeletion is not 1'''\n\n    # For each note\n    for id,title,create,modify,data in db.execute(nquery):\n        dn = id+'.qvnote'\n        \n        def get_attach(id):\n            \"Find attachment via db / filesystem, copy into note and return html to reference it\"\n            row = db.execute(mquery,(id,)).fetchone()\n            if not row:\n                print(\"Missed attachment\",id)\n                return \"\"\n\n            data, typ, id2, fname, url, title = row\n            if typ == 'com.apple.notes.table' and data:\n                doc = parse(decompress(data,47),s_table)\n                return render_table_html(doc['version'][0]['data'])\n            elif typ == 'public.url':\n                # there is a preview image somewhere too, but not sure I care\n                return E(\"a\",title or url,href=url)\n            elif fname:\n                fn = os.path.join('Media',id2,fname)\n            else:\n                fn = os.path.join('FallbackImages',id+'.jpg')\n\n            src = os.path.join(root,fn)\n            if os.path.exists(src):\n                data = open(src,'rb').read()\n                hc = md5(data).hexdigest().upper()\n                _,ext = os.path.splitext(src)\n                fn2 = hc+ext\n                write(data, dest, dn, 'resources', fn2)\n                if ext in ['.jpg','.jpeg','.png','.tiff']:\n                    return E('img', src=f'quiver-image-url/{fn2}',alt=fn)\n                else:\n                    return E('a',fn,href=f'quiver-file-url/{fn2}')\n            print(\"fail\",id,typ)\n            return E('span')\n\n        pb = decompress(data,47)\n        doc = parse(pb,s_doc)['version'][0]['data']\n        section = render_html(doc,get_attach)\n        section = ET.tostring(section,method=\"html\").decode('utf8')\n\n        unix_ts = 0\n        content = {'title': title, 'cells': [{ 'type': 'text', 'data': section }]}\n        meta = {'uuid':id, 'created_at': int(create)+978307200, 'tags': [], 'title': title, 'updated_at': int(modify)+978307200}\n        writej(content, dest, dn, 'content.json')\n        writej(meta, dest, dn, 'meta.json')\n\n    print(f\"wrote files to {dest}\")\n"
  }
]