Full Code of dunhamsteve/notesutils for AI

Repository: dunhamsteve/notesutils
Branch: master
Commit: 582e569c9d0a
Files: 6
Total size: 44.7 KB

Directory structure:
gitextract_cr0hplzn/

├── LICENSE
├── README.md
├── notes.md
├── notes2bear
├── notes2html
└── notes2quiver

================================================
FILE CONTENTS
================================================

================================================
FILE: LICENSE
================================================
This is free and unencumbered software released into the public domain.

Anyone is free to copy, modify, publish, use, compile, sell, or
distribute this software, either in source code form or as a compiled
binary, for any purpose, commercial or non-commercial, and by any
means.

In jurisdictions that recognize copyright laws, the author or authors
of this software dedicate any and all copyright interest in the
software to the public domain. We make this dedication for the benefit
of the public at large and to the detriment of our heirs and
successors. We intend this dedication to be an overt act of
relinquishment in perpetuity of all present and future rights to this
software under copyright law.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.

For more information, please refer to <http://unlicense.org>


================================================
FILE: README.md
================================================
# Apple Notes Export Tools

This repository includes a few python export tools for Apple's "Notes.app".  The scripts require python3, but I tried to make the scripts self-contained and concise, with no additional dependencies. I may revisit this and break the common code into a library in the future.

This repository is in the public domain.

## `notes.md`

Description of how notes data is stored.

## `notes2bear`

Writes a `notes.bearbk` file in Bear's backup format (a zip file with their own markup flavor).  It doesn't handle tables because Bear doesn't do tables yet. This shaves about 100 lines from the script.

## `notes2html` 

`notes2html` takes an destination directory and writes a tree of html files and images.  One html file per note and any associated media in the media directory. If you specify `--title` the files with be named with the title guessed by Apple (with / replaced by _).  If you specify `--svg`, any drawings will be rendered as inline SVG; otherwise, the fallback jpg files provided by Apple will be used.

Usage is:

```
notes2html [--svg] [--title] dest
```












================================================
FILE: notes.md
================================================
# Notes on Notes.app
For future reference and to aid anyone else who might want to extract data from the "Notes" app. I compiled this out of curiosity and a desire to backup my notes content. 

This document only covers the current format of notes, as synced with iCloud. The application also supports IMAP synced notes with reduced functionality. They are stored in a separate database (and on the IMAP server). You're on your own there, but it's pretty much just MIME and HTML.

## OSX Files

The notes are stored in `~/Library/Group Containers/group.com.apple.notes` in a sqlite database named `NoteStore.sqlite`. The database contains sync state from iCloud. It's a CoreData store, but I approached it from plain sqlite.

We're interested in the `ZICCLOUDSYNCINGOBJECT` table and the `ZICNOTEDATA` table. The former contains the sync state of each iCloud object - notes, attachments, and folders, and the latter contains the note data. Everything has a UUID identifier `ZIDENTIFIER`. The `ZICCLOUDSYNCINGOBJECT` table column `ZNOTEDATA` points to the `Z_PK` column in `ZICNOTEDATA`.
The meat of the data is in zlib compressed protobuf blobs. For documents, it's `ZDATA` in `ZNOTEDATA` and for tables/drawings, it's `ZMERGEABLEDATA` in `ZICCLOUDSYNCINGOBJECT`.

Apple uses a [CvRDT](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) for syncing (evidenced by various strings appearing in the data and analysis of what happens as you edit). I suspect the one used for documents, which they call "topotext" is not actually a CvRDT (they seem to order conflicts on a first to sync basis rather than vector clock) but I could be mistaken. It doesn't matter since everything goes through iCloud, which serializes the merges.

## Protobuf Data

**Document Wrapper**
Everything is wrapped with a versioned document object:

    message Document {
        repeated Version version = 2;
    }
    message Version {
        optional bytes data = 3;
    }

There are additional fields that aren't relevant to us, and I've never seen more than one `version`. The content of the `data` field is also protobuf but varies depending on whether we're looking at a note, table, or drawing.

**Notes**
The protobuf data for a note is a `String` as described below. Assume everything is optional and I’ve elided the CRDT stuff.  (Repeated field 3 of String is a sequence clock, length, attribute clock, tombstone, and children.  It forms a DAG. For chunks of length >1 the clock is implicitly incremented for each character.)


    message String {
        string string = 2;
        // these are in order, disjoint, and their length sums to the length of string
        repeated AttributeRun attributeRun = 5;
    }
    message AttributeRun {
        uint32 length = 1;
        ParagraphStyle paragraphStyle = 2;
        Font font = 3;    
        uint32 fontHints = 5; // 1:bold, 2:italic, 3:bold italic
        uint32 underline = 6;
        uint32 strikethrough = 7;
        int32 superscript = 8; // sign indicates super/subscript
        string link = 9;
        Color color = 10;
        AttachmentInfo attachmentInfo = 12;
    }
    message ParagraphStyle {
        // 0:title, 1:heading, 4:monospace, 100:dotitem, 101:dashitem, 102:numitem, 
        // 103:todoitem
        uint32 style = 1;
        uint32 alignment = 2; // 0:left, 1:center, 2:right, 3:justified
        int32 indent = 4;
        Todo todo = 5;
    }
    message Font {
        string name = 1;
        float pointSize = 2;
        uint32 fontHints = 3;
    }
    message AttachmentInfo {
        string attachmentIdentifier = 1;
        string typeUTI = 2;
    }
    message Todo {
        bytes todoUUID = 1;
        bool done = 2;
    }
    message Color {
        float red = 1;
        float green = 2;
        float blue = 3;
        float alpha = 4;
    }

**Drawings**
This info isn’t strictly necessary.  For each drawing, you’ll find a rendering in `FallbackImages/UUID.jpg`. I was curious whether I could recover / backup the original vector data, so I came up with the following. (The root object here is `Drawing`.)


    message Drawing {
        int64 serializationVersion = 1;
        repeated bytes replicaUUIDs = 2;
        repeated StrokeID versionVector = 3;
        repeated Ink inks = 4;
        repeated Stroke strokes = 5;
        int64 orientation = 6;
        StrokeID orientationVersion = 7;
        Rectangle bounds = 8;
        bytes uuid = 9;
    }
    message Color {
        float red = 1;
        float green = 2;
        float blue = 3;
        float alpha = 4;
    }
    message Rectangle {
        float height = 4;
        float originX = 1;
        float originY = 2;
        float width = 3;
    }
    message Transform {
        float a = 1;
        float b = 2;
        float c = 3;
        float d = 4;
        float tx = 5;
        float ty = 6;
    }
    message Ink {
        Color color = 1;
        string identifier = 2;
        int64 version = 3;
    }
    message Stroke {
        int64 inkIndex = 3;
        int64 pointsCount = 4;
        bytes points = 5;
        Rectangle bounds = 6;
        bool hidden = 9;
        double timestamp = 11;
        bool createdWithFinger = 12;
        Transform transform = 10;
    }

The byte array `points` is compactly encoded list of points. It is a sequence of this struct:


    struct PKCompressedStrokePoint {
        float timestamp;            // timestamp (delta from somethin, probably previous)
        float xpos;
        float ypos;
        unsigned short radius;      // radius*10
        unsigned short aspectRatio; // aspectRatio*1000
        unsigned short edgeWidth;   // edgeWidth*10
        unsigned short force;       // force*1000
        unsigned short azimuth;     // azimuth*10430.2191955274
        unsigned char altitude;     // altitude*162.338041953733
        unsigned char opacity;      // opacity*255
    };

The `inkIndex` field points into the array `inks`. The `identifier` in an `Ink` includes stuff like `com.apple.ink.marker`. I kinda fudged this in my svg generation - apple’s rendering code is much more sophisticated, taking azimuth/altitude into account.  My code works well enough for pen, but falls short on marker.  You may be better off with the jpeg.

**Tables**
This one is complicated, and I think a bit of explanation is in order to explain why it’s structured this way. They are trying to model a table with multiple people editing at the same time. Editing the contents of a cell is essentially a solved problem (it’s complicated, but it’s solved above - each cell is its own "document” - a `String` object from above).

But in addition to this, people are adding, removing, and reordering columns.  You want to ensure that if two people move a column or one person adds a column and another adds a row, things end up in a sane state, no matter which order you see the operations.

To do this, we consider the rows to be an ordered set of uuids. (And the same for the columns.) Then you have a map of column guid → row guid → String object. 

The data itself a pile of CRDTs encoded with something like NSKeyedArchiver, but built on top of protobuf for these CRDTs. The root object contains a few tables, and a list of objects (like NSKeyedArchiver), and reverenced by index via a variant time that apple calls `ObjectID`. (I managed to figure this out by generically decoding the protobuf data and looking at it, but later found they ship an older revision of the proto files to their web app.)

This is the variant type used below:


    message ObjectID {
        uint64 unsignedIntegerValue = 2;
        string stringValue = 4;
        uint32 objectIndex = 6;
    }

The `objectIndex` is a index into the list of `object` in `Document`. 

The root `Document` is:


    message Document {   
        repeated DocObject object = 3;
        repeated string keyItem = 4;
        repeated string typeItem = 5;
        repeated bytes uuidItem = 6;
    }
    
    message DocObject {
        RegisterLatest registerLatest = 1;
        Dictionary dictionary = 6;
        String string = 10;  // this is our fancy String above
        CustomObject custom = 13;
        OrderedSet orderedSet = 16;    
    }
    

The first object in the `object` field is the root object. A `CustomObject` is essentially a key/value map with a type.  The keys are indexed from `keyItem` and type is from `typeItem`.


    message CustomObject {
        int32 type = 1; // index into "typeItem" below
        message MapEntry {
            required int32 key = 1; // index into keyItem below
            required ObjectID value = 2;
        }
        repeated MapEntry mapEntry = 3;
    }

For a UUID, the type is `com.apple.CRDT.NSUUID` and there is a `UUIDIndex` field whose value is the index of the UUID in `uuidItem` of the `Document`.

A `RegisterLatest` is just a CRDT for a value.  There is a clock, not shown here, which helps with merging conflicts.  It’s last write wins. It only appears to be used to point at an NSString custom object holding “CRTableColumnDirectionLeftToRight”.  I’m ignoring this at the moment.


    message RegisterLatest {
        ObjectID contents = 2;
    }

A `Dictionary` object holds object id for both key and value. In practice the keys points to a UUID `CustomObject` and the value is either a UUID or another dictionary. (This is a last-write-wins CRDT, I’m leaving out the clock values.) 


    message Dictionary {
        message Element {
           ObjectID key = 1;
           ObjectID value = 2;
        }
        repeated Element element = 1;
    }

Which leaves us with `OrderedSet`.  An ordered set leverages `String` to provide an vector of UUIDs via `TTArray`.  Pairs of string position to UUID are stored in `attachments` of `TTArray`.  Surrounding that is `Array` which has a CRDT Dictionary to map the uuid of the TTArray to the content in that position of the array (which happens to also be a UUID in this case).  And surrounding that is `OrderedSet` which contains another `Dictionary` of UUID to UUID, but the keys and values are the same (uuids from the TTArray space). This seems to be used to filter out deleted items in the case where you simultaneously move and delete a column. (The move does delete + add and the delete does a delete, so a copy remains in the Array.)

Two conflicting moves will also create duplicates in the array. Apple appears to handle this by ignoring all but the first instance. (And cleaning it up on the next move of that column.)


    message OrderedSet {
        Array ordering = 1;
        Dictionary elements = 2; // set of elements that haven't been deleted
    }
    message Array {
        TTArray array = 1; // TTArray
        Dictionary contents = 2; // map of TTArray uuid to content uuid
    }
    message TTArray {
        String contents = 1; // we don't actually reference this.
        ArrayAttachment attachments = 2; // list of (position -> uuid
    }
    message ArrayAttachment {
        int64 index = 1;
        bytes uuid = 2;
    }


**Decoding Tables**

Ok, so to decode a table, the root object will be a CustomObject with the fields:

| Field       | Value                                                       |
| ----------- | ----------------------------------------------------------- |
| crRows      | OrderedSet for row uuids                                    |
| crColumns   | OrderedSet for column uuids                                 |
| cellColumns | Dictionary of column uuid → Dictionary of row uuid → String |

The `value` of these fields are referenced by `objectIndex`.  Both `crRows` and `crColumns` are a `CustomObject` of type `OrderedSet`. 

To get the list of uuids for these ordered sets, take each `ordering.array.attachments.uuid`, filter out values that don’t appear as keys in `elements` , and look up each of the resulting values in the dictionary `ordering.contents`. 

Then iterate through the column uuids and look them up in cellColumns. The result, for each column, will be a dictionary.  In this dictionary, look up each row uuid.  The result, if present, will be a String object (or rather an ObjectId pointing to a String object).  This is the content of the table cell.



## TODO

I still need to write up the attachment, folder, and encryption stuff.




================================================
FILE: notes2bear
================================================
#!/usr/bin/env python3
import zlib, os, sqlite3, re, zipfile, json
from struct import unpack_from
from datetime import datetime

# Simple script to export Notes.app to Bear.app backup format
# I have code to decode the tables and drawings (to svg), but not necessary for Bear

def uvarint(data,pos):
    x = s = 0
    while True:
        b = data[pos]
        pos += 1
        x = x | ((b&0x7f)<<s)
        if b < 0x80: return x,pos
        s += 7

def readbytes(data,pos):
    l,pos = uvarint(data,pos)
    return data[pos:pos+l], pos+l

def readstruct(fmt,l):
    return lambda data,pos: (unpack_from(fmt,data,pos)[0],pos+l)

readers = [ uvarint, readstruct('<d',8), readbytes, None, None, readstruct('<f',4) ]

def parse(data, schema):
    "parses a protobuf"
    obj = {}
    pos = 0
    while pos < len(data):
        val,pos = uvarint(data,pos)
        typ = val & 7
        key = val >> 3
        val, pos = readers[typ](data,pos)
        if key not in schema: 
            continue
        name, repeated, typ = schema[key]
        if isinstance(typ, dict):
            val = parse(val, typ)
        if typ == 'string':
            val = val.decode('utf8')
        if repeated:
            val = obj.get(name,[]) + [val]
        obj[name] = val
    return obj

def translate(data, media):
    styles = {0: '# ', 1: '## ',100: '* ', 101: '* ', 102: '1. ', 103: '- '}
    rval = []
    refs = []
    txt = data['string']
    pos = 0
    acc = None
    pre = False
    for run in data['attributeRun']:
        l = run['length']
        for frag in re.findall(r'\n|[^\n]+',txt[pos:pos+l]):
            if acc is None: # start paragraph
                pstyle = run.get('paragraphStyle',{}).get('style')
                indent = run.get('paragraphStyle',{}).get('indent',0)
                acc = "  "*indent+styles.get(pstyle,"")
                if pstyle == 103 and run['paragraphStyle']['todo']['done']:
                    acc = "  "*indent+"+ "
                if pstyle == 4:
                    if not pre: 
                        rval.append("```")
                elif pre:
                    rval.append("```")
                pre = pstyle == 4
            if frag == '\n': # end paragraph
                rval.append(acc)
                acc = None
            else: # accumulate and handle inline styles - although bear doesn't seem to support nested ones. 
                link = run.get('link')
                info = run.get('attachmentInfo')
                style = run.get('fontHints',0) + 4*run.get('underline',0) + 8*run.get('strikethrough',0)
                if style & 1: frag = f'*{frag}*'
                if style & 2: frag = f'/{frag}/'
                if style & 4: frag = f'_{frag}_'
                if style & 8: frag = f'~{frag}~'
                if link: frag = f'[{frag}]({link})'
                if info:
                    id = info.get('attachmentIdentifier')
                    fn = media.get(id)
                    if fn:
                        _,e = os.path.splitext(fn)
                        acc += f'[assets/{id}{e}]'
                        refs.append(id)
                    else:
                        acc += f"ATTACH {info}"
                else:
                    acc += frag
        pos += l
    if acc: rval.append(acc)
    rval = '\n'.join(rval)+"\n"
    return rval,refs

# The schema subset needed for bear export
docschema = { 
    2: [ "version", 1, { 
        3: [ "data", 0, {
            2: [ "string", 0, "string"],
            5: [ "attributeRun", 1, {
                1: ["length",0,0],
                2: ["paragraphStyle", 0, {
                    1: ["style", 0,0],
                    4: ["indent",0,0],
                    5: ["todo",0,{ 
                        1: ["todoUUID", 0, "bytes"],
                        2: ["done",0,0]
                    }]
                }],
                5: ["fontHints",0,0],
                6: ["underline",0,0],
                7: ["strikethrough",0,0],
                9: [ "link", 0, "string" ],
                12: [ "attachmentInfo", 0, {
                    1: [ "attachmentIdentifier", 0, "string"],
                    2: [ "typeUTI", 0, "string"]
                }]
            }]
        }]
    }]
}

if __name__ == '__main__':
    root = os.path.expanduser("~/Library/Group Containers/group.com.apple.notes")
    db = sqlite3.Connection(os.path.join(root,'NoteStore.sqlite'))
    media = {} # there is some indirection for attachments
    for a,b,fn in db.execute('select a.zidentifier, b.zidentifier, b.zfilename from ziccloudsyncingobject a left join ziccloudsyncingobject b on a.zmedia = b.z_pk'):
        if fn:
            full = os.path.join(root,'Media',b,fn)
        else:
            full = os.path.join(root,'FallbackImages',a+".jpg")
        if os.path.exists(full):
            media[a] = full

    count = 0
    with zipfile.ZipFile('notes.bearbk','w') as zip:
        for id, title, data, cdate, mdate in db.execute('select o.zidentifier, ztitle1, zdata, o.zcreationdate1, o.zmodificationdate1 from zicnotedata join ziccloudsyncingobject o on znotedata = zicnotedata.z_pk where zicnotedata.zcryptotag is null'):
            ze = f'notes.bearbk/{id}.textbundle'
            if data:
                pb = zlib.decompress(data, 47)
                doc = parse(pb, docschema)['version'][0]['data']
                if not doc['string']: continue # some are blank
                text,refs = translate(doc, media)
                info = {
                    "type":"public.plain-text", 
                    "creatorIdentifier": "net.shinyfrog.bear",
                    "net.shinyfrog.bear": { 
                        "modificationDate": datetime.fromtimestamp(int(mdate)+978307200).isoformat(),
                        "creationDate": datetime.fromtimestamp(int(cdate)+978307200).isoformat()
                    },
                    "version":2
                }
                zip.writestr(f'{ze}/info.json',json.dumps(info))
                zip.writestr(f'{ze}/text.txt',text)
                for ref in refs:
                    _,e = os.path.splitext(media[ref])
                    fn = f'{ze}/assets/{ref}{e}'
                    zip.write(media[ref],fn)
                count += 1
    print(f"wrote {count} notes to notes.bearbk")



================================================
FILE: notes2html
================================================
#!/usr/bin/env python3
import os, sqlite3, json, struct, re, zipfile, sys
from zlib import decompress
import xml.etree.ElementTree as ET

# HTML construction utils

def append(rval,a):
    "append a to rval and return a"
    if isinstance(a,str):
        i = len(rval)-1
        if i<0:
            rval.text = (rval.text or "")+a
        else:
            rval[i].tail = (rval[i].tail or "")+a
    elif isinstance(a,ET.Element):
        rval.append(a)
    elif isinstance(a,dict):
        rval.attrib.update(a)
    else:
        raise Exception(f"unhandled type {type(a)}")
    return a

def E(tag,*args):
    tag,*cc = tag.split('.')
    rval = ET.Element(tag)
    tail = None
    if cc: rval.set('class',' '.join(cc))
    for a in args:
        append(rval,a)
    return rval

# protobuf parser

def uvarint(data,pos):
    x = s = 0
    while True:
        b = data[pos]
        pos += 1
        x = x | ((b&0x7f)<<s)
        if b < 0x80: return x,pos
        s += 7

def readbytes(data,pos):
    l,pos = uvarint(data,pos)
    return data[pos:pos+l], pos+l

def readstruct(fmt,l):
    return lambda data,pos: (struct.unpack_from(fmt,data,pos)[0],pos+l)

readers = [ uvarint, readstruct('<d',8), readbytes, None, None, readstruct('<f',4) ]

def parse(data, schema):
    "parses a protobuf"
    obj = {}
    pos = 0
    while pos < len(data):
        val,pos = uvarint(data,pos)
        typ = val & 7
        key = val >> 3
        val, pos = readers[typ](data,pos)
        if key not in schema: 
            continue
        name, repeated, typ = schema[key]
        if isinstance(typ, dict):
            val = parse(val, typ)
        if typ == 'string':
            val = val.decode('utf8')
        if repeated:
            val = obj.get(name,[]) + [val]
        obj[name] = val
    return obj

def svg(drawing):
    "Convert note drawing to SVG"
    width = drawing['bounds']['width']
    height = drawing['bounds']['height']
    rval = E('svg',{'width':str(width),'height':str(height)})
    inks = drawing.get('inks')
    for stroke in drawing.get('strokes',[]):
        if stroke.get('hidden'):
            continue
        if 'points' in stroke:
            swidth=1
            ink = inks[stroke['inkIndex']]
            c = ink['color']
            red = int(c['red']*255)
            green = int(c['green']*255)
            blue = int(c['blue']*255)
            alpha = c['alpha']
            if ink['identifier'] == 'com.apple.ink.marker':
                swidth = 15
                alpha = 0.5

            color = f'rgba({red},{green},{blue},{alpha})'
            path = ''
            for _,x,y,*rest in struct.iter_unpack('<3f5H2B',stroke['points']):
                path += f"L{x:.2f} {y:.2f}"
            path = "M"+path[1:]
            
            rval.append(E('path',{'d':"M"+path[1:],'stroke':color,'stroke-width':str(swidth),'stroke-cap':'round','fill':'none'}))
            if 'transform' in stroke:
                rval[-1].set('transform',"matrix({a} {b} {c} {d} {tx:.2f} {ty:.2f})".format(**stroke['transform']))
    return rval

def render_html(note,attachments={}):
    if note is None:
        return ""
    "Convert note attributed string to HTML"
    # TODO
    # - attachments
    styles = {0:'h1',1:'h2',4:'pre',100:'li',101:'li',102:'li',103:'li'}
    rval = E('div')
    txt = note['string']
    pos = 0
    par = None
    for run in note.get('attributeRun',[]):
        l = run['length']
        for frag in re.findall(r'\n|[^\n]+',txt[pos:pos+l]):
            if par is None: # start paragraph
                pstyle = run.get('paragraphStyle',{}).get('style',-1)
                indent = run.get('paragraphStyle',{}).get('indent',0)
                if pstyle > 100: # this mess handles merging bulleted lists
                    tag = ['ul','ul','ol','ul'][pstyle - 100]
                    par = rval
                    while indent > 0:
                        last = par[-1]
                        if last.tag != tag:
                            break
                        par = last
                        indent -= 1
                    while indent >= 0:
                        par = append(par,E(tag))
                        indent -= 1
                    par = append(par,E('li'))
                elif pstyle == 4 and rval[-1].tag == 'pre':
                    par = rval[-1]
                    append(par,"\n")
                else:
                    par = append(rval,E(styles.get(pstyle,'p')))
                if pstyle == 103:
                    par.append(E('input',{"type":"checkbox"}))
                    if run.get('paragraphStyle',{}).get('todo',{}).get('done'):
                        par[0].set('checked','')
            if frag == '\n':
                par = None
            else:
                link = run.get('link')
                info = run.get('attachmentInfo')
                style = run.get('fontHints',0) + 4*run.get('underline',0) + 8*run.get('strikethrough',0)
                if style & 1: frag = E('b',frag)
                if style & 2: frag = E('em',frag)
                if style & 4: frag = E('u',frag)
                if style & 8: frag = E('strike',frag)
                if info:
                    attach = attachments.get(info.get('attachmentIdentifier'))
                    if attach and attach.get('html'):
                        frag = attach.get('html')
                append(par,frag)
        pos += l
    return rval

def process_archive(table):
    "Decode a 'CRArchive'"
    objects = []

    def dodict(v):
        rval = {}
        for e in v.get('element',[]):
            rval[coerce(e['key'])] = coerce(e['value'])
        return rval

    def coerce(o):
        [(k,v)]= o.items()
        if 'custom' == k:
            rval = dict((table['keyItem'][e['key']],coerce(e['value'])) for e in v['mapEntry'])
            typ = table['typeItem'][v['type']]
            if typ == 'com.apple.CRDT.NSUUID':
                return table['uuidItem'][rval['UUIDIndex']]
            if typ == 'com.apple.CRDT.NSString':
                return rval['self']
            return rval
        if k == 'objectIndex':
            return coerce(table['object'][v])
        if k == 'registerLatest':
            return coerce(v['contents'])
        if k == 'orderedSet':
            elements = dodict(v['elements'])
            contents = dodict(v['ordering']['contents'])
            rval = []
            for a in v['ordering']['array']['attachments']:
                value = contents[a['uuid']]
                if value not in rval and a['uuid'] in elements:
                    rval.append(value)
            return rval
        if k == 'dictionary':
            return dodict(v)
        if k in ('stringValue','unsignedIntegerValue','string'):
            return v
        raise Exception(f"unhandled type {k}")

    return coerce(table['object'][0])

def render_table(table):
    "Render a table to html"
    table = process_archive(table)
    rval = E('table')
    for row in table['crRows']:
        tr = E('tr')
        rval.append(tr)
        for col in table['crColumns']:
            cell = table.get('cellColumns').get(col,{}).get(row)
            td = E('td',render_html(cell))
            rval.append(td)
    return rval

s_string = {
    2: [ "string", 0, "string"],
    5: [ "attributeRun", 1, {
        1: ["length",0,0],
        2: ["paragraphStyle", 0, {
            1: ["style", 0,0],
            4: ["indent",0,0],
            5: ["todo",0,{ 
                1: ["todoUUID", 0, "bytes"],
                2: ["done",0,0]
            }]
        }],
        5: ["fontHints",0,0],
        6: ["underline",0,0],
        7: ["strikethrough",0,0],
        9: ["link",0,"string"],
        12: [ "attachmentInfo", 0, {
            1: [ "attachmentIdentifier", 0, "string"],
            2: [ "typeUTI", 0, "string"]
        }]
    }]
}

s_doc = { 2: ["version", 1, { 3: ["data", 0, s_string ]}]}

s_drawing = { 2: ["version", 1, { 3: ["data", 0, {
            4: ["inks",1, {
                1:["color",0,{1:["red",0,0],2:["green",0,0],3:["blue",0,0],4:["alpha",0,0]}],
                2:["identifier",0,"string"]
            }],
            5: ["strokes",1, {
                3:["inkIndex",0,0],
                5:["points",0,"bytes"],
                9:["hidden",0,0],
                10: ["transform",0,{1:["a",0,0],2:["b",0,0],3:["c",0,0],4:["d",0,0],5:["tx",0,0],6:["ty",0,0]}]
            }],
            8: ["bounds", 0, {1:["originX",0,0],2:["originY",0,0],3:["width",0,0],4:["height",0,0]}]
        }]
    }
]}

# this essentially is a variant type
s_oid = { 2:["unsignedIntegerValue",0,0], 4:["stringValue",0,'string'], 6:["objectIndex",0,0] }
s_dictionary = {1:["element",1,{ 1:["key",0,s_oid], 2:["value",0,s_oid]}]}
s_table = { 2: ["version", 1, { 3: ["data", 0, {
    3: ["object",1,{
        1:["registerLatest",0,{2:["contents",0,s_oid]}],
        6:["dictionary",0,s_dictionary],
        10:["string",0,s_string],
        13:["custom",0,{
            1:["type",0,0],
            3:["mapEntry",1,{
                1:["key",0,0],
                2:["value",0,s_oid]
            }]
        }],
        16:["orderedSet",0,{
            1: ["ordering",0, {
                1:["array",0,{
                    1:["contents",0,s_string],
                    2:["attachments",1,{1:["index",0,0],2:["uuid",0,0]}]
                }],
                2:["contents",0,s_dictionary]
            }],
            2: ["elements",0,s_dictionary]
        }]
    }],
    4:["keyItem",1,"string"],
    5:["typeItem",1,"string"],
    6:["uuidItem",1,"bytes"]
}]}]}

def write(data,*path):
    path = os.path.join(*path)
    os.makedirs(os.path.dirname(path),exist_ok=True)
    open(path,'wb').write(data)

if __name__ == '__main__':
    css = '''
.underline { text-decoration: underline; }
.strikethrough { text-decoration: line-through; }
.todo { list-style-type: none; margin-left: -20px; }
.dashitem { list-style-type: none; }
.dashitem:before { content: "-"; text-indent: -5px }
'''

    def help():
        print(f'Usage:\n')
        print(f'   {sys.argv[0]} [--svg] [--title] dest')
        print(f'   --svg    Use svg for drawings')
        print(f'   --title  Use title for filenames')
        print(f'   dest     destination directory')
        print()
        exit(-1)

    dest = None
    use_svg = False    
    use_title = False
    for x in sys.argv[1:]:
        if x == '--svg': use_svg = True
        elif x == '--title': use_title = True
        elif x.startswith('--'):
            help()
        else:
            dest = x

    if not dest:
        help()


    root = os.path.expanduser("~/Library/Group Containers/group.com.apple.notes")
    dbpath = os.path.join(root,'NoteStore.sqlite')
    write(open(dbpath,'rb').read(),dest,'NoteStore.sqlite')
    db = sqlite3.Connection(dbpath)

    # process attachments first
    attachments = {}
    mquery = '''select a.zidentifier, a.zmergeabledata, a.ztypeuti, b.zidentifier, b.zfilename, a.zurlstring,a.ztitle
        from ziccloudsyncingobject a left join ziccloudsyncingobject b on a.zmedia = b.z_pk
        where a.zcryptotag is null and a.ztypeuti is not null'''
    for id, data, typ, id2, fname, url,title in db.execute(mquery):
        if typ == 'com.apple.drawing' and data and use_svg:
            doc = parse(decompress(data,47),s_drawing)
            attachments[id] = {'html': svg(doc['version'][0]['data'])}
        elif typ == 'com.apple.notes.table' and data:
            doc = parse(decompress(data,47),s_table)
            attachments[id] = {'html': render_table(doc['version'][0]['data']) }
        elif typ == 'public.url':
            # there is a preview image somewhere too, but not sure I care
            attachments[id] = {'html': E('a',{'href':url},title or url)}
        elif fname:
            fn = os.path.join('Media',id2,fname)
            if typ in ['public.tiff','public.jpeg','public.png']:
                attachments[id] = {'html': E('img',{'src':fn})}
            else:
                attachments[id] = {'html': E('a',{'href':fn},fname)}
            src = os.path.join(root,fn)
            if os.path.exists(src):
                write(open(src,'rb').read(),dest,fn)
        else:
            fn = os.path.join('FallbackImages',id+'.jpg')
            src = os.path.join(root,fn)
            if os.path.exists(src):
                attachments[id] = {'html': E('img',{'src':fn})}
                write(open(src,'rb').read(),dest,fn)
            
    nquery = '''select a.zidentifier, a.ztitle1, n.zdata from zicnotedata n join ziccloudsyncingobject a on a.znotedata = n.z_pk 
        where n.zcryptotag is null and zdata is not null'''

    seen = set()
    count = 0
    for id,title,data in db.execute(nquery):
        pb = decompress(data,47)
        doc = parse(pb,s_doc)['version'][0]['data']
        section = render_html(doc,attachments)
        section.tag = 'section'
        hdoc = E('html',E('head',E('style',css)),E('body',section))
        fn = id
        if use_title and title:
            tmp = title.replace(':','_').replace('/','_')[:64]
            fn = tmp
            ix = 1
            while fn in seen:
                fn = tmp + '_' + str(ix)
                ix += 1
            seen.add(fn)
        html = ET.tostring(hdoc,method='html')
        try:
            write(html,dest,f'{fn}.html')
        except:
            print(f'write to {fn}.html failed, trying {id}.html')
            write(html,dest,f'{fn}.html')
        count += 1

    print(f"wrote {count} documents to {dest}")


================================================
FILE: notes2quiver
================================================
#!/usr/bin/env python3
import os, sqlite3, json, struct, re, zipfile, sys
from zlib import decompress
import xml.etree.ElementTree as ET

# protobuf parser

def uvarint(data,pos):
    x = s = 0
    while True:
        b = data[pos]
        pos += 1
        x = x | ((b&0x7f)<<s)
        if b < 0x80: return x,pos
        s += 7

def readbytes(data,pos):
    l,pos = uvarint(data,pos)
    return data[pos:pos+l], pos+l

def readstruct(fmt,l):
    return lambda data,pos: (struct.unpack_from(fmt,data,pos)[0],pos+l)

readers = [ uvarint, readstruct('<d',8), readbytes, None, None, readstruct('<f',4) ]

def parse(data, schema):
    "parses a protobuf"
    obj = {}
    pos = 0
    while pos < len(data):
        val,pos = uvarint(data,pos)
        typ = val & 7
        key = val >> 3
        val, pos = readers[typ](data,pos)
        if key not in schema: 
            continue
        name, repeated, typ = schema[key]
        if isinstance(typ, dict):
            val = parse(val, typ)
        if typ == 'string':
            val = val.decode('utf8')
        if repeated:
            val = obj.get(name,[]) + [val]
        obj[name] = val
    return obj

# HTML construction utils

def append(rval,a):
    "append a to rval and return a"
    if isinstance(a,str):
        i = len(rval)-1
        if i<0:
            rval.text = (rval.text or "")+a
        else:
            rval[i].tail = (rval[i].tail or "")+a
    elif isinstance(a,ET.Element):
        rval.append(a)
    elif isinstance(a,dict):
        rval.attrib.update(a)
    else:
        raise Exception(f"unhandled type {type(a)}")
    return a

def E(tag,*args,**attrs):
    tag,*cc = tag.split('.')
    rval = ET.Element(tag)
    tail = None
    if cc: rval.set('class',' '.join(cc))
    if attrs:
        append(rval,attrs)
    for a in args:
        append(rval,a)
    return rval

# Util for processing CRArchive

def process_archive(table):
    "Decode a 'CRArchive' (for tables)"
    objects = []

    def dodict(v):
        return {coerce(e['key']):coerce(e['value']) for e in v.get('element',[])}

    def coerce(o):
        [(k,v)] = o.items()
        if 'custom' == k:
            rval = dict((table['keyItem'][e['key']],coerce(e['value'])) for e in v['mapEntry'])
            typ = table['typeItem'][v['type']]
            if typ == 'com.apple.CRDT.NSUUID':
                return table['uuidItem'][rval['UUIDIndex']]
            if typ == 'com.apple.CRDT.NSString':
                return rval['self']
            return rval
        if k == 'objectIndex':
            return coerce(table['object'][v])
        if k == 'registerLatest':
            return coerce(v['contents'])
        if k == 'orderedSet':
            elements = dodict(v['elements'])
            contents = dodict(v['ordering']['contents'])
            rval = []
            for a in v['ordering']['array']['attachments']:
                value = contents[a['uuid']]
                if value not in rval and a['uuid'] in elements:
                    rval.append(value)
            return rval
        if k == 'dictionary':
            return dodict(v)
        if k in ('stringValue','unsignedIntegerValue','string'):
            return v
        raise Exception(f"unhandled type {k}")

    return coerce(table['object'][0])

# HTML

def render_html(note,get_attach=lambda x:None):
    if note is None:
        return ""
    "Convert note attributed string to HTML"
    # TODO
    # - attachments
    styles = {0:'h1',1:'h2',4:'pre',100:'li',101:'li',102:'li',103:'li'}
    rval = E('div')
    txt = note['string']
    pos = 0
    par = None
    for run in note.get('attributeRun',[]):
        l = run['length']
        for frag in re.findall(r'\n|[^\n]+',txt[pos:pos+l]):
            if par is None: # start paragraph
                pstyle = run.get('paragraphStyle',{}).get('style',-1)
                indent = run.get('paragraphStyle',{}).get('indent',0)
                if pstyle >= 100: # this mess handles merging todo lists
                    tag = ['ul','ul','ol','ul'][pstyle - 100]
                    par = rval
                    while indent > 0:
                        last = par[-1]
                        if last.tag != tag:
                            break
                        par = last
                        indent -= 1
                    while indent >= 0:
                        par = append(par,E(tag))
                        indent -= 1
                    par = append(par,E('li'))
                elif pstyle == 4 and rval[-1].tag == 'pre':
                    par = rval[-1]
                    append(par,"\n")
                else:
                    par = append(rval,E(styles.get(pstyle,'p')))
                if pstyle == 103:
                    par.append(E('input',{"type":"checkbox"}))
                    if run.get('paragraphStyle',{}).get('todo',{}).get('done'):
                        par[0].set('checked','')
            if frag == '\n':
                par = None
            else:
                link = run.get('link')
                info = run.get('attachmentInfo')
                style = run.get('fontHints',0) + 4*run.get('underline',0) + 8*run.get('strikethrough',0)
                if style & 1: frag = E('b',frag)
                if style & 2: frag = E('em',frag)
                if style & 4: frag = E('u',frag)
                if style & 8: frag = E('strike',frag)
                if info:
                    attach = get_attach(info.get('attachmentIdentifier'))
                    if attach is not None:
                        frag = attach
                if link:
                    frag = E('a',frag,href=link)

                append(par,frag)
        pos += l
    return rval

def render_table_html(table):
    "Render a table to html"
    table = process_archive(table)
    rval = E('table')
    for row in table['crRows']:
        tr = E('tr')
        rval.append(tr)
        for col in table['crColumns']:
            cell = table.get('cellColumns').get(col,{}).get(row)
            td = E('td',render_html(cell))
            rval.append(td)
    return rval


# protobuf schema

s_string = {
    2: [ "string", 0, "string"],
    5: [ "attributeRun", 1, {
        1: ["length",0,0],
        2: ["paragraphStyle", 0, {
            1: ["style", 0,0],
            4: ["indent",0,0],
            5: ["todo",0,{ 
                1: ["todoUUID", 0, "bytes"],
                2: ["done",0,0]
            }]
        }],
        5: ["fontHints",0,0],
        6: ["underline",0,0],
        7: ["strikethrough",0,0],
        9: ["link",0,"string"],
        12: [ "attachmentInfo", 0, {
            1: [ "attachmentIdentifier", 0, "string"],
            2: [ "typeUTI", 0, "string"]
        }]
    }]
}

s_doc = { 2: ["version", 1, { 3: ["data", 0, s_string ]}]}

# this essentially is a variant type
s_oid = { 2:["unsignedIntegerValue",0,0], 4:["stringValue",0,'string'], 6:["objectIndex",0,0] }
s_dictionary = {1:["element",1,{ 1:["key",0,s_oid], 2:["value",0,s_oid]}]}
s_table = { 2: ["version", 1, { 3: ["data", 0, {
    3: ["object",1,{
        1:["registerLatest",0,{2:["contents",0,s_oid]}],
        6:["dictionary",0,s_dictionary],
        10:["string",0,s_string],
        13:["custom",0,{
            1:["type",0,0],
            3:["mapEntry",1,{
                1:["key",0,0],
                2:["value",0,s_oid]
            }]
        }],
        16:["orderedSet",0,{
            1: ["ordering",0, {
                1:["array",0,{
                    1:["contents",0,s_string],
                    2:["attachments",1,{1:["index",0,0],2:["uuid",0,0]}]
                }],
                2:["contents",0,s_dictionary]
            }],
            2: ["elements",0,s_dictionary]
        }]
    }],
    4:["keyItem",1,"string"],
    5:["typeItem",1,"string"],
    6:["uuidItem",1,"bytes"]
}]}]}


if __name__ == '__main__':
    import uuid
    from hashlib import md5

    def write(data,*path):
        path = os.path.join(*path)
        os.makedirs(os.path.dirname(path),exist_ok=True)
        open(path,'wb').write(data)

    def writej(data,*path):
        write(json.dumps(data,indent=True).encode('utf8'),*path)

    dest = 'notes.qvnotebook'
    if len(sys.argv)>1:
        dest = sys.argv[1]

    root = os.path.expanduser("~/Library/Group Containers/group.com.apple.notes")
    dbpath = os.path.join(root,'NoteStore.sqlite')
    db = sqlite3.Connection(dbpath)

    fn = os.path.join(dest,'meta.json')
    if not os.path.exists(fn):
        import uuid
        writej({'name': 'Notes', 'uuid': str(uuid.uuid4())},fn)

    # process attachments first
    attachments = {}
    mquery = '''select a.zmergeabledata, a.ztypeuti, b.zidentifier, b.zfilename, a.zurlstring,a.ztitle
                  from ziccloudsyncingobject a left join ziccloudsyncingobject b on a.zmedia = b.z_pk
                 where a.zcryptotag is null and a.ztypeuti is not null and a.zidentifier = ?'''

    nquery = '''select a.zidentifier, a.ztitle1, a.zcreationdate1, a.zmodificationdate1, n.zdata 
                  from zicnotedata n 
                  join ziccloudsyncingobject a on a.znotedata = n.z_pk 
                 where n.zcryptotag is null and zdata is not null and zmarkedfordeletion is not 1'''

    # For each note
    for id,title,create,modify,data in db.execute(nquery):
        dn = id+'.qvnote'
        
        def get_attach(id):
            "Find attachment via db / filesystem, copy into note and return html to reference it"
            row = db.execute(mquery,(id,)).fetchone()
            if not row:
                print("Missed attachment",id)
                return ""

            data, typ, id2, fname, url, title = row
            if typ == 'com.apple.notes.table' and data:
                doc = parse(decompress(data,47),s_table)
                return render_table_html(doc['version'][0]['data'])
            elif typ == 'public.url':
                # there is a preview image somewhere too, but not sure I care
                return E("a",title or url,href=url)
            elif fname:
                fn = os.path.join('Media',id2,fname)
            else:
                fn = os.path.join('FallbackImages',id+'.jpg')

            src = os.path.join(root,fn)
            if os.path.exists(src):
                data = open(src,'rb').read()
                hc = md5(data).hexdigest().upper()
                _,ext = os.path.splitext(src)
                fn2 = hc+ext
                write(data, dest, dn, 'resources', fn2)
                if ext in ['.jpg','.jpeg','.png','.tiff']:
                    return E('img', src=f'quiver-image-url/{fn2}',alt=fn)
                else:
                    return E('a',fn,href=f'quiver-file-url/{fn2}')
            print("fail",id,typ)
            return E('span')

        pb = decompress(data,47)
        doc = parse(pb,s_doc)['version'][0]['data']
        section = render_html(doc,get_attach)
        section = ET.tostring(section,method="html").decode('utf8')

        unix_ts = 0
        content = {'title': title, 'cells': [{ 'type': 'text', 'data': section }]}
        meta = {'uuid':id, 'created_at': int(create)+978307200, 'tags': [], 'title': title, 'updated_at': int(modify)+978307200}
        writej(content, dest, dn, 'content.json')
        writej(meta, dest, dn, 'meta.json')

    print(f"wrote files to {dest}")