Repository: dunhamsteve/notesutils Branch: master Commit: 582e569c9d0a Files: 6 Total size: 44.7 KB Directory structure: gitextract_cr0hplzn/ ├── LICENSE ├── README.md ├── notes.md ├── notes2bear ├── notes2html └── notes2quiver ================================================ FILE CONTENTS ================================================ ================================================ FILE: LICENSE ================================================ This is free and unencumbered software released into the public domain. Anyone is free to copy, modify, publish, use, compile, sell, or distribute this software, either in source code form or as a compiled binary, for any purpose, commercial or non-commercial, and by any means. In jurisdictions that recognize copyright laws, the author or authors of this software dedicate any and all copyright interest in the software to the public domain. We make this dedication for the benefit of the public at large and to the detriment of our heirs and successors. We intend this dedication to be an overt act of relinquishment in perpetuity of all present and future rights to this software under copyright law. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. For more information, please refer to ================================================ FILE: README.md ================================================ # Apple Notes Export Tools This repository includes a few python export tools for Apple's "Notes.app". The scripts require python3, but I tried to make the scripts self-contained and concise, with no additional dependencies. I may revisit this and break the common code into a library in the future. This repository is in the public domain. ## `notes.md` Description of how notes data is stored. ## `notes2bear` Writes a `notes.bearbk` file in Bear's backup format (a zip file with their own markup flavor). It doesn't handle tables because Bear doesn't do tables yet. This shaves about 100 lines from the script. ## `notes2html` `notes2html` takes an destination directory and writes a tree of html files and images. One html file per note and any associated media in the media directory. If you specify `--title` the files with be named with the title guessed by Apple (with / replaced by _). If you specify `--svg`, any drawings will be rendered as inline SVG; otherwise, the fallback jpg files provided by Apple will be used. Usage is: ``` notes2html [--svg] [--title] dest ``` ================================================ FILE: notes.md ================================================ # Notes on Notes.app For future reference and to aid anyone else who might want to extract data from the "Notes" app. I compiled this out of curiosity and a desire to backup my notes content. This document only covers the current format of notes, as synced with iCloud. The application also supports IMAP synced notes with reduced functionality. They are stored in a separate database (and on the IMAP server). You're on your own there, but it's pretty much just MIME and HTML. ## OSX Files The notes are stored in `~/Library/Group Containers/group.com.apple.notes` in a sqlite database named `NoteStore.sqlite`. The database contains sync state from iCloud. It's a CoreData store, but I approached it from plain sqlite. We're interested in the `ZICCLOUDSYNCINGOBJECT` table and the `ZICNOTEDATA` table. The former contains the sync state of each iCloud object - notes, attachments, and folders, and the latter contains the note data. Everything has a UUID identifier `ZIDENTIFIER`. The `ZICCLOUDSYNCINGOBJECT` table column `ZNOTEDATA` points to the `Z_PK` column in `ZICNOTEDATA`. The meat of the data is in zlib compressed protobuf blobs. For documents, it's `ZDATA` in `ZNOTEDATA` and for tables/drawings, it's `ZMERGEABLEDATA` in `ZICCLOUDSYNCINGOBJECT`. Apple uses a [CvRDT](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) for syncing (evidenced by various strings appearing in the data and analysis of what happens as you edit). I suspect the one used for documents, which they call "topotext" is not actually a CvRDT (they seem to order conflicts on a first to sync basis rather than vector clock) but I could be mistaken. It doesn't matter since everything goes through iCloud, which serializes the merges. ## Protobuf Data **Document Wrapper** Everything is wrapped with a versioned document object: message Document { repeated Version version = 2; } message Version { optional bytes data = 3; } There are additional fields that aren't relevant to us, and I've never seen more than one `version`. The content of the `data` field is also protobuf but varies depending on whether we're looking at a note, table, or drawing. **Notes** The protobuf data for a note is a `String` as described below. Assume everything is optional and I’ve elided the CRDT stuff. (Repeated field 3 of String is a sequence clock, length, attribute clock, tombstone, and children. It forms a DAG. For chunks of length >1 the clock is implicitly incremented for each character.) message String { string string = 2; // these are in order, disjoint, and their length sums to the length of string repeated AttributeRun attributeRun = 5; } message AttributeRun { uint32 length = 1; ParagraphStyle paragraphStyle = 2; Font font = 3; uint32 fontHints = 5; // 1:bold, 2:italic, 3:bold italic uint32 underline = 6; uint32 strikethrough = 7; int32 superscript = 8; // sign indicates super/subscript string link = 9; Color color = 10; AttachmentInfo attachmentInfo = 12; } message ParagraphStyle { // 0:title, 1:heading, 4:monospace, 100:dotitem, 101:dashitem, 102:numitem, // 103:todoitem uint32 style = 1; uint32 alignment = 2; // 0:left, 1:center, 2:right, 3:justified int32 indent = 4; Todo todo = 5; } message Font { string name = 1; float pointSize = 2; uint32 fontHints = 3; } message AttachmentInfo { string attachmentIdentifier = 1; string typeUTI = 2; } message Todo { bytes todoUUID = 1; bool done = 2; } message Color { float red = 1; float green = 2; float blue = 3; float alpha = 4; } **Drawings** This info isn’t strictly necessary. For each drawing, you’ll find a rendering in `FallbackImages/UUID.jpg`. I was curious whether I could recover / backup the original vector data, so I came up with the following. (The root object here is `Drawing`.) message Drawing { int64 serializationVersion = 1; repeated bytes replicaUUIDs = 2; repeated StrokeID versionVector = 3; repeated Ink inks = 4; repeated Stroke strokes = 5; int64 orientation = 6; StrokeID orientationVersion = 7; Rectangle bounds = 8; bytes uuid = 9; } message Color { float red = 1; float green = 2; float blue = 3; float alpha = 4; } message Rectangle { float height = 4; float originX = 1; float originY = 2; float width = 3; } message Transform { float a = 1; float b = 2; float c = 3; float d = 4; float tx = 5; float ty = 6; } message Ink { Color color = 1; string identifier = 2; int64 version = 3; } message Stroke { int64 inkIndex = 3; int64 pointsCount = 4; bytes points = 5; Rectangle bounds = 6; bool hidden = 9; double timestamp = 11; bool createdWithFinger = 12; Transform transform = 10; } The byte array `points` is compactly encoded list of points. It is a sequence of this struct: struct PKCompressedStrokePoint { float timestamp; // timestamp (delta from somethin, probably previous) float xpos; float ypos; unsigned short radius; // radius*10 unsigned short aspectRatio; // aspectRatio*1000 unsigned short edgeWidth; // edgeWidth*10 unsigned short force; // force*1000 unsigned short azimuth; // azimuth*10430.2191955274 unsigned char altitude; // altitude*162.338041953733 unsigned char opacity; // opacity*255 }; The `inkIndex` field points into the array `inks`. The `identifier` in an `Ink` includes stuff like `com.apple.ink.marker`. I kinda fudged this in my svg generation - apple’s rendering code is much more sophisticated, taking azimuth/altitude into account. My code works well enough for pen, but falls short on marker. You may be better off with the jpeg. **Tables** This one is complicated, and I think a bit of explanation is in order to explain why it’s structured this way. They are trying to model a table with multiple people editing at the same time. Editing the contents of a cell is essentially a solved problem (it’s complicated, but it’s solved above - each cell is its own "document” - a `String` object from above). But in addition to this, people are adding, removing, and reordering columns. You want to ensure that if two people move a column or one person adds a column and another adds a row, things end up in a sane state, no matter which order you see the operations. To do this, we consider the rows to be an ordered set of uuids. (And the same for the columns.) Then you have a map of column guid → row guid → String object. The data itself a pile of CRDTs encoded with something like NSKeyedArchiver, but built on top of protobuf for these CRDTs. The root object contains a few tables, and a list of objects (like NSKeyedArchiver), and reverenced by index via a variant time that apple calls `ObjectID`. (I managed to figure this out by generically decoding the protobuf data and looking at it, but later found they ship an older revision of the proto files to their web app.) This is the variant type used below: message ObjectID { uint64 unsignedIntegerValue = 2; string stringValue = 4; uint32 objectIndex = 6; } The `objectIndex` is a index into the list of `object` in `Document`. The root `Document` is: message Document { repeated DocObject object = 3; repeated string keyItem = 4; repeated string typeItem = 5; repeated bytes uuidItem = 6; } message DocObject { RegisterLatest registerLatest = 1; Dictionary dictionary = 6; String string = 10; // this is our fancy String above CustomObject custom = 13; OrderedSet orderedSet = 16; } The first object in the `object` field is the root object. A `CustomObject` is essentially a key/value map with a type. The keys are indexed from `keyItem` and type is from `typeItem`. message CustomObject { int32 type = 1; // index into "typeItem" below message MapEntry { required int32 key = 1; // index into keyItem below required ObjectID value = 2; } repeated MapEntry mapEntry = 3; } For a UUID, the type is `com.apple.CRDT.NSUUID` and there is a `UUIDIndex` field whose value is the index of the UUID in `uuidItem` of the `Document`. A `RegisterLatest` is just a CRDT for a value. There is a clock, not shown here, which helps with merging conflicts. It’s last write wins. It only appears to be used to point at an NSString custom object holding “CRTableColumnDirectionLeftToRight”. I’m ignoring this at the moment. message RegisterLatest { ObjectID contents = 2; } A `Dictionary` object holds object id for both key and value. In practice the keys points to a UUID `CustomObject` and the value is either a UUID or another dictionary. (This is a last-write-wins CRDT, I’m leaving out the clock values.) message Dictionary { message Element { ObjectID key = 1; ObjectID value = 2; } repeated Element element = 1; } Which leaves us with `OrderedSet`. An ordered set leverages `String` to provide an vector of UUIDs via `TTArray`. Pairs of string position to UUID are stored in `attachments` of `TTArray`. Surrounding that is `Array` which has a CRDT Dictionary to map the uuid of the TTArray to the content in that position of the array (which happens to also be a UUID in this case). And surrounding that is `OrderedSet` which contains another `Dictionary` of UUID to UUID, but the keys and values are the same (uuids from the TTArray space). This seems to be used to filter out deleted items in the case where you simultaneously move and delete a column. (The move does delete + add and the delete does a delete, so a copy remains in the Array.) Two conflicting moves will also create duplicates in the array. Apple appears to handle this by ignoring all but the first instance. (And cleaning it up on the next move of that column.) message OrderedSet { Array ordering = 1; Dictionary elements = 2; // set of elements that haven't been deleted } message Array { TTArray array = 1; // TTArray Dictionary contents = 2; // map of TTArray uuid to content uuid } message TTArray { String contents = 1; // we don't actually reference this. ArrayAttachment attachments = 2; // list of (position -> uuid } message ArrayAttachment { int64 index = 1; bytes uuid = 2; } **Decoding Tables** Ok, so to decode a table, the root object will be a CustomObject with the fields: | Field | Value | | ----------- | ----------------------------------------------------------- | | crRows | OrderedSet for row uuids | | crColumns | OrderedSet for column uuids | | cellColumns | Dictionary of column uuid → Dictionary of row uuid → String | The `value` of these fields are referenced by `objectIndex`. Both `crRows` and `crColumns` are a `CustomObject` of type `OrderedSet`. To get the list of uuids for these ordered sets, take each `ordering.array.attachments.uuid`, filter out values that don’t appear as keys in `elements` , and look up each of the resulting values in the dictionary `ordering.contents`. Then iterate through the column uuids and look them up in cellColumns. The result, for each column, will be a dictionary. In this dictionary, look up each row uuid. The result, if present, will be a String object (or rather an ObjectId pointing to a String object). This is the content of the table cell. ## TODO I still need to write up the attachment, folder, and encryption stuff. ================================================ FILE: notes2bear ================================================ #!/usr/bin/env python3 import zlib, os, sqlite3, re, zipfile, json from struct import unpack_from from datetime import datetime # Simple script to export Notes.app to Bear.app backup format # I have code to decode the tables and drawings (to svg), but not necessary for Bear def uvarint(data,pos): x = s = 0 while True: b = data[pos] pos += 1 x = x | ((b&0x7f)<> 3 val, pos = readers[typ](data,pos) if key not in schema: continue name, repeated, typ = schema[key] if isinstance(typ, dict): val = parse(val, typ) if typ == 'string': val = val.decode('utf8') if repeated: val = obj.get(name,[]) + [val] obj[name] = val return obj def translate(data, media): styles = {0: '# ', 1: '## ',100: '* ', 101: '* ', 102: '1. ', 103: '- '} rval = [] refs = [] txt = data['string'] pos = 0 acc = None pre = False for run in data['attributeRun']: l = run['length'] for frag in re.findall(r'\n|[^\n]+',txt[pos:pos+l]): if acc is None: # start paragraph pstyle = run.get('paragraphStyle',{}).get('style') indent = run.get('paragraphStyle',{}).get('indent',0) acc = " "*indent+styles.get(pstyle,"") if pstyle == 103 and run['paragraphStyle']['todo']['done']: acc = " "*indent+"+ " if pstyle == 4: if not pre: rval.append("```") elif pre: rval.append("```") pre = pstyle == 4 if frag == '\n': # end paragraph rval.append(acc) acc = None else: # accumulate and handle inline styles - although bear doesn't seem to support nested ones. link = run.get('link') info = run.get('attachmentInfo') style = run.get('fontHints',0) + 4*run.get('underline',0) + 8*run.get('strikethrough',0) if style & 1: frag = f'*{frag}*' if style & 2: frag = f'/{frag}/' if style & 4: frag = f'_{frag}_' if style & 8: frag = f'~{frag}~' if link: frag = f'[{frag}]({link})' if info: id = info.get('attachmentIdentifier') fn = media.get(id) if fn: _,e = os.path.splitext(fn) acc += f'[assets/{id}{e}]' refs.append(id) else: acc += f"ATTACH {info}" else: acc += frag pos += l if acc: rval.append(acc) rval = '\n'.join(rval)+"\n" return rval,refs # The schema subset needed for bear export docschema = { 2: [ "version", 1, { 3: [ "data", 0, { 2: [ "string", 0, "string"], 5: [ "attributeRun", 1, { 1: ["length",0,0], 2: ["paragraphStyle", 0, { 1: ["style", 0,0], 4: ["indent",0,0], 5: ["todo",0,{ 1: ["todoUUID", 0, "bytes"], 2: ["done",0,0] }] }], 5: ["fontHints",0,0], 6: ["underline",0,0], 7: ["strikethrough",0,0], 9: [ "link", 0, "string" ], 12: [ "attachmentInfo", 0, { 1: [ "attachmentIdentifier", 0, "string"], 2: [ "typeUTI", 0, "string"] }] }] }] }] } if __name__ == '__main__': root = os.path.expanduser("~/Library/Group Containers/group.com.apple.notes") db = sqlite3.Connection(os.path.join(root,'NoteStore.sqlite')) media = {} # there is some indirection for attachments for a,b,fn in db.execute('select a.zidentifier, b.zidentifier, b.zfilename from ziccloudsyncingobject a left join ziccloudsyncingobject b on a.zmedia = b.z_pk'): if fn: full = os.path.join(root,'Media',b,fn) else: full = os.path.join(root,'FallbackImages',a+".jpg") if os.path.exists(full): media[a] = full count = 0 with zipfile.ZipFile('notes.bearbk','w') as zip: for id, title, data, cdate, mdate in db.execute('select o.zidentifier, ztitle1, zdata, o.zcreationdate1, o.zmodificationdate1 from zicnotedata join ziccloudsyncingobject o on znotedata = zicnotedata.z_pk where zicnotedata.zcryptotag is null'): ze = f'notes.bearbk/{id}.textbundle' if data: pb = zlib.decompress(data, 47) doc = parse(pb, docschema)['version'][0]['data'] if not doc['string']: continue # some are blank text,refs = translate(doc, media) info = { "type":"public.plain-text", "creatorIdentifier": "net.shinyfrog.bear", "net.shinyfrog.bear": { "modificationDate": datetime.fromtimestamp(int(mdate)+978307200).isoformat(), "creationDate": datetime.fromtimestamp(int(cdate)+978307200).isoformat() }, "version":2 } zip.writestr(f'{ze}/info.json',json.dumps(info)) zip.writestr(f'{ze}/text.txt',text) for ref in refs: _,e = os.path.splitext(media[ref]) fn = f'{ze}/assets/{ref}{e}' zip.write(media[ref],fn) count += 1 print(f"wrote {count} notes to notes.bearbk") ================================================ FILE: notes2html ================================================ #!/usr/bin/env python3 import os, sqlite3, json, struct, re, zipfile, sys from zlib import decompress import xml.etree.ElementTree as ET # HTML construction utils def append(rval,a): "append a to rval and return a" if isinstance(a,str): i = len(rval)-1 if i<0: rval.text = (rval.text or "")+a else: rval[i].tail = (rval[i].tail or "")+a elif isinstance(a,ET.Element): rval.append(a) elif isinstance(a,dict): rval.attrib.update(a) else: raise Exception(f"unhandled type {type(a)}") return a def E(tag,*args): tag,*cc = tag.split('.') rval = ET.Element(tag) tail = None if cc: rval.set('class',' '.join(cc)) for a in args: append(rval,a) return rval # protobuf parser def uvarint(data,pos): x = s = 0 while True: b = data[pos] pos += 1 x = x | ((b&0x7f)<> 3 val, pos = readers[typ](data,pos) if key not in schema: continue name, repeated, typ = schema[key] if isinstance(typ, dict): val = parse(val, typ) if typ == 'string': val = val.decode('utf8') if repeated: val = obj.get(name,[]) + [val] obj[name] = val return obj def svg(drawing): "Convert note drawing to SVG" width = drawing['bounds']['width'] height = drawing['bounds']['height'] rval = E('svg',{'width':str(width),'height':str(height)}) inks = drawing.get('inks') for stroke in drawing.get('strokes',[]): if stroke.get('hidden'): continue if 'points' in stroke: swidth=1 ink = inks[stroke['inkIndex']] c = ink['color'] red = int(c['red']*255) green = int(c['green']*255) blue = int(c['blue']*255) alpha = c['alpha'] if ink['identifier'] == 'com.apple.ink.marker': swidth = 15 alpha = 0.5 color = f'rgba({red},{green},{blue},{alpha})' path = '' for _,x,y,*rest in struct.iter_unpack('<3f5H2B',stroke['points']): path += f"L{x:.2f} {y:.2f}" path = "M"+path[1:] rval.append(E('path',{'d':"M"+path[1:],'stroke':color,'stroke-width':str(swidth),'stroke-cap':'round','fill':'none'})) if 'transform' in stroke: rval[-1].set('transform',"matrix({a} {b} {c} {d} {tx:.2f} {ty:.2f})".format(**stroke['transform'])) return rval def render_html(note,attachments={}): if note is None: return "" "Convert note attributed string to HTML" # TODO # - attachments styles = {0:'h1',1:'h2',4:'pre',100:'li',101:'li',102:'li',103:'li'} rval = E('div') txt = note['string'] pos = 0 par = None for run in note.get('attributeRun',[]): l = run['length'] for frag in re.findall(r'\n|[^\n]+',txt[pos:pos+l]): if par is None: # start paragraph pstyle = run.get('paragraphStyle',{}).get('style',-1) indent = run.get('paragraphStyle',{}).get('indent',0) if pstyle > 100: # this mess handles merging bulleted lists tag = ['ul','ul','ol','ul'][pstyle - 100] par = rval while indent > 0: last = par[-1] if last.tag != tag: break par = last indent -= 1 while indent >= 0: par = append(par,E(tag)) indent -= 1 par = append(par,E('li')) elif pstyle == 4 and rval[-1].tag == 'pre': par = rval[-1] append(par,"\n") else: par = append(rval,E(styles.get(pstyle,'p'))) if pstyle == 103: par.append(E('input',{"type":"checkbox"})) if run.get('paragraphStyle',{}).get('todo',{}).get('done'): par[0].set('checked','') if frag == '\n': par = None else: link = run.get('link') info = run.get('attachmentInfo') style = run.get('fontHints',0) + 4*run.get('underline',0) + 8*run.get('strikethrough',0) if style & 1: frag = E('b',frag) if style & 2: frag = E('em',frag) if style & 4: frag = E('u',frag) if style & 8: frag = E('strike',frag) if info: attach = attachments.get(info.get('attachmentIdentifier')) if attach and attach.get('html'): frag = attach.get('html') append(par,frag) pos += l return rval def process_archive(table): "Decode a 'CRArchive'" objects = [] def dodict(v): rval = {} for e in v.get('element',[]): rval[coerce(e['key'])] = coerce(e['value']) return rval def coerce(o): [(k,v)]= o.items() if 'custom' == k: rval = dict((table['keyItem'][e['key']],coerce(e['value'])) for e in v['mapEntry']) typ = table['typeItem'][v['type']] if typ == 'com.apple.CRDT.NSUUID': return table['uuidItem'][rval['UUIDIndex']] if typ == 'com.apple.CRDT.NSString': return rval['self'] return rval if k == 'objectIndex': return coerce(table['object'][v]) if k == 'registerLatest': return coerce(v['contents']) if k == 'orderedSet': elements = dodict(v['elements']) contents = dodict(v['ordering']['contents']) rval = [] for a in v['ordering']['array']['attachments']: value = contents[a['uuid']] if value not in rval and a['uuid'] in elements: rval.append(value) return rval if k == 'dictionary': return dodict(v) if k in ('stringValue','unsignedIntegerValue','string'): return v raise Exception(f"unhandled type {k}") return coerce(table['object'][0]) def render_table(table): "Render a table to html" table = process_archive(table) rval = E('table') for row in table['crRows']: tr = E('tr') rval.append(tr) for col in table['crColumns']: cell = table.get('cellColumns').get(col,{}).get(row) td = E('td',render_html(cell)) rval.append(td) return rval s_string = { 2: [ "string", 0, "string"], 5: [ "attributeRun", 1, { 1: ["length",0,0], 2: ["paragraphStyle", 0, { 1: ["style", 0,0], 4: ["indent",0,0], 5: ["todo",0,{ 1: ["todoUUID", 0, "bytes"], 2: ["done",0,0] }] }], 5: ["fontHints",0,0], 6: ["underline",0,0], 7: ["strikethrough",0,0], 9: ["link",0,"string"], 12: [ "attachmentInfo", 0, { 1: [ "attachmentIdentifier", 0, "string"], 2: [ "typeUTI", 0, "string"] }] }] } s_doc = { 2: ["version", 1, { 3: ["data", 0, s_string ]}]} s_drawing = { 2: ["version", 1, { 3: ["data", 0, { 4: ["inks",1, { 1:["color",0,{1:["red",0,0],2:["green",0,0],3:["blue",0,0],4:["alpha",0,0]}], 2:["identifier",0,"string"] }], 5: ["strokes",1, { 3:["inkIndex",0,0], 5:["points",0,"bytes"], 9:["hidden",0,0], 10: ["transform",0,{1:["a",0,0],2:["b",0,0],3:["c",0,0],4:["d",0,0],5:["tx",0,0],6:["ty",0,0]}] }], 8: ["bounds", 0, {1:["originX",0,0],2:["originY",0,0],3:["width",0,0],4:["height",0,0]}] }] } ]} # this essentially is a variant type s_oid = { 2:["unsignedIntegerValue",0,0], 4:["stringValue",0,'string'], 6:["objectIndex",0,0] } s_dictionary = {1:["element",1,{ 1:["key",0,s_oid], 2:["value",0,s_oid]}]} s_table = { 2: ["version", 1, { 3: ["data", 0, { 3: ["object",1,{ 1:["registerLatest",0,{2:["contents",0,s_oid]}], 6:["dictionary",0,s_dictionary], 10:["string",0,s_string], 13:["custom",0,{ 1:["type",0,0], 3:["mapEntry",1,{ 1:["key",0,0], 2:["value",0,s_oid] }] }], 16:["orderedSet",0,{ 1: ["ordering",0, { 1:["array",0,{ 1:["contents",0,s_string], 2:["attachments",1,{1:["index",0,0],2:["uuid",0,0]}] }], 2:["contents",0,s_dictionary] }], 2: ["elements",0,s_dictionary] }] }], 4:["keyItem",1,"string"], 5:["typeItem",1,"string"], 6:["uuidItem",1,"bytes"] }]}]} def write(data,*path): path = os.path.join(*path) os.makedirs(os.path.dirname(path),exist_ok=True) open(path,'wb').write(data) if __name__ == '__main__': css = ''' .underline { text-decoration: underline; } .strikethrough { text-decoration: line-through; } .todo { list-style-type: none; margin-left: -20px; } .dashitem { list-style-type: none; } .dashitem:before { content: "-"; text-indent: -5px } ''' def help(): print(f'Usage:\n') print(f' {sys.argv[0]} [--svg] [--title] dest') print(f' --svg Use svg for drawings') print(f' --title Use title for filenames') print(f' dest destination directory') print() exit(-1) dest = None use_svg = False use_title = False for x in sys.argv[1:]: if x == '--svg': use_svg = True elif x == '--title': use_title = True elif x.startswith('--'): help() else: dest = x if not dest: help() root = os.path.expanduser("~/Library/Group Containers/group.com.apple.notes") dbpath = os.path.join(root,'NoteStore.sqlite') write(open(dbpath,'rb').read(),dest,'NoteStore.sqlite') db = sqlite3.Connection(dbpath) # process attachments first attachments = {} mquery = '''select a.zidentifier, a.zmergeabledata, a.ztypeuti, b.zidentifier, b.zfilename, a.zurlstring,a.ztitle from ziccloudsyncingobject a left join ziccloudsyncingobject b on a.zmedia = b.z_pk where a.zcryptotag is null and a.ztypeuti is not null''' for id, data, typ, id2, fname, url,title in db.execute(mquery): if typ == 'com.apple.drawing' and data and use_svg: doc = parse(decompress(data,47),s_drawing) attachments[id] = {'html': svg(doc['version'][0]['data'])} elif typ == 'com.apple.notes.table' and data: doc = parse(decompress(data,47),s_table) attachments[id] = {'html': render_table(doc['version'][0]['data']) } elif typ == 'public.url': # there is a preview image somewhere too, but not sure I care attachments[id] = {'html': E('a',{'href':url},title or url)} elif fname: fn = os.path.join('Media',id2,fname) if typ in ['public.tiff','public.jpeg','public.png']: attachments[id] = {'html': E('img',{'src':fn})} else: attachments[id] = {'html': E('a',{'href':fn},fname)} src = os.path.join(root,fn) if os.path.exists(src): write(open(src,'rb').read(),dest,fn) else: fn = os.path.join('FallbackImages',id+'.jpg') src = os.path.join(root,fn) if os.path.exists(src): attachments[id] = {'html': E('img',{'src':fn})} write(open(src,'rb').read(),dest,fn) nquery = '''select a.zidentifier, a.ztitle1, n.zdata from zicnotedata n join ziccloudsyncingobject a on a.znotedata = n.z_pk where n.zcryptotag is null and zdata is not null''' seen = set() count = 0 for id,title,data in db.execute(nquery): pb = decompress(data,47) doc = parse(pb,s_doc)['version'][0]['data'] section = render_html(doc,attachments) section.tag = 'section' hdoc = E('html',E('head',E('style',css)),E('body',section)) fn = id if use_title and title: tmp = title.replace(':','_').replace('/','_')[:64] fn = tmp ix = 1 while fn in seen: fn = tmp + '_' + str(ix) ix += 1 seen.add(fn) html = ET.tostring(hdoc,method='html') try: write(html,dest,f'{fn}.html') except: print(f'write to {fn}.html failed, trying {id}.html') write(html,dest,f'{fn}.html') count += 1 print(f"wrote {count} documents to {dest}") ================================================ FILE: notes2quiver ================================================ #!/usr/bin/env python3 import os, sqlite3, json, struct, re, zipfile, sys from zlib import decompress import xml.etree.ElementTree as ET # protobuf parser def uvarint(data,pos): x = s = 0 while True: b = data[pos] pos += 1 x = x | ((b&0x7f)<> 3 val, pos = readers[typ](data,pos) if key not in schema: continue name, repeated, typ = schema[key] if isinstance(typ, dict): val = parse(val, typ) if typ == 'string': val = val.decode('utf8') if repeated: val = obj.get(name,[]) + [val] obj[name] = val return obj # HTML construction utils def append(rval,a): "append a to rval and return a" if isinstance(a,str): i = len(rval)-1 if i<0: rval.text = (rval.text or "")+a else: rval[i].tail = (rval[i].tail or "")+a elif isinstance(a,ET.Element): rval.append(a) elif isinstance(a,dict): rval.attrib.update(a) else: raise Exception(f"unhandled type {type(a)}") return a def E(tag,*args,**attrs): tag,*cc = tag.split('.') rval = ET.Element(tag) tail = None if cc: rval.set('class',' '.join(cc)) if attrs: append(rval,attrs) for a in args: append(rval,a) return rval # Util for processing CRArchive def process_archive(table): "Decode a 'CRArchive' (for tables)" objects = [] def dodict(v): return {coerce(e['key']):coerce(e['value']) for e in v.get('element',[])} def coerce(o): [(k,v)] = o.items() if 'custom' == k: rval = dict((table['keyItem'][e['key']],coerce(e['value'])) for e in v['mapEntry']) typ = table['typeItem'][v['type']] if typ == 'com.apple.CRDT.NSUUID': return table['uuidItem'][rval['UUIDIndex']] if typ == 'com.apple.CRDT.NSString': return rval['self'] return rval if k == 'objectIndex': return coerce(table['object'][v]) if k == 'registerLatest': return coerce(v['contents']) if k == 'orderedSet': elements = dodict(v['elements']) contents = dodict(v['ordering']['contents']) rval = [] for a in v['ordering']['array']['attachments']: value = contents[a['uuid']] if value not in rval and a['uuid'] in elements: rval.append(value) return rval if k == 'dictionary': return dodict(v) if k in ('stringValue','unsignedIntegerValue','string'): return v raise Exception(f"unhandled type {k}") return coerce(table['object'][0]) # HTML def render_html(note,get_attach=lambda x:None): if note is None: return "" "Convert note attributed string to HTML" # TODO # - attachments styles = {0:'h1',1:'h2',4:'pre',100:'li',101:'li',102:'li',103:'li'} rval = E('div') txt = note['string'] pos = 0 par = None for run in note.get('attributeRun',[]): l = run['length'] for frag in re.findall(r'\n|[^\n]+',txt[pos:pos+l]): if par is None: # start paragraph pstyle = run.get('paragraphStyle',{}).get('style',-1) indent = run.get('paragraphStyle',{}).get('indent',0) if pstyle >= 100: # this mess handles merging todo lists tag = ['ul','ul','ol','ul'][pstyle - 100] par = rval while indent > 0: last = par[-1] if last.tag != tag: break par = last indent -= 1 while indent >= 0: par = append(par,E(tag)) indent -= 1 par = append(par,E('li')) elif pstyle == 4 and rval[-1].tag == 'pre': par = rval[-1] append(par,"\n") else: par = append(rval,E(styles.get(pstyle,'p'))) if pstyle == 103: par.append(E('input',{"type":"checkbox"})) if run.get('paragraphStyle',{}).get('todo',{}).get('done'): par[0].set('checked','') if frag == '\n': par = None else: link = run.get('link') info = run.get('attachmentInfo') style = run.get('fontHints',0) + 4*run.get('underline',0) + 8*run.get('strikethrough',0) if style & 1: frag = E('b',frag) if style & 2: frag = E('em',frag) if style & 4: frag = E('u',frag) if style & 8: frag = E('strike',frag) if info: attach = get_attach(info.get('attachmentIdentifier')) if attach is not None: frag = attach if link: frag = E('a',frag,href=link) append(par,frag) pos += l return rval def render_table_html(table): "Render a table to html" table = process_archive(table) rval = E('table') for row in table['crRows']: tr = E('tr') rval.append(tr) for col in table['crColumns']: cell = table.get('cellColumns').get(col,{}).get(row) td = E('td',render_html(cell)) rval.append(td) return rval # protobuf schema s_string = { 2: [ "string", 0, "string"], 5: [ "attributeRun", 1, { 1: ["length",0,0], 2: ["paragraphStyle", 0, { 1: ["style", 0,0], 4: ["indent",0,0], 5: ["todo",0,{ 1: ["todoUUID", 0, "bytes"], 2: ["done",0,0] }] }], 5: ["fontHints",0,0], 6: ["underline",0,0], 7: ["strikethrough",0,0], 9: ["link",0,"string"], 12: [ "attachmentInfo", 0, { 1: [ "attachmentIdentifier", 0, "string"], 2: [ "typeUTI", 0, "string"] }] }] } s_doc = { 2: ["version", 1, { 3: ["data", 0, s_string ]}]} # this essentially is a variant type s_oid = { 2:["unsignedIntegerValue",0,0], 4:["stringValue",0,'string'], 6:["objectIndex",0,0] } s_dictionary = {1:["element",1,{ 1:["key",0,s_oid], 2:["value",0,s_oid]}]} s_table = { 2: ["version", 1, { 3: ["data", 0, { 3: ["object",1,{ 1:["registerLatest",0,{2:["contents",0,s_oid]}], 6:["dictionary",0,s_dictionary], 10:["string",0,s_string], 13:["custom",0,{ 1:["type",0,0], 3:["mapEntry",1,{ 1:["key",0,0], 2:["value",0,s_oid] }] }], 16:["orderedSet",0,{ 1: ["ordering",0, { 1:["array",0,{ 1:["contents",0,s_string], 2:["attachments",1,{1:["index",0,0],2:["uuid",0,0]}] }], 2:["contents",0,s_dictionary] }], 2: ["elements",0,s_dictionary] }] }], 4:["keyItem",1,"string"], 5:["typeItem",1,"string"], 6:["uuidItem",1,"bytes"] }]}]} if __name__ == '__main__': import uuid from hashlib import md5 def write(data,*path): path = os.path.join(*path) os.makedirs(os.path.dirname(path),exist_ok=True) open(path,'wb').write(data) def writej(data,*path): write(json.dumps(data,indent=True).encode('utf8'),*path) dest = 'notes.qvnotebook' if len(sys.argv)>1: dest = sys.argv[1] root = os.path.expanduser("~/Library/Group Containers/group.com.apple.notes") dbpath = os.path.join(root,'NoteStore.sqlite') db = sqlite3.Connection(dbpath) fn = os.path.join(dest,'meta.json') if not os.path.exists(fn): import uuid writej({'name': 'Notes', 'uuid': str(uuid.uuid4())},fn) # process attachments first attachments = {} mquery = '''select a.zmergeabledata, a.ztypeuti, b.zidentifier, b.zfilename, a.zurlstring,a.ztitle from ziccloudsyncingobject a left join ziccloudsyncingobject b on a.zmedia = b.z_pk where a.zcryptotag is null and a.ztypeuti is not null and a.zidentifier = ?''' nquery = '''select a.zidentifier, a.ztitle1, a.zcreationdate1, a.zmodificationdate1, n.zdata from zicnotedata n join ziccloudsyncingobject a on a.znotedata = n.z_pk where n.zcryptotag is null and zdata is not null and zmarkedfordeletion is not 1''' # For each note for id,title,create,modify,data in db.execute(nquery): dn = id+'.qvnote' def get_attach(id): "Find attachment via db / filesystem, copy into note and return html to reference it" row = db.execute(mquery,(id,)).fetchone() if not row: print("Missed attachment",id) return "" data, typ, id2, fname, url, title = row if typ == 'com.apple.notes.table' and data: doc = parse(decompress(data,47),s_table) return render_table_html(doc['version'][0]['data']) elif typ == 'public.url': # there is a preview image somewhere too, but not sure I care return E("a",title or url,href=url) elif fname: fn = os.path.join('Media',id2,fname) else: fn = os.path.join('FallbackImages',id+'.jpg') src = os.path.join(root,fn) if os.path.exists(src): data = open(src,'rb').read() hc = md5(data).hexdigest().upper() _,ext = os.path.splitext(src) fn2 = hc+ext write(data, dest, dn, 'resources', fn2) if ext in ['.jpg','.jpeg','.png','.tiff']: return E('img', src=f'quiver-image-url/{fn2}',alt=fn) else: return E('a',fn,href=f'quiver-file-url/{fn2}') print("fail",id,typ) return E('span') pb = decompress(data,47) doc = parse(pb,s_doc)['version'][0]['data'] section = render_html(doc,get_attach) section = ET.tostring(section,method="html").decode('utf8') unix_ts = 0 content = {'title': title, 'cells': [{ 'type': 'text', 'data': section }]} meta = {'uuid':id, 'created_at': int(create)+978307200, 'tags': [], 'title': title, 'updated_at': int(modify)+978307200} writej(content, dest, dn, 'content.json') writej(meta, dest, dn, 'meta.json') print(f"wrote files to {dest}")