Repository: andymatuschak/Bear-Markdown-Export Branch: master Commit: 0d2d91888337 Files: 7 Total size: 47.9 KB Directory structure: gitextract_jcda48_b/ ├── .gitattributes ├── .gitignore ├── Bear Import.md ├── LICENSE ├── README.md ├── bear_export_sync.py └── bear_import.py ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitattributes ================================================ # Auto detect text files and perform LF normalization * text=auto ================================================ FILE: .gitignore ================================================ .DS_Store ================================================ FILE: Bear Import.md ================================================ ## Bear Markdown and textbundle import – with tags from file and folder. ***bear_import.py*** *Version 1.0.0 - 2018-02-10 at 17:37 EST* *See also:* **[bear_export_sync.py](https://github.com/rovest/Bear-Markdown-Export/blob/master/README.md)** *for export with sync-back.* ### Features * Imports markdown or textbundles from nested folders under a `BearImport/input/' folder * Foldernames are converted to Bear tags * Also imports MacOS file tags as Bear tags * Imported notes are also tagged with `#.imported/yyyy-MM-dd` for convenience. * Import-files are then cleared to a `BearImport/done/' folder * Use for email input to Bear with Zapier's "Gmail to Dropbox" zap. * Or for import of nested groups and sheets from Ulysses, images and keywords included. ### Trigger script with Automator Folder Action 1. New Automator file as `Folder Action` 2. Set `Folder action receives files and folders added to`: `{user}/Dropbox/BearImport/Input` 3. Add action: `Run Shell Script` choose `bin/bash` 4. Insert one line with full paths to python and script (Use "" if spaces in paths!): `/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 "/Users/username/scripts/bear_import.py"` 5. Save as `Bear Import` or whatever you choose. Or skip all this and run it manually :) ### Get mail to Bear with "Zapier Gmail to Dropbox" action 1. Create a free zapier.com account. 2. Use a dedicated gmail account or setup a filter assigning a label used by zapier. 3. Make a Zapier zap. See: [Add new Gmail emails to Dropbox as text files](https://zapier.com/apps/dropbox/integrations/gmail/10323/add-new-gmail-emails-to-dropbox-as-text-files) 1. Set zap to monitor inbox with label (assigned by filter in step 2.) 2. Set zap Dropbox output to `{user}/Dropbox/BearImport/Input` - Zap will now check for new email (with matching gmail label) every 15 minutes and script above will import to Bear. - Alternately on iOS: use this workflow (import to Bear from same Dropbox folder): [Gmail-DB zap to Bear](https://workflow.is/workflows/827b9b2518d5476ca0158a67d5b492fa) ### Import from Ulysses’ external folders on Mac 1. Add `{user}/Dropbox/BearImport/Input` as external folder 2. Edit folder settings to `.textbundle` and `Inline Links`! 3. Drag any library group to this folder in Ulysses' sidebar. 4. Voilà – Imports to Bear with images and tags (both from group names and keywords). ================================================ FILE: LICENSE ================================================ MIT License Copyright (c) 2018 rovest Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: README.md ================================================ ## Markdown export and sync of Bear notes ***bear_export_sync.py*** *Version 1.4, 2020-01-11* Python script for export and roundtrip sync of Bear's notes to OneDrive, Dropbox, etc. and edit online with [StackEdit](https://stackedit.io/app), or use a markdown editor like *Typora* on Windows or a suitable app on Android. Remote edits and new notes get synced back into Bear with this script. **See also: [Bear Markdown and textbundle import – with tags from file and folder](https://github.com/rovest/Bear-Markdown-Export/blob/master/Bear%20Import.md)** Set up seamless syncing with Ulysses’ external folders on Mac, with images included! Write and add photos in Bear, then reorder, glue, and publish, export, or print with styles in Ulysses— bears and butterflies are best friends ;) (PS. The manual order you set for notes in Ulysses' external folder, is maintained during syncs, unless title is changed.) Suitable for use with https://github.com/andymatuschak/note-link-janitor. BEAR IN MIND! This is a free to use version, and please improve or modify as needed. But do be careful! both `rsync` and `shutil.rmtree` used here, are powerful commands that can wipe clean a whole folder tree or even your complete HD if paths are set incorrectly! Also, be safe, take a fresh backup of both Bear and your Mac before first run. *See also: [Bear Power Pack](https://github.com/rovest/Bear-Power-Pack/blob/master/README.md)* ## Usage ``` python bear_export_sync.py --out ~/Notes/Bear --backup ~/Notes/Backup ``` See `--help` for more. ## Features * Bear notes exported as plain Markdown or Textbundles with images. * Syncs external edits back to Bear with original image links intact. * New external `.md` files or `.textbundles` are added. (Tags created from sub folder name) * Export option: Make nested folders from tags. For first tag only, or all tags (duplicates notes) * Export option: Include or exclude export of notes with specific tags. * Export option: Export as `.textbundles` with images included. * Or as: `.md` with links to common image repository * Export option: Hide tags in HTML comments like: `` if `hide_tags_in_comment_block = True` * **NEW** Hybrid export: `.textbundles` of notes with images, otherwise regular `.md` (Makes it easier to browse and edit on other platforms.) * **NEW** Writes log to `bear_export_sync_log.txt` in `BearSyncBackup` folder. Edit your Bear notes online in browser on [OneDrive.com](https://onedrive.live.com). It has a ok editor for plain text/markdown. Or with [StackEdit](https://stackedit.io/app), an amazing online markdown editor that can sync with *Dropbox* or *Google Drive* Read and edit your Bear notes on *Windows* or *Android* with any markdown editor of choice. Remote edits or new notes will be synced back into Bear again. *Typora* works great on Windows, and displays images of text bundles as well. NOTE! If syncing with Ulysses’ external folders on Mac, remember to edit that folder settings to `.textbundle` and `Inline Links`! Run script manually or add it to a cron job for automatic syncing (every 5 – 15 minutes, or whatever you prefer). ([LaunchD Task Scheduler](https://itunes.apple.com/us/app/launchd-task-scheduler/id620249105?mt=12) Is easy to configure and works very well for this) ### Syncs external edits back into Bear Script first checks for external edits in Markdown files or textbundles (previously exported from Bear as described below): * It replaces text in original note with `bear://x-callback-url/add-text?mode=replace` command (That way keeping original note ID and creation date) If any changes to title, new title will be added just below original title. (`mode=replace` does not replace title) * Original note in `sqlite` database and external edit are both backed up as markdown-files to BearSyncBackup folder before import to bear. * If a sync conflict, both original and new version will be in Bear (the new one with a sync conflict message and link to original). * New notes created online, are just added to Bear (with the `bear://x-callback-url/create` command) * If a textbundle gets new images from an external app, it will be opened and imported as a new note in Bear, with message and link to original note. (The `subprocess.call(['open', '-a', '/applications/bear.app', bundle])` command is used for this) ### Markdown export to Dropbox, OneDrive, or other: Then exports all notes from Bear's database.sqlite as plain markdown files: * Checks modified timestamp on database.sqlite, so exports only when needed. * Sets Bear note's modification date on exported markdown files. * Appends Bear note's creation date to filename to avoid “title-filename-collisions” * Note IDs are included at bottom of markdown files to match original note on sync back: {BearID:730A5BD2-0245-4EF7-BE16-A5217468DF0E-33519-0000429ADFD9221A} (these ID's are striped off again when synced back into Bear) * Uses rsync for copying (from a temp folder), so only changed notes will be synced to Dropbox (or other sync services) * rsync also takes care of deleting trashed notes * "Hides” tags from being displayed as H1 in other markdown apps by adding `period+space` in front of first tag on a line: `. #bear #idea #python` * Or hide tags in HTML comment blocks like: `` if `hide_tags_in_comment_block = True` (these are striped off again when synced back into Bear) * Makes subfolders named with first tag in note if `make_tag_folders = True` * Files can now be copied to multiple tag-folders if `multi_tags = True` * Export can now be restricted to a list of spesific tags: `limit_export_to_tags = ['bear/github', 'writings']` or leave list empty for all notes: `limit_export_to_tags = []` * Can export and link to images in common image repository * Or export as textbundles with images included You have Bear on Mac but also want your notes on your Android phone, on Linux or Windows machine at your office. Or you want them available online in a browser from any desktop computer. Here is a solution (or call it workaround) for now, until Bear comes with an online, Windows, or Android solution ;) Happy syncing! ;) ================================================ FILE: bear_export_sync.py ================================================ # encoding=utf-8 # python3.6 # bear_export_sync.py # Developed with Visual Studio Code with MS Python Extension. import shlex import objc from AppKit import NSWorkspace, NSWorkspaceOpenConfiguration, NSURL ''' # Markdown export from Bear sqlite database Version 1.4, 2020-01-11 modified by: github/andymatuschak, andy_matuschak@twitter original author: github/rovest, rorves@twitter See also: bear_import.py for auto import to bear script. ## Sync external updates: First checks for changes in external Markdown files (previously exported from Bear) * Replacing text in original note with callback-url replace command (Keeping original creation date) If changes in title it will be added just below original title * New notes are added to Bear (with x-callback-url command) * New notes get tags from sub folder names, or `#.inbox` if root * Backing up original note as file to BearSyncBackup folder (unless a sync conflict, then both notes will be there) ## Export: Then exporting Markdown from Bear sqlite db. * check_if_modified() on database.sqlite to see if export is needed * Uses rsync for copying, so only markdown files of changed sheets will be updated and synced by Dropbox (or other sync services) * "Hides" tags with `period+space` on beginning of line: `. #tag` not appear as H1 in other apps. (This is removed if sync-back above) * Or instead hide tags in HTML comment blocks like: `` if `hide_tags_in_comment_block = True` * Makes subfolders named with first tag in note if `make_tag_folders = True` * Files can now be copied to multiple tag-folders if `multi_tags = True` * Export can now be restricted to a list of spesific tags: `limit_export_to_tags = ['bear/github', 'writings']` or leave list empty for all notes: `limit_export_to_tags = []` * Can export and link to images in common image repository * Or export as textbundles with images included ''' make_tag_folders = False # Exports to folders using first tag only, if `multi_tag_folders = False` multi_tag_folders = True # Copies notes to all 'tag-paths' found in note! # Only active if `make_tag_folders = True` hide_tags_in_comment_block = False # Hide tags in HTML comments: `` # The following two lists are more or less mutually exclusive, so use only one of them. # (You can use both if you have some nested tags where that makes sense) # Also, they only work if `make_tag_folders = True`. only_export_these_tags = [] # Leave this list empty for all notes! See below for sample # only_export_these_tags = ['bear/github', 'writings'] export_as_textbundles = False # Exports as Textbundles with images included export_as_hybrids = True # Exports as .textbundle only if images included, otherwise as .md # Only used if `export_as_textbundles = True` export_image_repository = True # Export all notes as md but link images to # a common repository exported to: `assets_path` # Only used if `export_as_textbundles = False` import os HOME = os.getenv('HOME', '') default_out_folder = os.path.join(HOME, "Work", "BearNotes") default_backup_folder = os.path.join(HOME, "Work", "BearSyncBackup") # NOTE! Your user 'HOME' path and '/BearNotes' is added below! # NOTE! So do not change anything below here!!! import sqlite3 import datetime import re import subprocess import urllib.parse import time import shutil import fnmatch import json import argparse parser = argparse.ArgumentParser(description="Sync Bear notes") parser.add_argument("--out", default=default_out_folder, help="Path where Bear notes will be synced") parser.add_argument("--backup", default=default_backup_folder, help="Path where conflicts will be backed up (must be outside of --out)") parser.add_argument("--images", default=None, help="Path where images will be stored") parser.add_argument("--skipImport", action="store_const", const=True, default=False, help="When present, the script only exports from Bear to Markdown; it skips the import step.") parser.add_argument("--excludeTag", action="append", default=[], help="Don't export notes with this tag. Can be used multiple times.") parser.add_argument("--hideTags", action="store_const", const=True, default=False, help="Wrap tags in ") parsed_args = vars(parser.parse_args()) set_logging_on = True # NOTE! if 'BearNotes' is left blank, all other files in my_sync_service will be deleted!! export_path = parsed_args.get("out") no_export_tags = parsed_args.get("excludeTag") # If a tag in note matches one in this list, it will not be exported. hide_tags_in_comment_block = parsed_args.get("hideTags"); # NOTE! "export_path" is used for sync-back to Bear, so don't change this variable name! multi_export = [(export_path, True)] # only one folder output here. # Use if you want export to severa places like: Dropbox and OneDrive, etc. See below # Sample for multi folder export: # export_path_aux1 = os.path.join(HOME, 'OneDrive', 'BearNotes') # export_path_aux2 = os.path.join(HOME, 'Box', 'BearNotes') # NOTE! All files in export path not in Bear will be deleted if delete flag is "True"! # Set this flag fo False only for folders to keep old deleted versions of notes # multi_export = [(export_path, True), (export_path_aux1, False), (export_path_aux2, True)] temp_path = os.path.join(HOME, 'Temp', 'BearExportTemp') # NOTE! Do not change the "BearExportTemp" folder name!!! bear_db = os.path.join(HOME, 'Library/Group Containers/9K33E3U3T4.net.shinyfrog.bear/Application Data/database.sqlite') sync_backup = parsed_args.get("backup") # Backup of original note before sync to Bear. log_file = os.path.join(sync_backup, 'bear_export_sync_log.txt') # Paths used in image exports: bear_image_path = os.path.join(HOME, 'Library/Group Containers/9K33E3U3T4.net.shinyfrog.bear/Application Data/Local Files/Note Images') assets_path = parsed_args.get("images") if parsed_args.get("images") else os.path.join(export_path, 'BearImages') sync_ts = '.sync-time.log' export_ts = '.export-time.log' sync_ts_file = os.path.join(export_path, sync_ts) sync_ts_file_temp = os.path.join(temp_path, sync_ts) export_ts_file_exp = os.path.join(export_path, export_ts) export_ts_file = os.path.join(temp_path, export_ts) gettag_sh = os.path.join(HOME, 'temp/gettag.sh') gettag_txt = os.path.join(HOME, 'temp/gettag.txt') def main(): init_gettag_script() if not parsed_args.get("skipImport"): sync_md_updates() if check_db_modified(): delete_old_temp_files() note_count = export_markdown() write_time_stamp() rsync_files_from_temp() if export_image_repository and not export_as_textbundles: copy_bear_images() # notify('Export completed') write_log(str(note_count) + ' notes exported to: ' + export_path) exit(1) else: print('*** No notes needed exports') exit(0) def write_log(message): if set_logging_on == True: if not os.path.exists(sync_backup): os.makedirs(sync_backup) time_stamp = datetime.datetime.now().strftime("%Y-%m-%d at %H:%M:%S") message = message.replace(export_path + '/', '') with open(log_file, 'a', encoding='utf-8') as f: f.write(time_stamp + ': ' + message + '\n') def check_db_modified(): if not os.path.exists(sync_ts_file): return True db_ts = get_file_date(bear_db) last_export_ts = get_file_date(export_ts_file_exp) return db_ts > last_export_ts def export_markdown(): with sqlite3.connect(bear_db) as conn: conn.row_factory = sqlite3.Row query = "SELECT * FROM `ZSFNOTE` WHERE `ZTRASHED` LIKE '0' AND `ZARCHIVED` LIKE '0'" c = conn.execute(query) note_count = 0 for row in c: title = row['ZTITLE'] md_text = row['ZTEXT'].rstrip() creation_date = row['ZCREATIONDATE'] modified = row['ZMODIFICATIONDATE'] uuid = row['ZUNIQUEIDENTIFIER'] pk = row['Z_PK'] filename = clean_title(title) file_list = [] if make_tag_folders: file_list = sub_path_from_tag(temp_path, filename, md_text) else: is_excluded = False for no_tag in no_export_tags: if ("#" + no_tag) in md_text: is_excluded = True break if not is_excluded: file_list.append(os.path.join(temp_path, filename)) if file_list: mod_dt = dt_conv(modified) md_text = hide_tags(md_text) md_text += '\n\n\n' for filepath in file_list: note_count += 1 # print(filepath) if export_as_textbundles: if check_image_hybrid(md_text): make_text_bundle(md_text, filepath, mod_dt) else: write_file(filepath + '.md', md_text, mod_dt, creation_date) elif export_image_repository: md_proc_text = process_image_links(md_text, filepath, conn, pk) write_file(filepath + '.md', md_proc_text, mod_dt, creation_date) else: write_file(filepath + '.md', md_text, mod_dt, creation_date) return note_count def check_image_hybrid(md_text): if export_as_hybrids: if re.search(r'\[image:(.+?)\]', md_text): return True else: return False else: return True def make_text_bundle(md_text, filepath, mod_dt): ''' Exports as Textbundles with images included ''' bundle_path = filepath + '.textbundle' assets_path = os.path.join(bundle_path, 'assets') if not os.path.exists(bundle_path): os.makedirs(bundle_path) os.makedirs(assets_path) info = '''{ "transient" : true, "type" : "net.daringfireball.markdown", "creatorIdentifier" : "net.shinyfrog.bear", "version" : 2 }''' matches = re.findall(r'\[image:(.+?)\]', md_text) for match in matches: image_name = match new_name = image_name.replace('/', '_') source = os.path.join(bear_image_path, image_name) target = os.path.join(assets_path, new_name) shutil.copy2(source, target) md_text = re.sub(r'\[image:(.+?)/(.+?)\]', r'![](assets/\1_\2)', md_text) write_file(bundle_path + '/text.md', md_text, mod_dt, 0) write_file(bundle_path + '/info.json', info, mod_dt, 0) os.utime(bundle_path, (-1, mod_dt)) def sub_path_from_tag(temp_path, filename, md_text): # Get tags in note: pattern1 = r'(?', md_text) return md_text def restore_tags(md_text): # Tags back to normal Bear tags, stripping the `period+space` at start of line: if hide_tags_in_comment_block: md_text = re.sub(r'(\n)', r'\1\2', md_text) return md_text def clean_title(title): title = title[:225].strip() if title == "": title = "Untitled" title = re.sub(r'[\/\\:]', r'-', title) title = re.sub(r'-$', r'', title) return title.strip() def write_file(filename, file_content, modified, created): with open(filename, "w", encoding='utf-8') as f: f.write(file_content) if modified > 0: os.utime(filename, (-1, modified)) if created > 0: newnum = dt_conv(created) dtdate = datetime.datetime.fromtimestamp(newnum) datestring = dtdate.strftime("%m/%d/%Y %H:%M:%S") command = 'SetFile -d "' + datestring + '" ' + shlex.quote(filename) subprocess.call(command, shell=True) def read_file(file_name): with open(file_name, "r", encoding='utf-8') as f: file_content = f.read() return file_content def get_file_date(filename): try: t = os.path.getmtime(filename) return t except: return 0 def dt_conv(dtnum): # Formula for date offset based on trial and error: hour = 3600 # seconds year = 365.25 * 24 * hour offset = year * 31 + hour * 6 return dtnum + offset def date_time_conv(dtnum): newnum = dt_conv(dtnum) dtdate = datetime.datetime.fromtimestamp(newnum) #print(newnum, dtdate) return dtdate.strftime(' - %Y-%m-%d_%H%M') def time_stamp_ts(ts): dtdate = datetime.datetime.fromtimestamp(ts) return dtdate.strftime('%Y-%m-%d at %H:%M') def date_conv(dtnum): dtdate = datetime.datetime.fromtimestamp(dtnum) return dtdate.strftime('%Y-%m-%d') def delete_old_temp_files(): # Deletes all files in temp folder before new export using "shutil.rmtree()": # NOTE! CAUTION! Do not change this function unless you really know shutil.rmtree() well! if os.path.exists(temp_path) and "BearExportTemp" in temp_path: # *** NOTE! Double checking that temp_path folder actually contains "BearExportTemp" # *** Because if temp_path is accidentally empty or root, # *** shutil.rmtree() will delete all files on your complete Hard Drive ;( shutil.rmtree(temp_path) # *** NOTE: USE rmtree() WITH EXTREME CAUTION! os.makedirs(temp_path) def rsync_files_from_temp(): # Moves markdown files to new folder using rsync: # This is a very important step! # By first exporting all Bear notes to an emptied temp folder, # rsync will only update destination if modified or size have changed. # So only changed notes will be synced by Dropbox or OneDrive destinations. # Rsync will also delete notes on destination if deleted in Bear. # So doing it this way saves a lot of otherwise very complex programing. # Thank you very much, Rsync! ;) for (dest_path, delete) in multi_export: if not os.path.exists(dest_path): os.makedirs(dest_path) if delete: subprocess.call(['rsync', '-r', '-t', '--crtimes', '-E', '--delete', '--exclude', 'BearImages/', '--exclude', '.obsidian/', '--exclude', '.Ulysses*', '--exclude', '*.Ulysses_Public_Filter', temp_path + "/", dest_path]) else: subprocess.call(['rsync', '-r', '-t', '-E', temp_path + "/", dest_path]) def sync_md_updates(): updates_found = False if not os.path.exists(sync_ts_file) or not os.path.exists(export_ts_file): return False ts_last_sync = os.path.getmtime(sync_ts_file) ts_last_export = os.path.getmtime(export_ts_file) # Update synced timestamp file: update_sync_time_file(0) file_types = ('*.md', '*.txt', '*.markdown') for (root, dirnames, filenames) in os.walk(export_path): if '.obsidian' in dirnames: dirnames.remove('.obsidian') ''' This step walks down into all sub folders, if any. ''' for pattern in file_types: for filename in fnmatch.filter(filenames, pattern): md_file = os.path.join(root, filename) ts = os.path.getmtime(md_file) if ts > ts_last_sync: if not updates_found: # Yet # Wait 5 sec at first for external files to finish downloading from dropbox. # Otherwise images in textbundles might be missing in import: time.sleep(5) updates_found = True md_text = read_file(md_file) backup_ext_note(md_file) if check_if_image_added(md_text, md_file): textbundle_to_bear(md_text, md_file, ts) write_log('Imported to Bear: ' + md_file) else: update_bear_note(md_text, md_file, ts, ts_last_export) write_log('Bear Note Updated: ' + md_file) if updates_found: # Give Bear time to process updates: time.sleep(3) # Check again, just in case new updates synced from remote (OneDrive/Dropbox) # during this process! # The logic is not 100% fool proof, but should be close to 99.99% sync_md_updates() # Recursive call return updates_found def check_if_image_added(md_text, md_file): if not '.textbundle/' in md_file: return False matches = re.findall(r'!\[.*?\]\(assets/(.+?_).+?\)', md_text) for image_match in matches: 'F89CDA3D-3FCC-4E92-88C1-CC4AF46FA733-10097-00002BBE9F7FF804_IMG_2280.JPG' if not re.match(r'[0-9A-F]{8}-([0-9A-F]{4}-){3}[0-9A-F]{12}-[0-9A-F]{3,5}-[0-9A-F]{16}_', image_match): return True return False def textbundle_to_bear(md_text, md_file, mod_dt): md_text = restore_tags(md_text) bundle = os.path.split(md_file)[0] match = re.search(r'\{BearID:(.+?)\}', md_text) if match: uuid = match.group(1) # Remove old BearID: from new note md_text = re.sub(r'\<\!-- ?\{BearID\:' + uuid + r'\} ?--\>', '', md_text).rstrip() + '\n' md_text = insert_link_top_note(md_text, 'Images added! Link to original note: ', uuid) else: # New textbundle (with images), add path as tag: md_text = get_tag_from_path(md_text, bundle, export_path) write_file(md_file, md_text, mod_dt, 0) os.utime(bundle, (-1, mod_dt)) subprocess.call(['open', '-a', '/applications/bear.app', bundle]) time.sleep(0.5) def backup_ext_note(md_file): if '.textbundle' in md_file: bundle_path = os.path.split(md_file)[0] bundle_name = os.path.split(bundle_path)[1] target = os.path.join(sync_backup, bundle_name) bundle_raw = os.path.splitext(target)[0] count = 2 while os.path.exists(target): # Adding sequence number to identical filenames, preventing overwrite: target = bundle_raw + " - " + str(count).zfill(2) + ".textbundle" count += 1 shutil.copytree(bundle_path, target) else: # Overwrite former bacups of incoming changes, only keeps last one: shutil.copy2(md_file, sync_backup + '/') def update_sync_time_file(ts): write_file(sync_ts_file, "Checked for Markdown updates to sync at: " + datetime.datetime.now().strftime("%Y-%m-%d at %H:%M:%S"), ts, 0) def update_bear_note(md_text, md_file, ts, ts_last_export): md_text = restore_tags(md_text) md_text = restore_image_links(md_text) uuid = '' match = re.search(r'\{BearID:(.+?)\}', md_text) sync_conflict = False if match: uuid = match.group(1) # Remove old BearID: from new note md_text = re.sub(r'\<\!-- ?\{BearID\:' + uuid + r'\} ?--\>', '', md_text).rstrip() + '\n' sync_conflict = check_sync_conflict(uuid, ts_last_export) if sync_conflict: link_original = 'bear://x-callback-url/open-note?id=' + uuid message = '::Sync conflict! External update: ' + time_stamp_ts(ts) + '::' message += '\n[Click here to see original Bear note](' + link_original + ')' x_create = 'bear://x-callback-url/create?show_window=no&open_note=no' bear_x_callback(x_create, md_text, message, '') else: # Regular external update orig_title = backup_bear_note(uuid) # message = '::External update: ' + time_stamp_ts(ts) + '::' x_replace = 'bear://x-callback-url/add-text?show_window=no&open_note=no&mode=replace&id=' + uuid bear_x_callback(x_replace, md_text, '', orig_title) # # Trash old original note: # x_trash = 'bear://x-callback-url/trash?show_window=no&id=' + uuid # subprocess.call(["open", x_trash]) # time.sleep(.2) else: # New external md Note, since no Bear uuid found in text: # message = '::New external Note - ' + time_stamp_ts(ts) + '::' md_text = get_tag_from_path(md_text, md_file, export_path) x_create = 'bear://x-callback-url/create?show_window=no' bear_x_callback(x_create, md_text, '', '') return def get_tag_from_path(md_text, md_file, root_path, inbox_for_root=False, extra_tag=''): # extra_tag should be passed as '#tag' or '#space tag#' path = md_file.replace(root_path, '')[1:] sub_path = os.path.split(path)[0] tags = [] if '.textbundle' in sub_path: sub_path = os.path.split(sub_path)[0] if sub_path == '': if inbox_for_root: tag = '#.inbox' else: tag = '' elif sub_path.startswith('_'): tag = '#.' + sub_path[1:].strip() else: tag = '#' + sub_path.strip() if ' ' in tag: tag += "#" if tag != '': tags.append(tag) if extra_tag != '': tags.append(extra_tag) for tag in get_file_tags(md_file): tag = '#' + tag.strip() if ' ' in tag: tag += "#" tags.append(tag) return md_text.strip() + '\n\n' + ' '.join(tags) + '\n' def get_file_tags(md_file): try: subprocess.call([gettag_sh, md_file, gettag_txt]) text = re.sub(r'\\n\d{1,2}', r'', read_file(gettag_txt)) tag_list = json.loads(text) return tag_list except: return [] open_config = NSWorkspaceOpenConfiguration.alloc().init() open_config.setActivates_(False) def bear_x_callback(x_command, md_text, message, orig_title): if message != '': lines = md_text.splitlines() lines.insert(1, message) md_text = '\n'.join(lines) if orig_title != '': lines = md_text.splitlines() title = re.sub(r'^#+ ', r'', lines[0]) if title != orig_title: md_text = '\n'.join(lines) else: md_text = '\n'.join(lines[1:]) x_command_text = x_command + '&text=' + urllib.parse.quote(md_text) url = NSURL.URLWithString_(x_command_text) NSWorkspace.sharedWorkspace().openURL_configuration_completionHandler_(url, open_config, None) time.sleep(.2) def check_sync_conflict(uuid, ts_last_export): conflict = False # Check modified date of original note in Bear sqlite db! with sqlite3.connect(bear_db) as conn: conn.row_factory = sqlite3.Row query = "SELECT * FROM `ZSFNOTE` WHERE `ZTRASHED` LIKE '0' AND `ZUNIQUEIDENTIFIER` LIKE '" + uuid + "'" c = conn.execute(query) for row in c: modified = row['ZMODIFICATIONDATE'] uuid = row['ZUNIQUEIDENTIFIER'] mod_dt = dt_conv(modified) conflict = mod_dt > ts_last_export return conflict def backup_bear_note(uuid): # Get single note from Bear sqlite db! with sqlite3.connect(bear_db) as conn: conn.row_factory = sqlite3.Row query = "SELECT * FROM `ZSFNOTE` WHERE `ZUNIQUEIDENTIFIER` LIKE '" + uuid + "'" c = conn.execute(query) title = '' for row in c: # Will only get one row if uuid is found! title = row['ZTITLE'] md_text = row['ZTEXT'].rstrip() modified = row['ZMODIFICATIONDATE'] mod_dt = dt_conv(modified) created = row['ZCREATIONDATE'] cre_dt = dt_conv(created) md_text = insert_link_top_note(md_text, 'Link to updated note: ', uuid) dtdate = datetime.datetime.fromtimestamp(cre_dt) filename = clean_title(title) + dtdate.strftime(' - %Y-%m-%d_%H%M') if not os.path.exists(sync_backup): os.makedirs(sync_backup) file_part = os.path.join(sync_backup, filename) # This is a Bear text file, not exactly markdown. backup_file = file_part + ".txt" count = 2 while os.path.exists(backup_file): # Adding sequence number to identical filenames, preventing overwrite: backup_file = file_part + " - " + str(count).zfill(2) + ".txt" count += 1 write_file(backup_file, md_text, mod_dt, created) filename2 = os.path.split(backup_file)[1] write_log('Original to sync_backup: ' + filename2) return title def insert_link_top_note(md_text, message, uuid): lines = md_text.split('\n') title = re.sub(r'^#{1,6} ', r'', lines[0]) link = '::' + message + '[' + title + '](bear://x-callback-url/open-note?id=' + uuid + ')::' lines.insert(1, link) return '\n'.join(lines) def init_gettag_script(): gettag_script = \ '''#!/bin/bash if [[ ! -e $1 ]] ; then echo 'file missing or not specified' exit 0 fi JSON="$(xattr -p com.apple.metadata:_kMDItemUserTags "$1" | xxd -r -p | plutil -convert json - -o -)" echo $JSON > "$2" ''' temp = os.path.join(HOME, 'temp') if not os.path.exists(temp): os.makedirs(temp) write_file(gettag_sh, gettag_script, 0, 0) subprocess.call(['chmod', '777', gettag_sh]) def notify(message): title = "ul_sync_md.py" try: # Uses "terminal-notifier", download at: # https://github.com/julienXX/terminal-notifier/releases/download/2.0.0/terminal-notifier-2.0.0.zip # Only works with MacOS 10.11+ subprocess.call(['/Applications/terminal-notifier.app/Contents/MacOS/terminal-notifier', '-message', message, "-title", title, '-sound', 'default']) except: write_log('"terminal-notifier.app" is missing!') return if __name__ == '__main__': main() ================================================ FILE: bear_import.py ================================================ # encoding=utf-8 # python3.6 # bear_import.py # Developed with Visual Studio Code with MS Python Extension. ''' # Markdown import to Bear from folder Version 1.0.0 - 2018-02-10 at 17:37 EST github/rovest, rorves@twitter ## NEW import function: * Imports markdown or textbundles from nested folders under a `BearImport/input/' folder * Foldernames are converted to Bear tags * Also imports MacOS file tags as Bear tags * Imported notes are also tagged with `#.imported/yyyy-MM-dd` for convenience. * Import-files are then cleared to a `BearImport/done/' folder * Use for email input to Bear with Zapier's "Gmail to Dropbox" zap. * Or for import of nested groups and sheets from Ulysses, images and keywords included. ''' my_sync_service = 'Dropbox' # Change 'Dropbox' to 'Box', 'Onedrive', # or whatever folder of sync service you need. # Your user "Home" folder is added below. use_filename_as_title = False # Set to `True` if importing Simplenotes synced with nvALT. set_logging_on = True # This tag is added for convenience (easy deletion of imported notes they are not wanted.) # (Easier to delete one tag, than finding a bunch of tagless imported notes.) import datetime import re import subprocess import urllib.parse import os import time import shutil import fnmatch import json import_tag = '#.imported/' + datetime.datetime.now().strftime('%Y-%m-%d') # import_tag = '' # Blank if not needed HOME = os.getenv('HOME', '') # Import folder for files from other apps, # or incoming emails via "Gmail to Dropbox" Zapier zap or IFTTT bear_import = os.path.join(HOME, my_sync_service, 'BearImport') import_path = os.path.join(bear_import, 'input') import_done = os.path.join(bear_import, 'done') gettag_sh = os.path.join(HOME, 'temp/gettag.sh') gettag_txt = os.path.join(HOME, 'temp/gettag.txt') def main(): if not os.path.exists(import_path): os.makedirs(import_path) print('New path, use it for import to Bear:', import_path) return False if not os.path.exists(import_done): os.makedirs(import_done) init_gettag_script() count = import_external_files() print(str(count), 'files imported. Job done!') def import_external_files(): files_found = False file_types = ('*.md', '*.txt', '*.markdown') count = 0 time.sleep(3) # Wait a little bit after being triggered by Automator Folder Action for (root, dirnames, filenames) in os.walk(import_path): ''' This step walks down into all sub folders, if any. ''' for pattern in file_types: for filename in fnmatch.filter(filenames, pattern): if not files_found: # Yet # Wait 5 sec at first for external files to finish downloading from dropbox. # Otherwise images in textbundles might be missing in import: time.sleep(5) files_found = True md_file = os.path.join(root, filename) mod_dt = os.path.getmtime(md_file) md_text = read_file(md_file) if pattern == '*.txt': # Replace rich text bullets to markdown: # (When using with IFTTT or Zapier and Gmail to Dropbox zap.) md_text = md_text.replace('\n• ', '\n- ') md_text = md_text.replace('\n • ', '\n\t- ') md_text = md_text.replace('\n • ', '\n\t\t- ') if re.search(r'!\[.*?\]\(assets/.+?\)', md_text) \ and '.textbundle/' in md_file: # New textbundle with images: bundle = os.path.split(md_file)[0] md_text = get_tag_from_path(md_text, bundle, import_path, False) write_file(md_file, md_text, mod_dt) os.utime(bundle, (-1, mod_dt)) subprocess.call(['open', '-a', '/applications/bear.app', bundle]) time.sleep(0.5) move_import_to_done(bundle, import_path, import_done) else: title = '' # No images, import markdown only even if textbundle: if '.textbundle/' in md_file: file_bundle = os.path.split(md_file)[0] else: file_bundle = md_file if use_filename_as_title: title = os.path.splitext(os.path.split(md_file)[1])[0] md_text = get_tag_from_path(md_text, file_bundle, import_path, False) x_create = 'bear://x-callback-url/create?show_window=no' bear_x_callback(x_create, md_text, title) move_import_to_done(file_bundle, import_path, import_done) write_log('Imported to Bear: ', file_bundle) count += 1 if files_found: # cleanup empty input sub folders here ??? # But quite tricky since new files may appear. Bette to do that manually when needed. # Recursive call to look for leftovers/newly downloaded files: count += import_external_files() return count def move_import_to_done(file_bundle, import_path, import_done): file_path = file_bundle.replace(import_path + '/', '') sub_path = os.path.split(file_path)[0] dest_path = os.path.join(import_done, sub_path) if not os.path.exists(dest_path): os.makedirs(dest_path) count = 2 file_name = os.path.split(file_bundle)[1] dest_file = os.path.join(dest_path, file_name) (file_raw, ext) = os.path.splitext(file_name) while os.path.exists(dest_file): # Adding sequence number to identical filenames, preventing overwrite: dest_file = os.path.join(dest_path, file_raw + " - " + str(count).zfill(2) + ext) count += 1 # dest_path = os.path.split(dest_file)[0] shutil.move(file_bundle, dest_file) def get_tag_from_path(md_text, file_bundle, root_path, inbox_for_root=True): path = file_bundle.replace(root_path, '')[1:] sub_path = os.path.split(path)[0] tags = [] if sub_path == '': if inbox_for_root: tag = '#.inbox' else: tag = '' elif sub_path.startswith('_'): tag = '#.' + sub_path[1:].strip() else: tag = '#' + sub_path.strip() if ' ' in tag: tag += "#" if tag != '': tags.append(tag) if import_tag != '': tags.append(import_tag) for tag in get_file_tags(file_bundle): tag = '#' + tag.strip() if ' ' in tag: tag += "#" tags.append(tag) return md_text.strip() + '\n\n' + ' '.join(tags) + '\n' def get_file_tags(file_bundle): try: subprocess.call([gettag_sh, file_bundle, gettag_txt]) tags_raw = read_file(gettag_txt) tags_text = re.sub(r'\\n\d{1,2}', r'', tags_raw) tag_list = json.loads(tags_text) return tag_list except: return [] def bear_x_callback(x_command, md_text, title): if title != '' and not title.startswith("#"): md_text = '# ' + title + '\n' + md_text x_command_text = x_command + '&text=' + urllib.parse.quote(md_text) subprocess.call(["open", x_command_text]) time.sleep(.2) def init_gettag_script(): gettag_script = \ '''#!/bin/bash if [[ ! -e $1 ]] ; then echo 'file missing or not specified' exit 0 fi JSON="$(xattr -p com.apple.metadata:_kMDItemUserTags "$1" | xxd -r -p | plutil -convert json - -o -)" echo $JSON > "$2" ''' temp = os.path.join(HOME, 'temp') if not os.path.exists(temp): os.makedirs(temp) write_file(gettag_sh, gettag_script, 0) subprocess.call(['chmod', '777', gettag_sh]) def write_log(message, file_bundle): if set_logging_on == True: log_file = os.path.join(import_done, 'bear_import_log.txt') time_stamp = datetime.datetime.now().strftime("%Y-%m-%d at %H:%M:%S") # file_name = os.path.split(file_path)[1] file_path = file_bundle.replace(import_path + '/', '') with open(log_file, 'a', encoding='utf-8') as f: f.write(time_stamp + ': ' + message + file_path +'\n') def write_file(filename, file_content, modified): with open(filename, "w", encoding='utf-8') as f: f.write(file_content) if modified > 0: os.utime(filename, (-1, modified)) def read_file(file_name): with open(file_name, "r", encoding='utf-8') as f: file_content = f.read() return file_content if __name__ == '__main__': main()