[
  {
    "path": "MIT-LICENSE",
    "content": "Copyright (c) 2019 Francois Payette\n\nPermission is hereby granted, free of charge, to any person obtaining\na copy of this software and associated documentation files (the\n\"Software\"), to deal in the Software without restriction, including\nwithout limitation the rights to use, copy, modify, merge, publish,\ndistribute, sublicense, and/or sell copies of the Software, and to\npermit persons to whom the Software is furnished to do so, subject to\nthe following conditions:\n\nThe above copyright notice and this permission notice shall be\nincluded in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND,\nEXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF\nMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND\nNONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE\nLIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION\nOF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION\nWITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n"
  },
  {
    "path": "README.md",
    "content": "# DuplexRsync\n\n🌟 Simple realtime 2-way sync.\n\n### Problem\n\nI often find myself editing quite a few files on remote hosts; for anything non-trivial I like to use local-running tools such as Sublime. I've used [rsub](https://github.com/henrikpersson/rsub), it's very nice and lightweight. Sometimes(often) the light editing turns heavier and more and more files are worked on. I have noticed that when the ssh tunnel dies and is recreated while a file is open, the file will be truncated to zlitch --a glitch to look out for that is more likely to occur when multiple files are open.\n\nWhen things keep getting heavier, I've then used [sshfs](https://github.com/osxfuse/osxfuse/wiki/SSHFS) to mount a remote directory and fuse it to the local filesystem. This usually works ok, but for some types of workflows such as sublime projects with a lot of files in subfolders (node_modules? --sometimes this one starts to feel like a whole Gentoo distro) it is inadequate. Search becomes extra slow. The SublimeText project tree spins and spins and spins, features that have become automatisms are unworkable. Also, open files prevent the tunnelling connection from exiting; and a broken tunnel (say you close your laptop without closing everything and unmounting) can leave the fuse subsystem in a weird state, where you cannot remount to the previous location until a reboot, as well as other minor glitches.\n\n### Solution\n\nDuplexRsync is a simple and pretty sweet (although only lightly tested as of 2019/03, PLEASE BE CAREFUL AND ALWAYS HAVE BACKUPS and/or VERSIONING!) solution based on fswatch and rsync. It's a single file you'll put in your local directory that will maintain (DropBox|GoogleDrive)-style 2-way sync between the current directory and a remote directory via SSH. This has the advantage to work fine when offline. This bash script is a bit macOSX-centric because that's what I use locally, please feel free to adapt. By default the script excludes node_modules and all folders that start with a period. (.git etc)\n\n### Merging\n\nIf a file has been edited on both ends while offline (duplexRsync not running), merging will simply crush the oldest edit; it will never results in conflict files. This is harsh but simpler; with git these days I think edits that have some value should be committed, so we delegate versioning there.\n\nIf you attempt to sync mismatched folders, a lot of files in the remote folder would get deleted. When launching duplexRsync you'll be prompted to either merge the folders (create these files in the local folder), or destroy all the extra files in the remote folder.\n\nLatency for multiple remote edits to propagate to local folder is set by default to 3 seconds, this prevents infinite cycling of change detection. Over very slow network connections you might need to increase this value.\n\n###  Setup\n\non your remote machine you'll need fswatch:\n\n\n    sudo add-apt-repository ppa:hadret/fswatch\n    sudo apt-get update\n    sudo apt-get install -y fswatch\n\non your local machine you'll need brew, that's it. This script will install the other required components (socat fswatch and gnu-getopt)\n\n    chmod u+x duplexRsync.sh\n    ./duplexRsync.sh --remoteHost user@192.168.0.2\n\n### Caveats\n\nThis is a simple solution, it does not implement any distributed locking. If you or processes are editing at both ends simultaneously, over and above the crushing of the oldest edit of the same file mentioned, there's a window while a newly created file can get deleted. Conversely but less serious, there's also a window during which a deleted file could be recreated. An argument to --delete-older-than \"seconds\" in rsync would mitigate the first edge case, I think the second one(zombie file coming back) is a an annoyance I can live with.\n\n### Related\n\nThanks for all the feedback in various forums. Here are a few related projects that have been brought to my attention. I have not tried any of these, they all look very well written; they could come in handy later.\n\n#### Heavier\n\n- [osync](https://github.com/deajan/osync)\n\n#### Heaviest\n\n- [Mutagen.io](https://mutagen.io/)\n- [Syncthing](https://github.com/syncthing/syncthing)\n- [Unison](https://github.com/bcpierce00/unison)\n\n\n\n\nThat's it!🔥 Cheers!\n\nPlease Note: A few hidden files are created to maintain the 2-way sync, they all start by .____*. The remote directory will be straight off the home of your remote user's home; there's an optional --remoteParent if you need to change that.\n\n\nLicense: MIT\n"
  },
  {
    "path": "duplexRsync.sh",
    "content": "#!/bin/bash\n\n# REQUIREMENT we need fswatch on both ends, run this to get it on ubuntu1604\n#sudo add-apt-repository ppa:hadret/fswatch\n#sudo apt-get update\n#sudo apt-get install -y fswatch\nprintHelp(){\n  echo \"USAGE: duplexRsync --remoteHost user@host\n\n  DuplexRsync requires fswatch on both ends, this tries to install it locally using brew(required).\n    on the remote end run:\n    sudo add-apt-repository ppa:hadret/fswatch\n    sudo apt-get update\n    sudo apt-get install -y fswatch\n\n  you need to specify:\n    --remoteHost        ex: user@192.168.0.2.\n\n  You can also optionally specify:\n    --remoteParent      contains/will contain the remoteDir\"\n}\n\n# if our arguments match this string, it's the socat fork trgger for remote change detection; increment sentinel and exit\nif [ \"$*\" =  \"sentinelIncrement\" ];\nthen\n  sentval=$(cat .____sentinel);sentval=$((sentval+1));echo $sentval > .____sentinel;\n  exit;\nfi\n\n\nif [ \"$*\" =  \"\" ];\nthen\n   printHelp;\n  exit;\nfi\n\n# we need brew on macosx\nif [ -z $(command -v brew) ];\nthen\n  printHelp;\n  exit\nfi\n\n# this is for macosx, we also need socat to create a socket to remote trigger rsync\nbrew install socat fswatch gnu-getopt\n\n\nfunction randomLocalPort() {\n  localPort=42\n  localPort=$RANDOM;\n  let \"localPort %= 999\";\n  localPort=\"42$localPort\"\n}\n\nfunction randomRemotePort() {\n  remotePort=42\n  remotePort=$RANDOM;\n  let \"remotePort %= 999\";\n  remotePort=\"42$remotePort\"\n}\n\n\nif ! options=$(/usr/local/Cellar/gnu-getopt/*/bin/getopt -u -o hr:p: -l help,remoteHost:,remoteParent: -- \"$@\")\nthen\n    # something went wrong, getopt will put out an error message for us\n    exit 1\nfi\n\n\nset -- $options\n\nwhile [ $# -gt 0 ]\ndo\n    case $1 in\n    # for options with required arguments, an additional shift is required\n    -h|--help ) printHelp; exit; shift;;\n    -r|--remoteHost ) remoteHost=$2; shift;;\n    -p|--remoteParent ) remoteParent=$2; shift;;\n    --) shift; break;;\n    #(-*) echo \"$0: error - unrecognized option $1\" 1>&2; exit 1;;\n    (*) break;;\n    esac\n    shift\ndone\n\nif [ -z \"$remoteHost\" ];\nthen\n  echo \"Missing Argument: --remoteHost\"\n  printHelp;\n  exit;\nfi\n\nremoteDir=${PWD##*/}\nremoteDir=\"$remoteParent$remoteDir\"\n\n\n\n\nif [ ! -f ~/.ssh/id_rsa.pub ];\nthen\n  echo \"You need a key pair to use duplexRsync. You can generate one using: ssh-keygen -t rsa\"\n  exit;\nfi\n\n# we'll need to ssh without pass - use public key crypto to ssh into remote end,  rsync needs this\n#we are copying our pubkey to ssh in without prompt\ncat ~/.ssh/id_rsa.pub | ssh \"$remoteHost\"  'mkdir .ssh;pubkey=$(cat); touch .ssh/authorized_keys; if grep -q \"$pubkey\" \".ssh/authorized_keys\"; then echo \"puublic key for this user already present\"; else echo $pubkey >> .ssh/authorized_keys;fi'\n\n\nfswatchPath=$(ssh \"$remoteHost\" 'command -v fswatch')\n#on macosx remote the $PATH variable is different when local or ssh, lets try with looking up the local path\nif [ -z \"$fswatchPath\" ];\nthen\n  fswatchPath=$(ssh \"$remoteHost\" 'command -v /usr/local/bin/fswatch')\nfi\n\nif [ -z \"$fswatchPath\" ];\nthen\n  echo \"ERROR: missing fswatch at remote end\"\n  printHelp;\n  exit;\nfi\n\n# kill all remote fswatches for this path that might be lingering\nssh $remoteHost \"pkill -P \\$(ps alx | egrep '.*pipe_w.*____rsyncSignal.sh --pwd $PWD --port $remotePort' | awk '{print \\$4}' | head -n 1)\"\n\nssh $remoteHost \"pkill -f '____rsyncSignal.sh --pwd $PWD'\"\n# if we have the ssh tunnel running this will match and we kill it; pwd args to prevent killing other folders being watched\npkill -f \"rsyncSignal.sh --pwd $PWD\"\n# if we have a lingering socat kill it\n# we shouldnt have one, this is a bad plan if using multple sockets\n#pkill -f \"sentinelIncrement.sh --pwd $PWD\"\n\necho '0' > .____sentinel\n#create localsocket to listen for remote changes\nsocatRes=\"not listening yet, we get a random port in the following loop\";\nwhile [ ! -z \"$socatRes\" ]\ndo\n  randomLocalPort;\n  socatRes=\"\";\n  # frok call this script with a special argument that simply inccrement snetinel and exits\n  socatRes=$(socat TCP-LISTEN:$localPort,fork EXEC:\"./duplexRsync.sh sentinelIncrement\" 2>&1 &) &\n  # result should be empty when listen works\ndone;\n\necho \"listening locally on:$localPort\"\n\n\n#for now we use the same port at both ends, this is a bit sloppy we should test to make sure it's not used with the ssh -R call\nremotePort=$localPort\n\n#we dump to a remote file the fswatch command that allows local running socat to get a signal of a remote change\n# modification to add the -r switch to all subs excluding node_modules. This is required because fswatch will still iterate over all subdirs because the -e switch is a pattern, not a path\n\n# if you get a bunch of: inotify_add_watch: No space left on device\n# you will need to https://github.com/guard/listen/wiki/Increasing-the-amount-of-inotify-watchers\n# check your current limit: cat /proc/sys/fs/inotify/max_user_watches\n# ATTENTION: you cannot change this kernel param if running in an unpriviledged container, you'll need to run this in the hosting kernel's env\n#  echo fs.inotify.max_user_watches=524288 | tee -a /etc/sysctl.conf && sysctl -p; echo \"increasing the limit of watches, cannot be done in unpriv container\"\n#echo \"$fswatchPath -r -e \\\"node_modules\\\" -o . | while read f; do echo 1 | nc localhost $remotePort; done\" | ssh $remoteHost  \"mkdir -p $remoteDir; cd $remoteDir; cat > .____rsyncSignal.sh\"\nabsPath=$(ssh $remoteHost \"mkdir -p $remoteDir; cd $remoteDir; pwd\")\n\n# we are exluding node_modules and folders starting with .\nssh $remoteHost \"mkdir -p $remoteDir; cd $remoteDir; find $absPath -maxdepth 1 -mindepth 1 -type d  ! -name \\\"node_modules\\\" ! -name \\\".*\\\"|  awk '{ print \\\"\\\\\\\"\\\"\\$0\\\"\\\\\\\"\\\"}' |  nl | awk -F\\\\\\\" '{printf \\\"/usr/bin/fswatch  -x --event Updated --event Created --event Removed --event Renamed --event MovedFrom --event MovedTo -r \\\\\\\"%s\\\\\\\"  | while read f; do  echo 1 | nc localhost $remotePort; done \\& \\n\\\", \\$2, \\$1, \\$1}' > .____rsyncSignal.sh\"\nssh $remoteHost \"cd $remoteDir; echo \\\"/usr/bin/fswatch -x --event Updated --event Created --event Removed --event Renamed --event MovedFrom --event MovedTo -o $absPath | while read f; do echo 1 | nc localhost $remotePort; done\\\" >> .____rsyncSignal.sh\"\n\n# we are exluding node_modules and folders starting with .\n# this should work, but there seems to be a bug in fswatch, so we are using multiple processes instead\n#ssh $remoteHost \"mkdir -p $remoteDir; cd $remoteDir; find $absPath -maxdepth 1 -mindepth 1 -type d  ! -name \\\"node_modules\\\" ! -name \\\".*\\\" |  awk '{ print \\\"\\\\\\\"\\\"\\$0\\\"\\\\\\\"\\\"}' | awk -F\\\\\\\" '{printf \\\" \\\\\\\"%s\\\\\\\" \\\", \\$2}'  | (echo -n \\\" /usr/bin/fswatch  -x --event Updated --event Created --event Removed --event Renamed --event MovedFrom --event MovedTo -r  \\\" && cat) > .____rsyncSignal.sh\"\n#ssh $remoteHost \"cd $remoteDir; echo \\\" | while read f; do if [ -z \\\\\\\"\\$skip\\\\\\\" ]; then skip=\\\\\\\"recursive first msg is spurious\\\\\\\"; else echo 1 | nc localhost $remotePort; fi done & /usr/bin/fswatch -o $absPath | while read f; do echo 1 | nc localhost $remotePort; done\\\" >> .____rsyncSignal.sh\"\n#exit 1;\n\n\nfunction duplex_rsync() {\n\n    # kill all remote fswatches, also supress kill notice in bash\n    ssh $remoteHost \"pkill -P \\$(ps alx | egrep '.*pipe_w.*____rsyncSignal.sh --pwd $PWD --port $remotePort' | awk '{print \\$4}' | head -n 1) >/dev/null 2&>1\"\n\n    # kill the remote fswatch while we sync, pwd arg used to prevent attempting to kill other watches; port prevent killing if 2 locals have the exact same path local\n    # also this discloses local path to remote end; dont think this is serious\n    ssh $remoteHost \"pkill -f '____rsyncSignal.sh --pwd $PWD --port $remotePort'\"\n\n\n    # also kill the tunnel\n    pkill -f \"rsyncSignal.sh --pwd $PWD\"\n\n    # order matters; if we got a remote trigger we'll process remote as src first to prevent restoring files that might have just been deleted\n    if [ \"$trigger\" = \"remote\" ];\n    then\n      rsync -auzP --exclude \".*/\" --exclude \".____*\"  --exclude \"node_modules\" --delete \"$remoteHost:$remoteDir/\" .;\n      rsync -auzP --exclude \".*/\" --exclude \".____*\"  --exclude \"node_modules\" --delete . \"$remoteHost:$remoteDir\";\n    else # local as src first\n      rsync -auzP --exclude \".*/\" --exclude \".____*\"  --exclude \"node_modules\" --delete . \"$remoteHost:$remoteDir\";\n      rsync -auzP --exclude \".*/\" --exclude \".____*\"  --exclude \"node_modules\" --delete \"$remoteHost:$remoteDir/\" .;\n    fi;\n\n\n    ssh  -R localhost:$localPort:127.0.0.1:$remotePort $remoteHost \"cd $remoteDir; bash .____rsyncSignal.sh --pwd $PWD --port $remotePort\"&\n    #tunnelPid=\"$!\"\n    # echo \"tunnelPid:$tunnelPid\"\n}\n\nlastSentinel=$(cat .____sentinel);\n\n# we always start from the local dir\ntrigger=local;\n# do a trial run to see if we'd delete files on the remote end\nwouldDeleteCount=$(rsync -anuzP --exclude \".*/\" --exclude \".____*\"  --exclude \"node_modules\" --delete . $remoteHost:$remoteDir/ | grep deleting | wc -l);\nwouldDeleteCount=\"$(echo -e \"${wouldDeleteCount}\" | tr -d '[:space:]')\"\n\nwouldDeleteRemoteFiles=$(rsync -anuzP --exclude \".*/\" --exclude \".____*\"  --exclude \"node_modules\" --delete . $remoteHost:$remoteDir/ | grep deleting);\nif [ ! -z \"$wouldDeleteRemoteFiles\" ];\nthen\n\n  unset destroyAhead\n  unset localFileCount\n  localFileCount=$(find . -type f | egrep -v '\\..+/' | egrep -v '\\./duplexRsync.sh' | egrep -v '\\./\\.____*' |  wc -l |  tr -d '[:space:]')\n  # if the local directory is empty using same pattern as rsync above we always merge\n  if [ \"$localFileCount\" -eq 0 ]\n  then\n    destroyAhead=\"merge\"\n  else\n    echo \"WOULD delete count: $wouldDeleteCount\"\n    echo \"$wouldDeleteRemoteFiles\"\n  fi\n\n  while ! [[ \"$destroyAhead\" =~ ^(destroy|merge|abort)$ ]]\n  do\n\n    if [ \"$wouldDeleteCount\" -gt 5 ]\n    then\n      major=\" ----MAJOR----- \";\n    fi\n\n    if [ \"$wouldDeleteCount\" -gt 42 ]\n    then\n      major=\" ----INTERSTELLAR BYPASS LEVEL----- \";\n    fi\n\n    echo \"ATTENTION $major DESTRUCTION  AHEAD: There is/are $wouldDeleteCount file(s) present in the remote folder that are not present locally. Could the remote folder be totally unrelated? Would you like to merge the folders by creeating these locally(merge),Sync and destroy(destroy) or abort?(merge/destroy/abort)\"\n    read destroyAhead\n  done\n  if [ \"$destroyAhead\" = \"abort\" ];\n  then\n    exit;\n  elif [ \"$destroyAhead\" = \"merge\" ];\n  then\n    # sync from remote without delete\n    rsync -auzP --exclude \".*/\" --exclude \".____*\"  --exclude \"node_modules\" \"$remoteHost:$remoteDir/\" .;\n  fi\nfi;\n\nduplex_rsync;fswatch -r -o . | while read f;\n  do\n    sentinel=$(cat .____sentinel);\n    echo \"sentinel $sentinel lastSentinel: $lastSentinel\"\n    sentinelInc=$((sentinel-lastSentinel));\n    # if the change is remote(incremented ____sentinel) lets slow down and wait to gobble multiple events\n    if [ $sentinelInc -gt 0 ]\n    then\n      echo 'remote change detected';\n      trigger=remote;\n      duplex_rsync;\n      sleep 3;\n    else\n      echo 'local change detected';\n      trigger=local;\n      duplex_rsync;\n    fi\n    lastSentinel=$sentinel;\n  done;\n"
  }
]