Repository: kmpm/nodemcu-uploader Branch: master Commit: 32187b04b757 Files: 37 Total size: 67.0 KB Directory structure: gitextract_j5fooql9/ ├── .editorconfig ├── .gitignore ├── .travis.yml ├── LICENSE ├── README.md ├── doc/ │ ├── DEVELOP.md │ ├── USAGE.md │ └── bash_completion.d/ │ └── nodemcu_uploader ├── nodemcu-uploader.py ├── nodemcu_uploader/ │ ├── __init__.py │ ├── __main__.py │ ├── exceptions.py │ ├── luacode.py │ ├── main.py │ ├── serialutils.py │ ├── term.py │ ├── uploader.py │ ├── utils.py │ ├── validate.py │ └── version.py ├── pylintrc ├── setup.cfg ├── setup.py ├── test_requirements.txt ├── tests/ │ ├── __init__.py │ ├── fixtures/ │ │ ├── big_file.txt │ │ ├── led_blink.lua │ │ ├── medium_file.txt │ │ ├── riazzerawifi.lua │ │ ├── signatur.tif │ │ ├── small_file.txt │ │ ├── testuploadfail.txt │ │ └── webserver.lua │ ├── misc.py │ ├── torture.py │ └── uploader.py └── tox.ini ================================================ FILE CONTENTS ================================================ ================================================ FILE: .editorconfig ================================================ root = true [*] end_of_line = lf insert_final_newline = true [*.py] indent_style = space indent_size = 4 charset = utf-8 trim_trailing_whitespace = true ================================================ FILE: .gitignore ================================================ # python-stuff *.egg-info/ .eggs dist/ build/ *.pyc __pycache__/ # testing .coverage env/ env3/ *.log tmp/ venv/ # editors .vscode/ .tox ================================================ FILE: .travis.yml ================================================ language: python python: - 3.6 - 3.7 # command to install dependencies install: "pip install -r test_requirements.txt" # command to run tests script: - coverage run setup.py test ================================================ FILE: LICENSE ================================================ The MIT License (MIT) Copyright (C) 2015-2020 Peter Magnusson Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: README.md ================================================ nodemcu-uploader.py =================== __Archival notice!!!__ This project is currently archived because of lack of time and interest. I no longer use NodeMCU in any form. But someone else might be. If you want to post a notice here about your active alternative then send me a DM here on github. --- A simple tool for uploading files to the filesystem of an ESP8266 running NodeMCU as well as some other useful commands. It should work on Linux, and OS X; and with any type of file that fits the filesystem, binary or text. | master | |--------| |[![Build Status](https://travis-ci.org/kmpm/nodemcu-uploader.svg?branch=master)](https://travis-ci.org/kmpm/nodemcu-uploader) | Please note that these tests is not complete and it might be the tests themselves that are having issues. Call for maintainers -------------------- Hi, This project is in need of maintenance and I (kmpm) do not have the time the project deserves. Look at https://github.com/kmpm/nodemcu-uploader/issues/90 for more information on what to do about it or email me@kmpm.se Installation ------------- Should be installable by PyPI (prefered) but there might be packaging issues still. pip install nodemcu-uploader nodemcu-uploader Otherwise clone from github and run directly from there git clone https://github.com/kmpm/nodemcu-uploader cd nodemcu-uploader python ./nodemcu-uploader.py Note that pip would install pyserial >= 2.7. The terminal command (using miniterm from pyserial) might not work depending on version used. This is a known issue. ### Notes for Windows There might be some [significant issues with Windows](https://github.com/kmpm/nodemcu-uploader/issues?q=is%3Aissue+is%3Aopen+label%3Aos%3Awindows). ### Notes for OS X To solve "ImportError: No module named serial", install the pyserial module: ```sh python easy_install pyserial ``` Usage ----- Download NodeMCU firmware from http://nodemcu-build.com/ . Since version v0.4.0 of the tool you will need a recent (june/july 2016) version of the firmware for nodemcu. The default baudrate was changed in firmware from 9600 to 115200 and this tool was changed as well. If you are using an older firmware you MUST use the option `--start-baud 9600` to the device to be recognized. Otherwise you will get a _Device not found or wrong port_ error. For more usage details see [USAGE.md in doc](doc/USAGE.md) Issues ------- When reporting issues please provide operating system (windows, mac, linux etc.), version of this tool `nodemcu-uploader --version` and version of the firmware on you device. If you got the firmware from http://nodemcu-build.com/ please tell if it was the dev or master branch and at what date it was created. As for firmware version I would like to have a dump of the details you get when connected using a terminal to the device at boot time. It would look something like this... ``` NodeMCU custom build by frightanic.com branch: master commit: b580bfe79e6e73020c2bd7cd92a6afe01a8bc867 SSL: false modules: crypto,file,gpio,http,mdns,mqtt,net,node,tmr,uart,wifi build built on: 2016-07-29 11:08 powered by Lua 5.1.4 on SDK 1.5.1(e67da894) ``` When you have as much of that as possible, create a issue in github, https://github.com/kmpm/nodemcu-uploader/issues Technical Details ----------------- This *almost* uses a implementation of xmodem protocol for the up-/download part. The main missing part is checksum and retransmission. This is made possible by first preparing the device by creating a set of helper functions using the ordinary terminal mode. These function utilize the built in uart module for the actual transfer and cuts up the transfers to a set of manageable blocks that are reassembled in the receiving end. ### Upload 1. Client calls the function recv() 2. NodeMCU disables echo and send a 'C' to tell that it's ready to receive data 3. Client sends a filename terminated with 0x00 4. NodeMCU sends ACK 5. Client send block of data according to the definition. 6. NodeMCU sends ACK 7. Step 5 and 6 are repeated until NodeMCU receives a block with 0 as size. 8. NodeMCU enables normal terminal again with echo ### Download 1. Client calls the function send(). 2. NodeMCU disables echo and waits for start. 2. Client send a 'C' to tell that it's ready to receive data 3. NodeMCU sends a filename terminated with 0x00 4. Client sends ACK 5. NodeMCU send block of data according to the definition. 6. Client sends ACK 7. Step 5 and 6 are repeated until client receives a block with 0 as size. 8. NodeMCU enables normal terminal again with echo. ### Data Block Definition __SOH__, __size__, __data[128]__ * SOH = 0x01 * Single byte telling how much of the 128 bytes data that are actually used. * Data padded with random bytes to fill out the 128 bytes frame. This gives a total 130 bytes per block. The block size was decided for... 1. Being close to xmodem from where the inspiration came 2. A fixed size allow the use of the uart.on('data') event very easy. 3. 130 bytes would fit in the receive buffer. 4. It would not waste that much traffic if the total size uploaded was not a even multiple of the allowed datasize. Disclaimer ----------- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: doc/DEVELOP.md ================================================ Develop and Test nodemcu-uploader ================================= Configure development environment ------- ```shell git clone https://github.com/kmpm/nodemcu-uploader cd nodemcu-uploader python3 -m venv . venv/bin/activate pip install -r test_requirements.txt pip install -e . ``` Testing ------- ```shell pip install -r test_requirements.txt coverage run setup.py test # or even better testing with tox tox ``` To run tests that actually communicate with a device you will need to set the __SERIALPORT__ environment variable to the port where you have an device connected. Linux ``` export SERIALPORT=/dev/ttyUSB0 ``` Publishing ---------- * https://packaging.python.org/tutorials/packaging-projects/ Please make sure to bump the version number in nodemcu_uploader/version.py as well as the testing of that number in tests/misc.py ```bash # python -m pip install --upgrade setuptools wheel twine python setup.py sdist bdist_wheel #test upload python -m twine upload -u __token__ --repository testpypi dist/* #real upload python -m twine upload -u __token__ dist/* ``` ================================================ FILE: doc/USAGE.md ================================================ Usage =================== This document is by no means complete. ## Common options * --help will show some help * --start_baud set at a default of 115200 (the speed of the nodemcu at boot in later versions of the firmware) * --baud are set at a default of 115200. This setting is used for transfers and such. * --port is by default __/dev/ttyUSB0__, __/dev/tty.SLAB_USBtoUART__ if on Mac and __COM1__ on Windows * the environment variable __SERIALPORT__ will override any default port Since version v0.4.0 of the tool you will need a recent (june/july 2016) version or later of the firmware for nodemcu. The default baudrate was changed in firmware from 9600 to 115200 and this tool was changed as well. Download a recent firmware from http://nodemcu-build.com/ . Since v0.2.1 the program works with 2 possible speeds. It connects at a default (--start_baud) of 115200 baud which is what the default firmware uses. Earlier versions of the firmware and this tool used 9600 as start baudrate. Immediately after first established communication it changes to a higher (--baud) speed, if neccesary, which defaults to 115200. This allows all communication to happen much faster without having to recompile the firmware or do any manual changes to the speed. When done and before it closes the port it changes the speed back to normal if it was changed. Since v0.4.0 of nodemcu-uploader it tries to use the auto-baudrate feature build in to the firmware by sending a character repetedly when initiating communication. This requires a firmware from june/july 2016 or later. ## Commands ### Upload From computer to esp device. ``` nodemcu-uploader upload init.lua ``` Uploading a number of files, but saving with a different file name. If you want an alternate destination name, just add a colon ":" and the new destination filename. ``` nodemcu-uploader upload init.lua:new_init.lua README.md:new_README.md ``` Uploading with wildcard and compiling to .lc when uploaded. ``` nodemcu-uploader upload lib/*.lua --compile ``` Uploading and verify successful uploading by downloading the file to RAM and comparing contents. ``` nodemcu-uploader.py upload init.lua --verify=raw ``` Uploading and verify successful uploading by calculating the sha1 checksum on the esp and compare it to the checksum of the original file. This requires the __crypto__ module in the firmware but it's more reliable than the _raw_ method. ``` nodemcu-uploader upload init.lua --verify=sha1 ``` ###Download From esp device to computer. Downloading a number of files. Supports multiple files. If you want an alternate destination name, just add a colon ":" and the new destination filename. ``` nodemcu-uploader download init.lua README.md nodemcu-uploader.py ``` Downloading a number of files, but saving with a different file name. ``` nodemcu-uploader download init.lua:new_init.lua README.md:new_README.md ``` ### List files Listing files, using serial port com1 on Windows ``` nodemcu-uploader --port com1 file list ``` ### Do (execute) a file `nodemcu-uploader file do runme.lua` This file has to exist on the device before, otherwise you will get an error. ### Print a file This will show the contents of an existing file. `nodemcu-uploader file print init.lua` ### Listing heap memory size `nodemcu-uploader node heap` ### Restarting the device `nodemcu-uploader node restart` ### Format filesystem ``` nodemcu-uploader file format ``` Note that this can take a long time depending on size of flash on the device. Even if the tool timeout waiting for response from the device it might have worked. The tool was just not waiting long enough. ### Remove specific files ``` nodemcu-uploader file remove foo.lua ``` ## Misc ### Setting default serial-port Using the environment variable `SERIALPORT` you can avoid having to type the `--port` option every time you use the tool. On Windows, if your devices was connected to COM3 this could be done like this. ```batch set SERIALPORT=com3 REM on all subsequent commands the default port `com3`would be assumed nodemcu-uploader file list ``` ================================================ FILE: doc/bash_completion.d/nodemcu_uploader ================================================ _nodemcu_uploader_remote_files() { nodemcu-uploader file list 2>&1 | awk '/\.lua\s+[0-9]+$/ { print $1 }' | tr "\n" ' ' } _nodemcu_uploader() { local cur prev opts COMPREPLY=() cur="${COMP_WORDS[COMP_CWORD]}" prev="${COMP_WORDS[COMP_CWORD-1]}" prev_prev="${COMP_WORDS[COMP_CWORD-2]}" opts="--help --verbose --version --port --baud --start_baud --timeout --autobaud_time" cmds="backup upload exec download file node terminal" node_options="heap restart" file_options="list do format remove print" if [[ ${cur} == -* ]] ; then COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) ) else case $prev in node ) COMPREPLY=( $(compgen -W "${node_options}" -- ${cur}) ) ;; file ) COMPREPLY=( $(compgen -W "${file_options}" -- ${cur}) ) ;; upload ) COMPREPLY=( $(compgen -f -- ${cur}) ) ;; download ) COMPREPLY=( $(compgen -W "$(_nodemcu_uploader_remote_files)" -- ${cur}) ) ;; *) if [[ ${prev_prev} == file ]] ; then case ${prev} in do | print | remove ) COMPREPLY=( $(compgen -W "$(_nodemcu_uploader_remote_files)" -- ${cur}) ) ;; esac else COMPREPLY=( $(compgen -W "${cmds}" -- ${cur}) ) fi esac fi return 0 } complete -F _nodemcu_uploader nodemcu-uploader ================================================ FILE: nodemcu-uploader.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # Copyright (C) 2015-2019 Peter Magnusson # pylint: disable=C0103 """makes it easier to run nodemcu-uploader from command line""" from nodemcu_uploader import main if __name__ == '__main__': main.main_func() ================================================ FILE: nodemcu_uploader/__init__.py ================================================ #!/usr/bin/env python # -*- coding: utf-8 -*- # Copyright (C) 2015-2019 Peter Magnusson """Library and util for uploading files to NodeMCU version 0.9.4 and later""" from .version import __version__ # noqa: F401 from .uploader import Uploader # noqa: F401 ================================================ FILE: nodemcu_uploader/__main__.py ================================================ from .main import main_func if __name__ == '__main__': main_func() ================================================ FILE: nodemcu_uploader/exceptions.py ================================================ # -*- coding: utf-8 -*- # Copyright (C) 2015-2019 Peter Magnusson # pylint: disable=C0111 """Various custom exceptions""" class CommunicationTimeout(Exception): def __init__(self, message, buf): super(CommunicationTimeout, self).__init__(message) self.buf = buf class BadResponseException(Exception): def __init__(self, message, expected, actual): message = message + ' expected:`%s` != actual: `%s`' % (expected, actual) super(BadResponseException, self).__init__(message) self.expected = expected self.actual = actual class NoAckException(Exception): pass class DeviceNotFoundException(Exception): pass class VerificationError(Exception): pass class PathLengthException(Exception): pass class ValidationException(Exception): def __init__(self, message, key, value): message = "Validation Exception. {key} was {message}. '{value}'".format(message=message, key=key, value=value) super(ValidationException, self).__init__(message) ================================================ FILE: nodemcu_uploader/luacode.py ================================================ # -*- coding: utf-8 -*- """This module contains all the LUA code that needs to be on the device to perform whats needed. They will be uploaded if they doesn't exist""" # Copyright (C) 2015-2019 Peter Magnusson # pylint: disable=C0301 # flake8: noqa LUA_FUNCTIONS = ['recv_block', 'recv_name', 'recv', 'shafile', 'send_block', 'send_file', 'send'] DOWNLOAD_FILE = "file.open('{filename}') print(file.seek('end', 0)) file.seek('set', {bytes_read}) uart.write(0, file.read({chunk_size}))file.close()" PRINT_FILE = "file.open('{filename}') print('---{filename}---') print(file.read()) file.close() print('---')" INFO_GROUP = "for key,value in pairs(node.info('{group}')) do k=tostring(key) print(k .. string.rep(' ', 20 - #k), tostring(value)) end" LIST_FILES = 'for key,value in pairs(file.list()) do print(key,value) end' # NUL = \000, ACK = \006 RECV_LUA = \ r""" function recv() local on,w,ack,nack=uart.on,uart.write,'\6','\21' local fd local function recv_block(d) local t,l = d:byte(1,2) if t ~= 1 then w(0, nack); fd:close(); return on('data') end if l >= 0 then fd:write(d:sub(3, l+2)); end if l == 0 then fd:close(); w(0, ack); return on('data') else w(0, ack) end end local function recv_name(d) d = d:gsub('%z.*', '') d:sub(1,-2) file.remove(d) fd=file.open(d, 'w') on('data', 130, recv_block, 0) w(0, ack) end on('data', '\0', recv_name, 0) w(0, 'C') end function shafile(f) print(crypto.toHex(crypto.fhash('sha1', f))) end """ # noqa: E122 SEND_LUA = \ r""" function send(f) uart.on('data', 1, function (data) local on,w=uart.on,uart.write local fd local function send_block(d) l = string.len(d) w(0, '\001' .. string.char(l) .. d .. string.rep('\0', 128 - l)) return l end local function send_file(f) local s, p fd=file.open(f) s=fd:seek('end', 0) p=0 on('data', 1, function(data) if data == '\006' and p """This module is the cli for the Uploader class""" from __future__ import print_function import argparse import logging import os import sys import glob import serial from .uploader import Uploader from .term import terminal from serial import VERSION as serialversion from .version import __version__ log = logging.getLogger(__name__) # pylint: disable=C0103 def destination_from_source(sources, use_glob=True): """ Split each of the sources in the array on ':' First part will be source, second will be destination. Modifies the the original array to contain only sources and returns an array of destinations. """ destinations = [] newsources = [] for i in range(0, len(sources)): srcdst = sources[i].split(':') if len(srcdst) == 2: destinations.append(srcdst[1]) newsources.append(srcdst[0]) # proper list assignment else: if use_glob: listing = glob.glob(srcdst[0]) for filename in listing: newsources.append(filename) # always use forward slash at destination destinations.append(filename.replace('\\', '/')) else: newsources.append(srcdst[0]) destinations.append(srcdst[0]) return [newsources, destinations] def operation_upload(uploader, sources, verify, do_compile, do_file, do_restart): """The upload operation""" if not isinstance(sources, list): sources = [sources] sources, destinations = destination_from_source(sources) if len(destinations) == len(sources): if uploader.prepare(): for filename, dst in zip(sources, destinations): if do_compile: uploader.file_remove(os.path.splitext(dst)[0]+'.lc') if not os.path.exists(filename) and not os.path.isfile(filename): raise Exception("File does not exist. {filename}".format(filename=filename)) uploader.write_file(filename, dst, verify) # init.lua is not allowed to be compiled if do_compile and dst != 'init.lua': uploader.file_compile(dst) uploader.file_remove(dst) if do_file: uploader.file_do(os.path.splitext(dst)[0]+'.lc') elif do_file: uploader.file_do(dst) else: raise Exception('Error preparing nodemcu for reception') else: raise Exception('You must specify a destination filename for each file you want to upload.') if do_restart: uploader.node_restart() log.info('All done!') return destinations def operation_download(uploader, sources, *args, **kwargs): """The download operation""" sources, destinations = destination_from_source(sources, False) # print('sources', sources) # print('destinations', destinations) dest = kwargs.pop('dest', '') if len(destinations) == len(sources): if uploader.prepare(): for filename, dst in zip(sources, destinations): dst = os.path.join(dest, dst) uploader.read_file(filename, dst) else: raise Exception('You must specify a destination filename for each file you want to download.') log.info('All done!') def operation_list_files(uploader): """List file on target""" files = uploader.file_list() for f in files: log.info("{file:30s} {size}".format(file=f[0], size=f[1])) def operation_file(uploader, cmd, filename=''): """File operations""" if cmd == 'list': operation_list_files(uploader) if cmd == 'do': for path in filename: uploader.file_do(path) elif cmd == 'format': uploader.file_format() elif cmd == 'remove': for path in filename: uploader.file_remove(path) elif cmd == 'print': for path in filename: uploader.file_print(path) elif cmd == 'remove_all': uploader.file_remove_all() def operation_port(args): if args.cmd == 'list': ports = serial.tools.list_ports.comports(include_links=False) print('device', 'vid', 'pid') for p in ports: print(p.device, p.vid, p.pid) def arg_auto_int(value): """parsing function for integer arguments""" return int(value, 0) def main_func(): """Main function for cli""" parser = argparse.ArgumentParser( description='NodeMCU Lua file uploader', prog='nodemcu-uploader' ) parser.add_argument( '--verbose', help='verbose output', action='store_true', default=False) parser.add_argument( '--silent', help='silent output. Errors and worse', action='store_true', default=False) parser.add_argument( '--version', help='prints the version and exists', action='version', version='%(prog)s {version} (serial {serialversion}, python {pv})'.format( version=__version__, serialversion=serialversion, pv=sys.version) ) parser.add_argument( '--port', '-p', help='Serial port device', default=Uploader.PORT) parser.add_argument( '--baud', '-b', help='Serial port baudrate', type=arg_auto_int, default=Uploader.BAUD) parser.add_argument( '--start_baud', '-B', help='Initial Serial port baudrate', type=arg_auto_int, default=Uploader.START_BAUD) parser.add_argument( '--timeout', '-t', help='Timeout for operations', type=arg_auto_int, default=Uploader.TIMEOUT) parser.add_argument( '--autobaud_time', '-a', help='Duration of the autobaud timer', type=float, default=Uploader.AUTOBAUD_TIME, ) subparsers = parser.add_subparsers( dest='operation', help='Run nodemcu-uploader {command} -h for additional help') backup_parser = subparsers.add_parser( 'backup', help='Backup all the files on the nodemcu board') backup_parser.add_argument('path', help='Folder where to store the backup') upload_parser = subparsers.add_parser( 'upload', help='Path to one or more files to be uploaded. Destination name will be the same as the file name.') upload_parser.add_argument( 'filename', nargs='+', help='Lua file to upload. Use colon to give alternate destination.' ) upload_parser.add_argument( '--compile', '-c', help='If file should be uploaded as compiled', action='store_true', default=False ) upload_parser.add_argument( '--verify', '-v', help='To verify the uploaded data.', action='store', nargs='?', choices=['none', 'raw', 'sha1'], default='none' ) upload_parser.add_argument( '--dofile', '-e', help='If file should be run after upload.', action='store_true', default=False ) upload_parser.add_argument( '--restart', '-r', help='If esp should be restarted', action='store_true', default=False ) exec_parser = subparsers.add_parser( 'exec', help='Path to one or more files to be executed line by line.') exec_parser.add_argument('filename', nargs='+', help='Lua file to execute.') download_parser = subparsers.add_parser( 'download', help='Path to one or more files to be downloaded. Destination name will be the same as the file name.') download_parser.add_argument( 'filename', nargs='+', help='Lua file to download. Use colon to give alternate destination.') file_parser = subparsers.add_parser( 'file', help='File functions') file_parser.add_argument( 'cmd', choices=('list', 'do', 'format', 'remove', 'print', 'remove_all'), help="""list=list files, do=dofile given path, format=formate file area, remove=remove given path, remove_all=delete all files""") file_parser.add_argument('filename', nargs='*', help='path for cmd') node_parse = subparsers.add_parser( 'node', help='Node functions') node_parse.add_argument( 'ncmd', choices=('heap', 'restart', 'info'), help="heap=print heap memory, restart=restart nodemcu, info=show node info") subparsers.add_parser( 'terminal', help='Run pySerials miniterm' ) port_parser = subparsers.add_parser( 'port', help='serial port stuff' ) port_parser.add_argument( 'cmd', choices=('list',) ) args = parser.parse_args() default_level = logging.INFO if args.silent: default_level = logging.ERROR if args.verbose: default_level = logging.DEBUG # formatter = logging.Formatter('%(message)s') logging.basicConfig(level=default_level, format='%(message)s') if args.operation == 'terminal': # uploader can not claim the port terminal(args.port, str(args.start_baud)) return elif args.operation == 'port': operation_port(args) return # let uploader user the default (short) timeout for establishing connection uploader = Uploader(args.port, args.baud, start_baud=args.start_baud, autobaud_time=args.autobaud_time) # and reset the timeout (if we have the uploader&timeout) if args.timeout: uploader.set_timeout(args.timeout) if args.operation == 'upload': operation_upload(uploader, args.filename, args.verify, args.compile, args.dofile, args.restart) elif args.operation == 'download': operation_download(uploader, args.filename) elif args.operation == 'exec': sources = args.filename for path in sources: uploader.exec_file(path) elif args.operation == 'file': operation_file(uploader, args.cmd, args.filename) elif args.operation == 'node': if args.ncmd == 'heap': uploader.node_heap() elif args.ncmd == 'restart': uploader.node_restart() elif args.ncmd == 'info': uploader.node_info() elif args.operation == 'backup': uploader.backup(args.path) # no uploader related commands after this point uploader.close() ================================================ FILE: nodemcu_uploader/serialutils.py ================================================ # -*- coding: utf-8 -*- # Copyright (C) 2015-2019 Peter Magnusson from platform import system from os import environ from serial.tools import list_ports def default_port(sysname=system(), detect=True): """This returns the default port used for different systems if SERIALPORT env variable is not set""" system_default = { 'Windows': 'COM1', 'Darwin': '/dev/tty.SLAB_USBtoUART' }.get(sysname, '/dev/ttyUSB0') # if SERIALPORT is set then don't even waste time detecting ports if 'SERIALPORT' not in environ and detect: try: ports = list_ports.comports(include_links=False) if len(ports) == 1: return ports[0].device else: # clever guessing, sort of # vid/pid # 4292/60000 adafruit huzzah for p in ports: if p.vid == 4292 and p.pid == 60000: return p.device # use last port as fallback return ports[-1].device except Exception: pass return environ.get('SERIALPORT', system_default) ================================================ FILE: nodemcu_uploader/term.py ================================================ # -*- coding: utf-8 -*- """Piggyback on pyserial terminal""" from serial.tools import miniterm import sys from .serialutils import default_port def terminal(port=default_port(), baud='9600'): """Launch minterm from pyserial""" testargs = ['nodemcu-uploader', port, baud] # TODO: modifying argv is no good sys.argv = testargs # resuse miniterm on main function miniterm.main() ================================================ FILE: nodemcu_uploader/uploader.py ================================================ # -*- coding: utf-8 -*- # Copyright (C) 2015-2019 Peter Magnusson """Main functionality for nodemcu-uploader""" # Not sure about it, because UnicodeEncodeError throws anyway # from __future__ import unicode_literals import time import logging import hashlib import os import errno import serial from . import validate from .serialutils import default_port from .exceptions import CommunicationTimeout, DeviceNotFoundException, \ BadResponseException, VerificationError, NoAckException from .utils import system, hexify, from_file, ENCODING from .luacode import RECV_LUA, SEND_LUA, LUA_FUNCTIONS, \ LIST_FILES, UART_SETUP, PRINT_FILE, INFO_GROUP, REMOVE_ALL_FILES log = logging.getLogger(__name__) # pylint: disable=C0103 __all__ = ['Uploader', 'default_port'] SYSTEM = system() MINIMAL_TIMEOUT = 0.001 BLOCK_START = b'\x01' NUL = b'\x00' ACK = b'\x06' class Uploader(object): """Uploader is the class for communicating with the nodemcu and that will allow various tasks like uploading files, formating the filesystem etc. """ BAUD = 115200 START_BAUD = 115200 TIMEOUT = 5 AUTOBAUD_TIME = 0.3 PORT = default_port() def __init__(self, port=PORT, baud=BAUD, start_baud=START_BAUD, timeout=TIMEOUT, autobaud_time=AUTOBAUD_TIME): self._timeout = Uploader.TIMEOUT self.set_timeout(timeout) log.info('opening port %s with %s baud', port, start_baud) if port == 'loop://': self._port = serial.serial_for_url(port, start_baud, timeout=timeout) else: self._port = serial.Serial(port, start_baud, timeout=timeout) # black magic aka proxifying # self._port = wrap(self._port) self.start_baud = start_baud self.baud = baud self.autobaud_time = autobaud_time # Keeps things working, if following connections are made: # RTS = CH_PD (i.e reset) # DTR = GPIO0 self._port.setRTS(False) self._port.setDTR(False) def __sync(): """Get in sync with LUA (this assumes that NodeMCU gets reset by the previous two lines)""" log.debug('getting in sync with LUA') self.__clear_buffers() try: self.__writeln('UUUUUUUUUUUU') # Send enough characters for auto-baud self.__clear_buffers() time.sleep(self.autobaud_time) # Wait for autobaud timer to expire self.__exchange(';') # Get a defined state self.__writeln('print("%sync%");') self.__expect('%sync%\r\n> ') except CommunicationTimeout: raise DeviceNotFoundException('Device not found or wrong port') __sync() if baud != start_baud: self.__set_baudrate(baud) # Get in sync again __sync() self.line_number = 0 def __set_baudrate(self, baud): """setting baudrate if supported""" log.info('Changing communication to %s baud', baud) self.__writeln(UART_SETUP.format(baud=baud)) # Wait for the string to be sent before switching baud time.sleep(0.1) try: self._port.setBaudrate(baud) except AttributeError: # pySerial 2.7 self._port.baudrate = baud def set_timeout(self, timeout): """Set the timeout for the communication with the device.""" timeout = int(timeout) # will raise on Error self._timeout = timeout == 0 and 999999 or timeout def __clear_buffers(self): """Clears the input and output buffers""" try: self._port.reset_input_buffer() self._port.reset_output_buffer() except AttributeError: # pySerial 2. self._port.flushInput() self._port.flushOutput() def __expect(self, exp='> ', timeout=None): """will wait for exp to be returned from nodemcu or timeout. Will use utils.ENCODING for encoding if not bytes. """ timeout_before = self._port.timeout timeout = timeout or self._timeout # do NOT set timeout on Windows if SYSTEM != 'Windows': # Checking for new data every 100us is fast enough if self._port.timeout != MINIMAL_TIMEOUT: self._port.timeout = MINIMAL_TIMEOUT if not isinstance(exp, bytes): exp = bytes(exp, ENCODING) end = time.time() + timeout # Finish as soon as either exp matches or we run out of time (work like dump, but faster on success) data = bytes() while not data.endswith(exp) and time.time() <= end: data += self._port.read() # msg = data.decode(ENCODING, 'ignore') now = time.time() log.debug('expect returned: `{0}`. wants: {1}'.format(data, exp)) if now > end: raise CommunicationTimeout('Timeout waiting for data', data) if not data.endswith(exp) and len(exp) > 0: raise BadResponseException('Bad response.', exp, data) if SYSTEM != 'Windows': self._port.timeout = timeout_before return str(data, ENCODING) def __write(self, output, binary=False): """write data on the nodemcu port. Strings will be converted to bytes using utils.ENCODING. If 'binary' is True the debug log will show the intended output as hex, otherwise as string""" if not binary: log.debug('write: %s', output) else: log.debug('write binary: %s', hexify(output)) if isinstance(output, str): output = bytes(output, ENCODING) self._port.write(output) self._port.flush() def __writeln(self, output): """write, with linefeed""" self.__write(output + '\n') def __exchange(self, output, timeout=None): """Write output to the port and wait for response Expects a str as input""" if not isinstance(output, str): raise TypeError("output should be a str") self.__writeln(output) self._port.flush() return self.__expect(timeout=timeout or self._timeout) def close(self): """restores the nodemcu to default baudrate and then closes the port""" try: if self.baud != self.start_baud: self.__set_baudrate(self.start_baud) self._port.flush() self.__clear_buffers() except serial.serialutil.SerialException: pass log.debug('closing port') self._port.close() def prepare(self): """ This uploads the protocol functions nessecary to do binary chunked transfer """ log.info('Preparing esp for transfer.') for func in LUA_FUNCTIONS: detected = self.__exchange('print({0})'.format(func)) if detected.find('function:') == -1: break else: log.info('Preparation already done. Not adding functions again.') return True functions = RECV_LUA + '\n' + SEND_LUA data = functions.format(baud=self._port.baudrate) # change any \r\n to just \n and split on that lines = data.replace('\r', '').split('\n') # remove some unneccesary spaces to conserve some bytes # TODO: a good minifier for lua for line in lines: line = line.strip().replace(', ', ',').replace(' = ', '=') if len(line) == 0: continue resp = self.__exchange(line) # do some basic test of the result if ('unexpected' in resp) or ('stdin' in resp) or len(resp) > len(functions)+10: log.error('error when preparing "%s"', resp) return False return True def download_file(self, filename): """Download a file from device to RAM Return 'bytes' of the full content """ validate.remotePath(filename) res = self.__exchange('send("{filename}")'.format(filename=filename)) if ('unexpected' in res) or ('stdin' in res): log.error('Unexpected error downloading file: %s', res) raise Exception('Unexpected error downloading file') # tell device we are ready to receive self.__write('C') # we should get a NUL terminated filename to start with sent_filename = self.__expect(NUL).strip() log.info('receiveing ' + sent_filename) # ACK to start download self.__write(ACK, True) buf = bytes() data = bytes() chunk, buf = self.__read_chunk(buf) # read chunks until we get an empty which is the end while len(chunk) > 0: self.__write(ACK, True) data = data + chunk chunk, buf = self.__read_chunk(buf) return data def read_file(self, filename, destination=''): """Downloading data from remote device into local file using the transfer protocol. """ if not destination: destination = filename log.info('Transferring %s to %s', filename, destination) data = self.download_file(filename) # Just in case, the filename may contain folder, so create it if needed. log.info(destination) dirpath1 = os.path.dirname(destination) if len(dirpath1) > 0 and not os.path.exists(dirpath1): try: os.makedirs(os.path.dirname(destination)) except OSError as e: # Guard against race condition if e.errno != errno.EEXIST: raise with open(destination, 'wb') as fil: try: fil.write(data) except Exception as e: log.error("Unexpected error writing file", e) raise def write_file(self, path, destination='', verify='none'): """Uploads a file to the remote device using the transfer protocol""" filename = os.path.basename(path) if not destination: destination = filename validate.remotePath(destination) log.info('Transferring %s as %s', path, destination) self.__writeln("recv()") res = self.__expect('C> ') if not res.endswith('C> '): log.error('Error waiting for esp "%s"', res) raise CommunicationTimeout('Error waiting for device to start receiving', res) log.debug('sending destination filename "%s"', destination) self.__write(destination + '\x00', True) if not self.__got_ack(): log.error('did not ack destination filename') raise NoAckException('Device did not ACK destination filename') content = from_file(path) log.debug('sending %d bytes in %s', len(content), filename) pos = 0 chunk_size = 128 while pos < len(content): rest = len(content) - pos if rest > chunk_size: rest = chunk_size data = content[pos:pos+rest] if not self.__write_chunk(data): resp = self.__expect() log.error('Bad chunk response "%s" %s', resp, hexify(resp)) raise BadResponseException('Bad chunk response', ACK, resp) pos += chunk_size log.debug('sending zero block') # zero size block self.__write_chunk() if verify != 'none': self.verify_file(path, destination, verify) def verify_file(self, local, remote, verify='none'): """Tries to verify if local has same checksum as remote. Valid options for verify is 'raw', 'sha1' or 'none' """ # get the local file contents self.__writeln(';') self.__expect('> ') content = from_file(local) log.info('Verifying using %s...' % verify) if verify == 'raw': data = self.download_file(remote) if content != data: log.error('Raw verification failed.') raise VerificationError('Verification failed.') else: log.info('Verification successful. Contents are identical.') elif verify == 'sha1': # Calculate SHA1 on remote file. Extract just hash from result data = self.__exchange('shafile("'+remote+'")').splitlines()[1] log.info('Remote SHA1: %s', data) # Calculate hash of local data filehashhex = hashlib.sha1(content).hexdigest() log.info('Local SHA1: %s', filehashhex) if data != filehashhex: log.error('SHA1 verification failed.') raise VerificationError('SHA1 Verification failed.') else: log.info('Verification successful. Checksums match') elif verify != 'none': raise Exception(verify + ' is not a valid verification method.') def exec_file(self, path): """execute the lines in the local file 'path'""" filename = os.path.basename(path) log.info('Execute %s', filename) content = from_file(path).replace('\r', '').split('\n') res = '> ' for line in content: line = line.rstrip('\n') retlines = (res + self.__exchange(line)).splitlines() # Log all but the last line res = retlines.pop() for lin in retlines: log.info(lin) # last line log.info(res) def __got_ack(self): """Returns true if ACK is received""" log.debug('waiting for ack') res = self._port.read(1) acked = res == ACK log.debug('ack read %s, comparing with %s. %s', hexify(res), hexify(ACK), acked) return acked def write_lines(self, data): """write lines, one by one, separated by \n to device""" lines = data.replace('\r', '').split('\n') for line in lines: self.__exchange(line) def __write_chunk(self, chunk=bytes()): """formats and sends a chunk of data to the device according to transfer protocol. Return result of ack check""" if not isinstance(chunk, bytes): raise TypeError() log.debug('writing %d bytes chunk', len(chunk)) data = BLOCK_START + bytes([len(chunk)]) + chunk if len(chunk) < 128: padding = 128 - len(chunk) log.debug('pad with %d characters', padding) data = data + (b'\x00' * padding) log.debug("packet size %d", len(data)) self.__write(data) self._port.flush() return self.__got_ack() def __read_chunk(self, buf): """Read a chunk of data""" log.debug('reading chunk') timeout_before = self._port.timeout if SYSTEM != 'Windows': # Checking for new data every 100us is fast enough if self._port.timeout != MINIMAL_TIMEOUT: self._port.timeout = MINIMAL_TIMEOUT end = time.time() + timeout_before if not isinstance(buf, bytes): raise Exception('Buffer is not instance of "bytes"') while len(buf) < 130 and time.time() <= end: r = self._port.read() if not isinstance(r, bytes): raise Exception('r is not instance of "bytes" is {t}'.format(t=type(r).__name__)) buf = buf + r if buf[0] != ord(BLOCK_START) or len(buf) < 130: log.debug('buffer binary: %s ', hexify(buf)) raise Exception('Bad blocksize or start byte') # else: # log.debug('buf binary: %s', hexify(buf)) if SYSTEM != 'Windows': self._port.timeout = timeout_before chunk_size = buf[1] data = buf[2:chunk_size+2] buf = buf[130:] return (data, buf) def file_list(self): """list files on the device""" log.info('Listing files') res = self.__exchange(LIST_FILES) res = res.split('\r\n') # skip first and last lines res = res[1:-1] files = [] for line in res: files.append(line.split('\t')) return files def file_do(self, filename): """Execute a file on the device using 'do'""" log.info('Executing '+filename) res = self.__exchange('dofile("'+filename+'")') log.info(res) return res def file_format(self): """Formats device filesystem""" log.info('Formating, can take minutes depending on flash size...') res = self.__exchange('file.format()', timeout=300) if 'format done' not in res: log.error(res) else: log.info(res) return res def file_print(self, filename): """Prints a file on the device to console""" log.info('Printing ' + filename) res = self.__exchange(PRINT_FILE.format(filename=filename)) log.info(res) return res def file_remove_all(self): log.info('Removing all files!!!') res = self.__exchange(REMOVE_ALL_FILES) log.info(res) return res def node_heap(self): """Show device heap size""" log.info('Heap') res = self.__exchange('print(node.heap())') log.info(res) return int(res.split('\r\n')[1]) def node_restart(self): """Restarts device""" log.info('Restart') res = self.__exchange('node.restart()') log.info(res) return res def node_info(self): """Node info""" log.info('Node info') res = self.node_info_group('hw') res += self.node_info_group('sw_version') res += self.node_info_group('build_config') return res def node_info_group(self, group): log.info('Node info %s', group) res = self.__exchange(INFO_GROUP.format(group=group)) log.info(res) return res def file_compile(self, path): """Compiles a file specified by path on the device""" log.info('Compile '+path) cmd = 'node.compile("%s")' % path res = self.__exchange(cmd) log.info(res) return res def file_remove(self, path): """Removes a file on the device""" log.info('Remove '+path) cmd = 'file.remove("%s")' % path res = self.__exchange(cmd) log.info(res) return res def backup(self, path): """Backup all files from the device""" log.info('Backing up in '+path) # List file to backup files = self.file_list() # then download each of then self.prepare() for f in files: self.read_file(f[0], os.path.join(path, f[0])) ================================================ FILE: nodemcu_uploader/utils.py ================================================ # -*- coding: utf-8 -*- # Copyright (C) 2015-2019 Peter Magnusson """Various utility functions""" from platform import system # from wrapt import ObjectProxy from sys import version_info __all__ = ['system', 'hexify', 'from_file', 'PY2', 'ENCODING'] PY2 = version_info.major == 2 if PY2: raise Exception("Python 2 is no longer supported") ENCODING = 'latin1' def to_hex(x): if isinstance(x, int): return hex(x) return hex(ord(x)) def hexify(byte_arr): if isinstance(byte_arr, int): return to_hex(byte_arr)[2:] else: return ':'.join((to_hex(x)[2:] for x in byte_arr)) def from_file(path): """Returns content of file as 'bytes'. """ with open(path, 'rb') as f: content = f.read() return content ================================================ FILE: nodemcu_uploader/validate.py ================================================ from .exceptions import ValidationException MAX_FS_NAME_LEN = 31 def remotePath(path): """Do various checks on the remote file name like max length. Raises exception if not valid """ if len(path) > MAX_FS_NAME_LEN: raise ValidationException('To long. >{0}'.format(MAX_FS_NAME_LEN), 'path', path) if len(path) < 1: raise ValidationException('To short', 'path', path) ================================================ FILE: nodemcu_uploader/version.py ================================================ # -*- coding: utf-8 -*- # Copyright (C) 2015-2019 Peter Magnusson """just keeper of current version""" # TODO: remember to update tests when version changes __version__ = '1.0.0' ================================================ FILE: pylintrc ================================================ [MASTER] persistent=yes [FORMAT] # Maximum number of characters on a single line. max-line-length=120 [DESIGN] max-args=6 max-statements=70 [MESSAGES CONTROL] disable=I0011 [REPORTS] msg-template={path}:{line}: [{msg_id}({symbol}), {obj}] {msg} include-ids=yes ================================================ FILE: setup.cfg ================================================ [metadata] description_file=README.md licence=LICENSE ================================================ FILE: setup.py ================================================ #!/usr/bin/python # -*- coding: utf-8 -*- # Copyright (C) 2015-2020 Peter Magnusson """Setup for nodemcu-uploader""" from setuptools import setup exec(open('nodemcu_uploader/version.py').read()) # pylint: disable=W0122 with open("README.md", "r") as fh: long_description = fh.read() setup(name='nodemcu-uploader', version=__version__, # noqa: F821 python_requires='>=3.5', install_requires=[ 'pyserial>=3.4' ], packages=['nodemcu_uploader'], # package_dir={'nodemcu_uploader': 'lib'}, url='https://github.com/kmpm/nodemcu-uploader', author='kmpm', author_email='me@kmpm.se', description='tool for uploading files to the filesystem of an ESP8266 running NodeMCU.', long_description=long_description, long_description_content_type="text/markdown", keywords=['esp8266', 'upload', 'nodemcu'], classifiers=[ 'Development Status :: 4 - Beta', 'Intended Audience :: Developers', 'Programming Language :: Python :: 3' ], license='MIT', test_suite="tests.get_tests", entry_points={ 'console_scripts': [ 'nodemcu-uploader=nodemcu_uploader.main:main_func' ] }, zip_safe=False, ) ================================================ FILE: test_requirements.txt ================================================ pyserial==3.4 coverage==4.0.3 flake8==3.7.9 ================================================ FILE: tests/__init__.py ================================================ # -*- coding: utf-8 -*- # Copyright (C) 2015-2019 Peter Magnusson """Add tests to include here""" import unittest import logging def get_tests(): """returns the tests to run""" return full_suite() def full_suite(): """creates a full suite of tests""" logging.basicConfig(filename='test.log', level=logging.INFO, format='%(asctime)s %(levelname)s %(module)s.%(funcName)s %(message)s') from .misc import MiscTestCase from . import uploader # from .serializer import ResourceTestCase as SerializerTestCase # from .utils import UtilsTestCase miscsuite = unittest.TestLoader().loadTestsFromTestCase(MiscTestCase) uploadersuite = unittest.TestLoader().loadTestsFromModule(uploader) return unittest.TestSuite([miscsuite, uploadersuite]) ================================================ FILE: tests/fixtures/big_file.txt ================================================ xyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyz ================================================ FILE: tests/fixtures/led_blink.lua ================================================ lighton=0 tmr.alarm(0,1000,1,function() if lighton==0 then lighton=1 led(512,512,512) -- 512/1024, 50% duty cycle else lighton=0 led(0,0,0) end end) ================================================ FILE: tests/fixtures/medium_file.txt ================================================ xyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyz 1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890 ================================================ FILE: tests/fixtures/riazzerawifi.lua ================================================ --riazzerawifi.lua -- Starts the portal to choose the wi-fi router to which we have -- to associate wifi.sta.disconnect() wifi.setmode(wifi.STATIONAP) --ESP SSID generated wiht its chipid wifi.ap.config({ssid="Mynode-"..node.chipid() , auth=wifi.OPEN}) enduser_setup.manual(true) enduser_setup.start( function() print("Connected to wifi as:" .. wifi.sta.getip()) end, function(err, str) print("enduser_setup: Err #" .. err .. ": " .. str) end ); ================================================ FILE: tests/fixtures/small_file.txt ================================================ xyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyz ================================================ FILE: tests/fixtures/testuploadfail.txt ================================================ if (anything == false and test > 10) then print ("This line should be fine") end -- Doesn't matter what is here, just needs to be enough for another block after the > sign -- Lorem Ipsum dolor sit amet consecutor adlipsing. Lorem Ipsum dolor sit amet consecutor adlipsing. -- Lorem Ipsum dolor sit amet consecutor adlipsing.Lorem Ipsum dolor sit amet consecutor adlipsing. -- Lorem Ipsum dolor sit amet consecutor adlipsing.Lorem Ipsum dolor sit amet consecutor adlipsing. ================================================ FILE: tests/fixtures/webserver.lua ================================================ -- webserver.lua --webserver sample from the nodemcu github if srv~=nil then srv:close() end gpio.mode(1, gpio.OUTPUT) srv=net.createServer(net.TCP) srv:listen(80,function(conn) conn:on("receive", function(client,request) local buf = "" local _, _, method, path, vars = string.find(request, "([A-Z]+) (.+)?(.+) HTTP") if(method == nil)then _, _, method, path = string.find(request, "([A-Z]+) (.+) HTTP") end local _GET = {} if (vars ~= nil)then for k, v in string.gmatch(vars, "(%w+)=(%w+)&*") do _GET[k] = v end end buf = buf.."

Hello, NodeMcu.

Turn PIN1
" client:send(buf) end) conn:on("sent", function (c) c:close() end) end) ================================================ FILE: tests/misc.py ================================================ # -*- coding: utf-8 -*- # Copyright (C) 2015-2019 Peter Magnusson import unittest from nodemcu_uploader.serialutils import default_port from nodemcu_uploader import __version__ import os from nodemcu_uploader import validate, exceptions class MiscTestCase(unittest.TestCase): def test_version(self): self.assertEqual(__version__, '1.0.0') def test_default_port(self): if os.environ.get('SERIALPORT', 'none') != 'none': # SERIALPORT should override any system defaults self.assertEqual(default_port(), os.environ['SERIALPORT']) else: # Test as if it were given system self.assertEqual(default_port('Linux', False), '/dev/ttyUSB0') self.assertEqual(default_port('Windows', False), 'COM1') self.assertEqual(default_port('Darwin', False), '/dev/tty.SLAB_USBtoUART') def test_remote_path_validation(self): validate.remotePath("test/something/maximum/len.jpeg") validate.remotePath("a") def v(p): validate.remotePath(p) self.assertRaises(exceptions.ValidationException, (lambda: v("test/something/maximum/leng.jpeg"))) self.assertRaises(exceptions.ValidationException, (lambda: v(""))) ================================================ FILE: tests/torture.py ================================================ # -*- coding: utf-8 -*- import unittest import logging import time import os from nodemcu_uploader import Uploader from nodemcu_uploader.main import operation_download, operation_upload import shutil log = logging.getLogger(__name__) logging.basicConfig( filename='test.log', level=logging.INFO, format='%(asctime)s %(levelname)s %(module)s.%(funcName)s %(message)s') LOOPPORT = 'loop://' # on which port should the tests be performed SERIALPORT = os.environ.get('SERIALPORT', LOOPPORT) def is_real(): if SERIALPORT.strip() == '': return False return str(SERIALPORT) != str(LOOPPORT) @unittest.skipUnless(is_real(), 'Needs a configured SERIALPORT') class TestTorture(unittest.TestCase): uploader = None def setUp(self): log.info("setUp") self.uploader = Uploader(SERIALPORT) def tearDown(self): log.info("tearDown") if is_real(): self.uploader.node_restart() self.uploader.close() time.sleep(1) def task_upload_verify_compile(self): """Upload lua code, verify and compile""" log.info('upload-verify-compile') self.assertTrue(self.uploader.prepare()) dests = operation_upload(self.uploader, "tests/fixtures/*.lua", 'sha1', True, False, False) return len(dests) def task_upload_verify(self): """Upload some text files and verify""" log.info('upload-verify') dests = operation_upload(self.uploader, "tests/fixtures/*_file.txt", 'sha1', False, False, False) return len(dests) def task_check_remote_files(self, wanted): """Check that the wanted number of files exists on the device""" lst = self.uploader.file_list() self.assertIsInstance(lst, type([])) self.assertEqual(len(lst), wanted) return lst def task_remove_all_files(self): """Remove all files on device""" log.info('remove all files') self.uploader.file_remove_all() def task_download_all_files(self, files): """Downloads all files on device and do a sha1 checksum""" log.info('download all files and verify. %s', files) dest = os.path.join('.', 'tmp') operation_download(self.uploader, files, dest=dest) for f in files: local = os.path.join(dest, f) self.assertTrue(os.path.isfile(local)) self.uploader.verify_file(local, f, 'sha1') def task_remove_tmp(self): """Removes local tmp folder""" dest = os.path.join('.', 'tmp') if os.path.isdir(dest): shutil.rmtree(dest) def test_for_long_time(self): """Run a sequence of steps a number of times""" testcount = 10 for x in range(testcount): print('test sequence {0}/{1}'.format(x+1, testcount)) log.info('--- test sequence {0}/{1} ---'.format(x+1, testcount)) self.task_remove_tmp() self.task_remove_all_files() self.task_check_remote_files(0) time.sleep(0.5) count = self.task_upload_verify_compile() self.assertEqual(count, 3) count += self.task_upload_verify() self.assertEqual(count, 6) files = self.task_check_remote_files(count) self.task_download_all_files(list(map(lambda x: x[0], files))) ================================================ FILE: tests/uploader.py ================================================ # -*- coding: utf-8 -*- # Copyright (C) 2015-2019 Peter Magnusson # pylint: disable=C0111,R0904 import unittest import time import os from nodemcu_uploader import Uploader # from serial import VERSION as serialversion # from distutils.version import LooseVersion LOOPPORT = 'loop://' # on which port should the tests be performed SERIALPORT = os.environ.get('SERIALPORT', LOOPPORT) def is_real(): if SERIALPORT.strip() == '': return False return str(SERIALPORT) != str(LOOPPORT) # @unittest.skipUnless(LooseVersion(serialversion) >= LooseVersion('3.0.0') , 'Needs pySerial >= 3.0.0') # class UploaderFakeTestCase(unittest.TestCase): # def test_init(self): # uploader = Uploader(SERIALPORT) # uploader.close() @unittest.skipUnless(is_real(), 'Needs a configured SERIALPORT') class UploaderTestCase(unittest.TestCase): uploader = None def setUp(self): self.uploader = Uploader(SERIALPORT) def tearDown(self): if is_real(): self.uploader.node_restart() self.uploader.close() time.sleep(1) def test_upload_and_verify_raw(self): self.uploader.prepare() self.uploader.write_file('tests/fixtures/big_file.txt', verify='raw') def test_upload_and_verify_sha1(self): self.uploader.prepare() self.uploader.write_file('tests/fixtures/big_file.txt', verify='sha1') def test_upload_strange_file(self): self.uploader.prepare() self.uploader.write_file('tests/fixtures/testuploadfail.txt', verify='raw') def test_file_list(self): lst = self.uploader.file_list() self.assertIsInstance(lst, type([])) self.assertGreaterEqual(len(lst), 1) self.assertLess(len(lst), 50) def test_node_heap(self): size = self.uploader.node_heap() self.assertGreater(size, 20000) self.assertLess(size, 60000) def test_node_info(self): result = self.uploader.node_info() self.assertNotIn("deprecated", result) ================================================ FILE: tox.ini ================================================ [tox] envlist = py36, py37, py38 [testenv] deps = -rtest_requirements.txt commands = python -m unittest -v tests.get_tests setenv = SERIALPORT= [flake8] include = nodemcu_uploader, tests # ignore = E501 max-line-length = 120